CONTACT: A Human-Grounded Benchmark and Surprisal-Based Predictive Scorer for Conversational Naturalness

CONTACT dataset subsets, human ratings and predictive scorer, from the paper.

P: participant · S: supervisor. H-H scores are individual, with video positions and disruption-instruction roles. Natural: both uninstructed; competitive floor conflict: both instructed. For unilateral H-H conditions, Combined averages uninstructed P and S; for natural conversations, P is both participants' mean. Competitive floor conflict uses both participants' mean with S for display only and is excluded from the role-resolved uninstructed-participant MOS sensitivity analysis. H-AI Combined is the mean of P and S. Scores are 1-5 and apply to the full recording.

Display videos: backgrounds are matted onto light gray using Robust Video Matting; chair and edge remnants may remain. Ratings and paper analyses use the original recordings. Faces and voices remain identifiable.

Awkward Turn-Taking

Natural Baseline

tt-019 · 4:11

Both participants uninstructed

Left · Speaker 005
Uninstructed
Right · Speaker 006
Uninstructed
RaterTurn Taking MOS
P - Left (uninstructed)5.00
P - Right (uninstructed)5.00
Supervisor5.00
Combined5.00
Prompt and event

One speaker asks for advice about a small everyday decision, such as whether to change a routine, accept an invitation, buy something, or bring up a minor issue. The partner should give advice naturally.

Target behavior. Both speakers should discuss the problem naturally. The advice seeker can add context and the partner can respond, but there should be no intentional blocking or interruption.

Both participants are uninstructed; individual scores are shown separately. Combined uses their mean and the supervisor rating. Ratings refer to the full recording.

Blocked Invited Response

tt-039 · 4:04

Instructed: left participant

Left · Speaker 009
Instructed
Right · Speaker 010
Uninstructed
RaterTurn Taking MOS
P - Left (instructed)1.00
P - Right (uninstructed)2.00
Supervisor2.00
Combined2.00
Prompt and event

Discuss how you would decide where to move if you had to move to a new city tomorrow. One speaker should invite the partner's thoughts during the reasoning process.

Target behavior. After inviting the partner's thoughts, the manipulated speaker should resume their own reasoning shortly after the partner begins responding.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Competitive Floor Conflict

tt-004 · 4:24

Both participants instructed

Left · Speaker 001
Instructed
Right · Speaker 002
Instructed
RaterTurn Taking MOS
P - Left (instructed)2.00
P - Right (instructed)2.00
Supervisor2.00
Combined2.00
Prompt and event

One speaker tells a story about a time when a plan did not go as expected. Explain what the original plan was, what went wrong, how people reacted, and how the situation ended.

Target behavior. The non-storytelling speaker should at some point begin speaking and continue with a meaningful contribution while the storyteller also continues. Both speakers should continue briefly before one yields or the conversation resets.

Both participants are instructed; individual scores are shown separately. Combined uses their mean and the supervisor rating. Ratings refer to the full recording.

Fragmented Turn Holding

tt-065 · 4:08

Instructed: left participant

Left · Speaker 017
Instructed
Right · Speaker 018
Uninstructed
RaterTurn Taking MOS
P - Left (instructed)2.00
P - Right (uninstructed)2.00
Supervisor2.00
Combined2.00
Prompt and event

Discuss whether trips should be carefully planned or spontaneous. Each speaker should explain their view and respond to the other person's reasons.

Target behavior. The manipulated speaker should defend their view using repeated fragmented pauses while continuing to hold the floor. The partner should respond naturally but may find it hard to know when to enter.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Premature Entry

tt-236 · 4:17

Instructed: right participant

Left · Speaker 079
Uninstructed
Right · Speaker 080
Instructed
RaterTurn Taking MOS
P - Left (uninstructed)3.00
P - Right (instructed)3.00
Supervisor3.00
Combined3.00
Prompt and event

One speaker explains how to troubleshoot something that is not working, such as a phone, computer, appliance, travel plan, or schedule problem. Walk through the steps and why each step would come next.

Target behavior. The manipulated listener should begin asking a question or responding slightly before the explainer has finished a step, roughly once every 1-2 turns when it feels natural.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Unmarked Turn Abandonment

tt-062 · 4:04

Instructed: left participant

Left · Speaker 015
Instructed
Right · Speaker 016
Uninstructed
RaterTurn Taking MOS
P - Left (instructed)2.00
P - Right (uninstructed)4.00
Supervisor2.00
Combined3.00
Prompt and event

Plan a weekend trip together. Decide where you would go, how long you would stay, what you would do, how much planning is needed, and what tradeoffs you would make if time or money were limited.

Target behavior. The manipulated speaker should stop during a proposal or decision point without finishing the idea and without clearly giving the floor to the partner. The partner should respond naturally to the unclear turn release.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Affective Mismatch

Natural Coordination

emo-359 · 4:07

Both participants uninstructed

Left · Speaker 107
Uninstructed
Right · Speaker 108
Uninstructed
RaterAffective Response MOS
P - Left (uninstructed)5.00
P - Right (uninstructed)4.00
Supervisor5.00
Combined4.75
Prompt and event

Talk about a situation where something you had been looking forward to was unexpectedly cancelled or could no longer happen. For example, a trip being cancelled, an event no longer happening, friends changing plans at the last minute, bad weather ruining an outdoor activity, or discovering that something you had anticipated was unavailable. Discuss why you were looking forward to it and how you would feel when the plan fell through. A hypothetical example is fine.

Target behavior. Both speakers should respond naturally. They do not need to exaggerate emotion; they should express the level of affect that would normally arise while discussing the topic.

Both participants are uninstructed; individual scores are shown separately. Combined uses their mean and the supervisor rating. Ratings refer to the full recording.

Valence Mismatch

emo-360 · 4:14

Instructed: left participant

Left · Speaker 107
Instructed
Right · Speaker 108
Uninstructed
RaterAffective Response MOS
P - Left (instructed)1.00
P - Right (uninstructed)4.00
Supervisor3.00
Combined3.50
Prompt and event

Talk about a hygiene or cleanliness situation that you would find particularly unpleasant or disgusting. For example, discovering bugs in a kitchen or bedroom, seeing residue on supposedly clean dishes, finding a shared kitchen covered in old food or oil, entering a very dirty bathroom, finding overflowing garbage, noticing spoiled food smell, or discovering that a place that should be clean has not been cleaned for a long time. Discuss what would bother you most and how you would react.

Target behavior. The topic speaker should discuss the assigned topic naturally. The manipulated responder should say things that are broadly appropriate, but with an emotional tone that clashes with the situation, such as cheerful toward something unpleasant or irritated toward something positive.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Under-Response

emo-361 · 4:19

Instructed: right participant

Left · Speaker 107
Uninstructed
Right · Speaker 108
Instructed
RaterAffective Response MOS
P - Left (uninstructed)2.00
P - Right (instructed)4.00
Supervisor3.00
Combined2.50
Prompt and event

Talk about a situation where you have to perform well in front of other people and do not know how it will go. For example, a job interview, an academic or professional interview, giving a presentation, speaking publicly, answering unexpected questions, or waiting before an evaluation. Discuss which part would make you most nervous. The situation can be hypothetical, and you should not name a real employer, school, interviewer, or identifying detail.

Target behavior. The topic speaker should discuss the assigned topic naturally. The manipulated responder should keep their words relevant and cooperative, but deliver them with unusually low emotional engagement: restrained, flat, unenthusiastic, or detached.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Disproportionate Emotional Overreaction

emo-362 · 4:03

Instructed: left participant

Left · Speaker 107
Instructed
Right · Speaker 108
Uninstructed
RaterAffective Response MOS
P - Left (instructed)2.00
P - Right (uninstructed)2.00
Supervisor4.00
Combined3.00
Prompt and event

Talk about an ordinary experience that unexpectedly turned out much better than you expected. For example, you might discuss trying a restaurant or food that was surprisingly good, visiting a place that was more enjoyable than expected, joining an activity you did not expect to like, attending an event that turned out to be fun, or receiving a small unexpected benefit or piece of good news. Discuss what happened, what surprised you, and why it made you feel happy.

Target behavior. The topic speaker should keep the situation ordinary and low-stakes. The manipulated responder should make reasonable conversational points, but react with noticeably excessive intensity for the situation.

Instructed and uninstructed participant ratings are reported separately. Combined uses the uninstructed participant and supervisor ratings. Ratings refer to the full recording.

Open-Ended Human-AI Conversations

Gemini 3.1 Flash Live Preview

hai-179 · 6:41
RatingPSCombined
Turn Taking5.003.004.00
Affective Response5.001.003.00
Overall5.003.004.00
Prompt and event

Share your experience being a fan of a sports team. What first drew you to them, what keeps you supporting them, and how you feel about them these days.

Participating human's rating. Ratings refer to the full recording.

Qwen3.5 Omni+

hai-181 · 5:53
RatingPSCombined
Turn Taking4.002.003.00
Affective Response5.003.004.00
Overall5.003.004.00
Prompt and event

Share a dating ‘rule’ you’ve noticed people assume (who texts first, who pays, how fast things move). Do you agree with it?

Participating human's rating. Ratings refer to the full recording.

Cascade V1

hai-182 · 5:05
RatingPSCombined
Turn Taking2.001.001.50
Affective Response3.001.002.00
Overall5.001.003.00
Prompt and event

Think about a technology you believe will significantly change everyday life in the future. What excites you about it, what worries you, and how confident you feel about those expectations.

Participating human's rating. Ratings refer to the full recording.

Gemini 3.1 Flash Live Preview

hai-185 · 4:33
RatingPSCombined
Turn Taking5.001.003.00
Affective Response5.004.004.50
Overall5.003.004.00
Prompt and event

Share an awkward or embarrassing moment. What happened, and how you look back on it now. Do you regret it?

Participating human's rating. Ratings refer to the full recording.

Qwen3.5 Omni+

hai-188 · 5:18
RatingPSCombined
Turn Taking4.002.003.00
Affective Response4.004.004.00
Overall4.002.003.00
Prompt and event

Share a dating ‘rule’ you’ve noticed people assume (who texts first, who pays, how fast things move). Do you agree with it?

Participating human's rating. Ratings refer to the full recording.

Cascade V2

hai-189 · 5:44
RatingPSCombined
Turn Taking2.003.002.50
Affective Response3.004.003.50
Overall2.003.002.50
Prompt and event

Think about a technology you believe will significantly change everyday life in the future. What excites you about it, what worries you, and how confident you feel about those expectations.

Participating human's rating. Ratings refer to the full recording.

Moshi

hai-190 · 5:01
RatingPSCombined
Turn Taking1.001.001.00
Affective Response1.001.001.00
Overall1.001.001.00
Prompt and event

Share what a normal day looks like for you. What parts feel natural, and which parts feel a bit forced.

Participating human's rating. Ratings refer to the full recording.

Gemini 3.1 Flash Live Preview

hai-192 · 5:13
RatingPSCombined
Turn Taking4.004.004.00
Affective Response3.003.003.00
Overall4.002.003.00
Prompt and event

Share an awkward or embarrassing moment. What happened, and how you look back on it now. Do you regret it?

Participating human's rating. Ratings refer to the full recording.