REVIEW 3 major objections 5 minor 36 references
Multimodal LLMs now match an aggregated human crowd at telling whether two people are familiar or strangers from 20-second clips, but only in accuracy: they lean toward 'stranger' and under-exploit visible behavior.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 01:42 UTC pith:GKTHOTQS
load-bearing objection Solid, carefully built benchmark; the 'statistically indistinguishable' parity claim is overreach from non-significance and post-hoc model selection, but the class-bias and modality findings are the real contribution. the 3 major comments →
FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that on this thin-slice familiarity task, current multimodal large language models have reached accuracy parity with an aggregated human crowd, but they solve the task differently. Across all three modalities, the best model's accuracy is statistically indistinguishable from the crowd's, yet the model reaches that level by over-answering 'stranger': per-class recall is highly skewed (e.g., 91.7% of strangers correct but only 20.8% of familiar pairs in text), whereas humans keep both answers nearly equal. Signal-detection statistics separate the two: discrimination (d') is comparable in video, but the criterion (c) leans strongly toward 'stranger' for the best models whil
What carries the argument
The machinery is the benchmark itself plus a decision-theoretic decomposition. FriendBench is built on a public corpus of dyadic interactions, restricted to a single 'Either-Or' ice-breaker task from three recording sites, with 96 dyads fully crossed by site, relationship status, and gender composition, each captured in a 20-second clip and presented in text-only, audio-only, and audiovisual conditions; the video condition includes the audio track, so it measures the added value of visible behavior on top of speech. On top of this, the analysis uses signal detection theory — d' to measure how well a rater discriminates familiar from stranger dyads, and the criterion c to measure response bia
Load-bearing premise
The whole comparison rests on the source dataset's familiarity labels being correct ground truth for the 96 dyads; if any label is wrong, or any acted or assigned-relationship interaction slipped past the exclusion of improvised sessions, then the human and model accuracy numbers, and the reported response-bias pattern, inherit the error.
What would settle it
Run the FriendBench evaluation again after independently verifying the relationship labels of the 96 dyads (e.g., by checking the dataset's recruitment records or re-contact protocol). If a meaningful share of labels is wrong, the accuracy and bias results are called into question. A second, cheaper test: re-prompt the top models with instructions that explicitly state that half of the pairs are strangers, and check whether the 'stranger' lean disappears — if it does, the bias is a prompt artifact; if it persists, the effective-prior interpretation is supported. Both results are observable fro
If this is right
- If the accuracy parity holds, a single overall accuracy figure is insufficient for evaluating social-perception models; per-class recall and signal-detection statistics are needed to see how a model reaches its score.
- Because the models' 'stranger' lean is a criterion shift rather than a discrimination deficit, recentering a model to the known base rate would raise its accuracy with no gain in d' — a correctable bias rather than a hard ceiling.
- Dyad difficulty is shared: humans and models tend to find the same conversations hard once class bias is partialled out, so what makes a dyad legible is at least partly a property of the interaction itself.
- The flat audio-to-video trajectory of the strongest models, against a reliable human gain, identifies visible behavior as the main untapped channel for multimodal social perception.
- For deployed agents that read social situations, the practical risk is concrete: they capture the vocal signal but miss the visible cues human observers rely on.
Where Pith is reading between the lines
- The paper's balanced 50/50 design means the models' 'stranger' bias is penalized by construction; a deployment-oriented evaluation with a realistic base rate could show either that the bias is harmless or that it is a genuine miscalibration — the paper notes this but does not test it.
- The shared-difficulty result hints at a common substrate, such as turn-timing or prosodic coordination; a direct test would feed models acoustic-prosodic or motion-capture features in isolation and see whether the within-class difficulty correlation with humans persists.
- Since the bias is idiosyncratic in direction across models, it suggests the lean reflects each model's instruction-following or verbalization habits rather than a universal prior; a prompt that explicitly states the base rate would test whether the bias is a framing artifact.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FriendBench, a benchmark for inferring whether a dyad of two people are familiar or strangers from a 20-second clip of an ice-breaker conversation, evaluated separately in text, audio, and audiovisual modalities. The benchmark is constructed from 96 participant-disjoint dyads drawn from the Seamless Interaction dataset, with a stratified design crossing site, relationship, and gender composition. The authors collect matched human ratings from three panels of roughly 90 raters each using a planned-missing incomplete-block design, and evaluate 26 multimodal LLMs zero-shot with the same task prompt. The central reported findings are that the best model and the human crowd are statistically indistinguishable on accuracy in every modality (McNemar p > .4), that humans keep a balanced response criterion while the strongest models lean toward 'stranger', and that richer channels help both humans and models, but only humans gain further accuracy from visible behavior on top of speech. The paper also analyzes rater individual differences and dyad difficulty, finding partially shared difficulty structure across humans and models. Stimuli, human ratings, and model predictions are released.
Significance. The benchmark construction is a genuine strength: balanced stratified sampling, participant-disjoint dyads, matched task wording for humans and models, and a planned-missing design that supports crossed random-effects modeling. The separation of discrimination (d') from response criterion (c) is a valuable analytic lens for multimodal social-perception benchmarks, and the per-dyad difficulty analysis is an informative step toward understanding what makes a pair legible. The release of stimuli, ratings, and predictions is a strong contribution to reproducibility. If the central parity claim survives proper equivalence and selection-aware inference, this would be a solid resource for the field. However, as it stands the headline statistical claim is not adequately supported, and the modality-specific claim about models under-exploiting visual cues is overstated relative to the reported within-model comparisons.
major comments (3)
- [§4, Table 1, and Discussion] The claim that the best model and the human crowd are 'statistically indistinguishable' on accuracy in every modality is an equivalence claim, but the only support is the statement that 'an exact McNemar test never approaches significance (p > .4 in every modality)'. Failing to reject the null of no difference is not evidence of absence. With 96 dyads, the power to detect a 5-point difference is limited, and the observed gaps (text 5.2, audio 3.2, video 5.2 points) are all within the reported Wilson CIs for the model accuracies. The paper should provide a prespecified equivalence margin, a TOST or a confidence interval for the accuracy difference, and a statement of the minimum detectable effect at N=96. Without this, the headline framing overstates the evidence.
- [§4, Table 1] The 'best model' in each modality is selected post hoc as the top-accuracy system among 26 models (or a subset) using the same 96 dyads on which the human–model comparison is then performed. This selection invalidates the nominal p-value of the McNemar test: the probability that the maximum of 26 sampled accuracies lands close to the human crowd is not the nominal p. Provide a selection-aware analysis (e.g., a permutation test comparing the observed best-model-minus-crowd gap against the distribution of such gaps over all models, or a holdout-based selection), or report all 26 comparisons with multiple-comparison control and interpret the 'best model' descriptively. This issue is load-bearing because the parity claim is the paper's headline contribution.
- [§5.3, Abstract, Discussion] The claim that 'only humans gain from visible behavior on top of speech' and that 'the strongest models are flat from audio to audiovisual' conflates two different systems: the top audio model is gemini_audio_pro (66.7%) and the top video model is gemini_video (66.7%), which are not the same model instance. The within-model trajectories, which the paper itself reports, are mixed: Gemini Flash improves 63.5→66.7, Gemini Pro declines 66.7→57.3, and Muse Spark declines 57.3→54.2. This does not establish a reliable 'flatness' for strong models. Either restrict the claim to the specific systems that are flat or perform a repeated-measures comparison over the matched model roster. As written, the abstract's 'only humans gain from visible behavior' is not supported by the presented evidence.
minor comments (5)
- [Abstract, §6] The phrase 'statistically indistinguishable' appears without qualification. Since the paper only reports null-hypothesis tests, the phrase should be either replaced by 'no significant difference was detected' or supported by the equivalence analysis requested in the major comments.
- [Appendix A, Table A1] The text says thresholds are listed in Table A1, but the table lists criteria without numeric thresholds (e.g., F2: '≥4 merged conversational turns' appears in the text, not in the table). Please move the actual threshold values into the table.
- [§3.1, Limitations] The relationship labels from the Seamless Interaction dataset are taken as ground truth. The LLM-based audit checks the prompt-content match, but not the familiar/stranger labels themselves. A short statement acknowledging this reliance, or a manual check of a subset of dyads, would strengthen the benchmark's validity.
- [§5.1, Table 1] The difference in criterion c between the human crowd and the best model is reported without uncertainty or a formal test. A bootstrap confidence interval on the difference, or a simple test, would support the claim of a difference in effective prior.
- [§4, Table 2] The identification of three models as 'above chance' is based on uncorrected exact binomial tests among 26 models. Under a conservative Bonferroni correction, none remain significant. Please add a caveat or report adjusted p-values.
Circularity Check
No circularity: this is an empirical measurement paper with no derivation whose conclusions are defined in terms of fitted parameters or self-cited theorems.
full rationale
FriendBench is an empirical benchmark paper, not a derivation. No headline number is defined in terms of a fitted parameter: model responses are zero-shot, human ratings are measured, and accuracy, d′, and c are direct transformations of observed responses. There is no fitted-input-called-prediction step, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The only author-overlapping citation is the Seamless Interaction dataset (Agrawal et al., 2025), which supplies the stimuli and the familiar/stranger labels. That is data provenance rather than a derived result: the labels are defined by recruitment (whether the dyad had met before), not inferred by the paper's own model or fitted to the paper's own outcome. The paper does not attempt to validate those labels, and the human/model comparisons are independent measurements against that fixed ground truth. The paper's self-cited dataset is load-bearing in the sense that all results depend on label quality, but this is a validity limitation, not circularity: the cited result is externally defined and not equivalent to the paper's predictions. The statistical concern that McNemar non-significance is not equivalence is a correctness risk, not a circularity risk, and would not change the circularity score under the review's definitions.
Axiom & Free-Parameter Ledger
free parameters (2)
- Benchmark quality-filter thresholds (F1–F5) =
Table A1 values (e.g., F2≥4 merged turns, F3≥3s speech per speaker, F4<35% aligned turn starts, F5≥90s interaction speec
- Clip duration and boundary snap window =
20 seconds with ±5s snap to nearest turn start
axioms (4)
- domain assumption The familiar/stranger labels in the Seamless Interaction dataset are accurate ground truth.
- domain assumption The Either-Or ice-breaker prompt suppresses semantic content equally across familiar and stranger dyads.
- domain assumption Crowdsourced raters made genuine, independent familiarity judgments under the instruction screen.
- standard math The log-linear correction and signal-detection statistics are valid for extreme per-class proportions.
read the original abstract
Reading a social situation often depends on behavior, not words alone. We introduce FriendBench, a benchmark for inferring whether two people are already familiar or are meeting as strangers, from a 20-second clip of a dyadic ice-breaker conversation. Every pair answers the same type of prompt, so only the manner of interaction can reveal the answer. Across text, audio, and video, we compare 26 models from seven companies against matched human panels over 96 balanced dyads. The best model and the human crowd are statistically indistinguishable on accuracy in every modality, but reach it differently: humans stay balanced across the two answers, while the strongest models lean toward "stranger"---a difference in effective prior, not discrimination. Richer channels help both unequally, and only humans gain from visible behavior on top of speech. We release the stimuli, human ratings, and model predictions.
Figures
Reference graph
Works this paper leans on
-
[1]
Agrawal, Vasu and Akinyemi, Akinniyi and Alvero, Kathryn and Behrooz, Morteza and Buffalini, Julia and Carlucci, Fabio Maria and Chen, Joy and Chen, Junming and Chen, Zhang and Cheng, Shiyang and Chowdary, Praveen and Chuang, Joe and D'Avirro, Antony and Daly, Jon and Dong, Ning and Duppenthaler, Mark and Gao, Cynthia and Girard, Jeff and Gleize, Martin a...
-
[2]
Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences:
Ambady, Nalini and Rosenthal, Robert , year = 1992, journal =. Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences:. doi:10.1037/0033-2909.111.2.256 , abstract =
-
[3]
Psychological Methods , volume =
Planned Missing Data Designs in Psychological Research , author =. Psychological Methods , volume =. doi:10.1037/1082-989x.11.4.323 , abstract =
-
[4]
Psychological Inquiry , volume =
The Nature of Rapport and Its Nonverbal Correlates , author =. Psychological Inquiry , volume =. doi:10.1207/s15327965pli0104_1 , abstract =
-
[5]
, year = 2017, pages =
Li, Junnan and Wong, Yongkang and Zhao, Qi and Kankanhalli, Mohan S. , year = 2017, pages =. Dual-. Proceedings of the
2017
-
[6]
Zadeh, Amir and Chan, Michael and Liang, Paul Pu and Tong, Edmund and Morency, Louis-Philippe , year = 2019, pages =. Social-. Proceedings of the
2019
-
[7]
Sun, Qianru and Schiele, Bernt and Fritz, Mario , year = 2017, pages =. A. Proceedings of the
2017
-
[8]
, year = 1995, month = mar, journal =
Hautus, Michael J. , year = 1995, month = mar, journal =. Corrections for Extreme Proportions and Their Biasing Effects on Estimated Values of d. doi:10.3758/BF03203619 , urldate =
-
[9]
Capretto, Tom. Bambi:. Journal of Statistical Software , volume =. doi:10.18637/jss.v103.i15 , urldate =
-
[10]
Bayesian Data Analysis , author =
-
[11]
Indices of Effect Existence and Significance in the
Makowski, Dominique and. Indices of Effect Existence and Significance in the. Frontiers in Psychology , volume =. doi:10.3389/fpsyg.2019.02767 , urldate =
arXiv 2019
-
[12]
Behavior Research Methods, Instruments, & Computers , volume =
Calculation of Signal Detection Theory Measures , author =. Behavior Research Methods, Instruments, & Computers , volume =. doi:10.3758/BF03207704 , urldate =
-
[13]
Handbook of Inter-Rater Reliability:
Gwet, Kilem L , year = 2021, edition =. Handbook of Inter-Rater Reliability:
2021
-
[14]
Silvera, David and Martinussen, Monica and Dahl, Tove I. , year = 2001, journal =. The. doi:10.1111/1467-9450.00242 , urldate =
arXiv 2001
-
[15]
Can Social Intelligence Be Measured?
Grieve, Rachel and Mahar, Doug , year = 2013, month = mar, journal =. Can Social Intelligence Be Measured?. doi:10.1080/03033910.2012.737758 , urldate =
arXiv 2013
-
[16]
Probabilistic
Rasch, Georg , year = 1960, publisher =. Probabilistic
1960
-
[17]
Nature Reviews Neuroscience , volume =
Cognitive Neuroscience of Human Social Behaviour , author =. Nature Reviews Neuroscience , volume =. doi:10.1038/nrn1056 , urldate =
-
[18]
and Frith, Uta , year = 2007, month = aug, journal =
Frith, Chris D. and Frith, Uta , year = 2007, month = aug, journal =. Social. doi:10.1016/j.cub.2007.05.068 , urldate =
-
[19]
Mathur, Leena and Liang, Paul Pu and Morency, Louis-Philippe , editor =. Advancing. Proceedings of the 2024. doi:10.18653/v1/2024.emnlp-main.1143 , urldate =
-
[20]
Sap, Maarten and Rashkin, Hannah and Chen, Derek and Le Bras, Ronan and Choi, Yejin , editor =. Social. Proceedings of the 2019. doi:10.18653/v1/D19-1454 , urldate =
-
[21]
Behavioral and Brain Sciences , volume =
Does the Chimpanzee Have a Theory of Mind? , author =. Behavioral and Brain Sciences , volume =. doi:10.1017/S0140525X00076512 , urldate =
-
[22]
Royal Society Open Science , volume =
Recognizing Affiliation in Colaughter and Cospeech , author =. Royal Society Open Science , volume =. doi:10.1098/rsos.201092 , urldate =
-
[23]
Cafaro, Angelo and Wagner, Johannes and Baur, Tobias and Dermouche, Soumia and Torres Torres, Mercedes and Pelachaud, Catherine and Andr. The. Proceedings of the 19th. doi:10.1145/3136755.3136780 , urldate =
-
[24]
Dunbar, R. I. M. and Robledo, Juan-Pablo and Tamarit, Ignacio and Cross, Ian and Smith, Emma , year = 2022, month = mar, journal =. Nonverbal. doi:10.1007/s10919-021-00386-y , urldate =
-
[25]
, year = 2021, month = may, journal =
Jia, Qi and Huang, Hongru and Zhu, Kenny Q. , year = 2021, month = may, journal =. doi:10.1609/aaai.v35i14.17551 , urldate =
-
[26]
Katerenchuk, Denys and Brizan, David Guy and Rosenberg, Andrew , year = 2014, pages =. ``Was That Your Mother on the Phone?'': Classifying Interpersonal Relationships between Dialog Participants with Lexical and Acoustic Properties , shorttitle =. Proc. doi:10.21437/Interspeech.2014-416 , urldate =
-
[27]
Kim, Eunsu and Park, Junyeong and Oh, Juhyun and Park, Kiwoong and Song, Seyoung and Do. Are They Lovers or Friends?. Proceedings of the 64th. doi:10.18653/v1/2026.acl-long.1074 , urldate =
-
[28]
Kong, Fanqi and Zu, Weiqin and Chen, Xinyu and Yang, Yaodong and Zhu, Song-Chun and Feng, Xue , editor =. Findings of the. doi:10.18653/v1/2026.findings-acl.1863 , urldate =
-
[29]
Latif, Nida and Barbosa, Adriano V. and. Movement. PLOS ONE , volume =. doi:10.1371/journal.pone.0105036 , urldate =
-
[30]
Palmero, Cristina and Selva, Javier and Smeureanu, Sorina and Junior, Julio C. S. Jacques and Clapes, Albert and Mosegui, Alexa and Zhang, Zejian and Gallardo, David and Guilera, Georgina and Leiva, David and Escalera, Sergio , year = 2021, pages =. Context-. Proceedings of the
2021
-
[31]
doi:10.1609/aaai.v40i30.39685 , urldate =
Qin, Zheng and Zheng, Ruobing and Wang, Yabing and Li, Tianqi and Yuan, Yi and Chen, Jingdong and Wang, Le , year = 2026, month = mar, journal =. doi:10.1609/aaai.v40i30.39685 , urldate =
-
[32]
doi:10.1109/TCSVT.2025.3616347 , urldate =
Tang, Wang and Dogan, Fethiye Irmak and Qing, Linbo and Gunes, Hatice , year = 2026, month = mar, journal =. doi:10.1109/TCSVT.2025.3616347 , urldate =
arXiv 2026
-
[33]
Philosophical Transactions of the Royal Society B: Biological Sciences , volume =
Long Gaps between Turns Are Awkward for Strangers but Not for Friends , author =. Philosophical Transactions of the Royal Society B: Biological Sciences , volume =. doi:10.1098/rstb.2021.0471 , urldate =
arXiv 2021
-
[34]
Zhang, Shuxiang and Yin, Yiting and Song, Wenxuan and Wu, Yuhang and Liu, Miao , year = 2026, month = jun, number =. doi:10.48550/arXiv.2606.23092 , urldate =. arXiv , keywords =:2606.23092 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.23092 2026
-
[35]
, year = 1998, month = oct, journal =
Dietterich, Thomas G. , year = 1998, month = oct, journal =. Approximate. doi:10.1162/089976698300017197 , urldate =
-
[36]
McNemar, Quinn , year = 1947, month = jun, journal =. Note on the. doi:10.1007/BF02295996 , urldate =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.