Pith. sign in

REVIEW 4 major objections 5 minor 91 references

Multimodal language models still fail to detect hidden dissent in multi-party meetings, according to the new MeetingToM benchmark.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:59 UTC pith:Q53MZFOL

load-bearing objection A solid, well-scoped benchmark for meeting-grounded ToM with a real third-party annotation caveat; worth peer review, but interpret the human ceiling as a measure of annotation convention, not ground truth. the 4 major comments →

arxiv 2607.19235 v1 pith:Q53MZFOL submitted 2026-07-21 cs.CL cs.CV

MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

Ziyi Wang , Yuhang Wu , Dongxu Piao , Xingyu Liu , Tianhui Zhou , Miao Liu This is my paper
classification cs.CL cs.CV
keywords theory of mindmultimodal language modelsmeeting understandingpseudo-consensussocial reasoningmental state predictionmultimodal fusiongroup dynamics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MeetingToM is a new benchmark that asks whether multimodal language models can do theory of mind in realistic multi-party meetings, where most social signals are implicit and distributed across speech and body language. It separates social reasoning into three levels: reading one participant's mental state, resolving who a speaker means by 'you' and that person's attitude, and judging whether group consensus is genuine or merely apparent. Its central construct is pseudo-consensus, a situation where verbal agreement masks private dissent under social pressure, a common workplace phenomenon that existing benchmarks mostly ignore. In evaluations, humans score near 80% overall, while the best models reach roughly 50–60% on subject- and group-level tasks and drop to the 20–30% range once class imbalance is accounted for. The authors conclude that current models lean on transcripts, miss subtle nonverbal dissent, and do not reliably tell true consensus from pseudo-consensus.

Core claim

The central discovery is the depth of the gap. MeetingToM transposes theory-of-mind tests from stories and dyads to naturalistic four-person meetings, and formalizes pseudo-consensus, a group that looks unanimous while a participant's verbal 'yes' contradicts visible tension, gaze aversion, or delayed nods. On this benchmark humans score about 80% overall; the strongest evaluated models reach 55–60% on subject-level mental-state prediction, about 51–52% on attitude inference, and 40–43% on consensus classification, and their macro scores collapse to the 22–30% range on imbalanced tasks. Modality ablations show that transcript-only input often beats video-plus-transcript, while visual input h

What carries the argument

The engine of the paper is the benchmark structure itself: three hierarchically ordered tasks built from a public corpus of four-person design meetings, with 1,800 clips and 600 instances per task. The tasks progress from mental-state prediction from 5-second close-ups, to 'you'-referent and attitude inference from roughly 50-second corner and mosaic views, to group consensus classification with dissenter identification. The load-bearing construct is pseudo-consensus, which forces a model to weigh verbal claims against observable nonverbal leakage and to decide when agreement is real. A distinct mechanism is the annotation pipeline: automatically generated questions, two independent human an

Load-bearing premise

The load-bearing premise is that an annotator watching a short clip can reliably infer what a participant privately believed or intended; the paper concedes it could not ask the participants themselves, so gold labels are third-person judgments that may differ from the participants' true private states.

What would settle it

Re-run the human and model evaluations on a subset of clips where all annotators fully agree; if the human–model gap largely disappears on that unanimous subset, the benchmark's difficulty is substantially driven by label ambiguity. A stronger test is to collect self-reports from the original meeting participants and compare their stated private attitudes to the annotator gold labels.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Meeting-grounded social reasoning will not be solved by stronger video encoders alone; models must learn to weigh verbal and non-verbal evidence against each other, since the two often conflict.
  • High overall accuracy overstates capability on socially diverse tasks; macro-averaged scores should be the headline number for mental-state and consensus prediction.
  • Multimodal fusion can hurt performance: adding video to transcript data was not consistently better than transcript alone, so future systems need task-appropriate fusion rather than naive concatenation.
  • Thinking prompts such as chain-of-thought and emotional chain-of-thought are not a cure: their gains are task- and model-dependent, and they mainly help group-level consensus reasoning.
  • Structured context, such as participant role, meeting phase, or dialogue act, is not automatically beneficial and can reduce performance when it conflicts with local evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Reader inference: because annotator agreement is only moderate on attitude and consensus tasks, some of the human–model gap is probably annotation ambiguity; a benchmark variant restricted to unanimous labels would likely produce smaller, cleaner gaps.
  • Reader inference: the conclusion that transcript-only input wins suggests a design for the next testbed, clip pairs where verbal and nonverbal cues point to different answers, to isolate when models should trust vision over text.
  • Reader inference: the pseudo-consensus construct should transfer to remote and hybrid meetings, where dissent leaks through camera status, response latency, and chat objections; a platform-mediated extension would test whether the same reasoning deficit appears when cues are digital rather than physical.
  • Reader inference: if private mental states cannot be self-reported, then any claim that a model 'gets' consensus should be tested against behavior downstream, for instance whether an assistant that identifies pseudo-consensus helps users realize the group never truly committed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MeetingToM, a multimodal benchmark built from AMI Meeting Corpus clips, targeting theory-of-mind reasoning in multi-party meetings. It defines three hierarchical task families: subject-level mental-state prediction, dyadic addressee identification and attitude inference, and group-level consensus classification with hidden-dissenter identification, including the novel construct of pseudo-consensus. The authors evaluate several proprietary and open-source MLLMs, report that models substantially underperform human annotators (human overall 79.67% vs. best models around 50–60% on most tasks), and present modality, prompting, and contextual-prior ablations. They conclude that current MLLMs struggle to integrate non-verbal cues, infer hidden attitudes, and distinguish genuine from pseudo-consensus.

Significance. If the benchmark is validated, MeetingToM fills a real gap: most multimodal ToM benchmarks target overt, externally verifiable states, whereas meeting dynamics require latent mental-state and group-level reasoning. The hierarchical design, use of the public AMI corpus, explicit leakage controls, and the detailed annotation guidelines are strengths. The introduction of pseudo-consensus as an evaluable construct is a useful contribution, and the paper is transparent about its limitations, including the third-person nature of mental-state labels. However, the current evidence does not yet fully support the central human–model gap claim: the human evaluation protocol is not described, model results lack statistical grounding, and the moderate annotation reliability on the key latent constructs needs more rigorous handling.

major comments (4)
  1. [Section 3.3 and Table 1] The human evaluation is load-bearing for the central claim, yet no protocol is reported: how many raters, were they the same annotators who created the gold labels, what was their training, how many instances per task, and how was the overall 79.67% computed? If the 'human' row is produced by the annotators who authored the gold labels, the comparison may overestimate human ability relative to a fresh rater population. This needs to be described and, ideally, supplemented with an independently recruited human evaluation.
  2. [Section 4.1 and Tables 1–3] All model results are reported as point estimates without variance, confidence intervals, or significance tests. Many headline comparisons are small: e.g., Gemini-3 Pro vs. GPT-5 on Task 2.1 (74.33 vs. 54.33) is large, but prompt gains such as Task-CoT on Task 1 for Gemini-3 Pro (59.67 vs. 59.00) and several Table 3 differences are within a few points. The paper should report multiple runs with mean/std, or at least a significance test, to support the cross-model and cross-prompt conclusions.
  3. [Section 3.3, Section 6, Section 7] The gold labels for attitude (Task 2.2, κ=0.5014), consensus (Task 3.1, κ=0.5681), and dissenter identification (Task 3.2, κ=0.5729) rest on moderate third-person agreement. The paper correctly states in Sections 6 and 7 that the labels are systematic third-person inferences, not ground truth about private states. Given this, the benchmark should provide additional evidence that the human–model gap is not an artifact of annotation ambiguity. Concretely, report model and human performance separately on instances with unanimous vs. adjudicated labels, or show that the gap persists when only high-agreement items are used. This would strengthen the claim that the gap reflects ToM reasoning rather than label noise.
  4. [Section 4.1 and Table 1, Task 3.2] Task 3.2 is evaluated conditionally on correctly answering Task 3.1, but the effective sample size per model is not reported. Because most models have low Task 3.1 accuracy, the Task 3.2 accuracy is computed over a small, model-dependent subset. Reporting only the conditional accuracy can be misleading; include the number of instances and, preferably, an unconditional accuracy (e.g., treating wrong Task 3.1 as a failure for Task 3.2).
minor comments (5)
  1. [Section 4.1 / Table 1] The metric named 'Macro' in Table 1 is described as 'Macro-F1' in the text. Use consistent terminology.
  2. [Table 1] The chance baseline row does not include a value for Task 3.2; the conditional chance level should be defined and reported, since the label space has 5 options but the evaluation is conditional.
  3. [Section 4.2 / Table 3] The sentence about Qwen3-VL-32B Task-CoT says it 'reduces Task 2.1 and group-level performance,' but Task 2.2 actually improves by 1.00 point. Clarify to avoid a minor inconsistency.
  4. [Section 3.3] Clarify how the 200-instance agreement subset was selected from the 600 instances per task family, and whether the reported kappa values include all three annotators before adjudication, as implied.
  5. [General] The paper could more clearly state the number of participants in the human annotation and human evaluation, and whether the 'human' result is an average over the same annotators or a separate sample.

Circularity Check

0 steps flagged

No derivation-level circularity; benchmark labels are human-annotated and model evaluations are independent of the construction inputs.

full rationale

MeetingToM is a benchmark-construction and evaluation paper, not a derivation of predicted quantities from fitted parameters. The gold labels are produced by a human annotation pipeline (two annotators plus adjudication) over the AMI Meeting Corpus, and the model scores are exact-match accuracies against those labels; no equation in the paper reduces its conclusions to its inputs. The nearest candidate for circularity is the fact that human annotators also form the human ceiling in Table 1, but this is standard benchmark calibration rather than a tautological result: the human row measures reproducibility of the annotation scheme, and the model–human gap is still an independent empirical measurement. The paper's own limitation (Section 6) concedes that labels for private mental states cannot be validated against participant self-reports, and Section 7 explicitly calls them 'systematic third-person inferences'; this is a validity/ambiguity concern, not a derivation-level circularity, and the prompt instructs to place such concerns under correctness risk rather than circularity. The only self-citation with author overlap is reference [51], cited once in Appendix D as a caution that standard benchmarks may capture limited ToM-like behavior; it is not load-bearing for the benchmark's design, its task definitions, or its headline results. I find no fitted-input-called-prediction, no uniqueness-theorem smuggling, no ansatz-by-citation, and no renaming of a known result presented as a derivation.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on the validity of third-person annotation of hidden mental states, on the choice of clip windows and sampling rates, and on the adequacy of the taxonomy. No fitted model parameters are used; the listed numbers are design choices that affect measured performance.

free parameters (3)
  • Task 1 clip duration = 5 seconds
    Hand-picked window to isolate a single participant's mental state; affects the difficulty and type of cues available.
  • Tasks 2/3 clip duration = 50–70 seconds
    Chosen manually to provide enough context for referential and group reasoning; changes the input length and may influence performance.
  • GPT-5 frame sampling rate = 0.5 FPS
    GPT-5 does not ingest video directly; uniformly sampled frames at 0.5 FPS are used, potentially disadvantaging it on visual-cue-heavy tasks.
axioms (4)
  • domain assumption AMI Meeting Corpus provides naturalistic multi-party meeting interactions with synchronized audio-video and transcripts.
    The benchmark is built entirely from AMI; if AMI scenarios are not representative of real meetings, the benchmark's ecological validity is limited (Appendix E).
  • domain assumption Third-person annotator judgments of latent mental states are reliable enough to serve as gold labels.
    The paper itself concedes labels may differ from participants' true private states (Section 6); inter-annotator kappa is moderate for attitude and consensus tasks, so this is a load-bearing assumption.
  • domain assumption The seven mental-state and four consensus classes cover the meeting-ToM phenomena of interest.
    The taxonomy is introduced in Section 3.1 and Appendix A; systematic gaps in the label set would make the benchmark unable to measure important aspects of meeting ToM.
  • domain assumption Multiple-choice VQA on short clips is a valid paradigm for eliciting Theory-of-Mind reasoning.
    The evaluation protocol treats model choices as ToM measures; prior work (Le et al., 2019) warns that QA datasets can be gamed by surface regularities, a risk not fully analyzed here.
invented entities (1)
  • Pseudo-consensus (benchmark construct) no independent evidence
    purpose: Defines the group-level task: apparent agreement masking private dissent under social pressure.
    It is a named construct introduced in Section 1 and Figure 1, operationalized via annotator-identified 'dissent leakage' cues; it is not independently measurable from participant self-report and depends on the annotation protocol.

pith-pipeline@v1.3.0-alltime-deepseek · 24838 in / 10616 out tokens · 112406 ms · 2026-08-01T12:59:33.740437+00:00 · methodology

0 comments
read the original abstract

Theory of Mind (ToM), the ability to infer other's beliefs, intentions, and states of knowledge, is central to social interaction, yet remains challenging for current Multimodal Large Language Models (MLLMs), especially in multi-party meetings where cues are distributed across speech and behavior. Existing multimodal ToM benchmarks mainly focus on video-grounded question answering over overt, externally verifiable signals, and provide limited coverage of latent social states and group dynamics. We introduce MeetingToM, a benchmark for complex social behavior reasoning in naturalistic multi-party meetings. MeetingToM targets meeting-specific phenomena such as \textbf{pseudo-consensus}, where apparent agreement masks private dissent under social pressure. The benchmark is hierarchically organized to evaluate ToM at increasing levels of social granularity, including (i) subject-level mental state prediction, (ii) dyadic-level addressee understanding, and (iii) group-level consensus reasoning. We provide a unified evaluation protocol and conduct systematic analyses of representative MLLMs, revealing persistent limitations in integrating non-verbal cues, inferring hidden attitudes, and distinguishing genuine consensus from pseudo-consensus. Our results highlight key challenges and establish MeetingToM as a testbed for advancing meeting-grounded ToM in multimodal models.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

91 extracted references · 10 canonical work pages

  1. [1]

    Allen and Nale Lehmann-Willenbrock

    Joseph A. Allen and Nale Lehmann-Willenbrock. The key features of workplace meetings: Conceptualizing the why, how, and what of meetings at work.Organizational Psychology Review, 13(4):355–378, 2023. doi: 10.1177/20413866221129231

    work page doi:10.1177/20413866221129231 2023

  2. [2]

    Allen, Nale Lehmann-Willenbrock, and Steven G

    Joseph A. Allen, Nale Lehmann-Willenbrock, and Steven G. Rogelberg, editors.The Cambridge Handbook of Meeting Science. Cambridge University Press, Cambridge, 2015. doi: 10.1017/ CBO9781107589735. URLhttps://doi.org/10.1017/CBO9781107589735

    work page doi:10.1017/cbo9781107589735 2015

  3. [3]

    Abdullah Almaatouq, Alejandro Noriega-Campero, Abdulrahman Alotaibi, P. M. Krafft, Mehdi Moussaid, and Alex Pentland. Adaptive social networks promote the wisdom of crowds. Proceedings of the National Academy of Sciences of the United States of America, 117(21): 11379–11386, 2020. doi: 10.1073/pnas.1917687117. URLhttps://www.pnas.org/doi/full/ 10.1073/pna...

    work page doi:10.1073/pnas.1917687117 2020

  4. [4]

    theory of mind

    Ian A. Apperly. What is “theory of mind”? concepts, cognitive processes and individual differences.The Quarterly Journal of Experimental Psychology, 65(5):825–839, 2012. doi: 10.1080/17470218.2012.676055. URLhttps://doi.org/10.1080/17470218.2012.676055

    arXiv 2012

  5. [5]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    Pith/arXiv arXiv 2025

  6. [6]

    Exercise of human agency through collective efficacy.Current Directions in Psychological Science, 9(3):75–78, 2000

    Albert Bandura. Exercise of human agency through collective efficacy.Current Directions in Psychological Science, 9(3):75–78, 2000. doi: 10.1111/1467-8721.00064. URL https:// journals.sagepub.com/doi/10.1111/1467-8721.00064

    arXiv 2000

  7. [7]

    Accuracy of deception judgments.Personality and social psychology Review, 10(3):214–234, 2006

    Charles F Bond Jr and Bella M DePaulo. Accuracy of deception judgments.Personality and social psychology Review, 10(3):214–234, 2006

    2006

  8. [8]

    Botvinick, Todd S

    Matthew M. Botvinick, Todd S. Braver, Deanna M. Barch, Cameron S. Carter, and Jonathan D. Cohen. Conflict monitoring and cognitive control.Psychological Review, 108(3):624–652, 2001. doi: 10.1037/0033-295X.108.3.624. Tsinghua University | 13

    doi:10.1037/0033-295x.108.3.624 2001

  9. [9]

    Broadbent.Perception and Communication

    Donald E. Broadbent.Perception and Communication. Pergamon Press, Oxford, 1958. ISBN 9780080090900

    1958

  10. [10]

    Iemocap: Interactive emo- tional dyadic motion capture database.Language Resources and Evaluation, 42(4):335–359,

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette Chang, Sungbok Lee, and Shrikanth Narayanan. Iemocap: Interactive emo- tional dyadic motion capture database.Language Resources and Evaluation, 42(4):335–359,

  11. [11]

    Misp- meeting: A real-world dataset with multimodal cues for long-form meeting transcription and summarization

    Hang Chen, Chao-Han Huck Yang, Jia-Chen Gu, Sabato Marco Siniscalchi, and Jun Du. Misp- meeting: A real-world dataset with multimodal cues for long-form meeting transcription and summarization. InProceedings of the 63st Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1–14, 2025

    2025

  12. [12]

    ToMBench: Benchmarking theory of mind in large language models

    Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. ToMBench: Benchmarking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15959–15983, Bangkok, Thai...

    doi:10.18653/v1/2024 2024

  13. [13]

    Theory of minds: early understanding of interacting minds

    Aaron Chuey and Hyowon Gweon. Theory of minds: early understanding of interacting minds. Annual Review of Developmental Psychology, 7(1):91–115, 2025

    2025

  14. [14]

    Clark and Jean E

    Herbert H. Clark and Jean E. Fox Tree. Usinguhandumin spontaneous speaking.Cognition, 84(1):73–111, 2002. doi: 10.1016/S0010-0277(02)00017-3

    doi:10.1016/s0010-0277(02)00017-3 2002

  15. [15]

    Carsten K. W. De Dreu. When too little or too much hurts: Evidence for a curvilinear relationship between task conflict and innovation in teams.Journal of Management, 32(1): 83–107, 2006. doi: 10.1177/0149206305277795

    work page doi:10.1177/0149206305277795 2006

  16. [16]

    what” and “why

    Edward L. Deci and Richard M. Ryan. The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior.Psychological Inquiry, 11(4):227–268, 2000. doi: 10.1207/S15327965PLI1104_01. URLhttps://doi.org/10.1207/S15327965PLI1104_01

    doi:10.1207/s15327965pli1104_01 2000

  17. [17]

    Dynamics of affective states during complex learning

    Sidney D’Mello and Arthur Graesser. Dynamics of affective states during complex learning. Learning and Instruction, 22(2):145–157, 2012. doi: 10.1016/j.learninstruc.2011.10.001

    doi:10.1016/j.learninstruc.2011.10.001 2012

  18. [18]

    Edmondson

    Amy C. Edmondson. Psychological safety and learning behavior in work teams.Administrative Science Quarterly, 44(2):350–383, 1999. doi: 10.2307/2666999. URL https://doi.org/10. 2307/2666999

    doi:10.2307/2666999 1999

  19. [19]

    Measuring nominal scale agreement among many raters.Psychological bulletin, 76(5):378–382, 1971

    Joseph L Fleiss. Measuring nominal scale agreement among many raters.Psychological bulletin, 76(5):378–382, 1971

    1971

  20. [20]

    the” as “thee

    Jean E. Fox Tree and Herbert H. Clark. Pronouncing “the” as “thee” to signal problems in speaking.Cognition, 62(2):151–167, 1997. doi: 10.1016/S0010-0277(96)00781-0

    work page doi:10.1016/s0010-0277(96)00781-0 1997

  21. [21]

    Frith and Uta Frith

    Chris D. Frith and Uta Frith. The neural basis of mentalizing.Neuron, 50(4):531–534, 2006. doi: 10.1016/j.neuron.2006.05.001. URLhttps://pubmed.ncbi.nlm.nih.gov/16701204/

    arXiv 2006

  22. [22]

    The multimodal information based speech Tsinghua University | 14 processing (misp) 2025 challenge: Audio-visual diarization and recognition.arXiv preprint arXiv:2505.13971, 2025

    Ming Gao, Shilong Wu, Hang Chen, Jun Du, Chin-Hui Lee, Shinji Watanabe, Jingdong Chen, Siniscalchi Sabato Marco, and Odette Scharenborg. The multimodal information based speech Tsinghua University | 14 processing (misp) 2025 challenge: Audio-visual diarization and recognition.arXiv preprint arXiv:2505.13971, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    The illusion of transparency: biased assessments of others’ ability to read one’s emotional states.Journal of personality and social psychology, 75(2):332, 1998

    Thomas Gilovich, Kenneth Savitsky, and Victoria Husted Medvec. The illusion of transparency: biased assessments of others’ ability to read one’s emotional states.Journal of personality and social psychology, 75(2):332, 1998

    1998

  24. [24]

    Meetingbank: A benchmark dataset for meeting summarization

    Yebowen Hu, Timothy Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, and Fei Liu. Meetingbank: A benchmark dataset for meeting summarization. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16409–16423, 2023

    2023

  25. [25]

    Groupthink.IEEE Engineering Management Review, 36(1):36, 2008

    Irving L Janis. Groupthink.IEEE Engineering Management Review, 36(1):36, 2008

    2008

  26. [26]

    Modeling dominance in group conversations using nonverbal cues.IEEE Transactions on Audio, Speech, and Language Processing, 17(3):501–513, 2009

    Dinesh Babu Jayagopi, Hayley Hung, Chuohao Yeo, and Daniel Gatica-Perez. Modeling dominance in group conversations using nonverbal cues.IEEE Transactions on Audio, Speech, and Language Processing, 17(3):501–513, 2009. doi: 10.1109/TASL.2008.2008238. URLhttps: //ieeexplore.ieee.org/document/4755455

    arXiv 2009

  27. [27]

    Ullman, Antonio Torralba, Joshua B

    Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer D. Ullman, Antonio Torralba, Joshua B. Tenenbaum, and Tianmin Shu. MMToM-QA: Multimodal theory of mind question answering. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16077–16102, Bangkok, Thailand,

  28. [28]

    Mm-soc: Benchmarking multimodal large language models in social media platforms

    Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, and Srijan Kumar. Mm-soc: Benchmarking multimodal large language models in social media platforms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Compu- tational Linguistics: ACL 2024, pages 6192–6210, Bangkok, Thailand, August 2024. As- sociation for Computatio...

    work page doi:10.18653/v1/2024.findings-acl.370 2024

  29. [29]

    Jones, Sean Trott, and Benjamin Bergen

    Cameron R. Jones, Sean Trott, and Benjamin Bergen. Comparing humans and large language models on an experimental protocol inventory for theory of mind evaluation (EPITOME). Transactions of the Association for Computational Linguistics, 12:803–819, 2024. doi: 10.1162/ tacl_a_00674. URLhttps://aclanthology.org/2024.tacl-1.45/

    2024

  30. [30]

    William A. Kahn. Psychological conditions of personal engagement and disengagement at work. Academy of Management Journal, 33(4):692–724, 1990

    1990

  31. [31]

    Siv- bench: A video benchmark for social interaction understanding and reasoning.arXiv preprint arXiv:2506.05425, 2025

    Fanqi Kong, Weiqin Zu, Xinyu Chen, Yaodong Yang, Song-Chun Zhu, and Xue Feng. Siv- bench: A video benchmark for social interaction understanding and reasoning.arXiv preprint arXiv:2506.05425, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024

    Michal Kosinski. Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024. doi: 10.1073/pnas.2405460121. URL https://doi.org/10.1073/pnas.2405460121

    doi:10.1073/pnas.2405460121 2024

  33. [33]

    Lazarus.Emotion and Adaptation

    Richard S. Lazarus.Emotion and Adaptation. Oxford University Press, New York, NY, 1991. ISBN 9780195069945. Tsinghua University | 15

    1991

  34. [34]

    Revisiting the evaluation of theory of mind through question answering

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-...

    doi:10.18653/v1/d19-1598 2019

  35. [35]

    Sangmin Lee, Minzhi Li, Bolin Lai, Wenqi Jia, Fiona Ryan, Xu Cao, Ozgur Kara, Bikram Boote, Weiyan Shi, Diyi Yang, and James M. Rehg. Towards social ai: A survey on understanding social interactions, 2024. URLhttps://arxiv.org/abs/2409.15316

    Pith/arXiv arXiv 2024

  36. [36]

    Nale Lehmann-Willenbrock and Joseph A. Allen. Modeling temporal interaction dynam- ics in organizational settings.Journal of Business and Psychology, 33(3):325–344, 2018. doi: 10.1007/s10869-017-9506-9. URL https://link.springer.com/article/10.1007/ s10869-017-9506-9

    work page doi:10.1007/s10869-017-9506-9 2018

  37. [37]

    Enhancing emotional generation capability of large language models via emotional chain-of-thought.arXiv preprint arXiv:2401.06836, 2024

    Zaijing Li, Gongwei Chen, Rui Shao, Yuquan Xie, Dongmei Jiang, and Liqiang Nie. Enhancing emotional generation capability of large language models via emotional chain-of-thought.arXiv preprint arXiv:2401.06836, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Leiter.The Truth About Burnout: How Organizations Cause Personal Stress and What to Do About It

    Christina Maslach and Michael P. Leiter.The Truth About Burnout: How Organizations Cause Personal Stress and What to Do About It. Jossey-Bass, San Francisco, CA, 1997. ISBN 9780787908744

    1997

  39. [39]

    Social genome: Grounded social reasoning abilities of multimodal models

    Leena Mathur, Marian Qian, Paul Pu Liang, and Louis-Philippe Morency. Social genome: Grounded social reasoning abilities of multimodal models. In Christos Christodoulopoulos, TanmoyChakraborty, CarolynRose, andVioletPeng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, page 24868–24891, Suzhou, China, Novemb...

    2025

  40. [40]

    The ami meeting corpus

    Iain McCowan, Jean Carletta, Wessel Kraaij, Simone Ashby, Sebastien Bourban, Michael Flynn, Maël Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Wilfried Post, Dennis Reidsma, and Pierre Wellner. The ami meeting corpus. InProceedings of Measuring Behavior 2005: 5th Internatio...

    2005

  41. [41]

    Griffiths

    Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Thomas L. Griffiths. Eval- uating theory of mind in question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2392–2400, 2018. doi: 10.18653/v1/D18-1261. URLhttps://aclanthology.org/D18-1261/

    doi:10.18653/v1/d18-1261 2018

  42. [42]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. OpenAI (product announcement), May 2024. URLhttps://openai. com/index/hello-gpt-4o/. May 13, 2024

    2024

  43. [43]

    Introducing gpt-5

    OpenAI. Introducing gpt-5. OpenAI (release announcement), August 2025. URL https: //openai.com/index/introducing-gpt-5/. August 7, 2025

    2025

  44. [44]

    Multi-speaker attention alignment for multimodal social interaction, 2025

    Liangyang Ouyang, Yifei Huang, Mingfang Zhang, Caixin Kang, Ryosuke Furuta, and Yoichi Sato. Multi-speaker attention alignment for multimodal social interaction, 2025. URLhttps: //arxiv.org/abs/2511.17952. Tsinghua University | 16

    arXiv 2025

  45. [45]

    MIT Press, 2008

    Alex Pentland.Honest Signals: How They Shape Our World. MIT Press, 2008. URLhttps: //mitpress.mit.edu/9780262515122/honest-signals/

    arXiv 2008

  46. [46]

    A new era of intelligence with gemini

    Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. A new era of intelligence with gemini

  47. [47]

    A review of affective computing: From unimodal analysis to multimodal fusion.Information Fusion, 37:98–125, 2017

    Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion.Information Fusion, 37:98–125, 2017. doi: 10.1016/j.inffus.2017.02.003. URL https://www.sciencedirect.com/science/article/pii/ S1566253517300738

    work page doi:10.1016/j.inffus.2017.02.003 2017

  48. [48]

    Does the chimpanzee have a theory of mind?Behavioral and Brain Sciences, 1(4):515–526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind?Behavioral and Brain Sciences, 1(4):515–526, 1978. doi: 10.1017/S0140525X00076512

    doi:10.1017/s0140525x00076512 1978

  49. [49]

    URLhttps://blog.google/products/ gemini/gemini-3/

    Google Blog (Products / Gemini), November 2025. URLhttps://blog.google/products/ gemini/gemini-3/. Nov 18, 2025 (The Keyword)

    2025

  50. [50]

    Qwen3 technical report, 2025

    Qwen-Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  51. [51]

    Position: Theory of mind benchmarks are broken for large language models.arXiv preprint arXiv:2412.19726, 2024

    Matthew Riemer, Zahra Ashktorab, Djallel Bouneffouf, Payel Das, Miao Liu, Justin D Weisz, and Murray Campbell. Position: Theory of mind benchmarks are broken for large language models.arXiv preprint arXiv:2412.19726, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    MindDial: Enhancing conversational agents with theory-of-mind for common ground alignment and negotiation

    Shuwen Qiu, Mingdian Liu, Hengli Li, Song-Chun Zhu, and Zilong Zheng. MindDial: Enhancing conversational agents with theory-of-mind for common ground alignment and negotiation. In Tatsuya Kawahara, Vera Demberg, Stefan Ultes, Koji Inoue, Shikib Mehri, David Howcroft, and Kazunori Komatani, editors,Proceedings of the 25th Annual Meeting of the Special Inte...

    doi:10.18653/v1/ 2024

  53. [53]

    James A. Russell. A circumplex model of affect.Journal of Personality and Social Psychology, 39(6):1161–1178, 1980. doi: 10.1037/h0077714

    doi:10.1037/h0077714 1980

  54. [54]

    A nonverbal behavior approach to identify emergent leaders in small groups.IEEE Transactions on Multimedia, 14(3):816–832, 2012

    Dairazalia Sanchez-Cortes, Oya Aran, Marianne Schmid Mast, and Daniel Gatica-Perez. A nonverbal behavior approach to identify emergent leaders in small groups.IEEE Transactions on Multimedia, 14(3):816–832, 2012. doi: 10.1109/TMM.2011.2181941. URLhttps://ieeexplore. ieee.org/document/6127885

    arXiv 2012

  55. [55]

    Rogelberg.The Surprising Science of Meetings: How You Can Lead Your Team to Peak Performance

    Steven G. Rogelberg.The Surprising Science of Meetings: How You Can Lead Your Team to Peak Performance. Oxford University Press, New York, NY, 2019. ISBN 9780190689216

    2019

  56. [56]

    Klaus R. Scherer. Appraisal considered as a process of multilevel sequential checking. In Klaus R. Scherer, Angela Schorr, and Tom Johnstone, editors,Appraisal Processes in Emotion: Theory, Methods, Research, pages 92–120. Oxford University Press, Oxford, 2001

    2001

  57. [57]

    Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker

    Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. Tsinghua University | 17 In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    work page doi:10.18653/v1/2023.ac 2023

  58. [58]

    Neural theory-of-mind? on the limits of social intelligence in large LMs

    Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, Abu Dhabi, United Arab Emirates, December 2022. Association ...

    doi:10.18653/v1/2022.emnlp-main.248 2022

  59. [59]

    Tomato: Verbalizing the mental states of role-playing llms for benchmarking theory of mind

    Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno, Keita Suzuki, Ryo Masumura, Hiroaki Sugiyama, and Kuniko Saito. Tomato: Verbalizing the mental states of role-playing llms for benchmarking theory of mind. InProceedings of the AAAI Conference on Artificial Intelligence, 2025. URLhttps://arxiv.org/abs/2501.08838. Also available as arXiv:2501.08838

    Pith/arXiv arXiv 2025

  60. [60]

    Evidence that altercentric biases in a continuous false belief task depend on highlighting the agent’s belief.Cognition, 256:106055, 2025

    Marie Luise Speiger, Katrin Rothmaler, Ulf Liszkowski, Hannes Rakoczy, and Charlotte Grosse Wiesmann. Evidence that altercentric biases in a continuous false belief task depend on highlighting the agent’s belief.Cognition, 256:106055, 2025

    2025

  61. [61]

    Muma-tom: Multi-modal multi-agent theory of mind

    Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. Muma-tom: Multi-modal multi-agent theory of mind. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1510–1519, 2025

    2025

  62. [62]

    Cognitive load during problem solving: Effects on learning.Cognitive Science, 12 (2):257–285, 1988

    John Sweller. Cognitive load during problem solving: Effects on learning.Cognitive Science, 12 (2):257–285, 1988. doi: 10.1207/s15516709cog1202_4

    doi:10.1207/s15516709cog1202_4 1988

  63. [63]

    Toney-Butler

    Karie Tennant, Ashley Long, and Tammy J. Toney-Butler. Active listening. InStatPearls. StatPearls Publishing, Treasure Island, FL, 2023. URL https://www.ncbi.nlm.nih.gov/ books/NBK442015/. [Updated 2023 Sep 13]

    2023

  64. [64]

    Testing theory of mind in large language models and humans.Nature human behaviour, 8(7):1285–1295, 2024

    James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. Testing theory of mind in large language models and humans.Nature human behaviour, 8(7):1285–1295, 2024

    2024

  65. [65]

    Elitr-bench: A meeting assistant benchmark for long-context language models

    Thibaut Thonet, Laurent Besacier, and Jos Rozen. Elitr-bench: A meeting assistant benchmark for long-context language models. InProceedings of the 31st International Conference on Computational Linguistics, pages 407–428, 2025

    2025

  66. [66]

    Emilio Villa-Cueva, S. M. Masrur Ahmed, Rendi Chevi, Jan Christian Blaise Cruz, Kareem Elzeky, Fermin Cristobal, Alham Fikri Aji, Skyler Wang, Rada Mihalcea, and Thamar Solorio. Moments: A comprehensive multimodal benchmark for theory of mind. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 22591–22611, 2025. doi: 10. 18653/...

    2025

  67. [67]

    Kenneth W. Thomas. Conflict and negotiation processes in organizations. In Marvin D. Dunnette and Leaetta M. Hough, editors,Handbook of Industrial and Organizational Psychology, volume 3, pages 651–717. Consulting Psychologists Press, Palo Alto, CA, 2 edition, 1992

    1992

  68. [68]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Tsinghua University | 18 Advances in Neural Information Processing Systems, volume 35, pages ...

    2022

  69. [69]

    Weiss and Russell Cropanzano

    Howard M. Weiss and Russell Cropanzano. Affective events theory: A theoretical dis- cussion of the structure, causes, and consequences of affective experiences at work. In Barry M. Staw and L. L. Cummings, editors,Research in Organizational Behavior, volume 18, pages 1–74. Elsevier Science/JAI Press, 1996. URLhttps://web.mit.edu/curhan/www/docs/ Articles/...

    1996

  70. [70]

    Social signals, their function, and automatic analysis: A survey

    Alessandro Vinciarelli, Maja Pantic, Hervé Bourlard, and Alex Pentland. Social signals, their function, and automatic analysis: A survey. InProceedings of the 10th International Conference on Multimodal Interfaces, ICMI ’08, pages 61–68, New York, NY, USA, October 2008. Association for Computing Machinery. doi: 10.1145/1452392.1452405

    arXiv 2008

  71. [71]

    Vcsum: A ver- satile chinese meeting summarization dataset

    Han Wu, Mingjie Zhan, Haochen Tan, Zhaohui Hou, Ding Liang, and Linqi Song. Vcsum: A ver- satile chinese meeting summarization dataset. InFindings of the Association for Computational Linguistics: ACL 2023, pages 6065–6079, 2023

    2023

  72. [72]

    Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10691–10706, 2023

    2023

  73. [73]

    Wickens, William S

    Christopher D. Wickens, William S. Helton, Justin G. Hollands, and Simon Banbury.Engineering Psychology and Human Performance. Routledge, New York, NY, 5 edition, 2021. ISBN 9781032011738. doi: 10.4324/9781003177616

    work page doi:10.4324/9781003177616 2021

  74. [74]

    M2met: The icassp 2022 multi-channel multi-party meeting transcription challenge

    Fan Yu, Shiliang Zhang, Yihui Fu, Lei Xie, Siqi Zheng, Zhihao Du, Weilong Huang, Pengcheng Guo, Zhijie Yan, Bin Ma, et al. M2met: The icassp 2022 multi-channel multi-party meeting transcription challenge. InICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6167–6171. IEEE, 2022

    2022

  75. [75]

    Persuasivetom: A benchmark for evaluating machine theory of mind in persuasive dialogues, 2025

    Fangxu Yu, Lai Jiang, Shenyi Huang, Zhen Wu, and Xinyu Dai. Persuasivetom: A benchmark for evaluating machine theory of mind in persuasive dialogues, 2025. URLhttps://arxiv.org/ abs/2502.21017

    Pith/arXiv arXiv 2025

  76. [76]

    OpenToM: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models

    Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. OpenToM: A comprehensive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8593–8623, Bangkok, Thailand, 2024. Association for Computa...

    doi:10.18653/v1/2024.acl-long.466 2024

  77. [77]

    Multi-modal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multi-modal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2236–2246, 2018. doi: 10.18653/v1/P18-1208. URL https://ac...

    doi:10.18653/v1/p18-1208 2018

  78. [78]

    Mug: A general meeting understanding and generation benchmark

    Qinglin Zhang, Chong Deng, Jiaqing Liu, Hai Yu, Qian Chen, Wen Wang, Zhijie Yan, Jinglin Liu, Yi Ren, and Zhou Zhao. Mug: A general meeting understanding and generation benchmark. InICASSP 2023-2023 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  79. [79]

    Social-IQ: A question answering benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-IQ: A question answering benchmark for artificial social intelligence. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8807– 8817, June 2019. URLhttps://openaccess.thecvf.com/content_CVPR_2019/html/Zadeh_ Social-IQ_A_Q...

    2019

  80. [82]

    MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi- domain meeting summarization. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages...

    2021

Showing first 80 references.