Pith. sign in

REVIEW 4 major objections 6 minor 19 references

XToM: Exploring the Multilingual Theory of Mind for Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces XToM, a five-language theory-of-mind benchmark, and uses it to show that large language models understand multiple languages well yet reason about beliefs unevenly and inconsistently across them.

desk verdict A useful multilingual ToM benchmark whose cross-lingual comparisons are shakier than the paper claims; the fact-belief gap is likely real, but translation difficulty is not psychometrically validated. read the letter →

arxiv 2506.02461 v1 pith:AKRIN4L5 submitted 2025-06-03 cs.CL

Chunkit Chan , Yauwai Yim , Hongchuan Zeng , Zhiying Zou , Xinyuan Cheng , Zhifan Sun , Zheye Deng , Kawai Chung
show 9 more authors
Yuzhuo Ao Yixiang Fan Cheng Jiayang Ercong Nie Ginny Y. Wong Helmut Schmid Hinrich Schütze Simon See Yangqiu Song
This is my paper
classification cs.CL
keywords multilingualtheoryofmindlargelanguagemodelsfalsebeliefreasoningcross-lingualevaluationbenchmarkconstructiontranslationqualitymentalstateinferenceXToM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

XToM is a human-annotated benchmark that translates 300 Theory-of-Mind stories and dialogues from three established English benchmarks into Chinese, German, French, and Japanese, giving each item semantically equivalent versions in five languages plus paired fact and belief questions. The paper uses it to test ten large language models and finds a pronounced dissonance: the models answer fact questions accurately in all five languages, but belief questions—questions about what a character thinks or will do—produce lower and unevenly distributed accuracy across languages. This gap is the paper's central evidence that LLMs' strong multilingual language understanding is not matched by robust multilingual reasoning about other minds. The authors position the result as a caution against reading fluent multilingual output as human-like social cognition, with direct consequences for deploying LLMs in multilingual social contexts.

What carries the argument

The load-bearing object is XToM itself: a parallel multilingual benchmark in which each story or dialogue from ToMi, FANToM, and NegotiationToM is translated into English, Chinese, German, French, and Japanese, corrected and scored by native-speaker annotators under the Multidimensional Quality Metrics protocol with a threshold of 95, and paired with a fact question and a belief question about the same narrative. This construction isolates the mechanism behind the claim: because the fact and belief questions share the same multilingual context, a model that answers the fact question but misses the belief question cannot be failing on language comprehension alone, and because the five versions are designed to be semantically equivalent, cross-language differences in belief accuracy can be attributed to reasoning rather than to surface vocabulary.

What would settle it

Give the same XToM items to human native speakers of all five languages and compare their answer accuracy and consistency with the LLM results: if human performance reproduces the models' cross-language pattern, the benchmark is measuring translation or cultural difficulty rather than a model-specific ToM deficit; if humans answer near-ceiling and uniformly across languages, the LLM gaps are real.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLMs do not exhibit robust Multilingual Theory of Mind: they can understand a story or dialogue in several languages while failing to answer questions about characters' false beliefs, and their belief-reasoning accuracy, as well as their actual answer choice, shifts with the language of the prompt. The empirical pattern is a systematic fact-versus-belief gap that holds across the three sub-benchmarks XToMi, XFANToM, and XNegotiationToM; models also show idiosyncratic language preferences and low cross-language consistency on belief questions even when fact-question consistency is high. The paper concludes that statistical mastery of language patterns does not equate to human-like social cognition.

Load-bearing premise

The load-bearing premise is that the five translated versions of each item are equally fair tests of theory of mind—same difficulty, same inferential demands, same cultural accessibility—rather than translations that differ in how easy it is to retrieve the correct answer.

Editorial extensions

If this is right

  • English-only evaluations of LLM theory of mind cannot be assumed to generalize; the same model can look competent in one language and weak in another.
  • Fact-question accuracy should not be used as evidence of social reasoning, since the belief-versus-fact gap appears even when multilingual comprehension is strong.
  • Scaling model size improves multilingual fact consistency but does not by itself improve belief consistency, so ToM may need targeted training.
  • Fine-tuning on theory-of-mind data in one language transfers to other languages, and multilingual fine-tuning can exceed monolingual fine-tuning in some languages.
  • Chain-of-thought prompting does not make multilingual ToM robust; performance remains language-dependent and can even decline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The fact-versus-belief pair design could be adapted to non-social reasoning, such as temporal or causal inference, to test whether the cross-language dissonance is specific to mental states or a general signature of reasoning that is not anchored in surface form.
  • Editorial inference: Because some cross-language differences may come from culture-laden content, a natural extension is to separate culturally neutral from culturally embedded items to identify how much of the ToM gap is linguistic rather than cultural.
  • Editorial inference: If cultural-linguistic context rather than raw language ability is the cause, adding more target-language text to pretraining may not close the gap; interventions would need to teach culturally grounded perspective-taking or adapt prompts to local norms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces XToM, a multilingual Theory-of-Mind benchmark covering English, Chinese, German, French, and Japanese, built by translating and human-correcting 300 sampled stories/dialogues from ToMi, FANToM, and NegotiationToM. Translation quality is assessed with MQM at a threshold of 95, and the benchmark is used to evaluate ten LLMs under zero-shot and chain-of-thought prompting. The central claim is a 'pronounced dissonance': models show high accuracy on multilingual fact questions but substantially lower and more language-dependent accuracy on belief questions, interpreted as evidence that LLMs' statistical language mastery does not amount to human-like or language-robust social cognition. The paper also analyzes cross-lingual answer consistency, first- versus second-order belief performance, task transfer through fine-tuning, and model-specific language preferences. The benchmark resource, human annotation pipeline, contamination checks on XFANToM and XNegotiationToM, and the breadth of models evaluated are genuine strengths, but the validity of the cross-lingual ToM comparisons rests on an assumption of psychometric equivalence across translations that the paper does not establish.

Significance. If the multilingual ToM comparisons are valid, XToM would fill a real gap: existing ToM benchmarks are predominantly English-only, and the paper's fact-vs-belief gap would be a useful falsifiable observation about LLM social reasoning. The human MQM annotation (with 95+ scores), the name-normalization step, the explicit contamination checks (Tables 4-5), and the release of code and data are concrete contributions that make the resource reusable. However, the significance of the cross-lingual finding is currently conditional: translation fidelity measured by MQM does not imply that the five language versions are equally difficult, equally inferentially demanding, or equally culturally accessible tests of ToM. Without human baselines or item-level equivalence analysis, the reported language gaps and the 'dissonance' conclusion could be artifacts of translation and item difficulty rather than properties of LLM social cognition. The paper is therefore best viewed as a promising benchmark construction plus an initial, clearly-labeled set of observations, not yet a validated cross-linguistic ToM measurement.

major comments (4)
  1. [§3.4 and Table 1] The MQM scores validate translation fidelity to the English source, but they do not establish psychometric equivalence of the five language versions. Equation (1) posits semantic equivalence as a construction assumption, yet the paper never reports human accuracy norms, item-difficulty comparisons, or differential-item-functioning analyses across languages. Table 19 makes the problem concrete: the same FANToM dialogue receives model answer A in EN/DE/FR and answer B in ZH/JA, and the authors attribute this to culture without ruling out translation-induced difficulty shifts. Because every cross-lingual comparison in Figures 2-4 and Section 5.6 is interpreted as ToM ability, the absence of equivalence validation undermines the central claim. The paper should either add per-language human baselines plus item-level equivalence tests, or explicitly soften all ToM-ability language comparisons to translation-faithful performance differences.
  2. [§5.1, Figure 5] The 'dissonance' argument relies on comparing fact-question accuracy with belief-question accuracy, but the fact questions are not matched controls: they are retrieval items with a different inferential load. High fact accuracy shows that models can understand and retrieve from the provided context, but it does not show equal language ability under the reasoning demand imposed by belief questions. If some translations make belief items systematically harder (e.g., via longer sentences, more ambiguous pronouns, or different inference markers), the fact-belief gap would be inflated. The paper should report matched item pairs (same story or dialogue, comparable surface length and vocabulary complexity) or statistically control for item difficulty across the two question types before concluding that LLMs lack multilingual ToM.
  3. [§3.1, A.4, Tables 4-5, 8-9] The contamination verification reports zero matches for XFANToM and XNegotiationToM but 0-36 matches for XToMi across models and languages, and the Limitations section acknowledges that ToMi may be contaminated. Nevertheless, XToMi results are reported alongside the uncontaminated subsets and the abstract and conclusion generalize to XToM as a whole. The paper should present XToMi as a reference-only subset, exclude it from any aggregate or headline claims, or provide a contamination-robust analysis before using XToMi results to support the main conclusion. As written, the inclusion of XToMi conflates a possibly memorization-prone benchmark with the uncontaminated portions.
  4. [Figures 2-4 and Table 8] Cross-language gaps are reported as point estimates without confidence intervals or significance tests. Given the per-language item counts (hundreds of questions per subset), some differences, such as the GPT-4o Japanese-vs-Chinese 11% gap cited in Section 5.6, appear substantial, but the reader cannot distinguish stable effects from sampling noise for the smaller gaps. The paper should report bootstrapped confidence intervals or a significance test for the headline cross-language comparisons, and for the fact-versus-belief gap in Section 5.1.
minor comments (6)
  1. [Figure 1] The figure contains duplicated text blocks and formatting artifacts (e.g., repeated French and Chinese story excerpts with a stray '?'), which should be cleaned for the camera-ready version.
  2. [§5.1, Figure 5] The parenthetical 'except DeepSeek R1' is not backed by a statistical interaction test or a held-out analysis; either add a test for the model-by-question-type interaction or remove the exception claim.
  3. [§3.2] The replacement of English names with 'Person number' is a sensible bias-mitigation step, but the paper should state whether possessive and pronoun references were systematically checked for ambiguity after replacement, since such ambiguity can change the difficulty of belief questions across languages.
  4. [Table 8] The layout of Table 8 is extremely dense and hard to read, with visual runs of model names and scores; consider splitting it by model or adding standard errors per entry.
  5. [§5.3] The observation that several models score higher on second-order than on first-order belief questions is counterintuitive and should be briefly explained or checked for label/option-order artifacts before being presented as a finding.
  6. [Abstract] The abstract calls the benchmark 'rigorously validated,' but Section 3.4 validates translation quality, not the psychometric properties of the ToM measure; suggesting 'human-validated translations' would be more precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark construction, human validation, and model evaluation form a self-contained pipeline; cross-lingual equivalence is an assumption, not a derived prediction.

full rationale

The paper does not derive its central claims from its own definitions or from a fitted parameter renamed as a prediction. The benchmark is built by sampling existing ToM datasets (ToMi, FANToM, NegotiationToM), translating them with GPT-4o, then having native-speaker annotators correct and evaluate the translations using the MQM framework. The later evaluation of LLMs, including GPT-4o, measures model performance on a human-corrected benchmark; GPT-4o's role as initial translator does not make the evaluation scores a fitted input, because no parameter is fitted to the evaluated responses and the human correction step breaks any direct statistical forcing. The cross-lingual comparisons in Section 5 do rely on the stated assumption in Equation (1) that language versions are semantically equivalent, but this is an explicit design assumption, not a conclusion derived from itself; the reported language differences are empirical observations. The fact-vs-belief gap in Section 5.1 is likewise an observed performance comparison, not a construction artifact. Contamination checks are reported and acknowledge the ToMi issue while arguing FANToM and NegotiationToM are uncontaminated. The use of the authors' prior NegotiationToM benchmark as a source is a normal external citation, and it is not used to justify a uniqueness claim or to forbid alternatives. The main validity concern, that MQM fidelity does not guarantee psychometric equivalence across languages, is a substantive measurement critique rather than a circularity. Therefore, no specific circular step can be exhibited under the required standard.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No invented entities or fitted model parameters are introduced. The central findings rest on design choices (translation protocol, MQM threshold, sample size, source benchmarks) and on domain assumptions about task validity and cross-lingual equivalence, all assumed rather than independently demonstrated.

free parameters (3)
  • MQM passing threshold = 95
    Hand-chosen threshold to accept translated instances (stricter than the standard 90); controls which items enter the benchmark, but was not derived from a predictive target (Section 3.4).
  • Sampled stories per source benchmark = 300
    Hand-selected sample size for ToMi, FANToM, and NegotiationToM; affects benchmark composition and statistical power (Appendix A.1).
  • Evaluation temperature settings = varies (GPT: 1.0; DeepSeek/Qwen: 0.5; Mistral: 0.2)
    Chosen per model based on defaults; affects response stochasticity, and no multiple-seed averaging is reported (Appendix C.1).
assumptions (4)
  • domain assumption A high MQM translation score ensures the translated items preserve the cognitive demands of the original ToM tasks.
    The paper validates translation quality, not task equivalence; a high MQM score does not guarantee equal difficulty or inferential structure across languages (Sections 3.3 and 3.4).
  • domain assumption ToMi, FANToM, and NegotiationToM are valid measures of theory of mind in their original English form.
    The benchmark inherits labels and construct validity from the source datasets; no independent validation of the ToM construct is provided (Sections 2.1 and 3.1).
  • domain assumption The contamination verification protocols of Golchin and Surdeanu (2024) and Li and Flanigan (2024) can detect benchmark leakage across five languages.
    Used to claim XFANToM and XNegotiationToM are uncontaminated; the protocols were not designed for multilingual leakage detection (Appendix A.4).
  • domain assumption The translated zero-shot and chain-of-thought prompts are equivalent in difficulty across languages.
    Prompt translations were checked by annotators, but no systematic comparison of prompt difficulty or bias across languages is reported (Appendix C.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of XToM: Exploring the Multilingual Theory of Mind for Large Language Models." pith.science (2026). https://pith.science/paper/AKRIN4L5

@misc{pith2026250602461,
  author       = {Pith},
  title        = {Pith review of: XToM: Exploring the Multilingual Theory of Mind for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AKRIN4L5}},
  note         = {Machine review of arXiv:2506.02461}
}
read the original abstract

Theory of Mind (ToM), the ability to infer mental states in others, is pivotal for human social cognition. Existing evaluations of ToM in LLMs are largely limited to English, neglecting the linguistic diversity that shapes human cognition. This limitation raises a critical question: can LLMs exhibit Multilingual Theory of Mind, which is the capacity to reason about mental states across diverse linguistic contexts? To address this gap, we present XToM, a rigorously validated multilingual benchmark that evaluates ToM across five languages and incorporates diverse, contextually rich task scenarios. Using XToM, we systematically evaluate LLMs (e.g., DeepSeek R1), revealing a pronounced dissonance: while models excel in multilingual language understanding, their ToM performance varies across languages. Our findings expose limitations in LLMs' ability to replicate human-like mentalizing across linguistic contexts.

Figures

Figures reproduced from arXiv: 2506.02461 by the authors.

Figure 1
Figure 1. A belief question and fact question (i.e., Real [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of different models on false belief questions across languages in XFANToM. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of different models on false belief questions across languages in XToMi. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Performance comparison of different models on belief choice across languages in XNegotiationToM. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: LLMs’ performance comparison of belief questions and fact questions in XFANToM. matches. First, as shown in Appendix A.4, many LLMs were likely exposed to the ToMi dataset during both pre-training and post-training phases, potentially facilitating some transfer of theo…
Figure 6
Figure 6. Figure 6: Consistency analysis across different lan [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison of different models on first- and second-order belief questions across languages [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison of task transfer on [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: LLMs Language Preference in XFANToM. essential questions about their underlying linguis￾tic biases. The experimental results in [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Interface for Human Correction and Validation [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of different models on false belief questions across languages in XFANToM. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of different models on false belief questions across languages in XToMi. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Performance comparison of different models on belief choice across languages in XNegotiationToM. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Performance comparison of different models on desire choice across languages in XNegotiationToM. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Performance comparison of different models on intention choice across languages in XNegotiationToM. [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Performance comparison of different models on first order and second order belief choice across [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Performance comparison of different models on first order and second order belief choice across [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: ToMi consistency analysis across different [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: Error analysis of different models on belief [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Error analysis of DeepSeek R1 on be￾lief question in XFANToM across language. With DeepSeek R1 receive zero error in Fact Understanding Error and Irrelevant Response Error; these two errors are ignored here. in language understanding. Consistent with prior research (S…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 10 canonical work pages

  1. [7]

    Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap

    Comparing humans and large language mod- els on an experimental protocol inventory for theory of mind evaluation (epitome).Transactions of the Association for Computational Linguistics, 12:803– 819. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap

    work page

  2. [8]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 14397–14413

    Fantom: A benchmark for stress-testing ma- chine theory of mind in interactions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Sin- gapore, December 6-10, 2023, pages 14397–14413. Association for Computational Linguistics. Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large ...

    arXiv 2023

  3. [10]

    Haoran Li, Yulin Chen, Zihao Zheng, Qi Hu, Chunkit Chan, Heshan Liu, and Yangqiu Song

    Privacy in large language models: Attacks, de- fenses and future directions.CoRR, abs/2310.10383. Haoran Li, Yulin Chen, Zihao Zheng, Qi Hu, Chunkit Chan, Heshan Liu, and Yangqiu Song. 2024a. Back- door removal for generative large language models. CoRR, abs/2405.07667. Haoran Li, Dadi Guo, Donghao Li, Wei Fan, Qi Hu, Xin Liu, Chunkit Chan, Duanyi Yao, Yu...

    arXiv

  4. [11]

    you gotta be a doctor, lin

    Analyzing leakage of personally identi- fiable information in language models.CoRR, abs/2302.00539. Xiaomeng Ma, Lingyu Gao, and Qihui Xu. 2023a. Tom- challenges: A principle-guided dataset and diverse evaluation tasks for exploring theory of mind. In Conference on Computational Natural Language Learning. Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai....

    arXiv 2023

  5. [13]

    Sandra Sandoval, Jieyu Zhao, Marine Carpuat, and Hal Daumé III

    Multi-tom: Evaluating multilingual theory of mind capabilities in large language models.CoRR, abs/2411.15999. Sandra Sandoval, Jieyu Zhao, Marine Carpuat, and Hal Daumé III. 2023. A rose by any other name would not smell as sweet: Social bias in names mis- translation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ...

    arXiv 2023

  6. [14]

    Improving dialog systems for negotiation with personality modeling. InProceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 681–693. Association for Com- putati...

    arXiv 2021

  7. [16]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q

    Multilingual brain surgeon: Large language models can be compressed leaving no language be- hind. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with BERT. In8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,

    work page 2020

  8. [18]

    let’s think step by step

    to assist annotators. The code will be released alongside the dataset. A.4 Verification of Potential Contamination Most of the existing available benchmarks in the NLP field were released prior to the initiation of the LLM training process, indicating that these datasets are likely to have been utilized during the pre-training phase and post-training phas...

    work page 2024

Show all 19 references
  1. [19]

    A”, “B”, “C

    to fine-tune LLMs using Low-Rank Adapta- tion (LoRA) (Hu et al., 2022) to achieve parameter- efficient training. The batch size is set to 64, and the learning rate is 5e-5. The LoRA rank is 8, with α set to 32. The maximum length for the input is set to 4,096. We set the warm-...

    2022

  2. [1286]

    Joseph L Fleiss

    IEEE. Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters.Psychological bulletin, 76(5):378. Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey

    1971

  3. [1985]

    theory of mind

    Does the autistic child have a “theory of mind” ?Cognition, 21:37–46. Simon Baron-Cohen, Michelle O’riordan, Valerie E. Stone, R. Jones, and Kate C. Plaisted. 1999. Recogni- tion of faux pas by normally developing children and children with asperger syndrome or high-functionin...

    1999 arXiv

  4. [2017]

    Michael Hanna and Ondrej Bojar

    How can memory-augmented neural networks pass a false-belief task?Cognitive Science. Michael Hanna and Ondrej Bojar. 2021. A fine-grained analysis of bertscore. InProceedings of the Sixth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, November 10-11, 2021, pa...

    2021 arXiv

  5. [2019]

    Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, No...

    2019

  6. [2020]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma

    OpenReview.net. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. Lla- mafactory: Unified efficient fine-tuning of 100+ lan- guage models.CoRR, abs/2403.13372. Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R. McK...

    2024 arXiv

  7. [2021]

    Experts, errors, and context: A large-scale study of human evaluation for machine translation. Trans. Assoc. Comput. Linguistics, 9:1460–1474. Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif- fiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Cheval...

    2023 arXiv

  8. [2022]

    Training language models to follow instruc- tions with human feedback. InAdvances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 - December 9, 2022. Ziqian Peng, ...

    2022 arXiv

  9. [2023]

    In32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023, Busan, Republic of Korea, August 28-31, 2023, pages 1279–

    Models and algorithms for human-aware task planning with integrated theory of mind. In32nd IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2023, Busan, Republic of Korea, August 28-31, 2023, pages 1279–

    2023

  10. [2024]

    Jiayang Cheng, Lin Qiu, Tsz Ho Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, and Zheng Zhang

    Tombench: Benchmarking theory of mind in large language models.ArXiv, abs/2402.15052. Jiayang Cheng, Lin Qiu, Tsz Ho Chan, Tianqing Fang, Weiqi Wang, Chunkit Chan, Dongyu Ru, Qipeng Guo, Hongming Zhang, Yangqiu Song, Yue Zhang, and Zheng Zhang. 2023. Storyanalogy: Deriving sto...

    2023 arXiv

  11. [2025]

    InProceedings of the 31st International Conference on Computational Linguistics, pages 10602–10617, Abu Dhabi, UAE

    Converging to a lingua franca: Evolution of linguistic regions and semantics alignment in mul- tilingual large language models. InProceedings of the 31st International Conference on Computational Linguistics, pages 10602–10617, Abu Dhabi, UAE. Association for Computational Lin...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.