{"id":"3ddb8487-1c4b-4a3b-9c2c-0e63fca8ac98","arxiv_id":"2501.01705","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark tests LLMs on theory-of-mind questions about novel characters, showing that humans with book knowledge outperform the best LLMs.","lead":"This paper introduces CHARTOM-QA, a benchmark of 1,035 theory-of-mind questions about characters in classic novels, and finds that people who have read the books answer significantly better than those who have not. It also reports that state-of-the-art LLMs score notably worse than well-read humans and fail to improve with longer context.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM evaluation never supplies the global context G defined in Eq. 1, so the human-LLM gap may reflect missing input rather than inability to use long-term background.","rationale":"The reader's weakest assumption focuses on the small human sample. That is a real limitation, but it is a statistical power issue that a larger study could fix; the design itself (each question answered by one reader and one non-reader) is reasonable. The more decisive threat is that the LLM evaluation never presents the global context the task is designed to test. The paper defines P(a|q,W,G) in Eq. 1 and argues that human ToM relies on G, but then evaluates LLMs with W only. The only channel for G is the pretraining corpus, which the authors do not measure or control. This creates a direct confound: if a model has not memorized the relevant backstory, its failure says nothing about whether it can reason from that backstory when given it. The human readers in Table 3 have the backstory in memory, so the comparison is not apples-to-apples. Supplying G as in-context input is the obvious control condition. Without it, the paper's headline claim that 'LLMs still perform notably worse than humans, despite having seen these stories' is not established. This concern is load-bearing because it targets the main empirical conclusion about LLMs, not just the precision of the human baseline. I therefore propose the full-context experiment as the decisive check. If the result confirms the gap, the benchmark and conclusions stand; if not, the paper would need to revise its central claim. The verdict should remain conditional pending this experiment.","tokens_in":21125,"tokens_out":10916,"duration_ms":114536,"concrete_test":"Run the same 150 multichoice questions used in Table 3 with GPT-4o, o1, and DeepSeek-R1, but supply the full novel text (or, within context limits, the concatenation of all passages mentioning the target character plus the associated plot window) as G in the prompt, holding the candidate choices fixed. If accuracy reaches or exceeds the human reader group (≥60% at the c=2k setting), the reported human-LLM gap is an artifact of withholding G; if accuracy remains below the reader group (≤55%), the claim that LLMs cannot use global context when it is available is supported. As a secondary check, also record c=0 accuracy without the novel to verify that the models have any parametric knowledge of these books.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The task formulation in Eq. 1 conditions the answer on global context G, and the paper's central claim is that LLMs cannot exploit such long-term background. However, every LLM experiment (Tables 1-3; Appendix K prompts) provides only a plot window W of 0/1k/2k tokens, never G. The abstract's assertion that the models 'have seen these stories during pre-training' is used as a substitute for G, but this is an unverified assumption: pretraining exposure to the Chinese translations is not checked, and parametric recall of a book is not equivalent to having the relevant background available during inference. In Table 3, human readers have actual memory of the whole book, so the comparison is asymmetric—readers effectively receive G, LLMs do not. The conclusion that LLMs struggle to capture nuanced historical context is therefore confounded with the absence of that context from the model input. This threatens the main LLM claim independently of the smaller-sample concern in the human study.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CHARTOM-QA, a new benchmark of 1,035 theory-of-mind (ToM) multiple-choice and generative questions about characters from classic novels. The benchmark is built with an AI-assisted human annotation pipeline that uses reader notes, GPT-4o for paraphrasing and question generation, and expert validation. The authors report a human study with 8 participants and 150 questions, finding that readers of the novels outperform non-readers by about 21%, and that non-readers perform at roughly the level of GPT-4o. They also evaluate six LLMs (including GPT-4o, GPT-3.5, Llama-3.1, Qwen2, Mistral, InternLM2) and the reasoning models o1 and DeepSeek-R1, finding that even the strongest models lag behind readers. The paper's central claim is that human ToM depends on long-term knowledge of a person's background—knowledge that current LLMs cannot recover from short plot windows despite possible pretraining exposure to the source texts.","tokens_in":21288,"tokens_out":2880,"duration_ms":29080,"significance":"If the central claim holds, the benchmark addresses a genuinely underexplored aspect of ToM evaluation: the need to integrate global character history rather than only local narrative cues. The dataset construction is creative, using authentic reader notes from online reading platforms, and the final benchmark spans diverse classic novels and 150 characters. The paper also proposes a bonus-point/penalty evaluation protocol and reports that it correlates more strongly with human judgments (Pearson 0.92) than Rouge or BERTScore. The inclusion of o1 and DeepSeek-R1 is timely. However, the significance is contingent on the validity of the human-LLM comparison and on the absence of confounds in the benchmark design; the paper's own evidence has several gaps that need to be addressed before the strong claims about LLM limitations can be accepted.","major_comments":[{"comment":"Equation (1) formally defines the task as P(a|q,W,G), conditioning on the global context G of the whole book, but every LLM experiment in Tables 1–3 and the prompts in Appendix K provide only a plot window W of 0/1k/2k tokens, never G. The abstract's assertion that LLMs 'have seen these stories during pre-training' is not verified for the specific Chinese translations and editions used, and parametric recall of a book is not equivalent to having its global context available during inference. Consequently, the human-LLM comparison in Table 3 is asymmetric: human readers have actual memory of G, while LLMs receive only W. The conclusion that LLMs fail to exploit long-term background is therefore confounded with the absence of G from the model input. The authors should either supply the full book (or a substantial relevant excerpt) to LLMs, add an explicit retrieval step, or probe whether the models can recall the relevant background; otherwise the main LLM claim is not supported.","section":"§3, §6.2, Appendix K"},{"comment":"The human study relies on only 8 participants and 150 questions, reports no significance tests, confidence intervals, or inter-rater reliability, and defines reading status by self-report. The ~21% gap between the 'w/. history' and 'w/o. history' groups is load-bearing for RQ1 and for the human-LLM comparison, but with this sample size the gap could be driven by individual differences, by partial cultural familiarity with classic novels, or by chance. The authors should report a mixed-effects model with participant and book random effects, bootstrap confidence intervals, or at least a screening quiz to confirm that the 'read' group actually remembers the books and the 'non-read' group does not. Without such evidence, the claim that book familiarity causes the human advantage is not established.","section":"§6.3, Table 3"},{"comment":"GPT-4o participates at nearly every stage of the benchmark: it paraphrases user notes into ToM descriptions, generates candidate questions, constructs distractors, extracts bonus points, and scores model responses. This creates a circularity risk: the benchmark may reward the kinds of answers that GPT-4o itself prefers. The reported 0.92 human correlation in Appendix E is computed on a subset, but the number of responses and human judgments, the selection procedure, and confidence intervals are not reported. The authors should report the sample size and design of the correlation study, validate a held-out set of final questions and answers with human experts (not just the underlying notes), and preferably include a non-GPT-4o evaluator (e.g., a smaller open-source LLM or human ratings) to confirm the relative rankings.","section":"§4.2, §5, Appendix E"}],"minor_comments":[{"comment":"The benchmark name is inconsistent across the manuscript: the abstract and main text use 'CHARTOM-QA' in some places and 'CharToM' in others; please unify the naming.","section":"Abstract, §1, Table 4"},{"comment":"The abbreviation 'indir.' in Table 3 is not defined until §6.4; please define it directly in the table caption.","section":"Table 3"},{"comment":"There is a typo: 'aformentioned' should be 'aforementioned'.","section":"§4.2"},{"comment":"The Pearson correlation of 0.92 is reported without the number of model responses or human judgments used; please add these details.","section":"Appendix E"},{"comment":"The table uses 'N/A', '–', and '!' symbols without a clear legend in the caption; please explain them explicitly.","section":"Table 5"},{"comment":"The references for Ma et al. 2023a and 2023b are both cited in §2 and appear in the bibliography; please check that all in-text citations (including 'Tomchallenges' and 'TOMBENCH') point to the intended entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important gap in ToM benchmarking, and the dataset construction is creative. The main risk is the confound between the formal task definition (conditioning on global context G) and the actual LLM inputs (only a plot window), which directly undermines the headline comparison with humans. The human study's small sample and lack of statistical inference compound this. I believe the issues are fixable within the manuscript's scope by running additional LLM experiments with full-book or retrieval-augmented inputs, and by strengthening the human study reporting, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark is a genuine contribution, but the strongest claim in the abstract needs a caveat: the LLMs are never given the global context your own Eq. 1 says the task requires. That doesn't sink the paper, but it changes what the LLM results can say.\n\nWhat's new: CHARTOM-QA draws 1,035 ToM questions from real novels and online reader notes, so it captures the kind of long-run character background that template-generated benchmarks miss. The human study is the clearest result: the same sort of educated participants score ~21 points higher when they've read the book than when they haven't, and non-readers land around GPT-4o's level. That's a useful, believable demonstration that background knowledge matters for ToM in this setting. The bonus-point/penalty evaluation protocol is also a reasonable move, and the reported 0.92 correlation with human judgments is solid evidence the metric tracks quality.\n\nThe soft spots are real. First, the LLM evaluation: the task is defined as P(a|q,W,G), but every prompt in Appendix K gives only W (0/1k/2k tokens). The paper leans on the claim that models 'have seen these stories during pre-training,' but that's unverified for the Chinese translations used, and parametric recall isn't the same as having the relevant background available at inference. So the comparison is asymmetric: human readers genuinely have G in memory; LLMs are being asked to answer without it. The conclusion that LLMs 'struggle to capture nuanced historical context' should be softened to 'struggle when that context is absent from the input and must be recovered from pretraining.' The paper actually nods at this in the Clever Hans discussion, but the abstract doesn't.\n\nSecond, the human study is thin: 8 participants, 150 questions, no significance tests or confidence intervals, and reading status is self-reported. The effect size is large, so I'd bet it replicates, but as it stands the central human-LLM gap is a point estimate, not a tested claim.\n\nThird, GPT-4o is used in generation, answer construction, and scoring. The 0.92 correlation with human judges mitigates the scoring part, and the human verification of notes and questions helps, but there's still a risk the benchmark is shaped by GPT-4o's preferences.\n\nOverall: the benchmark is worth having, the human finding is probably right, and the LLM comparison needs reframing rather than rejection. I'd send it to peer review with requests for a significance test, a larger or better-controlled human sample, and an LLM setting that either supplies relevant global context or explicitly tests retrieval of it.","headline":"A genuinely useful new ToM benchmark, but the LLM-versus-human claim is undercut because the models never receive the global context the task defines.","tokens_in":21832,"tokens_out":2739,"would_cite":true,"duration_ms":26758,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that human Theory of Mind relies on long-term knowledge of a person's background, and that current LLMs, including o1 and DeepSeek-R1, perform notably worse than human readers on questions that require this knowledge.","keywords":["Theory of Mind","question answering","large language models","contextual understanding","character understanding","benchmark","long-context reasoning","novel comprehension"],"falsifier":"A preregistered replication with a larger, balanced sample that finds no reader-versus-non-reader accuracy gap on the same 150 questions would falsify the central claim; a second decisive result would be non-readers matching readers once given a written character-background summary, which would show the effect is information access rather than memory integration.","tokens_in":1636,"feed_emoji":"📚","tokens_out":2759,"duration_ms":76486,"temperature":0.7,"pith_summary":"The paper argues that a core part of human Theory of Mind—the ability to attribute mental states to others—is understanding a person's long-term background and life story, and that existing benchmarks built on short, self-contained stories never test this. To support the point, the authors introduce CharToM-QA, 1,035 questions about characters from 20 classic novels, covering belief, intention, emotion, and desire. In a human study, educated participants answered far better when they had read the novel than when they had not, while even strong LLMs underperformed readers despite the stories being in their training data. If correct, this means long-term contextual understanding is a distinct, mostly missing axis in machine Theory of Mind evaluation.","feed_headline":"Readers beat GPT-4o on Theory of Mind questions about novels","feed_subtitle":"People who had read the book score ~21 points higher, and longer story windows do not close the LLM gap.","key_machinery":"The load-bearing object is the CharToM-QA benchmark itself: 1,035 Theory of Mind questions drawn from reader annotations on 20 classic novels, with each question tied to a plot window and to one of four Theory of Mind dimensions (belief, intention, emotion, desire). Because the questions concern characters whose full life stories are spread across an entire book, answering them requires global context rather than only the supplied window. The argument is carried by two comparisons: readers versus non-readers on the same questions, which isolates the contribution of long-term background knowledge, and humans versus LLMs under matched context windows, which tests whether models can substitute scale or reasoning for that knowledge. A secondary mechanism is the generative evaluation protocol, which grades responses by whether they cover extracted bonus points and penalizes factual or logical defects, because it correlates with human judgment better than token- or embedding-based metrics.","core_discovery":"The central discovery the paper reports is that Theory of Mind about a character is inseparable from knowledge of that character's personal history: readers who had read a novel beat non-readers by about 21 percent on the same multiple-choice questions about the characters' beliefs, intentions, emotions, and desires, while GPT-4o, o1, and DeepSeek-R1 all performed below the readers and roughly at or below the non-reader level. The paper also finds that human accuracy rises when people are given a longer story window, but LLM accuracy stays flat as the window grows, and that LLMs drop most on indirect questions whose answers are not literally present in the provided plot. The authors interpret this as evidence that current models rely on local textual clues or memorized story knowledge rather than the integrated, global understanding that human Theory of Mind uses.","pith_inferences":["If the reader-versus-non-reader gap replicates in a larger sample, then current 'Theory of Mind' results on short narratives likely overstate machine social reasoning, because they never engage the long-term person-model component.","A natural extension the paper does not run: give non-reader humans or LLMs a written biography of the character before the same questions; if the gap closes, the binding constraint is access to background information, not the ability to integrate it over a long text.","Another testable extension is retrieval: measure whether supplying relevant background passages, rather than a longer window, changes LLM accuracy; the paper's stable-window result predicts that retrieval helps only if it provides the causal links models currently miss.","The visible chain-of-thought finding that 76 percent of thoughts recall relevant background suggests that even when models locate the right context, they often fail to convert it into correct answers, pointing to integration rather than knowledge access as the bottleneck."],"forward_implications":["Theory of Mind benchmarks built on short, self-contained narratives miss the global-background component that this paper argues is central to human mental-state inference.","State-of-the-art LLMs, including reasoning models with chain-of-thought, remain below human readers on these questions and do not improve when given longer story windows.","The human advantage appears specifically on indirect questions, where the answer requires knowledge outside the given plot window, and this is exactly where LLM performance drops most.","The flat LLM performance across context lengths may reflect reliance on memorized story content rather than genuine contextual integration, which the paper flags as a Clever Hans risk.","The paper's grading-style evaluation protocol, based on bonus-point coverage with penalty detection, correlates with human judgment better than ROUGE or BERTScore and offers a reusable metric for generative Theory of Mind QA."],"supporting_citations":[{"why":"ToMI, the template-based belief QA benchmark that this paper contrasts with and builds on.","marker":"(Nematzadeh et al., 2018)"},{"why":"Social IQa, a commonsense social reasoning benchmark that represents the short-narrative evaluation style the paper argues is insufficient.","marker":"(Sap et al., 2019)"},{"why":"HI-TOM, the higher-order Theory of Mind benchmark that shows the recursive-belief focus of prior tests.","marker":"(Wu et al., 2023)"},{"why":"OpenToM, which adds explicit personality traits and preferences to stories and serves as the closest prior contrast for the paper's global-context claim.","marker":"(Xu et al., 2024a)"},{"why":"Clever Hans analysis used by the paper to explain LLMs' stable performance across context lengths via memorized book knowledge.","marker":"(Shapira et al., 2024)"},{"why":"Precedent for collecting human baselines on a subset of evaluation questions, used to justify the human study design.","marker":"(Thai et al., 2022)"},{"why":"Source of the user-note-as-annotation-proxy method used to build the benchmark from reader comments.","marker":"(Wan et al., 2019)"},{"why":"Prior work on personality understanding of fictional characters during book reading that motivates the character-centered QA design.","marker":"(Yu et al., 2023)"}],"fun_headline_variants":["ToM about story characters requires reading the book, LLMs fall short","Novel readers beat LLMs on character mindset questions by 21 points","LLMs fail Theory of Mind on novel characters, readers do better","Context is key: humans beat AI on ToM when they know the story","Reading the novel boosts ToM accuracy 21%, not for LLMs"],"cache_read_input_tokens":24064,"weakest_assumption_plain":"The claim that long-term familiarity drives the human advantage rests on a small human study (eight participants, 150 self-reported reading assignments, no significance test on the roughly 21 percent gap), so if that gap does not survive a larger sample the central human-LLM comparison weakens.","fun_headline_variants_meta":{"raw":{"variants":["ToM about story characters requires reading the book, LLMs fall short","Novel readers beat LLMs on character mindset questions by 21 points","LLMs fail Theory of Mind on novel characters, readers do better","Context is key: humans beat AI on ToM when they know the story","Reading the novel boosts ToM accuracy 21%, not for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2270,"prompt_tokens":955,"completion_tokens":1315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1219}},"tokens_in":571,"tokens_out":1315,"duration_ms":8548,"temperature":1.0,"reasoning_tokens":1219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:21:18.807957+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A preregistered replication with a larger, balanced sample that finds no reader-versus-non-reader accuracy gap on the same 150 questions would falsify the central claim; a second decisive result would be non-readers matching readers once given a written character-background summary, which would show the effect is information access rather than memory integration.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Clever Hans analysis used by the paper to explain LLMs' stable performance across context lengths via memorized book knowledge."}],"review_version":1}