{"id":"c1e4b9c6-7888-4b78-aca1-49d696bb2edd","arxiv_id":"2506.13641","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A temporal knowledge graph of character mental states is proposed to improve LLM performance on a new ToM benchmark, but the evaluation is circular because the same LLM generated the benchmark and the hints.","lead":"This paper introduces LitCharToM, a benchmark of multiple-choice questions about characters' beliefs, desires, emotions, and intentions in classic novels, and proposes EvolvTrip, a temporal knowledge graph of character mental states to improve LLM performance on such questions. The reported gains are likely inflated because the same AI model generated both the questions and the triples that are provided as hints.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The w Triple condition leaks the answer: question generation and triples come from the same GPT-4o process, and the correct answer appears almost verbatim in the supplied triples.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: the triples used for evaluation are the same triples used to generate the questions, so the w Triple condition leaks the correct answer. My reading of the appendix confirms this: Table A5's prompt takes 'Previous Character Relation Triples' as an explicit input, and the OOD example in Table A7 shows the gold triples mirroring the correct option. The central claim of the paper is that temporal ToM graphs improve LLM reasoning; to establish that, the additive value of triples must be measured against questions that were not constructed from those triples. The paper does not do this. Additionally, the abstract says 'consistently enhances performance' but multiple entries in Tables 1 and 2 degrade with triples, which further weakens the claim. The full text also contains an unrelated MedNLI/BioBERT table in the introduction, indicating a compilation error that, while not central to the scientific claim, reduces confidence in the manuscript's overall care. The benchmark resource itself may still be useful if independently validated, but the method's effectiveness is unproven under the current evaluation design. I therefore uphold the reader's REJECT verdict, with the concrete test above serving as the decisive check for whether a future revision could restore the claim.","tokens_in":19321,"tokens_out":3982,"duration_ms":40760,"concrete_test":"Regenerate the benchmark questions using the question-generation prompt with the 'Previous Character Relation Triples' input removed (or, better, use independently human-authored questions), then rerun the standard versus 'w Triple' comparisons of Tables 1 and 2. If the w Triple advantage disappears or shrinks substantially, the reported gains are leakage. As a complementary check, provide only the triples (without narrative context) to the same LLMs and measure their accuracy; if accuracy here is far above the no-triple baseline, the triples already encode the answers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported gains of EvolvTrip are not trustworthy because the evaluation protocol is circular. Table A5, the prompt for multiple-choice question generation, explicitly includes 'Previous Character Relation Triples' as an input and instructs GPT-4o to ground the question in those triples. The same triples are then fed to answer models in the 'w Triple' condition. The Appendix C example makes the leakage concrete: the correct answer to the Siddhartha question, 'To seek a deeper understanding of Atman beyond words and rituals', is essentially a paraphrase of the supplied triples '(Siddhartha, Desires, to truly know Atman beyond words and rituals)' and '(Siddhartha, IntendsToSeek, further understanding beyond traditional teachings)'. Thus the enhanced condition is closer to an answer-copying task than to a test of whether temporal theory-of-mind graphs improve narrative reasoning. The 'consistently enhances' claim is also contradicted by the paper's own tables: Table 1 shows Qwen2.5-72B-Ins at 64.47 to 64.39 and Llama3.3-70B-Ins at 63.60 to 63.53, while Table 2 shows Qwen3-32B at 64.41 to 64.15. Without a question set that is independent of the triples, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LitCharToM, a multiple-choice Theory-of-Mind (ToM) benchmark of 2,539 questions derived from 20 classic novels, and EvolvTrip, a temporal knowledge graph of character mental states expressed as subject-predicate-object triples. Experiments compare standard prompting with prompting augmented by EvolvTrip triples across a range of LLMs in two context-length settings, plus an out-of-distribution fine-tuning study. The authors claim that EvolvTrip consistently improves accuracy, particularly for smaller models and extended contexts, and that the benchmark captures long-narrative ToM reasoning.","tokens_in":19569,"tokens_out":5317,"duration_ms":47145,"significance":"If the evaluation were valid, LitCharToM would address an underexplored long-context ToM setting, and EvolvTrip's temporal-triple augmentation would be a practically useful contribution; the paper also releases data and code and reports human verification of a sample of the generated questions and triples. However, the central claim is not established because the question-generation prompt receives the same triples that are later supplied as answer-augmenting context, creating an answer-leakage loop. The reported 'consistent' improvements are also contradicted by several decreases in the paper's own tables. The out-of-distribution experiments inherit the leakage through their training and test protocols. As a result, the contributions currently amount to a dataset whose evaluation is contaminated by the condition it is meant to test.","major_comments":[{"comment":"The multiple-choice question generation prompt explicitly lists 'Previous Character Relation Triples' as an input field and instructs GPT-4o to ground each question in those triples, and Section 4.1 states that the 'w Triple' condition feeds 'relevant mental state triples' to the answer model. The same triples therefore both shape the correct answer and are given to the model at test time. The Appendix C Siddhartha example makes this concrete: the correct option B, 'To seek a deeper understanding of Atman beyond words and rituals,' is a near-verbatim restatement of the supplied triples '(Siddhartha, Desires, to truly know Atman beyond words and rituals)' and '(Siddhartha, IntendsToSeek, further understanding beyond traditional teachings)'. The reported gains are thus attributable to answer leakage rather than to EvolvTrip's temporal structure.","section":"§3.2, Table A5, Appendix C"},{"comment":"The abstract's claim that EvolvTrip 'consistently enhances performance' is contradicted by the paper's own numbers. In Table 1, Qwen2.5-72B-Instruct declines from 64.47 to 64.39 and Llama3.3-70B-Instruct from 63.60 to 63.53; in Table 2, Qwen3-32B declines from 64.41 to 64.15. Even under the paper's evaluation protocol, the effect is not consistent, so the headline claim needs qualification or the analysis needs correction.","section":"§5.1, Tables 1–2"},{"comment":"The out-of-distribution fine-tuning experiments inherit the same leakage. The training examples in Table A8 include the relevant triples followed by the correct answer, and the 'w Triple' test condition provides the same triples as input. Additionally, the OOD questions are generated with the same prompt in Table A5, so the OOD questions are also grounded in the triples that are later supplied. Consequently, the fine-tuning gains in Table 3 do not demonstrate transfer of EvolvTrip reasoning; they may reflect learning to select the option that matches the provided triples.","section":"§5.3, Tables A8–A9"}],"minor_comments":[{"comment":"The manuscript contains an unrelated table and text about BERT, BioBERT, and MedNLI (including 'DistilledReasoning' and 'T5-CoT' fragments) immediately after the Introduction. This appears to be a copy-paste error and must be removed.","section":"After Section 1"},{"comment":"There are several typos: 'charaters' in Section 1, 'corss-plot' in Section 1, and 'w / Triple' in the Table 1 caption.","section":"Throughout"},{"comment":"The human verification description is vague: it says a 'substantial portion' of questions was human-checked but does not state the fraction; please specify the exact proportion of questions and triples verified.","section":"Appendix A.2"},{"comment":"No statistical significance tests, confidence intervals, or variance estimates are reported for the accuracy differences; given the small gaps in some rows, these would help assess whether the improvements are meaningful.","section":"§4.2 and Tables 1–3"},{"comment":"The Limitations section acknowledges GPT-4o bias in triple extraction but does not mention the circularity between question generation and the 'w Triple' evaluation condition; this should be disclosed and addressed.","section":"Limitations"}],"recommendation":"reject","confidential_remarks":"The manuscript shows signs of insufficient proofreading, including an unrelated MedNLI table in the body. More importantly, the central evaluation is circular: the same triples that generate the questions and their correct answers are later provided to the answer models in the 'w Triple' condition. Even if the benchmark might be useful after substantial revision, the current version does not support the main claim. I recommend rejection, though a revised version that regenerates questions independently of the triples or evaluates with triples withheld could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper builds a nice new resource but the headline claim about EvolvTrip doesn't survive contact with its own evaluation. The 'w Triple' condition leaks the answer, so the reported gains are not interpretable.\n\nWhat's actually new: LitCharToM is a 2,539-question multiple-choice benchmark over 20 classic novels, with questions in belief/emotion/intention/desire and reported human verification above 92%. EvolvTrip extracts character mental-state triples via GPT-4o and links them across plot segments with temporal tags. That temporal linking is a genuine addition over prior KG-prompting work like COMET or ATOMIC, and the dataset could be useful for narrative ToM research if independently validated.\n\nWhere it falls apart: the evaluation. Table A5 shows the question-generation prompt receives 'Previous Character Relation Triples' as input, and the same triples are fed to answer models in the 'w Triple' condition. Appendix C makes the leak concrete: the correct Siddhartha answer is a near-verbatim paraphrase of the supplied triples. The 'consistent enhancement' claim is also contradicted by the paper's own tables: Qwen2.5-72B and Llama3.3-70B both drop slightly with triples in Table 1, and Qwen3-32B drops in Table 2. That pattern is consistent with leakage plus noise, not with a robust method. There's also a stray MedNLI table near the start that has no relation to the rest of the paper—an apparent copy-paste artifact that should have been caught.\n\nThe limitations section is honest about some issues but doesn't acknowledge the circularity. The core idea—explicitly tracking temporal mental-state triples—remains plausible, but this paper doesn't establish it.\n\nWho it's for: people building narrative AI benchmarks or KG-prompting methods. The benchmark could be a useful starting point after careful re-validation. The method paper needs a non-circular evaluation before its claims can be trusted.\n\nRecommendation: send to peer review with a request for major revision, mainly to fix the evaluation protocol. A referee should ask for question generation independent of the triples, or at least a held-out set where the triples are not used to write the questions, plus a per-model report of gains without cherry-picking. If the authors can't do that, the submission shouldn't be accepted.","headline":"A useful new benchmark, but the paper's headline claim about EvolvTrip is invalidated by a leak in its own evaluation design.","tokens_in":20112,"tokens_out":1947,"would_cite":false,"duration_ms":18858,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding temporal theory-of-mind triples to narrative context consistently raises LLM accuracy on character reasoning.","keywords":["theory of mind","temporal knowledge graph","narrative comprehension","large language models","multiple-choice QA","character mental states","long-context reasoning","literary benchmark"],"falsifier":"Regenerate the LitCharToM questions and correct answers without providing the previous character triples to the question-generation model, then re-run the standard versus triple-augmented comparison on the same models; if the accuracy gap largely disappears, the claimed benefit of the triples is not supported.","tokens_in":19133,"feed_emoji":"🧠","tokens_out":3985,"duration_ms":39169,"temperature":0.7,"pith_summary":"The paper sets out to show that large language models reason better about fictional characters when the characters' evolving mental states—beliefs, desires, intentions, and emotions—are made explicit as time-stamped triples in a knowledge graph. To test this, the authors build LitCharToM, a 2,539-question multiple-choice benchmark drawn from twenty classic novels, and EvolvTrip, a pipeline that generates such triples from plot summaries, dialogues, thoughts, and actions. Across fourteen models, from 7B to GPT-4o, adding the triples to the prompt raises accuracy in nearly every case, with the largest gains for smaller models and for emotion questions. The paper reads the result as evidence that explicit temporal representation of mental states helps LLMs track character psychology in long narratives.","feed_headline":"Mind-state triples boost LLM scores on literary characters","feed_subtitle":"A time-tagged knowledge graph of beliefs, desires, and intentions improves model accuracy across scales.","key_machinery":"The central object is the temporal knowledge graph of mental-state triples. Each triple has the form (character, predicate, object) with predicates restricted to four theory-of-mind dimensions, and each triple carries the number of the plot segment in which the mental state is active; inter-plot links connect the same character across segments so that evolving or contradicting states can be updated. The triples are generated by GPT-4o from plot summaries, scenario descriptions, dialogues, bracketed thoughts, and parenthesised actions, with previously extracted triples for the character fed back in to maintain consistency. The claim is that this graph supplies the model with explicitly resolved perspective-aware states that would otherwise have to be inferred from raw text, reducing the reasoning burden in long contexts.","core_discovery":"The authors claim that a perspective-aware temporal knowledge graph of character mental states improves LLM theory-of-mind reasoning in extended literary narratives. The graph represents each mental state as a (character, predicate, object) triple—predicates such as BelievesAbout, DesiresFor, FeelsTowards, and IntendsTo—tagged with the plot segment in which it arises, and links instances of the same character across plot segments so that updates and contradictions reflect psychological development. Adding these triples to the prompt alongside the narrative context improves multiple-choice accuracy on their LitCharToM benchmark for almost all tested models and settings, including the harder setting where previous plot summaries are included, and the gains transfer to five withheld books when models are fine-tuned to produce triples before answering.","pith_inferences":["The evaluation may overstate the gain: the same GPT-4o that generated the triples also wrote the questions and correct answers, so the 'w Triple' condition could be giving models the answer key in disguised form; a cleaner test would generate questions without access to the triples.","The approach is a form of context compression: triples condense the relevant mental states, so gains might replicate with any high-quality summary; testing against plot summaries of matched length would isolate the effect of the structured form.","The temporal tagging suggests a natural extension to tracking how a character's beliefs change in response to specific events, which could be evaluated with questions about state transitions rather than static states."],"forward_implications":["If the claim holds, supplying structured mental-state triples is a practical, prompt-level way to improve LLM performance on narrative comprehension without retraining.","Smaller models benefit disproportionately, so the method could partially close the capability gap with larger models on social reasoning.","The benchmark itself provides a reusable evaluation for future work on long-context theory-of-mind reasoning.","Fine-tuning small models to emit triples before answering transfers to books not seen in training, suggesting the representation is learnable."],"supporting_citations":[{"why":"Supplies the CoSER dataset of 81 literary works with plots, dialogues, thoughts, and actions that LitCharToM is built from.","marker":"[Wang et al., 2025]"},{"why":"CharToM is the prior benchmark that EvolvTrip extends by moving from brief vignettes to long narrative context.","marker":"[Zhou et al., 2025]"},{"why":"EnigmaToM provides the neural-knowledge-base-of-entity-states approach that EvolvTrip adapts to character mental states.","marker":"[Xu et al., 2025]"},{"why":"GPT-4o is the model used to generate the multiple-choice questions, the correct answers, and the mental-state triples.","marker":"[OpenAI, 2024]"},{"why":"COMET supplies the baseline knowledge-graph construction approach that EvolvTrip contrasts with by adding temporal depth.","marker":"[Bosselut et al., 2019]"},{"why":"ATOMIC models if-then social events and is cited as prior knowledge representation that lacks temporal character tracking.","marker":"[Sap et al., 2019a]"}],"fun_headline_variants":["Temporal character graph boosts LLM literary reasoning","Time-tagged mind triples help LLMs follow characters","Graphing character beliefs lifts LLM narrative scores","Character mind-graph enhances LLM story understanding","LLMs read characters better with temporal knowledge graph"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported improvements assume that the mental-state triples are an independent source of knowledge that helps LLMs reason; but the same model that generated the triples also generated the questions and their correct answers, so the triple-augmented evaluation may reveal leakage of the ground truth rather than genuine reasoning gains.","fun_headline_variants_meta":{"raw":{"variants":["Temporal character graph boosts LLM literary reasoning","Time-tagged mind triples help LLMs follow characters","Graphing character beliefs lifts LLM narrative scores","Character mind-graph enhances LLM story understanding","LLMs read characters better with temporal knowledge graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1309,"prompt_tokens":917,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":533,"tokens_out":392,"duration_ms":4347,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:27:55.672056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the LitCharToM questions and correct answers without providing the previous character triples to the question-generation model, then re-run the standard versus triple-augmented comparison on the same models; if the accuracy gap largely disappears, the claimed benefit of the triples is not supported.","supporting_citations":[],"review_version":1}