{"id":"b89ce9af-8592-4b92-b70e-26c8b45bcf47","arxiv_id":"2608.01598","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PICTURE prompts LLMs to generate a character's lack of knowledge in free-form chain-of-thought, improving false-belief reasoning by 7.3% over prior prompting methods.","lead":"The paper proposes a new way to prompt large language models for Theory of Mind tasks: instead of hiding story events the character doesn't know about, the model is asked to state, step by step, what the character does not know. The method improves accuracy on false-belief questions by about 7% on average across several benchmarks and models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing 'knows-only' free-form control leaves the causal role of generated lack-of-knowledge statements untested.","rationale":"The reader's weakest_assumption focuses on the reliability of generated lack-of-knowledge statements, backed only by manual inspection of 50 examples. That is a real concern, but I think the more load-bearing gap is the missing control that isolates the lack-of-knowledge instruction from free-form, non-hiding perspective-taking. The reader's rationale does mention the absent 'knows-only' baseline, so there is partial agreement, but it is not the stated weakest assumption. The Section 3 preliminary study gives some support for the causal role of explicit lack statements, and the ablations in Tables 4 and 8 show that event hiding is not the source of the gains and that CoT alone is insufficient. However, none of these experiments rule out the possibility that a free-form perspective-taking prompt that only asks about knowledge would be just as effective as PICTURE. That is why the central claim remains conditional rather than established. Since the reader already assigned CONDITIONAL, my concern does not move the verdict; it strengthens the reason for requiring the additional experiment.","tokens_in":29841,"tokens_out":5799,"duration_ms":71176,"concrete_test":"Run a 'knows-only free-form' control using the exact PICTURE pipeline, changing only the Step 1 instruction from 'What events does and does not {name} know about?' to 'What events does {name} know about?' (with no mention of lack/does-not-know), keeping the same rules, 'Think step by step' trigger, and answer-extraction prompt. Evaluate on the same four LLMs and three datasets as Table 1 (or at least GPT-3.5-Turbo and GPT-4o on BigToM/ToMi/FANToM). If this control matches or exceeds PICTURE on false-belief accuracy, the causal role of explicitly revealing lack of knowledge is not supported; if PICTURE remains significantly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that explicitly generating a character's lack of knowledge causes LLMs to inhibit responses to unknown events, and that this is why PICTURE outperforms baselines. The evidence does not currently isolate that component. Compared with SimToM, PICTURE varies two things at once: it drops event hiding (using free-form CoT instead) and it instructs the model to state what the character does not know. The reported ablations do not separate these factors. SimToM + CoT (Table 4) keeps event hiding, so it cannot show what would happen if the model generated only a character's knowledge in free-form without lack statements. PICTURE w/ Event Hiding (Table 8) keeps the lack-of-knowledge instruction and then adds hiding, so it likewise does not measure the contribution of lack statements alone. The Section 3 preliminary study does show a gain when oracle 'does not know that' phrases are inserted, but that is an upper-bound manipulation, not the actual PICTURE mechanism. In PICTURE the lack statements are generated by the same model that answers; their accuracy is checked only by manual inspection of 50 ToMi examples (Section 6.2, Appendix B.5). Without a 'knows-only free-form' baseline—same rules, same CoT, same answer extraction, but the Step 1 prompt asks only what the character knows and never asks for lack of knowledge—the 7.3% improvement may be attributable to free-form perspective-taking generally, not to the hypothesized lack-of-knowledge inhibition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PICTURE, a two-step prompting method for theory-of-mind (ToM) tasks. Instead of the prevailing event-hiding perspective-taking approach, PICTURE asks an LLM to generate free-form chain-of-thought reasoning that explicitly states both what a target character knows and what the character does not know, and then extracts the final answer from that reasoning. The authors first test the hypothesis that making a character's lack of knowledge explicit improves false-belief accuracy by augmenting Percept-ToMi stories with oracle 'does not know that' phrases. They then evaluate PICTURE on BigToM, ToMi, FANToM, OpenToM, ToMBench, and Hi-ToM across multiple backbone LLMs, reporting that PICTURE outperforms existing zero-shot prompting baselines by an average of 7.3% on false-belief questions. Additional experiments include ablations (SimToM+CoT and PICTURE w/ Event Hiding), prompt-robustness tests, comparisons with recent event-hiding methods, and manual analysis of generated perspective-taking outputs.","tokens_in":30197,"tokens_out":5074,"duration_ms":62267,"significance":"If the result holds, the paper makes a useful conceptual and empirical contribution: it reframes perspective-taking for LLM ToM as an inhibitory-control problem rather than an event-hiding problem, and it provides a simple prompt that yields consistent gains across several benchmarks and model families. The evaluation is unusually broad, with four primary backbones plus three additional recent LLMs, and the paper reports standard deviations, multiple seeds, and significance tests. The oracle-based preliminary study is a nice hypothesis-validation step, and the prompt-robustness study in Appendix B.7 is a genuine strength. The main unresolved issue is that the causal role of the specific 'does not know' component is not isolated by the current ablations; the reported gains could in principle come from free-form perspective-taking generally rather than from the explicit lack-of-knowledge statements.","major_comments":[{"comment":"The central claim is that explicitly generating a character's lack of knowledge causes the inhibition that drives PICTURE's gains. The current ablations do not isolate that component. SimToM + CoT differs from PICTURE in two ways: it keeps event hiding and it does not ask for lack-of-knowledge statements. PICTURE w/ Event Hiding keeps the lack-of-knowledge instruction and adds hiding, so it measures the cost of event hiding, not the contribution of lack statements. Neither is a free-form perspective-taking baseline that uses the same rules, the same CoT trigger, and the same two-step answer extraction but asks only what the character knows (e.g., 'What events does {character} know about?') and never requests lack knowledge. Without such a control, the 7.3% average improvement over baselines could be due to free-form perspective-taking generally, to the explicit knowledge-state decomposit","section":"§6.4, Table 4; Appendix B.3, Table 8"},{"comment":"The reliability of the generated lack-of-knowledge statements is load-bearing for PICTURE's effectiveness, but the only direct evidence is manual inspection of 50 sampled ToMi questions. The error analysis reports that in 80% of 50 failures the model produced an incorrect perspective, which shows that the method's success is sensitive to this intermediate step. The authors acknowledge this limitation, but the conclusion that PICTURE generates characters' lack of knowledge 'as expected' rests on a small, manually inspected sample, and the reported 2% versus 18% misclassification rates in §6.2 are not accompanied by confidence intervals or a larger automated evaluation. I would like to see a larger-scale, preferably automated, evaluation of the intermediate statements (e.g., using ToMi/Percept-ToMi perceiver annotations to compute precision/recall of the generated knows and does-not-know s","section":"§6.2; Appendix B.5; Limitations (3)"}],"minor_comments":[{"comment":"The preliminary study reports large accuracy differences but gives no number of examples, error bars, or significance test. Please state the sample size and add significance testing or confidence intervals.","section":"§3, Figure 2"},{"comment":"The 'Vanilla w/ Rules' and 'CoT w/ Rules' rows are potentially relevant controls, but they are only footnoted and only reported for GPT-3.5-Turbo on ToMi. Clarify exactly what these prompts contain (rules only, or also knowledge-state generation) and consider discussing them in the main ablation section, since they bear on the role of the hand-crafted rules.","section":"Appendix B.8, Table 16"},{"comment":"The prompts differ between GPT and Llama-2-chat, especially in the answer-extraction step for BigToM. Please state explicitly whether any of the reported cross-model differences could be affected by these prompt differences, and make the full prompt files available in the repository.","section":"Appendix C"},{"comment":"There are several spacing/formatting errors such as 'PICTUREoutperforms' and 'PICTUREachieves' that should be corrected in the camera-ready version.","section":"Abstract and text"}],"recommendation":"major_revision","confidential_remarks":"The missing knows-only free-form control is a straightforward but important addition. The paper's broad evaluation and honest limitations section make it a reasonable candidate for acceptance after the causal-role question is addressed. I would not reject on the current evidence, but the central claim is not fully supported until the 'does not know' component is isolated from free-form perspective-taking generally."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, thorough prompting paper and it deserves a serious referee. The headline result—making a character's lack of knowledge explicit in free-form CoT outperforms event-hiding baselines by roughly 7 points on false-belief tasks—is consistent across three benchmarks and several model families, and the paper includes enough ablations and robustness checks that I take the empirical claim seriously.\n\nThe genuinely new thing is the instruction to generate 'does not know' statements as part of the reasoning, without any event-hiding step. That's a simple idea that hasn't been tested directly before, and the paper makes a good conceptual case (Appendix A) that event hiding sidesteps inhibitory control rather than exercising it. The evaluation is broad: BigToM, ToMi, FANToM, plus OpenToM, ToMBench, Hi-ToM; four main backbones and three more recent ones. The ablations (SimToM+CoT, PICTURE w/ Event Hiding) and the prompt-variant study are useful. The code is available.\n\nThe soft spot is the one the stress-test note flags, and it's real: the mechanism is not cleanly isolated. Compared with SimToM, PICTURE changes two things at once—it drops event hiding for free-form CoT, and it instructs the model to state what the character does not know. No baseline keeps the free-form CoT and the same rules but only asks for what the character knows, without ever asking for lack of knowledge. The preliminary study inserts oracle 'does not know' phrases, which is an upper bound, not the actual generation. The manual check of 50 sampled ToMi questions suggests the generated statements are usually correct, but it's a small sample, and the error analysis says 80% of 50 failures had an incorrect perspective—so the intermediate step is far from perfect. These gaps are addressable with additional experiments, and the paper's own Limitations section acknowledges the small-scale check.\n\nMinor point: the hand-crafted knowledge-update rules are domain assumptions, shared with SimToM, not derived from data. That's not a flaw per se, but it limits how far the method can be claimed to generalize.\n\nWho's it for: anyone working on ToM prompting or evaluating LLM social reasoning. The method itself is cheap and easy to adopt. I'd want the missing 'knows-only' control before I'd sign off on the causal claim, but the paper is in good shape for a serious referee.","headline":"Solid, thorough prompting paper with consistent gains; the causal role of the 'does not know' generation is plausible but not cleanly isolated from the free-form CoT change.","tokens_in":30658,"tokens_out":2422,"would_cite":true,"duration_ms":26621,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Making a character's lack of knowledge explicit during reasoning improves LLM accuracy on false-belief questions by an average of 7.3%.","keywords":["theory of mind","large language models","false belief","chain-of-thought prompting","perspective-taking","inhibitory control","question answering"],"falsifier":"Run PICTURE on a large sample of ToMi or FANToM questions and automatically compare every generated lack-of-knowledge statement against ground-truth perceiver annotations (as Percept-ToMi provides): if many statements are wrong while accuracy stays high, the explicit ignorance sentences are not the operative mechanism. Then run PICTURE unchanged on stories where ignorance comes from deception or unspoken intentions rather than leaving a location: if accuracy collapses to baseline, the hand-written spatial rules, not the reveal-ignorance principle, are carrying the result.","tokens_in":29767,"feed_emoji":"🧠","tokens_out":12670,"duration_ms":126374,"temperature":0.7,"pith_summary":"This paper claims that large language models can answer false-belief questions more accurately when their intermediate reasoning is allowed to state, in plain language, what a character does not know, instead of being forced to filter the story down to only what the character does know. The proposed prompting method, PICTURE, first asks the model to enumerate which events the character knows and which it does not, then to think step by step to the answer, with no strict output format on the reasoning step. Across three theory-of-mind benchmarks and four LLMs, PICTURE outperforms existing zero-shot prompting methods, including the event-hiding baselines SimToM and PercepToM, by an average of 7.3 percentage points on false-belief questions. The paper's central bet is that the bottleneck in machine theory of mind is not access to the character's perspective but the failure to inhibit the pull of the true state of the world, and that explicit statements of ignorance supply that inhibition.","feed_headline":"Revealing characters' ignorance lifts LLM false-belief accuracy 7.3%","feed_subtitle":"Stating what a character does not know beats event-hiding and plain-CoT prompts on three benchmarks.","key_machinery":"The central mechanism is the generated lack-of-knowledge statement inside free-form Chain-of-Thought. Instead of removing events unknown to a character (event hiding), PICTURE keeps the full story visible and prompts the model to write out which events the character does not know—'Liam does not know that Owen moved the radish to the bottle'—before continuing with 'Think step by step' to the answer. These explicit ignorance statements act as inhibition hints: they name the events the model must keep from driving its answer. The two-step structure (perspective-taking that generates lack of knowledge, then answer extraction from the generated reasoning) avoids the rigid output formats—JSON arra","core_discovery":"The paper establishes two claims. A preliminary study on Percept-ToMi shows that augmenting stories with explicit statements of a character's lack of knowledge—'Liam does not know that Owen moved the radish to the bottle'—raises false-belief accuracy substantially without hurting true-belief accuracy. PICTURE turns this into a prompting method: the model generates, in free-form Chain-of-Thought with no event removal, both what the character knows and what it does not know, then reasons step by step to the answer. Across BigToM, ToMi, and FANToM with four LLMs, PICTURE beats all zero-shot baselines, including event-hiding SimToM and PercepToM, by an average of 7.3% on false-belief questions.","pith_inferences":["My inference: the reveal-ignorance move is a general recipe for suppression tasks, not only narrative theory of mind—questions where a system must not use information it has been given (negation, privacy-respecting answer generation, stale-state retrieval) may benefit from the same explicit 'X does not know / X is not relevant' step.","My inference: the headline average hides a steep model-size gradient, with the largest gains on the smallest models; the method may be raising the floor of weak reasoners more than the ceiling of strong ones, which is where deployment value would concentrate.","My inference: the causal story would be settled by an automated audit—comparing every generated ignorance statement against oracle perceiver annotations at scale—which would reveal how much of the gain survives when the generated statements are only partially correct."],"forward_implications":["LLM theory-of-mind failures look like an inhibition problem, not a knowledge problem: a model can know the character's perspective and still answer from reality unless the reasoning explicitly names what the character does not know.","Event hiding, the dominant perspective-taking strategy, is not necessary; keeping the full story visible and revealing ignorance outperforms filtering and sidesteps the reasoning cost of strict output formats.","The method is zero-shot and tool-free, so it applies directly to any instruction-following model; the paper's additional runs on newer LLMs show the gains persist as the underlying models change.","The rules written into the prompts (once a character leaves a location, later events there are unknown to them) determine what 'does not know' means, and the paper finds the largest gains exactly on location-based questions, with smaller gains where questions require multi-hop or attitude reasoning."],"supporting_citations":[{"why":"Introduces SimToM, the event-hiding perspective-taking baseline that PICTURE is designed to surpass, and supplies the evaluation setup and prompts used for the comparison.","marker":"Wilf et al., 2024"},{"why":"Introduces PercepToM and the Percept-ToMi dataset whose ground-truth perceiver annotations are used in the preliminary study that validates the lack-of-knowledge hypothesis.","marker":"Jung et al., 2024"},{"why":"Provides zero-shot Chain-of-Thought prompting, the source of the 'Think step by step' trigger and the two-stage reason-then-extract structure used by PICTURE.","marker":"Kojima et al., 2022"},{"why":"Supplies the psychological notion of inhibitory control that the paper uses to define and motivate 'inhibition' of responses to events unknown to a character.","marker":"Carlson and Moses, 2001"},{"why":"Evidence that strict output format constraints degrade LLM reasoning performance, the rationale for PICTURE's free-form, no-event-hiding design.","marker":"Tam et al., 2024"},{"why":"Supplies the BigToM benchmark of naturalistic narratives used to evaluate false-belief and true-belief questions.","marker":"Gandhi et al., 2023"},{"why":"Supplies the ToMi benchmark of templated multi-location stories, the main dataset for the case study and error analysis.","marker":"Le et al., 2019"},{"why":"Supplies the FANToM benchmark of multi-party conversations used to test PICTURE when characters leave and rejoin a dialogue.","marker":"Kim et al., 2023"},{"why":"Supplies the OpenToM benchmark with location, multi-hop, and attitude questions used to test generalization beyond physical-world false beliefs.","marker":"Xu et al., 2024"}],"fun_headline_variants":["Don't hide events, reveal ignorance: LLM ToM jumps 7.3%","Stating what characters don't know improves LLM reasoning 7.3%","Explicit 'does not know' beats event-hiding for LLM false beliefs","Free-form CoT with revealed ignorance outperforms event-hiding by 7.3%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"PICTURE's gains depend on the language model reliably writing truthful statements about what a character does not know during free-form reasoning; the paper's direct evidence for that reliability is a manual review of fifty sampled questions, and in four-fifths of the failure cases it inspected, the model had produced an incorrect perspective.","fun_headline_variants_meta":{"raw":{"variants":["Don't hide events, reveal ignorance: LLM ToM jumps 7.3%","Stating what characters don't know improves LLM reasoning 7.3%","Explicit 'does not know' beats event-hiding for LLM false beliefs","Free-form CoT with revealed ignorance outperforms event-hiding by 7.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001415,"raw_usage":{"total_tokens":5568,"prompt_tokens":780,"completion_tokens":4788,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":4697}},"tokens_in":524,"tokens_out":4788,"duration_ms":39440,"temperature":1.0,"reasoning_tokens":4697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:22:23.693772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PICTURE on a large sample of ToMi or FANToM questions and automatically compare every generated lack-of-knowledge statement against ground-truth perceiver annotations (as Percept-ToMi provides): if many statements are wrong while accuracy stays high, the explicit ignorance sentences are not the operative mechanism. Then run PICTURE unchanged on stories where ignorance comes from deception or unspoken intentions rather than leaving a location: if accuracy collapses to baseline, the hand-written spatial rules, not the reveal-ignorance principle, are carrying the result.","supporting_citations":[],"review_version":1}