PICTURE: Enhancing Theory-of-Mind in Large Language Models by Revealing, Not Hiding, Characters' Lack of Knowledge

Eojin Jeon; SangKeun Lee

REVIEW 2 major objections 4 minor 47 references

Making a character's lack of knowledge explicit during reasoning improves LLM accuracy on false-belief questions by an average of 7.3%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:22 UTC pith:3IGIBAB7

load-bearing objection Solid, thorough prompting paper with consistent gains; the causal role of the 'does not know' generation is plausible but not cleanly isolated from the free-form CoT change. the 2 major comments →

arxiv 2608.01598 v1 pith:3IGIBAB7 submitted 2026-08-03 cs.CL cs.AI

PICTURE: Enhancing Theory-of-Mind in Large Language Models by Revealing, Not Hiding, Characters' Lack of Knowledge

Eojin Jeon , SangKeun Lee This is my paper

classification cs.CL cs.AI

keywords theory of mindlarge language modelsfalse beliefchain-of-thought promptingperspective-takinginhibitory controlquestion answering

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that large language models can answer false-belief questions more accurately when their intermediate reasoning is allowed to state, in plain language, what a character does not know, instead of being forced to filter the story down to only what the character does know. The proposed prompting method, PICTURE, first asks the model to enumerate which events the character knows and which it does not, then to think step by step to the answer, with no strict output format on the reasoning step. Across three theory-of-mind benchmarks and four LLMs, PICTURE outperforms existing zero-shot prompting methods, including the event-hiding baselines SimToM and PercepToM, by an average of 7.3 percentage points on false-belief questions. The paper's central bet is that the bottleneck in machine theory of mind is not access to the character's perspective but the failure to inhibit the pull of the true state of the world, and that explicit statements of ignorance supply that inhibition.

Core claim

The paper establishes two claims. A preliminary study on Percept-ToMi shows that augmenting stories with explicit statements of a character's lack of knowledge—'Liam does not know that Owen moved the radish to the bottle'—raises false-belief accuracy substantially without hurting true-belief accuracy. PICTURE turns this into a prompting method: the model generates, in free-form Chain-of-Thought with no event removal, both what the character knows and what it does not know, then reasons step by step to the answer. Across BigToM, ToMi, and FANToM with four LLMs, PICTURE beats all zero-shot baselines, including event-hiding SimToM and PercepToM, by an average of 7.3% on false-belief questions.

What carries the argument

The central mechanism is the generated lack-of-knowledge statement inside free-form Chain-of-Thought. Instead of removing events unknown to a character (event hiding), PICTURE keeps the full story visible and prompts the model to write out which events the character does not know—'Liam does not know that Owen moved the radish to the bottle'—before continuing with 'Think step by step' to the answer. These explicit ignorance statements act as inhibition hints: they name the events the model must keep from driving its answer. The two-step structure (perspective-taking that generates lack of knowledge, then answer extraction from the generated reasoning) avoids the rigid output formats—JSON arra

Load-bearing premise

PICTURE's gains depend on the language model reliably writing truthful statements about what a character does not know during free-form reasoning; the paper's direct evidence for that reliability is a manual review of fifty sampled questions, and in four-fifths of the failure cases it inspected, the model had produced an incorrect perspective.

What would settle it

Run PICTURE on a large sample of ToMi or FANToM questions and automatically compare every generated lack-of-knowledge statement against ground-truth perceiver annotations (as Percept-ToMi provides): if many statements are wrong while accuracy stays high, the explicit ignorance sentences are not the operative mechanism. Then run PICTURE unchanged on stories where ignorance comes from deception or unspoken intentions rather than leaving a location: if accuracy collapses to baseline, the hand-written spatial rules, not the reveal-ignorance principle, are carrying the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

LLM theory-of-mind failures look like an inhibition problem, not a knowledge problem: a model can know the character's perspective and still answer from reality unless the reasoning explicitly names what the character does not know.
Event hiding, the dominant perspective-taking strategy, is not necessary; keeping the full story visible and revealing ignorance outperforms filtering and sidesteps the reasoning cost of strict output formats.
The method is zero-shot and tool-free, so it applies directly to any instruction-following model; the paper's additional runs on newer LLMs show the gains persist as the underlying models change.
The rules written into the prompts (once a character leaves a location, later events there are unknown to them) determine what 'does not know' means, and the paper finds the largest gains exactly on location-based questions, with smaller gains where questions require multi-hop or attitude reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

My inference: the reveal-ignorance move is a general recipe for suppression tasks, not only narrative theory of mind—questions where a system must not use information it has been given (negation, privacy-respecting answer generation, stale-state retrieval) may benefit from the same explicit 'X does not know / X is not relevant' step.
My inference: the headline average hides a steep model-size gradient, with the largest gains on the smallest models; the method may be raising the floor of weak reasoners more than the ceiling of strong ones, which is where deployment value would concentrate.
My inference: the causal story would be settled by an automated audit—comparing every generated ignorance statement against oracle perceiver annotations at scale—which would reveal how much of the gain survives when the generated statements are only partially correct.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid, thorough prompting paper with consistent gains; the causal role of the 'does not know' generation is plausible but not cleanly isolated from the free-form CoT change.

read the letter

Quick take: this is a solid, thorough prompting paper and it deserves a serious referee. The headline result—making a character's lack of knowledge explicit in free-form CoT outperforms event-hiding baselines by roughly 7 points on false-belief tasks—is consistent across three benchmarks and several model families, and the paper includes enough ablations and robustness checks that I take the empirical claim seriously.

The genuinely new thing is the instruction to generate 'does not know' statements as part of the reasoning, without any event-hiding step. That's a simple idea that hasn't been tested directly before, and the paper makes a good conceptual case (Appendix A) that event hiding sidesteps inhibitory control rather than exercising it. The evaluation is broad: BigToM, ToMi, FANToM, plus OpenToM, ToMBench, Hi-ToM; four main backbones and three more recent ones. The ablations (SimToM+CoT, PICTURE w/ Event Hiding) and the prompt-variant study are useful. The code is available.

The soft spot is the one the stress-test note flags, and it's real: the mechanism is not cleanly isolated. Compared with SimToM, PICTURE changes two things at once—it drops event hiding for free-form CoT, and it instructs the model to state what the character does not know. No baseline keeps the free-form CoT and the same rules but only asks for what the character knows, without ever asking for lack of knowledge. The preliminary study inserts oracle 'does not know' phrases, which is an upper bound, not the actual generation. The manual check of 50 sampled ToMi questions suggests the generated statements are usually correct, but it's a small sample, and the error analysis says 80% of 50 failures had an incorrect perspective—so the intermediate step is far from perfect. These gaps are addressable with additional experiments, and the paper's own Limitations section acknowledges the small-scale check.

Minor point: the hand-crafted knowledge-update rules are domain assumptions, shared with SimToM, not derived from data. That's not a flaw per se, but it limits how far the method can be claimed to generalize.

Who's it for: anyone working on ToM prompting or evaluating LLM social reasoning. The method itself is cheap and easy to adopt. I'd want the missing 'knows-only' control before I'd sign off on the causal claim, but the paper is in good shape for a serious referee.

Referee Report

2 major / 4 minor

Summary. The paper proposes PICTURE, a two-step prompting method for theory-of-mind (ToM) tasks. Instead of the prevailing event-hiding perspective-taking approach, PICTURE asks an LLM to generate free-form chain-of-thought reasoning that explicitly states both what a target character knows and what the character does not know, and then extracts the final answer from that reasoning. The authors first test the hypothesis that making a character's lack of knowledge explicit improves false-belief accuracy by augmenting Percept-ToMi stories with oracle 'does not know that' phrases. They then evaluate PICTURE on BigToM, ToMi, FANToM, OpenToM, ToMBench, and Hi-ToM across multiple backbone LLMs, reporting that PICTURE outperforms existing zero-shot prompting baselines by an average of 7.3% on false-belief questions. Additional experiments include ablations (SimToM+CoT and PICTURE w/ Event Hiding), prompt-robustness tests, comparisons with recent event-hiding methods, and manual analysis of generated perspective-taking outputs.

Significance. If the result holds, the paper makes a useful conceptual and empirical contribution: it reframes perspective-taking for LLM ToM as an inhibitory-control problem rather than an event-hiding problem, and it provides a simple prompt that yields consistent gains across several benchmarks and model families. The evaluation is unusually broad, with four primary backbones plus three additional recent LLMs, and the paper reports standard deviations, multiple seeds, and significance tests. The oracle-based preliminary study is a nice hypothesis-validation step, and the prompt-robustness study in Appendix B.7 is a genuine strength. The main unresolved issue is that the causal role of the specific 'does not know' component is not isolated by the current ablations; the reported gains could in principle come from free-form perspective-taking generally rather than from the explicit lack-of-knowledge statements.

major comments (2)

[§6.4, Table 4; Appendix B.3, Table 8] The central claim is that explicitly generating a character's lack of knowledge causes the inhibition that drives PICTURE's gains. The current ablations do not isolate that component. SimToM + CoT differs from PICTURE in two ways: it keeps event hiding and it does not ask for lack-of-knowledge statements. PICTURE w/ Event Hiding keeps the lack-of-knowledge instruction and adds hiding, so it measures the cost of event hiding, not the contribution of lack statements. Neither is a free-form perspective-taking baseline that uses the same rules, the same CoT trigger, and the same two-step answer extraction but asks only what the character knows (e.g., 'What events does {character} know about?') and never requests lack knowledge. Without such a control, the 7.3% average improvement over baselines could be due to free-form perspective-taking generally, to the explicit knowledge-state decomposit
[§6.2; Appendix B.5; Limitations (3)] The reliability of the generated lack-of-knowledge statements is load-bearing for PICTURE's effectiveness, but the only direct evidence is manual inspection of 50 sampled ToMi questions. The error analysis reports that in 80% of 50 failures the model produced an incorrect perspective, which shows that the method's success is sensitive to this intermediate step. The authors acknowledge this limitation, but the conclusion that PICTURE generates characters' lack of knowledge 'as expected' rests on a small, manually inspected sample, and the reported 2% versus 18% misclassification rates in §6.2 are not accompanied by confidence intervals or a larger automated evaluation. I would like to see a larger-scale, preferably automated, evaluation of the intermediate statements (e.g., using ToMi/Percept-ToMi perceiver annotations to compute precision/recall of the generated knows and does-not-know s

minor comments (4)

[§3, Figure 2] The preliminary study reports large accuracy differences but gives no number of examples, error bars, or significance test. Please state the sample size and add significance testing or confidence intervals.
[Appendix B.8, Table 16] The 'Vanilla w/ Rules' and 'CoT w/ Rules' rows are potentially relevant controls, but they are only footnoted and only reported for GPT-3.5-Turbo on ToMi. Clarify exactly what these prompts contain (rules only, or also knowledge-state generation) and consider discussing them in the main ablation section, since they bear on the role of the hand-crafted rules.
[Appendix C] The prompts differ between GPT and Llama-2-chat, especially in the answer-extraction step for BigToM. Please state explicitly whether any of the reported cross-model differences could be affected by these prompt differences, and make the full prompt files available in the repository.
[Abstract and text] There are several spacing/formatting errors such as 'PICTUREoutperforms' and 'PICTUREachieves' that should be corrected in the camera-ready version.

Circularity Check

0 steps flagged

No significant circularity: PICTURE is an empirical prompting method evaluated on external benchmarks; no fitted parameter or self-citation chain forces the results.

full rationale

The paper's derivation chain is experimental rather than analytic: it hypothesizes that making a character's lack of knowledge explicit helps LLMs inhibit responses to unknown events, tests this with oracle 'does not know that' insertions on Percept-ToMi, then proposes PICTURE, which prompts LLMs to generate such lack-of-knowledge statements in free-form CoT, and evaluates on BigToM, ToMi, FANToM, OpenToM, and other benchmarks. No parameter is fitted to the test data, and the main reported accuracies are external measurements of complete prompting methods. The PICTURE prompt does not define the target answer in terms of the output; it elicits intermediate reasoning that is then separately extracted. The closest concern is that the preliminary study's oracle statements can function as answer leakage for false-belief questions, since 'Liam does not know that Owen moved the radish to the bottle' entails the correct answer 'crate.' This is a validity/confound concern about the preliminary study, not a circular derivation: the main numerical claims about PICTURE are not obtained by that manipulation, and the paper's Limitations section (Section 7) explicitly acknowledges the manual 50-sample robustness check. There are no load-bearing self-citations and no uniqueness theorem imported from the authors. Therefore the central claim retains independent empirical content and no step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The method rests on several hand-crafted domain rules about how knowledge updates with location and action. These rules are stated in the prompts and are not derived from data. They are shared with SimToM, so they are not a new burden introduced by PICTURE, but they are assumptions the reader must accept. No free parameters (numerical fits) or invented entities are introduced.

axioms (4)

domain assumption A character knows about all events that they do.
Given in the ToMi/FANToM prompts (Appendix C.1, C.3), also used in SimToM. Establishes the epistemic rule that actions are known to the actor.
domain assumption If a character is in a location, they know about all other events that happen there.
Given as rule 2 in ToMi/FANToM prompts. Assumes co-presence implies full awareness of all events in that location.
domain assumption If a character leaves a location, they no longer know about any events that happen within it, unless they re-enter.
Given as rule 3 in ToMi/FANToM prompts. Central to false-belief inference, it encodes the update rule that makes the character's belief stale.
domain assumption If the story says the character is not aware of some events, the character knows only the rest of the events.
BigToM prompt (Appendix C.2) relies on the story's explicit unawareness statements to define the knowledge partition.

pith-pipeline@v1.3.0-daily-deepseek · 29500 in / 10776 out tokens · 99768 ms · 2026-08-05T00:22:23.693772+00:00 · methodology

0 comments

Cite this review

Pith. "Pith review of PICTURE: Enhancing Theory-of-Mind in Large Language Models by Revealing, Not Hiding, Characters' Lack of Knowledge." pith.science (2026). https://pith.science/paper/3IGIBAB7

@misc{pith2026260801598,
  author       = {Pith},
  title        = {Pith review of: PICTURE: Enhancing Theory-of-Mind in Large Language Models by Revealing, Not Hiding, Characters' Lack of Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IGIBAB7}},
  note         = {Machine review of arXiv:2608.01598}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

Simulating human-like Theory of Mind (ToM) has been a longstanding problem in natural language processing (NLP). To address this, existing works introduce a reasoning step of event hiding (a.k.a. perspective-taking), where events unknown to a character are removed before question answering. However, resorting to event hiding for ToM reasoning presents a performance degradation issue due to the strict output format constraints involved in event hiding. To mitigate this issue, we propose generating perspective-taking outputs as free-form explanations without event hiding, but this poses a notable yet underexplored challenge: LLMs need to inhibit responses to events unknown to characters, because the absence of event hiding exposes LLMs to these events throughout reasoning. To address this challenge, we hypothesize and empirically verify that LLMs can achieve such inhibition if a character's lack of knowledge about events is made explicit during reasoning. Based on this finding, we introduce PICTURE, a new prompting method that enables LLMs to generate a character's lack of knowledge within free-form Chain-of-Thought (CoT). Experimental results show that PICTURE outperforms existing prompting methods by an average of 7.3% on false-belief tasks.

discussion (0)

Reference graph

Works this paper leans on

47 extracted references · 9 canonical work pages

[1]

I’m sorry, I have to cut the conversation short. My delivery is arriving and I need to receive it. We’ll continue this chat soon though

Richard: "I’m sorry, I have to cut the conversation short. My delivery is arriving and I need to receive it. We’ll continue this chat soon though." - Richard knows about his own utterance
[2]

No worries, Richard. Catch you later

Kobe: "No worries, Richard. Catch you later." - Richard knows about Kobe’s utterance
[3]

Bye Richard!

Elena: "Bye Richard!" - Richard knows about Elena’s utterance
[4]

Bye, Kobe and Elena! Enjoy the rest of your conversation

Richard: "Bye, Kobe and Elena! Enjoy the rest of your conversation." - Richard knows about his own utterance
[5]

Now that it’s just us, Elena, what’s your biggest parenting challenge so far?

Kobe: "Now that it’s just us, Elena, what’s your biggest parenting challenge so far?" - Richard does not know about this conversation as he has left the location
[6]

InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 19794–19809

Perceptions to beliefs: Exploring precursory in- ferences for theory of mind in large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 19794–19809. Association for Computa- tional Linguistics. Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B....

2024
[7]

Oh, that’s a tough one. For me, it’s balancing my career responsibilities with being there for my child. But each challenge makes the journey more rewarding

Kobe: "Oh, that’s a tough one. For me, it’s balancing my career responsibilities with being there for my child. But each challenge makes the journey more rewarding." - Richard does not know about this conversation as he has left the location
[8]

I couldn’t agree more, Kobe. At the end of the day, all the struggles and challenges are more than worth it

Elena: "I couldn’t agree more, Kobe. At the end of the day, all the struggles and challenges are more than worth it." - Richard does not know about this conversation as he has left the location
[9]

Absolutely, Elena. Despite the ups and downs, I wouldn’t trade parenthood for anything

Kobe: "Absolutely, Elena. Despite the ups and downs, I wouldn’t trade parenthood for anything." - Richard does not know about this conversation as he has left the location
[10]

Hi guys, sorry about the interruption. I hope I didn’t miss too much of our chat. You both were talking about balancing challenges last I recall

Richard: "Hi guys, sorry about the interruption. I hope I didn’t miss too much of our chat. You both were talking about balancing challenges last I recall." - Richard knows about his own utterance. Based on the events, Richard does not know about the specific parenting challenge that Elena faced, which was maintaining a balance between wanting to protect ...
[14]

I think it’s maintaining a balance between wanting to protect and nurture my child but also letting him learn things by himself. You?

Elena: "I think it’s maintaining a balance between wanting to protect and nurture my child but also letting him learn things by himself. You?" - Richard does not know about this conversation as he has left the location
[19]

(Noah is now in the staircase.)

Noah enters the staircase. (Noah is now in the staircase.)
[20]

(Noah sees the slacks in the basket.)

The slacks are in the basket. (Noah sees the slacks in the basket.)
[21]

(Noah is no longer in the staircase, so he cannot see what happens inside.)

Noah exits the staircase. (Noah is no longer in the staircase, so he cannot see what happens inside.)
[22]

James moves the slacks to the treasure chest. (James moves the slacks to a new location, which Noah does not know about since he is no longer in the staircase.) Therefore, based on the information provided, Noah will not know where the slacks are located after he exits the staircase. Answer: Basket. (✓) Table 13: Additional qualitative results of Llama2-7...
[23]

- Emma knows that she is in the lounge

Emma entered the lounge. - Emma knows that she is in the lounge
[24]

- Emma knows that Alexander is in the lounge

Alexander entered the lounge. - Emma knows that Alexander is in the lounge
[25]

- Emma knows that the peas are in the crate

The peas is in the crate. - Emma knows that the peas are in the crate
[26]

- Emma does not know about this event because she exited the lounge before it happened

Alexander moved the peas to the bathtub. - Emma does not know about this event because she exited the lounge before it happened
[27]

- Emma knows that she loves the lime

Emma loves the lime. - Emma knows that she loves the lime
[28]

- Emma is no longer in the lounge, so she does not know about any events that happen in the lounge after she left

Emma exited the lounge. - Emma is no longer in the lounge, so she does not know about any events that happen in the lounge after she left. Based on the events that Emma knows about, she last knew that the peas were in the crate in the lounge. She exited the lounge before Alexander moved the peas to the bathtub. Answer: Emma thinks that Alexander searches ...
[29]

- Logan knows this

Logan likes the lemon. - Logan knows this
[30]

- Logan knows Lucas entered the crawlspace

Lucas entered the crawlspace. - Logan knows Lucas entered the crawlspace
[31]

- Logan knows Logan entered the crawlspace

Logan entered the crawlspace. - Logan knows Logan entered the crawlspace
[32]

- Logan knows the broccoli is in the drawer

The broccoli is in the drawer. - Logan knows the broccoli is in the drawer
[33]

- Logan knows Lucas exited the crawlspace

Lucas exited the crawlspace. - Logan knows Lucas exited the crawlspace
[34]

- Logan knows Logan moved the broccoli to the bucket

Logan moved the broccoli to the bucket. - Logan knows Logan moved the broccoli to the bucket
[35]

- Logan no longer knows about events in the crawlspace

Logan exited the crawlspace. - Logan no longer knows about events in the crawlspace
[36]

w/ Rules

Lucas entered the crawlspace. - Logan does not know about this event as he is not in the crawlspace Based on the events Logan knows about, he last knew that the broccoli was in the bucket in the crawlspace. Answer: Logan thinks that Lucas searches for the broccoli in the bucket. (✗) Table 14: Failure cases made by PICTUREon questions from the ToMi dataset...

2022
[39]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step by s...

work page
[42]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} This is the end of the main question. What events does and does not {name} know about? Output the answer for t...

work page
[45]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step b...

work page
[48]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step b...

work page
[51]

Noah exited the living room

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} What events does {name} know about? Only output the events according to the above rules, do not provide an explanation. SimToM (Inference) {perspective} You are {name}...

work page
[52]

A character knows about all events that they do
[53]

This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place

If a character is in a certain room/location, that character knows about all other events that happens in the room. This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place
[54]

Noah exited the living room

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. PercepToM (Question Answering) {perspective} You are {name}. Based on the above information, answer the following question: {question} Keep your answer concise, one sentence is enoug...

work page
[55]

A character knows about all utterances that they make

work page
[56]

If a character is in a certain room/location, that character knows about every utterance that occurs in the room

work page
[57]

Noah: Hi, Emma

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} What events does {name} know about? Only output the utterances according to the above rules, do not provide an explanation. SimToM (Inference) {perspective} You are...

work page
[1203]

Adele Diamond

Association for Computational Linguistics. Adele Diamond. 2013. Executive functions.Annual review of psychology, 64(1):135–168. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.CoRR, abs/2407.21783. Kanishk Gandhi, Jan...

Pith/arXiv arXiv 2013
[1985]

theory of mind

Does the autistic child have a “theory of mind”?Cognition, 21(1):37–46. Stephanie M Carlson and Louis J Moses. 2001. Individ- ual differences in inhibitory control and children’s theory of mind.Child development, 72(4):1032– 1053. Stephanie M Carlson, Louis J Moses, and Casey Breton

2001
[1994]

Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu

The relationship between cognition and action: performance of children 312–7 years old on a stroop- like day-night test.Cognition, 53(2):129–153. Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu. 2024. Timetom: Temporal space is the key to unlocking the door of large lan- guage models’ theory-of-mind. InFindings of the As- sociation fo...

2024
[2002]

Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang

How specific is the relation between execu- tive function and theory of mind? contributions of inhibitory control and working memory.Infant and Child Development: An International Journal of Re- search and Practice, 11(2):73–92. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, ...
[2019]

PICTUREw/ Event Hiding

Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 5871–5876. Associa- tion for Computational Linguistics. Huao Li, Yu Quan Chong, Simon Stepputt...

Pith/arXiv arXiv 2019
[2023]

InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 14397–14413

Fantom: A benchmark for stress-testing ma- chine theory of mind in interactions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 14397–14413. Association for Computational Lin- guistics. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage mo...

2023
[2024]

InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 15959–15983

ToMBench: Benchmarking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 15959–15983. Associ- ation for Computational Linguistics. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 202...

2024

[1] [1]

I’m sorry, I have to cut the conversation short. My delivery is arriving and I need to receive it. We’ll continue this chat soon though

Richard: "I’m sorry, I have to cut the conversation short. My delivery is arriving and I need to receive it. We’ll continue this chat soon though." - Richard knows about his own utterance

[2] [2]

No worries, Richard. Catch you later

Kobe: "No worries, Richard. Catch you later." - Richard knows about Kobe’s utterance

[3] [3]

Bye Richard!

Elena: "Bye Richard!" - Richard knows about Elena’s utterance

[4] [4]

Bye, Kobe and Elena! Enjoy the rest of your conversation

Richard: "Bye, Kobe and Elena! Enjoy the rest of your conversation." - Richard knows about his own utterance

[5] [5]

Now that it’s just us, Elena, what’s your biggest parenting challenge so far?

Kobe: "Now that it’s just us, Elena, what’s your biggest parenting challenge so far?" - Richard does not know about this conversation as he has left the location

[6] [6]

InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 19794–19809

Perceptions to beliefs: Exploring precursory in- ferences for theory of mind in large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 19794–19809. Association for Computa- tional Linguistics. Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B....

2024

[7] [7]

Oh, that’s a tough one. For me, it’s balancing my career responsibilities with being there for my child. But each challenge makes the journey more rewarding

Kobe: "Oh, that’s a tough one. For me, it’s balancing my career responsibilities with being there for my child. But each challenge makes the journey more rewarding." - Richard does not know about this conversation as he has left the location

[8] [8]

I couldn’t agree more, Kobe. At the end of the day, all the struggles and challenges are more than worth it

Elena: "I couldn’t agree more, Kobe. At the end of the day, all the struggles and challenges are more than worth it." - Richard does not know about this conversation as he has left the location

[9] [9]

Absolutely, Elena. Despite the ups and downs, I wouldn’t trade parenthood for anything

Kobe: "Absolutely, Elena. Despite the ups and downs, I wouldn’t trade parenthood for anything." - Richard does not know about this conversation as he has left the location

[10] [10]

Hi guys, sorry about the interruption. I hope I didn’t miss too much of our chat. You both were talking about balancing challenges last I recall

Richard: "Hi guys, sorry about the interruption. I hope I didn’t miss too much of our chat. You both were talking about balancing challenges last I recall." - Richard knows about his own utterance. Based on the events, Richard does not know about the specific parenting challenge that Elena faced, which was maintaining a balance between wanting to protect ...

[11] [14]

I think it’s maintaining a balance between wanting to protect and nurture my child but also letting him learn things by himself. You?

Elena: "I think it’s maintaining a balance between wanting to protect and nurture my child but also letting him learn things by himself. You?" - Richard does not know about this conversation as he has left the location

[12] [19]

(Noah is now in the staircase.)

Noah enters the staircase. (Noah is now in the staircase.)

[13] [20]

(Noah sees the slacks in the basket.)

The slacks are in the basket. (Noah sees the slacks in the basket.)

[14] [21]

(Noah is no longer in the staircase, so he cannot see what happens inside.)

Noah exits the staircase. (Noah is no longer in the staircase, so he cannot see what happens inside.)

[15] [22]

James moves the slacks to the treasure chest. (James moves the slacks to a new location, which Noah does not know about since he is no longer in the staircase.) Therefore, based on the information provided, Noah will not know where the slacks are located after he exits the staircase. Answer: Basket. (✓) Table 13: Additional qualitative results of Llama2-7...

[16] [23]

- Emma knows that she is in the lounge

Emma entered the lounge. - Emma knows that she is in the lounge

[17] [24]

- Emma knows that Alexander is in the lounge

Alexander entered the lounge. - Emma knows that Alexander is in the lounge

[18] [25]

- Emma knows that the peas are in the crate

The peas is in the crate. - Emma knows that the peas are in the crate

[19] [26]

- Emma does not know about this event because she exited the lounge before it happened

Alexander moved the peas to the bathtub. - Emma does not know about this event because she exited the lounge before it happened

[20] [27]

- Emma knows that she loves the lime

Emma loves the lime. - Emma knows that she loves the lime

[21] [28]

- Emma is no longer in the lounge, so she does not know about any events that happen in the lounge after she left

Emma exited the lounge. - Emma is no longer in the lounge, so she does not know about any events that happen in the lounge after she left. Based on the events that Emma knows about, she last knew that the peas were in the crate in the lounge. She exited the lounge before Alexander moved the peas to the bathtub. Answer: Emma thinks that Alexander searches ...

[22] [29]

- Logan knows this

Logan likes the lemon. - Logan knows this

[23] [30]

- Logan knows Lucas entered the crawlspace

Lucas entered the crawlspace. - Logan knows Lucas entered the crawlspace

[24] [31]

- Logan knows Logan entered the crawlspace

Logan entered the crawlspace. - Logan knows Logan entered the crawlspace

[25] [32]

- Logan knows the broccoli is in the drawer

The broccoli is in the drawer. - Logan knows the broccoli is in the drawer

[26] [33]

- Logan knows Lucas exited the crawlspace

Lucas exited the crawlspace. - Logan knows Lucas exited the crawlspace

[27] [34]

- Logan knows Logan moved the broccoli to the bucket

Logan moved the broccoli to the bucket. - Logan knows Logan moved the broccoli to the bucket

[28] [35]

- Logan no longer knows about events in the crawlspace

Logan exited the crawlspace. - Logan no longer knows about events in the crawlspace

[29] [36]

w/ Rules

Lucas entered the crawlspace. - Logan does not know about this event as he is not in the crawlspace Based on the events Logan knows about, he last knew that the broccoli was in the bucket in the crawlspace. Answer: Logan thinks that Lucas searches for the broccoli in the bucket. (✗) Table 14: Failure cases made by PICTUREon questions from the ToMi dataset...

2022

[30] [39]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step by s...

work page

[31] [42]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} This is the end of the main question. What events does and does not {name} know about? Output the answer for t...

work page

[32] [45]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step b...

work page

[33] [48]

However, they can re-enter the location

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} The main question to be solved is as follows: {question} What events does and does not {name} know about? Output the answer for the main question by thinking step b...

work page

[34] [51]

Noah exited the living room

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. Story: {story} What events does {name} know about? Only output the events according to the above rules, do not provide an explanation. SimToM (Inference) {perspective} You are {name}...

work page

[35] [52]

A character knows about all events that they do

[36] [53]

This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place

If a character is in a certain room/location, that character knows about all other events that happens in the room. This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place

[37] [54]

Noah exited the living room

If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location. PercepToM (Question Answering) {perspective} You are {name}. Based on the above information, answer the following question: {question} Keep your answer concise, one sentence is enoug...

work page

[38] [55]

A character knows about all utterances that they make

work page

[39] [56]

If a character is in a certain room/location, that character knows about every utterance that occurs in the room

work page

[40] [57]

Noah: Hi, Emma

If a character leaves a location, and is NOT in that location, they no longer know about any utterance that occurs within that location. However, they can re-enter the location. Story: {story} What events does {name} know about? Only output the utterances according to the above rules, do not provide an explanation. SimToM (Inference) {perspective} You are...

work page

[41] [1203]

Adele Diamond

Association for Computational Linguistics. Adele Diamond. 2013. Executive functions.Annual review of psychology, 64(1):135–168. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.CoRR, abs/2407.21783. Kanishk Gandhi, Jan...

Pith/arXiv arXiv 2013

[42] [1985]

theory of mind

Does the autistic child have a “theory of mind”?Cognition, 21(1):37–46. Stephanie M Carlson and Louis J Moses. 2001. Individ- ual differences in inhibitory control and children’s theory of mind.Child development, 72(4):1032– 1053. Stephanie M Carlson, Louis J Moses, and Casey Breton

2001

[43] [1994]

Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu

The relationship between cognition and action: performance of children 312–7 years old on a stroop- like day-night test.Cognition, 53(2):129–153. Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu. 2024. Timetom: Temporal space is the key to unlocking the door of large lan- guage models’ theory-of-mind. InFindings of the As- sociation fo...

2024

[44] [2002]

Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang

How specific is the relation between execu- tive function and theory of mind? contributions of inhibitory control and working memory.Infant and Child Development: An International Journal of Re- search and Practice, 11(2):73–92. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, ...

[45] [2019]

PICTUREw/ Event Hiding

Revisiting the evaluation of theory of mind through question answering. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 5871–5876. Associa- tion for Computational Linguistics. Huao Li, Yu Quan Chong, Simon Stepputt...

Pith/arXiv arXiv 2019

[46] [2023]

InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 14397–14413

Fantom: A benchmark for stress-testing ma- chine theory of mind in interactions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 14397–14413. Association for Computational Lin- guistics. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage mo...

2023

[47] [2024]

InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 15959–15983

ToMBench: Benchmarking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, pages 15959–15983. Associ- ation for Computational Linguistics. Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 202...

2024

PICTURE: Enhancing Theory-of-Mind in Large Language Models by Revealing, Not Hiding, Characters' Lack of Knowledge

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it