{"id":"ce643a50-ef5d-43e5-8955-ba23eacf08c3","arxiv_id":"2506.16755","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LIRAS synthesizes PDDL world models and agent configurations from language, parses video frames into symbolic states, and runs Bayesian inverse planning (SIAM) to produce human-like graded social inferences across five gridworld domains.","lead":"This paper introduces LIRAS, a pipeline that turns written rules and short videos into formal planning models and then uses Bayesian inverse planning to infer an agent's goals, beliefs, rewards, and costs. On several theory-of-mind benchmarks, LIRAS matches human judgments more closely than large vision-language models such as GPT-4o and OpenAI o3, despite using a lightweight VLM.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-sample synthesis with rejection sampling leaves the near-ceiling correlations unverified: the reported k=1 draw needs a distribution over compiled model samples.","rationale":"The reader's weakest assumption already identifies the joint correctness of synthesized PDDL domains, agent configurations, and VLM state parses as the load-bearing condition, noting that the paper reports hallucination and syntax errors and relies on rejection sampling. My stress-test sharpens this into a concrete, testable vulnerability: the single k=1 sample per stimulus means the reported correlations are a point estimate with no measured variance, and rejection sampling conditions the estimate on compilation success. If the distribution over compiled samples has high variance, the near-ceiling correlations would not reproduce, and the central claim that automated synthesis plus SIAM explains human judgments would be weakened; if the distribution is tight, the concern is resolved. I also considered the abstract's overclaim that LIRAS outperforms baselines on all domains, since o3 is numerically higher on m-DKG (0.81 vs. 0.78) and DKG-Inverse (0.79 vs. 0.74), but the confidence intervals overlap and this is a qualification rather than a threat to the core mechanism. Machine-checked proofs are not present, so the empirical robustness check is the decisive step. The appropriate verdict remains CONDITIONAL: the existing conditions should be expanded to require the sample-distribution and parse-accuracy analysis described above, plus release of the synthesized models for independent inspection.","tokens_in":19593,"tokens_out":7042,"duration_ms":78743,"concrete_test":"Re-run the full LIRAS pipeline with N=10 independent rejection-sampled draws per stimulus (same temperature=1.0, same prompts) across all DKG variants and Foodtruck/Astronaut, and compute the per-draw Pearson correlation with human means. Report the mean, SD, and minimum correlation over draws for each domain, plus the percentile of the paper's k=1 result within this distribution. If the SD exceeds roughly 0.05 or any draw falls below the corresponding CoT baseline, the near-ceiling results are not robust to synthesis stochasticity. As a secondary check, hand-label ground-truth grid states for a random 20% of stimuli and compare VLM parses against them to quantify whether residual parse errors distort inferred posteriors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 specifies that LIRAS is run with k=1 sample per stimulus, using rejection sampling until the full pipeline compiles, and asserts without quantitative support that semantic variation among models that actually compile is minimal. Section 4.1 concedes hallucination and syntax errors in world-model synthesis and visual parsing, errors that required resampling to overcome. The headline correlations in Tables 1 and 2 (e.g., r=0.79 on DKG-Single, versus split-half reliability 0.78) are therefore produced by one unexamined draw from a distribution over compiled symbolic models and parses. If the rejected samples differ systematically, or if the single retained draw is unusually favorable, the near-ceiling agreement could be an artifact of draw luck or selection rather than a property of the LIRAS architecture. This is load-bearing because the central claim is that automated synthesis plus Bayesian inverse planning, not a particular lucky synthesized model, explains human graded theory-of-mind judgments. Without variance or parse-accuracy data, the reader cannot distinguish a robust mechanism from a favorable sample. The stated claim that compiled models show minimal semantic variation is precisely the assumption that needs empirical support before the abstract-level claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LIRAS, a neuro-symbolic framework that synthesizes a PDDL environment model, a rational agent configuration, and cell-level visual state parses from language and video inputs, and then runs Bayesian inverse planning (SIAM) to output graded theory-of-mind ratings. It evaluates on two published cognitive-science datasets (Food Truck, Astronaut) and five DKG variants, including rule-modified and multi-agent versions, comparing correlations with human judgments against Gemini 2.0 Flash, GPT-4o, OpenAI o3, and an ablation that replaces the Bayesian engine with direct LLM inference. The reported LIRAS correlations (0.74–0.87) are close to or above the split-half reliability of the human data on DKG variants, and the qualitative example in Figure 2 illustrates that LIRAS, unlike baselines, shifts its goal posterior when the language changes from the DKG-Single to the DKG-Reuse rule.","tokens_in":19774,"tokens_out":7347,"duration_ms":72121,"significance":"If the reported results hold, the paper would make a valuable contribution: it demonstrates that a lightweight VLM, coupled with LLM-driven synthesis of structured world models and Bayesian inverse planning, can capture graded human mental-state judgments across diverse tasks without fitting any parameters to the human responses. The external human benchmark and the ablation isolating the Bayesian inference engine are notable strengths, as is the use of paired DKG stimuli to show sensitivity to linguistic rule changes. However, the central quantitative claim currently rests on a single stochastic synthesis sample per stimulus and on an unquantified assumption that residual parsing and synthesis errors are benign, so the significance cannot be fully assessed without additional sampling and parse-accuracy evidence.","major_comments":[{"comment":"All LIRAS results are generated with k=1 stochastic synthesis sample per stimulus, with no seed variance reported; because the PDDL domain, agent configuration, and visual state parses are all outputs of stochastic LLM/VLM sampling, the correlations in Tables 1 and 2 are one draw from a distribution over compiled models. The statement that 'semantic variation among models that actually compile is minimal' is asserted without quantitative support, and Section 4.1 confirms that hallucination and syntax errors were frequent enough to require resampling. Please report the distribution of correlations over multiple independent samples (or at least inter-sample agreement among ratings), the rejection and compile rates per stimulus, and an analysis showing that the retained samples are not selected in a way that systematically favors human agreement; without this, the near-ceiling correlations could reflect sampling luck rather than a property of the LIRAS architecture.","section":"Section 3.3 and Eqs. (1), (6), (7)"},{"comment":"The paper concedes hallucination and syntax errors in world-model synthesis and visual parsing, but provides no quantitative parse-accuracy measure or audit of residual errors. Since SIAM's posterior is computed over the parsed PDDL states and reconstructed actions, the claim that LIRAS matches human judgments at the split-half ceiling requires evidence that the residual errors are benign; for example, a manual audit of state and action sequences on a random stimulus subset, or an analysis of whether stimuli that required resampling produce ratings that differ systematically from those that did not.","section":"Section 4.1"},{"comment":"The evaluation protocol averages baselines over k=3 samples while reporting only k=1 for LIRAS; because the paper notes that single-sample baseline results are significantly more variable and worse, the single-sample LIRAS correlations are not directly comparable to the averaged baselines under sampling noise. Please report LIRAS results under the same k=3 protocol, or justify statistically that the single-sample estimates are stable, so the comparisons in Tables 1 and 2 are on equal footing.","section":"Sections 3.3 and 3.4"}],"minor_comments":[{"comment":"The paper says that results 'statistically significant from others are bolded' but never describes the significance test, the unit of analysis, or whether the confidence intervals are bootstrap intervals over stimuli; please specify the procedure.","section":"Tables 1 and 2"},{"comment":"The human data section reports n=20 per variant (100 total) but then states that 13 participants were excluded; please clarify the final sample size per variant and whether the split-half reliability in Table 2 is computed on the post-exclusion sample.","section":"Section 3.2"},{"comment":"The same reference appears as both 'Buschoff et al. (2025)' and 'Schulze Buschoff et al. (2025)'; please standardize. There is also a typo in Section 6.3 ('to to synthesize code') and a grammatical issue in the Figure 2 caption ('were tested').","section":"References and prose"},{"comment":"Because the method depends on explicitly provided action spaces and transition rules, and the paper acknowledges that humans can infer such rules implicitly, the title and abstract overstate the 'on-the-fly' generality; a more precise claim would be that LIRAS synthesizes models from explicit linguistic instructions on discrete gridworld tasks.","section":"Section 7 and Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The single-sample synthesis issue is the main barrier to accepting the quantitative claims. If the authors can supply seed variance, rejection rates, and a parse audit, I would be supportive; the architecture and evaluation design are otherwise promising. The paper's heavy reliance on the authors' own SIAM/SIPS lineage is not, by itself, a problem, especially given the external human benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, LIRAS is the first system I've seen that jointly synthesizes both the world model and the agent model from language into executable PDDL, then runs principled Bayesian inverse planning on top of VLM-parsed visual states. That integration is genuinely new and it works: correlations with human judgments on Foodtruck, Astronaut, and the DKG variants are mostly in the 0.74–0.87 range, close to the split-half reliability of the human data. The ablation replacing SIAM with direct LLM inference is a nice control—it shows the Bayesian engine is doing real work, not just the symbolic representation. The new DKG-Double/Reuse/Inverse stimuli with fresh human data are a useful resource, and the qualitative example in Figure 2 is convincing as a case where LIRAS adapts to language while o3 does not. Credit is earned here.\n\nThe soft spots are real but fixable. The biggest is that every LIRAS number comes from one synthesis sample per stimulus (k=1), with rejection sampling that only checks whether the pipeline compiles. The paper asserts without evidence that semantic variation among compiled models is minimal. That is load-bearing: if the single retained draw was lucky, the near-ceiling correlations could be draw luck, not architecture. I'd want variance across seeds, or at least a distribution of correlations from multiple samples, before trusting the headline numbers. Relatedly, the paper reports hallucination and syntax errors in synthesis and parsing but never quantifies parse accuracy or how often resampling was needed; some measurement of residual error would help. Second, the abstract says LIRAS outperforms SOTA models \"across all domains,\" but Table 2 shows o3 beats LIRAS on m-DKG (0.81 vs 0.78). That is a straightforward factual overclaim and should be fixed. Third, there are no hand-constructed symbolic model baselines on Foodtruck/Astronaut, so we can't tell how much automated synthesis loses relative to the original hand-crafted models. The ablation already shows the inference engine matters; a hand-crafted baseline would sharpen the claim about synthesis quality. Minor: the AutoToM comparison is acknowledged as unfair and dropped, which is fine, but the appendix discussion could be shorter. No code, stimuli, or human data are released; for this kind of pipeline work, that's a reproducibility gap.\n\nVerdict: this deserves a serious referee. The central idea is sound and the results are strong enough to be important if the single-sample concern resolves. I'd recommend conditional acceptance after the authors report synthesis variance, add hand-crafted baselines, fix the abstract, and release artifacts. I'd cite it if I were doing Bayesian ToM or language-conditioned planning.","headline":"A well-built LLM-to-PDDL plus Bayesian inverse planning pipeline that hits human noise ceilings on several theory-of-mind tasks, but the near-ceiling correlations rest on a single synthesis sample per stimulus and the abstract overclaims on m-DKG.","tokens_in":20369,"tokens_out":1541,"would_cite":true,"duration_ms":19085,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LIRAS synthesizes PDDL world and agent models from language and video, then runs Bayesian inverse planning to emit graded judgments of goals, beliefs, rewards, and costs.","keywords":["theory of mind","Bayesian inverse planning","multimodal reasoning","language-conditioned model synthesis","PDDL","rational agents","social cognition","sequential inverse agent modeling"],"falsifier":"Take one DKG-Reuse stimulus and deliberately change one generated rule so the PDDL contradicts the intended instructions, for example removing the reusability of keys while keeping visuals and queries fixed; if LIRAS's human correlation does not drop, the synthesized model is not what carries the inference. The paper already reports hallucination and syntax errors that are removed by rejection sampling, so the residual unsampled parses would be the place to look for systematic distortion.","tokens_in":19363,"feed_emoji":"🧠","tokens_out":7297,"duration_ms":68292,"temperature":0.7,"pith_summary":"LIRAS tries to show that grounded, situation-specific theory-of-mind reasoning can be automated by synthesizing a symbolic rational agent model from language and then inverting it against parsed visual evidence. Given a linguistic description and video frames, a multimodal LLM compiles a PDDL planning domain and an agent configuration (goal space, rewards, costs, observability, temperature), a VLM parses each grid cell into symbolic states, and the Sequential Inverse Agent Modeling engine runs Bayesian inverse planning to output graded probabilities over goals, beliefs, rewards, and costs. The paper reports that this pipeline correlates with human judgments between 0.74 and 0.87 across five cognitive-science domains, reaching the split-half noise ceiling of human data on DKG variants, and that it outperforms GPT-4o, OpenAI o3, and the same VLM used end-to-end. The wider point is that language can restructure social inference through explicit model construction rather than through pattern matching alone.","feed_headline":"Building world models from language matches human social reasoning","feed_subtitle":"A small vision-language model plus Bayesian planning beats GPT-4o on graded goals, beliefs, and costs.","key_machinery":"The carrying object is the synthesized (PO)MDP agent model: a PDDL domain generated from the linguistic description, agent parameters covering goal predicates, reward and action-cost tables, observability and belief configuration, and Boltzmann temperature, plus VLM-parsed environment states with derived actions. SIAM, Sequential Inverse Agent Modeling, extends Sequential Inverse Plan Search with joint inference over goals, rewards, costs, and beliefs; it enumerates hypotheses over mental states, updates belief states, computes expected utilities via memoized A* path costs, and weights each hypothesis by the Boltzmann probability of the observed action sequence. The mechanism shows how language can change the structure of the inference problem itself.","core_discovery":"The central claim is that the bottleneck in multimodal social reasoning is not perception but the absence of an explicit, task-specific generative model of the agent and environment. LIRAS has the LLM translate the instructions into PDDL and into an agent model with priors over goals, rewards, costs, and observability, parses the video frames cell by cell into matching PDDL states, and then uses SIAM to compute a posterior over mental states by Boltzmann-rational action likelihoods with memoized shortest-path utilities. On the food-truck, astronaut, DKG-Single/Double/Reuse/Inverse, and m-DKG tasks, the full LIRAS pipeline produces human-likeness correlations between about 0.74 and 0.87, while direct VLM prompting and even the ablated symbolic-synthesis-without-Bayes version fall much lower or negative. This is what the authors are trying to establish: a lightweight VLM plus explicit inverse planning can match the graded, language-sensitive social inferences people make, where larger end-to-end models cannot.","pith_inferences":["If the pattern holds beyond gridworlds, language-conditioned model synthesis plus Bayesian inversion could become a probe for how narration restructures mental models in humans, not just a benchmark.","A natural test is to feed LIRAS incomplete or implicit rule descriptions; if its posterior degrades the same way humans' does, the framework would help locate where commonsense inference takes over from explicit linguistic information.","The cell-by-cell VLM parsing and rejection sampling suggest a scalability bottleneck: continuous domains and visually rich scenes would require a different parser, so the contribution may be best framed as a proof-of-concept for discrete, language-defined environments.","The ablated result implies that neural models can assemble the right symbols yet still fail to reason over them; that suggests hybrid neuro-symbolic architectures may benefit human-AI collaboration even when the neural component is small."],"forward_implications":["Foundation-model baselines, including o3, give uneven and sometimes negative correlations with human judgments, whereas LIRAS with a lightweight VLM stays at 0.74 or above on every domain tested.","The ablation shows that synthesized symbolic models alone are not enough: asking the LLM to answer from the same PDDL and states performs at base-VLM level, so the explicit Bayesian inversion carries the human-likeness.","Changing the linguistic rule (key double, reuse, inverse) shifts human goal judgments, and LIRAS tracks those shifts while baselines degrade most on the unusual reuse variant.","Near-noise-ceiling correlation on the DKG variants means LIRAS accounts for nearly all reliable human variance in those stimulus sets."],"supporting_citations":[{"why":"Supplies the food-truck belief-desire task and the rational quantitative theory-of-mind framework whose human judgments LIRAS is evaluated against.","marker":"Baker et al. 2017"},{"why":"Defines the naive utility calculus and the astronaut domain with terrain costs and package rewards.","marker":"Jara-Ettinger et al. 2016"},{"why":"Introduces Sequential Inverse Plan Search and the original DKG stimuli that LIRAS extends.","marker":"Zhi-Xuan et al. 2020"},{"why":"Provides the language-augmented Bayesian theory of mind and belief-space SIPS that SIAM extends.","marker":"Ying et al. 2025b"},{"why":"Shows LLM translation of natural language into probabilistic world models, which LIRAS adapts to PDDL synthesis.","marker":"Wong et al. 2023"},{"why":"Formalizes (PO)MDP planning that underlies the agent model's action likelihoods.","marker":"Kaelbling et al. 1998"},{"why":"Supplies the PDDL interpreter that compiles and executes the synthesized planning domain.","marker":"Zhi-Xuan, 2022"},{"why":"Introduces inverse planning as action understanding, the inference principle LIRAS instantiates.","marker":"Baker et al. 2009"}],"fun_headline_variants":["Lightweight VLM plus Bayesian planning matches human social reasoning","LIRAS: language-aware inverse planning beats GPT-4o on social reasoning","Small model + Bayesian planning = human-like theory of mind","Explicit agent models from language and vision fix social reasoning","Why a small VLM beats GPT-4o at inferring goals, beliefs, and costs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on the assumption that the LLM-generated PDDL domain, the LLM-generated agent configuration, and the VLM's cell-by-cell state parses jointly recover the exact task content that human participants read and saw, with any residual errors staying benign enough not to distort the inferred posterior.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight VLM plus Bayesian planning matches human social reasoning","LIRAS: language-aware inverse planning beats GPT-4o on social reasoning","Small model + Bayesian planning = human-like theory of mind","Explicit agent models from language and vision fix social reasoning","Why a small VLM beats GPT-4o at inferring goals, beliefs, and costs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001294,"raw_usage":{"total_tokens":5271,"prompt_tokens":922,"completion_tokens":4349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":4255}},"tokens_in":538,"tokens_out":4349,"duration_ms":30365,"temperature":1.0,"reasoning_tokens":4255,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:19:19.466331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one DKG-Reuse stimulus and deliberately change one generated rule so the PDDL contradicts the intended instructions, for example removing the reusability of keys while keeping visuals and queries fixed; if LIRAS's human correlation does not drop, the synthesized model is not what carries the inference. The paper already reports hallucination and syntax errors that are removed by rejection sampling, so the residual unsampled parses would be the place to look for systematic distortion.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the food-truck belief-desire task and the rational quantitative theory-of-mind framework whose human judgments LIRAS is evaluated against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the naive utility calculus and the astronaut domain with terrain costs and package rewards."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Sequential Inverse Plan Search and the original DKG stimuli that LIRAS extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces inverse planning as action understanding, the inference principle LIRAS instantiates."}],"review_version":2}