{"id":"f8e7d6f6-616c-4b2f-ba82-289014e6752c","arxiv_id":"2506.01474","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A neuro-symbolic Rational Speech Act model with LLM proposers and evaluators predicts human question-answer patterns about as well as the fully hand-specified probabilistic model.","lead":"The authors replace hand-built components of a probabilistic model of pragmatic question answering with LLM modules, and test which combinations best predict human answers. Hybrid models match or beat the traditional model in this case study, but performance depends heavily on how the LLM is integrated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The flagship 'full NeSy' model's best fit depends on rule-based supplementation of 'no-options' and 'all-options' responses (Sec. 3.3), undermining the central claim that LLM modules replace hand-specified utterance sets.","rationale":"The reader's weakest_assumption was the compute-budget confound, which is real but secondary. The more load-bearing concern is that the flagship 'full NeSy' model—the model that most directly embodies the paper's claim of replacing hand-specified components—relies on rule-based supplementation of two response categories that the LLM proposer fails to generate at n=10. This is an internal inconsistency with the abstract's promise of eliminating manual specification. The paper remains a useful systematic case study, and the 'prompt-based questioner with goals' model also fits well, so a conditional verdict is appropriate. However, the headline claim as stated is not fully supported by the flagship model's construction. I recommend retaining the reader's CONDITIONAL verdict, with the condition that the authors either demonstrate that the LLM proposer alone (at sufficient n) covers all response types without manual intervention, or explicitly soften the open-endedness claim.","tokens_in":15142,"tokens_out":4173,"duration_ms":44049,"concrete_test":"Run the 'full NeSy' model with the response proposer set to n=50 (which the authors note ensures full coverage of response types) and do not append any rule-based 'no-options' or 'all-options' responses. Compute the JSD improvement over baseline and compare it to the PCM reference line and to the reported result with manual supplementation. If the JSD worsens substantially or becomes undefined because the model assigns zero probability to a human-used category, the reported fit is an artifact of manual response supplementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that neuro-symbolic models can 'eliminate the need for manual specification' (Abstract) is contradicted by the construction of the best-performing 'full NeSy' model. In Section 3.3, the authors state that because the LLM response proposer with n=10 was 'unlikely to contain' the 'no-options' and 'all-options' response types, they 'appended these two response types to set of alternatives manually nonetheless.' These two rule-based additions are exactly two of the five human response categories. Thus the model's ability to match human data (Figure 2) depends on hand-specified utterance types that the LLM proposer fails to supply, so the model does not actually replace hand-specified utterance sets. The other top-fitting model, 'prompt-based questioner with goals' (Section 3.4), also uses the pre-specified symbolic response set from the PCM, so it likewise does not demonstrate open-ended LLM generation of utterances. The compute-budget confound identified by the reader is a secondary concern: even if compute were matched, the manual supplementation would still threaten the open-endedness claim. The empirical fit of the 'full NeSy' model is therefore not a clean test of the paper's central contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a neuro-symbolic framework for pragmatic question-answering in which LLM modules replace or augment components of a Rational Speech Act-style probabilistic cognitive model (PCM). The authors systematically vary which components are neural (utility evaluation, literal semantics, proposers of responses/questions/goals, prompt-based questioner, one-shot chain-of-thought) and compare model predictions to human response-category proportions using Jensen-Shannon divergence. The central claim is that hybrid models can match or exceed the traditional PCM while 'eliminating the need for manual specification.' The paper also includes qualitative module analyses and an appendix with an open-source LLM (Qwen) replication. The main empirical finding is that the 'full NeSy' model and the 'prompt-based questioner with goals' achieve the best JSD fit, with several other scaffolded models close to the PCM baseline.","tokens_in":15384,"tokens_out":2450,"duration_ms":27808,"significance":"If the central claims hold, this is a valuable contribution to computational pragmatics: it provides a systematic architecture for combining LLMs with probabilistic cognitive models, tests multiple integration points, and validates an LLM utility evaluator against human ratings (R = 0.92). The paper is transparent about many limitations, includes prompts and materials, and reports an open-source LLM replication, which strengthens reproducibility. However, the flagship 'full NeSy' model's best fit relies on manual supplementation of the response set, directly undercutting the abstract's 'eliminating the need for manual specification' claim. The compute-budget confound is acknowledged but not addressed, and the headline comparisons lack inferential statistics. These issues affect the strength of the central contribution as currently stated, although they are addressable within the manuscript's scope.","major_comments":[{"comment":"The abstract claims that the framework eliminates the need for manual specification, but the best-performing 'full NeSy' model depends on exactly the kind of manual specification the paper aims to avoid. In Section 3.3, the authors state that because the LLM response proposer with n=10 was 'unlikely to contain' the 'no-options' and 'all-options' response types, they 'appended these two response types to set of alternatives manually nonetheless.' These two rule-based additions are two of the five human response categories, so the model's ability to match human data in Figure 2 is not a clean test of open-ended generation of utterance sets. The 'prompt-based questioner with goals' model also uses the pre-specified symbolic response set from the PCM. The open-endedness claim should be substantially tempered, or the authors should provide a version of the full NeSy model whose response types are generated entirely by the LLM proposer (e.g., with larger n or adjusted prompting) and show that the fit is preserved.","section":"§3.3, Abstract"},{"comment":"The paper acknowledges that 'one other potential source of improved performance with scaffolding of the LLM could be due to higher inference time compute budget that comes with decomposing the task into several LLM calls.' Since the monolithic one-shot CoT model uses a single LLM call while scaffolded models use several, the observed improvement in fit may be driven by additional compute rather than by the cognitive task decomposition. This confound is not controlled anywhere in the manuscript: there is no matching of total tokens, API calls, or sampling effort across conditions. The claim that task decomposition, rather than compute, drives the improvement is therefore unsupported. I recommend adding a control condition (e.g., sampling multiple responses from the monolithic model, or otherwise equating compute) or rephrasing the conclusion to present the result as a comparison of architectures under unequal compute budgets.","section":"§6 (last paragraph)"},{"comment":"The central quantitative comparisons are based on means and 95% bootstrapped confidence intervals over only five simulation iterations, with no inferential statistics. The statement in the abstract that hybrid models 'can match or exceed the performance of traditional probabilistic models' is stronger than what the data show: in Figure 2, the CIs of the best models overlap with or lie near the PCM reference line, and differences among models are not tested. I ask for either a permutation or bootstrap significance test on the JSD differences, or a rephrasing of the claim to 'comparable to' rather than 'match or exceed.' Reporting per-item JSD distributions would also help assess whether the aggregate fit is driven by a few vignettes.","section":"§4, Figure 2"}],"minor_comments":[{"comment":"The response category labels ('target,' 'similar,' 'unrelated,' 'all options,' 'no options') are introduced in the text but Figure 3's category names should be explicitly matched to these labels to avoid confusion, especially for readers not familiar with Tsvilodub et al. (2023).","section":"§3.1"},{"comment":"The definition Δ_i = JSD(B, humans) − JSD(M_i, humans) is clear, but the text should state explicitly that higher Δ means closer to human data (it is implied by the caption) and report the uncertainty in Δ (e.g., bootstrap CIs on the difference rather than on each JSD separately).","section":"§4, Formula for Δ"},{"comment":"The main results rely on a single closed-source LLM (GPT-4o-mini), and the open-source replication in Appendix D does not report results for the full neuro-symbolic model due to parsing errors. This limitation is acknowledged in the Discussion, but it would strengthen the paper to state in Section 4 that the headline 'full NeSy' result is specific to GPT-4o-mini and has not been replicated with an open-source model.","section":"§3.3 and Appendix D"},{"comment":"The phrase 'open-ended' is used both for the framework's goal (in principle unbounded sets) and for the actual implementations (n=10 or n=3 proposals with manual supplementation). Clarifying this distinction early would avoid overreading the results as demonstrating fully open-ended generation.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead 2506.01474. The genuinely new thing is the systematic comparison: the same pragmatic QA task, the same PCM scaffold, and a graded replacement of components (utilities, semantics, proposers, full NeSy, prompted variants). That is a useful map of where LLM modules do and don't slot into a rational speech act model. The utility evaluator correlates with human ratings at R=0.92, which is a concrete, reproducible result, and the appendix with Qwen is a good-faith check on closed-source dependence.\n\nThe soft spot is the central claim. The abstract says the framework eliminates manual specification, but the best-fitting full NeSy model manually appends the 'no-options' and 'all-options' response types because the LLM proposer rarely generates them. Those are two of the five human response categories, so the fit in Figure 2 is partly bought with hand-specified utterance types. The other top model, prompt-based questioner with goals, also reasons over the pre-specified symbolic response set. That doesn't kill the paper's usefulness—the modular comparison and design lessons survive—but it does mean 'open-ended natural language' is exaggerated. The reader's stress-test note is on target.\n\nOther soft spots are more ordinary. No inferential statistics; the JSD differences have bootstrapped CIs but no test of whether the full NeSy model is actually better than PCM. The compute-budget confound is acknowledged in Section 6, and it is a real alternative explanation for why scaffolding helps. Main results are GPT-4o-mini only; the Qwen results are exploratory and one full NeSy run failed due to parsing.\n\nThe paper is honest about most of these. The discussion flags the compute issue, the closed-source limitation, and the n-sensitivity of response proposals. That intellectual honesty counts for something.\n\nWho is this for? Anyone building hybrid LLM/PCM models in pragmatics or cognitive modeling. It's a good empirical starting point, not a finished theory. A serious referee could push them to (a) rephrase the open-endedness claim to match what was actually tested, (b) add statistical comparisons, and (c) ideally a compute-matched control. I'd send it to peer review—conditional, but worth the referees' time.\n\nBest.","headline":"A useful systematic comparison of LLM/PCM hybrids, but the 'open-ended' headline is held up by manual response supplementation.","tokens_in":15908,"tokens_out":1830,"would_cite":true,"duration_ms":19306,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM proposers and evaluators inside a probabilistic cognitive model can match or exceed the original model's fit to human question-answering data, with truth-conditional semantics as the weak link.","keywords":["neuro-symbolic models","pragmatic question answering","probabilistic cognitive models","large language models","rational speech act","LLM proposers","LLM evaluators","overinformative answers"],"falsifier":"Run a monolithic one-shot LLM and the decomposed neuro-symbolic model with matched total token or compute budgets and compare Jensen-Shannon divergence to human response categories; if the monolithic model matches the decomposed model's fit once budgets are equal, the paper's scaffolding explanation is undermined.","tokens_in":14881,"feed_emoji":"💬","tokens_out":7567,"duration_ms":71344,"temperature":0.7,"pith_summary":"The paper tries to show that hand-built probabilistic models of pragmatic question answering can be opened up to natural language by handing specific subtasks to large language models (LLMs) while keeping the symbolic reasoning scaffold intact. It tests a family of such hybrid models on a classic scenario: a person asks for an unavailable item and the respondent mentions a relevant alternative. Across systematic variations, several hybrids match or beat the original hand-specified model in predicting which response category humans produce. The paper also claims that the integration style matters: LLMs work best as proposers of alternative responses, questions, and goals, and as evaluators of utility, but are the weak link for truth-conditional semantic evaluation. If true, this is a path toward cognitive models that handle open-ended language instead of fixed vignettes.","feed_headline":"Hybrid models match symbolic pragmatics models on human data","feed_subtitle":"Task-decomposed LLM modules predict relevant answers as well as the original probabilistic model, without manual specification.","key_machinery":"The central object is the QA model, a probabilistic cognitive model that formalizes pragmatic question answering as recursive reasoning: a base-level respondent answers literally, a pragmatic questioner chooses a question by expected utility over a decision problem, and a pragmatic respondent infers the questioner's decision problem and chooses an answer that maximizes expected utility plus informativeness. The neuro-symbolic variants replace discrete components of this model, namely the utility function, literal semantics, and the sets of possible questions, answers, and goals, with LLM calls, while leaving the Bayesian inference steps symbolic. The key contrast is between models with LLM evaluators (utility and semantics) and LLM proposers (alternatives and goals), and between decomposed multiple-LLM architectures and a single monolithic prompt.","core_discovery":"The central claim is that a neuro-symbolic model, in which LLM modules propose alternatives and evaluate utility while symbolic Bayesian inference over those outputs is preserved, can match or exceed the traditional probabilistic model's fit to human answer patterns. In the main comparison, the fully decomposed neuro-symbolic model and a prompt-based questioner conditioned on inferred goals reach or exceed the probabilistic baseline in Jensen-Shannon improvement over chance. The paper further finds that the way LLMs are integrated is decisive: they are reliable for generating alternative utterances and questions, for proposing plausible questioner goals, and for rating the utility of options (correlating with human utility ratings at R = 0.92), but LLM-based semantic evaluation of the pragmatic respondent's answers overpredicts unrelated responses. This leads the authors to identify truth-conditional semantic evaluation as the bottleneck of the neuro-symbolic approach.","pith_inferences":["If the compute-budget confound is set aside, the results suggest a general design rule for neuro-symbolic cognitive models: use LLMs for generation and rating, and keep a symbolic or rule-based component for literal truth checking.","The goal-proposer analysis points to a testable bias: LLM-generated goals over-anchor on the target and underrepresent social goals, which could distort respondent inferences in domains where social reasons dominate.","A natural extension is to replace fixed n=10 response sampling with sampling until all conceptual response types are covered, which would test whether proposer coverage, rather than the symbolic scaffold, drives the fit.","The improvement observed as more LLM modules are added within the PCM family suggests an open question: whether a fully open-ended model with no manually specified response categories can reach human-level fit once semantic evaluation is fixed."],"forward_implications":["Hand-specified sets of utterances, questions, and goals can be replaced by LLM proposals without sacrificing predictive fit.","LLM utility ratings correlate strongly with human utility ratings, so one human rating elicitation step can be automated.","Models with at least some task decomposition fit human data better than a single monolithic chain-of-thought prompt.","Truth-conditional semantic evaluation is the weak point; models using LLM semantics overpredict unrelated responses.","Open-source LLMs can reproduce the main pattern when formatting failures are controlled, suggesting the framework is not tied to one proprietary model."],"supporting_citations":[{"why":"Provides the probabilistic QA model and the WebPPL implementation that serve as the symbolic scaffold and baseline.","marker":"Hawkins et al. (to appear)"},{"why":"Supplies the 30 vignettes, free-response human data, categorization scheme, and one-shot LLM prompt used for evaluation.","marker":"Tsvilodub et al. (2023)"},{"why":"Grounds the model's notion of relevance in decision problems and expected utility of answers.","marker":"van Rooy (2003)"},{"why":"Introduces the recursive rational-agent reasoning paradigm that the QA model extends to question answering.","marker":"Frank and Goodman (2012)"},{"why":"Provides the WebPPL probabilistic programming language in which the baseline PCM is implemented.","marker":"Goodman and Stuhlmüller (2014)"},{"why":"Cited for the difficulty of converting between neural and symbolic representations, which the paper uses to explain LLM semantic evaluator failures.","marker":"Bader et al. (2004)"},{"why":"Provides the notion of LLMs as proposers within cognitive-architecture scaffolding.","marker":"Sumers et al. (2023)"}],"fun_headline_variants":["Neuro-symbolic pragmatics matches classic on human answers","Hybrid pragmatics: LLM proposals, Bayesian inference match humans","LLM modules for pragmatics: match symbolic without manual specs","Pragmatic QA: neuro-symbolic equals traditional on human fit","LLM semantic evaluation: the pragmatics bottleneck"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the improved fit of the scaffolded models comes from the cognitive task decomposition, not from the larger inference-time compute budget that multiple LLM calls consume.","fun_headline_variants_meta":{"raw":{"variants":["Neuro-symbolic pragmatics matches classic on human answers","Hybrid pragmatics: LLM proposals, Bayesian inference match humans","LLM modules for pragmatics: match symbolic without manual specs","Pragmatic QA: neuro-symbolic equals traditional on human fit","LLM semantic evaluation: the pragmatics bottleneck"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001723,"raw_usage":{"total_tokens":6786,"prompt_tokens":889,"completion_tokens":5897,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":5812}},"tokens_in":505,"tokens_out":5897,"duration_ms":42610,"temperature":1.0,"reasoning_tokens":5812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:40:44.237653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a monolithic one-shot LLM and the decomposed neuro-symbolic model with matched total token or compute budgets and compare Jensen-Shannon divergence to human response categories; if the monolithic model matches the decomposed model's fit once budgets are equal, the paper's scaffolding explanation is undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 30 vignettes, free-response human data, categorization scheme, and one-shot LLM prompt used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the model's notion of relevance in decision problems and expected utility of answers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the recursive rational-agent reasoning paradigm that the QA model extends to question answering."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the WebPPL probabilistic programming language in which the baseline PCM is implemented."},{"cited_title":"The Integration of Connectionism and First-Order Knowledge Representation and Reasoning as a Challenge for Artificial Intelligence","cited_arxiv_id":"cs/0408069","evidence_quote":"Cited for the difficulty of converting between neural and symbolic representations, which the paper uses to explain LLM semantic evaluator failures."}],"review_version":1}