{"id":"4e76f0d4-9f35-4039-aaa2-aee7bcb492eb","arxiv_id":"2608.03388","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across four LLMs and six interaction modes, models solve hidden-function inference more often with upfront or oracle-provided evidence than with self-selected queries, and they struggle to know when to stop.","lead":"This paper introduces the Alien Abduction game, a six-mode benchmark in which language models must infer a hidden Python function from limited interaction. Most models succeed more often when evidence is handed to them upfront than when they must choose what to query, and they often either commit too early or keep probing without converging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HRA-based consistency and unclaimed-win findings rest on unvalidated self-reported hypotheses; the overfitting-to-self-selected-evidence claim is not yet established.","rationale":"I agree with the reader's weakest assumption and treat it as the most load-bearing issue. The benchmark's main novelty over prior work is its turn-level internal-consistency analysis (HRA) and the unclaimed-win observation, both of which depend on the reported current hypothesis being a faithful, executable candidate. The paper itself flags this in Limitations item 4, and the high parser-error rates make the measurement chain especially fragile. The success-rate findings (upfront evidence beats sequential; passive beats active for most models; some models commit early, others exhaust the budget) are largely independent of HRA and are plausible from the displayed data, so the overall picture of deficient active evidence acquisition is not threatened. What is threatened is the stronger interpretive claim about overfitting to self-selected evidence. A concrete validation—executing the last reported hypothesis as a submission—would settle the concern. If it passes, the interpretation is supported; if it fails, the paper should weaken the overfitting and unclaimed-win conclusions to behavioral-report claims. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not move it.","tokens_in":23117,"tokens_out":8051,"duration_ms":73375,"concrete_test":"In a replication, run an additional condition per active-mode episode: if the model submits SOLVE, first score the last reported current hypothesis by executing it against the same 100 test cases; if the episode ends without submission, force a final SOLVE constructed from the last reported hypothesis and score it. Then recompute Table 4's unclaimed-win count and Figure 5's final-turn HRA from these executed hypotheses. If the executed reported hypotheses fail at substantially higher rates than the reported HRA values suggest, the overfitting and unclaimed-win conclusions are artifacts of self-report rather than evidence of the model's operative hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's novel interpretive claim—that models 'form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives'—rests on HRA (Section 4.3) and on the unclaimed-wins analysis (Section 6, Table 4). Both are computed from the model's self-reported 'current hypothesis' field, which the Game Master never executes and which Limitations item 4 concedes 'may not fully represent the model's internal belief state.' This is not a peripheral caveat: the headline contrast (final-turn HRA of 0.85–1.00 for successful active-output episodes in Figure 5, alongside low success rates) is exactly the evidence for overfitting. If the reports are post-hoc summaries that retrodict observed evidence without being the operative program, high HRA is a report artifact. The problem is compounded by parser-error rates (Table 5: 40.6% for Qwen3.6-35B, 45.2% for Mistral-Large-3); the paper does not specify how non-executable or 'unknown' hypotheses are scored in HRA. Without a check that a reported hypothesis would actually pass held-out tests, the unclaimed-win figure (53.6% for GPT5.4-mini, Table 4) and the active/passive HRA gap do not establish the claimed deficiency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Alien Abduction, a multi-turn benchmark game for studying how LLMs acquire evidence, update hypotheses, and decide when to stop in abductive inference. A model must infer a hidden Python function from only its signature and a limited interaction budget, with six modes varying who controls evidence selection (active vs. passive) and the feedback format (exact outputs vs. binary verdicts), plus two single-turn baselines. The benchmark uses 50 automatically generated and validated target functions across five domains, with the same targets used across modes, sandboxed execution of submitted solutions, and held-out test evaluation. Across four models, the authors report success rates, turn-budget use, hypothesis retrodiction accuracy (HRA), and qualitative trace analyses. The main empirical findings are that providing evidence upfront or passively leads to higher success than active multi-turn acquisition; some models commit before using the available evidence while others exhaust the budget without converging; final hypotheses in active modes are more consistent with self-selected evidence; and some failures are 'unclaimed wins' where a correct hypothesis is apparently reached but never submitted.","tokens_in":23191,"tokens_out":5816,"duration_ms":51117,"significance":"If the behavioral interpretation holds, the paper fills a genuine gap: it jointly varies evidence control and feedback form while scoring intermediate hypotheses rather than only final answers, and it does so with a controlled, reproducible protocol. The evaluation is machine-checked in the important parts: every submitted solution is executed in a sandbox against held-out tests, targets are automatically validated, a fixed seed governs target sampling, and the same 50 targets are used across all modes and models. The authors also provide an unusually honest Limitations section, explicitly flagging the GPT5.4-as-generator distributional bias and the self-reported nature of HRA. Even if the interpretive claims about overfitting and stopping need tightening, the benchmark itself and the descriptive success-rate and turn-budget results are a useful contribution to the study of interactive LLM reasoning.","major_comments":[{"comment":"The headline contrast—final-turn HRA of 0.85–1.00 for successful active episodes versus 0.20–0.55 elsewhere—is computed from the model's self-reported 'current hypothesis' field. The manuscript does not state whether the reported hypothesis code is executed against the observed evidence, nor how non-executable reports (parser-error rates of 40.6% for Qwen3.6-35B and 45.2% for Mistral-Large-3, Table 5) or 'unknown' hypotheses are scored in HRA. Because the abstract's overfitting claim ('hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives') rests on this contrast, the authors should either report HRA computed by executing the reported hypotheses on the observed evidence, or explicitly restrict the claim to 'reported' hypotheses and add a sensitivity analysis excluding parser-error turns.","section":"Section 4.3, Section 5.3, Figure 5"},{"comment":"The unclaimed-win analysis calls an episode an unclaimed win when the model reaches a correct hypothesis but never submits it. As with HRA, 'correct hypothesis' is apparently determined from the self-reported field, not from executing the reported code. For GPT5.4-mini, 53.6% of failed instances are classified this way; if the report is a post-hoc paraphrase that happens to match the observed examples, the figure overstates the claim that the model 'does not reliably assess when its hypotheses are sufficiently supported.' The authors should validate these last-turn reports by executing them against the held-out test suite (or at least against all evidence observed in the episode) and report the agreement rate.","section":"Section 6, Table 4"},{"comment":"The ordering claim that success rates follow single-turn > passive > active is presented without significance testing. With 50 targets per mode-model cell, several mode contrasts appear within overlapping 95% confidence intervals in Figure 2, so the descriptive ordering alone does not establish the general conclusion. A paired or bootstrap test across the fixed 50 targets, or reported confidence intervals for the differences between modes, would strengthen the central comparative claim without changing the experimental design.","section":"Section 5.2, Figure 2"}],"minor_comments":[{"comment":"The TBU formula appears as 'TBU = (n-1 T)' in the text; it should be '(n-1)/T'.","section":"Section 4.3"},{"comment":"The sentence '7.56% of failed instances of GPT5.4-mini's end with a correct hypothesis' contains a possessive typo; it should read 'of GPT5.4-mini end'.","section":"Section 6"},{"comment":"The model name 'GPT5.4-Mini' is capitalized inconsistently with 'GPT5.4-mini' used throughout the rest of the paper.","section":"Table 4"},{"comment":"The statement that source code and test instances 'will be released upon the paper's acceptance' limits reproducibility during review; consider providing an anonymized artifact link or a detailed protocol appendix.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and its execution protocol are strong, and the descriptive results are worth publishing after revision. The main risk is that the interpretive framing—overfitting to self-selected evidence and unclaimed wins—goes beyond what the self-report-based HRA can support. I would ask the authors to validate HRA and unclaimed-win labels by executing reported hypotheses, and to add significance testing or interval estimates for the mode contrasts, rather than to expand the scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nYou should look at this paper for the benchmark, not for the headline. Alien Abduction is a well-built probe: six interaction modes factorize who selects evidence (active/passive) and what feedback looks like (outputs/verdicts) over the same 50 hidden Python functions, with submissions executed in a sandbox against held-out tests. That design is genuinely new relative to CodeARC, SYNTRA, Geng et al., and the rest, and the shared-target control is a real strength. The basic behavioral findings — upfront evidence beats sequential, passive beats active for most models, models either commit early or burn the whole budget — are visible in the data and don't depend on anything fancy. If you work on LLM agents, this is a useful controlled setting.\n\nThe soft spots are in the interpretive layer. The paper's central claim is that models \"form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives.\" That rests on HRA, which is computed from the model's self-reported current hypothesis, not from anything the Game Master executes. The authors admit in Limitations item 4 that these reports \"may not fully represent the model's internal belief state.\" If the reports are post-hoc rationalizations, high HRA in active modes is a report artifact, not evidence of overfitting. Relatedly, Table 4's \"unclaimed wins\" are actually just uncommitted failures with HRA=1 at the last turn. HRA=1 means the reported hypothesis reproduces the observed evidence — it does not mean the hypothesis is correct on held-out tests. Calling those \"correct hypotheses\" is not supported. That's a real overstatement, though the stopping problem might survive a corrected analysis.\n\nTwo more moderate issues. First, no code or data artifact is provided; release is promised upon acceptance, and the test-case pools aren't enumerated, so none of the numbers are currently reproducible. For a benchmark paper that's close to a deal-breaker until the artifact ships. Second, the mode and model contrasts are presented without significance tests, and some differences sit within the 95% CIs shown in Figure 2. That makes the stronger claims (\"providing evidence upfront leads to higher success rates\") sound more definitive than the evidence supports. Parser-error rates are also high for two models, and the paper doesn't say how non-executable or \"unknown\" hypotheses are scored in HRA.\n\nNone of this sinks the paper. The core design is careful, the limitations section is honest, and the target-generation bias is acknowledged. It deserves a serious referee, but the revision needs to ship the artifact, clarify HRA handling of parse errors, and either validate the self-reported hypotheses against held-out tests or soften the overfitting conclusion. I'd bring it to a reading group and would cite it once the artifact exists.\n\nRecommendation: send to peer review, with expectation of revision.","headline":"A genuinely useful interactive benchmark for active evidence acquisition, but the headline overfitting claim rests on unvalidated self-reports and an overstatement of what 'unclaimed wins' mean.","tokens_in":23925,"tokens_out":3872,"would_cite":true,"duration_ms":33297,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models are worse at abductive inference when they must actively gather evidence over multiple turns, forming hypotheses that fit self-selected evidence without discriminating among alternatives.","keywords":["abductive reasoning","active evidence acquisition","multi-turn interaction","black-box function induction","hypothesis retrodiction accuracy","LLM agents","stopping behaviour","query selection"],"falsifier":"Score the model's last reported hypothesis (or force it to submit that hypothesis) in every episode instead of the submitted answer. If the model with a 53.6% unclaimed-win rate turns most of those failures into wins and the active-passive success gap narrows sharply, the deficiency is in commitment and format rather than abductive evidence management; if the gap persists, the claim survives.","tokens_in":22747,"feed_emoji":"🔍","tokens_out":8586,"duration_ms":70129,"temperature":0.7,"pith_summary":"This paper tries to establish that standard single-turn evaluations miss a central weakness of LLM reasoning: when a model must acquire evidence itself over multiple turns, revise its hypotheses, and decide when to stop, it performs substantially worse than when the same evidence is supplied. The authors build the Alien Abduction game, in which a model must reconstruct a hidden Python function from its signature and a limited number of black-box queries, with six modes that vary who selects the evidence and whether feedback is exact outputs or true/false verdicts. Across four models, success is highest when evidence is shown upfront, intermediate when examples are fed passively, and lowest when the model chooses its own probes. Models that pick their own evidence end up with hypotheses that reproduce that narrow evidence well, but those hypotheses do not distinguish the true function from rivals; some models submit too early, while others exhaust the budget while holding a correct but unsubmitted hypothesis. If correct, the findings imply that active evidence management, query selection, hypothesis validation, and stopping, is a distinct bottleneck in agentic LLM reasoning.","feed_headline":"Self-chosen evidence misleads LLMs in multi-turn inference","feed_subtitle":"In a hidden-function game, models do better with provided examples than with their own queries, yet overfit what they picked.","key_machinery":"The load-bearing instrument is the Alien Abduction game, a black-box protocol in which a Game Master hides a Python function and the model either proposes test inputs or receives examples, with feedback as exact outputs or as true/false membership verdicts; six modes factor control over evidence selection (active versus passive) against feedback form (output versus verdict), holding the same 50 target functions across modes. Two metrics carry the argument: Turn Budget Use (TBU), the fraction of the 15-turn budget consumed before submission, and Hypothesis Retrodiction Accuracy (HRA), the fraction of observed evidence reproduced by the model's self-reported current hypothesis. The factorization allows the active-passive comparison to isolate the effect of self-directed exploration, and HRA is what separates inferring the wrong rule from ignoring one's own evidence.","core_discovery":"On the paper's own terms, the central discovery is the active-information-acquisition deficit: in the Alien Abduction game, providing evidence upfront leads to higher success rates than distributing it across turns, and across models the single-turn modes outperform the passive multi-turn modes, which in turn outperform the active modes. When models select their own queries, they achieve lower success than when examples are provided by the oracle, yet their final reported hypotheses are more consistent with the evidence they selected, as measured by Hypothesis Retrodiction Accuracy. The authors interpret this as evidence that models form hypotheses that fit self-selected evidence without sufficiently distinguishing them from alternatives, and that they struggle to validate and refine hypotheses or to determine when to stop. Supporting observations include early commitment before the available evidence is used, exhaustion of the turn budget without convergence, narrow input coverage of self-selected queries, and unclaimed wins in which a correct hypothesis is reached but never submitted.","pith_inferences":["A direct test of the overfitting explanation: if models are forced to include probes chosen by a broad random sampler or to state which live alternative each query discriminates, the active-passive gap should shrink; otherwise the gap is a query-selection problem, not a validation problem.","The behavioral pattern resembles confirmation bias in human hypothesis testing: models appear to prefer hypothesis-consistent probes over falsifying ones. Building a version of the game that counts whether queries target surviving alternatives would connect LLM active learning to that literature.","The unclaimed-win episodes suggest that for some models the binding constraint is commitment, not inference: prompting the model to submit whenever its reported hypothesis reproduces all evidence (HRA of 1) is a cheap intervention that would test whether success rises without any improvement in induction.","Parser errors were frequent for two of the four models, so part of the active-mode deficit may be protocol adherence rather than reasoning; comparing a free-form or tool-calling interface against the rigid format would separate the two."],"forward_implications":["Single-turn benchmarks overstate agentic competence: high performance on upfront-evidence tasks will not transfer to settings where an LLM agent must gather evidence, so interactive evaluations must include querying and stopping.","Deployments that let an LLM choose its own data points, such as tool use, data collection, or experiment design, should expect lower success than pipelines that feed curated examples, unless query selection is explicitly improved.","A hypothesis that fits all evidence the model itself collected is a weak success signal; agents need to seek evidence that discriminates among live alternatives, not just evidence consistent with the current guess.","Stopping is a real failure mode: models either commit too early or keep querying while holding a correct hypothesis, so explicit stopping criteria are as important as better induction.","Membership-verdict feedback is harder than exact-output feedback, especially when the model must propose the candidate pairs, meaning binary feedback regimes demand extra support."],"supporting_citations":[{"why":"Contrasts passive observation with active intervention in reverse-engineering settings; the direct baseline the active-vs-passive comparison extends.","marker":"Geng et al. (2025)"},{"why":"CodeARC benchmark where agents query a hidden function and refine candidate Python implementations; the closest interactive program synthesis setting.","marker":"Wei et al. (2025)"},{"why":"SYNTRA selects informative test inputs to eliminate competing program hypotheses; motivates the query-selection analysis.","marker":"Lee et al. (2025)"},{"why":"RULEARN frames hidden-rule learning as cycles of abduction, deduction, and induction; supplies the reasoning-cycle framing.","marker":"He et al. (2025)"},{"why":"ORACLE extends black-box environment interaction to code and other domains; part of the interactive-benchmark comparison.","marker":"Yin et al. (2026)"},{"why":"AR-Bench tests whether models ask useful questions under incomplete information; the question-asking baseline.","marker":"Zhou et al. (2025)"},{"why":"Active concept learning with predefined query-selection policies, showing the querying strategy is usually not model-chosen.","marker":"Chari and Pattanaik (2026)"},{"why":"Classic finding on failure to eliminate hypotheses; motivates the test of whether models seek disconfirming evidence.","marker":"Wason (1960)"}],"fun_headline_variants":["LLMs overfit self-chosen evidence in abductive games","Active evidence seeking hurts LLM abductive reasoning","LLMs pick narrow evidence, then overfit to it","Models overfit their own queries in abductive inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hypotheses a model reports in each turn accurately reflect what it actually believes; the paper itself concedes these reports may not represent the model's internal belief state, so the consistency and stopping findings would weaken if self-reports are noisy or decoupled from operative reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overfit self-chosen evidence in abductive games","Active evidence seeking hurts LLM abductive reasoning","LLMs pick narrow evidence, then overfit to it","Models overfit their own queries in abductive inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2305,"prompt_tokens":931,"completion_tokens":1374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1309}},"tokens_in":547,"tokens_out":1374,"duration_ms":10551,"temperature":1.0,"reasoning_tokens":1309,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:51:11.930702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the model's last reported hypothesis (or force it to submit that hypothesis) in every episode instead of the submitted answer. If the model with a 53.6% unclaimed-win rate turns most of those failures into wins and the active-passive success gap narrows sharply, the deficiency is in commitment and format rather than abductive evidence management; if the gap persists, the claim survives.","supporting_citations":[],"review_version":1}