{"id":"7d1a1841-d439-4241-bd22-a9ea1bae3aa6","arxiv_id":"2607.15766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"HypoArena is a 988-case benchmark that asks LLMs to generate hypothesis sets from conclusion-free reconstructed contexts and ranks 15 models via pairwise arena judgments.","lead":"This paper introduces HypoArena, a benchmark that tests whether LLMs can form plausible, testable hypotheses from pre-conclusion evidence rather than answer predefined questions. It reconstructs 988 discovery contexts from expert documents and ranks 15 LLMs with pairwise judgments that correlate with human experts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's central validity rests on Retrospective Context Regression producing genuinely conclusion-free contexts; leakage or memorization would collapse PHD to answer-restatement, and current evidence (single-model Forge-Audit, 60-case human audit, date-cutoff-only contamination control) is i","rationale":"The central claim of the paper is that HypoArena measures Prospective Hypothesis Discovery as a distinct competence. For that to be true, the model-visible Context must genuinely withhold the answer while retaining enough information to support hypotheses. The paper's own definition (§2.1) makes 'operational conclusion absence' a validity constraint, and the entire evaluation inherits it: if contexts leak, then high-performing models may simply be recognizing the source conclusion, and the benchmark measures answer retrieval rather than discovery. This is exactly the assumption the reader identified as weakest, and I agree.\n\nThe evidence offered for the absence of leakage is an Audit agent (§2.3) and a 60-case human audit (§2.4). Neither is sufficient. The Audit agent is an LLM prompt, not a formal guarantee; because the same model (gpt-5.4) generates the contexts, an auditor with shared priors may systematically miss what it does not consider leakage. The human audit covers only three domains and reports an 80% pass rate for Biomedical; 8% of the audited sample failed at least one criterion, and the other three domains—including ML, which is used for the ICLR external-validity analysis, and Safety, where stripped probable-cause language is the key risk—have no human audit at all. Appendix A concedes the single-model forge limitation. Temporal cutoffs also do not address memorization: many source documents (ICLR 2026 submissions, NTSB reports) are public and may be in training corpora, so a model could reproduce the reference hypothesis from memory even with a perfectly stripped context. The paper does not report any memorization probe.\n\nThe proposed forced-choice leakage test would settle the question: if independent readers cannot identify the held-out hypothesis from the context at better than chance, then the 'conclusion-free' claim is operational and the benchmark is valid as a measure of hypothesis generation on reconstructed contexts. If they can, the benchmark needs re-construction or filtering of leaky cases. Until that test is run, the appropriate verdict remains CONDITIONAL: the benchmark is a substantial and well-documented contribution, but its central construct validity is not yet established.","tokens_in":26474,"tokens_out":6481,"duration_ms":54464,"concrete_test":"Sample 200 HypoData cases stratified across all six domains (including un-audited ML, IT, Safety). For each, give only the model-visible Context to independent human experts (or a strong judge from outside the gpt family) and ask them to select the source-derived Reference hypothesis among 5 plausible distractors written in the same style. If accuracy significantly exceeds the 20% chance level (e.g., >35%, 95% CI excluding chance), leakage is present and the central claim fails; if near chance, the leakage concern is resolved. In parallel, run zero-shot memorization probes on all evaluated models (ask for the conclusion/probable cause of the source document without providing the Context) and test whether per-case arena performance correlates with memorization; a positive correlation indicates contamination rather than prospective discovery.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that Retrospective Context Regression yields contexts that are 'operationally conclusion-free' (§2.1, §2.3) and that the Forge–Audit loop removes answer-side cues. This premise is under-tested in three places. (1) Forge and Audit are both gpt-5.4 (Appendix A concedes the single-model construction); an auditor from the same model family can share the forger's blind spots, so a passed Leakage Check does not establish the absence of subtle cues. (2) The human quality audit covers only 60 cases in 3 of 6 domains (Biomedical, Financial, Social Science; §2.4), and even there the pass rate is 92% overall and 80% for Biomedical (Table 6); the remaining 928 cases, including ML and Safety, are un-audited. (3) Contamination control is temporal only (Appendices B/C): cutoffs exclude post-cutoff retrieval, but they do not test whether evaluated models memorized source documents (ICLR 2026 submissions, NTSB/CSB reports, post-mortems) and can recite the held-out hypothesis without seeing it in the context. There is also a structural issue: contexts are assembled from documents written after the conclusion, so fact selection/emphasis is guided by the eventual hypothesis; the paper disclaims exact historical recovery but does not bound the resulting bias. If any of these leaks operates, the task collapses to answer-restatement, and the claimed isolation of PHD fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Prospective Hypothesis Discovery (PHD), a task in which a model constructs grounded, discriminative, and testable hypothesis sets from pre-conclusion contexts, and presents HypoArena, a benchmark of 988 cases across six domains. HypoData is built via Retrospective Context Regression, a Forge–Audit pipeline that strips explicit conclusions from expert documents while preserving factual material. HypoEval combines pairwise LLM judging with Bradley–Terry–Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. The paper reports evaluations of 15 frontier LLMs, finding large BTD spreads, model-dependent effects of structured analytical skills, and correlations with human expert judgments and an external ICLR-acceptance signal. The central claim is that PHD is a distinct, largely unmeasured competence of LLMs.","tokens_in":26826,"tokens_out":6689,"duration_ms":59802,"significance":"If the construct validity of the reconstructed contexts can be established, HypoArena would be a genuinely useful public benchmark, filling a gap between QA benchmarks and open-ended idea-generation benchmarks. The paper is methodologically careful in several ways: it ships code and data, applies leakage/faithfulness/supportability audits, includes a human quality audit, replicates rankings with a second judge, and tests association with an independent peer-review signal. These are real strengths. The main risk is that the central premise—that Retrospective Context Regression yields operationally conclusion-free contexts—is under-tested. Since leakage or memorization would collapse PHD into answer-restatement, the benchmark's validity depends on closing this gap. With the additional validation described below, the paper could support its strong claims; as written, those claims outrun the evidence.","major_comments":[{"comment":"The benchmark's central validity rests on contexts that do not leak the held-out hypothesis. Current evidence is insufficient. Forge and Audit are both gpt-5.4 (Appendix A concedes single-model construction), so the auditor may share the forger's blind spots. The human audit covers only 60 of 988 cases and only Biomedical, Financial, and Social Science; Machine Learning, IT Operations, and Safety Investigation are unaudited. The Biomedical pass rate is 80% (Table 6), i.e., 4 of 20 audited cases failed. Because leakage would collapse PHD to answer-restatement, please add multi-model and/or human-in-the-loop construction for a validation subset, expand human audit to all six domains with adequate samples, and run explicit memorization/concealment tests (e.g., whether models can recover the reference hypothesis from context alone, or reproduce source conclusions above chance).","section":"§2.3, Appendix A, Tables 2/6"},{"comment":"Temporal cutoffs are the only contamination control. They exclude post-cutoff retrieval but do not test whether evaluated models memorized the source documents (ICLR 2026 submissions, NTSB/CSB reports, 2025–2026 papers). For IT Operations, the paper assumes unique incident configurations make temporal contamination low, but this is an assumption, not a measurement. Without a direct test—for example, a fuzzy-match or cloze-style probe comparing model outputs with the source-derived reference—one cannot rule out that apparent PHD performance partly reflects recitation. This is load-bearing for all subsequent rankings.","section":"Appendices B/C; §2.2"},{"comment":"Even if explicit conclusions are removed, contexts are assembled from documents written after the conclusion; the selection, ordering, and emphasis of facts is therefore informed by the eventual hypothesis. The paper acknowledges this in the Introduction ('Rather than recovering the exact historical information state') but does not bound the resulting bias. Because PHD is supposed to measure pre-conclusion reasoning, please validate construction against an independent pre-conclusion source in at least a subset of cases, or show that a context built from a randomly sampled factual subset of the source yields similar conclusions. Without such a check, the task may reward reconstructing the source's post-hoc narrative rather than discovering from genuinely inconclusive evidence.","section":"§2.1, Introduction"},{"comment":"The claim that arena evaluation 'resolves finer-grained differences among models' needs more than point estimates. Table 3 reports BTD ratings without confidence intervals or significance tests; adjacent systems are separated by as little as ~10–40 points, and pairwise cross-judge agreement is only 63–68% (Table 12). Per-domain human-judge alignment drops to τ=0.53 for Biomedical (Table 10). The aggregated rank correlations are encouraging, but the fine-grained separations may be within noise. Please report bootstrap or posterior intervals for BTD ratings and test whether adjacent ranks are statistically distinguishable. This is needed to support both 'clear capability stratification' and the 'finer-grained differences' claims.","section":"Table 3; §4.4; Table 12"}],"minor_comments":[{"comment":"The GitHub URL is duplicated; one of the two should be the HuggingFace dataset link.","section":"Abstract"},{"comment":"The axis labels and caption appear truncated; the per-domain Kendall τ values mentioned in the text (0.53, 0.97, 0.85) should be visible in the figure or caption.","section":"Figure 4"},{"comment":"The same human quality audit results appear as Table 2 and Table 6; unify the numbering to avoid confusion.","section":"Table 2 vs Table 6"},{"comment":"The mapping from 5-level verdicts to win shares {1.0, 0.75, 0.5, 0.25, 0.0} and the choice of BTD tie parameter θ are not sensitivity-analyzed; a short robustness check would strengthen the evaluation.","section":"§3.2"},{"comment":"The text says 'across all three reported measures' but the preceding sentence in the appendix lists three; in the main text this is clear only after reading Appendix H.2. Consider stating the three measures explicitly.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The core benchmark idea is promising and the execution is unusually transparent, but the conclusion-free premise is the foundation of the whole contribution. The paper's own limitations section concedes the single-model construction; the missing validation—expanded human audit, memorization probes, and uncertainty quantification—is within scope and should be addressed before the claims about PHD as a distinct measurable competence are accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Put it on your desk if you work on LLM evaluation. The paper defines Prospective Hypothesis Discovery (PHD) as a distinct task and builds HypoArena: 988 cases across six domains, contexts reconstructed from completed expert documents with explicit leakage/faithfulness/supportability checks, and a pairwise arena protocol with BTD aggregation. That combination is new. The arena-vs-rubric contrast is real: rubric scores compress into a one-point band while arena separates models by hundreds of BTD points, and the cross-judge replication (rho = 0.92) plus the ICLR-acceptance association make the ranking tool look usable.\n\nBut the load-bearing assumption is that Retrospective Context Regression produces contexts that are operationally conclusion-free. That is not as well tested as the headline suggests. The forge and audit are the same model, gpt-5.4, so an auditor can share the forger's blind spots. The human quality audit covers only 60 of 988 cases in three of six domains; biomedical pass rate is 80%, and the remaining 928 cases, including ML and Safety, get no human eyes. Contamination control is temporal only; nothing tests whether models memorized the source documents. And the structural problem stands: the contexts are built from documents written after the conclusion, so fact selection and emphasis are shaped by the eventual hypothesis. The paper disclaims exact historical recovery, but it doesn't bound the bias this introduces.\n\nNone of this kills the benchmark. The task is still meaningful, the arena protocol still ranks models more finely than rubrics, and the human-alignment data, while limited, points in the right direction. What it does mean is that the claim of isolating PHD is overstated until someone shows a larger human audit, a second forge model (or human-authored cases), and at least one memorization probe. The authors themselves list the single-model forge and text-centric data as limitations, which is to their credit.\n\nWorth a serious referee. Send it to review; expect the reviewers to push on the conclusion-free premise. I'd use it in my own work once the validity evidence thickens.","headline":"A credible first benchmark for prospective hypothesis discovery, with an arena protocol that genuinely beats rubric scoring—but the 'conclusion-free' premise is less proven than the experiments imply.","tokens_in":27379,"tokens_out":2285,"would_cite":true,"duration_ms":19366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper contends that Prospective Hypothesis Discovery—proposing grounded, testable hypotheses from incomplete, pre-conclusion evidence—is a distinct and largely unmeasured competence of large language models, and that HypoArena can meas","keywords":["prospective hypothesis discovery","hypothesis generation","LLM evaluation","benchmark","arena evaluation","pre-conclusion reasoning","open-ended output evaluation","Bradley-Terry-Davidson"],"falsifier":"Take a reader blind to the source document and present the reconstructed context; if they can recover the withheld hypothesis at a rate far above chance, the 'conclusion-free' context is leaking answer-side content. More directly, compare the reconstructed context against a genuinely contemporaneous pre-conclusion record (e.g., lab notebooks or early drafts) for the same cases; if the reconstructed contexts are not statistically interchangeable with the real ones, the benchmark measures reconstruction artifacts rather than discovery.","tokens_in":26327,"feed_emoji":"🧪","tokens_out":6539,"duration_ms":48254,"temperature":0.7,"pith_summary":"The paper is trying to establish that there is a meaningful, currently untested ability: whether an LLM, given raw evidence before any conclusion exists, can construct a space of plausible, testable hypotheses. It builds a benchmark that converts completed expert documents (papers, incident reports, analysis notes) into 'conclusion-free' contexts by stripping conclusions and causal attributions, then asks models to generate hypothesis sets. To score these open-ended outputs, it uses pairwise arena comparisons aggregated by a Bradley–Terry–Davidson model, which separates models more sharply than absolute rubric scores. The authors report clear stratification among 15 leading LLMs and rankings that align with human expert judgments. If correct, the field gains a way to measure, and eventually improve, the discovery-stage reasoning of LLMs.","feed_headline":"New benchmark measures how LLMs propose hypotheses before conclusions","feed_subtitle":"A 988-case benchmark reconstructs pre-conclusion contexts and ranks 15 models with pairwise arena judgments.","key_machinery":"Retrospective Context Regression: a Forge–Audit pipeline that reconstructs a pre-conclusion context from a completed expert document by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate; an Audit agent iteratively checks for leakage, faithfulness, and supportability. On the evaluation side, the central mechanism is a pairwise arena in which a judge compares two hypothesis sets under the same context, with position-debiased verdicts aggregated by the Bradley–Terry–Davidson model with a tie parameter to produce a global ranking.","core_discovery":"The core claim is that Prospective Hypothesis Discovery is a real, measurable competence that existing QA and idea-generation benchmarks do not capture. HypoArena operationalizes it: each case pairs a model-visible reconstructed pre-conclusion context with a withheld hypothesis–evidence reference, and evaluation compares submitted hypothesis sets head-to-head on grounding, insight, justification, breadth, distinctness, and utility. The paper reports that this arena protocol yields a stratified leaderboard across 15 models, that it distinguishes models that rubric scoring compresses into a single-point band, and that aggregated rankings strongly agree with human experts and with an independen","pith_inferences":["If Retrospective Context Regression is faithful, the same Forge–Audit loop could generate large-scale training data for pre-conclusion reasoning, letting models be fine-tuned on hypothesis generation before inference.","The arena evaluation protocol could transfer to other open-ended generation tasks (e.g., research idea evaluation, counterfactual reasoning), where multiple valid outputs make single-answer metrics misleading.","A testable extension: measure whether models' hypothesis sets change if the context is regenerated with a different random factual emphasis; the benchmark's internal consistency would predict stability across such perturbations."],"forward_implications":["If PHD is a distinct competence, then standard QA scores should not be used to infer whether a model can reason before conclusions; HypoArena supplies a separate measurement axis.","The benchmark's pairwise arena can separate models that rubric scoring cannot, suggesting that open-ended tasks generally benefit from relative comparison rather than absolute scores.","Structured analytic skills do not uniformly help: the paper finds gains for some models and regressions for others, so skill-driven prompting must be tuned per model rather than assumed beneficial.","The reference-side hypothesis sets, held out during generation, can serve as an external calibration signal, letting a fixed source-derived baseline be compared against models in the arena."],"fun_headline_variants":["New benchmark tests LLMs on pre-conclusion hypothesis discovery","Arena ranking reveals which LLMs form hypotheses best before answers","HypoArena: 988 cases to spot LLM hypothesis skills before conclusions","LLM benchmark measures hypothesis space building from ambiguous clues","Pre-conclusion thinking: new benchmark separates top LLM hypothesis proposers"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire benchmark depends on the premise that deleting conclusions and causal attributions from finished expert documents yields a context that faithfully represents the pre-conclusion reasoning state and contains no leaked cues; if the original documents do not encode such a state, or if leakage persists, the task collapses into answer restatement.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark tests LLMs on pre-conclusion hypothesis discovery","Arena ranking reveals which LLMs form hypotheses best before answers","HypoArena: 988 cases to spot LLM hypothesis skills before conclusions","LLM benchmark measures hypothesis space building from ambiguous clues","Pre-conclusion thinking: new benchmark separates top LLM hypothesis proposers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1150,"prompt_tokens":825,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":569,"tokens_out":325,"duration_ms":3682,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:23:21.859498+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a reader blind to the source document and present the reconstructed context; if they can recover the withheld hypothesis at a rate far above chance, the 'conclusion-free' context is leaking answer-side content. More directly, compare the reconstructed context against a genuinely contemporaneous pre-conclusion record (e.g., lab notebooks or early drafts) for the same cases; if the reconstructed contexts are not statistically interchangeable with the real ones, the benchmark measures reconstruction artifacts rather than discovery.","supporting_citations":[],"review_version":1}