Pith. sign in

REVIEW 4 major objections 5 minor 33 references

This paper contends that Prospective Hypothesis Discovery—proposing grounded, testable hypotheses from incomplete, pre-conclusion evidence—is a distinct and largely unmeasured competence of large language models, and that HypoArena can meas

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:23 UTC pith:2IGVGEON

load-bearing objection A credible first benchmark for prospective hypothesis discovery, with an arena protocol that genuinely beats rubric scoring—but the 'conclusion-free' premise is less proven than the experiments imply. the 4 major comments →

arxiv 2607.15766 v1 pith:2IGVGEON submitted 2026-07-17 cs.CL

Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

Tianyun Zhong , Wangyi Jiang , Wei Wang , Xuanang Chen , Yaojie Lu , Shiwei Ye , Yuzhen Shi , Boyu Yang
show 10 more authors
Jinghang Wang Han Li Weiqi Zhai Bing Zhao Hu Wei Haiyang Yu Yongbin Li Hongyu Lin Le Sun Xianpei Han
This is my paper
classification cs.CL
keywords prospective hypothesis discoveryhypothesis generationLLM evaluationbenchmarkarena evaluationpre-conclusion reasoningopen-ended output evaluationBradley-Terry-Davidson
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that there is a meaningful, currently untested ability: whether an LLM, given raw evidence before any conclusion exists, can construct a space of plausible, testable hypotheses. It builds a benchmark that converts completed expert documents (papers, incident reports, analysis notes) into 'conclusion-free' contexts by stripping conclusions and causal attributions, then asks models to generate hypothesis sets. To score these open-ended outputs, it uses pairwise arena comparisons aggregated by a Bradley–Terry–Davidson model, which separates models more sharply than absolute rubric scores. The authors report clear stratification among 15 leading LLMs and rankings that align with human expert judgments. If correct, the field gains a way to measure, and eventually improve, the discovery-stage reasoning of LLMs.

Core claim

The core claim is that Prospective Hypothesis Discovery is a real, measurable competence that existing QA and idea-generation benchmarks do not capture. HypoArena operationalizes it: each case pairs a model-visible reconstructed pre-conclusion context with a withheld hypothesis–evidence reference, and evaluation compares submitted hypothesis sets head-to-head on grounding, insight, justification, breadth, distinctness, and utility. The paper reports that this arena protocol yields a stratified leaderboard across 15 models, that it distinguishes models that rubric scoring compresses into a single-point band, and that aggregated rankings strongly agree with human experts and with an independen

What carries the argument

Retrospective Context Regression: a Forge–Audit pipeline that reconstructs a pre-conclusion context from a completed expert document by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate; an Audit agent iteratively checks for leakage, faithfulness, and supportability. On the evaluation side, the central mechanism is a pairwise arena in which a judge compares two hypothesis sets under the same context, with position-debiased verdicts aggregated by the Bradley–Terry–Davidson model with a tie parameter to produce a global ranking.

Load-bearing premise

The entire benchmark depends on the premise that deleting conclusions and causal attributions from finished expert documents yields a context that faithfully represents the pre-conclusion reasoning state and contains no leaked cues; if the original documents do not encode such a state, or if leakage persists, the task collapses into answer restatement.

What would settle it

Take a reader blind to the source document and present the reconstructed context; if they can recover the withheld hypothesis at a rate far above chance, the 'conclusion-free' context is leaking answer-side content. More directly, compare the reconstructed context against a genuinely contemporaneous pre-conclusion record (e.g., lab notebooks or early drafts) for the same cases; if the reconstructed contexts are not statistically interchangeable with the real ones, the benchmark measures reconstruction artifacts rather than discovery.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If PHD is a distinct competence, then standard QA scores should not be used to infer whether a model can reason before conclusions; HypoArena supplies a separate measurement axis.
  • The benchmark's pairwise arena can separate models that rubric scoring cannot, suggesting that open-ended tasks generally benefit from relative comparison rather than absolute scores.
  • Structured analytic skills do not uniformly help: the paper finds gains for some models and regressions for others, so skill-driven prompting must be tuned per model rather than assumed beneficial.
  • The reference-side hypothesis sets, held out during generation, can serve as an external calibration signal, letting a fixed source-derived baseline be compared against models in the arena.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If Retrospective Context Regression is faithful, the same Forge–Audit loop could generate large-scale training data for pre-conclusion reasoning, letting models be fine-tuned on hypothesis generation before inference.
  • The arena evaluation protocol could transfer to other open-ended generation tasks (e.g., research idea evaluation, counterfactual reasoning), where multiple valid outputs make single-answer metrics misleading.
  • A testable extension: measure whether models' hypothesis sets change if the context is regenerated with a different random factual emphasis; the benchmark's internal consistency would predict stability across such perturbations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Prospective Hypothesis Discovery (PHD), a task in which a model constructs grounded, discriminative, and testable hypothesis sets from pre-conclusion contexts, and presents HypoArena, a benchmark of 988 cases across six domains. HypoData is built via Retrospective Context Regression, a Forge–Audit pipeline that strips explicit conclusions from expert documents while preserving factual material. HypoEval combines pairwise LLM judging with Bradley–Terry–Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. The paper reports evaluations of 15 frontier LLMs, finding large BTD spreads, model-dependent effects of structured analytical skills, and correlations with human expert judgments and an external ICLR-acceptance signal. The central claim is that PHD is a distinct, largely unmeasured competence of LLMs.

Significance. If the construct validity of the reconstructed contexts can be established, HypoArena would be a genuinely useful public benchmark, filling a gap between QA benchmarks and open-ended idea-generation benchmarks. The paper is methodologically careful in several ways: it ships code and data, applies leakage/faithfulness/supportability audits, includes a human quality audit, replicates rankings with a second judge, and tests association with an independent peer-review signal. These are real strengths. The main risk is that the central premise—that Retrospective Context Regression yields operationally conclusion-free contexts—is under-tested. Since leakage or memorization would collapse PHD into answer-restatement, the benchmark's validity depends on closing this gap. With the additional validation described below, the paper could support its strong claims; as written, those claims outrun the evidence.

major comments (4)
  1. [§2.3, Appendix A, Tables 2/6] The benchmark's central validity rests on contexts that do not leak the held-out hypothesis. Current evidence is insufficient. Forge and Audit are both gpt-5.4 (Appendix A concedes single-model construction), so the auditor may share the forger's blind spots. The human audit covers only 60 of 988 cases and only Biomedical, Financial, and Social Science; Machine Learning, IT Operations, and Safety Investigation are unaudited. The Biomedical pass rate is 80% (Table 6), i.e., 4 of 20 audited cases failed. Because leakage would collapse PHD to answer-restatement, please add multi-model and/or human-in-the-loop construction for a validation subset, expand human audit to all six domains with adequate samples, and run explicit memorization/concealment tests (e.g., whether models can recover the reference hypothesis from context alone, or reproduce source conclusions above chance).
  2. [Appendices B/C; §2.2] Temporal cutoffs are the only contamination control. They exclude post-cutoff retrieval but do not test whether evaluated models memorized the source documents (ICLR 2026 submissions, NTSB/CSB reports, 2025–2026 papers). For IT Operations, the paper assumes unique incident configurations make temporal contamination low, but this is an assumption, not a measurement. Without a direct test—for example, a fuzzy-match or cloze-style probe comparing model outputs with the source-derived reference—one cannot rule out that apparent PHD performance partly reflects recitation. This is load-bearing for all subsequent rankings.
  3. [§2.1, Introduction] Even if explicit conclusions are removed, contexts are assembled from documents written after the conclusion; the selection, ordering, and emphasis of facts is therefore informed by the eventual hypothesis. The paper acknowledges this in the Introduction ('Rather than recovering the exact historical information state') but does not bound the resulting bias. Because PHD is supposed to measure pre-conclusion reasoning, please validate construction against an independent pre-conclusion source in at least a subset of cases, or show that a context built from a randomly sampled factual subset of the source yields similar conclusions. Without such a check, the task may reward reconstructing the source's post-hoc narrative rather than discovering from genuinely inconclusive evidence.
  4. [Table 3; §4.4; Table 12] The claim that arena evaluation 'resolves finer-grained differences among models' needs more than point estimates. Table 3 reports BTD ratings without confidence intervals or significance tests; adjacent systems are separated by as little as ~10–40 points, and pairwise cross-judge agreement is only 63–68% (Table 12). Per-domain human-judge alignment drops to τ=0.53 for Biomedical (Table 10). The aggregated rank correlations are encouraging, but the fine-grained separations may be within noise. Please report bootstrap or posterior intervals for BTD ratings and test whether adjacent ranks are statistically distinguishable. This is needed to support both 'clear capability stratification' and the 'finer-grained differences' claims.
minor comments (5)
  1. [Abstract] The GitHub URL is duplicated; one of the two should be the HuggingFace dataset link.
  2. [Figure 4] The axis labels and caption appear truncated; the per-domain Kendall τ values mentioned in the text (0.53, 0.97, 0.85) should be visible in the figure or caption.
  3. [Table 2 vs Table 6] The same human quality audit results appear as Table 2 and Table 6; unify the numbering to avoid confusion.
  4. [§3.2] The mapping from 5-level verdicts to win shares {1.0, 0.75, 0.5, 0.25, 0.0} and the choice of BTD tie parameter θ are not sensitivity-analyzed; a short robustness check would strengthen the evaluation.
  5. [§4.3] The text says 'across all three reported measures' but the preceding sentence in the appendix lists three; in the main text this is clear only after reading Appendix H.2. Consider stating the three measures explicitly.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's construction and evaluation are empirical, and its limitations are validity threats rather than definitional reductions.

full rationale

The paper does not contain a load-bearing derivation in which a claimed result is equivalent to an input by construction. Retrospective Context Regression is an LLM-based data-construction pipeline, not a mathematical derivation; the Forge-Audit loop produces contexts and reference hypotheses from source documents, and the reference is explicitly withheld during generation. The arena rankings are directly aggregated from pairwise LLM judgments via Bradley-Terry-Davidson; no parameter is fitted to a subset of data and then presented as a prediction of a closely related quantity. The paper provides external anchor points: a human quality audit (Table 6), human preference alignment (Table 10, Kendall tau = 0.90 overall), a second LLM judge (Appendix I), and an ICLR acceptance association (Appendix H.2) that uses an outcome not seen by the judge. The main caveats—single-model Forge and Audit (Appendix A), a 60-case human audit covering only three of six domains (Section 2.4, Table 6), and temporal rather than memorization-based contamination controls (Appendix B)—are genuine threats to the construct validity of 'conclusion-free' contexts, but they are not circular reductions: no equation, fitted parameter, or self-citation chain makes the benchmark's central claims equivalent to its own inputs. There are no load-bearing self-citations; all cited works are external. Therefore, under the hard rules requiring an exhibited specific reduction, no significant circularity is present.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical or formal entities are introduced; PHD is a task construct, and HypoData/HypoEval are artifacts rather than postulated entities. The central empirical claims depend on the listed evaluation design choices and on untested assumptions about leakage and contamination.

free parameters (5)
  • Win-share mapping = {1.0, 0.75, 0.5, 0.25, 0.0}
    Hand-chosen mapping from categorical pairwise verdicts to BTD win-shares; the leaderboard values depend on it. Robustness to this mapping is not reported.
  • BTD tie parameter theta = not reported
    Bradley-Terry-Davidson model includes a tie parameter; paper does not state its value or sensitivity, though tie handling affects ratings.
  • Rubric equal-weight aggregation = Qpair=(g+l+j)/3; Qset=(b+d+u)/3
    Diagnostic rubric scores are unweighted averages; alternative weights could change per-dimension profiles.
  • Domain-specific temporal cutoffs = 2025-2026 for most; none for IT Operations
    Hand-set cutoffs intended to reduce contamination; no sensitivity analysis is provided, and IT post-mortems have no cutoff.
  • Case cardinality K = 1 (scientific), open (analytical)
    Domain-dependent output cardinality is a design choice that shapes what is scored; single-conjecture cases may undervalue breadth.
axioms (4)
  • domain assumption A pre-conclusion information state is recoverable from post-hoc expert documents by deleting explicit conclusions and retrospective causal attributions.
    The entire HypoData construction rests on this reconstruction premise; validated only on a 60-case human sample (§2.4).
  • domain assumption The Audit agent's leakage/faithfulness/supportability checks are sufficient to prevent conclusion leakage.
    Leakage freedom is enforced by an LLM audit plus a small human check; no systematic test is reported.
  • domain assumption Temporal cutoffs prevent the evaluated models from having memorized the source documents.
    No contamination test is reported and IT Operations imposes no cutoff (Appendix B).
  • domain assumption LLM pairwise judgments are a valid proxy for expert judgment of open-ended hypothesis sets.
    Supported by 1,500 human judgments in 3 domains, but per-domain Kendall tau ranges 0.53-0.97 and cross-judge pairwise agreement is 63-68%.

pith-pipeline@v1.3.0-alltime-deepseek · 26139 in / 13768 out tokens · 109122 ms · 2026-08-01T22:23:21.859498+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured. We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discriminative, and testable hypothesis spaces from inconclusive evidence, including anomalous observations and fragmented records, to guide subsequent investigation. To evaluate this capability, we introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets. To construct HypoData at scale, we propose Retrospective Context Regression, a Forge--Audit pipeline that reconstructs pre-conclusion contexts from completed expert documents by removing explicit conclusions, target hypotheses, and retrospective causal attributions while preserving the factual substrate. Because PHD admits multiple valid outputs, HypoEval combines bidirectional pairwise judgments with Bradley--Terry--Davidson aggregation for ranking and six-dimensional rubric scoring for diagnosis. Experiments on 15 frontier LLMs reveal clear capability stratification and model-dependent effects of structured analytical skills, with gains for several lower-performing models on HypoArena but regressions for other systems, including a top-performing model. Compared with absolute rubric scoring, arena evaluation resolves finer-grained differences among models, with aggregated rankings showing strong agreement with human experts and an independent judge. Together, these results support treating PHD as a distinct target for evaluating how LLMs formulate investigative directions when final conclusions are withheld. Our code and data are publicly available at github.com/SKYLENAGE-AI/HypoArena and github.com/SKYLENAGE-AI/HypoArena.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 5 linked inside Pith

  1. [1]

    Dabstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719, 2025

    Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. Dabstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Dsbench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703, 2024

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Kosmos: An ai scientist for autonomous discovery

    Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C Landsness, Daniel L Barabasi, Siddharth Narayanan, Nicky Evans, et al. Kosmos: An ai scientist for autonomous discovery. arXiv preprint arXiv:2511.02824, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Discoverybench: Towards data-driven discovery with large language models

    Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  5. [5]

    Gaurav Sahu, Abhay Puri, Juan A. Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar, and Issam H. Laradji. Insightbench: Evaluating business analytics agents through multi-step insight generation. InThe Thirteenth In...

    2025

  6. [6]

    Hypobench: Towards systematic and principled benchmarking for hypothesis generation, 2026

    Haokun Liu, Sicong Huang, Jingyu Hu, Yangqiaoyu Zhou, and Chenhao Tan. Hypobench: Towards systematic and principled benchmarking for hypothesis generation, 2026

    2026

  7. [7]

    Sciarena: An open evaluation platform for non-verifiable scientific literature-grounded tasks

    Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Yixin Liu, Xiangru Tang, Joseph Chee Chang, Jesse Dodge, Jonathan Bragg, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. Sciarena: An open evaluation platform for non-verifiable scientific literature-grounded tasks. InThe Thirty-ninth Annual Conference on Neural Information Pr...

    2025

  8. [8]

    Rahmani, Yanshan Wang, Qiang Zhang, Keyan Ding, Jeff Z

    Shuofei Qiao, Yunxiang Wei, Xuehai Wang, Bin Wu, Boyang Xue, Ningyu Zhang, Hossein A. Rahmani, Yanshan Wang, Qiang Zhang, Keyan Ding, Jeff Z. Pan, Huajun Chen, and Emine Yilmaz. Innoeval: On research idea evaluation as a knowledge-grounded, multi-perspective reasoning problem, 2026

    2026

  9. [9]

    Fire-bench: Evaluating agents on the rediscovery of scientific insights.arXiv preprint arXiv:2602.02905, 2026

    Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, et al. Fire-bench: Evaluating agents on the rediscovery of scientific insights.arXiv preprint arXiv:2602.02905, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Williams, Stefan Bekiranov, and Aidong Zhang

    Sikun Guo, Amir Hassan Shariatmadari, Guangzhi Xiong, Albert Huang, Myles Kim, Corey M. Williams, Stefan Bekiranov, and Aidong Zhang. Ideabench: Benchmarking large language models for research idea generation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page 5888–5899, New York, NY, USA, 2025. Asso...

    2025

  11. [11]

    Ai idea bench 2025: Ai research idea generation benchmark, 2025

    Yansheng Qiu, Haoquan Zhang, Zhaopan Xu, Ming Li, Diping Song, Zheng Wang, and Kaipeng Zhang. Ai idea bench 2025: Ai research idea generation benchmark, 2025

    2025

  12. [12]

    Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition, 2025

    Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition, 2025

    2025

  13. [13]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109, 2024

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002

    2002

  15. [15]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81, 2004

    2004

  16. [16]

    G-eval: Nlg evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023

    2023

  17. [17]

    On extending the bradley-terry model to accommodate ties in paired comparison experiments

    Roger R Davidson. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329):317–328, 1970

    1970

  18. [18]

    Cq Press, 2019

    Randolph H Pherson and Richards J Heuer Jr.Structured analytic techniques for intelligence analysis. Cq Press, 2019

    2019

  19. [19]

    Moose-chem2: Exploring llm limits in fine-grained scientific hypothesis discovery via hierarchical search, 2025

    Zonglin Yang, Wanhao Liu, Ben Gao, Yujie Liu, Wei Li, Tong Xie, Lidong Bing, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. Moose-chem2: Exploring llm limits in fine-grained scientific hypothesis discovery via hierarchical search, 2025

    2025

  20. [20]

    Literature meets data: A synergistic approach to hypothesis generation

    Haokun Liu, Yangqiaoyu Zhou, Mingxuan Li, Chenfei Yuan, and Chenhao Tan. Literature meets data: A synergistic approach to hypothesis generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 245–...

    2025

  21. [21]

    Hy- pER: Literature-grounded hypothesis generation and distillation with provenance

    Rosni Vasu, Chandrayee Basu, Bhavana Dalvi Mishra, Cristina Sarasua, Peter Clark, and Abraham Bernstein. Hy- pER: Literature-grounded hypothesis generation and distillation with provenance. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language ...

    2025

  22. [22]

    Toward reliable scientific hypothesis generation: Evaluating truthfulness and hallucination in large language models

    Guangzhi Xiong, Eric Xie, Corey Williams, Myles Kim, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Toward reliable scientific hypothesis generation: Evaluating truthfulness and hallucination in large language models. In James Kwok, editor,Proceedings of the Thirty-FourthInternational Joint Conference on Artificial Intelligence,...

    2025

  23. [23]

    Grapheval: A lightweight graph-based LLM framework for idea evaluation

    Tao Feng, Yihang Sun, and Jiaxuan You. Grapheval: A lightweight graph-based LLM framework for idea evaluation. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  24. [24]

    No" (indicating a singular reasonable direction) is acceptable for specific deterministic cases, a high

    Jiefu Ou, William Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel 11 Khashabi, and Benjamin Van Durme. CLAIMCHECK: How grounded are LLM critiques of scientific papers? In Christos Christodoulopoulos, Tanmoy Chakrab...

    arXiv 2025

  25. [25]

    there is enough background and tension here to support a serious hypothesis, but the answer has not been given away

    Model Outputs. Two candidate sets of hypotheses and evidence (Output A and Output B), randomized and stripped of model metadata. 3.Evaluation Metrics. The standardized rubric used for assessment (see Appendix F.2). Labeling Requirements.For each pairwise matchup, annotators must provide: •Preference. Select from A>B, Tie, B>A. •Confidence Score. Annotator...

    arXiv 2026

  26. [26]

    Leaks answer-side content

  27. [27]

    Does not go beyond the source paper’s local framing

  28. [28]

    Carries retrieval residue (citations, URLs, bibliography)

  29. [29]

    passed”: true} or {“passed

    Too thin to support non-trivial hypothesis generation. 25 Output: {“passed”: true} or {“passed”: false, “summary”: “...”, “problems”: [...]} Context Audit — Safety Investigation You are AuditAgent. Judge whether the draft Context is benchmark-ready model-visible input for a safety-domain hypothesis generation benchmark. What Benchmark-Ready Safety Context...

  30. [30]

    Leaks the agency’s final probable cause or root-cause determination

  31. [31]

    caused,” “contributed to

    Contains analysis-layer causal language (“caused,” “contributed to”)

  32. [32]

    Reads like a summarized finding rather than a factual record

  33. [33]

    passed”: true} or {“passed

    Too thin to support non-trivial investigative hypothesis generation. Audit Philosophy: Protect factual density, timeline fidelity, unresolved tension, and answer-side separation. Do not ask for better prose or methodology sections. Output: {“passed”: true} or {“passed”: false, “summary”: “...”, “problems”: [...]} J.2 Stage 2: Hypothesis Construction J.2.1...