Pith. sign in

REVIEW 2 major objections 5 minor 15 cited by

FIRE-Bench evaluates AI agents on rediscovering established scientific insights from scratch and finds every evaluated agent below 50 F1, with the bottleneck in experimental planning and conclusion formation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:11 UTC pith:6HKXBQXU

load-bearing objection FIRE-Bench's constrained-rediscovery formulation is genuinely new and useful, but the shipped Medical Bias task has a prompt–ground-truth mismatch that undercuts its headline claims and needs repair before the numbers are trusted. the 2 major comments →

arxiv 2602.02905 v2 pith:6HKXBQXU submitted 2026-02-02 cs.AI

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

Zhen Wang , Fan Bai , Zhongyan Luo , Jinyan Su , Kaiser Sun , Xinle Yu , Jieyuan Liu , Kun Zhou
show 4 more authors
Claire Cardie Mark Dredze Zhiting Hu Eric P. Xing
This is my paper
classification cs.AI
keywords FIRE-BenchAI research agentsscientific discoverybenchmark evaluationLLM agentsexperimental designclaim-level scoringerror diagnosis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FIRE-Bench, a benchmark that asks AI agents to rediscover established findings from recent high-impact machine-learning studies, giving them only a high-level research question and withholding the original experimental design and analysis path. It reports that the strongest evaluated agent reaches a mean F1 of 46.7, that runs vary widely, and that failures concentrate in research planning and conclusion formation rather than coding. A sympathetic reader should care because this is a more objective, verifiable alternative to paper-generation benchmarks and because the failure pattern points to specific capabilities—designing controlled experiments and turning data into conclusions—that need improvement before agents can do unsupervised research.

Core claim

The paper claims that full-cycle empirical research—from planning through implementation, execution, and conclusion—remains unreliable for current agent systems, with even the best system scoring below 50 on claim-level F1 and with high variance across repeated runs. It further claims that the dominant failure modes are not low-level implementation but deficiencies in Research Planning (e.g., omitting necessary control or counterfactual designs) and Conclusion Formation (e.g., drawing unsupported or overgeneralized claims), and that most false-positive conclusions are contradictory or irrelevant rather than plausible alternatives. It reports no strong evidence of data contamination after str

What carries the argument

The central mechanism is constrained rediscovery: each task is built by parsing a published empirical paper into a research-problem tree, selecting a parent node as the high-level research question, and withholding the authors' concrete methodology. The agent's final conclusion is decomposed into atomic claims and matched against the ground-truth claims from the paper using an LLM-based entailment judge, yielding precision, recall, and F1. A second piece of machinery is the four-stage error taxonomy (Research Planning, Implementation, Experimental Execution, Conclusion Formation) that attributes each missed or spurious claim to a stage, turning the benchmark into a diagnostic tool.

Load-bearing premise

The benchmark's validity rests on each task's research-question prompt and its graded ground truth targeting the same empirical finding; this fails in the Medical Bias task, where the prompt asks about cost and length-of-stay predictions but the graded finding concerns procedure recommendations and patient perception.

What would settle it

Audit all 30 FIRE-Bench tasks for prompt–ground-truth alignment; if more than a couple of tasks show the same kind of mismatch as the Medical Bias task (e.g., prompt about one outcome, ground truth about another), then the reported sub-50 F1 is partly an artifact of grading misalignment rather than a pure measure of rediscovery ability. Alternatively, rescore the Medical Bias run that used race-label-only counterfactual variation with paired comparisons and FDR correction: if a correct experimental design is scored as a false-negative 'Method Deviation,' the diagnostic framework is not measuri

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is valid, current frontier agents are not reliable enough for unattended full-cycle research; human oversight of planning and conclusion-checking remains necessary.
  • Agent development should prioritize experimental design and evidence-based reasoning, not just code generation, since the error analysis identifies those stages as the bottleneck.
  • The benchmark offers a verifiable alternative to LLM-as-judge evaluation, because ground truth is anchored to published experimental findings rather than an automated reviewer's opinion.
  • Because no consistent pre-cutoff performance advantage was found, the reported scores are unlikely to be driven by simple memorization of the source papers, though the analysis is coarse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A concrete prompt–ground-truth mismatch in the Medical Bias task (the question asks about cost and length-of-stay predictions, while the graded finding concerns procedure recommendations and perception) suggests the reported F1s may partly measure whether agents answer the question the grader expects, not the question actually asked; a systematic audit of all 30 tasks would reveal how widespread t
  • The rediscovery format inherently treats the original paper's finding as the only correct answer, so a valid alternative conclusion that answers the question is penalized; the paper's own false-positive analysis reports 'Alternative' conclusions are rare, but the design may suppress them.
  • A natural extension would be to re-score runs against question-derived ground truth rather than paper-derived ground truth, isolating evaluation misalignment from agent capability; the gap between the two scores would quantify how much of the sub-50 F1 is benchmark noise.
  • The cost–performance data, which show a cheaper agent attaining near-best F1, imply that efficiency should be reported alongside accuracy in future agent evaluations, not just headline F1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. FIRE-Bench proposes a benchmark for evaluating LLM agents on end-to-end rediscovery of central findings from 30 recent empirical analysis papers. For each task, an agent receives a high-level research question, datasets, models, and constraints, but not the original design or conclusions. Agent conclusions are decomposed into atomic claims by an LLM and matched against ground-truth claims via an entailment classifier; precision/recall/F1 are reported, with human validation on a 33% subset. Four agent configurations (OpenHands with o4-mini/gpt-5, Codex, Claude Code) are evaluated over three runs each. The paper reports that the best agent, Claude Code, achieves a mean F1 of 46.7 and that all agents show large run-to-run variance. A fine-grained error taxonomy attributes most failures to Research Planning and Conclusion Formation. Stratification by task difficulty and knowledge cutoff shows no consistent pre-cutoff advantage, which the authors interpret as weak evidence against contamination.

Significance. If the measurement is valid, FIRE-Bench would fill an important gap: it offers a scalable, claim-level, process-diagnostic alternative to both leaderboard-style metrics and LLM-judged paper generation. The benchmark is carefully curated (30 top-venue papers with open, compute-light inputs), and the release of the extraction prompts, error taxonomy, and evaluation examples is valuable. The paper also contributes a positive step by validating parts of the LLM-based evaluation with human annotations (0.89 F1 on the sampled subset) and by conducting a difficulty-stratified contamination analysis. However, the central validity condition—that each task's research prompt and its graded ground truth correspond to the same empirical finding—is violated in at least one shipped task. Because the headline numbers and the §5.1 claims about control-design failures rely on that task, the current results cannot be taken at face value. The framework is promising, but the benchmark requires an audit and re-scoring before its conclusions are accepted.

major comments (2)
  1. [§3.1, Table 6, §5.1, Appendix F Example 2, Appendix E.2] The benchmark's validity condition is that the research-question prompt and the graded ground truth target the same empirical finding. This fails for the 'LLM Racial Bias in Medicine' task. Table 6 asks whether GPT-3.5 predicts higher medical costs and longer hospital stays disproportionately for certain racial groups, but Appendix F's Example 2 ground truth is about 'significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception'—a different dependent variable. Appendix E.2 shows an agent that constructed precisely the counterfactual race-label-only control, within-patient paired comparisons, and FDR correction that §5.1 says agents 'consistently fail' to build, and reported a null result on the prompted cost/LOS outcome. The agent was scored as a false negative under 'Method Deviation.' This is not a
  2. [§3.3, §6] The paper motivates FIRE-Bench as avoiding benchmarks that 'heavily rely on LLM-as-judge evaluations' and concludes that it enables 'objective, reproducible, end-to-end assessment ... without relying on subjective paper-level judgments.' Yet the evaluation pipeline uses gpt-5.2 both to extract atomic claims and to perform semantic entailment matching between agent and ground-truth claims. The reported human validation (0.95/0.86/0.89) is on a 33% subset, but the manuscript does not specify whether this validates the extraction stage alone or the full entailment matching, nor does it report per-task agreement or disagreement categories. The Medical Bias failure in Appendix E.2 suggests that the aggregate validation did not catch the prompt-ground-truth mismatch. To support the claimed objectivity, the paper should (a) clarify the exact role of human validation in the whole pipeline, (b) r
minor comments (5)
  1. [Table 7 vs Table 1] The average F1 for OpenHands (o4-mini) is reported as 31.8±17.3 in Table 7 and as 31.9±17.6 in Table 1 and the abstract. Please reconcile the inconsistency.
  2. [Appendix numbering] Appendix cross-references are inconsistent: §5.2 says a detailed breakdown is provided in 'Appendix G,' while the error taxonomy appears in Appendix H; §3.3 refers to 'Appendix G' for full extraction prompts. Please renumber or fix the cross-references.
  3. [§3.3] The definition of ground-truth text as 'the caption and relevant prose describing the corresponding figure or table' is clear enough, but please clarify whether the parsed leaf node's 'conclusion' field (from the problem-tree extraction) is also used as ground truth or whether it is discarded.
  4. [Table 4] Several cells in the difficulty × cutoff table have n=1 or n=2 (e.g., Easy post-cutoff for gpt-5, Easy pre-cutoff for o4-mini). The paper already notes limited sample sizes, but the table would benefit from explicit confidence intervals or a warning that these cells are not interpretable.
  5. [Appendix G.2] The research input example instructs agents to 'Design and execute FULL experiments.' The word 'FULL' may induce agents to overclaim or to run unnecessarily large protocols. Consider standardizing the prompt wording and reporting sensitivity to this instruction.

Circularity Check

2 steps flagged

Medical Bias prompt–ground-truth mismatch and the Method Deviation taxonomy make part of the headline failure-mode analysis self-defined.

specific steps
  1. self definitional [§3.1, §5.2, Appendix H, Appendix F Example 2]
    "§3.1: 'The agent is provided with the research question from v∗ ... but without access to the original authors’ specific implementations or conclusions. The empirical result reported at l∗ serves as the ground truth for evaluation.' Appendix H: 'Method Deviation: Agents employ a different methodology from that used by human researchers, e.g., omitting critical control conditions or using alternative experimental designs.' Appendix F Example 2: 'Error Type: "Method Deviation"'."

    The benchmark withholds the original method by design, yet defines the most common error category as using a method different from the original. Therefore any agent that independently designs an experiment—exactly what the benchmark asks for—can be labeled a Research Planning failure by construction. The claimed result that failures are 'dominated by Research Planning and Conclusion Formation' is thus partly an artifact of the taxonomy, not an independent empirical discovery. The Medical Bias example makes this concrete: the agent built the race-label-only counterfactual control with redaction and FDR-corrected paired comparisons, yet was scored as a false negative labeled 'Method Deviation.'

  2. other [Table 6 (#1) vs. Appendix F Example 2; §5.1]
    "Table 6: 'Does the GPT-3.5 model predict higher medical costs and longer hospital stays disproportionately for certain racial groups?' Appendix F Ground Truth: 'Assessment and plans created by the model showed significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception.' Agent false negative: 'GPT-3.5 did not show statistically reliable differences in predicted total hospital cost when only race/ethnicity labels were varied...'."

    The task prompt asks about medical costs and hospital stays, while the graded ground truth concerns recommendations for more expensive procedures and patient perception. An agent that answers the actually posed question with a clean null cost/LOS result is scored as a false negative against a different dependent variable. Under the protocol, 'the empirical result reported at l∗ serves as the ground truth,' so the F1 score for this task measures prompt–ground-truth alignment rather than rediscovery of the prompted insight. This directly undermines the §5.1 statement that agents 'consistently fail to recover this control-based experimental structure,' since the counterfactual structure was built and the failure was imposed by mismatched grading.

full rationale

FIRE-Bench is not wholly circular: the benchmark tasks, human-authored ground-truth claims, and 33% human validation provide substantial independent content, and the low F1 scores reflect real agent trajectories. However, two specific reductions are present. First, the error taxonomy defines 'Method Deviation' relative to an original methodology that agents are deliberately not shown, so the dominant failure category is guaranteed to some degree by construction rather than measured from agent behavior in isolation. Second, at least one shipped task (Medical Bias) has a research question whose dependent variables (cost, hospital stay) differ from the ground-truth target (procedure recommendations, patient perception); an agent that correctly answered the prompt with the prescribed counterfactual design was scored as a false negative. This makes a portion of the reported F1 values and the 'consistent failure to construct controls' claim self-referential to the benchmark's own grading choices. The headline <50 F1 is not entirely forced, because other tasks still show genuine failures, but the benchmark's diagnostic conclusions are partly produced by the evaluation machinery rather than independently observed.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 1 invented entities

The central claim does not rest on physics-like free parameters; it rests on the reliability of the LLM measurement machinery and on the alignment of 30 prompt/ground-truth pairs. One hand-set parameter (difficulty binning) enters a secondary analysis, and its counts are internally inconsistent. Five domain assumptions are load-bearing, of which the prompt-to-ground-truth alignment assumption is demonstrably violated in at least one task.

free parameters (1)
  • difficulty-bin thresholds = Easy: S=3-4; Medium: S=5-6; Hard: S=7-9 (S = D + C + M, each axis 1-3)
    Hand-chosen binning of the three-axis rubric in Appendix C; used only for the secondary contamination stratification, not for task scores. The resulting counts disagree with Table 4 (12 vs 13 Medium; 11 vs 10 Hard).
axioms (5)
  • domain assumption gpt-5 Pro problem-tree extraction faithfully represents source-paper structure and findings
    Used to derive every task prompt and ground truth (§3.1, Eq. 1); human-scored on only 5 of 30 papers (Appendix D, scores 4.8-5.0 on a 1-5 scale).
  • domain assumption gpt-5.2 claim extraction + entailment matching approximates human judgment of rediscovery
    All F1 scores are computed by this pipeline (§3.3); human validation covers 33% of reference instances (P=0.95, R=0.86, F1=0.89, Appendix E), a subset.
  • domain assumption Published findings used as ground truth are correct and computationally verifiable
    Selection criteria (§3.2) rely on peer review (ICLR/ICML/NeurIPS 2024-25) as a proxy; the benchmark does not independently verify the underlying findings.
  • ad hoc to paper Selected parent node v* plus inherited leaf scope targets one central, gradeable finding
    Central to benchmark validity; demonstrably violated in the Medical Bias task (§5.1 vs Appendix F), where the prompt and the graded ground truth concern different outcomes.
  • domain assumption Agents did not retrieve the source paper during runs
    Stated in §4: trajectory inspection 'confirms' adherence, but no systematic filter exists (blacklists are 'technically infeasible for proprietary agents'); the contamination analysis does not test this directly.
invented entities (1)
  • Research-problem tree T(P) with root/intermediate/leaf nodes no independent evidence
    purpose: Formalizes each source paper into a hierarchy; leaf l* supplies the ground truth and parent v* supplies the agent prompt
    Produced by the authors' own gpt-5 Pro extractor and validated by the authors' own human evaluation on 5 of 30 papers; no external falsifiable handle (e.g., a fixed released artifact set) is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 26369 in / 21560 out tokens · 223788 ms · 2026-08-03T05:11:49.399811+00:00 · methodology

0 comments
read the original abstract

Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery.

Figures

Figures reproduced from arXiv: 2602.02905 by Claire Cardie, Eric P. Xing, Fan Bai, Jieyuan Liu, Jinyan Su, Kaiser Sun, Kun Zhou, Mark Dredze, Xinle Yu, Zhen Wang, Zhiting Hu, Zhongyan Luo.

Figure 1
Figure 1. Figure 1: Comparing FIRE-BENCH with other types of bench￾marks for research automation, including full paper generation (Lu et al., 2024; Yamada et al., 2025), isolated stage automation (e.g., ideation, execution, etc.), and method replication & engineer￾ing (Starace et al., 2025; Chan et al., 2024). high-impact empirical analysis paper on LLM behavior, whose central findings are well documented, peer reviewed, and … view at source ↗
Figure 2
Figure 2. Figure 2: FIRE-BENCH presents an AI research agent with a high-level research question from a published study and evaluates its ability to autonomously rediscover the study’s central empirical finding. This formulation enables fine-grained comparison of the agent’s end-to-end research process with the original human workflow. These benchmarks evaluate an agent’s ability to replicate reported experiments and results,… view at source ↗
Figure 3
Figure 3. Figure 3: Error Distribution of four evaluated agents. Different agents exhibit similar error distributions, with failures in Research Planning and Conclusion Formation accounting for the majority of errors. scientific conclusions are required. Performance varies with task structure. We observe that agent performance differs systematically across tasks with distinct structural requirements. In particular, agents ten… view at source ↗
Figure 4
Figure 4. Figure 4: Agent performance stratified by difficulty level. The clear monotonic relationship supports the validity of the proposed difficulty measure. benchmark papers that may appear in model training data. Assessing this effect is non-trivial, as publication date alone is an imperfect proxy for training exposure and may be con￾founded with task difficulty. To disentangle these factors, we analyze performance strat… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D

    cs.AI 2026-07 conditional novelty 7.0

    In a four-task benchmark of AI research agents, sabotage planted in fine-tuning data is detected less than half the time even by monitors that can run the submitted artifact.

  2. NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

    cs.CL 2026-06 unverdicted novelty 7.0

    NatureBench evaluates ten frontier AI coding agents on 90 tasks from Nature papers under web-search-disabled conditions and finds the strongest agent surpasses published SOTA on only 17.8% of tasks, succeeding mainly ...

  3. NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

    cs.CL 2026-06 accept novelty 7.0

    Frontier coding agents surpass published Nature-family SOTA on only 17.8% of 90 sealed scientific tasks, mostly by recasting problems as supervised ML rather than inventing methods.

  4. ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

    cs.CL 2026-06 unverdicted novelty 7.0

    ReproRepo uses GitHub issues as natural supervision to benchmark LLM agents on detecting reproducibility blockers across 1,149 ML papers, with the top agent finding related issues for roughly 90% of cases.

  5. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 accept novelty 7.0

    Current AI agents fail to reliably invent transferable ML methods on a controlled 140-task benchmark, and more test-time compute or context does not remove the scientific-judgment bottleneck.

  6. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 unverdicted novelty 7.0

    MLS-Bench is a benchmark with 140 tasks that evaluates AI agents on inventing generalizable and scalable ML methods, finding they lag human performance especially in insight-driven invention rather than tuning.

  7. Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery

    cs.CL 2026-07 conditional novelty 6.0

    HypoArena is a 988-case benchmark that asks LLMs to generate hypothesis sets from conclusion-free reconstructed contexts and ranks 15 models via pairwise arena judgments.

  8. Towards Autonomous and Auditable Medical Imaging Model Development

    cs.CV 2026-07 conditional novelty 6.0

    AMID, a verification-guided multi-agent MLE system for medical imaging, outperforms general MLE agents on 20 ReX-MLE challenges and approaches human challenge solutions on several tasks.

  9. Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements

    cs.AI 2026-06 unverdicted novelty 6.0

    Closed-loop LM-agent auto research finds some transferable gains on molecular property prediction benchmarks via external data but shows non-transfer for model and feature edits selected on validation.

  10. ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

    cs.AI 2026-05 unverdicted novelty 6.0

    ScientistOne introduces Chain-of-Evidence and an audit system that achieves zero hallucinated references, perfect score verification, and top method-code alignment while matching or beating human experts on five front...

  11. MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

    cs.LG 2026-05 unverdicted novelty 6.0

    MLS-Bench shows that current AI agents fall short of reliably inventing generalizable ML methods, with engineering tuning easier than genuine invention.

  12. CellScientist: Dual-Space Hierarchical Orchestration for Closed-Loop Refinement of Virtual Cell Models

    cs.LG 2026-05 unverdicted novelty 6.0

    CellScientist introduces a dual-space hierarchical orchestration system that enables closed-loop refinement of virtual cell models by routing execution discrepancies back to hypothesis or implementation updates, yield...

  13. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.5

    A multi-LLM consensus pipeline turns 14,450 auto-generated candidate rubrics into 2,600 distinguishable gold rubrics that rank 10 financial deep-research systems from 58.58% to 22.23% pass rate.

  14. FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality

    cs.CL 2026-07 conditional novelty 5.0

    A benchmark whose 2,600 'gold' rubrics are generated, validated, and applied entirely by LLMs — with no human in the final loop — differentiates 10 financial deep-research systems across a 36-point pass-rate spread.

  15. AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery

    cs.AI 2026-05 unverdicted novelty 4.0

    A survey organizing AI-powered research automation into five workflow stages, defining AutoResearch and Vibe Research, and proposing five evaluation dimensions while noting domain-conditioned limits on autonomy.

Reference graph

Works this paper leans on

43 extracted references · 7 linked inside Pith · cited by 11 Pith papers

  1. [1]

    2.Experiment Completeness: Whether all key experiments from the paper are captured in the tree structure

    Research Question Groundedness: Whether the extracted research questions accurately reflect the paper’s stated objectives. 2.Experiment Completeness: Whether all key experiments from the paper are captured in the tree structure

  2. [2]

    URL https://openreview.net/forum? id=eb5pkwIB5i. Poster. Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. Chemcrow: Augmenting large- language models with chemistry tools.arXiv preprint arXiv:2304.05376, 2023. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwa...

    Pith/arXiv arXiv 2023

  3. [3]

    4.Structural Coherence: Whether the hierarchical decomposition follows a logical parent-child relationship

    Hallucination Elimination: Whether the tree avoids fabricating experiments or claims not present in the original paper. 4.Structural Coherence: Whether the hierarchical decomposition follows a logical parent-child relationship

  4. [4]

    URL https://openreview.net/forum? id=8zxGruuzr9. Poster. 10 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Ru, D., Qiu, L., Hu, X., Zhang, T., Shi, P., Chang, S., Ji- ayang, C., Wang, C., Sun, S., Li, H., Zhang, Z., Wang, B., Jiang, J., He, T., Wang, Z., Liu, P., Zhang, Y ., and Zhang, Z. RAGChecker: A fine-grained framework for d...

    Pith/arXiv arXiv 2024

  5. [5]

    URL https://openreview.net/forum? id=w6nlcS8Kkn. Poster. Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025. Stechly, K., Valmeekam, K., and Kambhampati, S. Chain of thoughtlessness...

    Pith/arXiv arXiv 2025

  6. [6]

    Wang, X., Chen, Y ., Yuan, L., Zhang, Y ., Li, Y ., Peng, H., and Ji, H

    URL https://openreview.net/forum? id=4Zt7S0B0Jp. Wang, X., Chen, Y ., Yuan, L., Zhang, Y ., Li, Y ., Peng, H., and Ji, H. Executable code actions elicit better LLM agents. InForty-first International Conference on Ma- chine Learning, 2024. URL https://openreview. net/forum?id=jJ9BoXAfFa. Wang, X., Li, B., Song, Y ., Xu, F. F., Tang, X., Zhuge, M., Pan, J....

    2024

  7. [7]

    Weng, Y ., Zhu, M., Bao, G., Zhang, H., Wang, J., Zhang, Y ., and Yang, L

    URL https://openreview.net/forum? id=OJd3ayDDoF. Weng, Y ., Zhu, M., Bao, G., Zhang, H., Wang, J., Zhang, Y ., and Yang, L. Cycleresearcher: Improving auto- mated research via automated review.arXiv preprint arXiv:2411.00816, 2024. Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., et al. Re- b...

    Pith/arXiv arXiv 2024

  8. [8]

    11 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Xu, T., Lu, P., Ye, L., Hu, X., and Liu, P

    URL https://openreview.net/forum? id=gjeQKFxFpZ. 11 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Xu, T., Lu, P., Ye, L., Hu, X., and Liu, P. Researcherbench: Evaluating deep ai research systems on the frontiers of sci- entific inquiry.arXiv preprint arXiv:2507.16280, 2025. Yamada, Y ., Lange, R. T., Lu, C., Hu, S., Lu, C., Foers...

    Pith/arXiv arXiv 2025

  9. [11]

    The results, shown in Table 9, demonstrate consistently high scores across all aspects, confirming the quality and reliability of the LLM-generated problem trees used in FIRE-Bench

    Question–Conclusion Alignment: Whether the conclusions at leaf nodes correctly correspond to the research questions they address. The results, shown in Table 9, demonstrate consistently high scores across all aspects, confirming the quality and reliability of the LLM-generated problem trees used in FIRE-Bench. Table 9.Human evaluation of problem-tree pars...

  10. [12]

    Placing gold document early boosts accuracy

  11. [13]

    Middle position performs worst

  12. [14]

    RAGChecker atomized claims:

    Last position shows a mild rebound (possible recency). RAGChecker atomized claims:

  13. [15]

    Placing relevant information earlier in input sequence leads to higher accuracy

  14. [16]

    Relevant information in middle positions performs worst

  15. [17]

    Metrics:TP = 3, Precision = 1.0, Recall = 1.0, F1 = 1.0 Justification:Human claim 1 matches RAGChecker claim 1 (both assert early placement improves accuracy)

    Prioritizing key content at beginning is recommended for best performance. Metrics:TP = 3, Precision = 1.0, Recall = 1.0, F1 = 1.0 Justification:Human claim 1 matches RAGChecker claim 1 (both assert early placement improves accuracy). Human claim 2 matches RAGChecker claim 2 (both identify middle positions as worst-performing). Human claim 3 matches RAGCh...

    2024

  16. [18]

    GPT-3.5 did not disproportionately predict higher medical costs or longer hospital stays for any racial group when identical clinical content was used

  17. [19]

    Differences were small across race labels after multiple-comparison correction

  18. [20]

    Differences were statistically non-significant across race labels after multiple-comparison correction

  19. [21]

    RAGChecker atomized claims:

    The model resulted in frequent identical outputs across race labels. RAGChecker atomized claims:

  20. [22]

    GPT-3.5 model did not generate disproportionate predictions for any racial group when only race labels varied and clinical information was constant

  21. [23]

    Differences in predictions across race labels were small

  22. [24]

    Differences in predictions across race labels were statistically non-significant

  23. [25]

    Model outputs across race labels were frequently identical. Metrics:TP = 4, Precision = 1.0, Recall = 1.0, F1 = 1.0 22 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Justification:All four claims have direct one-to-one correspondence. Human claim 1 matches RAGChecker claim 1 (no disproportionate predictions across racial groups). ...

  24. [26]

    Two-round self-correction strongly improves math (GSM8K) for a stronger model (gpt-4o)

  25. [27]

    Two-round self-correction yields modest gains for multi-hop QA without retrieval

  26. [28]

    Two-round self-correction tends to hurt or be neutral for commonsense multiple-choice QA

  27. [29]

    For gpt-3.5-turbo, effects are neutral/negative

  28. [30]

    RAGChecker atomized claims:

    Gpt-3.5-turbo resulted in small, inconsistent Hotpot improvements. RAGChecker atomized claims:

  29. [31]

    Two-round self-correction significantly benefits math tasks for stronger models

  30. [32]

    23 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights

    Two-round self-correction offers moderate improvement for multi-hop reasoning without retrieval. 23 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights

  31. [33]

    Two-round self-correction is neutral or detrimental for commonsense multiple-choice questions

  32. [34]

    Models better at using relevant information at beginning of input context and the performance drops in later position

    Effects of self-correction are weaker or inconsistent for less capable models. Metrics:TP = 4, FN = 1, Precision = 1.0, Recall = 0.80, F1 = 0.89 Justification:Human claim 1 matches RAGChecker claim 1 (math improvement for stronger models). Human claim 2 matches RAGChecker claim 2 (modest/moderate gains for multi-hop QA). Human claim 3 matches RAGChecker c...

  33. [35]

    **Locate the root ** in the title, abstract, introduction, or discussion

  34. [36]

    first... second

    **Recursively decompose ** each problem by following explicit textual cues ( headings, "first... second...", "to this end...", method overviews, figure/table captions, bullet lists, etc.)

  35. [37]

    **Identify leaves **: a node is a leaf *only if * it describes a concrete experiment and you can cite the corresponding Figure / Table / Section ID

  36. [38]

    **Capture all layers **--do **not** skip intermediate hypotheses, objectives, or analysis steps the paper explicitly discusses

  37. [39]

    **Stay faithful ** to the paper’s wording for technical terms; paraphrase only for brevity or clarity

  38. [40]

    paper": {

    **No outside invention **--derive every node from the paper alone. If information is missing, mark the node with [uncertain]. 25 FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Paper Parsing Prompt (Cont.) Strictly output the tree in a JSON format: ‘‘‘ { "paper": { "title": "", "authors": [], "venue": "", "year": "" }, "problem_tre...

  39. [42]

    {gt}". And the false negative conclusion missed by AI research agent:

    The logged trajectory of an AI agent doing research about the same question. The correct conclusion found by human researchers is: "{gt}". And the false negative conclusion missed by AI research agent: "{f_statment}". Based on the original research of human researchers and the logged trajectory of AI research agent, what error did the agent make so it get...

  40. [43]

    The original paper of research question "{query}"

  41. [44]

    {gt}". And the false positive conclusion generated by an AI research agent:

    The logged trajectory of an AI agent doing research about the same question. The correct conclusion found by human researchers is: "{gt}". And the false positive conclusion generated by an AI research agent: "{f_statment}". Based on the original research of human researchers and the logged trajectory of AI research agent, what error did the agent make so ...

  42. [2024]

    URL https:// aclanthology.org/2024.tacl-1.9/

    doi: 10.1162/tacl_a_00638. URL https:// aclanthology.org/2024.tacl-1.9/. Liu, R., Geng, J., Peterson, J., Sucholutsky, I., and Griffiths, T. L. Large language models assume peo- ple are more rational than we really are. InInterna- tional Conference on Learning Representations (ICLR), 2025a. URL https://openreview.net/forum? id=dAeET8gxqg. Poster. Liu, Y ....

    Pith/arXiv arXiv 2024

  43. [2025]

    URL https://openreview.net/forum? id=p2smPMRQae. Poster. Baek, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models.arXiv preprint arXiv:2404.07738, 2024. Binder, F. J., Chua, J., Korbak, T., Sleight, H., Hughes, J., Long, R., Perez, E., Turpin, M., and Ev...

    Pith/arXiv arXiv 2024