Pith. sign in

REVIEW 3 major objections 6 minor 33 references

VERITAS is a domain-agnostic coding-agent pipeline that replicates scientific papers from a paper, code, or both, and leads strong same-model baselines on two multi-domain replication benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 06:00 UTC pith:QBKTXD5H

load-bearing objection Solid systems paper: a real general-purpose replication pipeline that beats matched Claude Code baselines, with the SOTA native-score wording a bit ahead of the single-run margins. the 3 major comments →

arxiv 2607.02931 v1 pith:QBKTXD5H submitted 2026-07-03 cs.AI

VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

Haokun Liu , Filbert Aurelian Tjiaranata , Chenhao Tan This is my paper
classification cs.AI
keywords scientific replicationcoding agentscomputational reproducibilityclaim verificationagent pipelinespaper-to-codeautomated verificationreplication benchmarks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Independent verification of published research is growing more important as AI speeds publication and review systems fall behind, yet manual replication remains slow and costly. Most existing agent-based efforts ship as fixed benchmarks whose companion agents only run inside those pipelines, and no general-purpose replication tool has been available. This paper presents VERITAS, a domain-agnostic framework built around command-line coding agents. Given a paper, a repository, or both, it extracts structured claims, plans and runs the methodology while repairing failures as they arise, rates every fix, and judges each claim against run evidence. The pipeline returns an importance-weighted Replication Score, a severity-rated fix log, and a patched codebase. On CORE-Bench and ReplicationBench—65 papers across computer science, social science, medicine, and astrophysics—VERITAS leads two strong coding-agent baselines run on the same model and host on every reported metric, including a high per-capsule pass rate on CORE-Bench Hard and the best paper-only and full-mode match rates on ReplicationBench after a uniform cheating correction.

Core claim

The authors claim that a single multi-phase replication framework around CLI coding agents can handle flexible inputs (paper plus code, paper only, or code only), actively fix execution failures, and verify paper claims against experimental evidence in a domain-agnostic way—and that this design reaches state-of-the-art results and leads strong same-model baselines on every metric across two multi-domain replication benchmarks covering 65 papers.

What carries the argument

The six-phase pipeline: ANALYZE extracts typed, importance-weighted claims (withheld from the replicating agent); optional CODEGEN builds a codebase from the paper alone; PLAN drafts claim-linked steps without paper values; REPLICATE executes with active fixing under a bounded manager loop; ASSESS FIXES rates each repair; VERIFY extracts replicated values and grades them into an importance-weighted Replication Score.

Load-bearing premise

The general-purpose claim rests mainly on beating same-family coding-agent baselines on two fixed benchmarks, with claim extraction and the user-facing Replication Score bypassed in the scored runs and results reported from single trajectories.

What would settle it

Re-run VERITAS and the same baselines on both benchmarks under multi-seed conditions with the same cheating correction: if VERITAS no longer leads on native pass or match rates, faithfulness, and cheating counts, the performance claim fails; if the full claim-extraction and scoring path systematically fails on papers outside those suites, the general-purpose claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Computational claims in published papers can be checked end-to-end without building a new agent for each benchmark or domain.
  • Paper-only versus code-available replication can be compared under one tool to measure how much authors’ source code closes the gap.
  • Severity-rated fix logs make the hidden cost of getting released code to run visible as part of a replication report.
  • Trajectory checks for forbidden access and execution-grounded faithfulness can sit beside native scores when judging agent replications.
  • A single patched codebase plus per-claim verdicts can be handed to reviewers or auditors as a concrete verification artifact.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • As submission volumes rise, automated claim-level replication could become a routine pre- or post-review check rather than a rare manual project.
  • Withholding reported values from the executing agent is a reusable pattern for reducing leakage in any agentic evaluation of scientific results.
  • The same plan–replicate–verify loop the authors sketch for extension and rediscovery tasks would be a natural next stress test of whether the pipeline is truly domain-agnostic.
  • Binary benchmark scorers can hide near-complete replications; tools that report partial matches and fix severity may change how the community reads a ‘failed’ replication score.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces VERITAS, a domain-agnostic scientific replication framework built around CLI coding agents (Claude Code, with support for Codex and Gemini). Given a paper, a repository, or both, the system extracts structured claims (ANALYZE), optionally generates code (CODEGEN), plans and executes the methodology while actively fixing failures under a bounded manager loop (PLAN/REPLICATE), rates applied fixes (ASSESS FIXES), and judges each claim against run evidence (VERIFY), returning an importance-weighted Replication Score, a severity-rated fix log, and a patched codebase. The authors evaluate the replication core on CORE-Bench Hard (45 capsules; CS, social science, medicine) and ReplicationBench (20 astrophysics papers / 111 tasks), comparing against two Claude Code baselines (ZEROSHOT, RETRY) on the same model (Claude Opus 4.8) and host environment. After a uniform cheating correction, VERITAS leads reported native and trajectory metrics; under a zero-variance re-scoring rule it reaches 97.8% per-capsule pass rate on CORE-Bench Hard (88.9% under the official scorer).

Significance. If the empirical picture holds under more robust evaluation, this is a useful systems contribution: prior replication agents are largely benchmark-tied or domain-specific, whereas VERITAS targets flexible inputs (paper+code, paper-only, repo-only) and end-to-end claim verification. Strengths include matched model/environment controls, system-blind adapted cheating and faithfulness metrics from prior work, transparent cheating correction, per-cell tables, and substantive case studies and error analysis (e.g., instructed external loads, paper-only data upper bounds, multi-value all-or-nothing scoring). The trajectory-level separation—especially faithfulness and lower cheating—is more informative than the thin native-score margins and is a genuine process-quality contribution for agentic replication research.

major comments (3)
  1. Abstract, Figure 1, and Table 1: the headline 97.8% CORE-Bench Hard pass rate is produced by a zero-variance re-scoring rule (§D) that flips four VERITAS failures and zero baseline failures. Under the official scorer VERITAS is 88.9% vs RETRY 84.4% and ZEROSHOT 80.0%—a lead of roughly two capsules on n=45. Leading the abstract and Figure 1 with the re-scored number, and the unqualified claim that VERITAS “achieves state-of-the-art performance and leads on every metric,” overstates a fragile single-run margin. Primary claims should use the official scorer; the σ-rule may appear as a secondary, uniformly applied analysis with explicit cross-system comparability caveats already noted in §D.
  2. §3.1 “Scope and scoring”: benchmark task questions are supplied as claims, bypassing ANALYZE and the importance-weighted Replication Score (Eq. 1); the authors state that neither the Replication Score nor the severity-rated fix log is used to compute any reported number. Yet the abstract and contribution bullets present claim extraction, the Replication Score, and the fix log as core delivered outputs of a general-purpose tool. This is a load-bearing gap between product framing and what Tables 1–2 validate (PLAN/REPLICATE/VERIFY). Either evaluate ANALYZE quality and score calibration on a held-out claim set, or reframe the paper’s claims around the replication core that was actually measured.
  3. §3 Experiments and Limitations (“Run-to-run variance”): all native scores are single-run point estimates; Claude Code exposes no pinable temperature, and both benchmarks include stochastic computations (MCMC, unseeded training). On ReplicationBench paper-only the match-rate lead is 0.333 vs 0.315 (~2 of 111 tasks); full-mode gaps are similarly small before trajectory metrics. Given that native “leads every metric” is central to the abstract, the manuscript needs multi-run averages, uncertainty (e.g., bootstrap over capsules/tasks), or explicitly softer language that privileges faithfulness/cheating—where separation is large and better supported—over native SOTA wording.
minor comments (6)
  1. Figure 1 caption and §3.2: make the dual reporting (official 88.9% vs σ-corrected 97.8%) visually primary rather than star-noted, so readers cannot miss which number is comparable to the HAL leaderboard.
  2. Eq. (1) and §2.2: importance weights (headline=3, supporting=2) and verdict values (match=1, partial=0.5) are free parameters; a short sensitivity check or justification would help even though the score is not used in the tables.
  3. §B.1 ReplicationBench shape-coercion patch: state clearly in the main text (not only the appendix) that a deterministic scorer patch was applied, and confirm it was applied identically to all systems.
  4. §F.3 / Limitations: the hardcoding blind spot of the access-only cheating metric is well illustrated by fable_mps; consider promoting a one-sentence main-text caveat next to Table 2’s cheating column so readers do not over-read the count.
  5. Related Work: a compact comparison table (inputs, domain, execution, claim grading, standalone vs benchmark-tied) would make the “no general-purpose tool” positioning easier to audit against Kohler et al., AutoReproduce, PaperRepro, and PaperBench agents.
  6. Minor consistency: “VERITAS” sometimes loses spacing in the PDF text (“VERITASextracts”, “VERITASon”); clean typesetting throughout.

Circularity Check

0 steps flagged

No circularity: empirical systems evaluation against external benchmarks with independent scorers; no derivation reduces to its inputs by construction.

full rationale

VERITAS is an empirical agent-systems paper, not a first-principles derivation. Its load-bearing claims are native pass/match rates on CORE-Bench and ReplicationBench, scored by each benchmark’s own grader after a uniform cheating correction adapted from Kohler et al. (external authors). The Replication Score (Eq. 1) is a transparent importance-weighted average of per-claim verdicts and is explicitly not used to compute any reported number (§3.1); ANALYZE and that score are bypassed so all systems are graded on the same supplied claim set. Trajectory metrics (faithfulness from Bai et al., cheating from Kohler et al.) are applied system-blind and uniformly; even the Bai et al. checklist, which shares a co-author, is an evaluation instrument rather than a premise that forces the result. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no self-definitional loop. Methodological concerns (single-run variance, the σ-re-scoring rule that re-credits only VERITAS capsules, thin native-score margins) affect robustness of the SOTA wording, not circularity of a derivation chain. Score 0 is the correct honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

As a systems paper, load-bearing structure is engineering and evaluation design rather than physical postulates. The central empirical claim rests on fixed scoring weights and thresholds, access-based cheating rules, and the assumption that benchmark task questions stand in for paper claims. No new physical entities are introduced.

free parameters (4)
  • claim importance weights
    Headline claims weight 3 and supporting claims weight 2 in the Replication Score (Eq. 1). These are design choices that shape the user-facing score; not fitted to the benchmark tables but free in the product definition.
  • verdict value mapping
    Match=1.0, partial=0.5, no match/not attempted=0.0. Arbitrary discrete mapping that fully determines the scalar score once verdicts exist.
  • relative-error / sigma thresholds for scalar grading
    Deterministic grader uses relative-error thresholds (and sigma rules when uncertainty is available); the 1% relative-error rule in the σ-corrected CORE-Bench scorer is an explicit tolerance choice.
  • manager-loop iteration cap and early-stop rules
    Bounded manager loop accepts or revises REPLICATE/PLAN based on deterministic execution checks; cap and non-improvement stop are free control parameters of the pipeline.
axioms (4)
  • domain assumption CLI coding agents (Claude Code with Claude Opus 4.8 in experiments) can install dependencies, edit code, and iterate toward runnable scientific pipelines when scaffolded.
    Foundational tooling assumption for the entire framework; supported by cited coding-benchmark SOTA but not proved for arbitrary science domains.
  • ad hoc to paper Benchmark task questions mapped one-to-one as headline claims are a fair proxy for evaluating the replication core against published agents.
    Stated in §3.1 Scope and scoring; ANALYZE and Replication Score are deliberately not scored, so generality of the full product is assumed rather than measured.
  • domain assumption Access-based cheating detection (Kohler et al.) plus execution-grounded faithfulness checklists (Bai et al.) adequately separate legitimate replication from shortcuts for reporting purposes.
    Authors adapt these metrics and apply a uniform cheating correction; they also note hardcoding blind spots (§ Limitations, §C).
  • standard math Standard container/host isolation and read-only mounts of paper/code/data define the legitimate information boundary for replication.
    Operational security model for the evaluation environment (Appendix A); conventional rather than paper-specific physics.
invented entities (2)
  • VERITAS pipeline (ANALYZE–CODEGEN–PLAN–REPLICATE–manager loop–ASSESS FIXES–VERIFY) independent evidence
    purpose: Organize domain-agnostic automated replication and claim-level scoring around CLI agents.
    The system is the paper’s primary artifact; independent evidence is the benchmark evaluation, not an external physical prediction.
  • importance-weighted Replication Score no independent evidence
    purpose: Aggregate per-claim verdicts into a single paper-level summary for users.
    Defined in Eq. (1); not independently validated in the experiments, which use native benchmark scorers instead.

pith-pipeline@v1.1.0-grok45 · 25918 in / 3664 out tokens · 35242 ms · 2026-07-12T06:00:42.032973+00:00 · methodology

0 comments
read the original abstract

AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.

Figures

Figures reproduced from arXiv: 2607.02931 by Chenhao Tan, Filbert Aurelian Tjiaranata, Haokun Liu.

Figure 1
Figure 1. Figure 1: Native benchmark scores after the cheating correction (§ [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The VERITAS pipeline. ANALYZE extracts structured claims, PLAN drafts a replication procedure, REPLICATE runs it while resolving issues, ASSESS FIXES rates each fix, and VERIFY judges each claim with a comparator step that extracts the replicated value and a deterministic grader that assigns the status. Verdicts aggregate into an importance-weighted Replication Score. CODEGEN runs only in paper-only mode, … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 1 canonical work pages

  1. [1]

    Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025.arXiv preprint arXiv:2602.05930, 2026

    Samar Ansari. Compound deception in elite peer review: A failure mode taxonomy of 100 fabricated citations at NeurIPS 2025.arXiv preprint arXiv:2602.05930, 2026

    arXiv 2025

  2. [2]

    Claude Code.https://github.com/anthropics/claude-code, 2025

    Anthropic. Claude Code.https://github.com/anthropics/claude-code, 2025

    2025

  3. [3]

    The story is not the science: Execution-grounded evaluation of mechanistic interpretability research, 2026

    Xiaoyan Bai, Alexander Baumgartner, Haojia Sun, Ari Holtzman, and Chenhao Tan. The story is not the science: Execution-grounded evaluation of mechanistic interpretability research, 2026. URLhttps://arxiv.org/abs/2602.18458

    arXiv 2026

  4. [4]

    PHANGS-ML: The universal relation between PAH band and optical line ratios across nearby star-forming galaxies.arXiv preprint arXiv:2410.02864, 2024

    Dalya Baron et al. PHANGS-ML: The universal relation between PAH band and optical line ratios across nearby star-forming galaxies.arXiv preprint arXiv:2410.02864, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Mass reproducibility and replicability: A new hope

    Abel Brodeur, Derek Mikola, Nikolai Cook, et al. Mass reproducibility and replicability: A new hope. IZA Discussion Paper 16912, Institute of Labor Economics (IZA), 2024

    2024

  6. [6]

    Computational reproducibility via containers in psychology.Meta-Psychology, 3, 2019

    April Clyburne-Sherin, Xu Fei, and Seth Ariel Green. Computational reproducibility via containers in psychology.Meta-Psychology, 3, 2019. doi: 10.15626/MP.2018.892

    work page doi:10.15626/mp.2018.892 2019

  7. [7]

    RExBench: Can coding agents autonomously implement AI research extensions? InACL, 2026

    Nicholas Edwards, Yukyung Lee, Yujun Audrey Mao, Yulu Qin, Sebastian Schuster, and Najoung Kim. RExBench: Can coding agents autonomously implement AI research extensions? InACL, 2026

    2026

  8. [8]

    Gemini CLI.https://github.com/google-gemini/gemini-cli, 2025

    Google. Gemini CLI.https://github.com/google-gemini/gemini-cli, 2025

    2025

  9. [9]

    REPRO- Bench: Can agentic AI systems assess the reproducibility of social science research? InFindings of ACL, 2025

    Chuxuan Hu, Liyun Zhang, Yeji Lim, Aum Wadhwani, Austin Peters, and Daniel Kang. REPRO- Bench: Can agentic AI systems assess the reproducibility of social science research? InFindings of ACL, 2025

    2025

  10. [10]

    A retrospective on the ICLR 2026 review process

    ICLR 2026 Program Chairs. A retrospective on the ICLR 2026 review process. https: //blog.iclr.cc/2026/03/31/a-retrospective-on-the-iclr-2026-review-process/ ,

    2026

  11. [11]

    Accessed: 24 May 2026

    2026

  12. [12]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InICLR, 2024

    2024

  13. [13]

    Holistic agent leaderboard: The missing infrastructure for AI agent evaluation

    Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, et al. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. InICLR, 2026

    2026

  14. [14]

    Read the paper, write the code: Agentic reproduction of social-science results.arXiv preprint arXiv:2604.21965, 2026

    Benjamin Kohler, David Zollikofer, Johanna Einsiedler, Alexander Hoyle, and Elliott Ash. Read the paper, write the code: Agentic reproduction of social-science results.arXiv preprint arXiv:2604.21965, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    The impact of satellite trails on Hubble Space Telescope observations.Nature Astronomy, 2023

    Sandor Kruk et al. The impact of satellite trails on Hubble Space Telescope observations.Nature Astronomy, 2023. doi: 10.1038/s41550-023-01903-3

    doi:10.1038/s41550-023-01903-3 2023

  16. [16]

    Bourne, Leah Bigwood, and Debora Sijacki

    Sergio Martin-Alvarez, Vid Iršiˇc, Sophie Koudmani, Martin A. Bourne, Leah Bigwood, and Debora Sijacki. Stirring the cosmic pot: how black hole feedback shapes the matter power spectrum in the FABLE simulations.arXiv preprint arXiv:2407.18349, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Docker: Lightweight Linux containers for consistent development and deployment

    Dirk Merkel. Docker: Lightweight Linux containers for consistent development and deployment. Linux Journal, 2014(239), 2014. 10

    2014

  18. [18]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal- bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  19. [19]

    Fact sheet: 34th annual conference on neural information processing systems

    NeurIPS. Fact sheet: 34th annual conference on neural information processing systems. https://neurips.cc/media/Press/NeurIPS_2020-Fact_Sheet.pdf, 2020. Accessed: 24 May 2026

    2020

  20. [20]

    NeurIPS 2025 fact sheet

    NeurIPS. NeurIPS 2025 fact sheet. https://media.neurips.cc/Conferences/NeurIPS20 25/press/NeurIPS2025-Fact_Sheet.pdf, 2025. Accessed: 24 May 2026

    2025

  21. [21]

    Obadage, Zack Ranjan, Sai Koneru, Anna Szabelska, Adam Gill, Timothy M

    Bang Nguyen, Dominik Soós, Qian Ma, Rochana R. Obadage, Zack Ranjan, Sai Koneru, Anna Szabelska, Adam Gill, Timothy M. Errington, Shakhlo Nematova, Sarah Rajtmajer, Jian Wu, and Meng Jiang. ReplicatorBench: Benchmarking LLM agents for replicability in social and behavioral sciences.arXiv preprint arXiv:2602.11354, 2026

    Pith/arXiv arXiv 2026

  22. [22]

    Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    Open Science Collaboration. Estimating the reproducibility of psychological science.Science, 349(6251):aac4716, 2015

    2015

  23. [23]

    Codex CLI.https://github.com/openai/codex, 2025

    OpenAI. Codex CLI.https://github.com/openai/codex, 2025

    2025

  24. [24]

    Mariia Rizhko and Joshua S. Bloom. AstroM 3: A self-supervised multimodal model for astronomy.arXiv preprint arXiv:2411.08842, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Paper2Code: Automating code generation from scientific papers in machine learning

    Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2Code: Automating code generation from scientific papers in machine learning. InICLR, 2026

    2026

  26. [26]

    Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan

    Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-Bench: Fostering the credibility of published research through a computational repro- ducibility agent benchmark.Transactions on Machine Learning Research, 2025

    2025

  27. [27]

    PaperBench: Evaluating AI’s ability to replicate AI research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. arXiv preprint arXiv:2504.01848, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Walters and Esther Isabelle Wilder

    William H. Walters and Esther Isabelle Wilder. Fabrication and errors in the bibliographic citations generated by ChatGPT.Scientific Reports, 13(1):14045, 2023. doi: 10.1038/s41598-0 23-41032-5

    doi:10.1038/s41598-0 2023

  29. [29]

    Xing, and Zhiting Hu

    Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Eric P. Xing, and Zhiting Hu. FIRE-Bench: Evaluating agents on the rediscovery of scientific insights.arXiv preprint arXiv:2602.02905, 2026

    arXiv 2026

  30. [30]

    Q. Wei, S. Holt, J. Yang, M. Wulfmeier, and M. van der Schaar. The AI imperative: Scaling high-quality peer review in machine learning.arXiv preprint arXiv:2506.08134, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V . Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, Diyi Yang, Risa Wechsler, and Ioana Ciuc ˘a. ReplicationBench: Can AI agents replicate astrophysics research papers? arXiv preprint arXiv:2510.24591, 2025

    arXiv 2025

  32. [32]

    PaperRepro: Auto- mated computational reproducibility assessment for social science papers.arXiv preprint arXiv:2603.00058, 2026

    Linhao Zhang, Tong Xia, Jinghua Piao, Lizhen Cui, and Yong Li. PaperRepro: Auto- mated computational reproducibility assessment for social science papers.arXiv preprint arXiv:2603.00058, 2026

    arXiv 2026

  33. [33]

    increase

    Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. AutoReproduce: Automatic AI experiment reproduction with paper lineage. InACL, 2026. 11 A Environment and Runtime We describe the runtime environment shared by all systems within each benchmark. The environment is identical across VERITA...

    2026