Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fisher-R1 shows that small open-weight agents, trained with verified p-value rewards, can outperform frontier models on statistical hypothesis testing.

desk verdict P-Bench and Fisher-R1 fill a real gap in evaluating statistical reasoning of LLM agents, but the Strict metric's one-correct-p-value assumption needs validation before the headline numbers can be fully trusted. read the letter →

arxiv 2608.07437 v1 pith:6PMTCRO5 submitted 2026-08-07 cs.AI

Jiacheng Miao , Jin Mu , Guanhua Chen , James Zou This is my paper · ORCID
classification cs.AI
keywords hypothesistestingLLMagentp-valuereinforcementlearningstatisticalreasoningbenchmarkP-BenchFisher-R1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that current LLM agents, even frontier ones, cannot be trusted to carry out hypothesis testing end-to-end: they often pick a plausible statistical method, run it correctly, and report a confident but wrong conclusion. To make this failure measurable it builds P-Bench, 425 open-ended tasks with expert-audited answer keys from real economic, biological, and medical analyses. It then trains Fisher-R1, an open-weight agent, on synthetic tasks whose answer keys are produced by executing the canonical method, using reinforcement learning with a reward that scores closeness of the reported p-value to the key in z-score space. On P-Bench, Fisher-R1-14B beats strong proprietary and open-source baselines, including GPT-5.4 and DeepSeek-V4-Pro, with a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro. The broader claim is that reliable statistical inference is a skill distinct from code execution, and that outcome-verified reinforcement learning can instill it.

What carries the argument

The load-bearing machinery is the z-score re-labeling of p-values, $z(p)=\Phi^{-1}(1-p/2)$, used twice: as the training reward (exponential closeness between reported and key z-scores) and as the evaluation metric (Strict requires $|\Delta z|<0.5$). Because raw p-values are compressed near zero, this monotonic transformation makes small differences at high significance count as large evidence gaps; it lets a single reward signal compare p-values regardless of which test produced them. The second mechanism is the synthetic task generator: the answer key is the output of running the canonical statistical method on simulated data, giving an executable, programmatically checkable reward signal that keeps P-Bench fully out of the training set.

What would settle it

Take a P-Bench biology task with outliers, have a panel of statisticians pre-register an alternative valid method (e.g., a rank-based test where the key uses linear regression), and check whether their p-value, differing by more than |Δz|=0.5, is marked wrong; if correct inferences are marked wrong, the Strict metric is not method-agnostic.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that instruction-following and coding ability are not enough for statistical hypothesis testing: every evaluated model, including GPT-5.4, frequently reports a correct reject/fail-to-reject decision while producing a p-value far from the reference analysis, as shown by Raw scores far exceeding Strict scores, especially on hard tasks. The paper demonstrates that this gap can be closed substantially by training with a verified statistical reward: Fisher-R1-14B reaches 33.0% Strict pass@1 on the hard split versus 30.5% for GPT-5.4 and 26.3% for DeepSeek-V4-Pro, and the trained models also show much lower run-to-run variance. This supports the claim that targeted reinforcement learning on hypothesis-testing tasks with outcome-grounded rewards, rather than mere model scale, improves inferential reliability.

Load-bearing premise

The benchmark's answer keys assume there is one correct statistical analysis per task, and the Strict score requires the agent's p-value to land within |Δz|<0.5 of that key, so any task with several defensible methods can mark a correct inference as wrong.

Editorial extensions

If this is right

  • Conclusion-only evaluation overstates agent reliability: every baseline scores far lower on Strict than on Raw, so adding a p-value-closeness criterion to data-analysis benchmarks is a cheap and informative check.
  • Training a small open-weight model with an outcome-verified p-value reward can outperform much larger models, since Fisher-R1-7B/14B beat GPT-OSS-120B, Qwen3-32B, and DeepSeek-V4-Pro on every Strict metric.
  • Using synthetic tasks with executable answer keys gives a scalable, programmatically checkable reward that keeps the evaluation benchmark out of the training set, as the embedding-similarity analysis shows.
  • The same recipe can extend to multi-test pipelines and multiple-comparison correction, the next direction the authors identify in the discussion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Outside the paper: if several statistical analyses are defensible for one task, the single-key Strict metric will penalize valid alternative methods; an ensemble-keys variant of P-Bench would test how much of Fisher-R1's edge is method-agnostic.
  • Outside the paper: the z-space reward could be lifted to other inferential quantities such as effect sizes, confidence intervals, or Bayes factors, letting the same RL recipe train agents on calibration rather than a single p-value.
  • Outside the paper: ablating the adversarial perturbation variants from the training corpus would isolate how much of the hard-split gain comes from explicit handling of outliers and missingness rather than from general method selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces P-Bench, a benchmark of 425 open-ended hypothesis-testing tasks drawn from economics, biology, and medicine, in which an agent must choose a statistical method, execute the analysis in R, and report a p-value and a reject/fail-to-reject conclusion against a hidden answer key derived from a logged canonical reference analysis. The paper also introduces Fisher-R1, an open-weight agent initialized from Qwen2.5-Coder and trained with supervised fine-tuning on teacher trajectories followed by DAPO reinforcement learning on 8,642 synthetic tasks; the reward combines p-value closeness on a two-sided z-scale with conclusion agreement. The paper reports that Fisher-R1-14B improves substantially over its backbone and matches or exceeds GPT-5.4 and DeepSeek-V4-Pro on P-Bench, and it provides ablations, a cosine-similarity analysis, and a relaxed-threshold robustness check.

Significance. If the answer keys are accepted as unique ground truth, P-Bench fills a real gap in evaluating inferential method selection and p-value accuracy rather than code executability or factoid answers, and the reported training gains for Fisher-R1 are substantial and practically useful. The strengths include the execution-log grounding of answer keys, the held-out evaluation set, the three-stage reproducibility pipeline, the ablation showing that SFT and DAPO are both needed, the similarity analysis that addresses memorization, and the relaxed-threshold table. The main risk is that the evaluation criterion assumes a single canonical analysis is the only correct answer, while the paper's own training design acknowledges that multiple statistical procedures may be defensible; this makes the benchmark's validity, not just its reproducibility, load-bearing for the headline conclusions.

major comments (4)
  1. [§3.2, §5.1, §4.3] The validity of P-Bench as a measure of reliable hypothesis testing is not established against the possibility of multiple defensible analyses. Section 3.2 calls answer keys 'trustworthy by construction,' but Section 5.1 defines Strict accuracy as |z(p_hat) - z(p*)| < 0.5, which marks a reported p-value as wrong whenever it differs from the single canonical value by more than that threshold. Section 4.3, however, explicitly states that 'multiple statistical procedures may be defensible for the same hypothesis-testing task.' For observational tasks in P-Bench (logistic, Cox, mixed-effects, IV/2SLS), legitimate choices about covariate adjustment, robust versus model-based standard errors, clustering level, and missing-data handling can produce p-values differing by more than 0.5 in z-space; for example, p = 0.05 and p = 0.01 differ by about 0.62 in z. The same z-space closeness is used as the RL reward in Section 4.3 and as the evaluation metric in Section 5.1, so 'reliability' is operationalized as agreement with one canonical implementation. The paper should provide an audit of how often P-Bench tasks admit alternative defensible analyses with p-values beyond the Strict threshold, plus a sensitivity analysis that accepts a set of reasonable alternative answers.
  2. [§4.3 and §5.1] The reward and the evaluation metric are the same functional object: both score the reported p-value by closeness to a single p* in two-sided z-space. Because the synthetic training answer keys are produced by running 'canonical' method code on simulated data (Appendix C.2.1) and P-Bench answer keys are produced by the same style of canonical reference analysis (Section 3.2), the training signal optimizes for reproducing a particular statistical convention. This is not direct overfitting to P-Bench, since P-Bench is held out, but it means the reported gains are evidence of alignment with the reference canon rather than, by itself, evidence of statistical validity independent of that canon. To disentangle these, the paper should add an evaluation variant in which any p-value inside an interval derived from a reasonable range of defensible specifications counts as correct, and an ablation in which the RL reward is replaced by a method-validity score that does not depend on a single p*.
  3. [§3.2 and Appendix B.1] The audit and release details needed to verify P-Bench are missing. The text says every answer key is expert-audited and tasks failing review are repaired or removed, but it does not state the number of auditors, their qualifications, the audit protocol, or inter-auditor agreement; it also does not provide the benchmark itself, the answer keys, the datasets, or the reproducibility pipeline code. The automated pipeline described in Appendix B.1 is built on Claude Code, but no accuracy or failure-rate statistics are reported for the claim-parsing and reproduction steps. Without public release of the benchmark and an audit appendix, the 'expert-verified' status and the numerical scores cannot be independently checked.
  4. [Table 1 and §5.2] The specific claim that Fisher-R1-14B outperforms GPT-5.4 on three of four Strict metrics is not supported by the reported statistics. Table 1 reports pass@1 as mean ± stdev over three independent runs; on P-Easy Strict pass@1 the values are 64.2 ± 0.6 for Fisher-R1-14B and 64.7 ± 3.3 for GPT-5.4, and on P-Hard Strict pass@1 they are 33.0 ± 1.7 and 30.5 ± 1.6. These differences are within the reported run-to-run variability, no paired significance test is given, and the relaxed-threshold results in Table 5 even show GPT-5.4 ahead on P-Easy Strict (72.6 ± 2.5 vs 70.6 ± 1.4). Please report task-level bootstrap confidence intervals and paired tests across models, or soften the superiority claim accordingly.
minor comments (5)
  1. [§2.2] The sentence 'P-Bench targets a regime that neither captures the full open-ended hypothesis-testing loop...' appears to omit an intended contrast with existing benchmarks and should be rewritten for clarity.
  2. [Table 1 and Figure 1] Model names are sometimes concatenated without spaces (e.g., 'GPT-5.492.9' and 'DeepSeekV4-Pro'); fix the formatting throughout.
  3. [§5.4] The similarity analysis would be more informative if it reported numeric summaries of the separation between the eval-to-train and train-to-train distributions (e.g., mean and percentile values) rather than only a density plot.
  4. [Appendix C.3.2] For the perturbed outlier tasks, the answer key is computed using 'robust standard errors or trimming' as the handling protocol; this choice should be documented per task, since trimming versus robust standard errors can change both the p-value and the conclusion.
  5. [Appendix C.2.1] The cross-reference 'summarised in S4.3' should be '§4.3'; there is no 'S4.3' section in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: P-Bench is held out from training, the reward and evaluation metric share a statistical canon but that is a domain-alignment concern, not a circular derivation, and no load-bearing self-citations appear.

full rationale

The paper's central empirical claims are that Fisher-R1 improves over its backbone and beats baselines on P-Bench, and that RL with a p-value-closeness reward improves reliability. There is no circular derivation chain here. The synthetic training tasks (Section 4.1) use answer keys computed by running the canonical statistical method on simulated data, and the RL reward (Section 4.3) scores closeness to that key via r_p = exp(-|min{z(phat),5} - min{z(p*),5}|/sigma), with sigma=1. The held-out P-Bench evaluation (Section 5.1) uses a closely related Strict criterion |z(phat)-z(p*)| < 0.5. This shared p-value-closeness signal is not circular because the evaluation set is explicitly kept out of training ('By construction, it also keeps P-Bench fully out of the training set'), and the paper provides a similarity analysis (Section 5.4) showing the evaluation prompts are not near-duplicates of training prompts. The paper does not fit any parameter to P-Bench and then report P-Bench as a prediction; it reports out-of-distribution generalization of a model trained on synthetic tasks. The main fragility is a benchmark-validity concern rather than a circularity concern: Section 3.2 calls the answer keys 'trustworthy by construction' because they come from logged canonical reference analyses, while Section 4.3 concedes that 'multiple statistical procedures may be defensible for the same hypothesis-testing task.' This means Strict accuracy operationalizes correctness as agreement with one canonical analysis, and defensible alternative methods could be penalized. But the paper is explicit that it evaluates against a canonical reference analysis, and the Discussion openly lists limitations ('P-Bench currently evaluates a single hypothesis test per task' and 'recognize when no single hypothesis test is adequate'). No load-bearing self-citations appear; the algorithm citation DAPO, the teacher model Claude, and the source datasets are all external. There is no uniqueness theorem imported from the authors, no ansatz smuggled in via citation, and no renamed known result. A reviewer concerned about benchmark validity should attack P-Bench's construct validity, not the paper's derivation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the correctness and uniqueness of the canonical answer keys, the representativeness of the synthetic task generator, and the validity of the z-space scoring. The free parameters are training and evaluation hyperparameters chosen by hand; none are fitted to P-Bench data. No new physical or conceptual entities are postulated.

free parameters (4)
  • Reward weights w_p, w_c = 0.9, 0.1
    Hand-chosen in Section 4.3; p-value closeness is weighted more heavily than conclusion agreement because the conclusion is derived from the reported p-value.
  • z-space reward scale sigma = 1
    Set in Section 4.3; controls the sharpness of the exponential p-value closeness reward.
  • Strict evaluation tolerance |Delta z| = 0.5
    Chosen in Section 5.1 as the cutoff for a numerically close p-value; a relaxed threshold of 1.0 is also tested (Table 5). The metric is arbitrary but the qualitative conclusion is robust.
  • Asymmetric clipping bounds (eps_l, eps_h) = 0.20, 0.28
    DAPO hyperparameters from Section 4.3 and C.4; standard choices from the DAPO paper, not fitted to P-Bench.
assumptions (4)
  • domain assumption The canonical reference analysis in each source paper, as re-run on the provided data, is the correct ground truth for the task.
    P-Bench answer keys are built by reproducing published analyses (Section 3.2); if the original analysis is flawed or the reproduction is wrong, the key is wrong. No independent statistical audit is provided beyond the authors' expert review claim.
  • domain assumption The synthetic data-generating process templates, drafted by Claude Sonnet 4.6 and checked by the authors, are representative of real hypothesis-testing tasks.
    Section C.2.1: the entire RL training corpus relies on these templates covering 27 methods; if they omit key aspects of real data, the trained agent may not transfer to P-Bench.
  • standard math The z-score transform z(p) = Phi^{-1}(1 - p/2) is a distribution-free re-labeling suitable for scoring p-values from different tests.
    Section 4.3: the transform is monotone and does not assume Gaussian data; scoring in z-space is standard in meta-analysis but still assumes that proximity in z-space matches statistical evidence similarity.
  • domain assumption The semantic-embedding similarity analysis demonstrates that P-Bench gains are not from memorization of training prompts.
    Section 5.4: eval-to-train similarity lower than train-to-train is presented as evidence of generalization, but cosine similarity in embedding space does not rule out conceptual or methodological overlap between synthetic and real tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing." pith.science (2026). https://pith.science/paper/6PMTCRO5

@misc{pith2026260807437,
  author       = {Pith},
  title        = {Pith review of: Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6PMTCRO5}},
  note         = {Machine review of arXiv:2608.07437}
}
read the original abstract

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Figures

Figures reproduced from arXiv: 2608.07437 by the authors.

Figure 1
Figure 1. Open-ended hypothesis testing with coding agents. Left: GPT-5.4 identifies outliers but still applies linear regression, leading to a false discovery. Fisher-R1 instead uses a rank-based test and correctly fails to reject the null. Right: P-Bench accuracy on easy and hard splits. inspecting datasets, generating code, running analyses, and drafting reports [2–4]. However, hy￾pothesis testing is not merely code execut… view at source ↗
Figure 2
Figure 2. A P-Bench task is composed of an analysis request, a CSV dataset, and a hidden answer [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Task Composition of P-Bench No single category exceeds 19%, so an agent cannot succeed by defaulting to a fixed recipe. Cross-domain coverage. Tasks span eco￾nomics, biology, and medicine, covering both randomized experiments and observational studies. Each setting has its own modeling as￾sumptions and inferential targets, so an agent must transfer statistical reasoning across disci￾plines. Realistic statistical tra… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Top: Expert trajectories are collected for SFT. The model generates reasoning traces containing thoughts, code, observations, and final answers; quality control filters for high-quality trajectories. Bottom: The policy is initialized from the SFT model and optimized wi…
Figure 5
Figure 5. Figure 5: Similarity Analysis As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Data curation for P-bench [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Full Welch’s t-test simulation template. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 26 canonical work pages

  1. [1]

    Wasserstein and Nicole A

    Ronald L. Wasserstein and Nicole A. Lazar. The asa statement on p-values: Context, process, and purpose.The American Statistician, 70(2):129–133, 2016. doi: 10.1080/00031305.2016. 1154108. URLhttps://doi.org/10.1080/00031305.2016.1154108

    arXiv 2016

  2. [2]

    Data-copilot: Bridging billions of data and humans with autonomous workflow.arXiv preprint arXiv:2306.07209, 2023

    Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Data-copilot: Bridging billions of data and humans with autonomous workflow.arXiv preprint arXiv:2306.07209, 2023

    arXiv 2023

  3. [3]

    Deepanalyze: Agentic large language models for autonomous data science.arXiv preprint arXiv:2510.16872, 2025

    Shaolei Zhang, Ju Fan, Meihao Fan, Guoliang Li, and Xiaoyong Du. Deepanalyze: Agentic large language models for autonomous data science.arXiv preprint arXiv:2510.16872, 2025

    arXiv 2025

  4. [4]

    Data interpreter: An llm agent for data science

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, et al. Data interpreter: An llm agent for data science. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19796–19821, 2025

    work page 2025

  5. [5]

    Infiagent-dabench: Evaluating agents on data analysis tasks,

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. Infiagent-dabench: Evaluating agents on data analysis tasks,

    work page

  6. [6]

    Dabstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719, 2025

    Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. Dabstep: Data agent benchmark for multi-step reasoning.arXiv preprint arXiv:2506.23719, 2025

    arXiv 2025

  7. [7]

    DA-code: Agent data science code gen- eration benchmark for large language models

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. DA-code: Agent data science code gen- eration benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Proce...

    doi:10.18653/v1/2024.emnlp-main.748 2024

  8. [8]

    Are large language models good statisticians?Advances in Neural Information Processing Systems, 37:62697–62731, 2024

    Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians?Advances in Neural Information Processing Systems, 37:62697–62731, 2024

    work page 2024

Show all 61 references
  1. [9]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  2. [10]

    Ds-1000: A natural and reliable benchmark for 12 data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen- tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for 12 data science code generation. InInternational Conference on Machine Learning, pages 18319– 1834...

    2023

  3. [11]

    Infiagent-dabench: Evaluating agents on data analysis tasks

    Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. Infiagent-dabench: Evaluating agents on data analysis tasks. arXiv preprint arXiv:2401.05507, 2024

    2024 arXiv

  4. [12]

    Datascibench: An llm agent benchmark for data science

    Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. Datascibench: An llm agent benchmark for data science. arXiv preprint arXiv:2502.13897, 2025

    2025 arXiv

  5. [13]

    Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095, 2024

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095, 2024

    arXiv

  6. [14]

    Tapilot-crossing: Benchmarking and evolving llms towards interactive data analysis agents.arXiv preprint arXiv:2403.05307, 2024

    Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Yurong Wu, Chenhao Ma, Jian-Guang Lou, and Reynold Cheng. Tapilot-crossing: Benchmarking and evolving llms towards interactive data analysis agents.arXiv preprint arXiv:2403.05307, 2024

    2024 arXiv

  7. [15]

    Ida-bench: Evaluating llms on interactive guided data analysis.arXiv preprint arXiv:2505.18223, 2025

    Hanyu Li, Haoyu Liu, Tingyu Zhu, Tianyu Guo, Zeyu Zheng, Xiaotie Deng, and Michael I Jordan. Ida-bench: Evaluating llms on interactive guided data analysis.arXiv preprint arXiv:2505.18223, 2025

    2025 arXiv

  8. [16]

    Fact or fiction: Verifying scientific claims

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, 2020

    2020

  9. [17]

    Sciclaimhunt: A large dataset for evidence-based scientific claim verification

    Sujit Kumar, Anshul Sharma, Siddharth Hemant Khincha, Gargi Shroff, Sanasam Ranbir Singh, and Rahul Mishra. Sciclaimhunt: A large dataset for evidence-based scientific claim verification. In2025 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE, 2025

  10. [18]

    Musciclaims: Multimodal scientific claim verifi- cation

    Yash Kumar Lal, Manikanta Bandham, Mohammad Saqib Hasan, Apoorva Kashi, Mahnaz Koupaee, and Niranjan Balasubramanian. Musciclaims: Multimodal scientific claim verifi- cation. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Co...

    2025

  11. [19]

    Investi- gating the reproducibility of the social and behavioural sciences.Nature, 652(8108):126–134, 2026

    Olivia Miske, Anna Lou Abatayo, Mason Daley, Mirka Dirzo, Nicholas Fox, Noah Haber, Krystal M Hahn, Melissa Kline Struhl, Brinna Mawhinney, Priya Silverstein, et al. Investi- gating the reproducibility of the social and behavioural sciences.Nature, 652(8108):126–134, 2026

    2026

  12. [20]

    Towards end-to-end automation of ai research.Nature, 651(8107):914–919, 2026

    Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of ai research.Nature, 651(8107):914–919, 2026

    2026

  13. [21]

    Ai-researcher: Autonomous scientific innovation.arXiv preprint arXiv:2505.18705, 2025

    Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. Ai-researcher: Autonomous scientific innovation.arXiv preprint arXiv:2505.18705, 2025. 13

    2025 arXiv

  14. [22]

    The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    2025 arXiv

  15. [23]

    Development economics field experiments (dfeep)

    Abdul Latif Jameel Poverty Action Lab (J-PAL). Development economics field experiments (dfeep). Harvard Dataverse, 2026. URLhttps://dataverse.harvard.edu/dataverse/ DFEEP. Accessed: 2026-04-20

    2026

  16. [24]

    The cbio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data.Cancer discovery, 2(5):401–404, 2012

    Ethan Cerami, Jianjiong Gao, Ugur Dogrusoz, Benjamin E Gross, Selcuk Onur Sumer, Bü- lent Arman Aksoy, Anders Jacobsen, Caitlin J Byrne, Michael L Heuer, Erik Larsson, et al. The cbio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data....

    2012

  17. [25]

    Integrative analysis of complex cancer genomics and clinical profiles using the cbioportal.Science signal- ing, 6(269):pl1–pl1, 2013

    Jianjiong Gao, Bülent Arman Aksoy, Ugur Dogrusoz, Gideon Dresdner, Benjamin Gross, S Onur Sumer, Yichao Sun, Anders Jacobsen, Rileen Sinha, Erik Larsson, et al. Integrative analysis of complex cancer genomics and clinical profiles using the cbioportal.Science signal- ing, 6(26...

    2013

  18. [26]

    Biodsa-1k: Benchmarking data science agents for biomedical research, 2025

    Zifeng Wang, Benjamin Danek, and Jimeng Sun. Biodsa-1k: Benchmarking data science agents for biomedical research, 2025. URLhttps://arxiv.org/abs/2505.16100

    2025 arXiv

  19. [27]

    Frank E. Harrell. Vanderbilt biostatistics datasets (hbiostat.org).https://hbiostat.org/ data/, 2024. Accessed 2026-04-20

    2024

  20. [28]

    Introducing claude sonnet 4.6.https://www.anthropic.com/news/ claude-sonnet-4-6, February 2026

    Anthropic. Introducing claude sonnet 4.6.https://www.anthropic.com/news/ claude-sonnet-4-6, February 2026. Accessed: 2026-05-06

    2026

  21. [29]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. InForty-first International Conference on Machine Learning, 2024

    2024

  22. [30]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  23. [31]

    Qwen2.5-coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xu- ancheng Ren, Jingren Zh...

    2024 arXiv

  24. [32]

    Introducing gpt-5.4.https://openai.com/index/introducing-gpt-5-4/, March

    OpenAI. Introducing gpt-5.4.https://openai.com/index/introducing-gpt-5-4/, March

  25. [33]

    Deepseek-v4: Towards highly efficient million-token context intelligence

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro, April 2026. Model card and tech- nical report for DeepSeek-V4-Pro. Accessed: 2026-05-06

    2026

  26. [34]

    Introducing gpt-oss.https://openai.com/index/introducing-gpt-oss/, 2025

    OpenAI. Introducing gpt-oss.https://openai.com/index/introducing-gpt-oss/, 2025. Accessed: 2026-05-07

    2025

  27. [35]

    Qwen3-coder-30b-a3b-instruct.https://huggingface.co/Qwen/ Qwen3-Coder-30B-A3B-Instruct, 2025

    Qwen Team. Qwen3-coder-30b-a3b-instruct.https://huggingface.co/Qwen/ Qwen3-Coder-30B-A3B-Instruct, 2025. Model card. Accessed: 2026-05-06. 14

    2025

  28. [36]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  29. [37]

    Scaling generalist data- analytic agents, 2026

    Shuofei Qiao, Yanqiu Zhao, Zhisong Qiu, Xiaobin Wang, Jintian Zhang, Zhao Bin, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Scaling generalist data- analytic agents, 2026. URLhttps://arxiv.org/abs/2509.25084

    2026

  30. [38]

    Accessed: 2026-05-06

    OpenAI.text-embedding-3-smallmodel.https://developers.openai.com/api/docs/ models/text-embedding-3-small, 2026. Accessed: 2026-05-06

    2026

  31. [39]

    Self-refine: Iterative refinement with self-feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iter...

    2023 arXiv

  32. [40]

    Reflexion: Language agents with verbal reinforcement learning.Advances in neural informa- tion processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in neural informa- tion processing systems, 36:8634–8652, 2023

    2023

  33. [41]

    Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  34. [42]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated soft- ware engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  35. [43]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  36. [44]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  37. [45]

    Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  38. [46]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 15

    2024 arXiv

  39. [47]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  40. [48]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    2025 arXiv

  41. [49]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    2025 arXiv

  42. [50]

    Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    2025 arXiv

  43. [51]

    Agent- rlvr: Training software engineering agents via guidance and environment rewards.arXiv preprint arXiv:2506.11425, 2025

    Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent- rlvr: Training software engineering agents via guidance and environment rewards.arXiv preprint arXiv:2506.11425, 2025

    2025 arXiv

  44. [52]

    Effects of cognitive behavioral therapy and cash transfers on older persons living alone in india: a randomized trial.Annals of internal medicine, 176(5):632–641, 2023

    Madeline McKelway, Abhijit Banerjee, Erin Grela, Frank Schilbach, Miriam Sequeira, Garima Sharma, Girija Vaidyanathan, and Esther Duflo. Effects of cognitive behavioral therapy and cash transfers on older persons living alone in india: a randomized trial.Annals of internal med...

    2023

  45. [53]

    Genomic characterization of metastatic patterns from prospective clinical sequenc- ing of 25,000 patients.Cell, 185(3):563–575, 2022

    Bastien Nguyen, Christopher Fong, Anisha Luthra, Shaleigh A Smith, Renzo G DiNatale, Subhiksha Nandakumar, Henry Walch, Walid K Chatila, Ramyasree Madupuri, Ritika Kun- dra, et al. Genomic characterization of metastatic patterns from prospective clinical sequenc- ing of 25,000...

    2022

  46. [54]

    The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults.Annals of internal medicine, 122(3):191–203, 1995

    William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell S Phillips, Alfred F Connors, Neal V Dawson, William J Fulkerson, Robert M Califf, Norman Desbiens, et al. The support prognostic model: Objective estimates of survival for seriously ill hospitalized adults.An...

    1995

  47. [55]

    Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024. URLhttps://arxiv.org/abs/2403.13372

    2024 arXiv

  48. [56]

    claim”: “drug improves patient outcome

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297, 2025. 16 A Addition...

    2023

  49. [59]

    An intermediate analysis turn: <think> your reasoning about the data, assumptions, and the next analysis step </think> <code> one block of R code (use cat() or print() to display results) </code>

  50. [60]

    A final answer turn (no code): <think> brief summary of what you learned </think> <answer>

  51. [61]

    Analysis summary: key EDA findings, assumption check results, method choice and rationale

  52. [62]

    Results interpretation: effect estimate, direction, magnitude, p-value, confidence interval. FINAL ANSWER: - Treatment effect: [number] - P-value: [number] - 95% CI: [lower, upper] - Conclusion: [Reject H0 / Fail to reject H0] at alpha = 0.05 </answer> Rules: - After each <cod...

  53. [2026]

    Accessed: 2026-05-06

    2026

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.