Pith. sign in

REVIEW 4 major objections 5 minor 51 references

AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AbGen, the first benchmark for LLM ablation-study design, shows frontier models trail human experts and that automated judges are not yet reliable for this task.

desk verdict AbGen is a genuinely new benchmark with a careful construction pipeline, but its headline human-vs-model comparison rests on a statistically fragile 100-example gold standard. read the letter →

arxiv 2507.13300 v1 pith:IPT7SHSY submitted 2025-07-17 cs.CL cs.AI

Yilun Zhao , Weiyuan Chen , Zhijian Xu , Manasi Patwardhan , Yixin Liu , Chengye Wang , Lovekesh Vig , Arman Cohan This is my paper
classification cs.CLcs.AI
keywords ablationstudydesignLLMbenchmarkscientificexperimentalmeta-evaluationLLM-as-judgenaturallanguageprocessinghumanevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces AbGen, the first benchmark built to test whether large language models can design sound ablation studies for scientific research. Drawing on 1,500 expert-annotated examples from 807 NLP papers, it gives a model a research context and the name of a module, then asks it to propose an experiment that isolates that module's contribution. On a 100-example sample scored by experts, the best model, DeepSeek-R1-0528, averaged 4.11 on a 1-5 scale while human experts averaged 4.80, a gap the authors read as evidence that LLMs cannot yet plan controlled experiments on their own. The paper then builds AbGen-Eval, a meta-evaluation set of 1,800 human-scored outputs, and reports that current LLM-as-judge systems correlate only weakly with human ratings at the instance level, with Pearson correlations as low as 0.164. A careful reader takes away that AbGen offers a first standard yardstick for this scientific skill, and that automated evaluation of it is not yet trustworthy.

What carries the argument

The load-bearing machinery is the benchmark construction pipeline and the meta-evaluation protocol built on top of it. AbGen is built by filtering arXiv NLP papers, restructuring each paper into a three-part research context with all ablation content removed, annotating each reference ablation study as objective, experiment process, and result discussion, and validating the annotations with a separate annotator. Model outputs are scored by experts on three 1-5 Likert dimensions — importance, faithfulness, and soundness — with evaluators scoring the design without seeing the reference first and then adjusting their scores. AbGen-Eval then treats the 1,800 human-scored outputs as the gold standard and measures each automated evaluator by system-level Kendall correlation and instance-level Pearson correlation; the low instance-level correlations are the specific quantity that exposes current LLM-as-judge systems as unreliable.

What would settle it

Take a fresh sample of 200 examples from the held-out test set, have all four expert annotators score every model output, and recompute the model ranking and the instance-level Pearson correlation of GPT-4.1-mini; if the best-model lead over o4-mini becomes statistically insignificant, or if any LLM judge's instance-level correlation exceeds about 0.5, the paper's two central claims — that LLMs trail experts and that automated evaluation is unreliable — would need revision.

Watch

Extended reading notes

Core claim

AbGen establishes a measurable standard for one specific scientific skill: designing an ablation study. Given a research context divided into background, methodology, and main experiments, plus the name of a module, an LLM must produce a research objective and a replicable experimental procedure that would isolate that module's role. The paper's central finding is two-fold. First, frontier LLMs underperform human experts by a wide margin: the strongest system, DeepSeek-R1-0528, receives an average human score of 4.11 versus 4.80 for human experts, with the largest deficits in faithfulness and soundness rather than in picking an important question. Second, automated evaluation cannot yet stand in for human judgment: on AbGen-Eval, every LLM-based evaluator tested shows low instance-level Pearson correlation with human scores, so automatic ratings would misorder the systems on individual examples.

Load-bearing premise

The load-bearing premise is that expert human ratings collected on a 100-example sample from testmini, with partially overlapping annotators, define a stable gold standard that can rank 18 models and ground the meta-evaluation; if that sample is not representative, both the model gap and the evaluator-unreliability finding weaken.

Editorial extensions

If this is right

  • If AbGen is a fair yardstick, current frontier LLMs are not ready to replace a researcher in planning even a single controlled experiment: every evaluated model averages below human experts on all three criteria.
  • The benchmark's testmini/test split and publicly released model outputs let future systems be compared on the same 500 and 1,000 examples without rerunning expert evaluation.
  • An interactive use case is viable: expert feedback raised GPT-4o's faithfulness score from 2.1 to 4.2 and Llama-3.1-70B's from 1.8 to 3.9 in the user study, suggesting LLMs are useful as drafts that a researcher corrects.
  • Small user studies on biomedical and computer-network papers show similar performance to the NLP setting, indicating the benchmark pipeline generalizes even though current model capability does not.
  • Automated evaluation of ablation designs needs new methods before it can rank models on this task; the reported instance-level correlations rule out today's LLM-as-judge systems for that purpose.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the weak instance-level correlations suggest that holistic judges miss the hard part — whether a control condition would actually isolate the module — so future evaluators may need to check experimental logic step by step rather than assign an overall score.
  • Editorial inference: a natural harder test is to score designs by whether the proposed ablation would change the paper's known main result; this would replace subjective importance ratings with an objective, falsifiable criterion.
  • Editorial inference: stripping all ablation content from the research context may make AbGen harder than real-world design, where a researcher already knows which ablations were actually run; the benchmark measures generative design from a clean slate, not recognition of a missing experiment.
  • Editorial inference: with only 100 examples judged and annotators only partially overlapping, the ranking of closely spaced models (4.11 versus 4.00) is a point estimate; re-running all four annotators on the full testmini set could shift the ordering.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AbGen, a benchmark of 1,500 expert-annotated examples from 807 NLP papers, in which an LLM is given a research context (background, methodology, main experiments) and the name of a module or process, and must produce a detailed ablation study design. The authors evaluate 18 frontier LLMs using expert human ratings on importance, faithfulness, and soundness, and compare these with an LLM-as-judge automated evaluator. They also construct AbGen-Eval, a meta-evaluation benchmark built from the same human ratings, and report that current automated evaluators have low instance-level correlation with human judgments. Two user studies examine whether researcher feedback improves LLM-generated ablation designs and whether the approach transfers to biomedical and computer-network domains.

Significance. If the results hold, AbGen is a useful first benchmark for a genuinely scientific task—ablation study design—and the public data and code will let other groups build on it. The paper also ships a meta-evaluation benchmark that directly targets the reliability of LLM-as-judge scoring, which is a timely contribution. The benchmark construction is reasonably transparent: source selection, annotation, validation, and revision (273 of 1,500 examples revised) are described, and a 100-example quality check reports over 95% satisfaction. The main weakness is statistical: the human gold standard that supports both the model ranking and the meta-evaluation consists of 100 testmini examples per model, each scored by a single annotator, with no reported annotator balance across models and no human-human Pearson/ICC reliability ceiling. These issues are fixable and do not invalidate the benchmark, but they currently weaken the central quantitative claims.

major comments (4)
  1. [Section 4.2, Table 2] The model ranking is built on 100 testmini examples per model, with each output scored by a single expert evaluator, yet no information is given about how the four evaluators were assigned across the 18 models. Because annotator severity differences can move a model's mean by more than the observed 0.11-point gap between DeepSeek-R1-0528 (4.11) and o4-mini (4.00), the ordering is not robust unless assignment is balanced or annotator effects are modeled. Please report per-example scores with annotator IDs, test for balance across models, and provide severity-adjusted estimates (e.g., a mixed-effects model) or an equivalent analysis.
  2. [Section 4.1, 'Measuring Performance of Real Paper and Expert'] The human-expert baseline of 4.80 is computed on only 20 examples from 10 papers, solved by Annotators 1 and 4, and is not the same 100-item set used for LLM scoring. The headline comparison 'human expert 4.80 vs DeepSeek 4.11' is therefore not a same-items comparison; the only matched reference available on the 100-example LLM set is the original-paper ablation at 4.77. Please either collect expert scores on the same 100 examples used for LLM evaluation or clearly present the expert baseline as a separate, non-matched estimate rather than implying a direct gap.
  3. [Sections 3.2 and 5.1, Tables 2 and 5] Single-annotator human scores are used as the gold standard in AbGen-Eval, and the only inter-annotator statistic reported is Cohen's Kappa on 40 outputs. Without a human-human Pearson or ICC on the same scoring task, the low machine-human correlations (e.g., 0.164 to 0.329 for GPT-4.1-mini) cannot be definitively interpreted as evidence that automated evaluation is unreliable, because part of the disagreement may reflect noise in the single-annotator gold standard. Please report human-human reliability on a larger overlap, use weighted Kappa/ICC for ordinal 1-5 scores, and consider using aggregated multiple ratings for the gold standard before drawing conclusions about automated evaluator reliability.
  4. [Table 2 caption and Section 4.2] The automated evaluation scores are reported on the test subset (1,000 examples) while human scores are reported on 100 examples from testmini. The apparent automated-versus-human discrepancy in this table is therefore not a same-item comparison; the same-item evidence appears only later via AbGen-Eval. Please compute automated scores on the same 100 human-evaluated examples for Table 2, or explicitly label the comparison as cross-subset and move the reliability argument entirely to the matched AbGen-Eval setting.
minor comments (5)
  1. [Appendix A.1, Table 6] The caption says 'Human evaluation over 200 samples' while Section 2.5 says the final quality check was on 100 examples; please reconcile this discrepancy.
  2. [Equation (1)] Equation (1) writes the task as arg max over A of P(A|C,M) but does not define the probability distribution; clarify that this is the model's generation distribution or replace the notation.
  3. [Table 3] The caption says 'A summary of GPT-4o's failure cases' while Section 4.2 says the error analysis is based on top-performing LLMs more generally; clarify exactly which models contributed the 100 failure cases.
  4. [Section 5.1] The sentence '18 LLM outputs × 100 human assessments = 1,800 examples' is ambiguous because each output receives three human scores; please say '1,800 model-output assessments' or clarify that each assessment is one (output, score) pair.
  5. [Sections 2.4 and 3.1] There are minor typos, including 'requried' in Section 2.4 and 'ambiguious' in Section 3.1; a careful proofread would fix these.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AbGen is an empirical benchmark and meta-evaluation dataset, not a derivation, and no prediction is reduced to its own inputs.

full rationale

The paper builds a benchmark dataset by restructuring research papers into research contexts and reference ablation studies, then evaluates LLM-generated ablation designs through human ratings and LLM-as-judge correlations. None of the central claims are derived from a fitted parameter or from an imported theorem that already contains the conclusion. The reference ablation studies come from the same source papers as the research contexts, but model outputs are not scored by matching the reference; human evaluators initially score without seeing the reference, and the reference is used only as a post-hoc check. This design may reward reconstruction of original authors' choices, but that is a validity and construct concern, not circularity, because the evaluation signal is human judgment of importance, faithfulness, and soundness, not the reference itself. The AbGen-Eval meta-benchmark is defined with human judgments as the gold standard, which is the intended and standard definition of a meta-evaluation dataset; the paper then computes correlations between independent LLM-judge scores and those human scores. No parameter is fit to the evaluation targets and then reported as a prediction, no uniqueness theorem is imported from the authors' prior work to force a choice, and no known result is merely renamed. Self-citations appear only in related work and methodological precedents and are not load-bearing for the paper's conclusions. Therefore the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The central claim is an empirical evaluation built on expert annotations and human judgments, so the ledger is naturally about the quality and representativeness of those human-derived labels rather than mathematical or physical assumptions. The filter and annotation criteria introduce expert-judgment dependence, which is the main risk to the benchmark's validity.

assumptions (5)
  • domain assumption Expert annotators with NLP publications can reliably restructure ablation studies from source papers with sufficient fidelity.
    The entire benchmark rests on the quality of the expert-annotated research contexts and reference ablation studies; §2.3 and §2.4 describe the annotation process.
  • domain assumption Human expert Likert ratings on importance, faithfulness, and soundness are a valid gold standard for ablation study quality.
    The meta-evaluation in §5.1 treats human scores as ground truth, following prior meta-evaluation works, but the construct validity of these three criteria is not independently proven.
  • domain assumption The 100-example testmini sample and 20-example expert comparison are representative of the full benchmark.
    Human evaluation in §4.2 is performed on a random sample of 100 examples from testmini, and the expert-level estimate uses only 20 examples from 10 papers, a small basis for ranking 18 models.
  • domain assumption Papers with fewer than two ablation studies or deemed low quality by annotators can be excluded without biasing the benchmark.
    Manual filtering criteria in §2.2 are applied by expert judgment, which introduces a possible selection bias toward papers with well-defined ablations.
  • domain assumption Ablation study quality can be assessed by the three chosen criteria alone, excluding fluency and result analysis.
    The evaluation criteria in §3.1 were chosen through discussion with senior researchers, but the completeness of the construct is not empirically validated.
invented entities (2)
  • AbGen independent evidence
    purpose: Benchmark for evaluating LLMs in designing ablation studies.
    The dataset is publicly released, so others can compute on it. The predictive validity of the benchmark is what is being tested.
  • AbGen-Eval independent evidence
    purpose: Meta-evaluation benchmark for assessing LLM-as-Judge reliability on AbGen.
    Built from 1,800 human-scored outputs and released for reuse, so its measurements of evaluator reliability are checkable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research." pith.science (2026). https://pith.science/paper/IPT7SHSY

@misc{pith2026250713300,
  author       = {Pith},
  title        = {Pith review of: AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPT7SHSY}},
  note         = {Machine review of arXiv:2507.13300}
}
read the original abstract

We introduce AbGen, the first benchmark designed to evaluate the capabilities of LLMs in designing ablation studies for scientific research. AbGen consists of 1,500 expert-annotated examples derived from 807 NLP papers. In this benchmark, LLMs are tasked with generating detailed ablation study designs for a specified module or process based on the given research context. Our evaluation of leading LLMs, such as DeepSeek-R1-0528 and o4-mini, highlights a significant performance gap between these models and human experts in terms of the importance, faithfulness, and soundness of the ablation study designs. Moreover, we demonstrate that current automated evaluation methods are not reliable for our task, as they show a significant discrepancy when compared to human assessment. To better investigate this, we develop AbGen-Eval, a meta-evaluation benchmark designed to assess the reliability of commonly used automated evaluation systems in measuring LLM performance on our task. We investigate various LLM-as-Judge systems on AbGen-Eval, providing insights for future research on developing more effective and reliable LLM-based evaluation systems for complex scientific tasks.

Figures

Figures reproduced from arXiv: 2507.13300 by the authors.

Figure 1
Figure 1. Overview of the research: the ablation study [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of ABGEN construction pipeline. ing LLMs in the context of designing ablation stud￾ies for scientific research. The dataset consists of 1,500 examples derived from 807 scientific papers in natural language processing (NLP). Each exam￾ple is carefully annotated and validated by NLP experts and includes a comprehensive research con￾text along with a reference ablation study, both restructured from the orig… view at source ↗
Figure 3
Figure 3. Prompt for ablation study generation. amples when evaluation. We report the evaluation results on [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Prompt for LLM-researcher interaction [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: A Failure Example of Misalignment with Research Context [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: A Failure Example of Ambiguity and Difficulty in Reproduction [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: A Failure Example of Partial Ablation or Incomplete Experimentation [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: A Failure Example of Insignificant Ablation Module [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: A Failure Example of Inherent Logical Inconsistencies [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Laradji, Laurent Charlin, and Christopher Pal

    Shubham Agarwal, Issam H. Laradji, Laurent Charlin, and Christopher Pal. 2024. http://arxiv.org/abs/2402.01788 Litllm: A toolkit for scientific literature review

    arXiv 2024

  4. [4]

    AI@Meta. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    arXiv 2024

  5. [5]

    Signe Altmäe, Alberto Sola-Leyva, and Andres Salumets. 2023. https://doi.org/https://doi.org/10.1016/j.rbmo.2023.04.009 Artificial intelligence in scientific writing: a friend or a foe? Reproductive BioMedicine Online, 47(1):3--9

    doi:10.1016/j.rbmo.2023.04.009 2023

  6. [6]

    Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. 2025 a . http://arxiv.org/abs/2505.19955 Mlr-bench: Evaluating ai agents on open-ended machine learning research

    arXiv 2025

  7. [7]

    Qiguang Chen, Mingda Yang, Libo Qin, Jinhao Liu, Zheng Yan, Jiannan Guan, Dengyun Peng, Yiyan Ji, Hanjing Li, Mengkang Hu, Yimeng Zhang, Yihao Liang, Yuhang Zhou, Jiaqi Wang, Zhi Chen, and Wanxiang Che. 2025 b . http://arxiv.org/abs/2507.01903 Ai4research: A survey of artificial intelligence for scientific research

    arXiv 2025

  8. [8]

    Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.179 Are factuality checkers reliable? adversarial meta-evaluation of factuality in summarization . In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2082--2095, Punta Cana, Dominican Republic. Association for Computational Linguistics

    doi:10.18653/v1/2021.findings-emnlp.179 2021

Show all 51 references
  1. [9]

    Mike D'Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259

    2024 arXiv

  2. [10]

    Smith, and Matt Gardner

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.365 A dataset of information-seeking questions and answers anchored in research papers . In Proceedings of the 2021 Conference of the North Amer...

    2021 doi

  3. [11]

    DeepSeek-AI. 2024. http://arxiv.org/abs/2412.19437 Deepseek-v3 technical report

    2024 arXiv

  4. [12]

    DeepSeek-AI. 2025. http://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    2025 arXiv

  5. [13]

    Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al. 2024. Llms assist nlp researchers: Critique paper (meta-) reviewing. arXiv preprint arXiv:2406.16253

    2024 arXiv

  6. [14]

    Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 S umm E val: Re-evaluating summarization evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

    2021 doi

  7. [15]

    Sengamedu, and Christos Faloutsos

    Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamedu, and Christos Faloutsos. 2024. https://openreview.net/forum?id=IZnrCGF9WI Large language models ( LLM s) on tabular data: Prediction, generation, and understanding - a survey . Tra...

    2024

  8. [16]

    Gemini. 2024. http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    2024 arXiv

  9. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  10. [18]

    Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-In Lee, and Moontae Lee. 2023. https://proceedings.mlr.press/v202/lee23n.html QASA : Advanced question answering on scientific articles . In Proceedings of the 40th International Conference on Machine Le...

    2023

  11. [19]

    Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, and Arman Cohan. 2024 a . https://doi.org/10.18653/v1/2024.findings-emnlp.904 M 3 S ci QA : A multi-modal multi-document scientific QA benchmark for evaluating foundation models . In Findings of the Association for ...

    2024 doi

  12. [20]

    Ruochen Li, Teerth Patel, Qingyun Wang, and Xinya Du. 2024 b . http://arxiv.org/abs/2408.14033 Mlr-copilot: Autonomous machine learning research based on large language models agents

    2024

  13. [21]

    Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.280 Benchmarking generation and evaluation capabilities of large language models f...

    2024 doi

  14. [22]

    Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, et al. 2023. Ml-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv preprint arXiv:2311.09835

    2023 arXiv

  15. [23]

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. https://doi.org/10.18653/v1/2020.acl-main.447 S 2 ORC : The semantic scholar open research corpus . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969...

    2020 doi

  16. [24]

    Renze Lou, Hanzi Xu, Sijia Wang, Jiangshu Du, Ryo Kamoi, Xiaoxin Lu, Jian Xie, Yuxuan Sun, Yusen Zhang, Jihyun Janice Ahn, Hongchao Fang, Zhuoyang Zou, Wenchao Ma, Xi Li, Kai Zhang, Congying Xia, Lifu Huang, and Wenpeng Yin. 2025. https://openreview.net/forum?id=RHAWcjIyl2 AAA...

    2025

  17. [25]

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. http://arxiv.org/abs/2408.06292 The ai scientist: Towards fully automated open-ended scientific discovery

    2024 arXiv

  18. [26]

    Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. 2024. http://arxiv.org/abs/2402.13610 Data-driven discovery with large generative models

    2024 arXiv

  19. [27]

    Meta AI . 2025. https://www.llama.com/models/llama-4 Llama 4: Natively multimodal mixture‑of‑experts language model

    2025

  20. [28]

    Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao ...

    2025 arXiv

  21. [29]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  22. [30]

    OpenAI. 2025 a . https://openai.com/index/o3-o4-mini-system-card-addendum-operator-o3/ Addendum to openai o3 and o4-mini system card: Openai o3 operator

    2025

  23. [31]

    OpenAI. 2025 b . https://openai.com/index/gpt-4-1/ Introducing gpt-4.1 in the api

    2025

  24. [32]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. 2022. https://api.semanticscholar.org/CorpusID:250729995 Scienceqa: a novel resource for question answering on scholarly articles . International Journal on Digital Libraries, 23:289 -- 301

    2022

  25. [33]

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2025. https://openreview.net/forum?id=M23dTGWCZy Can LLM s generate novel research ideas? a large-scale human study with 100+ NLP researchers . In The Thirteenth International Conference on Learning Representations

    2025

  26. [34]

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. http://arxiv.org/abs/2305.13062 Table meets llm: Can large language models understand structured table data? a benchmark and empirical study

    2024 arXiv

  27. [35]

    Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z. Li. 2024. http://arxiv.org/abs/2406.05688 Peer review as a multi-turn and long-context dialogue with role-based interactions

    2024 arXiv

  28. [36]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  29. [37]

    Qwen Team. 2025. http://arxiv.org/abs/2505.09388 Qwen3 technical report

    2025 arXiv

  30. [38]

    Chengye Wang, Yifei Shen, Zexi Kuang, Arman Cohan, and Yilun Zhao. 2025. http://arxiv.org/abs/2506.15569 Sciver: Evaluating foundation models for multimodal scientific claim verification

    2025 arXiv

  31. [39]

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024 a . http://arxiv.org/abs/2305.14259 Scimon: Scientific inspiration machines optimized for novelty

    2024 arXiv

  32. [40]

    Xintao Wang, Jiangjie Chen, Nianqi Li, Lida Chen, Xinfeng Yuan, Wei Shi, Xuyang Ge, Rui Xu, and Yanghua Xiao. 2024 b . Surveyagent: A conversational system for personalized and efficient research survey. arXiv preprint arXiv:2404.06364

    2024 arXiv

  33. [41]

    Po-Cheng Wu, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2022. https://api.semanticscholar.org/CorpusID:252904822 Incorporating peer reviews and rebuttal counter-arguments for meta-review generation . Proceedings of the 31st ACM International Conference on Information & Know...

    2022

  34. [42]

    Fangyuan Xu, Kyle Lo, Luca Soldaini, Bailey Kuehl, Eunsol Choi, and David Wadden. 2024. Kiwi: A dataset of knowledge-intensive writing instructions for answering research questions. arXiv preprint arXiv:2403.03866

    2024 arXiv

  35. [43]

    Zhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig, and Arman Cohan. 2025. http://arxiv.org/abs/2507.02694 Can llms identify critical limitations within scientific research? a systematic evaluation on ai research papers

    2025 arXiv

  36. [44]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  37. [45]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024 b . Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv preprint arXiv:2405.15793

    2024 arXiv

  38. [46]

    Griffiths, Yuan Cao, and Karthik R Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. https://openreview.net/forum?id=5Xc1ecxO1h Tree of thoughts: Deliberate problem solving with large language models . In Thirty-seventh Conference on Neural Informat...

    2023

  39. [47]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. https://openreview.net/forum?id=tr0KidwPLc Evaluating large language models at evaluating instruction following . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Yilun Zhao, Chengye Wang, Chuhan Li, and Arman Cohan. 2025 a . http://arxiv.org/abs/2507.10787 Can multimodal foundation models understand schematic diagrams? an empirical study on information-seeking qa over scientific papers

    2025 arXiv

  41. [49]

    Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025 b . http://a...

    2025

  42. [50]

    Ruiyang Zhou, Lu Chen, and Kai Yu. 2024 a . https://api.semanticscholar.org/CorpusID:269803977 Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks . In International Conference on Language Resources and Evaluation

    2024

  43. [51]

    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024 b . http://arxiv.org/abs/2404.04326 Hypothesis generation with large language models

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.