Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read No current large language model can reliably find the errors in a scientific manuscript, according to a new benchmark of 83 real papers with author-confirmed flaws.

desk verdict SPOT is a genuinely useful benchmark and the recall ceiling is a solid negative result, but the 6.1% precision claim is undermined by the paper's own non-exhaustive gold set, and the 'confidence' analysis is not about model confidence. read the letter →

arxiv 2505.11855 v1 pith:Y6QLESE7 submitted 2025-05-17 cs.CL

Guijin Son , Jiwoo Hong , Honglu Fan , Heejeong Nam , Hyunwoo Ko , Seungwon Lim , Jinyeop Song , Jinha Choi
show 3 more authors
Gonçalo Paulo Youngjae Yu Stella Biderman
This is my paper · ORCID
classification cs.CL
keywords SPOTbenchmarkscientificerrordetectionLLMverificationmultimodalevaluationretractionanderratumanalysisAIco-scientistprecisionrecallmanuscript
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SPOT, a benchmark of 83 published scientific manuscripts carrying 91 confirmed errors that led to errata or retractions. It then asks ten state-of-the-art LLMs to read each full paper and flag every error. On this task the best model, o3, finds only about one in five known errors (21.1% recall) while keeping precision at 6.1%, and every other model is close to zero. Models also report low confidence and rarely rediscover the same errors across repeated runs, so the paper argues current AI systems are not dependable as scientific verifiers.

What carries the argument

The object that carries the argument is SPOT itself, a benchmark built by collecting errors from retraction notices and post-publication peer review, keeping only errors the original authors explicitly acknowledged, then normalizing full PDFs into interleaved text and image sets for the models. The evaluation machinery is the exhaustively annotated ground truth: each model's flags are matched against the 91 annotations, any unannotated flag is counted as a false positive, and precision, recall, and pass@K summarize the results. This design is what turns individual model outputs into the paper's central negative result.

What would settle it

Have domain experts independently re-read a random sample of the model predictions that SPOT scored as false positives; if a material fraction of those are genuine errors that the benchmark's annotations missed, the central claim of near-zero precision would be weakened because precision would rise once the annotations are corrected.

Watch

Extended reading notes

Core claim

The central discovery is that the verification pass, in which an AI reads a full manuscript and identifies flaws, is a distinctly hard task: on SPOT, no evaluated model surpasses 21.1% recall or 6.1% precision, far below what reliable academic verification requires. The 91 errors span equation/proof errors, figure duplication, data inconsistencies, statistical reporting, reagent identity, and experiment setup, and the papers average about 12,000 text tokens and 18 images each. Reasoning models are uneven: o3 leads on equation/proof errors with 62.6% pass@4 but scores 0% on figure duplication, while GPT-4.1 reaches 44.4% on figure duplication. The paper therefore concludes that current LLM capabilities fall far short of dependable AI-assisted academic verification.

Load-bearing premise

The evaluation treats the 91 annotations as a complete list of the errors in each paper, so a model flag that names a real but unlisted error is counted as a false positive; if unannotated true errors are common, the reported precision is too low.

Editorial extensions

If this is right

  • AI co-scientist pipelines cannot currently rely on LLMs as final verifiers of their own or others' outputs.
  • Deployed AI manuscript checkers will generate many false alarms and miss most real errors, so human expert review remains necessary.
  • SPOT can serve as a fixed, contamination-controlled testbed for measuring progress in automated scientific verification.
  • Model confidence and repeated-run agreement are too low to support trust in a single error-detection pass.
  • Different models are strong in different error categories, suggesting hybrid verification pipelines rather than a single model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If SPOT's annotations are not exhaustive, the reported precision figures are lower bounds, and a re-annotation exercise would likely move them upward; the paper's own materials-science case study already documents an unannotated unit error.
  • The segment-only ablation suggests a cheap diagnostic extension: scoring models on pages that contain the known error would separate core error-detection ability from long-context navigation failures.
  • A testable next benchmark could ask models to produce a full derivation or calculation before flagging an error, since the case studies show failures on implicit multi-step reasoning rather than simple misreading.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SPOT, a benchmark of 83 published scientific papers paired with 91 errors serious enough to have prompted errata or retraction, and evaluates ten multimodal LLMs on the task of independently detecting these errors in full-length, multimodal manuscripts. The authors report that the best model (o3) achieves only 21.1% recall and 6.1% precision, with most other models near zero, and they interpret this as evidence that current LLMs are far from dependable for automated academic verification. The paper also contains a calibration analysis, a text-only ablation, and expert-led case studies in mathematics and materials science that illustrate failure modes such as hallucinated criticisms and student-like misunderstandings.

Significance. If the recall finding is robust, SPOT is a valuable and genuinely hard benchmark for AI-assisted scientific verification. The dataset construction is a clear strength: all errors are confirmed by original authors or retraction notices, human annotators perform a second audit, the papers are recent enough to limit parametric contamination, and the multimodal preprocessing pipeline is documented. The paper also ships the benchmark and code, which supports reproducibility. However, the precision ceiling, which is part of the headline claim, rests on an exhaustiveness assumption that the paper itself contradicts, and the calibration analysis is internally circular. These issues do not necessarily invalidate the central negative result about recall, but they do require substantive rework before the abstract's claims can be accepted as stated.

major comments (3)
  1. [Section 2.3, Appendix A, Appendix D.2] The precision metric in Eq. (1) treats the SPOT gold set as exhaustive: Section 2.3 states that 'any model-reported error not matching an annotation is counted as a false positive.' Appendix A immediately concedes that 'some true errors may be unannotated,' and Appendix D.2 documents a concrete instance in which Gemini 2.5 Pro identified a genuine unit error in [59] (1606 cm^2/g should be m^2/g) that is not in the gold set and would therefore be scored as a false positive. Because o3's precision is only 6.1% and its true-positive count is small, even a small number of such unannotated genuine errors across the 83 papers could materially increase the measured precision. The abstract's claim that no model surpasses 6.1% precision is therefore not established. I recommend either a human relabeling of all model-reported false positives or a careful rephrasing of the claim as 'precision with respect to the SPOT gold annotations,' with the abstract and Section 3.1 revised accordingly.
  2. [Section 3.2, Figure 4, Appendix C] The calibration analysis is internally circular. The 'confidence' estimator in Eq. (5), p-hat_{i,g} = 1 - C(n - c_{i,g}, K)/C(n, K), is a deterministic function of c_{i,g}, the number of runs in which error g was detected. These are the same per-error detection counts that enter the pass@K definition in Eq. (2). Figure 4 (right) plots the aggregated confidence from Eq. (6) against pass@4, which is comparing two transforms of identical data; the weak correlation carries no information about whether models are well-calibrated in any model-intrinsic sense. The left panel of Figure 4 is labeled as the density of each model's 'reported confidence,' but Appendix C derives confidence from detection counts rather than from model outputs. To support the paper's claim that 'confidence estimates are uniformly low' and that models are miscalibrated, the authors should either collect explicit model-reported confidence values (e.g., token probabilities or an explicit confidence field in the prompt) or relabel this analysis as an estimate of pass@K variability and remove the calibration interpretation.
  3. [Abstract, Section 3] The abstract states that 'across eight independent runs, models rarely rediscover the same errors, undermining their reliability,' but I could not find a direct analysis of per-error rediscovery rates in the main text or appendices. Table 2 reports pass@1 and pass@4 and their standard deviations, which are indirect evidence at best, and the bootstrap standard deviations are not the same as a statement about how often a particular error is detected across runs. If this is intended as a central reliability claim, the authors should add a dedicated analysis, for example a histogram or table of the per-error detection counts c_{i,g} across the eight runs, or the fraction of errors detected in exactly one, two, ..., eight runs.
minor comments (4)
  1. [Abstract] There are several typos: 'theacademic verification' should be 'the academic verification,' and 'multiplehuman-annotatederrors' should be 'multiple human-annotated errors.'
  2. [Section 2.1, Stage 3] The sentence 'In all cases where the author themselves admits the problem, we take this acknowledgment as confirmation of a genuine error' is grammatically awkward; also 'as many of the work cover ungoing areas of research' should be 'as many of the works cover ongoing areas of research.'
  3. [Figure 4] The left panel shows kernel density estimates of confidence but does not state the bandwidth or binning; please provide the smoothing parameters or a histogram with explicit bins so the densities are reproducible.
  4. [Table 3] The caption says '13 models' but the multi-modal panel lists 10 models and the text-only panel adds three more; please clarify that the '13 models' refers only to the text-only setting.

Circularity Check

1 steps flagged · score 5.0 of 10

Confidence calibration in §3.2/Appendix C is self-referential (built from the same detection counts as pass@K); the headline recall/precision benchmark is otherwise self-contained.

  1. self definitional [Appendix C, Eqs. (4)-(6); used in §3.2 and Figure 4]
    "To assign each ground-truth error g∈G_i a confidence score, we perform n independent runs (here n=8) and let c_{i,g} be the number of runs in which g is detected. ... Hence the unbiased estimator for the pass@K probability of error g is \hat{p}_{i,g} = 1 - ... (5). We then aggregate these per-error confidences into an overall self-estimated confidence: Confidence = ... (6)."

    Appendix C defines 'confidence' from c_{i,g}, the exact per-error detection counts over the same n=8 runs that define pass@K in Eq. (2)/(3). Equation (5) is the unbiased pass@K estimator for each error, and Eq. (6) averages these values and labels the result 'overall self-estimated confidence.' Section 3.2 and Figure 4 then plot this derived quantity against pass@4 and read the scatter as calibration. Both axes are functions of identical data (the same c_{i,g} counts and runs), so the comparison reduces to checking one estimator of pass@K against another estimator of pass@K. The claims of 'uniformly low confidence' and weak correlation with pass@4 are therefore consequences of the definitional choice, not independent evidence about whether model-reported confidence tracks accuracy.

full rationale

SPOT's dataset construction is self-contained: seeds come from Withdrarxiv/PubPeer, errors are retained only after author acknowledgment and a two-stage human sanity check, and recall/pass@K are computed directly against the 91 gold annotations. The headline result that o3 reaches only 21.1% recall and 6.1% precision is an external, falsifiable measurement and is not circular. The one genuine circular step is the calibration analysis: Appendix C builds 'confidence' from the same eight-run detection counts used to compute pass@K, then §3.2/Figure 4 compare this derived quantity to pass@4. I do not count the exhaustiveness assumption (Section 2.3) as circularity: treating the gold set as exhaustive is a standard benchmark convention, though Appendix A's admission that 'some true errors may be unannotated' and Appendix D.2's documented unannotated unit typo are a real validity threat to the precision ceiling, not a derivation-circle issue. Using GPT-4.1 as the matcher while GPT-4.1 is itself an evaluated model is a self-evaluation design concern, but no equation-level reduction was shown. Self-citations such as [60] are motivational and not load-bearing. Because the central benchmark claim is independent, the overall score stays moderate despite the concrete self-referential calibration step.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters because it builds a benchmark rather than a fitted model. The load-bearing assumptions are about the validity and completeness of the error annotations, the reliability of an LLM judge, and the contamination cutoff.

assumptions (4)
  • domain assumption Author acknowledgment of an error is taken as ground truth for the existence of that error.
    Stage 3 of Section 2.1 retains only errors explicitly admitted by original authors, assuming those admissions are accurate and unbiased.
  • domain assumption SPOT annotations are exhaustive for scoring purposes.
    Section 2.3 counts all non-annotated predictions as false positives; the paper's own Limitations and case study acknowledge genuine unannotated errors exist.
  • domain assumption GPT-4.1 can reliably match predicted errors to annotated errors by location and description.
    Section 2.3 uses GPT-4.1 as an LLM judge for the similarity check, with no validation of its matching accuracy or inter-annotator agreement.
  • domain assumption Papers published from 2024 onward are not memorized by the evaluated models.
    Section 2.1 uses a 2024 publication cutoff to reduce contamination; this is a heuristic assumption that the models have not seen the errors during training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research." pith.science (2026). https://pith.science/paper/Y6QLESE7

@misc{pith2026250511855,
  author       = {Pith},
  title        = {Pith review of: When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y6QLESE7}},
  note         = {Machine review of arXiv:2505.11855}
}
read the original abstract

Recent advances in large language models (LLMs) have fueled the vision of automated scientific discovery, often called AI Co-Scientists. To date, prior work casts these systems as generative co-authors responsible for crafting hypotheses, synthesizing code, or drafting manuscripts. In this work, we explore a complementary application: using LLMs as verifiers to automate the \textbf{academic verification of scientific manuscripts}. To that end, we introduce SPOT, a dataset of 83 published papers paired with 91 errors significant enough to prompt errata or retraction, cross-validated with actual authors and human annotators. Evaluating state-of-the-art LLMs on SPOT, we find that none surpasses 21.1\% recall or 6.1\% precision (o3 achieves the best scores, with all others near zero). Furthermore, confidence estimates are uniformly low, and across eight independent runs, models rarely rediscover the same errors, undermining their reliability. Finally, qualitative analysis with domain experts reveals that even the strongest models make mistakes resembling student-level misconceptions derived from misunderstandings. These findings highlight the substantial gap between current LLM capabilities and the requirements for dependable AI-assisted academic verification.

Figures

Figures reproduced from arXiv: 2505.11855 by the authors.

Figure 1
Figure 1. Overview of SPOT. Green indicates benchmark construction process, from seed collection through validation to normalization; blue indicates evaluation, where LLM outputs are compared to ground-truth errors and classified as true positives, false positives, or false negatives. with multiple human-annotated errors. Given large-scale multi-modal inputs with 12,000 text tokens and 18 images on average, multi-modal LLMs (… view at source ↗
Figure 2
Figure 2. Distribution of annotated errors by research domain and error type. Paper Subjects We present general statistics in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Category-specific performance and calibration of six LLMs on SPOT. Left: Kernel density estimates of each model’s reported confidence; all six models predominantly express very low confidence. Right: Scatter plot of mean reported confidence (see Appendix C for further details) versus pass@4 for each model (color), broken down by error type (shape). The dashed diagonal marks perfect calibration. and Llama-4-Maverick,… view at source ↗
Figures from the paper (18 more)
Figure 3
Figure 3. Figure 3: Performance of o3 and Llama-4- Maverick across six challenging STEM bench￾marks. The short red horizontal lines mark the gap ∆ = o3−Llama-4-Maverick for each benchmark. A New Challenging Benchmark for STEM. Fig￾ure 3 illustrates the performance of o3 on six bench￾marks…
Figure 5
Figure 5. Figure 5: o3’s feedback on [58]. On the other hand, Gemini-2.5-Pro tackles Theorem 3.8 as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Gemini-2.5-Pro’s feedback on [58]. 4.2 Material science In our materials science case study, we select [59], which uses TfOH to synthesize covalent organic frameworks (COFs). However, it mislabels several samples in its figures. These errors are easier to spot than tho…
Figure 7
Figure 7. Figure 7: o3’s feedback on [59]. Errors pointed out by o3 exemplify failures in both long-range context comprehension and multi-hop reasoning. For instance, in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Two of Gemini 2.5 Pro’s feedback on [59]. In (A) of [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Impact of context length on error de￾tection. Each bar shows ∆ = (segment-only - full￾paper) for precision and recall across five models (o3, Gemini 2.5 Pro, Claude 3.7-Sonnet:Thinking, Qwen 2.5-VL-72B-Instruct, Llama-4-Maverick). In [PITH_FULL_IMAGE:figures/full_fig_…
Figure 10
Figure 10. Figure 10: Performance of o4-mini with vary￾ing reasoning effort. Performance is reported from three independent trials. In [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: o3’s initial feedback on [73]. • Section 2.5: Corollary 2.7 incorrectly describes the R-span of the algebraic representatives for the real primitive cohomology H n−1 prim(X, R). The basis elements given, [ωji ] and [ √ −1 ωji ], correspond under φC to ℜ(ωk) and i ℜ(ωk…
Figure 12
Figure 12. Figure 12: Gemini-2.5-Pro’s initial feedback on [73]. Since we have already analyzed the LLM responses for [58] in Section 4, here we only describe some additional analysis on [73]. Our first attempt at running our pipeline introduced many OCR errors, such as missing conjugation…
Figure 13
Figure 13. Figure 13: Gemini-2.5-Pro’s regenerated feedback on [73]. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: o3’s regenerated feedback on [59]. Most of the errors OpenAI’s o3 pointed out on [59] were on the experimental setup and the conclusions that the authors derived from them. Although the specific points raised by o3 seem to be plausible mistakes, all of them either ste…
Figure 15
Figure 15. Figure 15: Gemini-2.5-Pro’s regenerated feedback on [59]. COF and they show how it does not produce such a good results, because of this Gemini claims that the reagent can actually be used because the authors used it in a figure. Contrary to o3, it does find a real mistake by no…
Figure 16
Figure 16. Figure 16: o3’s regenerated feedback on [74]. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png]
Figure 17
Figure 17. Figure 17: Gemini-2.5-Pro’s regenerated feedback on [74]. In [PITH_FULL_IMAGE:figures/full_fig_p029_17.png]
Figure 18
Figure 18. Figure 18: o3’s regenerated feedback on [75]. D.3 Environmental Science We selected [75] in environmental science, the paper conductsa comprehensive evaluation of human contributions to atmospheric CO2. In a data-driven analysis, [75] argues that human-induced CO2 emissions are …
Figure 19
Figure 19. Figure 19: Publication dates against first error-notice dates for the 83 manuscripts. Each point denotes one paper; blue markers note papers published in 2024, while red markers are those otherwise. passed our automated filters because revisions were submitted after 2024; we ret…
Figure 20
Figure 20. Figure 20: Guideline provided to annotators. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 21
Figure 21. Figure 21: Example image of annotation platform. F Additional Details on Evaluation Evaluation consists of two phases. In the first phase, the target LLM is prompted to identify potential errors in each paper using our “Generation Prompt.” In the second phase, we employ GPT-4.1 …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 4 citations worldwide. Full citation record

  1. Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System

    cs.AI 2026-07 conditional novelty 6.0 of 10

    MEDA uses LLM agents to formalize biological constraints and guide symbolic regression, recovering correct ODE structures for canonical and extrapolated biological models, with structural recovery driven mainly by lit...

Reference graph

Works this paper leans on

92 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [59]

    Superacid in situ protected synthesis of covalent organic frameworks

    Xingyao Ye, Ruoyang Liu, Xinyu Mu, Shanshan Tao, Hao Yang, Xuejiao J Gao, Shuo-Wang Yang, and Donglin Jiang. Superacid in situ protected synthesis of covalent organic frameworks. Journal of the American Chemical Society, 2025

    work page 2025

  2. [1]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  3. [2]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  4. [3]

    R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiyohiro Nakayama, Zhengyang Geng, Chen Wang, et al. R-bench: Graduate-level multi-disciplinary benchmarks for llm & mllm complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    arXiv 2025

  5. [4]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  6. [5]

    Physics: Benchmarking foundation models on university-level physics problem solving.arXiv preprint arXiv:2503.21821, 2025

    Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, and Arman Cohan. Physics: Benchmarking foundation models on university-level physics problem solving.arXiv preprint arXiv:2503.21821, 2025

    arXiv 2025

  7. [6]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109, 2024

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109, 2024

    arXiv 2024

  8. [7]

    Can chatgpt be used to generate scientific hypotheses?Journal of Materiomics, 10(3):578–584, 2024

    Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. Can chatgpt be used to generate scientific hypotheses?Journal of Materiomics, 10(3):578–584, 2024

    2024

Show all 92 references
  1. [8]

    Pasa: An llm agent for comprehensive academic paper search.arXiv preprint arXiv:2501.10120, 2025

    Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al. Pasa: An llm agent for comprehensive academic paper search.arXiv preprint arXiv:2501.10120, 2025

    2025 arXiv

  2. [9]

    Generative ai in writing research papers: a new type of algorithmic bias and uncertainty in scholarly work

    Rishab Jain and Aditya Jain. Generative ai in writing research papers: a new type of algorithmic bias and uncertainty in scholarly work. InIntelligent Systems Conference, pages 656–669. Springer, 2024

    2024

  3. [10]

    Towards an ai co-scientist

    Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864, 2025

    2025 arXiv

  4. [11]

    The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  5. [12]

    Ai mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolu- tion.bioRxiv, pages 2025–02, 2025

    José R Penadés, Juraj Gottweis, Lingchen He, Jonasz B Patkowski, Alexander Shurick, Wei- Hung Weng, Tao Tu, Anil Palepu, Artiom Myaskovsky, Annalisa Pawlosky, et al. Ai mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolu- tion....

    2025

  6. [13]

    Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller

    Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools.Nature Machine Intelli- gence, 6(5):525–535, 2024

    2024

  7. [14]

    Quantum many-body physics calculations with large language models.Communications Physics, 8(1):49, 2025

    Haining Pan, Nayantara Mudur, William Taranto, Maria Tikhanovskaya, Subhashini Venu- gopalan, Yasaman Bahri, Michael P Brenner, and Eun-Ah Kim. Quantum many-body physics calculations with large language models.Communications Physics, 8(1):49, 2025

    2025

  8. [15]

    Alphaevolve: a gemini-powered coding agent for design- ing advanced algorithms

    DeepMind. Alphaevolve: a gemini-powered coding agent for design- ing advanced algorithms. https://deepmind.google/discover/blog/ alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/ ,

  9. [16]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 10

    2023

  10. [17]

    Tabfact: A large-scale dataset for table-based fact verification

    Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164, 2019

    1909 arXiv

  11. [18]

    A review on fact extraction and verification.ACM Computing Surveys (CSUR), 55(1):1–35, 2021

    Giannis Bekoulis, Christina Papagiannopoulou, and Nikos Deligiannis. A review on fact extraction and verification.ACM Computing Surveys (CSUR), 55(1):1–35, 2021

    2021

  12. [19]

    Poly- fever: A multilingual fact verification benchmark for hallucination detection in large language models.arXiv preprint arXiv:2503.16541, 2025

    Hanzhi Zhang, Sumera Anjum, Heng Fan, Weijian Zheng, Yan Huang, and Yunhe Feng. Poly- fever: A multilingual fact verification benchmark for hallucination detection in large language models.arXiv preprint arXiv:2503.16541, 2025

    2025 arXiv

  13. [20]

    Sciclaims: An end-to-end generative system for biomedical claim analysis.arXiv preprint arXiv:2503.18526, 2025

    Raúl Ortega and José Manuel Gómez-Pérez. Sciclaims: An end-to-end generative system for biomedical claim analysis.arXiv preprint arXiv:2503.18526, 2025

    2025

  14. [21]

    Sciclaimhunt: A large dataset for evidence-based scientific claim verification

    Sujit Kumar, Anshul Sharma, Siddharth Hemant Khincha, Gargi Shroff, Sanasam Ranbir Singh, and Rahul Mishra. Sciclaimhunt: A large dataset for evidence-based scientific claim verification. arXiv preprint arXiv:2502.10003, 2025

    2025 arXiv

  15. [22]

    Core- bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363, 2024

    Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core- bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363, 2024

    2024 arXiv

  16. [23]

    Nlpeer: A unified resource for the computa- tional study of peer review.arXiv preprint arXiv:2211.06651, 2022

    Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. Nlpeer: A unified resource for the computa- tional study of peer review.arXiv preprint arXiv:2211.06651, 2022

    2022 arXiv

  17. [24]

    Peerqa: A scientific question answering dataset from peer reviews.arXiv preprint arXiv:2502.13668, 2025

    Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. Peerqa: A scientific question answering dataset from peer reviews.arXiv preprint arXiv:2502.13668, 2025

    2025 arXiv

  18. [25]

    Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets

    Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of self-influence for learning from noisy NLP datasets. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natu...

    2023 doi

  19. [26]

    FEVER: a large-scale dataset for fact extraction and VERification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent, editors,Proceedings of the 2018 Conference of the North American Chapter of the Associ...

    2018 doi

  20. [27]

    Fact or fiction: Verifying scientific claims

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods ...

    2020

  21. [28]

    Moprd: A multidisciplinary open peer review dataset.Neural Computing and Applications, 35(34): 24191–24206, 2023

    Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Moprd: A multidisciplinary open peer review dataset.Neural Computing and Applications, 35(34): 24191–24206, 2023

    2023

  22. [29]

    Automatically evaluating the paper reviewing capability of large language models.arXiv preprint arXiv:2502.17086, 2025

    Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. Automatically evaluating the paper reviewing capability of large language models.arXiv preprint arXiv:2502.17086, 2025

    2025

  23. [30]

    Openai o3 and o4-mini system card

    OpenAI. Openai o3 and o4-mini system card. https://openai.com/index/ o3-o4-mini-system-card/, April 2025. Accessed: 2025-05-12

    2025

  24. [31]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innova- tion

    Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innova- tion. https://ai.meta.com/blog/llama-4-multimodal-intelligence/ , April 2025. Accessed: 2025-05-12. 11

    2025

  25. [32]

    Withdrarxiv: A large-scale dataset for retraction study.arXiv preprint arXiv:2412.03775, 2024

    Delip Rao, Jonathan Young, Thomas Dietterich, and Chris Callison-Burch. Withdrarxiv: A large-scale dataset for retraction study.arXiv preprint arXiv:2412.03775, 2024

    2024 arXiv

  26. [33]

    Classification and analysis of pubpeer comments: How a web journal club is used.Journal of the Association for Information Science and Technology, 73(5):655–670, 2022

    José Luis Ortega. Classification and analysis of pubpeer comments: How a web journal club is used.Journal of the Association for Information Science and Technology, 73(5):655–670, 2022

    2022

  27. [35]

    American Invitational Mathematics Examination – AIME

    MAA. American Invitational Mathematics Examination – AIME. InAmerican Invitational Math- ematics Examination – AIME 2024, February 2024, February 2024. URL https://maa.org/ math-competitions/american-invitational-mathematics-examination-aime

    2024

  28. [36]

    Proof or bluff? evaluating llms on 2025 usa math olympiad.arXiv preprint arXiv:2503.21934, 2025

    Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunovi´c, Nikola Jovanovi´c, and Martin Vechev. Proof or bluff? evaluating llms on 2025 usa math olympiad.arXiv preprint arXiv:2503.21934, 2025

    2025 arXiv

  29. [37]

    Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    2025 arXiv

  30. [38]

    Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192, 2025

    Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning.arXiv preprint arXiv:2504.17192, 2025

    2025

  31. [39]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  32. [40]

    tiktoken: A fast bpe tokeniser for use with openai’s models

    OpenAI. tiktoken: A fast bpe tokeniser for use with openai’s models. https://github.com/ openai/tiktoken, 2025. GitHub repository; version 0.9.0 (Feb. 14, 2025); accessed May 12, 2025

    2025

  33. [41]

    Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures.arXiv preprint arXiv:2406.06565, 2024

    Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures.arXiv preprint arXiv:2406.06565, 2024

    2024 arXiv

  34. [42]

    Spoc: Search-based pseudocode to code.Advances in Neural Information Processing Systems, 32, 2019

    Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code.Advances in Neural Information Processing Systems, 32, 2019

    2019

  35. [43]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  36. [44]

    OpenAI. GPT-4.1. https://openai.com/index/gpt-4-1/ , 2025. Accessed: May 15, 2025

    2025

  37. [45]

    Gemini 2.5 pro

    Google Cloud. Gemini 2.5 pro. https://cloud.google.com/vertex-ai/ generative-ai/docs/models/gemini/2-5-pro, 2025. Accessed: 2025-05-12

    2025

  38. [46]

    Gemini 2.0 Flash Lite

    Google Cloud. Gemini 2.0 Flash Lite. https://cloud.google.com/vertex-ai/ generative-ai/docs/models/gemini/2-0-flash-lite , 2025. Accessed: May 15, 2025

    2025

  39. [47]

    Claude 3.7 Sonnet System Card

    Anthropic. Claude 3.7 Sonnet System Card. https://assets.anthropic.com/ m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf , 2025. Ac- cessed: May 15, 2025. 13

    2025

  40. [48]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  41. [49]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  42. [50]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    arXiv

  43. [51]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  44. [52]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  45. [53]

    On calibration of modern neural networks

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InInternational conference on machine learning, pages 1321–1330. PMLR, 2017

    2017

  46. [54]

    Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.Advances in neural information processing systems, 32, 2019

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.Advances in neural information processin...

    2019

  47. [55]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  48. [56]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  49. [57]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  50. [58]

    Multiplicative chow-künneth decomposition and homology splitting of configuration spaces, 2024

    Dan Petersen and Orsola Tommasi. Multiplicative chow-künneth decomposition and homology splitting of configuration spaces, 2024. URLhttps://arxiv.org/abs/2401.06455

    2024 arXiv

  51. [60]

    Llm-as-a-judge & reward model: What they can and cannot do.arXiv preprint arXiv:2409.11239, 2024

    Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they can and cannot do.arXiv preprint arXiv:2409.11239, 2024

    2024 arXiv

  52. [61]

    Discourse-based objectives for fast unsupervised sentence representation learning.arXiv preprint arXiv:1705.00557, 2017

    Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning.arXiv preprint arXiv:1705.00557, 2017

    2017 arXiv

  53. [62]

    Self-instruct: Aligning language models with self-generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022. 14

    2022 arXiv

  54. [63]

    Reviewer2: Optimizing review genera- tion through prompt generation.arXiv preprint arXiv:2402.10886, 2024

    Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. Reviewer2: Optimizing review genera- tion through prompt generation.arXiv preprint arXiv:2402.10886, 2024

    2024 arXiv

  55. [64]

    Scientific opinion summarization: Paper meta-review generation dataset, methods, and evaluation

    Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, and Heng Ji. Scientific opinion summarization: Paper meta-review generation dataset, methods, and evaluation. In Artificial Intelligence for Research and Democracy: First International Workshop, AI4Research 2024, a...

    2024

  56. [65]

    Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.arXiv preprint arXiv:2109.09774, 2021

    Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.arXiv preprint arXiv:2109.09774, 2021

    2014 arXiv

  57. [66]

    A noise audit of the peer review of a scientific article: a wpom journal case study.WPOM-Working Papers on Operations Management, 14(2): 137–166, 2023

    Tomas Bonavia and Juan A Marin-Garcia. A noise audit of the peer review of a scientific article: a wpom journal case study.WPOM-Working Papers on Operations Management, 14(2): 137–166, 2023

    2023

  58. [67]

    Michelan- gelo: Long context evaluations beyond haystacks via latent structure queries.arXiv preprint arXiv:2409.12640, 2024

    Kiran V odrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelan- gelo: Long context evaluations beyond haystacks via latent structure queries.arXiv preprint arXiv:2409...

    2024 arXiv

  59. [68]

    Scaling scaling laws with board games.arXiv preprint arXiv:2104.03113, 2021

    Andy L Jones. Scaling scaling laws with board games.arXiv preprint arXiv:2104.03113, 2021

    2021 arXiv

  60. [69]

    Linguistic generalizability of test-time scaling in mathematical reasoning.arXiv preprint arXiv:2502.17407, 2025

    Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning.arXiv preprint arXiv:2502.17407, 2025

    2025 arXiv

  61. [70]

    Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  62. [71]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  63. [72]

    Finding flawed fictions: Evaluating complex reasoning in language models via plot hole detection.arXiv preprint arXiv:2504.11900, 2025

    Kabir Ahuja, Melanie Sclar, and Yulia Tsvetkov. Finding flawed fictions: Evaluating complex reasoning in language models via plot hole detection.arXiv preprint arXiv:2504.11900, 2025

    2025

  64. [73]

    Algebraic description of complex conjugation on cohomology of a smooth projective hypersurface, 2024

    Jeehoon Park, Junyeong Park, and Philsang Yoo. Algebraic description of complex conjugation on cohomology of a smooth projective hypersurface, 2024. URL https://arxiv.org/abs/ 2402.14546

    2024 arXiv

  65. [74]

    Reem Altuijri, A Atta, E Abdeltwab, and MM Abdelhamied. Impacts of low energy argon beam on enhancing the surface wettability and electrical performance of ca/pani films.ECS Journal of Solid State Science and Technology, 13(4):043017, 2024

    2024

  66. [75]

    Limitations

    Michael Simpson. The scientific case against net zero: Falsifying the greenhouse gas hypothesis. Journal of Sustainable Development, 17(6):137–157, 2024. doi: 10.5539/jsd.v17n6p137. URL https://ideas.repec.org/a/ibn/jsd123/v17y2024i6p137.html. 15 NeurIPS Paper Checklist 1.Clai...

    2024 doi

  67. [78]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  68. [79]

    The author’s note contains an error location that does not sufficiently cover all the affected results

  69. [80]

    Conversely, false positives may occur when:

    There exist smaller errors unrelated to the main technical error in the preprint. Conversely, false positives may occur when:

  70. [81]

    reasoning effort

    An LLM correctly points out a theorem that contains an error, but the content in the LLM’s response is still irrelevant. We therefore recommend a secondary expert review, particularly for domains with complex logical dependencies or deep specialization, to validate and refine ...

  71. [82]

    decreased from 0.116 eV. . . to 1.03 eV

    or derivation. Using an potentially incorrect or unsubstantiated formula invalidates the calculated Wm values. • Results and Discussion – Calculation ofWm:The text states that the estimated Wm “decreased from 0.116 eV. . . to 1.03 eV .” However, the authors’ reported values(0....

    2024

  72. [83]

    Complete the six annotation questions in the right panel

  73. [84]

    ClickSave Annotationto append toannotations.csv

  74. [85]

    Repeat until 3–5 rows are completed

  75. [86]

    Generation Prompt

    ClickSubmitto sendannotations.csvto the maintainer. Figure 20:Guideline provided to annotators. 31 Figure 21:Example image of annotation platform. F Additional Details on Evaluation Evaluation consists of two phases. In the first phase, the target LLM is prompted to identify p...

  76. [87]

    annotations

    "annotations": the ground-truth errors (each has "location" and "description")

  77. [88]

    predictions

    "predictions": the model’s reported errors (same format). Task

  78. [89]

    Compare each prediction against each annotation

  79. [90]

    location

    A match occurs only when both "location" and "description" are identical

  80. [91]

    matches": [ {

    Your output should be generated in the following format: <analysis> Analysis and comparison of each prediction and annotation. </analysis> <response> { "matches": [ { "location": the location of the matched object, which should be based on the annotated location, "description"...

  81. [92]

    In Table 4 to 13 we present detailed results of each model from Table 2

  82. [93]

    Table 4:Mean and standard deviation of pass@K for o3 (K∈{1,2,4} ) by error category (left) and paper category (right)

    In Table 14 to 26 we present detailed results of the text-only evaluation from Table 3. Table 4:Mean and standard deviation of pass@K for o3 (K∈{1,2,4} ) by error category (left) and paper category (right). Detailed evaluations results for Table 2. Error Category Paper Categor...

  83. [2020]

    doi: 10.18653/v1/2020.emnlp-main.609

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.609. URLhttps://aclanthology.org/2020.emnlp-main.609/

    2020 doi

  84. [2025]

    Accessed: 2025-05-15

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.