{"id":"06e652e1-90b9-4c71-bf2e-9620cb2e15ae","arxiv_id":"2505.11855","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SPOT shows that state-of-the-art AI models detect fewer than one in five known errors in full scientific papers, with precision below 7%.","lead":"The authors built SPOT, a set of 83 published papers with 91 confirmed errors, and asked ten advanced AI models to find the errors. The best model found only about one in five errors and produced many false alarms, suggesting AI systems are not yet trustworthy scientific verifiers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6.1% precision ceiling is not a valid capability bound because SPOT's gold set is treated as exhaustive even though the paper's own Limitations and case study document unannotated genuine errors, which are scored as false positives.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: SPOT's annotations are treated as exhaustive when computing precision, although the paper itself admits unannotated true errors may exist and documents one in Appendix D.2. This is the most direct threat to the abstract's quantitative claim that 'none surpasses 21.1% recall or 6.1% precision,' because the precision bound depends on counting genuine unannotated errors as false positives. The recall bound is largely immune to this issue, and the qualitative conclusion that current models are far from dependable would survive even a doubled or tripled precision estimate, so the central message is probably robust. However, the exact 6.1% ceiling is not trustworthy without a systematic audit. The reader's CONDITIONAL verdict already captures this need for rework, so the stress-test pass does not change the verdict. Secondary issues, such as reliance on GPT-4.1 as an error-matching judge and the questionable 'confidence' metric in Appendix C, also warrant attention, but the annotation-exhaustiveness problem is the most load-bearing because it is acknowledged and contradicted within the manuscript itself.","tokens_in":38077,"tokens_out":7709,"duration_ms":85822,"concrete_test":"Collect every model prediction that the GPT-4.1 matcher classified as a false positive across the full SPOT corpus (or a stratified random sample of at least 100 such flags). Have domain experts, blinded to the SPOT annotations, independently judge whether each flag identifies a genuine scientific error in the paper. Recompute precision for o3 and the other models using these human judgments, treating only flags judged not to be genuine errors as true false positives. If o3's recomputed precision exceeds 6.1% by more than a few points, the reported precision ceiling is an artifact of the non-exhaustive gold set; if it remains near 6.1%, the benchmark's precision claim is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 defines precision by counting any model-reported error that does not match a SPOT annotation as a false positive, explicitly assuming the annotations are exhaustive. Appendix A immediately concedes that 'some true errors may be unannotated,' and Appendix D.2 documents a concrete case: Gemini 2.5 Pro identified a genuine unit error in [59] (1606 cm^2/g should be m^2/g), which the protocol would classify as a false positive because it is not in the gold set. The paper asserts such cases are 'highly unlikely,' but the only systematic evidence offered is the small set of case studies, which already contains at least one counterexample. Since o3's precision is only 6.1% and its true-positive count is small, even a handful of unannotated genuine errors across the 83 papers could move precision materially. Without a human relabeling of all model-reported false positives, the abstract's claim that no model surpasses 6.1% precision is not established. The recall finding (max 21.1%) is less affected by this issue and likely robust, but the precision half of the headline claim rests on an assumption the paper itself contradicts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SPOT, a benchmark of 83 published scientific papers paired with 91 errors serious enough to have prompted errata or retraction, and evaluates ten multimodal LLMs on the task of independently detecting these errors in full-length, multimodal manuscripts. The authors report that the best model (o3) achieves only 21.1% recall and 6.1% precision, with most other models near zero, and they interpret this as evidence that current LLMs are far from dependable for automated academic verification. The paper also contains a calibration analysis, a text-only ablation, and expert-led case studies in mathematics and materials science that illustrate failure modes such as hallucinated criticisms and student-like misunderstandings.","tokens_in":38322,"tokens_out":3102,"duration_ms":32379,"significance":"If the recall finding is robust, SPOT is a valuable and genuinely hard benchmark for AI-assisted scientific verification. The dataset construction is a clear strength: all errors are confirmed by original authors or retraction notices, human annotators perform a second audit, the papers are recent enough to limit parametric contamination, and the multimodal preprocessing pipeline is documented. The paper also ships the benchmark and code, which supports reproducibility. However, the precision ceiling, which is part of the headline claim, rests on an exhaustiveness assumption that the paper itself contradicts, and the calibration analysis is internally circular. These issues do not necessarily invalidate the central negative result about recall, but they do require substantive rework before the abstract's claims can be accepted as stated.","major_comments":[{"comment":"The precision metric in Eq. (1) treats the SPOT gold set as exhaustive: Section 2.3 states that 'any model-reported error not matching an annotation is counted as a false positive.' Appendix A immediately concedes that 'some true errors may be unannotated,' and Appendix D.2 documents a concrete instance in which Gemini 2.5 Pro identified a genuine unit error in [59] (1606 cm^2/g should be m^2/g) that is not in the gold set and would therefore be scored as a false positive. Because o3's precision is only 6.1% and its true-positive count is small, even a small number of such unannotated genuine errors across the 83 papers could materially increase the measured precision. The abstract's claim that no model surpasses 6.1% precision is therefore not established. I recommend either a human relabeling of all model-reported false positives or a careful rephrasing of the claim as 'precision with respect to the SPOT gold annotations,' with the abstract and Section 3.1 revised accordingly.","section":"Section 2.3, Appendix A, Appendix D.2"},{"comment":"The calibration analysis is internally circular. The 'confidence' estimator in Eq. (5), p-hat_{i,g} = 1 - C(n - c_{i,g}, K)/C(n, K), is a deterministic function of c_{i,g}, the number of runs in which error g was detected. These are the same per-error detection counts that enter the pass@K definition in Eq. (2). Figure 4 (right) plots the aggregated confidence from Eq. (6) against pass@4, which is comparing two transforms of identical data; the weak correlation carries no information about whether models are well-calibrated in any model-intrinsic sense. The left panel of Figure 4 is labeled as the density of each model's 'reported confidence,' but Appendix C derives confidence from detection counts rather than from model outputs. To support the paper's claim that 'confidence estimates are uniformly low' and that models are miscalibrated, the authors should either collect explicit model-reported confidence values (e.g., token probabilities or an explicit confidence field in the prompt) or relabel this analysis as an estimate of pass@K variability and remove the calibration interpretation.","section":"Section 3.2, Figure 4, Appendix C"},{"comment":"The abstract states that 'across eight independent runs, models rarely rediscover the same errors, undermining their reliability,' but I could not find a direct analysis of per-error rediscovery rates in the main text or appendices. Table 2 reports pass@1 and pass@4 and their standard deviations, which are indirect evidence at best, and the bootstrap standard deviations are not the same as a statement about how often a particular error is detected across runs. If this is intended as a central reliability claim, the authors should add a dedicated analysis, for example a histogram or table of the per-error detection counts c_{i,g} across the eight runs, or the fraction of errors detected in exactly one, two, ..., eight runs.","section":"Abstract, Section 3"}],"minor_comments":[{"comment":"There are several typos: 'theacademic verification' should be 'the academic verification,' and 'multiplehuman-annotatederrors' should be 'multiple human-annotated errors.'","section":"Abstract"},{"comment":"The sentence 'In all cases where the author themselves admits the problem, we take this acknowledgment as confirmation of a genuine error' is grammatically awkward; also 'as many of the work cover ungoing areas of research' should be 'as many of the works cover ongoing areas of research.'","section":"Section 2.1, Stage 3"},{"comment":"The left panel shows kernel density estimates of confidence but does not state the bandwidth or binning; please provide the smoothing parameters or a histogram with explicit bins so the densities are reproducible.","section":"Figure 4"},{"comment":"The caption says '13 models' but the multi-modal panel lists 10 models and the text-only panel adds three more; please clarify that the '13 models' refers only to the text-only setting.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The recall finding is likely robust and the dataset is a useful contribution, but the precision ceiling and the calibration story are both load-bearing parts of the abstract's claims. The authors' own case study inadvertently provides a concrete counterexample to the exhaustiveness assumption, which makes the precision issue particularly serious. I would encourage the editor to request a revision that either re-annotates the false positives or carefully rewrites the claims, and that replaces the circular calibration analysis with model-reported confidences or removes the calibration claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the SPOT benchmark is genuinely useful: 83 recent full papers, 91 author-confirmed errors across ten fields, multimodal inputs, and a released dataset. The curation is careful, and the recall numbers are a solid negative result — o3 finds only 21% of known errors after four tries, most open models are near zero. Second, the precision ceiling of 6.1% is not established, and the calibration analysis in Section 3.2 is mislabeled. Trust the recall finding and the benchmark itself; do not quote the precision or confidence claims without fixing them.\n\nWhat is new here is real. Prior benchmarks stop at sentence-level claims or CS-only review generation; SPOT asks models to verify whole papers, including figures and equations, with errors checked against the original authors. That is the right next test for AI co-scientist pipelines. The multi-modality ablation is thoughtful: o3 and Gemini actually use images, while most other models treat figures as distractors. The expert case studies are honest and show genuine student-like errors, not just cherry-picked failures.\n\nThe soft spots are in the interpretation. Section 2.3 treats the gold set as exhaustive: anything a model flags that is not an annotation is a false positive. The Limitations appendix concedes that some true errors may be unannotated, and Appendix D.2 documents exactly one: Gemini 2.5 Pro found a real unit typo (cm^2/g should be m^2/g) that the protocol would count as a false positive. The paper says such cases are highly unlikely, but the case study already contradicts that. With o3 at 6.1% precision and small true-positive counts, even a handful of such errors changes the number materially. The recall number (21.1%) is less sensitive to this and probably robust.\n\nBigger problem: the confidence analysis. Appendix C defines an unbiased estimator of per-error pass@K probability from the same detection counts that produce the pass@4 metric. Figure 4 then compares that estimator to pass@4 and calls it 'self-estimated confidence.' This is not model confidence; it is two transforms of identical data. The finding that confidence is uniformly low is an artifact of the estimator, not a discovery about calibration. The authors should either ask models for actual confidence judgments or drop the calibration framing.\n\nThe missing human baseline is a gap but a minor one for a benchmark paper; the absolute numbers are already telling.\n\nThis paper deserves peer review. The benchmark is a real asset and the negative result on recall is important. A serious referee should ask for a relabeling or candid caveat of the precision claim, a rewrite of the calibration section, and preferably a human expert baseline. Even with those changes, the central message — current LLMs cannot reliably verify scientific manuscripts — holds up.","headline":"SPOT is a genuinely useful benchmark and the recall ceiling is a solid negative result, but the 6.1% precision claim is undermined by the paper's own non-exhaustive gold set, and the 'confidence' analysis is not about model confidence.","tokens_in":38834,"tokens_out":4805,"would_cite":true,"duration_ms":44033,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No current large language model can reliably find the errors in a scientific manuscript, according to a new benchmark of 83 real papers with author-confirmed flaws.","keywords":["SPOT benchmark","scientific error detection","LLM verification","multimodal LLM evaluation","retraction and erratum analysis","AI co-scientist","precision and recall evaluation","scientific manuscript verification"],"falsifier":"Have domain experts independently re-read a random sample of the model predictions that SPOT scored as false positives; if a material fraction of those are genuine errors that the benchmark's annotations missed, the central claim of near-zero precision would be weakened because precision would rise once the annotations are corrected.","tokens_in":37921,"feed_emoji":"🔬","tokens_out":6142,"duration_ms":61264,"temperature":0.7,"pith_summary":"This paper introduces SPOT, a benchmark of 83 published scientific manuscripts carrying 91 confirmed errors that led to errata or retractions. It then asks ten state-of-the-art LLMs to read each full paper and flag every error. On this task the best model, o3, finds only about one in five known errors (21.1% recall) while keeping precision at 6.1%, and every other model is close to zero. Models also report low confidence and rarely rediscover the same errors across repeated runs, so the paper argues current AI systems are not dependable as scientific verifiers.","feed_headline":"Best AI finds only 21% of errors in flawed papers","feed_subtitle":"Testing LLMs as manuscript verifiers on 83 real papers, precision tops out near 6 percent.","key_machinery":"The object that carries the argument is SPOT itself, a benchmark built by collecting errors from retraction notices and post-publication peer review, keeping only errors the original authors explicitly acknowledged, then normalizing full PDFs into interleaved text and image sets for the models. The evaluation machinery is the exhaustively annotated ground truth: each model's flags are matched against the 91 annotations, any unannotated flag is counted as a false positive, and precision, recall, and pass@K summarize the results. This design is what turns individual model outputs into the paper's central negative result.","core_discovery":"The central discovery is that the verification pass, in which an AI reads a full manuscript and identifies flaws, is a distinctly hard task: on SPOT, no evaluated model surpasses 21.1% recall or 6.1% precision, far below what reliable academic verification requires. The 91 errors span equation/proof errors, figure duplication, data inconsistencies, statistical reporting, reagent identity, and experiment setup, and the papers average about 12,000 text tokens and 18 images each. Reasoning models are uneven: o3 leads on equation/proof errors with 62.6% pass@4 but scores 0% on figure duplication, while GPT-4.1 reaches 44.4% on figure duplication. The paper therefore concludes that current LLM capabilities fall far short of dependable AI-assisted academic verification.","pith_inferences":["If SPOT's annotations are not exhaustive, the reported precision figures are lower bounds, and a re-annotation exercise would likely move them upward; the paper's own materials-science case study already documents an unannotated unit error.","The segment-only ablation suggests a cheap diagnostic extension: scoring models on pages that contain the known error would separate core error-detection ability from long-context navigation failures.","A testable next benchmark could ask models to produce a full derivation or calculation before flagging an error, since the case studies show failures on implicit multi-step reasoning rather than simple misreading."],"forward_implications":["AI co-scientist pipelines cannot currently rely on LLMs as final verifiers of their own or others' outputs.","Deployed AI manuscript checkers will generate many false alarms and miss most real errors, so human expert review remains necessary.","SPOT can serve as a fixed, contamination-controlled testbed for measuring progress in automated scientific verification.","Model confidence and repeated-run agreement are too low to support trust in a single error-detection pass.","Different models are strong in different error categories, suggesting hybrid verification pipelines rather than a single model."],"supporting_citations":[{"why":"Supplies the WITHDRARXIV retraction records used as the first seed source for candidate papers with acknowledged critical errors.","marker":"[32]"},{"why":"Provides the keyword-query crawling method used to harvest PubPeer comments that seed the second candidate pool.","marker":"[33]"},{"why":"The GPT-4o filtering passes that select seed comments tied to specific locations and remove externally verifiable errors are built on this model.","marker":"[34]"},{"why":"o3 is the strongest evaluated model and sets the reported ceiling of 21.1% recall and 6.1% precision on SPOT.","marker":"[30]"},{"why":"The Llama-4 family results supply the near-zero open-model comparison that the paper contrasts with proprietary models.","marker":"[31]"},{"why":"Provides the LLM-based similarity-check approach used to decide which predicted error descriptions match benchmark annotations.","marker":"[41]"},{"why":"Defines the pass@K estimation procedure that the paper adopts for repeated error-detection runs and confidence estimation.","marker":"[43]"}],"fun_headline_variants":["AI verifiers miss 79% of real paper errors","SPOT benchmark shows AI can't reliably verify papers","Top AI scores just 21% recall spotting paper flaws","AI manuscript checkers: 21% recall, 6% precision","Why AI co-scientists fail at finding paper errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the 91 annotations as a complete list of the errors in each paper, so a model flag that names a real but unlisted error is counted as a false positive; if unannotated true errors are common, the reported precision is too low.","fun_headline_variants_meta":{"raw":{"variants":["AI verifiers miss 79% of real paper errors","SPOT benchmark shows AI can't reliably verify papers","Top AI scores just 21% recall spotting paper flaws","AI manuscript checkers: 21% recall, 6% precision","Why AI co-scientists fail at finding paper errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000192,"raw_usage":{"total_tokens":1333,"prompt_tokens":919,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":535,"tokens_out":414,"duration_ms":3753,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:46:04.870000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have domain experts independently re-read a random sample of the model predictions that SPOT scored as false positives; if a material fraction of those are genuine errors that the benchmark's annotations missed, the central claim of near-zero precision would be weakened because precision would rise once the annotations are corrected.","supporting_citations":[{"cited_title":"Classification and analysis of pubpeer comments: How a web journal club is used.Journal of the Association for Information Science and Technology, 73(5):655–670, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the keyword-query crawling method used to harvest PubPeer comments that seed the second candidate pool."},{"cited_title":"Openai o3 and o4-mini system card","cited_arxiv_id":null,"evidence_quote":"o3 is the strongest evaluated model and sets the reported ceiling of 21.1% recall and 6.1% precision on SPOT."},{"cited_title":"The llama 4 herd: The beginning of a new era of natively multimodal ai innova- tion","cited_arxiv_id":null,"evidence_quote":"The Llama-4 family results supply the near-zero open-model comparison that the paper contrasts with proprietary models."}],"review_version":1}