{"id":"6e8a43a8-2635-4e62-b1ed-5b3fc689e4b3","arxiv_id":"2506.08235","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CLAIM-BENCH evaluates six LLMs on claim-evidence extraction from 100 AI/ML papers, finding that iterative prompting improves recall at high computational cost.","lead":"This paper introduces CLAIM-BENCH, a benchmark that tests how well large language models can match scientific claims to supporting evidence in full research papers. It finds that closed-source models and multi-step prompting improve performance, but all models struggle with accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation metrics lack an operational span-matching rule, so the reported precision, recall, F1, and model rankings are not well-defined.","rationale":"The reader’s weakest assumption focuses on low inter-annotator agreement (kappa = 0.30 for evidence, F1 = 0.617 for links) and the resulting noise in gold labels. That is a valid concern and is indeed load-bearing for any benchmark. However, the manuscript’s primary quantitative results have an even more basic precondition: the evaluation protocol must define what counts as a match. Without that definition, precision, recall, and F1 are not well-defined even if the gold annotations were perfect. The paper explicitly defines the matching rule only for the auxiliary sentence_gap metric, leaving the central metrics underspecified. This is an internal inconsistency rather than a disagreement with external consensus, and it affects every experimental claim. The reader did mention ‘clarify the matching rule’ in the rationale, so there is partial agreement, but it was not presented as the weakest assumption. Because the issue can be fixed by specification and rerunning, the appropriate verdict remains conditional (as the reader already concluded), not outright rejection. The post-hoc selection of best-performing strategy per model is a further threat to the specific claim that closed-source models consistently outperform open-source ones, since Table 1 shows LLaMA-70B achieving higher claim F1 than GPT-4-Turbo. Still, the matching rule is the single most load-bearing concern because it undermines the validity of all measurements, including the strategy comparison and the benchmark’s reliability claim.","tokens_in":14309,"tokens_out":5224,"duration_ms":65407,"concrete_test":"Obtain the released dataset and evaluation code (or ask the authors to specify the rule) and recompute Table 1 and Figure 2 using two clearly defined matching rules: (a) exact substring match of spans and (b) token-level IoU ≥ 0.5. If the model rankings and the closed-vs-open and Single-Pass-vs-iterative conclusions are stable across both rules, the concern is resolved. If they flip, the headline claims are artifacts of the matching choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CLAIM-BENCH is a reliable diagnostic tool rests on the precision/recall/F1 measurements in §3.2 and Tables/Figures built from them. Yet the paper never specifies how a predicted span is counted as a true positive, false positive, or false negative. Section 3.2 defines P and R symbolically (Eqs. 1–2) but does not state the matching criterion: exact character match, token-level overlap, IoU ≥ threshold, or semantic equivalence. The only explicit matching rule in the manuscript is for the auxiliary sentence_gap metric (Eq. 3, “using Intersection over Union matching rule”), not for the headline P/R/F1 results. This gap is especially consequential because the annotation process itself acknowledges that annotators often select different but valid evidence spans (Appendix B.2). A strict matching rule would penalize models for legitimate annotation variability; a loose rule would inflate scores. Without a stated rule, Figure 2 and Table 1 cannot be interpreted, reproduced, or compared across models. The benchmark’s reliability claim therefore depends on an unstated evaluation assumption. A related but secondary issue is that Table 1 reports each model’s best performance across the three prompting strategies, which breaks comparability; for example, open-source LLaMA-70B’s best claim F1 (0.58) exceeds GPT-4-Turbo’s (0.56), undermining the abstract’s “consistently outperform” claim. The matching rule is the more fundamental gap because it affects every reported number, not just one comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CLAIM-BENCH, a benchmark of 100 AI/ML papers annotated with claims, evidence spans, and claim–evidence links by PhD students, and uses it to evaluate six large language models (GPT-4-Turbo, Claude 3.5 Sonnet, Gemini, LLaMA-70B, Ministral-8B, and Phi-3.5-MoE) under three prompting strategies: Single-Pass, Three-Pass, and One-by-One. The headline findings are that closed-source models generally outperform open-source models on precision and recall for claim and evidence identification, and that iterative prompting strategies improve the linking of dispersed evidence at higher computational cost. The paper also reports a new sentence_gap metric, execution times, and an analysis of recall as a function of document token length.","tokens_in":14559,"tokens_out":3995,"duration_ms":46142,"significance":"If the benchmark is valid, it fills a real gap in evaluating scientific comprehension: no prior benchmark directly tests whether LLMs can extract and link claims to their supporting evidence across full scientific papers. The paper has several strengths: the dataset is released, the annotation protocol and tool are described in detail, human agreement is measured explicitly, multiple model families and prompting strategies are compared, and the analysis includes practical dimensions such as runtime and token-length sensitivity. However, the benchmark's reliability claim is undermined by the low inter-annotator agreement on evidence spans (Cohen's kappa = 0.30) and by the absence of an operational span-matching rule for the headline precision/recall/F1 results. These issues are central because every reported model comparison and strategy conclusion depends on those numbers.","major_comments":[{"comment":"The precision, recall, and F1 metrics in Eqs. (1)–(2) are not operationally defined because the manuscript never states how a predicted span is matched to a gold span (e.g., exact character match, token-level overlap, IoU threshold, or semantic equivalence). Since Eq. (3) explicitly invokes an IoU matching rule for sentence_gap, the omission is conspicuous, and Figure 2 and Table 1 cannot be reproduced or compared across models without this rule. Please specify the matching criterion and report results under it.","section":"§3.2, Eqs. (1)–(2)"},{"comment":"The validation phase reports Cohen's kappa = 0.30 for evidence identification and F1 = 0.617 for claim–evidence links, yet the text describes CLAIM-BENCH as a dependable ground-truth resource. With only fair agreement on the very spans used as gold labels, the reported LLM precision and recall are partly artifacts of annotation noise. The paper needs a quantitative sensitivity analysis (for example, evaluating models against each annotator's labels separately or restricting evaluation to spans with multi-annotator agreement) and a discussion of how robust the model rankings are to this noise.","section":"§3.1 and Appendix B.2"},{"comment":"Table 1 reports each model's best performance across the three prompting strategies, while the abstract and §5.1 claim that closed-source models consistently outperform open-source counterparts. This is contradicted by the table itself: LLaMA-70B's best claim F1 of 0.58 exceeds GPT-4-Turbo's 0.56. Reporting only best-case results with no error bars or significance tests makes the central 'consistently outperform' claim unsupported; please report all strategy-specific results with variance and statistical comparisons.","section":"Table 1 and §5.1–5.2"},{"comment":"The strategy and token-length analyses rest on aggregate recall and pair-count comparisons without confidence intervals or significance tests; for example, the claim that iterative prompting 'neutralizes length effects' is based on a few mean values. Please include per-strategy standard deviations and pairwise significance tests, and reconcile the inconsistent strategy labels ('One-pass' in Figure 7 versus 'Single-Pass' and 'Three-Pass' in §4.1).","section":"§5.4–5.5 and Figure 6"}],"minor_comments":[{"comment":"The phrase 'length between 0 to 20 pages' should read 'length up to 20 pages'; also, the abstract's claim of 'over 300 claim–evidence pairs' should be reconciled with the dataset description in Section 3.1.","section":"§3.1"},{"comment":"Section 5.2 cites GPT-4-Turbo achieving precision 0.68 and recall 0.81, but Table 1 lists claim precision 0.66 and recall 0.57; please clarify which prompting strategy and content type these numbers refer to.","section":"§5.2 and Table 1"},{"comment":"The strategy names are inconsistent across figures: Figure 4 uses '1-pass' and Figure 7 uses 'One-pass', while Section 4.1 defines 'Single-Pass' and 'Three-Pass'; unify the terminology.","section":"Figures 4 and 7"},{"comment":"The Limitations section opens with an incomplete sentence ('While CLAIM-BENCH provides comprehensive insights into the capabilities of LLMs in scientific claim-evidence reasoning. Despite these insights...'); please rewrite for grammatical completeness.","section":"§8"},{"comment":"The reference list contains inconsistent formatting (e.g., 'LI et al., 2025' vs. 'Li et al., 2025') and at least one truncated entry ('Helpsteer2-preference'); please correct these issues.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle to acceptance is not the novelty or scope of the benchmark but the well-definedness and validity of the headline evaluation numbers. The missing span-matching rule and the low evidence-level agreement directly affect every precision/recall/F1 result and the resulting model rankings. These are fixable in a revision, so I am not recommending rejection, but they are not cosmetic presentation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core asset here is the dataset: 100 full 2024 ML/AI papers with claim–evidence annotations, spanning multiple domains, built with a purpose-made tool and checked by a second annotation pass. That is real work and it fills a genuine gap—no existing benchmark, at least among those cited, tests claim-to-evidence linking across whole papers. The three prompting strategies and the sentence-gap analysis are reasonable additions, and the authors are candid about the task's difficulty and their own limitations.\n\nThe problem is that the reported precision, recall, and F1 values are not well-defined. Section 3.2 defines P and R symbolically but never states the matching criterion for a predicted span: exact text match? token overlap? IoU threshold? The only explicit rule, IoU, is introduced for the auxiliary sentence_gap metric, not for the headline results. Given that annotators themselves often select different but valid evidence spans (Appendix B.2), this ambiguity is consequential. Without a matching rule, Figure 2 and Table 1 cannot be reproduced, compared across models, or even interpreted as real measurements. This is a load-bearing flaw, not a nitpick.\n\nTwo other soft spots matter, though less. Cohen's kappa for evidence is 0.30—fair at best—and the authors' defense (sparse sentences, legitimate variation) is plausible but does not remove the concern that gold evidence labels are noisy. And Table 1 reports each model's best performance across the three strategies; that post-hoc selection directly undercuts the abstract's claim that closed-source models “consistently outperform” open-source ones. In fact, LLaMA-70B's best claim F1 (0.58) beats GPT-4-Turbo's (0.56). The pairwise comparisons in Figure 2 are more honest, but the paper's own summary leans on the best-case numbers.\n\nNovelty is also overstated: the related work section discusses scientific fact-checking without citing SciFact, which is the obvious prior benchmark for claim verification (albeit against abstracts/snippets rather than full papers). This should be fixed.\n\nWho gets value from this? Researchers building scientific-knowledge benchmarks, and anyone studying long-context reasoning or prompt decomposition. It deserves a serious referee, but the revision must specify the span-matching rule, report variance or error bars, and soften the “consistently outperform” claim. I would not cite the numbers as-is.","headline":"A genuinely useful new resource with an unstated evaluation rule that makes the headline numbers uninterpretable until fixed.","tokens_in":15101,"tokens_out":2040,"would_cite":false,"duration_ms":25897,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLAIM-BENCH, a new benchmark of claim-evidence pairs from 100 AI papers, shows that LLMs validate scientific claims far better with iterative prompting, and that closed-source models currently lead.","keywords":["CLAIM-BENCH","claim-evidence reasoning","scientific comprehension","LLM benchmark","long-context reasoning","prompting strategies","inter-annotator agreement","sentence gap"],"falsifier":"Re-annotate the same 30-paper subset with stricter, boundary-resolved evidence definitions and a consensus adjudication step, then recompute all models' precision, recall, and F1; if the closed-source advantage and the iterative-prompting gains persist under cleaner labels, the annotation noise is not driving the conclusions. A cheaper check is to compute each model's F1 variance across random halves of the dual-annotated papers and test whether the reported ordering survives resampling.","tokens_in":14130,"feed_emoji":"⚗️","tokens_out":8296,"duration_ms":87385,"temperature":0.7,"pith_summary":"The paper introduces CLAIM-BENCH, a human-annotated benchmark built from 100 recent AI/ML papers and containing over 300 claim-evidence pairs, designed to test whether large language models can perform the global reasoning that scientific validation requires: finding the sentences that support or contradict a claim anywhere in a full paper. The authors find that every model falls well short of reliable performance, with best claim F1 scores around 0.56–0.59 and evidence F1 scores around 0.35–0.47. Closed-source models such as GPT-4-Turbo and Claude 3.5 balance precision and recall better than open-source models, and iterative prompting strategies—Three-Pass and One-by-One—recover dispersed evidence that Single-Pass prompting misses, at substantially higher compute cost. If the benchmark is sound, it gives the research community a diagnostic for scientific comprehension in LLMs and a concrete way to measure progress.","feed_headline":"Benchmark shows LLMs fall short on claim-evidence science reasoning","feed_subtitle":"Closed-source models lead, and iterative prompting recovers long-range evidence links—at a steep compute cost.","key_machinery":"The central object is CLAIM-BENCH itself: a ground-truth dataset in which four PhD annotators marked claims and linked supporting evidence spans in 100 recent AI/ML papers, yielding more than 300 claim-evidence pairs. The evaluation protocol around it uses precision, recall, F1, and a new sentence_gap metric—the average absolute sentence-level distance between a predicted claim and its matched evidence—which is meant to quantify how far a model must reason across a document. Three prompting strategies carry the experimental comparison: Single-Pass (one prompt for claims, evidence, and conclusions), Three-Pass (sequential prompts for claims, then evidence, then conclusions), and One-by-One (a separate evidence pass for each individual claim). The ground truth's reliability is reported through inter-annotator agreement, with Cohen's κ = 0.66 for claims and κ = 0.30 for evidence.","core_discovery":"On its own terms, the paper claims that LLMs' scientific claim-evidence reasoning is currently limited and uneven, and that the pattern of failure is diagnostic. Across six models, no configuration reaches an F1 of 0.7; claims are extracted with higher precision than evidence, while evidence extraction yields higher recall than precision. The paper further claims that closed-source models such as GPT-4-Turbo and Claude 3.5 outperform open-source models like LLaMA, Ministral, and Phi, and that splitting the task into sequential passes—first claims, then evidence, then conclusions—materially improves linking of dispersed evidence, especially for smaller models on long documents. A new metric, sentence_gap, which measures the average sentence distance between a predicted claim and its matched evidence, shows that iterative prompting produces longer-range links, which the authors read as broader coverage but also as increased noise.","pith_inferences":["The paper's own agreement numbers imply that reported model differences of about 0.1 F1 may sit within annotation noise: with Cohen's κ = 0.30 on evidence spans, a re-analysis using adjudicated evidence labels would show whether the model rankings are robust.","The sentence_gap metric is a natural general-purpose diagnostic for long-context reasoning in any document genre, since it measures the dependency distance a system must bridge rather than just end-task accuracy.","A testable extension of the prompting result is adaptive pass allocation—running expensive One-by-One passes only for claims whose evidence is predicted to be dispersed, which would trade compute against recall more cheaply than the paper's fixed strategies.","The benchmark's structure could be extended to cross-paper evidence graphs, linking a claim in one paper to evidence in another; the authors mention this direction in discussion but do not evaluate it."],"forward_implications":["Iterative prompting is a model-agnostic lever for long-context scientific reasoning: Three-Pass and One-by-One recover recall that Single-Pass loses on documents above 20k tokens, including for small open-source models.","For automated peer review or literature verification, closed-source models are currently the more balanced choice, but even their best F1 implies that extracted claim-evidence links require human oversight.","Because all models show higher recall than precision on evidence, downstream systems that consume extracted evidence should expect false positives and build in filtering.","The sentence_gap metric offers a quantitative handle on reasoning distance, so future systems can be designed to keep gaps short while maintaining recall, which would indicate more controlled evidence linking."],"supporting_citations":[{"why":"LongGenBench supplies the long-context generation benchmark whose length-related failures motivate the paper.","marker":"(Wu et al., 2025)"},{"why":"XL2Bench shows long-range dependency limits that CLAIM-BENCH's dispersed evidence is designed to test.","marker":"(Ni et al., 2024)"},{"why":"Chain of Evidence frames the multi-hop, structured reasoning needed to validate dispersed information.","marker":"(Chang et al., 2024)"},{"why":"The scientific fact-checking survey defines the localized validation task that the paper extends to full-paper reasoning.","marker":"(Vladika and Matthes, 2023)"},{"why":"ScienceAgentBench is the scientific reasoning benchmark used to position CLAIM-BENCH's distinct claim-evidence task.","marker":"(Chen et al., 2025)"},{"why":"SCBench is a long-context benchmark whose coverage of reasoning the paper distinguishes from claim-evidence validation.","marker":"(LI et al., 2025)"},{"why":"MMLongBench-Doc is a long-context document benchmark whose scope motivates the new task design.","marker":"(Ma et al., 2024)"}],"fun_headline_variants":["Benchmark exposes LLMs' weak link between science claims and evidence","LLMs can't reliably validate science claims, benchmark shows","Closed models lead, open ones lag in claim-evidence benchmark","Iterative prompting boosts LLM evidence linking at high compute cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is only as reliable as its human annotations, and annotators reached only fair agreement on which sentences count as evidence (Cohen's κ = 0.30), so every measured model score inherits that labelling noise.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark exposes LLMs' weak link between science claims and evidence","LLMs can't reliably validate science claims, benchmark shows","Closed models lead, open ones lag in claim-evidence benchmark","Iterative prompting boosts LLM evidence linking at high compute cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":2029,"prompt_tokens":962,"completion_tokens":1067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":996}},"tokens_in":578,"tokens_out":1067,"duration_ms":9942,"temperature":1.0,"reasoning_tokens":996,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:16:07.388447+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 30-paper subset with stricter, boundary-resolved evidence definitions and a consensus adjudication step, then recompute all models' precision, recall, and F1; if the closed-source advantage and the iterative-prompting gains persist under cleaner labels, the annotation noise is not driving the conclusions. A cheaper check is to compute each model's F1 variance across random halves of the dual-annotated papers and test whether the reported ordering survives resampling.","supporting_citations":[],"review_version":1}