{"id":"8559d051-3cf6-4dfb-ac21-a5c4cc1e36f7","arxiv_id":"2608.01409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On the five-source CARE-XAI benchmark, classifiers beat LLMs at verdict-only prediction, fine-tuning improves evidence-generating LLMs more than retrieval, and PubMed retrieval helps only source-aligned claims.","lead":"An evaluation of language models for biomedical fact-checking finds that standard classifiers still predict verdicts best, while fine-tuned LLMs produce the strongest claim-and-evidence outputs. The paper introduces a metric, Bio-GRACE, showing that PubMed retrieval helps scientific-abstract claims but often distracts on public-health claims, motivating selective retrieval.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bio-GRACE's UR may subtract probabilities from separately trained classifiers, not a fixed verifier, so the source-dependent 'distraction' finding could be a training-condition artifact and needs a same-verifier recomputation before it is load-bearing.","rationale":"The reader's weakest assumption points to reference-evidence validity, which the paper itself flags in Limitations and is a real concern. But the more load-bearing issue is that Bio-GRACE may not even measure what it claims: Eq. 4–6 require probabilities from a single verifier, while the reported classifier 'regimes' appear to be separately trained models. If so, subtracting p^C from p^R or p^G measures differences between classifiers, not the causal effect of adding evidence. This directly threatens the central claim that 'Bio-GRACE shows retrieval utility is source-dependent' and the 'distraction' interpretation. I am not recommending rejection because the paper has independent support: the label-only LLM retrieval deltas in Table 9 and the source-router simulation in Table 10 also suggest source-dependent retrieval effects, and the leakage-filtered analyses preserve the broad ordering. Those results could survive even if Bio-GRACE is flawed. However, the paper's main novel diagnostic must be re-run under a same-verifier protocol, with matched decoding for LLMs or input-dropout-trained classifiers, and the source-level UR values re-reported. If the pattern holds, the central claim is strengthened; if not, Bio-GRACE's headline result is an artifact and the contribution shifts to the weaker, but still useful, direct retrieval comparisons. Hence a CONDITIONAL verdict rather than ACCEPT or REJECT.","tokens_in":38236,"tokens_out":9072,"duration_ms":88576,"concrete_test":"Recompute source-stratified Bio-GRACE (Eq. 6) with a single fixed verifier: one instruction-tuned LLM prompted identically in claim-only, claim+retrieved, and claim+gold conditions (same weights/seed/decoding), or one classifier trained with input dropout so all three input types are in-distribution. Compare UR/CW-UR on PUBHEALTH and HealthFC to Tables 8/9. If the negative UR persists, retrieval distraction is confirmed; if it flips to non-negative or shrinks toward zero, Bio-GRACE is confounded by comparing different classifiers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central novelty is Bio-GRACE, defined in Eqs. 4–6 and Appendix F.1 as a counterfactual intervention 'on a verifier': p^C_i, p^R_i, p^G_i are supposed to be probabilities from the same verifier under different inputs. But the implementation in §4/§5 and Table 7 lists 'Classifier claim-only,' 'Classifier Retrieved PubMed,' and 'Classifier Gold evidence' as separate 9-run classifier families, each trained on a different input condition. If these are separate models, then U^R_i = p^R_i − p^C_i and U^*_i = p^G_i − p^C_i subtract probabilities across different models, conflating retrieval/gold evidence effects with training-condition, model-capacity, and seed differences. The large gaps in Table 7 (claim-only 0.547, retrieved 0.386, gold 0.692) are then partly artifacts of the training condition, not of the evidence itself. Consequently, negative UR for PUBHEALTH and HealthFC in Table 8 may reflect that the 'retrieved-condition' classifiers are weaker models, not that PubMed retrieval distracts a fixed verifier. The paper's Limitations statement that reference evidence 'may itself be incomplete or heterogeneous' also means source-level UR can be driven by reference-evidence quality; but the more immediate internal problem is that the same-verifier premise of Bio-GRACE is not evidently honored. Since the headline 'Bio-GRACE shows retrieval utility is source-dependent' rests on this metric, the diagnostic needs validation before the central claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates evidence-generating LLMs for biomedical claim verification on CARE-XAI, a unified benchmark of 17,803 examples from PubMedQA, SciFact, HealthVer, PUBHEALTH, and HealthFC. It compares base instruction LLMs, PubMed RAG LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared protocol with a frozen retrieval cache, explicit failure manifest, and leakage audit. The main empirical conclusions are: (i) biomedical classifiers remain strongest for verdict-only prediction; (ii) fine-tuned LLMs are the most reliable evidence generators; and (iii) PubMed retrieval helps on PubMed-aligned sources (PubMedQA, SciFact) but distracts on public-health sources (PUBHEALTH, HealthFC). The paper introduces Bio-GRACE, a gold-reference-normalized diagnostic intended to measure how much of the decision benefit of reference evidence is recovered by retrieved context, and uses it to argue that retrieval utility is source-dependent and that retrieval should be selective rather than always-on.","tokens_in":38645,"tokens_out":5925,"duration_ms":60378,"significance":"If the central claims hold, the paper makes a useful contribution to biomedical fact-checking evaluation. The controlled evaluation design is a genuine strength: a frozen retrieval cache decouples retrieval from answer generation, prompts and output contracts are shared across families, failed runs are retained in a manifest rather than discarded, and a leakage audit is reported. The finding that verdict-only biomedical classifiers outperform evidence-generating LLMs on label prediction is credible and important, as is the observation that retrieval metrics such as Recall@10 do not imply decision utility. Bio-GRACE is an interesting diagnostic idea, but its current implementation has a load-bearing internal inconsistency: the three probabilities it subtracts come from separately trained classifier families, not from a single verifier under different evidence conditions. The paper also lacks error bars and significance tests for its main LLM comparisons. These issues are fixable within the scope of the manuscript, but they must be addressed before the source-dependent retrieval-utility claim is accepted.","major_comments":[{"comment":"Bio-GRACE is defined as a counterfactual intervention on a verifier: p^C_i, p^R_i, and p^G_i should be probabilities from the same verifier under claim-only, retrieved, and gold-evidence inputs. The implementation, however, uses three separately trained 9-run classifier families, labeled 'Classifier claim-only,' 'Classifier Retrieved PubMed,' and 'Classifier Gold evidence' in Table 7. Each family is trained on a different input condition. Consequently, U^R_i = p^R_i − p^C_i and U*_i = p^G_i − p^C_i subtract probabilities across different models, conflating the evidence intervention with training-condition, model-capacity, and seed differences. The large gaps in Table 7 (0.547, 0.386, 0.692) are exactly what one would expect from separately trained models and do not establish what a fixed verifier would do. This is load-bearing because the headline 'retrieval utility is source-dependent'","section":"§5, Eqs. (4)–(6); §4/§5, Tables 7–8"},{"comment":"The main LLM comparisons are single runs: no seeds are reported for LLM rows, and the mean macro-F1 values in Table 6 are presented without confidence intervals or significance tests. The best fine-tuned LLM (0.483) and best base LLM (0.450) differ by 0.033, and the RAG-vs-base differences vary by prompt; without multiple seeds or paired bootstrap tests over examples, 'fine-tuning is the most reliable adaptation strategy' is not statistically established. The same issue applies to the source-level UR estimates in Table 9, where the effective sample sizes are only 90–105 unique rows per source. Please provide seed-level variation or bootstrap intervals, and paired significance tests for the matched prompt conditions, especially the zero-shot comparisons that the Limitations section identifies as the cleanest retrieval-effect estimate.","section":"§6, Tables 5–6, Figure 3; Table 9"},{"comment":"The leakage audit reports only regime-level macro-F1 sensitivity (Table 11), but the paper's central claim is that retrieval utility is source-dependent. Given that 588 test rows have exact evidence overlap and the stricter evidence-group-safe filter removes 591 rows, the source-level Bio-GRACE and NRI values could change substantially after filtering, particularly for PUBHEALTH, which dominates the test set. The sentence in §7 that 'retrieval remains source-dependent' is not supported by the table actually presented. Please report leakage-safe Bio-GRACE/NRI by source, or at least show that the source-level ordering in Table 9 is preserved on the evidence-group-safe subset.","section":"§7, Table 11; Appendix H"}],"minor_comments":[{"comment":"The source-router simulation is transparently labeled 'no-new-inference,' but because the router is constructed from the same test-set UR values it is meant to predict, the modest gains partly reflect the diagnostic signal by construction. The paper already cautions this; I would make that caution more prominent in the main text and avoid wording such as 'motivates selective biomedical RAG' being read as a validated routing result.","section":"§6.3, Table 10"},{"comment":"SciFact has Hit@10 = 0.000 but positive UR = 0.435. This is not contradictory, since retrieval can help without recovering source PMIDs, but it is surprising enough to deserve an explicit explanation in the text.","section":"Table 9"},{"comment":"The human-consensus denominators are small (21–29 per regime after exact agreement), so the regime percentages in Table 34 have wide uncertainties. Reporting bootstrapped intervals or including the raw counts more prominently would prevent over-reading.","section":"Appendix P, Table 34"},{"comment":"There are typographical issues in the references, e.g., 'V olume' in the ARES entry and 'F oundations' in Robertson and Zaragoza (2009). The spelling of 'synthesised' vs. 'synthesized' is inconsistent. These are minor.","section":"References and formatting"}],"recommendation":"major_revision","confidential_remarks":"The same-verifier issue is the main barrier to acceptance, but it is addressable: the authors can recompute Bio-GRACE with a fixed verifier or carefully re-frame the metric. The paper's evaluation infrastructure, retrieval cache, manifest policy, and leakage audit are strengths, and the verdict-only comparison is solid. I would not reject; I would require the Bio-GRACE recomputation and significance-aware reporting before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee, but the headline diagnostic has a load-bearing problem. Bio-GRACE is defined as a counterfactual intervention on a single verifier, yet the implementation trains separate classifier families on claim-only, retrieved, and gold-evidence inputs. The differences in Table 7, and the negative UR values in Table 8, can therefore reflect training-condition effects rather than the effect of evidence on a fixed verifier. The source-dependent story may survive a same-verifier redo, but as it stands the paper's central claim is not supported by the numbers as presented.\n\nThat said, the evaluation itself is well done. The frozen retrieval cache, shared prompts, leakage audits, and the manifest of failed runs are all thoughtful. Keeping incomplete and template-copy runs visible is the right way to evaluate generative models. The verdict that classifiers remain strongest for label prediction and that fine-tuning is more reliable than always-on RAG is consistent with the data and not the part I'd push back on. The source-stratified retrieval pattern, with PubMed-aligned sources benefiting and public-health sources not, is plausible and matches the low hit rates on PUBHEALTH and HealthFC.\n\nThe other soft spots are minor by comparison but worth naming. The LLM runs have no error bars or significance tests, so the fine-tuning-vs-RAG gap is reported without uncertainty. The routing simulation uses a test-set-derived source signal and is honestly labeled as a no-new-inference diagnostic, but the modest gains there should not be over-read. The human evaluation is small, deliberately enriched, and has low inter-annotator agreement, so the human-evidence-quality numbers can only be suggestive. The paper's own limitation that reference evidence may be incomplete is relevant: if gold evidence is weak on public-health sources, some of the negative UR could be an evidence-quality artifact. But the same-verifier issue is the immediate blocker.\n\nI'd send this to peer review because the benchmark, protocol, and evaluation design are solid enough to deserve referee time, and the source-dependent retrieval question matters. But the authors need to recompute Bio-GRACE with a shared verifier—either one model under different inputs or a matched parameter-sharing setup—before the headline claim can be accepted. As written, the diagnostic is the weakest link, not the empirical observations around it.","headline":"Careful evaluation and a transparent protocol, but Bio-GRACE's central finding rests on subtracting probabilities from separately trained classifiers, so the 'retrieval distracts' result needs a same-verifier recomputation before it can be trusted.","tokens_in":39064,"tokens_out":1608,"would_cite":false,"duration_ms":18751,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that PubMed retrieval helps verify claims aligned with scientific abstracts but actively distracts on broader public-health claims, and that its gold-reference-normalized diagnostic, Bio-GRACE, exposes when retrieval","keywords":["biomedical claim verification","evidence generation","retrieval-augmented generation","PubMed retrieval","Bio-GRACE","selective retrieval","CARE-XAI","fact-checking"],"falsifier":"On a stratified sample of PUBHEALTH rows with negative UR, have two experts judge whether each retrieved PubMed abstract actually addresses the claim; if the abstracts address the claims and UR remains negative, retrieval distraction is confirmed, while if the abstracts are off-topic the negative UR reflects source mismatch rather than retrieval-induced distraction.","tokens_in":38145,"feed_emoji":"🔎","tokens_out":8940,"duration_ms":74933,"temperature":0.7,"pith_summary":"The paper asks what it takes for a biomedical fact-checking system to do more than pick a verdict: it must also produce evidence that is faithful, complete, and useful. To answer, it builds CARE-XAI, a unified benchmark of 17,803 claims from five fact-checking sources, and compares base LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical classifiers under one protocol. The central result is that verdict-only classifiers are still the strongest label predictors, fine-tuned LLMs are the most reliable evidence generators, and PubMed retrieval is source-dependent: it recovers much of the reference-evidence benefit on PubMed-aligned sources but actively distracts on public-health and misinformation claims. If correct, this means always-on PubMed RAG is not a safe default and should be replaced by selective retrieval, and it motivates a new diagnostic, Bio-GRACE, that measures decision utility rather than topical relevance.","feed_headline":"PubMed RAG distracts on public-health claim verification","feed_subtitle":"Bio-GRACE shows retrieval recovers reference evidence on PubMedQA and SciFact but moves the wrong way on PUBHEALTH and HealthFC.","key_machinery":"Bio-GRACE (Biomedical Gold-Reference Assessment of Contextual Evidence) is the paper's central diagnostic. For each claim it records the verifier's probability of the true label under three inputs: claim-only $p^C_i(y_i)$, retrieved context $p^R_i(y_i)$, and gold reference evidence $p^G_i(y_i)$. Oracle utility is $U^\\star_i = p^G_i(y_i) - p^C_i(y_i)$; retrieved utility is $U^R_i = p^R_i(y_i) - p^C_i(y_i)$. On evidence-sensitive examples $H = \\{i : U^\\star_i > 0\\}$, utility recovery is $UR = \\frac{1}{|H|}\\sum_{i \\in H}\\mathrm{clip}\\!\\left(\\frac{U^R_i}{U^\\star_i + \\epsilon}; -1, 1\\right)$. Positive UR means retrieval recovers some of the reference-evidence benefit; negative UR means retrieval","core_discovery":"On CARE-XAI, reference/gold evidence is decision-useful: supplying it lifts classifier macro-F1 from 0.547 (claim-only) to 0.692, so the task is evidence-sensitive. Retrieved PubMed context does not supply that benefit: it leaves classifier macro-F1 at 0.386. Bio-GRACE makes the pattern explicit by comparing, on evidence-sensitive examples, how much retrieved context raises the verifier's true-label probability relative to gold evidence. The score is strongly positive for PubMedQA (UR=0.676) and SciFact (0.435), near zero for HealthVer, and negative for HealthFC (-0.222) and PUBHEALTH (-0.378); all three encoder verifiers have negative aggregate UR. The paper concludes that retrieval utility","pith_inferences":["Editorial inference: the same source-gating logic could be applied per claim rather than per source, using retriever confidence or verifier uncertainty as a gate; the paper's no-new-inference router already shows headroom over always-retrieval.","Editorial inference: adding a non-PubMed retriever (news, guidelines, policy documents) for public-health sources is the most direct testable extension; if it turns PUBHEALTH and HealthFC Bio-GRACE UR positive, the paper's source-mismatch reading is confirmed.","Editorial inference: because Bio-GRACE uses supervised verifier probabilities, its sign should be checked under a calibrated verifier; if negative UR flips sign with calibration, part of the distraction effect is attributable to miscalibrated confidence rather than retrieval content.","Editorial inference: the low inter-annotator agreement in the paper's 100-output human evaluation suggests that evidence-faithfulness standards vary even among experts; a practical extension would be adjudicated multi-rater evidence labels that can serve as a higher-confidence test bed for Bio-GRACE."],"forward_implications":["Deployers of biomedical fact-checking should treat always-on PubMed RAG as a liability on public-health and misinformation claims; the paper's source-router diagnostic raises label-only LLM macro-F1 from 0.343 to 0.381 by disabling retrieval on negative-utility sources.","Retrieval evaluation should include a decision-utility metric like Bio-GRACE alongside recall and nDCG, because the paper shows high query success (0.871) with low source-PMID recall (0.224 at 10) can coexist with negative utility.","Fine-tuning on the target evidence format is a more reliable adaptation than retrieval for evidence-generating LLMs: mean macro-F1 across complete fine-tuned runs is 0.447 versus 0.385 for RAG runs.","Reference evidence is decision-useful (classifier macro-F1 rises from 0.547 claim-only to 0.692 with gold evidence), so the observed RAG failures are not evidence-insensitivity in the verifier but a supply-side source-matching problem.","The UNADDRESSED verdict class is a distinct challenge: systems that retrieve topically related abstracts must avoid over-predicting SUPPORTED, and Bio-GRACE's evidence-sensitive set makes this explicit."],"supporting_citations":[{"why":"Supplies the CARE-XAI benchmark with its five-source composition, splits, and reference evidence used in every comparison.","marker":"Singh, Prabhjot, 2026"},{"why":"PubMedQA is one of the two PubMed-aligned sources where Bio-GRACE utility recovery is strongly positive.","marker":"Jin et al., 2019"},{"why":"SciFact is the other PubMed-aligned source in the benchmark and anchors the positive-utility side of the retrieval result.","marker":"Wadden et al., 2020"},{"why":"PUBHEALTH is the dominant public-health source and the main contributor to negative Bio-GRACE utility.","marker":"Kotonya and Toni, 2020"},{"why":"HealthVer provides the health-claim verification examples that sit between abstract-aligned and public-health retrieval needs.","marker":"Sarrouti et al., 2021"},{"why":"HealthFC supplies the smaller public-health fact-checking source with negative retrieval utility.","marker":"Vladika et al., 2024"},{"why":"Motivates the PubMed-oriented query rewriting and evidence-retrieval pipeline used for the RAG systems.","marker":"Deka et al., 2022b"},{"why":"BGE-M3 reranker selects the top contexts inserted into RAG prompts and populates the frozen retrieval cache.","marker":"Chen et al., 2024"},{"why":"Prior evidence that irrelevant context can degrade generation, which the paper extends by measuring decision utility rather than topical relevance.","marker":"Yoran et al., 2024"}],"fun_headline_variants":["PubMed retrieval distracts on public-health claims","Bio-GRACE: a diagnostic for source-dependent retrieval utility","When RAG hurts: retrieval fails on broad health claims","Retrieval helps some biomed claims, distracts on others","Gold evidence outperforms PubMed retrieval in verification"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that CARE-XAI's reference evidence is a valid gold standard and that the decision benefit of that evidence is faithfully captured by the change in a supervised classifier's probability of the true label; the paper itself notes the reference evidence may be incomplete or heterogeneous.","fun_headline_variants_meta":{"raw":{"variants":["PubMed retrieval distracts on public-health claims","Bio-GRACE: a diagnostic for source-dependent retrieval utility","When RAG hurts: retrieval fails on broad health claims","Retrieval helps some biomed claims, distracts on others","Gold evidence outperforms PubMed retrieval in verification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1451,"prompt_tokens":744,"completion_tokens":707,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":488,"tokens_out":707,"duration_ms":6764,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:11:57.034610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a stratified sample of PUBHEALTH rows with negative UR, have two experts judge whether each retrieved PubMed abstract actually addresses the claim; if the abstracts address the claims and UR remains negative, retrieval distraction is confirmed, while if the abstracts are off-topic the negative UR reflects source mismatch rather than retrieval-induced distraction.","supporting_citations":[],"review_version":1}