{"id":"14d9e6ac-7819-49d1-af8e-3ff05bf10a81","arxiv_id":"2509.04304","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs more often reproduce older medical conclusions than updated ones, as shown by a new 512-question dataset of Cochrane reviews whose verdicts changed over time.","lead":"This paper builds two medical question-answering datasets from Cochrane systematic reviews and finds that large language models more often give the older, outdated answer when medical consensus has changed. The results suggest that LLMs can memorize stale medical guidance, which matters for any clinical or patient-facing use of medical AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 contradicts 'consistent reliance across all models': Llama (+7.4) and OLMo (+2.9) score better on latest labels; no significance tests or chance baselines support the claim.","rationale":"The reader's weakest assumption about clustering is important, but the paper's manual correction of all 512 MedChangeQA labels mitigates label noise; the grouping into same-question clusters remains opaque and would need transparency. However, the more immediate problem is that the headline is an empirical claim about the models, and the paper's own numbers contradict it. Two models show positive F1 differences; no statistics. If re-analysis shows these differences are within noise, the abstract's 'consistent reliance across all models' is an overstatement. The conditional verdict is appropriate: the dataset is potentially useful, but the outdated-knowledge conclusion requires the proposed robustness analysis before being taken at face value. Hence UNCHANGED relative to the reader's conditional verdict, with a partial agreement because our load-bearing concern differs from the reader's clustering concern.","tokens_in":14868,"tokens_out":5272,"duration_ms":46247,"concrete_test":"Run a non-parametric bootstrap (10,000 resamples) over the 512 MedChangeQA questions to compute 95% CIs for each model's outdated-vs-latest macro-F1 difference, and compare to a chance baseline (e.g., random labels and always-most-frequent-outdated-label). If the CIs for Llama and OLMo exclude zero in the positive direction, or if the intervals for the other models include zero, the 'consistent reliance across all models' claim is not supported. Also recompute the 32-40% 'outdated answers' rate against a prior-based baseline; report p-values or CIs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that all eight models 'consistently rely on outdated knowledge' is not supported by the paper's own Table 2. Using the paper's F1-difference proxy (outdated labels minus latest labels as ground truth), only five models show negative differences (GPT-4o -3.0, Qwen -4.8, DeepSeek -1.7, PMC-Llama -2.8, BioMistral -1.5); Mistral is essentially flat (-0.2); and Llama 3.3 (+7.4) and OLMo 2 (+2.9) perform better on the latest labels. Thus the 'across all models' and 'consistent reliance' statements are internally contradicted. No confidence intervals, bootstrap estimates, or significance tests are reported, so with n=512 the small negative differences could easily be sampling noise. The supporting 'F1 Outdated Answers' column (32-40%) is only slightly above the 33% random baseline for a three-class choice and is not tested against a majority-class or label-prior baseline, so it does not establish reliance. Even granting the MedChangeQA label construction and the same-question clustering, the reported numbers fail to support the abstract's strongest claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two QA datasets derived from Cochrane systematic reviews: MedRevQA (16,501 QA pairs) and MedChangeQA (512 QA pairs where the conclusion label changed across review versions). Eight LLMs are evaluated on three tasks: full MedRevQA, MedChangeQA with outdated labels as ground truth, and MedChangeQA with latest labels as ground truth. The difference in macro-F1 between the outdated- and latest-label conditions is used as a proxy for memorization of outdated medical knowledge. The paper claims that all eight models show 'consistent reliance on outdated knowledge,' and further analyzes pretraining corpora and a simple RAG mitigation.","tokens_in":15134,"tokens_out":2226,"duration_ms":20557,"significance":"The datasets are a potentially valuable resource for studying temporal decay of medical knowledge in LLMs, and the public release of code and data is a strength. The gold-label checking of all 512 MedChangeQA instances and the OLMo pretraining-corpus analysis are careful elements. However, the central claim—that all eight models consistently rely on outdated knowledge—is not supported by the paper's own Table 2, and the evaluation lacks significance testing. If the claim is revised and the analysis strengthened, the dataset contribution could be useful to the community.","major_comments":[{"comment":"The abstract states 'consistent reliance on outdated knowledge across all models,' but Table 2 shows that Llama 3.3 has an F1 difference of +7.4 (better on latest labels) and OLMo 2 has +2.9; Mistral is essentially flat at -0.2. Only five of the eight models show negative differences. The text in Section 5 acknowledges the positive differences for Llama and OLMo, yet the conclusion and abstract retain the unqualified 'all models' claim. This is a load-bearing inconsistency that must be resolved, either by softening the claim to a majority of models or by providing a statistical justification for treating the two positive cases as noise.","section":"Abstract and Section 5, Table 2"},{"comment":"No confidence intervals, bootstrap estimates, or significance tests are reported. With n=512, the differences of -1.5 to -4.8 for the five negative models could plausibly be sampling variation, and the positive differences for Llama and OLMo are likewise not assessed. The 'F1 Outdated Answers' column (32.0–40.6%) is near the 33% random baseline for a three-class choice, and no comparison to a majority-class or label-prior baseline is given. Without such baselines, the numbers do not establish that models are specifically choosing outdated labels rather than randomly guessing or following a prior.","section":"Section 5, Table 2"},{"comment":"The construction of MedChangeQA relies on grouping 4,379 SLRs into 1,535 clusters of 'the same research question,' but the grouping procedure is not described. If clusters differ in population, intervention, comparator, or outcome, a 'verdict change' may simply reflect a different review scope rather than a genuine reversal of medical consensus. The paper should specify how the grouping was performed (e.g., matching on PICO elements or title similarity) and provide evidence that the 512 changed-verdict instances indeed represent the same clinical question across versions.","section":"Section 3, Changed Knowledge"}],"minor_comments":[{"comment":"The caption refers to 'five LLMs' while the paper evaluates eight. Clarify which models are included in the figure and why, or update the caption to 'eight.'","section":"Figure 1 / Figure 4"},{"comment":"The phrase 'Our dateset' appears to be a typo for 'dataset.'","section":"Section 3, Dataset Construction"},{"comment":"The column heading 'F1 Outdated Answers' is ambiguous: it actually reports the percentage of answers (in the latest-label condition) that match the outdated label, not an F1 score. Consider renaming to 'Outdated-label match rate' for clarity.","section":"Section 5, Table 2"},{"comment":"The n-gram counts report 'mean and median amount of n-gram counts per year,' but the figure only shows one line. Specify whether it is mean or median, or plot both.","section":"Section 6, Inspection of OLMo"},{"comment":"In the example, Llama 3.3 is said to predict 'Refuted' when the latest label is 'Supported,' which is indeed outdated/incorrect. This is a good illustration, but the text labels it 'outdated and incorrect' while the table places it in the context of the outdated label 'Not Enough Information.' Clarify which prediction is being compared against which gold label.","section":"Appendix D, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's dataset contribution is potentially solid, and the gold-label subset is a meaningful resource. However, the headline claim is overstated relative to Table 2, and the lack of statistical testing is a serious gap. I would advise the editor that the paper requires a revised central claim and additional analysis before it can be considered for publication. The grouping method for MedChangeQA also needs to be made explicit; without it, the validity of the 'changed verdict' labels is not fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful contribution here is the dataset, not the headline result. MedRevQA (16.5k QA pairs from Cochrane systematic reviews) and MedChangeQA (512 manually verified verdict changes) are new, well-motivated, and likely to be reused. Releasing the code and data helps. The OLMo/Dolma n-gram analysis, showing older reviews are more prevalent in training, is a nice piece of evidence, and the qualitative examples of models citing decade-old Cochrane reviews are genuinely instructive.\n\nThe paper's core empirical claim, however, is overstated. The abstract says 'consistent reliance on outdated knowledge across all models,' but Table 2 shows two of eight models (Llama +7.4, OLMo +2.9) score higher against latest labels than outdated ones, and Mistral is essentially flat (-0.2). With n=512, no confidence intervals, and no significance tests, the negative differences for the other five are within sampling noise. The '32-40% outdated answers' column is only slightly above the 33% random baseline for three classes, and no majority-class or label-prior baseline is reported. So the evidence supports 'some models show a tendency,' not 'all models consistently rely.'\n\nThere is also an opaque construction step: 4,379 SLRs are grouped into 1,535 'same question' clusters with no stated criteria (PICO matching, title similarity, etc.). If those clusters mix different populations or interventions, some 'verdict changes' may just be scope changes, which would weaken the outdated-vs-latest distinction. This needs to be documented and ideally released.\n\nTo their credit, the authors do acknowledge the proxy is imperfect in the Limitations section, and the manual checking of all 512 changed-verdict labels is real work. The qualitative analysis is suggestive, not conclusive.\n\nBottom line: this deserves a serious referee and, after revision, likely acceptance. I'd ask for significance testing or bootstraps, a chance-level baseline, a transparent clustering procedure, and an abstract that matches the actual mixed pattern. The benchmark is worth keeping; the marketing claim is not.\n\nRecommendation: send to peer review.","headline":"Valuable benchmark, but the abstract's 'consistent reliance' claim doesn't survive Table 2.","tokens_in":15636,"tokens_out":1597,"would_cite":true,"duration_ms":14625,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All eight tested LLMs rely on outdated medical knowledge when answers change over time, the paper demonstrates with a new benchmark built from systematic reviews.","keywords":["large language models","medical question answering","outdated knowledge","memorization","systematic reviews","temporal knowledge change","Cochrane reviews","retrieval-augmented generation"],"falsifier":"Take a random sample of the 512 MedChangeQA pairs and have two independent medical reviewers check, for each pair, whether the older and newer reviews have the same population, intervention, comparator, and outcome. If more than a small fraction (say 10%) are judged to differ in scope, then the outdated-vs-latest label distinction—and the F1-difference proxy built on it—would be measuring review drift, not outdated memorization.","tokens_in":14753,"feed_emoji":"🩺","tokens_out":8285,"duration_ms":68918,"temperature":0.7,"pith_summary":"The paper claims that large language models, when asked directly about medical questions whose evidence base has shifted, fall back on the older, superseded conclusion. To test this, the authors built MedRevQA, a 16,501-question benchmark from medical systematic reviews, and MedChangeQA, a gold-labeled subset of 512 questions where the verdict of the underlying review changed over time. On MedChangeQA, all eight tested models—proprietary, open-weights, and domain-fine-tuned—scored at least as well when graded against outdated labels as against the latest labels, and 32–40% of their answers carried the outdated label. The paper reads this as consistent evidence that LLM medical knowledge lags current consensus, and traces part of the cause to the heavier presence of older reviews in pre-training data. The result matters because it converts a vague worry about stale medical AI into a measured, reproducible phenomenon.","feed_headline":"LLMs answer with outdated medical verdicts 32-40% of the time","feed_subtitle":"A benchmark of 512 changed medical verdicts shows every model lags current consensus.","key_machinery":"The evaluative core is MedChangeQA, a dataset of 512 question-verdict pairs where the verdict flipped between versions of a systematic review; the measurement is the F1-difference proxy, in which each model is scored against the outdated labels and then against the latest labels, and the gap (or the outdated-answer rate) is taken as the extent of outdated memorization. The explanatory core is the pre-training-data analysis: for the fully open model, n-gram counts against the open pre-training corpus show all 16,501 underlying reviews appear in training, with older reviews more frequent—offering a concrete mechanism for why older consensus wins.","core_discovery":"The paper's central discovery is that outdated medical knowledge is not a niche failure but a systematic property of current LLMs: for 512 questions where the underlying Cochrane systematic reviews changed their verdict between versions, every one of the eight models performed better when scored against the outdated labels than against the latest labels, and produced the outdated label in 32–40% of answers. The most common pattern is a shift from Not Enough Information to Supported/Refuted as new evidence accumulates, but the reverse also occurs—for example, probiotics for necrotising enterocolitis went from Supported in 2014 to Not Enough Information in 2023, and models cited the 2014 revie","pith_inferences":["If the same measurement were applied to other fast-moving consensus domains—clinical guidelines, drug safety warnings, nutrition or physics consensus—the same 'older-is-better' pattern should appear, because the mechanism (higher training frequency of older text) is general; this is an extrapolation the paper does not test.","The F1-difference proxy probably understates the real rate of outdated reliance: a model that is wrong for non-temporal reasons counts against the latest labels without being counted as an outdated answer, and the paper only manually audited a sample of explanations.","MedChangeQA's outdated/latest label pairs give knowledge-editing methods a concrete success criterion they currently lack: after an edit or unlearning step, the model should flip from the outdated label to the latest label on the same question."],"forward_implications":["Zero-shot medical QA from current LLMs will, on questions whose evidence has shifted, deliver the superseded verdict in roughly a third of cases; this is the direct clinical-safety consequence.","Domain-specific models (PMC-LLaMa, BioMistral) do not escape the problem; pre-training on biomedical papers actually increases the tendency to cite specific, often decade-old, studies.","Retrieval augmentation with a single related abstract helps but does not cure the problem (3–16 F1 points), so recency-aware retrieval and conflict-resolution are needed.","The result reframes LLM medical QA as a temporal-knowledge problem, not just a coverage or reasoning problem; knowledge editing, unlearning, and continual learning are the paper's suggested next steps."],"supporting_citations":[{"why":"Supplies the Cochrane systematic-review source and methods definition that the datasets are built from.","marker":"(Cumpston et al., 2022)"},{"why":"Provides the prior finding that a substantial share of Cochrane reviews change conclusions, motivating the change-detection subset.","marker":"(Hughes et al., 2012)"},{"why":"Defines the MedREQAL QA construction approach that MedRevQA extends and repurposes for temporal change.","marker":"(Vladika et al., 2024b)"},{"why":"Establishes the temporal-QA methodology on encyclopedic knowledge that this paper adapts to the medical domain.","marker":"(Vu et al., 2024)"},{"why":"Supplies the memorization framework used to explain why older, more frequent documents are more strongly encoded.","marker":"(Carlini et al., 2023)"},{"why":"Releases the fully open Dolma corpus, which lets the paper verify that the review texts appear in OLMo's training data.","marker":"(Soldaini et al., 2024)"},{"why":"Provides the n-gram search tool used to count review mentions in the corpus and trace outdated knowledge to training exposure.","marker":"(Liu et al., 2024)"},{"why":"Supports the background claim that LLMs encode clinical knowledge, the premise this paper complicates with temporal decay.","marker":"(Singhal et al., 2023)"}],"fun_headline_variants":["LLMs stuck in medical past: 32-40% outdated verdicts","Outdated medical verdicts in 32-40% of LLM answers","Medical AI is stuck in the past: 32-40% outdated calls","All tested LLMs lean on obsolete medical verdicts"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The measure of 'outdated knowledge' depends on treating different versions of a systematic review as asking the exact same question; if two versions actually differ in scope (e.g., different patients, treatments, or outcomes), a changed conclusion is not evidence that the model is behind the times.","fun_headline_variants_meta":{"raw":{"variants":["LLMs stuck in medical past: 32-40% outdated verdicts","Outdated medical verdicts in 32-40% of LLM answers","Medical AI is stuck in the past: 32-40% outdated calls","All tested LLMs lean on obsolete medical verdicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3737,"prompt_tokens":678,"completion_tokens":3059,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":2981}},"tokens_in":422,"tokens_out":3059,"duration_ms":21715,"temperature":1.0,"reasoning_tokens":2981,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:13:02.398240+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 512 MedChangeQA pairs and have two independent medical reviewers check, for each pair, whether the older and newer reviews have the same population, intervention, comparator, and outcome. If more than a small fraction (say 10%) are judged to differ in scope, then the outdated-vs-latest label distinction—and the F1-difference proxy built on it—would be measuring review drift, not outdated memorization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Cochrane systematic-review source and methods definition that the datasets are built from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prior finding that a substantial share of Cochrane reviews change conclusions, motivating the change-detection subset."}],"review_version":1}