{"id":"aac6139d-fd6a-46bb-842a-a4a3cfb67cc5","arxiv_id":"2505.07968","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On a synthetic benchmark of 4,290 clinical scenarios, seven LLMs often endorsed outdated advice and contradicted themselves, and combining retrieval-augmented generation with preference tuning reduced both failures.","lead":"This paper builds a synthetic benchmark that flips current medical recommendations into plausible outdated versions and finds that seven large language models often fail to reject the outdated ones. It then shows that combining retrieval-augmented generation with preference fine-tuning reduces these failures more than either method alone.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-outdated foils are the load-bearing risk: without validation against real guideline reversals, the temporal-reliability and RoD mitigation claims do not generalize.","rationale":"The reader's conditional verdict already identifies pseudo-outdated validity as the weakest assumption; this stress-test agrees and sharpens the failure mode. The synthetic foils mix two distinct constructs: genuinely outdated historical advice and merely modified current wording, with the Recommendation Intensity category being the clearest case. Because 27.2% of the benchmark's items are intensity variants that Appendix A.5 admits are not logical contradictions, the measured 'rejection failure' is partly an artifact of scoring. This is not a dispute with clinical consensus; it is an internal-validity risk in how the ground-truth labels were generated. The concrete test with historical reversals would settle it. Independent support: the baseline evaluation is straightforward, the mitigation table is internally consistent, and the dataset is released, so the paper has value as a benchmark. Still, the central 'temporal reliability' claim and the RoD generality claim should remain conditional on real-guideline validation and held-out knowledge, matching the reader's CONDITIONAL verdict.","tokens_in":19373,"tokens_out":6742,"duration_ms":71584,"concrete_test":"Build a validation set of 30–50 date-stamped real guideline reversals (e.g., NICE-SUGAR glucose targets, ADA HbA1c thresholds, HCV ribavirin recommendations). Convert the old versions into the same scenario template with a neutral factor, run the seven base models, and compare ECDArej and IKCR against ConflictMedQA. If per-model ECDArej shifts by more than ~0.1 or model rankings change, the pseudo-outdated foils are not representative and the temporal-reliability conclusion fails to generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LLMs are temporally unreliable because they fail to reject outdated recommendations, and that retrieval-plus-DPO mitigates this—rests on the pseudo-outdated advice constructed in Section 3.1. This construction has not been validated as a proxy for real guideline reversals. The five alteration strategies are editorial transformations of current recommendations; no clinician or historical source confirms any pseudo-outdated item was ever actually recommended. The largest category, Recommendation Intensity (N=53, 27.2%), includes pairs such as 'should recommend' versus 'may consider'; Appendix A.5 itself concedes these are 'not strictly contradictory in formal logic.' Scoring endorsement of the weaker wording as a temporal-reliability error conflates recommendation strength with outdatedness, inflating ECDArej and the headline asymmetry. The distribution of categories is also not shown to match real reversals such as NICE-SUGAR glucose targets or HCV regimen changes. If these foils are unrepresentative, the study demonstrates only that models struggle to reject mutated current advice, not that they are temporally unreliable. The mitigation claim is additionally weakened because DPO training and the RAG knowledge base are built from the same 195 items used to generate all test scenarios, so RoD gains may reflect memorization and exact-match retrieval rather than adaptation to genuinely new guidelines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ConflictMedQA (DriftMedQA in the abstract), a benchmark of 195 synthetic up-to-date/pseudo-outdated clinical advice pairs and 4,290 scenario-based QA items, and evaluates seven LLMs on three ECDA metrics (endorsement, rejection, overall alignment) and the IKCR internal-conflict metric. It reports that models are much better at endorsing current guidelines than rejecting pseudo-outdated ones, that larger models do not consistently reduce internal conflicts, and that a combination of retrieval-augmented generation and DPO-based preference tuning (RoD) yields the largest improvements in alignment and consistency. The dataset is publicly released, and the metric definitions are explicit.","tokens_in":19623,"tokens_out":3158,"duration_ms":33542,"significance":"If the benchmark validly simulates clinical guideline evolution, the paper would make a useful contribution: it quantifies a temporal-reliability failure mode that standard static medical QA benchmarks miss, and it provides a principled comparison of RAG, DPO, and their combination. The work is significant in highlighting the asymmetry between endorsing current advice and rejecting outdated advice, and in showing that scale does not monotonically improve rejection behavior. The dataset release, the clear metric definitions, and the inclusion of both open-weight and proprietary models are strengths. However, the central empirical claims rest entirely on the validity of the manually constructed 'pseudo-outdated' foils, which are not validated against any real guideline reversal or clinician judgment, and the mitigation evaluation is largely in-distribution with the benchmark construction. These issues make the headline reliability and mitigation conclusions larger than the current evidence supports.","major_comments":[{"comment":"The pseudo-outdated recommendations are the load-bearing ground truth for the ECDA_rej metric (Eq. 2), but no clinician or historical guideline source validates that any of these pseudo-outdated items was ever an actual prior recommendation. The largest category, Recommendation Intensity (N=53, 27.2%), is explicitly conceded in Appendix A.5 to be 'not strictly contradictory in formal logic'; scoring models as temporally unreliable when they endorse 'may consider' over 'should recommend' conflates recommendation strength with outdatedness and may inflate the headline ECDA_rej asymmetry. The authors should either validate the pseudo-outdated set against documented guideline reversals (e.g., NICE-SUGAR glucose targets, HCV regimen changes, hormone replacement therapy), obtain clinician/expert annotations that each pseudo-outdated item is a plausible historical recommendation, or report ECDA_rej separately for each alteration category and show that the asymmetry persists when Recommendation Intensity items are excluded.","section":"§3.1 and Appendix A.5"},{"comment":"The mitigation evidence is substantially circular. DPO preference pairs are generated by inserting the same 195 up-to-date/pseudo-outdated advice pairs into a template (§4.2: 'directly inserted the original advice into a template'), and the RAG knowledge base is the same 195-item set (§4.3), while all 4,290 evaluation scenarios are paraphrases generated from those same 195 pairs (§3.1). Consequently, the reported gains for DPO, RAG, and RoD may reflect memorization of the training pairs and exact-match retrieval rather than adaptation to genuinely new guideline content. The evaluation should be repeated on a held-out set derived from real historical guideline changes that were not in the training KB or in the DPO preference data, and the RAG KB should be independent of the scenario-generation source.","section":"§4.2, §4.3, Eq. (5), Eq. (7)"},{"comment":"The numeric presentation of Table 1 is unreliable and undermines the quantitative mitigation claims. Cells such as '9198(+07)' for Qwen2.5-72B ECDA_adh and '66 96 (+30)' for Llama-3.3-70B are not valid decimal values, making it impossible to verify the reported absolute improvements. Additionally, the claim that RoD 'consistently yielded the highest ECDAall scores' is presented without confidence intervals or significance tests, even though the improvements over the best single method are often small (e.g., Qwen2.5-7B RoD vs. RAG: 0.807 vs. 0.720 in Table 5, a gap that should be tested for statistical significance). The table should be corrected and the aggregate comparisons should include uncertainty estimates.","section":"Table 1 and §5.2"},{"comment":"The 'authority signal hypothesis' introduced in Appendix A.7 to explain scale-dependent rejection failures is a post hoc, untested mechanism. For example, the observation that Qwen2.5-72B has lower ECDA_rej than Qwen2.5-7B could equally be explained by differences in RLHF tuning, instruction-following behavior, or decoding settings rather than by scale-dependent authority bias. The manuscript should either present a controlled test of this hypothesis or clearly label it as speculation rather than a conclusion drawn from the evaluation.","section":"§5.1 and Appendix A.7"}],"minor_comments":[{"comment":"The abstract refers to the benchmark as 'DriftMedQA' while the body and contributions section use 'ConflictMedQA'; the naming should be made consistent throughout.","section":"Abstract vs. Section 1"},{"comment":"The model name is written inconsistently as 'Ministral-8B' in most places but 'Mistral-8B' in Table 3 and 'Ministral-8B-Instruct' in Appendix B; please standardize to the official model name.","section":"Throughout"},{"comment":"The sentence 'The training continued until the model achieved 100% accuracy on the pseudo-outdated versus up-to-date advice pairs' is an unusual stopping criterion and likely induces overfitting; the authors should report the number of epochs and whether early stopping or validation was used.","section":"§4.2"},{"comment":"Only the dataset link is provided; no code or evaluation harness is released, which limits reproducibility of the RAG and DPO pipelines and the metric computations.","section":"Availability"},{"comment":"The radar plots in Figure 2 lack axis labels and a clear legend tying each color and marker to a model; readers cannot unambiguously reconstruct the per-category ECDA values from the figure.","section":"Figure 2"},{"comment":"Table 3 reports LoRA rank ablation but only for 'Mistral-8B', and the text says 'Rank 16 achieves the best balance' while the table shows monotonically improving scores; please clarify whether rank 8 was used in the main experiments and why the main table reports different values than the ablation table.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a careful, useful benchmark-and-mitigation paper, but its headline claim — that LLMs are temporally unreliable in medicine — rests on invented 'pseudo-outdated' foils, and the mitigation results are partly in-distribution. Worth refereeing; not worth taking at face value.\n\nWhat is actually new: ConflictMedQA pairs 195 current clinical recommendations with manually mutated 'pseudo-outdated' versions across five change categories, wraps them in 4,290 contextualized scenarios, and scores seven models with two purpose-built metrics (ECDA, IKCR). That is a real addition to the static medical QA literature. The metrics are well-defined, and the internal comparisons in Table 1 are coherent: RoD consistently beats RAG or DPO alone on small open models, and the ablations (LoRA rank, factor-wise breakdowns) give useful details. The limitations section is honest about synthetic scenarios, which I respect.\n\nThe soft spot is exactly where the stress test points. The pseudo-outdated advice is constructed by editorial transformation; no clinician or historical source confirms any of it was ever genuinely recommended. That matters most for the largest category, Recommendation Intensity (27.2%), for which Appendix A.5 itself concedes the pairs are 'not strictly contradictory in formal logic.' Scoring a model that endorses 'may consider' instead of 'should recommend' as a temporal-reliability error conflates wording strength with outdatedness, and it inflates the ECDA_rej asymmetry that drives the paper's main claim. Without validation against real guideline reversals (e.g., NICE-SUGAR glucose targets, HCV regimens), the study shows models struggle to reject mutated current advice — which is real but not the same as being temporally unreliable.\n\nThe mitigation section has a milder version of the same problem. DPO trains on preference pairs derived from the same 195 advice items that generate all test scenarios, and RAG retrieves from that same 195-item KB. The RoD gains could reflect memorization plus exact-match retrieval rather than adaptation to genuinely new guidelines. The paper does not claim otherwise, but the framing in the abstract and discussion goes beyond what the evidence supports.\n\nOne more minor thing: the abstract (and the Hugging Face link) says DriftMedQA while the body says ConflictMedQA. That sort of mismatch costs credibility even when the science is fine.\n\nWho benefits: clinical NLP groups that want a concrete stress test for knowledge recency and a baseline recipe (RAG + DPO) for small models. It is a solid starting point, and I would send it to peer review — but with a request either to validate the foils against real guideline reversals or to reframe the claims as synthetic stress-test results rather than temporal reliability in the wild.","headline":"Useful benchmark and mitigation study whose headline temporal-reliability claim is undercut by invented pseudo-outdated foils and in-distribution mitigation evaluation.","tokens_in":20168,"tokens_out":2422,"would_cite":false,"duration_ms":23900,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models are not temporally reliable in medicine: they systematically fail to reject outdated clinical recommendations, while a benchmark pairing current and pseudo-outdated guidelines shows that retrieval-augmented…","keywords":["medical knowledge drift","clinical guidelines","knowledge conflicts","retrieval-augmented generation","direct preference optimization","temporal reliability","ConflictMedQA","LLM evaluation"],"falsifier":"Construct a test set from genuine superseded guideline statements before documented reversals—for example, intensive glucose-control targets before the NICE-SUGAR trial or hormone-replacement-therapy recommendations before the early-2000s safety findings—and run the same seven models through the same endorsement/rejection protocol. If rejection rates on these real outdated statements differ substantially from rejection rates on the pseudo-outdated items, or if models that scored well on synthetic items fail on real ones, the benchmark's central claim about how LLMs handle medical knowledge drift is not supported.","tokens_in":19166,"feed_emoji":"🩺","tokens_out":7558,"duration_ms":65679,"temperature":0.7,"pith_summary":"Large language models are not temporally reliable in medicine. Across 4,290 scenario-based questions built from 195 pairs of current and \"pseudo-outdated\" clinical recommendations, seven evaluated models were often good at endorsing up-to-date advice but systematically bad at rejecting outdated advice, and sometimes endorsed mutually exclusive recommendations at the same time. The paper introduces ConflictMedQA to expose these failures and tests two mitigations: retrieval-augmented generation, direct preference optimization, and their combination. The combination consistently gave the most reliable alignment with current guidelines and the fewest internal contradictions. If the finding holds, clinical deployment of LLMs will need retrieval plus preference adaptation rather than scale alone.","feed_headline":"LLMs endorse outdated medical advice, but RAG plus DPO fixes it","feed_subtitle":"New benchmark pairs current and pseudo-outdated guidelines; retrieval plus preference tuning restores consistency.","key_machinery":"The load-bearing instrument is ConflictMedQA, a benchmark built from 195 clinical recommendation pairs in infectious and chronic disease. For each current recommendation, the authors hand-construct a mutually exclusive \"pseudo-outdated\" version using five strategies that mirror how guidelines actually change: clinical context, diagnostic thresholds, implementation approach, recommendation intensity, and treatment modality. Each pair is then turned into 4,290 scenario-based QA prompts through eleven cognitive and social conditioning factors. The benchmark's metrics separate external drift alignment (endorsing current advice versus rejecting outdated advice, combined into ECDA_all) from internal consistency (IKCR, the fraction of active pairs where a model endorses both versions). On the mitigation side, the machinery is a hybrid \"RAG on DPO\" pipeline: DPO with LoRA teaches the model to prefer current over outdated advice, and RAG supplies retrieved guideline snippets at inference time, which the paper shows activates the learned preference knowledge.","core_discovery":"The central discovery is that current LLMs are temporally unreliable in clinical knowledge: across 4,290 scenario-based QA pairs derived from 195 current-versus-pseudo-outdated recommendation pairs, the seven evaluated models systematically endorsed up-to-date guidelines while failing to reject outdated ones, and all models sometimes endorsed both members of a mutually exclusive pair simultaneously. Scale did not cure the problem: larger models did not consistently have lower internal conflict rates. The paper also finds that retrieval-augmented generation and direct preference optimization each improve alignment, but RAG alone can degrade smaller models' ability to reject outdated advice, while applying RAG to a DPO-tuned model consistently yields the best alignment and the lowest conflict rates.","pith_inferences":["One extension the authors leave implicit is that clinical safety metrics should weight rejection errors more heavily than endorsement errors, since acting on an outdated recommendation is the more dangerous failure; a cost-sensitive variant of ECDA_all would make this explicit.","The benchmark methodology is transferable to other rapidly changing knowledge domains, such as drug interaction databases, legal regulations, or cybersecurity advisories, where the same current-versus-superseded pair construction could test temporal reliability.","The observation that larger models show no consistent reduction in IKCR suggests that internal conflict rates may be driven by training-data coverage across time rather than raw capability; a testable prediction is that models trained on timestamped or temporally filtered corpora will show lower IKCR at fixed scale.","Because RAG alone sometimes increased internal conflicts in smaller models, a further testable extension would be retrieval that explicitly flags a retrieved snippet's temporal status rather than returning top-k nearest chunks."],"forward_implications":["Current LLMs cannot be assumed safe for answering guideline-related clinical questions, because endorsing a withdrawn recommendation is a recurring failure mode across all five change types.","Static knowledge benchmarks overstate clinical readiness: they measure adherence to current facts but not the rejection of superseded ones.","RAG alone is not a universal fix; for smaller models it can lower rejection accuracy, so grounding must be paired with the model's ability to weigh retrieved evidence.","Preference fine-tuning internalizes guideline updates, but knowledge learned from simple advice pairs does not automatically transfer to complex, context-rich scenarios.","A hybrid pipeline (RAG on DPO) gives the largest gains, consistently exceeding the additive contributions of either method alone."],"supporting_citations":[{"why":"Supplies the premise that clinical guidelines become outdated within a few years, motivating the drift benchmark.","marker":"Shekelle et al., 2001"},{"why":"Provides the NICE-SUGAR glucose-control reversal used as the paradigmatic guideline reversal that LLMs must handle.","marker":"Investigators, 2009"},{"why":"Establishes that training on conflicting evidence creates knowledge conflicts that models struggle to resolve.","marker":"Chen et al., 2022"},{"why":"Documents how LLMs behave under knowledge conflicts, framing the internal-conflict measurement.","marker":"Xie et al., 2024"},{"why":"Supplies the retrieval-augmented generation method used as the non-parametric mitigation.","marker":"Lewis et al., 2020"},{"why":"Supplies direct preference optimization, the parametric mitigation the paper combines with RAG.","marker":"Rafailov et al., 2023"},{"why":"Provides the Sentence-BERT encoder used to retrieve guideline snippets.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides the LoRA parameter-efficient fine-tuning used for DPO training.","marker":"Hu et al., 2022"}],"fun_headline_variants":["LLMs can't reject outdated medical advice; RAG plus DPO restores reliability","New DriftMedQA shows LLM conflicts on guideline drift; combined mitigation wins","LLMs often endorse conflicting medical guidance; RAG+DPO ensures consistent care","Medical knowledge drift trips up LLMs; retrieval plus preference tuning fixes it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusions about temporal reliability rest on the assumption that the manually created \"pseudo-outdated\" recommendations, made by altering current guidelines in five specified ways, are faithful stand-ins for real historical guideline reversals; if real reversals differ in wording, context, or subtlety, the measured rejection failures and conflict rates may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't reject outdated medical advice; RAG plus DPO restores reliability","New DriftMedQA shows LLM conflicts on guideline drift; combined mitigation wins","LLMs often endorse conflicting medical guidance; RAG+DPO ensures consistent care","Medical knowledge drift trips up LLMs; retrieval plus preference tuning fixes it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1405,"prompt_tokens":851,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":467,"tokens_out":554,"duration_ms":5146,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:06:59.732283+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set from genuine superseded guideline statements before documented reversals—for example, intensive glucose-control targets before the NICE-SUGAR trial or hormone-replacement-therapy recommendations before the early-2000s safety findings—and run the same seven models through the same endorsement/rejection protocol. If rejection rates on these real outdated statements differ substantially from rejection rates on the pseudo-outdated items, or if models that scored well on synthetic items fail on real ones, the benchmark's central claim about how LLMs handle medical knowledge drift is not supported.","supporting_citations":[],"review_version":1}