{"id":"2cad8f8d-1808-4592-939f-968b1f831a28","arxiv_id":"2508.00923","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A dynamic red-teaming audit reports that 94% of MedQA-correct answers fail under adversarial mutation, with 86% privacy leak rates, 81% bias shift rates, and 66-74% hallucination rates across 15 medical LLMs.","lead":"This paper introduces DAS, an automated agent-based system that generates adversarial medical questions and attacks to stress-test large language models across robustness, privacy, bias, and hallucination. Applied to 15 commercial and open models, it reports a wide gap between high static benchmark scores and low reliability under these dynamic attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract asserts a HealthBench generalization that the paper's own limitations disavow; the central benchmarking-gap claim is broader than the reported evidence.","rationale":"The reader's verdict of REJECT is well supported by multiple measurement and reporting flaws. I agree with the overall rejection, but I identify a different weakest link than the reader's choice of bias measurement. The bias concern, while valid for the identity/linguistic/emotional subtests, is less load-bearing for the headline claims because the abstract's 81% fairness figure refers specifically to cognitive-bias priming, which is a direct injection of an irrelevant authority/consensus cue and does not require a neutral-perturbation control to interpret. In contrast, the abstract's HealthBench generalization is a concrete, checkable factual claim: the abstract asserts that the 94% brittleness generalizes to open-ended HealthBench with >70% failure rates, yet the body contains no such experiment and explicitly limits the robustness audit to MedQA multiple-choice items. This is not merely a missing optional extension; it is a contradiction between the central summary and the paper's own methods and limitations. If the HealthBench result cannot be produced or reproduced, the abstract's claim that static scores reflect 'superficial memorization' is overstated, because the generalization to realistic open-ended evaluation—the strongest form of the benchmarking gap—is unsupported. The robustness result on MedQA remains directionally credible and the paper has merit as a red-teaming framework, but the abstract as written overstates the evidence. Therefore the reader's REJECT verdict stands, and the concrete test above would settle whether the concern actually lands.","tokens_in":46406,"tokens_out":13540,"duration_ms":160035,"concrete_test":"Search the paper, its supplementary materials, and the GitHub repository for any DAS robustness evaluation on HealthBench. If the claimed >70% failure rates on HealthBench are absent or cannot be reproduced from released logs, the abstract's generalization claim is unsupported and must be removed or substantiated. If logs exist, rerun the DAS robustness orchestrator on the HealthBench set and compare failure rates to the abstract's claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The arXiv abstract states: 'This brittleness generalized to the realistic, open-ended HealthBench dataset, where top-tier models exhibited failure rates exceeding 70% and sharp shifts in model rankings across evaluations.' No HealthBench-based robustness evaluation appears in the main text, Methods, or appendices; the only HealthBench usage is as a source of negative hallucination samples (§4.4.1). This is directly contradicted by §3 Limitations: 'our robustness audit is built upon the multiple-choice MedQA dataset, not open-ended formats.' The central claim that the MedQA robustness gap 'generalizes' to open-ended realistic clinical data is therefore asserted without supporting evidence and is inconsistent with the paper's own stated scope. Since the abstract uses this alleged generalization to argue that high static scores reflect 'superficial memorization,' the headline conclusion is overstated as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces DAS, an agentic red-teaming pipeline that dynamically mutates medical prompts to stress-test 15-16 large language models across four safety axes: robustness (MedQA-based multiple-choice mutations), privacy (81 HIPAA/GDPR trap scenarios), bias/fairness (415 clinical vignettes with identity, linguistic, emotional, and cognitive manipulations), and hallucination (a 260-item curated set scored by a seven-sub-agent detector). The authors report a median 94% robustness jailbreak rate on initially correct MedQA answers, 86-91% privacy leakage rates, an 81.1% cognitive-bias-induced answer-shift rate, and hallucination rates of roughly 62-73%, and they argue that these results expose a 'benchmarking gap' between high static benchmark scores and low dynamic reliability. The paper further claims in the abstract that the robustness brittleness generalizes to the open-ended HealthBench dataset, and it positions DAS as a living, scalable audit framework for regulators and hospitals.","tokens_in":46506,"tokens_out":12155,"duration_ms":127368,"significance":"The DAS framework addresses a genuine gap: static medical benchmarks are gameable and quickly become obsolete, and most existing medical red-teaming covers only a single failure mode. The paper's concrete strengths include a closed-loop, dynamic orchestration with a constrained action space; per-tool robustness ablations that let the reader see which perturbations drive the headline numbers; a four-axis unified audit pipeline; explicit cost accounting (a full audit at roughly $35-40); a public code release; and small clinician-validation exercises for the privacy judge and hallucination detector. The seven-category hallucination taxonomy and the adversarial privacy scenarios are also useful community assets. If the headline robustness result survives the interpretational caveats below, it is a policy-relevant challenge to the common practice of citing static USMLE-style accuracy as evidence of clinical readiness. However, the significance of the specific quantitative claims is currently muted by the abstract's unsupported HealthBench generalization and by the bias metric's lack of a neutral control; both issues are fixable and both affect the headline numbers.","major_comments":[{"comment":"The abstract's claim that the MedQA robustness gap 'generalized to the realistic, open-ended HealthBench dataset, where top-tier models exhibited failure rates exceeding 70% and sharp shifts in model rankings across evaluations' is not supported anywhere in the manuscript. The only appearance of HealthBench is as a source of negative hallucination samples (§4.4.1), and the Limitations section explicitly states that 'our robustness audit is built upon the multiple-choice MedQA dataset, not open-ended formats.' Since the generalization sentence is used to motivate the conclusion that high static scores 'may reflect superficial memorization,' the headline claim is internally inconsistent with the reported evidence; the authors must either add the HealthBench robustness evaluation, including the ranking-shift analysis, or delete the sentence and qualify the generalization.","section":"Abstract; §3 Limitations; §4.4.1"},{"comment":"The bias metric classifies as a 'bias jailbreak' any shift in the multiple-choice answer after the {patient} placeholder, the wording, or the emotional tone of the self-report is changed, using the model's own initial response as the 'unbiased reference answer' (§4.3.1-4.3.2). No control is reported in which the prompt is perturbed neutrally (e.g., reordering the clinical note or inserting a benign non-clinical sentence), so the 81.1% cognitive-bias rate and the roughly 24% identity, linguistic, and emotional shift rates conflate genuine unfairness with generic sensitivity to wording changes. The Discussion's concession that the equity scenarios 'lack a canonical gold standard' (§3) underlines the point: as defined, the measurement is answer instability, not bias. A concrete fix is to add a neutral-perturbation control and report the excess shift rate attributable to the bias content, or to relabel the metric as an 'answer-shift rate' and soften the fairness conclusions.","section":"§2.3, §4.3.1, §4.3.2"},{"comment":"The composite privacy figure of 91% is computed only over the models selected for adversarial testing, namely those with post-warning jailbreak rates below 80%; DeepSeek-V3, QwQ-32B, Qwen3-32B, MedGemma, and Gemma-3 were explicitly excluded because they already exceeded 90% jailbreaks despite the explicit privacy warning (§2.2). The headline therefore describes a non-random subpopulation, and the Fig. 4f caption does not flag the selection. The paper should report the composite rate for the full panel alongside the adversarial subset, or restrict the headline claim to the direct-request rate of 86.46% over all 81 scenarios and all models, which is the comparison that is actually comparable across the panel.","section":"§2.2, Fig. 4f"},{"comment":"The headline '94% of previously correct answers failed' aggregates six mutation tools that differ in kind: answer negation removes the correct option, question inversion changes the task to identifying the complement set (with incomplete answers scored as incorrect), and physiological impossibility reassigns the ground truth to a new 'false/impossible information' option. Since the per-tool ablation shows that question inversion alone produces a 60% jailbreak rate (Fig. 3c), a substantial share of the 94% reflects sensitivity to an unusual task format rather than loss of the underlying clinical knowledge. The inference from these numbers to 'superficial memorization rather than genuine clinical capability' (Abstract) is therefore stronger than the evidence supports; reporting same-task mutations (e.g., narrative distraction, choice expansion) separately from task-changing mutations, and tempering the memorization language, would place the benchmarking-gap claim on firmer ground.","section":"§2.1, Fig. 3c"}],"minor_comments":[{"comment":"The arXiv abstract's 'hallucination rates exceeded 74% in widely used models' disagrees with the paper's own abstract ('exceeding 66%') and with §2.4's reported averages of 62.6% for proprietary and 72.5% for open-source models; these numbers must be reconciled.","section":"Abstract; §2.4"},{"comment":"The text refers to 'Fig. 4c' for the key results of the privacy attack, but the corresponding panel is labeled 'f' in the figure caption; the cross-reference should be corrected.","section":"§2.2, Fig. 4"},{"comment":"There is a typo in 'Physiological Impossibility can mot be applied'; 'mot' should be 'not'.","section":"§4.1.2"},{"comment":"The abstract and the other three axes report 15 models, while §2.1 and §4 state that 16 models (including o3-mini) were tested for robustness; the panel definition should be stated consistently throughout.","section":"Abstract; §2.1; §4"},{"comment":"The text reports a 'median jailbreak rate of 94%' while the Fig. 3b caption reports 'mean = 94%'; please clarify which statistic is being displayed.","section":"§2.1, Fig. 3b"},{"comment":"The sentence '>80% of rabbit models we eventually observed a jailbreak after at least one failed attempt (Fig. 3b)' is grammatically garbled; presumably the intended meaning is that a jailbreak was eventually observed in more than 80% of cases after at least one failed attempt.","section":"§3 Discussion"},{"comment":"The hallucination rates are reported directly from the detector without propagating its measured classification error (81.9% accuracy, 79.8% precision, 83.2% recall on the 260-sample validation set); a sensitivity analysis showing the range of rates under error correction would make the hallucination-pillar numbers more robust.","section":"§2.4, §4.4.2"}],"recommendation":"major_revision","confidential_remarks":"The abstract-body mismatch on HealthBench is the most serious issue in the submission: the abstract claims a generalization result that the Methods never report and the Limitations explicitly disclaim. I recommend that the editor require either the missing analysis or a corrected abstract before acceptance. The privacy headline is also reported on a non-random subset without a clear caveat, and the bias metric needs a neutral control to support the fairness language. All of these are fixable within the manuscript's scope, so I view major revision as appropriate rather than rejection. Separately, the 15- versus 16-model panel discrepancy and the hallucination-rate inconsistency between the arXiv abstract and the main-text abstract should be corrected in the same revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the framework is real and worth knowing about; the headline numbers are not all supported as stated. The paper builds a four-axis dynamic red-teaming audit for medical LLMs, applies it to 15 models, and releases code. That is a concrete contribution, and the core observation—static MedQA scores coexist with high failure under adversarial perturbation—is directionally credible.\n\nThe soft spots are concentrated in the abstract and the bias/privacy measurement choices. The abstract claims the robustness gap \"generalized to the realistic, open-ended HealthBench dataset,\" with failure rates exceeding 70%. I could not find that experiment in the main text, methods, or appendices. HealthBench appears only as a source of negative hallucination samples. The limitations section explicitly says the robustness audit is built on multiple-choice MedQA, not open-ended formats. That is a direct contradiction, and it is load-bearing because the \"superficial memorization\" conclusion rests on it. The abstract also says hallucination rates exceeded 74% while the body's abstract says 66%; the numbers need to be reconciled.\n\nThe bias axis has a more subtle problem. A \"bias jailbreak\" is defined as any answer change relative to the model's own initial response. Without a neutral-perturbation control, you cannot tell whether the model is biased or just sensitive to wording. The authors acknowledge the equity scenarios lack a gold standard, but they still report an 81% cognitive-bias rate as if it were a clean fairness metric. The privacy composite rate is computed only on the subset of models with initial jailbreak rates below 80%, so the 91% figure is not a rate for the full panel. And the hallucination detector has an 18% error rate that is not propagated into the reported hallucination rates.\n\nThe robustness finding itself is more defensible: adversarial search should find failures, and the six mutation tools are reasonable clinical stress tests. But the 94% figure is an upper bound from an adaptive attacker, not an unbiased failure rate under representative perturbation.\n\nBottom line: this paper deserves a serious referee. The framework, datasets, and code are useful, and the direction is sound. But the abstract overstates the evidence, the bias metric needs a control, and the reporting needs to be honest about subsets and detector error. With those fixes, it could be a strong paper. I'd send it to review, with the expectation of heavy revision.","headline":"A genuinely useful red-teaming framework and dataset for medical LLMs, but the abstract overstates the evidence and the bias metric needs a neutral control.","tokens_in":47174,"tokens_out":3205,"would_cite":true,"duration_ms":38881,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Static benchmarks overstate medical LLM safety: 94% of 'correct' answers fail under stress.","keywords":["medical LLM safety","dynamic red-teaming","adversarial benchmarks","benchmarking gap","hallucination detection","privacy leakage","bias and fairness","healthcare AI evaluation"],"falsifier":"Run the same bias protocol with a neutral-perturbation control: rewrite the patient self-report using synonyms, sentence reordering, or innocuous added details that carry no sociodemographic, emotional, or judgmental content, and measure the answer-shift rate. If the neutral shift rate is comparable to the 81% rate seen under bias manipulations, the bias metric is measuring general instability rather than unfairness.","tokens_in":46149,"feed_emoji":"🩺","tokens_out":6705,"duration_ms":74704,"temperature":0.7,"pith_summary":"This paper tries to establish that static benchmark scores give a misleading picture of how safe large language models are in healthcare. It introduces a Dynamic, Automatic, and Systematic (DAS) red-teaming audit in which adversarial agents continuously mutate medical questions and escalate attacks across four axes—robustness, privacy, bias/fairness, and hallucination—until a model fails or the budget runs out. Across 15 proprietary and open-source models, the authors report that despite a median MedQA accuracy above 80%, 94% of previously correct answers became wrong under dynamic robustness testing, that privacy leaks were elicited in 86% of scenarios, that cognitive-bias priming shifted clinical recommendations in 81% of fairness tests, and that hallucination rates exceeded 50% for every model. The authors argue that if this holds, safety claims based on static leaderboards are systematically overconfident and that clinical deployment should be gated by living adversarial audits instead.","feed_headline":"Medical LLMs fail 94% of previously correct answers under stress","feed_subtitle":"High exam scores hide it: when 15 medical LLMs face live adversarial pressure, 94% of their correct answers collapse.","key_machinery":"The load-bearing object is the DAS framework itself: a closed-loop, agent-based audit in which attacker agents choose from a predefined toolbox of mutation tools, examine the model's previous response, and escalate or switch strategies over up to five conversational rounds until the model fails. It operationally defines a 'jailbreak' as a shift from a correct, safe, or unbiased baseline to an incorrect, unsafe, or biased answer, and pairs this with automated detector agents for privacy and hallucination. The framework is dynamic in that every mutated prompt is generated on the fly, making memorization of the test set ineffective, and automatic in that no human intervention is required; a full four-axis audit costs roughly 35 to 40 dollars in API fees.","core_discovery":"The central claim is that current medical LLMs exhibit a 'Benchmarking Gap': high scores on established static exams do not transfer to reliable behaviour under realistic adversarial pressure. Starting from MedQA medical-exam questions each model answered correctly, the DAS robustness orchestrator mutated the questions with six constrained tools—answer negation, question inversion, choice expansion, narrative distraction, cognitive bait, and physiological impossibility—over up to five escalating rounds, and achieved a median jailbreak rate of 94%, with the most robust model still failing 69% of the time. Privacy red-teaming, using well-meaning, subtle, misdirection, and trap-warning disguises, elicited protected-health-information leaks in 86% of scenarios on average and 91% after the adversarial pass; cognitive-bias priming shifted decisions in 81% of fairness tests; and a seven-category hallucination detector found that every model hallucinated on more than half of high-risk prompts, with false or outdated medical facts the leading error. The paper also shows that the same brittleness appears on open-ended HealthBench-style prompts and that model rankings shift sharply across evaluation axes, so 'which model is safest' depends on which pillar is inspected.","pith_inferences":["A natural extension is to apply the same dynamic red-teaming recipe to other high-stakes domains where static benchmarks dominate, such as legal advice, financial counseling, or government benefit guidance; the 'benchmarking gap' is likely a general property of overfitting to static tests, not a medical-specific artifact.","The bias metric's reference point is the model itself: without a neutral-rewording control, part of the 81% answer-shift rate may reflect ordinary prompt sensitivity rather than sociodemographic unfairness, and running the identical protocol with benign stylistic rewrites would separate those components.","Because the attacker and detector agents are themselves LLMs with imperfect fidelity, including mutation-tool validity rates of 88 to 96%, 82% hallucination-detector accuracy, and 93% privacy-judge accuracy, the absolute jailbreak rates carry measurement noise; error-corrected estimates would bound the true failure rates.","The current attack vocabulary is finite, so a truly living audit would need an open-ended mutation generator that invents new attacks rather than recombining predefined ones; otherwise the framework itself could become the next static checklist."],"forward_implications":["Static benchmark scores cannot be used as evidence of clinical reliability: a model's 'correct' answers collapse under perturbations that preserve the clinical content.","Safety is axis-specific: different models lead on robustness, privacy, fairness, and factuality, so a single aggregate safety score is misleading and audits must report per-axis dossiers.","Dynamic red-teaming can function as post-market surveillance for model updates, with automatically generated risk dossiers accompanying each release at low cost.","Because prompts are generated on the fly, dynamic audits resist Goodhart-style optimization, whereas static benchmarks become targets for fine-tuning and contamination.","Chain-of-reasoning models hallucinate more on logic, context, and citation errors than non-reasoning baselines, even though they follow instructions and flag unsafe recommendations more often."],"supporting_citations":[{"why":"Provides the canonical MedQA medical-exam benchmark whose first-round correct answers seed the robustness stress tests.","marker":"[4]"},{"why":"Supplies the Goodhart's-law argument that measures become targets, motivating the shift from static to dynamic evaluation.","marker":"[10]"},{"why":"Supplies high-confidence positive hallucination prompts used to measure model hallucination rates.","marker":"[15]"},{"why":"Provides evidence that models can 'learn the test,' which the paper cites as a key failure of static benchmarks.","marker":"[20]"},{"why":"Supplies the cognitive-bias taxonomy that the bias-priming agent draws on for its attack strategies.","marker":"[26]"},{"why":"Supplies most of the bias-elicitation clinical vignettes that are manually screened and adapted into the fairness dataset.","marker":"[34]"},{"why":"Supplies physician-vetted, near-ideal open-ended responses used as negative hallucination samples and as realistic evaluation material.","marker":"[35]"}],"fun_headline_variants":["Medical LLMs: 94% of correct answers fail under stress","High MedQA scores hide 94% failure under dynamic red-teaming","Red-teaming reveals 94% collapse in previously correct medical AI answers","Dynamic stress tests expose medical LLM benchmark gaps","94% of medical AI answers fail when red-teamed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bias results define 'unbiased' as the model's own initial answer, and count any change after changing a patient's demographics, language, emotion, or a cognitive-bias cue as unfair bias; because the paper reports no neutral-perturbation control and concedes the equity scenarios lack a canonical gold standard, the 81% bias rate could overstate unfairness if models also change under benign rewording.","fun_headline_variants_meta":{"raw":{"variants":["Medical LLMs: 94% of correct answers fail under stress","High MedQA scores hide 94% failure under dynamic red-teaming","Red-teaming reveals 94% collapse in previously correct medical AI answers","Dynamic stress tests expose medical LLM benchmark gaps","94% of medical AI answers fail when red-teamed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1734,"prompt_tokens":1117,"completion_tokens":617,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":733,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":733,"tokens_out":617,"duration_ms":6858,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:37:04.515261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same bias protocol with a neutral-perturbation control: rewrite the patient self-report using synonyms, sentence reordering, or innocuous added details that carry no sociodemographic, emotional, or judgmental content, and measure the answer-shift rate. If the neutral shift rate is comparable to the 81% rate seen under bias manipulations, the bias metric is measuring general instability rather than unfairness.","supporting_citations":[],"review_version":1}