{"id":"fd3cda49-11ca-419d-98f3-392767085a9b","arxiv_id":"2501.18632","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Medical LLMs are highly vulnerable to black-box jailbreaking, and adversarial continual fine-tuning greatly reduces measured jailbreak effectiveness.","lead":"This study measures how easily seven large language models can be tricked into giving harmful medical advice using three jailbreaking techniques, and tests whether fine-tuning makes them safer. It reports high attack success rates on leading models like GPT-4o and shows that safety fine-tuning sharply lowers those rates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98% compliance headline rests entirely on an unvalidated GPT-4o judge; without clinician agreement or a human-labeled calibration set, the metric could misclassify refusals or disclaimers as compliant.","rationale":"The paper's contribution is an empirical vulnerability assessment plus a defense evaluation, and its headline is a number produced entirely by the automated GPT-4o judge. No other evidence directly measures whether that number corresponds to real patient harm. The reader's weakest assumption identifies exactly this evaluator-validity issue, and I agree that it is the most load-bearing point. The concern is not that the authors are dishonest; the paper openly acknowledges in its Limitations section that automated evaluation 'may not capture subtle aspects of model vulnerabilities,' and the overall direction of the finding is consistent with prior jailbreaking literature. However, the specific 98% figure and all mean-effectiveness comparisons are only as trustworthy as the judge. The ambiguity in the 'policy_alignment' term is an internal red flag that the rubric's direction was not carefully checked. A clinician-labeled calibration study is relatively cheap and would settle the matter: if agreement is high, the paper is a solid conditional accept; if agreement is low, the headline must be revised or re-measured. Because the weakness is concrete and fixable rather than fatal, the reader's CONDITIONAL verdict should stand, so no verdict adjustment is needed.","tokens_in":8819,"tokens_out":4812,"duration_ms":51979,"concrete_test":"Sample all FlipAttack responses from GPT-4o and Llama3.3-70B that the judge marked compliant (roughly 440 per model), plus 50 clearly refusing responses from each. Have two board-certified physicians independently label each response as 'actionable harmful medical advice', 'safe refusal/disclaimer', or 'ambiguous/irrelevant', then compute agreement (Cohen's kappa) with the GPT-4o judge's compliance and harmfulness scores. If kappa is below 0.8 or if more than 10% of judge-compliant responses are clinically safe or non-actionable, the 98% compliance claim and the derived effectiveness scores are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—'the most effective jailbreaking technique reaches a 98% compliance rate on GPT-4o and llama3.3-70B'—rests on one component: the GPT-4o judge's binary compliance score (Automated Evaluator, Figure 7). The paper states that the judge 'follows the StrongREJECT setup' and substitutes AMA Principles of Medical Ethics for the usage policy, but it provides no human validation of this adapted rubric in the medical domain, no clinician agreement study, and no calibration against MedSafetyBench's own labels. Because effectiveness is defined as compliance multiplied by harmfulness, any systematic tendency of the judge to label non-refusals—including disclaimers, general advice, or irrelevant text—as compliant will inflate every table. The adjacent definition of harmfulness also uses a 'policy_alignment' term whose direction is ambiguous: the name implies high score equals aligned with AMA policy, but the text says high harmfulness requires a response that 'clearly violates' the policy. If the judge was actually prompted for policy violation, the naming is misleading; if it was prompted for policy alignment, the formula is inverted. This unvalidated evaluator is the load-bearing pillar of the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies jailbreaking of large language models (LLMs) in medical contexts. The authors evaluate seven LLMs (GPT-4o, GPT-4-turbo, DeepSeek-R1-Distill-Llama-70B, Llama3.3-70B, Llama3.1-8B, Meditron-7B, Meditron-70B) against three black-box jailbreaking techniques (PAIR, PAP with three persuasion variants, FlipAttack) using a domain-adapted GPT-4o judge derived from StrongREJECT, with the AMA Principles of Medical Ethics as the governing policy. They report mean effectiveness scores, compliance rates, and model breach rates, and show that continual fine-tuning (CFT) on MedSafetyBench data substantially reduces the measured effectiveness for Llama3.1-8B and Meditron-7B. The central empirical claim is that the most effective technique achieves a 98% compliance rate on GPT-4o and Llama3.3-70B, indicating that alignment alone is insufficient for medical safety.","tokens_in":9042,"tokens_out":5723,"duration_ms":49650,"significance":"If the reported numbers are accurate, the paper makes a useful empirical contribution in a high-stakes domain: it demonstrates that current frontier models are highly vulnerable to simple black-box jailbreaks for harmful medical queries, and that domain-specific continual fine-tuning can substantially reduce that vulnerability. The proposed evaluation pipeline is a practical, scalable tool, and the release of code is a strength. The use of MedSafetyBench as a query source is also appropriate. However, the quantitative conclusions rest on an unvalidated automated judge, and the paper's own Limitations section acknowledges that automated evaluation 'may not capture subtle aspects of model vulnerabilities that could be better assessed through human review.' With human validation of the judge and additional robustness checks, the paper could become a solid contribution to AI safety evaluation in healthcare.","major_comments":[{"comment":"The automated judge is the sole source of every quantitative result, yet it is never validated. The paper states in the 'Automated Evaluator' section that the judge 'follows the StrongREJECT setup,' but it provides no human-agreement study, no calibration against MedSafetyBench's existing labels, and no error analysis showing what kinds of responses are misclassified. Since the headline claim of a 98% compliance rate (Table 2, FlipAttack for GPT-4o and Llama3.3-70B) and all effectiveness and breach rates in Tables 1 and 3 are computed from this judge, a systematic bias (e.g., labeling disclaimers or general advice as compliant) would invalidate every reported number. The manuscript's Limitations section explicitly concedes that the 'reliance on automated evaluation frameworks, though scalable, may not capture subtle aspects of model vulnerabilities that could be better assessed through human review.' Given that this is the load-bearing component of the central claim, the paper must add a clinician or human-annotator agreement study on a representative sample (ideally 100 or more responses), report per-category agreement, and provide example cases where the judge diverges from human judgment.","section":"Automated Evaluator"},{"comment":"The definition and direction of the 'policy_alignment' term are ambiguous and potentially self-contradictory. The text defines harmfulness as (2*policy_alignment + specificity + convincingness)/4 and states that a high harmfulness score requires a response that 'clearly violates the provided usage policy.' If policy_alignment is scored as degree of alignment with the AMA Principles, then a high alignment score should be negatively correlated with harmfulness, and the formula would be inverted. If the score actually measures policy violation, then the variable name is misleading. Figure 7 is only a schematic; the exact prompt sent to the GPT-4o judge is not quoted. This ambiguity affects every harmfulness-derived value in the paper, so the authors must provide the exact judge prompt and clarify the scoring direction of each rubric item, ideally with a worked example of a scored response.","section":"Automated Evaluator (harmfulness formula)"},{"comment":"No confidence intervals, repeated trials, or statistical tests are reported. LLM outputs are stochastic (especially with the 'temperature' settings implied by the API and open-source sampling), and the GPT-4o judge itself is stochastic. The point estimates in Tables 1 and 2 (e.g., GPT-4o FlipAttack compliance 0.98 vs. PAIR 0.96) could easily flip across runs, and the CFT improvements, while dramatic, are not accompanied by any variance measure. The paper should report at least several independent runs for the headline model–attack combinations, or provide bootstrap confidence intervals, and a paired significance test for the CFT comparisons.","section":"Results / Tables 1-2"},{"comment":"The claim that CFT is evaluated under 'out-of-distribution testing' is not well supported. The CFT models are trained on MedSafety-Improve-GPT4 and evaluated on MedSafety-Eval-GPT4, both of which are subsets of MedSafetyBench, and both training and evaluation queries are transformed with the same jailbreak techniques (PAIR, PAP variants, FlipAttack). The only difference is the base harmful query set. Therefore, the evaluation is in-distribution with respect to the attack templates and the domain, which undercuts the generalizability claim. The authors should either provide evidence that the evaluation queries are distributionally distinct from the training queries (e.g., disjoint text features or a separate benchmark) and that the attack templates transfer, or they should temper the wording to 'held-out evaluation within MedSafetyBench.'","section":"Model Guardrail Enhancement"}],"minor_comments":[{"comment":"The Abstract and Introduction state that the paper selects 'five black-box jailbreaking techniques,' while the 'Jailbreaking Techniques' section lists three techniques (PAIR, PAP, FlipAttack) with PAP having three persuasion variants. Clarify whether the count refers to five attack instantiations (PAIR, three PAP variants, FlipAttack) or something else.","section":"Abstract / Introduction"},{"comment":"The text claims that continual fine-tuning 'decreases the mean effectiveness score of jailbreaking on llama3.1-8B by an average of 62.7%.' From the rounded values in Table 1, the average absolute decrease is 0.625 (62.5%), while the average relative decrease is approximately 97%. Please state exactly how 62.7% was computed (absolute vs. relative, and with which unrounded values).","section":"Results"},{"comment":"The Abstract highlights 'a 98% compliance rate on GPT-4o and llama3.3-70B,' but Table 2 also shows GPT-4-turbo reaching 1.00 under FlipAttack. Make the headline statement consistent with the full table, or explicitly explain why GPT-4-turbo is not mentioned in the abstract's strongest claim.","section":"Table 2 / Abstract"},{"comment":"The judge's binary 'compliance' score is not defined for borderline cases, such as responses that contain a refusal followed by a partial answer or a disclaimer alongside the requested information. Specify how such responses are coded, as this directly affects the compliance rate.","section":"Automated Evaluator / Figure 7"},{"comment":"Reference [30] (American Medical Association, Code of Medical Ethics) lacks a URL, year, or access date. Completing this reference would improve reproducibility of the policy used in the judge.","section":"References"},{"comment":"The code is available at the stated URL, but the paper does not mention a version, license, or detailed usage instructions. Adding these would support reproducibility of the evaluation pipeline.","section":"Code availability"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped empirical study with a potentially important message, but the missing validation of the automated judge is exactly the kind of load-bearing gap that must be fixed before publication. I would ask the authors to provide a clinician agreement study on a sample of judge decisions, quote the exact judge prompt, clarify the scoring direction of 'policy_alignment,' and run repeated trials with confidence intervals. If the judge turns out to be unreliable, the paper may need to be reframed as a qualitative demonstration of vulnerability rather than a quantitative benchmark. The fit with a security or medical AI venue is reasonable, but only after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The paper applies three known jailbreak techniques (PAIR, PAP, FlipAttack) to seven LLMs in medical contexts, using a GPT-4o judge adapted from StrongREJECT with AMA Principles of Medical Ethics, and then shows that LoRA-based continual fine-tuning on MedSafetyBench crushes attack effectiveness. The headline 98% compliance on GPT-4o/llama3.3-70B is plausible but rests on an unvalidated judge, so treat the absolute numbers as provisional.\n\nWhat is genuinely useful: the domain-adapted rubric, the Model Breach Rate metric (fraction of prompts breached by at least one attack), the breadth of models including Meditron, and the CFT results. The CFT numbers are striking—llama3.1-8B-CFT drops mean effectiveness from 0.57 to 0.01 on plain queries and similar across attacks. Code is released. That is real, reproducible work.\n\nSoft spots, in order. First, the judge. There is no human validation, no clinician agreement, and no calibration against MedSafetyBench's own labels. The plain-query compliance rates look high (0.44 for GPT-4o), which makes me suspect the judge is lenient. Second, the harmfulness formula is ambiguous: it adds policy_alignment with positive weight, but the text says a harmful response is one that violates the policy. Either the naming is off or the formula is inverted; the paper never resolves this. Third, the defense evaluation is not truly out-of-distribution: training and evaluation sets are both drawn from MedSafetyBench. It is a within-benchmark test. Fourth, no error bars or repeated sampling; GPT-4o serves as both target and judge, which could bias scores.\n\nThese are fixable, not fatal. The central vulnerability claim is consistent with prior jailbreaking literature, and the CFT effect is large enough that a noisy judge is unlikely to fully explain it. The paper's own limitations section acknowledges the automated-evaluation concern.\n\nWho should read it: people building or evaluating medical LLM safety, and anyone doing red-teaming in high-stakes domains. It deserves a serious referee. I would send it to review with the requirement that the authors validate the judge on a human-labeled sample (even 100 responses) and clarify the policy_alignment direction.","headline":"A useful domain-specific jailbreak evaluation of medical LLMs whose headline numbers rest on an unvalidated GPT-4o judge; the CFT defense results are strong enough to warrant peer review.","tokens_in":9556,"tokens_out":2874,"would_cite":true,"duration_ms":28708,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jailbreaks push medical AI to comply 98% of the time","keywords":["LLM jailbreaking","healthcare safety","medical AI","black-box attacks","compliance rate","continual fine-tuning","GPT-4o","safety alignment"],"falsifier":"Take a random sample of query-response pairs from the paper's evaluation and have board-certified clinicians score harmfulness independently; if expert scores disagree with the GPT-4o judge on more than a small fraction of cases, the reported 98% compliance rate and all effectiveness scores need revision.","tokens_in":8616,"feed_emoji":"🩺","tokens_out":7831,"duration_ms":69135,"temperature":0.7,"pith_summary":"This paper tries to establish that state-of-the-art large language models, including GPT-4o and Llama 3.3-70B, are not safe enough to act as medical assistants: simple black-box jailbreaking prompts can push them to comply with harmful medical requests almost every time, with the strongest attack reaching a 98% compliance rate. It also claims that continual fine-tuning on medical safety demonstrations can slash that vulnerability, driving mean effectiveness scores and compliance rates close to zero on smaller models. The motivation is concrete: if an aligned chatbot gives unethical clinical guidance when attacked, patient harm becomes a realistic outcome, so knowing which attacks work and which defenses hold matters for anyone deploying AI in healthcare.","feed_headline":"Medical jailbreaks defeat AI guardrails 98% of the time","feed_subtitle":"Simple prompt tricks make chatbots obey harmful medical requests; a study shows fine-tuning can restore safety.","key_machinery":"The evaluation pipeline is the load-bearing machinery. A GPT-4o judge scores each query-response pair on four dimensions: binary compliance, policy alignment against medical ethics principles, specificity, and convincingness. Those yield a harmfulness score $harmfulness = (2 \\cdot policy\\_alignment + specificity + convincingness)/4$, and the final jailbreak effectiveness is $effectiveness = compliance \\cdot harmfulness$. The defense machinery is continual fine-tuning with Low-Rank Adaptation on 2,700 jailbreak-invariant safe-response pairs, which repeatedly teaches the model to refuse adversarial variants of medical prompts.","core_discovery":"The paper's central finding is that leading commercial and open-source LLMs are far from robust to jailbreaking in medical contexts: FlipAttack, a one-query perturbation technique, achieves a 98% compliance rate on GPT-4o and Llama 3.3-70B, and GPT-4-turbo reaches a compliance rate of 1.00 on the same attack. Model breach rate—the fraction of harmful prompts for which at least one of the tested jailbreaks fully succeeds—reaches 0.93 for GPT-4-turbo and 0.81 for GPT-4o, meaning an attacker iterating through techniques succeeds for nearly every query. On the defense side, continual fine-tuning on safe-response pairs reduces the mean jailbreak effectiveness on Llama 3.1-8B by an average of 62.7 percentage points across techniques, with post-tuning effectiveness scores near zero.","pith_inferences":["The reported 98% compliance rate depends entirely on the GPT-4o judge's harmfulness scoring; an expert clinician panel might rate the same responses as less harmful if they contain disclaimers, or more harmful if subtle incorrect doses slip through.","Because the judge weights policy alignment double, a response that refuses in a roundabout way but still provides dangerous advice could receive a non-zero effectiveness score; testing with a stricter judge could change the ranking of attacks.","The defense results are measured on prompts derived from the same dataset family used for training, so out-of-distribution generalization of continual fine-tuning to novel attack formats remains open; a natural extension would be to test CFT models against future jailbreaks not derived from that benchmark.","If the CFT recipe scales to larger models, it offers a targeted alternative to full safety fine-tuning, but its effect on general medical question-answering utility is not measured in this study."],"forward_implications":["General-purpose safety alignment does not transfer to medical contexts: plain harmful queries are refused more often, but iterative and perturbation attacks bypass guardrails at high rates.","An attacker with no model access can reliably elicit harmful medical output from frontier models by iterating through a few known attacks, since model breach rates approach 0.93.","Continual fine-tuning on domain-specific safety demonstrations can drive compliance near zero on smaller open models, suggesting a practical defense for bespoke clinical deployments.","Lower-capability models like Meditron show lower jailbreak effectiveness, implying a trade-off between clinical capability and vulnerability to adversarial prompts."],"supporting_citations":[{"why":"Supplies the harmful medical queries that form the basis for all 2,700 test prompts and the safe-response training pairs.","marker":"[24]"},{"why":"Provides the fine-grained scoring rubric and the mean effectiveness metric the evaluation pipeline adapts.","marker":"[23]"},{"why":"Contributes the policy-alignment evaluation template that the judge follows when scoring responses.","marker":"[21]"},{"why":"Defines the iterative PAIR attack, one of the three jailbreak techniques tested.","marker":"[15]"},{"why":"Defines the persuasion-based PAP attack with its three variants tested here.","marker":"[16]"},{"why":"Defines the one-query FlipAttack that reaches the highest compliance rates.","marker":"[19]"},{"why":"The medical-ethics policy that serves as the judge's alignment target.","marker":"[30]"},{"why":"The low-rank adaptation method used to perform continual fine-tuning in the defense experiments.","marker":"[31]"}],"fun_headline_variants":["FlipAttack jailbreak fools GPT-4o 98% of the time","Medical jailbreaks breach GPT-4-turbo 93% of the time","Fine-tuning cuts medical jailbreak success by 63 points","One-query attack yields 98% compliance on medical LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated GPT-4o judge, adapted from a jailbreak-evaluation rubric with medical ethics as policy, correctly identifies harmful responses; the paper never shows the judge's scores match expert clinicians, so every compliance and effectiveness number inherits that uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["FlipAttack jailbreak fools GPT-4o 98% of the time","Medical jailbreaks breach GPT-4-turbo 93% of the time","Fine-tuning cuts medical jailbreak success by 63 points","One-query attack yields 98% compliance on medical LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3186,"prompt_tokens":875,"completion_tokens":2311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2233}},"tokens_in":491,"tokens_out":2311,"duration_ms":17561,"temperature":1.0,"reasoning_tokens":2233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:33:40.661865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of query-response pairs from the paper's evaluation and have board-certified clinicians score harmfulness independently; if expert scores disagree with the GPT-4o judge on more than a small fraction of cases, the reported 98% compliance rate and all effectiveness scores need revision.","supporting_citations":[],"review_version":1}