Pith. sign in

REVIEW 4 major objections 4 minor 44 references

Watermarking LLMs for traceability can silently corrupt clinical reasoning—fabricating diagnoses, misusing terms, and misreading images—even when benchmark accuracy barely moves.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:48 UTC pith:EE32JH4D

load-bearing objection Solid empirical evidence that watermarking degrades clinical reasoning even when accuracy benchmarks stay flat; the judge calibration is the main caveat, not a fatal one. the 4 major comments →

arxiv 2607.20462 v1 pith:EE32JH4D submitted 2026-05-16 cs.AI

Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

Melanie Rieff , Robin Staab , Thibaud Gloaguen , Stefan Hegselmann , Martin Vechev This is my paper
classification cs.AI
keywords LLM watermarkingmedical NLPclinical reasoning evaluationhallucination detectionmultimodal medical benchmarksquality degradationreasoning tracesLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Watermarking is the leading mechanism for tracing AI-generated text, and clinical deployments of large language models will increasingly be watermarked. This paper asks whether the token-level perturbations that watermarks introduce are harmless in medicine, where one altered word can change a diagnosis. It benchmarks five watermarking schemes across text-only and multimodal medical models, and it claims the answer is no: accuracy on medical multiple-choice benchmarks often stays within the unwatermarked baseline, yet watermarked outputs show more fabricated medical entities, misapplied terminology, corrupted spellings, and contradicted image findings. The paper introduces three physician-validated judge audits to see past letter accuracy, and argues that aggregate benchmarks systematically mask these failures. If correct, the implication is direct: evaluating a watermark's safety in medicine requires domain-specific reasoning audits, not just accuracy.

Core claim

The paper's central claim is that applying a text watermark to a medical LLM can degrade the clinical quality of the output even when the model's benchmark accuracy remains statistically unchanged. Across five watermarking schemes, eleven language models, and seven vision-language models, the authors find that watermarks increase the rate of correct answers backed by flawed reasoning, inject fabricated or misapplied medical terms—'pentaprazole' for pantoprazole, a non-existent antibiotic 'naftifloxacin'—and shift visual grounding so that supported image findings fall while contradicted claims rise. The damage is scheme-specific and model-specific: one scheme mostly corrupts surface spelling,

What carries the argument

The load-bearing instrument is a set of three physician-validated LLM-as-judge audits that look past the multiple-choice letter. A single-response audit scores each completion for fabricated entities, misapplied real terms, corrupted spellings, confident errors, clue misreads, contradictions, and—for multimodal inputs—classifies each perceptual image claim as supported, contradicted, or unverifiable against the image. A pairwise divergence audit compares watermarked and unwatermarked answers to the same question and flags mutually exclusive diagnostic interpretations even when both pick the correct letter. A reasoning-trace audit rates the hidden chain of thought of reasoning models on effic

Load-bearing premise

The entire harm estimate rests on the LLM judge's flags being a valid proxy for clinically meaningful degradation; physician agreement is strong for fabricated entities but only moderate for misapplied terms and image-contradicted claims, so if those flags carry systematic bias the reported degradation magnitudes could be overstated.

What would settle it

A prospective blinded study in which board-certified physicians score a large sample of watermarked and unwatermarked outputs on open-ended clinical documentation, not multiple choice, could settle the claim: if expert-rated harm rates do not rise with watermark presence at the strongest detectability settings, the degradation reported here is an artifact of the judge rather than a clinical effect.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Accuracy on medical benchmarks is not a sufficient safety signal: a watermark can raise the faulty-but-correct rate from 11.4% to 26.3% on a 14B model while accuracy drops only 3.2 points.
  • Watermarking vision-language models corrupts visual reporting: supported image claims fall and contradicted claims rise in most configurations, even for models whose accuracy stays within the baseline confidence band.
  • For reasoning models, watermarking only the final answer is essentially free, while watermarking the reasoning trace inflates output length by up to 69% and degrades terminology and efficiency.
  • Degradation is model- and scheme-specific: general-purpose 70B models can be stable while medical-specialised 70Bs collapse, and each scheme damages a different axis; even 'distortion-free' schemes show significant degradation.
  • Domain-specific evaluation should be a prerequisite for deploying watermarked models in medicine, because current benchmarks can otherwise obscure clinically consequential failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the branching-point mechanism generalizes, watermark harm is concentrated at decision-critical tokens, so a token-importance-aware watermark—preserving clinical content while watermarking low-stakes tokens—could maintain traceability with less clinical damage; the paper leaves this design open.
  • The findings imply that regulatory operating points defined only by detectability (for example, 99% true-positive rate at 1% false-positive rate) are insufficient; compliance benchmarks should include domain-quality audits, or a watermark can pass detection requirements while degrading the text clinicians read.
  • The judge-validation numbers suggest which axes to trust: fabrication counts and pairwise divergence show substantial agreement with physicians, while misapplied-term and image-contradiction flags show only moderate agreement; a larger prospective physician study on those low-agreement axes would sharpen or revise the reported magnitudes.
  • For multi-turn clinical workflows, watermarked outputs become context for subsequent generations; the paper notes it remains unclear whether degradation compounds or attenuates, so testing watermark harm in interactive report-writing and follow-up dialogue settings is a direct extension.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents a large-scale empirical evaluation of five text-watermarking schemes (KGW, DipMark, AAR, PPL, SynthID) applied to medical LLMs and VLMs on MedQA and MedXpertQA-MM. The authors introduce three LLM-as-judge protocols—single-response audit, pairwise divergence audit, and reasoning-trace audit—to measure reasoning quality, terminology misuse, hallucinations, and multimodal grounding beyond MCQ accuracy. Their central claim is that watermarking can substantially degrade clinical reasoning even when benchmark accuracy remains within baseline confidence intervals: e.g., faulty-but-correct rates rise from 11.4% to 26.3% on PHI-4-14B under SynthID while accuracy drops only 3.2 pp, and fabricated entities increase by up to 39.2 per 100 questions on LINGSHU-7B. They also report that watermarking reasoning traces in thinking models inflates length and degrades terminology, while restricting the watermark to the final answer preserves reasoning quality. The paper concludes that accuracy-only evaluation is insufficient for safe deployment of watermarked models in medicine.

Significance. If the central claim holds, this is an important result for the trustworthy-deployment literature: it shows that standard accuracy benchmarks can mask clinically consequential watermark-induced degradation, and it identifies answer-only watermarking as a practical mitigation for reasoning models. The study is carefully designed in several respects: paired seeds isolate the watermark effect, bootstrapped CIs are reported, full per-configuration grids are in the appendix, and the judge validation effort is substantial. The branching-point experiment is a useful mechanistic probe. However, the load-bearing audit quantities rest on LLM-judge agreement that is moderate on some headline axes, and the paper does not yet rule out condition-dependent judge bias.

major comments (4)
  1. [Sec. 4.3, Table 10] The headline deltas in Table 2 and Figure 4 rely on the 'misapplied terms' and 'contradicted image claims' axes. Physician agreement for misapplied terms is only κ=0.42/ρ=0.47, and judge–judge agreement for contradicted image claims is only ρ=0.50. The paper attributes low κ to base-rate sensitivity, but κ is prevalence-dependent and this does not address whether disagreement is condition-dependent. Since the reported quantities are differences between watermarked and unwatermarked outputs, a judge that flags more errors in less-fluent watermarked text would inflate the deltas; paired anchoring removes constant bias but not differential bias. Please report agreement stratified by condition (watermarked vs. unwatermarked), and ideally re-score the headline axes with physician adjudication or a second physician.
  2. [Sec. 4.3 vs. Introduction/Conclusion] The Introduction and Conclusion state that the judges were validated against 'two board-certified physicians,' but Sec. 4.3 describes a single board-certified physician annotating the 650 samples, and no physician–physician agreement is reported. The acknowledgments thank one physician for annotation. This discrepancy overstates the validation basis. Please clarify exactly how many physicians annotated which subsets, report per-physician agreement if two were used, or remove the 'two physicians' claim.
  3. [Sec. 4.2, Tables 7 and 8] Statistical significance is reported at p<0.01 for many cells, but the experiment spans 7 models × 5 schemes × roughly 12 axes, yielding hundreds of comparisons. No multiple-testing correction is applied, so some of the scheme-specific 'different axis' conclusions may be false positives. Please report FDR-adjusted q-values or pre-specify a small set of primary axes; at minimum, state the total number of comparisons and discuss the expected number of p<0.01 findings under the null.
  4. [App. D.2, Figure 12] The clinician validation panel is explicitly not statistically powered (5 scored outputs per configuration), and the text says the deployment operating point is TPR≈99% while the figure caption labels the dotted vertical guide as TPR=80%. This panel is a qualitative check, not a confirmation of the automated judge's dose–response. Please either strengthen the panel (more items, more physicians) or soften the claim that the clinician panel confirms the main findings.
minor comments (4)
  1. [Abstract vs. Sec. 3.1] The abstract says 11 LLMs, but Sec. 3.1 lists 6 instruction-tuned + 4 reasoning models = 10; BIOMISTRAL-7B appears only in the appendix. Please reconcile the model count.
  2. [Sec. 4.1, footnote 1] The footnote states BIOMISTRAL-7B collapses with KGW −37.5 and SynthID −14.2 pp, but Table 7 reports −38.7 and −30.4. Please correct the inconsistency.
  3. [App. D.2, Figure 12 caption] The caption says 'Dotted vertical guide marks TPR = 80%' while the body text discusses TPR≈99% as the deployment point. If intentional, explain; otherwise align the caption with the text.
  4. [Sec. A.2] The reproducibility section says code is released, but I did not see a repository URL or artefact identifier. Please add the link or a statement about availability.

Circularity Check

0 steps flagged

No derivative chain to collapse; empirical evaluation with one minor, non-load-bearing self-citation.

full rationale

This is an empirical measurement study, not a derivation-from-principles paper. The central claim (watermarking degrades medical reasoning even when accuracy holds) is established by paired generation experiments across 11 LLMs and 7 VLMs, five watermark schemes, and three LLM-as-judge audits. The judge audits are not defined in terms of watermark status: GEMINI-3-FLASH is blind to condition and gold answer in the single-response audit, and the pairwise audit strips condition labels and randomizes response order; validation is against a human physician (Sec. 4.3, Table 10). No equation in the paper reduces to its own input. The only self-citation is [20] (Gloaguen et al., same research group), which supplies the PPL watermark implementation and an optimality statement for AAR among sum-based distortion-free detectors; both are auxiliary descriptions of the evaluated methods and do not carry the conclusion. Even if [20] were set aside, the empirical contrasts (e.g., PHI-4-14B F&C 11.4%->26.3% under SynthID; LINGSHU-7B +39.2 fabricated entities/100) stand on the reported generation and blind-judge data. The judge-agreement limitations (kappa 0.42 for misapplied terms, rho 0.50 for image-contradicted claims) are a validity or measurement concern, not circularity: the judge is independent of the watermarked models, and the paper transparently reports the weak agreements. Thus no step satisfies the standard of an input-output equivalence by construction; score 1 reflects one minor, non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted to the outcome: watermark strengths are swept and the ≥99% TPR operating point is selected on detectability alone, independent of accuracy or reasoning metrics. The axioms are the main load-bearing assumptions of this empirical study, centered on judge validity and the representativeness of the benchmarks.

axioms (5)
  • domain assumption Paired-seed generation isolates the watermark as the only source of divergence between watermarked and unwatermarked completions.
    App. A.2: 'paired watermarked/unwatermarked completions for a given question therefore share both prompt and seed, isolating the watermark as the only source of divergence.' Requires vLLM sampling to be deterministic under a fixed seed given modified logits.
  • domain assumption The GEMINI-3-FLASH judge's audit axes, after physician validation on 650 samples, are valid measures of clinical reasoning quality.
    Sec. 4.3 and Table 10: the headline claims rely on judge flags for fabricated entities, misapplied terms, and image-contradicted claims, yet agreement is only moderate on several of these axes.
  • domain assumption TPR ≥99% at 1% FPR is the appropriate regulatory operating point for evaluating watermark degradation.
    Sec. 3.2: audits are run at the operating point 'which prior work has considered the operating point required for regulatory deployment.' This is a borrowed standard, not derived.
  • domain assumption MedQA and MedXpertQA-MM multiple-choice accuracy is a meaningful proxy for the clinical performance relevant to watermark safety.
    Sec. 3.1: MedQA is selected partly because a meta-analysis identifies it as most predictive of real-world performance; the multiple-choice single-turn format is itself a limitation the authors acknowledge.
  • domain assumption The five watermark implementations faithfully match the published schemes.
    App. A.1: KGW, DipMark, AAR, PPL, and SynthID are implemented via a custom vLLM LogitsProcessor; incorrect implementation of any scheme would affect its measured degradation.

pith-pipeline@v1.3.0-alltime-deepseek · 35519 in / 11846 out tokens · 105145 ms · 2026-08-02T13:48:37.605531+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking. Yet, most watermarks are evaluated on general-purpose benchmarks, leaving domains like medicine, where small token-level perturbations can result in significant semantic changes, underexplored. In this work, we present the first rigorous study of how LLM watermarks affect medical performance, benchmarking 5 watermarking schemes across 11 LLMs and 7 VLMs on various tasks spanning unimodal and multimodal clinical reasoning. Importantly, we complement existing evaluations by introducing a human-expert-validated pipeline for systematically auditing medical reasoning quality, terminological precision, and induced hallucinations. Our results reveal that watermarking can induce substantial degradation across multiple failure modes, including lexical corruption, hallucinated terminology, and amplified misattribution or omission of image findings. Notably, we find that the absence of domain-specific analyses, combined with aggregate metrics that miss failures inherent to clinical text, can systematically obscure practical watermark-induced degradations. Our findings establish domain-specific evaluation as a prerequisite for the safe deployment of watermarked models in medicine, where current benchmarks can otherwise mask clinically consequential failures.

Figures

Figures reproduced from arXiv: 2607.20462 by Martin Vechev, Melanie Rieff, Robin Staab, Stefan Hegselmann, Thibaud Gloaguen.

Figure 1
Figure 1. Figure 1: Overview of our evaluation pipeline. Given a multimodal benchmark instance, we prompt VLMs (using chain-of-thought prompting) with and without watermarks for reference. We first evaluate the detectability of watermarks in the answers and their accuracy. We then examine the reasoning traces in more depth using three specific LLM-as-judges, validated by clinical experts. medicine, where correctness depends j… view at source ↗
Figure 2
Figure 2. Figure 2: MedQA accuracy vs. detectability (TPR@1% FPR), non-reasoning LLMs across five schemes. Dashed line and grey band: unwatermarked mean and 95% CI; dotted vertical: TPR = 99%. LLAMA-3.1-70B, GEMMA-3-12B, and PHI-4-14B stay within the band across all schemes; the two specialised 70Bs collapse under complementary schemes; LLAMA-3.1-8B degrades broadly. labels stripped. The judge records the correctness configur… view at source ↗
Figure 3
Figure 3. Figure 3: MedXpertQA-MM accuracy vs. detectability, seven VLMs; conventions as in Figure [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of watermarking on visual un￾derstanding. We show the difference between wa￾termarked and unwatermarked reasoning in the av￾erage number of supported (top) and contradicted (bottom) visual claims on MedXpertQA. Harm grows with strength, but the trajec￾tory is scheme-specific. We find in App. C and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: MedQA: generation-perplexity shift vs. watermark detectability, six non-reasoning LLMs. ∆ PPL = PPLwm−PPLbase. Monotonic in strength on every model×scheme cell; absolute magnitude spans an order of magnitude (note differing y-axes). LLAMA-3.1-8B has the steepest increases; GEMMA-3-12B and ULTRAMEDICAL-70B show contained shifts despite their sharply different accuracy robustness. 19 [PITH_FULL_IMAGE:figure… view at source ↗
Figure 6
Figure 6. Figure 6: MedXpertQA-MM: generation-perplexity shift vs. watermark detectability, seven VLMs. Same overall shape as MedQA; KGW and SynthID drive the steepest high-TPR increases. LINGSHU-32B and LINGSHU-7B show pronounced jumps at the highest strengths despite minimal accuracy changes (cf [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Audit outcomes vs. watermark detectability on PHI-4-14B (MedQA, top) and MEDGEMMA-4B (MedXpertQA-MM, bottom) under KGW (blue) and SynthID (purple). Each point is one (scheme, strength); dashed line and grey band give the unwatermarked baseline mean and 95%-bootstrap CI; dotted vertical marks TPR = 99%. Top: per-response confident-error rate restricted to right-letter responses (left); pairwise vignette-fab… view at source ↗
Figure 8
Figure 8. Figure 8: Pairwise vignette-fabrication ratio per scheme (MedQA, log scale). WM/Base rate ratio per (model, scheme) cell; dashed line at 1 (no effect). Cells with very low baseline rates have wide CIs. AAR DipMark KGW PPL SynthID 10 0 10 1 Fab ratio (WM / Base) Gemma 4 31B (FabBase=0.1%) AAR DipMark KGW PPL SynthID 10 0 10 1 Lingshu 7B (FabBase=5.1%) AAR DipMark KGW PPL SynthID 10 0 Lingshu 32B (FabBase=2.3%) AAR Di… view at source ↗
Figure 9
Figure 9. Figure 9: Pairwise vignette-fabrication ratio per scheme (MedXpertQA-MM, log scale). Con￾ventions as in [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-scheme ∆ in image-supported and image-contradicted visual claims, all 7 VLMs (MedXpertQA-MM). All-VLM companion to [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Pairwise mean Full WM–reference score on six reasoning-quality axes, four rea￾soning models × three KGW strengths. Blue: Full WM scored against the unwatermarked Base response; orange: Full WM scored against the FA-only response. Per-question scores are integers in {−2, −1, 0, +1, +2} (rubric in Sec. F.2.3); panel values are sample means. Negative bar = Full WM judged worse than the reference. Error bars:… view at source ↗
Figure 12
Figure 12. Figure 12: Clinician validation panel – harm rate vs. watermark detectability. Each watermarked [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Branching at the next decision token (Q2, PSGN). The prefix describes FSGS-like fea￾tures that are inconsistent with the correct diagnosis (PSGN). Unwatermarked, the model recognises the misdirection and overrides the visual cue using the clinical context (recent streptococcal infection, oedema, haematuria), returning the correct answer. Under PPL ε=0.5 watermarking, the perturbed next-token distribution … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 4 canonical work pages

  1. [1]

    2026 physician survey on augmented intelligence

    American Medical Association. 2026 physician survey on augmented intelligence. https:// www.ama-assn.org/system/files/physician-ai-sentiment-report.pdf , 2026. Accessed: 2026-04-30

    2026

  2. [2]

    Gowda, Meer C

    Anirudh S. Gowda, Meer C. Chhabria, and Ryan C. Lam. Use of large language models on radiology reports: A scoping review.Journal of the American College of Radiology, 23(3): 437–454, 2026. doi: 10.1016/j.jacr.2025.10.005. Published online 2025

    work page doi:10.1016/j.jacr.2025.10.005 2026

  3. [3]

    Weissman, Toni Mankowitz, and Genevieve P

    Gary E. Weissman, Toni Mankowitz, and Genevieve P. Kanter. Unregulated large language models produce medical device-like output.npj Digital Medicine, 8(1):148, 2025. doi: 10. 1038/s41746-025-01544-y

    2025

  4. [4]

    Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (AI Act)

    European Union. Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union, L 2024/1689, 2024

    2024

  5. [5]

    Second draft code of practice on marking and labelling of AI-generated content

    European Commission. Second draft code of practice on marking and labelling of AI-generated content. https://digital-strategy.ec.europa.eu/en/library/ commission-publishes-second-draft-code-practice-marking-and-labelling-ai-generated-content ,

  6. [6]

    WaterBench: Towards holistic evaluation of watermarks for large language models

    Shangqing Tu, Yuliang Sun, Yushi Bai, Jifan Yu, Lei Hou, and Juanzi Li. WaterBench: Towards holistic evaluation of watermarks for large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://ac...

    2024

  7. [7]

    Adams, and Keno K

    Johannes Moll, Markus Graf, Tristan Lemke, Nicolas Lenhart, Daniel Truhn, Jean-Benoit Delbrouck, Jiazhen Pan, Daniel Rueckert, Lisa C. Adams, and Keno K. Bressem. Evaluating reasoning faithfulness in medical vision-language models using multimodal perturbations. In Proceedings of Machine Learning for Health (ML4H), volume 297 ofProceedings of Machine Lear...

    arXiv 2025

  8. [8]

    Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicine, 30:2613–2622, 2024

    Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, and Daniel Rueckert. Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicine, 30:2613–2622, 2024. doi: 10.1038/s41591-024-03097-1

    doi:10.1038/s41591-024-03097-1 2024

  9. [9]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    2021

  10. [10]

    Siun Kim and Hyung-Jin Yoon. Questioning our questions: How well do medical QA bench- marks evaluate clinical capabilities of language models? InProceedings of the 24th Workshop on Biomedical Language Processing (BioNLP), pages 274–296, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.bionlp-1.24

    work page doi:10.18653/v1/2025.bionlp-1.24 2025

  11. [11]

    Hu et al

    Y . Hu et al. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 11

    2024

  12. [12]

    GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical AI.arXiv preprint arXiv:2408.03361, 2024

    Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, et al. GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical AI.arXiv preprint arXiv:2408.03361, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    MedXpertQA: Benchmarking expert-level medical reasoning and understanding

    Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. MedXpertQA: Benchmarking expert-level medical reasoning and understanding. InProceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 80961–80990. PMLR, 2025. arXiv:2...

    Pith/arXiv arXiv 2025

  14. [14]

    Right prediction, wrong reasoning: Uncovering LLM misalignment in RA disease diagnosis.arXiv preprint arXiv:2504.06581, 2025

    Umakanta Maharana, Sarthak Verma, Avarna Agarwal, Prakashini Mruthyunjaya, Dwarikanath Mahapatra, Sakir Ahmed, and Murari Mandal. Right prediction, wrong reasoning: Uncovering LLM misalignment in RA disease diagnosis.arXiv preprint arXiv:2504.06581, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Med-HALT: Medical domain hallucination test for large language models

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-HALT: Medical domain hallucination test for large language models. InProceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 314–334, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.conll-1.21

    doi:10.18653/v1/2023.conll-1.21 2023

  16. [16]

    Clinical large language model evaluation by expert review (CLEVER): Framework development and validation.JMIR AI, 4(1):e72153, 2025

    Veysel Kocaman, Mustafa Aytu ˘g Kaya, Andrei Marian Feier, and David Talby. Clinical large language model evaluation by expert review (CLEVER): Framework development and validation.JMIR AI, 4(1):e72153, 2025. doi: 10.2196/72153

    work page doi:10.2196/72153 2025

  17. [17]

    Arora, Jason Wei, Rebecca S

    Rahul K. Arora, Jason Wei, Rebecca S. Hicks, et al. HealthBench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. A survey of text watermarking in the era of large language models.ACM Computing Surveys, 57:47:1–47:36, 2024. doi: 10.1145/3691626. URLhttps://arxiv.org/abs/2312.07913

    Pith/arXiv arXiv 2024

  19. [19]

    A watermark for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. InInternational Conference on Machine Learning, pages 17061–17084. PMLR, 2023

    2023

  20. [20]

    A unified framework for llm watermarks, 2026

    Thibaud Gloaguen, Robin Staab, Nikola Jovanovi´c, and Martin Vechev. A unified framework for llm watermarks, 2026. URLhttps://arxiv.org/abs/2602.06754

    arXiv 2026

  21. [21]

    Dipmark: A stealthy, efficient and resilient watermark for large language models

    Yihan Wu, Zhengmian Hu, Hongyang Zhang, and Heng Huang. Dipmark: A stealthy, efficient and resilient watermark for large language models. 2023

    2023

  22. [22]

    Watermarking of large language models

    Scott Aaronson. Watermarking of large language models. InWorkshop on Large Language Models and Transformers, Simons Institute, UC Berkeley, 2023

    2023

  23. [23]

    Scalable watermarking for identifying large language model outputs.Nature, 634(8035):818–823, 2024

    Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. Scalable watermarking for identifying large language model outputs.Nature, 634(8035):818–823, 2024

    2024

  24. [24]

    Robust distortion- free watermarks for language models

    Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion- free watermarks for language models. 2024

    2024

  25. [25]

    Enhancing watermarking quality for LLMs via contextual generation states awareness.arXiv preprint arXiv:2506.07403, 2025

    Peiru Yang, Xintian Li, Wanchun Ni, Jinhua Yin, Huili Wang, Guoshun Nan, Shangguang Wang, Yongfeng Huang, and Tao Qi. Enhancing watermarking quality for LLMs via contextual generation states awareness.arXiv preprint arXiv:2506.07403, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Who wrote this code? watermarking for code generation

    Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, 2024

    2024

  27. [27]

    WatME: Towards lossless watermarking through lexical redundancy

    Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, and Kam-Fai Wong. WatME: Towards lossless watermarking through lexical redundancy. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9166–9180, Bangkok, Thailand, August 2024. Association for Computational Linguistic...

    work page doi:10.18653/v1/2024.acl-long.496 2024

  28. [28]

    Distilling the thought, watermarking the answer: A principle semantic guided watermark for reasoning large language models.OpenReview, 2026

    Shuliang Liu, Xingyu Li, Hongyi Liu, Dong Fang, Duan Bingchen, Zheng Qi, Lingfeng Su, and Xuming Hu. Distilling the thought, watermarking the answer: A principle semantic guided watermark for reasoning large language models.OpenReview, 2026. URL https: //openreview.net/forum?id=T6NVogsXCZ. ICLR 2026 Poster

    2026

  29. [29]

    Factuality beyond coherence: Evaluating LLM watermarking methods for medical texts

    Rochana Prih Hastuti, Rian Adam Rajagede, Mansour Al Ghanim, Mengxin Zheng, and Qian Lou. Factuality beyond coherence: Evaluating LLM watermarking methods for medical texts. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15129–15147, Suzhou, China, 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025. fin...

    arXiv 2025

  30. [30]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences

    Ankit Pal and Malaikannan Sankarasubbu. OpenBioLLMs: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/ Llama3-OpenBioLLM-70B, 2024

    2024

  34. [34]

    UltraMedical: Building specialized generalists in biomedicine

    Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Jin-Fang Hu, Zhiyuan Liu, and Bowen Zhou. UltraMedical: Building specialized generalists in biomedicine. InAdvances in Neural Infor- mation Processing Systems 37: Datasets and Benchmarks Track (NeurIPS), 2024. Spotlight; arX...

    Pith/arXiv arXiv 2024

  35. [35]

    DeepSeek-R1: Incentivizing reason- ing capability in LLMs via reinforcement learning.Nature, 645:633–638, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1: Incentivizing reason- ing capability in LLMs via reinforcement learning.Nature, 645:633–638, 2025. doi: 10.1038/s41586-025-09422-z

    doi:10.1038/s41586-025-09422-z 2025

  36. [36]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5. Blog post

    2026

  37. [37]

    Qwen3 technical report.arXiv preprint, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint, 2025

    2025

  38. [38]

    Gemma 4: Expanding the gemmaverse with Apache 2.0, April 2026

    Google Open Source. Gemma 4: Expanding the gemmaverse with Apache 2.0, April 2026. URL https://opensource.googleblog.com/2026/03/ gemma-4-expanding-the-gemmaverse-with-apache-20.html . Google Open Source Blog

    2026

  39. [39]

    Lingshu: A generalist foun- dation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Cheng- hao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foun- dation model for unified multimodal medical understanding and reasoning.arXiv preprint arXiv:2506.07044, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    MedGemma technical report.arXiv preprint arXiv:2507.05201, 2025

    Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. MedGemma technical report.arXiv preprint arXiv:2507.05201, 2025. 13 A Additional Experimental Details A.1 Watermarking Scheme Details All five schemes share a common structure: at each token posit...

    Pith/arXiv arXiv 2025

  41. [42]

    the answer is X

    draws (gu ∈ {0,1}), so that in expectation half of the tokens are green. A constant logit bias δ is added to the green-list tokens before sampling, yielding the soft logit-bias watermark q(g)∝pexp(δ g).(1) This is the soft Kirchenbauer scheme; we do not use the hard variant that excludes red-list tokens entirely. Detection tests whether the proportion of ...

    2000

  42. [43]

    Correctness configuration:BOTH_CORRECT,BOTH_WRONG_SAME_ANSWER,32 BOTH_WRONG_DIFF_ANSWER,R1_ONLY_CORRECT,R2_ONLY_CORRECT, or33 ONE_OR_BOTH_NONE.34

  43. [44]

    configuration

    Diagnostic interpretation divergence(NONE/MINOR/MAJOR): whether the two responses35 make mutually exclusive claims about the same clinical data (e.g., different mechanism attribu-36 tions, image findings, or value interpretations).37 Output schema (abbreviated).38 32 Output JSON - pairwise divergence audit { "configuration": { "response _1_answer", "respo...

  44. [2026]

    Accessed: 2026-04-30

    Published: 2026-03-05. Accessed: 2026-04-30

    2026