Pith. sign in

REVIEW 3 major objections 5 minor 33 references

Robustness tests for biomedical foundation models should tailor to specifications

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that robustness tests for biomedical foundation models should be chosen from task-specific priority specifications, not generic distance-bounded threat models.

desk verdict A well-scoped position piece that usefully frames robustness evaluation as a specification problem; the load-bearing conversion step is asserted rather than shown, but the framework is solid enough to deserve a serious referee. read the letter →

arxiv 2502.10374 v3 pith:DDMFV3IF submitted 2025-02-14 cs.SE cs.CY

R. Patrick Xian , Noah R. Baker , Tom David , Qiming Cui , A. Jay Holmgren , Stefan Bauer , Madhumita Sushil , Reza Abbasi-Asl This is my paper · ORCID
classification cs.SEcs.CY
keywords robustnesstestingbiomedicalfoundationmodelsdistributionshifttestcaseprioritizationAIregulationmodellifecycleadversarialclinical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that robustness testing of biomedical foundation models is misaligned with how these models actually fail. Generic distance-bounded adversarial tests and consistent multi-dataset performance do not tell whether a model will survive realistic, task-relevant disruptions. The authors propose a robustness specification: for each target task, a curated list of priority scenarios such as drug interactions, scanner artifacts, typos, and off-topic requests, broken into operationalizable units that are each convertible into a small number of quantitative tests with guarantees. If correct, this gives developers, users, and regulators a common standard for what 'robust' means for a given application and replaces ad hoc evaluations. A survey of over 50 biomedical foundation models motivates the proposal by finding that most models lack rigorous robustness assessments.

What carries the argument

The central object is the robustness specification, defined as the collection of priority scenarios that demand testing for an individual task. Each priority is meant to be an operationalizable unit convertible into a small number of quantitative tests with guarantees, and the specification can be updated as model capabilities and risk assessments change. The paper contrasts priority-based test design, which uses realistic artifacts such as typos, entity substitution, scanner differences, and imaging artifacts, with threat-based design, which uses distance bounds such as edit distance for text or Euclidean distance for images; the two overlap, but priority-based tests are argued to reflect deployment realities more directly. Two worked examples in Box 2 show how a specification mixes domain-specific and general failure scenarios.

What would settle it

Take one of the paper's own specifications, such as the pharmacy chatbot with its nine priority scenarios, and attempt to convert each priority into a fixed, small number of quantitative tests with a stated guarantee. If any priority, for example the refusal behavior for non-OTC drugs, requires an unbounded family of prompts or has no clear metric, the framework's core claim fails. A concrete check would be to count the test inputs needed to cover the priority 'up to 12 turns of conversation' under a guarantee; combinatorial growth with turn count would falsify the 'small number' assertion.

Watch

Extended reading notes

Core claim

The paper's central claim is that a task-dependent robustness specification can replace generic robustness testing for biomedical foundation models. The specification collects the degradation mechanisms and performance metrics that matter for a specific task and breaks robustness evaluation into operationalizable units, where each unit can be converted into a small number of quantitative tests with guarantees. The authors support the need for this by surveying over 50 models and reporting that about 31.4% have no robustness assessment and that the most common evidence, consistent performance across datasets, does not amount to a robustness guarantee because dataset relationships are unknown. They illustrate the approach with two specifications, a pharmacy chatbot for over-the-counter medication and an MRI radiology report copilot, showing how realistic priorities overlap with but are not captured by distance-bounded threat models.

Load-bearing premise

The load-bearing premise is that every priority in a robustness specification can be converted into a small number of quantitative tests with guarantees; the paper asserts this conversion but does not provide a worked procedure or demonstration of it.

Editorial extensions

If this is right

  • Robustness evaluation can be standardized across the model lifecycle: the same specification guides development-stage checks, deployment tests, and updates.
  • Abstract regulatory requirements such as 'resilience' and 'cybersecurity' become concrete test procedures tied to quantitative guarantees.
  • Existing specialized robustness tests, including typo attacks, entity substitution, and imaging artifacts, are reusable building blocks within a priority-based specification.
  • Specifications can be updated as models gain capabilities or risk assessments change, so evaluation keeps pace with deployment.
  • Compound AI systems can be tested per subsystem, with specifications for each component and for part-whole interactions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If regulators adopted such specifications, they could name a concrete test suite instead of vague resilience requirements; the paper hints at but does not spell out this translation.
  • The same specification format could support post-market surveillance by mapping reported failures back to specific priority scenarios, revealing which degradation mechanisms actually occur in practice.
  • A natural experiment would run both threat-based and priority-based tests on the same model family and compare which failures are caught; the paper predicts substantial non-overlap, with priority tests catching more clinically salient errors.
  • Specifications would need versioning because priorities change as models gain capabilities; the paper notes accommodation of capability expansion but does not design the versioning mechanism.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript is a position piece on robustness evaluation for biomedical foundation models (BFMs). The authors survey over 50 BFMs and report that roughly a third contain no robustness assessments, and that the most common evidence of robustness is consistent performance across datasets, which they argue is an unreliable proxy. They propose that robustness tests be designed from task-dependent 'robustness specifications': a collection of priority scenarios that are, according to the manuscript, broken down into operationalizable units each convertible into a small number of quantitative tests with guarantees. Two example specifications are given (a pharmacy chatbot and an MRI radiology report copilot), and the proposal is discussed in relation to knowledge integrity, population structure, uncertainty awareness, compound AI systems, and regulatory frameworks such as the EU AI Act and the US NIST AI Risk Management Framework.

Significance. If the conversion step were made concrete, the proposal could provide a practical bridge between high-level regulatory robustness requirements and test implementation, and could support standardization across model types and lifecycles. Strengths of the paper include the public release of the underlying survey data, the use of realistic biomedical artifacts (imaging artifacts, drug interactions) rather than generic distance-bounded perturbations, the explicit connection to existing work on test-case prioritization, and the attempt to cover both monolithic and compound systems. The paper is clearly written, and the two example specifications are useful anchors. However, the central operational claim is not demonstrated, and the survey statistics are presented without a methodology section, which limits the force of the motivation.

major comments (3)
  1. [Specifying robustness by priorities] The sentence 'The specification breaks down robustness evaluation into operationalizable units such that each is convertible into a small number of quantitative tests with guarantees' is the load-bearing claim of the paper, but it is only asserted. The terms 'operationalizable unit', 'small number', and 'guarantee' are never defined, and no worked conversion is provided. In Box 2, none of the 19 priorities across the two examples is connected to a concrete test protocol. Item 7 of the pharmacy chatbot specification ('Limited prescription authority (will refuse the task if asked for non-OTC or prescription drugs)') is a behavioral constraint, not an input perturbation; converting it into a quantitative test requires deciding the query distribution, the acceptable refusal rate, the coverage of drug classes, and the consequences of an incorrect non-refusal, none of which is specified. Without a mapping from each priority to a metric, a perturbation set, and a threshold, the proposal cannot standardize evaluation and is not distinguishable from an ad hoc checklist.
  2. [The robustness evaluation challenges (Fig. 1, Supplementary Data 1)] The motivation rests on the survey statistics in this section, but the methodology behind 'over 50 existing BFMs' is not described. The manuscript does not state the inclusion/exclusion criteria, the search date, the definition of 'robustness assessment', or the inter-rater reliability of the classification. With a sample size of roughly 50, the reported proportions (31.4%, 33.3%, 5.9%, 3.9%, 9.8%) also carry wide confidence intervals; the absence of any measure of uncertainty is misleading. The authors should either present the survey as an illustrative convenience sample (and say so) or provide a reproducible protocol together with the per-model coding in Supplementary Data 1. As written, the empirical foundation for the proposal is not checkable.
  3. [Robustness framework limitations; Specifying robustness by priorities] The manuscript repeatedly attaches 'guarantees' to the proposed tests (e.g., 'quantitative tests with guarantees') without specifying the nature of the guarantee, despite the acknowledgement in 'Robustness framework limitations' that existing theoretical guarantees are asymptotic and 'don't necessarily translate into effective robustness.' If the guarantee is statistical, what coverage and confidence are intended? If it is worst-case, how is the adversary or input set defined for behavioral priorities such as refusals and off-topic handling? The paper should state the type of guarantee it envisages and discuss the assumptions needed (e.g., a bounded deployment query distribution) for each class of priority.
minor comments (5)
  1. [Box 2] The two example specifications are presented with the phrase 'has been extensively tested for robustness under the following scenarios', but these are proposed test plans, not reports of tests that have already been run; the wording should be changed to 'would be tested' or 'should be tested' to avoid implying existing results.
  2. [Fig. 1a] The treemap shows topical areas, but the caption does not define how models were assigned to topics; the categorization scheme should be stated, and the number of surveyed models should be given in the caption (e.g., N=53) rather than only 'over 50'.
  3. [Supplementary Data 1] The data file is referenced but not included in the arXiv posting; ensure that the supplement is actually provided to readers and that the column definitions are documented so that the survey classifications can be audited.
  4. [Specifying robustness by priorities] The analogy to test-case prioritization [7] is useful, but the paper should note a key difference: in regression testing, test cases already exist, whereas the proposal requires creating tests from priorities; this makes the conversion step more central than in the original software engineering setting.
  5. [Bridging policy with implementation] The statement that the EU AI Act and the NIST framework 'recognize the relation between natural and adversarial notions of robustness' could be supported with the specific recitals or sections beyond the quoted Article 15 and the NIST measure process; currently the generalization goes slightly beyond the cited excerpts.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity: the framework is a proposal, not a derivation; the sole self-citation is illustrative, not load-bearing.

full rationale

This is a perspective/comment paper with no quantitative derivation, no fitted parameters, and no predictive claims, so the standard circularity failure modes do not arise. Its central assertion, that a robustness specification 'breaks down robustness evaluation into operationalizable units such that each is convertible into a small number of quantitative tests with guarantees,' is a design proposal for future test design, not a claim derived from the robustness concepts it lists. The two Box 2 examples are illustrative, not demonstrations, and the acknowledged absence of a worked conversion procedure is a completeness or falsifiability weakness, not circularity. The only self-citation, ref. [18], is used as one among several examples of existing entity-substitution tests ('substituting biomedical entities [18]' to induce erroneous model behaviors) and is not load-bearing: the framework does not depend on the truth of that cited work. No equations, no fitted-input-as-prediction, no uniqueness argument imported from the authors' prior work, and no ansatz smuggled in via self-citation. Accordingly, no circular step is identified; the score reflects only the presence of a minor non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters and no physical or computational entities. It relies on several domain assumptions, the most load-bearing being that priority scenarios can be converted into quantitative tests with guarantees. The motivating survey is used as evidence but its selection criteria are not specified.

assumptions (4)
  • domain assumption Biomedical foundation models will eventually become integral to healthcare automation.
    Opening paragraph of the introduction asserts this trajectory; it motivates the need for standardized robustness testing but is not proven.
  • domain assumption Test case prioritization from software engineering transfers to ML robustness evaluation.
    Section 'Specifying robustness by priorities' states the inspiration from ref [7] without establishing that the transfer preserves cost-effectiveness guarantees.
  • ad hoc to paper Each priority scenario in a specification is convertible into a small number of quantitative tests with guarantees.
    Section 'Specifying robustness by priorities' makes this the operational core of the framework; no procedure or worked conversion is provided.
  • domain assumption Community endorsement is a necessary condition for adoption of the framework.
    The implementation considerations argue that specifications should seek community endorsement, but no evidence is given that consensus will emerge or that it is required for success.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robustness tests for biomedical foundation models should tailor to specifications." pith.science (2026). https://pith.science/paper/DDMFV3IF

@misc{pith2026250210374,
  author       = {Pith},
  title        = {Pith review of: Robustness tests for biomedical foundation models should tailor to specifications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDMFV3IF}},
  note         = {Machine review of arXiv:2502.10374}
}
read the original abstract

The rise of biomedical foundation models creates new hurdles in model testing and authorization, given their broad capabilities and susceptibility to complex distribution shifts. We suggest tailoring robustness tests according to task-dependent priorities and propose to integrate granular notions of robustness in a predefined specification to guide implementation. Our approach facilitates the standardization of robustness assessments in the model lifecycle and connects abstract AI regulatory frameworks with concrete testing procedures.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 28 canonical work pages

  1. [1]

    Tocchetti, A. et al. A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities. ACM Comput. Surv. 57, 141:1–141:38 (2025). URL https://dl.acm.org/doi/10. 1145/3665926

    work page 2025

  2. [2]

    Kostick-Quenet, K. M. & Gerke, S. AI in the hands of imperfect users. npj Digital Medicine 5, 197:1– 6 (2022). URL https://www.nature.com/articles/s41746-022-00737-z. Publisher: Nature Publishing Group

    work page 2022

  3. [3]

    Ladkin, P. B. Robustness of Software. Digital Evidence and Electronic Signature Law Review 17, 15–24 (2020). URL https://heinonline.org/HOL/P?h=hein.journals/digiteeslr17&i=17

    work page 2020

  4. [4]

    J., Tazbaz, T

    Warraich, H. J., Tazbaz, T. & Califf, R. M. FDA Perspective on the Regulation of Artificial Intelli- gence in Health Care and Biomedicine. JAMA 333, 241–247 (2025). URL https://doi.org/10.1001/ jama.2024.21451

    work page arXiv 2025

  5. [5]

    C., Kather, J

    Freyer, O., Wiest, I. C., Kather, J. N. & Gilbert, S. A future role for health applications of large language models depends on regulators enforcing safety standards.The Lancet Digital Health 6, e662– e672 (2024). URL https://www.thelancet.com/journals/landig/article/PIIS2589-7500(24)00124-9/ fulltext. Publisher: Elsevier

    work page 2024

  6. [6]

    & Magrabi, F

    Lyell, D., Wang, Y., Coiera, E. & Magrabi, F. More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. Journal of the American Medical Informatics Association 30, 1227–1236 (2023). URL https://doi.org/10.1093/jamia/ocad065

    work page doi:10.1093/jamia/ocad065 2023

  7. [7]

    & Harrold, M

    Rothermel, G., Untch, R., Chu, C. & Harrold, M. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering 27, 929–948 (2001). URL https://ieeexplore.ieee.org/ document/962562

    work page 2001

  8. [8]

    & Paul, S

    Chen, P.-Y., Liu, S. & Paul, S. Foundational Robustness of Foundation Models . NeurIPS Tutorial (2022). URL https://research.ibm.com/publications/foundational-robustness-of-foundation-models

    work page 2022

Show all 33 references
  1. [9]

    Qi, X. et al. AI Risk Management Should Incorporate Both Safety and Security (2024). URL http://arxiv.org/abs/2405.19524. ArXiv:2405.19524 [cs]. 14

    2024 arXiv

  2. [10]

    Yang, J. et al. Poisoning medical knowledge using large language models.Nature Machine Intelligence 6, 1156–1168 (2024). URL https://www.nature.com/articles/s42256-024-00899-3. Publisher: Nature Publishing Group

    2024

  3. [11]

    Jin, R., Huang, C.-Y., You, C. & Li, X. Backdoor Attack on Unpaired Medical Image-Text Foundation Models: A Pilot Study on MedCLIP . 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 272–285 (2024). URL https://ieeexplore.ieee.org/document/10516621

    2024

  4. [12]

    Chowdhury, A. G. et al. Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models (2024). URL http://arxiv.org/abs/2403.04786. ArXiv:2403.04786 [cs]

    2024

  5. [13]

    & Chawla, S

    Karunanayake, N., Gunawardena, R., Seneviratne, S. & Chawla, S. Out-of-Distribution Data: An Acquaintance of Adversarial Examples - A Survey. ACM Comput. Surv. 57, 210:1–210:40 (2025). URL https://dl.acm.org/doi/10.1145/3719292

    2025 doi

  6. [14]

    Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine 30, 2613–2622 (2024). URL https://www.nature.com/articles/ s41591-024-03097-1. Publisher: Nature Publishing Group

    2024

  7. [15]

    Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nature Medicine 31, 77–86 (2025). URL https://www.nature.com/articles/ s41591-024-03328-5. Publisher: Nature Publishing Group

    2025

  8. [16]

    Han, T. et al. Medical large language models are susceptible to targeted misinformation attacks. npj Digital Medicine 7, 288:1–9 (2024). URL https://www.nature.com/articles/s41746-024-01282-7. Publisher: Nature Publishing Group

    2024

  9. [17]

    & Wang, X

    Yan, Q., He, X., Yue, X. & Wang, X. E. Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA . Findings of the Association for Compu- tational Linguistics: ACL 2025, 19188–19205 (Association for Computational Linguistics, Vien...

    2025

  10. [18]

    Xian, R. P. et al. Assessing biomedical knowledge robustness in large language models by query- efficient sampling attacks. Transactions on Machine Learning Research (2024). URL https: //openreview.net/forum?id=pvol5JyVYB

    2024

  11. [19]

    Boone, L. et al. ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI. NeuroImage 278, 120289 (2023). URL https: //www.sciencedirect.com/science/article/pii/S1053811923004408

    2023

  12. [20]

    & Ghassemi, M

    Yang, Y., Zhang, H., Katabi, D. & Ghassemi, M. Change is hard: a closer look at subpopulation shift . Proceedings of the 40th International Conference on Machine Learning, 39584–39622 (Honolulu, Hawaii, USA, 2023). 15

    2023

  13. [21]

    Chandu, K. et al. CertainlyUncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness. The Thirteenth International Conference on Learning Representations (2025). URL https://openreview.net/forum?id=cQ25MQQSNI

    2025

  14. [22]

    Wang, W. et al. A Survey of LLM-based Agents in Medicine: How far are we from Baymax? Find- ings of the Association for Computational Linguistics: ACL 2025, 10345–10359 (Association for Computational Linguistics, Vienna, Austria, 2025). URL https://aclanthology.org/2025.findin...

    2025

  15. [23]

    Mukherjee, S. et al. Polaris: A Safety-focused LLM Constellation Architecture for Healthcare (2024). URL http://arxiv.org/abs/2403.13313. ArXiv:2403.13313 [cs]

    2024 arXiv

  16. [24]

    C., Barr-Walker, J

    Radcliffe, K., Lyson, H. C., Barr-Walker, J. & Sarkar, U. Collective intelligence in medical decision- making: a systematic scoping review. BMC Medical Informatics and Decision Making 19, 158 (2019). URL https://doi.org/10.1186/s12911-019-0882-0

    2019 doi

  17. [25]

    & Anderljung, M

    Koessler, L., Schuett, J. & Anderljung, M. Risk thresholds for frontier AI (2024). URL http: //arxiv.org/abs/2406.14713. ArXiv:2406.14713

    2024 arXiv

  18. [26]

    Miller, R. A. in Diagnostic Decision Support Systems (ed.Berner, E. S.) Clinical Decision Support Systems: Theory and Practice 181–208 (Springer International Publishing, Cham, 2016). URL https: //doi.org/10.1007/978-3-319-31913-1 11

    2016 doi

  19. [27]

    ROUGE: A Package for Automatic Evaluation of Summaries

    Lin, C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries . Text Summarization Branches Out, 74–81 (Association for Computational Linguistics, Barcelona, Spain, 2004). URL https://aclanthology.org/W04-1013/

    2004

  20. [28]

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: Evaluating Text Generation with BERT . International Conference on Learning Representations (2019). URL https://openreview.net/forum?id=SkeHuCVFDr

    2019

  21. [29]

    Yu, F. et al. Evaluating progress in automatic chest X-ray radiology report generation. Patterns 4, 100802 (2023). URL https://www.sciencedirect.com/science/article/pii/S2666389923001575

    2023

  22. [30]

    & McDonald, R

    Maynez, J., Narayan, S., Bohnet, B. & McDonald, R. On Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919 (Association for Computational Linguistics, Stroudsburg, PA, USA, ...

    1906

  23. [31]

    The EU AI Act and the Wager on Trustworthy AI

    Bellog ´ ın, A.et al. The EU AI Act and the Wager on Trustworthy AI. Commun. ACM 67, 58–65 (2024). URL https://dl.acm.org/doi/10.1145/3665322. 16

    2024 doi

  24. [32]

    & Finck, M

    Nolte, H., Rateike, M. & Finck, M. Robustness and Cybersecurity in the EU Artificial Intelligence Act. F AccT ’25, 283–295 (Association for Computing Machinery, New York, NY, USA, 2025). URL https://dl.acm.org/doi/10.1145/3715275.3732020

    2025

  25. [33]

    A., Mitchell, C., Walton, M

    Rawal, A., Johnson, K. A., Mitchell, C., Walton, M. & Nwankwo, D. Responsible Artificial Intel- ligence (RAI) in US Federal Government : Principles, Policies, and Practices. NeurIPS 2024 Workshop on Regulatable ML (2024). URL https://openreview.net/forum?id=OrwvUD7p5q. 17

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.