Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs

Raj Krishnan Vijayaraj

REVIEW 4 major objections 5 minor 96 references

Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes LDFR, a geometry-aware metric that exposes latent diagnostic fragility in clinical LLMs: small clinically plausible edits can flip the embedded diagnosis while surface similarity stays high.

desk verdict The paper's central metric LDFR is not computed as defined: at zero perturbation it should be zero, not 0.91, and the reported trend is backwards for a flip rate, so the main claim of latent fragility is unsupported as written. read the letter →

arxiv 2507.21188 v1 pith:RENGBHLK submitted 2025-07-27 cs.LG cs.AI

Raj Krishnan Vijayaraj This is my paper · ORCID

classification cs.LGcs.AI

keywords clinicalLLMevaluationlatentspacerobustnessdiagnosticfragilityperturbationtestingembeddingdecisionboundariesPCAprobingBERTScoregapLDFR

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that clinical large language models can be latently fragile even when they look robust on standard text metrics. To show this, it introduces LAPD, a perturbation-and-probing pipeline, and LDFR, a metric that counts how often a small clinically grounded edit—masking a symptom, negating a finding, swapping a synonym, or nudging a numeric value—moves the model's embedding across a diagnostic boundary. The measured gap is stark: at 25% entity masking, BERTScore stays above 0.89 while the latent diagnostic signal falls from about 91% to about 55%, and the pattern holds on 90 real discharge summaries from MIMIC-IV as well as on synthetic notes. If the paper is right, high benchmark scores and high surface similarity are not evidence that a clinical assistant will reason stably, and geometry-aware auditing should be part of pre-deployment checks.

What carries the argument

The load-bearing object is the LDFR probe: embeddings are extracted from frozen encoders (ClinicalBERT for synthetic notes, Clinical-Longformer for real ones), reduced by PCA to the 90%-variance subspace, and a logistic-regression classifier trained on the unperturbed projections marks a linear diagnostic boundary. LDFR counts, per perturbation threshold, the fraction of perturbed embeddings whose boundary assignment differs from the unperturbed assignment, and the paper reports it as an agreement signal between probe and LLM. The complementary mechanism is dimensional collapse: under masking and negation, one or two PCA dimensions absorb up to 20–30% of total variance, so the embeddings compress into a narrow subspace where small edits cross the boundary. The structured perturbations themselves—masking, negation, synonym replacement, and numeric variation, applied at controlled intensities—are the stress tests that drive the geometry.

What would settle it

Have clinicians score a sample of perturbed notes whose embeddings crossed the boundary. If the majority of flips are judged clinically unwarranted, or if retraining the boundary on the LLM's own final-layer hidden states makes the flips disappear, then the fragility the paper reports is an artifact of the linear PCA probe rather than a property of the LLM.

Watch

Extended reading notes

Core claim

The paper's central claim is that robustness in clinical LLMs has a latent, geometric component that output-level metrics do not see. It defines LDFR as the proportion of perturbed samples whose latent representations cross a diagnostic decision boundary, with that boundary estimated by a logistic-regression probe trained on PCA-reduced embeddings of unperturbed notes. In the reported measurements, this probe tracks the LLM's own behavior: under entity masking the probe's agreement with the model's diagnoses falls from roughly 91% to 55% at only 25% perturbation intensity, while BERTScore remains above 0.89; negation produces a smaller but consistent decline; synonym substitution is model-dependent; and numerical edits leave both surface and latent metrics nearly flat. The paper reads the last result as evidence that these models lean on explicitly stated symptoms and underuse quantitative cues. The same fragility pattern appears on 90 real discharge notes from a MIMIC-IV-based benchmark, which the paper takes as evidence that LDFR captures a real deployment-relevant weakness rather than a synthetic-data artifact; the paper's own limitations section concedes that no clinician adjudicated whether the perturbation-induced flips are clinically warranted.

Load-bearing premise

The whole signal depends on a fitted line in embedding space standing in for the model's true diagnostic reasoning, and the paper concedes that no clinician adjudicated whether the measured flips are clinically warranted—if that line is misplaced, the flips describe the probe, not the model.

Editorial extensions

If this is right

High BERTScore, ROUGE-L, or entity F1 cannot certify a clinical LLM; the paper's measurements show surface similarity and diagnostic stability coming apart under masking and negation.
Entity masking is the most effective stressor, so an audit for clinical deployment should include symptom-omission edits before trusting a model's diagnosis.
LDFR can be computed from frozen embeddings without access to model weights or output logits, making it a practical model-agnostic screen across proprietary and open models.
Numerical perturbations leave latent structure intact, suggesting current clinical LLMs underuse vitals and lab values; training that forces quantitative grounding might close the gap.
The same fragility pattern on real discharge summaries implies that LDFR-based screening can be run on real documentation, not only on controlled synthetic notes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the paper's own limitation section concedes that no clinician adjudicated whether the perturbation-induced flips are clinically appropriate; if most flips turn out to be warranted, LDFR measures useful sensitivity rather than failure, which would change the recommended audit response from suppressing the model to flagging cases for human review.
Editorial extension: a natural check is to fit the boundary not with PCA plus logistic regression but with the LLM's own hidden states or a nonlinear classifier; if the flips vanish, the linear probe manufactured the apparent fragility.
Editorial extension: the dimensional-collapse result suggests a monitoring heuristic—track per-dimension variance of embeddings during deployment and alert when variance concentrates into a few axes—which could catch fragile inputs in real time.
Editorial extension: LDFR is label-agnostic, so the same boundary-crossing logic could be applied to other high-stakes text tasks whose outcome space is discrete, such as legal classification or radiology triage.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know about arXiv:2507.21188. First, the central metric, LDFR, is not computed as defined. Eq 4.3 defines it as a flip rate between the probe classifier's predictions on original and perturbed notes; at zero perturbation that rate must be zero. Table 1 reports 0.9125 at threshold zero for all four perturbation types, and the values decrease as perturbations increase. The text reads the decrease as 'latent agreement breaks down,' which is an agreement rate, not a flip rate. So the column labeled LDFR appears to be an agreement rate between the probe and the LLM, and the actual boundary-crossing metric is never reported. This is load-bearing, because every headline claim about latent fragility is expressed in terms of LDFR.

What the paper does well: the evaluation pipeline is thoughtfully assembled. It generates synthetic clinical notes from DDXPlus with forward and backward reasoning, applies four clinically motivated perturbation axes (masking, negation, synonym, numeric), tests five models, and includes 90 real DiReCT notes. The idea that surface metrics like BERTScore stay high while representations cross diagnostic boundaries is worth testing, and the variance decomposition analysis is a reasonable way to look for low-dimensional instability. The synthetic note generation is creative, and the human evaluation, while tiny, is a good instinct.

Where it falls down beyond the metric issue: the probe's decision boundary is only validated against the same LLMs under audit. Section 8 admits no clinician adjudicated whether perturbation-induced flips are clinically warranted, so the 'boundary' may be an artifact of the probe. The real-note evidence is thin: 90 notes, correlations as low as 0.16. There are also small internal inconsistencies (10-fold vs 5-fold cross-validation, 13.2 vs 83.5 entities per DiReCT note) that undermine confidence.

Bottom line: the direction is salvageable but the paper as written does not support its claims. I would not cite it, and I would not publish it in this form. If it comes to you as a referee, my recommendation is reject with constructive comments: redefine LDFR, recompute the tables, and add an external validation anchor. It might then become a useful workshop paper.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes LAPD, a geometry-aware framework for probing the latent robustness of clinical LLMs under four structured perturbation types (masking, negation, synonym replacement, and numerical variation). The central contribution is the Latent Diagnosis Flip Rate (LDFR), defined as the proportion of perturbed samples whose embeddings cross a PCA-based diagnostic decision boundary. Experiments on synthetic notes (DDXPlus) and 90 real notes (DiReCT/MIMIC-IV) claim that LDFR reveals fragility hidden by surface metrics such as BERTScore, and that masking and negation cause large latent shifts even when surface similarity remains high.

Significance. If valid, LDFR would offer a genuinely useful auditing signal for clinical LLMs, going beyond surface-level agreement to probe internal representation stability. The paper includes several positive elements: a concrete, reproducible pipeline; evaluation on both synthetic and real clinical notes; and an explicit acknowledgment of limitations in Section 8. However, the central metric is internally inconsistent as defined and as reported, and its validation is circular. These issues undermine the main claim, so the positive elements cannot currently be credited as supporting a sound contribution.

major comments (4)

[Section 4.3, Eq. (1) vs. Table 1] The definition LDFR(t) = (1/N) Σ_i I[d0_i ≠ dt_i] forces LDFR(0) = 0, because at threshold 0 the perturbed note is identical to the unperturbed note, making d0_i = dt_i for every i. Table 1 reports LDFR = 0.9125 at threshold 0.00 for all four perturbation types, and Table 2 reports values around 0.69. These numbers are impossible for the quantity defined in Eq. (1). The tables must be reporting a different metric, such as an agreement rate between the PCA classifier and the LLM, but this is not stated. This direct contradiction invalidates the quantitative foundation of the paper.
[Section 6.1, Section 7, Tables 1–2] If LDFR were a flip rate, one would expect it to increase with perturbation intensity under fragility, because stronger perturbations should drive more embeddings across the decision boundary. Tables 1 and 2 show the opposite: LDFR decreases monotonically (e.g., masking from 0.9125 to 0.3525). The text interprets this decrease as 'latent agreement breaks down' (Section 6.1) and states that 'the latent classifier no longer follows the LLM’s diagnosis reliably' (Section 7). That interpretation describes an agreement or alignment rate (1 − flip rate), not a flip rate. The reversed interpretation is load-bearing: the paper's headline claim that masking and negation trigger latent fragility is based on a quantity that behaves opposite to the definition.
[Section 4.2 and Section 8] The PCA-logistic probe is trained on unperturbed embeddings with ground-truth diagnosis labels, not on the target LLM's internal decision regions. Section 8 explicitly admits that clinician reviewers did not adjudicate whether the perturbation-induced flips are clinically warranted. The validation of LDFR is therefore circular: LDFR is validated by correlating it with the same LLM's output flips, but the LLM is the system under audit, and there is no external ground truth for whether the flips should occur. LDFR may be measuring probe–LLM misalignment rather than genuine latent fragility, and the manuscript offers no test to distinguish these alternatives.
[Section 4.2] The assumption that a logistic regression classifier on PCA-reduced embeddings faithfully represents the LLM's diagnostic decision boundary is untested. The probe is a separate model trained on dataset labels, so its boundary could reflect dataset statistics rather than the audited LLM's reasoning. A concrete test would be to compare the probe's boundary to per-model decision regions (e.g., by training probes on each model's own outputs) or to define LDFR using the LLM's own embeddings. Without such a check, the central 'fragility' signal may be an artifact of the probe rather than a property of the LLM.

minor comments (5)

[Section 4.2 vs. Section 5.4] Section 4.2 reports 10-fold cross-validation for the latent classifier, while Section 5.4 reports 5-fold cross-validation. These statements should be reconciled.
[Appendix A.6.3] The solver is listed as 'lgfbs'; this appears to be a typo for 'lbfgs'.
[Section 8] The phrase 'another direction illucidated' contains a typo ('illucidated' should be 'elucidated').
[Table 1 caption] The caption claims the table shows 'Boundary crossings in latent space predict real diagnostic flips', but the table reports Pearson/Spearman correlations and LDFR values; the operational link between these columns and 'real diagnostic flips' is never defined.
[Abstract] The term 'model-agnostic' is misleading: LDFR depends on the selected encoder (ClinicalBERT or Clinical-Longformer), the PCA threshold, and a classifier trained on a specific dataset, so the metric is not agnostic to these choices.

Circularity Check

2 steps flagged · score 7.0 of 10

LDFR as tabulated is not the flip rate defined in Eq. 4.3; it behaves as the agreement rate between the fitted PCA probe and the audited LLM's outputs, so the central fragility claim reduces to a probe-LLM agreement artifact rather than a measured latent boundary-crossing rate.

other [Section 4.3, Eq. (4.3); Tables 1-2; Section 6.1 and Section 7]
"We define LDFR as the proportion of perturbed samples whose latent representations cross the diagnostic boundary: LDFR(𝑡) = 1/𝑁 ∑ I[𝑑(𝑖)0 ≠ 𝑑(𝑖)𝑡] where 𝑑(𝑖)0 and 𝑑(𝑖)𝑡 are classifier predictions for unperturbed and perturbed notes, respectively."

Under this definition, threshold 0 compares each note with itself, so the indicator is identically 0 for every sample and LDFR(0)=0. Tables 1 and 2 instead report LDFR=0.9125 and about 0.69 at threshold 0.00, and Section 7 interprets the decreasing column as "the latent classifier no longer follows the LLM's diagnosis reliably." Thus the reported values are not boundary-flip rates; they are the agreement rates between the PCA probe and the LLM's output labels. Every headline result (masking/negation trigger large LDFR changes, surface metrics hide instability, generalization to DiReCT) is expressed in terms of this re-labeled agreement number.
fitted input called prediction [Section 6.5, Table 2; Section 4.3; Section 8]
"As shown in Table 2, correlations between LDFR and model predictions consistently drop as perturbations increase, confirming that our metric captures instability beyond synthetic settings."

The confirming evidence is the correlation of LDFR with the very model under audit: the LLM outputs serve as the "model predictions" that define the reference flips, and Section 8 concedes "we did not assess whether clinician reviewers agree with the diagnosis flips induced by perturbations." The logistic probe is fitted on unperturbed notes, so a declining agreement between that fixed probe and the perturbed model's outputs is the expected consequence of input drift, not an externally validated signal of fragility. The framework therefore validates its fragility claim against the same system whose stability it is supposed to test, making the prediction forced by the choice of reference rather than by clinician-adjudicated ground truth.

full rationale

The paper's central contribution, LDFR, is defined in Section 4.3 as a classifier flip rate between unperturbed and perturbed embeddings, which is identically zero at threshold 0. The reported LDFR values in Tables 1 and 2 are nonzero at threshold 0 and decrease monotonically under perturbation, and the text explicitly reads this as "the latent classifier no longer follows the LLM's diagnosis reliably." The quantity actually tabulated is therefore the agreement/alignment between the PCA probe and the audited LLM's outputs, not the defined flip rate. Because that same LLM's outputs are used as the reference for "real diagnostic flips," and because Section 8 states that clinician reviewers did not adjudicate perturbation-induced flips, the central empirical claim reduces to a fixed probe agreeing less with the audited model as inputs drift, which is an expected baseline behavior rather than a demonstrated latent fragility. No self-citation chain or imported uniqueness theorem is involved; the circularity is internal to the metric's definition, operationalization, and validation. If the LDFR column were recomputed as the Eq. 4.3 flip rate and validated against clinician-adjudicated outcomes, the framework could supply independent evidence, but as reported the headline results are forced by the metric's construction. Score 7 reflects that the central claim is substantially undermined by the definitional mismatch and self-referential validation, while leaving the underlying probe-plus-perturbation methodology as a potentially salvageable idea.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a chain of unverified proxies: LLM-generated notes stand in for real clinical text, a frozen BERT encoder stands in for each model's internal geometry, and a linear probe in a PCA subspace stands in for the LLM's diagnostic reasoning. The first two are acknowledged in the paper's own limitations; the third is not validated externally. No new physical entities are introduced, but the choice of the 90% variance threshold and the probe's regularization constant are hand-set and would alter LDFR values if changed.

free parameters (3)

PCA variance threshold = 0.90
Set from an elbow plot in Appendix A.9; controls the dimensionality of the space in which all flips are counted.
Logistic regression inverse-regularization C = 10.0
Reported in Appendix A.6.3; the boundary location and hence every LDFR flip depend on this choice.
Perturbation intensity thresholds = 0%, 25%, 50%, 75%, 100%
Experimental settings that define the LDFR trend, and the fraction of entities perturbed depends on an unspecified clinical NER entity count.

assumptions (4)

domain assumption PCA of note embeddings provides a faithful low-dimensional representation of semantic diagnostic content.
Used in Sections 4.2 and A.9; all flips are measured in the 90% variance subspace, so if this projection discards diagnostic signal, LDFR is meaningless.
domain assumption LLM-generated synthetic notes from DDXPlus are representative test inputs for evaluating clinical LLM robustness.
The test distribution is generated by LLM prompts, and Section 8 concedes synthetic notes may not reflect real-world clinical variability.
domain assumption A frozen ClinicalBERT or Clinical-Longformer embedding space is a valid geometry for auditing model-specific LLMs.
Acknowledged in Section 8 as a limitation: a fixed BERT space may misalign with model-specific representations.
ad hoc to paper The linear logistic decision boundary in PCA space is a faithful proxy for the LLM's internal diagnostic reasoning.
This is the load-bearing proxy assumption; no independent evidence ties the fitted boundary to clinical decision regions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs." pith.science (2026). https://pith.science/paper/RENGBHLK

@misc{pith2026250721188,
  author       = {Pith},
  title        = {Pith review of: Embeddings to Diagnosis: Latent Fragility under Agentic Perturbations in Clinical LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RENGBHLK}},
  note         = {Machine review of arXiv:2507.21188}
}

read the original abstract

LLMs for clinical decision support often fail under small but clinically meaningful input shifts such as masking a symptom or negating a finding, despite high performance on static benchmarks. These reasoning failures frequently go undetected by standard NLP metrics, which are insensitive to latent representation shifts that drive diagnosis instability. We propose a geometry-aware evaluation framework, LAPD (Latent Agentic Perturbation Diagnostics), which systematically probes the latent robustness of clinical LLMs under structured adversarial edits. Within this framework, we introduce Latent Diagnosis Flip Rate (LDFR), a model-agnostic diagnostic signal that captures representational instability when embeddings cross decision boundaries in PCA-reduced latent space. Clinical notes are generated using a structured prompting pipeline grounded in diagnostic reasoning, then perturbed along four axes: masking, negation, synonym replacement, and numeric variation to simulate common ambiguities and omissions. We compute LDFR across both foundation and clinical LLMs, finding that latent fragility emerges even under minimal surface-level changes. Finally, we validate our findings on 90 real clinical notes from the DiReCT benchmark (MIMIC-IV), confirming the generalizability of LDFR beyond synthetic settings. Our results reveal a persistent gap between surface robustness and semantic stability, underscoring the importance of geometry-aware auditing in safety-critical clinical AI.

Figures

Figures reproduced from arXiv: 2507.21188 by the authors.

**Figure 1.** Overview of the LAPD evaluation pipeline. Synthetic or real clinical notes are processed through structured perturba [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Overview of our framework (LAPD) through a constructed example illustrating latent diagnostic fragility. The [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** LLMs show inconsistent resilience to clinically realistic perturbations. This figure illustrates how diagnostic accuracy [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (17 more)

**Figure 4.** Figure 4: Global embedding shifts do not directly predict diagnostic failures. This figure shows how perturbations move [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Perturbations induce structured shifts in GPT-4O-mini’s latent space. This figure shows how variance is distributed [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 9.** Figure 9: Prompt for Aggregation and Verification of Reason [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 6.** Figure 6: Prompt for Clinical Note Generation. You are a diagnostic reasoning assistant. Read the following clinical note and generate a step-by-step reasoning process that extracts key observations and logical deductions leading to a diagnosis. Clinical Note: <-----Clinical Not…

**Figure 7.** Figure 7: Prompt for Forward Reasoning Generation. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: Prompt for Backward Reasoning Generation. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 10.** Figure 10: Entity-level recognition performance degrades under perturbation, particularly with masking and synonym edits. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 14.** Figure 14: Forward reasoning for GERD "1. Patient reports symptoms worsening after eating. ( Observation) 2. Patient describes pain as haunting, sensitive, and radiating to the upper chest. (Observation) 3. Patient reports a burning sensation from stomach to throat with a bitter…

**Figure 15.** Figure 15: Backward reasoning for GERD explained variance across perturbation thresholds (0%, 25%, 50%, 75%, 100%). Illustrated through [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]

**Figure 16.** Figure 16: Aggregated reasoning for GERD "Patient Details: Age: 57 Sex: [MASK] Chief Complaint: The patient reports experiencing [MASK] that worsen after eating. History of Present Illness: The patient describes her [MASK] as haunting, sensitive, a knife stroke, sickening, and s…

**Figure 19.** Figure 19: Backward reasoning for Masked Perturbation of [PITH_FULL_IMAGE:figures/full_fig_p014_19.png]

**Figure 17.** Figure 17: Masked Clinical Note for diagnosis GERD "1. Patient is a 57-year-old [MASK] with a chief complaint of unexplained discomfort worsening after eating. 2. The discomfort is described as haunting, sensitive, a knife stroke, sickening, and scary. 3. The discomfort is locat…

**Figure 20.** Figure 20: Final diagnosistic reasoning for Masked Perturba [PITH_FULL_IMAGE:figures/full_fig_p015_20.png]

**Figure 21.** Figure 21: Variance explained by top 5 latent dimensions for [PITH_FULL_IMAGE:figures/full_fig_p016_21.png]

**Figure 22.** Figure 22: Variance explained by top 5 latent dimensions for [PITH_FULL_IMAGE:figures/full_fig_p017_22.png]

**Figure 23.** Figure 23: Variance explained by top 5 latent dimensions for [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]

**Figure 24.** Figure 24: PCA Elbow Plot comparing cumulative explained variance for real (DiReCT) and synthetic clinical note embeddings. [PITH_FULL_IMAGE:figures/full_fig_p019_24.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 61 canonical work pages

[1]

Yu-Wen Chen and Julia Hirschberg. 2024. Exploring Robustness in Doctor- Patient Conversation Summarization: An Analysis of Out-of-Domain SOAP Notes. In Proceedings of the 6th Clinical Natural Language Processing Workshop , Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, and Danielle Bitterman (Eds.). Association for Computational Linguisti...

doi:10.18653/v1/2024.clinicalnlp-1.1 2024
[2]

Youngduck Choi, Chill Yi-I Chiu, and David Sontag. 2016. Learning Low- Dimensional Representations of Medical Concepts. AMIA Joint Summits on Translational Science Proceedings 2016 (2016), 41–50. https://www.ncbi.nlm.nih. gov/pmc/articles/PMC5001761/ eCollection 2016

2016
[3]

Sherry J. H. Feng, Edmund M-K. Lai, and Weihua Li. 2024. Geometry of Textual Data Augmentation: Insights from Large Language Models. Electronics 13, 18 (2024). doi:10.3390/electronics13183781

work page doi:10.3390/electronics13183781 2024
[4]

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. arXiv:2212.07919 [cs.CL] https://arxiv.org/ abs/2212.07919

arXiv 2023
[5]

Approximating mutual information of high-dimensional variables using learned representations

Gokul Gowri, Xiao-Kang Lun, Allon M. Klein, and Peng Yin. 2024. Approximating mutual information of high-dimensional variables using learned representations. arXiv:2409.02732 [q-bio.QM] https://arxiv.org/abs/2409.02732

work page Pith review arXiv 2024
[6]

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv:2009.13081 [cs.CL] https://arxiv.org/abs/2009.13081

arXiv 2020
[7]

Ginder, Peter Hong, Razi Sayeed, Barbra Y

Robert Johnson, Uri Gottlieb, Gal Shaham, Liron Eisen, Jonathan Waxman, Shay Devons-Sberro, Charles R. Ginder, Peter Hong, Razi Sayeed, Barbra Y. Reis, Ran D. Balicer, Noa Dagan, and Marinka Zitnik. 2024. Unified Clinical Vocabulary Embeddings for Advancing Precision Medicine. medRxiv (10 December 2024). doi:10.1101/2024.12.03.24318322 Preprint

work page doi:10.1101/2024.12.03.24318322 2024
[8]

Linyang Li, Ke Ren, Yunfan Shao, Pengyu Wang, and Xipeng Qiu. 2023. Perturb- Score: Connecting Discrete and Continuous Perturbations in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6638–6648. doi:10.18653/v1/2023.findi...

doi:10.18653/v1/2023.findings-emnlp.442 2023

Show all 96 references

[9]

Xin Li and Anand Sarwate. 2025. Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of- Experts. arXiv:2502.13577 [cs.LG] https://arxiv.org/abs/2502.13577

2025 arXiv
[10]

Smith, Thomas Lo, and Bang Liu

Yizhan Li, Sifan Wu, Christopher W. Smith, Thomas Lo, and Bang Liu. 2024. Improving Clinical Note Generation from Complex Doctor-Patient Conversa- tion. ArXiv abs/2408.14568 (2024). https://api.semanticscholar.org/CorpusID: 271962802

2024 arXiv
[11]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013/

2004
[12]

Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, and Nigel Collier. 2025. Measuring, Evaluating and Improving Logical Consistency in Large Language Models. https://openreview.net/forum?id=kJgi5ykK3t

2025
[13]

Isabel Lopez, Arjun Swaminathan, Karan Vedula, et al . 2025. Clinical entity augmented retrieval for clinical information extraction. npj Digital Medicine 8 (2025), 45. doi:10.1038/s41746-024-01377-1 Published: January 19, 2025

2025 doi
[14]

Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi

John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi
[15]

Priebe, and Eric Horvitz

Robert Osazuwa Ness, Katie Matton, Hayden Helm, Sheng Zhang, Junaid Bajwa, Carey E. Priebe, and Eric Horvitz. 2024. MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering. arXiv:2406.06573 [cs.CL] https://arxiv.org/abs/2406.06573

2024 arXiv
[16]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Bal- com, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jef...

2025 arXiv
[17]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds...

2002 doi
[18]

Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2022. On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation. https: //api.semanticscholar.org/CorpusID:253735234

2022
[19]

Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. 2023. ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , Houda Bouamor, Juan Pino, and Kalika Bali (...

2023 doi
[20]

Clément Péalat, Guillaume Bouleux, Vincent Cheutet, Maxime Maignan, Luc Provoost, Sylvie Pillet, and Olivier Mory. 2023. Using a manifold-based ap- proach to extract clinical codes associated with winter respiratory viruses at an emergency department. Expert Systems with Appli...

2023
[21]

Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schlu...

2020 doi
[22]

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansf...

2023
[23]

Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. 2022. DDXPlus: A New Dataset For Automatic Medical Diagnosis. arXiv:2205.09148 [cs.CL] https://arxiv.org/abs/2205.09148

2022 arXiv
[24]

Bowen Wang, Jiuyang Chang, Yiming Qian, Guoxin Chen, Junhao Chen, Zhouqiang Jiang, Jiahao Zhang, Yuta Nakashima, and Hajime Nagahara. 2024. DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models. In The Thirty-eight Conference on Neural Information Processin...

2024
[25]

Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin...

2021
[26]

Wong, and Rui Wang

Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. 2025. Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation. arXiv:2410.13640 [cs.CL] https://arxiv.org/abs/2410.13640

2025 arXiv
[27]

Grabowska, Iftikhar J

Shang Xian, Maria E. Grabowska, Iftikhar J. Kullo, Yuan Luo, Jordan W. Smoller, Wei-Qi Wei, Gail Jarvik, Sean Mooney, and David Crosslin. 2024. Language-model- based patient embedding using electronic health records facilitates phenotyping, disease forecasting, and progression...

2024 doi
[28]

Zi Yin and Yuanyuan Shen. 2018. On the Dimensionality of Word Embedding. arXiv:1812.04224 [cs.LG] https://arxiv.org/abs/1812.04224

2018 arXiv
[29]

Weinberger, and Yoav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs.CL] https://arxiv.org/abs/1904.09675

2020 arXiv
[30]

Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoor- thi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber
[33]

She reports symptoms worsening after eating
[34]

The pain is described as haunting, sensitive, a knife stroke, sickening, and scary
[35]

The pain is located in the lower chest, upper chest, hypochondrium (left), and epigastric regions
[36]

The pain radiates to the upper chest
[37]

The intensity and duration of the pain are unknown
[38]

The pain is alleviated while sitting up
[39]

A burning sensation is reported, starting in the stomach and going up into the throat
[40]

The patient has a bitter taste in the mouth
[41]

She has a history of hiatal hernia and asthma
[42]

She has used a bronchodilator for asthma in the past
[43]

She is currently pregnant
[44]

No current medications were reported
[45]

She has no known allergies
[46]

The patient reports being significantly overweight
[47]

She drinks alcohol excessively
[48]

She smokes cigarettes
[51]

Figure 14: Forward reasoning for GERD

These factors (overweight, alcohol consumption, smoking) may be relevant to the patient 's current symptoms." Figure 14: Forward reasoning for GERD "1. Patient reports symptoms worsening after eating. ( Observation)
[52]

(Observation)

Patient describes pain as haunting, sensitive, and radiating to the upper chest. (Observation)
[53]

(Observation)

Patient reports a burning sensation from stomach to throat with a bitter taste. (Observation)
[54]

(Past Medical History)

Patient has a hiatal hernia. (Past Medical History)
[55]

(Knowledge of GERD)

Symptoms of GERD (gastroesophageal reflux disease) include heartburn, regurgitation, and pain that worsens after eating. (Knowledge of GERD)
[56]

( Knowledge of GERD and hiatal hernia)

GERD symptoms can be exacerbated by hiatal hernia. ( Knowledge of GERD and hiatal hernia)
[57]

(Logical deduction)

The patient 's symptoms, including pain, burning sensation, and bitter taste, are consistent with GERD. (Logical deduction)
[58]

(Logical deduction)

The patient 's history of hiatal hernia, along with her symptoms, increases the likelihood of GERD. (Logical deduction)
[59]

(Logical deduction)

The patient 's symptoms are not typical of other conditions, such as heart disease or gallbladder disease, given her age and lack of other risk factors. (Logical deduction)
[60]

(Final deduction)" Figure 15: Backward reasoning for GERD explained variance across perturbation thresholds (0%, 25%, 50%, 75%, 100%)

Therefore, the most likely diagnosis is GERD. (Final deduction)" Figure 15: Backward reasoning for GERD explained variance across perturbation thresholds (0%, 25%, 50%, 75%, 100%). Illustrated through Fig 22, Fig 21 and Fig 23. Key Insights • Dimensions such as 273 and 752 con...

2025
[61]

Symptoms worsen after eating
[62]

Pain described as haunting, sensitive, and radiating to the upper chest
[63]

Logical Deductions:

Burning sensation from stomach to throat with a bitter taste. Logical Deductions:
[64]

Patient 's symptoms are consistent with GERD
[65]

Patient 's history of hiatal hernia increases the likelihood of GERD
[66]

Patient 's symptoms are not typical of other conditions, such as heart disease or gallbladder disease
[67]

The patient 's lifestyle factors (overweight, alcohol consumption, smoking) may be relevant to her symptoms
[69]

The patient 's use of a bronchodilator for asthma does not directly contribute to the diagnosis of GERD, but it does not rule it out either
[70]

Figure 16: Aggregated reasoning for GERD

The patient 's current pregnancy may not directly affect the diagnosis of" Figure 16: Aggregated reasoning for GERD "Patient Details: Age: 57 Sex: [MASK] Chief Complaint: The patient reports experiencing [MASK] that worsen after eating. History of Present Illness: The patient ...
[71]

The discomfort is described as haunting, sensitive, a knife stroke, sickening, and scary
[72]

The discomfort is located in the epigastric region, left hypochondrium, and umbilical region
[73]

The intensity and duration of the discomfort are unknown, but it radiates to the back
[74]

The discomfort is alleviated while sitting up
[75]

The patient also reports a burning sensation starting in the throat and going up into the mouth, associated with a sensation in the mouth
[76]

The patient has a history of hypertension and diabetes, and has used a diuretic in the past
[77]

No current medications were reported, but the patient has a history of using a diuretic for hypertension
[78]

The patient has no known allergies
[79]

The patient reports feeling significantly underweight, drinking alcohol excessively, and having a poor appetite
[80]

No further physical examination findings were provided
[81]

The patient has not traveled out of the country in the last 4 weeks
[82]

Figure 18: Forward reasoning for Masked Perturbation of 100% GERD

These factors may be relevant to the patient 's current symptoms and should be considered in the diagnostic and management process." Figure 18: Forward reasoning for Masked Perturbation of 100% GERD "1. The patient reports symptoms of heartburn that worsen after eating
[83]

The heartburn is described as haunting, sensitive, and radiating to the left hypochondrium and back
[84]

The patient has a history of GERD and has used a proton pump inhibitor (PPI) in the past
[85]

The patient 's symptoms are alleviated while sitting up, which is a common symptom of GERD
[86]

The patient 's excessive alcohol consumption and obesity may contribute to the worsening of GERD symptoms
[87]

Logical deductions:

No physical examination findings suggest an alternative diagnosis, such as a cardiac issue. Logical deductions:
[88]

The patient 's symptoms of heartburn, worsened by eating and alleviated by sitting up, are consistent with GERD
[89]

The patient 's history of GERD and use of a PPI support the diagnosis of GERD
[90]

The patient 's obesity and excessive alcohol consumption may exacerbate GERD symptoms
[91]

Full-resolution figures per model are available upon request

The absence of physical examination findings suggesting an alternative diagnosis supports the diagnosis of GERD." Figure 19: Backward reasoning for Masked Perturbation of 100% GERD perturbation type. Full-resolution figures per model are available upon request. A.9 Elbow Plot ...

2025
[92]

Patient 's discomfort worsens after eating and is alleviated while sitting up
[93]

Discomfort is described as haunting, sensitive, and radiates to the left hypochondrium and back
[94]

Patient has a history of hypertension, diabetes, and past use of a diuretic
[95]

Logical deductions:

Patient reports excessive alcohol consumption, poor appetite, and feeling significantly underweight. Logical deductions:
[96]

Symptoms are consistent with GERD
[97]

Patient 's history of GERD and use of a PPI support the diagnosis
[98]

Patient 's lifestyle factors may exacerbate GERD symptoms
[99]

For synthetic notes, 90% of variance is typically captured within 30–35 components

Absence of physical examination findings suggesting an alternative diagnosis supports the diagnosis of GERD." Figure 20: Final diagnosistic reasoning for Masked Perturba- tion of 100% GERD Figure 24 shows the cumulative explained variance as a function of the number of princip...

2025
[2020]

arXiv:2005.05909 [cs.CL] https://arxiv.org/abs/ 2005.05909

TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. arXiv:2005.05909 [cs.CL] https://arxiv.org/abs/ 2005.05909

2005 arXiv
[2024]

Do you have pain somewhere, related to your reason for consulting?

Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934 [cs.AI] https://arxiv.org/abs/2410.10934 A Appendix A.1 Code and reproduction All prompting templates and code to reproduce our results are available at: https://github.com/unni12345/geometric_diagnostics A.2 The ...

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Yu-Wen Chen and Julia Hirschberg. 2024. Exploring Robustness in Doctor- Patient Conversation Summarization: An Analysis of Out-of-Domain SOAP Notes. In Proceedings of the 6th Clinical Natural Language Processing Workshop , Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, and Danielle Bitterman (Eds.). Association for Computational Linguisti...

doi:10.18653/v1/2024.clinicalnlp-1.1 2024

[2] [2]

Youngduck Choi, Chill Yi-I Chiu, and David Sontag. 2016. Learning Low- Dimensional Representations of Medical Concepts. AMIA Joint Summits on Translational Science Proceedings 2016 (2016), 41–50. https://www.ncbi.nlm.nih. gov/pmc/articles/PMC5001761/ eCollection 2016

2016

[3] [3]

Sherry J. H. Feng, Edmund M-K. Lai, and Weihua Li. 2024. Geometry of Textual Data Augmentation: Insights from Large Language Models. Electronics 13, 18 (2024). doi:10.3390/electronics13183781

work page doi:10.3390/electronics13183781 2024

[4] [4]

Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. arXiv:2212.07919 [cs.CL] https://arxiv.org/ abs/2212.07919

arXiv 2023

[5] [5]

Approximating mutual information of high-dimensional variables using learned representations

Gokul Gowri, Xiao-Kang Lun, Allon M. Klein, and Peng Yin. 2024. Approximating mutual information of high-dimensional variables using learned representations. arXiv:2409.02732 [q-bio.QM] https://arxiv.org/abs/2409.02732

work page Pith review arXiv 2024

[6] [6]

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv:2009.13081 [cs.CL] https://arxiv.org/abs/2009.13081

arXiv 2020

[7] [7]

Ginder, Peter Hong, Razi Sayeed, Barbra Y

Robert Johnson, Uri Gottlieb, Gal Shaham, Liron Eisen, Jonathan Waxman, Shay Devons-Sberro, Charles R. Ginder, Peter Hong, Razi Sayeed, Barbra Y. Reis, Ran D. Balicer, Noa Dagan, and Marinka Zitnik. 2024. Unified Clinical Vocabulary Embeddings for Advancing Precision Medicine. medRxiv (10 December 2024). doi:10.1101/2024.12.03.24318322 Preprint

work page doi:10.1101/2024.12.03.24318322 2024

[8] [8]

Linyang Li, Ke Ren, Yunfan Shao, Pengyu Wang, and Xipeng Qiu. 2023. Perturb- Score: Connecting Discrete and Continuous Perturbations in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6638–6648. doi:10.18653/v1/2023.findi...

doi:10.18653/v1/2023.findings-emnlp.442 2023