Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry

Alireza Nik; Andreea Grama; Bogdan Ionescu; Ciprian I. B\u{a}cil\u{a}; Jaya Ojha; Larisa Irimin\u{a}; Michael A. Riegler; Oriana Presacan; Vajira Thambawita

REVIEW 4 major objections 4 minor 26 references

LLMs commit to psychiatric diagnoses before the evidence is in, a new benchmark shows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:46 UTC pith:FXDB3IQK

load-bearing objection A careful, useful benchmark with a credible main finding; the retrospective sufficiency labels make the timing results approximate, but the paper handles that honestly and deserves refereeing. the 4 major comments →

arxiv 2607.13036 v1 pith:FXDB3IQK submitted 2026-05-06 cs.CL cs.AI

Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry

Oriana Presacan , Andreea Grama , Larisa Irimin\u{a} , Alireza Nik , Jaya Ojha , Vajira Thambawita , Ciprian I. B\u{a}cil\u{a} , Bogdan Ionescu

show 1 more author

Michael A. Riegler

This is my paper

classification cs.CL cs.AI

keywords LLM safetypsychiatric diagnosissequential evaluationabstentioncalibrationclarification-seekingclinical decision supportbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Safe-Psych is a benchmark for whether language models know when not to diagnose in psychiatry. It presents 1,048 real clinical notes one section at a time, with psychiatrist-defined labels telling whether a model should diagnose, ask for clarification, or abstain at each step. Across nine models, the paper finds that capability does not track calibration: most models diagnose more than 60% of the cases that psychiatrists judged insufficient, even when explicitly allowed to abstain, and they rarely request more information unless prompted. Telling models to be cautious delays diagnoses and lowers premature commitment, but only by making them abstain on many cases that did have enough information. The paper argues that recognizing incomplete evidence is a separate skill from diagnostic accuracy and that current prompting does not supply it.

Core claim

The paper's central claim is that LLMs used for psychiatric decision support are poorly calibrated when clinical evidence arrives over time: they commit to diagnoses before enough information exists, under-abstain even when abstention is explicitly allowed, and only shift error patterns rather than improve them when prompted to be cautious. In the benchmark's sequential protocol, models that diagnose at the expert-defined 'sufficient' stage are consistently more accurate than those that diagnose early, so premature commitment carries a concrete accuracy cost. The authors find diagnostic capability and uncertainty calibration decouple: strong models can be accurate on full notes while still f

What carries the argument

The load-bearing object is the expert action sequence derived from psychiatrist annotations: for each case, the earliest section where information becomes sufficient yields CLARIFY for earlier steps, DIAGNOSE from that step onward, and ABSTAIN at the end if it never becomes sufficient. Model trajectories are judged by an LLM judge that extracts whether the response commits to a diagnosis, requests specific information, or gives none, and maps that to the same three actions. This setup converts a one-shot QA task into a timing-and-calibration task, letting the paper measure under-abstention (diagnosing insufficient cases), over-abstention (abstaining on sufficient cases), and premature versus

Load-bearing premise

The claim rests on treating psychiatrists' retrospective 'earliest sufficient section' labels as true decision points; if those labels mark information sufficient earlier than it would be in real time, premature diagnoses can be mislabeled as on-time or late.

What would settle it

A prospective study in which psychiatrists judge sufficiency while reading a note section-by-section, without seeing later sections, could check whether the retrospective labels shift. If under real-time labeling premature diagnoses are no less accurate than on-time ones, or under-abstention drops below the reported rates, the central timing and calibration story would need revision.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Deploying current LLMs as intake or triage assistants in psychiatry risks early commitment: diagnoses made before evidence is sufficient are measurably less accurate than on-time ones.
Adding abstention instructions does not fix calibration; it just trades unsafe early diagnoses for missed ones, so prompting alone is not a safety mechanism.
Models generally will not ask for missing information on their own; clarification-seeking must be explicitly prompted or trained.
Diagnostic accuracy on full notes overstates real-world usefulness, because in practice evidence arrives piecemeal; sequential evaluation is needed to see the gap.
Psychiatrist disagreement on a case correlates with lower model accuracy, suggesting some benchmark errors reflect genuine case ambiguity rather than model fault alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the same failure generalizes beyond psychiatry, the sequential-disclosure design could be adapted to emergency triage or primary care, where information also arrives over time; that transfer is untested.
The under-/over-abstention tradeoff resembles a calibration frontier; future work could treat abstention thresholds as tunable and evaluate whether models can be calibrated per case difficulty rather than only per prompt.
Because sufficiency labels were made retrospectively, a prospective study where clinicians mark sufficiency in real time would test whether the reported premature-diagnosis accuracy gap survives stricter timing labels.
The benchmark's judge maps 'committed plus specific question' to CLARIFY; this definition likely rewards models that hedge a diagnosis with a question, so results may be sensitive to response style, an artifact worth testing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A careful, useful benchmark with a credible main finding; the retrospective sufficiency labels make the timing results approximate, but the paper handles that honestly and deserves refereeing.

read the letter

Two things you should know. First, this is a genuinely useful artifact: a 1,048-note psychiatric benchmark that presents evidence incrementally and asks models to clarify, diagnose, or abstain. The authors validate their LLM judge against human annotations, report inter-rater agreement, release code and data, and disclose their limitations in unusual detail. Second, the main empirical claim—that strong models under-abstain under incomplete information and that safety prompts merely shift errors into over-abstention—is credible and likely robust. That finding alone makes the paper worth reading.

The under/over-abstention result (Figure 3) is the cleanest contribution because it depends only on the final sufficient/insufficient label, which has 90% raw agreement and is less sensitive to timing assumptions. The sequential results—clarification rates, premature-diagnosis timing, and the premature-vs-on-time accuracy gap—are more fragile because they rest on the psychiatrists' retrospective "earliest sufficient section" labels. Annotators saw the full note before marking that section, and disagreements were resolved by taking the earlier annotation. That biases the benchmark toward early sufficiency, which the paper correctly notes is lenient. In practice, this likely makes the premature-diagnosis finding conservative: some diagnoses labeled "on-time" would be premature under stricter prospective thresholds, so the accuracy gap in Figure 7 is probably understated, not overstated. Still, the paper does not quantify how much the timing or accuracy conclusions would shift under different threshold choices, and Figure 7 has no confidence intervals or significance tests. That is the main soft spot, and it is a moderate one, not fatal.

Other soft spots are minor and mostly disclosed: only 9% of cases are insufficient-information cases, so under-abstention estimates for that subset are unstable; the judge reaches macro-F1 0.84 on action labels, so small cross-model differences may be noise; and the data come from one Romanian hospital with a 77% female sample, translated to English. None of these undermine the central story.

Who is this for? Anyone building or evaluating clinical LLM safety benchmarks. It extends AbstentionBench and MedAbstain in a way that is more than cosmetic: the clarify option and the sequential disclosure actually change what you measure. I would bring it to a reading group and would cite it if I worked in this area.

Recommendation: send it to peer review. The referees should ask for a sensitivity analysis on the sufficiency thresholds—for example, using the later annotation instead of the earlier one, or a small prospective validation—and for uncertainty intervals on Figure 7. But this is a solid, honest piece of benchmark work that deserves a serious referee, not a desk rejection.

Referee Report

4 major / 4 minor

Summary. Safe-Psych is a sequential benchmark for evaluating LLM decision behavior under evolving clinical information in psychiatry. It releases 1,048 anonymized, translated psychiatric notes segmented into five sections, with psychiatrist-derived ground-truth labels for diagnosis, the earliest sufficient section, and sufficiency of the full note. Nine LLMs are evaluated under full-information and sequential settings, with responses mapped by a validated LLM judge to DIAGNOSE, CLARIFY, or ABSTAIN. The paper's main claims are that strong models under-abstain (diagnose despite insufficient evidence), rarely seek clarification unless explicitly prompted, diagnose prematurely relative to expert sufficiency labels, that premature diagnoses are less accurate than on-time ones, and that abstention-aware prompting shifts errors toward over-abstention rather than improving calibration.

Significance. If the central claims hold, Safe-Psych is a valuable resource: it addresses a gap left by static QA and abstention benchmarks by measuring how model behavior changes as evidence accumulates. The dataset is large, real-world, and externally annotated; the judge is validated against human annotations; prompting, evaluation, and reproducibility code are described in unusual detail. The distinction between capability and calibration, and the explicit measurement of over-abstention alongside under-abstention, are strengths. However, the sequential timing and calibration results depend on retrospectively assigned sufficiency labels, and the paper's own limitation section concedes that these may be systematically earlier than prospective clinical judgments. That pivot requires additional validation or sensitivity analysis before the headline timing claims can be regarded as fully established.

major comments (4)

[§2.3, Appendix B.6] The expert stopping step tau_exp is the load-bearing label for all premature/on-time classifications, yet it was assigned after annotators viewed the full note, and disagreements were resolved by taking the earlier section. Appendix B.6 concedes this 'may mark information as sufficient earlier than it would be judged in a prospective clinical setting.' The paper argues the bias is conservative for cases already labeled premature, but it does not quantify boundary misclassification. Please add a sensitivity analysis: re-annotate a subset with section-by-section prospective judgments, or re-run the timing metrics with tau_exp shifted one section later and with adjudicated disagreements excluded. Without this, Figures 5–7 do not fully support the sequential calibration story.
[§4.4, Fig. 7] The claim that 'on-time diagnoses are consistently more accurate than premature ones' is presented without confidence intervals, case counts, or a statistical test. Since accuracy is computed only for cases where the model produced a diagnosis, the premature and on-time groups may differ systematically in case difficulty or diagnostic mix. A model that diagnoses early may do so disproportionately on easy cases, while a model that waits may only diagnose cases it is confident about. Please report n and 95% CIs for each timing group, and stratify by expert tau_exp stage or diagnosis category. A paired comparison using cases where the same model eventually reaches an on-time diagnosis would directly test whether waiting itself improves accuracy.
[§4.1, Fig. 3; §6] The under-abstention and over-abstention rates rely on only 9% insufficient cases (roughly 95 of 1,048). Error bars in Figure 3 are shown only for the four models with repeated seed runs; the remaining five models have no uncertainty quantification. Because decoding is stochastic and the insufficient-case denominator is small, the statement that 'under-abstention exceeds 60% for most models' needs binomial or bootstrap confidence intervals for every model, or at least a statement of whether single-run estimates for MedGemma-27B, Gemma-3-4B, Claude-Opus-4.6, and Gemini-Flash-2.5 are robust to sampling variability. The appendix currently gives repeated-run data only for the full-information setting, not for the sequential setting most central to the paper.
[§3.4.1, Table 1, Appendix C.1.1] The LLM judge is GPT-5.4, which is also one of the evaluated models. Its macro-F1 on the final action label is 0.84, below the human agreement of 0.89. The validation set of 149 responses is deliberately enriched for boundary cases, but no per-model or per-strategy judge agreement is reported. Since every behavioral metric passes through this judge, differential judge accuracy across models or inference strategies could confound cross-model comparisons. Please report judge agreement stratified by evaluated model (at least for GPT-5.4 vs. the other models), and include a sensitivity analysis using human labels on a random subset of responses.

minor comments (4)

[§3.1] Please state explicitly whether an intermediate ABSTAIN terminates the trajectory or whether the model continues to receive subsequent sections. The current text records 'premature abstention' but does not define the resulting transition, which is needed for exact reproduction.
[Appendix Fig. 1b] The axis label 'Final S' is ambiguous; rename it 'Final section' to match the earlier sections S1–S4.
[Appendix Table 3] F10.2 (alcohol dependence syndrome) is listed twice in the diagnosis-specific agreement table; remove the duplicate.
[References] References [24] and [25] both cite the Qwen3 Technical Report with the same arXiv identifier; one should be removed or merged.

Circularity Check

0 steps flagged

No significant circularity: all benchmark outcomes are anchored to external psychiatrist labels and a human-validated judge; no fitted parameters or self-citation derivations.

full rationale

Safe-Psych's central quantities—under-/over-abstention, premature/on-time timing, and diagnostic accuracy—are defined against external human annotations, not against the evaluated models' outputs. Section 2.3 describes independent psychiatrist annotation of information sufficiency and earliest sufficient section; Section 3.2 derives expert action labels from these annotations. Model responses are classified by an LLM judge (Section 3.4.1) that was validated against majority-vote human annotations (Table 1: macro-F1 0.84, Fleiss' kappa 0.86 on actions), so the judge is a measured instrument rather than the source of ground truth. The claims that models 'diagnose before sufficient evidence is available' and that 'premature diagnoses are less accurate than on-time diagnoses' are empirical comparisons of model behavior against these external labels, not equivalences by construction. There are no fitted parameters renamed as predictions and no load-bearing self-citations; the reference list contains no prior work by the present authors. The retrospective-sufficiency limitation (Appendix B.6: 'sufficiency labels were assigned retrospectively after annotators viewed the full note... making the benchmark relatively lenient') is a validity/bias concern about the external gold standard, not a circularity: the label is still independent of the evaluated model. Judge noise is likewise acknowledged in Section 6 and bounded by human agreement. Thus no circular step can be quoted, and no result reduces to its own input by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The benchmark's central claims depend on external assumptions about the staged-disclosure protocol, the reliability of retrospective psychiatrist annotations, the accuracy of the LLM judge, and translation fidelity. The paper states and partially quantifies the risks (moderate κ, 79.9% ICD agreement, judge macro-F1 0.84, single-hospital data), but these remain load-bearing premises.

axioms (4)

domain assumption Psychiatric diagnosis is inherently uncertain and unfolds sequentially; staged evidence disclosure is a valid proxy for clinical workflow.
Motivates the benchmark design in the Introduction and Section 3.1. If false, the protocol does not measure clinically meaningful behavior.
domain assumption Psychiatrist re-annotation of anonymized, translated notes provides reliable ground truth for sufficiency, timing, and diagnosis.
Used to derive all expert action labels. Agreement is moderate on sufficiency (Cohen's κ=0.593) and 79.9% on full ICD-10; the labels are retrospective (Section 2.3, Appendix B.6).
domain assumption The LLM judge's mapping of free-form responses to DIAGNOSE/CLARIFY/ABSTAIN is accurate enough for cross-model comparisons.
Judge validated on 149 examples; action macro-F1 is 0.84, below human agreement of 0.89. Small model differences may be judge noise (Table 1, Section 6).
domain assumption English translations of Romanian notes preserve the uncertainty cues relevant to clinical decisions.
All experiments use translated text; COMET is 0.889, but the authors note that translation may alter hedging and symptom description (Appendix B.1, B.6).

pith-pipeline@v1.3.0-alltime-deepseek · 20304 in / 9936 out tokens · 98275 ms · 2026-08-02T14:46:23.691612+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) are increasingly used for decision support in healthcare, but clinical evidence is often incomplete or evolving. When the available information is insufficient to support a reliable answer, models should request clarification or abstain rather than provide unsupported responses. Existing medical benchmarks, however, typically assume that complete information is available upfront. We introduce Safe-Psych, a sequential benchmark for evaluating how LLMs handle evolving diagnostic uncertainty in clinical psychiatry. Safe-Psych contains over 1,000 real-world psychiatric clinical notes segmented to simulate incremental evidence disclosure, with psychiatrist-derived action labels at each stage: DIAGNOSE, CLARIFY, or ABSTAIN. We evaluate multiple state-of-the-art LLMs in full-information and sequential settings. Our findings show that capability does not ensure calibration: even strong models struggle under incomplete clinical information, with under-abstention exceeding 60% for most models and safety-aware prompting reducing premature commitment only by shifting errors toward excessive abstention. In sequential evaluation, models frequently diagnose before sufficient evidence is available and rarely seek clarification unless explicitly prompted; these premature diagnoses are less accurate than on-time diagnoses. Overall, Safe-Psych reveals a limitation across the evaluated models: recognizing when clinical evidence is incomplete and additional information is needed. We release Safe-Psych to support research on improving LLM safety in healthcare.

Figures

Figures reproduced from arXiv: 2607.13036 by Alireza Nik, Andreea Grama, Bogdan Ionescu, Ciprian I. B\u{a}cil\u{a}, Jaya Ojha, Larisa Irimin\u{a}, Michael A. Riegler, Oriana Presacan, Vajira Thambawita.

**Figure 1.** Figure 1: Traditional benchmarks vs. Safe-Psych. (a) Traditional benchmarks provide complete information upfront. (b–c) Safe-Psych reveals information incrementally and evaluates whether models clarify, diagnose when evidence is sufficient (b), or abstain when it remains incomplete (c). healthcare. Similarly, MedAbstain [15] introduces missing-information variants of clinical multiplechoice questions and shows that… view at source ↗

**Figure 2.** Figure 2: Human annotation-to-action pipeline. Psychiatrists determine information sufficiency, earliest sufficient section, and diagnosis; which are converted into CLARIFY, DIAGNOSE, and ABSTAIN. Evaluator ANS. REQ. ACT. Humans 0.84 0.86 0.89 GPT-5.4 0.83 0.85 0.86 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Under- and over-abstention reveal that capability does not ensure calibration. Under-abstention = diagnosing insufficient cases; over-abstention = abstaining on sufficient cases. # Clarify at Stage 1 0 20 40 60 80 100 % of samples Mistral 24B Claude 4.6 Gemma 3 4B Gemma 3 27B GPT-5.4 Qwen 32B MedGemma 27B Gemini Flash 2.5 Med42-8B [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 5.** Figure 5: Abstention prompting reduces premature diagnoses but increases missed diagnoses. S1 S2 S3 S4 Final 0 20 40 60 80 100 Cumulative cases (%) Neutral Abstention Expert sufficient [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

**Figure 8.** Figure 8: Sequential information reduces diagnostic accuracy relative to full information. 3-character ICD accuracy across static and sequential settings, with and without abstention-aware prompting. Diagnosis-level ICD 3-character ICD High-level class 0 20 40 60 80 100 Diagnosis accuracy (%) Doctor agreement Doctor disagreement [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗

**Figure 1.** Figure 1: Annotators agreement on information sufficiency and expert action label distribution. [PITH_FULL_IMAGE:figures/full_fig_p015_1.png] view at source ↗

**Figure 2.** Figure 2: Dataset characteristics and expert-derived action label distribution. [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗

**Figure 3.** Figure 3: Under- and over-abstention in the Sequential Information - Abstention Aware setting. [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

**Figure 4.** Figure 4: Abstention rates differ for sufficient vs. insufficient information. Abstention rate by model, separated by cases with insufficient (higher is better) and sufficient (lower is better) clinical information. D.3 Diagnosis accuracy [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗

**Figure 5.** Figure 5: Three-character ICD accuracy for the six highest- and lowest-performing ground-truth diagnostic categories. Case counts are shown next to each bar. 0 128 2048 Gemini 2.5 Flash thinking budget 0 20 40 60 80 100 Rate (%) 3-character ICD accuracy Under-abstention Over-abstention [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

26 extracted references · 3 canonical work pages

[1]

Abi-Dargham, S

A. Abi-Dargham, S. J. Moeller, F. Ali, C. DeLorenzo, K. Domschke, G. Horga, A. Jutla, R. Kotov, M. P. Paulus, J. M. Rubio, G. Sanacora, J. Veenstra-VanderWeele, and J. H. Krystal. Candidate biomarkers in psychiatric disorders: State of the field.World Psychiatry, 22(2): 236–262, June 2023. doi: 10.1002/wps.21078

work page doi:10.1002/wps.21078 2023
[2]

American Psychiatric Association Publishing, Washington, DC, 5th ed., text rev

American Psychiatric Association.Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR). American Psychiatric Association Publishing, Washington, DC, 5th ed., text rev. edition, 2022. doi: 10.1176/appi.books.9780890425787

doi:10.1176/appi.books.9780890425787 2022
[3]

Claude Opus 4.6 System Card

Anthropic. Claude Opus 4.6 System Card. Anthropic model system card, 2026. URL https: //www.anthropic.com/system-cards. Accessed 2026-05-04

2026
[4]

Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero- Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URLhttps://arxiv.org/abs/2505.08775

Pith/arXiv arXiv 2025
[5]

Simulating viva voce examinations to evaluate clinical reasoning in large language models, 2025

Christopher Chiu, Silviu Pitis, and Mihaela van der Schaar. Simulating viva voce examinations to evaluate clinical reasoning in large language models, 2025. URL https://arxiv.org/ abs/2510.10278

arXiv 2025
[6]

Kanithi, Tathagata Raha, Shadab Khan, and Marco A

Clément Christophe, Praveen K. Kanithi, Tathagata Raha, Shadab Khan, and Marco A. F. Pimentel. Med42-v2: A Suite of Clinical LLMs.arXiv preprint arXiv:2408.06142, 2024. URL https://arxiv.org/abs/2408.06142

Pith/arXiv arXiv 2024
[7]

Euroeval multilingual european leaderboard

EuroEval. Euroeval multilingual european leaderboard. https://euroeval.com/ leaderboards/Multilingual/european/, 2024. Accessed: 2026-04-05

2024
[8]

Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025

Gemma Team. Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025. URL https://arxiv.org/abs/2503.19786

Pith/arXiv arXiv 2025
[9]

Gemini 2.5 Flash

Google. Gemini 2.5 Flash. Google Cloud Vertex AI documentation, 2025. URLhttps://docs. cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash . Accessed 2026-04-21

2025
[10]

Cohen, and Xinghua Lu

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China,

2019
[11]

Phillips, and Thomas R

Shitij Kapur, Anthony G. Phillips, and Thomas R. Insel. Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it?Molecular Psychiatry, 17(12): 1174–1179, 2012. doi: 10.1038/mp.2012.105

work page doi:10.1038/mp.2012.105 2012
[12]

Kendell and A

R. Kendell and A. Jablensky. Distinguishing between the validity and utility of psychiatric diagnoses.American Journal of Psychiatry, 160(1):4–12, January 2003. doi: 10.1176/appi.ajp. 160.1.4

work page doi:10.1176/appi.ajp 2003
[13]

Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions.arXiv preprint, 2025. URL https: //arxiv.org/abs/2506.09038. arXiv:2506.09038 [cs.CL]

Pith/arXiv arXiv 2025
[14]

Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov

Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. MEDIQ: question-asking LLMs and a benchmark for reliable interactive clinical reasoning. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA, 2025. Curran Associate...

2025
[15]

Knowing when to abstain: Medical llms under clinical uncertainty, 2026

Sravanthi Machcha, Sushrita Yerra, Sahil Gupta, Aishwarya Sahoo, Sharmin Sultana, Hong Yu, and Zonghai Yao. Knowing when to abstain: Medical llms under clinical uncertainty, 2026. URLhttps://arxiv.org/abs/2601.12471. 10

arXiv 2026
[16]

Mistral Small 3.1

Mistral AI. Mistral Small 3.1. Mistral AI, 2025. URL https://mistral.ai/news/ mistral-small-3-1. Accessed 2026-05-04

2025
[17]

Initial psychiatric assess- ment

MSD Manual Professional Edition. Initial psychiatric assess- ment. MSD Manual Professional Edition, 2026. URL https: //www.msdmanuals.com/professional/psychiatric-disorders/ approach-to-the-patient-with-psychiatric-symptoms/ initial-psychiatric-assessment . Reviewed/Revised October 2024; Modified January 2026; accessed 2026-05-04

2026
[18]

GPT-5.4 Model

OpenAI. GPT-5.4 Model. OpenAI API documentation, 2026. URL https://developers. openai.com/api/docs/models/gpt-5.4. Accessed 2026-04-21

2026
[19]

Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. InProceedings of the Conference on Health, Inference, and Learning, volume 174 ofProceedings of Machine Learning Research, pages 248–260. PMLR, 2022

2022
[20]

Regier, William E

Darrel A. Regier, William E. Narrow, Diana E. Clarke, Helena C. Kraemer, S. Janet Kuramoto, Emily A. Kuhl, and David J. Kupfer. Dsm-5 field trials in the united states and canada, part ii: Test-retest reliability of selected categorical diagnoses.American Journal of Psychiatry, 170(1): 59–70, January 2013. doi: 10.1176/appi.ajp.2012.12070999

Pith/arXiv arXiv 2013
[21]

Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis.Psychiatry Research, 355:116844, 2026

Karthik V Sarma, Kaitlin E Hanss, Andrew J M Halls, Andrew Krystal, Daniel F Becker, Anne L Glowinski, and Atul J Butte. Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis.Psychiatry Research, 355:116844, 2026. ISSN 0165-1781. doi: 10.1016/j.psychres.2025.116844

arXiv 2026
[22]

MedGemma Technical Report.arXiv preprint arXiv:2507.05201, 2025

Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, et al. MedGemma Technical Report.arXiv preprint arXiv:2507.05201, 2025. URL https: //arxiv.org/abs/2507.05201

Pith/arXiv arXiv 2025
[23]

World Health Organization, Geneva, 2019

World Health Organization.International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10). World Health Organization, Geneva, 2019

2019
[25]

Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505. 09388

Pith/arXiv arXiv 2025
[26]

committed

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/ abs/2306.05685. 11 A Ethical approval and data governance A.1 Institutional oversight and et...

Pith/arXiv arXiv 2023
[2019]

doi: 10.18653/v1/D19-1259

Association for Computational Linguistics. doi: 10.18653/v1/D19-1259

doi:10.18653/v1/d19-1259

[1] [1]

Abi-Dargham, S

A. Abi-Dargham, S. J. Moeller, F. Ali, C. DeLorenzo, K. Domschke, G. Horga, A. Jutla, R. Kotov, M. P. Paulus, J. M. Rubio, G. Sanacora, J. Veenstra-VanderWeele, and J. H. Krystal. Candidate biomarkers in psychiatric disorders: State of the field.World Psychiatry, 22(2): 236–262, June 2023. doi: 10.1002/wps.21078

work page doi:10.1002/wps.21078 2023

[2] [2]

American Psychiatric Association Publishing, Washington, DC, 5th ed., text rev

American Psychiatric Association.Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR). American Psychiatric Association Publishing, Washington, DC, 5th ed., text rev. edition, 2022. doi: 10.1176/appi.books.9780890425787

doi:10.1176/appi.books.9780890425787 2022

[3] [3]

Claude Opus 4.6 System Card

Anthropic. Claude Opus 4.6 System Card. Anthropic model system card, 2026. URL https: //www.anthropic.com/system-cards. Accessed 2026-05-04

2026

[4] [4]

Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero- Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URLhttps://arxiv.org/abs/2505.08775

Pith/arXiv arXiv 2025

[5] [5]

Simulating viva voce examinations to evaluate clinical reasoning in large language models, 2025

Christopher Chiu, Silviu Pitis, and Mihaela van der Schaar. Simulating viva voce examinations to evaluate clinical reasoning in large language models, 2025. URL https://arxiv.org/ abs/2510.10278

arXiv 2025

[6] [6]

Kanithi, Tathagata Raha, Shadab Khan, and Marco A

Clément Christophe, Praveen K. Kanithi, Tathagata Raha, Shadab Khan, and Marco A. F. Pimentel. Med42-v2: A Suite of Clinical LLMs.arXiv preprint arXiv:2408.06142, 2024. URL https://arxiv.org/abs/2408.06142

Pith/arXiv arXiv 2024

[7] [7]

Euroeval multilingual european leaderboard

EuroEval. Euroeval multilingual european leaderboard. https://euroeval.com/ leaderboards/Multilingual/european/, 2024. Accessed: 2026-04-05

2024

[8] [8]

Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025

Gemma Team. Gemma 3 Technical Report.arXiv preprint arXiv:2503.19786, 2025. URL https://arxiv.org/abs/2503.19786

Pith/arXiv arXiv 2025

[9] [9]

Gemini 2.5 Flash

Google. Gemini 2.5 Flash. Google Cloud Vertex AI documentation, 2025. URLhttps://docs. cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash . Accessed 2026-04-21

2025

[10] [10]

Cohen, and Xinghua Lu

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China,

2019

[11] [11]

Phillips, and Thomas R

Shitij Kapur, Anthony G. Phillips, and Thomas R. Insel. Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it?Molecular Psychiatry, 17(12): 1174–1179, 2012. doi: 10.1038/mp.2012.105

work page doi:10.1038/mp.2012.105 2012

[12] [12]

Kendell and A

R. Kendell and A. Jablensky. Distinguishing between the validity and utility of psychiatric diagnoses.American Journal of Psychiatry, 160(1):4–12, January 2003. doi: 10.1176/appi.ajp. 160.1.4

work page doi:10.1176/appi.ajp 2003

[13] [13]

Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions.arXiv preprint, 2025. URL https: //arxiv.org/abs/2506.09038. arXiv:2506.09038 [cs.CL]

Pith/arXiv arXiv 2025

[14] [14]

Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov

Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. MEDIQ: question-asking LLMs and a benchmark for reliable interactive clinical reasoning. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY , USA, 2025. Curran Associate...

2025

[15] [15]

Knowing when to abstain: Medical llms under clinical uncertainty, 2026

Sravanthi Machcha, Sushrita Yerra, Sahil Gupta, Aishwarya Sahoo, Sharmin Sultana, Hong Yu, and Zonghai Yao. Knowing when to abstain: Medical llms under clinical uncertainty, 2026. URLhttps://arxiv.org/abs/2601.12471. 10

arXiv 2026

[16] [16]

Mistral Small 3.1

Mistral AI. Mistral Small 3.1. Mistral AI, 2025. URL https://mistral.ai/news/ mistral-small-3-1. Accessed 2026-05-04

2025

[17] [17]

Initial psychiatric assess- ment

MSD Manual Professional Edition. Initial psychiatric assess- ment. MSD Manual Professional Edition, 2026. URL https: //www.msdmanuals.com/professional/psychiatric-disorders/ approach-to-the-patient-with-psychiatric-symptoms/ initial-psychiatric-assessment . Reviewed/Revised October 2024; Modified January 2026; accessed 2026-05-04

2026

[18] [18]

GPT-5.4 Model

OpenAI. GPT-5.4 Model. OpenAI API documentation, 2026. URL https://developers. openai.com/api/docs/models/gpt-5.4. Accessed 2026-04-21

2026

[19] [19]

Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. InProceedings of the Conference on Health, Inference, and Learning, volume 174 ofProceedings of Machine Learning Research, pages 248–260. PMLR, 2022

2022

[20] [20]

Regier, William E

Darrel A. Regier, William E. Narrow, Diana E. Clarke, Helena C. Kraemer, S. Janet Kuramoto, Emily A. Kuhl, and David J. Kupfer. Dsm-5 field trials in the united states and canada, part ii: Test-retest reliability of selected categorical diagnoses.American Journal of Psychiatry, 170(1): 59–70, January 2013. doi: 10.1176/appi.ajp.2012.12070999

Pith/arXiv arXiv 2013

[21] [21]

Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis.Psychiatry Research, 355:116844, 2026

Karthik V Sarma, Kaitlin E Hanss, Andrew J M Halls, Andrew Krystal, Daniel F Becker, Anne L Glowinski, and Atul J Butte. Integrating expert knowledge into large language models improves performance for psychiatric reasoning and diagnosis.Psychiatry Research, 355:116844, 2026. ISSN 0165-1781. doi: 10.1016/j.psychres.2025.116844

arXiv 2026

[22] [22]

MedGemma Technical Report.arXiv preprint arXiv:2507.05201, 2025

Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, et al. MedGemma Technical Report.arXiv preprint arXiv:2507.05201, 2025. URL https: //arxiv.org/abs/2507.05201

Pith/arXiv arXiv 2025

[23] [23]

World Health Organization, Geneva, 2019

World Health Organization.International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10). World Health Organization, Geneva, 2019

2019

[24] [25]

Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. Qwen3 Technical Report.arXiv preprint arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505. 09388

Pith/arXiv arXiv 2025

[25] [26]

committed

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/ abs/2306.05685. 11 A Ethical approval and data governance A.1 Institutional oversight and et...

Pith/arXiv arXiv 2023

[26] [2019]

doi: 10.18653/v1/D19-1259

Association for Computational Linguistics. doi: 10.18653/v1/D19-1259

doi:10.18653/v1/d19-1259