Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking

Alexander Baumann; Finn Fassbender; Ip Chi Wang; Johanna Reimer; Lukas Goede; Markus Hobert; Martje Pauly; Rebecca Herzog; Ronald B\"ock; Sebastian Fudickar

arxiv: 2607.01103 · v1 · pith:7XEZO4ZAnew · submitted 2026-07-01 · 💻 cs.CL

Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking

William Philipp , Finn Fassbender , Thorsten Langer , Martje Pauly , Rebecca Herzog , Alexander Baumann , Markus Hobert , Theresa Paulus

show 6 more authors

Ip Chi Wang Lukas Goede Johanna Reimer Sebastian L\"ons Ronald B\"ock Sebastian Fudickar

This is my paper

Pith reviewed 2026-07-02 12:39 UTC · model grok-4.3

classification 💻 cs.CL

keywords LLM evaluatorsmedical AI benchmarkingclinical cautionGerman medical QAinter-rater agreementabstention behaviorlineage bias

0 comments

The pith

LLM evaluators reach physician-level agreement on a German clinical benchmark but never abstain on difficult items.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MedQADE, an open-response benchmark of 3,800 German clinical items rated by ten physicians. It shows that the strongest LLM evaluator achieves kappa alignment close to the physician ceiling. Physicians increase abstention as item difficulty rises, while every tested frontier model issues a definitive score in all cases. The work also documents systematic biases in which models give higher scores to outputs from architecturally related models. These patterns indicate that matching agreement statistics does not guarantee the presence of clinical metacognition in automated evaluators.

Core claim

Statistical alignment between LLM judges and physician raters on the MedQADE benchmark does not imply equivalent clinical metacognition, because physicians scale abstention with item difficulty while frontier models assign definitive scores in every case; the benchmark additionally reveals lineage-dependent scoring biases that are independent of language.

What carries the argument

The MedQADE benchmark of 3,800 open-response items with ten-physician annotations, used to measure both kappa agreement and abstention rates across LLM evaluators.

If this is right

Agreement metrics alone cannot certify an LLM evaluator as clinically reliable.
Abstention behavior must be measured separately from scoring accuracy.
Lineage biases in scoring occur across languages and require explicit checks.
Open-response medical benchmarks need evaluators that replicate human caution thresholds.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Current LLM-as-judge pipelines may systematically overestimate clinical model performance by never flagging uncertain cases.
Benchmarks in other medical languages or specialties could test whether the metacognition gap is general.
Training or prompting methods that force abstention might close the gap but could lower overall agreement.
The lineage bias suggests that evaluator choice may favor certain model families in multi-model comparisons.

Load-bearing premise

The 3,800 items and ten-physician annotations form a representative ground truth for measuring clinical caution across German medical reasoning.

What would settle it

A replication in which a fresh panel of physicians rates the same 3,800 items and shows no difficulty-linked abstention pattern, or in which an LLM is modified to abstain and then matches physician abstention rates.

read the original abstract

Open-response evaluation provides stronger clinical validity than multiple-choice benchmarks but creates a scoring bottleneck that motivates automated LLM-asa-Judge approaches. Whether such evaluators replicate clinical calibration and caution, however, remains untested. We introduce MedQADE, the first standardised open-response clinical benchmark for German, a major clinical language lacking native evaluation infrastructure, comprising 3,800 items annotated by ten practising physicians and nine Large Language Model (LLM) evaluators. The top-performing evaluator model, Gemini 3 Flash, reached alignment consistent with the physician ceiling (\k{appa} = 0.694 vs. \k{appa} = 0.709), though wide confidence intervals limit interpretation. Despite this statistical alignment, automated evaluators exhibited near-absent clinical metacognition: physicians scaled abstention with item difficulty, while frontier models assigned definitive scores in every case. We additionally quantified systematic lineage-dependent biases, where models preferentially scored architectural siblings, an effect independent of language. These results show that statistical alignment does not ensure clinical caution, and that evaluator independence requires explicit verification.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper's core point is that LLM judges can hit physician-level agreement on a new German clinical benchmark but show no equivalent caution in abstaining on hard items.

read the letter

The paper introduces MedQADE, the first standardized open-response clinical benchmark for German, with 3,800 items annotated by ten physicians. It then runs nine LLM evaluators and reports that the best one (Gemini 3 Flash) reaches kappa close to the human inter-rater ceiling, while also documenting a lineage bias where models score their own architectural relatives higher. The abstention result is the main claim: physicians withheld scores more often on difficult items, but the frontier models gave definitive answers every time.

This is genuinely new infrastructure for a language that had little native evaluation material, and the direct comparison of metacognition-style behavior is a useful angle that prior LLM-as-judge papers have not tested in this setting. The lineage bias finding is also concrete and independent of language.

The work is straightforward in its presentation and the cautionary message about statistical alignment not equaling clinical caution is worth hearing for anyone building medical benchmarks.

The soft spots are real but not fatal. The abstract gives no information on whether the LLM prompts included instructions or options for abstention comparable to what the physicians received; if the models were simply told to score everything, the difference is a prompt artifact rather than evidence of missing metacognition. Item selection, physician inter-rater reliability numbers, and confidence interval methods are also not described, which makes it hard to judge how stable the ground truth is. Wide intervals on the kappa values already limit how strongly the alignment claim can be read.

This is for groups working on medical AI evaluation protocols or non-English clinical NLP. It deserves peer review because the benchmark itself is new and the practical issue it raises is important, provided the prompt and annotation details are clarified in revision.

Referee Report

2 major / 2 minor

Summary. The paper introduces MedQADE, the first standardised open-response clinical benchmark for German comprising 3,800 items annotated by ten practising physicians and nine LLM evaluators. It reports that the top-performing LLM (Gemini 3 Flash) reaches kappa agreement (0.694) close to the physician inter-rater ceiling (0.709), but physicians scale abstention with item difficulty while frontier models assign definitive scores in every case; it also quantifies lineage-dependent biases where models preferentially score architectural siblings.

Significance. If the methodological controls support the claims, the work would be significant for medical AI benchmarking by providing the first native open-response resource for German and demonstrating that statistical alignment with human evaluators does not imply clinical metacognition or caution in automated judges. This could influence standards for LLM-as-judge pipelines in clinical domains and underscore the need to verify evaluator independence explicitly.

major comments (2)

[Abstract] Abstract: The central claim of near-absent clinical metacognition in LLMs (physicians scale abstention with difficulty while models assign definitive scores in every case) is load-bearing for the main conclusion. The manuscript provides no information on whether the nine LLM evaluators received prompts that permitted or encouraged abstention on uncertain items in a manner comparable to the physician instructions; without this, the observed difference may reflect prompt design rather than inherent evaluator limits.
[Abstract] Abstract: Kappa values are reported (0.694 vs. 0.709) along with abstention differences, but the text supplies no details on item selection for the 3,800 benchmark items, inter-rater reliability among the ten physicians, confidence interval calculations, or controls for prompt variation. These omissions directly affect assessment of ground-truth stability and the robustness of the alignment claim.

minor comments (2)

[Abstract] Abstract: The notation \k{appa} should be clarified as Cohen's kappa on first use for readability.
[Abstract] Abstract: The statement that wide confidence intervals limit interpretation is noted but no numerical ranges or calculation method is supplied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful comments. We address each major comment below and have revised the manuscript to incorporate additional methodological details for improved transparency.

read point-by-point responses

Referee: [Abstract] Abstract: The central claim of near-absent clinical metacognition in LLMs (physicians scale abstention with difficulty while models assign definitive scores in every case) is load-bearing for the main conclusion. The manuscript provides no information on whether the nine LLM evaluators received prompts that permitted or encouraged abstention on uncertain items in a manner comparable to the physician instructions; without this, the observed difference may reflect prompt design rather than inherent evaluator limits.

Authors: We agree that the abstract omitted explicit details on abstention instructions for the LLM evaluators. The methods section of the manuscript specifies that LLM prompts included instructions to abstain on uncertain items using language parallel to the physician protocol. To directly address the concern, we have added the full prompt templates used for both physicians and all nine LLM evaluators to a new Appendix A. This revision allows verification that the prompts were designed to be comparable and supports the interpretation that the abstention difference reflects model behavior. revision: yes
Referee: [Abstract] Abstract: Kappa values are reported (0.694 vs. 0.709) along with abstention differences, but the text supplies no details on item selection for the 3,800 benchmark items, inter-rater reliability among the ten physicians, confidence interval calculations, or controls for prompt variation. These omissions directly affect assessment of ground-truth stability and the robustness of the alignment claim.

Authors: We agree these details were not included in the abstract due to space constraints. The main text covers item selection in Section 2, reports the physician inter-rater kappa, and references bootstrap-derived confidence intervals. Prompt variation was controlled via fixed templates. In revision we have expanded the abstract with a brief methods summary and added a dedicated methods subsection plus appendix material detailing item curation criteria, full inter-rater statistics, CI computation, and prompt consistency checks. These changes strengthen assessment of ground-truth stability. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark comparison is self-contained

full rationale

The paper introduces an external benchmark (MedQADE with 3,800 items and ten-physician annotations) and directly measures abstention rates and agreement (kappa values) between physicians and LLM evaluators. No equations, fitted parameters, or self-citations are used to derive the central claim about absent clinical metacognition; the physician scaling of abstention with difficulty is reported as an observed empirical pattern against which model behavior is compared. The derivation chain consists of data collection and statistical comparison rather than any reduction of outputs to inputs by construction. This is a standard empirical benchmarking study with no load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the assumption that physician annotations constitute reliable ground truth and that the sampled items capture clinical difficulty variation; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Physician annotations on the 3,800 items provide a stable and representative measure of clinical judgment quality.
The paper treats the ten-physician labels as the reference standard for both agreement and abstention behavior.

pith-pipeline@v0.9.1-grok · 5764 in / 1174 out tokens · 16314 ms · 2026-07-02T12:39:23.982855+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

40 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Fudickar, S.et al.Natural language processing of referral letters for machine learning–based triaging of patients with low back pain to the most appropri- ate intervention: Retrospective study.Journal of Medical Internet Research26, e46857 (2024)

2024
[2]

Klug, K.et al.From admission to discharge: a systematic review of clinical natural language processing along the patient journey.BMC Medical Informatics and Decision Making24(2024)

2024
[3]

D., Glas, H

Maarseveen, T. D., Glas, H. K., Veris-van Dieren, J., van den Akker, E. & Knevel, R. Improving musculoskeletal care with ai enhanced triage through data driven screening of referral letters.npj Digital Medicine8(2025)

2025
[4]

Busch, F.et al.Current applications and challenges in large language models for patient care: a systematic review.Communications Medicine5(2025)

2025
[5]

D.et al.Development and evaluation of a clinical note summarization system using large language models.Communications Medicine5(2025)

Oliveira, J. D.et al.Development and evaluation of a clinical note summarization system using large language models.Communications Medicine5(2025)

2025
[6]

Mandal, S.et al.Utilization of generative ai-drafted responses for managing patient-provider communication.npj Digital Medicine8(2025)

2025
[7]

& Lu, X.Pubmedqa: A dataset for biomedical research question answering

Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X.Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International JointConferenceonNaturalLanguageProcessing(EMNLP-IJCNLP),2567–2577 (Association for Computational Linguistics, 2019)

2019
[8]

Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa : A large-scale multi- subject multi-choice dataset for medical domain question answering.arXiv preprint arXiv:2203.14371(2022)

work page arXiv 2022
[9]

Jin, D.et al.What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences11, 6421 (2021)

2021
[10]

& Beskow, J

Moëll, B., Farestam, F. & Beskow, J. Swedish medical llm benchmark: develop- ment and evaluation of a framework for assessing large language models in the swedish medical domain.Frontiers in Artificial Intelligence8(2025)

2025
[11]

S., Łaba, J., Korzeniewski, K

Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K. & Młyńczak, M. Evaluation of the performance of gpt-3.5 and gpt-4 on the polish medical final examination. Scientific Reports13(2023). 23

2023
[12]

Luo, D.et al.Evaluating the performance of gpt-3.5, gpt-4, and gpt-4o in the chinese national medical licensing examination.Scientific Reports15(2025)

2025
[13]

Germediq: A resource for simulated and synthesized anam- nesis interview responses in german

Hofenbitzer, J.et al. Germediq: A resource for simulated and synthesized anam- nesis interview responses in german. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), 1064–1078 (Association for Computational Linguistics, 2025)

2025
[14]

Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?

Doll, N.et al.Can continual pre-training bridge the performance gap between general-purpose and specialized language models in the medical domain? (2026). URL https://arxiv.org/abs/2604.19394

work page internal anchor Pith review Pith/arXiv arXiv 2026
[15]

Measuring massive multitask language understanding

Hendrycks, D.et al. Measuring massive multitask language understanding. Pro- ceedings of the International Conference on Learning Representations (ICLR) (2021)

2021
[16]

Rowland, C. A. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.Psychological Bulletin140, 1432–1463 (2014)

2014
[17]

W., McCarthy, T., Schlinsog, A

Procop, G. W., McCarthy, T., Schlinsog, A. & Ghofrani, M. A comparison of free-response and multiple-choice questions on the american board of pathology primary certification examinations.Academic Pathology13, 100248 (2026)

2026
[18]

Sheaffer, E. A. & Addo, R. T. Pharmacy student performance on constructed- response versus selected-response calculations questions.American Journal of Pharmaceutical Education77, 6 (2013)

2013
[19]

Singh, S.et al.The pitfalls of multiple-choice questions in generative ai and medical education.Scientific Reports15(2025)

2025
[20]

Cocchieri, A., Ragazzi, L., Tagliavini, G. & Moro, G.Remedqa: Are we done with medical multiple-choice benchmarks?Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2706–2738 (Association for Computational Linguistics, 2026)

2026
[21]

Croxford, E.et al.Evaluating clinical ai summaries with large language models as judges.npj Digital Medicine8(2025)

2025
[22]

& Zulkernine, F

Chen, Y., Wen, B. & Zulkernine, F. A multiagent summarization and auto- evaluation framework for medical text: Development and evaluation study.JMIR AI4, e75932–e75932 (2025)

2025
[23]

& Panickssery, A.Llm evaluators recognize and favor their own generations

Bowman, S., Feng, S. & Panickssery, A.Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems 37, NeurIPS2024,68772–68802(NeuralInformationProcessingSystemsFoundation, Inc. (NeurIPS), 2024). 24

2024
[24]

& Panickssery, N

Ackerman, C. & Panickssery, N. Inspection and control of self-generated-text recognition ability in llama3-8b-instruct (2024)

2024
[25]

Gemini 3 flash – model card

Google DeepMind. Gemini 3 flash – model card. https://deepmind.google/ models/model-cards/gemini-3-flash/ (2025)

2025
[26]

Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data.Biometrics33, 159–174 (1977)

1977
[27]

Gemma 3 technical report (2025)

Gemma Team, Google DeepMind. Gemma 3 technical report (2025)

2025
[28]

Qwen3 technical report (2025)

Qwen Team, Alibaba Group. Qwen3 technical report (2025)

2025
[29]

Computing krippendorff’s alpha-reliability

Krippendorff, K. Computing krippendorff’s alpha-reliability. Technical report, University of Pennsylvania (2011)

2011
[30]

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025)

Gemini Team, Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025)

2025
[31]

Gpt-5 system card (2025)

OpenAI. Gpt-5 system card (2025)

2025
[32]

Gpt-5.4 thinking system card

OpenAI. Gpt-5.4 thinking system card. https://openai.com/index/ gpt-5-4-thinking-system-card/ (2026)

2026
[33]

Approbationsordnung für Ärzte (Äappro)

Bundesministerium der Justiz. Approbationsordnung für Ärzte (Äappro). https: //www.gesetze-im-internet.de/_appro_2002/ (2002). Zuletzt geändert durch Art. 1 V v. 12.1.2023 (BGBl. 2023 I Nr. 18)

2002
[34]

Ankizin: Offene karteikarten für das medizinstudium

Ankizin Project Team. Ankizin: Offene karteikarten für das medizinstudium. https://ankizin.de (2024). Accessed: 2026-05-18

2024
[35]

Better zero-shot reasoning with role-play prompting

Kong, A.et al. Better zero-shot reasoning with role-play prompting. Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 4099–4113 (Association for Computational Linguistics, 2024)

2024
[36]

Li, C.et al.Large language models understand and can be enhanced by emotional stimuli (2023)

2023
[37]

Davis, J., Van Bulck, L., Durieux, B. N. & Lindvall, C. The temperature feature of chatgpt: Modifying creativity for clinical research.JMIR Human Factors11, e53559 (2024)

2024
[38]

Kim,S.et al.Prometheus:Inducingfine-grainedevaluationcapabilityinlanguage models (2023). 25

2023
[39]

Chain-of-thought prompting elicits reasoning in large lan- guage models

Bosma, M.et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in Neural Information Processing Systems 35, NeurIPS 2022, 24824–24837 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022)

2022
[40]

& Carlin, J

Byrt, T., Bishop, J. & Carlin, J. B. Bias, prevalence and kappa.Journal of Clinical Epidemiology46, 423–429 (1993). 26

1993

[1] [1]

Fudickar, S.et al.Natural language processing of referral letters for machine learning–based triaging of patients with low back pain to the most appropri- ate intervention: Retrospective study.Journal of Medical Internet Research26, e46857 (2024)

2024

[2] [2]

Klug, K.et al.From admission to discharge: a systematic review of clinical natural language processing along the patient journey.BMC Medical Informatics and Decision Making24(2024)

2024

[3] [3]

D., Glas, H

Maarseveen, T. D., Glas, H. K., Veris-van Dieren, J., van den Akker, E. & Knevel, R. Improving musculoskeletal care with ai enhanced triage through data driven screening of referral letters.npj Digital Medicine8(2025)

2025

[4] [4]

Busch, F.et al.Current applications and challenges in large language models for patient care: a systematic review.Communications Medicine5(2025)

2025

[5] [5]

D.et al.Development and evaluation of a clinical note summarization system using large language models.Communications Medicine5(2025)

Oliveira, J. D.et al.Development and evaluation of a clinical note summarization system using large language models.Communications Medicine5(2025)

2025

[6] [6]

Mandal, S.et al.Utilization of generative ai-drafted responses for managing patient-provider communication.npj Digital Medicine8(2025)

2025

[7] [7]

& Lu, X.Pubmedqa: A dataset for biomedical research question answering

Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X.Pubmedqa: A dataset for biomedical research question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International JointConferenceonNaturalLanguageProcessing(EMNLP-IJCNLP),2567–2577 (Association for Computational Linguistics, 2019)

2019

[8] [8]

Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa : A large-scale multi- subject multi-choice dataset for medical domain question answering.arXiv preprint arXiv:2203.14371(2022)

work page arXiv 2022

[9] [9]

Jin, D.et al.What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences11, 6421 (2021)

2021

[10] [10]

& Beskow, J

Moëll, B., Farestam, F. & Beskow, J. Swedish medical llm benchmark: develop- ment and evaluation of a framework for assessing large language models in the swedish medical domain.Frontiers in Artificial Intelligence8(2025)

2025

[11] [11]

S., Łaba, J., Korzeniewski, K

Rosoł, M., Gąsior, J. S., Łaba, J., Korzeniewski, K. & Młyńczak, M. Evaluation of the performance of gpt-3.5 and gpt-4 on the polish medical final examination. Scientific Reports13(2023). 23

2023

[12] [12]

Luo, D.et al.Evaluating the performance of gpt-3.5, gpt-4, and gpt-4o in the chinese national medical licensing examination.Scientific Reports15(2025)

2025

[13] [13]

Germediq: A resource for simulated and synthesized anam- nesis interview responses in german

Hofenbitzer, J.et al. Germediq: A resource for simulated and synthesized anam- nesis interview responses in german. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), 1064–1078 (Association for Computational Linguistics, 2025)

2025

[14] [14]

Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?

Doll, N.et al.Can continual pre-training bridge the performance gap between general-purpose and specialized language models in the medical domain? (2026). URL https://arxiv.org/abs/2604.19394

work page internal anchor Pith review Pith/arXiv arXiv 2026

[15] [15]

Measuring massive multitask language understanding

Hendrycks, D.et al. Measuring massive multitask language understanding. Pro- ceedings of the International Conference on Learning Representations (ICLR) (2021)

2021

[16] [16]

Rowland, C. A. The effect of testing versus restudy on retention: A meta-analytic review of the testing effect.Psychological Bulletin140, 1432–1463 (2014)

2014

[17] [17]

W., McCarthy, T., Schlinsog, A

Procop, G. W., McCarthy, T., Schlinsog, A. & Ghofrani, M. A comparison of free-response and multiple-choice questions on the american board of pathology primary certification examinations.Academic Pathology13, 100248 (2026)

2026

[18] [18]

Sheaffer, E. A. & Addo, R. T. Pharmacy student performance on constructed- response versus selected-response calculations questions.American Journal of Pharmaceutical Education77, 6 (2013)

2013

[19] [19]

Singh, S.et al.The pitfalls of multiple-choice questions in generative ai and medical education.Scientific Reports15(2025)

2025

[20] [20]

Cocchieri, A., Ragazzi, L., Tagliavini, G. & Moro, G.Remedqa: Are we done with medical multiple-choice benchmarks?Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2706–2738 (Association for Computational Linguistics, 2026)

2026

[21] [21]

Croxford, E.et al.Evaluating clinical ai summaries with large language models as judges.npj Digital Medicine8(2025)

2025

[22] [22]

& Zulkernine, F

Chen, Y., Wen, B. & Zulkernine, F. A multiagent summarization and auto- evaluation framework for medical text: Development and evaluation study.JMIR AI4, e75932–e75932 (2025)

2025

[23] [23]

& Panickssery, A.Llm evaluators recognize and favor their own generations

Bowman, S., Feng, S. & Panickssery, A.Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems 37, NeurIPS2024,68772–68802(NeuralInformationProcessingSystemsFoundation, Inc. (NeurIPS), 2024). 24

2024

[24] [24]

& Panickssery, N

Ackerman, C. & Panickssery, N. Inspection and control of self-generated-text recognition ability in llama3-8b-instruct (2024)

2024

[25] [25]

Gemini 3 flash – model card

Google DeepMind. Gemini 3 flash – model card. https://deepmind.google/ models/model-cards/gemini-3-flash/ (2025)

2025

[26] [26]

Landis, J. R. & Koch, G. G. The measurement of observer agreement for categorical data.Biometrics33, 159–174 (1977)

1977

[27] [27]

Gemma 3 technical report (2025)

Gemma Team, Google DeepMind. Gemma 3 technical report (2025)

2025

[28] [28]

Qwen3 technical report (2025)

Qwen Team, Alibaba Group. Qwen3 technical report (2025)

2025

[29] [29]

Computing krippendorff’s alpha-reliability

Krippendorff, K. Computing krippendorff’s alpha-reliability. Technical report, University of Pennsylvania (2011)

2011

[30] [30]

Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025)

Gemini Team, Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities (2025)

2025

[31] [31]

Gpt-5 system card (2025)

OpenAI. Gpt-5 system card (2025)

2025

[32] [32]

Gpt-5.4 thinking system card

OpenAI. Gpt-5.4 thinking system card. https://openai.com/index/ gpt-5-4-thinking-system-card/ (2026)

2026

[33] [33]

Approbationsordnung für Ärzte (Äappro)

Bundesministerium der Justiz. Approbationsordnung für Ärzte (Äappro). https: //www.gesetze-im-internet.de/_appro_2002/ (2002). Zuletzt geändert durch Art. 1 V v. 12.1.2023 (BGBl. 2023 I Nr. 18)

2002

[34] [34]

Ankizin: Offene karteikarten für das medizinstudium

Ankizin Project Team. Ankizin: Offene karteikarten für das medizinstudium. https://ankizin.de (2024). Accessed: 2026-05-18

2024

[35] [35]

Better zero-shot reasoning with role-play prompting

Kong, A.et al. Better zero-shot reasoning with role-play prompting. Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 4099–4113 (Association for Computational Linguistics, 2024)

2024

[36] [36]

Li, C.et al.Large language models understand and can be enhanced by emotional stimuli (2023)

2023

[37] [37]

Davis, J., Van Bulck, L., Durieux, B. N. & Lindvall, C. The temperature feature of chatgpt: Modifying creativity for clinical research.JMIR Human Factors11, e53559 (2024)

2024

[38] [38]

Kim,S.et al.Prometheus:Inducingfine-grainedevaluationcapabilityinlanguage models (2023). 25

2023

[39] [39]

Chain-of-thought prompting elicits reasoning in large lan- guage models

Bosma, M.et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in Neural Information Processing Systems 35, NeurIPS 2022, 24824–24837 (Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022)

2022

[40] [40]

& Carlin, J

Byrt, T., Bishop, J. & Carlin, J. B. Bias, prevalence and kappa.Journal of Clinical Epidemiology46, 423–429 (1993). 26

1993