Pith. sign in

REVIEW 3 major objections 5 minor 76 references

Can open source large language models be used for tumor documentation in Germany? -- An evaluation on urological doctors' notes

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Open-source language models with 7 to 12 billion parameters can carry out the core German tumor documentation tasks at around 90 percent accuracy with prompt design alone.

desk verdict The released German urology dataset is the real contribution; the headline accuracies are optimistic upper bounds because best prompt variants were selected on the same test set, but the paper is a solid, honest incremental evaluation. read the letter →

arxiv 2501.12106 v4 pith:OVPRUH7N submitted 2025-01-21 cs.CL cs.AI

Stefan Lenz , Arsenij Ustjanzew , Marco Jeray , Meike Ressing , Torsten Panholzer This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelstumordocumentationGermanmedicalNLPICD-10codinginformationextractionfew-shotpromptingurologydoctors'notesclinicaltextmining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether open-source large language models can take over parts of the German tumor documentation process, which is now done by staff who read doctors' letters and enter structured data into cancer registries. On an annotated set of 149 real urology text snippets, it runs eleven models from 1 to 70 billion parameters through three tasks: detecting tumor diagnoses, assigning the corresponding three-character ICD-10 codes, and extracting the date of first diagnosis. The central finding is that mid-sized models—Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B—reach roughly 90 percent accuracy on diagnosis detection and coding and up to 92 percent on the date task, while models with fewer than 7 billion parameters or much less training data lag behind and models up to 70 billion parameters show no clear gain. Because German data-protection law makes cloud LLMs impractical for patient data, the paper argues that locally deployable mid-sized models offer the best balance between performance and resource use, and it releases the annotated dataset as a benchmark for German medical NLP.

What carries the argument

The argument rests on a three-step prompting pipeline plus a public evaluation dataset. Step 1 instructs the model to return tumor diagnoses from a text snippet as a JSON array; Step 2 asks for the three-character ICD-10 code, the standard international disease classification, for each extracted label; Step 3 extracts the first diagnosis date either directly or by asking the model to verify dates found by a regular expression. Ground truth is a set of 149 snippets from anonymized urological doctors' notes, annotated by four independent raters with agreement around 0.84–0.90 on the key variables. The prompts vary the number and domain of few-shot examples, contrasting urology with gynecology, which lets the paper separate model knowledge from prompt guidance.

What would settle it

Run the same three tasks on the released dataset with Llama 3.1 70B and Mixtral 8x7B at full precision instead of 8-bit; if they then clearly outscore Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B, the claim that mid-sized models are the optimal resource-performance balance would lose its main evidence.

Watch

Extended reading notes

Core claim

The paper claims that the three tasks at the heart of German tumor documentation—extracting tumor diagnoses from text, coding them with ICD-10 codes, and finding the date of first diagnosis—can be handled by open-source LLMs in the 7–12 billion parameter range without any fine-tuning, using carefully designed prompts alone. On its 149-snippet urology dataset, Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B all perform comparably, with diagnosis detection and ICD-10 coding around 90 percent accuracy and first-diagnosis-date extraction reaching 92 percent for Mistral NeMo 12B in the best configuration. The authors further claim that larger models such as Mixtral 8x7B and Llama 3.1 70B do not consistently outperform these mid-sized models, that smaller models or models trained on less data clearly fall behind, and that additional German or medical training (as in BioMistral and SauerkrautLM) does not automatically beat the base models. They present this as evidence that mid-sized open-source models are the practical starting point for automating tumor documentation under German data protection constraints, and they release the annotated dataset so other groups can build on the comparison.

Load-bearing premise

The comparison assumes that compressing the three largest models to use less memory (8-bit quantization) does not noticeably hurt their scores, so the conclusion that bigger models are no better is not an artifact of the compression.

Editorial extensions

If this is right

  • On these three documentation subtasks, a hospital could run the whole pipeline locally on a single modern GPU, avoiding the legal problem of sending patient text to a cloud LLM.
  • The absence of a consistent accuracy gain from 46B- and 70B-parameter models means that, for tumor documentation, spending additional compute on larger models is not justified by accuracy alone.
  • Since the fine-tuned medical or German models did not beat the corresponding base models, choosing a strong general-purpose base model and then designing or tuning the prompt may matter more than buying a domain-specific model.
  • Cross-domain few-shot examples from gynecology worked as well as domain-matched examples for urology notes, implying that the model learns the abstract definition of a tumor diagnosis rather than memorizing vocabulary.
  • The released dataset gives German clinical NLP a small but authentic public benchmark, so later work can compare new models or prompting strategies on the same real-world text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension the paper does not run is fine-tuning the best 7-12B models on its released dataset; if fine-tuning closes the remaining error gap, the same pipeline could plausibly reach clinical usability.
  • The size comparison is only as clean as the quantization step; re-running Llama 3.1 70B and Mixtral 8x7B at full precision on this dataset would show whether the 'no gain from larger models' result is robust.
  • The dataset's small size and urology-only origin mean the absolute accuracy numbers are anchors rather than population estimates; replicating the three-step pipeline on other tumor entities and note types is the natural next check of generality.
  • A retrieval-augmented variant that feeds the model relevant ICD-10 catalogue entries before coding could address the remaining coding errors without moving to larger models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper evaluates eleven open-source large language models (1B–70B parameters) on three tasks relevant to German tumor documentation: detecting tumor diagnoses in urological doctors' notes, assigning ICD-10 codes, and extracting the first diagnosis date. Using a newly released dataset of 149 annotated German text snippets from 25 patients, the authors compare multiple prompt variants (zero-shot and few-shot, with urology-specific or gynecological examples). They report that Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B achieve around 90% accuracy for diagnosis detection and ICD-10 coding and up to 92% for first-diagnosis-date extraction, and that larger models do not show clear gains. The paper concludes that 7–12B open-source models could offer an optimal balance of performance and resource efficiency for local deployment, and it releases code and data.

Significance. If the quantitative findings were reliable, this study would provide a useful practical benchmark for German-language clinical NLP and guidance for local LLM deployment in cancer registries. The paper's strengths include a clearly documented annotation process with high inter-rater agreement (kappa about 0.90 for diagnosis presence, Krippendorff's alpha about 0.85 for ICD-10 codes), release of the dataset and evaluation code, and a careful description of prompt variants. However, the central performance claims rest on the selection of the best prompt variant per model on the same test set used for reporting accuracy, and on a very small, domain-restricted sample; these issues limit the strength of the conclusions as they stand.

major comments (3)
  1. [Methods (Prompt design), Figures 2–5, Tables A-1–A-3] The reported accuracy for each model is the maximum over the 7–14 prompt variants evaluated on the same 149-snippet test set, and the methods state that prompts were manually adjusted after 'preliminary results.' Selecting the best variant on the test set and then reporting that value as the model's performance inflates the results through multiple comparisons; for example, Llama 3.1 8B Step 1 accuracy ranges from 0.74 (zero-shot) to 0.98 (best few-shot) and Mistral 7B from 0.76 to 0.93 in Table A-1. The paper's central claims about '~90% accuracy' and the absence of gains from larger models are therefore based on selected maxima rather than a pre-specified evaluation protocol. The authors should report the full distribution of results across prompt variants, use a held-out validation set or nested cross-validation for prompt selection, or otherwise demonstrate that the reported values are not inflated by selection.
  2. [Results, Table 1, Table A-2] The evaluation set contains only 149 snippets from 25 patients, with substantial clustering (multiple snippets per patient) and class imbalance (75% of snippets contain a tumor diagnosis; 69% are prostate cancer). The differences between the best-performing 7–12B models and the larger models are within binomial sampling error at this sample size; for instance, in Table A-2 the 'Snippet correct' values for Llama 3.1 8B and Mixtral 8x7B differ by only 0.01–0.03, which is not significant without confidence intervals. The manuscript should report confidence intervals or statistical comparisons that account for snippet clustering and multiple testing before claiming that 7–12B models exhibit an optimal performance/resource trade-off.
  3. [Software implementation, Discussion] The largest models (Mistral NeMo 12B, Mixtral 8x7B, Llama 3.1 70B) were run with 8-bit quantization, while the smaller models were run unquantized. Because quantization can degrade model quality, the finding that larger models 'did not display performance gains' is confounded: it may reflect the effects of quantization rather than a true property of model scale. The authors should either run the larger models without quantization, add a control experiment measuring the effect of 8-bit quantization on a subset of the small models, or substantially soften the size-related conclusions. This concern also affects the recommendation of the 7–12B sweet spot, since one of the recommended models (Mistral NeMo 12B) was quantized.
minor comments (5)
  1. [Background, first paragraph] The sentence 'LLMs shown impressive results' should read 'LLMs have shown impressive results'.
  2. [Methods, Data set preparation; Discussion, Contribution to openly available German clinical data] The text states that the initial data basis comprises 153 doctors' notes and that the dataset contains 149 annotated text snippets; in the Discussion the authors write 'The number of letters in our data set is very similar (153)'. Please clarify whether 'letters' refers to the original PDFs or to the snippet set, as the two numbers appear conflated.
  3. [Figures 2–5] The bars indicate the best value with respect to accuracy, but the figure legends do not list all the colored prompt-type markers; the reader must consult the interactive supplement. Consider adding a complete legend or a separate table mapping marker numbers to prompt types.
  4. [Methods, Software implementation] The choice of temperature 0 is reported, but the sampling parameters (e.g., top-p, max tokens) are not specified; since reproducibility is claimed, these should be stated.
  5. [Results, Step 3] The text reports that 'the best results were achieved by Mistral NeMo 12B, which identified the first diagnosis date with an accuracy of 92% in the zero-shot approach,' but Table A-3 shows that this 0.92 value is obtained when using the 'results from best models in previous steps'; the sentence should specify the pipeline configuration to avoid ambiguity.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the reported accuracies are empirical evaluations, and the disclosed prompt-selection bias is a methodological limitation, not a circular reduction.

full rationale

This paper is an empirical benchmark, not a derivation: it evaluates eleven open-source LLMs on three information-extraction tasks against a manually annotated dataset of German urological doctors' notes. There is no mathematical derivation in which an output quantity is shown to be equivalent to an input quantity by construction. The only self-citation is reference [9], the authors' own prior conference abstract, cited for the data preparation steps ('The Excel document was then transformed into an XML file that served as the primary basis for the experiments [9]' and 'The cases with differing annotations were analyzed and discussed, which resulted in a limited amount of changes to the original annotation [9]'). This is not load-bearing for the central performance claims: the current paper describes the annotation process in detail, reports re-annotation by three independent annotators with inter-rater agreement statistics, and releases the dataset publicly, so the benchmark stands on its own description and data rather than on the self-citation. The more substantive concern is that prompts were manually adjusted after 'preliminary results' and that Figures 2-5 report the best prompt variant per model on the same 149 snippets used for the final accuracy numbers. This is a real evaluation-bias problem (optimistic selection over prompt variants and multiple comparisons), and the paper partly acknowledges it by cautioning that 'the absolute numbers of the different metrics reported here might not reflect the exact overall performance of the models.' However, this is not circularity under the definitions used here: no accuracy value is definitionally equal to a fitted parameter, no 'prediction' is forced by the input data, and the central claim is an empirical comparison, not a theorem resting on a self-citation. Therefore the appropriate finding is no significant circularity, with a low score reflecting the minor non-load-bearing self-citation and the disclosed prompt-selection limitation.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new entities are introduced. The central empirical claims rest on the quality of the annotation, the completeness of the text extraction, and the fairness of the model comparison. The main unmodeled factors are prompt selection on the test set and the quantization of larger models.

free parameters (1)
  • Per-model best prompt variant = Selected from 7 prompt variants (e.g., 6-uro for Llama 3.1 8B in Step 1)
    The paper reports the best accuracy over 7 prompt variants per model (Figure 2 caption: 'bars indicate the best value/prompt type'). This selection is made on the same test set used for reporting, effectively fitting a discrete prompt hyperparameter to the data.
assumptions (4)
  • domain assumption The four annotators' consensus is an accurate ground truth for tumor diagnoses, ICD-10 codes, and first diagnosis dates.
    The whole evaluation compares model outputs to this annotation. The paper reports high inter-rater agreement (Fleiss kappa 0.90, Krippendorff alpha 0.85 for codes, 0.84 for dates), but ambiguity remained, especially for dates, which were harmonized by excluding uncertain cases.
  • domain assumption The PDF text extraction using PyMuPDF preserved the relevant information needed for the three tasks.
    The snippets were automatically extracted by clustering text blocks with 'diagnosis' headings; if the extraction dropped or garbled context, model performance could be underestimated.
  • domain assumption The German tumor documentation definition of first diagnosis date is correctly operationalized in the annotation and prompts.
    The paper notes difficulty in deciding whether dates refer to treatment or diagnosis and chose only clear cases, so the annotation may not match the full regulatory definition.
  • ad hoc to paper The instruction to avoid C77, C78, C79 is appropriate and does not bias the coding task.
    This instruction was added after observing that some models returned C78 for metastases. It prevents a class of errors, but it is a task-specific rule that would not apply to general ICD coding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can open source large language models be used for tumor documentation in Germany? -- An evaluation on urological doctors' notes." pith.science (2026). https://pith.science/paper/OVPRUH7N

@misc{pith2026250112106,
  author       = {Pith},
  title        = {Pith review of: Can open source large language models be used for tumor documentation in Germany? -- An evaluation on urological doctors' notes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OVPRUH7N}},
  note         = {Machine review of arXiv:2501.12106}
}
read the original abstract

Tumor documentation in Germany is largely done manually, requiring reading patient records and entering data into structured databases. Large language models (LLMs) could potentially enhance this process by improving efficiency and reliability. This evaluation tests eleven different open source LLMs with sizes ranging from 1-70 billion model parameters on three basic tasks of the tumor documentation process: identifying tumor diagnoses, assigning ICD-10 codes, and extracting the date of first diagnosis. For evaluating the LLMs on these tasks, a dataset of annotated text snippets based on anonymized doctors' notes from urology was prepared. Different prompting strategies were used to investigate the effect of the number of examples in few-shot prompting and to explore the capabilities of the LLMs in general. The models Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12 B performed comparably well in the tasks. Models with less extensive training data or having fewer than 7 billion parameters showed notably lower performance, while larger models did not display performance gains. Examples from a different medical domain than urology could also improve the outcome in few-shot prompting, which demonstrates the ability of LLMs to handle tasks needed for tumor documentation. Open source LLMs show a strong potential for automating tumor documentation. Models from 7-12 billion parameters could offer an optimal balance between performance and resource efficiency. With tailored fine-tuning and well-designed prompting, these models might become important tools for clinical documentation in the future. The code for the evaluation is available from https://github.com/stefan-m-lenz/UroLlmEval. We also release the dataset as a new valuable resource that addresses the shortage of authentic and easily accessible benchmarks in German-language medical NLP.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 54 canonical work pages

  1. [1]

    Bekanntmachung - Aktualisierter einheitlicher onkologischer Basisdatensatz der Arbeitsgemeinschaft Deutscher Tumorzentren e

    Holland J. Bekanntmachung - Aktualisierter einheitlicher onkologischer Basisdatensatz der Arbeitsgemeinschaft Deutscher Tumorzentren e. V. (ADT) und der Gesellschaft der epidemiologischen Krebsregister in Deutschland e. V. (GEKID). Bundesanzeiger. 2021;BAnz AT 12.07.2021 B4

    work page 2021

  2. [2]

    Der einheitliche onkologische Basisdatensatz (oBDS)

    Klinkhammer-Schalke M, Kleihues van Tol K, Jurkschat R, Meyer M, Katalinic A, Holleczek B, et al. Der einheitliche onkologische Basisdatensatz (oBDS). Forum [Internet]. 2024 [cited 2025 May 5];39:191–5. Available from: https://doi.org/10.1007/s12312-024-01320-1

    work page doi:10.1007/s12312-024-01320-1 2024

  3. [3]

    ICD-10-GM Version 2024, Systematisches Verzeichnis, Internationale statistische Klassifikation der Krankheiten und verwandter Gesundheitsprobleme, 10

    Bundesinstitut für Arzneimittel und Medizinprodukte. ICD-10-GM Version 2024, Systematisches Verzeichnis, Internationale statistische Klassifikation der Krankheiten und verwandter Gesundheitsprobleme, 10. Revision [Internet]. 2024 [cited 2024 Jun 12]. Available from: https://www.bfarm.de/DE/Kodiersysteme/Klassifikationen/ICD/ICD-10-GM/_node.html

    work page 2024

  4. [4]

    GPT-4 Technical Report [Internet]

    OpenAI, Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, et al. GPT-4 Technical Report [Internet]. arXiv; 2024 [cited 2024 Oct 24]. Available from: http://arxiv.org/abs/2303.08774

    arXiv 2024

  5. [5]

    A systematic evaluation of large language models of code

    Xu FF, Alon U, Neubig G, Hellendoorn VJ. A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming [Internet]. New York, NY, USA: Association for Computing Machinery; 2022 [cited 2025 Jan 9]. p. 1–10. Available from: https://dl.acm.org/doi/10.1145/3520312.3534862

    arXiv 2022

  6. [6]

    Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step

    Zhong L, Wang Z, Shang J. Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step. In: Ku L-W, Martins A, Srikumar V, editors. Findings of the Association for Computational Linguistics ACL 2024 [Internet]. Bangkok, Thailand and virtual meeting: Association for Computational Linguistics; 2024 [cited 2024 Oct 24]. p....

    2024

  7. [7]

    Monitoring Patients with Glioblastoma by Using a Large Language Model: Accurate Summarization of Radiology Reports with GPT-4

    Laukamp KR, Terzis RA, Werner J-M, Galldiks N, Lennartz S, Maintz D, et al. Monitoring Patients with Glioblastoma by Using a Large Language Model: Accurate Summarization of Radiology Reports with GPT-4. Radiology [Internet]. 2024 [cited 2024 Oct 24];312:e232640. Available from: https://pubs.rsna.org/doi/10.1148/radiol.232640

    work page doi:10.1148/radiol.232640 2024

  8. [8]

    Leveraging GPT-4 for Post Hoc Transformation of Free-Text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study

    Adams LC, Truhn D, Busch F, Kader A, Niehues SM, Makowski MR, et al. Leveraging GPT-4 for Post Hoc Transformation of Free-Text Radiology Reports into Structured Reporting: A Multilingual Feasibility Study. Radiology [Internet]. 2023 [cited 2023 Apr 12];230725. Available from: https://pubs.rsna.org/doi/10.1148/radiol.230725

    doi:10.1148/radiol.230725 2023

Show all 76 references
  1. [9]

    Gesundheit - gemeinsam

    Lenz S, Ustjanzew A, Jeray M, Panholzer T. Few-Shot-Prompting von Large Language Models zur Extraktion von Daten zu Tumordiagnosen aus urologischen Arztbriefen – eine Evaluation. GMDS Kooperationstagung “Gesundheit - gemeinsam” 2024 [Internet]. German Medical Science GMS Publi...

    2024 doi

  2. [10]

    Privacy-preserving large language models for structured medical information retrieval

    Wiest IC, Ferber D, Zhu J, van Treeck M, Meyer SK, Juglan R, et al. Privacy-preserving large language models for structured medical information retrieval. npj Digit Med [Internet]. 2024 [cited 2024 Sep 26];7:1–9. Available from: https://www.nature.com/articles/s41746-024-01233- 2

    2024

  3. [11]

    On the Opportunities and Risks of Foundation Models

    Bommasani R, Hudson DA, Adeli E, Altman R, Arora S, von Arx S, et al. On the Opportunities and Risks of Foundation Models. arXiv:210807258 [cs] [Internet]. 2021 [cited 2021 Sep 21]; Available from: https://crfm.stanford.edu/assets/report.pdf

    2021

  4. [12]

    Schwerpunkt künstliche Intelligenz in der Medizin – rechtliche Aspekte bei der Nutzung großer Sprachmodelle im klinischen Alltag

    Weicken E, Mittermaier M, Hoeren T, Kliesch J, Wiegand T, Witzenrath M, et al. Schwerpunkt künstliche Intelligenz in der Medizin – rechtliche Aspekte bei der Nutzung großer Sprachmodelle im klinischen Alltag. Inn Med (Heidelb) [Internet]. 2025 [cited 2025 Apr 7];66:436–41. Ava...

    2025

  5. [13]

    Gesetz zur Zusammenführung von Krebsregisterdaten [Law on the Consolidation of Cancer Registry Data] [Internet]

    Federal Republic of Germany. Gesetz zur Zusammenführung von Krebsregisterdaten [Law on the Consolidation of Cancer Registry Data] [Internet]. Bundesgesetzblatt Jahrgang 2021 Teil I Nr

    2021

  6. [14]

    The TRIPOD-LLM reporting guideline for studies using large language models

    Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med [Internet]. 2025 [cited 2025 Jan 9];1–10. Available from: https://www.nature.com/articles/s41591-024- 03425-5

    2025

  7. [15]

    [Large Language Models for Rapid Simplification of Quality Assurance Data Input: Field Trial with Real Data in the Context of Tumour Documentation in Urology]

    Frank J, Merseburger AS, Landmesser J, Brozat-Essen S, Schramm P, Freimann L, et al. [Large Language Models for Rapid Simplification of Quality Assurance Data Input: Field Trial with Real Data in the Context of Tumour Documentation in Urology]. Aktuelle Urol [Internet]. 2024 [...

    2024 doi

  8. [16]

    Lost in the Middle: How Language Models Use Long Contexts

    Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics [Internet]. 2024 [cited 2025 Apr 17];12:157–73. Available from: https://aclanthology.o...

    2024

  9. [17]

    statsmodels: Econometric and statistical modeling with python

    Seabold S, Perktold J. statsmodels: Econometric and statistical modeling with python. 9th Python in Science Conference. 2010

    2010

  10. [18]

    Fast Krippendorff: Fast computation of Krippendorff’s alpha agreement measure [Internet]

    Castro S. Fast Krippendorff: Fast computation of Krippendorff’s alpha agreement measure [Internet]. GitHub repository. GitHub; 2017. Available from: https://github.com/pln-fing- udelar/fast-krippendorff

    2017

  11. [19]

    The Measurement of Observer Agreement for Categorical Data

    Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics [Internet]. 1977 [cited 2025 May 2];33:159–74. Available from: https://www.jstor.org/stable/2529310

    1977

  12. [20]

    PyMuPDF documentation [Internet]

    Artifex Software Inc. PyMuPDF documentation [Internet]. 2025 [cited 2025 Apr 23]. Available from: https://pymupdf.readthedocs.io

    2025

  13. [21]

    Model card of Command R+ on HuggingFace [Internet]

    Cohere For AI. Model card of Command R+ on HuggingFace [Internet]. 2024 [cited 2024 Apr 8]. Available from: https://huggingface.co/CohereForAI/c4ai-command-r-plus

    2024

  14. [22]

    The Llama 3 Herd of Models [Internet]

    Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, et al. The Llama 3 Herd of Models [Internet]. arXiv; 2024 [cited 2024 Aug 1]. Available from: http://arxiv.org/abs/2407.21783

    2024 arXiv

  15. [23]

    Mistral 7B [Internet]

    Jiang AQ, Sablayrolles A, Mensch A, Bamford C, Chaplot DS, Casas D de las, et al. Mistral 7B [Internet]. arXiv; 2023 [cited 2024 Jan 3]. Available from: http://arxiv.org/abs/2310.06825

    2023 arXiv

  16. [24]

    BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains [Internet]

    Labrak Y, Bazoge A, Morin E, Gourraud P-A, Rouvier M, Dufour R. BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains [Internet]. arXiv; 2024 [cited 2024 Sep 24]. Available from: http://arxiv.org/abs/2402.10373

    2024 arXiv

  17. [25]

    Model card of LeoLM 7B on HuggingFace [Internet]

    Plüster B, Schuhmann C. Model card of LeoLM 7B on HuggingFace [Internet]. 2023 [cited 2024 Sep 25]. Available from: https://huggingface.co/LeoLM/leo-hessianai-7b

    2023

  18. [26]

    Llama 2: Open Foundation and Fine-Tuned Chat Models [Internet]

    Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models [Internet]. arXiv; 2023 [cited 2024 Sep 26]. Available from: http://arxiv.org/abs/2307.09288

    2023 arXiv

  19. [27]

    VAGOsolutions - organization card on HuggingFace [Internet]

    Golchinfar D, Vaziri D, Hennekeuser D, Fernandes Neto F, Atkins L, Marquardt A. VAGOsolutions - organization card on HuggingFace [Internet]. 2024 [cited 2024 Sep 4]. Available from: https://huggingface.co/VAGOsolutions

    2024

  20. [28]

    Model card of Mistral Nemo on HuggingFace [Internet]

    Jiang A, Sablayrolles A, Tacnet A, Kothari A, Roux A, Mensch A, et al. Model card of Mistral Nemo on HuggingFace [Internet]. 2024 [cited 2024 Sep 25]. Available from: https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407 - 26 - This version of the article has been accept...

    2024 doi

  21. [29]

    Mixtral of Experts [Internet]

    Jiang AQ, Sablayrolles A, Roux A, Mensch A, Savary B, Bamford C, et al. Mixtral of Experts [Internet]. arXiv; 2024 [cited 2024 Jan 9]. Available from: http://arxiv.org/abs/2401.04088

    2024 arXiv

  22. [30]

    EuroLLM: Multilingual Language Models for Europe [Internet]

    Martins PH, Fernandes P, Alves J, Guerreiro NM, Rei R, Alves DM, et al. EuroLLM: Multilingual Language Models for Europe [Internet]. arXiv; 2024 [cited 2024 Sep 26]. Available from: http://arxiv.org/abs/2409.16235

    2024 arXiv

  23. [31]

    Model card of Llama 3.2 1B on Hugging Face [Internet]

    Meta, Inc. Model card of Llama 3.2 1B on Hugging Face [Internet]. 2024 [cited 2024 Nov 21]. Available from: https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct

    2024

  24. [32]

    Model card of Llama 3.2 3B on Hugging Face [Internet]

    Meta, Inc. Model card of Llama 3.2 3B on Hugging Face [Internet]. 2024 [cited 2024 Nov 21]. Available from: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct

    2024

  25. [33]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]

    Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]. arXiv; 2019 [cited 2024 Oct 17]. Available from: http://arxiv.org/abs/1810.04805

    2019 arXiv

  26. [34]

    Multitask Prompted Training Enables Zero-Shot Task Generalization

    Sanh V, Webson A, Raffel C, Bach S, Sutawika L, Alyafeai Z, et al. Multitask Prompted Training Enables Zero-Shot Task Generalization. International Conference on Learning Representations [Internet]. 2022. Available from: https://openreview.net/forum? id=9Vrb9D0WI4

    2022

  27. [35]

    Model card of EuroLLM 1.7B on Hugging Face [Internet]

    Martins PH, Fernandes P, Alves J, Guerreiro NM, Rei R, Alves DM, et al. Model card of EuroLLM 1.7B on Hugging Face [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/utter-project/EuroLLM-1.7B-Instruct

    2024

  28. [36]

    Model card of BioMistral on Hugging Face [Internet]

    Labrak Y, Bazoge A, Morin E, Gourraud P-A, Rouvier M, Dufour R. Model card of BioMistral on Hugging Face [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/BioMistral/BioMistral-7B

    2024

  29. [37]

    Model card of Mistral 7B on Hugging Face [Internet]

    Jiang AQ, Sablayrolles A, Mensch A, Bamford C, Chaplot DS, Casas D de las, et al. Model card of Mistral 7B on Hugging Face [Internet]. 2025 [cited 2025 Jan 15]. Available from: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

    2025

  30. [38]

    Model card of Llama 3.1 8B on Hugging Face [Internet]

    Meta, Inc. Model card of Llama 3.1 8B on Hugging Face [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

    2024

  31. [39]

    Model card of Llama 3.1 8B SauerkrautLM on HuggingFace [Internet]

    Golchinfar D, Vaziri D, Hennekeuser D, Fernandes Neto F, Atkins L, Marquardt A. Model card of Llama 3.1 8B SauerkrautLM on HuggingFace [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct

    2024

  32. [40]

    Model card of Mixtral 8x7B on Hugging Face [Internet]

    Jiang AQ, Sablayrolles A, Mensch A, Bamford C, Chaplot DS, Casas D de las, et al. Model card of Mixtral 8x7B on Hugging Face [Internet]. 2025 [cited 2025 Jan 15]. Available from: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

    2025

  33. [41]

    Model card of Llama 3.1 70B on Hugging Face [Internet]

    Meta, Inc. Model card of Llama 3.1 70B on Hugging Face [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct - 27 - This version of the article has been accepted for publication after peer review but is not the Version o...

    2024 doi

  34. [42]

    Model card of Llama 3 8B SauerkrautLM on HuggingFace [Internet]

    Golchinfar D, Vaziri D, Hennekeuser D, Fernandes Neto F, Atkins L, Marquardt A. Model card of Llama 3 8B SauerkrautLM on HuggingFace [Internet]. 2024 [cited 2025 Jan 15]. Available from: https://huggingface.co/VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct

    2024

  35. [43]

    Calibrate Before Use: Improving Few-shot Performance of Language Models

    Zhao Z, Wallace E, Feng S, Klein D, Singh S. Calibrate Before Use: Improving Few-shot Performance of Language Models. In: Meila M, Zhang T, editors. Proceedings of the 38th International Conference on Machine Learning [Internet]. PMLR; 2021. p. 12697–706. Available from: https...

    2021

  36. [44]

    Mitigating Label Biases for In-context Learning

    Fei Y, Hou Y, Chen Z, Bosselut A. Mitigating Label Biases for In-context Learning. In: Rogers A, Boyd-Graber J, Okazaki N, editors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) [Internet]. Toronto, Canada: Asso...

    2023

  37. [45]

    Transformers: State-of-the- Art Natural Language Processing

    Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: State-of-the- Art Natural Language Processing. In: Liu Q, Schlangen D, editors. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations [Internet]...

    2020

  38. [46]

    Data Structures for Statistical Computing in Python

    McKinney W. Data Structures for Statistical Computing in Python. In: Walt S van der, Millman J, editors. Proceedings of the 9th Python in Science Conference. 2010. p. 56–61

    2010

  39. [47]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale

    Dettmers T, Lewis M, Belkada Y, Zettlemoyer L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Proceedings of the 36th International Conference on Neural Information Processing Systems [Internet]. Red Hook, NY, USA: Curran Associates Inc.; 2024 [cited 2024 S...

    2024

  40. [48]

    LeoLM: Igniting German-Language LLM Research | LAION Blog Post [Internet]

    Plüster B. LeoLM: Igniting German-Language LLM Research | LAION Blog Post [Internet]. 2023 [cited 2024 Sep 26]. Available from: https://laion.ai/blog/leo-lm

    2023

  41. [49]

    Introducing Meta Llama 3: The most capable openly available LLM to date [Internet]

    Meta AI. Introducing Meta Llama 3: The most capable openly available LLM to date [Internet]. [cited 2024 Apr 25]. Available from: https://ai.meta.com/blog/meta-llama-3/

    2024

  42. [50]

    Scaling Laws for Neural Language Models [Internet]

    Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R, et al. Scaling Laws for Neural Language Models [Internet]. arXiv; 2020 [cited 2024 Oct 16]. Available from: http://arxiv.org/abs/2001.08361

    2020 arXiv

  43. [51]

    Can Large Language Models Understand Real-World Complex Instructions? Proceedings of the AAAI Conference on Artificial Intelligence [Internet]

    He Q, Zeng J, Huang W, Chen L, Xiao J, He Q, et al. Can Large Language Models Understand Real-World Complex Instructions? Proceedings of the AAAI Conference on Artificial Intelligence [Internet]. 2024 [cited 2024 Oct 14];38:18188–96. Available from: https://ojs.aaai.org/index....

    2024

  44. [52]

    Parameter-efficient fine-tuning of large-scale pre-trained language models

    Ding N, Qin Y, Yang G, Wei F, Yang Z, Su Y, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nat Mach Intell [Internet]. 2023 [cited 2024 Sep 26];5:220–35. Available from: https://www.nature.com/articles/s42256-023-00626-4 - 28 - This version ...

    2023

  45. [53]

    Gesundheit - gemeinsam

    Melnik S, Brix T, Storck M, Riepenhausen S, Varghese J, Rudack C. Overview of German Clinical Text Corpora for Large Language Models – Scoping Review. GMDS Kooperationstagung “Gesundheit - gemeinsam” 2024 [Internet]. German Medical Science GMS Publishing House

    2024

  46. [54]

    Announcement of the German Medical Text Corpus Project (GeMTeX)

    Meineke F, Modersohn L, Loeffler M, Boeker M. Announcement of the German Medical Text Corpus Project (GeMTeX). Stud Health Technol Inform. 2023;302:835–6

    2023

  47. [55]

    A distributable German clinical corpus containing cardiovascular clinical routine doctor’s letters

    Richter-Pechanski P, Wiesenbach P, Schwab DM, Kiriakou C, He M, Allers MM, et al. A distributable German clinical corpus containing cardiovascular clinical routine doctor’s letters. Sci Data [Internet]. 2023 [cited 2024 Jan 31];10:207. Available from: https://www.nature.com/ar...

    2023

  48. [56]

    Annotation and initial evaluation of a large annotated German oncological corpus

    Kittner M, Lamping M, Rieke DT, Götze J, Bajwa B, Jelas I, et al. Annotation and initial evaluation of a large annotated German oncological corpus. JAMIA Open [Internet]. 2021 [cited 2024 Feb 14];4:ooab025. Available from: https://academic.oup.com/jamiaopen/article/doi/10.1093...

    2021 doi

  49. [57]

    GERNERMED: An open German medical NER model

    Frei J, Kramer F. GERNERMED: An open German medical NER model. Software Impacts [Internet]. 2022 [cited 2024 Oct 16];11:100212. Available from: https://www.sciencedirect.com/science/article/pii/S2665963821000944

    2022

  50. [58]

    GRASCCO - The First Publicly Shareable, Multiply- Alienated German Clinical Text Corpus

    Modersohn L, Schulz S, Lohr C, Hahn U. GRASCCO - The First Publicly Shareable, Multiply- Alienated German Clinical Text Corpus. Stud Health Technol Inform [Internet]. 2022;296:66–72. Available from: https://doi.org/10.3233/SHTI220805

    2022 doi

  51. [59]

    Available from: https://dip.bundestag.de/vorgang/gesetz-zur-zusammenf %C3%BChrung-von-krebsregisterdaten/273932

    Aug 18, 2021. Available from: https://dip.bundestag.de/vorgang/gesetz-zur-zusammenf %C3%BChrung-von-krebsregisterdaten/273932

    2021

  52. [60]

    medBERT.de: A comprehensive German BERT model for the medical domain

    Bressem KK, Papaioannou J-M, Grundmann P, Borchert F, Adams LC, Liu L, et al. medBERT.de: A comprehensive German BERT model for the medical domain. Expert Systems with Applications [Internet]. 2024 [cited 2024 Oct 17];237:121598. Available from: https://www.sciencedirect.com/s...

    2024

  53. [61]

    Critical assessment of transformer-based AI models for German clinical notes

    Lentzen M, Madan S, Lage-Rupprecht V, Kühnel L, Fluck J, Jacobs M, et al. Critical assessment of transformer-based AI models for German clinical notes. JAMIA Open [Internet]. 2022 [cited 2024 Oct 17];5:ooac087. Available from: https://doi.org/10.1093/jamiaopen/ooac087

    2022 doi

  54. [62]

    German’s Next Language Model

    Chan B, Schweter S, Möller T. German’s Next Language Model. In: Scott D, Bel N, Zong C, editors. Proceedings of the 28th International Conference on Computational Linguistics [Internet]. Barcelona, Spain (Online): International Committee on Computational Linguistics; 2020 [cit...

    2020

  55. [63]

    Scaling instruction-finetuned language models

    Chung HW, Hou L, Longpre S, Zoph B, Tay Y, Fedus W, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research [Internet]. 2024 [cited 2024 Sep 25];25:1–53. Available from: https://www.jmlr.org/papers/v25/23-0870.html

    2024

  56. [64]

    Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying

    Soroush A, Glicksberg BS, Zimlichman E, Barash Y, Freeman R, Charney AW, et al. Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. NEJM AI [Internet]. 2024 [cited 2024 Jun 5];1:AIdbp2300040. Available from: https://ai.nejm.org/doi/full/10.10...

    2024 doi

  57. [65]

    [Automatic ICD-10 coding : Natural language processing for German MRI reports]

    Mittermeier A, Aßenmacher M, Schachtner B, Grosu S, Dakovic V, Kandratovich V, et al. [Automatic ICD-10 coding : Natural language processing for German MRI reports]. Radiologie (Heidelb) [Internet]. 2024;64:793–800. Available from: https://doi.org/10.1007/s00117-024- 01349-2

    2024 doi

  58. [66]

    How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment

    Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Medical Education [Internet]...

    2023

  59. [67]

    Modellprojekt Kodierqualität

    Dirschedl P, Reichle M, Röther M. Modellprojekt Kodierqualität. Gesundheitswesen [Internet]. 2003 [cited 2024 Sep 26];65:1–7. Available from: http://www.thieme-connect.de/DOI/DOI?10.1055/s-2003-36914

    2003 doi

  60. [68]

    Retrieval-augmented generation for knowledge-intensive NLP tasks

    Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Proceedings of the 34th International Conference on Neural Information Processing Systems [Internet]. Red Hook, NY, USA: Curran Associates Inc....

    2020

  61. [69]

    Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments

    Brin D, Sorin V, Vaid A, Soroush A, Glicksberg BS, Charney AW, et al. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci Rep [Internet]. 2023 [cited 2024 Oct 18];13:16492. Available from: https://www.nature.com/articles/s41598-023-43436-9

    2023

  62. [70]

    Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening

    Unlu O, Shin J, Mailly CJ, Oates MF, Tucci MR, Varugheese M, et al. Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening. NEJM AI [Internet]. 2024 [cited 2024 Jun 26];0:AIoa2400181. Available from: https://ai.nejm.org/doi/full/10.1056/AIoa2400181

    2024 doi

  63. [71]

    KRAGEN: a knowledge graph-enhanced RAG framework for biomedical problem solving using large language models

    Matsumoto N, Moran J, Choi H, Hernandez ME, Venkatesan M, Wang P, et al. KRAGEN: a knowledge graph-enhanced RAG framework for biomedical problem solving using large language models. Bioinformatics [Internet]. 2024 [cited 2024 Oct 18];40:btae353. Available from: https://doi.org...

    2024 doi

  64. [72]

    A Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare

    Budler LC, Chen H, Chen A, Topaz M, Tam W, Bian J, et al. A Brief Review on Benchmarking for Large Language Models Evaluation in Healthcare. WIREs Data Mining and Knowledge Discovery [Internet]. 2025 [cited 2025 May 5];15:e70010. Available from: https://onlinelibrary.wiley.com...

    2025 doi

  65. [73]

    Assessing Retrieval- Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders [Internet]

    Klang E, Tessler I, Apakama DU, Abbott E, Glicksberg BS, Arnold M, et al. Assessing Retrieval- Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to Human Coders [Internet]. medRxiv; 2024 [cited 2024 Oct 22]. p. 2024.10.15.24315526. Av...

    2024 doi

  66. [74]

    Digitaler FortschrittsHub Gesundheit – DECIDE

    Liu L, Lian L, Hao Y, Pace A, Kim E, Homsi N, et al. Human-level information extraction from clinical reports with fine-tuned language models [Internet]. Health Informatics; 2024 [cited 2025 May 5]. Available from: http://medrxiv.org/lookup/doi/10.1101/2024.11.18.24317466 - 31...

    2024 doi

  67. [75]

    A Survey on Evaluation of Large Language Models

    Chang Y, Wang X, Wang J, Wu Y, Yang L, Zhu K, et al. A Survey on Evaluation of Large Language Models. ACM Trans Intell Syst Technol [Internet]. 2024 [cited 2025 May 5];15:39:1- 39:45. Available from: https://dl.acm.org/doi/10.1145/3641289

    2024 doi

  68. [2024]

    DocAbstr

    p. DocAbstr. 743. Available from: https://doi.org/10.3205/24gmds079

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.