{"id":"7ca7fbe8-509d-4698-a7c8-7462827ea6ad","arxiv_id":"2501.12106","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Open source LLMs with 7 to 12 billion parameters can identify tumor diagnoses, assign ICD-10 codes, and find first diagnosis dates in German urology notes with roughly 90% accuracy.","lead":"This study tested eleven open source large language models on three basic tasks from German tumor documentation: detecting tumor diagnoses, assigning ICD-10 codes, and extracting the date of first diagnosis. Models with 7 to 12 billion parameters performed best, suggesting that locally run open source models could support cancer registry data entry while keeping patient data on hospital hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt variants are selected on the same test set, so the reported ~90% accuracies likely overstate real-world performance; a held-out evaluation is needed before the 7–12B recommendation is accepted.","rationale":"Reading the paper in good faith, it is a careful empirical study that releases a useful German urology benchmark and documents inter-rater agreement. The central claim that 7–12B open LLMs are promising for tumor documentation is plausible and consistent with the observed strong performance of Llama 3.1 8B, Mistral 7B and Mistral NeMo 12B. However, the quantitative support is weaker than the headline suggests because the prompt variants are selected on the same test data. The paper's own tables show large prompt-to-prompt variation (e.g., Llama 3.1 8B Step 1 ranges 0.74–0.98; Mistral NeMo Step 3 ranges 0.60–0.92). Reporting the maximum over these variants for each model converts prompt exploration into an optimistic bias. This is not an internal inconsistency, but it is a correctness risk for the 'around 90%' and 'optimal balance' claims. The quantization of larger models is a further confound for the size comparison, but it is secondary: prompt selection touches every reported number. A patient-level cross-validation would settle whether the selected prompts generalize. Given the small sample, confidence intervals are also essential. The released code and dataset are real strengths, and the reader's CONDITIONAL verdict is appropriate; the condition should explicitly include held-out prompt selection.","tokens_in":39873,"tokens_out":4163,"duration_ms":42310,"concrete_test":"Split the 149 text snippets by patient (25 patients) into 5 folds. For each model, select the best prompt variant for each of the three steps using only the training folds (e.g., by accuracy on a validation fold), then evaluate the selected variant on the held-out test fold. Compare the cross-validated accuracy to the reported best-variant accuracy (e.g., Llama 3.1 8B Step 1 0.98, Mistral NeMo 12B Step 3 0.92). If the held-out accuracy drops by more than 5 percentage points, the headline numbers are overfit and the central claim should be weakened. Also report 95% Wilson confidence intervals for the key accuracies to assess whether model differences exceed sampling noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is prompt selection on the evaluation set. The study evaluates 7–14 prompt variants per step (Appendix B) and reports, for each model, the accuracy of the best variant on the same 149 snippets used for the final results (Tables A-1–A-3, Figures 2–5). The Methods also state that prompts were manually adjusted after 'preliminary results.' With 11 models and multiple steps, taking the maximum accuracy over variants on the test set inflates the reported numbers through multiple comparisons and overfitting. For example, Llama 3.1 8B Step 1 accuracy goes from 0.74 (zero-shot) to 0.98 (best few-shot); Mistral 7B from 0.76 to 0.93. The central claim that 7–12B models reach ~90% accuracy and that larger models show no gains rests on these selected maxima. The dataset is small (149 snippets, 25 patients), so the differences between models are also within sampling noise. The paper itself notes that 'the absolute numbers ... might not reflect the exact overall performance,' but it still headlines the best-variant accuracies. A held-out validation or cross-validated prompt selection is needed before the quantitative claims can be taken at face value.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates eleven open-source large language models (1B–70B parameters) on three tasks relevant to German tumor documentation: detecting tumor diagnoses in urological doctors' notes, assigning ICD-10 codes, and extracting the first diagnosis date. Using a newly released dataset of 149 annotated German text snippets from 25 patients, the authors compare multiple prompt variants (zero-shot and few-shot, with urology-specific or gynecological examples). They report that Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B achieve around 90% accuracy for diagnosis detection and ICD-10 coding and up to 92% for first-diagnosis-date extraction, and that larger models do not show clear gains. The paper concludes that 7–12B open-source models could offer an optimal balance of performance and resource efficiency for local deployment, and it releases code and data.","tokens_in":40109,"tokens_out":5664,"duration_ms":53767,"significance":"If the quantitative findings were reliable, this study would provide a useful practical benchmark for German-language clinical NLP and guidance for local LLM deployment in cancer registries. The paper's strengths include a clearly documented annotation process with high inter-rater agreement (kappa about 0.90 for diagnosis presence, Krippendorff's alpha about 0.85 for ICD-10 codes), release of the dataset and evaluation code, and a careful description of prompt variants. However, the central performance claims rest on the selection of the best prompt variant per model on the same test set used for reporting accuracy, and on a very small, domain-restricted sample; these issues limit the strength of the conclusions as they stand.","major_comments":[{"comment":"The reported accuracy for each model is the maximum over the 7–14 prompt variants evaluated on the same 149-snippet test set, and the methods state that prompts were manually adjusted after 'preliminary results.' Selecting the best variant on the test set and then reporting that value as the model's performance inflates the results through multiple comparisons; for example, Llama 3.1 8B Step 1 accuracy ranges from 0.74 (zero-shot) to 0.98 (best few-shot) and Mistral 7B from 0.76 to 0.93 in Table A-1. The paper's central claims about '~90% accuracy' and the absence of gains from larger models are therefore based on selected maxima rather than a pre-specified evaluation protocol. The authors should report the full distribution of results across prompt variants, use a held-out validation set or nested cross-validation for prompt selection, or otherwise demonstrate that the reported values are not inflated by selection.","section":"Methods (Prompt design), Figures 2–5, Tables A-1–A-3"},{"comment":"The evaluation set contains only 149 snippets from 25 patients, with substantial clustering (multiple snippets per patient) and class imbalance (75% of snippets contain a tumor diagnosis; 69% are prostate cancer). The differences between the best-performing 7–12B models and the larger models are within binomial sampling error at this sample size; for instance, in Table A-2 the 'Snippet correct' values for Llama 3.1 8B and Mixtral 8x7B differ by only 0.01–0.03, which is not significant without confidence intervals. The manuscript should report confidence intervals or statistical comparisons that account for snippet clustering and multiple testing before claiming that 7–12B models exhibit an optimal performance/resource trade-off.","section":"Results, Table 1, Table A-2"},{"comment":"The largest models (Mistral NeMo 12B, Mixtral 8x7B, Llama 3.1 70B) were run with 8-bit quantization, while the smaller models were run unquantized. Because quantization can degrade model quality, the finding that larger models 'did not display performance gains' is confounded: it may reflect the effects of quantization rather than a true property of model scale. The authors should either run the larger models without quantization, add a control experiment measuring the effect of 8-bit quantization on a subset of the small models, or substantially soften the size-related conclusions. This concern also affects the recommendation of the 7–12B sweet spot, since one of the recommended models (Mistral NeMo 12B) was quantized.","section":"Software implementation, Discussion"}],"minor_comments":[{"comment":"The sentence 'LLMs shown impressive results' should read 'LLMs have shown impressive results'.","section":"Background, first paragraph"},{"comment":"The text states that the initial data basis comprises 153 doctors' notes and that the dataset contains 149 annotated text snippets; in the Discussion the authors write 'The number of letters in our data set is very similar (153)'. Please clarify whether 'letters' refers to the original PDFs or to the snippet set, as the two numbers appear conflated.","section":"Methods, Data set preparation; Discussion, Contribution to openly available German clinical data"},{"comment":"The bars indicate the best value with respect to accuracy, but the figure legends do not list all the colored prompt-type markers; the reader must consult the interactive supplement. Consider adding a complete legend or a separate table mapping marker numbers to prompt types.","section":"Figures 2–5"},{"comment":"The choice of temperature 0 is reported, but the sampling parameters (e.g., top-p, max tokens) are not specified; since reproducibility is claimed, these should be stated.","section":"Methods, Software implementation"},{"comment":"The text reports that 'the best results were achieved by Mistral NeMo 12B, which identified the first diagnosis date with an accuracy of 92% in the zero-shot approach,' but Table A-3 shows that this 0.92 value is obtained when using the 'results from best models in previous steps'; the sentence should specify the pipeline configuration to avoid ambiguity.","section":"Results, Step 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's own limitation statement in the Discussion acknowledges that the absolute numbers might not reflect exact overall performance, but the abstract and conclusions headline the best-variant accuracies. In my view, the prompt-selection issue is the key barrier; the dataset/code release is a positive contribution. The manuscript can be made acceptable with a revised analysis based on a held-out protocol or cross-validation, or with substantially softened claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know here is that the dataset is genuinely useful, and the evaluation is carefully done, but the headline accuracies come from selecting the best prompt per model on the same test set, so treat them as upper bounds rather than expected performance.\n\nWhat's actually new: 149 annotated snippets from real German urology notes, with four annotators and agreement in the 0.84–0.90 range, released on HuggingFace. That addresses a real shortage of German clinical NLP benchmarks. They also ship code, use temperature zero, and compare 11 models across three tasks with sensible metrics. The cross-domain few-shot experiment (urology vs. gynecology examples) is a nice touch and the finding that examples from a different specialty can help is worth reporting.\n\nThe soft spots are real but not disqualifying. The main one: they tried multiple prompt variants per step and report, for each model, the best variant evaluated on the same 149 snippets used for the final numbers. They also say prompts were manually adjusted after seeing preliminary results. With 11 models and multiple steps, that is multiple comparisons on the test set. The gap between zero-shot and best few-shot for Llama 3.1 8B (0.74 to 0.98) shows how much the prompt can move the number. Second, the three largest models were 8-bit quantized while the 7–8B models were not, so the \"larger models show no gains\" conclusion is partly a quantization comparison. Third, 149 snippets from 25 patients is small; differences between close models are within sampling noise. The paper acknowledges the absolute numbers are not exact, but the abstract still headlines the selected maxima.\n\nNone of this sinks the paper. The conclusion that 7–12B models are promising candidates for local deployment is plausible and appropriately hedged in the discussion. The dataset alone justifies the work.\n\nWho this is for: anyone building German medical NLP benchmarks, cancer registry people thinking about local LLM assistance, and groups comparing open source models for clinical extraction.\n\nMy call: it deserves serious review. The dataset and reproducible code are enough. I would ask for a held-out validation set (or cross-validated prompt selection) and a quantization ablation, but the paper is a solid incremental contribution with an honest limitations section.","headline":"The released German urology dataset is the real contribution; the headline accuracies are optimistic upper bounds because best prompt variants were selected on the same test set, but the paper is a solid, honest incremental evaluation.","tokens_in":40639,"tokens_out":2201,"would_cite":true,"duration_ms":25115,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source language models with 7 to 12 billion parameters can carry out the core German tumor documentation tasks at around 90 percent accuracy with prompt design alone.","keywords":["large language models","tumor documentation","German medical NLP","ICD-10 coding","information extraction","few-shot prompting","urology doctors' notes","clinical text mining"],"falsifier":"Run the same three tasks on the released dataset with Llama 3.1 70B and Mixtral 8x7B at full precision instead of 8-bit; if they then clearly outscore Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B, the claim that mid-sized models are the optimal resource-performance balance would lose its main evidence.","tokens_in":39668,"feed_emoji":"🩺","tokens_out":8872,"duration_ms":88091,"temperature":0.7,"pith_summary":"The paper tests whether open-source large language models can take over parts of the German tumor documentation process, which is now done by staff who read doctors' letters and enter structured data into cancer registries. On an annotated set of 149 real urology text snippets, it runs eleven models from 1 to 70 billion parameters through three tasks: detecting tumor diagnoses, assigning the corresponding three-character ICD-10 codes, and extracting the date of first diagnosis. The central finding is that mid-sized models—Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B—reach roughly 90 percent accuracy on diagnosis detection and coding and up to 92 percent on the date task, while models with fewer than 7 billion parameters or much less training data lag behind and models up to 70 billion parameters show no clear gain. Because German data-protection law makes cloud LLMs impractical for patient data, the paper argues that locally deployable mid-sized models offer the best balance between performance and resource use, and it releases the annotated dataset as a benchmark for German medical NLP.","feed_headline":"Mid-size open-source LLMs hit ~90% on tumor documentation tasks","feed_subtitle":"On real German urology notes, 7-12B models matched or beat much larger ones, suggesting local, privacy-safe automation.","key_machinery":"The argument rests on a three-step prompting pipeline plus a public evaluation dataset. Step 1 instructs the model to return tumor diagnoses from a text snippet as a JSON array; Step 2 asks for the three-character ICD-10 code, the standard international disease classification, for each extracted label; Step 3 extracts the first diagnosis date either directly or by asking the model to verify dates found by a regular expression. Ground truth is a set of 149 snippets from anonymized urological doctors' notes, annotated by four independent raters with agreement around 0.84–0.90 on the key variables. The prompts vary the number and domain of few-shot examples, contrasting urology with gynecology, which lets the paper separate model knowledge from prompt guidance.","core_discovery":"The paper claims that the three tasks at the heart of German tumor documentation—extracting tumor diagnoses from text, coding them with ICD-10 codes, and finding the date of first diagnosis—can be handled by open-source LLMs in the 7–12 billion parameter range without any fine-tuning, using carefully designed prompts alone. On its 149-snippet urology dataset, Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B all perform comparably, with diagnosis detection and ICD-10 coding around 90 percent accuracy and first-diagnosis-date extraction reaching 92 percent for Mistral NeMo 12B in the best configuration. The authors further claim that larger models such as Mixtral 8x7B and Llama 3.1 70B do not consistently outperform these mid-sized models, that smaller models or models trained on less data clearly fall behind, and that additional German or medical training (as in BioMistral and SauerkrautLM) does not automatically beat the base models. They present this as evidence that mid-sized open-source models are the practical starting point for automating tumor documentation under German data protection constraints, and they release the annotated dataset so other groups can build on the comparison.","pith_inferences":["A direct extension the paper does not run is fine-tuning the best 7-12B models on its released dataset; if fine-tuning closes the remaining error gap, the same pipeline could plausibly reach clinical usability.","The size comparison is only as clean as the quantization step; re-running Llama 3.1 70B and Mixtral 8x7B at full precision on this dataset would show whether the 'no gain from larger models' result is robust.","The dataset's small size and urology-only origin mean the absolute accuracy numbers are anchors rather than population estimates; replicating the three-step pipeline on other tumor entities and note types is the natural next check of generality.","A retrieval-augmented variant that feeds the model relevant ICD-10 catalogue entries before coding could address the remaining coding errors without moving to larger models."],"forward_implications":["On these three documentation subtasks, a hospital could run the whole pipeline locally on a single modern GPU, avoiding the legal problem of sending patient text to a cloud LLM.","The absence of a consistent accuracy gain from 46B- and 70B-parameter models means that, for tumor documentation, spending additional compute on larger models is not justified by accuracy alone.","Since the fine-tuned medical or German models did not beat the corresponding base models, choosing a strong general-purpose base model and then designing or tuning the prompt may matter more than buying a domain-specific model.","Cross-domain few-shot examples from gynecology worked as well as domain-matched examples for urology notes, implying that the model learns the abstract definition of a tumor diagnosis rather than memorizing vocabulary.","The released dataset gives German clinical NLP a small but authentic public benchmark, so later work can compare new models or prompting strategies on the same real-world text."],"supporting_citations":[{"why":"Supplies the earlier annotation and XML evaluation basis from which this dataset and few-shot prompt design grew.","marker":"[9]"},{"why":"Supplies the anonymized urological doctor letters and the prior GPT-4 documentation evaluation context.","marker":"[15]"},{"why":"Provides the Llama 3.1 family used in the evaluation and the training-data and German-language statistics used to interpret model-size effects.","marker":"[22]"},{"why":"Defines Mistral 7B, one of the three best-performing models.","marker":"[23]"},{"why":"Defines BioMistral, the medical fine-tune whose lack of improvement over Mistral 7B supports the claim about domain training.","marker":"[24]"},{"why":"Defines LeoLM, the German-trained model whose lower performance supports the training-data-size interpretation.","marker":"[25]"},{"why":"Defines Mistral NeMo 12B, the model with the best first-diagnosis-date extraction.","marker":"[28]"},{"why":"Supplies the 8-bit quantization method used on the three largest models, the key setup choice for the size comparison.","marker":"[47]"},{"why":"Provides the earlier 72.2 percent maximum accuracy for automatic ICD-10 coding on German radiology reports, the direct comparison for the coding accuracy result.","marker":"[63]"}],"fun_headline_variants":["Mid-size open-source LLMs match giants on tumor docs","7-12B LLMs hit ~90% on German tumor documentation","Open-source LLMs: 7-12B sweet spot for tumor coding","Smaller LLMs rival larger ones on tumor documentation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that compressing the three largest models to use less memory (8-bit quantization) does not noticeably hurt their scores, so the conclusion that bigger models are no better is not an artifact of the compression.","fun_headline_variants_meta":{"raw":{"variants":["Mid-size open-source LLMs match giants on tumor docs","7-12B LLMs hit ~90% on German tumor documentation","Open-source LLMs: 7-12B sweet spot for tumor coding","Smaller LLMs rival larger ones on tumor documentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1384,"prompt_tokens":1134,"completion_tokens":250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":177}},"tokens_in":750,"tokens_out":250,"duration_ms":3052,"temperature":1.0,"reasoning_tokens":177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:29:33.393763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three tasks on the released dataset with Llama 3.1 70B and Mixtral 8x7B at full precision instead of 8-bit; if they then clearly outscore Llama 3.1 8B, Mistral 7B, and Mistral NeMo 12B, the claim that mid-sized models are the optimal resource-performance balance would lose its main evidence.","supporting_citations":[{"cited_title":"Gesundheit - gemeinsam","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier annotation and XML evaluation basis from which this dataset and few-shot prompt design grew."},{"cited_title":"[Large Language Models for Rapid Simplification of Quality Assurance Data Input: Field Trial with Real Data in the Context of Tumour Documentation in Urology]","cited_arxiv_id":null,"evidence_quote":"Supplies the anonymized urological doctor letters and the prior GPT-4 documentation evaluation context."},{"cited_title":"Model card of LeoLM 7B on HuggingFace [Internet]","cited_arxiv_id":null,"evidence_quote":"Defines LeoLM, the German-trained model whose lower performance supports the training-data-size interpretation."},{"cited_title":"Model card of Mistral Nemo on HuggingFace [Internet]","cited_arxiv_id":null,"evidence_quote":"Defines Mistral NeMo 12B, the model with the best first-diagnosis-date extraction."},{"cited_title":"Scaling instruction-finetuned language models","cited_arxiv_id":null,"evidence_quote":"Provides the earlier 72.2 percent maximum accuracy for automatic ICD-10 coding on German radiology reports, the direct comparison for the coding accuracy result."}],"review_version":1}