Pith. sign in

REVIEW 2 major objections 1 minor 12 references

sebis at CRF Filling 2026: A Two-Stage Local LLM Pipeline for Medical CRF Filling

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-stage local LLM pipeline extracts structured clinical data from EHR notes at 0.55 macro-F1 without external APIs or fine-tuning.

desk verdict A practical engineering report on a local two-stage LLM pipeline that places second in one medical shared task but applies mostly standard techniques. read the letter →

arxiv 2606.13082 v1 pith:M6T6I7QM submitted 2026-06-11 cs.CL

Katharina Sommer , Tristan Till , Florian Matthes This is my paper · ORCID
classification cs.CL
keywords clinicalNLPCRFfillinglocalLLMtwo-stagepipelinemedicalinformationextractionfew-shotin-contextlearningprivacy-preservingEHRprocessing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper demonstrates a fully local pipeline that turns unstructured electronic health record notes into filled case report forms using the MedGemma-27B model. The method runs entirely on-premise with item-specific few-shot examples to decide first whether a data item is present and then to pull its value only when the text supports it. This design produces fixed outputs for missing, negated, or uncertain cases instead of guessing. A reader would care because clinical data extraction has long required either sending sensitive notes to cloud services or accepting hallucinations that violate source evidence. The reported score places the system second among all locally hosted open-source entries on the English test track.

What carries the argument

The two-stage architecture that first classifies whether an item is present in the note and only then extracts its value when the classification is positive.

What would settle it

Running the pipeline on a set of notes rich in negations and uncertainties and finding any extracted values that do not appear verbatim in the source text would show the adherence claim does not hold.

Watch

Extended reading notes

Core claim

The central claim is that a two-stage architecture separating binary presence classification from subsequent value extraction, executed via item-specific few-shot in-context learning on a local model, enforces strict textual grounding and yields deterministic outputs for negated, uncertain, or absent information while reaching a macro-F1 of 0.55 on the official English test track of the CL4Health 2026 CRF filling task.

Load-bearing premise

The separation into presence classification followed by value extraction will reliably block any output that lacks direct textual support.

Editorial extensions

If this is right

  • Hospitals can keep all clinical note processing inside their own infrastructure while still producing structured forms.
  • Item-specific few-shot prompts allow adaptation to new CRF items without retraining the underlying model.
  • Outputs for uncertain or negated items become fixed rather than variable, reducing downstream review effort.
  • Open-source local models become practical substitutes for proprietary services on privacy-regulated medical tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same staged design could be applied to other languages once comparable local models exist for them.
  • Embedding the pipeline inside existing hospital record systems would let staff review only the cases the model flags as present.
  • Measuring performance separately on notes with high versus low negation density would test whether the two-stage split truly drives the determinism.
  • Replacing the base model with a larger local variant while keeping the same two-stage logic would show how much headroom remains without losing on-premise constraints.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces a two-stage local LLM pipeline based on MedGemma-27B for the CL4Health 2026 CRF filling task. It uses item-specific few-shot in-context learning to separate binary presence classification from value extraction, with the goal of enforcing textual evidence adherence and producing deterministic outputs for negated, uncertain, or unknown states. The system reports a macro-F1 of 0.55 on the English test track and second place among locally-hosted open-source submissions.

Significance. If reproducible, the result demonstrates that fully local, privacy-preserving LLM pipelines without fine-tuning or external APIs can reach competitive performance on clinical information extraction. The explicit supply of prompt templates, output schemas, and per-item handling rules is a clear strength for reproducibility in this domain.

major comments (2)
  1. [Abstract] Abstract: The assertion that the two-stage design 'enforces strict adherence to textual evidence' and 'ensures deterministic outputs' for negated/uncertain states lacks any supporting error analysis, ablation against a single-stage baseline, or breakdown on negated/uncertain cases; the aggregate macro-F1 does not isolate or verify this property.
  2. [Evaluation/Results section] Evaluation/Results section: No baseline comparisons (rule-based, single-stage LLM, or prior shared-task systems) or per-item/per-category metrics are reported, which is load-bearing for claims about the architecture's contribution to the observed ranking and evidence adherence.
minor comments (1)
  1. [Method section] Method section: The output schemas and few-shot example selection rules would benefit from one or two concrete worked examples in the main text rather than only in an appendix.

Simulated Author's Rebuttal

2 responses · 0 unresolved

Thank you for the constructive feedback on our manuscript. We appreciate the emphasis on empirical validation for the claims regarding the two-stage architecture. We address each major comment below and commit to revisions that strengthen the paper without misrepresenting the current results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The assertion that the two-stage design 'enforces strict adherence to textual evidence' and 'ensures deterministic outputs' for negated/uncertain states lacks any supporting error analysis, ablation against a single-stage baseline, or breakdown on negated/uncertain cases; the aggregate macro-F1 does not isolate or verify this property.

    Authors: We agree that the abstract asserts these properties of the two-stage design without dedicated supporting analysis in the manuscript. The architecture incorporates explicit per-item rules for negated, uncertain, and unknown states in the second stage to promote evidence adherence and determinism, but the reported macro-F1 of 0.55 does not isolate this effect. In the revised manuscript, we will add an error analysis section with a breakdown on negated/uncertain cases and an ablation comparing the two-stage pipeline to a single-stage baseline to substantiate these claims. revision: yes

  2. Referee: [Evaluation/Results section] Evaluation/Results section: No baseline comparisons (rule-based, single-stage LLM, or prior shared-task systems) or per-item/per-category metrics are reported, which is load-bearing for claims about the architecture's contribution to the observed ranking and evidence adherence.

    Authors: We acknowledge that the Evaluation/Results section reports only the aggregate macro-F1 without baselines or granular metrics, which limits attribution of the second-place ranking among local submissions to the two-stage design. The revised version will incorporate per-item and per-category F1 scores. We will also add comparisons to a rule-based baseline and a single-stage LLM implementation. For prior shared-task systems, we will expand the related work discussion with available context, though direct comparisons depend on task-specific data availability. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

The manuscript is an empirical system paper describing a two-stage LLM pipeline for CRF filling. It reports a macro-F1 of 0.55 on an official test track using item-specific few-shot ICL on MedGemma-27B. No equations, fitted parameters, or derivations are present; the central result is a direct performance measurement on held-out data. No self-citation chains, ansatzes, or uniqueness claims reduce the result to its inputs by construction. The architecture description (binary classification then value extraction) is operationalized via explicit prompt templates and rules that are independently verifiable against the reported outputs.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

This is an empirical systems paper describing a pipeline for a clinical NLP shared task. No free parameters, axioms, or invented entities are introduced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of sebis at CRF Filling 2026: A Two-Stage Local LLM Pipeline for Medical CRF Filling." pith.science (2026). https://pith.science/paper/M6T6I7QM

@misc{pith2026260613082,
  author       = {Pith},
  title        = {Pith review of: sebis at CRF Filling 2026: A Two-Stage Local LLM Pipeline for Medical CRF Filling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6T6I7QM}},
  note         = {Machine review of arXiv:2606.13082}
}
read the original abstract

The extraction of structured clinical information from unstructured EHR notes is a persistent bottleneck in healthcare informatics. While large language models (LLMs) offer high performance, their deployment in clinical settings is hindered by privacy risks, inference costs, and the tendency to hallucinate beyond textual evidence. We address these challenges for the CL4Health 2026 Case Report Form (CRF) filling task by proposing a fully local, domain-adapted pipeline using the MedGemma-27B model. Our two-stage architecture, which separates binary presence classification from value extraction, enforces strict adherence to textual evidence and ensures deterministic outputs for negated, uncertain, or unknown states. By leveraging item-specific, few-shot in-context learning without external API calls or fine-tuning, our approach achieves a macro-F1 score of 0.55 on the official English test track. This result secures second place among all locally-hosted, open-source submissions. Our work demonstrates that privacy-preserving, on-premise LLM pipelines can achieve near-competitive performance with proprietary frontier models, providing a practical, data-sovereign framework for clinical NLP.

Figures

Figures reproduced from arXiv: 2606.13082 by the authors.

Figure 1
Figure 1. Overview of the two-stage classification-then-extraction pipeline. For each clinical note and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Label distribution of the training set across [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Diagram of the CL4Health 2026 CRF-filling leaderboard, split into detection language and open [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages

  1. [1]

    Catalan Speecon database

    Speecon Consortium. Catalan Speecon database. 2011

    2011

  2. [2]

    The EMILLE/CIIL Corpus

    Anthony McEnery and others. The EMILLE/CIIL Corpus. 2004

    2004

  3. [3]

    The OrienTel Moroccan MCA (Modern Colloquial Arabic) database

    Khalid Choukri and Niklas Paullson. The OrienTel Moroccan MCA (Modern Colloquial Arabic) database. 2004

    2004

  4. [4]

    ItalWordNet v.2

    Roventini, Adriana and Marinelli, Rita and Bertagna, Francesca. ItalWordNet v.2

  5. [5]

    MedGemma Technical Report

    Medgemma technical report , author=. arXiv preprint arXiv:2507.05201 , year=

    work page Pith review arXiv

  6. [6]

    Overview of the CRF 2026 Shared Task on Clinical Case Report Forms filling

    Ferrazzi, Pietro and Ghosh, Soumitra and Lavelli, Alberto and Magnini, Bernardo. Overview of the CRF 2026 Shared Task on Clinical Case Report Forms filling. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health). 2026

    2026

  7. [7]

    2026 , eprint=

    Toward Automatic Filling of Case Report Forms: A Case Study on Data from an Italian Emergency Department , author=. 2026 , eprint=

    2026

  8. [8]

    Converting Annotated Clinical Cases into Structured Case Report Forms

    Ferrazzi, Pietro and Lavelli, Alberto and Magnini, Bernardo. Converting Annotated Clinical Cases into Structured Case Report Forms. Proceedings of the 24th Workshop on Biomedical Language Processing. 2025. doi:10.18653/v1/2025.bionlp-1.26

    work page doi:10.18653/v1/2025.bionlp-1.26 2025

Show all 12 references
  1. [9]

    Proceedings of the 2022 conference on empirical methods in natural language processing , pages=

    Large language models are few-shot clinical information extractors , author=. Proceedings of the 2022 conference on empirical methods in natural language processing , pages=

    2022

  2. [10]

    MedRxiv , year=

    LLM-AIx: An open source pipeline for Information Extraction from unstructured medical text based on privacy preserving Large Language Models , author=. MedRxiv , year=

  3. [11]

    JAMIA open , volume=

    Leveraging open-source large language models for clinical information extraction in resource-constrained settings , author=. JAMIA open , volume=. 2025 , publisher=

    2025

  4. [12]

    Journal of biomedical informatics , pages=

    Medication information extraction using local large language models , author=. Journal of biomedical informatics , pages=. 2025 , publisher=

    2025

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.