Pre-training A Neural Language Model Improves The Sample Efficiency of an Emergency Room Classification Model

Binbin Xu; C\'edric Gil-Jardin\'e; Emmanuel Lagarde; Eric Tellier; Frantz Thiessard; Marta Avalos

REVIEW 4 major objections 5 minor 16 references

Pre-training A Neural Language Model Improves The Sample Efficiency of an Emergency Room Classification Model

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pre-training on unlabeled ER notes cuts labeled data needs tenfold

desk verdict A useful French clinical confirmatory study of pre-training's sample-efficiency gain, but the headline factor-of-ten is not measured cleanly because the test set doubles as the model-selection set. read the letter →

arxiv 1909.01136 v5 pith:CZEWUFWF submitted 2019-08-30 cs.CL cs.AIcs.LG

Binbin Xu , Cédric Gil-Jardiné , Frantz Thiessard , Eric Tellier , Marta Avalos , Emmanuel Lagarde This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords sampleefficiencypre-trainingGPT-2clinicaltextclassificationemergencyroomtraumaself-supervisedlearningFrenchnotes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether a neural language model trained from scratch on unlabeled French emergency-room notes can learn to classify a visit as traumatic or non-traumatic using far fewer expert-labeled examples than a model trained only on labeled data. The authors retrain a GPT-2 model from random weights on 151,930 unlabeled clinical notes, then fine-tune it on small labeled sets, and compare this two-stage approach with training the same model from scratch directly on labeled notes. They report that the pretrained model reaches a discrimination score (AUC) above 0.95 with 600 labeled notes, whereas the fully supervised model needs about 6,000 notes to reach the same level—a tenfold reduction in annotation load. If this holds, it would make large-scale automated injury surveillance practical, since expert annotation is the most expensive step in building such classifiers.

What carries the argument

The load-bearing mechanism is the GPT-2 Transformer language model (117M-parameter version) retrained from a random initialization on 151,930 unlabeled French clinical notes via self-supervised next-token prediction, followed by supervised fine-tuning on labeled notes. Classification is implemented as a question-answering-style next-token prediction: each note is suffixed with the control token "TARPON" and the model learns to output "1" or "0" as the next token. The pre-training phase supplies the model with the vocabulary, abbreviations, and phrasing of emergency-room notes, so the fine-tuning phase only needs to learn the classification decision boundary instead of language understanding from scratch.

What would settle it

Re-run the two scenarios with a separate validation set for early stopping and keep the 10,000-note test set untouched; if the pre-trained model then needs more than 600 labeled notes to reach AUC 0.949, or the fully supervised model needs fewer than 6,000, the factor-of-ten claim fails. Alternatively, report the AUC at a fixed iteration budget or at the validation-set-selected iteration instead of the maximum over iterations.

Watch

Extended reading notes

Core claim

The central claim is that a generative self-supervised pre-training phase on unlabeled clinical notes sharply improves sample efficiency for a downstream classification task. Using a 117-million-parameter GPT-2 retrained from scratch on French emergency-room notes, the authors cast classification as next-token prediction: each note is suffixed with the control keyword "TARPON" and the model is trained to emit the token "1" for trauma or "0" for non-trauma. In Scenario B (pre-training plus fine-tuning), 600 labeled notes yield an AUC of 0.949 and an F1 score of 0.852, matching or exceeding what Scenario A (fully supervised from scratch) achieves with 6,000 labeled notes. With all 10,000 labeled notes, Scenario B reaches an AUC of 0.970, comparable to Scenario A trained on more than 100,000 notes. The authors conclude that the pre-training phase reduces the required number of annotated samples by roughly an order of magnitude.

Load-bearing premise

The central comparison assumes that taking the maximum AUC over training iterations on the fixed 10,000-note test set is a valid way to measure achievable performance; if that test set doubles as the model-selection set, the reported 600-versus-6,000 gap may be inflated.

Editorial extensions

If this is right

A self-supervised pre-training phase on unlabeled clinical notes reduces the number of expert-annotated notes needed for trauma classification by about a factor of 10.
The GPT-2 question-answering setup works for French clinical text without using OpenAI's English pretrained weights, showing the approach transfers to non-English medical domains.
With only 10,000 labeled notes after pre-training, the model reaches performance comparable to a fully supervised model trained on more than 100,000 labeled notes.
Using the larger 345M-parameter GPT-2 did not significantly improve classification, so the smaller model is sufficient for this task.
The same pipeline is intended to be extended to multi-label coding of injury mechanisms, pointing toward a national automated trauma observatory.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported factor-of-ten gain may be optimistic because the paper selects the best iteration using the frozen 10,000-note test set; re-evaluating with a separate validation set for early stopping could shrink or eliminate the gap.
The unusual next-token classification head (rather than a standard softmax layer over the hidden state) may understate what pre-training enables, so a conventional classifier head could show even larger gains.
Since the pre-trained representations are task-agnostic, the same unlabeled corpus should also reduce labeling needs for other ER classification targets such as injury mechanism, intent, or product involved, which the authors plan to test.
The dependence of the gain on the volume of unlabeled notes (151,930 in this study) is untested; a sweep over pre-training corpus sizes would show whether the tenfold gain persists with fewer unlabeled notes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read the GPT-2 ER classification paper. Bottom line: the qualitative claim is almost certainly right, but the quantitative factor-of-ten is not well supported as written.

What's new: they retrain GPT-2 from scratch on French clinical notes, then fine-tune on small labeled sets, and show the pre-trained model reaches similar AUC with about a tenth of the labels. That's a useful domain/language confirmation of what ULMFiT already showed in English. The paper does it carefully in one sense: they froze a 10k test set, used ICD-10-derived labels to avoid noise, and compared 117M vs 345M. They also cite Howard and Ruder and acknowledge their result is in line.

The soft spot is the one the stress-test flags. They take the maximum test AUC over the training trajectory for each labeled-set size, and use that maximum to draw the curves. That means the test set is doing double duty as validation set. For small labeled sets the training curve is noisy, so the max over up to 330k iterations will be biased upward, more so for the 600-note run than the 6,000-note run. The factor-of-ten gap could be inflated. No repeated seeds, no confidence intervals, so we have no sense of variance. Also the abstract says AUC>0.95, but the 600-note result is 0.949, which doesn't cross that threshold. Minor, but sloppy.

There's also the novelty question. The Howard and Ruder result (100 labeled matching 20,000) is a stronger version of the same claim in a general domain. So the paper is confirmatory rather than breaking new ground. That's fine for an applied clinical NLP audience, but it should be framed as such.

Recommendation: I'd send it to a serious referee, but with the expectation that the authors need to rerun with a validation set or at least report results from a fixed early-stopping rule and repeated seeds. The central direction is likely right, and the French clinical setting is underserved. As is, the specific factor-of-ten claim is not solid enough to rely on.

Referee Report

4 major / 5 minor

Summary. This paper assesses whether a self-supervised pre-training phase on unlabeled French emergency-room (ER) notes improves the sample efficiency of a supervised binary classifier that predicts whether an ER visit is traumatic from free-text notes. The authors fine-tune a GPT-2 model retrained from scratch using a question-answering-style output ('TARPON' followed by a 0/1 token) and compare two scenarios: fully supervised training from scratch (Scenario A) versus pre-training on unlabeled notes followed by supervised fine-tuning on a smaller labeled set (Scenario B). Using 171,930 ER notes and a fixed 10,000-note test set, they report that Scenario B reaches AUC 0.949 with 600 labeled notes, while Scenario A requires about 6,000 labeled notes to match, implying a factor-of-10 reduction in annotation load. They also report that the GPT-2 345M model offers no significant gain over the 117M model.

Significance. The question addressed is practically important for injury surveillance and clinical NLP: if pre-training on unlabeled clinical text can reduce expert annotation effort by an order of magnitude, that has direct operational value for the French TARPON project and similar systems. The study has strengths: it is grounded in a real clinical corpus, the authors retrain GPT-2 from scratch rather than relying on English pre-trained weights, they vary the labeled-set size over a wide grid, and they evaluate with AUC and F1 score. The qualitative finding that generative pre-training improves sample efficiency is plausible and consistent with prior work (e.g., ULMFiT). However, the quantitative factor-of-10 claim rests on an evaluation protocol that uses the test set for model selection, as detailed below; the magnitude should therefore be treated as unverified rather than established.

major comments (4)

[Methods – Sampling strategy; Results] The test set is used both for model selection and for evaluation. The Methods state that the 10,000-note test set 'was used to estimate the number of notes needed to achieve maximum prediction performance,' and the Results state that for each training-set size 'the maximum AUC/F1 score value was retained' to build Figures 6 and 9. Taking the maximum over up to 330,000 iterations on a fixed test set makes the test set an early-stopping/model-selection set; the reported point is an order statistic biased upward relative to any fixed stopping rule or validation-selected checkpoint. Because the bias is expected to be larger for small training sets (noisier curves), the 600-note versus 6,000-note comparison that anchors the factor-of-10 claim is not a valid estimate of achievable performance. A validation set (or repeated splits with a principled early-stopping rule) is needed to support the headline claim.
[Results (Figures 4–9)] Each configuration is run once and no confidence intervals or repeated-seed estimates are provided. The learning curves in Figures 4 and 5 are highly non-monotonic; the reported point per training-set size is the maximum over a trajectory of up to 330,000 iterations of a single run. Under run-to-run variance, a maximum-of-one-trajectory estimate is unstable, and the 0.949-versus-~0.95 difference between 600 and 6,000 notes is within plausible noise. Please provide repeated runs (at least for the critical sample sizes) and report means with confidence intervals or an equivalent variance estimate.
[Abstract; Results] The abstract states the target is 'AUC > 0.95' and claims a factor-of-10 reduction in data needed to reach that level, but the result used to support the claim is AUC = 0.949 with 600 notes in Scenario B. Please correct the threshold or the reported value, and specify exactly which target underlies the factor-of-10 statement.
[Adapting the GPT-2 model to our setting] The manuscript does not report the training hyperparameters for the GPT-2 117M model (optimizer, learning rate, gradient clipping, sequence length, vocabulary size, number of pre-training epochs or iterations, and the exact stopping criterion for pre-training in Scenario B). Since the authors retrain from scratch, these choices define the pre-training phase and can affect the measured gain; please provide the full configuration in an appendix or public repository.

minor comments (5)

[Abstract] The word 'ginve' should be 'given'.
[Figure 1 caption; Discussion] 'S CENARIO' in the Figure 1 caption should be 'SCENARIO', and 'Standford Question Answering Dataset' should be 'Stanford Question Answering Dataset'.
[Figures 4–10] The axis labels contain rendering artifacts such as '0/period.math40'; please regenerate the figures so the iteration and metric values are readable.
[Methods – Sampling strategy] Please state explicitly whether the smaller labeled samples are nested subsets of the larger ones, and whether the same sampled notes are used in both scenarios; this affects the interpretation of the learning curves.
[Results] The sentence 'At the end of Scenario B, with a training of all 10,000 notes, AUC and F1 score are respectively 0.970 and 0.889, corresponding the cases of more than 100,000 notes in Scenario A' is grammatically unclear; please specify the exact comparison being made.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the sample-efficiency comparison is an empirical holdout measurement, not a fit or self-citation chain.

full rationale

The paper's central claim is an empirical comparison, not a derivation. In Scenario A the GPT-2 model is trained from random weights directly on labeled notes, while in Scenario B it is first pre-trained on unlabeled notes and then fine-tuned on labeled notes; both are evaluated on the same frozen 10,000-note test set with genuine holdout labels. The reported AUC and F1 values are measured predictions, not quantities defined in terms of the target result. No parameter is fitted to the test labels, and no result is imported from prior work by the same authors. The only external citations (Radford et al., Devlin et al., Howard and Ruder) are used for motivation and comparison, not as load-bearing proof of this paper's effect. A methodological caveat exists: the Methods state that the test set 'was used to estimate the number of notes needed to achieve maximum prediction performance,' and the Results state that 'the maximum AUC/F1 score value was retained to build Figure 6.' This means the test set doubles as a checkpoint-selection set, which can bias the reported maximum upward, especially for small training sets. That is a validity/selection-bias concern about the evaluation protocol, not circularity: the pre-training effect is not true by construction, and the reported numbers are still genuine holdout evaluations. Per the hard rules, such protocol concerns belong to correctness risk rather than to a circularity score; there is no chain of self-citation or definitional equivalence that forces the conclusion. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

This is a purely empirical paper with no theoretical derivation. The central claim rests on experimental-design assumptions: test-set-based selection of iterations, ICD-10-derived labels, a single split, and distributional alignment between pretraining and fine-tuning corpora. No new theoretical entities are introduced.

free parameters (2)

Training configuration of GPT-2 117M (unreported hyperparameters) = not reported
The central comparison is an empirical claim that depends on the training setup; without learning rate, optimizer, sequence length, and vocabulary details, the result cannot be reproduced or assessed for optimality.
Selection of best iteration by test AUC = per-configuration maximum
The reported performance values are maxima over training iterations on the test set; this hand-chosen selection rule directly affects the claimed factor of ten.

assumptions (4)

ad hoc to paper The maximum AUC/F1 over training iterations on the fixed test set is a valid measure of achievable model performance for each training-set size.
Used to build Figures 6 and 9 and to derive the factor-10 comparison; no validation set or early-stopping rule is used.
domain assumption ICD-10 code ranges (S, T1-T35, V vs A, C, D, E, G, H, I, J, L, N) provide accurate binary ground truth for traumatic vs non-traumatic ER visits.
Labels are derived automatically from ICD-10 codes assigned by clinicians; excluded codes are considered semantically uncertain, so the selected subset may not represent the full clinical population.
domain assumption A single random split into 10,000 test and 161,930 training notes is representative enough to support the quantitative comparison.
No cross-validation or repeated splits are performed, so the reported gap could be affected by the particular split.
domain assumption The unlabeled pre-training corpus and the labeled fine-tuning/test sets come from the same distribution, making self-supervised pretraining transferable to the classification task.
Both are drawn from the same hospital ER records, 2011-2018, which is plausible but not statistically verified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pre-training A Neural Language Model Improves The Sample Efficiency of an Emergency Room Classification Model." pith.science (2026). https://pith.science/paper/CZEWUFWF

@misc{pith2026190901136,
  author       = {Pith},
  title        = {Pith review of: Pre-training A Neural Language Model Improves The Sample Efficiency of an Emergency Room Classification Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZEWUFWF}},
  note         = {Machine review of arXiv:1909.01136}
}

read the original abstract

To build a French national electronic injury surveillance system based on emergency room visits, we aim to develop a coding system to classify their causes from clinical notes in free-text. Supervised learning techniques have shown good results in this area but require a large amount of expert annotated dataset which is time consuming and costly to obtain. We hypothesize that the Natural Language Processing Transformer model incorporating a generative self-supervised pre-training step can significantly reduce the required number of annotated samples for supervised fine-tuning. In this preliminary study, we test our hypothesis in the simplified problem of predicting whether a visit is the consequence of a traumatic event or not from free-text clinical notes. Using fully re-trained GPT-2 models (without OpenAI pre-trained weights), we assess the gain of applying a self-supervised pre-training phase with unlabeled notes prior to the supervised learning task. Results show that the number of data required to achieve a ginve level of performance (AUC>0.95) was reduced by a factor of 10 when applying pre-training. Namely, for 16 times more data, the fully-supervised model achieved an improvement <1% in AUC. To conclude, it is possible to adapt a multi-purpose neural language model such as the GPT-2 to create a powerful tool for classification of free-text notes with only a small number of labeled samples.

Figures

Figures reproduced from arXiv: 1909.01136 by the authors.

**Figure 2.** SCENARIO A: supervised training … Pre-training on 151,930 unlabeled clinical notes e model is able to generate artificial text Prediction TARPON 1 TARPON 0 TARPON 1 … Training on labeled clinical notes (from a dataset of 10,000 notes) Building prompts ended with TARPON key-word for prediction TARPON TARPON 0 Prediction [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** SCENARIO B: self-supervised training + supervised training For both scenarios (Figures 2 and 3), the test phase consists in feeding the models with prompts by adding the task identifier at the end of each test clinical note and ask the model to predict the next token right after the task identifier. Ideally, this newly generated token should be one of the classification codes (tokens). On the first iterations, du… view at source ↗

**Figure 5.** Scenario B: AUC by number of iterations. 19 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

Figures from the paper (4 more)

**Figure 6.** Figure 6: Comparison of AUC by cases (number of notes) [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 7.** Figure 7: Scenario A: F1 score by number of iterations. 26 [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]

**Figure 8.** Figure 8: Scenario B: F1 score by number of iterations. 19 [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]

**Figure 10.** Figure 10: Comparison of GPT-2 117M models and 345M [PITH_FULL_IMAGE:figures/full_fig_p005_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

Cho, K.; van Merrienboer, B.; G \" u l c ehre, C .; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078

arXiv 2014
[3]

Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805

arXiv 2018
[4]

Erhan, D.; Bengio, Y.; Courville, A.; Manzagol, P.-A.; Vincent, P.; and Bengio, S. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11(Feb):625--660

work page 2010
[5]

Fouillet, A.; Bousquet, V.; Pontais, I.; Gallay, A.; and Schonemann. 2015. The french emergency department oscour network: Evaluation after a 10-year existence. Online Journal of Public Health Informatics 7(1):e74

work page 2015
[6]

Howard, J., and Ruder, S. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 328--339. Melbourne, Australia: Association for Computational Linguistics

work page 2018
[7]

Huang, J.; Osorio, C.; and Sy, L. W. 2019. An empirical evaluation of deep learning for icd-9 code assignment using mimic-iii clinical notes. Computer Methods and Programs in Biomedicine 177:141 -- 153

work page 2019
[8]

Li , M.; Fei , Z.; Zeng , M.; Wu , F.; Li , Y.; Pan , Y.; and Wang , J. 2019. Automated icd-9 coding via a deep learning approach. IEEE/ACM Transactions on Computational Biology and Bioinformatics 16(4):1193--1202

work page 2019

Show all 16 references

[9]

Organization, W. H. 2015. International statistical classification of diseases and related health problems : 10th revision (ICD-10), Fifth edition, 2016 . World Health Organization

2015
[10]

E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L

Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized word representations. CoRR abs/1802.05365

2018 arXiv
[11]

Powers, D. M. 2011. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation

2011
[12]

Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training

2018
[13]

Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1(8)

2019
[14]

Rothe, S.; Narayan, S.; and Severyn, A. 2019. Leveraging pre-trained checkpoints for sequence generation tasks. CoRR abs/1907.12461

2019 arXiv
[15]

N.; Kaiser, L

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; and Polosukhin, I. 2017. Attention is all you need. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Infor...

2017
[16]

G.; Salakhutdinov, R.; and Le, Q

Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J. G.; Salakhutdinov, R.; and Le, Q. V. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR abs/1906.08237

2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

Cho, K.; van Merrienboer, B.; G \" u l c ehre, C .; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078

arXiv 2014

[3] [3]

Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805

arXiv 2018

[4] [4]

Erhan, D.; Bengio, Y.; Courville, A.; Manzagol, P.-A.; Vincent, P.; and Bengio, S. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11(Feb):625--660

work page 2010

[5] [5]

Fouillet, A.; Bousquet, V.; Pontais, I.; Gallay, A.; and Schonemann. 2015. The french emergency department oscour network: Evaluation after a 10-year existence. Online Journal of Public Health Informatics 7(1):e74

work page 2015

[6] [6]

Howard, J., and Ruder, S. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 328--339. Melbourne, Australia: Association for Computational Linguistics

work page 2018

[7] [7]

Huang, J.; Osorio, C.; and Sy, L. W. 2019. An empirical evaluation of deep learning for icd-9 code assignment using mimic-iii clinical notes. Computer Methods and Programs in Biomedicine 177:141 -- 153

work page 2019

[8] [8]

Li , M.; Fei , Z.; Zeng , M.; Wu , F.; Li , Y.; Pan , Y.; and Wang , J. 2019. Automated icd-9 coding via a deep learning approach. IEEE/ACM Transactions on Computational Biology and Bioinformatics 16(4):1193--1202

work page 2019