{"id":"17e3e2ff-5a3e-419e-b8ad-cdfe31dee0cb","arxiv_id":"1909.01136","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Self-supervised pre-training on unlabeled clinical notes reduced the labeled data needed for ER trauma classification by roughly 10x, but the estimate is weakened by test-set-based model selection and missing variance.","lead":"This paper tests whether pre-training a GPT-2 language model on unlabeled French emergency room notes reduces how many labeled notes are needed to classify visits as traumatic or not. It reports a 10x reduction in labeled data needed to reach about 0.95 AUC, but the comparison uses the test set to pick the best training iteration and has no error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set is used to select the best iteration, so the reported 600-vs-6,000 labeled-note gap may be inflated by selection bias; a validation-based rerun is needed.","rationale":"The reader's weakest assumption directly identifies the most load-bearing issue: the 10,000-note test set is used both to select the best training iteration and to report final performance. Because the paper explicitly says the test set was used to estimate the number of notes needed and that the maximum AUC/F1 over iterations was retained, the reported crossover between Scenario B at 600 notes and Scenario A at 6,000 notes is an upper envelope of test-set-dependent maxima, not an unbiased estimate of achievable performance under a realistic early-stopping rule. This is especially problematic for the small-data regime, where test AUC curves are volatile and the maximum over many iterations can overfit the test set. The absence of repeated seeds or confidence intervals leaves no way to quantify the selection bias. The qualitative direction is consistent with prior work such as ULMFiT, and the fully retrained GPT-2 setup is a useful contribution, but the factor-of-10 claim cannot be accepted as quantitatively supported without a validation-set-based rerun. The reader's CONDITIONAL verdict is appropriate; I would not move it to ACCEPT or REJECT on the current evidence. A secondary wording issue, that the abstract claims AUC>0.95 while the cited result is 0.949, reinforces the need for careful recalculation but is not the central concern.","tokens_in":9047,"tokens_out":4621,"duration_ms":44384,"concrete_test":"Hold out a validation set (e.g., 2,000 notes) disjoint from the 10,000-note test set. Re-run Scenario A with 6,000 labeled notes and Scenario B with 600 labeled notes, using at least 5 random seeds per condition. For each run, select the early-stopping iteration by maximum validation AUC (or by a fixed iteration budget), then evaluate that checkpoint on the original test set. Report mean and 95% CI of test AUC for both conditions. If the Scenario B 600-note mean is not within about 0.01 AUC of the Scenario A 6,000-note mean, the order-of-magnitude claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Methods section 'Sampling strategy' states that the 10,000-note test set 'was used to estimate the number of notes needed to achieve maximum prediction performance,' and the Results state that for each training-set size 'the maximum AUC/F1 score value was retained' to build Figures 6 and 9. This means the test set is doing double duty: it is the evaluation set and the early-stopping/model-selection set. For each labeled-set size, the reported point is the maximum over a long training trajectory (up to 330,000 iterations) of a metric evaluated on the same fixed test set. That maximum is an order statistic and is biased upward relative to the performance of any fixed stopping rule or of a validation-selected checkpoint. The bias is likely larger for the 600-note pre-trained model than for the 6,000-note supervised model, because with fewer training examples the test-AUC curve is noisier and the max over many iterations can select a checkpoint that partly fits the test set. Since the headline factor-of-10 rests on comparing these two maxima, the gap may be substantially inflated. No repeated-seed or confidence-interval information is provided, so the actual variance of the 600-note estimate is unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper assesses whether a self-supervised pre-training phase on unlabeled French emergency-room (ER) notes improves the sample efficiency of a supervised binary classifier that predicts whether an ER visit is traumatic from free-text notes. The authors fine-tune a GPT-2 model retrained from scratch using a question-answering-style output ('TARPON' followed by a 0/1 token) and compare two scenarios: fully supervised training from scratch (Scenario A) versus pre-training on unlabeled notes followed by supervised fine-tuning on a smaller labeled set (Scenario B). Using 171,930 ER notes and a fixed 10,000-note test set, they report that Scenario B reaches AUC 0.949 with 600 labeled notes, while Scenario A requires about 6,000 labeled notes to match, implying a factor-of-10 reduction in annotation load. They also report that the GPT-2 345M model offers no significant gain over the 117M model.","tokens_in":9309,"tokens_out":6627,"duration_ms":55998,"significance":"The question addressed is practically important for injury surveillance and clinical NLP: if pre-training on unlabeled clinical text can reduce expert annotation effort by an order of magnitude, that has direct operational value for the French TARPON project and similar systems. The study has strengths: it is grounded in a real clinical corpus, the authors retrain GPT-2 from scratch rather than relying on English pre-trained weights, they vary the labeled-set size over a wide grid, and they evaluate with AUC and F1 score. The qualitative finding that generative pre-training improves sample efficiency is plausible and consistent with prior work (e.g., ULMFiT). However, the quantitative factor-of-10 claim rests on an evaluation protocol that uses the test set for model selection, as detailed below; the magnitude should therefore be treated as unverified rather than established.","major_comments":[{"comment":"The test set is used both for model selection and for evaluation. The Methods state that the 10,000-note test set 'was used to estimate the number of notes needed to achieve maximum prediction performance,' and the Results state that for each training-set size 'the maximum AUC/F1 score value was retained' to build Figures 6 and 9. Taking the maximum over up to 330,000 iterations on a fixed test set makes the test set an early-stopping/model-selection set; the reported point is an order statistic biased upward relative to any fixed stopping rule or validation-selected checkpoint. Because the bias is expected to be larger for small training sets (noisier curves), the 600-note versus 6,000-note comparison that anchors the factor-of-10 claim is not a valid estimate of achievable performance. A validation set (or repeated splits with a principled early-stopping rule) is needed to support the headline claim.","section":"Methods – Sampling strategy; Results"},{"comment":"Each configuration is run once and no confidence intervals or repeated-seed estimates are provided. The learning curves in Figures 4 and 5 are highly non-monotonic; the reported point per training-set size is the maximum over a trajectory of up to 330,000 iterations of a single run. Under run-to-run variance, a maximum-of-one-trajectory estimate is unstable, and the 0.949-versus-~0.95 difference between 600 and 6,000 notes is within plausible noise. Please provide repeated runs (at least for the critical sample sizes) and report means with confidence intervals or an equivalent variance estimate.","section":"Results (Figures 4–9)"},{"comment":"The abstract states the target is 'AUC > 0.95' and claims a factor-of-10 reduction in data needed to reach that level, but the result used to support the claim is AUC = 0.949 with 600 notes in Scenario B. Please correct the threshold or the reported value, and specify exactly which target underlies the factor-of-10 statement.","section":"Abstract; Results"},{"comment":"The manuscript does not report the training hyperparameters for the GPT-2 117M model (optimizer, learning rate, gradient clipping, sequence length, vocabulary size, number of pre-training epochs or iterations, and the exact stopping criterion for pre-training in Scenario B). Since the authors retrain from scratch, these choices define the pre-training phase and can affect the measured gain; please provide the full configuration in an appendix or public repository.","section":"Adapting the GPT-2 model to our setting"}],"minor_comments":[{"comment":"The word 'ginve' should be 'given'.","section":"Abstract"},{"comment":"'S CENARIO' in the Figure 1 caption should be 'SCENARIO', and 'Standford Question Answering Dataset' should be 'Stanford Question Answering Dataset'.","section":"Figure 1 caption; Discussion"},{"comment":"The axis labels contain rendering artifacts such as '0/period.math40'; please regenerate the figures so the iteration and metric values are readable.","section":"Figures 4–10"},{"comment":"Please state explicitly whether the smaller labeled samples are nested subsets of the larger ones, and whether the same sampled notes are used in both scenarios; this affects the interpretation of the learning curves.","section":"Methods – Sampling strategy"},{"comment":"The sentence 'At the end of Scenario B, with a training of all 10,000 notes, AUC and F1 score are respectively 0.970 and 0.889, corresponding the cases of more than 100,000 notes in Scenario A' is grammatically unclear; please specify the exact comparison being made.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a preliminary study from a clinical surveillance group. The test-set-based model selection is a serious but fixable flaw; the qualitative direction has strong prior support, so a validation-based reanalysis would be more appropriate than rejection. Please also encourage the authors to provide a data/code availability statement, as none is currently included."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the GPT-2 ER classification paper. Bottom line: the qualitative claim is almost certainly right, but the quantitative factor-of-ten is not well supported as written.\n\nWhat's new: they retrain GPT-2 from scratch on French clinical notes, then fine-tune on small labeled sets, and show the pre-trained model reaches similar AUC with about a tenth of the labels. That's a useful domain/language confirmation of what ULMFiT already showed in English. The paper does it carefully in one sense: they froze a 10k test set, used ICD-10-derived labels to avoid noise, and compared 117M vs 345M. They also cite Howard and Ruder and acknowledge their result is in line.\n\nThe soft spot is the one the stress-test flags. They take the maximum test AUC over the training trajectory for each labeled-set size, and use that maximum to draw the curves. That means the test set is doing double duty as validation set. For small labeled sets the training curve is noisy, so the max over up to 330k iterations will be biased upward, more so for the 600-note run than the 6,000-note run. The factor-of-ten gap could be inflated. No repeated seeds, no confidence intervals, so we have no sense of variance. Also the abstract says AUC>0.95, but the 600-note result is 0.949, which doesn't cross that threshold. Minor, but sloppy.\n\nThere's also the novelty question. The Howard and Ruder result (100 labeled matching 20,000) is a stronger version of the same claim in a general domain. So the paper is confirmatory rather than breaking new ground. That's fine for an applied clinical NLP audience, but it should be framed as such.\n\nRecommendation: I'd send it to a serious referee, but with the expectation that the authors need to rerun with a validation set or at least report results from a fixed early-stopping rule and repeated seeds. The central direction is likely right, and the French clinical setting is underserved. As is, the specific factor-of-ten claim is not solid enough to rely on.","headline":"A useful French clinical confirmatory study of pre-training's sample-efficiency gain, but the headline factor-of-ten is not measured cleanly because the test set doubles as the model-selection set.","tokens_in":9831,"tokens_out":1962,"would_cite":false,"duration_ms":17192,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training on unlabeled ER notes cuts labeled data needs tenfold","keywords":["sample efficiency","pre-training","GPT-2","clinical text classification","emergency room","trauma classification","self-supervised learning","French clinical notes"],"falsifier":"Re-run the two scenarios with a separate validation set for early stopping and keep the 10,000-note test set untouched; if the pre-trained model then needs more than 600 labeled notes to reach AUC 0.949, or the fully supervised model needs fewer than 6,000, the factor-of-ten claim fails. Alternatively, report the AUC at a fixed iteration budget or at the validation-set-selected iteration instead of the maximum over iterations.","tokens_in":8866,"feed_emoji":"🏥","tokens_out":7713,"duration_ms":55397,"temperature":0.7,"pith_summary":"This paper tests whether a neural language model trained from scratch on unlabeled French emergency-room notes can learn to classify a visit as traumatic or non-traumatic using far fewer expert-labeled examples than a model trained only on labeled data. The authors retrain a GPT-2 model from random weights on 151,930 unlabeled clinical notes, then fine-tune it on small labeled sets, and compare this two-stage approach with training the same model from scratch directly on labeled notes. They report that the pretrained model reaches a discrimination score (AUC) above 0.95 with 600 labeled notes, whereas the fully supervised model needs about 6,000 notes to reach the same level—a tenfold reduction in annotation load. If this holds, it would make large-scale automated injury surveillance practical, since expert annotation is the most expensive step in building such classifiers.","feed_headline":"Pre-training on unlabeled ER notes cuts labeled data needs tenfold","feed_subtitle":"A GPT-2 retrained from scratch on French clinical notes matches supervised performance with 600 labels instead of 6,000.","key_machinery":"The load-bearing mechanism is the GPT-2 Transformer language model (117M-parameter version) retrained from a random initialization on 151,930 unlabeled French clinical notes via self-supervised next-token prediction, followed by supervised fine-tuning on labeled notes. Classification is implemented as a question-answering-style next-token prediction: each note is suffixed with the control token \"TARPON\" and the model learns to output \"1\" or \"0\" as the next token. The pre-training phase supplies the model with the vocabulary, abbreviations, and phrasing of emergency-room notes, so the fine-tuning phase only needs to learn the classification decision boundary instead of language understanding from scratch.","core_discovery":"The central claim is that a generative self-supervised pre-training phase on unlabeled clinical notes sharply improves sample efficiency for a downstream classification task. Using a 117-million-parameter GPT-2 retrained from scratch on French emergency-room notes, the authors cast classification as next-token prediction: each note is suffixed with the control keyword \"TARPON\" and the model is trained to emit the token \"1\" for trauma or \"0\" for non-trauma. In Scenario B (pre-training plus fine-tuning), 600 labeled notes yield an AUC of 0.949 and an F1 score of 0.852, matching or exceeding what Scenario A (fully supervised from scratch) achieves with 6,000 labeled notes. With all 10,000 labeled notes, Scenario B reaches an AUC of 0.970, comparable to Scenario A trained on more than 100,000 notes. The authors conclude that the pre-training phase reduces the required number of annotated samples by roughly an order of magnitude.","pith_inferences":["The reported factor-of-ten gain may be optimistic because the paper selects the best iteration using the frozen 10,000-note test set; re-evaluating with a separate validation set for early stopping could shrink or eliminate the gap.","The unusual next-token classification head (rather than a standard softmax layer over the hidden state) may understate what pre-training enables, so a conventional classifier head could show even larger gains.","Since the pre-trained representations are task-agnostic, the same unlabeled corpus should also reduce labeling needs for other ER classification targets such as injury mechanism, intent, or product involved, which the authors plan to test.","The dependence of the gain on the volume of unlabeled notes (151,930 in this study) is untested; a sweep over pre-training corpus sizes would show whether the tenfold gain persists with fewer unlabeled notes."],"forward_implications":["A self-supervised pre-training phase on unlabeled clinical notes reduces the number of expert-annotated notes needed for trauma classification by about a factor of 10.","The GPT-2 question-answering setup works for French clinical text without using OpenAI's English pretrained weights, showing the approach transfers to non-English medical domains.","With only 10,000 labeled notes after pre-training, the model reaches performance comparable to a fully supervised model trained on more than 100,000 labeled notes.","Using the larger 345M-parameter GPT-2 did not significantly improve classification, so the smaller model is sufficient for this task.","The same pipeline is intended to be extended to multi-label coding of injury mechanisms, pointing toward a national automated trauma observatory."],"supporting_citations":[{"why":"Introduces the two-phase generative pre-training and fine-tuning framework that this paper applies.","marker":"Radford et al. 2018"},{"why":"Provides the GPT-2 model architecture and the claim that language models are unsupervised multitask learners.","marker":"Radford et al. 2019"},{"why":"Supplies the Transformer architecture with self-attention that GPT-2 is built on.","marker":"Vaswani et al. 2017"},{"why":"Establishes the transfer of self-attention blocks from pre-trained models, the BERT baseline that motivates the approach.","marker":"Devlin et al. 2018"},{"why":"Reports a comparable text classification result with 100 labeled samples versus 20,000 from scratch, the closest prior evidence for the claimed gain.","marker":"Howard and Ruder 2018"},{"why":"Provides ELMo, another contextualized representation method supporting the value of pre-training.","marker":"Peters et al. 2018"},{"why":"Foundational result that unsupervised pre-training helps deep learning, cited as the origin of the benefit.","marker":"Erhan et al. 2010"},{"why":"Defines the ICD-10 coding standard used to derive the trauma/non-trauma labels from clinical codes.","marker":"Organization 2015"}],"fun_headline_variants":["Tenfold fewer labels with GPT-2 pretraining on ER notes","Unlabeled ER notes cut labeling needs 10x for GPT-2","600 labeled notes match 6,000 via GPT-2 pretraining","Pre-training on unlabeled ER notes: 10x sample efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that taking the maximum AUC over training iterations on the fixed 10,000-note test set is a valid way to measure achievable performance; if that test set doubles as the model-selection set, the reported 600-versus-6,000 gap may be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Tenfold fewer labels with GPT-2 pretraining on ER notes","Unlabeled ER notes cut labeling needs 10x for GPT-2","600 labeled notes match 6,000 via GPT-2 pretraining","Pre-training on unlabeled ER notes: 10x sample efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1787,"prompt_tokens":993,"completion_tokens":794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":717}},"tokens_in":609,"tokens_out":794,"duration_ms":6934,"temperature":1.0,"reasoning_tokens":717,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:05:50.255457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two scenarios with a separate validation set for early stopping and keep the 10,000-note test set untouched; if the pre-trained model then needs more than 600 labeled notes to reach AUC 0.949, or the fully supervised model needs fewer than 6,000, the factor-of-ten claim fails. Alternatively, report the AUC at a fixed iteration budget or at the validation-set-selected iteration instead of the maximum over iterations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPT-2 model architecture and the claim that language models are unsupervised multitask learners."},{"cited_title":"N.; Kaiser, L","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer architecture with self-attention that GPT-2 is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports a comparable text classification result with 100 labeled samples versus 20,000 from scratch, the closest prior evidence for the claimed gain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundational result that unsupervised pre-training helps deep learning, cited as the origin of the benefit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ICD-10 coding standard used to derive the trauma/non-trauma labels from clinical codes."}],"review_version":1}