{"id":"a8918eff-814d-42e1-893c-4630ee3178ef","arxiv_id":"2412.11477","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Contrastive pre-training that aligns medical note text with ICD-10 code sequences outperforms prior models on MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full, with the clearest gains on rare codes.","lead":"A pre-training method that matches doctors' notes with their ICD-10 diagnostic codes improves automatic medical coding on three benchmark tasks, with the largest gains on rare conditions. It matters because faster and more sensitive coding could reduce manual documentation and billing work in hospitals and improve data for clinical research.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rare50 gains may be driven by ICD-10 encoder vocabulary coverage rather than contrastive alignment; the attribution needs an in-/out-of-vocabulary split of the 50 rare labels to rule this out.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the ICD-10 encoder's vocabulary cutoff and random ICD-9-to-ICD-10 mapping may misrepresent the rare codes that dominate the rare50 benchmark, and the NoteLM versus NoteContrast ablation does not control for encoder pretraining data, vocabulary, or institution. My pass sharpens this into a specific, falsifiable confound: the attribution of the rare-code gains to data-driven code-sequence pretraining is secure only if the improvement is not concentrated in labels whose mapped ICD-10 codes are absent from the encoder vocabulary. If the rare50 advantage vanishes for out-of-vocabulary labels, the benchmark numbers could be correct while the paper's explanatory claim is wrong; if it persists, the transfer concern is substantially weakened. I do not see a reason to change the reader's CONDITIONAL verdict: the concern is concrete and testable, and conditional acceptance with the proposed vocabulary-split analysis is the appropriate bar. I would not escalate to rejection because the internal arithmetic, five-seed averaging, and train/test patient exclusion are all consistent with the reported results.","tokens_in":17722,"tokens_out":7267,"duration_ms":68040,"concrete_test":"Use the authors' ICD-9-to-ICD-10 mapping table and the ICD-10 encoder vocabulary file (the vocabulary is shareable even if weights are not) to partition the 50 MIMIC-III-rare50 labels into in-vocabulary and out-of-vocabulary sets. Recompute macro-AUC and macro-F1 on the 142-note rare50 test set for NoteContrast 8k ICD and for the matched KEPT/NoteLM baselines separately on each subset. If the NoteContrast advantage disappears or reverses on out-of-vocabulary labels, the reported rare-code gains reflect encoder coverage rather than contrastive alignment; if it persists on out-of-vocabulary labels, the transfer concern is not the driver. Additionally, report label-level counts of positive test instances in each subset to confirm the split has enough statistical power.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest evidence for the central claim is Table 2's rare50 results, where NoteContrast 8k ICD reaches macro-AUC 88.92 versus 82.30 for KEPT and macro-F1 40.26 versus 28.94. This comparison is meant to show that aligning notes with a data-driven ICD-10 code-sequence encoder beats ontology-based code distances. That attribution is load-bearing and insecure for the following reason. The code encoder was trained on real-world ICD-10 sequences with a vocabulary cutoff; Section 5.1 states that 'certain rare codes are likely missing from our diagnostic code model.' The contrastive pre-training pairs MIMIC notes with ICD-10 codes obtained by an ICD-9-to-ICD-10 mapping in which ambiguous mappings are resolved at random (Section 3.5). If the mapped ICD-10 codes for the 50 rare labels are absent from the encoder vocabulary, the contrastive step never provides a code-side representation for those labels, so any rare50 gain cannot be attributed to the data-driven code-sequence encoder. Conversely, if those labels are present, the large gains could still reflect the encoder's real-world pretraining rather than contrastive alignment; the NoteLM versus NoteContrast ablation controls for the training objective but not for encoder pretraining data, vocabulary, or institution. The fixed 142-note rare50 test set (Table 8) makes the per-label comparisons especially sensitive to which labels are covered. This is not an accusation of leakage; it is an unidentified confound in the attribution of the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NoteContrast, a contrastive pretraining method that aligns medical-note text with ICD-10 diagnostic-code sequences. The diagnostic-code encoder is a RoBERTa-style transformer trained on 7.5 million real-world sequences from MassGeneral Brigham, and the text encoder is a BioLM-based BigBird model pretrained on MIMIC-III notes; the two are jointly trained with an InfoNCE loss plus masked language modeling, followed by a short contrastive fine-tuning step using ICD-10 code descriptions. The authors evaluate on MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding, reporting that NoteContrast outperforms prior state-of-the-art models such as KEPT, MSMN, ISD, TreeMAN, and DiscNet+RE on most metrics. The central claim is that aligning note text with a data-driven code-sequence encoder rather than ontology-based code distances is what drives the improvement.","tokens_in":17796,"tokens_out":6386,"duration_ms":55920,"significance":"If the results hold, the paper makes a useful contribution: it provides a clean recipe for contrastive language-diagnostic pretraining, with five-seed averages, dev-set threshold selection, explicit exclusion of MIMIC-III test patients from pretraining, and publicly available code. The NoteLM versus NoteContrast ablation does isolate the contrastive objective from the text-encoder architecture, and the gains on MIMIC-III-50 and especially MIMIC-III-rare50 are large. However, the mechanistic attribution to the data-driven ICD-10 encoder is partially confounded by encoder vocabulary coverage and by the random ICD-9-to-ICD-10 mapping, and the abstract overstates performance on the MIMIC-III-full task. The contribution is therefore significant but needs qualification and additional analysis before the paper's main attribution can be accepted.","major_comments":[{"comment":"The paper's central mechanistic claim—that aligning notes with a data-driven ICD-10 sequence encoder, rather than ontology-based code distances, drives the rare-code gains—is not fully supported by the current evidence. Section 5.1 states that 'certain rare codes are likely missing from our diagnostic code model' due to vocabulary cutoff, and Section 3.5 maps ICD-9 codes to ICD-10 with 'selected an ICD10 code at random' for ambiguous mappings. Since the rare50 labels are rare by construction, the macro-AUC/F1 improvements over KEPT (88.92 vs 82.30 and 40.26 vs 28.94 in Table 2) could reflect which rare labels happen to have a code-side token in the MassGeneral Brigham encoder vocabulary, rather than the contrastive alignment itself. The NoteLM versus NoteContrast ablation controls for the training objective but not for encoder pretraining data, vocabulary, or institution. Please report Table 2 results split by whether each label's mapped ICD-10 code is in the encoder vocabulary, and ideally compare with a code encoder trained without the real-world MGB sequences or with a vocabulary-based coverage control.","section":"§3.5, §5.1, Table 2"},{"comment":"The abstract's claim that the contrastive approach 'improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks' is contradicted by the paper's own Table 3 for the full task: NoteContrast 8k ICD achieves macro-F1 11.9 ± 0.3, while DiscNet+RE achieves 14 and ISD achieves 11.90 ± 0.2. The body correctly says the model 'improved over the previous state-of-the-art method for most metrics,' but the abstract and conclusion are not so qualified. The claim should be restricted to micro-averaged metrics and precision/recall@K, or the macro-F1 deficit should be discussed explicitly.","section":"Abstract; §4.3, Table 3"},{"comment":"The rare50 headline result is obtained from a NoteContrast 8k ICD model that was first fine-tuned on MIMIC-III-50, which the authors report 'performed better than fine-tuning the NoteContrast 8k ICD model on the MIMIC-III-rare50 dataset alone.' The paper does not state whether this initialization choice was made using the development set or the test set. If the choice was made after inspecting test performance, the reported rare50 gains are inflated by test-set model selection. Please clarify the selection rule and report both variants as pre-specified or with a dev-based selection criterion. Given the 142-note test set, per-label breakdowns and bootstrap confidence intervals would also help gauge stability.","section":"Table 2, §4.3"}],"minor_comments":[{"comment":"'2,083,180 million de-identified notes' should read '2,083,180 de-identified notes'; the word 'million' is erroneous.","section":"Appendix A.2.2"},{"comment":"Footnote 1's URL is run into the text after 'ICD-9 codes to ICD-10 codes 1.'; please add a space or restructure the sentence.","section":"§3.5"},{"comment":"The task names 'MIMIC-50', 'MIMIC-III-50', and 'MIMIC-50 rare' are used inconsistently; please define the three benchmarks once and use the same names throughout.","section":"Throughout"},{"comment":"Table 5 groups four model variants under a single set of hyperparameters, but 'Maximum Sequence Length' differs by model; please state clearly that all other listed hyperparameters are identical across variants, or indicate exceptions in a footnote.","section":"Tables 5–7"},{"comment":"The UMAP axis labels appear as 'UMAP1' and 'UMAP2' without a separating space; also, the caption should state whether the 5000-code sample is fixed across the four panels so the Procrustes alignment is well defined.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The main reason for major revision is the rare50 attribution issue: the vocabulary-coverage confound is real and should be addressed empirically before the paper's mechanistic claim is accepted. The abstract overclaim about MIMIC-III-full is easy to fix but should be corrected. The paper is otherwise carefully benchmarked, and the code release is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper, the strongest of the three benchmark claims holds up, and the rare-code result is real but the attribution needs one more analysis before I'd take it at face value. Worth sending to review.\n\nWhat's new: most prior work injects ontology structure into coding models; this paper instead pretrains a transformer on 60 million real-world ICD-10 sequences and contrastively aligns note text with those code-sequence embeddings. That combination is genuinely absent from the cited literature. The paper is also careful where it counts: MIMIC test patients are excluded from pretraining, the ICD-10 encoder is trained on an independent hospital system, and the NoteLM/NoteContrast ablation with five seeds isolates the contrastive objective. The arithmetic in the abstract and tables is internally consistent, and the limitations section is honest about the vocabulary cutoff and batch-size constraints.\n\nWhere it's soft: the abstract says 'improves performance ... for MIMIC-III-full' but Table 3 shows macro-F1 below DiscNet+RE; that is an overstatement. The full task is a re-ranker over MSMN candidates, not an end-to-end coding system, so claims on that benchmark should be framed as re-ranking. The rare50 test set is 142 notes; a 40 vs 29 macro-F1 difference is large but carries unquantified sampling variance. Most importantly, the attribution of the rare gains to the data-driven code encoder is not fully isolated: the encoder has a vocabulary cutoff that likely misses rare codes, the ICD-9-to-ICD-10 mapping resolves ambiguous mappings at random, and the NoteLM baseline does not control for encoder pretraining data or institution. An in-/out-of-vocabulary split of the 50 rare labels would settle whether the gains come from contrastive alignment or from which codes the encoder happened to cover. That is a request, not a refutation.\n\nBottom line: clinical NLP and ML4H readers will get real value; the method is reproducible enough (code on GitHub) and the benchmark math is sane. Send it to a serious referee, ask for the vocab split and clearer full-task framing.","headline":"Genuinely new contrastive pretraining recipe for medical coding, with solid benchmark work and honest limitations; the rare-code attribution needs an in/out-of-vocabulary split before I'd fully trust it.","tokens_in":18581,"tokens_out":2373,"would_cite":true,"duration_ms":21202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that contrastive pretraining aligning medical notes with ICD-10 diagnosis sequences improves automated diagnostic coding, with the largest gains on rare codes.","keywords":["automatic diagnostic coding","medical text","contrastive learning","ICD-10","ICD-9","MIMIC-III","representation learning","rare disease coding"],"falsifier":"Run the same text encoder and prompt-based fine-tuning but replace the contrastively pretrained ICD-10 encoder with a static embedding derived from the ICD-10 hierarchy, and compare macro-F1 on MIMIC-III-rare50; if the rare-code advantage disappears, the claim that contrastive alignment of data-driven code sequences is the driver is falsified. A second check: replace the random ICD-9-to-ICD-10 mapping with a deterministic, frequency-weighted mapping and see whether the rare-code numbers move.","tokens_in":17283,"feed_emoji":"🩺","tokens_out":10637,"duration_ms":81781,"temperature":0.7,"pith_summary":"The paper claims that pre-training a medical-text model together with a model of ICD-10 diagnosis sequences, using a contrastive objective, improves automated diagnostic coding over prior state-of-the-art models. The authors evaluate on the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full benchmarks and report gains across macro and micro AUC and F1, with the largest improvements on rare codes. They argue that data-driven representations of diagnosis codes, learned from a large real-world corpus of hospital encounters, capture structure that static ontology-based code distances miss. If true, this means that the routine pairing of clinical notes with their billed diagnostic codes is a strong pretraining signal for medical language understanding, and that contrastive alignment is a practical route to better code assignment.","feed_headline":"Rare-code ICD coding jumps 11 F1 points with contrastive pretraining","feed_subtitle":"Aligning medical notes with ICD-10 diagnosis sequences beats prior methods on MIMIC-III-50, rare50, and full.","key_machinery":"The central mechanism is the contrastive alignment of two encoders: a medical-text encoder for long notes and a diagnosis-code encoder for sequences of ICD-10 codes from the same clinical encounters. The diagnosis encoder is a transformer pretrained with a masked-language objective on millions of real-world hospital encounters, using relative position encodings to capture temporal proximity to the encounter of interest. During contrastive pre-training, the InfoNCE loss maximizes cosine similarity between the text embedding of a note and the code-sequence embedding of its associated codes, while minimizing similarity to other notes in the batch; the model also keeps a masked-language loss on the text side, with the two losses combined by uncertainty weighting. Prompt-based fine-tuning with textual descriptions of ICD-9 codes then adapts the aligned representations to the target coding tasks.","core_discovery":"The best NoteContrast model, which combines a long-document text encoder, a data-driven ICD-10 sequence encoder, and contrastive fine-tuning, reaches macro-AUC 93.8 and macro-F1 69.2 on MIMIC-III-50, and on MIMIC-III-rare50 it reaches macro-AUC 88.92 and macro-F1 40.26 when initialized from MIMIC-50 fine-tuning, versus 82.30 and 28.94 for the previous best prompt-based model under the same initialization. The authors attribute the gains to the contrastive pre-training step: aligning note representations with ICD-10 code-sequence representations pulls matching note-code pairs together and separates non-matching pairs, and this signal, combined with an auxiliary masked-language objective, produces embeddings that transfer to ICD-9 coding at fine-tuning time.","pith_inferences":["The random selection among ambiguous ICD-9-to-ICD-10 mappings is a potential confound: a systematic mapping might change which codes the encoder sees and could either inflate or deflate rare-code scores; re-running with a deterministic mapping would isolate this effect.","Because the contrastive batch size is small (64) relative to large-scale multimodal pretraining, the InfoNCE objective may be under-sampled; larger batches could sharpen the rare-code alignment and further improve the tail.","The same architecture could be turned into a retrieval system: given a note, retrieve likely codes directly from the shared embedding space, which would make the model's coding decisions more inspectable than a flat classifier."],"forward_implications":["Automated coding systems can be built from routinely available billing data: the note-code pairs used for contrastive pretraining require no manual annotation beyond the codes already assigned for billing.","Rare-disease coding, where ontologies are sparse and static distances mislead, is the regime that benefits most: macro-F1 on MIMIC-III-rare50 jumps by more than eleven points over the prior best model.","Longer document contexts matter: scaling the text encoder from 4096 to 8192 tokens further improves performance, suggesting that full discharge summaries carry coding-relevant information beyond the first few thousand tokens.","The pretraining recipe is portable: the same contrastive objective can be applied to any structured label with noisy correspondences to clinical text, without ontology engineering."],"supporting_citations":[{"why":"Supplies the prompt-based fine-tuning framework and the MIMIC-III-50/rare50 baselines that the paper compares against.","marker":"Yang et al., 2022"},{"why":"Provides the MIMIC-III notes used for text pretraining, contrastive pretraining, and all three coding benchmarks.","marker":"Johnson et al., 2016"},{"why":"Defines the InfoNCE contrastive loss used to align note and code-sequence embeddings.","marker":"Oord et al., 2018"},{"why":"Provides the medical language model checkpoint used to initialize the text encoder.","marker":"Lewis et al., 2020"},{"why":"Offers the long-document transformer conversion that extends the text encoder to 8192 tokens.","marker":"Beltagy et al., 2020"},{"why":"Defines the sinusoidal positional embeddings used to encode relative encounter timing in ICD-10 sequences.","marker":"Vaswani et al., 2017"},{"why":"Justifies the 20% masking rate used when pretraining the ICD-10 sequence encoder.","marker":"Wettig et al., 2023"},{"why":"Supplies the uncertainty weighting that combines the contrastive and masked-language losses.","marker":"Kendall et al., 2018"}],"fun_headline_variants":["Contrastive pre-training boosts rare ICD-10 coding by 11 F1","Aligning notes and ICD-10 codes beats prior coding models","NoteContrast: new SOTA for MIMIC-III diagnostic coding tasks","From text to ICD-10: contrastive model tops MIMIC-III benchmarks","Rare-code coding gains 11 F1 via note-code contrastive alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the data-driven ICD-10 sequence encoder, pretrained with a vocabulary that omits some rare codes and connected to ICD-9 through a random mapping for ambiguous cases, transfers accurately to the MIMIC-III coding tasks; if those codes are missing or mistranslated, the reported rare-code improvements may come from encoder coverage rather than from the contrastive objective.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive pre-training boosts rare ICD-10 coding by 11 F1","Aligning notes and ICD-10 codes beats prior coding models","NoteContrast: new SOTA for MIMIC-III diagnostic coding tasks","From text to ICD-10: contrastive model tops MIMIC-III benchmarks","Rare-code coding gains 11 F1 via note-code contrastive alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1616,"prompt_tokens":938,"completion_tokens":678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":578}},"tokens_in":554,"tokens_out":678,"duration_ms":5994,"temperature":1.0,"reasoning_tokens":578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:53:47.768688+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same text encoder and prompt-based fine-tuning but replace the contrastively pretrained ICD-10 encoder with a static embedding derived from the ICD-10 hierarchy, and compare macro-F1 on MIMIC-III-rare50; if the rare-code advantage disappears, the claim that contrastive alignment of data-driven code sequences is the driver is falsified. A second check: replace the random ICD-9-to-ICD-10 mapping with a deterministic, frequency-weighted mapping and see whether the rare-code numbers move.","supporting_citations":[{"cited_title":"Should you mask 15 \\ In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985--3000, Dubrovnik, Croatia, 2023","cited_arxiv_id":null,"evidence_quote":"Justifies the 20% masking rate used when pretraining the ICD-10 sequence encoder."}],"review_version":1}