{"id":"faa5fe49-e119-4404-b991-ba6b8fe14cf6","arxiv_id":"2505.18708","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A general knowledge injection framework that uses synthesized code-knowledge guidelines and multi-task learning to improve ICD coding, achieving state-of-the-art results on MIMIC-III benchmarks.","lead":"GKI-ICD is a training framework that improves automated ICD coding by feeding the model a synthesized guideline built from code descriptions, synonyms, and hierarchy, alongside the original medical text. It reports state-of-the-art results on MIMIC-III, but most of the gain comes from the description knowledge, not the much-touted combination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic-similarity loss (Eq. 14) is the core of the proposed multi-task mechanism, but no ablation removes it; gains may come entirely from Lguide plus R-Drop, leaving the central knowledge-alignment claim untested.","rationale":"I focused on the missing Lsim ablation because it targets the mechanism that distinguishes GKI-ICD from simply training on extra description text. The reader's weakest assumption (domain gap breaking Lsim) is a plausible failure mode, but the prior question is whether Lsim has any measurable effect. Other concerns (unfair PLM-CA comparison, missing error bars, overclaimed SOTA) are real and already noted by the reader; they affect the magnitude and presentation of results but not the identity of the proposed mechanism. If Lsim is unnecessary, the paper's central 'knowledge alignment' contribution and its applicability claim are significantly weakened. The proposed λ=0 check is cheap and decisive. Since this is an experimental gap rather than a demonstrated error, CONDITIONAL acceptance remains appropriate.","tokens_in":11638,"tokens_out":7841,"duration_ms":70935,"concrete_test":"Run the MIMIC-III-Full description-only condition with all settings identical to the 'w/ desc' row of Table 3 but set λ=0 (equivalently drop Lsim from Eq. 15). Compare macro/micro F1 and P@8/P@15 to Table 3. If the metrics remain within ~0.1 macro F1 and ~0.5 micro F1 of the λ>0 run across three seeds, the similarity constraint is not load-bearing and the method's contribution reduces to auxiliary guideline supervision; if they drop substantially, the alignment mechanism is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.2 presents the framework as two coupled mechanisms: supervised prediction from the synthesized guideline (Lguide, Eq. 11) and a cosine-similarity constraint (Lsim, Eq. 14) that aligns raw-text code evidence with guideline evidence. The paper's novelty claim is that this alignment injects knowledge into the encoder. However, Table 3 only ablates knowledge types (desc/syn/hie), never the loss terms: every 'w/ knowledge' row includes Lraw + Lguide + λLsim, and the 'w/o knowledge' row already includes R-Drop and 8,192-token truncation, so the contribution of Lsim is completely confounded with Lguide and with R-Drop. It is entirely possible that Lguide alone (supervising the model on clean code descriptions) produces the reported gains, and that Lsim is neutral or harmful; the reported λ value is also omitted. Without this ablation, the central claim that cross-sample semantic alignment is what transfers knowledge to raw-text inference is not supported by the experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GKI-ICD, a training-time knowledge-injection framework for ICD coding. For each training document, it synthesizes a 'guideline' string from the ground-truth ICD codes' official descriptions, UMLS synonyms, and ICD-9 hierarchical descriptions. The ICD coding model (RoBERTa-PM encoder with cross-attention label queries, following PLM-CA) is then trained with three losses: binary cross-entropy on the raw text, binary cross-entropy on the synthetic guideline, and a cosine-similarity constraint between the code-specific evidence vectors extracted from the raw text and from the guideline. Experiments on MIMIC-III-Full and MIMIC-III-Top-50 report improvements over several baselines and particularly large gains for rare codes, and the code is publicly released. The manuscript's own Limitations section acknowledges that the method is evaluated only on ICD-9 and that it does not use the Alphabetic Index.","tokens_in":11821,"tokens_out":6410,"duration_ms":48624,"significance":"If the central claim holds, GKI-ICD is a useful contribution: it is architecture-agnostic, requires no extra parameters or specialized modules, leaves inference-time cost unchanged, and shows a large rare-code improvement (Table 4, F1 0.132 vs. 0.029 for codes occurring at most 10 times). The paper ships code, states the loss terms explicitly, and honestly lists its scope limitations (ICD-9 only, no Alphabetic Index). However, the current experimental evidence does not isolate the mechanism that is claimed to be central---the semantic-similarity alignment of Eq. (14)---and the reported SOTA claim is stronger than the table supports. These gaps are fixable but require additional experiments and a more careful presentation of results.","major_comments":[{"comment":"The claim that 'our method outperforms state-of-the-arts significantly on all evaluation metrics' is not supported by the table. On MIMIC-III-Top-50, CoRelation has higher Macro F1 (0.693 vs. 0.692) and higher P@5 (0.683 vs. 0.681), and MRR has higher P@5 (0.685). On MIMIC-III-Full, AKIL and MRR have higher P@15 (0.637 and 0.623 vs. 0.624 for GKI-ICD) and AKIL has higher P@8 (0.784 vs. 0.777). Even setting aside the methods that use additional human annotations, CoRelation beats GKI-ICD on two Top-50 metrics. The paper should state precisely on which metrics and against which baselines GKI-ICD is state-of-the-art, and should not claim superiority on all metrics.","section":"Section 4.2, Table 2"},{"comment":"The ablation study never removes the semantic-similarity loss L_sim. Every 'w/ knowledge' row in Table 3 is trained with the full objective L_raw + L_guide + lambda*L_sim, while the 'w/o knowledge' row already includes R-Drop regularization and 8,192-token truncation. Consequently, the contribution of Eq. (14)---the mechanism claimed to align raw-text evidence with guideline evidence and thereby inject knowledge---is completely confounded with L_guide and with R-Drop. The reported gains could in principle come entirely from supervising the model on clean code-description strings (L_guide) rather than from the cross-sample similarity constraint. An ablation with L_raw + L_guide but without L_sim, and ideally also a variant with L_sim alone or with lambda=0, is required to support the paper's central claim about knowledge alignment.","section":"Section 4.3, Table 3 and Eq. (15)"},{"comment":"The hyperparameter lambda that weights the semantic-similarity loss is never reported. Eq. (15) defines the total loss with lambda, but the implementation-details paragraph gives only the learning rate, batch size, epochs, and the R-Drop coefficient alpha. Since L_sim is the central novel term, the reader cannot tell whether lambda was tuned, what value was used, or whether the reported results are sensitive to it. A sensitivity study over lambda (e.g., 0.1, 1, 10) should be added, and the selected value must be stated.","section":"Section 4.1, Eq. (15)"},{"comment":"The main comparison against PLM-CA is not apples-to-apples. The note under Table 3 says that all models, including the 'w/o knowledge' baseline, use R-Drop and truncate input to 8,192 tokens, whereas PLM-CA used 6,144 tokens and no R-Drop. The reported 4.6% Macro-AUC improvement over PLM-CA on MIMIC-III-Full therefore mixes the effect of knowledge injection with the effects of additional regularization and a longer truncation window. The authors should either re-run PLM-CA under the same training recipe, or use the 'w/o knowledge' row as the direct control in the SOTA table. Additionally, no multiple-seed results or significance tests are reported, so the word 'significantly' is not justified even where the improvements are numerically positive.","section":"Section 4.2, Table 3 note"}],"minor_comments":[{"comment":"The dimension of the attention matrix A_i is inconsistent: Eq. (17) computes a vector of attention weights over the N input tokens, but the text and Eq. (18) describe A_i as an R^{C x N} matrix. A_i should be a vector in R^N for each code i, and the notation should be corrected.","section":"Section 3.3, Eqs. (17)-(18)"},{"comment":"There are several typos and corrupted figure strings: 'knowldge' in the contributions list, 'fisrt' in Section 2.1, 'Prediciton' and 'Esophegeal Reflux' in Figure 2, and the nonsensical synonym strings 'sahedfoufhlsreflux' and 'shcofiaodi' in Figure 2. The figure should be regenerated and proofread.","section":"Throughout"},{"comment":"The phrase 'state-of-the-arts' is nonstandard; 'state-of-the-art methods' or 'state of the art' would be clearer.","section":"Section 4.2"},{"comment":"The implementation-details paragraph does not state the number of random seeds, the random seed, or how hyperparameters (including lambda and alpha) were selected. Adding these details would improve reproducibility.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The framework idea is clean and the paper is likely of interest to the clinical NLP community, but the experimental support for the central alignment mechanism is incomplete. The most important fix is an ablation that removes L_sim while keeping L_guide, plus reporting lambda and multi-seed variance. I do not see a fundamental flaw in the approach itself, so the appropriate decision is major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: GKI-ICD is a clean, architecture-agnostic way to inject ICD code knowledge by synthesizing a guideline from the ground-truth codes and training the model to predict from both the raw text and the guideline, with a cosine alignment between the two. The headline result — rare-code F1 from 0.029 to 0.132 on MIMIC-III — is genuinely useful, and the main claim that adding description-based guidelines improves performance holds up in their ablations. The paper is worth a look.\n\nWhat's new: they avoid the usual graph networks and multi-synonym attention, and instead turn each code's description, synonyms, and hierarchy into a synthetic training document. That is clean and should transfer to other multi-label tasks. They also ship code, which is more than many papers in this area do.\n\nSoft spots, in order of importance. First, the stress-test is right: the similarity loss (L_sim) is the conceptual core of the \"alignment,\" but no ablation removes it. Every 'with knowledge' row includes L_guide and λL_sim together, so we cannot tell whether the gains come from the alignment or simply from the extra supervised signal on clean descriptions. The λ value is never reported. That is the largest gap. Second, the SOTA claim is overstated: they say \"outperforms state-of-the-arts significantly on all evaluation metrics,\" but their own Table 2 shows AKIL and MRR beat them on Micro-AUC and P@K. The caveat about extra human annotations is fair, but the sentence should be softened. Third, the ablation shows synonym and hierarchy knowledge contribute almost nothing on macro AUC (0.960 to 0.962) and only a couple of points on F1, which undercuts the \"complementarity\" framing. Fourth, no error bars or multiple seeds; given the small gains on some metrics, a single run is not convincing. The comparison with PLM-CA also uses different truncation and R-Drop, though they disclose this and the w/o knowledge row is a proper controlled baseline. The circularity concern is not real: the guideline is built from labels, but that is a training-time technique and the knowledge itself is external.\n\nAll of these are fixable. The rare-code result and the simplicity of the method make it a serious candidate for publication, but the authors need to ablate L_sim, report λ and seeds, and tone down the SOTA claim.\n\nRecommendation: send to peer review. It needs revision, but there is a usable core.","headline":"A simple, architecture-agnostic knowledge-injection framework for ICD coding that clearly helps rare codes, but the central similarity-loss mechanism is never ablated and the SOTA claim is overstated.","tokens_in":12371,"tokens_out":3485,"would_cite":true,"duration_ms":26312,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GKI-ICD turns ICD descriptions, synonyms, and hierarchy into synthetic guidelines that teach coding models during training alone, improving accuracy, especially on rare codes.","keywords":["ICD coding","knowledge injection","multi-task learning","guideline synthesis","long-tail distribution","rare codes","clinical text classification","multi-label classification"],"falsifier":"Replace every code's description, synonym, and hierarchy in the synthesized guidelines with random text drawn from the same source vocabulary and retrain; if validation F1 does not drop relative to the full framework, the specific knowledge content is not what drives the gain. A complementary check is to hold out all examples of a rare code from training and see whether the rare-code F1 gain over the base model disappears, as the paper's mechanism predicts it should.","tokens_in":11446,"feed_emoji":"🩺","tokens_out":8572,"duration_ms":67256,"temperature":0.7,"pith_summary":"The paper tries to show that the knowledge needed for automatic ICD coding — what each code means, what words patients and doctors actually use for it, and where it sits in the code hierarchy — can be injected into a model without adding any new neural network components. The trick is to build a clean synthetic guideline from the codes assigned to a training document, then train the model to make the same predictions from the guideline and from the messy original text, while aligning the evidence it attends to in both. If the framework works as claimed, knowledge injection stops being a collection of incompatible specialized modules and becomes a general training recipe, with the largest payoff on rare codes that have few training examples. That would matter because ICD coding is a bottleneck in medical billing and research, and rare codes are exactly where annotators and models struggle.","feed_headline":"Three knowledge types, no new modules: better ICD coding","feed_subtitle":"Synthesized guidelines from descriptions, synonyms, and hierarchy boost accuracy without changing the model.","key_machinery":"The load-bearing object is the synthesized guideline $\\hat{x}$: for each training document, the assigned codes are shuffled, and each code is expanded into its official description, a randomly chosen synonym, and the descriptions of all parent groups in the ICD hierarchy, then concatenated into a single text string. The training signal combines three losses: binary cross-entropy on the raw clinical text, binary cross-entropy on the guideline, and a cosine-similarity term $L_{\\text{sim}} = 1 - \\cos(E, \\hat{E})$ that aligns the aggregated code-specific evidence from both inputs. The framework's novelty is that the knowledge enters only through this synthetic text and these losses, not through graph networks, synonym-attention modules, or prompt engineering.","core_discovery":"The paper's central claim is that a training-time framework, GKI-ICD, can inject three complementary types of ICD knowledge — the official description of each assigned code, randomly sampled synonyms from a biomedical terminology source, and the code's place in the ICD hierarchy — by synthesizing a clean guideline text and training the model to do three things at once: predict codes from that guideline, predict codes from the raw clinical note, and pull the code-specific evidence representations of the two inputs toward each other with a cosine similarity loss. No new network modules are introduced, so the same recipe attaches to any existing encoder-and-label-attention coding model. The paper reports that this beats its strongest baselines across nearly all metrics on the standard full and top-50 clinical benchmarks, and that the biggest gain appears on codes with ten or fewer training examples.","pith_inferences":["A testable extension is applying the same recipe to ICD-10 or other coded vocabularies; the paper only validates on ICD-9, so whether the gains survive the larger, more granular code space is open.","The cosine alignment is one of many possible constraints; a contrastive or distribution-matching alternative might be more robust when the guideline and the note are semantically distant, but the paper does not compare such variants.","The rare-code gain hints that the framework could reduce dependence on extra human-annotated inputs such as DRG or procedure codes, but the paper only shows comparable performance, not that the framework can replace those annotations.","Because synonym selection is random, the method could be made stronger with a curriculum that prefers synonyms hard to align to the raw text; the paper does not test this."],"forward_implications":["Any ICD coding model that already extracts code-specific evidence from text can adopt the framework without architectural changes, so knowledge injection becomes compatible with future backbones.","Combining description, synonym, and hierarchy knowledge yields higher accuracy than any single knowledge source alone, which the ablation tables support.","The largest measured improvement is on rare codes (ten or fewer training examples), suggesting the framework directly targets the long-tail problem that dominates ICD coding.","Because the knowledge enters only at training time, inference speed and memory use are unchanged relative to the base model.","The framework is a general multi-label recipe: any task with label-specific descriptions, synonyms, or hierarchies could reuse the same guideline-synthesis and alignment scheme."],"supporting_citations":[{"why":"Supplies the base model architecture — encoder, cross-attention code queries, and evaluation setup — that GKI-ICD wraps.","marker":"(Edin et al., 2024)"},{"why":"Provides the MIMIC-III train/dev/test split, the label-attention design, and the F1/AUC/P@K metrics used throughout.","marker":"(Mullenbach et al., 2018)"},{"why":"Representative hierarchy-aware baseline that injects ICD hierarchy via graph networks, the approach GKI-ICD replaces without extra modules.","marker":"(Xie et al., 2019)"},{"why":"Source of the synonym-knowledge idea and the multi-synonym matching baseline; GKI-ICD substitutes random synonym sampling.","marker":"(Yuan et al., 2022)"},{"why":"Supplies the biomedical terminology resource (UMLS) from which synonyms and Concept Unique Identifiers are retrieved.","marker":"(Bodenreider, 2004)"},{"why":"Gives the pretrained biomedical language model used as the text encoder for all experiments.","marker":"(Lewis et al., 2020)"}],"fun_headline_variants":["ICD coding: three knowledge types, zero new modules","Rare ICD codes: three fused knowledge types lift accuracy","Synthesize guidelines from ICD description, synonym, hierarchy","No architecture changes: fuse three ICD knowledge types","Better ICD coding with fused knowledge, no module redesign"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach assumes that training on clean summary sentences built from the true codes makes the model better at finding evidence in messy real clinical notes, rather than simply memorizing the summary format.","fun_headline_variants_meta":{"raw":{"variants":["ICD coding: three knowledge types, zero new modules","Rare ICD codes: three fused knowledge types lift accuracy","Synthesize guidelines from ICD description, synonym, hierarchy","No architecture changes: fuse three ICD knowledge types","Better ICD coding with fused knowledge, no module redesign"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000776,"raw_usage":{"total_tokens":3400,"prompt_tokens":883,"completion_tokens":2517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2439}},"tokens_in":499,"tokens_out":2517,"duration_ms":21086,"temperature":1.0,"reasoning_tokens":2439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:26:22.053499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace every code's description, synonym, and hierarchy in the synthesized guidelines with random text drawn from the same source vocabulary and retrain; if validation F1 does not drop relative to the full framework, the specific knowledge content is not what drives the gain. A complementary check is to hold out all examples of a rare code from training and see whether the rare-code F1 gain over the base model disappears, as the paper's mechanism predicts it should.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the synonym-knowledge idea and the multi-synonym matching baseline; GKI-ICD substitutes random synonym sampling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the biomedical terminology resource (UMLS) from which synonyms and Concept Unique Identifiers are retrieved."}],"review_version":1}