{"id":"dca4ceb2-bd05-4f7a-9149-dd9def66604e","arxiv_id":"2507.01802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An empirical study of the MDACE dataset and current explainable ICD coding models, introducing match measures and showing that supervised models align with human evidence in most test cases.","lead":"This paper analyzes the MDACE dataset of clinical notes annotated with ICD code evidence and tests how well explainable medical coding models extract evidence that matches human annotations. It proposes proximity-aware match measures and gives recommendations for evaluating evidence extraction in ICD coding.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central '~80% overlap' claim is not controlled for extraction breadth: proximate/partial matches require only one overlapping token, so broad or duplicate-emitting explanations count as successes; no precision baseline is reported.","rationale":"The reader's weakest assumption (gold-standard completeness and single-seed sampling) is real, but the more load-bearing issue is that the proposed match measures—the paper's central methodological contribution—do not control for precision. The 'at least one correct token' headline counts broad/duplicate outputs as success, and the paper's own qualitative observation of duplicate-heavy model evidence (Sec. 4.5) shows this is not a hypothetical failure mode. I therefore do not object to the CONDITIONAL verdict: the quantitative claims need either a precision-controlled match measure or a trivial baseline before they can support the recommendation that supervised evidence training produces meaningfully better evidence. The reader's rationale already notes 'no false-positive evaluation,' so my concern partially overlaps; the reader emphasizes seed/gold-standard more, whereas I view the metric's lack of breadth control as the decisive gap. A concrete sensitivity/baseline test would settle whether the 80% overlap is informative.","tokens_in":12982,"tokens_out":5500,"duration_ms":65279,"concrete_test":"Re-run the match-category analysis on the same 61-document test set with two baseline extractors matched to the supervised model's output volume: (1) a random token selector, and (2) the top-k attribution tokens for k chosen so that the baseline emits the same total token count as the supervised model. Recompute Figure 6 and the 80% statistic; if either baseline reaches comparable proximate+partial rates, the measures do not discriminate evidence quality. Also report mean evidence-spans-per-case and exact-match counts after removing duplicate spans.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Sec. 4.3: 'In roughly 80% of test cases, the best model (supervised) identified at least one correct token') is computed from exact, proximate, and partial matches. But the taxonomy is built on extremely permissive success conditions: a proximate match only requires each ground-truth sequence to share at least one token, with unmatched tokens inside a fixed context window of k=10; a partial match is triggered by any single missed sequence or any single machine token outside the window. No precision or span-count statistic is reported per match type, and the paper itself observes in Sec. 4.5 that 'models tend to extract more evidence than humans annotated, often echoing the same or similar information at different positions.' A model that emits generously many tokens, or repeats the same finding at multiple locations, will therefore be scored as proximate or partial on a large fraction of cases even though its explanation would not be usable by a coder. Without a trivial high-recall baseline (e.g., top-k attribution tokens or all tokens above a low threshold) and without false-positive counts, the 80% figure and the supervised-vs-unsupervised comparison do not yet establish that the match measures capture evidence quality better than F1/IOU.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the MDACE dataset and evaluates current explainable ICD coding models. It studies where human-annotated evidence is located, how much evidence overlaps with code descriptions, and whether sufficient evidence is a subset of complete evidence. For model evaluation, it introduces match measures (empty, exact, proximate, partial, no match) and reports that the best supervised model identifies at least one correct token in roughly 80% of test cases, with supervised training producing more exact matches and fewer empty or non-matches than an unsupervised input-gradient-regularization model. The paper closes with recommendations for practitioners and researchers, including the reporting of false positives.","tokens_in":13228,"tokens_out":5700,"duration_ms":66668,"significance":"If substantiated, the paper offers a useful application-oriented evaluation protocol for evidence extraction in medical coding and a helpful characterization of the MDACE dataset. The strengths are the public code, the transparent match counts, the explicit split and seed details, and the qualitative error analysis. However, the headline overlap figure and the supervised-versus-unsupervised comparison depend on permissive match definitions and on a single selected seed per training strategy, and the paper does not yet provide a baseline or precision analysis that would show the match measures capture evidence quality better than F1/IOU. The contribution is therefore potentially valuable to the community, but the central empirical claims need additional support before the recommendations can be taken as established.","major_comments":[{"comment":"The headline 80% overlap result counts exact, proximate, and partial matches as success, but the proximate and partial definitions are very permissive. Proximate requires each ground-truth sequence to share at least one token, with unmatched tokens inside a context window of k=10; partial is triggered whenever at least one ground-truth sequence is unmatched or any machine token falls outside the window. No constraint limits the total number of machine-extracted tokens or false positives, and §4.5 itself notes that models \"tend to extract more evidence than humans annotated, often echoing the same or similar information at different positions.\" Without a trivial high-recall baseline (e.g., top-k attribution tokens or all tokens above a low threshold) and without precision or false-positive counts per match type, the 80% figure and the claim that the proposed measures are more meaningful than F1/IOU are not established; the results may reflect extraction breadth rather than evidence quality.","section":"§4.3, §4.1, §4.5"},{"comment":"The comparison between supervised and unsupervised training strategies is based on one seed per strategy, selected according to explanation metrics. Because ten model seeds are available for each strategy and Figure 8 shows nontrivial variation across seeds in F1, precision, and recall, the observed differences in exact, empty, and no-match counts cannot currently be attributed to the training strategy rather than to the selected seed or to selection effects. The recommendation that investing in evidence annotation improves exact matches would require an aggregate comparison over seeds with appropriate error bars or a paired statistical test.","section":"§4.1, §5.1, Appendix C.1"},{"comment":"The semantic relabeling of no-match cases was performed by two authors with \"basic medical knowledge,\" without inter-annotator agreement and without considering the surrounding context; the authors themselves state that expert assessments are necessary. The reported 46% and 53% \"semantically meaningful\" rates should therefore be treated as exploratory and should not carry quantitative weight in the narrative that model evidence is closer to human evidence than token-level matching suggests.","section":"§4.3"}],"minor_comments":[{"comment":"\"we selected the 52 document with common hospital admission ID\" appears to be a typo for \"52 documents.\"","section":"§3.3"},{"comment":"The figure uses the abbreviation \"prox.\" while the text defines \"proximate\"; please use the same term consistently.","section":"Figure 6"},{"comment":"The figure caption does not identify the blue/orange color scheme that the text refers to; a legend or explicit caption labels would improve readability.","section":"Figure 5"},{"comment":"The 74% agreement figure is computed for two selected seeds; the text should state explicitly that this is a seed-specific observation rather than a property of the training strategies in general.","section":"§4.4"},{"comment":"The fixed context window k=10 is acknowledged in the limitations, but a short sensitivity analysis varying k would substantially strengthen the proximate-match definition.","section":"§5, Limitations"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, honest evaluation paper, not a breakthrough, but it ships something usable — new match measures for evidence extraction in ICD coding, plus dataset analyses no one else has done. The 80% overlap figure is real but means 'at least one correct token,' so treat it as recall, not precision.\n\nWhat's actually new: the proximity-aware match taxonomy (empty/exact/proximate/partial/no match) and the RQ1–RQ3 dataset findings — evidence position distributions, code-description overlap, and the surprisingly small subset relationship between sufficient and complete annotations. The match measures are simple but could be adopted by other researchers; the code is public. They also include true positives and false negatives in evaluation, which is more transparent than prior work, and their limitations section acknowledges most of the weaknesses.\n\nSoft spots, in order of severity. First, the headline 80% is computed on exact+proximate+partial, and partial just requires any overlap; proximate requires one token per ground-truth sequence with unmatched tokens within k=10. So a model that emits lots of tokens will score well on this metric without being useful in practice. The paper notes duplicates and over-extraction in Sec. 4.5, but it never reports a precision/span-count statistic or a trivial high-recall baseline. That gap matters if someone uses these measures to compare systems. Second, the supervised vs. unsupervised comparison rests on one seed per strategy, chosen by explanation performance on the validation set — the seed-selection bias is real, though the appendix does show per-seed F1/precision/recall. Third, the manual semantic labeling of non-matches by two authors with 'basic medical knowledge' has no expert validation or inter-annotator agreement; the paper says expert assessment is needed, so this is acknowledged. Fourth, the test set is 61 documents / 586 spans, so the quantitative claims have wide error bars; the paper says this itself.\n\nDoes anything fail? The central empirical claims — that the supervised model has more exact matches and fewer empty/non-matches than the unsupervised baseline, and that match rates give a usable evaluation protocol — hold up, within the scope the authors claim. The stress-test worry about the 80% being permissive is legitimate, but the paper never calls it precision; it explicitly defines overlap as 'at least one correct token.' The real omission is not the definition, it's the absence of a baseline and false-positive counts.\n\nWho's this for: researchers working on explainable evaluation in clinical NLP, or on evidence extraction with MDACE. It's not a new architecture or benchmark, but it's a fair empirical contribution. I'd send it to a serious referee — there's enough new, reproducible content here, and the recommendations are useful even if the quantitative claims need tightening. For me: reading-group maybe, cite if I work on explanation evaluation.","headline":"A useful, honest evaluation paper with new match measures and dataset analyses; the 80% headline is recall-oriented, and the lack of a precision baseline is the main gap.","tokens_in":13763,"tokens_out":3158,"would_cite":true,"duration_ms":31757,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Proximity-aware match measures show explainable ICD models align with human evidence in about 80% of test cases.","keywords":["ICD coding","explainability","evidence extraction","MDACE dataset","match measures","plausibility evaluation","medical coding","feature attribution"],"falsifier":"Re-annotate a random sample of the 61 test discharge summaries with an independent team of professional coders under the same guidelines, then compute inter-annotator agreement between the new annotations and the MDACE spans; if that agreement is no higher than the supervised model's match rate on the same documents, the ground truth is too noisy to support the paper's comparative claims.","tokens_in":12792,"feed_emoji":"🩺","tokens_out":3623,"duration_ms":42769,"temperature":0.7,"pith_summary":"The paper argues that evaluating explainable medical coding with proximity-aware match measures—empty, exact, proximate, partial, and no match—reveals a practically useful alignment between extracted evidence and human-annotated ground truth. Using the MDACE dataset, it shows that a supervised model trained on evidence annotations identifies at least one correct token in roughly 80% of test cases, and that supervised training produces more exact matches and fewer empty or no-match cases than an unsupervised baseline. This matters because medical coding needs transparency for coders and regulators, and the paper provides an evaluation protocol that captures whether an explanation guides a human to the right context, which is more meaningful than F1 or intersection-over-union scores. The paper also analyzes where evidence appears in documents, how much it overlaps with code descriptions, and how sufficient versus complete annotations differ, yielding recommendations for developing and evaluating explainable ICD coding systems.","feed_headline":"Explanations from ICD models match human evidence in 80% of cases","feed_subtitle":"Proximity-aware scoring shows supervised training yields more exact evidence matches, guiding trust in medical coding.","key_machinery":"The central object is a taxonomy of five match measures that compare model-extracted evidence spans against ground truth spans: Empty (no evidence predicted), Exact (the sets of spans are equal), Proximate (every ground truth span has at least one matching token and all unmatched tokens fall within a context window of k=10 tokens), Partial (at least one ground truth span lacks a match or some model tokens fall outside the context window), and No match (no token overlap). This taxonomy carries the evaluation argument because it treats evidence as useful when it guides attention to the right context window, rather than requiring exact span equality as F1 and IOU do. The supporting machinery is the MDACE dataset with its sufficient and complete code-evidence annotations, the PLM-ICD-based architecture modified with standard cross-attention, and the AttInGrad feature attribution method (attention weights multiplied by input-times-gradient scores) with a threshold chosen on the validation set.","core_discovery":"The central claim is that current explainable ICD coding models, when evaluated with proximity-aware match categories instead of F1/IOU, align substantially with human-annotated evidence: the best supervised model achieves some form of match (exact, proximate, or partial) in about 80% of test cases, with relatively few empty or no-match predictions. The supervised approach, which uses annotated evidence in its training objective, yields more exact matches and fewer empty and no-match cases than an unsupervised baseline that only regularizes gradients. The paper also finds that human evidence is not systematically located at the document start, that many codes have little overlap between evidence and their official descriptions, and that the 'sufficient' (Inpatient) annotation scheme is not a subset of the 'complete' (Profee) scheme in the common documents. These findings support a recommendation for investment in evidence annotation when exact matches are desired, and for reporting false positives in future evaluation.","pith_inferences":["A natural extension of the match taxonomy would be to apply it to other evidence-extraction tasks in clinical NLP, such as procedure coding or coding in languages other than English, where the same proximity semantics could guide evaluation design.","The 80% overlap figure is dominated by partial matches; if a stricter bar requiring exact or proximate matches were used, the practical usefulness of current models would appear considerably lower, which is worth stating explicitly when comparing systems.","The paper's observation that models extract drugs and co-morbid conditions as evidence for diagnoses suggests that feature attribution methods reflect correlational rather than causal structure; regulatory acceptance of such explanations may require additional causal or counterfactual validation.","The finding that sufficient evidence is not a subset of complete evidence, while the authors attribute it to different coding rules, may also indicate that the annotation guidelines themselves need formalization before a single gold standard can be assumed for benchmark comparisons."],"forward_implications":["If the central claim holds, supervised training with evidence annotations should be the recommended strategy when deployment requires exact or near-exact evidence matches, and investing in evidence annotation is a concrete way to improve that outcome.","The proximity-aware match measures constitute a ready-to-use evaluation protocol for explainable medical coding that is more interpretable to end users than F1 or IOU, and can be adopted as a default reporting standard alongside existing metrics.","The finding that many codes have high overlap between evidence and code descriptions suggests that a subset of the ICD coding task can be solved efficiently with rule-based or hybrid approaches, freeing deep learning for the harder, low-overlap codes.","The observed relationship between short model evidence and lower recall indicates that explanation length can serve as a lightweight warning signal for likely classification errors in deployed systems.","The low probabilities associated with empty evidence predictions and the absence of a probability signal for exact matches give a concrete direction for future work on model calibration and its interaction with explanation quality."],"supporting_citations":[{"why":"Supplies the MDACE dataset, the human-annotated code-evidence spans that serve as ground truth for the plausibility evaluation.","marker":"Cheng et al. (2023)"},{"why":"Provides the fine-tuned model weights, the supervised and unsupervised training strategies including input gradient regularization, the AttInGrad explanation method, and the prior faithfulness/plausibility evaluation that this paper builds on.","marker":"Edin et al. (2024)"},{"why":"Defines the PLM-ICD architecture that, with cross-attention replacements, underlies all models evaluated in the paper.","marker":"Huang et al. (2022)"},{"why":"Establishes the ERASER benchmark and the F1/IOU plausibility metrics that the paper argues are less meaningful for end users and to which the proposed match measures are contrasted.","marker":"DeYoung et al. (2020)"},{"why":"Provides the MIMIC-III database from which the MDACE subset is drawn, establishing the clinical documentation foundation of the dataset.","marker":"Johnson et al. (2016)"}],"fun_headline_variants":["Proximity scoring shows ICD evidence matches humans in 80% of cases","Explainable ICD coding: evidence match approaches 80% with new scoring","New evaluation reveals high ICD evidence overlap with human annotations","Supervised ICD models yield more exact evidence matches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the MDACE human-annotated evidence spans are a complete and reliable gold standard, and that the 61-document test set with 586 evidence spans, together with one selected model seed per training strategy, is enough to compare supervised versus unsupervised evidence extraction.","fun_headline_variants_meta":{"raw":{"variants":["Proximity scoring shows ICD evidence matches humans in 80% of cases","Explainable ICD coding: evidence match approaches 80% with new scoring","New evaluation reveals high ICD evidence overlap with human annotations","Supervised ICD models yield more exact evidence matches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3216,"prompt_tokens":909,"completion_tokens":2307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2236}},"tokens_in":525,"tokens_out":2307,"duration_ms":20980,"temperature":1.0,"reasoning_tokens":2236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:42:03.598035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the 61 test discharge summaries with an independent team of professional coders under the same guidelines, then compute inter-annotator agreement between the new annotations and the MDACE spans; if that agreement is no higher than the supervised model's match rate on the same documents, the ground truth is too noisy to support the paper's comparative claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-III database from which the MDACE subset is drawn, establishing the clinical documentation foundation of the dataset."}],"review_version":1}