{"id":"cab24bb6-903b-4543-9ca6-99c01bcb5b74","arxiv_id":"2412.17803","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Subgroup performance in clinical ICD-10 coding tracks the similarity of the group's label distribution to the overall data, not the group's sample size.","lead":"This paper tests how imbalanced patient groups affect three clinical language models that predict diagnosis codes from hospital discharge notes. It finds that how similar a group's conditions are to the majority, not how many patients are in the group, best tracks performance differences.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a Pearson correlation over 16 overlapping subgroup partitions with the 'All' row as a deterministic anchor; this invalidates the reported p-values and the feature-similarity conclusion.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the Section V-C correlation analysis violates independence by pooling overlapping demographic partitions and including the 'All' point as if it were an independent observation. This is not a minor methodological quibble; it is the only statistical evidence for the paper's central claim that dissimilarity to the overall distribution, rather than subgroup proportion, drives performance. The reported p-values are extreme (e.g., F1-macro p=2.76e-10) but are computed from n=16 non-independent points with a deterministic anchor. Removing the 'All' row or modeling the partition structure could easily eliminate significance. The paper also selects only Clinical Longformer post hoc and does not correct for multiple comparisons across 15 metrics, further weakening the evidence. The descriptive performance tables and fairness measurements are useful, but the central causal/conclusion claim is not supported by the analysis as presented. Therefore the reader's REJECT verdict is appropriate and needs no change.","tokens_in":20174,"tokens_out":3318,"duration_ms":34705,"concrete_test":"Recompute the Table V Pearson correlations for Clinical Longformer after excluding the 'All' row, and also run a permutation test that resamples patients within each partition to account for the overlapping structure. Specifically check whether the F1-macro correlation with cosine distance remains significant at α=0.05 when 'All' is removed; if the p-value rises above 0.05 or the correlation becomes weak, the paper's headline conclusion is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline conclusion in Section V-C(a) is that model performance correlates significantly with cosine distance to the overall label distribution but not with subgroup proportion. This conclusion rests entirely on the Pearson correlations in Table V and Table VI. The 'Overall Data' analysis in Table V treats 16 rows as independent observations: All, 2 gender groups, 5 age groups, 5 race/ethnicity groups, and 3 insurance groups. These rows are not independent—they are overlapping partitions of the same test set, and the 'All' row is not an observation at all. Its cosine distance is 0 by construction and its performance equals the overall mean, so it is a deterministic anchor point that can artificially strengthen the negative correlation between distance and performance. With only n=16, the reported p-values such as F1-macro p=2.76e-10 are extremely sensitive to this one non-independent point. The paper does not report the correlation after removing 'All', does not account for the nested structure of the partitions, and does not correct for multiple testing across the many metrics in Table V. Since the paper's central claim—that feature similarity to the majority class is more critical than subgroup size—depends on the contrast between Table V and Table VI, the statistical invalidity of Table V is load-bearing. Even if the correlation survives removal of 'All', the distance used is between globally averaged label vectors, not text features, so the abstract's 'feature similarity' language goes beyond what was measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how data imbalance affects the performance and demographic fairness of clinical language models on the ICD-10 phenotype inference task using MIMIC-IV. The authors analyze label and demographic imbalances, evaluate ClinicalBERT, GatorTron, and Clinical Longformer across gender, age, race/ethnicity, and insurance subgroups, and measure fairness with an equality-difference metric. The central claim is that subgroup performance is more strongly associated with the cosine distance between the subgroup's globally averaged label vector and the overall test data than with the subgroup's proportion in the test data, which they interpret as evidence that feature similarity to the majority class is a more critical factor than subgroup size.","tokens_in":20418,"tokens_out":6114,"duration_ms":61911,"significance":"The paper addresses an important and under-studied question: how demographic and social-determinant imbalances affect clinical language model performance and fairness. Its strengths include a large, standard benchmark dataset, evaluation of three established clinical models, a broad set of performance metrics, a released codebase, and a descriptive analysis of intersectional imbalance patterns in Tables I and II. If the main comparative claim were well supported, the study would be a useful reference for the health NLP community. However, the primary statistical evidence for that claim, the Pearson correlation analysis in Section V-C, has load-bearing methodological flaws. The non-independence of the observations, the deterministic anchor of the 'All' row, post-hoc model selection, and the mismatch between 'feature similarity' and label-vector distance all undermine the central conclusion as currently stated. The descriptive findings on performance disparities and fairness remain informative, but the paper's headline claim requires substantially reworked analysis and more careful framing.","major_comments":[{"comment":"The Pearson correlations treat the 16 rows ('All', male, female, five age groups, five race/ethnicity groups, three insurance groups) as independent observations, but these are overlapping partitions of the same test set. The 'All' row is not an independent observation: its cosine distance is 0 by construction and its performance equals the overall mean, so it acts as a deterministic anchor that can manufacture or exaggerate the negative slope. The reported p-values (e.g., F1-macro p=2.76e-10) are therefore not trustworthy, and the contrast with the non-significant proportion correlations is not evidence for the paper's central claim. The analysis should account for the nested and overlapping structure, for example with a mixed-effects model or a cluster bootstrap, and should report results with the 'All' row removed.","section":"Section V-C(a), Tables V and VI"},{"comment":"The correlation analysis is conducted only for Clinical Longformer, which the authors selected after observing that it outperformed the other two models ('As previously determined... we used Clinical Longformer for this correlation analysis'). This post-hoc model selection is not accounted for, and it weakens the inferential value of the correlation results. The central claim would be more credible if the correlations were reported for all three models, or if the selection were justified a priori with an appropriate multiple-comparison adjustment.","section":"Section V-C, model selection"},{"comment":"The conclusion that 'feature similarity to the majority class may be a more critical factor' is not supported by the measurement actually used. The paper computes cosine distance between globally averaged label vectors, which is a label-distribution distance, not a feature similarity measure. In addition, the 'majority class' is never defined: the distance is taken to the overall test data, not to the majority demographic group or a majority label vector. The wording overclaims what the analysis can establish. Either the claims should be limited to label-distribution similarity, or the analysis should be extended to textual feature representations.","section":"Abstract and Section V-C(a"},{"comment":"Many significance tests are performed across multiple performance metrics and several subgroup partitions without any correction for multiple testing. Since the paper's main conclusion rests on the significance in Table V and the non-significance in Table VI, the alpha=0.05 comparisons should be adjusted or explicitly justified as exploratory. This issue compounds the non-independence problem: even if the rows were independent, the sheer number of tests would make chance findings likely.","section":"Section V-C, Tables V and VI"}],"minor_comments":[{"comment":"The equality difference definition uses ED_{g,m} on the left-hand side but sums over all groups g in G on the right-hand side, so the left-hand index is misleading. The notation should clarify that ED is computed per demographic category, not per individual group.","section":"Section V-B, Eq. (1)"},{"comment":"The caption says each value is the 'mean and margin of error of the confidence interval,' but no confidence level is stated. Please specify the confidence level or report standard deviations/standard errors.","section":"Table II caption"},{"comment":"The notation around x_i and y is confusing: x_i is described as a vector of performance metrics across groups, while y is a vector of distances or proportions. Please define both variables explicitly and state which quantity corresponds to which axis in Tables V and VI.","section":"Section V-C, Pearson correlation formula"},{"comment":"The label 'Overall Data' for the first analysis block is misleading, since the rows include all partitions plus the 'All' row. Consider renaming it to something like 'All partitions' or 'All groups combined' to avoid implying it is a single overall test set.","section":"Tables V and VI"},{"comment":"The conclusion repeats the 'feature similarity' phrasing without acknowledging that the analysis measured label-vector distance rather than text or feature similarity. The terminology should be aligned with the actual method or the method should be extended.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper's statistical core is not yet sound, but the flaws are fixable in principle: the authors could redo the correlation analysis with appropriate non-independence-aware methods, report all three models, and align their claims with the label-distance measure they actually use. If the authors can provide a valid statistical analysis and adjust the 'feature similarity' wording, the paper could become a useful empirical contribution. As it stands, the central comparative claim is unsupported, so I recommend major revision rather than acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look for its descriptive content, but the headline conclusion does not survive contact with the statistics. The authors evaluate three clinical language models on ICD-10 prediction across demographic and SDoH subgroups in MIMIC-IV, and the performance tables are genuinely informative: the youngest age group's odd accuracy/F1 pattern, the 90+ group doing well despite tiny representation, and the insurance-by-ethnicity breakdowns are all useful observations. The intersectional analysis is the first I've seen that separates group proportion from label-distribution distance in this setting. Code is released, preprocessing is documented, and the limitations section is honest about single-dataset scope.\n\nThe soft spot is Section V-C, and it is load-bearing. The correlation between subgroup performance and cosine distance to the overall label vector treats 16 rows as independent observations when they are heavily overlapping partitions of one test set, plus the 'All' row, which has distance 0 and overall-mean performance by construction. That deterministic anchor can manufacture a significant negative slope, and with n=16 the reported p-values like 2.76e-10 are not believable without an independence argument. The paper never reports the correlation with 'All' removed, and it does not correct for multiple testing across the many metrics. On top of that, the analysis is run only on Clinical Longformer, chosen because it performed best, which is post hoc selection. And the abstract's 'feature similarity' language overreaches: the cosine distance is computed on averaged label vectors, not on text features. The paper's own limitation section acknowledges the cosine-distance choice but not the independence problem.\n\nNone of this kills the descriptive value. The performance disparities and the ED fairness numbers are real measurements. But the central claim—that feature similarity to the majority matters more than subgroup size—is not established by the current analysis. It is an interesting hypothesis that deserves a proper test.\n\nI would send this to peer review, not desk-reject it, because the dataset work and the descriptive findings are useful and the flaw is fixable. The authors should redo the correlation with non-overlapping groups, exclude the 'All' anchor, correct for multiple testing, and report all three models. If the correlation survives that, fine; if not, the claim should be downgraded to an observation. As it stands, I would not cite the headline result, but I would cite the descriptive tables and would bring the paper to a reading group as a cautionary example of correlation pitfalls in fairness analysis.","headline":"A useful descriptive study of demographic imbalance in ICD coding whose headline claim rests on a statistically shaky correlation; the central finding needs a redo.","tokens_in":20959,"tokens_out":2395,"would_cite":false,"duration_ms":25069,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For clinical ICD coding, similarity to the majority class, not group size, predicts performance","keywords":["data imbalance","clinical language models","ICD code prediction","demographic fairness","cosine distance","equality difference","MIMIC-IV","long-tailed labels"],"falsifier":"Recompute the Pearson correlation between cosine distance and F1-macro on the five non-overlapping age bands alone, dropping the 'All' row; if the negative correlation loses significance, the paper's central claim does not survive.","tokens_in":19911,"feed_emoji":"🏥","tokens_out":7656,"duration_ms":66010,"temperature":0.7,"pith_summary":"The paper asks why clinical language models perform unevenly across demographic and insurance groups when predicting ICD-10 codes from hospital discharge notes. It argues that data imbalance matters, but the dominant driver is not how many patients a subgroup contains; it is how far that subgroup's typical label pattern sits from the rest of the dataset. Using three clinical models on a large intensive-care corpus, the authors show that subgroup F1, precision, recall, and AUC all fall as the cosine distance between the subgroup's average label vector and the overall test set's average label vector rises, while subgroup size shows no significant correlation. They conclude that models overfit majority feature patterns, so fairness work should target feature similarity, and that accuracy is an unreliable metric under long-tailed ICD labels.","feed_headline":"Similarity to the majority class, not size, drives model gaps","feed_subtitle":"Subgroup F1 scores track label-vector closeness to the whole test set; group size shows no link.","key_machinery":"The machine doing the work is a two-step comparison. First, each demographic, age, insurance, and intersectional subgroup is summarized by a single vector: the average, over all patients in the subgroup, of the 100-dimensional ICD-10 label vector. The paper then measures dissimilarity as the cosine distance between that subgroup-average vector and the average vector of the entire test set, producing one number per subgroup. Second, for each performance metric it computes the Pearson correlation coefficient across subgroups between those cosine distances and the subgroup's metric value, testing the null hypothesis of no correlation at a 0.05 significance level. The fairness side uses Equality Difference, the summed absolute deviation of a group's performance from the overall performance, to rank which demographic splits are least fair (age is worst). These two tools—cosine distance between label vectors, and Pearson r—carry the central argument.","core_discovery":"On the paper's own terms, the central discovery is a correlation result: for the top-100 ICD-10 phenotype prediction task, the F1-macro score of Clinical Longformer across all demographic, age, insurance, and intersectional subgroups is strongly negatively correlated with the cosine distance between the subgroup's globally averaged label vector and the whole test set's averaged label vector (r = -0.973, p = 2.76e-10 for F1-macro on the overall grouping), while no such correlation appears with the subgroup's proportion of the test data. The same pattern holds for micro and weighted F1, precision, recall, and AUC, and it is consistent with the descriptive performance tables for ClinicalBERT and GatorTron. The paper reads this as evidence that data imbalance affects performance and fairness chiefly through feature similarity to the majority class: subgroups whose health-condition signature resembles the bulk of the data are predicted well even if small, and subgroups with dissimilar signatures are predicted poorly even if reasonably represented. It also documents that accuracy behaves oppositely, rising with dissimilarity, which it attributes to the exact-match nature of accuracy on long-tailed multi-label data.","pith_inferences":["The paper's correlation is computed across overlapping partitions with an 'All' anchor point; a direct robustness check would rerun the correlations on non-overlapping subgroups only, such as the five age bands alone, to see whether the negative slope survives without the anchor.","A natural extension the paper does not run: train the same Clinical Longformer on data reweighted to match the majority label distribution and test whether subgroup F1 gaps shrink, which would experimentally separate the dissimilarity effect from the sample-size effect.","If the mechanism is genuine, the same cosine-distance analysis should predict performance gaps in other long-tailed clinical tasks, such as mortality or readmission prediction; that is a testable prediction beyond this paper.","The positive accuracy–dissimilarity correlation implies that dashboards reporting only accuracy for clinical models can hide severe subgroup underperformance, a practical warning for deployment."],"forward_implications":["Subgroup performance in ICD-code prediction is explained more by how different a subgroup's label distribution is from the whole dataset than by that subgroup's share of the data.","Accuracy should be used with caution in long-tailed clinical multi-label settings, since it can rise for dissimilar subgroups even as recall and F1 fall, masking real underperformance.","Fairness audits should report equality differences and correlation with dissimilarity, not just subgroup proportions, to detect the kind of disparity documented here.","Clinical Longformer's longer input window gives it an overall performance edge, but the dissimilarity-driven performance gap persists across all three models, suggesting the pattern is not an artifact of a single architecture."],"supporting_citations":[{"why":"Supplies the clinical notes corpus (MIMIC-IV Notes v2.2) whose discharge summaries are the task input.","marker":"[12]"},{"why":"Supplies patient demographics, insurance, and diagnoses from MIMIC-IV used to define subgroups and labels.","marker":"[2]"},{"why":"Clinical Longformer, the best-performing model on which the central correlation analysis is run.","marker":"[15]"},{"why":"ClinicalBERT, one of the three models compared in the performance and fairness tables.","marker":"[13]"},{"why":"GatorTron, the third model used in the comparison, providing a replication of the performance pattern.","marker":"[14]"},{"why":"Defines the Equality Difference fairness measure used to quantify disparity across groups.","marker":"[32]"}],"fun_headline_variants":["Similarity to majority class, not size, drives subgroup F1 gaps","Cosine distance to majority class explains fairness gaps, not group size","For clinical LMs, subgroup F1 tracks similarity to majority, not size","Label-vector closeness to majority, not data size, drives subgroup performance","Subgroup F1 correlates with majority-class similarity, not group size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The correlation analysis treats the sixteen subgroup rows as independent measurements even though they are overlapping slices of one test set, with the 'All' row at zero distance and average performance anchoring the regression; if those points are not independent, the reported p-values do not establish the conclusion.","fun_headline_variants_meta":{"raw":{"variants":["Similarity to majority class, not size, drives subgroup F1 gaps","Cosine distance to majority class explains fairness gaps, not group size","For clinical LMs, subgroup F1 tracks similarity to majority, not size","Label-vector closeness to majority, not data size, drives subgroup performance","Subgroup F1 correlates with majority-class similarity, not group size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000727,"raw_usage":{"total_tokens":3251,"prompt_tokens":930,"completion_tokens":2321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2241}},"tokens_in":546,"tokens_out":2321,"duration_ms":16906,"temperature":1.0,"reasoning_tokens":2241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:08:30.293271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the Pearson correlation between cosine distance and F1-macro on the five non-overlapping age bands alone, dropping the 'All' row; if the negative correlation loses significance, the paper's central claim does not survive.","supporting_citations":[{"cited_title":"Mimic-iv-note: Deidentified free-text clinical notes (version 2.2),","cited_arxiv_id":null,"evidence_quote":"Supplies the clinical notes corpus (MIMIC-IV Notes v2.2) whose discharge summaries are the task input."},{"cited_title":"Clinical- longformer and clinical-bigbird: Transformers for long clinical se- quences,","cited_arxiv_id":null,"evidence_quote":"Clinical Longformer, the best-performing model on which the central correlation analysis is run."},{"cited_title":"Publicly available clinical BERT embeddings,","cited_arxiv_id":null,"evidence_quote":"ClinicalBERT, one of the three models compared in the performance and fairness tables."},{"cited_title":"Measuring and mitigating unintended bias in text classification,","cited_arxiv_id":null,"evidence_quote":"Defines the Equality Difference fairness measure used to quantify disparity across groups."}],"review_version":1}