{"id":"7bde68de-788b-4b28-b63b-4b6f3ee6afbf","arxiv_id":"2505.00592","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-expert distillation method with feature alignment and uncertainty weighting improves imbalanced grading on two medical datasets, but its state-of-the-art claim only holds for selected metrics in one scenario.","lead":"This paper combines two ways of teaching a small model from larger expert models: matching image features and matching predictions, with experts trusted less when they are uncertain. Tests on prostate and retina grading show gains over several baselines, though the claimed top performance is partly contradicted by the paper's own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"UDD uncertainty weighting is not validated as the cause of gains; a control fixing Eq. (7) weights would settle it.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I identify: U_T is asserted, not demonstrated, to capture imbalance-induced teacher unreliability, and no experiment isolates the uncertainty weighting from the rest of UDD or from generic spatial reweighting. That is the weakest link in the central claim because the method's novelty and its claimed advantage over plain decoupled distillation both ride on Eq. (7). The SOTA claim is also overstated in target-imbalanced APTOS (RKD beats UMKD on OA and F1), but that is a claim-scoping issue that is correctable by revision and does not by itself sink the method. The proposed control experiment is cheap, does not require code or new data, and would settle whether U_T specifically is responsible for the gains. I agree with CONDITIONAL: the method may well work, but the mechanism is unvalidated and the headline claim is overbroad. I would not downgrade to REJECT because the full UMKD pipeline does outperform baselines on SICAPv2 across all metrics and on APTOS on mAcc, which is the metric most indicative of minority-class performance.","tokens_in":8373,"tokens_out":1808,"duration_ms":15720,"concrete_test":"Run an ablation on SICAPv2 that keeps SFA+CFA+UDD's spatial partitioning but replaces Eq. (7) weights with (a) fixed weights (2+U0, 1-U0) for a constant U0 = 0.3, (b) random per-cell weights with the same mean, and (c) U_T computed from the student's own softmax instead of the teacher's. If all controls match or beat the reported UMKD numbers (OA 91.02 / 91.75), the uncertainty mechanism is not doing the work attributed to it; if the teacher-uncertainty version is clearly better across OA, mAcc, and F1, the mechanism survives. Also recompute Table 2 target-imbalanced row to explicitly mark RKD's higher OA and F1, and restate the SOTA claim per metric.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central modeling mechanism, UDD in Eq. (7), weights TCKD by (2+U_T) and NCKD by (1-U_T), where U_T = 1 - max(softmax(teacher logits)) is claimed to quantify expert bias from class imbalance. The paper never shows that this softmax-concentration score tracks actual teacher unreliability, minority-class error, or per-region misclassification; support is only indirect end-task accuracy. A simpler hypothesis fits the data equally well: UDD merely reweights the two DKD terms on a per-cell basis, and the gain comes from any spatially varying reweighting or from the logit-map partitioning itself, not from U_T specifically. The ablation in Table 3 only removes UDD entirely, so it cannot distinguish 'uncertainty weighting works' from 'any dynamic weighting of TCKD/NCKD works' or 'spatial partition helps'. The classification loss L_cls also trains the student on the same imbalanced labels, so it is unclear how much of the bias mitigation is due to UDD versus the student's own supervised training. Without a control that fixes the functional form of Eq. (7) (e.g., constant weights, random weights, or U_T computed from the student instead of the teacher), the paper's mechanistic claim that uncertainty-aware weighting reduces imbalance-induced bias is unsubstantiated. The authors even overclaim in Table 2's target-imbalanced APTOS results: RKD has higher OA and F1 than UMKD, so 'new state-of-the-art' holds only for mAcc/MAE and is contradicted by the abstract's sweeping claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UMKD, a multi-expert knowledge distillation framework for imbalanced disease image grading, combining two feature-space alignment modules (SFA in the frequency domain and CFA in a spherical space) with an output-space uncertainty-aware decoupled distillation (UDD) loss. The UDD loss reweights DKD-style TCKD and NCKD terms per spatial cell using a teacher softmax-concentration uncertainty coefficient U_T, and the total loss adds a classification loss with weighting hyperparameters alpha and beta. The method is evaluated on SICAPv2 prostate grading and APTOS fundus grading under two scenarios: source-imbalanced distillation (experts trained on imbalanced data, distillation on balanced data) and target-imbalanced distillation (experts trained on balanced data, distillation on imbalanced data), reporting OA, mAcc, F1, and MAE against several KD baselines plus an ablation study on SICAPv2.","tokens_in":8714,"tokens_out":2622,"duration_ms":26089,"significance":"If the central claims held, the paper would make a useful contribution to an important practical problem: enabling a compact student model to inherit multi-expert knowledge while mitigating class-imbalance bias in medical image grading. The use of public datasets, external baselines, and ablation components is a strength, and the reported SICAPv2 gains are large. However, the current evidence does not fully support the abstract's state-of-the-art claim, and the paper's central mechanism—uncertainty-aware weighting—is not directly validated as the cause of the observed gains. The missing control experiments and the inconsistency in the 'balanced' subset sizes are load-bearing issues for the paper's main narrative.","major_comments":[{"comment":"The abstract and Section 3.2 claim 'a new state-of-the-art in both source-imbalanced and target-imbalanced scenarios,' but Table 2 shows that in the target-imbalanced APTOS task RKD achieves higher OA (85.00 vs. 83.91) and higher F1 (84.38 vs. 84.03) than UMKD. The claim therefore holds only for mAcc and MAE, and the 'state-of-the-art' statement is overstated as written. Please either revise the claim to be metric-specific or provide a clearly defined aggregated criterion under which UMKD is best.","section":"Abstract and Table 2 (APTOS target-imbalanced)"},{"comment":"The core novelty is the UDD weighting in Eq. (7), where U_T = 1 - max(softmax(teacher logits)) is asserted to quantify expert bias from class imbalance and to justify the weights (2+U_T) and (1-U_T). The paper provides no evidence that this softmax-concentration score tracks per-region teacher unreliability or minority-class error; the only support is end-task accuracy. The ablation in Table 3 removes UDD entirely, so it cannot distinguish 'uncertainty-aware weighting helps' from 'any spatially varying reweighting of TCKD/NCKD helps' or 'the logit-map partitioning helps.' To substantiate the mechanistic claim, please add controls that fix the functional form, e.g., constant weights, random per-cell weights, U_T computed from the student instead of the teacher, and, ideally, a direct analysis correlating U_T with teacher misclassification on minority classes.","section":"§2, Eq. (7) and §3.3, Table 3 (UDD mechanism)"},{"comment":"The text says the authors 'generate balanced subsets from original imbalanced datasets' and lists SICAPv2-balanced (2500, 2222, 2500, 948) and APTOS-balanced (600, 370, 300, 193, 295). These class counts are not equal across classes, so the datasets are not class-balanced as the term is normally used. This ambiguity affects the interpretation of both the source-imbalanced and target-imbalanced setups. Please clarify the exact class distribution used in each scenario, or relabel the subsets with their true counts.","section":"§3.1, Implementation details (balanced subsets)"},{"comment":"The text states that the reconstruction loss L_MSE is used to 'ensure that the expert models remain unchanged due to privacy constraints,' but Eq. (2) defines L_MSE as a loss that measures the change between original and decoded expert features. If the expert models are frozen and unchanged, it is unclear why a reconstruction loss on teacher features is needed or what it constrains. Please clarify whether the experts are trainable in the alignment modules, and how the reconstruction loss is computed without modifying the experts.","section":"§2, Eq. (2) and surrounding text"}],"minor_comments":[{"comment":"The phrase 'As shown in 7-th row' is unclear; the row numbers are not labeled in Table 2, and the sentence would be clearer with the method name (RKD) and metric values stated directly.","section":"§3.2, Results on APTOS Grading"},{"comment":"The ablation study is reported only on SICAPv2, with APTOS results deferred to 'space limitations.' Since APTOS is the dataset with the strongest imbalance and where the target-imbalanced claim is weakest, providing the APTOS ablation table in a supplement would substantially strengthen the paper.","section":"§3.3, Ablation Study"},{"comment":"None of the tables report standard deviations, confidence intervals, or statistical significance tests. Given that many of the reported gaps between methods are small (e.g., below 1 point), adding variance estimates from multiple runs would materially improve the reliability of the comparison.","section":"Tables 1 and 2"},{"comment":"The phrase 'sources-imbalanced' is used inconsistently ('sources-imbalanced' versus 'source-imbalanced'); please standardize the terminology. Also, the caption of Fig. 1 describes 'sources-imbalanced' and 'target-imbalanced KD tasks' but the plot itself is not fully explained in the text.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things before you read it: the abstract says SOTA in both source- and target-imbalanced scenarios, but Table 2's target-imbalanced APTOS row shows RKD with higher OA (85.00 vs 83.91) and F1 (84.38 vs 84.03). So the headline claim is narrower than stated. And the core mechanism, UDD's uncertainty weighting, is not actually validated as the active ingredient; a control with fixed weights would be needed to show it's the softmax-confidence term doing the work.\n\nWhat's genuinely new: the combination of multi-scale low-pass feature alignment (from the one-for-all lineage), MMD projection into a spherical space, and an uncertainty-weighted decoupled distillation. Individually these are known, but the package is not, and the SICAPv2 results support it: UMKD beats SDD, DKD, and KD on all four metrics in both tasks, and the ablation shows SFA and CFA each contribute. The writing is clear, the medical-imbalance motivation is real, and the authors honestly acknowledge the RKD trade-off on APTOS.\n\nSoft spots, in order: first, the SOTA overclaim. Second, the 'balanced subset' description is internally inconsistent: SICAPv2-balanced lists (2500, 2222, 2500, 948), which are not balanced class counts; same for APTOS (600, 370, 300, 193, 295). Either the data prep is wrong or the text is, and it matters for interpreting the target-imbalanced experiments. Third, no error bars or significance tests anywhere; I'd treat all the deltas as point estimates. Fourth, the UDD mechanism: U_T = 1 - max(softmax) is asserted to capture imbalance-induced bias, but there's no evidence it correlates with teacher error, and the ablation only removes UDD as a whole. A fixed-weight or random-weight control would tell us whether the uncertainty term is actually the cause or just the spatial partitioning.\n\nNone of these are fatal. The method is plausible and the SICAPv2 gains are real. A serious referee should engage with it, mainly to force the authors to fix the claims and run the control. I'd accept it for peer review.","headline":"A decent multi-expert KD method for imbalanced disease grading, but the abstract overclaims SOTA and the uncertainty-weighting mechanism is not yet proven to be the cause of the gains.","tokens_in":9209,"tokens_out":4012,"would_cite":false,"duration_ms":35409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"UMKD distills several expert models into one compact student and reports the best mean accuracy on imbalanced disease grading.","keywords":["knowledge distillation","multi-expert distillation","disease image grading","class imbalance","uncertainty-aware distillation","diabetic retinopathy grading","prostate cancer grading","feature alignment"],"falsifier":"Create a validation set with a known rare class, record the teacher's $U_{T_t}$ per image region and its per-region misclassification, and check whether high uncertainty concentrates where errors occur; alternatively, run UMKD with random or constant weights in place of $(2+U_{T_t})$ and $(1-U_{T_t})$ and compare accuracy. If uncertainty does not track errors, or the random-weight control matches UMKD, the uncertainty mechanism is not the source of the gains.","tokens_in":8175,"feed_emoji":"🩺","tokens_out":9507,"duration_ms":92015,"temperature":0.7,"pith_summary":"The paper proposes UMKD, a multi-expert knowledge-distillation framework that compresses several expert models into one compact student for disease-image grading under class imbalance. Its central claim is that a student trained by UMKD reaches the best published trade-off among distillation baselines in both source-imbalanced and target-imbalanced settings on prostate histology (SICAPv2) and diabetic-retinopathy fundus (APTOS) grading. The key move is to treat teacher uncertainty as a signal for what to transfer: where a teacher's local prediction is ambiguous, the student is pushed to match the teacher's class distribution harder; where the teacher is confident, the student preserves exact logit alignment. On target-imbalanced APTOS the edge is metric-dependent, since a feature-based baseline keeps higher overall accuracy and F1 while UMKD leads in mean accuracy. If correct, this gives a practical recipe for deploying a small model in clinical settings without losing minority-class performance to the bias that imbalance injects into teachers.","feed_headline":"Uncertainty weighting distills smarter disease-grading models","feed_subtitle":"UMKD weighs teacher uncertainty to beat KD baselines on prostate and fundus grading.","key_machinery":"The load-bearing mechanism is the uncertainty-aware decoupled distillation (UDD) loss of Eq. (7): for each teacher $T_t$, scale $w$, and spatial cell $n$, the loss is $L_{\\mathrm{UDD}}(w,n) = (2+U_{T_t}) L_{\\mathrm{TCKD}} + (1-U_{T_t}) L_{\\mathrm{NCKD}}$, where $U_{T_t} = 1 - \\max(\\sigma(\\psi_{T_t}(w,n)))$ is the teacher's local ambiguity measured by how far its softmax output is from a one-hot vector. The intuition the paper relies on is that imbalance makes teachers unreliable precisely where their softmax is flat, so the student should intensify learning on those cells via $L_{\\mathrm{TCKD}}$ and relax exact-logit matching via $L_{\\mathrm{NCKD}}$. Two feature-space modules carry the rest: shallow feature alignment (SFA) uses multi-scale low-pass filtering to align task-agnostic structural features, and compact feature alignment (CFA) projects penultimate-layer features into a shared spherical space to align task-specific features; both are trained with maximum mean discrepancy plus a reconstruction loss that keeps the frozen experts unchanged. Together they are meant to handle architectural heterogeneity and source/target distribution mismatch while UDD handles imbalance bias.","core_discovery":"The paper's central discovery is that class-imbalance bias in multi-expert distillation can be countered by a per-region, uncertainty-aware reweighting of decoupled knowledge. In the output space, UDD partitions each teacher's logit map into multi-scale cells, defines $U_{T_t} = 1 - \\max(\\sigma(\\psi_{T_t}(w,n)))$, and scales the target-class distillation term $L_{\\mathrm{TCKD}}$ by $(2+U_{T_t})$ while scaling the non-target term $L_{\\mathrm{NCKD}}$ by $(1-U_{T_t})$; ambiguous cells therefore dominate training, while confident cells keep the fine-grained logit alignment that decoupled distillation is designed to protect. In feature space, shallow feature alignment (SFA) suppresses high-frequency details with multi-scale low-pass filtering to align structural, task-agnostic features, and compact feature alignment (CFA) maps penultimate-layer features into a shared spherical space to align task-specific semantic features; both use maximum mean discrepancy plus a reconstruction loss. The paper reports that this combination beats all compared baselines on SICAPv2 and APTOS for the source-imbalanced task, and on APTOS target-imbalanced it reports the best mean accuracy while noting that feature-based RKD still has higher overall accuracy and F1. On SICAPv2 target-imbalanced it reports the best performance on all four metrics.","pith_inferences":["The authors do not test whether $U_{T_t}$ tracks actual teacher errors; a direct diagnostic would be to compare per-patch softmax uncertainty against per-patch misclassification on an artificially imbalanced validation set, and a calibrated confidence measure would be a natural upgrade if the correlation is weak.","The same $(2+U)/(1-U)$ reweighting idea transfers to other teacher ensembles, such as federated medical models or temporally averaged teachers, wherever domain shift creates spatially local ambiguity; nothing in the mechanism is specific to disease grading.","Because feature-based RKD and UMKD lead on complementary metrics in target-imbalanced APTOS, combining the uncertainty reweighting with relational knowledge distillation is a testable hybrid that might keep both high overall accuracy and high mean accuracy.","Before relying on the state-of-the-art claim, a reader should fix one primary metric per task and recompute the comparison tables, since no single method dominates every metric in both datasets."],"forward_implications":["A compact student can replace a larger expert ensemble: across the paper's tables, UMKD's ResNet18 student hits or exceeds the per-expert ResNet50 results on the imbalanced tasks.","Uncertainty-aware weighting gives a rule for combining many teachers: instead of averaging all outputs, the student can trust each teacher where it is confident and learn class structure from it where it is uncertain.","Source-imbalanced distillation, where the teachers themselves learned from skewed data, is the harder setting and is where the UDD weighting shows its largest margin over SDD on SICAPv2.","The framework transfers across heterogeneous architectures without changing the experts, because SFA and CFA align features in normalized common spaces and the reconstruction loss keeps expert weights fixed, a property the paper ties to privacy constraints.","Reported gains are metric-dependent in one cell of the tables: on target-imbalanced APTOS, UMKD leads in mean accuracy and MAE, while RKD leads in overall accuracy and F1."],"supporting_citations":[{"why":"supplies the TCKD/NCKD decomposition that UDD reweights with uncertainty.","marker":"[23]"},{"why":"the most recent logits-based distillation baseline that UMKD is compared against and whose scaled decoupling it extends.","marker":"[21]"},{"why":"a feature-based hint distillation baseline and the ancestor of the fitnet-style feature alignment used in SFA and CFA.","marker":"[18]"},{"why":"the relational feature-based baseline that remains the strongest competitor on target-imbalanced APTOS, setting the comparison the paper must address.","marker":"[16]"},{"why":"the standard logits distillation baseline forming the common denominator of all logits-based comparisons.","marker":"[11]"},{"why":"supplies the maximum mean discrepancy and reconstruction losses used by both feature alignment modules.","marker":"[5]"},{"why":"motivates aligning heterogeneous expert and student features across architectures, which SFA adapts to the shallow layers.","marker":"[9]"},{"why":"provides the SICAPv2 prostate histology dataset used for source-imbalanced and target-imbalanced grading experiments.","marker":"[19]"},{"why":"provides the APTOS diabetic-retinopathy fundus dataset used for the second set of grading experiments.","marker":"[12]"}],"fun_headline_variants":["Uncertainty-aware weights fix imbalanced disease grading","Weigh teacher uncertainty to fix imbalanced disease grading","Uncertainty decoupling boosts imbalanced disease grading","Multi-expert uncertainty weighting stabilizes disease grading"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a teacher's flat softmax output, measured as one minus its top probability, really tracks the bias that class imbalance injects into that teacher's local predictions, so that weighting the distillation loss by this number is what improves the student.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty-aware weights fix imbalanced disease grading","Weigh teacher uncertainty to fix imbalanced disease grading","Uncertainty decoupling boosts imbalanced disease grading","Multi-expert uncertainty weighting stabilizes disease grading"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001237,"raw_usage":{"total_tokens":5137,"prompt_tokens":1062,"completion_tokens":4075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":4013}},"tokens_in":678,"tokens_out":4075,"duration_ms":30776,"temperature":1.0,"reasoning_tokens":4013,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:39:01.296710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Create a validation set with a known rare class, record the teacher's $U_{T_t}$ per image region and its per-region misclassification, and check whether high uncertainty concentrates where errors occur; alternatively, run UMKD with random or constant weights in place of $(2+U_{T_t})$ and $(1-U_{T_t})$ and compare accuracy. If uncertainty does not track errors, or the random-weight control matches UMKD, the uncertainty mechanism is not the source of the gains.","supporting_citations":[{"cited_title":"Information Sciences 669, 120564 (2024)","cited_arxiv_id":null,"evidence_quote":"supplies the maximum mean discrepancy and reconstruction losses used by both feature alignment modules."},{"cited_title":"Advances in Neural Information Processing Systems 36 (2024)","cited_arxiv_id":null,"evidence_quote":"motivates aligning heterogeneous expert and student features across architectures, which SFA adapts to the shallow layers."},{"cited_title":"Computer methods and programs in biomedicine 195, 105637 (2020)","cited_arxiv_id":null,"evidence_quote":"provides the SICAPv2 prostate histology dataset used for source-imbalanced and target-imbalanced grading experiments."},{"cited_title":"https://kaggle.com/ competitions/aptos2019-blindness-detection (2019), kaggle","cited_arxiv_id":null,"evidence_quote":"provides the APTOS diabetic-retinopathy fundus dataset used for the second set of grading experiments."}],"review_version":1}