{"id":"fb21cb14-e91b-4683-bef1-d75c89905666","arxiv_id":"2412.08139","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Wasserstein-distance-based logit and feature distillation losses outperform KL-divergence-based counterparts on image classification and object detection.","lead":"The authors propose replacing Kullback-Leibler divergence with Wasserstein distance for knowledge distillation, both on final logits and on intermediate features. On ImageNet, CIFAR-100, and MS-COCO, their student models outperform KL-divergence-based distillation baselines by small but consistent margins.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: WKD-L gains over KL-Div are confounded by the added CKA interrelation cost; plain WD with a uniform cost matrix is never evaluated.","rationale":"The reader accepted the paper with moderate confidence and identified the Gaussian assumption in WKD-F as the weakest assumption. I disagree that this is the most load-bearing concern. Even if deep features are not Gaussian, the diagonal-covariance WKD-F loss reduces to matching per-channel means and variances, which is a standard and well-defined moment-matching objective; the geometric interpretation is a motivation, not a logical requirement. The more serious issue is in WKD-L, where the comparison against KL-Div baselines is not controlled. The paper's headline claim is about the distance metric itself, yet every WD experiment includes a teacher-derived CKA cost matrix that the KL baselines lack. This is not an internal inconsistency, but it is a missing control that determines whether the central claim is true. The paper does many things well: extensive experiments on ImageNet, CIFAR-100, COCO, ablations of hyperparameters, and comparisons to strong baselines. The code release is promised. But for the logit-distillation claim, the reported improvements are equally compatible with the hypothesis that 'adding inter-class similarity information helps' and the hypothesis that 'Wasserstein distance is a better loss than KL divergence.' A single uniform-cost WD ablation would adjudicate. Without it, the acceptance should be conditional on providing that control or on softening the central claim. I therefore recommend CONDITIONAL rather than outright ACCEPT or REJECT.","tokens_in":26720,"tokens_out":4228,"duration_ms":47111,"concrete_test":"Run the WKD-L ablation on ImageNet (ResNet34→ResNet18, same hyperparameters as Section C.2) replacing the CKA-derived cost matrix in Eq. (3) with a uniform cost, c_ij = 0 if i=j and c_ij = 1 otherwise. Compare top-1 accuracy to the KL-Div baseline (71.03) and to CKA-cost WKD-L (72.49). If uniform-cost WD does not substantially outperform KL-Div, the claimed advantage of the Wasserstein metric is not established. A complementary control would incorporate the same CKA cost into a KL-style loss (e.g., sum_{i,j} p_i c_ij log(p_i/p_j)) to determine whether the IR information alone accounts for the gain.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim that Wasserstein distance outperforms KL-Divergence for logit distillation is not isolated from a key confound. WKD-L (Eqs. 3–5) augments discrete WD with a cost matrix c_ij = 1 − exp(−κ(1 − IR_T(C_i,C_j))) derived from teacher features via CKA. All KL-Div baselines (KD, DKD, NKD, WTTM) use no such inter-class cost information. Consequently, Table 2a's 'WD versus KL-Div' comparisons conflate two variables: the distance metric (WD vs KL) and the presence of teacher-derived category interrelations. The ablation in Table 2b shows that all IR modelings (CKA linear/poly/RBF, cosine with classifier weights or centroids) improve over KL-Div, but it never tests WD with a uniform cost (c_ij = 0 for i=j, 1 otherwise) or KL augmented with the same IR cost. Without such a control, the reported gains (72.04 vs 71.03 without target separation; 72.49 vs 71.70/71.96 with separation) could be entirely due to the IR information, not to the optimal-transport mechanism that motivates the paper. This directly weakens the strongest conceptual contribution: that WD's cross-category comparison, rather than the injected inter-class similarities, is what rivals KL-Div. The Gaussian assumption flagged in the reader's verdict is secondary, since for diagonal covariances WKD-F reduces to mean and variance matching, a well-defined loss even if features are not Gaussian.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Wasserstein-distance-based knowledge distillation in two forms: WKD-L for logits, using entropy-regularized discrete optimal transport with a teacher-derived category-interrelation cost matrix; and WKD-F for intermediate features, modeling per-image feature distributions as Gaussians and minimizing their continuous Wasserstein distance. The authors evaluate WKD on ImageNet and CIFAR-100 image classification, self-knowledge distillation, and MS-COCO object detection, reporting consistent improvements over KL-divergence-based baselines and several state-of-the-art distillation methods. The manuscript includes extensive ablations of the cost construction, distribution modeling, matching strategy, distillation position, hyperparameters, and runtime.","tokens_in":27052,"tokens_out":5154,"duration_ms":59035,"significance":"If the central claim holds, the paper makes a useful empirical contribution by showing that optimal-transport-based losses are a practical and efficient alternative to KL-based distillation, with gains on standard benchmarks and across heterogeneous architectures. The strengths of the paper are its breadth of evaluation (ImageNet, CIFAR-100, COCO, self-KD), the inclusion of multiple ablations, the public code release, and the careful reporting of latency and complexity. The main weakness is that the headline logit-distillation comparison conflates the choice of divergence with the injection of category-interrelation information, so the paper's conceptual claim is not yet isolated.","major_comments":[{"comment":"The central claim that Wasserstein distance rivals KL divergence for logit distillation is not tested independently of the added category-interrelation (IR) cost. WKD-L differs from KD, DKD, and NKD in two ways: it replaces KL with entropy-regularized WD, and it inserts a teacher-derived cost c_ij = 1 - exp(-kappa(1 - IR_T(C_i,C_j))) into the transport problem. The reported gains (72.04 vs 71.03 without target separation; 72.49 vs 71.70/71.96 with separation) could therefore come entirely from the IR information rather than from the optimal-transport cross-category mechanism. Table 2b varies only the IR model inside WD; it never evaluates WD with a uniform cost matrix (e.g., c_ij = 0 for i=j and 1 otherwise) nor a KL-based loss augmented with the same pairwise IR cost. Please add both control experiments. If uniform-cost WD still outperforms KD, the title claim is supported; if not, the paper should reframe WKD-L as a method that combines WD with category interrelations rather than as evidence that the WD metric itself is superior to KL.","section":"§2.1, Eqs. (3)–(5); Tables 2a–2b"},{"comment":"The geometric interpretation of WKD-F relies on the assumption that deep features are Gaussian, an assumption the authors themselves flag as an open problem in Section F. The paper should add an explicit robustness discussion noting that with diagonal covariances the loss reduces to weighted mean and variance matching, which is a well-defined moment-matching objective even when the Gaussian assumption is violated. A concrete stress test would be to compare WKD-F against a pure moment-matching baseline with the same mean and variance terms but without the Wasserstein framing; if the performance is identical, the Riemannian-metric interpretation is not doing the explanatory work claimed in the text.","section":"§2.2, Eq. (9); Section F"}],"minor_comments":[{"comment":"Several typos remain, including 'supervisor' where 'superior' is intended in the contributions list, and 'sharping parameter' in the hyperparameter summary.","section":"Section 1 and Section C.2"},{"comment":"ImageNet results are reported as single runs with margins of roughly 0.5–0.9% over strong baselines; adding multiple seeds or error bars, as already done for CIFAR-100 in Table 6, would strengthen the headline comparisons.","section":"Table 4"},{"comment":"Hyperparameters are tuned separately for each architecture and setting; the paper should state in the main text how much of the reported gain survives when a single hyperparameter configuration is fixed across settings, at least for the two ImageNet settings.","section":"Sections C.2, C.5, C.6"},{"comment":"The choice of 18×18 RoIAlign features is justified by saturation at 28×28, but the improvement over 7×7 is small (0.27 mAP) and detection results are single-run; some variance information would help.","section":"Table 12b"},{"comment":"The notation for the feature matrix is difficult to parse due to formatting issues ('k9th column' and similar artifacts); please clarify the indexing and the dimensions of the kernel matrices.","section":"Equation (1)"},{"comment":"The limitation paragraph is appropriately candid about the Gaussian assumption and the open question of the true feature distribution; please also note the diagonal-covariance reduction to moment matching there, since it mitigates the concern.","section":"Section F"}],"recommendation":"major_revision","confidential_remarks":"The missing control experiment is the single substantive obstacle to acceptance, and it is a standard, short experimental addition. The rest of the empirical work is strong and the paper is likely to be competitive once the authors disentangle the effect of the metric from the effect of the injected category-interrelation cost."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a solid piece of KD engineering. The new combination—discrete WD for logit distillation with a teacher-derived CKA inter-class cost, plus continuous WD between per-image Gaussians for feature distillation—is genuinely novel relative to WCoRD and EMD+IPOT, which only did cross-instance discrete WD for features. The experiments are extensive: ImageNet, CIFAR-100 across CNN/Transformer pairs, self-KD, and object detection, with careful ablations. The gains over strong baselines (ReviewKD, DKD, WTTM) are consistent, and the method is competitive on latency. That's real value.\n\nThe soft spot is that the paper's central mechanistic claim—that WD outperforms KL because it does cross-category comparison—is not actually isolated. WKD-L's transport cost encodes category interrelations from teacher features via CKA; all the KL baselines use no such side information. So the headline comparison conflates the distance metric with the presence of an inter-class cost. Table 2b shows that every IR modeling variant beats KD, but there is no control with a uniform cost matrix (plain WD), nor a KL variant augmented with the same IR cost. Without those, the gain could come entirely from the injected inter-class similarities, not from optimal transport. This is a real gap in the argument, and it is fixable: one or two extra rows in Table 2a would settle it.\n\nMinor issues: ImageNet and COCO results are single-run, with no error bars; hyperparameters are tuned per setting, which is standard practice in this subfield but worth flagging. The Gaussian assumption is acknowledged in the appendix, and since the diagonal version reduces to mean and variance matching, that concern is secondary.\n\nWho is this for: anyone in knowledge distillation. The method itself looks competitive and useful. But the story about why it works currently overshoots the evidence. I would send it to review, and in the revision require the uniform-cost control. If that control shows plain WD with a uniform cost also beats KL, the paper's thesis is strongly supported. If not, the paper needs to be reframed as a method paper that combines WD with IR, not a paper demonstrating the superiority of WD over KL.","headline":"Good KD method, but the central claim that WD itself beats KL is not isolated from the added inter-class cost; the mechanism story needs a control before it should be taken at face value.","tokens_in":27551,"tokens_out":2699,"would_cite":true,"duration_ms":27738,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Wasserstein distance rivals Kullback-Leibler divergence as a distillation objective, with new logit and feature losses beating KL-based baselines on ImageNet, CIFAR-100, and MS-COCO.","keywords":["knowledge distillation","Wasserstein distance","optimal transport","Kullback-Leibler divergence","logit distillation","feature distillation","category interrelations","Gaussian feature modeling"],"falsifier":"Replace WKD-L's interrelation-based transport costs with a uniform cost on ImageNet setting (a); if top-1 accuracy stays at 72.49%, the cross-category mechanism is not the source of the gain. Alternatively, train WKD-F on a feature distribution known to be heavy-tailed, such as quantized features, and compare against KL-based feature distillation; if the gap disappears, the Gaussian modeling premise is what carries the result.","tokens_in":26498,"feed_emoji":"🧠","tokens_out":8908,"duration_ms":81048,"temperature":0.7,"pith_summary":"Knowledge distillation trains a small student network to copy a large teacher network, and the standard objective has long been Kullback-Leibler divergence, which compares each category's probability separately and so cannot use similarities between categories. The paper argues that Wasserstein distance, the minimal cost of moving one probability distribution onto another, fixes both of KL-divergence's main shortcomings. For logits, the proposed WKD-L uses discrete Wasserstein distance with a transport cost derived from measured interrelations among categories, so similar classes are treated as closer. For intermediate features, WKD-F fits a Gaussian to each image's feature map and uses the closed-form Wasserstein distance between Gaussians, which respects the geometry of the distribution space. On ImageNet, CIFAR-100, and MS-COCO, both losses beat their KL-divergence counterparts, and their combination reaches 72.76% top-1 accuracy for ResNet34-to-ResNet18 distillation.","feed_headline":"Wasserstein distance beats KL divergence for AI model distillation","feed_subtitle":"New logit and feature losses push ImageNet top-1 accuracy past KL-based baselines.","key_machinery":"The central object is the Wasserstein distance (earth mover's distance), used in two forms. In discrete form, it is an entropy-regularized optimal-transport problem between probability vectors, with transport cost $c_{ij}=1-\\exp(-\\kappa(1-\\mathrm{IR}_T(C_i,C_j)))$ where $\\mathrm{IR}_T$ is the CKA similarity between categories $C_i$ and $C_j$ computed once from the teacher; this is what enables cross-category comparison. In continuous form, it is the closed-form Wasserstein distance between Gaussians, $D_{\\mathrm{WD}}(\\mathcal{N}_T,\\mathcal{N}_S)=\\|\\mu_T-\\mu_S\\|^2+\\mathrm{tr}(\\Sigma_T+\\Sigma_S-2((\\Sigma_T^{1/2}\\Sigma_S\\Sigma_T^{1/2})^{1/2}))$, with the diagonal-covariance version used in practice. The machinery carries the argument by turning category similarities and feature statistics into a true metric that respects the geometry of the underlying space, which is the property KL-divergence lacks.","core_discovery":"The paper's central claim is that Wasserstein distance can serve as a strong alternative to Kullback-Leibler divergence as the workhorse of knowledge distillation. WKD-L treats the teacher's and student's softmax outputs as discrete probability measures and minimizes the entropy-regularized optimal-transport cost between them, with the transport cost built from centered kernel alignment (CKA) similarities between categories, so the student learns that dog is closer to wolf than to car. WKD-F reshapes each image's feature maps into a Gaussian with a mean and a diagonal covariance, then minimizes the closed-form Wasserstein distance between teacher and student Gaussians, balancing the mean and covariance terms with a ratio. The empirical conclusion is that WKD-L reaches 72.49% top-1 accuracy on ImageNet ResNet34-to-ResNet18 distillation versus 71.03% for classic KD and 71.96% for NKD, while WKD-F reaches 72.50% versus 71.61% for ReviewKD; combining both gives 72.76%, and the same pattern holds on CIFAR-100 and MS-COCO object detection.","pith_inferences":["As an extension, one could strip WKD-L's category-interrelation cost down to a uniform cost; if accuracy does not drop, the claimed cross-category mechanism is not what carries the gain.","As an extension, WKD-F's Gaussian assumption could be tested directly by fitting a non-Gaussian parametric family and comparing; the paper itself flags that what distribution deep features follow is an open problem.","As an extension, the same interrelation matrix could be applied to other objectives, such as label smoothing or contrastive learning, where KL-divergence to a uniform distribution is currently standard.","As an extension, WKD-F's diagonal-covariance choice could be compared against plain mean-and-variance matching without the Wasserstein metric, which would separate the effect of the metric from the effect of matching statistics."],"forward_implications":["Logit distillation can exploit cross-category structure: WKD-L lifts ImageNet ResNet34-to-ResNet18 top-1 accuracy from 71.03% for classic KD and 71.96% for NKD to 72.49%.","Feature distillation can be cast as matching Gaussians: WKD-F reaches 72.50% on the same ImageNet setting, surpassing ReviewKD's 71.61%.","The two losses are complementary: combining WKD-L and WKD-F reaches 72.76% top-1 on ImageNet and outperforms DKD+ReviewKD and FCFD on MS-COCO object detection.","WKD also works when teacher and student share an architecture: WKD-L self-distillation on ImageNet reaches 71.35%, above 70.50% for Born-Again Networks.","WKD can be added to existing strong methods: WKD-F raises NKD to 72.68% on ImageNet, and WKD-L plus WKD-F improves ReviewKD and FCFD on MS-COCO."],"supporting_citations":[{"why":"Defines the KL-divergence distillation baseline that WKD-L replaces and outperforms.","marker":"[2]"},{"why":"Supplies the target/non-target separation scheme that WKD-L adopts and the strong DKD baseline.","marker":"[3]"},{"why":"Provides the normalized KL-divergence baseline NKD and the self-distillation baseline USKD.","marker":"[4]"},{"why":"Gives the optimal-transport formulation and the closed-form Wasserstein distance between Gaussians used in WKD-F.","marker":"[14]"},{"why":"Establishes CKA as a similarity measure for deep representations, used to quantify category interrelations.","marker":"[17]"},{"why":"Defines the centered kernel alignment/HSIC formula that the interrelation matrix in WKD-L is built from.","marker":"[18]"},{"why":"Provides the entropy-regularized optimal-transport algorithm used to solve the discrete Wasserstein problem in WKD-L.","marker":"[23]"},{"why":"ReviewKD is the previous top-performing feature-distillation competitor and the detection baseline WKD-F improves on.","marker":"[29]"}],"fun_headline_variants":["Wasserstein beats KL for knowledge distillation","Wasserstein distance improves model distillation","WD rivals KL in distillation, wins on accuracy","Wasserstein method tops KL for AI teaching","Wasserstein outruns KL in distillation tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the feature-distillation branch, the load-bearing premise is that each image's feature map can be modeled as a Gaussian distribution; if deep features depart strongly from Gaussianity, the Wasserstein distance between Gaussians loses its geometric meaning as a teaching signal.","fun_headline_variants_meta":{"raw":{"variants":["Wasserstein beats KL for knowledge distillation","Wasserstein distance improves model distillation","WD rivals KL in distillation, wins on accuracy","Wasserstein method tops KL for AI teaching","Wasserstein outruns KL in distillation tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1336,"prompt_tokens":1004,"completion_tokens":332,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":620,"tokens_out":332,"duration_ms":3865,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:09:27.583377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace WKD-L's interrelation-based transport costs with a uniform cost on ImageNet setting (a); if top-1 accuracy stays at 72.49%, the cross-category mechanism is not the source of the gain. Alternatively, train WKD-F on a feature distribution known to be heavy-tailed, such as quantized features, and compare against KL-based feature distillation; if the gap disappears, the Gaussian modeling premise is what carries the result.","supporting_citations":[{"cited_title":"Similarity of neural network representations revisited","cited_arxiv_id":null,"evidence_quote":"Establishes CKA as a similarity measure for deep representations, used to quantify category interrelations."},{"cited_title":"Algorithms for learning kernels based on centered alignment","cited_arxiv_id":null,"evidence_quote":"Defines the centered kernel alignment/HSIC formula that the interrelation matrix in WKD-L is built from."},{"cited_title":"Sinkhorn distances: Lightspeed computation of optimal transport","cited_arxiv_id":null,"evidence_quote":"Provides the entropy-regularized optimal-transport algorithm used to solve the discrete Wasserstein problem in WKD-L."},{"cited_title":"Distilling knowledge via knowl- edge review","cited_arxiv_id":null,"evidence_quote":"ReviewKD is the previous top-performing feature-distillation competitor and the detection baseline WKD-F improves on."}],"review_version":1}