{"id":"d3067c49-fbd0-45f6-bafb-60e2a5e5db14","arxiv_id":"2508.04539","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A logit-based distillation method that amplifies and decouples the teacher's Top-K outputs reports gains over several existing losses and can be added to other methods.","lead":"This paper proposes a training loss that makes student models pay extra attention to the teacher model's top-ranked class predictions, instead of treating all output scores equally. The result is a simple plug-in that reports improved accuracy on image classification benchmarks and across several model families.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TSM scaling (Eq. 2) depends on wi and Delta defined only in a missing Algorithm ??, so TopKD's central mechanism and reported SOTA gains are not independently verifiable.","rationale":"Good-faith reading: TopKD is a logit-based distillation method with a plausible motivation—teacher Top-K logits carry semantic structure. The paper is clearly written at the level of equations, and the contrastive + cosine decoupled loss is a reasonable design. The main weakness is not an internal contradiction but a missing specification: the TSM's wi and Delta are not defined in the text. The reader's verdict already flags this. My stress-test agrees and adds that this is the single load-bearing issue because Eq. 2 is the only place where the method differs from a generic contrastive logit loss; without it, the reported improvements cannot be attributed to Top-K scaling. The proposed check (reproduce one key result with the algorithm in hand) would settle whether the concern lands. If the algorithm is provided and results reproduce, the CONDITIONAL verdict can be upgraded; if not, the empirical claims remain unverified. Other issues (no error bars, missing appendix tables, hyperparameter search on validation) are secondary and do not change the verdict.","tokens_in":11749,"tokens_out":3002,"duration_ms":32616,"concrete_test":"Request the missing Algorithm 1 from the authors or obtain the code release. Then implement Eq. 2 exactly with the given w_i and Delta, using the paper's default hyperparameters (K=10, alpha=3, beta=1, batch size 64, 240 epochs, SGD) on CIFAR-100 with ResNet32x4 as teacher and ResNet8x4 as student, and run three seeds. If the mean Top-1 accuracy does not match the reported 78.18 within ±0.3, or if the algorithm is not available, the central claim is not reproducible from the submitted text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—TopKD consistently surpasses SOTA—rests on the Top-K Scaling Module (TSM). Equation (2) defines the rescaled teacher logits z'_i = z_i * w_i + Delta for i in Top-K union {y_g}, but w_i ('rank-dependent scaling factor') and Delta ('bias term proportional to the mean difference between Top-K and Non-Top-K logits') are not specified. The actual computation is deferred to 'Algorithm ?? in the appendix,' and that algorithm is absent from the submitted text. Since z'_t enters the TDL loss (Eq. 4) and thereby the total TopKD loss (Eq. 5), every reported result—main tables, ablations, plug-and-play gains, transformer experiments—depends on this unspecified computation. Without w_i and Delta, a reader cannot implement TSM, cannot distinguish the method from simple contrastive logit distillation, and cannot verify whether the reported gains are due to the proposed mechanism rather than to ad hoc choices or hyperparameter tuning on the benchmark. The paper also defers other details (detailed results, transformer/detection tables) to a missing appendix, but the missing algorithm is the most load-bearing because it defines the core novelty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TopKD, a logit-based knowledge-distillation method. TopKD combines (i) a Top-K Scaling Module (TSM) that rescales the teacher's top-K logits, with a ground-truth boost when the teacher's top-1 prediction is wrong, and (ii) a Top-K Decoupled Loss (TDL) that applies separate cosine-similarity terms to positive top-K, negative top-K, and non-top-K logit dimensions, added to a contrastive batch-alignment loss. The authors claim consistent state-of-the-art results on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet, plug-and-play improvements when TSM/TDL are added to existing KD methods, and further gains for Vision Transformers and object detection. The paper contains the main loss equations, several tables of classification results, ablations over K, α, and β, and qualitative visualizations. However, the central TSM computation is deferred to an appendix algorithm that is absent from the submitted text, and other detailed experiments are also referenced to a missing appendix.","tokens_in":12053,"tokens_out":3955,"duration_ms":42273,"significance":"If the mechanism were fully specified and the reported results reproducible, the paper would make a useful contribution: it offers a lightweight, architecture-agnostic logit-distillation recipe that could complement feature-based methods, and the modular TSM/TDL idea is simple enough to be adopted by practitioners. The contrastive formulation and the emphasis on top-K structure are reasonable and potentially interesting directions. The paper is also honest in reporting ablations over the main hyperparameters. However, the significance is currently capped by three issues: (1) the core TSM scaling factors w_i and Δ are not defined anywhere in the manuscript, so the proposed method cannot be implemented or independently evaluated; (2) the headline claim of \"consistently surpasses state-of-the-art\" is contradicted by several cells in the paper's own tables; and (3) large parts of the experimental evidence, including ViT and detection results, are deferred to a nonexistent appendix. These are fixable in revision, which is why I am not recommending rejection, but they are load-bearing for the central claims.","major_comments":[{"comment":"The definition of the Top-K Scaling Module is incomplete. Equation (2) rescales teacher logits using a rank-dependent factor w_i and a bias Δ, but neither quantity is specified. The text says the procedure is \"provided in Algorithm ?? in the appendix,\" and no such algorithm or appendix appears in the manuscript. Since z'_t enters TDL (Eq. 4) and the final loss (Eq. 5), every reported result depends on this unspecified computation. This is the paper's central novelty and must be fully specified, including defaults, exact formulas, and any edge cases (e.g., when y_g is already in I_top). Without this, the method is not reproducible and cannot be distinguished from simpler logit-scaling heuristics.","section":"§3.2, Eq. (2)"},{"comment":"The abstract and text claim TopKD \"consistently surpasses state-of-the-art distillation methods,\" but the paper's own tables contain counterexamples. In Table 2, the WRN-40-2 → WRN-40-1 cell reports TopKD at 75.09, while FCFD achieves 75.53. In Table 3, setting (a) (ResNet-34/ResNet-18) reports TopKD Top-1 at 71.52, below DKD+LS (71.88), WTTM (72.19), and WKD-L (72.49). These cells undermine the \"consistently surpasses\" claim. The text should be revised to accurately state where TopKD is best and where it is competitive but not superior, or the experiments should be re-run and reported with corrected claims.","section":"§4.1, Tables 1–3 and Abstract"},{"comment":"The hyperparameters K, α, and β are selected by ablations on CIFAR-100 and then adopted as defaults for all experiments, including ImageNet, STL-10, Tiny-ImageNet, and the ViT/detection experiments reported only in the missing appendix. This is a selection-on-the-benchmark concern for the main claims. The paper should provide, at minimum, the sensitivity of the method on at least one additional dataset or architecture, and should disclose clearly that the defaults were tuned on CIFAR-100. The current presentation overstates the generality of the chosen values.","section":"§4.2, Tables 5–6 and §4.3"},{"comment":"Several load-bearing experimental claims are not verifiable because the supporting material is absent. Section 4.1 refers to \"Detailed results are in the appendix (Table??)\"; Section 4.3 states that ViT and object-detection results are \"see appendix for details\"; and the training details section says \"More implementation details are provided in the appendix.\" No appendix is present in the submitted text. The ViT experiments and detection experiments are announced but not reported, so the claims of \"substantial effectiveness\" and \"broad applicability\" are unsupported. The authors must include the appendix or remove these claims.","section":"§4.3 and 'Detailed results ... appendix (Table??)'"},{"comment":"Reference [44] (Wang and Zhou, arXiv:2411.10693, \"Multi-perspective Contrastive Logit Distillation\") is listed in the bibliography but never cited in the body. The contrastive loss in Eq. (1) appears conceptually close to that prior work, and the reader is left to guess how TopKD differs. The authors should cite and contrast their method with [44] explicitly, and state what is new in TopKD relative to that paper. This is essential for assessing novelty and for the reader to understand the incremental contribution.","section":"References / §3.3"}],"minor_comments":[{"comment":"The notation y ∈ R^{B×1} with indices 0 to B−1 is unusual for a cross-entropy label; it would be clearer to write y as a vector of class indices or one-hot labels. Also, the temperature τ is introduced but its default value is not stated here or in the training details.","section":"§3.1, Eq. (1)"},{"comment":"The table formatting is hard to read: the colon-aligned header \"SettingMetricTch. Stu.\" and the merged Feature/Logits columns make it difficult to tell which baseline belongs to which setting. Please reformat and add clear column separators.","section":"§4.1, Table 3"},{"comment":"The sentence \"We report LSKD [37], select DKD+LS [52, 37] as the representative setting\" is grammatically unclear and does not explain why LSKD itself is not compared directly. Please clarify the baseline-selection protocol.","section":"§4.1, Baselines"},{"comment":"The t-SNE and logit-correlation visualizations are qualitative and would be more informative with a quantitative metric (e.g., k-nearest-neighbor accuracy or mean absolute difference). Also, the figure captions do not specify which model is the teacher and which is the student for each panel.","section":"§4.3, Figures 5–6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear structure and a plausible direction, but the missing Algorithm ?? and absent appendix are serious. The authors appear to have built the paper on an earlier preprint (ref [44]) without citing it in the body, which is a transparency issue the editor should monitor during revision. The inconsistent SOTA claims in their own tables are also concerning and suggest the conclusions were written before the final numbers were collected. If the authors supply the missing algorithm, add the appendix, and adjust the claims to match the data, the paper could become acceptable; in its current form it should not be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe punchline: this is a workable idea shipwrecked by a missing appendix. TopKD claims that emphasizing the teacher's top-K logits with a decoupled cosine loss and contrastive alignment beats state-of-the-art logit- and feature-based KD across CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet. If the numbers are real, that would be a useful, deployment-friendly result. But the central mechanism—the TSM scaling in Eq. (2), specifically how w_i and Delta are computed—is deferred to \"Algorithm ?? in the appendix,\" and the appendix is absent. Without that, the method is unverifiable from the submitted text. You cannot implement it, you cannot separate the proposed mechanism from the contrastive core, and you cannot tell whether the gains come from the top-K scaling or from hyperparameter tuning on the benchmark.\n\nCredit where it's due. The intuition is sound: teacher misclassifications often land on semantically related classes, so explicit emphasis on top-K logits is a reasonable extension of DKD's positive/negative decomposition. The loss structure is clearly written at the equation level. The experiments are extensive in scope—many teacher/student pairs, four datasets, and plug-and-play gains on existing methods. The ablations for K and α/β are thoughtful and show a sensible pattern, with performance collapsing at K=50, which supports the premise that top-K knowledge is the informative part.\n\nSoft spots, in proportion. The missing algorithm is the load-bearing one; it is a conditional issue, not a fatal contradiction. The paper also defers detailed results and transformer/detection tables to the same missing appendix, so the boldest claims (ViT and detection gains) are unsubstantiated in the text. There are no error bars or significance tests, and the hyperparameters (α=3, β=1, K=10) are tuned on CIFAR-100, which is also a main benchmark—a legitimate concern for a \"consistent SOTA\" claim. Finally, the contrastive loss in Eq. (1) closely resembles the authors' own prior MCLD (ref [44]), which appears in the bibliography but is never cited in the body. That is a worrying citation gap and it weakens the novelty claim. None of these are fatal by themselves, but together they mean the paper is not ready as submitted.\n\nWho's this for? Anyone working on logit-based distillation. It deserves a serious referee—the idea is plausible and the preliminary numbers are good enough to warrant a careful look at the missing appendix and code. But it needs major revision before I'd trust the claims. Recommendation: send to peer review, but the reviewers should demand the actual algorithm and code before accepting.","headline":"Plausible top-K distillation idea, but the core TSM algorithm is missing from the appendix—unverifiable as submitted.","tokens_in":12559,"tokens_out":3365,"would_cite":false,"duration_ms":29362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the most valuable signal for knowledge distillation lives in the teacher's Top-K logits, and that scaling these logits and supervising with a contrastive plus decoupled-cosine loss lets logit-based distillation match","keywords":["knowledge distillation","logit-based distillation","Top-K logits","contrastive learning","cosine similarity loss","vision transformers","image classification","model compression"],"falsifier":"Run TopKD once the appendix's Algorithm ?? is supplied: if Table 1's accuracies reproduce, the claim is supported. A sharper test is to replace the rank-dependent $w_i$ with a single constant scale on all Top-K logits and a fixed boost on the ground-truth class; if the simplified scaling matches full TopKD within noise, then rank-dependence is not the operative mechanism.","tokens_in":1713,"feed_emoji":"🎓","tokens_out":2616,"duration_ms":104305,"temperature":0.7,"pith_summary":"The paper is trying to show that the most useful knowledge a teacher passes to a student sits in the teacher's Top-K output logits, not in the full softened probability vector that classic distillation matches. The authors propose TopKD, which rescales those top logits—giving an extra boost to the ground-truth class when the teacher's own Top-1 guess is wrong—and trains the student with a contrastive loss plus a decoupled cosine loss that treats positive Top-K, negative Top-K, and non-Top-K logits separately. They report that this logit-only recipe outperforms both logit-based and feature-based distillation methods on CIFAR-100 and ImageNet, transfers better to STL-10 and Tiny-ImageNet, works for Vision Transformer students, and can be dropped into existing distillation methods as a plug-in. If true, it would mean that lightweight logit distillation no longer needs to lag behind feature-matching approaches.","feed_headline":"Distill only the teacher's Top-K logits to beat feature methods","feed_subtitle":"A scaling module plus a decoupled cosine loss lifts students above state-of-the-art KD baselines.","key_machinery":"The central objects are the Top-K Scaling Module (TSM) and the Top-K Decoupled Loss (TDL). TSM rescales the teacher logits as $z'_i = z_i \\cdot w_i + \\Delta$ on the Top-K indices plus the ground-truth class, where $w_i$ is a rank-dependent weight and $\\Delta$ a bias tied to the mean Top-K/non-Top-K gap. TDL splits the student-teacher cosine similarity into positive Top-K, negative Top-K, and non-Top-K components with weights $\\alpha$ and $\\beta$. Together they amplify the semantically informative part of the teacher's logits and align the student's logit geometry directionally rather than by exact value matching.","core_discovery":"The central claim is that Top-K knowledge—the few highest-confidence and most-negative entries in a teacher's logit vector—carries an inter-class semantic structure that KL-Div-based distillation erases. TopKD operationalizes this in two pieces. The Top-K Scaling Module (TSM) multiplies the teacher's Top-K logits by a rank-dependent weight and adds a bias $\\Delta$ proportional to the gap between Top-K and non-Top-K logits; when the teacher's Top-1 prediction is wrong, it also amplifies the ground-truth logit to correct the teacher's bias. The Top-K Decoupled Loss (TDL) replaces the overall cosine similarity with three decoupled terms—positive Top-K, negative Top-K, and non-Top-K—so that high","pith_inferences":["A direct extension not tested in the paper: on very large label spaces, such as ImageNet-21k or long-tail datasets, the optimal $K$ may need to vary per sample or per teacher confidence rather than staying fixed.","The sharp accuracy drop at $K=50$ in the paper's own ablation suggests Top-K knowledge has a sweet spot; an adaptive or confidence-weighted $K$ could be a natural follow-up.","Because TSM and TDL only modify the loss, the same idea could transfer to other domains where top-of-the-distribution knowledge dominates, such as language-model token distributions or retrieval logits.","The exact computation of $w_i$ and $\\Delta$ is deferred to an Algorithm ?? that does not appear in this submission, so the quantitative claims are contingent on the authors' implementation being reproduced."],"forward_implications":["Logit-based distillation can match or beat feature-based distillation on standard classification benchmarks, removing the need for architectural alignment layers and feature-matching overhead.","The TSM and TDL components can be inserted into existing distillation pipelines as plug-ins, raising accuracy without retraining the teacher or changing student backbones.","Students trained with TopKD transfer better to new datasets such as STL-10 and Tiny-ImageNet, indicating that the learned representations are more reusable.","TopKD also improves Vision Transformer students, suggesting that the Top-K signal generalizes across convolutional and attention-based architectures.","With stronger teachers, TopKD students continue to improve where several comparison methods stagnate or degrade, helping to mitigate the 'bigger teacher is not always better' problem."],"supporting_citations":[{"why":"Introduces logit-based knowledge distillation and the KL-Div matching that TopKD argues washes out Top-K structure.","marker":"[16]"},{"why":"Decoupled knowledge distillation; the main logit-based baseline and the work TopKD extends by decoupling supervision.","marker":"[52]"},{"why":"Argues that KL-Div alignment is fundamentally limited, providing TopKD's motivating critique of KL-Div.","marker":"[29]"},{"why":"DOT, a logit-based distillation baseline that TopKD compares against and reports surpassing.","marker":"[51]"},{"why":"WTTM, a transformed-teacher logit matching baseline used as a comparison in the main tables.","marker":"[53]"},{"why":"SimCLR supplies the contrastive-learning formulation TopKD adapts for instance-level alignment.","marker":"[5]"},{"why":"MoCo provides the contrastive objective and negative-pair mechanism referenced in Equation 1.","marker":"[12]"},{"why":"CRD, a feature-based contrastive distillation baseline that TopKD claims to match or beat without feature matching.","marker":"[39]"},{"why":"ReviewKD, a feature-based distillation baseline that TopKD claims to surpass on several settings.","marker":"[4]"},{"why":"FCFD, a feature-based distillation method used as a strong comparison and in transfer experiments.","marker":"[28]"}],"fun_headline_variants":["Top-K logit accent: a logit-only boost that beats feature KD","Why distill all logits? Top-K scaling does more with less","Teacher's top-K logits hide a structure KL destroys","TopKD: keep the top logits, drop the noise, win distillation","A logit-only trick that outclassed feature transfer"],"cache_read_input_tokens":14336,"weakest_assumption_plain":"The load-bearing premise is that the paper's specific rank-dependent scaling of the teacher's Top-K logits—including a larger boost to the ground-truth class when the teacher's Top-1 is wrong—is the right way to expose Top-K knowledge, and the exact computation of that scaling is deferred to an Algorithm ?? that is not present in this submission, so the central mechanism cannot be independently checked from the text.","fun_headline_variants_meta":{"raw":{"variants":["Top-K logit accent: a logit-only boost that beats feature KD","Why distill all logits? Top-K scaling does more with less","Teacher's top-K logits hide a structure KL destroys","TopKD: keep the top logits, drop the noise, win distillation","A logit-only trick that outclassed feature transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000495,"raw_usage":{"total_tokens":2271,"prompt_tokens":752,"completion_tokens":1519,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":1428}},"tokens_in":496,"tokens_out":1519,"duration_ms":15496,"temperature":1.0,"reasoning_tokens":1428,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:54:47.581675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TopKD once the appendix's Algorithm ?? is supplied: if Table 1's accuracies reproduce, the claim is supported. A sharper test is to replace the rank-dependent $w_i$ with a single constant scale on all Top-K logits and a fixed boost on the ground-truth class; if the simplified scaling matches full TopKD within noise, then rank-dependence is not the operative mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Decoupled knowledge distillation; the main logit-based baseline and the work TopKD extends by decoupling supervision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Argues that KL-Div alignment is fundamentally limited, providing TopKD's motivating critique of KL-Div."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DOT, a logit-based distillation baseline that TopKD compares against and reports surpassing."},{"cited_title":"Zheng and E.-H","cited_arxiv_id":null,"evidence_quote":"WTTM, a transformed-teacher logit matching baseline used as a comparison in the main tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MoCo provides the contrastive objective and negative-pair mechanism referenced in Equation 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CRD, a feature-based contrastive distillation baseline that TopKD claims to match or beat without feature matching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ReviewKD, a feature-based distillation baseline that TopKD claims to surpass on several settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FCFD, a feature-based distillation method used as a strong comparison and in transfer experiments."}],"review_version":1}