{"id":"5e8e69ec-1fa2-4079-bafe-b7fa3c024bf8","arxiv_id":"2411.10693","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MCLD uses three contrastive losses on raw logits, rather than softmax probabilities, to distill knowledge from a teacher network to a student network.","lead":"This paper introduces a knowledge distillation method, MCLD, that compares a small student network's raw class scores with a large teacher network's scores using three contrastive loss terms. The authors report small accuracy gains over existing distillation methods on several image classification benchmarks while using less training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main tables compare LSKD+MLKD at 240 epochs although its published recipe uses 480; until the 480-epoch results are in the main comparison, the SOTA claim is not secured.","rationale":"The reader's verdict is CONDITIONAL, and my concern reinforces that conditional status rather than overturning it. The reader explicitly flagged the LSKD+MLKD reproduction choice in the rationale as 'questionable baseline reproduction choice', but the reader's stated weakest assumption was the lack of normalization in the Instance-wise CLD dot products. I treat the baseline-epoch mismatch as the more load-bearing concern because the paper's central claim is empirical SOTA; a method's design intended for 480 epochs will look arbitrarily bad at 240, and the main tables use that unfavorable comparison to highlight MCLD. The paper says 480-epoch results exist in supplements, so this is not yet a demonstrated error, but the main-text evidence for SOTA is incomplete until those numbers are presented alongside the 240-epoch numbers. I would keep the verdict CONDITIONAL: the paper should either include the 480-epoch LSKD+MLKD results in the main comparison or soften the SOTA claim. The scale-normalization issue is real and merits an ablation, but even if the semantic interpretation is imperfect, the empirical comparisons could still stand; the baseline-epoch issue directly threatens the headline claim, so it is the single most load-bearing concern.","tokens_in":13242,"tokens_out":5651,"duration_ms":61252,"concrete_test":"Retrieve the 480-epoch LSKD+MLKD results from the supplements for every teacher-student pair in Tabs. 1 and 2, run with the method's original code and hyperparameters, and re-insert those entries into the main tables before marking best/second-best. Then check whether any LSKD+MLKD value exceeds the corresponding MCLD entry. If it does, the SOTA claim should be explicitly qualified or removed; if MCLD remains ahead across the table, the concern is settled and the main text should show the 480-epoch numbers for transparency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is empirical state-of-the-art performance. In Sec. 5.1, Tabs. 1 and 2 report LSKD+MLKD reproduced at 240 epochs, with entries collapsing or failing to converge (e.g., 19.63 on ResNet50/MN-V2 in Tab. 1 and 54.55 on VGG13/VGG8 in Tab. 2). The paper itself states: 'LSKD+MLKD [39], when trained for 240 epochs (480 epochs in the original report), exhibited significant performance degradation... we also reproduce the same epoch (480 epochs) as in the original report. More results are attached in the supplements.' A method whose published schedule is 480 epochs will be arbitrarily handicapped at 240 epochs, so its poor 240-epoch number does not demonstrate that MCLD outperforms it. The main tables, however, use only the 240-epoch runs for the best/second-best comparisons. If the supplemental 480-epoch runs show LSKD+MLKD beating MCLD on a substantial subset of teacher-student pairs, then 'state-of-the-art' is not established by the main text. This is load-bearing because the headline result, not the individual loss components, is the paper's primary contribution. The fairness of every other comparison also matters, but this is the clearest case where the published protocol of a named baseline is knowingly changed and the favorable comparison is foregrounded while the protocol-consistent results are deferred to supplements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Multi-perspective Contrastive Logit Distillation (MCLD), a logit-only knowledge distillation method. Instead of KL divergence on softened probabilities, MCLD computes three contrastive losses on raw logits: instance-wise CLD using a queue of teacher logits, sample-wise CLD aligning same-sample teacher/student pairs within a batch, and category-wise CLD treating same-class samples as positives. The total loss is L_Inst + L_Samp + ω*L_Cate, with ω warmed up from 0 to 1. Experiments on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet report accuracy gains over feature- and logit-distillation baselines, together with training-efficiency and transferability analyses. The central claim is that a logit-only contrastive loss can rival and often outperform feature-level distillation and attain state-of-the-art results.","tokens_in":13486,"tokens_out":9134,"duration_ms":87437,"significance":"If the claims hold, this is a valuable empirical contribution to knowledge distillation: it shows that raw-logit contrastive losses can capture more than KL-based logit distillation while avoiding the extra mapping modules and sample-indexing overhead of feature distillation. The ablation in Table 4 demonstrates that each of the three loss terms contributes, and the transfer-learning and efficiency experiments are useful. However, the state-of-the-art claim is not yet fully established: the main comparison handicaps one baseline by using a shorter training schedule, the loss equations contain an internal inconsistency, and the raw-dot-product formulation weakens the semantic interpretation. The paper is within the journal's scope, and the core idea is promising, but these points require attention.","major_comments":[{"comment":"LSKD+MLKD is compared at 240 epochs although its published recipe uses 480. The text acknowledges that at 240 epochs this baseline 'exhibited significant performance degradation' and even failed to converge in some cases, and the 480-epoch results are deferred to the supplements. Since the headline claim is state-of-the-art accuracy, comparing against a knowingly handicapped baseline in the main tables does not support that claim. Please include the 480-epoch results for LSKD+MLKD in the main comparison, or state explicitly that the 480-epoch runs still fall below MCLD across the evaluated teacher-student pairs, and adjust the state-of-the-art claim if this is not the case.","section":"Sec. 5.1, Tables 1-2 and the paragraph following Table 2"},{"comment":"Eq. (7) is not algebraically equivalent to Eq. (5). In Eq. (5), the positive similarity σi appears in the denominator as exp(σi/τ), but in Eq. (7) the positive pair has γ=0 and therefore contributes exp(0)=1 to the denominator; samples from the same class (also γ=0) likewise contribute 1, not their similarity. The text says Eq. (7) reformulates Eq. (5) as a (K+1)-way cross-entropy task, but the denominator is not the normalization over the positive and negative logits. This is load-bearing for the method definition: please correct the formula, define the actual denominator and label vector, and state which equation is used in the experiments.","section":"Sec. 4.2.1, Eqs. (5)-(7)"},{"comment":"All three contrastive losses use raw dot products z_s·z_t without normalization. If teacher and student logits have different magnitudes, the contrastive score is dominated by scale rather than by the semantic direction that the paper uses to motivate the method. The manuscript does not report an ablation with L2-normalized, or otherwise scale-aligned, logits, nor any analysis of logit-norm behavior during training. Please add such an ablation or analysis, or temper the claim that the method captures the 'semantic properties' of logits.","section":"Sec. 4.2.1-4.2.3, Eqs. (4), (8), (10)"},{"comment":"The main accuracy tables report no error bars or significance measures. For CIFAR-100 the text states that results are averaged over three trials, but no standard deviations are given, and several margins over the second-best method are very small, for example Ours 71.90 vs LSKD+DKD 71.88 in the ResNet34/ResNet18 ImageNet row of Table 3. ImageNet results appear to be single runs. To support a state-of-the-art claim, please report standard deviations or confidence intervals, or repeated trials, at least for the closest comparisons.","section":"Sec. 5.1, Tables 1-3"}],"minor_comments":[{"comment":"The sentence 'all values below zero are lost' is imprecise: softmax preserves the ordering of the values but loses sign and relative-magnitude information. Rewording in terms of information diversity would be more accurate.","section":"Sec. 3, after Eq. (3)"},{"comment":"The label vector yinst is described as one-hot with a 1 at position 0, but the formula does not show how yinst combines with the γ mask; please clarify the exact cross-entropy implementation and whether the positive similarity appears in the denominator.","section":"Sec. 4.2.1, Eq. (7)"},{"comment":"The denominator in LCate sums only over the negative set N, rather than over positives and negatives as in standard supervised contrastive learning; please justify this design choice and comment on its effect.","section":"Sec. 4.2.3, Eq. (11)"},{"comment":"The ω end epoch is tuned on the validation set, with Table 5 searching values from 1 to 240 and selecting 155. This should be disclosed in the main text, and the sentence claiming 'no deliberate tuning of hyperparameters' should be revised.","section":"Sec. 5.2, Table 5"},{"comment":"The abstract claims 'state-of-the-art performance' while several results are described as 'comparable or even superior' and MCLD is not the best method in every teacher-student pair; please align the wording with the actual pattern of results.","section":"Abstract and Sec. 5.1"},{"comment":"The WTTM Top-5 entry is listed as n/a; please either provide the value or explain why it is unavailable.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the 480-epoch LSKD+MLKD comparison: if the supplemental runs show MCLD still wins on every teacher-student pair, the paper can be fixed by moving those numbers into the main text; if not, the central state-of-the-art claim needs substantial revision. The Eq. (5)/(7) discrepancy should also be resolved before publication. The paper is otherwise within scope, and the multi-perspective idea is worth publishing after these corrections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the paper with the stress-test note in mind. The core idea is genuinely new: decomposing logit distillation into instance-wise, sample-wise, and category-wise contrastive losses on raw logits, with a queue of stored negative pairs and a target-mask queue to handle label consistency. The ablations are clean and each module contributes, which supports the claim that the design is not just a bag of tricks. The transfer experiments and training-efficiency comparison, while partly deferred to supplements, also point in the right direction.\n\nThat said, the state-of-the-art claim is not secured by the main tables. The paper itself discloses that LSKD+MLKD was trained for 240 epochs in the main comparison even though its published recipe uses 480, and it points to supplements for the 480-epoch runs. If those runs show competitive or superior numbers, the headline claim is overstated. This is load-bearing, not cosmetic. I also share the reader's concern about raw dot products without normalization: the contrastive signal could be dominated by logit magnitude rather than semantic direction. An ablation with L2-normalized or standardized logits would clarify this. Minor issues: the warm-up endpoint is tuned on validation (Table 5), so the 'no hyperparameter tuning' claim is soft, and the queue length and temperature are also free parameters. Error bars are absent on ImageNet and transfer tables, and no code or weights are provided.\n\nOn the other hand, the paper is transparent about the LSKD+MLKD epoch mismatch, cites the relevant literature, and the loss equations are coherent. The circularity burden is low—there is no fitting to a prior result or to the evaluation metric.\n\nI would send this to peer review. The authors should be asked to put the 480-epoch LSKD+MLKD results in the main comparison, report error bars or at least seeds, and add a normalization ablation or justify the raw-dot-product choice. The paper is a solid empirical contribution that could become a useful benchmark, but it needs that revision before the SOTA claim is credible.","headline":"MCLD is a coherent logit-only contrastive distillation method with good empirical results, but the SOTA claim rests on a baseline handicapped by using 240 instead of 480 epochs for LSKD+MLKD, so the main comparison needs revision.","tokens_in":14048,"tokens_out":2226,"would_cite":true,"duration_ms":22813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a contrastive loss computed directly on raw teacher and student logits from three perspectives—instance, sample, and category—lets logit-only distillation match or beat feature-level distillation on four…","keywords":["knowledge distillation","logit distillation","contrastive learning","raw logits","image classification","transfer learning","Vision Transformers","CIFAR-100"],"falsifier":"A reader could rerun the CIFAR-100 ResNet32$\\times$4-to-ResNet8$\\times$4 experiment with all three terms computed on L2-normalized logits; if accuracy falls well below the reported 78.19, the unnormalized scale is doing the work. A second check is to multiply teacher logits by a constant factor (say 2 or 0.5) and observe whether accuracy changes substantially, which would confirm magnitude sensitivity rather than semantic contrast.","tokens_in":12986,"feed_emoji":"🎯","tokens_out":11025,"duration_ms":99117,"temperature":0.7,"pith_summary":"Logit distillation has been underused, this paper argues, because the standard pipeline—softmax with temperature, then KL divergence—discards information and ignores the semantic structure in raw logits. The paper proposes Multi-perspective Contrastive Logit Distillation (MCLD), a loss that keeps raw logits and contrasts them from three views: each image against a queue of teacher-logit negatives, each sample against its own teacher counterpart in the batch, and each category against same-class teacher logits. On CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10, MCLD is reported to reach the best accuracy among logit-only methods and to match or exceed feature-distillation methods, while requiring no extra alignment modules or sampled indices. It also transfers well to new datasets and shows strong results when distilling Vision Transformers. If these results hold, the expensive feature-matching machinery of modern distillation is unnecessary for top-tier transfer.","feed_headline":"Three-way logit contrast beats feature distillation","feed_subtitle":"A cheap logit-only loss with three contrastive views reaches top accuracy and trains faster.","key_machinery":"The load-bearing object is the MCLD loss, a sum of three logit-contrast terms acting on raw logits $z_s$ and $z_t$. The instance-wise term is an InfoNCE-style contrastive cross-entropy—pull the matching student–teacher logit pair together against a queue of negative pairs—using stored logit products and a target-mask queue to avoid false negatives; the sample-wise term is a $B$-way classification over the diagonal of the student–teacher similarity matrix; the category-wise term is a supervised contrastive loss over same-class and cross-class teacher logits. A warm-up weight $\\omega$ on the category term grows from 0 to 1 during training, following a schedule borrowed from earlier logit-distillation methods. Together these terms turn logit distillation into a structured ranking problem over instances, samples, and categories rather than a single per-sample KL divergence.","core_discovery":"The paper's central claim is that logits encode high-level semantic structure that can be distilled directly with contrastive learning, without softmax or intermediate features. The MCLD objective is $L_{\\text{MCLD}}=L_{\\text{Inst}}+L_{\\text{Samp}}+\\omega L_{\\text{Cate}}$, where each term is a dot-product comparison between raw student and teacher logits. Instance-wise CLD runs InfoNCE between a student logit and a queue of teacher-logit negatives, with a target-mask queue so same-class samples are treated as positives; sample-wise CLD aligns same-image pairs across the batch with a $B$-way classification on the similarity matrix; category-wise CLD uses ground-truth labels to treat all same-class teacher logits as positives. The loss needs no mapping modules and no positive/negative indexing beyond labels, and the category weight $\\omega$ is ramped up through a warm-up so it acts once the student can discriminate within categories. The paper reports that this decomposition consistently outperforms logit-only baselines and matches or beats feature distillation on the standard benchmarks.","pith_inferences":["An implication left implicit is that all three terms use unnormalized dot products, so the method's success may depend on teacher and student logit scales; L2-normalizing the logits before computing $\\sigma_i$, $\\eta_i$, and $\\psi_p$ is a direct test of whether direction or magnitude carries the signal.","Because the queue makes instance-wise CLD a logit-level analogue of momentum contrastive learning, the same loss could be adapted to other output spaces, such as multimodal embeddings or structured prediction heads, where KL divergence is not well defined.","The warm-up on the category term suggests a curriculum on contrastive hardness; applying the same ramp to the instance and sample terms, or making it adaptive to the student's discriminative ability, could improve early-training stability.","The paper's evidence is mostly classification and transfer; extending MCLD to dense prediction tasks such as object detection would test whether logit-level contrast retains its advantage when spatial structure matters."],"forward_implications":["Logit-only distillation can match or exceed feature-based distillation, so alignment modules for intermediate feature maps are optional rather than necessary for state-of-the-art transfer.","Ground-truth labels replace sampled positive and negative indices, which removes a data-sampling bottleneck and makes each training batch faster.","The three-perspective loss transfers to Vision Transformer students, where the paper reports consistent gains over both logit and feature distillation baselines.","Representations distilled with MCLD transfer better to STL-10 and Tiny-ImageNet than those from several feature-based and logit-based methods.","MCLD reduces the 'bigger teacher, worse student' phenomenon, with accuracy improving as teacher capacity increases in the reported WRN experiments."],"supporting_citations":[{"why":"Defines the standard knowledge-distillation baseline, using temperature-scaled KL divergence on softmax logits, that MCLD is designed to replace.","marker":"[16]"},{"why":"Supplies the InfoNCE contrastive loss that the instance-wise and category-wise terms adapt.","marker":"[33]"},{"why":"Provides the queue mechanism for holding a large set of negative logit pairs drawn from the whole dataset.","marker":"[13]"},{"why":"The contrastive feature-distillation baseline that motivates contrastive learning for distillation and that MCLD compares against.","marker":"[40]"},{"why":"A recent logit-distillation baseline and the source of the warm-up learning strategy used for the category weight.","marker":"[51]"},{"why":"A recent logit-distillation baseline compared throughout the experiments, also cited for the same warm-up schedule.","marker":"[52]"},{"why":"A feature-distillation method representing the state of the art that MCLD claims to match without feature aligners.","marker":"[4]"},{"why":"A feature-distillation baseline used in the main tables, transfer experiments, and training-efficiency comparison.","marker":"[29]"}],"fun_headline_variants":["Logit-only contrastive loss outshines feature distillation","Multi-view logit distillation tops feature methods","Cheap logit contrast wins without feature maps","Three-way logit compare beats complex distillation","Contrastive logits: faster, better than feature distills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that raw dot products between teacher and student logits measure semantic similarity even though the two models' logits have different scales; if magnitude differences dominate, the loss compares scale rather than meaning.","fun_headline_variants_meta":{"raw":{"variants":["Logit-only contrastive loss outshines feature distillation","Multi-view logit distillation tops feature methods","Cheap logit contrast wins without feature maps","Three-way logit compare beats complex distillation","Contrastive logits: faster, better than feature distills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1336,"prompt_tokens":987,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":603,"tokens_out":349,"duration_ms":4221,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:25:21.872310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could rerun the CIFAR-100 ResNet32$\\times$4-to-ResNet8$\\times$4 experiment with all three terms computed on L2-normalized logits; if accuracy falls well below the reported 78.19, the unnormalized scale is doing the work. A second check is to multiply teacher logits by a constant factor (say 2 or 0.5) and observe whether accuracy changes substantially, which would confirm magnitude sensitivity rather than semantic contrast.","supporting_citations":[{"cited_title":"Con- trastive representation distillation","cited_arxiv_id":null,"evidence_quote":"The contrastive feature-distillation baseline that motivates contrastive learning for distillation and that MCLD compares against."},{"cited_title":"DOT: A Distillation-Oriented Trainer","cited_arxiv_id":"2307.08436","evidence_quote":"A recent logit-distillation baseline compared throughout the experiments, also cited for the same warm-up schedule."},{"cited_title":"Distilling knowledge via knowledge review","cited_arxiv_id":null,"evidence_quote":"A feature-distillation method representing the state of the art that MCLD claims to match without feature aligners."},{"cited_title":"Function-consistent feature distillation","cited_arxiv_id":null,"evidence_quote":"A feature-distillation baseline used in the main tables, transfer experiments, and training-efficiency comparison."}],"review_version":1}