{"id":"15c64041-b47b-425d-9bd8-bea0460962e7","arxiv_id":"2412.16493","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CRLD adds within-view and cross-view consistency regularisation plus confidence-based soft label mining to logit-based knowledge distillation, achieving state-of-the-art results on three benchmarks.","lead":"CRLD is a new knowledge distillation method that forces a compact student model to agree with a large teacher model on both normal and heavily altered views of the same image, plus a confidence gate that filters out unreliable teacher predictions. On CIFAR-100, Tiny-ImageNet, and ImageNet, it outperforms prior logit-based distillation methods without adding parameters, and it can be layered on top of several existing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical support for SOTA and universal boosting is not yet settled: no error bars are reported, and Table 6 itself shows DKD+CRLD dropping accuracy on two CIFAR-100 pairs.","rationale":"I focused on what would most directly falsify the abstract's central claim. The reader's weakest assumption is about the theoretical mechanism of cross-view consistency, which matters for explaining the method, but the claim under test is explicitly empirical: CRLD sets SOTA and boosts existing methods. The most load-bearing issue is therefore whether the reported numbers are reliable and universal. The paper's own Table 6 provides a concrete counterexample to one part of the claim: DKD+CRLD is worse than DKD alone on two of six pairs. That is an internal inconsistency in the evidence as presented, not a disagreement with external consensus. The lack of error bars compounds the problem, because several SOTA margins are smaller than what is commonly expected from seed variation in CIFAR-100 and ImageNet distillation experiments. A single, well-defined multi-seed reproduction with paired intervals would settle whether the drops persist and whether the gains exceed noise. This does not accuse the authors of any misconduct; it only asks for the statistical evidence that an empirical SOTA claim requires. I therefore keep the reader's CONDITIONAL verdict, since the method is plausible and the code is released, but the verification is incomplete. The reader and I only partially agree because the reader elevates the cross-view mechanism as the weakest assumption, whereas I see the unverified reliability and universality of the empirical gains as more decisive for the central claim.","tokens_in":19461,"tokens_out":6423,"duration_ms":56680,"concrete_test":"Rerun CRLD, DKD, DKD+CRLD, NormKD, and CRLD-NormKD on the six CIFAR-100 teacher-student pairs in Table 6 with 10 independent seeds, and on the ImageNet ResNet34 to ResNet18 and ResNet50 to MobileNetV1 pairs with at least 3 seeds, reporting per-seed values, mean plus/minus standard deviation, and paired 95% bootstrap intervals computed over the same seed schedule. Accept the universal-boosting claim only if DKD+CRLD's drop on ResNet56/20 and ResNet110/32 disappears or reverses, and only if every claimed SOTA margin exceeds the paired interval width. If a margin is within noise, weaken the SOTA claim to 'competitive with, but not statistically separable from, prior methods.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: CRLD sets state-of-the-art results and, as stated in the abstract, 'without bells and whistles, boosts the performance of various existing approaches by considerable margins.' The paper reports only means over three runs, with no standard deviations, confidence intervals, or significance tests. Many decisive margins are small: in Table 1, CRLD vs DKD on WRN-40-2 to WRN-16-2 is 76.45 vs 76.24; in Table 4, CRLD vs LSKD on ResNet34 to ResNet18 is 72.37 vs 72.08. Without uncertainty quantification, these differences cannot be separated from run-to-run noise. More directly, Table 6 contradicts the universal-boosting claim: grafting CRLD onto DKD lowers accuracy on ResNet56 to ResNet20 (71.49 to 70.70) and on ResNet110 to ResNet32 (73.95 to 73.45), even though the average improves. Since the paper's own reported evidence contains a counterexample to 'boosts various existing approaches', and the headline SOTA margins are statistically unverified, the central claim is not yet established. If those drops persist, or if the small gains are within seed variance, the abstract's claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRLD, a logit-based knowledge distillation method that augments standard KD losses with within-view and cross-view consistency regularisation between teacher and student predictions on weak and strongly augmented views, plus a confidence-based soft-label mining (SLS) step that discards low-confidence teacher predictions. The authors evaluate CRLD on CIFAR-100, Tiny-ImageNet, and ImageNet across a range of homogeneous and heterogeneous teacher-student pairs, and also graft CRLD onto existing logit-based methods (KD, NKD, MLLD, DKD, NormKD). The central claims are that CRLD sets new state-of-the-art results, introduces no extra network parameters, and boosts the performance of various existing methods by considerable margins.","tokens_in":19727,"tokens_out":3407,"duration_ms":31200,"significance":"If the central claims are correct, CRLD is an attractive plug-in regulariser for logit-based distillation: it is simple, adds no parameters, applies to multiple base methods, and is evaluated on several standard benchmarks. The paper has genuine strengths: the empirical sweep is broad (multiple datasets, many teacher-student pairs, ablations in Table 5, sensitivity analyses in Figures 6-7, a ViT-teacher experiment in Table 9, and a label-free KD study in Table 8), the method is clearly specified in Algorithm 1, and the authors state that code and models will be released. However, the headline conclusions are empirical claims, and the evidence as reported is not yet strong enough to support the 'state-of-the-art' and 'boosts various existing approaches' statements: there are no error bars or significance tests, and one of the paper's own tables contains counterexamples to the universal-boosting claim.","major_comments":[{"comment":"The abstract claims that CRLD 'boosts the performance of various existing approaches by considerable margins', but Table 6 directly contradicts this for DKD: on ResNet56->ResNet20, DKD+CRLD scores 70.70 versus DKD's 71.49, and on ResNet110->ResNet32, DKD+CRLD scores 73.45 versus DKD's 73.95. These are not marginal differences (0.79 and 0.50 accuracy points) and they appear in the same table used to support the generalisation claim. The universal-boosting statement must be qualified, or the two counterexamples must be explained and shown to be attributable to a specific identifiable mechanism rather than left as exceptions.","section":"Abstract; Section 4.3, Table 6"},{"comment":"The paper reports only means over 3 runs with no standard deviations, confidence intervals, or significance tests, yet the central claims rest on small margins. For example, in Table 1 the CRLD versus DKD gap on WRN-40-2->WRN-16-2 is 76.45 versus 76.24 (0.21 points), and in Table 4 the CRLD versus LSKD gap on ResNet34->ResNet18 is 72.37 versus 72.08 (0.29 points). With three seeds, these differences are well within plausible run-to-run noise. At minimum, the authors should report per-seed results or error bars for the headline comparisons and state whether the reported margins are stable across seeds; otherwise the 'new state-of-the-art' claim is not statistically supported.","section":"Section 4.2; Tables 1-4, 6"},{"comment":"The core novelty, cross-view consistency (Eq. (2)), is shown in Table 5 to be harmful when used alone (Expt C: 74.10, Expt D: 75.38) relative to using only within-view terms (Expt A: 76.26, Expt B: 76.75), yet beneficial when combined with within-view terms (Expts E-J). The paper offers no principled account of why cross-view matching is only useful in combination, beyond the assertion in Section 4.4 that 'cross-view consistencies are harmful when used individually, but are rather beneficial when applied in concert with within-view consistencies'. This is load-bearing because the method's claimed superiority depends on the interaction, not on the cross-view term per se. The authors should either provide a mechanistic explanation or add targeted experiments that isolate the conditions under which the interaction helps (e.g., varying the relative strengths of the two views, the confidence thresholds, or the teacher's accuracy), so that the result is not simply an unexplained hyperparameter-dependent phenomenon.","section":"Section 3.2, Eq. (2); Section 4.4, Table 5"},{"comment":"Several decisive hyperparameters are tuned on validation data (tau_w, tau_s, lambda_WV_KD, lambda_CV_KD, n, and p_s), and Figure 6 shows that performance varies across the threshold space even if the method is 'not highly sensitive'. Since the SOTA comparisons in Tables 1-4 presumably use tuned thresholds, the paper should state clearly which hyperparameter values were used for each dataset/architecture pair and whether the same values transfer across pairs. Without this information, the reader cannot judge whether the reported gains are robust to reasonable hyperparameter choices or partly reflect per-pair tuning.","section":"Section 4.4; Figure 6; Supplementary B"}],"minor_comments":[{"comment":"The text says 'in Table 3 we plot the evolution of training and test accuracies', but the corresponding visualisation is Figure 3, not Table 3.","section":"Section 4.5, first paragraph"},{"comment":"The phrase 'over the entire hyperaprameter space' contains a typo; it should be 'hyperparameter space'.","section":"Section 4.4, 'Sensitivity to tau_w and tau_s'"},{"comment":"The description says ImageNet images are 'annotated in 100 classes', but ImageNet is a 1000-class dataset; the text appears to be a typo.","section":"Section 4.1, 'ImageNet'"},{"comment":"Equation (1) defines the within-view loss without the SLS mask, while Algorithm 1 applies masks M_w and M_s to the same within-view terms. Since Section 3.3 states that 'other objectives are defined like-wise', the notation should be made consistent so that the reader knows whether the masks apply to Eq. (1) as well.","section":"Section 3.2, Eq. (1); Algorithm 1"},{"comment":"The supplementary text says 'From the results in Table 9', but the relevant table in the main text is Table 7; the cross-reference should be corrected.","section":"Supplementary Materials, 'Strengths of view transformations'"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the ACM MM 2024 conference version (the PDF includes the MM '24 copyright block and DOI). If this is under consideration as a journal submission, the editors should clarify the relationship to the prior conference publication and assess whether there is sufficient new material for a journal version. The technical concerns in the major comments are, in my view, addressable with additional experiments and revised claims, so I do not recommend rejection on technical grounds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on arXiv:2412.16493. It's a logit-based KD paper that takes consistency regularisation from semi-supervised learning and applies it to teacher-student logit matching. The specific CRLD formulation—within-view and cross-view consistency plus confidence-based soft label selection—is new relative to the cited literature. Each ingredient exists (FixMatch-style weak/strong views, pseudo-label thresholds), but the teacher-student cross-view matching is not something SSKD or HSAKD do. That makes it a legitimate, simple contribution.\n\nThe paper does a lot right. The evaluation is broad: three datasets, many teacher-student pairs, ablations, sensitivity analysis, a ViT teacher, and a label-free setting. Code is released. The authors also explicitly address an apparent contradiction with Beyer et al. and show their cross-view term is harmful alone but helpful in combination. That is honest, and the ablations in Table 5 are informative.\n\nThe soft spots are real but not fatal. First, there are no error bars or significance tests anywhere. Several headline margins are small (76.45 vs 76.24 on WRN-40-2→WRN-16-2; 72.37 vs 72.08 on ResNet34→ResNet18), so I cannot tell if CRLD is actually better than DKD or LSKD on those pairs. Second, Table 6 contradicts the abstract's universal-boosting claim: DKD+CRLD is worse than DKD on ResNet56→ResNet20 (70.70 vs 71.49) and ResNet110→ResNet32 (73.45 vs 73.95). That is a direct counterexample to 'boosts various existing approaches by considerable margins'. Third, the label-free CRLD (Table 8) beats the supervised CRLD (Table 1) on both shared pairs (77.82 vs 77.60, 75.36 vs 75.27). That needs an explanation; it makes me suspect a configuration difference or a typo. Fourth, the text says 'a notable +1.56% accuracy gain' from Expt B to J, but Table 5 shows 76.75→78.18, which is 1.43. Minor, but sloppy.\n\nMy bottom line: the core idea is useful, the experiments are extensive, and the code is available. The paper deserves a serious referee. I would send it to review, but I would ask the authors to add variance information, reconcile the DKD drops and the label-free anomaly, and soften the universal-boosting language.","headline":"Solid empirical KD paper with a simple consistency-regularisation idea; the universal-boosting claim overreaches and the lack of error bars blunts the SOTA claim.","tokens_in":20257,"tokens_out":4114,"would_cite":true,"duration_ms":32155,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-view consistency regularisation sets new logit-distillation records on CIFAR-100, Tiny-ImageNet, and ImageNet, adding no parameters.","keywords":["knowledge distillation","logit distillation","consistency regularisation","cross-view learning","data augmentation","confirmation bias","image classification"],"falsifier":"Re-run the paper's Table 5 ablation on Tiny-ImageNet (within-view terms alone versus within-view plus cross-view terms, keeping the strong augmentation and thresholds fixed): the central claim predicts a clear accuracy gain from the two cross-view terms. If the gain is absent or negative on this second dataset, the claim that cross-view consistency is the key ingredient would be refuted.","tokens_in":19244,"feed_emoji":"📊","tokens_out":9003,"duration_ms":67963,"temperature":0.7,"pith_summary":"This paper tries to establish that a simple extra training signal—forcing the student's predictions to agree with the teacher's across both the same and differently augmented views of an image—pushes logit-based knowledge distillation past current methods on CIFAR-100, Tiny-ImageNet, and ImageNet. If the claim holds, it matters because the method adds no network parameters and no inference cost, and it can be grafted onto existing logit-based distillation methods to improve their accuracy. The paper also claims the method keeps working when ground-truth labels are unavailable, and that it transfers from a vision-transformer teacher to a convolutional student. A sympathetic reading of the evidence is that the key contribution is showing that cross-view consistency, previously thought to be inferior to same-view matching, becomes beneficial when combined with same-view consistency and confidence-filtered soft labels.","feed_headline":"Cross-view consistency lifts distilled students past prior records","feed_subtitle":"Matching weak and strong views at once, with no extra parameters and clear gains over existing logit methods.","key_machinery":"The load-bearing object is a four-way prediction-matching loss defined on a weakly augmented view $T_w(x)$ and a strongly augmented view $T_s(x)$ of each input. Within-view terms use $\\mathrm{KLD}(p_S^w, p_T^w)$ and $\\mathrm{KLD}(p_S^s, p_T^s)$; cross-view terms use $\\mathrm{KLD}(p_S^w, p_T^s)$ and $\\mathrm{KLD}(p_S^s, p_T^w)$, where $p$ denotes the teacher's or student's softmax predictions. Each term is multiplied by a binary mask from confidence-based soft-label mining, keeping only teacher predictions whose top class probability exceeds thresholds $\\tau_w$ and $\\tau_s$. The full objective sums these gated KL terms with cross-entropy on both views to the ground-truth label. The mechanism is claimed to work because the strong view amplifies dark knowledge, the confidence masks suppress confirmation bias, and the cross-view terms regularise the student against overconfident teacher outputs.","core_discovery":"The paper's central claim is that consistency regularisation can be moved from semi-supervised learning into logit-based knowledge distillation with strong results. CRLD takes a weakly and a strongly augmented view of each image, passes both through teacher and student, and enforces four matching terms: the student's weak prediction should match the teacher's weak prediction, the student's strong should match the teacher's strong, and the two cross-view pairs should also match. A confidence-based soft-label mining step discards teacher predictions whose maximum class probability falls below a threshold, so that unreliable supervision does not reinforce confirmation bias. On CIFAR-100, Tiny-ImageNet, and ImageNet, across homogeneous and heterogeneous teacher-student architecture pairs, the paper reports new state-of-the-art accuracies, and it reports that adding CRLD to NormKD, DKD, NKD, MLLD, and vanilla KD improves each of them. The paper also positions CRLD as a refinement of the earlier finding that teacher and student must see identical views: cross-view matching is harmful alone but helps in combination with within-view matching.","pith_inferences":["Beyond the paper: the cross-view terms may be functioning mostly as a strong regulariser rather than as a richer knowledge channel, since they are harmful alone but beneficial on top of within-view terms; that would explain the ablation pattern and predict that the gain shrinks as the student already generalises well.","Beyond the paper: the two confidence thresholds make the method depend on teacher calibration; a natural extension is to make the thresholds class-conditional or adapt them during training, which the paper does not explore.","Beyond the paper: the strong augmentation strength is a critical knob—the paper shows performance drops when too many RandAugment operations are applied—so the method may be sensitive to the augmentation policy on datasets unlike CIFAR-100 or ImageNet."],"forward_implications":["Grafting CRLD onto existing logit-based methods such as NormKD, DKD, NKD, MLLD, and vanilla KD improves their accuracy on CIFAR-100 by about one to three points, with no extra parameters or inference overhead.","CRLD outperforms both logit-based and feature-based distillation baselines on CIFAR-100, Tiny-ImageNet, and ImageNet across many teacher-student architecture pairs.","The method remains effective in a label-free setting where ground-truth labels are unavailable during distillation, a practical scenario for proprietary teachers.","CRLD transfers knowledge from a ViT-L teacher to a ResNet-18 student, suggesting it generalises across architectures, not just within a family."],"supporting_citations":[{"why":"Supplies the original logit-KD objective and the notion of dark knowledge that CRLD's design targets.","marker":"[27]"},{"why":"Reports that teacher and student must receive identical views for best distillation; CRLD uses this as the rival view it claims is incomplete.","marker":"[5]"},{"why":"NormKD is one of the strongest logit baselines and the main method CRLD is grafted onto.","marker":"[10]"},{"why":"DKD is a state-of-the-art logit method whose target and non-target decomposition CRLD boosts in experiments.","marker":"[75]"},{"why":"NKD provides another logit baseline and soft-label formulation that CRLD improves.","marker":"[65]"},{"why":"MLLD supplies the training configuration under which CRLD is also tested.","marker":"[32]"},{"why":"Supplies the confidence-threshold pseudo-labelling mechanism CRLD adapts to soft-label mining.","marker":"[50]"},{"why":"Provides evidence that strong augmentation amplifies dark information and mitigates confirmation bias, motivating cross-view consistency.","marker":"[62]"},{"why":"Defines the RandAugment operations used to build the strong view in CRLD.","marker":"[13]"}],"fun_headline_variants":["Cross-view consistency regularisation fixes two logit-distillation flaws","No-param regulariser tames teacher overconfidence and confirmation bias","Cross-view regularisation kills two KD bugs: overconfidence and confirmation bias","Cross-view consistency lifts logit distillation without extra parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that matching the student's prediction on one augmentation to the teacher's prediction on a different augmentation transfers useful knowledge when paired with same-view matching; the paper's own ablations show cross-view terms alone degrade accuracy and offer no mechanism to explain why the combination works.","fun_headline_variants_meta":{"raw":{"variants":["Cross-view consistency regularisation fixes two logit-distillation flaws","No-param regulariser tames teacher overconfidence and confirmation bias","Cross-view regularisation kills two KD bugs: overconfidence and confirmation bias","Cross-view consistency lifts logit distillation without extra parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000977,"raw_usage":{"total_tokens":4169,"prompt_tokens":982,"completion_tokens":3187,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":3115}},"tokens_in":598,"tokens_out":3187,"duration_ms":20389,"temperature":1.0,"reasoning_tokens":3115,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:31:07.601115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's Table 5 ablation on Tiny-ImageNet (within-view terms alone versus within-view plus cross-view terms, keeping the strong augmentation and thresholds fixed): the central claim predicts a clear accuracy gain from the two cross-view terms. If the gain is absent or negative on this second dataset, the claim that cross-view consistency is the key ingredient would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DKD is a state-of-the-art logit method whose target and non-target decomposition CRLD boosts in experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NKD provides another logit baseline and soft-label formulation that CRLD improves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MLLD supplies the training configuration under which CRLD is also tested."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the confidence-threshold pseudo-labelling mechanism CRLD adapts to soft-label mining."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that strong augmentation amplifies dark information and mitigates confirmation bias, motivating cross-view consistency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the RandAugment operations used to build the strong view in CRLD."}],"review_version":1}