{"id":"f60edf10-80e4-46e8-ac09-fb12dbfcbaa2","arxiv_id":"2501.07040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Distilling with retrieved in-context teacher logits as extra regularization improves student accuracy in offline, online, and teacher-free knowledge distillation.","lead":"This paper proposes IC-KD, a knowledge distillation method that trains a student network with teacher predictions for retrieved in-context samples: similar same-class neighbors and different-class negatives. It reports accuracy gains over several distillation baselines on CIFAR-100, ImageNet, and Cityscapes, across offline, online, and teacher-free settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval by teacher feature similarity is the load-bearing mechanism, yet the paper never tests whether top-K retrieval outperforms random or least-similar same-class selection; without that control, the observed gains may come from generic in-context regularization rather than retrieval.","rationale":"The reader's weakest_assumption and my load-bearing concern converge: the paper assumes without derivation that teacher feature cosine similarity selects in-context samples whose logits are good distillation targets. This is the core of the method's novelty; if it fails, the paper reduces to adding an arbitrary same-class soft-target regularizer, which would not support the 'in-context sample retrieval perspective' claimed in the title and abstract. The paper's Fig. 7 shows a qualitative correlation but does not establish that this correlation is necessary for the observed gains. A randomized retrieval control is the minimal experiment that would separate the retrieval mechanism from the regularization effect. I agree with the reader's conditional verdict: the method is plausible and well-tested in aggregate, but the missing control leaves the central mechanism unverified. No new concern beyond the reader's was found that would change the verdict.","tokens_in":18586,"tokens_out":9932,"duration_ms":98381,"concrete_test":"On a fixed CIFAR-100 teacher-student pair (e.g., ResNet32x4→ResNet8x4) with the paper's default hyperparameters, run three PICD variants while keeping all other losses (standard KD, L_ce, NICD) fixed: (A) K positive in-context samples selected by teacher cosine similarity as in Eq. 7; (B) K random same-class samples; (C) K least-similar same-class samples. Also run a fourth variant with the NICD negative samples selected randomly instead of by dissimilarity. If variant (A) does not outperform (B) and (C) by a margin comparable to the reported IC-KD gain over KD (e.g., Table IV, ResNet32x4→ResNet8x4: +3.89), the retrieval step is not the active ingredient and the 'in-context' framing of the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is the unproven assumption that cosine similarity in the teacher feature space identifies in-context samples whose teacher logits serve as good soft targets for the student (Eq. 7, used in PICD Eq. 8 and NICD Eq. 10). The paper motivates this with Fig. 7 but provides no quantitative control: it never ablates the retrieval rule itself. Concretely, the paper does not compare Top-K same-class retrieval with random same-class selection or with least-similar same-class selection. If random or least-similar same-class teacher logits produce the same gains, then the improvement is caused by the added regularization (any same-class soft target acts as a learned label smoother), not by the retrieval-based 'in-context' definition. Because the paper's novelty and title rest entirely on the retrieval perspective, this untested assumption is load-bearing. The concern is amplified in online KD (Table IX), where the memory bank is rebuilt from a student that changes every epoch, so the teacher-feature-similarity rationale no longer even applies directly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes In-Context Knowledge Distillation (IC-KD), a method that reformulates the knowledge transferred in distillation as relationships between a query sample and retrieved \"in-context\" samples, rather than per-sample teacher outputs. A teacher feature memory bank is built, and for each training sample the method retrieves Top-K same-class neighbors and Top-N different-class neighbors by cosine similarity in the teacher feature space. Two auxiliary losses are added: Positive In-Context Distillation (PICD, Eq. 8), which matches the student's prediction to a similarity-weighted aggregate of teacher predictions over same-class neighbors, and Negative In-Context Distillation (NICD, Eq. 10), which pushes the student's prediction away from different-class teacher neighbors. The method is evaluated on CIFAR-100 and ImageNet classification, online and teacher-free KD, semantic segmentation on Cityscapes, and heterogeneous teacher-student architectures. The central claim is that this in-context-sample regularization improves student accuracy consistently across diverse KD paradigms.","tokens_in":31,"tokens_out":5647,"duration_ms":115958,"significance":"If the central claim holds, the paper's main contribution is a simple, task-agnostic regularization that can be plugged into offline, online, and teacher-free KD and that improves accuracy across many teacher-student pairs. The experimental coverage is broad: 13 CIFAR-100 pairs, ImageNet, Cityscapes, online KD, teacher-free KD, and heterogeneous architectures, with consistent gains when IC-KD is added to KD and to CRD. The method is also relatively efficient, as shown by the training-time comparison in Table VII, and the proposed losses are easy to implement. The paper gives useful ablations of its loss components (Tables IV and V) and of the weighting terms (Table VI). However, the retrieval mechanism that defines the paper's novelty is not directly validated, the theoretical section is not a derivation, and the abstract's state-of-the-art claim is stronger than the tables support. No code or error bars are provided, which limits the certainty of the empirical claims.","major_comments":[{"comment":"The central claim that retrieval by teacher feature similarity is responsible for the observed gains is not tested. The paper never compares Top-K same-class retrieval with random same-class selection or with least-similar same-class selection. Without such a control, the accuracy improvements could come from any same-class soft-target regularization, not from the \"in-context retrieval\" mechanism on which the title and novelty rest. Figure 7 is qualitative and does not provide the missing control. I request an ablation on at least two representative pairs (e.g., ResNet32x4→ResNet8x4 and VGG13→VGG8) comparing (a) teacher-similarity Top-K, (b) random same-class, and (c) least-similar same-class selection, keeping all other loss terms fixed.","section":"III-B, Eq. (7), and Tables I-II"},{"comment":"In the online KD setting, the feature memory bank is rebuilt from student 1, which changes every epoch, rather than from the fixed teacher model. The paper's rationale that teacher feature similarity identifies useful in-context samples therefore does not directly apply, yet the method still reports gains. This is a load-bearing gap: please either justify why student-derived features can substitute for teacher features, or add an ablation comparing a fixed teacher-derived memory bank with the dynamic student-derived bank in the online setting.","section":"IV-B, Table IX (online KD)"},{"comment":"The section titled \"Rethinking 'Knowledge' in Distillation\" is presented as a theoretical analysis, but it does not derive the central claim. Equations (1)-(4) restate the known Tf-KD result that KD is a form of learned label smoothing regularization. The subsequent statement that \"the discrepancy between the teacher's predictions for in-context samples and the student's predictions for corresponding samples decreases as the similarity between these paired samples increases\" is asserted, not proven, and the proposed losses in Eqs. (5)-(6) do not follow from the LSR equivalence. The authors should either provide a formal derivation under explicit assumptions (e.g., Lipschitz continuity of the logit map in the feature space) or reframe this section as motivation rather than theory.","section":"III-A"},{"comment":"The abstract claims that IC-KD \"consistently achieves state-of-the-art performance on CIFAR-100 and ImageNet datasets.\" This is contradicted by the reported numbers: in Table I, DKD achieves 76.24 on WRN-40-2→WRN-16-2 while IC-KD achieves 76.21, and in Table II, several feature-based methods outperform IC-KD on multiple pairs (e.g., SimKD 78.39 vs. 77.52 on ResNet32x4→ShuffleNetV2; CAT-KD 78.41 vs. 77.52 on the same pair). Please replace \"consistently achieves state-of-the-art\" with a qualified claim such as \"competitive with state-of-the-art methods,\" or specify the subset of settings for which the SOTA claim holds.","section":"Abstract and Tables I-III"},{"comment":"Equation (10) is ambiguous as written: L_nicd = 1 - cos(p_s_i, p_t_i) + b_i,j cos(p_s_i, p_t_j). The term b_i,j is indexed by j, but the expression has no summation or averaging over the retrieved negative set, even though the text refers to N negative in-context samples. Please clarify whether the intended loss is an average over retrieved negatives, and specify how the j index is selected in the notation. This is the definition of one of the two proposed losses, so the equation needs to be unambiguous.","section":"III-B, Eq. (10)"}],"minor_comments":[{"comment":"Line 6 of Algorithm 1 reads \"for t ← −1 to T\"; this is likely a typo for \"1 to T\" or \"0 to T−1\" and should be corrected.","section":"Algorithm 1"},{"comment":"The caption lists the second teacher-student pair as \"ResNet101 → ResNet18\" twice; the second should refer to PSPNet-ResNet18, not ResNet101 again.","section":"Table X caption"},{"comment":"The paper states that each CIFAR-100 experiment is repeated three times, but no standard deviations are reported anywhere, and no code is released. Please report mean ± std or provide a reproducibility statement with code to support the claimed gains.","section":"IV-A and all result tables"},{"comment":"The aggregated teacher prediction is denoted \\hat p_i^s, which is confusing because it is constructed from teacher logits and used as a target for the student. Renaming it to \\hat p_i^t or another notation would improve clarity.","section":"III-B, Eq. (7)"},{"comment":"In the teacher-free setting, the text says a trained baseline model serves as the teacher and an untrained model as the student; this is essentially offline KD and should be clarified, since \"teacher-free\" normally implies no pretrained teacher is used.","section":"IV-A, teacher-free KD"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's scope and the empirical breadth is impressive, but the missing retrieval-rule ablation is a substantive gap that must be addressed before publication. The theoretical section also overstates its status as a derivation, and the abstract's SOTA claim needs to be reconciled with the tables. I do not see a fatal flaw in the method itself; the issues are fixable with additional experiments and careful rewriting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. This paper proposes a plug-in regularization for knowledge distillation: build a teacher feature memory bank, retrieve top-K same-class neighbors for each sample, aggregate their teacher logits, and use that as a soft target (PICD); then push the student away from top-N different-class teacher logits via cosine separation (NICD). The empirical work is broad and mostly consistent—CIFAR-100 (7 same-arch + 6 cross-arch pairs), ImageNet, Cityscapes segmentation, plus offline, online, and teacher-free KD. Gains over CRD are often 0.3–1.9 points, and the component ablations show both losses help. That is a genuine incremental contribution, not a repackaging of a single existing method.\n\nThe big soft spot is exactly what the stress-test says: the paper never tests whether retrieval by teacher feature similarity actually matters. No comparison to random same-class selection or least-similar same-class selection. If random same-class teacher logits give the same accuracy, then the improvement is generic label smoothing from extra soft targets, and the 'in-context' framing is decoration. This is the central claim of the paper, so the omission is not minor. A second, related issue is online KD, where the memory bank is built from a student that changes every epoch; the teacher-feature rationale no longer applies cleanly, yet the method still works, which only deepens the question of what the retrieval is doing.\n\nOther issues are more ordinary. Section III-A re-derives KD as label smoothing and then asserts that in-context samples contribute; it is not a derivation. No code, no error bars. And the abstract's 'consistently achieves state-of-the-art' is contradicted by Table I, where DKD and LSKD beat IC-KD on several CIFAR-100 pairs.\n\nOn balance, the paper deserves a serious referee. The experiments are extensive and the loss construction is reasonable. But the missing retrieval-rule ablation is load-bearing: a referee should ask for it, along with error bars and a toned-down abstract. I'd send it out, not desk-reject.","headline":"Broad, consistent KD gains from an in-context retrieval regularizer, but the central retrieval mechanism is never ablated and the SOTA claim is too strong.","tokens_in":19355,"tokens_out":2890,"would_cite":false,"duration_ms":27391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that regularizing a student with teacher logits of retrieved in-context neighbors—same-class positives pulled close, different-class negatives pushed apart—improves accuracy across offline, online, and teacher-free…","keywords":["knowledge distillation","in-context sample retrieval","label smoothing regularization","contrastive learning","feature memory bank","model compression","image classification","semantic segmentation"],"falsifier":"Replace teacher-feature retrieval with random same-class neighbor selection and re-run the IC-KD protocol on CIFAR-100; if accuracy stays the same, the retrieval mechanism is not what drives the reported gains.","tokens_in":18383,"feed_emoji":"🔍","tokens_out":8015,"duration_ms":73329,"temperature":0.7,"pith_summary":"Conventional distillation asks the student to imitate the teacher's output on each individual sample. This paper argues that the knowledge worth transferring also includes how a sample relates to its in-context samples—similar examples of the same or different classes. Viewing distillation as learned label smoothing, it proposes IC-KD, which retrieves in-context samples from a teacher feature memory bank and regularizes the student with two extra losses: one aligns the student with an aggregation of same-class teacher logits, and the other repels the student from different-class teacher logits. The paper reports consistent accuracy gains over existing KD baselines on CIFAR-100 and ImageNet, plus gains on Cityscapes segmentation, across offline, online, and teacher-free KD. If the claim holds, distillation is best understood as retrieval-augmented regularization rather than pointwise imitation.","feed_headline":"Distilling from retrieved neighbors beats per-sample distillation","feed_subtitle":"Same-class and different-class neighbor logits add regularization in offline, online, and teacher-free KD.","key_machinery":"The load-bearing object is the in-context sample: a set of similar training samples retrieved by cosine similarity in the teacher's feature space and separated by label into same-class positives and different-class negatives. A feature memory bank stores teacher features for the whole training set; for each query sample, Eq. (7) keeps the top-K same-class similarities, softmax-weights them, and aggregates the corresponding teacher logits into one softened target. Eq. (8), positive in-context distillation, is the KL divergence between that aggregated target and the student's prediction, and Eq. (10), negative in-context distillation, is a cosine-separation loss against different-class teacher predictions. The theoretical bridge is the label-smoothing view of distillation: if KD regularizes like label smoothing, then teacher predictions on other samples are legitimate regularizers, not noise to be ignored.","core_discovery":"On its own terms, the central discovery is that a student trained to match the teacher on a sample's retrieved in-context neighbors learns better than a student trained to match only per-sample outputs. For each sample, IC-KD builds a teacher feature memory bank, retrieves top-K same-class and top-N different-class neighbors by cosine similarity, and forms an aggregated teacher soft target from the positive neighbors. Positive in-context distillation (PICD) minimizes the KL divergence between that aggregated target and the student prediction, while negative in-context distillation (NICD) uses cosine separation to push the student away from different-class teacher predictions. The paper reports top-1 accuracy improvements over CRD and other baselines on CIFAR-100 and ImageNet, and mIoU improvements on Cityscapes, in offline, online, and teacher-free distillation settings.","pith_inferences":["The paper leaves implicit that the same retrieval regularizer could be applied in feature space; its own limitation section identifies feature-space regularization as future work.","The negative loss's cosine repulsion resembles contrastive learning, so part of the gain may come from sharpening the student's representations rather than from label smoothing; a controlled study that swaps the negative loss for a plain cross-entropy term would separate these effects.","The fixed teacher feature bank could be replaced by a momentum-updated or student-aware bank, which may matter when teacher and student architectures differ strongly.","Since the method needs only a memory bank plus labels, it should transfer to semi-supervised or self-distillation settings without an external teacher, though the paper does not test this."],"forward_implications":["Any existing KD objective can be augmented with positive and negative in-context losses without changing the teacher or student architectures.","The label-smoothing framing predicts that the identity of the source of the aggregated target matters less than the target's quality, which is why the same regularizer transfers to teacher-free and online settings.","Retrieval from a precomputed teacher feature memory bank keeps the added cost mostly offline, so the method scales to ImageNet-sized problems.","Because the gains appear on classification and segmentation, in-context retrieval regularization is a general training signal rather than a task-specific trick."],"supporting_citations":[{"why":"Defines the vanilla KD loss that IC-KD augments with in-context regularization.","marker":"[14]"},{"why":"Contrastive representation distillation baseline whose training protocol and results IC-KD compares against.","marker":"[22]"},{"why":"Provides the learned-label-smoothing view of KD that motivates using teacher predictions as regularizers.","marker":"[28]"},{"why":"Establishes the regularization benefit of label smoothing that the paper extends to in-context soft targets.","marker":"[27]"},{"why":"Supplies the infoNCE cosine-similarity formulation adapted for the negative in-context distillation loss.","marker":"[52]"},{"why":"Cross-image relational KD baseline for semantic segmentation whose protocol and results IC-KD uses for Cityscapes.","marker":"[16]"},{"why":"Motivates retrieval of optimal in-context samples in visual in-context learning, which IC-KD adapts to distillation.","marker":"[51]"}],"fun_headline_variants":["Retrieve teacher's neighbors to distill better students","In-context sample retrieval boosts distillation performance","Distilling from neighbor logits outperforms per-sample KD","IC-KD: Learn from similar and dissimilar samples for KD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that images judged similar by the teacher's feature space are the right images to borrow soft labels from, even after the student's own representation has drifted away from the teacher's during training.","fun_headline_variants_meta":{"raw":{"variants":["Retrieve teacher's neighbors to distill better students","In-context sample retrieval boosts distillation performance","Distilling from neighbor logits outperforms per-sample KD","IC-KD: Learn from similar and dissimilar samples for KD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1438,"prompt_tokens":1004,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":380}},"tokens_in":620,"tokens_out":434,"duration_ms":5057,"temperature":1.0,"reasoning_tokens":380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:15.993130+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace teacher-feature retrieval with random same-class neighbor selection and re-run the IC-KD protocol on CIFAR-100; if accuracy stays the same, the retrieval mechanism is not what drives the reported gains.","supporting_citations":[{"cited_title":"Revisiting knowledge distillation via label smoothing regularization,","cited_arxiv_id":null,"evidence_quote":"Provides the learned-label-smoothing view of KD that motivates using teacher predictions as regularizers."},{"cited_title":"When does label smoothing help?","cited_arxiv_id":null,"evidence_quote":"Establishes the regularization benefit of label smoothing that the paper extends to in-context soft targets."},{"cited_title":"What makes good examples for visual in- context learning?","cited_arxiv_id":null,"evidence_quote":"Motivates retrieval of optimal in-context samples in visual in-context learning, which IC-KD adapts to distillation."}],"review_version":1}