{"id":"e53ecbe3-7c9c-43c9-a0d9-64dea23947c8","arxiv_id":"2412.11017","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Displacement Knowledge Distillation matches the distribution of feature differences between teacher and student, and a dual distillation network applies it to novel classes in few-shot class-incremental learning, achieving state-of-the-art retention on CIFAR-100, miniImageNet, and CUB-200.","lead":"A machine learning paper proposes Displacement Knowledge Distillation (DKD), which transfers knowledge between models by matching distributions of sample-to-sample differences instead of sample similarities. Combined with a dual-branch network for few-shot class-incremental learning, it reports state-of-the-art accuracy on three image benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DKD's central gain rests on matching displacement distributions of a few-shot teacher; the paper neither bounds the teacher noise nor controls for it, and its own Table III shows a ~10-point branch drop exactly when DKD starts.","rationale":"I read the paper as claiming DKD itself is the reason for better retention. The weakest point is not whether displacement encodes more structure than similarity—that argument is plausible—but whether the specific teacher that FSCIL provides is good enough to make those displacement targets worth matching. The paper's own Sec. V-D flags the session-2 degradation, and no analysis or ablation isolates teacher quality. This is internally distinguishable from the central claim: DKD could be a good loss with a reliable teacher yet fail in FSCIL because the teacher is unreliable. The empirical evidence is otherwise consistent (Table I shows SOTA KR; Table II shows DKD>RKD in the final configuration), so I do not think the paper should be rejected, but it should not be accepted without either bounding the teacher error or adding a teacher-quality control. The reader's verdict CONDITIONAL remains appropriate; my stress-test does not move it.","tokens_in":20581,"tokens_out":12745,"duration_ms":118279,"concrete_test":"Teacher-replacement ablation on CIFAR-100: retrain the session-1 model as in the paper, and also train a reference teacher for the same 5 novel classes using full supervision; compute held-out displacement distributions from both teachers and measure their distance (e.g., MMD/KL). Then run DDNet with each teacher as the DKD target and compare session-2 branch accuracy and final KR over at least three seeds. If the few-shot teacher's displacement distribution is far from the full-data reference and the stronger teacher removes the session-2 drop, teacher noise is the load-bearing failure; if results are equivalent, the assumption holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that displacement knowledge distillation preserves structural information better than similarity-based distillation. For that to hold in FSCIL, the teacher displacement vectors z^{τ-1}_{f,i} - z^{τ-1}_{f,j} in Eqs. (5)-(6) must be reliable distillation targets. But the teacher at session τ-1 has seen only K shots per novel class, and the paper provides no bias/variance analysis or error bound for these targets. Its robustness experiment (Table III, Sec. V-D) shows the DDNet current-branch accuracy dropping by roughly ten points between sessions 1 and 2 (70.11 to 60.50 at 0% outliers), exactly when DKD begins; the paper attributes this to a 'semantic gap' but the pattern is also exactly what noise propagation from a weak teacher would predict. The theory in Sec. IV-B/IV-C counts polluted gradient terms but never tracks how teacher displacement error enters the student through the KL loss. Without controlling for teacher quality, the observed final-KR improvement could come from the dual-branch/selector architecture rather than from DKD's structural target, and the paper's own admitted early-session fragility remains unresolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Displacement Knowledge Distillation (DKD), a relational distillation objective that matches distributions of pairwise displacement vectors between teacher and student features, and combines it with individual logit distillation in a Dual Distillation Network (DDNet) for few-shot class-incremental learning. The authors argue that displacement preserves more structural information than similarity-based distillation, that it is more robust to outliers, and that a dual-branch design with a learned sample selector improves knowledge retention. The method is evaluated on CIFAR-100, miniImageNet, and CUB-200, with additional experiments on general class-incremental learning and standard knowledge distillation.","tokens_in":20800,"tokens_out":8232,"duration_ms":65534,"significance":"If the empirical claims hold, DKD is a simple and potentially general distillation objective: it is well-defined, it has a clean gradient interpretation, and the paper demonstrates consistent gains over IKD and RKD in FSCIL, CIL, and model-compression settings. The dual-branch design with an instance-aware selector is a reasonable response to the base/novel performance gap, and the robustness analysis is a useful addition. The main weaknesses are reporting inconsistencies in the central ablation table and an unresolved confound between the DKD objective and teacher quality in the few-shot setting; these need to be addressed before the state-of-the-art claim can be accepted.","major_comments":[{"comment":"The KR column in Table II is not computed according to the definition given in Section V-B. KR is defined as Acc_tau / Acc_0 x 100%, but the first row of Table II reports KR = 62.34% while 51.61 / 76.97 = 67.05%; the reported value is instead the mean of the nine per-session accuracies (561.04 / 9 = 62.34). This discrepancy affects every ablation row and the text's quantitative claims about KR gains, so all ablation KR values need to be recomputed from the final-session accuracy and the conclusions rechecked.","section":"Section V-B / Table II"},{"comment":"The theoretical analysis in Eqs. (23)-(24) counts how many gradient terms an outlier pollutes, but it does not model how errors in the teacher's displacement targets propagate to the student through the KL loss. The 0%-outlier row of Table III shows the current branch dropping from 70.11 at session 1 to 60.50 at session 2, which is exactly the first session where DKD operates on the pre-order dataset; the paper attributes this to a 'semantic gap,' but it is equally consistent with a noisy few-shot teacher. Without a control that matches teacher quality or ablates DKD at session 2, the final KR improvement cannot be cleanly attributed to DKD's structural target rather than to the dual-branch/selector architecture.","section":"Sections IV-B, IV-C and V-D"},{"comment":"The outlier-robustness conclusion is based only on current-branch accuracy, while BiDist and DDNet differ in their objective functions and branch design. This makes the claimed reduction in average accuracy drop from 0.986% to 0.526% potentially confounded. The authors should report the outlier-induced accuracy drop for the full DDNet model and for a baseline that differs only in the distillation loss, so that the robustness gain can be attributed to DKD.","section":"Section V-D / Table III"}],"minor_comments":[{"comment":"The subscripts and superscripts in Eq. (6) are inconsistent (z^tau_{f i} vs. z^{tau-1}_{f i}); define z^tau_{f,i} and z^{tau-1}_{f,i} explicitly as logits of the i-th and j-th samples.","section":"Eq. (6)"},{"comment":"The caption of Fig. 1 contains a duplicated older caption passage ('Figures a and b show two different structures...') that should be removed.","section":"Fig. 1 caption"},{"comment":"The sentence 'respectively leading FCIL (M=2), BiDist (M=1), and BiDist (M=5) by 0.94%, 2.32%, 2.64%' does not match the numbers in Table I; for example, the average KR difference between Ours (M=1) and FCIL is 0.94% if compared with the average KR, but FCIL has no M parameter in the table, and the other quoted differences also need to be verified against the recomputed averages.","section":"Section V-B"},{"comment":"The phrase 'considering the opposite direction as the same pair' is confusing because Eq. (20) sums over ordered pairs and yields N(N-1) terms; please clarify whether the count refers to ordered or unordered pairs.","section":"Section IV-C"},{"comment":"The statement that RKD and DKD increase Acc_n by averages of 1.33% and 4.81% is not directly verifiable from Table II, which reports only the session-8 Acc_n, not per-session averages; provide the per-session Acc_n curves or state explicitly how the averages were computed.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical paper with a clear central idea, but the metric inconsistency in the central ablation table and the unresolved teacher-quality confound are load-bearing issues. I would ask the authors to recompute all KR-related numbers, add a control experiment that isolates the DKD effect from the sample selector, and clarify the robustness comparison before considering acceptance. I do not see grounds for rejection: the method is well-motivated, the experiments are broad, and the reported gains are plausible if the ablations are recomputed correctly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take: this paper has a genuinely new idea—distilling the distribution of displacement vectors (z_i - z_j) instead of similarities or logits—and it backs it with consistent gains across three FSCIL benchmarks, plus transfer experiments on general CIL and model compression. The core loss in Eq. (20) is not in the RKD/BiDist/ERL line; the dual-branch scheme (IKD for base, DKD for novel) is a reasonable extension of BiDist. The authors did the right experiments: ablations, robustness, and a clean model-compression test where DKD beats RKD variants, which suggests the metric itself, not just the FSCIL setup, is doing the work.\n\nThat said, there are three soft spots, in increasing order of seriousness.\n\nMinor: the gradient analysis in Sec. IV-C claims an outlier pollutes 1/N of DKD pairs vs 100% for RKD. Counting differences, a single outlier appears in about 2(N-1) terms, so closer to 2/N. Not a big deal, but the paper should be accurate.\n\nMore serious: Table II's KR column is not the defined metric. KR is defined as Acc_tau/Acc_0, but the reported values are the mean session accuracy. For the first ablation row, 51.61/76.97 = 67.05%, yet they list 62.34%. This is a load-bearing reporting error—KR is the headline metric—and it makes me wonder if other numbers have the same issue.\n\nMost serious: the robustness experiment in Table III shows DDNet's current-branch accuracy dropping about 10 points from session 1 to 2 at 0% outliers (70.11 to 60.50), while BiDist drops only about 3. The paper attributes this to a \"semantic gap,\" but it happens exactly when DKD starts, and a plausible alternative is that the few-shot teacher's displacement distributions are noisy in the early sessions. The authors provide no bound or sensitivity analysis for teacher quality. This is not fatal—final accuracy is still good, and the CIL/compression results are positive—but it needs to be addressed, not hand-waved.\n\nAlso, no code and no error bars. For a paper claiming SOTA, that's non-negotiable in 2024.\n\nBottom line: this paper deserves a serious referee. The idea is novel, the experiments are broad, and the flaws are addressable. I'd send it out and ask the authors to fix the KR definition, provide code and multi-seed runs, and explain the session-2 dip. My own verdict on the current version would be conditional, not reject.\n\nI'd bring it to the reading group—the displacement-loss idea is simple enough to test quickly.","headline":"Genuinely new distillation loss with real empirical gains, but the KR metric inconsistency, missing code/error bars, and the unexplained session-2 drop need to be fixed before the headline numbers are fully trustworthy.","tokens_in":21393,"tokens_out":4425,"would_cite":true,"duration_ms":38820,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that matching displacement vectors between sample pairs, instead of similarity scores, preserves structural knowledge better during few-shot class-incremental learning, and that splitting distillation by class type…","keywords":["few-shot class-incremental learning","knowledge distillation","displacement distillation","relational knowledge distillation","catastrophic forgetting","structural information","dual distillation network"],"falsifier":"Run the CIFAR-100 FSCIL protocol with the teacher's novel-class features deliberately corrupted (for example, permute the features of a random 10% of novel samples before computing displacement targets): if knowledge retention does not drop, the robustness claim would be contradicted; alternatively, train the same DDNet with DKD applied only from session 3 onward and compare with DKD from session 2, and if the early-session accuracy drop at session 2 is not recovered, the advantage attributed to DKD during the first incremental step fails to appear.","tokens_in":20315,"feed_emoji":"🧠","tokens_out":9860,"duration_ms":76224,"temperature":0.7,"pith_summary":"Few-shot class-incremental learning (FSCIL) asks a model to keep recognizing previously learned classes while learning new classes from just a handful of examples, and knowledge distillation is the usual tool to avoid forgetting. Classic distillation matches the teacher's and student's outputs for each sample, while structured distillation matches similarity scores between sample pairs; this paper argues that both discard information that matters in the few-shot regime. The central claim is that matching the displacement vector between two samples—the pointwise difference of their features, treated as a probability distribution—preserves distance and direction information that a scalar similarity collapses, and that this is especially valuable for the few-shot novel classes whose features are poorly learned. Building on that, the paper proposes a dual network that distills base-class knowledge from output scores and novel-class knowledge from displacements, with a learned sample selector deciding how much to trust each branch at inference. The authors report that this design holds more knowledge across sessions than prior methods on CIFAR-100, miniImageNet, and CUB-200, and that the displacement loss tolerates outlier samples better than similarity-based relational distillation.","feed_headline":"Displacement beats similarity in few-shot knowledge retention","feed_subtitle":"Matching differences between sample pairs, not similarities, cuts forgetting when new classes appear with few examples.","key_machinery":"The load-bearing object is the displacement vector $z_i - z_j$ between two samples' feature vectors, which the method normalizes into a probability distribution and matches between teacher and student with KL divergence. Unlike an inner-product similarity, which collapses a pair to a scalar, the displacement lives in the original $d$-dimensional feature space and keeps both the magnitude and the direction of the relation between samples. The second load-bearing mechanism is the dual-branch design: per-sample logit matching (IKD) is applied to the base classes because their features are already discriminative, while DKD is applied to the novel classes because their few-shot features are not, and an instance-aware sample selector trained with triplet and binary cross-entropy losses over momentum-updated base/novel prototypes weights the two branches at test time.","core_discovery":"Structured knowledge distillation for FSCIL has been done with relational knowledge distillation (RKD), which measures a sample pair by their similarity (inner product or distance) and matches the teacher's and student's similarity matrices. This paper's discovery is that the right structural quantity to distill is the displacement between samples: for any pair of feature vectors $z_i, z_j$, the displacement $z_i - z_j$ is a $d$-dimensional structure that retains the full geometric relation, including direction and magnitude, whereas similarity reduces the relation to a single scalar. DKD turns each displacement into a probability distribution and minimizes the KL divergence between the teacher's and the student's displacement distributions, which multiplies the number of matched teacher-student pairs by $N-1$ and removes the coupling that makes an RKD gradient fully contaminated by any single outlier. Observing a performance gap between base and novel class features, the paper assigns per-sample logit matching (IKD) to the base classes and DKD to the novel classes within one network, DDNet, whose instance-aware sample selector fuses the two streams during inference. On three benchmarks the method raises knowledge retention to 69.39%, 72.91%, and 76.84%, and under an outlier-attack experiment the average accuracy drop falls from 0.986% to 0.526%.","pith_inferences":["Editorial inference: the robustness argument generalizes beyond FSCIL: under a single outlier, an $N(N-1)$-term displacement loss has only $1/N$ of its terms contaminated, while a similarity matrix couples every term, so displacement-based losses should be preferred in any noisy or adversarial distillation setting.","Editorial inference: the transient ~10% accuracy drop at session 2, which the paper itself reports when DKD first activates, suggests the few-shot teacher's displacement distribution is noisy early on; a natural extension is to anneal the DKD weight or gate it on a teacher-confidence estimate during the first incremental sessions.","Editorial inference: the sample selector adds triplet loss, momentum prototypes, and binary cross-entropy yet improves knowledge retention by only about half a percentage point in ablations; a simpler fixed or distance-based branch weight may capture most of the benefit, which would be worth testing.","Editorial inference: applying DKD to normalized features (removing magnitude) would isolate whether the gains come from distance information or from direction information; if retention collapses under normalization, the distance component is doing the work."],"forward_implications":["Replacing similarity-based relational distillation with displacement distillation should improve any incremental-learning method that currently matches sample similarities; the paper shows gains when DKD replaces the standard loss inside LwF, iCaRL, and WA on general class-incremental learning.","The displacement formulation densifies supervision by giving each sample $N-1$ matched pairs instead of one, which should be especially useful when training data are scarce, without needing synthetic samples.","Splitting the distillation strategy by class domain—logit matching for well-trained base classes, displacement matching for few-shot novel classes—should extend to other base/novel splits beyond the three benchmarks tested.","Outside incremental learning, displacement matching should compete with relational KD as a general model-compression tool; on CIFAR-10 it beats RKD variants in four of five teacher-student configurations."],"supporting_citations":[{"why":"Defines the FSCIL problem and the benchmark protocol (60 base classes plus 40 novel classes in a 5-way 5-shot setup) that the experiments and dataset splits build on.","marker":"[4]"},{"why":"Supplies the standard knowledge distillation loss (KL divergence over softened outputs) that DKD reuses on displacement distributions.","marker":"[5]"},{"why":"Defines relational knowledge distillation (RKD), the similarity-based structural distillation that DKD contrasts with and whose IKD/RKD trichotomy the paper adopts.","marker":"[6]"},{"why":"BiDist, the dual-branch FSCIL baseline that DDNet extends; its M=1 and M=5 results are the direct comparisons in Table I, and its code is used for fair comparison.","marker":"[7]"},{"why":"Applies relation knowledge distillation to FSCIL via an exemplar relation graph; the direct predecessor that motivates relational structure preservation in this setting.","marker":"[8]"},{"why":"Provides the empirical basis for the paper's claim that individual logit distillation is suboptimal for few-shot novel classes, motivating the displacement loss for that branch.","marker":"[53]"}],"fun_headline_variants":["Displacement distillation beats similarity for few-shot learning","New distillation method uses sample differences to prevent forgetting","DDNet: Dual distillation for few-shot class incremental learning","Displacement knowledge: a better way to distill in few-shot settings","From similarity to displacement: boosting few-shot class incremental learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The teacher model trained on a few samples per novel class must produce displacement patterns that are reliable distillation targets; if the few-shot teacher's displacement vectors are noisy or uninformative, the DKD loss teaches the student to reproduce that noise, and the paper offers no bound or analysis on that teacher noise.","fun_headline_variants_meta":{"raw":{"variants":["Displacement distillation beats similarity for few-shot learning","New distillation method uses sample differences to prevent forgetting","DDNet: Dual distillation for few-shot class incremental learning","Displacement knowledge: a better way to distill in few-shot settings","From similarity to displacement: boosting few-shot class incremental learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1689,"prompt_tokens":1053,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":669,"tokens_out":636,"duration_ms":5690,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:23:16.396429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the CIFAR-100 FSCIL protocol with the teacher's novel-class features deliberately corrupted (for example, permute the features of a random 10% of novel samples before computing displacement targets): if knowledge retention does not drop, the robustness claim would be contradicted; alternatively, train the same DDNet with DKD applied only from session 3 onward and compare with DKD from session 2, and if the early-session accuracy drop at session 2 is not recovered, the advantage attributed to DKD during the first incremental step fails to appear.","supporting_citations":[{"cited_title":"Few-shot class-incremental learning,","cited_arxiv_id":null,"evidence_quote":"Defines the FSCIL problem and the benchmark protocol (60 base classes plus 40 novel classes in a 5-way 5-shot setup) that the experiments and dataset splits build on."},{"cited_title":"Relational knowledge distilla- tion,","cited_arxiv_id":null,"evidence_quote":"Defines relational knowledge distillation (RKD), the similarity-based structural distillation that DKD contrasts with and whose IKD/RKD trichotomy the paper adopts."},{"cited_title":"Few- shot class-incremental learning via class-aware bilateral distillation,","cited_arxiv_id":null,"evidence_quote":"BiDist, the dual-branch FSCIL baseline that DDNet extends; its M=1 and M=5 results are the direct comparisons in Table I, and its code is used for fair comparison."},{"cited_title":"Few- shot class-incremental learning via relation knowledge distillation,","cited_arxiv_id":null,"evidence_quote":"Applies relation knowledge distillation to FSCIL via an exemplar relation graph; the direct predecessor that motivates relational structure preservation in this setting."},{"cited_title":"Incremental few-shot learning via vector quantization in deep embedded space,","cited_arxiv_id":null,"evidence_quote":"Provides the empirical basis for the paper's claim that individual logit distillation is suboptimal for few-shot novel classes, motivating the displacement loss for that branch."}],"review_version":1}