Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning

Deyu Meng; Quanziang Wang; Renzhen Wang; Wenyang Liao; Yichen Wu

REVIEW 4 major objections 5 minor 18 references

Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Data-Distill-Net claims that replay-based continual learning improves when a hypernetwork distills the memory buffer into learnable soft labels, and demonstrates consistent gains on four baselines.

desk verdict The plug-in idea is new and the experiments are broad, but the implemented algorithm does not match the proposed objective, so the reported gains are not yet attributable to the method. read the letter →

arxiv 2505.20135 v2 pith:G4WULR3Q submitted 2025-05-26 cs.LG

Wenyang Liao , Quanziang Wang , Yichen Wu , Renzhen Wang , Deyu Meng This is my paper

classification cs.LG

keywords continuallearningcatastrophicforgettingreplay-basedmethodsdatasetdistillationsoftlabelshypernetworkbi-leveloptimizationmemorybuffer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Replay-based continual learning assumes that a small memory buffer can stand in for all previously seen data, but limited capacity and heuristic selection make that assumption shaky. This paper argues that the fix is not to choose better samples but to distill the buffer itself: at each task, a hypernetwork called Data-Distill-Net (DDN) rewrites the buffer's one-hot labels into learnable soft labels, trained so that a model trained on the distilled buffer behaves like a model trained on the whole history. The paper shows this soft-label distillation is equivalent, via gradient matching, to directly optimizing the entire buffer as parameters, while costing far less. Plugging DDN into four replay baselines (ER, DER++, CLSER, ER-ACE) improves average accuracy and lowers forgetting on Split CIFAR-10, Split CIFAR-100, and Split Tiny-ImageNet in both online and offline settings. A sympathetic reader would care because DDN is a drop-in plugin that attacks the root assumption of replay methods rather than adding another selection heuristic.

What carries the argument

The load-bearing object is the hypernetwork Data-Distill-Net (DDN), an MLP with two hidden layers of 200 ReLU units that maps classifier logits or probabilities of buffer samples to soft labels. The soft labels are blended with an EMA-cached old generator to stabilise training and then normalized with the one-hot label into a valid probability distribution. The mechanism that carries the argument is the bi-level optimization: an inner loop trains a temporary classifier on the buffer with generated soft labels, and an outer loop updates DDN's weights so that this classifier performs well on the previous buffer plus the current task. Theorems 5.1 and 5.2 show, by first-order Taylor expansion, that both full-buffer parameterization and DDN parameterization are equivalent to minimizing the negative inner product of the buffer gradient and the global-data gradient.

What would settle it

Take a continual-learning benchmark where the reservoir buffer is deliberately made unrepresentative of early tasks (for example, by removing most samples of one old class from the buffer before task n), then compare ER plus DDN against ER trained with full access to the old class data for distillation. If ER plus DDN does not outperform plain ER on the underrepresented class, or if a version of DDN validated on the full historical data clearly beats it, the claim that DDN distills global information from all seen tasks is contradicted. Equivalently, an ablation that replaces the outer-loop validation set $M^{{n-1}}$ ∪ T^n with $T^{{1:n}}$ should show a large accuracy gap if the buffer-proxy assumption is the real bottleneck.

Watch

Extended reading notes

Core claim

The central claim is that catastrophic forgetting in replay-based continual learning can be mitigated by treating the memory buffer as a distillation target rather than a fixed sample set. Instead of parameterizing the whole buffer, Data-Distill-Net (DDN), a two-hidden-layer MLP, consumes each buffered sample's current predicted class probabilities and emits a soft label; the soft labels are the only learnable buffer content. DDN is trained by a bi-level objective whose outer loop validates on the union of the previous buffer and the current task, and Theorem 5.2 proves this objective reduces to matching the gradient of the distilled buffer to the gradient of that union. The paper therefore claims that generating soft labels is functionally equivalent to full-buffer parameterization for distillation, while avoiding the compute and overfitting risks of direct optimization. Empirically, the claim is that applying DDN on top of ER, DER++, CLSER, and ER-ACE improves accuracy and reduces forgetting across standard benchmarks, with the largest gains at small buffer sizes.

Load-bearing premise

The whole method leans on the assumption that the previous memory buffer plus the current task is a faithful enough stand-in for all data ever seen, so the soft labels can learn from information that is actually missing from the buffer.

Editorial extensions

If this is right

Any replay-based baseline can be upgraded by attaching DDN, since DDN does not change how buffer samples are chosen or stored; the paper demonstrates this on ER, DER++, CLSER, and ER-ACE.
At small buffer sizes (0.05K to 0.2K on Split CIFAR-10), the improvements are largest, suggesting soft-label distillation is most valuable exactly when the buffer is least representative.
Because DDN is only used during training, the deployed classifier incurs no extra test-time cost.
The gradient-matching equivalence means the benefit should transfer to other replay losses that are differentiable in the classifier parameters, not only cross-entropy.
Distilling the previous buffer together with the current task preserves cross-task relationships that per-task dataset distillation (for example Mnemonics and LoDM) misses, which is the paper's stated reason for better continual-learning performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper leaves open is combining DDN with coreset-selection strategies rather than reservoir sampling, so that the outer-loop validation set is more representative of all past tasks; the theory does not require reservoir sampling specifically.
The soft labels DDN produces encode inter-class relationships, so DDN could be interpreted as a continual, data-dependent label-smoothing mechanism; that view suggests it may also improve calibration of the final classifier.
The gradient-matching view opens the possibility of applying the same DDN idea to other replay objectives, such as logit distillation or contrastive losses, with the same bi-level derivation.
If the buffer is strongly biased, the outer loop's validation set M^{n-1} ∪ T^n may not expose missing old-class information, so the distilled soft labels would inherit rather than repair the bias; this is the paper's own weakest assumption and a concrete risk in long task sequences.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read the paper. The plug-in idea is coherent and worth taking seriously: a lightweight hypernetwork that maps classifier predictions to soft labels for buffer data, trained by bi-level optimization, slots into ER/DER++/CLSER/ER-ACE. That combination is new relative to the cited distillation literature. The empirical spread is broad—three datasets, two settings, four baselines—and there are useful ablations (random soft labels, label smoothing, direct label optimization). The per-task accuracy and probability-distribution analyses give some insight into why soft labels might help. If the numbers are reproducible, this would be a practical contribution.

However, there is a load-bearing inconsistency between the proposed objective and the implementation. Eqn (6) defines the inner loop as training a temporary classifier only on soft labels generated by the current G^n_omega over the current buffer, and the outer loop as validating on M^{n-1} ∪ T^n. Algorithm 2 instead samples from M^n, uses the stale generator G^{n-1}_omega, and adds a hard-label loss in the inner step. Appendix A's Eqn (11) writes the inner update with only soft labels from the current generator and no hard-label term. As written, no part of the paper optimizes Eqn (6). The reported gains therefore cannot be attributed to the proposed soft-label distillation; the extra hard-label loss, the reservoir-updated inner distribution, or the omitted EMA could explain them. The theoretical equivalence in Theorems 5.1-5.2 is a standard bi-level-to-gradient-matching Taylor expansion around a one-step SGD update; it does not bound the approximation error and does not fix the algorithm-objective mismatch.

The experiments also have weaker points: hyperparameters appear selected on test performance, no error bars are reported, and the code is not yet available. Those are fixable, but until they are fixed the central claim is not established.

This paper is for a continual-learning reader who wants the plug-in idea and is willing to help reconcile the algorithm with the stated objective. I would send it to peer review—the idea is plausible and the issues are addressable—but I would ask for a major revision that clarifies exactly what is optimized, adds error bars, and releases code. With that, the empirical story might hold.

Referee Report

4 major / 5 minor

Summary. The paper proposes Data-Distill-Net (DDN), a lightweight hypernetwork that generates soft labels for memory-buffer samples in replay-based continual learning. The stated goal is to distill global information from the current task and the previous buffer via a bi-level optimization objective, and the paper claims that DDN is theoretically equivalent to directly parameterizing the buffer. The method is evaluated as a plug-in for ER, DER++, CLSER, and ER-ACE on Split CIFAR-10, Split CIFAR-100, and Split Tiny-ImageNet under online and offline settings, reporting accuracy gains and reduced forgetting in many configurations. The paper also includes ablations on the trade-off parameter, the EMA weight, small buffer sizes, and a comparison with other soft-label baselines.

Significance. If the method works as described, it would be a useful and low-cost addition to replay-based continual learning: it avoids parameterizing the entire buffer and instead learns soft labels, which is a plausible way to improve old-task retention. The experimental study is broad, covering four baselines, two settings, three datasets, several buffer sizes, and multiple ablations, and the planned code release is a positive feature. However, the central claim is not currently established because the pseudocode does not optimize the stated objective, and the theoretical equivalence results rely on unsupported approximations. The empirical gains may therefore be due to confounds rather than the proposed distillation mechanism.

major comments (4)

[Section 4.2 / Appendix A / Algorithm 2] Algorithm 2 does not implement the objective in Eqn (6). Eqn (6) and Appendix A define the inner update as a single SGD step on L(f_theta; {X^n_buf, Y_tilde^n_buf(omega)}) with no hard-label term (Eqn 11), while Algorithm 2's inner loop minimizes L(f_theta; {X_in, Y_in}) + alpha * L(f_theta; {X_in, Y_tilde_in(omega)}), where Y_tilde_in comes from the stale generator G^{n-1}_omega and X_in is sampled from M^n. The reported gains are therefore not attributable to the proposed soft-label distillation; they could be caused by the extra hard-label loss, the reservoir-updated buffer distribution, or the previous generator. In addition, Algorithm 1 and Algorithm 2 omit the EMA combination defined in Eqn (8), so the beta ablation in Fig. 4 does not correspond to any pseudocode. The authors should either correct Algorithm 2 to optimize Eqn (6) and rerun the experiments, or rewrite both the method and the theoretical claims to match the algorithm actually used.
[Section 5 / Appendix B] The proofs of Theorems 5.1 and 5.2 replace the inner argmin with a single SGD step and then apply a first-order Taylor expansion (Eqns 13-15 and 18-20) without any bound on the approximation error. This does not establish the claimed equivalence between the bi-level frameworks and the gradient-matching objectives. Furthermore, even if each bi-level problem is approximately equivalent to its gradient-matching form, the claimed equivalence between direct buffer parameterization (Eqn 4) and hypernetwork parameterization (Eqn 6) does not follow, because the two gradient-matching objectives are optimized over different parameter spaces and no argument shows that their optima coincide. The chain of approximate equalities in Section 5 is therefore unsupported. Please either provide a rigorous statement with error bounds or substantially weaken the theoretical claims to describe a heuristic approximation.
[Section 4.3 and Eqn (5)] The architectural description of DDN is inconsistent with the mathematical formulation. Section 4.3 states that DDN takes the predicted probabilities from the classifier network (with gradients stopped) as input, whereas Eqn (5), Algorithm 2, and the proof of Theorem 5.2 define \tilde Y^n_buf(omega) = G^n_omega(X^n_buf), i.e., as a function of the raw buffer input X. If the actual input is the classifier's predicted probabilities, then the gradient computations and the bi-level derivation are incorrect as written, because G would depend on theta. If the actual input is X^n_buf, then the architecture description should be corrected. This discrepancy must be resolved before the method can be reproduced or the theory assessed.
[Section 4.2, Eqn (6)] The outer-loop validation set in Eqn (6) is M^{n-1} ∪ T^n, not the full set of previously seen task data T^{1:n}. In the failure mode that motivates the paper, an unrepresentative buffer, DDN has no access to the missing information and may simply inherit the buffer's bias. The abstract and Section 4.1 claim that DDN 'distills global information from all seen tasks,' but the objective only uses the buffer plus the current task. Please either justify the 'global information' claim more carefully or add experiments that manipulate buffer representativeness to show when the method can and cannot recover the true distribution.

minor comments (5)

[Title and throughout] The title uses 'Reply-based' but the body consistently uses 'Replay-based'; please make the terminology uniform.
[Algorithm 2] Algorithm 2 does not clarify whether theta(omega) is a temporary clone of the classifier network or the actual classifier theta from Algorithm 1; please state this explicitly to avoid confusion about which network is updated in the inner loop.
[Table 2] The claim that DDN improves baselines 'across almost all datasets and baselines' should be qualified: on Split Tiny-ImageNet in the offline setting, ER-ACE DDN is worse than ER-ACE on both ACC and FM (e.g., M=5K: 37.67 vs 38.92 ACC, 39.37 vs 28.65 FM). These cases should be discussed rather than glossed over.
[Eqn (8) and Fig. 4] The EMA variable Gold is used in Eqn (8) before it is formally defined; please define Gold in the text or in a footnote. Also, the beta axis in Fig. 4(b) appears to range from 0.0 to 1.0, but the tick labels are not readable in the current figure.
[Eqn (7) and Eqn (6)] Eqn (7) introduces alpha as the trade-off weight for the classifier update, but Eqn (6) has no alpha inside the inner objective; please clarify whether alpha is also used in the DDN inner loop and how it relates to Eqn (11).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: soft labels are meta-optimized on hard-labeled validation data and evaluated on held-out test sets; Algorithm 2's divergence from Eqn (6) is a correctness gap, not a circular reduction.

full rationale

The claimed derivation chain is not circular. In Eqn (6), the DDN parameters ω are meta-optimized so that a classifier trained (inner loop) on generated soft labels Ỹ^n_buf(ω) = G^n_ω(X^n_buf) minimizes, in the outer loop, a hard-label loss on M^{n-1} ∪ T^n, and all reported ACC/FM numbers are measured on held-out test splits of Split CIFAR-10/100 and Tiny-ImageNet; hence no reported 'prediction' reduces to a fitted value or to the training objective. One-hot hard labels are a reachable special case of the soft-label parameterization (the paper adds the one-hot label to ỹ before normalization), so the gains over ER/DER++/CLSER/ER-ACE are non-tautological empirical outcomes. Theorems 5.1 and 5.2 are proven in Appendix B via a one-step inner SGD update plus a first-order Taylor expansion; they are self-contained, and the self-citations in the paper (e.g., Wang et al. 2023 in Appendix A for gradient-based bi-level updates) are standard methodology references, not load-bearing premises. The 'global information' claim rests on the assumption that the reservoir buffer M^{n-1} plus T^n is a faithful proxy for T^{1:n} — a modeling assumption, not a definitional collapse. The one serious gap is a missing-support issue, not circularity: Algorithm 2's inner loop updates θ(ω) by minimizing L(fθ; {X_in, Y_in}) + α·L(fθ; {X_in, Ỹ_in(ω)}), with soft labels from the stale generator G^{n-1}_ω and an extra hard-label replay term, whereas Eqn (6) and Appendix A Eqn (11) define the inner objective using only soft labels from the current G^n_ω with no hard-label term. As written, the paper never optimizes Eqn (6), so the reported ER DDN improvement (36.05→41.93, Split CIFAR-10, M=0.2K) cannot be attributed to the claimed soft-label distillation objective; it could stem from the auxiliary hard-label loss or from the reservoir-updated M^n. This severs the link from derivation to empirical result but is an internal inconsistency, not a reduction of a result to an input, so it does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim depends on the soft-label parameterization, the buffer-as-validation assumption, and the single-step inner approximation in the theory. None of these are externally validated. The only free parameters are hyperparameters tuned on the test set, which introduces a risk of optimistic results.

free parameters (4)

alpha (trade-off weight) = 1.0 (Split CIFAR-10, M=0.2K)
Trade-off weight between the new task loss and the buffer loss in Eqn (7). Chosen by sweeping in Fig 4(a) on the test set, then fixed for all other experiments.
beta (EMA weight) = 0.9
EMA weight for the old DDN checkpoint in Eqn (8). Chosen by sweeping in Fig 4(b) on the test set.
DDN learning rates = 0.001 (CIFAR-10), 0.01 (CIFAR-100), 0.0001 (Tiny-ImageNet)
Per-dataset Adam learning rates for the hypernetwork, selected manually without sensitivity analysis.
DDN architecture = MLP with 2 hidden layers of 200 units, ReLU
Architecture chosen by hand; no ablation over width or depth is reported.

assumptions (4)

ad hoc to paper The inner argmin in Eqn (4)/(6) can be replaced by a single gradient step, and the Taylor expansion error is negligible.
This is the step in the proofs of Theorem 5.1 and 5.2 (Appendix B, Eqns (13)-(21)). Without it, the claimed equivalence to gradient matching does not hold.
domain assumption The buffer M^{n-1} plus current task T^n is a faithful validation set for the global empirical risk over all seen tasks.
The outer objective in Eqn (4)/(6) uses M^{n-1} ∪ T^n, not the full T^{1:n}. This is precisely the surrogate that the paper's own motivation says can be unreliable.
domain assumption The hypernetwork G_omega has sufficient capacity to represent the optimal soft labels for this task.
The paper invokes universal approximation in Sec 4.3 but does not analyze the capacity needed for the soft-label mapping in the CL setting.
standard math The outer loss is smooth enough for first-order Taylor expansion to be valid.
Used in Appendix B to derive the gradient matching objective; no regularity conditions are stated or verified.

invented entities (1)

Data-Distill-Net (DDN), a hypernetwork G_omega that maps classifier predictions to soft labels
purpose: Generates learnable soft labels for memory buffer samples, replacing direct buffer parameterization with a lightweight module
Its usefulness is only demonstrated by the paper's own experiments; there is no externally observable prediction or artifact outside the paper that would independently confirm its role.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning." pith.science (2026). https://pith.science/paper/G4WULR3Q

@misc{pith2026250520135,
  author       = {Pith},
  title        = {Pith review of: Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4WULR3Q}},
  note         = {Machine review of arXiv:2505.20135}
}

read the original abstract

Replay-based continual learning (CL) methods assume that models trained on a small subset can also effectively minimize the empirical risk of the complete dataset. These methods maintain a memory buffer that stores a sampled subset of data from previous tasks to consolidate past knowledge. However, this assumption is not guaranteed in practice due to the limited capacity of the memory buffer and the heuristic criteria used for buffer data selection. To address this issue, we propose a new dataset distillation framework tailored for CL, which maintains a learnable memory buffer to distill the global information from the current task data and accumulated knowledge preserved in the previous memory buffer. Moreover, to avoid the computational overhead and overfitting risks associated with parameterizing the entire buffer during distillation, we introduce a lightweight distillation module that can achieve global information distillation solely by generating learnable soft labels for the memory buffer data. Extensive experiments show that, our method can achieve competitive results and effectively mitigates forgetting across various datasets. The source code will be publicly available.

Figures

Figures reproduced from arXiv: 2505.20135 by the authors.

**Figure 1.** Comparison between traditional dataset distillation (a) and our proposed method (b). At the n-th task T n , the distilled data E n in the traditional approach contains only task-specific information from T n , while the samples in Mn fail to capture inter-task relationships as illustrate in (a). In contrast, our method distills samples for the memory buffer Mn from both the current task T n and the previous buffer M… view at source ↗

**Figure 2.** Overall framework of the proposed DDN. By generating refined soft labels for buffer samples, DDN enhances the classifier’s training, thereby mitigating catastrophic forgetting of prior task knowledge. The training algorithms for the classifier and DDN are detailed in Algorithm 1 and Algorithm 2, respectively. for these buffer data. We then update the classifier network fθ using the resulting learnable buffer Mn−1 … view at source ↗

**Figure 3.** Under online CL, comparison of the predicted probability distributions for different class samples between ER and ER DDN(ours) on Split CIFAR-10 with buffer size M = 0.2K. 0.0 0.5 1.0 1.5 2.0 30 35 40 45 50 ACC 25 30 35 40 45 FM ACC FM (a) 0.0 0.5 1.0 30 35 40 45 50 ACC 25 30 35 40 45 FM ACC FM (b) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Under online CL, ACC and FM of different α and β of our method on Split CIFAR-10 with buffer size M = 0.2K. (a) Effect of α with fixed β = 0.9. (b) Effect of β with fixed α = 1.0. the performance of ER, it cannot outperform our method with the Gω, indicating the import…

**Figure 5.** Figure 5: The normalized confusion matrix of ER and ER DDN based on Split CIFAR-10 with buffer size M=0.2K. dataset was partitioned into 10 tasks, each with 20 classes, and its test set contains 10,000 images, referred to as Split Tiny-ImageNet. C.2. Training Details As we afore…

**Figure 6.** Figure 6: The accuracy of each task in the whole training process on Split CIFAR-10 with buffer size M=0.2K . ER DDN(ours) in [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 9 canonical work pages

[1]

Online continual learning with maximal interfered retrieval

Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Cac- cia, M., Lin, M., and Page-Caccia, L. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a. Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y . Gradi- ent based sample selection for online continual learning. Advances in neura...

arXiv
[5]

dataset was partitioned into 10 tasks, each with 20 classes, and its test set contains 10,000 images, referred to as Split Tiny-ImageNet

The normalized confusion matrix of ER and ER DDN based on Split CIFAR-10 with buffer size M=0.2K. dataset was partitioned into 10 tasks, each with 20 classes, and its test set contains 10,000 images, referred to as Split Tiny-ImageNet. C.2. Training Details As we aforementioned in Sec. 6.1, the primary settings of our experiments are follow DER++ (Buzzega...

work page 2020
[6]

Connectionist models of recognition memory: constraints imposed by learning and forgetting functions

Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990a. Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990b. Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and ...

work page 2001
[9]

Improving task-free continual learning by distributionally robust memory evolution

Wang, Z., Shen, L., Fang, L., Suo, Q., Duan, T., and Gao, M. Improving task-free continual learning by distributionally robust memory evolution. In International conference on machine learning, pp. 22985–22998. PMLR, 2022b. Wang, Z., Wu, Y ., Wang, R., Lin, H., Wang, Q., Zhao, Q., and Meng, D. Singular value fine-tuning for few-shot class-incremental lear...

arXiv
[10]

doi: 10.1109/CVPR.2019. 00046. Wu, Y ., Huang, L.-K., Wang, R., Meng, D., and Wei, Y . Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Repre- sentations, 2024a. Wu, Y ., Wang, H., Zhao, P., Zheng, Y ., Wei, Y ., and Huang, L.-K. Mitigating cata...

doi:10.1109/cvpr.2019 2019
[11]

Continual Learning for Segment Anything Model Adaptation

Yang, J., Wu, Y ., Cen, J., Huang, W., Wang, H., and Zhang, J. Continual learning for segment anything model adapta- tion. arXiv preprint arXiv:2412.06418,

work page Pith review arXiv
[12]

Yoon, J., Madaan, D., Yang, E., and Hwang, S. J. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085,

arXiv
[14]

Zhang, L., Zhang, J., Lei, B., Mukherjee, S., Pan, X., Zhao, B., Ding, C., Li, Y ., and Xu, D

doi: 10.1109/CVPR42600.2020.00396. Zhang, L., Zhang, J., Lei, B., Mukherjee, S., Pan, X., Zhao, B., Ding, C., Li, Y ., and Xu, D. Accelerating dataset distillation via model augmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11950–11959,

arXiv 2020

Show all 18 references

[15]

R., and Bilen, H

Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929,

2006 arXiv
[18]

proposed a plugin module for previous dataset distillation methods. For example, based on DM (Zhao & Bilen, 2023), LoDM decomposed the synthetic images as two low-rank matrices to reduce the optimization parameters in image space, which significantly reduces the calculation in...

2023
[1998]

C., Wang, Z., and Lin, D

Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D. Learning a unified classifier incrementally via rebalancing. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 831–839,

2019
[2018]

2017.2773081

doi: 10.1109/TPAMI. 2017.2773081. Liu, Y ., Su, Y ., Liu, A.-A., Schiele, B., and Sun, Q. Mnemon- ics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 12245– 12254,

2017
[2019]

Jiang, Z., Gu, J., Liu, M., and Pan, D. Z. Delving into effec- tive gradient matching for dataset condensation. In 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS), pp. 1–6. IEEE,

2023
[2020]

New insights on reducing abrupt representation change in online continual learning

Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., and Belilovsky, E. New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025,

arXiv
[2021]

E., Li, G., Wang, T., and Feng, J

Yuan, L., Tay, F. E., Li, G., Wang, T., and Feng, J. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902–3910,

2020
[2022]

Optimization of Eqn

11 Supplementary Materials Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning A. Optimization of Eqn. (6) We first recall Eqn. (6) in the main text, representing our pro- posed distillation framework tailored for continual learning, whic...

2017
[2023]

Dual-cba: Improving online continual learning via dual continual bias adaptors from a bi-level optimization perspective

Wang, Q., Wang, R., Wu, Y ., Jia, X., Zhou, M., and Meng, D. Dual-cba: Improving online continual learning via dual continual bias adaptors from a bi-level optimization perspective. arXiv preprint arXiv:2408.13991,

arXiv
[2024]

Wang, T., Zhu, J.-Y ., Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint arXiv:1811.10959,

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Online continual learning with maximal interfered retrieval

Aljundi, R., Belilovsky, E., Tuytelaars, T., Charlin, L., Cac- cia, M., Lin, M., and Page-Caccia, L. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a. Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y . Gradi- ent based sample selection for online continual learning. Advances in neura...

arXiv

[2] [5]

dataset was partitioned into 10 tasks, each with 20 classes, and its test set contains 10,000 images, referred to as Split Tiny-ImageNet

The normalized confusion matrix of ER and ER DDN based on Split CIFAR-10 with buffer size M=0.2K. dataset was partitioned into 10 tasks, each with 20 classes, and its test set contains 10,000 images, referred to as Split Tiny-ImageNet. C.2. Training Details As we aforementioned in Sec. 6.1, the primary settings of our experiments are follow DER++ (Buzzega...

work page 2020

[3] [6]

Connectionist models of recognition memory: constraints imposed by learning and forgetting functions

Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990a. Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2):285, 1990b. Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and ...

work page 2001

[4] [9]

Improving task-free continual learning by distributionally robust memory evolution

Wang, Z., Shen, L., Fang, L., Suo, Q., Duan, T., and Gao, M. Improving task-free continual learning by distributionally robust memory evolution. In International conference on machine learning, pp. 22985–22998. PMLR, 2022b. Wang, Z., Wu, Y ., Wang, R., Lin, H., Wang, Q., Zhao, Q., and Meng, D. Singular value fine-tuning for few-shot class-incremental lear...

arXiv

[5] [10]

doi: 10.1109/CVPR.2019. 00046. Wu, Y ., Huang, L.-K., Wang, R., Meng, D., and Wei, Y . Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Repre- sentations, 2024a. Wu, Y ., Wang, H., Zhao, P., Zheng, Y ., Wei, Y ., and Huang, L.-K. Mitigating cata...

doi:10.1109/cvpr.2019 2019

[6] [11]

Continual Learning for Segment Anything Model Adaptation

Yang, J., Wu, Y ., Cen, J., Huang, W., Wang, H., and Zhang, J. Continual learning for segment anything model adapta- tion. arXiv preprint arXiv:2412.06418,

work page Pith review arXiv

[7] [12]

Yoon, J., Madaan, D., Yang, E., and Hwang, S. J. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085,

arXiv

[8] [14]

Zhang, L., Zhang, J., Lei, B., Mukherjee, S., Pan, X., Zhao, B., Ding, C., Li, Y ., and Xu, D

doi: 10.1109/CVPR42600.2020.00396. Zhang, L., Zhang, J., Lei, B., Mukherjee, S., Pan, X., Zhao, B., Ding, C., Li, Y ., and Xu, D. Accelerating dataset distillation via model augmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11950–11959,

arXiv 2020