Pith. sign in

REVIEW 3 major objections 4 minor 82 references

Expert Routing with Synthetic Data for Continual Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Generate to Discriminate (G2D) trains an inference-time domain router on synthetic samples, and this use of synthetic data outperforms generative replay on every benchmark tested.

desk verdict The method and benchmark are real contributions, but the headline claim that synthetic data is better for routing than for replay is not actually isolated by the experiments. read the letter →

arxiv 2412.17009 v3 pith:DUY73GRC submitted 2024-12-22 cs.LG

Yewon Byun , Sanket Vaibhav Mehta , Saurabh Garg , Emma Strubell , Michael Oberst , Bryan Wilder , Zachary C. Lipton This is my paper · ORCID
classification cs.LG
keywords continuallearningdomain-incrementalcatastrophicforgettingsyntheticdatagenerativereplayexpertroutingdomaindiscriminatordermatologybenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to keep a model accurate on earlier domains when institutions can share models but not data. Its answer, Generate to Discriminate (G2D), is to spend synthetic data on learning which expert to call at test time rather than on retraining the label classifier. Across vision and text benchmarks, routing on synthetic samples outperforms generative replay with the same samples, and the gap grows on datasets with large domain shifts. The paper also introduces a dermatology benchmark to show the failure of prompt-based methods when pretraining does not cover the domain.

What carries the argument

The load-bearing object is the domain discriminator (router) trained on the union of synthetic sets $M_1 \cup \dots \cup M_t$, where $M_t$ is sampled from a generator finetuned on domain $D_t$. At inference, the router predicts a domain identity and dispatches the sample to the expert finetuned on that domain; for out-of-distribution samples it chooses the most similar seen domain. The same synthetic samples, when instead used to augment the label classifier (generative replay), give consistently worse results, which the paper reads as evidence that modeling domain differences is easier than modeling class-label differences for the generators considered.

What would settle it

Train the G2D router on synthetic samples and evaluate it on real held-out samples domain by domain; if its routing accuracy approaches chance on any domain while a router trained on real samples from that domain is accurate, the transfer premise fails. A sharper test is to replace the per-domain generator with a class-conditional but domain-blind generator; if G2D still routes well, the signal is not domain-specific and the method's stated mechanism is wrong.

Watch

Extended reading notes

Core claim

The central claim is that in domain-incremental continual learning, if a per-domain generator produces a fixed set of synthetic samples, training a domain discriminator on those samples and using it to route test inputs to domain-specific experts yields higher average accuracy than feeding the same samples back into the classifier as rehearsal data. The authors call this Generate to Discriminate (G2D). They report consistent gains over generative replay, experience replay, regularization, and prompt-based baselines on DomainNet, CORe50, the new DermCL benchmark, and a four-dataset question-answering sequence, and they show the synthetic-trained router approaches the accuracy of a router trained on real data.

Load-bearing premise

The synthetic samples generated from each domain's fine-tuned generator carry enough domain-specific signal that a router trained only on them will recognize real test samples from that domain.

Editorial extensions

If this is right

  • With no real data or domain labels from earlier domains, a deployment can still route each new input to the right expert; G2D needs only the current domain's real data plus generated samples from all seen domains.
  • For a fixed number of generated examples, the paper's results say the highest-accuracy use is building the router, not augmenting the classifier: G2D beats Generative Replay on DomainNet, CORe50, DermCL, and QA.
  • The synthetic-trained router nearly matches a router trained on real samples, so the routing step is not the bottleneck; remaining gaps to the multi-task upper bound come from expert training, not domain identification.
  • On out-of-distribution test domains (CORe50), the router behaves as a similarity function and sends samples to the most similar seen expert, extending the method beyond strict domain membership.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If domain discrimination is genuinely easier than class discrimination for the same generator, then generative replay's known weakness is not sample quality alone; the target task itself is harder, so synthetic data should be allocated to auxiliary identification tasks.
  • The same router-on-synthetic-samples design transfers naturally to federated or institution-separated settings, where each site can share a local generator or generated samples rather than raw data.
  • A direct extension is to measure router accuracy against the number of synthetic samples used; if it saturates early, G2D could operate with far fewer generated samples than generative replay needs, lowering compute and exposure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Generate to Discriminate (G2D), a domain-incremental continual learning method in which synthetic data is used to train a domain discriminator that routes test inputs to a set of per-domain expert classifiers. The experts are trained sequentially, each on real data from its own domain, and the router is trained on synthetic samples produced by a per-domain-finetuned generator (Stable Diffusion for vision, T5 for text). The authors report results on DomainNet, CORe50, a new dermatology benchmark (DermCL), and a question-answering continual learning benchmark, and they claim that using the same synthetic samples for routing consistently outperforms using them to augment the training data of a single classifier (generative replay). The central empirical claim is that synthetic data is more valuable for inference-time routing than for rehearsal.

Significance. If the central claim were established, the paper would offer a genuinely new perspective on the role of synthetic data in domain-incremental learning and would be of practical interest in privacy-constrained settings. The paper makes two concrete contributions: (i) a simple method that is easy to implement, and (ii) a new benchmark, DermCL, assembled from four public dermatology datasets with realistic distribution shifts. The use of the same generator (starting from the same pretrained checkpoint) for both G2D and the generative replay baseline is a good methodological choice that reduces, though does not eliminate, the risk of an unfair comparison. However, the central claim is not currently supported by the reported experiments because the comparison changes two variables at once (the use of synthetic data and the model architecture), and the claimed equality of the synthetic data sets is not documented.

major comments (3)
  1. [Section 6, Tables 1 and 2] The headline comparison between G2D and Generative Replay confounds the role of synthetic data with the choice of model architecture. G2D maintains T domain-specific experts trained on real data from their own domain, while Generative Replay trains a single shared classifier on synthetic replay plus current real data. These are different systems, so the observed performance difference cannot be attributed to the routing use of synthetic data. The Oracle Router row in Table 1 (same experts, router trained on real data) nearly matches the MTL upper bound, which shows that the expert-ensemble architecture is itself very strong. To support the claim, the authors should hold the architecture fixed and vary only the role of the synthetic samples, for example by comparing (a) the expert ensemble with a router trained on synthetic samples (current G2D), (b) the expert ensemble with a router trained on real data (Oracle Router), and (c) the expert ensemble with a generative-replay classifier trained on synthetic samples from previous domains (i.e., each expert trained on its own real data plus synthetic replay from previous domains), or, alternatively, a single-classifier variant of G2D. Only such an ablation can isolate the contribution claimed. Additionally, the two arms differ in that at domain t the generative replay classifier is trained on real samples from the current domain, whereas the G2D router is trained on synthetic samples from the current domain; this difference in input distribution is another uncontrolled variable.
  2. [Section 4.1 and Appendix B.2] The claim that G2D and Generative Replay use 'the same set of synthetic samples' is not supported by the text. Section 5.2 states that the Generative Replay buffer is capped at 15/class for DomainNet, 100/class for DermCL, 50/class for CORe50, and 1% for QA, but the number of synthetic samples M_t used to train the G2D router is never specified anywhere, including the hyperparameter tables in Appendix C. If the router is trained on the full set of generated samples while the replay buffer is capped, the two arms do not consume the same amount of synthetic data and the central comparison is not 'same set.' The authors must report M_t and the effective number of samples seen by each arm, and ideally run an additional comparison in which the replay buffer size equals the number of samples used for router training.
  3. [Section 4.2] The paper does not state whether the union M_1 ∪ ... ∪ M_t used to train the router at time t requires storing all past synthetic samples or regenerating them. If past synthetic samples are stored, the method maintains a synthetic rehearsal buffer of unbounded size, which should be compared against the memory budgets of the replay baselines; if they are regenerated, the computational cost is different. Either way, this information is needed to evaluate the claimed advantage over generative replay and to understand the method's memory footprint.
minor comments (4)
  1. [Section 3, Eq. (1)] The sum in Eq. (1) runs from i=0 to N_t and computes a sum rather than an average; the standard risk definition would average over N_t and the index should start at 1.
  2. [Section 5.2] In the text, the baseline is referred to as 'Oracle Gate' but in Table 1 it appears as 'Oracle Router'; please use one term throughout.
  3. [Section 6, footnote 5] The reader cannot verify the 'same set' claim from the example generations in Figure 3 and Table 11 alone; please add a statistical summary of the synthetic data (e.g., number of samples, quality metrics) to the appendix.
  4. [Section 5.1 (DermCL)] Since the DermCL benchmark uses average AUC due to class imbalance, it would be helpful to include per-domain label distributions or a class-imbalance table, so that the difficulty of each domain is transparent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central G2D claim is an empirical benchmark comparison, not a reduction to a fitted input or self-citation.

full rationale

The paper's central claim is that synthetic samples are more effective for training a domain discriminator than for augmenting downstream classifier training. This is presented as an empirical observation from benchmark comparisons (Tables 1 and 2), not derived from a fitted equation or from an assumption that already contains the result. The domain discriminator in Section 4.2 is trained on synthetic samples whose source-domain labels are known by construction, and then evaluated on held-out real test samples and against an oracle discriminator trained on real data (Table 3), which is an external check rather than a circular input. The comparison with Generative Replay uses the same generators and synthetic-sample generation procedure (Section B.2), making the comparison a genuine experimental test. The only self-citations, such as [44] and [70], appear in related-work or baseline context and are not load-bearing for the central claim. Concerns about the expert-ensemble architecture confounding the G2D-versus-Generative-Replay comparison are experimental-design or correctness issues, not circularity, and do not change the circularity verdict.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical comparison, so the ledger records the tuned hyperparameters and the data-generation assumptions the method depends on. No new theoretical entities are postulated; the domain discriminator is a standard classifier trained on synthetic data.

free parameters (5)
  • Number of synthetic samples per domain (M_t)
    The main text defines M_t as the synthetic sample set but never states how many samples are generated per domain. The size of M_t affects both the discriminator and the generative replay comparison, and is a key variable that is left unspecified.
  • G2D vision hyperparameters = lr 0.005-0.07, epochs 10-50, LoRA rank 16
    Selected via source hold-out performance (Appendix C.1, Table 7). The router and expert training depend on these choices, which are tuned per benchmark rather than fixed.
  • G2D text hyperparameters = lr 5e-4, epochs 3-5, LoRA rank 32
    Selected via source hold-out performance (Appendix C.2, Table 9). Used for the QA expert and discriminator.
  • Generator finetuning hyperparameters = 150k steps, lr 1e-5, batch 16 for SD; prompt length 400 for T5
    Appendix C.1-C.2. The quality of synthetic samples, and hence the router, depends on these choices.
  • Baseline replay buffer sizes = 15/class DomainNet, 100/class DermCL, 50/class CORe50, 1% QA
    Chosen according to lower bound on samples per class or prior practice. These affect generative replay and experience replay baselines, and therefore the strength of the comparison.
assumptions (4)
  • domain assumption The label space is fixed across domains and domain identities are known during training but not at inference
    This defines the domain-incremental scenario in Section 3. G2D is only applicable where this assumption holds.
  • domain assumption Per-domain finetuned generative models produce synthetic samples whose domain distribution matches real domain distributions closely enough for a discriminator to transfer
    The whole router depends on this. Stated as a design choice in Section 4.1, tested empirically in Section 6.1.
  • domain assumption Pretrained backbones (ViT-B/16, BERT-Base) and generators (Stable Diffusion, T5) provide sufficient quality for the considered tasks
    All experts, routers, and generators are initialized from pretrained checkpoints; the method inherits their capabilities and biases.
  • domain assumption Source hold-out performance is a valid criterion for hyperparameter selection
    The paper selects hyperparameters based on source hold-out (Appendix C), assuming this generalizes to target domains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expert Routing with Synthetic Data for Continual Learning." pith.science (2026). https://pith.science/paper/DUY73GRC

@misc{pith2026241217009,
  author       = {Pith},
  title        = {Pith review of: Expert Routing with Synthetic Data for Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUY73GRC}},
  note         = {Machine review of arXiv:2412.17009}
}
read the original abstract

In many real-world settings, regulations and economic incentives permit the sharing of models but not data across institutional boundaries. In such scenarios, practitioners might hope to adapt models to new domains, without losing performance on previous domains (so-called catastrophic forgetting). While any single model may struggle to achieve this goal, learning an ensemble of domain-specific experts offers the potential to adapt more closely to each individual institution. However, a core challenge in this context is determining which expert to deploy at test time. In this paper, we propose Generate to Discriminate (G2D), a domain-incremental continual learning method that leverages synthetic data to train a domain-discriminator that routes samples at inference time to the appropriate expert. Surprisingly, we find that leveraging synthetic data in this capacity is more effective than using the samples to \textit{directly} train the downstream classifier (the more common approach to leveraging synthetic data in the lifelong learning literature). We observe that G2D outperforms competitive domain-incremental learning methods on tasks in both vision and language modalities, providing a new perspective on the use of synthetic data in the lifelong learning literature.

Figures

Figures reproduced from arXiv: 2412.17009 by the authors.

Figure 1
Figure 1. Generate to Discriminate (G2D); During training (black text and arrows), we (i) finetune the generator and expert classifier and (ii) finetune a domain discriminator on synthetic images produced by our generator. At inference time (blue text and arrows), we route test samples to the corresponding expert, based on our discriminator’s prediction. at test time. Some recent methods rely on the implicit domain discrimina… view at source ↗
Figure 2
Figure 2. (Three left-most plots) Domain discrimination visualizations for CORe50 benchmark (8 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Comparison of sampled synthetic images and real images from the DomainNet benchmark. [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 39 canonical work pages

  1. [1]

    Unsupervised domain clusters in pretrained language models

    Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters in pretrained language models. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747–7763, 2020

    2020

  2. [2]

    Expert gate: Lifelong learning with a network of experts

    Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3366–3375, 2017

    2017

  3. [3]

    Learning to route for dynamic adapter composition in continual learning with language models.arXiv preprint arXiv:2408.09053, 2024

    Vladimir Araujo, Marie-Francine Moens, and Tinne Tuytelaars. Learning to route for dynamic adapter composition in continual learning with language models.arXiv preprint arXiv:2408.09053, 2024

    arXiv 2024

  4. [4]

    Don’t generate me: Training differentially private generative models with sinkhorn divergence.Advances in Neural Information Processing Systems, 34:12480–12492, 2021

    Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don’t generate me: Training differentially private generative models with sinkhorn divergence.Advances in Neural Information Processing Systems, 34:12480–12492, 2021

    work page 2021

  5. [5]

    Analysis of the isic image datasets: Usage, benchmarks and recommendations.Medical image analysis, 75:102305, 2022

    Bill Cassidy, Connah Kendrick, Andrzej Brodzicki, Joanna Jaworek-Korjakowska, and Moi Hoon Yap. Analysis of the isic image datasets: Usage, benchmarks and recommendations.Medical image analysis, 75:102305, 2022

    work page 2022

  6. [6]

    Efficient lifelong learning with a-GEM

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. InInternational Conference on Learning Representations, 2019

    2019

  7. [7]

    On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486, 2019

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486, 2019

    arXiv 1902

  8. [8]

    Synthetic data in machine learning for medicine and healthcare.Nature Biomedical Engineering, 5 (6):493–497, 2021

    Richard J Chen, Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare.Nature Biomedical Engineering, 5 (6):493–497, 2021

    work page 2021

Show all 82 references
  1. [9]

    Quac: Question answering in context

    Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, 2018

    2018

  2. [10]

    Aaron S Coyner, Jimmy S Chen, Ken Chang, Praveer Singh, Susan Ostmo, R V Paul Chan, Michael F Chiang, Jayashree Kalpathy-Cramer, J Peter Campbell, Imaging, Informatics in Retinopathy of Prematurity Consortium, et al. Synthetic medical images for robust, privacy-preserving trai...

    2022

  3. [11]

    Disparities in dermatology ai performance on a diverse, curated clinical image set.Science advances, 8(31): eabq6147, 2022

    Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai performance on a diverse, curated clinical image set.Science advances,...

    2022

  4. [12]

    A continual learning survey: Defying forgetting in classification tasks.IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleˇ s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks.IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021

    2021

  5. [13]

    Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019

    Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019. 9

    2019

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  7. [15]

    Continual learning beyond a single model

    Thang Doan, Seyed Iman Mirzadeh, and Mehrdad Farajtabar. Continual learning beyond a single model. InConference on Lifelong Learning Agents, pages 961–991. PMLR, 2023

    2023

  8. [16]

    Differentially private diffusion models.arXiv preprint arXiv:2210.09929, 2022

    Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models.arXiv preprint arXiv:2210.09929, 2022

    2022 arXiv

  9. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:...

    2010 arXiv

  10. [18]

    Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation.NPJ digital medicine, 4(1):141, 2021

    August DuMont Sch¨ utte, J¨ urgen Hetzel, Sergios Gatidis, Tobias Hepp, Benedikt Dietz, Stefan Bauer, and Patrick Schwab. Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation.NPJ digital medicine, 4(1):141, 2021

    2021

  11. [19]

    Now Publishers Inc., 2014

    Cynthia Dwork and Aaron Roth.The Algorithmic Foundations of Differential Privacy. Now Publishers Inc., 2014

    2014

  12. [20]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. InTheory of Cryptography Conference, pages 265–284. Springer, 2006

    2006

  13. [21]

    Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al. M 3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design.Advances in Neural Information Processing Systems, 35:2844...

    2022

  14. [22]

    Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    Robert M French. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4):128–135, 1999

    1999

  15. [23]

    Domain adaptation for medical image analysis: a survey.IEEE Transactions on Biomedical Engineering, 69(3):1173–1185, 2021

    Hao Guan and Mingxia Liu. Domain adaptation for medical image analysis: a survey.IEEE Transactions on Biomedical Engineering, 69(3):1173–1185, 2021

    2021

  16. [24]

    Improved schemes for episodic memory-based lifelong learning.Advances in Neural Information Processing Systems, 33:1023–1035, 2020

    Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning.Advances in Neural Information Processing Systems, 33:1023–1035, 2020

    2020

  17. [25]

    Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning.Advances in Neural Information Processing Systems, 34:29335–29347, 2021

    Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning.Advances in Neural Information Processing Syste...

    2021

  18. [26]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  19. [27]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601...

    2017

  20. [28]

    Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study.bmj, 376, 2022

    Fahad Kamran, Shengpu Tang, Erkin Otles, Dustin S McEvoy, Sameh N Saleh, Jen Gong, Benjamin Y Li, Sayon Dutta, Xinran Liu, Richard J Medford, et al. Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and mult...

    2022

  21. [29]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT, pages 4171–4186, 2019

    2019

  22. [30]

    Adam: A Method for Stochastic Optimization.arXiv Preprint arXiv:1412.6980, 2014

    Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization.arXiv Preprint arXiv:1412.6980, 2014. 10

    2014 arXiv

  23. [31]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting i...

    2017 doi

  24. [32]

    Mixture of experts meets prompt-based continual learning.arXiv preprint arXiv:2405.14124, 2024

    Minh Le, An Nguyen, Huy Nguyen, Trang Nguyen, Trang Pham, Linh Van Ngo, and Nhat Ho. Mixture of experts meets prompt-based continual learning.arXiv preprint arXiv:2405.14124, 2024

    2024 arXiv

  25. [33]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021

    2021

  26. [34]

    Theory on mixture-of-experts in continual learning.arXiv preprint arXiv:2406.16437, 2024

    Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B Shroff. Theory on mixture-of-experts in continual learning.arXiv preprint arXiv:2406.16437, 2024

    2024 arXiv

  27. [35]

    Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting

    Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. InInternational conference on machine learning, pages 3925–3934. PMLR, 2019

    2019

  28. [36]

    Learning without forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017

    Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017

    2017

  29. [37]

    The clear benchmark: Continual learning on real-world imagery

    Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. InThirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021

    2021

  30. [38]

    Core50: a new dataset and benchmark for continuous object recognition

    Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. InConference on robot learning, pages 17–26. PMLR, 2017

    2017

  31. [39]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017

    2017

  32. [40]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  33. [41]

    Differentially private latent diffusion models.arXiv preprint arXiv:2305.15759, 2023

    Saiyue Lyu, Margarita Vinaroz, Michael F Liu, and Mijung Park. Differentially private latent diffusion models.arXiv preprint arXiv:2305.15759, 2023

    2023 arXiv

  34. [42]

    Modeling task relationships in multi-task learning with multi-gate mixture-of-experts

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930–1939, 2018

    1930

  35. [43]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. InPsychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989

    1989

  36. [44]

    An empirical investigation of the role of pre-training in lifelong learning.Journal of Machine Learning Research, 24(214):1–50, 2023

    Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning.Journal of Machine Learning Research, 24(214):1–50, 2023. URLhttp://jmlr.org/papers/v24/22-0496.html

    2023

  37. [45]

    Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from smartphones.Data in brief, 32:106221, 2020

    Andre GC Pacheco, Gustavo R Lima, Amanda S Salomao, Breno Krohling, Igor P Biral, Gabriel G de Angelo, F´ abio CR Alves Jr, Jos´ e GM Esgario, Alana C Simora, Pedro BC Castro, et al. Pad-ufes-20: A skin lesion dataset composed of patient data and clinical images collected from...

    2020

  38. [46]

    Learning more generalized experts by merging experts in mixture-of-experts.arXiv preprint arXiv:2405.11530, 2024

    Sejik Park. Learning more generalized experts by merging experts in mixture-of-experts.arXiv preprint arXiv:2405.11530, 2024

    2024 arXiv

  39. [47]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. InProceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019. 11

    2019

  40. [48]

    Can we trust deep learning based diagnosis? the impact of domain shift in chest radiograph classification

    Eduardo HP Pooch, Pedro Ballester, and Rodrigo C Barros. Can we trust deep learning based diagnosis? the impact of domain shift in chest radiograph classification. InThoracic Image Analysis: Second International Workshop, TIA 2020, Held in Conjunction with MICCAI 2020, Lima, P...

    2020

  41. [49]

    LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5

    Chengwei Qin and Shafiq Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5. InInternational Conference on Learning Representations, 2022

    2022

  42. [50]

    Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Machine Learning Research, 21(1):5485–5551, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Machine Learning Research, 21(1):5485–5551, 2020

    2020

  43. [51]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016

    2016

  44. [52]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  45. [53]

    Divide and not forget: Ensemble of selectively trained experts in continual learning.arXiv preprint arXiv:2401.10191, 2024

    Grzegorz Rype´ s´ c, Sebastian Cygert, Valeriya Khan, Tomasz Trzci´ nski, Bartosz Zieli´ nski, and Bart lomiej Twardowski. Divide and not forget: Ensemble of selectively trained experts in continual learning.arXiv preprint arXiv:2401.10191, 2024

    2024 arXiv

  46. [54]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  47. [55]

    Continual learning with deep generative replay.Advances in neural information processing systems, 30, 2017

    Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay.Advances in neural information processing systems, 30, 2017

    2017

  48. [56]

    Continual diffusion: Continual customization of text-to-image diffusion with c-lora.arXiv preprint arXiv:2304.06027, 2023

    James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of text-to-image diffusion with c-lora.arXiv preprint arXiv:2304.06027, 2023

    2023 arXiv

  49. [57]

    Coda-prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning

    James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention-based prompting for rehearsal-free continual learning. InProceedings of the IE...

    2023

  50. [58]

    An introduction to lifelong supervised learning.arXiv preprint arXiv:2207.04354, 2022

    Shagun Sodhani, Mojtaba Faramarzi, Sanket Vaibhav Mehta, Pranshu Malviya, Mohamed Abdel- salam, Janarthanan Janarthanan, and Sarath Chandar. An introduction to lifelong supervised learning.arXiv preprint arXiv:2207.04354, 2022

    2022 arXiv

  51. [59]

    {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning

    Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning. InInternational Conference on Learning Representations, 2020

    2020

  52. [60]

    The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Scientific data, 5(1):1–9, 2018

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Scientific data, 5(1):1–9, 2018

    2018

  53. [61]

    Alvaro E Ulloa-Cerna, Linyuan Jing, John M Pfeifer, Sushravya Raghunath, Jeffrey A Ruhl, Daniel B Rocha, Joseph B Leader, Noah Zimmerman, Greg Lee, Steven R Steinhubl, et al. Rechommend: an ecg-based machine learning approach for identifying patients at increased risk of undia...

    2022

  54. [62]

    Three scenarios for continual learning.arXiv preprint arXiv:1904.07734, 2019

    Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning.arXiv preprint arXiv:1904.07734, 2019

    1904 arXiv

  55. [63]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008. 12

    2008

  56. [64]

    Development and multi-site external validation of a generalizable risk prediction model for bipolar disorder.medRxiv, pages 2023–02, 2023

    Colin G Walsh, Michael A Ripperger, Yirui Hu, Yi-han Sheu, Drew Wilimitis, Amanda B Zheutlin, Daniel Rocha, Karmel W Choi, Victor M Castro, H Lester Kirchner, et al. Development and multi-site external validation of a generalizable risk prediction model for bipolar disorder.me...

    2023

  57. [65]

    Coscl: Cooperation of small continual learners is stronger than a big one

    Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Coscl: Cooperation of small continual learners is stronger than a big one. InEuropean Conference on Computer Vision, pages 254–271. Springer, 2022

    2022

  58. [66]

    Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.Advances in Neural Information Processing Systems, 36, 2024

    Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.Advances in Neural Information Processing Systems, 36, 2024

    2024

  59. [67]

    S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning

    Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022....

    2022

  60. [68]

    Dualprompt: Complementary prompting for rehearsal-free continual learning

    Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. InEuropean Conference on Computer Vision, pages 631–648. Springer, 2022

    2022

  61. [69]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022

    2022

  62. [70]

    Efficient meta lifelong-learning with limited memory

    Zirui Wang, Sanket Vaibhav Mehta, Barnab´ as Pocz´ os, and Jaime G Carbonell. Efficient meta lifelong-learning with limited memory. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 535–548, 2020

    2020

  63. [71]

    Benchmd: A benchmark for modality-agnostic learning on medical images and sensors.arXiv preprint arXiv:2304.08486, 2023

    Kathryn Wantlin, Chenwei Wu, Shih-Cheng Huang, Oishi Banerjee, Farah Dadabhoy, Veeral Vipin Mehta, Ryan Wonhee Han, Fang Cao, Raja R Narayan, Errol Colak, et al. Benchmd: A benchmark for modality-agnostic learning on medical images and sensors.arXiv preprint arXiv:2304.08486, 2023

    2023 arXiv

  64. [72]

    Boosting continual learning of vision-language models via mixture-of-experts adapters

    Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219–23230, 2024

    2024

  65. [73]

    Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.PLoS medicine, 15(11):e1002683, 2018

    John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.PLoS medicine, 15(11):e1002683, 2018

    2018

  66. [74]

    Continual learning through synaptic intelligence

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. InInternational conference on machine learning, pages 3987–3995. PMLR, 2017

    2017

  67. [75]

    Continual learning with pre-trained models: A survey.arXiv preprint arXiv:2401.16386, 2024

    Da-Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey.arXiv preprint arXiv:2401.16386, 2024

    2024 arXiv

  68. [76]

    Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems, 35:7103–7114, 2022

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems, 35:7103–7114, 2022. 13 A Dataset Details In this section, we pr...

    2022

  69. [77]

    QuAC→TriviaQA (Web)→TriviaQA (Wiki)→SQuAD

  70. [78]

    SQuAD→TriviaQA (Wiki)→QuAC→TriviaQA (Web)

  71. [79]

    TriviaQA (Web)→TriviaQA (Wiki)→SQuAD→QuAC

  72. [80]

    Generate article, question and answer

    TriviaQA (Wiki)→QuAC→TriviaQA (Web)→SQuAD A.3 DermCL Benchmark Details This benchmark offers a sequence of four dermatology imaging tasks. Distribution shifts are present across all four domains (HAM10000[ 60], BCN2000[ 5], PAD-UEFS-20[ 45], and, DDI[ 11]), in both demographic...

    2010

  73. [81]

    (2) In the original Generative Replay implementation, the generator is sequentially finetuned (in addition to the classifier) on each domain in the sequence

    for text data. (2) In the original Generative Replay implementation, the generator is sequentially finetuned (in addition to the classifier) on each domain in the sequence. This introduces the challenge of catastrophic forgetting in the generative model and introduces addition...

  74. [82]

    The BERT-base architecture has 12 Transformer layers, 12 self-attention heads, and 768 hidden dimensions (110M parameters)

    backbone. The BERT-base architecture has 12 Transformer layers, 12 self-attention heads, and 768 hidden dimensions (110M parameters). For our generative model, we use prompt tuning to learn parameter-efficient models [ 33]. We use the pretrained T5-Large v1.1 checkpoint adapte...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.