{"id":"36ffb635-5006-4f02-a393-9b284a4a1e24","arxiv_id":"2412.17009","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"G2D trains a domain discriminator on synthetic data to route test samples to domain-specific experts, outperforming generative replay on several continual learning benchmarks.","lead":"The paper proposes G2D, a continual learning method that uses synthetic data to train a domain router, which sends each test sample to the expert model trained for its domain. Across vision and text benchmarks, routing this way beats using the same synthetic data to train the classifier directly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison confounds the role of synthetic data with the expert-ensemble architecture: G2D's experts train on real domain data only, so the gain over Generative Replay may be due to specialist experts, not to the routing use of synthetic data.","rationale":"The paper's central contribution is the empirical observation that synthetic data is more valuable for training an inference-time router than for replay in the downstream classifier. The reader's weakest_assumption focused on whether synthetic samples preserve domain-discriminative signal that transfers to real inputs. That is a legitimate concern, and the paper's discriminator accuracy tables provide some support. However, the more load-bearing issue is experimental confounding: the G2D pipeline and the Generative Replay baseline differ not only in how synthetic data is used, but also in the underlying model architecture. G2D uses a set of domain-specific experts, each trained exclusively on real data from its own domain; Generative Replay uses one shared classifier trained on synthetic replay plus current real data. The expert ensemble itself prevents forgetting by construction, independent of the router. The Oracle Router results show that this architecture is nearly as strong as the multi-task upper bound, so the observed gains over Generative Replay cannot be attributed solely to the routing role of synthetic data. This is not an ad hominem or a rejection of the method's practical value; the empirical results may still hold. But the central claim as stated requires a controlled comparison. The proposed ablation isolates the role of synthetic data while holding the architecture fixed. The reader already noted 'asymmetry in the generative replay comparison' and the missing sample counts, so the verdict of CONDITIONAL remains appropriate; no change in verdict is needed, but the condition should explicitly include this ablation.","tokens_in":17782,"tokens_out":7221,"duration_ms":71130,"concrete_test":"Run a matched ablation on DomainNet and DermCL, keeping the G2D expert set, LoRA ranks, and per-domain training budgets identical. Arm A (current G2D): train the router on the synthetic samples and route test inputs as in the paper. Arm B: remove the router; train the same experts on their real domain plus the same synthetic samples from all previous domains as replay (or, alternatively, train one shared classifier on real current data plus the same synthetic replay), and evaluate with oracle domain identity. If Arm A does not outperform Arm B, the headline claim that routing is the better use of synthetic data is unsupported; the gain would be attributable to the expert ensemble. Also report the exact M_t sizes used for the router so the 'same set' premise can be checked.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6's headline empirical claim—that the same synthetic samples are more effective when used to train a domain discriminator than to augment downstream classifier training—rests entirely on Table 1/2 comparisons between G2D and Generative Replay. But these two arms differ in a second, uncontrolled variable: G2D maintains T domain-specific experts, each trained only on real data from its own domain, while Generative Replay trains a single shared classifier on synthetic replay plus current real data. The expert ensemble has a structural advantage independent of synthetic data: each expert never forgets its own domain. The Oracle Router row in Table 1 (same experts, real-data router) nearly matches the MTL upper bound, confirming the architecture itself is very strong. Thus the observed G2D > Generative Replay does not isolate the contribution of using synthetic data for routing; it could be driven by the specialist-expert setup. The paper also does not report the number of synthetic samples fed to the router versus the capped 15/100 per-class replay buffer, so 'same set' is not yet verified. To support the central claim, the comparison must hold the expert architecture fixed and vary only the role of the synthetic samples.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Generate to Discriminate (G2D), a domain-incremental continual learning method in which synthetic data is used to train a domain discriminator that routes test inputs to a set of per-domain expert classifiers. The experts are trained sequentially, each on real data from its own domain, and the router is trained on synthetic samples produced by a per-domain-finetuned generator (Stable Diffusion for vision, T5 for text). The authors report results on DomainNet, CORe50, a new dermatology benchmark (DermCL), and a question-answering continual learning benchmark, and they claim that using the same synthetic samples for routing consistently outperforms using them to augment the training data of a single classifier (generative replay). The central empirical claim is that synthetic data is more valuable for inference-time routing than for rehearsal.","tokens_in":17975,"tokens_out":7149,"duration_ms":56791,"significance":"If the central claim were established, the paper would offer a genuinely new perspective on the role of synthetic data in domain-incremental learning and would be of practical interest in privacy-constrained settings. The paper makes two concrete contributions: (i) a simple method that is easy to implement, and (ii) a new benchmark, DermCL, assembled from four public dermatology datasets with realistic distribution shifts. The use of the same generator (starting from the same pretrained checkpoint) for both G2D and the generative replay baseline is a good methodological choice that reduces, though does not eliminate, the risk of an unfair comparison. However, the central claim is not currently supported by the reported experiments because the comparison changes two variables at once (the use of synthetic data and the model architecture), and the claimed equality of the synthetic data sets is not documented.","major_comments":[{"comment":"The headline comparison between G2D and Generative Replay confounds the role of synthetic data with the choice of model architecture. G2D maintains T domain-specific experts trained on real data from their own domain, while Generative Replay trains a single shared classifier on synthetic replay plus current real data. These are different systems, so the observed performance difference cannot be attributed to the routing use of synthetic data. The Oracle Router row in Table 1 (same experts, router trained on real data) nearly matches the MTL upper bound, which shows that the expert-ensemble architecture is itself very strong. To support the claim, the authors should hold the architecture fixed and vary only the role of the synthetic samples, for example by comparing (a) the expert ensemble with a router trained on synthetic samples (current G2D), (b) the expert ensemble with a router trained on real data (Oracle Router), and (c) the expert ensemble with a generative-replay classifier trained on synthetic samples from previous domains (i.e., each expert trained on its own real data plus synthetic replay from previous domains), or, alternatively, a single-classifier variant of G2D. Only such an ablation can isolate the contribution claimed. Additionally, the two arms differ in that at domain t the generative replay classifier is trained on real samples from the current domain, whereas the G2D router is trained on synthetic samples from the current domain; this difference in input distribution is another uncontrolled variable.","section":"Section 6, Tables 1 and 2"},{"comment":"The claim that G2D and Generative Replay use 'the same set of synthetic samples' is not supported by the text. Section 5.2 states that the Generative Replay buffer is capped at 15/class for DomainNet, 100/class for DermCL, 50/class for CORe50, and 1% for QA, but the number of synthetic samples M_t used to train the G2D router is never specified anywhere, including the hyperparameter tables in Appendix C. If the router is trained on the full set of generated samples while the replay buffer is capped, the two arms do not consume the same amount of synthetic data and the central comparison is not 'same set.' The authors must report M_t and the effective number of samples seen by each arm, and ideally run an additional comparison in which the replay buffer size equals the number of samples used for router training.","section":"Section 4.1 and Appendix B.2"},{"comment":"The paper does not state whether the union M_1 ∪ ... ∪ M_t used to train the router at time t requires storing all past synthetic samples or regenerating them. If past synthetic samples are stored, the method maintains a synthetic rehearsal buffer of unbounded size, which should be compared against the memory budgets of the replay baselines; if they are regenerated, the computational cost is different. Either way, this information is needed to evaluate the claimed advantage over generative replay and to understand the method's memory footprint.","section":"Section 4.2"}],"minor_comments":[{"comment":"The sum in Eq. (1) runs from i=0 to N_t and computes a sum rather than an average; the standard risk definition would average over N_t and the index should start at 1.","section":"Section 3, Eq. (1)"},{"comment":"In the text, the baseline is referred to as 'Oracle Gate' but in Table 1 it appears as 'Oracle Router'; please use one term throughout.","section":"Section 5.2"},{"comment":"The reader cannot verify the 'same set' claim from the example generations in Figure 3 and Table 11 alone; please add a statistical summary of the synthetic data (e.g., number of samples, quality metrics) to the appendix.","section":"Section 6, footnote 5"},{"comment":"Since the DermCL benchmark uses average AUC due to class imbalance, it would be helpful to include per-domain label distributions or a class-imbalance table, so that the difficulty of each domain is transparent.","section":"Section 5.1 (DermCL)"}],"recommendation":"major_revision","confidential_remarks":"The key concern is the confound in the central comparison. I would encourage the editor to prioritize the proposed ablation (hold architecture fixed, vary the role of synthetic data) as a condition for acceptance. The DermCL benchmark is a useful contribution and should be retained. The paper is otherwise well-written and the experiments are extensive, but the main claim needs stronger isolation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read of arXiv:2412.17009 (G2D). The genuinely new thing here is the idea itself: instead of using synthetic samples to rehearse the label classifier, train a domain discriminator on them and route test examples to per-domain experts. That framing is clean, and the paper shows the synthetic-trained router gets close to a router trained on real data (Table 3, Figure 2). The DermCL benchmark is also a useful addition, giving the community a medical-imaging sequence with real distribution shifts.\n\nBut the central empirical claim in Section 6 — that the same synthetic samples are better used for domain discrimination than for augmenting downstream classifier training — is not supported by the comparison as run. G2D and Generative Replay differ on two axes at once: how synthetic data is used, and whether the system is an ensemble of per-domain experts or a single sequentially-updated classifier. The Oracle Router rows show the ensemble alone is strong (95.44 on CORe50 vs 97.19 MTL; 71.24 vs 75.10 on DomainNet), so you cannot attribute the G2D-over-GR margin to the routing use of synthetic data. A controlled comparison would hold the expert architecture fixed and vary only what the synthetic samples train. Without that, the 'surprising finding' is an artifact of confounding.\n\nTwo smaller issues: the number of synthetic samples per domain (M_t) is never reported, so the 'same set' claim is unverified — if the router sees more samples than the 15/class replay buffer, the comparison is not even sample-count-matched. And there is no code release, which hurts reproducibility given the heavy pipeline.\n\nThe paper is not sloppy; the motivation is well-argued and the analysis of router accuracy is a genuine step forward. But the abstract overstates what is demonstrated. I would send this to review, asking for the controlled experiment or a scaled-back claim, plus the sample counts and code. A solid conditional-accept candidate if the authors can disentangle the confound.","headline":"The method and benchmark are real contributions, but the headline claim that synthetic data is better for routing than for replay is not actually isolated by the experiments.","tokens_in":18561,"tokens_out":4095,"would_cite":false,"duration_ms":36120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generate to Discriminate (G2D) trains an inference-time domain router on synthetic samples, and this use of synthetic data outperforms generative replay on every benchmark tested.","keywords":["continual learning","domain-incremental learning","catastrophic forgetting","synthetic data","generative replay","expert routing","domain discriminator","dermatology benchmark"],"falsifier":"Train the G2D router on synthetic samples and evaluate it on real held-out samples domain by domain; if its routing accuracy approaches chance on any domain while a router trained on real samples from that domain is accurate, the transfer premise fails. A sharper test is to replace the per-domain generator with a class-conditional but domain-blind generator; if G2D still routes well, the signal is not domain-specific and the method's stated mechanism is wrong.","tokens_in":17550,"feed_emoji":"🧭","tokens_out":6511,"duration_ms":58219,"temperature":0.7,"pith_summary":"The paper asks how to keep a model accurate on earlier domains when institutions can share models but not data. Its answer, Generate to Discriminate (G2D), is to spend synthetic data on learning which expert to call at test time rather than on retraining the label classifier. Across vision and text benchmarks, routing on synthetic samples outperforms generative replay with the same samples, and the gap grows on datasets with large domain shifts. The paper also introduces a dermatology benchmark to show the failure of prompt-based methods when pretraining does not cover the domain.","feed_headline":"Synthetic data routes better than it replays","feed_subtitle":"In continual learning, G2D uses generated samples to train a domain router and beats generative replay on vision and text benchmarks.","key_machinery":"The load-bearing object is the domain discriminator (router) trained on the union of synthetic sets $M_1 \\cup \\dots \\cup M_t$, where $M_t$ is sampled from a generator finetuned on domain $D_t$. At inference, the router predicts a domain identity and dispatches the sample to the expert finetuned on that domain; for out-of-distribution samples it chooses the most similar seen domain. The same synthetic samples, when instead used to augment the label classifier (generative replay), give consistently worse results, which the paper reads as evidence that modeling domain differences is easier than modeling class-label differences for the generators considered.","core_discovery":"The central claim is that in domain-incremental continual learning, if a per-domain generator produces a fixed set of synthetic samples, training a domain discriminator on those samples and using it to route test inputs to domain-specific experts yields higher average accuracy than feeding the same samples back into the classifier as rehearsal data. The authors call this Generate to Discriminate (G2D). They report consistent gains over generative replay, experience replay, regularization, and prompt-based baselines on DomainNet, CORe50, the new DermCL benchmark, and a four-dataset question-answering sequence, and they show the synthetic-trained router approaches the accuracy of a router trained on real data.","pith_inferences":["If domain discrimination is genuinely easier than class discrimination for the same generator, then generative replay's known weakness is not sample quality alone; the target task itself is harder, so synthetic data should be allocated to auxiliary identification tasks.","The same router-on-synthetic-samples design transfers naturally to federated or institution-separated settings, where each site can share a local generator or generated samples rather than raw data.","A direct extension is to measure router accuracy against the number of synthetic samples used; if it saturates early, G2D could operate with far fewer generated samples than generative replay needs, lowering compute and exposure."],"forward_implications":["With no real data or domain labels from earlier domains, a deployment can still route each new input to the right expert; G2D needs only the current domain's real data plus generated samples from all seen domains.","For a fixed number of generated examples, the paper's results say the highest-accuracy use is building the router, not augmenting the classifier: G2D beats Generative Replay on DomainNet, CORe50, DermCL, and QA.","The synthetic-trained router nearly matches a router trained on real samples, so the routing step is not the bottleneck; remaining gaps to the multi-task upper bound come from expert training, not domain identification.","On out-of-distribution test domains (CORe50), the router behaves as a similarity function and sends samples to the most similar seen expert, extending the method beyond strict domain membership."],"supporting_citations":[{"why":"It defines generative replay, replaying synthetic samples into the label classifier; this is the baseline the paper's central claim is measured against.","marker":"[55]"},{"why":"It is the prompt-based routing baseline that uses K-means centroids and KNN at inference; G2D's router is compared against it on vision benchmarks.","marker":"[67]"},{"why":"It introduces the question-answering domain-incremental benchmark and the episodic-memory MbPA baseline used in the text experiments.","marker":"[13]"},{"why":"It provides the text-to-image diffusion generator that produces the synthetic vision samples used to train the domain discriminator.","marker":"[52]"},{"why":"It provides the pretrained T5 backbone that, with prompt tuning, generates the synthetic text samples for the QA experiments.","marker":"[50]"},{"why":"It supplies the ViT backbone used for both expert classifiers and the domain discriminator in vision.","marker":"[17]"},{"why":"It supplies the BERT backbone used for the text experts and discriminator.","marker":"[29]"},{"why":"It defines the experience replay baseline with real stored samples, a quasi-oracle comparator against G2D.","marker":"[7]"}],"fun_headline_variants":["Synthetic data: better as a router than as a rehearsal set","G2D: Turn synthetic data into a domain router, not a replayer","In continual learning, use synthetic data to discriminate, not to replay","Domain experts: synthetic data picks the right one better than replay","Why synthetic data is more powerful for routing than for training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic samples generated from each domain's fine-tuned generator carry enough domain-specific signal that a router trained only on them will recognize real test samples from that domain.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data: better as a router than as a rehearsal set","G2D: Turn synthetic data into a domain router, not a replayer","In continual learning, use synthetic data to discriminate, not to replay","Domain experts: synthetic data picks the right one better than replay","Why synthetic data is more powerful for routing than for training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000733,"raw_usage":{"total_tokens":3241,"prompt_tokens":871,"completion_tokens":2370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2279}},"tokens_in":487,"tokens_out":2370,"duration_ms":13554,"temperature":1.0,"reasoning_tokens":2279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:52:47.390580+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the G2D router on synthetic samples and evaluate it on real held-out samples domain by domain; if its routing accuracy approaches chance on any domain while a router trained on real samples from that domain is accurate, the transfer premise fails. A sharper test is to replace the per-domain generator with a class-conditional but domain-blind generator; if G2D still routes well, the signal is not domain-specific and the method's stated mechanism is wrong.","supporting_citations":[{"cited_title":"S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning","cited_arxiv_id":null,"evidence_quote":"It is the prompt-based routing baseline that uses K-means centroids and KNN at inference; G2D's router is compared against it on vision benchmarks."},{"cited_title":"Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019","cited_arxiv_id":null,"evidence_quote":"It introduces the question-answering domain-incremental benchmark and the episodic-memory MbPA baseline used in the text experiments."}],"review_version":1}