REVIEW 4 major objections 3 minor 1 cited by
Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning
T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that a two-level routing grouped mixture-of-experts can make CLIP learn sequential multi-domain tasks while suffering less catastrophic and forward forgetting than prior methods, using fewer trainable parameters.
desk verdict Plausible integration of known MoE ideas, but the abstract's performance claims are unsupported and the MLLM routing dependency is load-bearing; deserves a referee look, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two-Level Routing Grouped Mixture-of-Experts (TRGE): a dynamically expanding expert architecture built on a frozen CLIP backbone. Each task receives a dedicated expert group; an intra-group router picks the most appropriate expert within a group, and an inter-group router aggregates outputs from several groups selected by task identity and prototype similarity. A training-progress-weighted fusion layer blends the frozen CLIP output with the routed adapter output for unseen samples.
What would settle it
Run TRGE on unseen test images but give the MLLM a deliberately wrong task identifier (or corrupt the semantic description). If accuracy drops sharply compared to the oracle-identifier setting, then the reported gains depend on near-perfect task recognition. A simpler check: measure TRGE's accuracy when the MLLM's task classification accuracy falls below some threshold (e.g., 90 percent) on a held-out domain.
Extended reading notes
Core claim
The central claim is that separation and collaboration can be combined rather than traded off. TRGE separates knowledge by giving each task a dedicated expert group, while the static size of the group keeps routing complexity bounded, and an intra-group router prevents overfitting as the number of experts grows. Collaboration is achieved by an inter-group router that takes the MLLM-recognized task identifier plus task prototype distances to select and weight relevant expert groups. The method further introduces training-progress-based dynamic fusion between the frozen CLIP features and the TRGE adapter output, which they argue mitigates forward forgetting on unseen samples. The paper reports
Load-bearing premise
The method works only if a multimodal language model reliably recognizes which task a given test image belongs to, because the inter-group router uses that identifier to select expert groups.
Editorial extensions
If this is right
- If TRGE is right, a single frozen CLIP can be incrementally specialized to many domains without retraining the backbone, which makes continual learning cheap in parameters and memory.
- The inter-group routing means tasks are not learned in isolation: later tasks can actively reuse earlier expert groups, so positive transfer is possible instead of only protecting old tasks.
- Because the number of experts grows but the group size stays fixed, the routing decision space does not blow up, so the method scales to long task sequences.
- Dynamic fusion with training progress gives a direct lever to control how much pre-trained knowledge to trust at any moment, which addresses forward forgetting on unseen data.
Reading between the lines
- The reliance on an MLLM for task identification is the main external dependency; if the MLLM is removed or fails, TRGE degrades to a prototype-distance-only router, and the paper's own comparison would be a useful ablation to report.
- The same two-level routing could be applied beyond vision-language models to other frozen foundation models (e.g., text or speech encoders), since the routing and fusion logic is not CLIP-specific.
- An explicit test of the 'collaboration' claim would be to measure whether later tasks actually improve earlier-task accuracy via inter-group routing, rather than only matching single-task baselines.
- The training-progress fusion schedule could be tuned per-domain; one could test whether a learned or uncertainty-based fusion weight outperforms the fixed progress schedule.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TRGE, a two-level routing grouped mixture-of-experts method for multi-domain continual learning (MDCL) built on a frozen CLIP backbone. TRGE dynamically expands expert groups per task, uses an intra-group router to limit routing complexity, and uses an inter-group routing policy based on task identifiers and task prototype distances to combine relevant expert groups. Task identifiers are obtained from a Multimodal Large Language Model (MLLM) that generates semantic task descriptions. A dynamic fusion of frozen CLIP and learned adapter outputs is introduced to mitigate forward forgetting. The abstract claims that TRGE outperforms other advanced methods with fewer trainable parameters.
Significance. If substantiated, the proposed method would be a meaningful contribution to parameter-efficient continual learning under both domain and class shifts, especially the combination of grouped MoE with MLLM-based task identification and prototype-based inter-group routing. However, the abstract contains no experimental evidence: no datasets, task sequences, baseline names, metrics, or error bars are reported. The significance therefore remains entirely conditional on the full manuscript providing such evidence.
major comments (4)
- [Abstract] The central claim 'our method outperforms other advanced methods with fewer trainable parameters' is asserted without any supporting experimental detail. No datasets, domain/class shift protocols, baselines, metrics, or statistical significance information are presented. Because this claim is the paper's headline, it is load-bearing. The full text must contain a complete evaluation; the abstract alone cannot support it.
- [Abstract] The parameter-efficiency claim is ambiguous under the described dynamic expansion. If a new expert group is assigned for each task, total parameters grow with the number of tasks. The claim of 'fewer trainable parameters' must specify whether parameters are measured at a fixed task, at the end of the sequence, or per task, and must include all routing components (intra-group, inter-group, and the MLLM if it is trainable or used for inference). The abstract does not define the parameter accounting, making the comparison unverifiable.
- [Abstract] Inter-group routing relies on task identifiers generated by an MLLM 'to recognize the correct task identifier.' The reliability of this component is load-bearing: if the MLLM misidentifies the task for an unseen sample, the wrong expert groups will be activated and the claimed collaboration benefit collapses. The abstract reports no MLLM recognition accuracy, no ablation against oracle or random task identifiers, and no failure analysis for out-of-domain inputs. These are needed to assess whether the method degrades gracefully under realistic MLLM error rates.
- [Abstract] The dynamic fusion schedule is described only as 'based on training progress' and is claimed to mitigate forward forgetting. No definition of forward forgetting, no measurement protocol, and no description of the fusion schedule (e.g., linear, inverse progress, trainable weights) are given. Without these details, the forward-forgetting claim is not assessable even from the abstract.
minor comments (3)
- [Abstract] Grammar: 'maintains the static experts count within the group' should be 'maintains the static expert count within the group.'
- [Abstract] Phrasing: 'MLLMs which own powerful multimodal comprehension capabilities' could be simplified to 'MLLMs that possess strong multimodal comprehension capabilities.'
- [Abstract] The term 'forward forgetting' is used without definition. If this is a standard term in the field, a brief clarification or citation would help the reader.
Circularity Check
No circularity detectable from the abstract; the method's components are externally evaluated benchmarks, not derived from their own outputs.
full rationale
The provided text is abstract-only, with no equations, no fitted parameters renamed as predictions, and no derivation chain that reduces to its own inputs. The TRGE method is described as a system architecture: per-task expert groups, intra-group and inter-group routers, MLLM-based task identification, and dynamic fusion. Each component is proposed as a mechanism, not as a quantity fitted to the evaluation data. Comparisons are made against other methods on multi-domain continual learning benchmarks, which are external to the paper. The MLLM-based task identifier is a potential robustness risk, but it is not circular: using a task label to route experts is not equivalent to defining the task label in terms of the routing output. No self-citation is mentioned. Without access to the full text, no specific reduction can be exhibited, and per the hard rules, speculation about hidden circularity is not permitted. Therefore the honest finding is no significant circularity, score 0.
Assumptions & free parameters
free parameters (3)
- Static expert group size
- Inter-group routing combination weights
- Fusion schedule
assumptions (2)
- domain assumption A frozen CLIP encoder provides a transferable feature space across all domains in the sequence.
- domain assumption An MLLM can correctly map each input to a semantic task description and recover the true task identifier at inference.
Cite this review
Pith. "Pith review of Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning." pith.science (2026). https://pith.science/paper/N2PG4VLZ
@misc{pith2026250807738,
author = {Pith},
title = {Pith review of: Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/N2PG4VLZ}},
note = {Machine review of arXiv:2508.07738}
}
read the original abstract
Multi-Domain Continual Learning (MDCL) acquires knowledge from sequential tasks with shifting class sets and distribution. Despite the Parameter-Efficient Fine-Tuning (PEFT) methods can adapt for this dual heterogeneity, they still suffer from catastrophic forgetting and forward forgetting. To address these challenges, we propose a Two-Level Routing Grouped Mixture-of-Experts (TRGE) method. Firstly, TRGE dynamically expands the pre-trained CLIP model, assigning specific expert group for each task to mitigate catastrophic forgetting. With the number of experts continually grows in this process, TRGE maintains the static experts count within the group and introduces the intra-group router to alleviate routing overfitting caused by the increasing routing complexity. Meanwhile, we design an inter-group routing policy based on task identifiers and task prototype distance, which dynamically selects relevant expert groups and combines their outputs to enhance inter-task collaboration. Secondly, to get the correct task identifiers, we leverage Multimodal Large Language Models (MLLMs) which own powerful multimodal comprehension capabilities to generate semantic task descriptions and recognize the correct task identifier. Finally, to mitigate forward forgetting, we dynamically fuse outputs for unseen samples from the frozen CLIP model and TRGE adapter based on training progress, leveraging both pre-trained and learned knowledge. Through extensive experiments across various settings, our method outperforms other advanced methods with fewer trainable parameters.
Forward citations
Cited by 1 Pith paper
-
Enhancing Small-Scale Dataset Expansion with Triplet-Connection-based Sample Re-Weighting
A triplet-connection re-weighting scheme for AI-generated training images that, per the abstract, never hurts augmentation pipelines and beats prior methods on natural and medical benchmarks.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.