Pith. sign in

REVIEW 4 major objections 7 minor 3 cited by

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Placing low-rank adapters at the layers a task's gradients highlight cuts forgetting to near zero and lifts average continual-tuning performance by 15 points.

desk verdict Per-task frozen LoRA experts plus autoencoder routing likely explain the near-zero forgetting; the dynamic allocator's contribution is not yet isolated, but the paper is worth refereeing. read the letter →

arxiv 2506.11672 v1 pith:TNJ36VQG submitted 2025-06-13 cs.CV

Chendi Ge , Xin Wang , Zeyang Zhang , Hong Chen , Jiapei Fan , Longtao Huang , Hui Xue , Wenwu Zhu This is my paper
classification cs.CV
keywords continualmultimodalinstructiontuningmixtureofLoRAexpertsdynamicexpertallocationzero-costproxiescatastrophicforgettinginter-modalcurriculumvision-languagemodelsparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a multimodal large language model (the kind that answers questions about images) can keep learning new instruction-following tasks in sequence without losing earlier ones, if the model is allowed to change its own adaptation architecture for each task. Its proposed method, D-MoLE, takes a 1% sample of each new task, measures how strongly every layer of the language model and the vision encoder reacts to that sample, and spends a fixed budget of small low-rank adapters (LoRA experts) only on the most reactive layers and the modality that needs more learning. The claimed payoff is a 73.87 average score versus 58.79 for the best fixed-architecture baseline, with backward transfer of only −1.49 instead of −21.31. If true, this recasts continual multimodal tuning as an architecture-allocation problem that can be solved cheaply at the start of each task, without replay buffers or regularization losses.

What carries the argument

The load-bearing mechanism is a two-part allocator driven by training-free zero-cost proxies, each proxy being a single forward and backward pass on a 1% random subset that yields per-layer L2 gradient norms. The dynamic layer-wise expert allocator ranks all LLM and vision-encoder layers by these norms and places LoRA experts only in the top-ranked layers within each module's budget, so every task receives a different network layout. The gradient-based inter-modal continual curriculum turns the same norms into an update ratio between the LLM and the vision encoder, splitting the total parameter budget by task-specific difficulty. A set of lightweight task-specific autoencoders acts as a router: their reconstruction losses select which prior experts to activate and signal unseen tasks, which fall back to the pretrained backbone.

What would settle it

Run D-MoLE on the same nine-task benchmark with an equal parameter budget but with LoRA experts placed at the layers the proxy ranks worst, or at random layers: if average score and backward transfer stay near the proxy-guided result, the gradient signal is not what drives the improvement. A sharper version is to recompute the per-layer gradient norms on a new task after several earlier tasks have been trained, using the current adapted model instead of the pretrained one, and check whether the chosen layers change and whether the new choice improves final performance.

Watch

Extended reading notes

Core claim

The central discovery claimed is that the two obstacles to continual multimodal instruction tuning are not primarily forgetting but task architecture conflict and modality imbalance—different tasks want adaptation in different transformer layers, and different tasks lean more on language or on vision. D-MoLE is the proposed resolution: per-layer gradient norms computed on a tiny subset, without any training, rank the layers of both the LLM and the vision encoder; a gradient-based inter-modal curriculum converts those norms into a budget split between the two modules; and the LoRA experts are installed only at the top-ranked layers, with previous experts kept frozen and reused through an autoencoder router that selects the most relevant prior task by reconstruction loss. On the paper's nine-task benchmark this yields near-zero backward transfer, higher average and final-task scores than all compared LoRA, MoLE, and regularization baselines, and general multimodal benchmarks close to the pretrained zero-shot model.

Load-bearing premise

The load-bearing premise is that gradient norms from a 1% random sample of the new task, computed on the frozen pretrained model, reliably reveal which layers and which modality will actually need adaptation for that task.

Editorial extensions

If this is right

  • D-MoLE with the same trainable-parameter budget as baselines raises average CMIT score from 58.79 (best baseline) to 73.87, final-task score from 62.04 to 82.18, and BWT from −21.31 to −1.49.
  • General MLLM abilities degrade far less under continual tuning: MME, MMMU, and POPE scores stay close to the zero-shot pretrained model, outperforming Seq-FT and O-LoRA.
  • Because only a subset of layers is adapted, total training time is reduced (12.40 h vs 14.87 h for O-LoRA and 23.03 h for Sparse MoLE), so the dynamic architecture is not just better but cheaper.
  • Ablations attribute the gain to both components: removing the layer-wise allocator (v4) or the inter-modal curriculum (v3) drops all metrics, and fine-tuning only one modality (v1, v2) is worse than the balanced dynamic version.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the proxy is measured on the pretrained model before any task-specific learning, its accuracy could decay as the model drifts; a natural test is to recompute gradient norms on the current adapted model and compare expert placements.
  • The near-zero backward transfer may partly reflect the autoencoder router providing strong task identity; a stricter evaluation would blur or interleave task boundaries so routing must rely on content rather than dataset signatures.
  • The inter-modal budget split extends directly to more than two modalities; scoring audio, video, or depth encoders the same way would generalize the curriculum, though the router would need a joint embedding.
  • A cheap causal check, not reported in the paper, is reversed allocation: spending the budget on the lowest-ranked layers under the same curriculum should significantly hurt if the proxy is genuinely informative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes D-MoLE, a method for continual multimodal instruction tuning (CMIT) that evolves the architecture of a multimodal LLM under a parameter budget. Two components are introduced: (1) a dynamic layer-wise expert allocator that uses gradient-norm zero-cost proxies on a 1% sample of the new task to decide which transformer layers receive new LoRA experts, and (2) a gradient-based inter-modal curriculum that splits the parameter budget between the LLM and vision encoder according to estimated per-module difficulty. Task-specific autoencoders route inputs to previously trained (frozen) LoRA experts at test time. The authors report large gains over baselines on a nine-task benchmark (AVG 73.87 vs. 58.79 for the best baseline, BWT -1.49), plus ablation and efficiency analyses. The claimed contribution is the first architectural-evolution perspective on CMIT.

Significance. If the central claim holds, the paper would make a useful contribution by showing that budgeted, task-specific architectural allocation can mitigate forgetting in multimodal instruction tuning without replay. The paper is also transparent in several respects: it provides a full algorithmic description, a sensitivity analysis of the routing threshold (Table 7), a parameter-budget comparison (Table 11), and time-cost breakdowns (Tables 8-10). The zero-cost proxy and autoencoder router are concrete and reproducible design choices. However, the significance is currently undercut by how the headline results are computed and by the absence of a control that isolates the dynamic-allocation mechanism from the simpler effect of frozen per-task experts.

major comments (4)
  1. [§5.1, Evaluation Metrics and Table 2] The AVG metric is an unnormalized mean of task scores with incompatible scales: CIDEr for captioning (roughly 0-150), top-1 accuracy for VQA (0-100), and IoU for grounding (0-100). The headline gain of 15.08 points (73.87 vs. 58.79) is therefore dominated by the two highest-scale tasks, VizWiz-Cap (148.19 vs. 99.07, a +49.12 CIDEr gap) and SK-VG (57.29 vs. 35.25, a +22.04 IoU gap). On lower-scale tasks such as PMC-VQA the gain is only 1.13 accuracy points. The abstract's '15% average improvement' also appears to be an absolute raw-score gap, not a relative percent gain. To make the aggregate claim meaningful, the authors should report per-task normalized scores (e.g., relative to the joint-learning or zero-shot bound) or another scale-invariant aggregation, and should state the aggregate as a point gap with the relative percentage computed correctly.
  2. [§4.3, Eq. (2); §5.4, Figure 4] Because previous-task LoRA experts are frozen and only newly allocated experts are trainable, the near-zero BWT (-1.49) is largely a structural consequence of parameter isolation, not necessarily of the dynamic layer-wise allocator or the inter-modal curriculum. The paper does not provide a control that isolates these components: a baseline that trains one frozen uniform LoRA expert per task (no gradient-based allocation, no curriculum) and uses either ground-truth task IDs or the same autoencoder router for retrieval. The v4 ablation removes both dynamic allocation and task-specific routing simultaneously, so it cannot show that dynamic allocation itself is what drives the gap. Without this control, the central claim that 'dynamic layer-wise expert allocation with an inter-modal curriculum mitigates forgetting' is not fully supported; the reported Last and BWT gains might be replicated by frozen per-task expert isolation alone.
  3. [All experiments (Tables 2, 3, 7, 11; Figure 4)] No multiple seeds, standard deviations, or significance tests are reported anywhere in the manuscript. The claimed improvements are large in aggregate, but several per-task differences are small (e.g., AVG KVQA 40.87 vs. 40.56; Last VizWiz-VQA 69.16 vs. 68.18), and continual instruction tuning is known to be sensitive to initialization and data ordering. The authors should rerun their main comparison (at least D-MoLE, O-LoRA, and EWC-LoRA) with 3 seeds and report mean ± std, and state whether the AVG/Last/BWT gaps are statistically distinguishable.
  4. [§4.1, Eq. (3) and §4.3, Step 2] The allocation proxy is computed as the gradient norm with respect to the pretrained weights W0 on a 1% sample, but at task t the model already contains LoRA experts from tasks 1..t-1. The paper does not test whether this pretrained-gradient proxy remains predictive of where LoRA is needed at later stages of the continual sequence. The authors should provide a direct check, for example by correlating the proxy ranking with post-training per-layer LoRA weight change or validation improvement, or by comparing the proxy-allocated architecture against an oracle that selects layers by validation performance. Without such a check, the core mechanism of the dynamic allocator is an untested assumption.
minor comments (7)
  1. [Abstract and §5.2] The abstract states a '15% average improvement over the best baseline', but §5.2 reports a 15.08-point gap on the AVG metric; please clarify whether the claimed improvement is in raw points or relative percent, and ensure the abstract matches the actual calculation.
  2. [Table 2] The BWT column for PMC-VQA is listed as '-' for every method, and the average BWT appears to be computed over the first eight tasks only. Please state this explicitly in the table footnote or metric description.
  3. [§5.4, Figure 4] Figure 4 reports ablation results without numeric values. Please provide the exact AVG, Last, and BWT numbers for v1-v4 and the best baseline in the text or in a table, so that the magnitude of each ablation effect can be assessed.
  4. [§4.1, Eqs. (4)-(5) and Eq. (15)] The budget B^t_M from Eq. (15) is a product of a ratio r^t_M and B_total, and Eq. (5) uses B^t_M as an integer rank cutoff. The rounding rule for non-integer budgets should be specified.
  5. [§4.1, Eq. (10) and Appendix G] The threshold tau^t is described as set 'moderately above the observed loss range' on the training set, but no concrete procedure is given for how 'moderately' is operationalized. Please specify the exact construction, even if it is heuristic.
  6. [Appendix K, Proof of Theorem 3.1] The proof concludes that the assumption of equal gradient norms 'would necessitate reconciling incompatible updates—a requirement that violates the sequential training paradigm.' This is not a formal mathematical contradiction; it is a paraphrase of the intuition. Either make the argument rigorous (e.g., by showing the updates increase the loss on one task in expectation) or state Theorem 3.1 as a motivating observation rather than a theorem.
  7. [§1 and §6] The claim of being 'the first study of continual learning for MLLMs from an architectural perspective' is strong given that related work (CoIN, MoA, LoraMoE, Continual LLaVA) also addresses architecture or expert allocation in continual instruction tuning. Please soften the claim or clearly differentiate the specific architectural-evolution contribution.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity: the reported gains are measured against external baselines, with only minor self-citations and a non-valid auxiliary proof.

full rationale

The central claim of D-MoLE is an empirical comparison against external baselines (Seq-FT, EWC-LoRA, Dense/Sparse MoLE, MoLA, O-LoRA) on the nine-task CMIT benchmark (Table 2). The reported AVG, Last, and BWT numbers are measured outcomes, not quantities derived from the method's own equations, so the headline 15.08-point improvement cannot reduce to an input. The allocator (Eq. 4-5) and inter-modal budget (Eq. 13-15) use gradient norms computed on a 1% sample of the current task's training data; this is a training-time heuristic, and no reported metric is fitted to it or renamed as a prediction. Self-citations (Ge et al. 2025, Wang et al. 2025, and the group's curriculum/NAS papers) appear only in framing or related work and are not load-bearing for the empirical result. Two caveats are worth recording but are not circular. First, near-zero BWT (-1.49) is largely a design consequence of Eq. (2), which freezes all previous-task LoRA experts; BWT then measures mainly routing stability, so it should not be read as independent evidence for the dynamic allocator. Second, the 'formal' proof in Appendix K is not logically valid: the step 'By Assumption K.2 ... This implies the existence of at least one critical layer l* where their expected gradients differ fundamentally' imports the target phenomenon, and equal-norm non-collinear gradients do not force norm inequality. This is a proof gap, not a circular reduction of the empirical claim. Verdict: no significant circularity; score 2 reflects the minor self-citations and the theory-quality caveat.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The method rests on heuristic proxies (gradient norms on 1% samples) and a task router. The theoretical analysis (Theorem 3.1) adds no predictive power. The benchmark aggregate metric is a raw average of incompatible task metrics.

free parameters (4)
  • Total parameter budget B_total = 24 LoRA experts (allocation ratio 0.5, rank 8)
    Chosen manually to keep trainable parameters equal across baselines; controls how many layers get experts.
  • TopK for expert activation = 2
    Number of prior-task experts activated at evaluation (Eq. 12).
  • Autoencoder reconstruction threshold tau^t = Set moderately above observed loss range (Appendix G)
    Threshold for task filtering in Eq. (10); sensitivity analysis shows robustness.
  • Sampling ratio for zero-cost proxy = 1% of training data
    Size of the subset used to compute gradient norms; if too small the proxy is noisy.
assumptions (3)
  • domain assumption Gradient norms on a 1% subset of task data, computed on the pretrained model, identify the layers and modalities most in need of adaptation for that task.
    Used throughout Section 4.1/4.2 to pick layers and budgets; no theoretical or empirical validation beyond the final benchmark result.
  • domain assumption Task identity can be inferred from pooled image and text features via per-task autoencoders, and tasks are separable by reconstruction error.
    Needed for routing (Eq. 6-12); supported only by t-SNE and sensitivity analysis in Appendices F/G.
  • ad hoc to paper Assumptions K.2 (task heterogeneity) and K.3 (non-collinear gradients) from the proof of Theorem 3.1.
    These assumptions essentially assert the conclusion (that tasks differ gradient-wise), so the theorem is a restatement rather than a derivation.
invented entities (1)
  • Task-specific autoencoder routers
    purpose: Detect task identity and route inputs to the correct LoRA experts during evaluation without task labels.
    Their effectiveness is shown only on the paper's self-constructed benchmark via t-SNE and sensitivity analysis; no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning." pith.science (2026). https://pith.science/paper/TNJ36VQG

@misc{pith2026250611672,
  author       = {Pith},
  title        = {Pith review of: Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TNJ36VQG}},
  note         = {Machine review of arXiv:2506.11672}
}
read the original abstract

Continual multimodal instruction tuning is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving tasks. However, most existing methods adopt a fixed architecture, struggling with adapting to new tasks due to static model capacity. We propose to evolve the architecture under parameter budgets for dynamic task adaptation, which remains unexplored and imposes two challenges: 1) task architecture conflict, where different tasks require varying layer-wise adaptations, and 2) modality imbalance, where different tasks rely unevenly on modalities, leading to unbalanced updates. To address these challenges, we propose a novel Dynamic Mixture of Curriculum LoRA Experts (D-MoLE) method, which automatically evolves MLLM's architecture with controlled parameter budgets to continually adapt to new tasks while retaining previously learned knowledge. Specifically, we propose a dynamic layer-wise expert allocator, which automatically allocates LoRA experts across layers to resolve architecture conflicts, and routes instructions layer-wisely to facilitate knowledge sharing among experts. Then, we propose a gradient-based inter-modal continual curriculum, which adjusts the update ratio of each module in MLLM based on the difficulty of each modality within the task to alleviate the modality imbalance problem. Extensive experiments show that D-MoLE significantly outperforms state-of-the-art baselines, achieving a 15% average improvement over the best baseline. To the best of our knowledge, this is the first study of continual learning for MLLMs from an architectural perspective.

Figures

Figures reproduced from arXiv: 2506.11672 by the authors.

Figure 1
Figure 1. Sensitivity of different transformer layers across [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Overall framework of D-MoLE. During training on task [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Results of ablation experiments. The results in [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: t-sne of the reconstructed sequence embeddings of the task-specific autoencoders. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Architecture evolution dynamics during continual multimodal instruction tuning. Each transformer layer is divided [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: Expert activation dynamics at different stages of continual multimodal instruction tuning. Each cell indicates how [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A private-plus-shared LoRA MoE with layer-adaptive momentum transfer enables continual anomaly detection on MLLMs and beats prior continual-learning baselines across class, domain, and modality shifts.

  2. PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

    cs.LG 2026-01 conditional novelty 5.0 of 10

    Routing MoE-LoRA experts by the strength of their low-rank activation and regularizing important rank directions reduces forgetting in continual instruction tuning of multimodal LLMs.

  3. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Reference graph

Works this paper leans on

29 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [6]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y ., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394,

    arXiv

  2. [7]

    Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,

    Gao, C., Chen, K., Rao, J., Sun, B., Liu, R., Peng, D., Zhang, Y ., Guo, X., Yang, J., and Subrahmanian, V . Higher layers need more lora experts.arXiv preprint arXiv:2402.08562,

    arXiv

  3. [8]

    To reduce the number of training samples in the OCR-VQA dataset, we consolidate QA pairs for the same image into multi-round conversations. The image feature dimension dv is 1024, and the text feature dimension dt is 2048, resulting in an input and output dimension of 3072 for the autoencoder, with a hidden dimension of

    work page

  4. [11]

    Llaca: Multimodal large language continual assistant

    Qiao, J., Zhang, Z., Tan, X., Qu, Y ., Ding, S., and Xie, Y . Llaca: Multimodal large language continual assistant. arXiv preprint arXiv:2410.10868,

    arXiv

  5. [12]

    Alphalora: Assigning lora experts based on layer training quality

    Qing, P., Gao, C., Zhou, Y ., Diao, X., Yang, Y ., and V osoughi, S. Alphalora: Assigning lora experts based on layer training quality. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20511–20523,

    work page 2024

  6. [13]

    Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 2001–2010,

    work page 2001

  7. [16]

    Conpet: Continual parameter- efficient tuning for large language models.arXiv preprint arXiv:2309.14763,

    Song, C., Han, X., Zeng, Z., Li, K., Chen, C., Liu, Z., Sun, M., and Yang, T. Conpet: Continual parameter- efficient tuning for large language models.arXiv preprint arXiv:2309.14763,

    arXiv

  8. [17]

    Team, K., Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599,

    arXiv

Show all 29 references
  1. [18]

    Orthogonal subspace learning for language model continual learning

    Wang, X., Chen, T., Ge, Q., Xia, H., Bao, R., Zheng, R., Zhang, Q., Gui, T., and Huang, X. Orthogonal subspace learning for language model continual learning. InFind- ings of the Association for Computational Linguistics: EMNLP 2023, pp. 10658–10671, 2023a. Wang, X., Liu, Y .,...

    2023

  2. [19]

    Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions

    Wang, Y ., Liu, Y ., Shi, C., Li, H., Chen, C., Lu, H., and Yang, Y . Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computat...

    2024 arXiv

  3. [20]

    Be- yond anti-forgetting: Multimodal continual instruction tuning with positive forward transfer.arXiv preprint arXiv:2401.09181,

    Zheng, J., Ma, Q., Liu, Z., Wu, B., and Feng, H. Be- yond anti-forgetting: Multimodal continual instruction tuning with positive forward transfer.arXiv preprint arXiv:2401.09181,

    arXiv

  4. [21]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

    Zhu, J., Wang, W., Chen, Z., Liu, Z., Ye, S., Gu, L., Duan, Y ., Tian, H., Su, W., Shao, J., et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479,

    arXiv

  5. [24]

    By combining LLMs with multimodal encoders, they support tasks such as image captioning and visual question answering

    have attracted attention for their ability to handle text along with other modalities, such as images and videos. By combining LLMs with multimodal encoders, they support tasks such as image captioning and visual question answering. Pretraining MLLMs requires extensive data an...

    2023

  6. [25]

    Model expansion increases capacity to handle new tasks while preserving prior information

    apply constraints to important parameters. Model expansion increases capacity to handle new tasks while preserving prior information. Mixture of ExpertsThe Mixture of Experts (MoE) model, introduced by (Jacobs et al., 1991), uses a gating mechanism to determine each expert’s c...

    1991

  7. [26]

    Recent research has extended the Mixture of Experts (MoE) framework (Jacobs et al., 1991; Shazeer et al.,

    further optimize expert activation for large NLP and vision models. Recent research has extended the Mixture of Experts (MoE) framework (Jacobs et al., 1991; Shazeer et al.,

    1991

  8. [27]

    These models are called Mixture of LoRA Experts (MoLE)

    as experts within parameter-efficient fine-tuning (PEFT) frameworks. These models are called Mixture of LoRA Experts (MoLE). Curriculum LearningCurriculum learning organizes training in an easy-to-hard order to improve optimization and generalization (Wang et al., 2021). It ha...

    2021

  9. [28]

    Recently, curriculum learning is also widely adopted in LLMs’ pretraining process, e.g., Kimi K1.5 (Team et al., 2025), DeepSeek-Prover-V2 (Ren et al., 2025), Seed-Coder

    further enable systematic evaluation. Recently, curriculum learning is also widely adopted in LLMs’ pretraining process, e.g., Kimi K1.5 (Team et al., 2025), DeepSeek-Prover-V2 (Ren et al., 2025), Seed-Coder. Our work introduces a gradient-based inter-modal curriculum for CMIT...

    2025

  10. [128]

    Each autoencoder is trained for 100 epochs with a learning rate of 1e-3. B. Additional Related Work Multimodal Large Language ModelsMultimodal large language models (Dai et al., 2023; McKinzie et al., 2024; Zhu et al., 2025; Bai et al.,

    2023

  11. [1991]

    Jung, M. J. and Kim, J. Pmoe: Progressive mixture of ex- perts with asymmetric transformer for continual learning. arXiv preprint arXiv:2407.21571,

    arXiv

  12. [2014]

    adaptability and its capacity to maintain performance on previous tasks while learning new ones

    Visual Grounding SK-VG (Chen et al., 2023b) <image> Based on the Knowledge:[KNOWLEDGE] Please provide the bounding box coordinates of the region<ref>[DESCRIPTION]</ref>. adaptability and its capacity to maintain performance on previous tasks while learning new ones. F. Analysi...

  13. [2017]

    Deepseek- prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition.arXiv preprint arXiv:2504.21801,

    Ren, Z., Shao, Z., Song, J., Xin, H., Wang, H., Zhao, W., Zhang, L., Fu, Z., Zhu, Q., Yang, D., et al. Deepseek- prover-v2: Advancing formal mathematical reasoning via reinforcement learning for subgoal decomposition.arXiv preprint arXiv:2504.21801,

    arXiv

  14. [2018]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923,

    arXiv

  15. [2019]

    A., Rabinowitz, N

    Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks.arXiv preprint arXiv:1606.04671,

    arXiv

  16. [2020]

    Continual instruc- tion tuning for large multimodal models.arXiv preprint arXiv:2311.16206,

    He, J., Guo, H., Tang, M., and Wang, J. Continual instruc- tion tuning for large multimodal models.arXiv preprint arXiv:2311.16206,

    arXiv

  17. [2021]

    X., and Wen, J.-R

    Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  18. [2022]

    Continual llava: Con- tinual instruction tuning in large vision-language models

    Cao, M., Liu, Y ., Liu, Y ., Wang, T., Dong, J., Ding, H., Zhang, X., Reid, I., and Liang, X. Continual llava: Con- tinual instruction tuning in large vision-language models. arXiv preprint arXiv:2411.02564,

    arXiv

  19. [2023]

    Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin

    Dou, S., Zhou, E., Liu, Y ., Gao, S., Shen, W., Xiong, L., Zhou, Y ., Wang, X., Xi, Z., Fan, X., Pu, S., Zhu, J., Zheng, R., Gui, T., Zhang, Q., and Huang, X. Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin. InProceedings of the 62...

    1932

  20. [2024]

    Improving multi-modal learning with uni-modal teachers.arXiv preprint arXiv:2106.11059,

    Du, C., Li, T., Liu, Y ., Wen, Z., Hua, T., Wang, Y ., and Zhao, H. Improving multi-modal learning with uni-modal teachers.arXiv preprint arXiv:2106.11059,

    arXiv

  21. [2025]

    Multimodal continual graph learning with neural architecture search

    Cai, J., Wang, X., Guan, C., Tang, Y ., Xu, J., Zhong, B., and Zhu, W. Multimodal continual graph learning with neural architecture search. InProceedings of the ACM Web Conference 2022, pp. 1292–1300,

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.