Pith. sign in

REVIEW 3 major objections 6 minor 52 references

DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DistMoE composes independently trained private experts at the token level, with no rehearsal of any client's private data.

desk verdict A plausible rehearsal-free MoE composition method for MLLMs; the Stage-II isotropic residual calibration is genuinely new, but the paper never proves that per-client router logits are cross-comparable, and the main Table has a copy-paste error. read the letter →

arxiv 2608.09907 v1 pith:HM4B64GM submitted 2026-08-10 cs.CV

Mainak Singha , Niccolò Biondi , Elisa Ricci , Subhankar Roy This is my paper · ORCID
classification cs.CV
keywords mixtureofexpertsdistributedvisualinstructiontuningmultimodallargelanguagemodelsrehearsal-freecompositionroutercalibrationmodelmergingquestionansweringprivacy-preservingadaptation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a practical constraint: adapting a multimodal language model to many specialized visual-language domains normally requires gathering all data in one place, which is impossible when each domain's data is private. DistMoE's proposal is that each data owner trains its own expert subnetwork locally, and the separately trained pieces are combined at inference time by a single router, with no one else's data ever rehearsed. The load-bearing trick is a two-step procedure: each client first trains its private expert against a shared public expert, then a cheap second stage recalibrates that client's router and a small linear adapter so every expert's contribution is expressed as a well-behaved deviation from the public expert. On image-question-answering benchmarks, the paper reports DistMoE generally gives the strongest results among distributed baselines that bypass joint training. If that holds, adapting a large multimodal model to new domains becomes a local per-client step followed by inference-time composition that never exposes private data.

What carries the argument

The load-bearing identity is the residual decomposition of the routed output, $o = h_{\text{pub}} + \sum_{m=1}^M \alpha_m \Delta_m$, which follows from the weight normalization $\alpha_{\text{pub}} + \sum_m \alpha_m = 1$. It shows that each private expert influences the mixture only through its calibrated residual $\Delta_m$, so influence is governed jointly by the router's weight $\alpha_m$ and the residual norm $\|\Delta_m\|_2$; Proposition 1 bounds the deviation from the public expert by $\sum_m \alpha_m \|\Delta_m\|_2$. Two training stages carry the argument: Stage-I per-client expert acquisition, and Stage-II public-anchored composition calibration, which freezes experts and updates only routers, biases, and linear adapters under a next-token-prediction loss plus an isotropic Gaussian loss on residuals, computed on the client's own data plus a small public subset. The isotropic loss is an Epps-Pulley discrepancy between one-dimensional projections of the residuals and a standard Gaussian, and its role is to make residual scales comparable across clients so that the merged router's softmax scores remain trustworthy.

What would settle it

Run the five calibrated clients on a shared holdout set and measure the per-client private-route logit distributions and residual norms; if the logit means or variances differ systematically across clients, the merged softmax is biased. Then train a joint router on a small rehearsal set drawn from all five clients and compare it with DistMoE on the same benchmarks: a clear gap in favor of the rehearsal-trained router would show the per-client calibration does not transfer to the merged M+1-expert router.

Watch

Extended reading notes

Core claim

The central claim is that independently trained experts can be composed without rehearsing other clients' data, provided the experts are represented as calibrated residuals around a shared public expert instead of as raw outputs. In Stage-I, each client builds a two-expert mixture: a frozen public feedforward network plus a private one initialized from it, trained on the client's own data so the local router learns when the private expert is useful. In Stage-II, all experts are frozen and only the two-way router, its bias, and a linear adapter on the private expert are updated on a mix of the client's data and a small public subset; an isotropic regularization loss pulls the per-token residuals $\Delta_m = T_m(E_m(x)) - E_{\text{pub}}(x)$ toward a common Gaussian-shaped distribution across clients. At inference the one public and $M$ private experts are gated by a top-$k$ router whose public-route weights are the mean of the per-client public weights, and the routed output decomposes as $o = h_{\text{pub}} + \sum_m \alpha_m \Delta_m$, so private experts act as shifts around the shared anchor. On image-QA benchmarks with Qwen-1.8B and Phi-2.7B backbones, the paper reports DistMoE generally gives the strongest results among the distributed baselines, beating rehearsal-free FlexOlmo and approaching a rehearsal-based upper bound.

Load-bearing premise

That the router scores and expert-output adjustments each client calibrates on its own data remain numerically comparable when all experts are combined at inference, so the merged router's choices stay trustworthy; Section 4.3 motivates this with a bound and routing plots, but does not prove the transfer.

Editorial extensions

If this is right

  • A multimodal model can be adapted per client on strictly local data and composed into a single model at inference, so domain adaptation no longer requires centralized or joint training.
  • Because routing is token-wise and needs no domain labels, individual client experts can be added, removed, or reused at inference; the removal study shows no single expert is load-bearing.
  • Only a small public calibration set (about 19k samples) is needed, since the public-data study shows gains saturate quickly.
  • A lightweight LoRA variant stays competitive, particularly on instruction-following benchmarks, so the calibration recipe transfers to parameter-efficient experts.
  • The results replicate across three language backbones, indicating the mechanism is not tied to one model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The residual view implies a general recipe the paper stops short of stating: what must be aligned across independently trained modules is not their outputs but their deviations from a shared anchor, and the calibration target could be any distributional statistic of those deviations, not only scale.
  • A directly testable extension would use residual-norm spread as a diagnostic: even when in-domain benchmarks look fine, merged-router quality should degrade as the cross-client spread of residual norms grows.
  • The privacy guarantee is a data-flow property, not an information-theoretic one, since router weights, biases, and adapters from every client are shared at inference; the guarantee holds only if those artifacts are treated as non-sensitive.
  • The rehearsal-free property is conditioned on a shared public checkpoint and a small public calibration set, so the method's reach is bounded by the availability of those ingredients - a point the paper itself raises for open-weight models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DistMoE, a mixture-of-experts method for distributed visual instruction tuning in which each client privately trains a domain-specific FFN expert initialized from a shared public expert, followed by a Stage-II calibration that updates only per-client routers, router biases, and lightweight linear adapters using a mix of the client's own data and a small public subset. An isotropic residual loss is introduced to align the distributions of residual shifts between private and public expert outputs. At inference, the public router weights are averaged and used together with the per-client private router weights to perform token-wise sparse routing over the public and all private experts, with no rehearsal of other clients' private data. Experiments on Qwen-1.8B, Phi-2.7B, and StableLM-1.6B across image QA and instruction-following benchmarks show that DistMoE generally outperforms model-merging and FlexOlmo-style baselines, while remaining below centralized references. The central claim is that public-anchored calibration makes both expert representations and routing scores comparable across clients, enabling rehearsal-free cross-client composition.

Significance. If the central claim holds, DistMoE addresses a practically important problem: adapting multimodal LLMs to multiple private, domain-specific clients and composing the resulting knowledge at inference without accessing any client's private data. The paper contributes a concrete training recipe, extensive experiments across three backbones, ablations of each Stage-II component, routing-behavior visualizations, and publicly available code, which I regard as strengths. However, the formal support for the key algorithmic claim is thin: Proposition 1 is an elementary triangle-inequality consequence of the algebraic identity in Eq. (9), and it does not establish comparability of the joint router's logits across clients. The empirical results are encouraging but the mechanism that is claimed to enable rehearsal-free composition is not directly validated, and there is an internal inconsistency in the main table that must be addressed. The significance is therefore conditional on the authors providing additional evidence that the joint inference router is actually well calibrated.

major comments (3)
  1. [Algorithm 1, lines 31-37; Sec. 4.2; Proposition 1] The load-bearing claim that Stage-II makes routing scores comparable across clients is not supported by the training objective. Each client trains a two-expert softmax router (public vs. its own private expert) on D_m ∪ D_pub (Algorithm 1, lines 22-28), producing private logits z_m = x^T r_m + b_m. At inference, the joint router is a single softmax over the averaged public logit x^T \bar r_pub and all private logits z_m (lines 31-37). The Stage-II objective in Eq. (7) — next-token prediction plus the isotropic loss of Eq. (6) — constrains only the expert-output residuals Δ_m = T_m(E_m(x)) − E_pub(x); it imposes no constraint on the relative scale or offset of the router logits across clients. The per-client bias b_m is trained independently and enters directly into the joint softmax, so a client with larger z_m − z_pub can dominate regardless of token content. Proposition 1 bounds the representation shift given routing weights α_m; it says nothing about whether those α_m are jointly calibrated. The routing-mass plots in Figs. 7-9 are aggregate and cannot detect token-level miscalibration. Please provide a direct test of the compositional assumption (e.g., token-level agreement between the joint router and independent client routers, or sensitivity to artificially inflating one client's router bias), or add an explicit cross-client calibration mechanism and analyze it.
  2. [Sec. 4.2; Algorithm 1, lines 32-37] The inference-time public logit uses the averaged router weights \bar r_pub, whereas during Stage-II each client's private logit was calibrated against that client's own public router weights r_pub,m. For client m, the relative logit difference changes from (x^T r_m + b_m) − x^T r_pub,m during training to (x^T r_m + b_m) − x^T \bar r_pub at inference. The paper does not analyze the magnitude or effect of this shift, which means even the per-client two-way decision boundary is altered in the merged router. This is a concrete, unquantified source of miscalibration that compounds the cross-client issue raised above. Please measure the discrepancy (e.g., compare routing decisions with per-client public weights vs. the averaged public weights) or justify why it is negligible.
  3. [Table 1; Sec. 5.1] Table 1 contains an internal inconsistency: the MoE-LLaVA row for Phi-2.7B is identical to the Qwen-1.8B row (72.81, 57.29, 63.22, 44.15, 89.32, 1230.16, 57.08, 62.16, 15.61), yet Table 8 reports different values for the Phi-2.7B MoE-LLaVA model (e.g., VQAv2 74.99, GQA 56.69, SQA IMG-Acc 68.37, VQAT 46.52). In addition, the Phi-2.7B DistMoE-LoRA row in Table 1 is nearly identical to the Public model row (VQAv2 42.89 vs 42.87, SQA 69.52 vs 69.26, POPE 80.84 vs 80.48), yet Sec. 5.1 describes DistMoE-LoRA as "highly competitive". Please correct Table 1 and clarify whether the LoRA variant actually adapts for Phi-2.7B or whether it degenerates to the public model in this configuration.
minor comments (6)
  1. [Sec. 4.1.2] The text contains a typo: "istropic Gaussian" should read "isotropic Gaussian".
  2. [Table 1] "SQA I" should be defined in the table caption; in the main text it is described as image-grounded accuracy, but the abbreviation is not expanded in the caption.
  3. [Sec. C.3] The sentence "We use top-k routing with m=4 for full experts and k=4R=64 for LoRA experts" contains an undefined variable m; it should be k=4 for full experts.
  4. [Abstract and Sec. 1] The term "rehearsal-free" is potentially misleading because Stage-II uses a public calibration set; the paper should explicitly define it as "cross-client rehearsal-free" (no access to other clients' private data) at first use.
  5. [Sec. 4.3 and Appendix B] Proposition 1 is an elementary consequence of the triangle inequality and the algebraic identity in Eq. (9); presenting it as a formal proposition overstates its support for the method. The discussion should clearly state that the bound is a necessary but not sufficient condition for cross-client calibration.
  6. [Figs. 7-9] The color legends for the five private experts are difficult to distinguish in the routing-mass plots; adding numerical annotations or a different palette would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: DistMoE's formal bound is a definitional identity, but the paper's empirical claims are evaluated on external benchmarks, and no load-bearing self-citation or fitted-input-as-prediction step exists.

full rationale

The paper's only formal result, Proposition 1, is an exact algebraic consequence of the residual definition and routing-weight normalization: Eq. 9 rewrites the routed output as the public expert plus a routing-weighted sum of residuals, and Eq. 10 follows from the triangle inequality. This is a mathematical identity, not an empirical prediction, and the paper presents it as motivation for the isotropic residual loss rather than as a benchmark-tested claim. Stage-II training uses a next-token loss and an isotropic regularization loss on each client's own data plus a public subset; its effectiveness is then tested on external QA and instruction-following benchmarks with ablations over adapters, router bias, and public-data size, so the central method is not fitted to the target result. The paper builds on FlexOlmo, MoE-LLaVA, and LeJEPA, but these are external prior works with no author overlap, and no load-bearing argument reduces to a self-citation or to an imported uniqueness theorem. The separate concern that per-client router biases are not explicitly cross-calibrated is a validity or robustness risk, not a circularity, because the paper does not derive the joint router's calibration from those biases by construction. Overall, the derivation chain is self-contained and the empirical evaluation is externally benchmarked, so no significant circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two ad hoc assumptions that are pragmatic rather than proven: isotropic residual calibration transfers across clients, and averaged router weights are valid at inference. The method also assumes the public anchor setup. These are reasonable engineering assumptions, but they carry the empirical weight of the paper.

free parameters (5)
  • lambda (isotropic loss weight) = 1e-4
    Weights L_iso in Eq. (7); set by hand, no sweep reported in the paper.
  • public calibration subset size |D_pub| = 19k
    Number of public samples per client in Stage-II; Fig. 4 shows saturation, but the exact size is a design choice.
  • top-k routing budget = k=4 (full), k=64 (LoRA)
    Selected from Fig. 5 as the best accuracy/efficiency trade-off; not derived.
  • LoRA rank and scaling = r=16, alpha=32, R=16
    LoRA expert configuration in DISTMOE-LoRA, chosen by the authors (Sec. 5, Appendix E).
  • number of sampled unit directions S in Eq. (6) = not reported
    Affects the Epps-Pulley estimate of residual isotropy; unspecified in the paper.
assumptions (5)
  • domain assumption A shared public expert and small public calibration set are available to all clients.
    The problem setup in Sec. 3.1 and Stage-II in Sec. 4.1.2 require D_pub at every client; the limitations section admits this may fail for some open-weight models.
  • ad hoc to paper Imposing isotropic Gaussian structure on residuals makes cross-client residual scales comparable.
    Introduced in Sec. 4.1.2 with a loss based on Epps-Pulley projections; no proof that this guarantees cross-client comparability, only empirical support in Tables 1-3 and Fig. 3.
  • ad hoc to paper Averaging per-client public-route weights yields a valid shared public route.
    Sec. 4.2 and Algorithm 1 lines 31-37 average r_pub,m without justification; the resulting router is used for all clients.
  • domain assumption Top-k softmax routing over post-hoc merged experts is a valid inference procedure.
    The paper assumes MoE routing remains well-behaved when experts are trained independently and composed without joint training, following FlexOlmo and BTX.
  • domain assumption The Epps-Pulley discrepancy is a suitable and differentiable surrogate for isotropy.
    Borrowed from LeJEPA [15]; the paper does not discuss the statistical or optimization properties in the MoE setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning." pith.science (2026). https://pith.science/paper/HM4B64GM

@misc{pith2026260809907,
  author       = {Pith},
  title        = {Pith review of: DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HM4B64GM}},
  note         = {Machine review of arXiv:2608.09907}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have shown strong multimodal instruction-following ability, but adapting them to diverse visual-language domains typically assumes centralized data access and costly joint training. This is restrictive when data is distributed across private, domain-specific, or permission-limited clients. To this end, we propose DistMoE, a mixture-of-experts (MoE) approach for distributed visual instruction tuning. In each layer of the language decoder it augments the public feedforward network (FFN) with a client-specific private FFN expert, with the goal to acquire domain-specific knowledge. However, independent expert training causes the private FFNs to learn representation of different scale and magnitudes, making merging the experts difficult. To reduce client-specific drift, we introduce a public-anchored expert composition stage that updates only routers and lightweight private projection adapters on a mix of local client data and public data, via an isotropic regularization loss, therefore making it cross-client rehearsal-free composition. During inference, DistMoE performs modular routing over public and private experts, enabling token-wise domain composition without explicit domain labels. Experiments across diverse visual-language benchmarks show that DistMoE enables flexible expert reuse, effective domain adaptation, and competitive performance while preserving modular control over client-specific knowledge. Codes are available at https://github.com/mainaksingha01/DistMoE.

Figures

Figures reproduced from arXiv: 2608.09907 by the authors.

Figure 1
Figure 1. Centralized MoE tuning vs. DISTMOE. Left: Centralized MoE methods [1] rely on joint access to domain-specific data. Right: DISTMOE independently trains domain-specific private experts with distributed data access and composes them via a distributed router. This enables experts to be flexibly added, removed, or reused at inference time depending on the application needs maintained [11]. Moreover, once such data is ut… view at source ↗
Figure 2
Figure 2. Overview of DISTMOE. Left. In Stage-I each client independently trains a private FFN by initializing from a shared public FFN. In Stage-II we freeze the experts, and recalibrate routing using isotropic residual regularization and next-token prediction loss utilizing a mix of public and private data. Right. At inference, DISTMOE composes the shared public expert and calibrated private experts through sparse routing w… view at source ↗
Figure 3
Figure 3. Top-4 routing mass on MME with Qwen-1.8B, computed as the aggregate normalized [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Public-data size in Stage-II. Effect of public-data anchoring [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Top-k routing budget. Top-k expert selection. In addition, we study the effect of the sparse routing budget by varying the number of selected experts in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Number of LoRA experts. We analyze the effect of using multiple LoRA experts per client in [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Top-4 routing mass of DISTMOE with Qwen-1.8B. 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 MoE layer idx 0% 25% 50% 75% 100% Percentage Public COCO GQA TextVQA OCRVQA VG (a) GQA 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 MoE layer idx 0% 25% 50% 75% 100% Percentage Publi…
Figure 8
Figure 8. Figure 8: Top-4 routing mass of DISTMOE with Phi-2.7B. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Top-4 routing mass of DISTMOE with StableLM-1.6B. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 16 canonical work pages

  1. [1]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. IEEE Transactions on Multimedia, 2026

    work page 2026

  2. [2]

    The revolution of multimodal large language models: A survey.Findings of the association for computational linguistics: ACL 2024, pages 13590–13618, 2024

    Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: A survey.Findings of the association for computational linguistics: ACL 2024, pages 13590–13618, 2024

    work page 2024

  3. [3]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    2024

  4. [4]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024

    arXiv 2024

  5. [5]

    Svit: Scaling up visual instruction tuning

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023

    arXiv 2023

  6. [6]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  7. [7]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017

    2017

  8. [8]

    Scaling vision-language models with sparse mixture of experts

    Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 11329–11344, 2023

    work page 2023

Show all 52 references
  1. [9]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  2. [10]

    Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.arXiv preprint arXiv:2403.07816, 2024

    Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.arXiv preprint arXiv:2403.07816, 2024

    2024 arXiv

  3. [11]

    Biomedical visual instruction tuning with clinician preference alignment.Advances in neural information processing systems, 37:96449–96467, 2024

    Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment.Advances in neural information processing systems, 37:96449–96467, 2024

    2024

  4. [12]

    Flexolmo: Open language models for flexible data use.arXiv preprint arXiv:2507.07024, 2025

    Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, et al. Flexolmo: Open language models for flexible data use.arXiv preprint arXiv:2507.07024, 2025

    2025 arXiv

  5. [13]

    Train separately, merge together: Modular post-training with mixture-of-experts

    Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia, Matei Zaharia, Noah A Smith, and Sewon Min. Train separately, merge together: Modular post-training with mixture-of-experts. arXiv preprint arXiv:2604.18473, 2026. 10

    2026 arXiv

  6. [14]

    Learning to instruct for visual instruction tuning.Advances in neural information processing systems, 2025

    Zhihan Zhou, Feng Hong, Jiaan Luo, Jiangchao Yao, Dongsheng Li, Bo Han, Ya Zhang, and Yanfeng Wang. Learning to instruct for visual instruction tuning.Advances in neural information processing systems, 2025

    2025

  7. [15]

    Lejepa: Provable and scalable self-supervised learning without the heuristics.arXiv preprint arXiv:2511.08544, 2025

    Randall Balestriero and Yann LeCun. Lejepa: Provable and scalable self-supervised learning without the heuristics.arXiv preprint arXiv:2511.08544, 2025

    2025 arXiv

  8. [16]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  9. [17]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  10. [18]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023)...

    2023

  11. [19]

    Coin: A benchmark of continual instruction tuning for multimodel large language models.Advances in neural information processing systems, 37:57817–57840, 2024

    Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao. Coin: A benchmark of continual instruction tuning for multimodel large language models.Advances in neural information processing systems, 37:57817–57840, 2024

    2024

  12. [20]

    Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning.arXiv preprint arXiv:2505.02486, 2025

    Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ho Shing Ip, and Sam Kwong. Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning.arXiv preprint arXiv:2505.02486, 2025

    2025 arXiv

  13. [21]

    On token’s dilemma: Dynamic moe with drift-aware token assignment for continual learning of large vision language models

    Chongyang Zhao, Mingsong Li, Haodong Lu, and Dong Gong. On token’s dilemma: Dynamic moe with drift-aware token assignment for continual learning of large vision language models. arXiv preprint arXiv:2603.27481, 2026

    2026

  14. [22]

    Kss-moe: Knowledge space synergy framework in mixture of experts for continual visual instruction tuning

    Lingyun Song, Ziyao Chen, Kang Pan, Xiaolin Han, Xinbiao Gan, Yudai Pan, Xiaofan Sun, Xiaoqi Wang, and Xuequn Shang. Kss-moe: Knowledge space synergy framework in mixture of experts for continual visual instruction tuning. InProceedings of the AAAI Conference on Artificial Int...

    2026

  15. [23]

    Continual instruction tuning for large multimodal models.IEEE Transactions on Image Processing, 2026

    Jinghan He, Haiyun Guo, Kuan Zhu, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large multimodal models.IEEE Transactions on Image Processing, 2026

    2026

  16. [24]

    Model tailor: Mitigating catastrophic forgetting in multi-modal large language models

    Didi Zhu, Zhongyisun Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. InInternational Conference on Machine Learning, pages 62581–62598. PMLR, 2024

    2024

  17. [25]

    Diloco: Distributed low- communication training of language models.arXiv preprint arXiv:2311.08105, 2023

    Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models.arXiv preprint arXiv:2311.08105, 2023

    2023 arXiv

  18. [26]

    Task formulation matters when learning continually: A case study in visual question answering

    Mavina Nikandrou, Lu Yu, Alessandro Suglia, Ioannis Konstas, and Verena Rieser. Task formulation matters when learning continually: A case study in visual question answering. arXiv preprint arXiv:2210.00044, 2022

    2022 arXiv

  19. [27]

    Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportu- nities.ACM Computing Surveys, 58(8):1–41, 2026

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportu- nities.ACM Computing Surveys, 58(8):1–41, 2026

    2026

  20. [28]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  21. [29]

    Branch-train-merge: Embarrassingly parallel training of expert language models.arXiv preprint arXiv:2208.03306, 2022

    Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models.arXiv preprint arXiv:2208.03306, 2022

    2022 arXiv

  22. [30]

    Ties-merging: Resolving interference when merging models.Advances in neural information processing systems, 36:7093–7115, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in neural information processing systems, 36:7093–7115, 2023

    2023

  23. [31]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025

    2025 arXiv

  24. [32]

    Scaling vision with sparse mixture of experts

    Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595, 2021

    2021

  25. [33]

    Multi- modal contrastive learning with limoe: the language-image mixture of experts.Advances in Neural Information Processing Systems, 35:9564–9576, 2022

    Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-image mixture of experts.Advances in Neural Information Processing Systems, 35:9564–9576, 2022

    2022

  26. [34]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014

    2014

  27. [35]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  28. [36]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 International Conference on Document Analysis and Recognition (ICDAR), pages 947–952, 2019

    2019

  29. [37]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  30. [38]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123(1):32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer v...

    2017

  31. [39]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  32. [40]

    Phi-2: The surprising power of small language models

    Mojan Javaheripi and Sébastien Bubeck. Phi-2: The surprising power of small language models. https://www.microsoft.com/en-us/research/blog/ phi-2-the-surprising-power-of-small-language-models/ , December 2023. Microsoft Research Blog

    2023

  33. [41]

    Stable lm 2 1.6 b technical report.arXiv preprint arXiv:2402.17834, 2024

    Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b technical report.arXiv preprint arXiv:2402.17834, 2024

    2024 arXiv

  34. [42]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  35. [43]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in neural information processing systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in neural information processing systems, 35:2507–252...

    2022

  36. [44]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 292–305, 2023

    2023

  37. [45]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  38. [46]

    Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  39. [47]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  40. [48]

    Open technical problems in open-weight ai model risk management.Transactions on Machine Learning Research, 2025

    Stephen Casper, Kyle O’Brien, Shayne Longpre, Elizabeth Seger, Kevin Klyman, Rishi Bom- masani, Aniruddha Nrusimha, Ilia Shumailov, Sören Mindermann, Steven Basart, et al. Open technical problems in open-weight ai model risk management.Transactions on Machine Learning Research, 2025

    2025

  41. [49]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023

    2023 arXiv

  42. [50]

    Aligning large multi-modal model with robust instruction tuning.arXiv preprint arXiv:2306.14565, 2(3):6, 2023

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning.arXiv preprint arXiv:2306.14565, 2(3):6, 2023

    2023 arXiv

  43. [51]

    Mimic-it: Multi-modal in-context instruction tuning.arXiv preprint arXiv:2306.05425, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning.arXiv preprint arXiv:2306.05425, 2023

    2023 arXiv

  44. [52]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 13 A Pseudocode Algorithm for DISTMOE Algorithm 1DISTMOE: Pri...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.