{"id":"acfaf29c-a2e9-4bb8-bb1a-189e91e7a2fb","arxiv_id":"2608.09907","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DistMoE composes independently trained per-client Mixture-of-Experts specialists for vision-language models using a shared public anchor and isotropic residual calibration, without rehearsal of other clients' private data.","lead":"DistMoE trains one private expert per client on local data, then calibrates routers and adapters against a shared public expert so the experts can be composed without any client sharing private data. It gives multimodal LLM teams a rehearsal-free route to merge domain-specific instruction-tuned models while keeping data local.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-client router biases and logit scales are never cross-calibrated; the joint inference softmax (Alg. 1, lines 31-37) can be dominated by one client's offset, so rehearsal-free composition rests on an unverified compositional assumption.","rationale":"The reader's weakest assumption—that per-client Stage-II calibration transfers to the joint router—is exactly the load-bearing concern. I agree with the conditionality. The paper's own discussion (Sec. 4.3) motivates Stage-II as making residuals comparable, but Eq. (9) and Proposition 1 only relate output deviation to residuals; they do not establish comparability of the router logits. The per-client bias b_m is a concrete, uncalibrated parameter that directly controls the joint softmax. This is not a disagreement with an external consensus; it is an internal gap between the method's stated goal and its training objective. The empirical evaluation does provide some support: DistMoE outperforms rehearsal-free FlexOlmo, the routing-mass figures show qualitatively sensible public/private usage, and Ablation Table 2 shows all Stage-II components contribute. Those facts suggest the approach may work in practice, but they do not rule out the possibility that its gains stem from a few dominant clients or from biases rather than token-level routing. The concern is addressable with a small sensitivity experiment, and the manuscript should also release code and seed variance per the reader's recommendation. Because the central claim is plausible but unproven on this specific point, CONDITIONAL remains the right verdict; no change from the reader's verdict.","tokens_in":26425,"tokens_out":7945,"duration_ms":75994,"concrete_test":"Run the DistMoE Qwen-1.8B evaluation on GQA, SQAI, VQAT, and POPE under an inference variant in which all private-router biases b_m (Algorithm 1, line 35) are set to zero instead of their trained values. If any benchmark shifts by more than one point, the joint router is sensitive to uncalibrated per-client logit offsets that Stage-II never aligned, and the rehearsal-free composition claim is not established. As a secondary probe, compare DistMoE's per-token top-4 expert selections on 1k public tokens with FlexOlmo-RT's selections; low agreement would confirm that independent two-way routers do not compose into a calibrated multiclass router.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the joint M+1-expert router at inference is well calibrated, but no mechanism in Stage-II enforces this. Each client m trains a two-expert softmax (public vs. its own private expert) on D_m ∪ D_pub (Algorithm 1, lines 22-28). At inference, the joint router is a single softmax over the averaged public logit z_pub = x^T r̄_pub and the private logits z_m = x^T r_m + b_m (lines 31-37). The Stage-II objective Eq. (7) and the isotropic residual loss Eq. (6) regularize the residuals Δ_m = T_m(E_m(x)) − E_pub(x); they impose no constraint on the relative offset or scale of the router logits z_m across clients. In particular, the per-client bias b_m is trained independently, is unbounded, and enters directly into the joint softmax (line 35). Even if each two-way router were individually well calibrated on its own mix, two-expert softmax scores are not compositionally consistent: a client with a larger z_m − z_pub will dominate the joint softmax regardless of token content. Proposition 1 bounds the effect of residuals given the routing weights α_m; it says nothing about whether those α_m are jointly calibrated. The routing-mass plots (Figs. 7-9) show aggregate usage only and cannot detect token-level miscalibration. The claimed cross-client rehearsal-free composition therefore rests on an unverified compositional assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DistMoE, a mixture-of-experts method for distributed visual instruction tuning in which each client privately trains a domain-specific FFN expert initialized from a shared public expert, followed by a Stage-II calibration that updates only per-client routers, router biases, and lightweight linear adapters using a mix of the client's own data and a small public subset. An isotropic residual loss is introduced to align the distributions of residual shifts between private and public expert outputs. At inference, the public router weights are averaged and used together with the per-client private router weights to perform token-wise sparse routing over the public and all private experts, with no rehearsal of other clients' private data. Experiments on Qwen-1.8B, Phi-2.7B, and StableLM-1.6B across image QA and instruction-following benchmarks show that DistMoE generally outperforms model-merging and FlexOlmo-style baselines, while remaining below centralized references. The central claim is that public-anchored calibration makes both expert representations and routing scores comparable across clients, enabling rehearsal-free cross-client composition.","tokens_in":26744,"tokens_out":9582,"duration_ms":75642,"significance":"If the central claim holds, DistMoE addresses a practically important problem: adapting multimodal LLMs to multiple private, domain-specific clients and composing the resulting knowledge at inference without accessing any client's private data. The paper contributes a concrete training recipe, extensive experiments across three backbones, ablations of each Stage-II component, routing-behavior visualizations, and publicly available code, which I regard as strengths. However, the formal support for the key algorithmic claim is thin: Proposition 1 is an elementary triangle-inequality consequence of the algebraic identity in Eq. (9), and it does not establish comparability of the joint router's logits across clients. The empirical results are encouraging but the mechanism that is claimed to enable rehearsal-free composition is not directly validated, and there is an internal inconsistency in the main table that must be addressed. The significance is therefore conditional on the authors providing additional evidence that the joint inference router is actually well calibrated.","major_comments":[{"comment":"The load-bearing claim that Stage-II makes routing scores comparable across clients is not supported by the training objective. Each client trains a two-expert softmax router (public vs. its own private expert) on D_m ∪ D_pub (Algorithm 1, lines 22-28), producing private logits z_m = x^T r_m + b_m. At inference, the joint router is a single softmax over the averaged public logit x^T \\bar r_pub and all private logits z_m (lines 31-37). The Stage-II objective in Eq. (7) — next-token prediction plus the isotropic loss of Eq. (6) — constrains only the expert-output residuals Δ_m = T_m(E_m(x)) − E_pub(x); it imposes no constraint on the relative scale or offset of the router logits across clients. The per-client bias b_m is trained independently and enters directly into the joint softmax, so a client with larger z_m − z_pub can dominate regardless of token content. Proposition 1 bounds the representation shift given routing weights α_m; it says nothing about whether those α_m are jointly calibrated. The routing-mass plots in Figs. 7-9 are aggregate and cannot detect token-level miscalibration. Please provide a direct test of the compositional assumption (e.g., token-level agreement between the joint router and independent client routers, or sensitivity to artificially inflating one client's router bias), or add an explicit cross-client calibration mechanism and analyze it.","section":"Algorithm 1, lines 31-37; Sec. 4.2; Proposition 1"},{"comment":"The inference-time public logit uses the averaged router weights \\bar r_pub, whereas during Stage-II each client's private logit was calibrated against that client's own public router weights r_pub,m. For client m, the relative logit difference changes from (x^T r_m + b_m) − x^T r_pub,m during training to (x^T r_m + b_m) − x^T \\bar r_pub at inference. The paper does not analyze the magnitude or effect of this shift, which means even the per-client two-way decision boundary is altered in the merged router. This is a concrete, unquantified source of miscalibration that compounds the cross-client issue raised above. Please measure the discrepancy (e.g., compare routing decisions with per-client public weights vs. the averaged public weights) or justify why it is negligible.","section":"Sec. 4.2; Algorithm 1, lines 32-37"},{"comment":"Table 1 contains an internal inconsistency: the MoE-LLaVA row for Phi-2.7B is identical to the Qwen-1.8B row (72.81, 57.29, 63.22, 44.15, 89.32, 1230.16, 57.08, 62.16, 15.61), yet Table 8 reports different values for the Phi-2.7B MoE-LLaVA model (e.g., VQAv2 74.99, GQA 56.69, SQA IMG-Acc 68.37, VQAT 46.52). In addition, the Phi-2.7B DistMoE-LoRA row in Table 1 is nearly identical to the Public model row (VQAv2 42.89 vs 42.87, SQA 69.52 vs 69.26, POPE 80.84 vs 80.48), yet Sec. 5.1 describes DistMoE-LoRA as \"highly competitive\". Please correct Table 1 and clarify whether the LoRA variant actually adapts for Phi-2.7B or whether it degenerates to the public model in this configuration.","section":"Table 1; Sec. 5.1"}],"minor_comments":[{"comment":"The text contains a typo: \"istropic Gaussian\" should read \"isotropic Gaussian\".","section":"Sec. 4.1.2"},{"comment":"\"SQA I\" should be defined in the table caption; in the main text it is described as image-grounded accuracy, but the abbreviation is not expanded in the caption.","section":"Table 1"},{"comment":"The sentence \"We use top-k routing with m=4 for full experts and k=4R=64 for LoRA experts\" contains an undefined variable m; it should be k=4 for full experts.","section":"Sec. C.3"},{"comment":"The term \"rehearsal-free\" is potentially misleading because Stage-II uses a public calibration set; the paper should explicitly define it as \"cross-client rehearsal-free\" (no access to other clients' private data) at first use.","section":"Abstract and Sec. 1"},{"comment":"Proposition 1 is an elementary consequence of the triangle inequality and the algebraic identity in Eq. (9); presenting it as a formal proposition overstates its support for the method. The discussion should clearly state that the bound is a necessary but not sufficient condition for cross-client calibration.","section":"Sec. 4.3 and Appendix B"},{"comment":"The color legends for the five private experts are difficult to distinguish in the routing-mass plots; adding numerical annotations or a different palette would improve readability.","section":"Figs. 7-9"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the unverified compositional assumption about the joint router, which is the core of the paper's claim. The authors should be asked to provide a direct empirical test or a concrete calibration step. The Table 1 inconsistency (Phi-2.7B MoE-LLaVA row identical to Qwen-1.8B) is the kind of error that should have been caught before submission; while fixable, it undermines confidence in the reported numbers. I recommend major revision rather than rejection because the proposed method is well-motivated and the empirical results are promising, but the central mechanism needs stronger support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DistMoE is a plausible, well-scoped extension of FlexOlmo-style distributed MoE to MLLMs. The genuinely new part is the Stage-II calibration: a linear adapter plus an isotropic Gaussian residual loss that anchors each private expert's output to the public expert, and averaging the public-route weights at inference. That recipe is not in the BTX/FlexOlmo line, and the paper shows it helps on several benchmarks, sometimes a lot (MME on Phi-2.7B). The authors run a sensible ablation and report results across three backbones. The writing is clear and the limitation section is honest.\n\nThe main soft spot is exactly what the stress-test says: the paper never establishes that the per-client router logits are comparable across clients. Each client trains a two-expert softmax (public vs its own private expert) on D_m ∪ D_pub. At inference, the joint router is a softmax over the averaged public logit and the M private logits, using per-client biases b_m directly. Nothing in Stage-II aligns the scale or offset of those logits; the isotropic loss constrains residuals Δ_m, not the router logits. Proposition 1 bounds the output deviation given routing weights, but says nothing about whether those weights are jointly calibrated. The routing-mass plots show aggregate usage, which cannot detect token-level miscalibration. So the central claim—cross-client rehearsal-free composition—rests on an unverified compositional assumption. This does not make the empirical results wrong, but it means the mechanism is not fully supported.\n\nTwo data issues: Table 1 lists identical MoE-LLaVA numbers for Qwen-1.8B and Phi-2.7B, contradicting the detailed Phi-2.7B results in Table 8 (a copy-paste error). Also, the LoRA variants on Phi-2.7B are essentially indistinguishable from the public model, so those numbers add little. No error bars anywhere.\n\nWho is this for? People working on distributed or privacy-preserving adaptation of MLLMs. The paper deserves peer review; the authors should fix the table, report variance across seeds, specify the Epps-Pulley implementation, and ideally add a diagnostic showing that per-client router scores are comparable on a shared public set, or at least discuss why the two-expert calibration transfers. With those fixes, this could be a solid contribution.","headline":"A plausible rehearsal-free MoE composition method for MLLMs; the Stage-II isotropic residual calibration is genuinely new, but the paper never proves that per-client router logits are cross-comparable, and the main Table has a copy-paste error.","tokens_in":27333,"tokens_out":5661,"would_cite":false,"duration_ms":47738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DistMoE composes independently trained private experts at the token level, with no rehearsal of any client's private data.","keywords":["mixture of experts","distributed visual instruction tuning","multimodal large language models","rehearsal-free composition","router calibration","model merging","visual question answering","privacy-preserving adaptation"],"falsifier":"Run the five calibrated clients on a shared holdout set and measure the per-client private-route logit distributions and residual norms; if the logit means or variances differ systematically across clients, the merged softmax is biased. Then train a joint router on a small rehearsal set drawn from all five clients and compare it with DistMoE on the same benchmarks: a clear gap in favor of the rehearsal-trained router would show the per-client calibration does not transfer to the merged M+1-expert router.","tokens_in":26154,"feed_emoji":"🧩","tokens_out":12025,"duration_ms":94432,"temperature":0.7,"pith_summary":"The paper addresses a practical constraint: adapting a multimodal language model to many specialized visual-language domains normally requires gathering all data in one place, which is impossible when each domain's data is private. DistMoE's proposal is that each data owner trains its own expert subnetwork locally, and the separately trained pieces are combined at inference time by a single router, with no one else's data ever rehearsed. The load-bearing trick is a two-step procedure: each client first trains its private expert against a shared public expert, then a cheap second stage recalibrates that client's router and a small linear adapter so every expert's contribution is expressed as a well-behaved deviation from the public expert. On image-question-answering benchmarks, the paper reports DistMoE generally gives the strongest results among distributed baselines that bypass joint training. If that holds, adapting a large multimodal model to new domains becomes a local per-client step followed by inference-time composition that never exposes private data.","feed_headline":"Merges private AI experts with no rehearsal data","feed_subtitle":"Each client calibrates its expert against a shared anchor, so one router composes them at inference.","key_machinery":"The load-bearing identity is the residual decomposition of the routed output, $o = h_{\\text{pub}} + \\sum_{m=1}^M \\alpha_m \\Delta_m$, which follows from the weight normalization $\\alpha_{\\text{pub}} + \\sum_m \\alpha_m = 1$. It shows that each private expert influences the mixture only through its calibrated residual $\\Delta_m$, so influence is governed jointly by the router's weight $\\alpha_m$ and the residual norm $\\|\\Delta_m\\|_2$; Proposition 1 bounds the deviation from the public expert by $\\sum_m \\alpha_m \\|\\Delta_m\\|_2$. Two training stages carry the argument: Stage-I per-client expert acquisition, and Stage-II public-anchored composition calibration, which freezes experts and updates only routers, biases, and linear adapters under a next-token-prediction loss plus an isotropic Gaussian loss on residuals, computed on the client's own data plus a small public subset. The isotropic loss is an Epps-Pulley discrepancy between one-dimensional projections of the residuals and a standard Gaussian, and its role is to make residual scales comparable across clients so that the merged router's softmax scores remain trustworthy.","core_discovery":"The central claim is that independently trained experts can be composed without rehearsing other clients' data, provided the experts are represented as calibrated residuals around a shared public expert instead of as raw outputs. In Stage-I, each client builds a two-expert mixture: a frozen public feedforward network plus a private one initialized from it, trained on the client's own data so the local router learns when the private expert is useful. In Stage-II, all experts are frozen and only the two-way router, its bias, and a linear adapter on the private expert are updated on a mix of the client's data and a small public subset; an isotropic regularization loss pulls the per-token residuals $\\Delta_m = T_m(E_m(x)) - E_{\\text{pub}}(x)$ toward a common Gaussian-shaped distribution across clients. At inference the one public and $M$ private experts are gated by a top-$k$ router whose public-route weights are the mean of the per-client public weights, and the routed output decomposes as $o = h_{\\text{pub}} + \\sum_m \\alpha_m \\Delta_m$, so private experts act as shifts around the shared anchor. On image-QA benchmarks with Qwen-1.8B and Phi-2.7B backbones, the paper reports DistMoE generally gives the strongest results among the distributed baselines, beating rehearsal-free FlexOlmo and approaching a rehearsal-based upper bound.","pith_inferences":["The residual view implies a general recipe the paper stops short of stating: what must be aligned across independently trained modules is not their outputs but their deviations from a shared anchor, and the calibration target could be any distributional statistic of those deviations, not only scale.","A directly testable extension would use residual-norm spread as a diagnostic: even when in-domain benchmarks look fine, merged-router quality should degrade as the cross-client spread of residual norms grows.","The privacy guarantee is a data-flow property, not an information-theoretic one, since router weights, biases, and adapters from every client are shared at inference; the guarantee holds only if those artifacts are treated as non-sensitive.","The rehearsal-free property is conditioned on a shared public checkpoint and a small public calibration set, so the method's reach is bounded by the availability of those ingredients - a point the paper itself raises for open-weight models."],"forward_implications":["A multimodal model can be adapted per client on strictly local data and composed into a single model at inference, so domain adaptation no longer requires centralized or joint training.","Because routing is token-wise and needs no domain labels, individual client experts can be added, removed, or reused at inference; the removal study shows no single expert is load-bearing.","Only a small public calibration set (about 19k samples) is needed, since the public-data study shows gains saturate quickly.","A lightweight LoRA variant stays competitive, particularly on instruction-following benchmarks, so the calibration recipe transfers to parameter-efficient experts.","The results replicate across three language backbones, indicating the mechanism is not tied to one model family."],"supporting_citations":[{"why":"Supplies the shared public MoE-LLaVA stage-2 checkpoint used as the anchor, the centralized MoE reference model, and the joint-training setting DistMoE removes.","marker":"[1]"},{"why":"Defines the sparsely-gated mixture-of-experts layer and top-k routing that DistMoE adopts for inference-time composition.","marker":"[7]"},{"why":"The rehearsal-based Branch-Train-Mix baseline that DistMoE matches while eliminating the rehearsal step.","marker":"[10]"},{"why":"Provides the public-anchor expert-initialization idea used in Stage-I and the FlexOlmo/FlexOlmo-RT baselines that rehearse private data to train the router.","marker":"[12]"},{"why":"Source of the isotropic Gaussian structure idea that Stage-II applies to client residuals to make them comparable across clients.","marker":"[15]"}],"fun_headline_variants":["Calibrate private experts to a shared anchor for easy merging","Private experts tuned together without sharing data","Mix-of-experts training that skips data sharing","Distributed fine-tuning with zero rehearsal data","No rehearsal: just anchor private experts to public base"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the router scores and expert-output adjustments each client calibrates on its own data remain numerically comparable when all experts are combined at inference, so the merged router's choices stay trustworthy; Section 4.3 motivates this with a bound and routing plots, but does not prove the transfer.","fun_headline_variants_meta":{"raw":{"variants":["Calibrate private experts to a shared anchor for easy merging","Private experts tuned together without sharing data","Mix-of-experts training that skips data sharing","Distributed fine-tuning with zero rehearsal data","No rehearsal: just anchor private experts to public base"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1478,"prompt_tokens":1081,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":324}},"tokens_in":697,"tokens_out":397,"duration_ms":3954,"temperature":1.0,"reasoning_tokens":324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:22:33.835503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the five calibrated clients on a shared holdout set and measure the per-client private-route logit distributions and residual norms; if the logit means or variances differ systematically across clients, the merged softmax is biased. Then train a joint router on a small rehearsal set drawn from all five clients and compare it with DistMoE on the same benchmarks: a clear gap in favor of the rehearsal-trained router would show the per-client calibration does not transfer to the merged M+1-expert router.","supporting_citations":[{"cited_title":"Moe-llava: Mixture of experts for large vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the shared public MoE-LLaVA stage-2 checkpoint used as the anchor, the centralized MoE reference model, and the joint-training setting DistMoE removes."}],"review_version":2}