{"id":"e4938d19-f4b2-4acd-8e08-d22676bfa215","arxiv_id":"2608.00574","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"The standard mixed image-text load-balancing loss lets modality errors cancel at one token mix; ReBA balances each modality separately and per image, reducing load across resolutions and tiles with comparable accuracy.","lead":"Vision-language MoE models route image and text tokens to expert networks, but the standard load-balancing loss can hide large opposing errors that reappear when the token mix changes. This paper derives the exact load curve under token-mix shifts and introduces ReBA, a loss that balances image and text separately and per image, lowering expert load without hurting accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard–soft alignment is only approximate in Appendix B.2 (ReBA median cosine 0.770/0.831), so the formal conditional-CV bound does not strictly hold for trained ReBA; empirical true-top-k load results are unaffected, so verdict stays ACCEPT.","rationale":"The paper's central empirical claim—ReBA lowers mean layer CV on every benchmark/backbone and keeps accuracy comparable—is supported by broad evidence: all four backbones and seven benchmarks in Table 1, a full coefficient grid in Table S4, paired physical sweeps in Fig. 5 and Tables S8/S10, and the ablation in Table 3 isolating the two design choices. The composition-shift law (Eq. 3) is an exact quadratic identity for fixed profiles; its only assumption is fixed q_I and q_T, which the paper explicitly acknowledges physical preprocessing can violate, and the native diagnostics (Tables S14/S15) quantify that drift. I therefore do not see a refutation of the central claim. The only real soft spot is the hard–soft alignment assumption in Appendix B.2. The reader identified the same point, and Table S5 confirms it is only approximate for ReBA. But the empirical load numbers are obtained with true top-k dispatch counts, so this does not overturn those numbers. It does mean the theoretical guarantee 'one modality cannot cancel the other' is weaker than stated for trained ReBA and for transfer to other architectures. This is a limitation, not a fatal flaw; the verdict should remain ACCEPT, with the theoretical section interpreted as a heuristic formally valid only under alignment. If the proposed numerical check shows large deviations, I would move to CONDITIONAL and require the paper to soften B.2; absent that evidence, the current disclosure is sufficient.","tokens_in":27642,"tokens_out":6328,"duration_ms":74608,"concrete_test":"On the fixed 500-sample probe, for each of the 36 Split-Qwen3VL-4B ReBA layers and each modality, compute L_r = N F_r^T P_r and compare it with the true hard-load surrogate 1 + CV^2(F_r). Report the per-layer relative error |L_r - (1 + CV^2(F_r))|/(1 + CV^2(F_r)), and the correlation between (δF_r)^T δP_r and ||δF_r|| ||δP_r||. If all layers are within ~10%, the bound is numerically adequate; if several layers deviate by >50%, Appendix B.2's guarantee fails on the actual ReBA checkpoint and the explanation should be reframed as empirically motivated rather than proven. A second check: repeat the comparison on Split-InternVL3-8B ReBA; if the median cosine falls below ~0.5 there, the portability claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix B.2's central theoretical step is the identity N(F_r)^T P_r = 1 + N(δF_r)^T δP_r (Eq. S19), which makes L_ReBA a squared-CV surrogate only when hard top-k deviations δF_r and soft gate deviations δP_r are aligned. For trained ReBA, Table S5 reports median cosines of 0.770 (image) and 0.831 (text), with only 31/36 and 29/36 positive layers; several layers have cosines near 0.38. When δF and δP are partially or negatively aligned, the product N F_r^T P_r can be close to its minimum even though true hard-load CV is large, so the claim that 'a small ReBA loss therefore requires both modality loads to approach balance' is not guaranteed by the math for the actual trained routers. This is the load-bearing soft spot because it is the paper's formal explanation for why separate modality terms prevent cancellation. However, it is an acknowledged approximation (the paper itself calls Table S5 support for an approximation), and every empirical load table/curve is computed from true top-k counts, not from the soft surrogate. The fixed-profile law (Eq. 3) is unaffected because it is purely algebraic and is verified by physical runs. Thus the concern limits the theoretical guarantee and portability, not the main empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies expert load balance in split vision-language MoE models under changing image/text token mixes. It identifies a failure mode of the standard token-level Switch auxiliary loss: because only the mixed image+text load is penalized, large opposing image and text load errors can cancel at a particular mix. Section 3.4 derives an exact composition-shift law (Eq. 3): for fixed conditional profiles, the squared aggregate RMS-CV is a parabola in image-token fraction a, with curvature κ = (N/L) Σ_l ||q^I_l − q^T_l||²₂, so the image–text load gap controls sensitivity to token mix. The paper proposes ReBA, which computes separate image and text auxiliary losses and treats each image as one equal-weight routing instance, pooling text. Experiments on four split backbones (Split-Qwen3VL-4B, Split-Qwen2.5VL-3B, Split-Qwen2VL-7B, Split-InternVL3-8B) show ReBA lowers mean layer CV on all reported benchmarks and physical resolution/tiling shifts while keeping mean accuracy comparable to Std-Aux; native MoE diagnostics show similar routing geometry. The paper is careful to distinguish fixed-profile predictions from physical profile drift and to report true top-k counts.","tokens_in":27996,"tokens_out":7872,"duration_ms":102490,"significance":"The contribution is significant if the claims hold: the fixed-profile law is simple, exact, and clearly separated from profile drift; ReBA is parameter-free apart from the global auxiliary coefficient and adds no router or expert parameters; the empirical evaluation is broad across backbones, benchmarks, resolutions, tiling counts, and coefficient values; code is provided. The paper also includes useful negative controls, such as showing that every coefficient in the sweep improves over Std-Aux, and clearly labels the idealized speedup proxy as not a latency measurement. The main caveat, acknowledged in Appendix B.2/Table S5, is that the squared-CV interpretation of the ReBA loss relies on approximate hard–soft alignment; this limits the formal guarantee but does not undermine the true-top-k load reductions.","major_comments":[],"minor_comments":[{"comment":"The statement in Sec. 4.1 that 'a small ReBA loss therefore requires both modality loads to approach balance' is stronger than the algebra supports when hard–soft alignment is not exact. Table S5 reports median cosines of 0.770 (image) and 0.831 (text) and only 31/36 and 29/36 positive layers. I suggest either softening this claim to a heuristic motivation or adding an explicit bound relating L_ReBA to true hard-load CV via the measured alignment. The empirical load tables, which are computed from true top-k counts, are unaffected.","section":"Appendix B.2, Eq. (S19), Table S5"},{"comment":"The final coefficient λ_aux = 0.1 is selected on the same fixed 500-sample probe used for diagnostics, which is a selection-on-the-evaluation-set concern. Table S4 partly addresses this by showing ReBA improves over Std-Aux at every coefficient value, so the conclusion is not driven by this choice; please state this explicitly in the main text.","section":"Sec. 5.1 / Appendix C.1"},{"comment":"The phrase 'The text defines every variant' is cryptic. Clarify whether 'text' refers to the text column, the text routing profile, or the text modality, and define each variant in the caption or surrounding text.","section":"Table 3 caption"},{"comment":"'Physical runs preserve the method order' is not strictly true at the lowest resolution, where ReBA (0.0755) is slightly above Std-Aux (0.0690) in Table S8. Rephrase as 'preserve the broad curve structure' or 'at four of five settings.'","section":"Figure 1 / Sec. 5.4, Table S8"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, modest contribution. The exact composition-shift law — mixed profile is a convex combination, squared CV is a parabola with curvature proportional to the squared image-text profile gap — is simple algebra, but as a stated diagnosis of VL MoE load imbalance it is genuinely new and not in the cited literature. ReBA’s two design choices, separate modality terms and per-image equal-weight routing instances, follow from measured routing geometry, and the experiments back them up. Across four split backbones, ReBA lowers mean layer CV on every benchmark input while keeping mean task accuracy comparable to Std-Aux. The paper also earns credit for reporting a seven-point coefficient sweep, bootstrap intervals, explicit result provenance, and released code.\n\nThe soft spot is the formal guarantee in Appendix B.2. The claim that separate terms prevent one modality from canceling the other is exact only under hard-soft alignment. Table S5 shows trained ReBA routers have median cosines of 0.770 (image) and 0.831 (text), with several layers near 0.38 and a few negative. So the derived \"both modality loads must approach balance\" bound does not strictly hold for the actual trained routers. The authors acknowledge this in the text, and it does not damage the main empirical result, since every load table and curve is computed from true top-k counts. Still, the portability argument to other architectures is weaker than the title suggests. Two minor issues: the final auxiliary coefficient is selected on the same fixed probe used for diagnosis (disclosed), and there is no latency measurement, only an idealized expert-compute proxy that excludes communication, kernel effects, and non-MoE work.\n\nNet: the central claim — ReBA lowers load while keeping accuracy comparable — is well supported. Anyone working on VL MoE serving or training should read this. It is a useful training-side fix, not a new inference paradigm or a capability gain. It deserves peer review, with a request to verify code and to state plainly that the theoretical guarantee is approximate under hard-soft misalignment.","headline":"Solid, modest training-side fix for VL MoE load imbalance; the composition-shift law is a genuinely useful diagnostic and the empirical results are consistent, but the formal guarantee in Appendix B.2 is weaker than advertised because hard-soft alignment is only approximate.","tokens_in":28450,"tokens_out":1613,"would_cite":true,"duration_ms":20982,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard vision-language MoE load balancing can hide opposing image and text errors that emerge as token mix changes; the paper derives the exact curve and proposes a separate-modality, per-image balancing loss that flattens it.","keywords":["mixture-of-experts","load balancing","vision-language models","auxiliary loss","token mix","routing geometry","image-text load gap","MoE inference"],"falsifier":"Take a trained ReBA checkpoint and compute, per layer and modality, the cosine between the hard profile deviation F-u and the soft profile deviation P-u; if many layers show near-zero or negative cosines, the squared-CV surrogate argument for 'one modality cannot cancel the other' loses its force. Alternatively, run a physical resolution sweep on a native large MoE trained with ReBA and check whether the mixed RMS-CV stays flat; if it rises comparably to Std-Aux at high image fraction, the central claim fails.","tokens_in":27587,"feed_emoji":"⚖️","tokens_out":7149,"duration_ms":84102,"temperature":0.7,"pith_summary":"Vision-language MoE batches mix image and text tokens in ratios that change with resolution, tiling, image count, and prompt length. The paper shows that the standard token-level Switch auxiliary loss balances only the mixed load, so large opposing image and text load errors can cancel at one ratio and re-emerge sharply when the ratio changes. It derives an exact fixed-profile law: with image and text load profiles held fixed, squared aggregate load follows a parabola in the image-token fraction, with curvature set by the image-text load gap. The proposed remedy, ReBA, uses separate image and text balance terms and one equal-weight routing instance per image, and the paper reports that it lowers load on every benchmark and backbone while keeping mean accuracy comparable.","feed_headline":"Mix-balanced MoE routers hide a 5x load swing; ReBA flattens it","feed_subtitle":"Separate image and text load terms keep vision-language routers balanced as resolution and tile counts change.","key_machinery":"The composition-shift law is the load-bearing object: the parabola identity R(a) = R(a*) + kappa (a - a*)^2 separates the best achievable load from sensitivity to token mix and identifies the image-text gap kappa as the cause of sensitivity. ReBA's objective is the other central mechanism: L_ReBA = lambda_I L_inst(M_img) + lambda_T L_inst(M_txt), where each L_inst uses the Switch-style hard-soft product N sum_i bar-F_i bar-P_i averaged over routing instances; M_img holds one instance per image and M_txt is a pooled text instance. This converts the standard token-level mixed loss into modality-separated, image-level balance terms.","core_discovery":"The central claim is that mixed-load balancing in vision-language MoEs is structurally blind to modality-specific imbalance. For fixed conditional image and text profiles qI and qT, the mixed profile is q(a) = a qI + (1 - a) qT; if it is uniform at a reference ratio a0, then q(a) - u = (a - a0)(qI - qT). Squared aggregate RMS-CV therefore follows R(a) = R(a*) + kappa (a - a*)^2 with kappa = (N/L) sum_l ||qI_l - qT_l||^2_2, so the image-text load gap controls how quickly load imbalance appears as the token mix shifts. ReBA implements two geometry-guided choices: separate image and text auxiliary losses, so one modality cannot cancel the other's error, and one equal-weight routing instance per","pith_inferences":["Beyond the paper: a direct untested extension is to train ReBA on native sparse MoEs with 128-256 experts and top-8 routing, where the paper only runs routing diagnostics; the separate-modality terms would face a much larger expert space.","Beyond the paper: the fixed-profile law suggests a calibration procedure—record per-layer image and text load profiles on a probe set, then predict load for any deployment ratio without rerunning the full model—which could guide preprocessing defaults such as pixel budgets and tile counts.","Beyond the paper: because expected load grows with variance of the image-token fraction, load-aware request batching or scheduling by image content could complement training-side balancing.","Beyond the paper: the measured within-image flattening and between-image variation suggest instance-level balancing may extend to other token-grouping boundaries, such as video frames or document pages, wherever tokens cluster by source."],"forward_implications":["If the law is right, an MoE router trained to balance a single mixed load has a balance point, not a balance range; deployments with varying image resolutions should evaluate load across token-mix ranges rather than at one mix.","Because curvature kappa is set by the image-text conditional gap, any method that shrinks those conditional gaps flattens the load curve; separate modality terms are one such method.","Expected squared load over a deployment grows with ratio variance plus the squared mismatch between the deployment mean and the checkpoint's best ratio, so workload-mix variability is a first-order cost.","On expert-parallel systems where the busiest expert sets layer time, ReBA's lower mixed load corresponds to an idealized expert-compute speedup of 1.23-1.25x at medium and high visual loads.","The routing diagnosis appears in released native MoEs, where roughly half of experts show a strong modality preference, suggesting the same failure mode and remedy may transfer beyond split models."],"supporting_citations":[{"why":"Supplies the standard token-level Switch auxiliary loss L_aux that the paper diagnoses and modifies.","marker":"Fedus, Zoph, and Shazeer 2022"},{"why":"Defines the sparse MoE routing and expert-parallel setting the paper studies.","marker":"Shazeer et al. 2017"},{"why":"Provides the dynamic-resolution visual tokenization that changes image-text token mixes.","marker":"Wang et al. 2024b"},{"why":"Supplies the Cambrian training corpus used for one-epoch finetuning of the split backbones.","marker":"Tong et al. 2024"},{"why":"Provides the VLMEvalKit benchmark evaluation and scoring protocol.","marker":"Duan et al. 2024"},{"why":"Provides the InternVL backbone and tiling setting used in physical-shift experiments.","marker":"Chen et al. 2023"},{"why":"Reports modality-aware preferences under the standard auxiliary loss, which the paper turns into a hard-load-gap diagnosis.","marker":"Xia et al. 2025"},{"why":"Provides the dense-to-MoE splitting method used to construct the disjoint-expert split backbones.","marker":"Zhang et al. 2022"}],"fun_headline_variants":["MoE load balance is mix-blind; ReBA splits it into image and text","5x MoE load swing comes from image-text mix; ReBA decouples them","Image-text load gap drives MoE imbalance; ReBA balances each modality"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The no-cancellation guarantee requires hard dispatch profiles and soft gate profiles to point in the same direction, but trained ReBA models satisfy this only approximately (median layer cosine 0.77 for image, 0.83 for text), and the fixed-profile load law itself assumes conditional profiles stay constant when resolution or tiling changes.","fun_headline_variants_meta":{"raw":{"variants":["MoE load balance is mix-blind; ReBA splits it into image and text","5x MoE load swing comes from image-text mix; ReBA decouples them","Image-text load gap drives MoE imbalance; ReBA balances each modality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3483,"prompt_tokens":821,"completion_tokens":2662,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":2594}},"tokens_in":565,"tokens_out":2662,"duration_ms":22964,"temperature":1.0,"reasoning_tokens":2594,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:32:08.953481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained ReBA checkpoint and compute, per layer and modality, the cosine between the hard profile deviation F-u and the soft profile deviation P-u; if many layers show near-zero or negative cosines, the squared-CV surrogate argument for 'one modality cannot cancel the other' loses its force. Alternatively, run a physical resolution sweep on a native large MoE trained with ReBA and check whether the mixed RMS-CV stays flat; if it rises comparably to Std-Aux at high image fraction, the central claim fails.","supporting_citations":[],"review_version":1}