{"id":"56427008-05ac-4680-9729-62a69f076490","arxiv_id":"2505.23830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"EvoMoE creates MoE experts as decaying averages of a single trained FFN and routes tokens with hypernetwork-generated weights, yielding small benchmark gains over MoE-LLaVA.","lead":"This paper introduces EvoMoE, a way to turn a dense multimodal language model into a mixture-of-experts model by creating extra experts as smoothed copies of one trained expert and routing tokens through a hypernetwork-generated router. The authors report small but consistent gains over MoE-LLaVA across five model sizes on standard multimodal benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (1) is ambiguous to the point of non-reproducibility, and the paper's activated-parameter counts are internally inconsistent; the central expert-evolution mechanism cannot be evaluated as written.","rationale":"The reader's weakest_assumption identifies Equation (1) as the linchpin of the expert-evolution contribution. All reported accuracy gains are attributed to the combination of expert evolution and DTR; if the evolution rule cannot be interpreted unambiguously, the method cannot be reimplemented and the central claim is unverifiable. The paper provides no pseudocode, no code, and no explicit statement of whether the formula is applied per training step or once. Under the most natural reading (raw gradient), the equation is dimensionally inconsistent; under the alternative (parameter update), the experts would be near-identical to θ1, which the paper's own Table 3 does not support, as it shows evolved experts outperforming Expert 1 on several benchmarks. Thus the concern is not merely cosmetic: it affects the core mechanism. I agree with the reader's conditional verdict. I also note a secondary data-integrity issue: Table 1 reports 3.0B activated parameters for EvoMoE with Phi-2.7B, whereas supplementary Table 8 reports 4.5B. Together with the 0.5B case, where EvoMoE activates 0.7B versus MoE-LLaVA's 0.6B, the paper's claim of 'fewer activated parameters' is unreliable. However, the accuracy gains are consistently positive across all model sizes, so a conditional verdict requiring code and clarification remains appropriate.","tokens_in":17568,"tokens_out":12524,"duration_ms":127566,"concrete_test":"Request the exact implementation of Eq. (1) (or the released code), and verify whether ∇θ1 denotes the raw gradient, the parameter update, or the updated parameter vector. Then, for the Qwen-1.8B checkpoint, compute the pairwise cosine similarity of the four expert FFN weight matrices after Stage II. If the clarified rule produces experts with cosine similarity above 0.99, or if the authors cannot supply a precise per-step rule, the expert-evolution mechanism is not well-defined and the diversity-based explanation of the gains is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core novelty of EvoMoE is expert evolution, defined entirely by Eq. (1): θn ← β·θ1 + (1−β)·∇θ1, where ∇θ1 is never precisely defined. If it is the raw gradient, the RHS mixes parameters and gradients with incompatible scales, so θn is not a valid FFN weight vector. If it is a gradient-based update of θ1 (e.g., θ1 - lr·g), then each expert is at most a single parameter step from θ1, yielding near-copies rather than the 'diverse' experts claimed. The paper also does not state whether Eq. (1) is applied once at the end of Stage II or at each training step; if per-step, the experts would track the current gradient and no accumulation mechanism is specified. Because this formula is the sole mechanism for generating experts 2–4 and is not accompanied by code, the reported improvements cannot be independently reproduced. A secondary data-integrity concern: Table 1 reports 3.0B activated parameters for the Phi-2.7B EvoMoE model, while supplementary Table 8 reports 4.5B for the same configuration, directly contradicting the paper's claim that EvoMoE activates fewer parameters.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EvoMoE, a mixture-of-experts tuning framework for multimodal large language models, with two main components: an expert evolution strategy that produces multiple FFN experts from a single trainable expert via Eq. (1), and a Dynamic Token-aware Router (DTR) that uses hypernetworks to generate token-specific routing weights. The method is evaluated on five LLM backbones (0.5B to 7B) across seven multimodal benchmarks, reporting consistent average gains over the MoE-LLaVA baseline, alongside ablations for each component, expert diversity strategies, top-k choices, and expert counts.","tokens_in":17805,"tokens_out":5518,"duration_ms":56217,"significance":"If the empirical results hold, the paper offers a simple and apparently effective recipe for improving sparse MLLMs: the two proposed mechanisms each contribute measurable gains, and the improvements are consistent across five model sizes. The paper also includes useful ablations (noise, dropout, contrastive loss, local loss, router variants) and supplementary training details. However, the central expert-evolution equation is not precisely defined, the activated-parameter claims are internally contradictory, and some ablation numbers differ between the main text and the supplement. These issues prevent the reader from verifying the method's core mechanism and its headline efficiency claim, so the current version is not ready for publication.","major_comments":[{"comment":"The symbol ∇θ1 is never defined precisely. If it denotes the raw gradient vector, then θn is a sum of parameters and gradients of incompatible scales and is not a valid FFN weight vector; if it denotes a gradient-based update of θ1 (e.g., θ1 − η∇θ), then each θn is at most a single step from θ1, and the claimed functional diversity of experts is unverified. The paper also does not state whether Eq. (1) is applied once at the end of Stage II or iteratively at each training step; the text in §3.2 and §4.1 says β is randomly assigned 'at each training step,' which, combined with Eq. (1), would produce experts that change every step with no accumulation rule. Because this formula is the sole mechanism for generating experts 2–N, the method is not reproducible as written.","section":"Section 3.2, Eq. (1)"},{"comment":"The paper repeatedly claims that EvoMoE achieves better performance 'with fewer activated parameters' (Section 4.2 and contribution list). Table 1 contradicts this for the 0.5B model: EvoMoE activates 0.7B parameters versus MoE-LLaVA's 0.6B. For Phi-2.7B, Table 1 reports 3.0B activated parameters for EvoMoE, while supplementary Table 8 reports 4.5B for the same configuration, which would be more than the MoE-LLaVA baseline's 3.6B. These discrepancies directly undermine the paper's central efficiency claim and must be reconciled.","section":"Section 4.2 and Tables 1, 8"},{"comment":"The ablation of the evolution value β reports different numbers for the same conditions in the main paper and the supplement. For example, 'Expert 2' with β=0.9 has VQAv2 76.4 and MME 1305.7 in Table 3, but Table 15 reports VQAv2 76.8 and MME 1290.7 for β=0.9; similarly, 'Expert 3'/β=0.8 has MME 1297.3 in Table 3 but 1277.3 in Table 15. If these are the same experiment, the discrepancy is a data-integrity issue; if they are different settings, the paper should say so explicitly.","section":"Section 4.3, Tables 3 and 15"},{"comment":"The DTR module cannot be reconstructed from the text because the hypernetwork architecture is unspecified: the layer widths, output dimensions of Θ_up and Θ_down, and the exact parameter count of the router are not reported (Table 8 gives only a single aggregate 'Router' value). Please provide these details or release code, since DTR is one of the two core contributions.","section":"Section 3.3, Eqs. (5)–(8)"}],"minor_comments":[{"comment":"The first sentence contains a typo: 'peform' should be 'perform'.","section":"Section 4.3"},{"comment":"The abbreviation 'M-T[25]' in the table header is not defined in the caption; please spell out 'MoE-tuning' and clarify the reference.","section":"Table 2"},{"comment":"In the Qwen-1.8B row, the MME value for the MoE baseline is printed as '12881', which appears to be missing a decimal point and should likely be '1288.1'.","section":"Supplementary Table 9"},{"comment":"The main text states a batch size of 4 with gradient accumulation 2, while Table 7 lists global batch sizes of 256, 64, and 64 for the three stages; please clarify the relationship between local batch size, gradient accumulation, and the reported global batch size.","section":"Section 4.1 and Supplementary Table 7"},{"comment":"The caption notes that MME values are divided by 20 for the AVG computation, but the table displays raw MME values; please state this explicitly in the main text near the table to avoid confusion.","section":"Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The activated-parameter discrepancy is the most serious issue: if Table 8 is correct, the paper's headline claim of efficiency is false for Phi-2.7B, and the 0.5B row already shows the opposite of 'fewer activated parameters.' The ambiguity in Eq. (1) is also fundamental, as it is the entire expert-generation mechanism. Both issues are fixable in principle with clarifications and corrected tables, but the authors should be asked to provide precise equations, pseudocode or code, and a reconciliation of all parameter counts. The paper would also benefit from a diversity metric for experts to support the claim that evolution yields functionally diverse experts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read EvoMoE. Bottom line: it's a reasonable incremental contribution to sparse MLLM tuning, with consistent small gains across five model sizes, but the paper's core mechanism is not described precisely enough to reproduce, and the parameter-efficiency claims don't hold in every row of its own tables.\n\nWhat's actually new: the combination of an evolution-style expert initialization (EMA-like updates from a single trainable FFN) and a hypernetwork-generated token-aware router. The ablations are genuinely informative: Table 2 shows each component contributes, Table 3 shows evolved experts beat the original across many benchmarks, and Table 5 shows that generic diversity strategies (noise, dropout, contrastive) don't work as well. That's real experimental work, and the improvements, though around one point on average, are consistent across Qwen2-0.5B through OpenChat-7B.\n\nThe soft spots are serious. Equation (1), the entire expert-evolution mechanism, defines θn ← β·θ1 + (1−β)·∇θ1, but ∇θ1 is never defined. If it's the raw gradient, the update mixes parameters and gradients with incompatible scales. If it's a post-optimizer parameter vector, then the 'evolved' experts are just EMA copies of one expert's trajectory—which might work, but the paper provides no diversity metric or analysis to show the experts are actually diverse. The paper also doesn't say whether Eq. (1) is applied once at the end of Stage II or at every training step. Without that, the method is not reproducible.\n\nSecond, the parameter story is inconsistent. Table 1 reports 0.6B activated for MoE-LLaVA-0.5B and 0.7B for EvoMoE-0.5B, yet Section 4.2 claims 'all with fewer activated parameters.' Supplementary Table 8 lists 4.5B activated for Phi-2.7B EvoMoE, while Table 1 says 3.0B for the same configuration. One of those is wrong, and either way the paper overstates the parameter advantage.\n\nThird, there are no error bars or multiple seeds, and the β ranges are tuned on the same benchmarks reported. That's a limitation, not a fatal flaw.\n\nThe core idea is plausible and the experiments are decent. A serious referee should see it, but the paper needs a major revision to clarify Eq. (1), reconcile the parameter counts, and release code. I'd send it to review, not desk-reject it, but I'd want to see the ambiguity resolved before trusting the numbers.","headline":"EvoMoE is a plausible incremental MoE-tuning recipe with consistent but small gains; the central evolution formula is too ambiguous to reproduce as written, and the parameter-efficiency claims are overstated in places.","tokens_in":18381,"tokens_out":2784,"would_cite":false,"duration_ms":27628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EvoMoE evolves four experts from a single trainable FFN and routes each token by modality, outperforming MoE-LLaVA with fewer activated parameters.","keywords":["Mixture of Experts","Multimodal Large Language Models","Expert Diversity","Token-aware Routing","Hypernetwork","MoE-tuning","Sparse Activation","Vision-Language Instruction Tuning"],"falsifier":"Train the same MoE-LLaVA baseline but replace expert evolution with a plain exponential moving average of the single expert's weights (or EMA plus dropout), keeping the DTR unchanged; if benchmark scores match EvoMoE's within noise, the evolutionary-diversity story is not what drives the gains. Alternatively, measure a functional diversity metric, such as pairwise token-routing overlap or output cosine distance between the four experts after training; if evolved experts are as similar to expert 1 as replicated experts are, the uniformity problem is not actually solved.","tokens_in":17351,"feed_emoji":"🧠","tokens_out":5343,"duration_ms":48969,"temperature":0.7,"pith_summary":"The paper claims that two persistent failure modes of mixture-of-experts tuning for multimodal LLMs—experts that end up interchangeable, and a router that ignores whether tokens come from images or text—can be fixed by two changes. Instead of cloning one feed-forward network into several experts, the authors evolve extra experts from a single trainable expert by mixing its current weights with its gradient update at a random rate $\\beta$. They also replace the shared linear router with a Dynamic Token-aware Router whose weights are generated per token by two modality-specific hypernetworks. On top of the MoE-LLaVA baseline, EvoMoE reports average gains of roughly 0.9 to 1.4 percentage points across seven benchmarks for five backbone LLMs from 0.5B to 7B while activating only a top-1 expert rather than top-2. If the recipe holds, sparse multimodal models can be tuned to be more accurate and cheaper to run than the replication-based MoE baseline.","feed_headline":"Evolving experts beats replicated MoE tuning for MLLMs","feed_subtitle":"One trainable FFN grows into four diverse experts; a hypernetwork router cuts activated params.","key_machinery":"The central mechanism is the expert-evolution update $\\theta_n \\leftarrow \\beta \\theta_1 + (1-\\beta) \\nabla \\theta_1$, where $\\theta_1$ is the single trainable FFN from Stage I, $\\nabla \\theta_1$ is described as the gradient update for that expert, and $\\beta$ is an evolution rate randomly sampled each training step from three ranges (0.9–0.99, 0.8–0.89, 0.7–0.79) to produce experts 2, 3, and 4. This operation is what is claimed to turn one expert's trajectory into functionally diverse experts. The second load-bearing component is the Dynamic Token-aware Router (DTR): two hypernetworks $H_V$ and $H_T$, each a two-layer MLP, generate the weights of down- and up-projection layers separately for visual and text tokens, and a final MLP $\\phi$ converts the projected token into expert probabilities. DTR is what makes routing depend on modality and per-token content rather than a fixed linear map.","core_discovery":"EvoMoE's central claim is that expert uniformity and router rigidity, not the MoE structure itself, are what hold MoE-tuned multimodal LLMs back. As evidence, the authors show that shuffling the router weights of MoE-LLaVA leaves average benchmark performance essentially unchanged, meaning replicated-initialized experts converge to near-identical functions. Their remedy is a two-part construction: expert evolution, which derives $N$ experts by taking a single trained feed-forward network $\\theta_1$ and applying $\\theta_n \\leftarrow \\beta \\theta_1 + (1-\\beta) \\nabla \\theta_1$ with $\\beta$ drawn from different ranges for different experts; and the Dynamic Token-aware Router (DTR), which uses two small hypernetworks, one for visual tokens and one for text tokens, to generate down- and up-projection weights per token before a final MLP picks the top-1 expert. Across the pipeline, only expert 1 and the DTR are ever trained, while the evolved experts stay frozen, so the method adds sparse capacity without training extra experts. On Qwen2-0.5B, StableLM-1.6B, Qwen-1.8B, Phi-2.7B, and OpenChat-7B, EvoMoE outperforms MoE-LLaVA on average while activating fewer parameters.","pith_inferences":["If $\\nabla \\theta_1$ is the raw gradient, equation (1) cannot directly produce valid network weights because the gradient has the wrong shape, so the actual implementation likely uses the post-optimizer parameter vector; under that reading the method is a form of weight averaging, and the claimed 'evolution' may be better described as randomized exponential moving average. This is our inference fr","A testable extension of the paper's logic: the same evolution-plus-modality-router recipe should transfer to dense-to-sparse conversion of purely text LLMs, where the modality distinction in DTR would collapse to token-position or token-type differences; the paper does not test this.","The authors report that randomly sampling $\\beta$ across ranges works better than any single fixed $\\beta$; our inference is that what matters is creating experts whose routing distributions differ, so an even simpler diversity-promoting prior, such as different learning rates per expert or different dropout masks, might match EvoMoE without the evolution formula."],"forward_implications":["EvoMoE's reported results imply that sparse multimodal LLMs can beat both their dense counterparts and the MoE-LLaVA baseline while activating only one expert per token, lowering inference cost for the same or better accuracy.","The DTR ablation, which replaces the linear router with modality-specific hypernetwork routing, gives a concrete recipe: separating visual and text token routing helps more than any shared-router variant, so future MoE MLLMs should route by modality first.","The shuffled-router experiment implies that replicated initialization in MoE-tuning produces redundant experts; any method that provably increases expert diversity, whether by evolution or otherwise, should recover similar gains, making diversity a testable design target.","The authors' negative results—unfreezing all experts in Stage III, adding shared experts, and removing the first MoE layer all hurt—suggest that the freeze-evolved-experts plus top-1 protocol is load-bearing rather than incidental."],"supporting_citations":[{"why":"MoE-LLaVA: the MoE-tuning baseline and pipeline that EvoMoE builds on and compares against across all experiments.","marker":"[25]"},{"why":"HyperLLaVA: prior hypernetwork-based expert design in MLLMs that motivates the Dynamic Token-aware Router.","marker":"[49]"},{"why":"DeepSeek-V3: source of MoE insights (shared experts, first-layer removal) tested in the MoE Exploration ablations.","marker":"[26]"},{"why":"LLaVA-mix-665k: the instruction dataset used in both Stage II expert evolution and Stage III DTR training.","marker":"[19]"},{"why":"SimCLR contrastive loss: one of the expert-diversity baselines in the increasing-expert-diversity ablation.","marker":"[8]"}],"fun_headline_variants":["Evolving experts from a single FFN beats replicated MoE init","One expert evolves into diverse specialists for multimodal LLMs","EvoMoE: experts evolve, routers adapt per token","Fixing expert uniformity and router rigidity in MoE MLLMs","Dynamic routing and expert evolution boost sparse MLLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire expert-evolution recipe rests on equation (1) being a well-defined way to convert one expert's training signal into new valid expert weights, but the paper never specifies what $\\nabla \\theta_1$ means: if it is the raw gradient it has the wrong shape to be a weight vector, and if it is the post-update parameter vector the method reduces to exponential averaging of one expert's own trajectory, which is not shown to produce functional diversity.","fun_headline_variants_meta":{"raw":{"variants":["Evolving experts from a single FFN beats replicated MoE init","One expert evolves into diverse specialists for multimodal LLMs","EvoMoE: experts evolve, routers adapt per token","Fixing expert uniformity and router rigidity in MoE MLLMs","Dynamic routing and expert evolution boost sparse MLLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2691,"prompt_tokens":1135,"completion_tokens":1556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":751,"completion_tokens_details":{"reasoning_tokens":1472}},"tokens_in":751,"tokens_out":1556,"duration_ms":12620,"temperature":1.0,"reasoning_tokens":1472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:15:28.575010+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same MoE-LLaVA baseline but replace expert evolution with a plain exponential moving average of the single expert's weights (or EMA plus dropout), keeping the DTR unchanged; if benchmark scores match EvoMoE's within noise, the evolutionary-diversity story is not what drives the gains. Alternatively, measure a functional diversity metric, such as pairwise token-routing overlap or output cosine distance between the four experts after training; if evolved experts are as similar to expert 1 as replicated experts are, the uniformity problem is not actually solved.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"LLaVA-mix-665k: the instruction dataset used in both Stage II expert evolution and Stage III DTR training."}],"review_version":1}