{"id":"c585ed6e-666f-4bd5-a0a1-791859a9380a","arxiv_id":"2505.22582","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A layer-wise expert allocation algorithm based on hidden-state similarity, plus a routing classifier, improves parameter efficiency and reduces forgetting when expanding LLMs to new languages.","lead":"This paper proposes a way to add new languages to an existing AI language model using fewer extra expert modules, by measuring how similar the model's internal representations are across languages at each layer. It reports better or comparable performance with up to 60% fewer added parameters than a prior state-of-the-art method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never tests the directional claim behind Eq. (10): similarity-guided allocation is only compared to random or uniform schedules, not to reversed or alternative signals, so the reported gains may owe to non-uniformity plus tuned K and the classifier rather than to HSA similarity.","rationale":"I read the paper in good faith. The method is concrete, the code is released, and the comparisons are mostly reproduced with matched data. The ablation results are consistent with the claim that allocation and classifier placement matter, and the Llama-3.2-3B result is a useful external check. However, the paper's central contribution is not just 'non-uniform expert allocation helps'; it is specifically that HSA similarity should be used in an inverse relation, as encoded in Eq. (10). That directional claim is the least secure part of the argument. The random-allocation ablation can only show that the specific similarity-guided schedule is better than one arbitrary schedule; it cannot distinguish the inverse-similarity hypothesis from the general benefit of a carefully chosen non-uniform allocation. The uniform baseline MoE-LPR(3x24) is likewise not a directional control, and MOLA is not matched in total parameters. A reversed-allocation experiment is the minimal counterfactual that would actually test the paper's mechanism. The issue is compounded by the fact that K, the number of classifier layers, is tuned on the same benchmark set used for the headline claim, and no variance or significance information is reported. These factors together mean that the observed gains could plausibly arise from selection and non-uniformity rather than from the semantic content of the similarity signal. Because the required experiments are well-defined and feasible, conditional acceptance remains the right posture; the concern does not by itself warrant rejection, but the paper's acceptance should be contingent on the directional counterfactual and on a held-out hyperparameter selection procedure.","tokens_in":20135,"tokens_out":6880,"duration_ms":88158,"concrete_test":"Fix the G0->G1 single-expansion setting with delta=72 total experts and K=7 classifiers, and vary only the allocation rule: (a) inverse similarity per Eq. (10), (b) proportional similarity (reversed direction), (c) uniform 3 experts per layer, and (d) MOLA's depth schedule. Train each variant with at least three random seeds and evaluate on a held-out half of the benchmarks (or on a held-out language pair) so classifier K is not selected on the test set. If the reversed or uniform allocation matches LayerMoE within seed error (e.g., average score within about 0.2 points), the inverse-similarity mechanism is unsupported and the 60%-fewer claim should be attributed to non-uniform allocation plus the tuned classifier. Also verify whether summing the rounded N_i from Eq. (10) exactly equals 72; if it exceeds 72, recompute the actual parameter reduction before comparing with MoE-LPR.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is parameter-efficiency: with 60% fewer added experts than MoE-LPR(6x24), LayerMoE achieves better old- and new-language averages. The mechanism justifying that reduction is Eq. (10), which sets per-layer expert counts inversely proportional to HSA similarity. The load-bearing assumption is that this direction and functional form are causal: high similarity reliably means fewer experts are needed. The paper's evidence is Figure 1 plus the random-allocation ablation, but neither tests the direction. Random allocation is a weak control because it confounds 'similarity content' with 'non-uniformity'; any sensible non-uniform schedule, or the tuned classifier placement, could produce the same advantage. The uniform baseline MoE-LPR(3x24) is not a directional control, and MOLA uses a different allocation rule with different total parameters. No reversed allocation (N_i proportional to S_i), no allocation transferred from held-out language pairs, and no post-training check of whether the original similarity still predicts routing load is reported. The generalization experiment (Table 5) covers only one setting and has no ablation. Compounding this, K is selected on the same benchmarks used for the headline results (top-7 chosen from Table 4), and no seed variance is reported, so part of the gap may be selection noise. These issues are addressable, but unless the directional hypothesis survives a counterfactual test, the headline improvement is not tied to the proposed cause.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LayerMoE, a method for expanding an existing LLM with new languages by adding layer-wise MoE experts. The method first measures cosine similarity between hidden states of old and new languages at each layer, then allocates more experts to layers with lower similarity and fewer to layers with higher similarity (Eq. 10). On high-similarity layers, a classifier is inserted before the router to route old-language tokens to the frozen old expert. Experiments on Qwen1.5-1.8B and Llama-3.2-3B across single-expansion and lifelong-expansion settings report that LayerMoE outperforms MoE-LPR baselines while using fewer added parameters, and the authors include ablations showing degradation under random expert allocation and without the classifier.","tokens_in":20471,"tokens_out":6174,"duration_ms":70460,"significance":"If the results hold, LayerMoE is a practically valuable contribution to parameter-efficient multilingual expansion: it reduces the parameter cost of adding languages and improves old-language retention compared to a strong MoE baseline. The paper ships code and reports experiments on four benchmarks, two models, and two expansion settings, with an ablation study and an honest limitations section. However, the significance is tempered by the small size of the reported gains, the absence of variance estimates, and the fact that the central mechanism—that HSA similarity should drive expert allocation in the specific inverse direction—is not directly tested.","major_comments":[{"comment":"The central directional claim—higher HSA similarity requires fewer new experts—is not tested. The ablation 'w/random' replaces the similarity-based allocation with a random schedule, but this only varies the allocation pattern while preserving non-uniformity; it does not control for the direction of the relation. A uniform baseline such as MoE-LPR(3*24) is also not a directional control. To establish that Eq. (10)'s inverse relation, rather than merely non-uniformity or the presence of a classifier, drives the gains, the paper should include a reversed allocation (N_i proportional to S_i), an allocation using a shuffled similarity vector, or an allocation transferred from a different language pair. Without such a counterfactual, the reported gains could equally be attributed to the tuned non-uniform schedule and classifier placement.","section":"§3.2, Eq. (10); §5.2, Table 3"},{"comment":"The number of classifier layers K (top-7 for single-expansion, top-5 for lifelong-expansion) is selected on the same evaluation benchmarks used for the headline results, with the text stating that these settings are taken 'according to the best performance.' Table 4 shows that the differences among top-6, top-7, and top-8 (avgs 40.23, 40.26, 40.17) are very small, and the paper reports no multiple seeds or error bars anywhere. The headline improvements compared to MoE-LPR (e.g., 0.19/0.44 points in New/Old-avg for G1) could therefore partly reflect selection noise. The authors should report seed variance or, better, select K on a validation set and evaluate on a held-out test set.","section":"§4.1; Table 4"},{"comment":"The abstract's claim of '60% fewer experts' and '33.3% fewer experts' is inaccurate. The tables report parameter reductions (1.6B vs 4B = 60% parameter reduction; 3.2B vs 4.8B = 33.3% parameter reduction), but the corresponding expert counts are 72 vs 144 (50% fewer) and 144 vs 168 (14.3% fewer) under the paper's δ=72-per-expansion allocation. The abstract and the 'fewer experts' phrasing should be corrected to refer to parameter counts, or the expert-count percentages should be computed correctly. This matters because the title and headline claim are explicitly about being parameter-efficient.","section":"Abstract; Tables 1 and 2"},{"comment":"The description of the similarity computation is not reproducible as written. The paper says Q=100,000 tokens are sampled per language and that similarity is computed as the average over all Q^2 pairs. With Q=10^5, this is 10^10 cosine computations per layer per language pair, which is computationally infeasible for the 24-layer model and multiple language pairs described. Please clarify whether pairs are subsampled, whether the mean HSA vector is used, or whether some other approximation is applied.","section":"§3.1, Eq. (8)"},{"comment":"The generalization study on Llama-3.2-3B covers only one setting (G0→G1) and compares only against MoE-LPR(3*24), with no ablation of the allocation or classifier components. This supports the claim that the overall pipeline transfers, but it does not provide evidence that the similarity direction or the classifier design generalizes. The text should be scoped accordingly rather than claiming broad cross-model effectiveness.","section":"§5.4, Table 5"}],"minor_comments":[{"comment":"In Eq. (13), cls(x) is defined as argmax(x·Wr), but the classifier network is Wc; it should be argmax(x·Wc).","section":"§3.3, Eq. (13)"},{"comment":"The text says the classifier placement uses S_new&old computed 'following Eq.(9)', but Eq.(9) defines S as the average of S_new&old and S_new&new. Please clarify whether the classifier uses only the new-old component or the averaged value.","section":"§3.3 and §3.2"},{"comment":"For lifelong expansion, it is not explicitly stated whether L_old at each step includes all previously acquired languages (G0 plus earlier groups) or only the original G0. This affects Eq.(9) and should be clarified.","section":"§4.1, lifelong-expansion"},{"comment":"The text refers to the 'lm-evaluation-hardness' framework, which appears to be a typo for 'lm-evaluation-harness.'","section":"§4.1"},{"comment":"The sentence 'Gao et al. proposes MOLA' is missing the citation marker; it should cite (Gao et al., 2024a) at the first mention of MOLA.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and an honest limitations section, but the headline percentages overstate the expert-count savings, and the central directional mechanism is not yet supported by a proper counterfactual. I would like to see the reversed-allocation experiment and seed variance before considering acceptance. The K-selection issue is also important because the reported gains are small; a validation-based selection would materially strengthen the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the pairing of similarity-based layer-wise expert allocation with a routing classifier that sends old-language tokens straight to the old expert. That combination is not in MoE-LPR or MOLA, and the ablation gives it some support: random allocation hurts new-language scores, and removing the classifier hurts old-language preservation. The code is public, which is real credit.\n\nThat said, the core mechanism is under-tested. Eq. (10) assumes higher HSA similarity means fewer experts needed, but the paper never tests the direction. Comparing to random allocation is a weak control because it changes both the content and the uniformity of the schedule; a reversed allocation (more experts where similarity is high) or an allocation from a held-out language pair would be the real test. The classifier placement also suffers from selection: K is tuned on the same benchmarks used for the headline numbers (Table 4), and no seeds or error bars are reported. The gains are small—New-avg for G1 goes from 34.52 (MoE-LPR 6x24) to 34.71, and Old-avg from 45.36 to 45.80—so with no variance, those differences could easily be noise. The Llama-3.2-3B generalization is a single setting with no ablations, and the avg gain over MoE-LPR (3x24) is 0.17 points, essentially null.\n\nThere is also a replication problem in Eq. (8): as written, it is O(Q^2) per layer pair, and with Q = 100,000 that is 10^10 cosine computations per layer. Either the description is wrong or there is an approximation that must be stated. Eq. (10) also has a rounding/constraint issue: ceiling each N_i will break the promised sum delta, and the paper doesn't say how the constraint is actually enforced. Minor but worth fixing.\n\nTo the paper's credit, the Limitations section honestly acknowledges that delta is fixed to match the baseline and that improvements are inconsistent. That is the kind of transparency that makes the work salvageable.\n\nThe target audience is researchers working on multilingual continual learning, MoE upcycling, and low-resource language expansion. The idea is worth borrowing, but the evidence presented does not yet establish that the inverse-similarity rule is what drives the gains.\n\nRecommendation: send it to peer review. A serious referee should require seeds and error bars, a reversed-allocation control, details on how Eq. (8) is actually computed, and a clarification of the rounding procedure. The paper is not ready in its current form, but the core question is worth referee time.","headline":"A plausible parameter-efficient multilingual expansion method with an untested core heuristic and thin empirical margins; worth refereeing but in need of major revisions.","tokens_in":20962,"tokens_out":3190,"would_cite":true,"duration_ms":39533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LayerMoE claims that allocating new MoE experts per layer in inverse proportion to cross-lingual hidden-state similarity lets an LLM expand to new languages with 60% fewer added parameters while better preserving old ones.","keywords":["layer-wise expert allocation","mixture-of-experts","multilingual LLM expansion","catastrophic forgetting","cross-lingual hidden state similarity","MoE-LPR","lifelong language expansion","parameter-efficient expansion"],"falsifier":"Compute, per layer and with the total expert budget held fixed, the marginal gain in new-language benchmark score from adding one expert, and correlate that gain with the layer's measured new-old hidden-state similarity; if higher-similarity layers show equal or larger marginal gains, the inverse-allocation premise is false. A cheaper check: compare LayerMoE's allocation against a random allocation with the same total budget and the same top-$K$ classifier placement, repeated over several seeds; if the random allocation matches it, the similarity signal contributes nothing.","tokens_in":19973,"feed_emoji":"🌐","tokens_out":7109,"duration_ms":65313,"temperature":0.7,"pith_summary":"The paper tries to establish that when an LLM is expanded with new languages by adding Mixture-of-Experts (MoE) layers, the number of new experts should vary by layer, and the layer-wise signal for how many to add is how similar the hidden states of different languages are after the attention module. It proposes LayerMoE, which measures pairwise cosine similarity of hidden-state activations at each layer and allocates experts in inverse proportion to that similarity, giving more experts where languages look most different and fewer where representations already overlap. It also inserts a small classifier before the router on the most similar layers to send old-language tokens to the original expert, mitigating forgetting. On Qwen1.5-1.8B and Llama-3.2-3B, the paper reports that this beats the uniform-allocation MoE-LPR baseline with 60% fewer added experts in a single expansion and 33.3% fewer in lifelong expansion. If right, this makes continual multilingual expansion cheaper and safer for existing models.","feed_headline":"Similarity-based expert allocation cuts MoE expansion cost by 60%","feed_subtitle":"LayerMoE sizes each layer's new experts by cross-lingual similarity and guards old languages with a router classifier.","key_machinery":"The load-bearing mechanism is the per-layer indicated similarity $S^i$, computed as the average pairwise cosine similarity of hidden states after the attention module between sampled token sets of the new and old language groups (and within the new group). The allocation formula $N^i = \\lceil ( (S^i)^{-1} / \\sum_j (S^j)^{-1} ) \\cdot \\delta \\rceil$ distributes a total expert budget $\\delta$ inversely to similarity, so high-similarity layers get few new experts and low-similarity layers get many. On the top-$K$ layers by new-old similarity, a linear classifier $W_c \\in \\mathbb{R}^{h \\times 2}$ in front of the router decides whether a token belongs to an old language and, if so, routes it to expert 0.","core_discovery":"The paper's central claim is that representation similarity between languages at each layer is a usable predictor of how much fresh capacity that layer needs when a language is added. The authors compute, per layer, the average cosine similarity between randomly sampled hidden states of old and new languages, call this the indicated similarity, and set the number of new experts proportional to the inverse of that similarity under a fixed total budget. They also find that layers with higher new-old similarity are exactly where the router gets confused, so they train a binary classifier there that routes old-language tokens straight to the frozen old expert. The reported experiments show that this scheme outperforms the previous method on both old-language retention and new-language acquisition while using far fewer added parameters.","pith_inferences":["The inverse-similarity rule reallocates a fixed parameter budget to layers where representations are most language-specific; an implication the paper leaves implicit is that the advantage over uniform allocation should shrink as the total budget grows very large, since capacity ceases to be the binding constraint.","The offline similarity computation could be reused as a prior for language pairs: the same vector of per-layer similarities might predict good allocations for other base models of the same family without retraining.","Since the classifier helps most on high-similarity layers, a testable extension is whether a single language-identification head shared across layers, or a head trained on new-new pairs as well, further improves routing in lifelong settings."],"forward_implications":["At a fixed total number of added experts, similarity-based per-layer allocation beats uniform allocation on both old-language preservation and new-language acquisition.","Adding the router-front classifier on high-similarity layers reduces old-language forgetting beyond what language-prior routing alone achieves.","The method transfers to a different base model (Llama-3.2-3B) and to a generative task (machine translation on FLORES), not just multiple-choice benchmarks.","In lifelong expansion, LayerMoE stays ahead with one-third fewer total experts, and the order in which language groups are added changes which languages gain most."],"supporting_citations":[{"why":"Supplies the two-stage MoE upcycling scheme, the language-prior routing loss, and the training recipes and baselines that LayerMoE extends and compares against.","marker":"(Zhou et al., 2024)"},{"why":"Provides MOLA, the alternative layer-wise expert allocation baseline that assigns more experts to higher layers; LayerMoE's similarity-based allocation is positioned against it.","marker":"(Gao et al., 2024a)"},{"why":"The Qwen1.5-1.8B model whose per-layer hidden-state similarity is measured in Figure 1 and which hosts the main single- and lifelong-expansion experiments.","marker":"(Team, 2024)"},{"why":"CulturalX, the multilingual corpus from which the sampled tokens for similarity calculations and the per-language training data are drawn.","marker":"(Nguyen et al., 2024a)"},{"why":"Prior evidence of language-agnostic and language-specific neurons in LLMs, cited to support the interpretation that high similarity means a layer is language-agnostic and needs fewer experts.","marker":"(Zhang et al., 2024b)"}],"fun_headline_variants":["LayerMoE: 60% fewer experts, same or better language gains","Similarity-guided experts slash multilingual LLM expansion cost","Per-layer expert sizing cuts MoE expansion parameters by 60%","Allocate experts by language similarity to shrink LLM expansion","Cross-lingual similarity predicts expert needs, cutting cost 60%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that cross-lingual similarity of hidden states after attention is a valid inverse signal for how many new experts a layer needs, with no other factor — such as the total expert budget, the choice of which layers receive the classifier, or the training data size — doing the real work.","fun_headline_variants_meta":{"raw":{"variants":["LayerMoE: 60% fewer experts, same or better language gains","Similarity-guided experts slash multilingual LLM expansion cost","Per-layer expert sizing cuts MoE expansion parameters by 60%","Allocate experts by language similarity to shrink LLM expansion","Cross-lingual similarity predicts expert needs, cutting cost 60%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1274,"prompt_tokens":953,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":232}},"tokens_in":569,"tokens_out":321,"duration_ms":3890,"temperature":1.0,"reasoning_tokens":232,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:04:13.505200+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, per layer and with the total expert budget held fixed, the marginal gain in new-language benchmark score from adding one expert, and correlate that gain with the layer's measured new-old hidden-state similarity; if higher-similarity layers show equal or larger marginal gains, the inverse-allocation premise is false. A cheaper check: compare LayerMoE's allocation against a random allocation with the same total budget and the same top-$K$ classifier placement, repeated over several seeds; if the random allocation matches it, the similarity signal contributes nothing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Qwen1.5-1.8B model whose per-layer hidden-state similarity is measured in Figure 1 and which hosts the main single- and lifelong-expansion experiments."}],"review_version":1}