{"id":"91acaff4-56b1-4d7d-b6e4-3b52f9d6bfb5","arxiv_id":"2505.20225","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"This paper releases seven open-source MoE language models with full training artifacts and reports up to 3.4 point accuracy gains over dense baselines at equal FLOPs.","lead":"FLAME-MoE is an open-source suite of seven mixture-of-experts language models, from 38M to 1.7B active parameters, with all training data, code, checkpoints, and logs released. The paper reports that FLAME-MoE outperforms compute-matched dense baselines by up to 3.4 points on six downstream tasks, and includes initial analyses of expert specialization and routing stability.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 3.4-point gain comes from an 8e19-FLOPs dense baseline (Dense-411M at ~79 tokens/param) that is far below its compute-optimal size; a fairer dense baseline would likely shrink the claimed advantage.","rationale":"The reader's weakest assumption correctly flags the fairness of the dense baselines, but the more precise and testable failure is visible from the paper's own Table 3 and Eq. (6): the dense baselines are not selected at their compute-optimal sizes, especially at the 8e19 budget that yields the maximum reported gain. At 2.4e20 the dense baseline runs at ~20 tokens/param, while at 8e19 Dense-411M runs at ~79 tokens/param, an internal inconsistency in how 'identical FLOPs' baselines are chosen. The MoE models, by contrast, were chosen via the paper's fitted scaling law. This makes the comparison asymmetric and likely inflates the MoE advantage, though it does not by itself invalidate the platform's transparency contribution or the smaller gains at other budgets. The proposed retraining test is expensive but decisive: if a compute-optimal dense baseline closes most of the gap, the central accuracy claim is materially weakened; if the gap persists, the concern is resolved. This supports keeping the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":16568,"tokens_out":9709,"duration_ms":98205,"concrete_test":"Train a dense checkpoint of about 822M parameters at 8e19 FLOPs (D ≈ 1.62e10 tokens, ~20 tokens/param) using the same DCLM data, Adam/WSD settings, sequence length, and 6-task harness as §5.1; if the average of the six tasks reaches 0.47 or above, the 3.4-point claim is largely a baseline-selection artifact. Run three seeds to bound noise, and optionally repeat at 3e19 with a ~500M dense model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim ('up to 3.4 points over dense baselines trained with identical FLOPs') is load-bearing on the choice of dense baselines in Table 3. Equation (6) defines D = C/(κ·N_active), and for Dense-411M at 8.0e19 this gives about 3.24e10 tokens, i.e., 79 tokens per parameter. By contrast, Dense-1.4B at 2.4e20 gets about 2.86e10 tokens, i.e., 20 tokens per parameter. Under the same Chinchilla-style accounting used in the paper, the compute-optimal dense size for 8e19 would be roughly sqrt(8e19/(6*20)) ≈ 8e8 parameters, not 4.11e8. The 8e19 row is exactly the row producing the headline maximum: FLAME-MoE-721M-3.8B averages 0.4782 versus 0.4447 for Dense-411M. The MoE side was selected through the paper's own isoFLOP scaling law, but no equivalent isoFLOP study is reported for dense models, so the comparison pits a compute-optimal MoE against an overtrained dense model. The same issue affects the 3e19 row (Dense-411M at 30 tokens/param) to a lesser degree. This is not a dispute about the FLOPs formula; even accepting Eq. (6) at face value, the dense baselines are not at their compute-optimal allocation, which understates dense performance and inflates the reported MoE gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FLAME-MoE introduces a fully open research platform for Mixture-of-Experts language models: seven decoder-only MoE models (38M–1.7B active parameters, 64 experts, top-8 routing, 2 shared experts), trained on DCLM data, with all code, data pipelines, checkpoints, logs, and routing traces released. The paper fits an empirical scaling law (IsoFLOP profiles plus a parametric loss) to select supposedly compute-optimal model sizes, compares the MoE models against dense baselines trained at the same nominal FLOPs on six downstream tasks, and reports average accuracy gains up to 3.4 points. It also presents analyses of expert specialization, expert co-activation, and router saturation using the released checkpoints.","tokens_in":16951,"tokens_out":8464,"duration_ms":87383,"significance":"If the claims hold, FLAME-MoE is a valuable community asset: it is, to my knowledge, the first MoE suite to release checkpoints plus full routing traces across seven scales, matching the Pythia-style transparent platform that the dense community has long had. The scaling-law fitting is transparent, the evaluation uses standard tasks, and the router-dynamics analyses are a useful template. The central empirical claim—that MoE improves average accuracy over identically-FLOPs dense baselines—is plausible and would be an important data point, but it currently rests on assumptions about dense-baseline optimality, FLOPs accounting, and statistical significance that are not fully validated in the manuscript.","major_comments":[{"comment":"The headline 'up to 3.4 points' gain comes from the 8.0e19 row, where Dense-411M is trained with D = C/(κN) ≈ 3.24e10 tokens, i.e., about 79 tokens per parameter. Under the same Chinchilla-style accounting used in the paper, the compute-optimal dense model at this budget would have about 8.2e8 parameters and about 20 tokens per parameter. No dense isoFLOP sweep is reported, so the comparison pits a compute-optimal MoE (selected by the paper's scaling law) against an overtrained dense model. To support the claim, the authors should provide dense isoFLOP profiles for the same budgets or otherwise demonstrate that the dense baselines in Table 3 are at their compute-optimal allocation.","section":"§5.1, Table 3, Eq. (6)"},{"comment":"All evaluation results are single runs with no error bars, confidence intervals, or significance tests. Some reported gaps are small (e.g., 0.69 points at 1.0e18, 2.3 points at 6.0e18) and could easily be within run-to-run variance on six tasks. The abstract's 'up to 3.4 points' is a single realization. At minimum, multiple seeds for both MoE and dense models, or a bootstrap analysis over tasks, are needed to establish that the improvements are not noise.","section":"Table 3, §5.2"},{"comment":"The 'compute-optimal' label for the released family relies on a power law fitted to four budgets (1e18, 3e18, 6e18, 3e19) and extrapolated to 8e19 and 2.4e20—factors of 2.7 and 8 beyond the largest fitted budget. No confidence intervals are reported for the fitted exponents (a=0.69, b=0.31), and no held-out budget is used to validate the extrapolation. The claim that FLAME-MoE-721M-3.8B and FLAME-MoE-1.7B-10.3B are compute-optimal is therefore not established; the authors should report uncertainty in the scaling-law parameters and, if possible, validate on an intermediate held-out budget.","section":"§4.2, §4.3, Fig. 1c, Table 4"},{"comment":"The 'identical FLOPs' premise is defined by Eq. (6), FLOPs = κ·N_active·D with κ=6, but the paper does not validate this formula against measured FLOPs for either architecture. For MoE models, attention, embedding, router, and communication overheads are not captured by κ·N_active·D, and §5.3 shows that MoE training has lower realized throughput than dense training. Since the central comparison is explicitly 'identical FLOPs,' the authors should report profiler-measured FLOPs for at least the rows in Table 3, or clearly state that the comparison is theoretical and explain why the approximation is unbiased.","section":"§5.1, Eq. (6)"}],"minor_comments":[{"comment":"The footnotes 'DCLM 400M-1x', 'DCLM 400M-4x', and 'DCLM 1B-1x' are never defined; please clarify what these data-mixture/repetition labels mean and how they map to the token counts implied by Eq. (6).","section":"Table 3 footnotes"},{"comment":"The summation index i in the router z-loss appears to range over tokens, but i is also used for expert indices in Eqs. (2)–(3); using t or j for the token index would remove ambiguity.","section":"Eq. (4)"},{"comment":"The IsoFLOP curves are plotted against active parameters, but the four FLOPs budgets are only given in the legend; labeling each curve directly would improve readability.","section":"Fig. 1a"},{"comment":"The text says the best configuration is PP=1 and EP=8, but the figure also shows PP=2 variants; please state the criterion used to select PP=1/EP=8 (e.g., throughput, memory, or stability).","section":"§5.3, Fig. 3"},{"comment":"The phrase 'up to 3.4 points' is the maximum row-wise gain, not the average; please also report the mean gain across the seven comparisons so readers can gauge the typical effect.","section":"Abstract, §5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's value as an open platform is real and the release is unusually complete, but the empirical comparison in Table 3 is the main scientific claim and it depends on the fairness of the dense baselines. The 8e19 row is the most problematic: Dense-411M is clearly overtrained under the paper's own accounting. I would not reject because the issue is fixable—adding dense isoFLOP profiles, multiple seeds, and uncertainty bounds on the scaling law would address it. The 'up to' phrasing in the abstract should be scrutinized by the editor, as it highlights a single favorable row; however, the consistent positive direction across rows is reassuring."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real value here is the artifact: a seven-model open MoE suite with checkpoints, training logs, routing traces, and code. That genuinely fills a gap and gives the MoE community something like Pythia. The scaling-law selection of model sizes is careful, and the paper is transparent about its setup. The behavioral analyses of specialization, co-activation, and router saturation are competent but mostly confirm what OLMoE and OpenMoE already showed. So the newness is the platform, not the findings.\n\nThe soft spot is the central accuracy claim. The stress-test note is correct and it matters. In Table 3, at the 8e19 FLOPs budget, Dense-411M is trained on about 79 tokens per parameter. Under the same Chinchilla-style accounting the paper uses, a compute-optimal dense model for 8e19 FLOPs would be roughly twice as large and trained on half as many tokens per parameter. That row is the one that produces the headline 3.4-point gap. Comparing a MoE selected by your own isoFLOP law against a dense model that is overtrained by a factor of four is not a fair equal-compute comparison. The same issue affects the 3e19 row to a lesser extent. This does not sink the platform, but it means the abstract's claim is overstated.\n\nOther concerns are secondary: single-run evaluations with no error bars, and a scaling law fitted to four budgets extrapolated to 2.4e20. These are addressable and don't undermine the release.\n\nWho is this for? Anyone doing MoE routing analysis, scaling studies, or sparsity research who wants a fully instrumented training suite. That audience gets real value. But the paper should be revised to either add compute-optimal dense baselines or explicitly report the dense-side token/parameter ratios and justify why the chosen dense sizes are appropriate. As it stands, the empirical efficiency claim needs qualification.\n\nRecommendation: send it to peer review, but with a clear request to fix the dense baseline comparison and add uncertainty estimates. The artifacts deserve careful refereeing; the 3.4-point claim does not deserve to stand as written.","headline":"Useful open MoE platform, but the headline 3.4-point accuracy gain over dense baselines is inflated by overtrained dense models at the key compute budget.","tokens_in":17462,"tokens_out":1317,"would_cite":true,"duration_ms":15634,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that FLAME-MoE, a fully open family of seven Mixture-of-Experts language models (38M to 1.7B active parameters), improves average accuracy by up to 3.4 points over dense baselines trained at identical FLOPs, and releases…","keywords":["mixture-of-experts","language models","sparse routing","scaling laws","expert specialization","router saturation","open-source research platform","training dynamics"],"falsifier":"Re-train the same dense-MoE pairs while measuring actual FLOPs with a profiler and equalizing total compute, wall-clock time, or energy, then re-run the six-task evaluation; if the average accuracy gap drops below or reverses the reported 1.8 to 3.4 points, the central comparison fails. A cheaper falsifier is to check whether the power-law exponents reproduce when $\\kappa$ is set from measured FLOPs rather than the standard value of 6.","tokens_in":16375,"feed_emoji":"🧩","tokens_out":8153,"duration_ms":79052,"temperature":0.7,"pith_summary":"This paper introduces FLAME-MoE, a fully open suite of seven decoder-only Mixture-of-Experts language models (38M to 1.7B active parameters) with 64 experts per layer, top-8 routing, and two shared experts. Its central claim is that at equal training FLOPs the MoE models outperform dense transformers of comparable size by up to 3.4 average accuracy points across six tasks. The paper also reports three training-dynamics findings from full checkpoints: token-level expert specialization grows over time, expert co-activation remains sparse, and router selections converge early. If true, FLAME-MoE gives academic researchers a reproducible platform to study MoE scaling, routing, and expert behavior across model sizes rather than only benchmark final checkpoints.","feed_headline":"Open MoE family beats dense baselines by up to 3.4 points","feed_subtitle":"Seven fully open MoE models from 38M to 1.7B active parameters, with full training traces released.","key_machinery":"The load-bearing object is the MoE layer: $N_E = 64$ expert FFNs per layer, top-$k = 8$ routing, and two shared experts that fire for every token, giving $k = 8$ active experts per token. The scaling-law machinery treats active parameters $N_{\\text{active}}$ and tokens $D$ as the compute variables through $C = \\kappa N_{\\text{active}} D$ with $\\kappa = 6$, fitting an IsoFLOP profile and a parametric loss $L_{\\text{val}}(N_{\\text{active}}, D) = A/N_{\\text{active}}^\\alpha + B/D^\\beta + L_0$ to select compute-optimal sizes. The trace analyses define specialization as the fraction of a token's occurrences routed to one expert, co-activation as the conditional probability that two experts appear together in a token's top-k set, and saturation as the step-by-step overlap of top-k selections with the final checkpoint.","core_discovery":"On the paper's own terms, the central discovery is that a compute-optimal MoE family with 64 experts and top-8 routing consistently beats dense transformers trained at the same FLOPs budget, with average accuracy gains of 1.8 to 3.4 points that widen at larger scales. The paper attributes this to effective capacity without proportional compute cost, and it supports the claim by training all models in the same codebase and releasing every checkpoint. Its trace analyses add three behavioral findings: token-level expert specialization intensifies during pretraining, expert co-activation is sparse overall yet stronger in deeper layers, and router selections converge to near-final assignments within the first few thousand steps. The paper presents the full release of code, data pipelines, logs, and checkpoints as the infrastructure that makes these observations reproducible.","pith_inferences":["The paper measures FLOPs as $\\kappa \\cdot N_{\\text{active}} \\cdot D$, so the headline 3.4-point gain should not be read as a wall-clock or memory advantage; the paper itself reports that MoE throughput lags dense models, so the efficiency claim is about theoretical compute, not realized speed.","Because the released traces include checkpoints, a natural extension is to test whether freezing or pruning the router after saturation preserves downstream accuracy; the paper does not run this experiment.","The specialization score's token-frequency normalization suggests a sharper check: measure per-token accuracy on rare versus frequent tokens to see whether expert specialization improves handling of rare tokens specifically.","The seven-scale suite could support scaling-law analyses of routing dynamics themselves, such as how the saturation step count grows with active parameters; the paper only reports descriptive trends."],"forward_implications":["At equal FLOPs, the MoE advantage should persist and widen as the training budget grows, since the reported gains rise from 1.8 points at smaller budgets to 3.4 points at the largest.","The released checkpoints allow independent replication of the three training-dynamics findings at every scale, not just at the final model.","The fitted scaling law with exponents $a \\approx 0.690$ and $b \\approx 0.310$ gives a resource-allocation rule: for a target FLOPs budget, the compute-optimal active-parameter count and token count follow these power laws.","Router saturation within the first few thousand steps implies that routing behavior is largely determined early; comparisons of routing interventions should therefore focus on early training."],"supporting_citations":[{"why":"Supplies the IsoFLOP and parametric-loss methodology used to choose compute-optimal model sizes.","marker":"[11]"},{"why":"Supplies the 64-expert top-8 architecture template, the evaluation task list, and the specialization and co-activation definitions that FLAME-MoE adapts.","marker":"[25]"},{"why":"Supplies the shared-expert design (two of the eight active experts shared across tokens) adopted in the MoE layer.","marker":"[20]"},{"why":"Provides the dense baseline architectures and the open-suite model that FLAME-MoE seeks to parallel for MoE research.","marker":"[3]"},{"why":"Provides the pretraining corpus and the dense baseline configurations used for the equal-FLOPs comparisons.","marker":"[19]"},{"why":"Provides the training framework on which the pretraining implementation and parallelization experiments are built.","marker":"[31]"},{"why":"Supplies the evaluation harness adapted for the six downstream accuracy tasks.","marker":"[8]"},{"why":"Supplies the router z-loss used to stabilize MoE training.","marker":"[40]"},{"why":"Supplies the load-balancing loss that keeps token dispatch across experts even.","marker":"[29]"}],"fun_headline_variants":["Open MoE family tops dense by up to 3.4 accuracy points","MoE beats dense by 3.4 pts, all open source","MoE with 64 experts: +3.4 accuracy over dense, fully open","Full training traces released: MoE beats dense by up to 3.4 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The equal-FLOPs comparison assumes that FLOPs computed as $\\kappa \\times \\text{active parameters} \\times \\text{tokens}$ with $\\kappa = 6$ captures the full training cost and that the dense baselines are as well tuned as the MoE models; if that accounting favors MoE or the dense baselines are undertuned, the reported accuracy advantage is not evidence of intrinsic MoE efficiency.","fun_headline_variants_meta":{"raw":{"variants":["Open MoE family tops dense by up to 3.4 accuracy points","MoE beats dense by 3.4 pts, all open source","MoE with 64 experts: +3.4 accuracy over dense, fully open","Full training traces released: MoE beats dense by up to 3.4 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001367,"raw_usage":{"total_tokens":5553,"prompt_tokens":962,"completion_tokens":4591,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":4505}},"tokens_in":578,"tokens_out":4591,"duration_ms":34183,"temperature":1.0,"reasoning_tokens":4505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:56:27.455936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the same dense-MoE pairs while measuring actual FLOPs with a profiler and equalizing total compute, wall-clock time, or energy, then re-run the six-task evaluation; if the average accuracy gap drops below or reverses the reported 1.8 to 3.4 points, the central comparison fails. A cheaper falsifier is to check whether the power-law exponents reproduce when $\\kappa$ is set from measured FLOPs rather than the standard value of 6.","supporting_citations":[{"cited_title":"Rae, and Laurent Sifre","cited_arxiv_id":null,"evidence_quote":"Supplies the IsoFLOP and parametric-loss methodology used to choose compute-optimal model sizes."},{"cited_title":"OLMoE: Open mixture-of- experts language models","cited_arxiv_id":null,"evidence_quote":"Supplies the 64-expert top-8 architecture template, the evaluation task list, and the specialization and co-activation definitions that FLAME-MoE adapts."},{"cited_title":"DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model.ArXiv preprint, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the shared-expert design (two of the eight active experts shared across tokens) adopted in the MoE layer."},{"cited_title":"Pythia: A suite for analyzing large language models across training and scaling","cited_arxiv_id":null,"evidence_quote":"Provides the dense baseline architectures and the open-suite model that FLAME-MoE seeks to parallel for MoE research."},{"cited_title":"DataComp-LM: In search of the next generation of training sets for language models","cited_arxiv_id":null,"evidence_quote":"Provides the pretraining corpus and the dense baseline configurations used for the equal-FLOPs comparisons."},{"cited_title":"Megatron-lm: Training multi-billion parameter language models using model parallelism.ArXiv preprint, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the training framework on which the pretraining implementation and parallelization experiments are built."},{"cited_title":"ST-MoE: Designing stable and transferable sparse expert models.ArXiv preprint, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the router z-loss used to stabilize MoE training."},{"cited_title":"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.ArXiv preprint, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the load-balancing loss that keeps token dispatch across experts even."}],"review_version":1}