{"id":"505229ed-e033-4184-95b6-ed0c4c7efefe","arxiv_id":"2607.22577","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A pipeline-level soft mixture of LLM streams implemented as dynamic convolution improves GPT-2-scale perplexity/GLUE/SQuAD, but only at substantially higher compute and without reported ParaScale/AltUp comparisons.","lead":"Researchers propose cMoLLM, a language-model architecture that treats mixture-of-experts layers as dynamic convolutions and routes tokens over parallel streams with soft, differentiable gating. They report perplexity and downstream-score gains over a dense GPT-2 baseline, but the comparisons use up to ~4.5x more compute and omit the named ParaScale/AltUp baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing: the 'matched compute' claim is contradicted by the paper's own Eq. (21): N=8 costs ~4.5x dense FLOPs, so Tables 2/3 compare unequal compute, not matched.","rationale":"The reader's verdict is REJECT with high correctness risk; I agree that rejection is warranted. I weight the matched-compute contradiction as the single most load-bearing issue because it is fully internal to the paper: Eq. (21) and the reported N=8 configuration imply a 4.5x compute ratio, so the abstract's 'under matched compute' and §5.3's 'matched training setups' cannot both be true. This alone invalidates the efficiency claim. The theory-pipeline gap (Theorem 4.1 covers a single linear-expert layer; §4.2 Eq. (12) describes a different structure) is a real and compounding concern, but the compute arithmetic is simpler and decisive. My verdict therefore does not change the reader's REJECT.","tokens_in":17118,"tokens_out":10526,"duration_ms":110590,"concrete_test":"Recompute per-token FLOPs in §4.7 for the N=8 configuration in Tables 2/3 using the stated d=768, d_ff=3072 (Table 5 / Appendix F): dense = 2·768·3072 ≈ 4.72M; cMoLLM = 9·768·3072 ≈ 21.2M plus gate, a 4.5x ratio. If 'matched compute' is intended as equal total FLOPs, this arithmetic falsifies the claim; if instead the comparison is at fixed parameter count or equal steps, the authors must state this and remove all 'matched compute' wording. Stronger version: run the released/reimplemented cMoLLM N=8 and dense baseline with total FLOPs held equal and check whether the PPL/GLUE gains in Table 2 persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central promise (abstract, §1, §5.3, conclusion) is that cMoLLM improves PPL/GLUE/SQuAD under matched compute. §4.7 gives FLOPs_cMoLLM ≈ (N+1)dd_ff + gate and FLOPs_dense ≈ 2dd_ff. For N=8, the configuration reported as best in Table 2 and used in scaling Table 3 (Appendix F: 'representative configuration'), the ratio is 9/2 = 4.5, not a matched-compute comparison. Eq. (22) is vacuous: it asserts a bound only for N ≤ C−1, but the real ratio is (N+1)/2, so the constant C would need to grow with N, and the 'within a constant factor' statement carries no content. If 'matched compute' means equal total FLOPs, the empirical gains in Tables 2–3 may simply reflect 4.5x more compute (and, with separate per-stream K_k, ~4.5x more parameters) than dense. This directly falsifies the headline efficiency claim. A separate, compounding gap is that Algorithm 1 implements only an FFN up-projection mixture (U = H(ΣgK)^T) whereas §4.2 Eq. (12) describes Transformers on concatenated branch outputs; Theorem 4.1 does not cover that structure. But the compute contradiction alone is decisive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes cMoLLM, a pipeline-level mixture-of-LLMs in which each of N 'streams' is associated with a 1x1 convolution kernel and a soft router mixes the kernels. The authors claim a formal equivalence between MoE layers and dynamic convolutions (Theorem 4.1), derive a 'horizontal scaling law' (Section 4.7, Eq. (22)), and report GPT-2-style experiments on FineWeb where cMoLLM improves perplexity, GLUE, and SQuAD 'under matched compute' relative to a dense baseline. The architecture is described in Section 4.2 and Algorithm 1, with four gating variants and scaling experiments at 85M, 350M, and 760M parameters.","tokens_in":17504,"tokens_out":6423,"duration_ms":64795,"significance":"If the matched-compute claim held, cMoLLM would be a useful contribution to conditional-computation scaling. The paper is also transparent in reporting a FLOP model, hyperparameters, and 3-seed mean/std statistics. However, the central claim is not supported: the paper's own Eq. (21) shows that for N=8 the proposed model costs about 4.5x the FLOPs of a dense FFN, so Tables 2-3 are not matched-compute comparisons. Moreover, Eq. (22) is mathematically impossible as stated, and Theorem 4.1 is a definitional identity that does not license the multi-layer architecture. The advertised comparisons to ParaScale and AltUp are not reported numerically. On the evidence in the manuscript, the headline efficiency/scaling claims fail.","major_comments":[{"comment":"The central 'matched compute' claim is contradicted by the paper's own FLOP model. Eq. (21) gives FLOPs_cMoLLM ≈ (N+1) d d_ff + gate versus FLOPs_dense ≈ 2 d d_ff. For N=8, the configuration with the best PPL in Table 2 and called 'representative' in Appendix F, the ratio is 9/2 = 4.5 before gating. No iso-FLOP comparison (e.g., reduced width, depth, or d_ff to equalize compute) is reported, so the gains in Tables 2-3 could simply reflect 4.5x more compute and parameters. This directly undermines the abstract and §5.3 headline claim of improvement 'under matched compute'.","section":"§4.7, Eq. (21); Tables 2-3"},{"comment":"The horizontal scaling-law bound is mathematically impossible as stated. Eq. (22) asserts existence of a constant C such that FLOPs_cMoLLM ≤ C·FLOPs_dense whenever N ≤ C^{-1}. But Eq. (21) implies the LHS/RHS ratio is about (N+1)/2. For N=8 this ratio is ≈4.5, requiring C ≥ 4.5; the condition N ≤ C^{-1} then requires C ≤ 1/8. No such C exists for N≥2. The 'within a constant factor' claim is therefore false for the experimental configurations, and the paper's 'horizontal scaling law' is not established.","section":"§4.7, Eq. (22)"},{"comment":"Theorem 4.1 is a definitional identity: by defining K~(x) := Σ_k g_k(x) W_k, Eq. (4) holds by linearity, so it is not a substantive equivalence that licenses the full cMoLLM pipeline. Algorithm 1 computes U_i = H_i K~_i^T, then a shared nonlinearity and down-projection, which is a single linear-expert MoE layer. In contrast, §4.2 Eq. (12) describes applying Transformer blocks to concatenated per-branch outputs X_input = [X_1,...,X_N]. The theorem does not cover the Eq. (12) structure, and no approximation or equivalence is proved for the actual multi-layer architecture used in the experiments.","section":"§4.1, Theorem 4.1; §4.2 Eq. (12); Algorithm 1"},{"comment":"Corollary 4.2 is stated for general Lσ-Lipschitz activations satisfying Assumption 3.2, but Appendix A proves the 'data-dependent mask' representation only for ReLU. The key step uses the identity ReLU(z) = I(z>0)·z, which has no analogue for GELU or other smooth Lipschitz activations. The corollary is therefore unproved in the stated generality; the statement should be restricted to ReLU or a proof for the general case must be supplied.","section":"Corollary 4.2; Appendix A"},{"comment":"The abstract and §5.3 claim that cMoLLM is better than 'ParaScale- and AltUp-style pipeline mixtures,' but Tables 2-3 contain only dense and cMoLLM results. Appendix F says ParaScale and AltUp were reimplemented and compared, yet no numerical results for those baselines appear anywhere in the manuscript. The comparative claims are unsupported by the presented evidence.","section":"§5.2, §5.3, Appendix F"}],"minor_comments":[{"comment":"Corollary 4.2 says 'Under Theorem 3.2'; the reference should be to Assumption 3.2, since no Theorem 3.2 is stated.","section":"Corollary 4.2"},{"comment":"The proof refers to 'as discussed in Theorem 4.3' for the binary mask; the intended reference appears to be Remark 4.3.","section":"Proposition 4.4 proof"},{"comment":"Theorem B.2 says 'as in Theorem B.1,' but the limitation of a single linear classifier is Proposition B.1, not Theorem B.1.","section":"Appendix B, Theorem B.2"},{"comment":"The sentence 'Extended results on downstream tasks and scaling are provided in ??' contains a broken cross-reference; the target section is missing.","section":"Section 6"},{"comment":"The caption says the right panel shows 'validation loss and perplexity vs. stream count n or training steps,' but no training-step variation is displayed or discussed; the caption should match the actual content.","section":"Figure 3"}],"recommendation":"reject","confidential_remarks":"The paper's two central advertised contributions—matched-compute gains and a horizontal scaling law—are contradicted by its own equations. The FLOP ratio in Eq. (21) and the impossible bound in Eq. (22) are load-bearing, and the claimed comparisons to ParaScale and AltUp are absent. These are not purely presentational issues; they concern the validity of the main claims. A resubmission would need, at minimum, genuine iso-FLOP experiments, a corrected scaling statement, and numerical baseline comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central theorem is a definition, and the paper's own FLOPs formula destroys the matched-compute claim. I read it after the stress-test note, and the note lands.\n\nWhat's honest here: the soft-gated mixture-of-streams idea is a reasonable engineering variant of MoE, and the authors are upfront that experiments are GPT-2 scale. The toy cluster model in Appendix B is a standard argument and, as far as it goes, fine. The writing is mostly clear.\n\nBut the load-bearing content doesn't hold up. Theorem 4.1 defines K~(x) := Σ g_k(x) W_k and then observes that Σ g_k(x) x W_k^T = x K~(x)^T. That's true by construction; it doesn't give you anything. Corollary 4.2 is only proved for ReLU even though it claims general Lipschitz activations; the appendix's mask argument is specifically the ReLU sign pattern. Proposition 4.4 is confused: it rewrites y = σ(Wx) as a sum of 'experts' with gating weights fixed to 1, which is just the same layer wearing a hat.\n\nThe empirical core fails on its own terms. Eq (21) gives FLOPs_cMoLLM ≈ (N+1) d d_ff vs ≈2 d d_ff for dense. For N=8, the configuration used in the scaling tables, that's 4.5× the compute. The paper's abstract and conclusion say 'under matched compute'; it is not. Even N=4 is 2.5×. So Tables 2 and 3 compare unequal compute, and the gains may simply be the usual 'more compute/parameters help' effect. Eq (22) is vacuous: the constant C is chosen after the fact to absorb the (N+1)/2 ratio, so the 'within a constant factor' claim is empty.\n\nLess central but still fair to note: ParaScale and AltUp are named as baselines, but no results for them appear in any table; the paper only compares to dense. Code is promised 'upon acceptance.' Those are exactly the places where a reader would want to see support.\n\nWho gets value from this? Nobody, as a research result; maybe a methods course as an example of how to disguise an identity as a theorem and then compare unequal compute. It's not a serious contender for publication in its current form. The authors could redo this with true iso-FLOP comparisons, report the baselines, release the code, and drop the 'scaling law' language — but that's a rewrite, not a revision.\n\nMy recommendation: desk-reject. Don't spend referee time on a paper whose central efficiency claim is contradicted by its own equations.","headline":"The headline theorem is a tautology and the 'matched compute' claim is contradicted by the paper's own FLOPs formula — the empirical gains are ~4.5x compute at N=8, not matched.","tokens_in":18019,"tokens_out":3493,"would_cite":false,"duration_ms":37130,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A normalized mixture of linear experts is exactly an input-dependent 1×1 convolution; cMoLLM uses this to scale LLM capacity by adding streams at near-dense compute.","keywords":["mixture-of-experts","dynamic convolution","soft routing","pipeline-level scaling","horizontal scaling law","conditional computation","language model scaling","1x1 convolution"],"falsifier":"Train cMoLLM with N=8 streams and a dense baseline whose per-token FLOPs are matched to cMoLLM's actual cost (about 4.5 times the dense FFN cost at N=8) on the same data; if the dense model matches or beats cMoLLM's perplexity, the horizontal scaling law is an artifact of an unequal compute comparison rather than a genuine capacity gain.","tokens_in":16936,"feed_emoji":"🔀","tokens_out":8638,"duration_ms":76261,"temperature":0.7,"pith_summary":"The paper tries to establish a precise identity: a mixture-of-experts layer with linear experts and normalized routing is not merely analogous to a dynamic 1×1 convolution—it is one, with the effective kernel equal to the input-weighted average of the expert kernels. On top of that identity it builds cMoLLM, which treats entire LLM streams as experts, mixes them with soft differentiable gating, and applies the mixed kernel through grouped pointwise convolution. If the identity extends to the full pipeline, model capacity can be scaled by adding streams while per-token computation stays near the dense baseline, roughly (N+1) times the FFN cost for N streams. In decoder-only models trained on a large public web corpus, the paper reports consistent perplexity and downstream accuracy improvements over a dense baseline at three model sizes, with stable stream utilization and no Top-K routing, virtual tokens, or auxiliary heads. A sympathetic reader would care because this points to a scaling axis—stream count—that is much cheaper than scaling width or depth.","feed_headline":"MoE layers equal dynamic convolutions — cheap LLM scaling follows","feed_subtitle":"One input-tuned kernel replaces many expert streams, so capacity grows without multiplying per-token compute.","key_machinery":"The load-bearing object is the MoE–dynamic-convolution equivalence: a 1×1 convolution applies the same matrix at every token position, and the theorem shows that mixing expert outputs with normalized weights is exactly convolution by the input-dependent averaged kernel. The architecture built on it is cMoLLM, where N parallel streams each own a kernel K_k, a lightweight gating network produces soft weights g_k(x), and the effective kernel K~(x)=Σ_k g_k(x)K_k is applied through grouped pointwise convolution. This identity does the work of replacing discrete expert routing with a differentiable, hardware-friendly kernel-averaging operation, and it motivates the load-balancing auxiliary loss th","core_discovery":"At the center is Theorem 4.1: for linear experts E_k(x)=xW_k^T and routing weights g_k(x) summing to one, MoE(x)=Σ_k g_k(x)E_k(x)=Conv1×1(x; K~(x)) with K~(x)=Σ_k g_k(x)W_k. Routing is kernel averaging. cMoLLM instantiates this by giving each of N end-to-end streams its own 1×1 kernel, computing soft mixture weights from the input, and applying the averaged kernel via grouped pointwise convolution; a load-balancing loss keeps the streams from collapsing. The paper's horizontal scaling law follows from the FLOP count: N stream convolutions plus one down-projection cost about (N+1)dd_ff per token, versus about 2dd_ff for a dense FFN, so for bounded N the effective capacity grows roughly linear","pith_inferences":["Not proven in the paper: the single-layer identity is exact for linear experts, and Corollary 4.2 extends it to two-layer nonlinear experts via a data-dependent mask, but the full multi-layer pipeline in Algorithm 1 and Eq. (12) routes over already-mixed stream outputs and inserts Transformer layers between convolutions; the theorem does not literally cover that architecture, so its validity at th","The 'matched compute' comparison is generous to cMoLLM: N=8 streams cost about 4.5 times the dense FFN FLOPs, so a dense model given the same compute budget is a stronger baseline than the one reported.","If the identity holds at scale, the most direct testable extension is to measure whether the per-stream perplexity gain persists when a dense baseline is given matching FLOPs and parameters; another is to see whether the advantage concentrates on token clusters that correlate with the learned routing patterns, as the toy cluster model suggests.","The equivalence also suggests a cross-fertilization: convolution-optimized hardware and kernel-fusion libraries could accelerate MoE-style LLM training directly, since the mixture reduces to one grouped pointwise convolution."],"forward_implications":["Soft MoE routing and dynamic convolution become the same operation, so any soft mixture layer can be implemented as a single input-conditioned convolution without Top-K truncation.","For bounded N, adding streams grows effective capacity roughly linearly in N while per-token FLOPs stay within a constant factor of the dense baseline—a horizontal scaling law alongside the usual vertical one.","The design eliminates virtual tokens and auxiliary prediction branches, which the paper argues removes the compute overhead and training instability those mechanisms introduce.","At every model size tested (85M, 350M, 760M parameters), the multi-stream model improves loss, perplexity, and downstream accuracy over the dense baseline, suggesting the benefit does not fade as models grow.","The paper itself flags that its experiments stop at 760M parameters and that validation at 7B+ is still needed, so the scaling claim is currently supported only at small scale."],"fun_headline_variants":["MoE = dynamic convolutions, enabling cheaper LLM scaling","cMoLLM: linear capacity growth without per-token compute explosion","Routing as kernel averaging: new scaling law for LLMs","Dynamic convolution reformulation makes MoE scaling efficient","Mixture-of-LLMs via dynamic convs: scale without compute blowup"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the per-layer identity—a weighted sum of linear experts equals one input-dependent 1×1 convolution—still holds for the full multi-stream cMoLLM pipeline as implemented, and that comparing N=8 streams to a dense model is a fair 'matched compute' comparison.","fun_headline_variants_meta":{"raw":{"variants":["MoE = dynamic convolutions, enabling cheaper LLM scaling","cMoLLM: linear capacity growth without per-token compute explosion","Routing as kernel averaging: new scaling law for LLMs","Dynamic convolution reformulation makes MoE scaling efficient","Mixture-of-LLMs via dynamic convs: scale without compute blowup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00037,"raw_usage":{"total_tokens":1850,"prompt_tokens":809,"completion_tokens":1041,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":953}},"tokens_in":553,"tokens_out":1041,"duration_ms":10331,"temperature":1.0,"reasoning_tokens":953,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:06:03.141108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train cMoLLM with N=8 streams and a dense baseline whose per-token FLOPs are matched to cMoLLM's actual cost (about 4.5 times the dense FFN cost at N=8) on the same data; if the dense model matches or beats cMoLLM's perplexity, the horizontal scaling law is an artifact of an unequal compute comparison rather than a genuine capacity gain.","supporting_citations":[],"review_version":1}