{"id":"baea16a9-354c-4266-af17-0e639d2c0b19","arxiv_id":"2412.11834","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hybrid SSD-attention-MoE architecture, Cheems, is reported to outperform LLaMA3, Mamba2, and Jamba on matched-size language-model evaluations, with a proposed RoPE-for-SSD proof and a new dynamic-mask attention.","lead":"Cheems, a new foundation-model architecture, combines state-space duality, rotary position embeddings, a proposed dynamic-mask attention, and a sparse cross-domain mixture of experts. The paper reports perplexity, associative-recall, and downstream-evaluation gains over LLaMA3, Mamba2, and Jamba at small (320M and 1.3B) parameter scales, plus speedups in expert retrieval.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own disclosure that all models were trained before a gradient-accumulation fix undercuts Table 4; downstream benchmarks from those runs cannot bear the claim that Cheems beats LLaMA3/Mamba2/Jamba.","rationale":"The reader's verdict identified the same weakest assumption, and I agree. The paper is internally explicit that the runs used for Table 4 are contaminated by the gradient-accumulation bug; this is a limitation statement that must be weighed as evidence. Since the entire 'competitor' claim rests on those downstream scores, the fair-comparison premise is the most load-bearing element. I do not see a separate concern that would overturn the conditional verdict: the RoPE-for-SSD proof in Appendix A is mathematically standard but correct; the module ablations (Tables 1-3, Figures 6-7) are consistent with the architecture's components being plausible; and the code, though incomplete in the appendix, is open-sourced. The mismatch between prose ('attenuate or enhance') and the hard-threshold implementation in Listing 5 is a real internal inconsistency, but it weakens the DMAttn novelty story rather than the Table 4 ordering, so it is secondary. The decisive test is reproduction after the fix with multiple seeds. Therefore the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":21585,"tokens_out":3460,"duration_ms":31860,"concrete_test":"Retrain the four 1.3B architectures (LlaMa3, Mamba2, Jamba, Cheems) with the gradient-accumulation fix applied, using identical code, token order, optimizer settings, and at least three seeds per architecture; then recompute the seven downstream benchmarks and report mean and standard deviation. If Cheems's Table 4 advantage (avg 52.44 vs 51.07 for Jamba) does not replicate outside the seed spread, the central competitor claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('Cheems can be a competitor to popular model architectures', abstract; Table 4 avg 52.44 vs 50.36/49.07/51.07 at 1.3B) depends on the benchmark ordering being produced under fair training conditions. The Table 4 caption explicitly states that all models were trained before a gradient-accumulation error fix in the Transformers library, that pretraining perplexity is therefore not comparable, and that 'if our training results are reproduced in the future, the scores on these verification metrics may rise.' Nevertheless, the downstream numbers in Table 4 and the Figure 8 throughput comparisons come from those same pre-fix runs. Gradient accumulation changes the effective batch size and optimizer noise; there is no evidence that the bug affected the pure-transformer, pure-SSM, hybrid, and CDMoE models identically. Because the reported margins are small (e.g., Cheems-1.3B average 52.44 vs Jamba 51.07; MMLU 39.08 vs 37.43), training-condition asymmetry or single-seed noise could plausibly reverse the ordering. The DMAttn code/description mismatch and the partial appendix listings are secondary; the load-bearing premise is that Table 4's ordering is meaningful, and the authors' own note undercuts exactly that premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid foundation-model architecture called Cheems that combines State Space Duality (SSD) with quadratic causal self-attention, a new Dynamic Mask Attention (DMAttn), and a Cross Domain Mixture of Experts (CDMoE). It also proves that rotary position embedding can be applied to the SSD semiseparable matrix, reports perplexity and speed comparisons for the individual modules, and presents downstream benchmark results at the 320M and 1.3B scales claiming that Cheems outperforms LLaMA3, Mamba2, and Jamba on most evaluated tasks. The central architectural idea is to unify position encoding for hybrid sequence transformations, add a learnable mask to attention, and combine dense and embedding-based experts to improve parameter utilization and retrieval speed.","tokens_in":21727,"tokens_out":4332,"duration_ms":41522,"significance":"If the empirical claims were fully supported, the paper would make a useful contribution: a concrete hybrid architecture that combines the strengths of SSMs and attention, a clean proof that RoPE is compatible with SSD, and an expert-retrieval design that scales to thousands of experts at low activation cost. Credit is due for open-sourcing the code, for presenting a self-contained derivation in Appendix A, and for being transparent about a training bug that affects the reported results. However, the significance is currently limited by the fact that the main comparative claim rests on models trained before a known gradient-accumulation fix, with single runs and no error bars, so the headline ordering in Table 4 is not yet established.","major_comments":[{"comment":"The central claim that Cheems outperforms LLaMA3, Mamba2, and Jamba is load-bearing but is based on runs that the authors themselves state were trained before a gradient-accumulation error fix in the Transformers library. The caption says that pretraining perplexity is therefore not comparable and that reproduced runs 'may rise' on the verification metrics, yet Table 4 and Figure 8 are produced from exactly those pre-fix runs. Gradient accumulation changes effective batch size and optimizer noise, and the paper provides no evidence that the bug affected all four architectures identically. Given that the reported margins are small (e.g., Cheems-1.3B average 52.44 vs Jamba 51.07; MMLU 39.08 vs 37.43), the ordering in Table 4 could plausibly reverse under fair training conditions. This issue must be resolved by retraining the models with the fixed library or by providing a convincing argument that the bug is architecture-symmetric; otherwise the headline comparison is unsupported.","section":"Table 4 caption and Section 4.2"},{"comment":"The mathematical description of DMAttn does not match the provided implementation. Equation (5) defines the dynamic mask as exp(τΔ(concat(V_cache, V) WΔ) A), and Eq. (7) applies it as a multiplicative, continuously attenuating or enhancing factor over the concatenated past and current value states. In Listing 5, however, dt is computed from the current V only (dt = self.dt_proj(rearrange(V, 'bhtp->btd'))), the dynamic mask is thresholded to a binary mask (dynamic_mask = ... < 1.0), and it is then added to the causal mask as -inf rather than multiplied with the attention scores. Because the reported MSE and MQAR results could correspond to either the equations or the code but not clearly to both, the method as presented is under-specified. The authors should align the description and the implementation, or explicitly state which version was evaluated.","section":"Section 3.2, Eqs. (5)-(7), and Appendix B.3, Listing 5"},{"comment":"The abstract claims that DMAttn 'maintains 100% accuracy' and improves by 'more than 150%' on the multi-query associative recall task, but Figure 6 shows only point estimates without error bars or multiple seeds, and no numerical table is provided from which these percentages can be verified. The same single-run limitation applies to Tables 1, 2, and 3 and to Figures 7 and 8. Given that some of the reported differences are small, the absence of variance information makes it difficult to assess whether the observed improvements are significant. Please provide seed-averaged results with confidence intervals, or at least state the number of runs and the observed spread.","section":"Section 4.1 and Figure 6"}],"minor_comments":[{"comment":"The column header 'a t' appears to be a typo; the caption refers to 'at' without explaining what this positional-encoding baseline is, and the table layout makes the comparison hard to parse.","section":"Table 1"},{"comment":"The column header 'learing rate' is misspelled as 'learing rate'; it should be 'learning rate'.","section":"Table 7"},{"comment":"The variable name 'mamba_libray' in the SSD listing is a typo for 'mamba_library'.","section":"Appendix B.4, Listing 6"},{"comment":"The discussion claims that the Doge architecture 'has no significant decrease in most verification metrics' without presenting any corresponding table or figure; either add the supporting results or soften this claim to a qualitative observation.","section":"Section 5 and Figure 9"},{"comment":"The model name is spelled inconsistently as 'LlaMa3' in tables and captions and 'Llama 3' in the references; please unify the spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The authors' candor about the gradient-accumulation bug is commendable, but it directly undercuts the central comparative claim. This is fixable by retraining the compared models under identical fixed conditions, or by removing the downstream comparison and reframing the paper as a proposal plus ablations. If the authors cannot retrain, the manuscript should not claim to beat LLaMA3/Mamba2/Jamba. I would also encourage the authors to check whether the DMAttn implementation and equations describe the same algorithm, since that ambiguity affects the reproducibility of the core contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jingze and Bingheng's paper is a mixed bag. The headline claim—Cheems beats LLaMA3, Mamba2, and Jamba at matched parameter counts—does not survive contact with the paper's own Table 4 caption. That caption says all models were trained before a gradient-accumulation fix, that pre-training perplexity is not comparable, and that 'if our training results are reproduced in the future, the scores on these verification metrics may rise.' Gradient accumulation changes the effective batch size and optimizer noise; the authors offer no evidence the bug affected the three baselines and Cheems identically. With margins of 1-3 points, that is enough to reverse the ordering. The stress-test note is right: this is the load-bearing flaw.\n\nThat said, the paper is not empty. The RoPE-for-SSD proof in Appendix A is correct and cleanly written, even if it is a direct application of the standard RoPE two-dimensional rotation property to C and B—RetNet already did something similar. The DMAttn module shows genuinely strong performance on the multi-query associative recall task (Figure 6), and the CDMoE speed scaling in Figure 7 is a real empirical observation. The code is open-sourced, which is more than many submissions do.\n\nThe soft spots beyond the bug: all experiments are single runs with no error bars, so even the synthetic task and the module ablation tables (1-3) are statistically thin. The prose says DMAttn 'attenuates or enhances' states, but Listing 5 applies a hard threshold (masked_fill with -inf) rather than continuous scaling. That is a discrepancy worth flagging. The appendix code listings are partial and contain undefined variables, so reproduction is not turnkey.\n\nWho should read this: anyone working on hybrid SSD-attention architectures or parameter-efficient MoE. The paper's ideas are plausible and the modular evaluations are suggestive, but the central comparative claim needs a retrain or a strong explanation of why the bug didn't matter. I think a serious editor should send this to review—the architecture is worth a look, and the authors' honesty about the bug is a point in their favor. But I would not cite the Cheems-vs-baselines numbers until they are reproduced on clean runs.","headline":"The architecture has interesting pieces, but the central 'Cheems beats the baselines' claim is undercut by the authors' own disclosure of a training bug, so the paper is conditional at best.","tokens_in":22407,"tokens_out":4081,"would_cite":false,"duration_ms":35656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cheems, a hybrid of state space duality, dynamic mask attention, and cross-domain mixture of experts, is claimed to outperform LLaMA3, Mamba2, and Jamba on most evaluated benchmarks at 320M and 1.3B scale.","keywords":["state space duality","rotary position embedding","dynamic mask attention","cross domain mixture of experts","hybrid foundation model architecture","multi-query associative recall","language modeling","mixture of experts"],"falsifier":"Rerun the 320M and 1.3B training runs for Cheems, LLaMA3, Mamba2, and Jamba after the gradient-accumulation fix, using the same data, token counts, and hyperparameters, and compare the seven benchmark averages. If Cheems's margin over Mamba2 and Jamba shrinks or reverses, the central competitive claim is settled.","tokens_in":21218,"feed_emoji":"⚡","tokens_out":14002,"duration_ms":110735,"temperature":0.7,"pith_summary":"The paper sets out to show that a foundation model gets both more efficient and more effective when each of its two halves—sequence transformation and state transformation—is a combination rather than a single choice. It proposes three matrix-level components: rotary position embeddings proved to work inside state space duality, dynamic mask attention that lets causal attention selectively filter past states, and cross-domain mixture of experts that keeps retrieval fast when the expert count reaches thousands. Assembled into the Cheems architecture, these components are claimed to beat LLaMA3, Mamba2, and Jamba on most of seven downstream benchmarks at the 320M and 1.3B scales while training faster than LLaMA3 and Jamba. If the claim holds, a hybrid design can get state-space efficiency and attention effectiveness in one architecture rather than trading one for the other.","feed_headline":"Hybrid Cheems tops LLaMA3, Mamba2, Jamba at same size","feed_subtitle":"Combining rotary SSD, dynamic-mask attention, and cross-domain MoE lifts the 1.3B average to 52.44.","key_machinery":"The central mechanism is the semiseparable matrix identity $M = L \\circ (CB^\\top)$ that rewrites state space duality as causal linear attention, together with the proof that rotating $C$ and $B$ by position-dependent angles preserves relative position information, so $\\langle f_C(x_m,m), f_B(x_n,n)\\rangle = g(x_m,x_n,m-n)$. Dynamic mask attention adds a zero-order-hold mask, a mask whose value is held until the next non-continuous state, computed as $\\exp(A\\,\\tau_\\Delta([V_{\\text{cache}},V]W_\\Delta))$ on the attention score, letting the value state itself decide which past keys are attenuated or enhanced. Cross-domain mixture of experts projects each token against product keys, retrieves the top-$k$ expert rows from embedding tables, and adds a shared dense cross-domain path, so retrieval cost stays nearly flat as the expert count grows. The Cheems stack places one dynamic-mask attention layer after every seven SSD layers, with a CDMoE state transformation between modules.","core_discovery":"On the paper's own terms, the discovery is that sequence transformation and state transformation can be unified through compatible matrix algorithms: RoPE applies to the $C$ and $B$ matrices of SSD so the semiseparable product $L\\circ(CB^\\top)$ and the attention product $L\\circ(QK^\\top)$ carry the same relative position information; dynamic mask attention gives quadratic causal attention a trainable, value-derived mask with the selectivity of a gated state space model; and cross-domain mixture of experts combines dense cross-domain linear layers with sparse product-key embedding retrieval so expert count can scale to thousands without linear retrieval slowdown. The paper reports that Cheems-1.3B averages 52.44 on seven downstream tasks versus 50.36 for LLaMA3, 49.07 for Mamba2, and 51.07 for Jamba, and Cheems-320M averages 45.22 versus 43.99, 43.07, and 44.31 respectively, under matched parameter counts. It also reports that Cheems trains faster than LLaMA3 and Jamba while remaining close to Mamba2 in throughput.","pith_inferences":["Editorial extension: the architecture is only demonstrated at 320M and 1.3B, so the paper leaves open whether CDMoE's flat retrieval cost keeps its advantage at larger widths and longer contexts, where the reported efficiency gap suggests it might.","Editorial extension: the Discussion's Doge variant—dynamic-mask attention with repeated CDMoE and no SSD—retains most benchmark scores, implying the SSD branch may be partly redundant; training Doge and Cheems under identical conditions would isolate SSD's contribution.","Editorial extension: DMAttn's 100% accuracy on multi-query associative recall is shown only up to 2048-token sequences and $d_{\\text{model}}=32$; longer sequences or smaller heads would test whether the mechanism generalizes beyond the reported range."],"forward_implications":["At matched parameter counts, a hybrid SSD-attention-MoE stack can match or beat pure transformer, pure SSM, and transformer-SSM hybrid baselines on the evaluated downstream tasks, so hybridity is not just an efficiency compromise.","Rotary position embedding can serve as the single position-encoding scheme for both attention and SSD branches, which simplifies hybrid models and enables linear-time generation with relative positions.","Dynamic mask attention gives transformer-style attention a selectivity mechanism that preserves associative recall at sequence lengths where both quadratic causal attention and SSD degrade.","Cross-domain mixture of experts keeps expert retrieval fast beyond 1024 experts, so the number of stored experts can grow without the linear retrieval slowdown seen in routing MoE.","The same architecture that scores higher on average than the three baselines also trains and evaluates faster than LLaMA3 and Jamba, meaning efficiency and effectiveness are obtained together rather than traded off."],"supporting_citations":[{"why":"Supplies the state space duality (SSD) matrix formulation and the Mamba2 baseline compared in Table 4.","marker":"Dao and Gu 2024"},{"why":"Introduces the Mamba selective state space model whose position handling the paper replaces with RoPE, and whose SSD component is hybridized.","marker":"Gu and Dao 2023"},{"why":"Defines rotary position embedding, the position encoding whose applicability to SSD the paper proves.","marker":"Su et al. 2021"},{"why":"Describes parameter-efficient expert retrieval (PEER) that CDMoE adapts and that serves as a speed baseline.","marker":"He 2024"},{"why":"Defines shared expert isolation MoE, the state transformation baseline SEIMoE used in Tables 3 and 4.","marker":"Dai et al. 2024"},{"why":"Defines the multi-query associative recall task that dynamic mask attention is tested on.","marker":"Arora et al. 2024"},{"why":"Provides the SmolLM-Corpus training data used for all four architectures in Table 4.","marker":"Ben Allal et al. 2024"},{"why":"Defines the LLaMA3 architecture used as a matched-size baseline.","marker":"Grattafiori et al. 2024"},{"why":"Defines the Jamba hybrid transformer-Mamba architecture used as a matched-size baseline.","marker":"Lieber et al. 2024"},{"why":"Provides the Transformers training framework used for all runs, and the gradient-accumulation caveat that qualifies the Table 4 results.","marker":"Wolf et al. 2020"}],"fun_headline_variants":["Cheems unifies sequence and state transforms to beat LLaMA3, Mamba2, Jamba","Cheems: unifying attention and state spaces for efficient foundation models","RoPE works with SSD, dynamic masks, and cross-domain MoE for 1.3B win","Hybrid attention-state model tops LLaMA3 by 2 points at 1.3B scale","Matrix fusion: RoPE, dynamic mask, and cross-domain MoE push Cheems ahead"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four Table 4 models were trained under conditions fair enough that their downstream score differences reflect the architectures; the paper itself says the runs predate a gradient-accumulation error fix and that perplexity is not comparable, yet the downstream claims come from those same runs.","fun_headline_variants_meta":{"raw":{"variants":["Cheems unifies sequence and state transforms to beat LLaMA3, Mamba2, Jamba","Cheems: unifying attention and state spaces for efficient foundation models","RoPE works with SSD, dynamic masks, and cross-domain MoE for 1.3B win","Hybrid attention-state model tops LLaMA3 by 2 points at 1.3B scale","Matrix fusion: RoPE, dynamic mask, and cross-domain MoE push Cheems ahead"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001412,"raw_usage":{"total_tokens":5716,"prompt_tokens":972,"completion_tokens":4744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":4625}},"tokens_in":588,"tokens_out":4744,"duration_ms":30651,"temperature":1.0,"reasoning_tokens":4625,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:31:51.696298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 320M and 1.3B training runs for Cheems, LLaMA3, Mamba2, and Jamba after the gradient-accumulation fix, using the same data, token counts, and hyperparameters, and compare the seven benchmark averages. If Cheems's margin over Mamba2 and Jamba shrinks or reverses, the central competitive claim is settled.","supporting_citations":[{"cited_title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","cited_arxiv_id":null,"evidence_quote":"Supplies the state space duality (SSD) matrix formulation and the Mamba2 baseline compared in Table 4."},{"cited_title":"Zoology: Measuring and Improving Recall in Efficient Language Models","cited_arxiv_id":null,"evidence_quote":"Defines the multi-query associative recall task that dynamic mask attention is tested on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SmolLM-Corpus training data used for all four architectures in Table 4."},{"cited_title":"Transformers: State-of-the-Art Natural Language Processing","cited_arxiv_id":null,"evidence_quote":"Provides the Transformers training framework used for all runs, and the gradient-accumulation caveat that qualifies the Table 4 results."}],"review_version":1}