{"id":"fce8cee4-6a42-495d-9e76-2e820bad04e4","arxiv_id":"2509.10406","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MuSe approximates softmax attention by K-means clustering queries and keys separately with a covariance-based dipole correction, speeding up 16k-context pretraining by 12.7% with 0.36% loss degradation in a 30M-parameter run.","lead":"MuSe is a fast attention approximation that clusters queries and keys separately and adds a covariance-based dipole correction, cutting 16k-context pretraining step time by about 12% for a 30M-parameter model with a 0.36% loss increase. It matters because it offers a drop-in way to make long-context training cheaper, though the submission abstract claims far larger gains at 64k context and 1B parameters that the manuscript itself never tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.36% loss-degradation claim rests on a single unseeded run; a 0.012-nat difference is within typical seed noise for a 30M model.","rationale":"The reader's weakest_assumption focuses on cluster tightness and backward-pass error, which is a legitimate scaling concern. However, the single most load-bearing weakness for the central full-text claim is statistical: the entire 'matching loss' assertion rests on one unseeded comparison, and the reported difference (0.012 nats) is small enough to be explained by seed noise. The reader did flag single-run comparisons as a separate issue (fourth reason), but did not make it the central load-bearing concern. My check directly tests whether the 0.36% number is meaningful. I keep the verdict CONDITIONAL (via UNCHANGED) because the speedup claim and the method's mechanics are plausible, but the quality-preservation claim needs multi-seed evidence. I also note the metadata abstract's unsupported 64k/1B/Llama claims as a separate submission-integrity issue, but that is not the load-bearing technical assumption for the full-text claim evaluated here.","tokens_in":14286,"tokens_out":10551,"duration_ms":121749,"concrete_test":"Run at least 5 independent pretraining runs of the Pallas Flash 16k and MuSe 16k configurations from Section 7.3/Appendix A with different random seeds, using the same 2B-token budget and hyperparameters; also include 5 runs of the CUDNN Flash 256 control. Report mean and standard deviation of final loss and step time for each condition. If the 95% confidence interval for (loss_MuSe - loss_Pallas) includes 0, the '0.36% degradation' claim is not statistically supported; if it excludes 0 and is ≤0.5%, the claim survives this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4 reports final losses of 3.310 (Pallas Flash 16k) and 3.322 (MuSe 16k), a difference of 0.012 nats, called '0.36% loss degradation'. Section 7.3 and Appendix A describe one training run per condition, with no seeds or variance reported. The central claim that MuSe preserves baseline loss while reducing runtime depends on this 0.012-nat gap being real and not a product of initialization/order noise. For a 29.4M-parameter model trained on 2B tokens, run-to-run variation in final NLL is typically comparable to or larger than 0.012 nats; the manuscript provides no basis to rule this out. Notably, the paper's own forward approximation error is ~0.19 late in training (Table 1), so the close final loss is a surprising result that needs replication before it can support the 'viability' conclusion. This concern is independent of cluster tightness: even if clusters remain tight in future scales, the current evidence cannot statistically distinguish MuSe from exact attention on quality; if clusters degrade at larger scale, the transfer is even less established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MuSe, an approximate softmax attention mechanism based on separate K-means clustering of queries and keys, using monopole (centroid) summaries plus dipole (covariance) corrections, and a hierarchical block decomposition for causal attention. The authors derive the approximation from a cumulant generating function expansion (Sections 3-4), analyze its complexity as O(NCD) for acausal and O(NCD log N) for causal attention, and report microbenchmarks on isolated attention layers plus an end-to-end pretraining experiment of a 29.4M-parameter model at 16k context (Section 7.3), where they claim a 12.2% (later 12.7%) runtime reduction with a 0.36% loss degradation. The abstract additionally claims 64k-context 36% speedups, validation on Llama 3.1-8B/3.2-1B without retraining, and pretraining up to 1B parameters at 64k context, but these results are not present in the body.","tokens_in":14467,"tokens_out":7664,"duration_ms":77876,"significance":"The theoretical framing is a genuine contribution: connecting softmax attention to multipole expansions via the cumulant generating function is elegant and yields a plausible error analysis, and the paper is transparent about the partial dipole specialization and its costs. The microbenchmarks show substantial speedups at long context (20-30x at 64k) and the method has the desirable property of converging to exact attention as cluster count grows. However, the central empirical claim of loss preservation rests on a single unseeded run, and several abstract claims go beyond what the body presents. If the training result were replicated over multiple seeds, the method would be a useful addition to efficient-attention literature.","major_comments":[{"comment":"The abstract claims 'accelerates 64k-context pretraining by 36% while matching baseline loss', validation on Llama 3.1-8B and 3.2-1B without retraining, and pretraining up to 1B parameters at 64k context. None of these experiments appear in the body. The only end-to-end training is a 29.4M model at 16k context (Section 7.3), and 64k appears only in isolated-layer benchmarks (Table 2). These unsupported claims overstate the scope and must be corrected or substantiated.","section":"Abstract vs. Sections 5-7"},{"comment":"The headline '0.36% loss degradation' compares a single MuSe run (3.322 nats) with a single Pallas Flash run (3.310 nats). No seeds or variance are reported for either condition. For a 29.4M model trained on 2B tokens, run-to-run variation is typically comparable to or larger than 0.012 nats, so the difference is not statistically distinguishable from noise. Please provide multiple seeds (at least 3) or another statistical comparison to support the claim that MuSe preserves pretraining quality.","section":"Section 7.3, Table 4"},{"comment":"The paper explicitly lists as a key question whether forward and backward approximation errors are sufficiently small, but only forward relative squared error is measured (Tables 1 and 3). Backward-pass gradient error is never quantified. The end-to-end loss is the only indirect evidence, and it is confounded by the single-run issue. Add a gradient-error measurement on recorded activations, or provide additional training runs, to support the claim that training quality is preserved.","section":"Section 7.2 and Section 7, Q4"}],"minor_comments":[{"comment":"The sentence claiming the error quantity is 'exactly the quantity minimized by K-means' is imprecise: K-means minimizes the trace of within-cluster covariance (sum of squared distances), not the product of query and key covariance traces. Rephrase as 'related to' or 'a quantity that K-means tends to reduce'.","section":"Section 4, after Eq. (7)"},{"comment":"Typo: 'desribed' should be 'described'.","section":"Section 4 title"},{"comment":"The covariance C[k] is computed over the U dimension unweighted by the attention probabilities. Clarify in the comment or pseudocode that this is Cov_j(v,k), not the exponentially tilted Cov_{ij}(v,k) used in Eq. (6).","section":"Algorithm 3"},{"comment":"The K-means++ initialization described as 'sampling points proportional to their squared norm' is equivalent to the first K-means++ sample only for zero-mean data. State whether the queries/keys are mean-centered or whether this is an approximation.","section":"Section 5, Clustering"},{"comment":"The inset showing convergence in the 1-2B token range is important but appears too small to read in the preprint. Ensure legibility in the camera-ready version.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The abstract/body mismatch is severe enough that the authors must either supply the missing 64k/1B/Llama experiments or rewrite the abstract to match the actual contributions. The single-run training experiment is the main technical blocker; with 2-3 additional seeds and a corrected abstract, the paper could become a solid contribution. The theoretical derivation is sound and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the core idea is genuinely clever and the derivation is sound. Clustering queries and keys separately, then using a dipole correction from the cumulant generating function, is a principled approximation to softmax attention. That's new and worth taking seriously. The ablations are honest: the two-stage and dipole terms each pull their weight, and the authors are upfront that the dipole is only partially specialized. The math in Sections 3-4 checks out; the error bound O(Tr(Cov(˜q,˜q) Cov(˜k,˜k))) is a real Taylor expansion, not a heuristic. The paper also makes a good point about why separate clustering of queries and keys is more natural than unified clustering.\n\nThe submission has a serious problem, though: the abstract claims 36% acceleration of 64k-context pretraining, 1B-parameter models, and drop-in validation on Llama 3.1-8B and 3.2-1B without retraining. None of that appears in the full text. The full text is a 30M-parameter, 16k-context run with a 12.2% speedup. That's a huge discrepancy. Either the abstract belongs to a different paper or the authors are overclaiming. The reader's report caught this, and it is correct.\n\nThe central empirical claim—0.36% loss degradation with 12.7% faster step time—rests on a single training run per condition. The 0.012-nat difference between 3.310 and 3.322 is within typical seed noise for a 30M model trained on 2B tokens. The stress-test note is right: there is no statistical basis to distinguish MuSe from exact attention on quality. Also, the speedup is measured against the authors' Pallas Flash Attention, which their own microbenchmarks show is ~31% slower than CUDNN at 8k. That doesn't kill the result, but it means the practical speedup over a strong baseline is smaller than advertised.\n\nThe causal O(NCD log N) claim is only exercised at one block level, so the asymptotic claim is plausible but untested. And the backward-pass approximation error is never measured, which matters for training.\n\nSo: this is a paper with a sound theoretical contribution and a modest, suggestive experiment, wrapped in an abstract that oversells. If the authors fix the abstract and add a few seeded runs, the core idea deserves peer review. As is, I'd engage with the math but treat the empirical claims with caution. For peer review, I'd send it out—the theory is solid and the idea is novel, even though the experiments need strengthening.","headline":"A genuinely novel multipole derivation with a sound error bound, but the abstract overclaims the empirical results and the single-run loss difference is within noise.","tokens_in":15102,"tokens_out":2698,"would_cite":false,"duration_ms":27679,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MuSe clusters queries and keys to approximate softmax attention, cutting 16k-context pretraining time by 12.2% while degrading final loss by only 0.36%.","keywords":["multipole expansion","softmax attention approximation","semantic clustering","long-context pretraining","K-means clustering","dipole correction","transformer efficiency","causal attention"],"falsifier":"Run the same 30M/16k pretraining comparison at 64k context and with a 1B-parameter model, and check whether the runtime gain matches the abstract's claimed 36% and the loss gap stays near 0.36%; if the gap grows substantially, the claim that MuSe preserves quality at scale fails. Independently, instrument the backward pass to measure the relative error of gradients; a large gradient error would show that training degradation is driven by something other than forward approximation fidelity.","tokens_in":13981,"feed_emoji":"⚡","tokens_out":9023,"duration_ms":86165,"temperature":0.7,"pith_summary":"This paper tries to show that the quadratic cost of softmax attention can be approximated during training by clustering queries and keys separately and keeping only summary statistics. The proposed method, MuSe, uses a two-stage attention scheme plus a covariance-based dipole correction to capture directional detail within clusters. In a 30M-parameter model trained on 16k-token book-length sequences, MuSe reduces wall-clock pretraining time by 12.2% while increasing final loss by only 0.36%. Because it is a drop-in replacement for the attention layer, it offers a path to longer-context pretraining without changing the architecture.","feed_headline":"Multipole attention cuts 16k-context pretraining time 12%","feed_subtitle":"Clustering queries and keys separately makes long-context attention nearly linear, adding just 0.36% to final loss.","key_machinery":"The load-bearing identity is the cumulant expansion of attention. Treating each key-value cluster as a probability distribution, the log-normalizer and the value output are derivatives of the joint cumulant generating function of keys and values. Truncating after the first cumulants yields the monopole terms (exponentially tilted cluster centroids) and the dipole term (the key-value covariance contracted with the residual query). The two-stage scheme drops only the residual-residual term q̃·k̃, giving an error of order Tr(Cov(q̃,q̃) Cov(k̃,k̃)), which K-means is designed to minimize.","core_discovery":"The paper's central claim is that softmax attention can be faithfully approximated by a two-stage semantic clustering in the representation spaces of queries and keys. Queries are partitioned into coarse clusters; each coarse query centroid attends to fine-grained key-value clusters to form query-dependent summary centroids; then individual residual queries refine those summaries. A dipole term, computed from the within-cluster covariance of keys and values, adds the first-order directional correction. The approximation error is governed by the product of intra-cluster query and key variances, so increasing the number of clusters drives the method toward exact attention. Empirically, on isol","pith_inferences":["If the error bound transfers, MuSe should scale to 64k context and billion-parameter models, but the paper's own evidence for such scales is only a projection; the abstract's 36% speedup at 64k is not backed by an experiment in the text.","The backward pass uses the same approximations but its error is never measured; training dynamics could be sensitive to gradient error even when forward error is modest.","Because the ideal metric for clustering queries depends on key variance and vice versa, replacing Euclidean K-means with a variance-weighted clustering could reduce error at fixed cluster count.","The separate clustering of queries and keys, justified by attention's invariance to basis changes of the keys, suggests a general principle for designing efficient attention beyond this specific implementation."],"forward_implications":["Training on long sequences becomes nearly linear in context length instead of quadratic, so 64k and longer contexts can be trained at a fraction of the previous cost.","Because MuSe is a drop-in attention replacement, existing pretrained transformers can be adapted to longer contexts without retraining from scratch.","The dipole correction provides a principled way to recover directional information lost by centroid-only methods, improving the accuracy of clustering-based attention.","Users can trade speed against accuracy by choosing the number of clusters; the paper reports a roughly fourfold increase in clusters halves the relative squared error.","The observed 12.2% end-to-end speedup at 16k context suggests larger relative gains at longer contexts, where the exact attention cost dominates more."],"fun_headline_variants":["MuSe: 36% faster 64k pretraining, same loss, no retraining","Semantic query-key clustering: 36% faster attention pretraining","Drop-in attention approximation cuts 64k pretraining time 36%","MuSe makes 64k-context pretraining 36% faster with same loss","36% faster 64k pretraining via semantic query-key clustering"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The error bound assumes that a single iteration of K-means yields clusters tight enough in both query and key space for the first-order dipole truncation to be accurate; the paper's measurements show a relative squared error of about 0.19 at its chosen settings, and the backward-pass error is never measured.","fun_headline_variants_meta":{"raw":{"variants":["MuSe: 36% faster 64k pretraining, same loss, no retraining","Semantic query-key clustering: 36% faster attention pretraining","Drop-in attention approximation cuts 64k pretraining time 36%","MuSe makes 64k-context pretraining 36% faster with same loss","36% faster 64k pretraining via semantic query-key clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001206,"raw_usage":{"total_tokens":4756,"prompt_tokens":646,"completion_tokens":4110,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":390,"completion_tokens_details":{"reasoning_tokens":4008}},"tokens_in":390,"tokens_out":4110,"duration_ms":27993,"temperature":1.0,"reasoning_tokens":4008,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:53:00.837008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 30M/16k pretraining comparison at 64k context and with a 1B-parameter model, and check whether the runtime gain matches the abstract's claimed 36% and the loss gap stays near 0.36%; if the gap grows substantially, the claim that MuSe preserves quality at scale fails. Independently, instrument the backward pass to measure the relative error of gradients; a large gradient error would show that training degradation is driven by something other than forward approximation fidelity.","supporting_citations":[],"review_version":1}