{"id":"713545ff-a190-47a5-a207-e063c3610548","arxiv_id":"2608.13426","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Selecting the highest-norm columns of the activation matrix inside each matrix product preserves quality at reduced compute, with attention far more reducible than MLP layers.","lead":"This paper proposes a training-free way to speed up large language model inference by computing only the most informative parts of each matrix multiplication, selected from the current input. If it holds up, it offers a controllable accuracy versus speed trade-off and suggests attention layers contain more removable redundancy than feed-forward layers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Minimax theorem assumes B is unobservable, but B is available in every RMM application; the unmeasured B-side energy term is what actually determines whether RMM works.","rationale":"The reader's conditional verdict already centers on the B-side alignment assumption. My stress-test confirms that this is the load-bearing gap: the minimax optimality theorem is valid only for a surrogate problem in which B is adversarial and unobserved, but in every RMM deployment B is actually available before the reduced product is computed. This makes the theorem's information-asymmetry premise false and leaves the method's effectiveness dependent on an empirical alignment between B's row energy and A's high-norm columns that the paper never measures. The Appendix B.4 MLP-Up result is the clearest internal evidence that A-side retained energy does not predict accuracy: 82.24% retained energy with a 16.32-point drop, versus 89.69% retained energy with a 3.52-point drop for attention. This directly contradicts the implicit assumption that high A-side energy concentration is sufficient for low error. A concrete epsilon_B measurement would settle whether the missing B-side term explains the discrepancy. Since the reader already assigned a conditional verdict on essentially this basis, my read does not change the verdict; it sharpens the reason and adds the specific observation that the theorem's one-sided framing is not merely conservative but factually inapplicable to the paper's own targets.","tokens_in":26235,"tokens_out":5596,"duration_ms":62862,"concrete_test":"On LLaMA 3.1 8B, for each component in Table 16 (Q projection, QKV, Attention, MLP Up/Gate/Down, Whole MLP) at RR=0.7 on ARC-Easy, compute the discarded B-side energy ratio epsilon_B(rho)=||B_{bar I,:}||_F^2 / ||B||_F^2 using the same index set I selected by RMM, and report it beside the accuracy drops. If MLP-Up has markedly larger epsilon_B than attention-side or Down, the alignment assumption is the cause; if epsilon_B is small everywhere, then the bound is too loose and the minimax framing still does not validate the method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E.1's Theorem 1 is correct for the stated surrogate, but the one-sided information asymmetry it assumes does not hold in RMM's own applications. In QK^T, the key matrix K is already computed before selection; in P V, the attention matrix P is available; in MLP projections, the weight matrix W is fixed and in memory. Selection before the GEMM does not make B unobservable. RMM deliberately ignores B, so TopK by ||A_{:,j}|| is not optimal among rules that could use B. More importantly, the practical claim needs the unmeasured assumption that B's row energy is small on the discarded columns. Eq. (17) bounds relative error by sqrt(epsilon_A epsilon_B), but epsilon_B is never reported. Appendix B.4 shows the A-side diagnostic fails: reducing MLP-Up at RR=0.7 retains 82.24% of activation energy yet drops 16.32 accuracy points, while attention-side reduction retains 89.69% and drops only 3.52. Without the B-side term, the central claim that RMM preserves model behavior across Transformer matrix products is not established; it holds empirically for some attention-side products and not for others.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reduced Matrix Multiplication (RMM), a training-free, input-adaptive method that reduces the contraction dimension of matrix products in Transformer inference. For a product Y = AB, RMM selects a subset I of dimension indices by the L2 column norms of A and computes A_{:,I} B_{I,:}, with the retention ratio rho controlling how many indices are kept. The authors apply RMM to attention-internal products (QK^T, PV) and to MLP/linear projections, and evaluate it on models from 1B to 70B across discriminative QA, summarization, long-context, and vision-language benchmarks. They report that attention-side computations are substantially more reducible than MLP components, that larger models tolerate more aggressive reduction, and that custom Triton kernels yield wall-clock speedups. A minimax optimality theorem (Appendix E.1) and an approximation error bound (Proposition 1, Corollary 1) are provided as theoretical support.","tokens_in":26386,"tokens_out":3796,"duration_ms":41405,"significance":"If the central claims held in their full generality, RMM would be a useful, simple addition to the inference-time pruning toolbox: it is training-free, requires no weight modification, has a single user-controlled retention ratio, and is evaluated across a broad sweep of models and tasks. The paper has notable strengths: it ships a large empirical matrix (Tables 3, 9, 10, 16), includes controlled ablations that separate dynamic vs. static and activation-aware vs. random selection, reports compute-normalized component analysis (Table 12), and explicitly acknowledges several limitations, including the need for heterogeneous retention policies and the fact that RMM does not reduce stored weights. The theoretical statements are transparently derived and are correct for the surrogate objective they address. However, the scope of the empirical support is narrower than the abstract claims, and the theoretical justification rests on an information-asymmetry premise that does not hold in the method's own applications. These issues are load-bearing for the paper's headline claim that RMM reduces 'Transformer matrix products' generally.","major_comments":[{"comment":"The abstract and Section 1 state that RMM reduces 'the high-dimensional matrix products in attention and feed-forward layers' and 'Transformer matrix products' generally, but Section 4 says the main-paper results apply reduction to attention-side matrix multiplications only. This is not a minor scope remark: Appendix Table 16 shows that applying RMM to the whole MLP block at RR=0.7 and RR=0.5 drops average QA accuracy from 69.79 to 55.93 and 40.28 respectively, far worse than the attention-side reductions reported in the main text. The general claim is therefore unsupported, and the paper should either restrict its headline claims to attention-side reductions or provide component-specific retention policies and report their aggregate behavior.","section":"Abstract and Section 4"},{"comment":"Theorem 1 justifies TopK selection by column norm as minimax optimal under an 'information asymmetry' in which B is not observed at selection time. But in every application of RMM in this paper, B is available before the GEMM is executed: in QK^T the key matrix K is already computed, in PV the attention matrix P is available, and in MLP projections the weight matrix W is fixed in memory. Thus the one-sided premise does not hold for RMM's own applications, and a B-aware selection rule could in principle do better. The practical claim therefore rests on the unmeasured assumption that the discarded rows of B carry little energy. Equation (17) bounds the relative error by sqrt(epsilon_A epsilon_B), but epsilon_B is never reported anywhere in the paper. Appendix B.4 (Table 12) shows the problem: reducing MLP-Up at RR=0.7 retains 82.24% of activation energy yet loses 16.32 accuracy points, while attention-side reduction retains 89.69% and loses only 3.52 points, so the A-side energy diagnostic alone cannot predict whether RMM preserves model behavior. I ask the authors to report epsilon_B for the tested components or otherwise provide empirical evidence about B-side energy concentration, and to compare RMM against a B-aware selection baseline to make the practical claim credible.","section":"Section 3.2 and Appendix E.1"},{"comment":"The comparison against SparseGPT, Wanda, SliceGPT, and magnitude pruning under a 'fixed retention ratio' of RR=0.5 does not match computation or memory budgets across methods. The static baselines remove weights permanently at a given sparsity, while RMM removes activation columns dynamically and retains all weights; the same nominal '0.5' therefore corresponds to different FLOPs, memory footprints, and architectural structures. The reported advantages over baselines may reflect the type of pruning (activation-adaptive vs. weight-static) rather than RMM as such, but the current exposition does not state what is matched. The authors should specify the matched quantity (e.g., measured FLOPs, measured latency, or parameter count) and, if possible, report runtime for each method under identical backends.","section":"Section 5.1, Tables 1 and 2"},{"comment":"The end-to-end wall-clock comparison in Table 8 uses a HuggingFace SDPA dense baseline against a custom Triton RMM implementation, so the reported speedup conflates algorithmic reduction with kernel engineering. The kernel-level GEMM numbers in Table 7 are more controlled, but the end-to-end claim should use the same implementation backend for both the dense and reduced variants, or at least include a dense Triton baseline. Otherwise the 1.05x speedup at sequence length 1024 and the larger numbers at longer sequences are not attributable solely to RMM's contraction-axis reduction.","section":"Section 6.3 and Table 8"}],"minor_comments":[{"comment":"The H2O rows list identical ROUGE and BERTScore values at RR=0.8 and RR=0.5; the paper explains this by H2O's fixed token budget, but the footnote should be placed directly in the table caption so readers do not misinterpret the duplicated numbers as a copy-paste error.","section":"Table 2 and Table 10"},{"comment":"In the Qwen3-1 7B block of Table 10, the H2O row reports ROUGE-2 = 9.10 with ROUGE-1 = 4.20, which is odd because ROUGE-2 should generally not exceed ROUGE-1; please verify these numbers and correct or annotate the table.","section":"Table 10, Qwen3-1 7B H2O row"},{"comment":"The 'Retained energy' column reports only A-side activation energy; since the text in Section B.4 uses this diagnostic to compare MLP-Up and MLP-Down, the column label should clarify that it is the retained activation energy of the input operand, not the energy of the discarded B rows.","section":"Table 12"},{"comment":"The phrase 'MLP reduction should be applied selectively across the Up, Gate, and Down projections' is a useful recommendation, but the main-text conclusion in Section 6.2 should state the same nuance rather than saying flatly that 'attention-side computations are substantially more reducible than MLP components', since MLP-Down at RR=0.7 loses only 3.51 points.","section":"Appendix B.4"},{"comment":"The speculative theoretical perspective is clearly labeled as heuristic, which is good; however, the term 'local tangent space' is used without a precise definition and could be simplified or removed, as it does not connect to any formal statement used elsewhere in the paper.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core for attention-side reduction and a clear set of controlled ablations, but the current framing substantially overclaims the scope of the method. The theoretical guarantee in Appendix E.1 does not apply to the actual applications because B is observable at selection time, and the missing epsilon_B measurement is the key piece of evidence needed to support the method's practical rationale. I believe these issues are fixable with a revised scope statement, additional B-side diagnostics, and a more careful baseline comparison, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RMM is a simple, training-free idea: at each matrix product, keep the top-k columns of the left operand by L2 norm and multiply only those. The genuinely new pieces are applying this to attention-internal products QK^T and PV, not just projections, and proving that TopK by column norm is minimax optimal for the activation-only selection problem. That proof is correct as stated, and the attention-vs-MLP asymmetry is a real finding worth taking seriously.\n\nCredit where it is due: the attention-side results are believable and reasonably thorough, with multiple models, retention ratios, tasks, and long-context settings. The 70B scaling trend is plausible, and the latency numbers at longer sequence lengths are consistent with the complexity analysis. The paper is honest about some of its limits, including the LLM-side caution and the dependency of the trade-off on model, task, and component.\n\nThe soft spots are not fatal, but they matter. First, the abstract and title overreach. The paper says it reduces Transformer matrix products generally, but the main scaling, generation, and long-context experiments apply RMM only to attention-side products. The MLP reductions collapse much faster, as Appendix B.4 shows clearly: reducing the Up projection at RR=0.7 retains 82.24% of activation energy yet drops accuracy by 16.32 points, while attention-side reduction retains 89.69% and drops only 3.52. That undercuts the general framing. Second, the theoretical guarantee is narrower than the way the paper uses it. The minimax theorem assumes B is unobservable at selection time, but in every RMM application B is already in memory before the GEMM. More importantly, the practical error bound depends on the product of discarded A-energy and discarded B-energy, and the paper never reports epsilon_B. The B.4 example shows that A-side energy alone can look fine while the model degrades sharply, so the theory justifies the selection rule as minimax for an A-only surrogate, not as a guarantee of model preservation. Third, the comparisons with SparseGPT, Wanda, SliceGPT, and magnitude pruning are not budget-matched in wall-clock or MAC terms; they are weight-pruning methods with different system costs, so the comparison flatters RMM. Fourth, the latency benchmark compares a custom Triton implementation against a generic SDPA baseline, which conflates the method with the kernel quality.\n\nNone of this kills the paper. The attention-side claims are likely reproducible, and the mechanistic asymmetry is a useful guide for future efficient-inference designs. But the paper should be revised to scope claims to attention-side reduction, report B-side residuals where possible, and add cost-matched baselines. Code would help resolve the remaining uncertainty.\n\nThis deserves a serious referee. It is not a desk reject, and a competent referee can push on the scoping and baselines without throwing out the contribution. I would cite the attention-side result, not the general claim.","headline":"A solid, clearly-written paper on an attention-side reduction trick that is over-scoped to all Transformer matrix products, with a correct but narrow theoretical guarantee and a few comparison/latency issues that revision can fix.","tokens_in":26963,"tokens_out":1825,"would_cite":true,"duration_ms":22292,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing each Transformer matrix product with a slice along the input's highest-energy columns preserves model behavior under moderate retention, with attention-side products far more reducible than MLPs.","keywords":["reduced matrix multiplication","input-adaptive inference","activation-aware dimension selection","Transformer inference","attention redundancy","MLP sensitivity","retention ratio","training-free acceleration"],"falsifier":"Measure, for every targeted matrix product at $\\rho=0.7$ in a small model, the product of discarded-energy fractions $\\sqrt{\\epsilon_A(\\rho)\\epsilon_B(\\rho)}$ and compare it with the actual output perturbation or downstream accuracy drop; if a component with high retained activation energy (for example, the MLP Up projection, which retains 82.24\\% of activation energy) still shows a much larger drop than the bound predicts, the alignment assumption is false for that component and the general claim 'moderate reduction preserves behavior' fails there.","tokens_in":25970,"feed_emoji":"✂️","tokens_out":15594,"duration_ms":129252,"temperature":0.7,"pith_summary":"Reduced Matrix Multiplication (RMM) is a training-free scheme for cutting Transformer inference cost: for every matrix product $Y=AB$ in attention and MLP layers, it keeps only the $\\lceil\\rho d\\rceil$ columns of $A$ with the largest L2 norms and computes $A_{:,I}B_{I,:}$. The paper claims that under moderate retention ratios ($\\rho=0.7$ to $0.9$) this preserves model behavior across discriminative QA, summarization, long-context, and vision-language tasks in models spanning 1B to 70B parameters, with degradation that grows smoothly as $\\rho$ falls. A central empirical finding is the structural asymmetry inside Transformers: attention-side products such as $QK^\\top$ and $PV$ tolerate aggressive reduction, while MLP projections---especially the Up projection---degrade sharply. If the claim holds, RMM gives practitioners a predictable accuracy--efficiency dial for inference without retraining or weight modification.","feed_headline":"Matrix-product slicing to top-k columns keeps LLM answers intact","feed_subtitle":"Choosing the highest-energy columns before each multiply preserves model behavior, especially in attention.","key_machinery":"The object that carries the argument is the reduced product $\\mathrm{RMM}_\\rho(A,B)=A_{:,I}B_{I,:}$ with $I=\\mathrm{TopK}(\\{\\|A_{:,j}\\|_2\\}_{j=1}^d,\\lceil\\rho d\\rceil)$. The analysis decomposes $AB=\\sum_{j=1}^d A_{:,j}B_{j,:}$, so discarding dimension $j$ costs $\\|A_{:,j}\\|_2\\|B_{j,:}\\|_2$. Because $B$ is unobserved at selection time, Theorem 1 treats selection as a minimax game in which an adversary may place all of $B$'s row energy on the discarded columns; TopK minimizes the worst-case residual $\\|A_{:,\\bar I}\\|_F$. The resulting bound, relative error $\\le\\sqrt{\\epsilon_A(\\rho)\\epsilon_B(\\rho)}$, is what lets the paper link activation-energy concentration to output fidelity and justify the method without seeing the weights or keys/values in advance.","core_discovery":"The central discovery is that the contraction axis of a Transformer matrix product carries input-dependent redundancy, and that a deterministic, activation-aware slice exploits it. Formally, for $Y=AB$, RMM selects $I=\\mathrm{TopK}(\\{\\|A_{:,j}\\|_2\\}_{j=1}^d,\\lceil\\rho d\\rceil)$ and returns $A_{:,I}B_{I,:}$. The paper proves (Theorem 1) that among all selection rules that see only $A$, this choice is minimax optimal against a worst-case $B$ whose row norms concentrate on the discarded columns, and bounds the relative error by $\\sqrt{\\epsilon_A(\\rho)\\epsilon_B(\\rho)}$, the product of the discarded-energy fractions of $A$ and $B$. Empirically, at $\\rho=0.8$ the reduced model stays close to the unmodified model across the evaluated benchmarks, and attention-side components remain stable even at $\\rho=0.5$; MLP components, and the Up projection in particular, do not. The paper interprets this as evidence that redundancy is not uniform across Transformer components and that component-aware retention policies are needed.","pith_inferences":["The authors leave implicit that the practical success of RMM should be predictable from activation-energy concentration alone, so measuring $\\epsilon_A(\\rho)$ at each layer could serve as a cheap, label-free proxy for how aggressively that layer can be reduced.","The method is a special case of a more general principle---contract only the dimensions where the input's energy is concentrated---so the same selection rule could be applied to KV-cache compression or to other bilinear operations, with per-component retention ratios rather than one global $\\rho$.","The minimax theorem concerns a worst-case adversary; real models are not adversarial. A testable extension is to measure the actual discarded row-energy fraction $\\epsilon_B(\\rho)$ of weights, keys, and values and check whether it stays small; if it does, a tighter data-dependent error bound than the minimax one would hold.","Because RMM recomputes indices every decoding step, it effectively performs a local, input-dependent projection. This suggests that training objectives that encourage energy concentration in activations could make inference-time reduction substantially more aggressive than post-hoc selection alone."],"forward_implications":["At $\\rho=0.8$, RMM stays within a few points of the full model on commonsense QA, MMLU, GSM8K, and HumanEval across the evaluated 7B to 70B models, and larger models generally tolerate more aggressive reduction.","Attention-side matrix products ($QK^\\top$, $PV$, and the Q projection) degrade only mildly even at $\\rho=0.5$, while MLP reduction---especially the Up projection---causes sharp accuracy drops, so safe deployment should apply component-specific retention ratios.","In autoregressive generation and long-context RULER tasks, degradation is smooth rather than abrupt, with no systematic increase in error up to 30K tokens.","The same principle transfers to vision-language models: at $\\rho=0.8$ RMM matches the dense baseline on POPE and BLINK subsets, and at $\\rho=0.5$ it clearly outperforms static and random pruning.","Wall-clock gains grow with sequence length: around $1.05\\times$ at 1K tokens, $1.27\\times$ at 2K, and $1.40\\times$ at 4K on the 8B model in single-GPU latency tests, and RMM avoids out-of-memory at 4K on the 70B model."],"supporting_citations":[{"why":"Defines the randomized matrix-product approximation paradigm whose importance-sampling formulation RMM replaces with deterministic, activation-aware selection.","marker":"Drineas et al., 2006"},{"why":"Provides the Transformer architecture whose attention and MLP matrix products RMM targets, establishing the $QK^\\top$, $PV$, and projection structure.","marker":"Vaswani et al., 2017"},{"why":"Wanda, a static pruning baseline RMM is compared against under matched retention, and the source of the latency evaluation protocol.","marker":"Sun et al., 2024"},{"why":"SparseGPT, a one-shot static pruning baseline that RMM outperforms on discriminative QA at RR=0.5.","marker":"Frantar and Alistarh, 2023"},{"why":"SliceGPT, a structural pruning baseline used to show that fixed, non-adaptive reduction degrades more than input-adaptive selection.","marker":"Ashkboos et al., 2024"},{"why":"TEAL, the closest training-free activation-sparsity method; Appendix B.3 compares RMM against it to show the matrix-product formulation extends reduction to $QK^\\top$ and $PV$.","marker":"Liu et al., 2025"},{"why":"H2O, the dynamic KV-cache baseline RMM outperforms in abstractive summarization.","marker":"Zhang et al., 2023"},{"why":"Supplies the LLaMA 3.1/3.2 model family (1B, 3B, 8B, 70B) on which the scaling and component-wise experiments are run.","marker":"Grattafiori et al., 2024"},{"why":"Supplies the Qwen2.5-VL-7B model on which the vision-language generalization experiments are run.","marker":"Bai et al., 2025"},{"why":"RULER, the long-context benchmark used to show RMM does not systematically degrade as context grows to 30K tokens.","marker":"Hsieh et al., 2024"}],"fun_headline_variants":["Top-k column picks cut LLM multiply cost, keep answers","Slicing contraction axes: cheap LLM inference, same outputs","Activation-aware column slicing trims LLM matrix work","Input-adaptive reduction: skip redundant multiply columns","Attention tolerates column drops; MLP does not"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical claim rests on the assumption that the information in the unobserved operand $B$ lines up with the high-norm columns of the input $A$, so that the energy discarded with the low-norm columns is small; the paper proves only that picking the largest columns is the best defence against a worst-case $B$, not that real $B$ behaves this way.","fun_headline_variants_meta":{"raw":{"variants":["Top-k column picks cut LLM multiply cost, keep answers","Slicing contraction axes: cheap LLM inference, same outputs","Activation-aware column slicing trims LLM matrix work","Input-adaptive reduction: skip redundant multiply columns","Attention tolerates column drops; MLP does not"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1410,"prompt_tokens":985,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":601,"tokens_out":425,"duration_ms":4326,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:14:06.283738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, for every targeted matrix product at $\\rho=0.7$ in a small model, the product of discarded-energy fractions $\\sqrt{\\epsilon_A(\\rho)\\epsilon_B(\\rho)}$ and compare it with the actual output perturbation or downstream accuracy drop; if a component with high retained activation energy (for example, the MLP Up projection, which retains 82.24\\% of activation energy) still shows a much larger drop than the bound predicts, the alignment assumption is false for that component and the general claim 'moderate reduction preserves behavior' fails there.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the randomized matrix-product approximation paradigm whose importance-sampling formulation RMM replaces with deterministic, activation-aware selection."}],"review_version":1}