{"id":"a44e34d3-1921-4415-a08b-66045c122cb7","arxiv_id":"2611.26363","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HiSA sparsifies the hidden dimension of RoPE attention by dropping high-frequency pairs at large token distances, improving loss and efficiency in LLMs.","lead":"The paper proposes Hidden Sparse Attention (HiSA), a method that makes attention in large language models sparse in the hidden dimension rather than over tokens, by computing only a partial dot product between queries and keys based on the rotation angle of Rotary Position Embeddings (RoPE).","discovery_kind":"new_method","skeptic_critique":{"audit":{"quote":"A ratio above 1 means that the true score part moves the output more than an uninformative one (content-specific signal); below 1, it moves the output \\emph{less} than an unrelated key would, so the model has shaped its queries and keys to damp it.","status":"reviewer_error","question":null,"consequence":null,"objection_id":"O1","published_as":"withdrawn"},"model":"deepseek/deepseek-v4-flash","headline":"Withdrawn: Pith's reviewer misread the paper.","rationale":"","tokens_in":27934,"tokens_out":846,"duration_ms":9282,"concrete_test":"","verdict_should_be":null,"load_bearing_attack":"The paper's words: “A ratio above 1 means that the true score part moves the output more than an uninformative one (content-specific signal); below 1, it moves the output \\emph{less} than an unrelated key would, so the model has shaped its queries and keys to damp it.”","agreement_with_reader":null},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes HiSA, a static parameter-free sparsity pattern in the hidden dimension of RoPE attention. It keeps query and key projections intact but computes only a partial dot product based on the RoPE rotation angle: high-frequency pairs are dropped when the rotation exceeds a budget (default 90°). The method is evaluated by training 135M and 360M models from scratch on OpenWebText, showing improved validation loss while computing ~30% of the dot products. It also demonstrates benefits in long-context extension, KV-cache reduction, and decoding speed. An interpretability analysis shows that dense models learn to suppress the same score parts that HiSA drops.","tokens_in":28256,"tokens_out":2774,"duration_ms":19518,"significance":"Not published. Pith's referee wrote this assessment before its objections were checked against the paper, and 4 of its 4 major objections did not survive that check.","major_comments":[],"minor_comments":[{"quote":null,"comment":"Figure 1 (teaser) is described but not visible. The caption is clear, but the figure would benefit from a more detailed description in the text for readers who cannot see it.","section":"§1, Figure 1","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":null,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"s^{\\text{\\method{}}}_{ij} = \\frac{1}{\\sqrt{d}} \\sum_{l=0}^{P-1} \\mathbf{1}\\big[\\Delta\\,\\theta_l < \\Phimax\\big]\\, \\big\\langle R_{i\\theta_l}\\pairq{l}_i,\\, R_{j\\theta_l}\\pairk{l}_j \\big\\rangle","comment":"The notation $R_{i\\theta_l}\\pairq{l}_i$ is slightly redundant: the query pair is already rotated by $i\\theta_l$ in RoPE. Consider clarifying that $R_{i\\theta_l}$ is the rotation applied by RoPE to the unrotated pair.","section":"§3.2, Equation (3)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":null,"comment":"The table reports 'Loss by position' with bins 0-512, 512-2K, 2K-4K. The bin boundaries are not aligned with the 4K context length (the last bin goes to 4K but the context is 4K tokens, so positions 0-4095). Clarify whether the bins are inclusive/exclusive and whether the last bin includes position 4095.","section":"§4.1, Table 1","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":false,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"After continued training (\\cref{tab:convert}), the converted model matches the dense one while computing 29\\% of the dot products.","comment":"The conversion experiment shows that a dense model converted to HiSA matches the dense continued training loss (2.836 vs 2.838). However, the dense continued training itself improves from 2.906 to 2.838, which is a larger drop than the HiSA advantage in Table 1. The paper should discuss whether the conversion result is simply due to the continued training budget being sufficient to adapt to the new pattern, and whether the converted model would maintain its advantage with more training.","section":"§4.5, Table 5","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"u_j = a_{ij}\\big(e^{-s^{(l)}_{ij}} - 1\\big)\\,\\mathbf{1}[i - j \\in \\mathcal{B}]","comment":"The derivation of Equation (8) uses $u_j = a_{ij}(e^{-s^{(l)}_{ij}} - 1)\\mathbf{1}[i - j \\in \\mathcal{B}]$. The term $e^{-s^{(l)}_{ij}}$ can be numerically unstable if $s^{(l)}_{ij}$ is large negative. The paper mentions using fp64 for evaluation, but the potential for overflow/underflow should be noted.","section":"§5, Equation (8)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Every finite budget beats dense attention, even $45^\\circ$, which computes under a third of the dot product. $45^\\circ$, $90^\\circ$ and $180^\\circ$ are within 0.012 of each other, about the spread between seeds (below), while a full rotation period ($360^\\circ$, the window of \\citealp{du2026waveprune}) keeps less of the gain.","comment":"The ablation table shows that at 2K context, the 180° budget gives the best validation loss (3.010) while computing 44.4% of the dot product. The paper chooses 90° as default. The justification for 90° over 180° should be stronger, perhaps by showing the trade-off at 4K context where the compute savings are larger.","section":"Appendix C, Table 7","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"After the extension, \\method{} keeps its advantage on books and web text while computing a fifth of the dot product (\\cref{tab:long-exact}).","comment":"The long-context extension uses a RoPE base of $2\\cdot10^4$ and extends to 16K context. The paper reports that HiSA computes 20.2% of the dot product. However, the validation loss on PG19 is 3.170 for HiSA vs 3.205 for dense. The absolute improvement is 0.035, which is smaller than the 0.026 improvement at 4K (2.880 vs 2.906). The paper should discuss whether the advantage diminishes at longer contexts.","section":"§4.3, Table 3","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Part of this comes from our kernel's smaller query tiles; with the same kernel and no pair dropped, Flash-\\method{} is still 13\\% faster.","comment":"The decoding throughput improvement is reported as +42% at 16K and batch 128. However, the paper notes that part of this comes from smaller query tiles in the kernel, not from the sparsity itself. The paper should separate the speedup due to sparsity from the speedup due to kernel optimizations.","section":"§4.4, Table 5","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Both works skip computation and key reads, but neither reduces the KV cache: every key and value stays in memory.","comment":"The related work section mentions concurrent works ATFlash and WavePrune. The paper should also discuss how HiSA differs from these in terms of training vs inference-only, and whether the KV-cache reduction is unique to HiSA.","section":"§2, Related Work","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Open questions such as scaling to bigger models, downstream tasks benchmarking and better kernel design are left for future work.","comment":"The conclusion states 'Open questions such as scaling to bigger models, downstream tasks benchmarking and better kernel design are left for future work.' This is an honest acknowledgment, but the paper would be strengthened by including at least one downstream task evaluation (e.g., perplexity on a standard benchmark like WikiText-103) to validate that the loss improvement translates to better language modeling.","section":"§6, Conclusion","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null}],"open_questions":null,"recommendation":"uncertain"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper asks a clean question: does attention need the full query–key dot product? With RoPE, the dot product decomposes into frequency pairs, and HiSA keeps each pair only while its rotation angle stays below a budget. The sparsity pattern is static—determined by positions and RoPE frequencies alone—so it needs no learned router or runtime selection. That is a genuinely new idea, and the paper develops it thoroughly: exact and block-sparse kernels, training from scratch, long-context extension, KV-cache reduction, and an interpretability analysis showing that dense models learn to suppress the same score parts that HiSA drops.\n\nThe experiments are well-designed: controlled comparisons, multiple seeds, ablations, and the code is released. At 135M and 360M parameters, HiSA reaches a lower validation loss than dense attention while computing 29–34% of the dot products. The gain holds after long-context extension and when converting a pretrained dense model. The FlashAttention-3 implementation trains at dense speed and improves decoding throughput by up to 42% with a third less KV-cache memory.\n\nThe main soft spot is scale. The experiments stop at 360M parameters and 20 tokens per parameter. The authors acknowledge this, but the central claim—that dropping high-frequency pairs at long range is harmless or beneficial—needs confirmation at larger scales where attention patterns may differ. The theoretical motivation (citing Khandelwal et al. 2018 for recurrent LMs) is suggestive but not a proof for modern attention-based LLMs. The backward pass does not yet benefit from the sparsity, which limits training speedups. These are limitations the paper states, not hidden flaws.\n\nThis paper is for researchers working on efficient attention and LLM inference. It deserves a serious referee. The method is novel, the experiments are honest, and the claims are proportional to the evidence shown. I would send it to review.","headline":"HiSA proposes a static, parameter-free sparsity pattern in the hidden dimension of RoPE attention that improves validation loss while computing about a third of the dot product, with careful experiments at 135M and 360M scales.","tokens_in":28558,"tokens_out":471,"duration_ms":6165},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper argues that high-frequency RoPE pairs can be dropped at large token distances without harming—and even improving—language model performance.","keywords":["sparse attention","rotary position embedding","RoPE","hidden dimension sparsity","efficient attention","KV cache","large language models","FlashAttention"],"falsifier":"Train a model with HiSA but with the angle budget set to 360° (full rotation) and compare to dense attention. If the loss improvement disappears, it would confirm that the benefit comes from removing high-frequency pairs at long range, not from some other aspect of the method. Alternatively, train a model where the dropped pairs are replaced by random noise; if the loss is worse than HiSA, it would confirm that the dropped pairs are indeed harmful.","tokens_in":29723,"feed_emoji":"🧠","tokens_out":1297,"duration_ms":10590,"temperature":0.2,"pith_summary":"The paper proposes Hidden Sparse Attention (HiSA), a static, parameter-free sparsity pattern in the hidden dimension of RoPE-based attention. It decomposes the query-key dot product into frequency pairs and keeps each pair only while its rotation stays below a fixed angle budget (default 90°). This means fast pairs see only nearby keys, slow pairs see far. Training LLMs from scratch with HiSA yields lower validation loss than dense attention while computing about a third of the dot products. The method also reduces KV-cache memory and speeds up decoding. An interpretability analysis shows that dense attention itself learns to suppress the same score parts that HiSA drops.","feed_headline":"HiSA: Dropping high-frequency RoPE pairs at long range improves LLM attention","feed_subtitle":"A static, parameter-free sparsity pattern in the hidden dimension computes a third of the dot product with better loss","key_machinery":"The central object is the decomposition of the RoPE-based attention score into a sum over frequency pairs, where each pair's contribution is weighted by a cosine of the rotation angle. HiSA applies an indicator function that keeps a pair only when its rotation is below a fixed angle budget (default 90°). This creates per-pair windows that grow with the pair's wavelength: fast pairs see only nearby keys, slow pairs see far. The sparsity pattern is static and parameter-free, depending only on positions and RoPE frequencies.","core_discovery":"The paper claims that the complete query-key dot product in RoPE-based attention is not necessary: high-frequency (fast) RoPE pairs can be omitted at large token distances without harming performance. The authors propose HiSA, which keeps each frequency pair only while its rotation is below an angle budget (default 90°). This creates a static, parameter-free sparsity pattern in the hidden dimension. Training from scratch with HiSA gives lower validation loss than dense attention while computing 29–34% of the dot products. The method also reduces KV-cache memory and speeds up decoding. The paper further shows that dense attention learns to suppress the same score parts that HiSA drops, providing an interpretability rationale.","headline_check":{"rule":"report_a_claim_v1","sent_back":false,"unattributed":{}},"pith_inferences":["The paper's interpretability analysis suggests that the benefit of HiSA may come from removing a source of noise (high-frequency pairs at long range) rather than from compute savings alone. This could be tested by training with HiSA but computing the full dot product (no compute saving) to see if the loss improvement persists.","The static sparsity pattern could be combined with token-level sparsity methods (e.g., sliding windows, top-k selection) for further gains, though the paper does not explore this.","The paper's results are on models up to 360M parameters; scaling to billions of parameters would test whether the benefit holds for larger models with different training dynamics.","The paper does not evaluate on downstream tasks (e.g., reasoning, coding, long-context benchmarks); such evaluations would be needed to confirm that the loss improvement translates to practical gains."],"forward_implications":["If HiSA scales to larger models, it could reduce the computational cost of attention in LLMs by roughly two-thirds during training and more during inference.","The KV-cache reduction (about one-third smaller) could enable longer context lengths on the same hardware.","The static sparsity pattern means no routing or selection overhead, making it compatible with existing efficient attention kernels like FlashAttention.","The finding that dense attention learns to suppress high-frequency pairs at long range suggests that HiSA's rule aligns with what models naturally learn, potentially making it a more efficient inductive bias."],"supporting_citations":[{"why":"Defines RoPE, the positional encoding that HiSA exploits.","marker":"su2021roformer"},{"why":"Provides the intuition that language models use distant context coarsely, motivating the hypothesis that high-frequency pairs are not needed at long range.","marker":"khandelwal2018sharp"},{"why":"FlashAttention-3 is the kernel that Flash-HiSA extends for efficient training and inference.","marker":"shah2024flashattention3"},{"why":"FlashAttention provides the IO-aware attention framework that HiSA's kernels build on.","marker":"dao2022flashattention"},{"why":"Provides the SmolLM architectures used for experiments.","marker":"allal2024smollm"}],"cache_read_input_tokens":0,"weakest_assumption_plain":"The core hypothesis is that high-frequency RoPE pairs are not needed at large token distances, and that removing them is harmless or beneficial. The paper provides empirical evidence but the theoretical justification relies on the intuition that language models do not need exact positional patterns at long range, citing Khandelwal et al. (2018) for recurrent language models. The extent to which this holds for modern LLMs with attention is not fully established.","cache_creation_input_tokens":0},"created_at":"2026-11-20T06:56:10.392454+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00","input_kind":"tex","input_chars":80781,"input_total_chars":80781,"passes_missing":[],"published_standing":"established","published_confidence":"moderate","objection_audit":{"version":"pith-objection-audit@1","counts":{"raised":5,"confirmed":0,"questions":0,"withdrawn":5,"moved_to_minor":0,"math_error_not_certified":0,"withdrawn_reviewer_error":2,"typesetting_minor_dropped":0,"withdrawn_already_addressed":3,"referee_assessment_unpublished":1},"open_questions":[],"standing":{"basis":["first_pass_verdict"],"standing":"established","confidence":"moderate"},"standing_rule":"1. No standing is published (not_assessed) when the review read only the abstract, when the referee pass or the objection check did not run, or when the first pass gave no verdict. 2. The first pass's verdict sets the start: ACCEPT is established, CONDITIONAL is established_narrower, REJECT is not_supported. 3. Each objection that survived the check against the paper's text caps the standing: one that breaks the central claim caps it at not_supported, one that narrows it at established_narrower, one that weakens it at supported_with_gaps. The standing is the lowest of the start and the caps. 4. A first-pass REJECT stands only if a surviving objection breaks the central claim. Without one, the standing is the lowest cap of the surviving objections, or not_assessed when none survived. 5. The confidence is the lowest of the first pass's confidence and the confidence of each objection that set the standing, and is no higher than moderate while a question is open or part of the paper's text was not supplied. 6. If any objection was withdrawn because Pith's reviewer misread the paper, the confidence is no higher than moderate, whatever the first pass said: a review that misread the paper once may have misread it elsewhere.","input_kind":"tex","nothing_to_audit":false,"bar":{"enabled":false}},"run_record":{"models":["deepseek/deepseek-v4-flash"],"passes":{"reader":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003155,"tokens_in":32803,"tokens_out":2165,"duration_ms":16060,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56","message_sha256":"619208179afd2c13f56df3aaad8a0db0e1681c07e2e08ccc89c20bab9c964428","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"referee":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003076,"tokens_in":28256,"tokens_out":2774,"duration_ms":19518,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"a852d99aea0ddfd900e1483f76e20777aae3afde9c016d5b8d6e5db01064c728","message_sha256":"e60af0ccfcbb83fc27164a5e2353fb363d26aaa4b2ac7fe619b9bee87844bef9","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"skeptic":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.002696,"tokens_in":27934,"tokens_out":846,"duration_ms":9282,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"8f88c0f61488b1e764c317998cddf35fffc33a1f20a3b1be54330d7843386fc7","message_sha256":"e1864b059e90124a4bf505ddd98fe9f99f55a5854fbac69ce0d8af2aee199e80","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"circularity":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.0025,"tokens_in":28108,"tokens_out":194,"duration_ms":4113,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"e9946e39d67b76cc60476379bc32abdc44665b5f0c438659796547bb1041a01f","message_sha256":"a7e98ed41171645481b98d19679d96ee066e5c1835901fd714113e7afce71bc1","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"desk_editor":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.002684,"tokens_in":28558,"tokens_out":471,"duration_ms":6165,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"e7211f6a3d2734c1025ce7e7e6d48f78b0944d0a32d5021f81be63b27721420a","message_sha256":"19ad89e2f19699a5e6619e20cb3a818d412ce475af75a3ef0d6840a364d5d4b9","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"objection_audit":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.003211,"tokens_in":29489,"tokens_out":2897,"duration_ms":18456,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"6c7a6883d5517cf9e4945d83947200a36dcd152b8b56c32baf7500d8b3841d0a","message_sha256":"83e9b50bb04ce286089b0ade94472391fcbb8b0cd72d4196b2f591f1f84c9841","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"pith_extraction":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.002941,"tokens_in":29723,"tokens_out":1297,"duration_ms":10590,"temperature":0.2,"input_sha256":"fa3f9bb8b8e356cd481e87fc5ed59485054674393d4937a670823b2951d258ee","prompt_sha256":"e4f56bf568e7854d0a1d2efb6ec96f8697d5132737cdb31cb7fc64acbd923974","message_sha256":"c255998d22892551867ff7a36a2d24de58dfe3f04bbd17ca8fe77c2a3769f62a","cache_read_input_tokens":0,"cache_creation_input_tokens":0}},"version":"pith-run-record/v1","prompt_set_sha256":"d60f54226f003b7308ab6f0f9f2951c09c726f00cc66fc05af3456f212247e93","configured_pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium"},"basis":{"kind":"full_text","source":"recorded","chars_given":80781,"chars_total":80781},"receipt":{"review_sha256":"0acfefd4e941dfec1ed0476af221c472533ae9423126fe79068fd50858ed7888","rule":"pith-review-sha256/v1","hashed_at":"2026-11-20T06:57:34.564010+00:00","fields":["id","arxiv_id","paper_version","created_at","pipeline_version","model_set","run_record","input_kind","input_chars","input_total_chars","passes_missing","verdict","confidence","novelty_score","correctness_risk","formal_verification","parameter_count","discovery_kind","one_line_summary","lead","skeptic_critique","referee_report","author_rebuttal","desk_editor","pith_extraction","objection_audit","published_standing","published_confidence"],"matches_current":true},"falsifier":"Train a model with HiSA but with the angle budget set to 360° (full rotation) and compare to dense attention. If the loss improvement disappears, it would confirm that the benefit comes from removing high-frequency pairs at long range, not from some other aspect of the method. Alternatively, train a model where the dropped pairs are replaced by random noise; if the loss is worse than HiSA, it would confirm that the dropped pairs are indeed harmful.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":"tex"},"url":"https://pith.science/accuracy"},"review_version":1}