Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference

Faramarz Fekri; Joe Zou; Siheng Xiong; Yae Jee Cho

REVIEW 4 major objections 6 minor 17 references

Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dynamic Hierarchical Sparse Attention predicts attention sparsity online by scoring chunk-level importance and upsampling to token pairs, letting frozen language models keep near-dense accuracy at a fraction of the compute.

desk verdict A genuinely new hierarchical sparse-attention mechanism with a solid LongBench result, undermined by an overclaiming abstract and a length-bias in the chunk scoring that needs quantitative attention. read the letter →

arxiv 2510.24606 v2 pith:5SUGOE43 submitted 2025-10-28 cs.CL

Siheng Xiong , Joe Zou , Faramarz Fekri , Yae Jee Cho This is my paper

classification cs.CL

keywords dynamicsparseattentionhierarchicalroutingchunk-levelsimilaritylong-contextinferencememory-constrainedLLMprefillspeedupsparsitypredictionfrozenmodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper contends that the quadratic cost of attention can be sidestepped without retraining the model: DHSA partitions a sequence into variable-length chunks, scores each chunk pair with a length-normalized average of query and key vectors, then propagates those scores to every token pair inside the chunk pair. A per-query token budget keeps only the most important interactions. On standard long-context benchmarks, this recovers nearly all of dense attention's accuracy while cutting prefill latency by 20–60% and peak memory by roughly 35%, and it outperforms fixed block-sparse patterns by 12–20% relative accuracy at similar cost. If correct, a frozen, open-weight language model could handle much longer contexts on memory-constrained hardware.

What carries the argument

The load-bearing object is the chunk-level similarity matrix S_c. It is built from boundary-predicted chunks, each represented by q_c = √|C|·(mean of queries) and k_c = √|C|·(mean of keys), so dot products are length-normalized and independent of chunk size. The upsample function tiles each S_c entry across the corresponding token submatrix to produce a token-level importance map S_t, and TopK selection with budget N_b turns S_t into the binary sparsity mask M. A roughly 20MB boundary predictor—two multi-head attention windows, feature fusion, and an MLP—locates chunk ends using local shifts in attention mass, trained with soft labels derived from accumulated attention ratios.

What would settle it

Construct a synthetic long context where one critical token (the 'needle') sits in a chunk whose neighboring tokens are irrelevant or contradictory, so the chunk-average similarity is low. If DHSA with a small budget prunes that token and fails the task while dense attention succeeds, the chunk-to-token proxy is falsified. A more direct test: compare DHSA's top-k mask to an oracle that selects the true top-k token pairs; the accuracy gap on LongBench quantifies how much the hierarchy loses.

Watch

Extended reading notes

Core claim

The central discovery is that a single chunk-level importance score—the dot product of length-normalized mean query and key vectors per chunk—can predict which token-level attention entries deserve the budget. DHSA first finds adaptive boundaries with a lightweight learned predictor, aggregates tokens into chunks with √|C|-scaled averaging, computes a chunk×chunk similarity matrix, then tiles each entry into the corresponding token submatrix and applies TopK selection. This hierarchical routing preserves salient dependencies without ever scoring all L×L pairs. Experiments show accuracy within a point of dense attention on LongBench with a 2k budget, near-perfect needle-in-a-haystack coverage

Load-bearing premise

The load-bearing premise is that a single chunk-level score—the dot product of averaged, length-normalized query and key vectors—faithfully represents the importance of every token-level interaction inside that chunk pair, so a rare but critical token can be pruned if its chunk averages out the signal.

Editorial extensions

If this is right

A frozen model can run long-context prompts at high sparsity with near-dense accuracy, enabling much longer contexts on memory-limited devices.
The per-query budget N_b can be set dynamically from available hardware, giving a direct accuracy–compute tradeoff without modifying model weights.
Because chunk boundaries align with discourse shifts (sentence and paragraph endings), DHSA can be combined with KV-cache compression methods to reduce both prefill and decode costs.
The boundary predictor is shared across layers and datasets, suggesting the learned sparsity pattern transfers across tasks and models.
A tiled or blocked backend can exploit the predicted mask to achieve large prefill speedups (up to 10× at 128k context) on consumer GPUs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the chunk-to-token proxy holds, the same hierarchy could be repurposed for KV-cache pruning during decoding, potentially multiplying memory savings beyond prefill.
The √|C| length normalization suggests a general principle that mean-pooled vector similarity over-weights short chunks; this correction could improve other hierarchical attention and retrieval systems.
One could probe whether the sparsity masks are stable across checkpoints and seeds; if they are not, the hierarchy may be latching onto surface regularities rather than true causal importance.
The boundary predictor's reliance on attention-ratio shifts implies it might be distilled into a single-pass token classifier, eliminating the extra O(L) routing pass at inference time.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The one genuinely new thing in this paper is the hierarchical sparsity prediction: a learned boundary predictor that segments the sequence into variable-length chunks, then a chunk-level similarity matrix upsampled to token-level top-k selection. That combination isn't in the sparse-attention literature I know, and it's a sensible way to get input-dependent sparsity without per-query token scoring. The LongBench table (Table 1) is the paper's real result: with a 2k per-query budget on Gemma2-2b-it, DHSA averages about 34.0 vs dense's 35.4 and clearly beats block sparse. That's a believable, useful number. The training details for the boundary predictor are also thorough — soft labels from attention ratios, focal loss, NMS — and the code is linked, which is more than many papers do.

That said, the paper has two soft spots that in my view are load-bearing. First, the abstract claims results that don't appear in the body: 10x speedup at 128K, LLaMA-3.1-8B on 100K, 12-20% accuracy gains, and "30-35% peak memory reduction." The body only tests Gemma2-2b-it and Gemma3-1b-it, and Tables 2-3 show DHSA using more peak memory than dense attention under torch.sdpa (6.99 vs 6.33 GB on NarrativeQA, 16.99 vs 15.18 at 32K). The 128K and 100K numbers are simply absent. That's a serious credibility problem — an abstract that contradicts its own tables.

Second, the chunk representation in Eq. 4 is length-confounded, just as the stress-test note says. Multiplying the average query by sqrt(|C|) makes the dot product scale with sqrt(|C_l||C_k|), so larger chunks get systematically higher scores. The paper calls this "length normalization," but it's actually a length bias. The consequence is exactly what you'd predict: the largest accuracy drops relative to dense are on retrieval-heavy tasks (Qasper, HotpotQA, Musique), where one salient token matters. The paper gives no quantitative evidence that chunk-level scores correlate with token-level top-k importance. This doesn't kill the method — the overall LongBench number is still respectable — but it's a hole that needs a direct analysis.

Also minor: evaluation on the first 100 validation samples without error bars is a selection choice that should be justified, and the 16K/32K results in Table 3 seem to contradict the stated 8K max context for Gemma2.

Who should read this: anyone working on efficient long-context inference. The mechanism is worth knowing, and the LongBench result is a useful data point. But I wouldn't cite it as-is until the claims are reconciled and the length-bias question is addressed. I'd send it to peer review — a serious referee can push on those points — but if I were the editor, I'd flag the abstract mismatch prominently and ask for a major revision before acceptance.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dynamic Hierarchical Sparse Attention (DHSA), a plug-in attention-sparsification module for frozen LLMs. DHSA partitions the sequence into variable-length chunks using a learned boundary predictor, aggregates token embeddings into chunk representations, computes a chunk-level similarity matrix, and upsamples it to token-level importance scores before applying a per-query top-k budget. Experiments on Gemma2-2b-it and Gemma3-1b-it report near-dense accuracy on LongBench and needle-in-a-haystack tests at reduced prefill latency and memory, compared against dense attention and block sparse attention.

Significance. If the core claims hold, DHSA offers a practical way to run long-context inference on memory-constrained devices with small accuracy loss. The LongBench table (Table 1) supports the near-dense claim at a 2k budget (DHSA average ≈34.0 vs dense ≈35.4 on Gemma2-2b-it). The paper includes code, evaluates on two model families, and provides latency/memory comparisons with various baselines. However, the central chunk-representation design contains a length-bias flaw that undermines the stated rationale, and several headline abstract claims are not supported by the body experiments. These issues need to be resolved before the paper can be accepted.

major comments (4)

[Abstract vs. §3 Evaluation] The abstract (page 1) claims evaluation on RULER, 12–20% relative accuracy gains over Block Sparse Attention, up to 10× prefill speedup at 128K, and LLaMA-3.1-8B 4-bit scaling to 100K on a single 24GB GPU. None of these appear in the body: the experiments (§3, Tables 1–3, Appendix D) use Gemma2-2b-it/Gemma3-1b-it, maximum context length 32K, and no RULER results. The full-text abstract also claims 6–18% accuracy gains and 20–60% latency reduction; Table 2 shows 17–28% latency reduction at 8K (eager: 1.65→1.19s; sdpa: 1.10→0.91s), with larger reductions only at 16K/32K in Table 3. The paper must either add the missing experiments or temper the claims.
[§2.3, Eq. (4)] The 'length-normalized aggregation' q_c = sqrt(|C|)·\bar{q} injects a chunk-size bias. Since (S_c)_{l,k} = sqrt(|C_l||C_k|)·(\bar{q}_l·\bar{k}_k), larger chunks receive systematically higher scores. With the constant upsampling in §2.1 (S_t[bl:bl+1,bk:bk+1] = S_c[l,k]), a large low-signal chunk can outrank a small chunk containing a single salient token, pruning that token. This is the opposite of the stated intent of 'avoiding the bias introduced by varying chunk lengths.' The failure pattern in Table 1 (largest drops vs. dense on Qasper −5.12, HotpotQA −2.85, Musique −3.15) is consistent with this bias. Provide quantitative evidence that chunk-level scores faithfully rank token-level importance (e.g., correlation with dense-attention top-k recall), or change the pooling to a genuinely length-invariant form and re-evaluate.
[Tables 2 and 3] The memory-efficiency claim is implementation-dependent. With torch.sdpa, dense attention uses 6.33 GB vs DHSA 6.99 GB at 8K (Table 2), and at 16K dense uses 8.38 GB vs DHSA 9.69 GB (Table 3). The 35% peak-memory reduction only holds against the eager implementation. Since the paper frames DHSA as a solution for memory-constrained inference, this overstates the benefit. Clarify which implementation the headline refers to and report memory for both eager and sdpa paths.
[§2.2 and Fig. 5] Dynamic chunking is one of the paper's three stated contributions, but its benefit over fixed-size chunking is only shown visually in Fig. 5. The appendix reports boundary-detection metrics (precision, recall, F1, top-K overlap), not end-task accuracy. Add a quantitative ablation comparing static vs. dynamic chunking on LongBench or needle-in-a-haystack; otherwise the contribution claim is unsupported.

minor comments (6)

[Eq. (2)] The feature vector uses sim(k_left, k_right) but the similarity kernel is not defined until later in §B. Define it at first use.
[Algorithm 1] Line 12 says 'Top Nb proceeding keys'; should read 'preceding keys'.
[Algorithm 2] In the loop over j = 0..Nc+1, the assignment s_t,new[bj:bj+1] = s_c,new[j] needs clarification for the last chunk containing the current query token; the boundary list includes indices b_{Nc+1} and b_{Nc+2}.
[Table 1] The entry 'DHSA (+bs)' is defined in the caption but not in the main text. Explain what 'boundary sharing across layers' means and why it is evaluated.
[Figure 2] The colorbar labels are clipped in the printed version; ensure the full scale (0.0–1.0) is visible.
[§B] The claimed 'strong alignment' between predicted boundaries and sentence/paragraph endings is not quantified. Add a numeric measure or rephrase as a qualitative observation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the boundary predictor is an explicit self-distillation surrogate, and final claims are tested against external benchmarks; the sqrt(|C|) chunk-scaling issue is a correctness concern, not a circular reduction.

full rationale

DHSA's derivation chain does not reduce to its own inputs. The boundary predictor is trained on soft labels derived from dense attention ratios (Eqs. 5-8) using Long Data Collections, TriviaQA, and ChatQA2, then evaluated on Needle-in-a-Haystack and LongBench; final accuracy is an external benchmark, not a refit of those labels. The paper explicitly frames this as knowledge distillation ('This approach is in line with knowledge distillation'). The chunk-level similarity scores in Eq. 4 are computed from the current layer's query/key embeddings, upsampled to token level (Sec. 2.1), and turned into a top-k mask; no fitted parameter is renamed as a prediction of the final metric. The only author self-citation (Yang et al. 2024) supports the general importance of long-context modeling and is not load-bearing. The limitations section notes that context-length extension beyond the Gemma family was not achieved and that hyperparameters remain important; this undercuts some abstract claims but is not circularity. The length-scaling in Eq. 4 (sqrt(|C|) weighting) may systematically favor large chunks and could hurt needle-style retrieval, as the reader's skeptic notes, but that is a modeling/correctness risk rather than an equation-level circular reduction. Accordingly, no circular step is exhibited.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method contributes no new physical entities; its free parameters are budget, chunk size, and a set of boundary-predictor hyperparameters tuned on validation data. The central approximation is that chunk-level averages and local boundary cues can reconstruct token-level attention importance.

free parameters (7)

token budget N_b = 512 / 1k / 2k
Per-query per-layer number of keys retained; controls accuracy/cost tradeoff and is set by hand per benchmark.
query chunk size = 64 or 256
Granularity of block-sparse implementation and chunk-size for DHSA; changes with experiment.
boundary window w = 4
Receptive field (8 tokens) for the boundary predictor; selected via Long Data Collections validation.
attention ratio threshold theta_r = 1.1
Candidate boundary threshold for automatic labels; chosen by hand.
soft-label scalars alpha, beta = alpha=2.0, beta=log10(2)≈0.301
Sigmoid parameters converting attention ratios to training probabilities; fit to training data.
focal loss and class weight = gamma=2.0, w=1.3
Loss hyperparameters for training the boundary predictor.
NMS window size = 8 or 64
Non-maximum suppression window for boundary post-processing; task-dependent.

assumptions (4)

domain assumption Chunk-level mean embeddings with length normalization preserve enough information to rank token-level attention importance.
Used in Section 2.3 and Step 2; if false, upsampling produces incorrect masks.
domain assumption Local key windows are sufficient to detect semantic chunk boundaries.
Section 2.2/B: the boundary predictor uses only two windows of w tokens each, ignoring global context.
domain assumption Dense attention ratios define correct chunk boundaries.
Section B automatic labelling: ground-truth boundaries are derived from attention mass of the same LLM; no external ground truth.
domain assumption TopK on predicted St approximates topK on true attention.
Section 2.1 Step 2; the entire method relies on this approximation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference." pith.science (2026). https://pith.science/paper/5SUGOE43

@misc{pith2026251024606,
  author       = {Pith},
  title        = {Pith review of: Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5SUGOE43}},
  note         = {Machine review of arXiv:2510.24606}
}

abstract

The quadratic cost of attention limits the scalability of long-context LLMs, especially under limited hardware memory budgets. While attention is often sparse, existing static sparse methods cannot adapt to task- or input-dependent variations, and recent dynamic approaches rely on predefined templates or heuristics that may sacrifice generality. We propose Dynamic Hierarchical Sparse Attention (DHSA), a data-driven framework that predicts attention sparsity online while keeping the LLM backbone frozen. DHSA performs hierarchical routing by estimating importance at the chunk level and propagating it to token-level interactions, preserving causally important dependencies while enabling efficient sparsification. Across Needle-in-a-Haystack test, LongBench and RULER, DHSA maintains near-dense accuracy in highly sparse regimes, achieving 12--20% relative accuracy gains over Block Sparse Attention at comparable prefill cost. With a memory-efficient tiled backend, DHSA delivers up to $10\times$ prefill speedup at 128K context length. On LLaMA-3.1-8B (4-bit), DHSA scales to 100K context on a single 24GB GPU, where dense attention fails. We provide complementary GPU and CPU backends, enabling DHSA to run across diverse hardware environments and multiple open-weight model families. These results demonstrate DHSA as an efficient and adaptable solution for memory-constrained long-context LLM inference.

Figures

Figures reproduced from arXiv: 2510.24606 by the authors.

**Figure 2.** Needle-in-a-haystack results on Gemma2-2b-it (maximum context length = 8k). The budget is specified per query per layer. Single-Document QA Multi-Document QA Summarization Few-shot Learning Method NrtvQA Qasper Mf-en HotpotQA 2WikiMQA Musique GovReport QMSum MultiNews TriviaQA SAMSum Dense 22.37 35.32 37.32 41.63 32.05 19.05 27.08 21.08 25.48 87.00 41.26 Block sparse 16.74 26.15 32.83 35.74 31.93 14.44 26.20 19.54 2… view at source ↗

**Figure 4.** illustrates our automatic labeling strategy. For each candidate position, we examine the accumulated attention mass patterns in its left window and right window, where each row in the heatmap corresponds to a token and color intensity indicates the magnitude of accumulated attention mass. In the left example, the left and right windows (both outlined in blue) exhibit highly similar attention profiles, indicating tha… view at source ↗

Figures from the paper (6 more)

**Figure 5.** Figure 5: Token-level similarity matrices with static (left) and dynamic (right) chunking in Gemma2- [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Needle-in-a-haystack results on Gemma2-2b-it (maximum context length = 8k). The budget [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: Latency comparison on Gemma2-2b-it with varying context lengths. Sliding-window [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: Peak memory usage comparison on Gemma2-2b-it with varying context lengths. Sliding [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Latency and peak memory usage comparison across different baselines with varying context [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: Potential factors affecting the latency of KV compression methods on Gemma2-2b-it. The [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 linked inside Pith

[1]

The compressor-retriever architecture for language model os.arXiv preprint arXiv:2409.01495,

Yuan Yang, Siheng Xiong, Ehsan Shareghi, and Faramarz Fekri. The compressor-retriever architecture for language model os.arXiv preprint arXiv:2409.01495,

arXiv
[5]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al

Accessed: 2025-08-27. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

arXiv 2025
[6]

Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

arXiv
[8]

Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities.arXiv preprint arXiv:2407.14482,

Peng Xu, Wei Ping, Xianchao Wu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities.arXiv preprint arXiv:2407.14482,

arXiv
[10]

Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

arXiv
[11]

and a global attention layer (layer 1). Using the conventional Binary Cross-Entropy (BCE) loss for training our boundary predictor in dynamic chunking, we identify two key issues: (1)class imbalance, where boundary tokens are much fewer than non-boundary tokens, and (2)varying sample difficulty, where some boundaries are easier to detect than others. To a...

2017
[12]

On Gemma2-2b-it, retaining the top 1k tokens per layer, DHSA matches dense attention while substantially outperforming block sparse attention

Among sparse attention methods, sliding window attention [Beltagy et al., 2020] relies on static patterns that perform poorly when the actual attention deviates from the predefined one. On Gemma2-2b-it, retaining the top 1k tokens per layer, DHSA matches dense attention while substantially outperforming block sparse attention. Even with a budget of 512, D...

2020
[14]

Sliding-window attention uses a budget of 2048, block sparse attention 512 and KV compression (streaming LLM, h2o, pyramidKV)

(a) Prefill-stage sparse attention 0 5000 10000 15000 20000 25000 30000 # Context T oken 0 2 4 6 8 10Time (s) Latency Comparison between Different Methods dense attention prefill dense attention decode sliding window attn prefill sliding window attn decode streaming llm prefill streaming llm decode h2o prefill h2o decode pyramid kv prefill pyramid kv deco...

Show all 17 references

[15]

Sliding- window attention uses a budget of 2048, block sparse attention 512 and KV compression (streaming LLM, h2o, pyramidKV)

(a) Prefill-stage sparse attention 0 5000 10000 15000 20000 25000 30000 # Context T oken 6 8 10 12 14 16Peak Allocated Memory (GB) Peak Memory Usage Comparison between Different Methods dense attention prefill dense attention decode sliding window prefill sliding window decode...
[16]

Finally, we note several potential influencing factors in KV compression methods (Fig

The number of generated tokens is 100, and all methods are imple- mented withtorch.sdpa. Finally, we note several potential influencing factors in KV compression methods (Fig. 10). Large changes in wall-clock inference time occur only with substantial reductions in KV cache si...
[17]

0 2000 4000 6000 8000 10000 12000 14000 16000 Max KV Cache Capacity 3.2 3.4 3.6 3.8 4.0 4.2Time (s) Latency v.s

The number of generated tokens is 100, and all methods are implemented withtorch.sdpa. 0 2000 4000 6000 8000 10000 12000 14000 16000 Max KV Cache Capacity 3.2 3.4 3.6 3.8 4.0 4.2Time (s) Latency v.s. Max KV Cache Capacity sliding window attn decode h2o decode (a) Latency vs. M...

2000
[128]

On the other hand, for all decode-stage methods, prefill time increases with context length since KV cache compression only applies during decoding

Block sparse attention with larger block sizes can also reduce prefill latency. On the other hand, for all decode-stage methods, prefill time increases with context length since KV cache compression only applies during decoding. Decoding time remains constant for KV cache comp...
[2006]

Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453,

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453,

arXiv
[2017]

Longformer: The long-document transformer

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150,

2004 arXiv
[2023]

Longbench: A bilingual, multitask benchmark for long context understanding.arXiv preprint arXiv:2308.14508,

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding.arXiv preprint arXiv:2308.14508,

arXiv
[2024]

Lm- infinite: Zero-shot extreme length generalization for large language models.arXiv preprint arXiv:2308.16137,

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- infinite: Zero-shot extreme length generalization for large language models.arXiv preprint arXiv:2308.16137,

arXiv
[2025]

triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv e-prints, art

5 Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.arXiv e-prints, art. arXiv:1705.03551,

arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

The compressor-retriever architecture for language model os.arXiv preprint arXiv:2409.01495,

Yuan Yang, Siheng Xiong, Ehsan Shareghi, and Faramarz Fekri. The compressor-retriever architecture for language model os.arXiv preprint arXiv:2409.01495,

arXiv

[2] [5]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al

Accessed: 2025-08-27. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,

arXiv 2025

[3] [6]

Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

arXiv

[4] [8]

Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities.arXiv preprint arXiv:2407.14482,

Peng Xu, Wei Ping, Xianchao Wu, Zihan Liu, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa 2: Bridging the gap to proprietary llms in long context and rag capabilities.arXiv preprint arXiv:2407.14482,

arXiv

[5] [10]

Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.arXiv preprint arXiv:2406.02069,

arXiv

[6] [11]

and a global attention layer (layer 1). Using the conventional Binary Cross-Entropy (BCE) loss for training our boundary predictor in dynamic chunking, we identify two key issues: (1)class imbalance, where boundary tokens are much fewer than non-boundary tokens, and (2)varying sample difficulty, where some boundaries are easier to detect than others. To a...

2017

[7] [12]

On Gemma2-2b-it, retaining the top 1k tokens per layer, DHSA matches dense attention while substantially outperforming block sparse attention

Among sparse attention methods, sliding window attention [Beltagy et al., 2020] relies on static patterns that perform poorly when the actual attention deviates from the predefined one. On Gemma2-2b-it, retaining the top 1k tokens per layer, DHSA matches dense attention while substantially outperforming block sparse attention. Even with a budget of 512, D...

2020

[8] [14]

Sliding-window attention uses a budget of 2048, block sparse attention 512 and KV compression (streaming LLM, h2o, pyramidKV)

(a) Prefill-stage sparse attention 0 5000 10000 15000 20000 25000 30000 # Context T oken 0 2 4 6 8 10Time (s) Latency Comparison between Different Methods dense attention prefill dense attention decode sliding window attn prefill sliding window attn decode streaming llm prefill streaming llm decode h2o prefill h2o decode pyramid kv prefill pyramid kv deco...