Pith. sign in

REVIEW 3 major objections 3 minor 54 references

ChunkLLM keeps 98.64% of long-context accuracy while cutting KV cache to 48.58% and speeding up 120K-token generation by up to 4.48x.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:38 UTC pith:R2G6RHI5

load-bearing objection ChunkLLM offers a genuinely novel plug-and-play chunked-attention scheme, but the headline quality and speedup claims are demonstrated in different configurations, and one PPL result is misreported. the 3 major comments →

arxiv 2510.02361 v2 pith:R2G6RHI5 submitted 2025-09-28 cs.CL cs.AI

ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inference

Haojie Ouyang , Jianwei Lv , Lei Ren , Chen Wei , Xiaojie Wang , Fangxiang Feng This is my paper
classification cs.CL cs.AI
keywords LLM inference accelerationKV cache compressionchunk selectionattention distillationsparse attentionlong-context transformerchunk boundary detectionIntra-Chunk Attention Consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that a plug-in, train-only-lightweight framework can make ordinary Transformers fast on very long inputs without giving up much accuracy. It claims that splitting text into semantic chunks, compressing each chunk to a single key, and selecting key chunks during inference preserves 98.64% of long-context benchmark performance while retaining only 48.58% of the KV cache and reaching a 4.48x speedup on 120K-token generation. The method is meant to be attached to existing LLMs with frozen backbones, so it offers a low-cost route to long-context inference acceleration. A sympathetic reader should see this as an attempt to solve the semantic-incompleteness problem of fixed-size chunks and the training-inference inefficiency of prior sparse-attention and separator-based methods.

Core claim

ChunkLLM is a pluggable framework that adds two lightweight modules to an existing Transformer: a Chunk Adapter at the bottom layer that predicts sentence-level chunk boundaries from contextual semantics, and per-layer QK Adapters that compress queries and chunk-representative keys and produce chunk-level attention scores. The QK Adapters are trained by attention distillation, matching their chunk attention to the full attention aggregated over oracle chunk boundaries, while the backbone stays frozen. At inference, only the top-k chunks chosen by a layer-wise voting mechanism are stored in the KV cache, and a discovered pattern called Intra-Chunk Attention Consistency (ICAC) means chunk sele

What carries the argument

The central mechanism is the QK Adapter pair: a Q-Adapter that projects each query into a low-dimensional space and a K-Adapter that projects the key vector of each chunk's final token into the same space, so chunk attention scores can be computed as softmax over compressed query-chunk-key products. A Chunk Adapter, a one-layer feed-forward classifier on the first layer's hidden states, identifies where chunks end. Attention distillation uses KL divergence to make the student chunk attention approximate the teacher's full attention aggregated per chunk, and the ICAC pattern prevents recomputing chunk selection inside a chunk. Together, these turn the problem of attending to long context into

Load-bearing premise

The whole scheme rests on assuming that a chunk's final token reliably summarizes that chunk, so that selecting chunks by their final token's key picks the right information; if that summary fails, top-k chunk recall and downstream accuracy collapse.

What would settle it

Take a 120K prompt and replace or mask the last word of every semantic chunk before feeding it through ChunkLLM, then measure needle-in-a-haystack accuracy and LongBench score; the method predicts a large drop if chunk keys carry the load, whereas a method that does not rely on boundary tokens would degrade far less.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • ChunkLLM achieves 98.64% of the vanilla Transformer's long-context benchmark score while using only 48.58% of the KV cache, and stays above 99.5% of vanilla performance on short-text benchmarks.
  • In 120K-token generation, per-token time remains roughly flat instead of growing linearly, yielding 3.84x and 4.48x speedups on two 7-8B open-weight LLMs.
  • Because the adapters are trained on 4K contexts and transfer to 120K, long-context acceleration can be obtained without expensive long-context training data.
  • ChunkLLM outperforms separator-based and sink-window sparse-attention baselines on LongBench and the 64K needle-in-a-haystack test while using less KV cache.
  • The framework keeps all backbone parameters frozen, so it can be bolted onto any Transformer-based LLM by training only the two small adapters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because each chunk is represented solely by the key of its last token, the method should be most sensitive to how well boundary tokens summarize their chunks; a testable extension is to represent chunks by a learned summary vector or by pooling several token keys, which could improve recall on texts where boundaries are not informative.
  • Editorial inference: ICAC implies that attention output within a chunk is nearly constant, which suggests one could also reuse or cache the attention output itself for all tokens inside a chunk, not just skip chunk re-selection; the paper does not measure this additional saving.
  • Editorial inference: the chunk boundary detector is trained on well-formed sentence boundaries, so code, tables, speech transcripts, or other non-sentence text may require re-training the Chunk Adapter; a quick check is to measure boundary-prediction F1 and downstream retrieval on those genres.
  • Editorial inference: the reported speedups are for vanilla autoregressive decoding; combining ChunkLLM with speculative decoding or parallel decoding could compound the gains, but that combination is beyond the paper's experiments.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces ChunkLLM, a pluggable framework that adds two lightweight modules to a frozen Transformer backbone: a Chunk Adapter that predicts sentence-like chunk boundaries from the bottom-layer hidden states, and QK Adapters at each layer that compress chunk keys and produce chunk-level attention scores. The QK Adapters are trained by KL distillation against full attention aggregated over oracle sentence boundaries. At inference, the model selects top-k chunks via the adapted attention scores, uses a voting mechanism across layers, and only re-selects chunks at predicted boundary tokens (the ICAC pattern). On LongBench, the authors report 98.64% of vanilla performance with 48.58% KV-cache retention for Qwen2.5-7B, and a maximum 4.48x speedup for 120K-token generation on PG19 with Llama3.1-8B. Ablations address the voting mechanism, ICAC, and fixed vs. semantic chunks.

Significance. If the central claims are sustained, ChunkLLM is a practically attractive recipe: it keeps the backbone frozen, trains on only 4K-length data, and reports strong long-context retention with a large speedup, while also outperforming SepLLM and StreamingLLM on the evaluated LongBench tasks. The attention-distillation objective for chunk-level selection and the ICAC observation are useful and plausible ideas, and the paper is honest in reporting KV-cache usage alongside accuracy. However, the evidence as presented does not yet establish the joint '98.64% accuracy + 4.48x speedup' claim, because the two numbers come from different hyperparameter configurations and the 120K regime shows perplexity degradation. The paper does not provide code, seeds, or confidence intervals, which limits reproducibility and the strength of the comparative claims.

major comments (3)
  1. [§3.2.3, Table 2] The text states that ChunkLLM 'yields a slight enhancement in ppl' relative to the vanilla model, but Table 2 shows the opposite: PPL increases from 14.41 to 16.23 for Qwen2.5-7B and from 11.93 to 12.89 for Llama3.1-8B. This is not a wording issue only: at 120K, the regime used for the headline 4.48x speedup, the only reported quality metric is worse for ChunkLLM. The abstract's joint claim of 'maintains 98.64% of the performance ... while preserving a 48.58% key-value cache retention rate' is based on LongBench with top-k ratio 45% and local chunks 15 (§3.2.1), whereas the speedup experiment uses top-k=256 and local chunks=16 (§3.2.3). The two halves of the central claim are therefore demonstrated in different configurations, and no quality metric at 120K supports the 'maintains performance' part.
  2. [§2.2, Eq. (3)] The student chunk-attention scores use only one transformed key per chunk, namely the key of the boundary token (Index_c in Eq. 3), while the teacher in Eq. (4) aggregates full attention over oracle sentence boundaries. The student is therefore trained to reproduce a quantity that is more informative than the representation it is allowed to use. The supporting recall evidence in Appendix 6.5 is given only for 4K and 32K inputs with top-k up to 45; the 120K generation experiments use top-k=256 over a much larger number of chunks, and no recall or retrieval-quality measurement is reported at that regime. Since the entire inference pipeline depends on the boundary-token key being sufficient for top-k chunk selection, the claim that 120K inference maintains performance is not established by the reported evidence.
  3. [§3.2.1–§3.2.3] The operating point changes between benchmarks: LongBench uses a 45% top-k ratio and 15 local chunks, while NIAH and the 120K PG19 experiments use top-k=256 and 16 local chunks. No sensitivity analysis, multiple seeds, or confidence intervals are provided. This matters because several reported advantages over baselines are small on individual subtasks (e.g., in Table 1, many entries differ by less than 1 point), and it is unclear whether the 'maximum speedup of 4.48x' is obtainable at the LongBench setting that supports the 98.64% accuracy claim. The paper should either report a single operating point for all headline claims or provide a sensitivity analysis showing that the conclusions are robust to these hyperparameter changes.
minor comments (3)
  1. [§3.2.4, Table 3] The table uses abbreviations 'Heag' and 'WG' without full names; these should be HellaSwag and WinoGrande for consistency with the text. The column spacing in Tables 1 and 3 is also inconsistent and makes some entries hard to read.
  2. [§2.3] The 'chunk voting mechanism' is described only verbally. It would help to specify exactly how the per-layer top-k sets are combined into the 'global top-k chunks'—e.g., by frequency counts, averaged scores, or a threshold—since this step is ablated and claimed to be important.
  3. [Appendix 6.5] Figure 10 reports recall for top-k values up to 45, but the NIAH and PG19 configurations use top-k=256. The recall behavior at larger top-k values, and particularly at 120K length, should be shown or discussed. Also, there is a typo in the caption: 'aross' should be 'across'.

Circularity Check

0 steps flagged

No significant circularity: the headline claims are empirical measurements under explicit hyperparameter settings, with training labels from an external sentence-boundary tool and evaluation on external task benchmarks.

full rationale

ChunkLLM's derivation chain is not circular by construction. The Chunk Adapter is trained with binary cross-entropy against pySBD sentence-boundary labels (Eq. 2), an external rule-based annotation tool, and the QK Adapter is trained by KL distillation to approximate the chunk-aggregated full-attention teacher A_t = Aggregate(A) (Eqs. 4-5). This is a standard distillation setup: the teacher is built from the same boundary annotation used to supervise the boundary predictor, but at inference the chunk boundaries come from the learned Chunk Adapter, and the headline results (LongBench 98.64%, NIAH accuracy, PG19 4.48x speedup) are measured on external task benchmarks using task labels, not on the training annotation. The top-k ratio and local-chunk counts are hyperparameters chosen separately for each benchmark (45%/15 for LongBench; 256/16 for NIAH and PG19), so the reported retention rates and speedups are measurements at those operating points, not predictions forced by a fitted parameter. The single-token-key representation (Eq. 3) is a modeling assumption, and the paper tests it empirically via recall curves (App. 6.5) and downstream accuracy; a load-bearing assumption is not the same as a circular definition. There is a reporting inconsistency: Table 2 shows PPL increasing (Qwen2.5-7B: 14.41 to 16.23; Llama3.1-8B: 11.93 to 12.89) while the text calls this a 'slight enhancement in ppl'; that is a correctness/analysis concern, not a circularity concern. No load-bearing self-citation chain is present. The central claims therefore stand on independent empirical evaluation rather than reducing to their inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

Everything in the central speed and quality claims rests on the adapter training and benchmark-specific budgets; the ledger lists the tuned budgets and the unproven representational assumptions about boundary tokens.

free parameters (5)
  • top_k_ratio = 45% (LongBench)
    Chunk selection budget for LongBench experiments; sets KV cache retention near 48.58%.
  • top_k_count = 256 (NIAH, PG19)
    Chunk selection budget for 64K NIAH and 120K PG19 runs; directly affects speedup and quality.
  • local_chunks = 15 (LongBench) / 16 (NIAH, PG19)
    Number of local chunks kept, a hyperparameter tuned per experiment.
  • boundary_threshold_alpha = 0.5
    Sigmoid threshold for chunk boundary prediction (Eq. 1); changes which tokens trigger chunk updates.
  • adapter_dimensions = 3584 (Qwen), 4096 (Llama)
    Hidden dimension of QK and Chunk adapters, chosen to match model width.
axioms (5)
  • domain assumption Layer-1 hidden states carry enough semantic signal to detect sentence-level chunk boundaries.
    Chunk Adapter consumes only the first-layer output (Sec 2.1); if boundaries are not detectable there, the whole chunking fails.
  • domain assumption A chunk can be represented by the key vector of its last token alone.
    In Eq. 3, K_hat contains only boundary-token keys; the student attention has no access to non-boundary tokens.
  • domain assumption Sum-aggregated token-level attention over oracle sentence chunks is a valid training target for chunk attention.
    Teacher A_t = Aggregate(A) in Eq. 4 defines the distillation signal.
  • domain assumption Intra-Chunk Attention Consistency holds at generation time.
    ICAC pattern (Sec 2.3) underlies the decision to skip chunk selection except at boundaries; if false, the speedup claim collapses.
  • domain assumption Frozen pretrained model weights remain valid when attention is restricted to selected chunks and local context.
    No backbone adaptation is performed; the paper relies on the pretrained Q/K/V being usable under sparse chunk attention.

pith-pipeline@v1.3.0-alltime-deepseek · 17852 in / 14791 out tokens · 331755 ms · 2026-08-04T14:38:08.560720+00:00 · methodology

0 comments
read the original abstract

Transformer-based large models excel in natural language processing and computer vision, but face severe computational inefficiencies due to the self-attention's quadratic complexity with input tokens. Recently, researchers have proposed a series of methods based on block selection and compression to alleviate this problem, but they either have issues with semantic incompleteness or poor training-inference efficiency. To comprehensively address these challenges, we propose ChunkLLM, a lightweight and pluggable training framework. Specifically, we introduce two components: QK Adapter (Q-Adapter and K-Adapter) and Chunk Adapter. The former is attached to each Transformer layer, serving dual purposes of feature compression and chunk attention acquisition. The latter operates at the bottommost layer of the model, functioning to detect chunk boundaries by leveraging contextual semantic information. During the training phase, the parameters of the backbone remain frozen, with only the QK Adapter and Chunk Adapter undergoing training. Notably, we design an attention distillation method for training the QK Adapter, which enhances the recall rate of key chunks. During the inference phase, chunk selection is triggered exclusively when the current token is detected as a chunk boundary, thereby accelerating model inference. Experimental evaluations are conducted on a diverse set of long-text and short-text benchmark datasets spanning multiple tasks. ChunkLLM not only attains comparable performance on short-text benchmarks but also maintains 98.64% of the performance on long-context benchmarks while preserving a 48.58% key-value cache retention rate. Particularly, ChunkLLM attains a maximum speedup of 4.48x in comparison to the vanilla Transformer in the processing of 120K long texts.

Figures

Figures reproduced from arXiv: 2510.02361 by Chen Wei, Fangxiang Feng, Haojie Ouyang, Jianwei Lv, Lei Ren, Xiaojie Wang.

Figure 1
Figure 1. Figure 1: The framework of ChunkLLM. The Chunk Adapter is a one-layer forward neural network (FNN) classifier for chunk boundary prediction. Its input is the output of the first layer of the LLM, and the out￾put is if or not the token is a chunk bound￾ary, as depicted in the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The inference process of ChunkLLM. We employ the Kullback-Leibler (KL) diver￾gence as the loss function for attention dis￾tillation to guide the student model A s in approximating the teacher model’s attention scores A t : L N AD = KL(A t ||A s ) (5) We average the KL divergence losses across the N layers to obtain the final attention dis￾tillation loss: LAD = 1 N X N i L i AD (6) During the training phase… view at source ↗
Figure 3
Figure 3. Figure 3: Attention visualization of chunk selec￾tion during the inference phase. The sample is de￾rived from the passkey retrieval task. The inference phase of ChunkLLM is depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Needle-in-a-Haystack retrieval accuracy across context positions with 64k context length. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of inference time per 10k tokens in the generation process on PG19 test set. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Inference efficiency of ICAC on PG19 test set. We validate the proposed vote mechanism and ICAC pattern based on the Qwen2.5-7B us￾ing the LongBench, with experimental results shown in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of fixed chunks and semantic chunks in NIAH test. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Compare with StreamingLLM on Needle-in-a-Haystack. The last column represents the [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Case study of ICAC. The test case is from the LongBench summarization task. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Top-k chunks aross all layers. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    2026

  5. [5]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for ...

    2024

  6. [7]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020 b

    Pith/arXiv arXiv 2004

  7. [8]

    Transformers to ssms: Distilling quadratic knowledge to subquadratic models

    Aviv Bick, Kevin Li, Eric Xing, J Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. Advances in Neural Information Processing Systems, 37: 0 31788--31812, 2024

    2024

  8. [9]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

    2020

  9. [10]

    Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic KV cache compression based on pyramidal information funneling. CoRR, abs/2406.02069, 2024. doi:10.48550/ARXIV.2406.02069. URL https://doi.org/10.48550/arXiv.2406.02069

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.02069 2024

  10. [11]

    Sepllm: Accelerate large language models by compressing one segment into one separator

    Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. Sepllm: Accelerate large language models by compressing one segment into one separator. CoRR, abs/2412.12094, 2024. doi:10.48550/ARXIV.2412.12094. URL https://doi.org/10.48550/arXiv.2412.12094

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.12094 2024

  11. [12]

    Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2 0 (3): 0 6, 2023

    2023

  12. [13]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  13. [14]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

  14. [15]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp.\ arXiv--2407, 2024

    2024

  15. [16]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. CoRR, abs/2006.05525, 2020. URL https://arxiv.org/abs/2006.05525

    Pith/arXiv arXiv 2006

  16. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  17. [18]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531

    Pith/arXiv arXiv 2015

  18. [19]

    Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37: 0 52481--52515, 2024

    2024

  19. [20]

    Wise: Weak-supervision-guided step-by-step explanations for multimodal llms in image classification

    Yiwen Jiang, Deval Mehta, Siyuan Yan, Yaling Shen, Zimu Wang, and Zongyuan Ge. Wise: Weak-supervision-guided step-by-step explanations for multimodal llms in image classification. arXiv preprint arXiv:2509.17740, 2025

    arXiv 2025

  20. [21]

    G. Kamradt. Llmtest needleinahaystack. 2023. URL https://github.c om/gkamradt/LLMTest_NeedleInAHaystack

    2023

  21. [22]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, pp.\ 1317--1327, 2016

    2016

  22. [23]

    Minimax-01: Scaling foundation models with lightning attention

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313, 2025

    Pith/arXiv arXiv 2025

  23. [24]

    Snapkv: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing S...

    2024

  24. [25]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024

    Pith/arXiv arXiv 2024

  25. [26]

    Fineweb-edu: the finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  26. [27]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y. Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. Moba: Mixture of block attention for long-conte...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.13189 2025

  27. [28]

    Linearizing large language models

    Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models. arXiv preprint arXiv:2405.06640, 2024

    Pith/arXiv arXiv 2024

  28. [29]

    Llama 3 model card

    Meta. Llama 3 model card. 2024

    2024

  29. [30]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - Novemb...

    doi:10.18653/v1/d18-1260 2018

  30. [31]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023 a

    Pith/arXiv arXiv 2023

  31. [32]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023 b

    Pith/arXiv arXiv 2023

  32. [33]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 2024

    Pith/arXiv arXiv 2024

  33. [34]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019

    Pith/arXiv arXiv 1911

  34. [35]

    Rusu, Sergio Gomez Colmenarejo, C aglar G \" u l c ehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell

    Andrei A. Rusu, Sergio Gomez Colmenarejo, C aglar G \" u l c ehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference T...

    Pith/arXiv arXiv 2016

  35. [36]

    P y SBD : Pragmatic sentence boundary disambiguation

    Nipun Sadvilkar and Mark Neumann. P y SBD : Pragmatic sentence boundary disambiguation. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pp.\ 110--114, Online, November 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.nlposs-1.15

    2020

  36. [37]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Art...

    doi:10.1609/aaai.v34i05.6399 2020

  37. [38]

    Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019. URL http://arxiv.org/abs/1910.01108

    Pith/arXiv arXiv 1910

  38. [39]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin...

    doi:10.18653/v1/ 2019

  39. [40]

    Instruction-tuning llms for event extraction with annotation guidelines

    Saurabh Srivastava, Sweta Pati, and Ziyu Yao. Instruction-tuning llms for event extraction with annotation guidelines. arXiv preprint arXiv:2502.16377, 2025

    Pith/arXiv arXiv 2025

  40. [41]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  41. [42]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-...

    2019

  42. [43]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    Pith/arXiv arXiv 2024

  43. [44]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  44. [45]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  45. [46]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  46. [47]

    Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression

    Haoyu Wang, Tong Teng, Tianyu Guo, An Xiao, Duyu Tang, Hanting Chen, and Yunhe Wang. Unshackling context length: An efficient selective attention approach through query-key compression. CoRR, abs/2502.14477, 2025. doi:10.48550/ARXIV.2502.14477. URL https://doi.org/10.48550/arXiv.2502.14477

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.14477 2025

  47. [48]

    The mamba in the llama: Distilling and accelerating hybrid models

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37: 0 62432--62457, 2024

    2024

  48. [49]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pp.\ 94--106. Association for Computational Linguistics, 2017. doi:10.18653/...

    doi:10.18653/v1/w17-4413 2017

  49. [50]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=NG7sS51zVF

    2024

  50. [51]

    Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference

    Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid KV cache compression for high-throughput LLM inference. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , pp.\ 3258--3270. A...

    doi:10.18653/v1/2024.findings-acl.195 2024

  51. [52]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehva...

    2025

  52. [53]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pp...

    doi:10.18653/v1/p19-1472 2019

  53. [54]

    Event temporal relation extraction based on retrieval-augmented on llms

    Xiaobin Zhang, Liangjun Zang, Qianwen Liu, Shuchong Wei, and Songlin Hu. Event temporal relation extraction based on retrieval-augmented on llms. In 2024 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--8. IEEE, 2024

    2024

  54. [55]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (ed...

    2023