Pith. sign in

REVIEW 4 major objections 5 minor 78 references

LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that long-context fine-tuning can be made far cheaper by identifying and dropping low-information tokens, and presents LeMo, a system that turns this token-level sparsity into up to 1.93x memory savings and up to 1.36x…

desk verdict Plausible systems paper with real memory savings, but the accuracy claim rests on an underexamined predictor; worth refereeing with requests for artifacts and a drift analysis. read the letter →

arxiv 2501.09767 v1 pith:MGKTOKFW submitted 2025-01-15 cs.CL cs.AI

Tuowei Wang , Xingyu Chen , Kun Li , Ting Cao , Ju Ren , Yaoxue Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords long-contextfine-tuningactivationmemorytokensparsitycontextualLoRAattentionparameter-efficientGPUoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LeMo is a fine-tuning system built on the claim that long-context sequences contain tokens that are redundant for learning, and that those tokens can be excluded from attention and MLP computation without meaningful accuracy loss. The reason this matters is that activation memory, not model parameters, is the main bottleneck when extending context windows: every token that participates in a layer leaves an activation footprint even if attention is sparse, an effect the paper calls "shadowy activation." LeMo directly reduces token involvement, which cuts activation memory and, as a side effect, speeds up training. The system identifies informative tokens with attention-score sums, predicts sparsity patterns with small neural networks, and uses kernel-level optimizations to make the elimination cheap. Evaluations on OPT and Llama families across three GPU platforms report memory reductions up to 1.93x, speedups up to 1.36x, and only small perplexity increases relative to LoRA.

What carries the argument

The load-bearing object is the token informativeness score, $I(T_j) = \sum_{i\neq j} S_{ij} = \sum_{i\neq j} Q_i K_j$, the column sum of raw pre-softmax attention scores: a token is considered redundant when its aggregate interaction with all other tokens is low. LeMo applies this score block-wise, takes the maximum positive attention score within each block, sums across blocks by column, and compares the result to a layer-specific threshold; the same idea is applied to MLP activations. Because computing full attention scores is expensive, each layer gets a pair of small low-rank ReLU predictors that map token embeddings to approximate query and key block scores, and their product estimates the block informativeness; an elastic size transformation prunes predictor neurons that are always zero. The supporting system machinery is the permutation-free kernel, which fuses selection, padding, and residual addition to avoid global memory movement, and segment-based gradient computation, which splits loss-gradient computation to cut activation memory peaks.

What would settle it

Run a Llama2-7B fine-tuning run at 16K sequence length and, every 500 steps, compare the tokens LeMo's predictors eliminate against the tokens with the highest true informativeness scores under current weights; if the recall of eliminated-but-truly-informative tokens falls materially below the reported 95.13% as training progresses, or if a full-attention control model and LeMo diverge beyond the reported perplexity differences on LongBench, the claim that elimination is accuracy-preserving would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that token-level sparsity, which it names Contextual Token Sparsity, exists and is exploitable during long-context fine-tuning: the tokens that matter are a small subset of the sequence, and this subset shifts with the input text and with the model layer. It defines a token's informativeness as the column sum of raw pre-softmax attention scores, aggregates these scores in blocks, and eliminates the low-scoring blocks using thresholds tuned per layer. Two lightweight predictors per layer estimate the informative scores of query and key blocks from contextual embeddings, so the full attention matrix never has to be materialized. LeMo then makes the elimination practical with a permutation-free kernel that avoids global memory movement and a segment-based scheme that cuts the activation peak from loss gradient computation. The paper reports that this design reduces memory consumption by up to 1.93x and achieves up to 1.36x speedups over state-of-the-art fine-tuning, with perplexity on PG19 and Proof-Pile and LongBench accuracy staying close to standard LoRA.

Load-bearing premise

The system assumes that a token's column sum of raw pre-softmax attention scores measures how much the model needs it, and that this ordering stays reliable as LoRA fine-tuning changes the weights; if the true set of informative tokens shifts during training, dropped tokens could take important learning signal with them.

Editorial extensions

If this is right

  • LeMo doubles the maximum sequence length that fits on one GPU for the tested models: OPT 1.3B goes from 16K to 32K and OPT 350M from 32K to 64K without recomputation or offloading.
  • Activation memory savings grow with sequence length, averaging 38.2% at 4K and 50.5% at 8K against LoRA, because longer sequences are more redundant.
  • Fine-tuning becomes faster as well as cheaper, with up to 1.36x end-to-end speedup and up to 2.04x when token sparsity is combined with hidden-dimension sparsity.
  • The overhead of detecting sparsity is small enough to be practical: predictors reach 95.13% average recall, are pruned to about a third of their original size, and add negligible time and memory.
  • LeMo composes with existing memory techniques instead of competing with them: the paper notes compatibility with activation recomputation, offloading, and other optimization methods, so the gains can be stacked.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the paper tracks whether the predictor's 95.13% recall holds as LoRA weights drift during fine-tuning; a direct test would be to measure overlap between predicted and true sparsity patterns at every training step, and if recall decays as weights move far from the base model, the accuracy guarantees would need re-validation.
  • The same token-level sparsity could plausibly be applied during continued pre-training or instruction tuning on 128K-plus contexts, where activation memory grows linearly; the paper does not evaluate that regime, but the mechanism does not depend on LoRA specifically.
  • Because LeMo changes which training tokens the model sees, it may act as a regularizer or curriculum that alters what the model learns beyond the perplexity numbers reported; comparing downstream task behavior of LeMo-tuned and dense-tuned models would separate sparsity effects from efficiency effects.
  • Combining LeMo with activation compression or recomputation could give multiplicative reductions beyond the reported 1.93x; the authors state compatibility but do not measure those combinations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LeMo, a system for long-context LLM fine-tuning that reduces activation memory by eliminating low-information tokens from attention and MLP computation. Token informativeness is defined from attention-score column sums (Eq. 1), tokens are eliminated block-wise with layer-specific thresholds (Algorithm 1), and the resulting sparsity patterns are approximated by lightweight per-layer predictors (Section 4.3) so that full attention scores need not be materialized. Kernel optimizations include permutation-free token movement and a segment-based method for the final loss gradient. The evaluation reports up to 1.93x memory reduction and up to 1.36x speedups over LoRA/LongLoRA on OPT and Llama models, with perplexity and LongBench results presented as showing no meaningful accuracy loss.

Significance. If the accuracy claims hold, the paper makes a useful systems contribution: it identifies a token-level sparsity axis for fine-tuning that is complementary to hidden-dimension sparsity, and it validates memory and speed effects with direct measurements on several GPU platforms and model families. The kernel-level ablations, the scalability measurements, and the explicit overhead accounting are concrete strengths. However, the central accuracy-preservation claim currently rests on single-run, mixed task-level scores and on an unvalidated assumption that predictors trained on base-model attention remain accurate throughout LoRA fine-tuning, so the significance is conditional on additional evidence.

major comments (4)
  1. [§6.2, Table 6] The LongBench evaluation consists of a single run per task and shows mixed task-level outcomes: gov_report drops from 27.44 to 25.92, qmsum from 22.64 to 20.33, and lcc from 71.28 to 70.32, while musique and 2wikimqa improve. No confidence intervals, significance tests, or aggregate statistics are reported, so the claim in §6.2 that LeMo achieves accuracy comparable to the original LoRA is not supported by the evidence as presented. Please report multiple seeds, error bars, and an aggregate summary (for example, mean and standard deviation across tasks, or per-category averages) so readers can assess whether the observed drops are within noise.
  2. [§4.3, §6.3] The reported predictor recall of 95.13% is computed against ground-truth informativeness derived from the base model's attention scores, but the predictors are used throughout LoRA fine-tuning while Q and K projections shift. The paper does not track whether recall, or the predicted sparsity pattern, stays accurate as training progresses, nor whether the layer-specific thresholds optimized in Algorithm 1 remain valid after weight updates. Because both the efficiency and the accuracy claims pass through this predictor, please add an experiment that re-measures predictor recall (or the overlap between predicted and true eliminated token sets) at several fine-tuning checkpoints, and report whether thresholds need to be re-tuned.
  3. [§4.2, Eq. (1)] The definition of token informativeness as the column sum of raw pre-softmax attention scores is an assumption that is never directly validated. The paper shows that predicted scores approximate these raw scores, but it does not show that eliminating tokens with low scores preserves loss or downstream accuracy better than, say, random or fixed-position elimination at the same sparsity. An ablation with random or fixed baselines at matched token-elimination ratios would establish that the proposed score, rather than mere sparsity, is responsible for the reported accuracy behavior.
  4. [§4.4] The segment-based peak-cutting method is described as processing each segment independently and later aggregating gradients, which is ambiguous: if each segment's loss is computed from a forward pass that starts at the segment boundary, the autoregressive context from earlier tokens is lost and the fine-tuning objective changes; if instead the prefix is recomputed or a KV cache is carried over, the memory-savings accounting needs to state that explicitly. Please clarify the implementation and provide a semantic-equivalence or loss-match check against the non-segmented objective.
minor comments (5)
  1. [Title and Abstract] The paper uses both 'LeMo' (title, abstract) and 'LEMO' (body, figures); please standardize the capitalization.
  2. [§4.3] The 'Comprehensive Overhead Analysis' discusses 'online inference', but the predictors are used during fine-tuning; the terminology should be 'runtime' or 'online fine-tuning'.
  3. [General] No code or artifact link is provided; given the systems nature of the contribution, providing one would aid reproducibility.
  4. [References] References [68] and [69] are duplicate entries of the same paper; please remove one.
  5. [§1] The phrase 'long-hanging fruit' should be 'low-hanging fruit'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the efficiency and accuracy claims rest on independent measurements, with only routine validation-set threshold tuning and a non-load-bearing self-citation.

full rationale

LeMo's derivation chain is not circular. Token informativeness is defined in Eq. 1 from raw attention scores, token elimination removes low-score blocks, predictors are trained to approximate those scores, and kernels implement the resulting sparse computation. The central efficiency claims (1.93x memory reduction, 1.36x speedup) are measured end-to-end against LoRA and LongLoRA on A800, A40, and 4090 platforms; these measurements do not reduce to any fitted parameter or definition. The accuracy-preservation claim is supported by held-out perplexity on PG19 and Proof-Pile and by LongBench scores, which are independent of the threshold-tuning objective used in Algorithm 1. Algorithm 1 does tune layer-specific thresholds using validation accuracy via finite-difference gradients, but this is standard hyperparameter fitting on a validation signal, not a prediction that is forced by construction, and the reported test metrics come from separate evaluation datasets. The paper cites the authors' prior work [65] on shadowy sparsity, but that citation supplies background and predictor-convergence evidence rather than the load-bearing justification for the measured results; the system is implemented and benchmarked against external baselines. No equation was found in which a claimed output is identical to an input by definition, and no fitted parameter is renamed as a prediction. Consequently, the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claim rests on empirical assumptions about language redundancy and on a heuristic informativeness measure. The memory measurements are direct, but the accuracy-maintenance result is partly produced by fitting layer thresholds to validation accuracy and by training predictors on the same model's attention distributions. The ledger therefore shows fitted parameters rather than a parameter-free derivation.

free parameters (4)
  • Layer-specific thresholds T_i = not reported
    Algorithm 1 tunes each layer's threshold by finite-difference gradient ascent on validation accuracy; these thresholds determine which tokens are eliminated and hence the memory/accuracy tradeoff.
  • Predictor weights (per-layer Q and K predictors) = not reported
    Trained offline to approximate attention-score informativeness; predictor recall is reported as 95.13%, but weights and training data are not released.
  • Token block size b and loss-segment size N = not reported
    Block size controls prediction overhead O(s^2/b^2) and elimination granularity; segment size sets the memory peak reduction to 1/N. No default values are given.
  • Elastic size transformation pruning schedule = not reported
    Neurons in predictors are pruned based on zero-frequency tracking; the pruning criterion and schedule are not specified, yet they determine the reported 64.6% average predictor size reduction.
assumptions (3)
  • domain assumption Natural language in long contexts is redundant enough that removing low-informativeness tokens from attention and MLP computation preserves fine-tuning quality.
    Stated in Section 3 as the basis for contextual token sparsity and supported only by empirical sparsity ratios (Table 3), not by a guarantee or bound.
  • ad hoc to paper The column sum of raw pre-softmax attention scores (Eq. 1) is a valid measure of token informativeness.
    Eq. 1 defines I(T_j) as sum_{i != j} Q_i K_j without validation against an optimal importance metric; the choice is a heuristic.
  • domain assumption Pattern predictors trained on base-model attention generalize across inputs and across fine-tuning updates.
    Section 4.3 assumes well-trained predictors can approximate sparsity patterns at runtime, but no experiment tracks predictor accuracy during fine-tuning as LoRA weights change.
invented entities (2)
  • Contextual Token Sparsity
    purpose: Claimed mechanism that token importance varies per input and per layer, making token-level elimination viable in fine-tuning.
    Introduced and visualized in Figures 4 and 7 and Table 3, but no independent falsifiable prediction outside this paper; it is a descriptive label for observed sparsity.
  • Shadowy Activation
    purpose: Concept explaining why hidden-dimension sparsity does not reduce activation memory: any participating token's activations are stored.
    An observational term that frames the paper's motivation, but it is not independently testable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning." pith.science (2026). https://pith.science/paper/MGKTOKFW

@misc{pith2026250109767,
  author       = {Pith},
  title        = {Pith review of: LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGKTOKFW}},
  note         = {Machine review of arXiv:2501.09767}
}
read the original abstract

The escalating demand for long-context applications has intensified the necessity of extending the LLM context windows. Despite recent fine-tuning approaches successfully expanding context lengths, their high memory footprints, especially for activations, present a critical practical limitation. Current parameter-efficient fine-tuning methods prioritize reducing parameter update overhead over addressing activation memory constraints. Similarly, existing sparsity mechanisms improve computational efficiency but overlook activation memory optimization due to the phenomenon of Shadowy Activation. In this paper, we propose LeMo, the first LLM fine-tuning system that explores and exploits a new token-level sparsity mechanism inherent in long-context scenarios, termed Contextual Token Sparsity. LeMo minimizes redundant token involvement by assessing the informativeness of token embeddings while preserving model accuracy. Specifically, LeMo introduces three key techniques: (1) Token Elimination, dynamically identifying and excluding redundant tokens across varying inputs and layers. (2) Pattern Prediction, utilizing well-trained predictors to approximate token sparsity patterns with minimal overhead. (3) Kernel Optimization, employing permutation-free and segment-based strategies to boost system performance. We implement LeMo as an end-to-end fine-tuning system compatible with various LLM architectures and other optimization techniques. Comprehensive evaluations demonstrate that LeMo reduces memory consumption by up to 1.93x and achieves up to 1.36x speedups, outperforming state-of-the-art fine-tuning systems.

Figures

Figures reproduced from arXiv: 2501.09767 by the authors.

Figure 1
Figure 1. Illustration of shadowy activation. (a) LoRA per [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Memory breakdown during LLM mixed-precision [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An illustrated example of LoRA for showcasing the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: LEMO overview. At each layer, token embeddings are first partitioned into blocks and fed into the pattern pre￾dictors (❷). Using the predicted informativeness scores from these predictors, the token elimination algorithm (❶) effec￾tively identifies and retains only the…
Figure 7
Figure 7. Figure 7: The average informativeness scores (normalized) of [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Comparison between naive token movement and [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Memory peak during loss gradient computation, [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 12
Figure 12. Figure 12: Memory footprints comparison on A800. as sequence length increases. This aligns with the observation that longer text sequences typically exhibit greater redun￾dancy. The enhanced efficiency of LEMO largely extends the fine-tuning sequence length achievable under GPU …
Figure 13
Figure 13. Figure 13: End-to-end speedup of LEMO on A800 and A40 [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 17
Figure 17. Figure 17: Performance of LEMO’s permutation-free kernel. Origin LeMo 70 60 50 60 50 Memory Footprint (GB) 40 [PITH_FULL_IMAGE:figures/full_fig_p011_17.png]
Figure 16
Figure 16. Figure 16: (a) Training loss curve on LongAlign (LA) / Red [PITH_FULL_IMAGE:figures/full_fig_p011_16.png]
Figure 19
Figure 19. Figure 19: Performance improvements from two extensions. [PITH_FULL_IMAGE:figures/full_fig_p012_19.png]
Figure 20
Figure 20. Figure 20: Strong scalability evaluation of LEMO. Extension 2: Sparsity-sensitive Offload. We compare the performance of our sparsity-sensitive offloading against a naive uniform offloading strategy. Unlike the naive approach, LEMO considers the varying sparsity ratios across la…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 21 canonical work pages

  1. [65]

    Long expo- sure: Accelerating parameter-efficient fine-tuning for llms under shadowy sparsity

    Tuowei Wang, Kun Li, Zixu Hao, Donglin Bai, Ju Ren, Yaoxue Zhang, Ting Cao, and Mao Yang. Long expo- sure: Accelerating parameter-efficient fine-tuning for llms under shadowy sparsity. In 2024 SC24: Interna- tional Conference for High Performance Computing, Networking, Storage and Analysis SC, pages 1176–1193. IEEE Computer Society, 2024

    work page 2024

  2. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [2]

    Deep learning using rectified linear units (relu)

    AF Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018

    arXiv 2018

  4. [3]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023

    2023

  5. [4]

    LongAlign: A recipe for long context alignment of large language mod- els

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language mod- els. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376–1395, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics

    work page 2024

  6. [5]

    Longbench: A bilingual, multi- task benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multi- task benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023

    arXiv 2023

  7. [6]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  8. [7]

    Token merging: Your vit but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022

    arXiv 2022

Show all 78 references
  1. [8]

    Language models are few-shot learn- ers

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learn- ers. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [9]

    Actnn: Reducing training memory footprint via 2-bit activation compressed training

    Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, and Joseph Gon- zalez. Actnn: Reducing training memory footprint via 2-bit activation compressed training. In International Conference on Machine Learning , pages 1803–1813. PMLR, 2021

    2021

  3. [10]

    Extending context window of large language models via positional interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023

    2023 arXiv

  4. [11]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016

    2016 arXiv

  5. [12]

    Longlora: Effi- cient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Effi- cient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023

    2023 arXiv

  6. [13]

    Llm-assisted content anal- ysis: Using large language models to support deductive coding

    Robert Chew, John Bollenbacher, Michael Wenger, Jes- sica Speer, and Annice Kim. Llm-assisted content anal- ysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924, 2023

    2023 arXiv

  7. [14]

    Redpajama: an open dataset for training large language models, 2023

    Together Computer. Redpajama: an open dataset for training large language models, 2023

    2023

  8. [15]

    FlashAttention-2: Faster attention with bet- ter parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with bet- ter parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  9. [16]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  10. [17]

    Attention is naturally sparse with gaussian distributed input

    Yichuan Deng, Zhao Song, and Chiwun Yang. Attention is naturally sparse with gaussian distributed input. arXiv preprint arXiv:2404.02690, 2024

    2024 arXiv

  11. [18]

    Longrope: Extending llm context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024

    2024 arXiv

  12. [19]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  13. [20]

    Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning. Neural networks, 107:3–11, 2018

    2018

  14. [21]

    Ac-gc: Lossy activation compression with guaranteed convergence

    R David Evans and Tor Aamodt. Ac-gc: Lossy activation compression with guaranteed convergence. Advances in Neural Information Processing Systems, 34:27434– 27448, 2021

    2021

  15. [22]

    Data en- gineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Han- naneh Hajishirzi, Yoon Kim, and Hao Peng. Data en- gineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171, 2024. 13

    2024 arXiv

  16. [23]

    Metadata condi- tioning accelerates language model pre-training

    Tianyu Gao, Alexander Wettig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Metadata condi- tioning accelerates language model pre-training. arXiv preprint arXiv:2501.01956, 2025

    2025 arXiv

  17. [24]

    Seerattention: Learning intrinsic sparse attention in your llms

    Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Hay- den Kwok-Hay So, Ting Cao, Fan Yang, and Mao Yang. Seerattention: Learning intrinsic sparse attention in your llms. arXiv preprint arXiv:2410.13276, 2024

    2024 arXiv

  18. [25]

    Power-bert: Accelerating bert inference via progressive word-vector elimination

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In Interna- tional Conference on Machine Learning, pages 3690–

  19. [26]

    Autotm: Automatic tensor movement in heterogeneous memory systems using integer linear programming

    Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, and Venkatesh Akella. Autotm: Automatic tensor movement in heterogeneous memory systems using integer linear programming. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Prog...

    2020

  20. [27]

    Parameter- efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter- efficient transfer learning for nlp. In International con- ference on machine learning, pages 2790–2799. PMLR, 2019

    2019

  21. [28]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  22. [29]

    Swapad- visor: Pushing deep learning beyond the gpu memory limit via smart swapping

    Chien-Chin Huang, Gu Jin, and Jinyang Li. Swapad- visor: Pushing deep learning beyond the gpu memory limit via smart swapping. In Proceedings of the Twenty- Fifth International Conference on Architectural Support for Programming Languages and Operating Systems , pages 1341–1355, 2020

    2020

  23. [30]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    arXiv

  24. [31]

    Llmlingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023

    2023 arXiv

  25. [32]

    Length-adaptive transformer: Train once with length drop, use anytime with search

    Gyuwan Kim and Kyunghyun Cho. Length-adaptive transformer: Train once with length drop, use anytime with search. arXiv preprint arXiv:2010.07003, 2020

    2010 arXiv

  26. [33]

    Learned token pruning for transformers

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gho- lami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 784–794, 2022

    2022

  27. [34]

    Reducing activation re- computation in large transformer models

    Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation re- computation in large transformer models. Proceedings of Machine Learning and Systems, 5:341–353, 2023

    2023

  28. [35]

    Efficient rematerialization for deep networks

    Ravi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, and Joshua Wang. Efficient rematerialization for deep networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  29. [36]

    Inducing and exploiting activation sparsity for fast inference on deep neural networks

    Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine...

    2020

  30. [37]

    {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache manage- ment

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. {InfiniGen}: Efficient generative inference of large language models with dynamic {KV} cache manage- ment. In 18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 24), pages 155– 172, 2024

    2024

  31. [38]

    Prefix-tuning: Optimiz- ing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimiz- ing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  32. [39]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201, 2023

    2023 arXiv

  33. [40]

    The lazy neuron phenomenon: On emergence of activation sparsity in transformers

    Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint arXiv:2210.06313, 2022

    arXiv

  34. [41]

    Retrievalattention: Accel- erating long-context llm inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhen- hua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accel- erating long-context llm inference via vector retrieval. arXiv preprint arXiv:2409.10516, 2024

    2024 arXiv

  35. [42]

    Scaling laws of rope-based extrapolation

    Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023. 14

    2023 arXiv

  36. [43]

    Gact: Activation com- pressed training for generic network architectures

    Xiaoxuan Liu, Lianmin Zheng, Dequan Wang, Yukuo Cen, Weize Chen, Xu Han, Jianfei Chen, Zhiyuan Liu, Jie Tang, Joey Gonzalez, et al. Gact: Activation com- pressed training for generic network architectures. In International Conference on Machine Learning, pages 14139–14152. PMLR, 2022

    2022

  37. [44]

    Deja vu: Con- textual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Con- textual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pages 22137–22176....

    2023

  38. [45]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  39. [46]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gre- gory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  40. [47]

    Relu strikes back: Exploiting activation sparsity in large lan- guage models

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large lan- guage models. arXiv preprint arXiv:2310.04564, 2023

    arXiv

  41. [48]

    Adapler: Speeding up inference by adaptive length reduction

    Ali Modarressi, Hosein Mohebbi, and Mohammad Taher Pilehvar. Adapler: Speeding up inference by adaptive length reduction. arXiv preprint arXiv:2203.08991 , 2022

    2022 arXiv

  42. [49]

    Leave no context behind: Efficient infinite con- text transformers with infini-attention

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite con- text transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024

    2024 arXiv

  43. [50]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  44. [51]

    ChatGPT: Get instant answers, find creative inspiration, learn something new

    OpenAI. ChatGPT: Get instant answers, find creative inspiration, learn something new. https://openai. com/chatgpt, 2022

    2022

  45. [52]

    Yarn: Efficient context window ex- tension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and En- rico Shippole. Yarn: Efficient context window ex- tension of large language models. arXiv preprint arXiv:2309.00071, 2023

    2023 arXiv

  46. [53]

    Ca- puchin: Tensor-based gpu memory management for deep learning

    Xuan Peng, Xuanhua Shi, Hulin Dai, Hai Jin, Weiliang Ma, Qian Xiong, Fan Yang, and Xuehai Qian. Ca- puchin: Tensor-based gpu memory management for deep learning. In Proceedings of the Twenty-Fifth Inter- national Conference on Architectural Support for Pro- gramming Languages ...

    2020

  47. [54]

    Training large neural networks with constant memory using a new execution algorithm

    Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jin- wen Xi, and Sujeeth Bharadwaj. Training large neural networks with constant memory using a new execution algorithm. arXiv preprint arXiv:2002.05645, 2020

    2002 arXiv

  48. [55]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  49. [56]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems , 34:13937–13949, 2021

    2021

  50. [57]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  51. [58]

    Prediction and entropy of printed english

    Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal , 30(1):50–64, 1951

    1951

  52. [59]

    Flexgen: High-throughput generative inference of large language models with a single gpu

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, page...

    2023

  53. [60]

    Powerinfer: Fast large language model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Princi- ples, pages 590–606, 2024

    2024

  54. [61]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  55. [62]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  56. [63]

    15 Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 15 Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  57. [64]

    Focused transformer: Contrastive training for context scaling

    Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miło´s. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Pro- cessing Systems, 36, 2024

    2024

  58. [66]

    What is linguistic redun- dancy

    EC Wit and Marie Gillette. What is linguistic redun- dancy. University of Chicago, 1999

    1999

  59. [67]

    Infllm: Training-free long-context extrap- olation for llms with an efficient context memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrap- olation for llms with an efficient context memory. InThe Thirty-eighth Annual Conference on Neural Information Processing Sy...

    2024

  60. [69]

    Effective long-context scaling of foundation mod- els

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation mod- els. arXiv preprint arXiv:2309.16039, 2023

    2023 arXiv

  61. [70]

    Tr-bert: Dynamic token reduction for accelerating bert inference

    Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. Tr-bert: Dynamic token reduction for accelerating bert inference. arXiv preprint arXiv:2105.11618, 2021

    2021 arXiv

  62. [71]

    A survey on recent advances in llm-based multi-turn dialogue systems

    Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. A survey on recent advances in llm-based multi-turn dialogue systems. arXiv preprint arXiv:2402.18013, 2024

    2024 arXiv

  63. [72]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283–17297, 2020

    2020

  64. [73]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  65. [74]

    Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning

    Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023

    2023 arXiv

  66. [75]

    Long context com- pression with activation beacon

    Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context com- pression with activation beacon. arXiv preprint arXiv:2401.03462, 2024

    2024 arXiv

  67. [76]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  68. [77]

    H2o: Heavy- hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy- hitter oracle for efficient generative inference of large language models. Advances in Neural Information Pro- cessing Sy...

    2023

  69. [78]

    In- former: Beyond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021

    2021

  70. [79]

    Pose: Efficient context window extension of llms via positional skip-wise train- ing

    Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wen- hao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise train- ing. arXiv preprint arXiv:2309.10400, 2023. 16

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.