Pith. sign in

REVIEW 4 major objections 4 minor 35 references

Semantic token merging lets LLMs read up to 2.4x fewer tokens without losing quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A semantic-aware tokenizer that merges similar and low-entropy text spans cuts long-context token counts by up to 59% and inference latency by roughly 2x, with no reported quality loss.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A plausible efficiency idea whose central mechanism—how a pretrained LLM actually consumes the merged tokens—is never specified, so the headline compression and speedup numbers are unsupported as stated. the 4 major comments →

arxiv 2508.15190 v1 pith:FHE4BGGY submitted 2025-08-21 cs.CL cs.AI

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling

Dong Liu , Yanxuan Yu This is my paper
classification cs.CL cs.AI
keywords semantic tokenizationtoken merginglong-context language modelingsemantic entropyadaptive token granularityinference speedupKV cache compressiontoken redundancy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SemToken claims that long-context efficiency can be improved at the tokenization stage rather than in the attention kernel: a frozen lightweight encoder embeds each token, adjacent tokens with similar contextual embeddings are merged into single tokens, and local semantic entropy decides how coarse or fine the remaining tokens should be. The authors report that this preprocessing cuts token counts by up to 59% and inference latency by up to 1.9x (2.7x when combined with FlashAttention2) across WikiText-103, LongBench, BookSum, and ChartQA, without degrading perplexity or task accuracy. If correct, the insight is that a large share of long-context tokens are semantically redundant, and removing them before attention makes every downstream optimization—sparse attention, KV caching, quantization—cheaper by the same factor.

Core claim

The paper's central claim is that semantic redundancy in long contexts can be detected and removed at the input level without retraining the language model. SemToken does this in four steps: compute contextual fingerprints with a frozen encoder; form spans by greedily merging adjacent tokens whose fingerprint cosine similarity exceeds tau; score each span by the trace of the covariance of its token embeddings (semantic entropy); and under a token budget, keep the highest-entropy spans, merging the rest into single tokens. The resulting stream x'_C = merge(C) is asserted to be directly consumable by existing LLMs. On the paper's benchmarks, the compressed stream yields a 2.4x token reduction

What carries the argument

Local semantic merging driven by cosine-similarity span formation and entropy-ranked budget selection. The span-formation step uses contextual embeddings h_i from a frozen encoder and greedily extends a span while sim(h_t, h_j) > tau; the selection step scores each span by H(C) = Tr(Cov({h_i | x_i in C})) and keeps the top-B spans by entropy. The unspecified merge() operator is the load-bearing detail: it must produce a token that the pretrained model's embedding and vocabulary can represent.

Load-bearing premise

The load-bearing premise is that a pretrained LLM can directly consume the merged token stream produced by SemToken's unspecified merge() operation, and that merging spans whose embedding cosine similarity is above tau never discards task-critical information.

What would settle it

Reproduce the pipeline on LLaMA-2-7B with the exact merge() definition and the reported threshold tau on WikiText-103: if perplexity moves more than roughly 0.3 PPL away from the reported 17.0, or if feeding the compressed stream to the unmodified model requires any vocabulary or embedding patch, the direct-consumption claim is refuted. A second check: run with tau = 0.99 (almost no merging) and confirm that token count and perplexity match the BPE baseline.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Token and KV-cache memory shrink by roughly the compression ratio r = n'/n; for r in [0.3, 0.5], the paper predicts 2-3.3x compute and memory gains before any attention accelerator is applied.
  • SemToken composes multiplicatively with FlashAttention2 and H2O-style pruning, reaching 2.7x and 3.3x latency speedups in the reported stack.
  • Because compression happens before attention, every downstream cost—prefill and decode FLOPs, KV reads, cache lookups—is reduced by the same factor.
  • Ablations suggest each of the three modules (clustering, density scoring, autoregressive budgeting) contributes to both quality and speed; removing any one degrades perplexity or latency.
  • The method transfers to multimodal inputs: ChartQA shows 61% compression with equal exact-match accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported speedup treats the encoder's prefill cost as negligible; for long inputs the frozen encoder adds its own latency and memory, so the net gain in a real deployment depends on the encoder-to-LLM cost ratio, which the paper does not break out.
  • If merge() is defined as replacing a span with its mean embedding and a new pseudo-token id, the pretrained model must either have a corresponding embedding or an id-mapping layer; a testable extension is to define merge() as concatenating the span's first and last subword tokens with a length marker, which any BPE vocabulary already supports.
  • The same semantic-density scoring could serve as a cheap router for retrieval, telling a RAG system which passages to expand and which to keep compressed.
  • The trace-of-covariance entropy could be replaced by a single-number proxy like next-token log-probability from a small LM, which would remove the need for a separate embedding encoder.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SemToken, a semantic-aware tokenization framework intended to reduce token count, KV-cache memory, and inference latency for long-context LLMs. The method first computes contextual embeddings with a lightweight encoder, merges adjacent tokens whose cosine similarity exceeds a threshold τ, assigns fine or coarse granularity based on a trace-of-covariance entropy score, and applies a budget-constrained selection. The authors claim up to 2.4× token reduction, 1.9× speedup, and 2.7× latency reduction when combined with FlashAttention, with negligible or no quality degradation. Experiments are reported on WikiText-103, LongBench, BookSum, ChartQA, and other benchmarks, with ablations and integration with FlashAttention2 and H2O-style pruning.

Significance. The high-level idea—exploiting semantic redundancy in long-context token streams—is plausible and could be a useful complement to attention-level accelerators. The paper correctly identifies a real bottleneck: over-tokenization of repetitive or boilerplate spans. However, the contribution as written is not reproducible: the core merge operation is undefined, no representation rule for feeding merged tokens to a frozen LLM is given, no threshold values or budgets are reported, and no code, data, or error bars are provided. Moreover, the headline efficiency gains are definitional consequences of the compression ratio rather than independently measured results. The quality claims rest on an unvalidated assumption and on differences that are within plausible noise. For these reasons, the manuscript does not currently establish its central claim.

major comments (4)
  1. [Algorithm 1, line 22; §3.2; §3.5] The representation of merged tokens is never specified. Algorithm 1 outputs x'_C = merge(C) for each selected span C, but merge() is not defined anywhere. The paper repeatedly claims that SemToken's output 'can be consumed directly by existing LLMs' (§1, §3.2) and is 'deployable without retraining.' A frozen LLaMA-style model accepts token IDs through an embedding lookup, not arbitrary continuous vectors. If merge(C) selects an existing token ID, the model receives a token distribution sharply different from its training distribution, and no argument is given that quality is preserved. If merge(C) is a continuous embedding, standard forward passes cannot ingest it. Section 3.5's 'offset metadata' concerns decoding, not input-side consumption. This gap is load-bearing: the speedup and cache reductions in Tables 1–6 assume the model can actually condition on the compressed stream.
  2. [§3.6; Tables 1 and 3] The 'Compute Gain = n/n' = 1/r' and 'Memory Gain = n/n' = 1/r' equations are definitional restatements of the measured compression ratio r, not independent empirical confirmations. The claimed speedups and cache reductions in Tables 1 and 3 therefore do not validate the method; they simply re-express the token count reduction. Furthermore, the 'Stacked Speedup = g_token · g_attn' formula is not supported by the paper's own Table 3: SemToken + Vanilla gives 2.0× and BPE + FlashAttention2 gives 1.6×, but the combination gives 2.7×, not 3.2×. The integration is therefore not multiplicatively composable, and the paper offers no explanation. The latency numbers also omit the overhead of semantic embedding, clustering, and budget selection, so the claimed end-to-end speedups are likely optimistic even if the representation gap were closed.
  3. [§4.1, §4.2, Tables 1, 4, 5] The experimental reporting is insufficient to support the quality claims. No error bars, number of runs, seeds, or statistical significance tests are provided; differences such as 17.0 vs. 17.3 perplexity and +0.5 F1 are small enough to be noise. The baselines are also problematic: 'VQ-Tok' is cited to Ronen et al. (2023), a vision-transformer tokenization paper, and 'TofuTok' is cited to Fu et al. (2021), a video-language model paper; neither reference describes the token-compression baseline used. Table 4 additionally introduces 'BPE+Chunk (Ours)' with no description anywhere in the methodology. The definition of compression ratio is inconsistent across tables: Table 1 reports token count 41% for SemToken, while Tables 4 and 5 report compression ratio 59% for the same result. These issues make the empirical comparisons and the stated quality preservation claims unverifiable.
  4. [§3.1–§3.3; §4.4] The core quality premise is unvalidated. The paper assumes that (i) adjacent tokens with cosine similarity above τ are semantically redundant and can be merged without loss, and (ii) trace-of-covariance entropy H(T) correctly identifies spans that can be coarsely tokenized. No evidence is given to support either assumption. The ablations in §4.4 only show that removing clustering or density scoring changes PPL and latency; they do not show that cosine-similarity merging preserves task-relevant information better than a random or entropy-agnostic merging baseline at the same compression ratio. The free parameters τ, δ, B, and k are never reported, so the results cannot be reproduced or the claims tested.
minor comments (4)
  1. [Throughout] Inconsistent compression-ratio notation: Table 1 reports 'Token Count (%)' of 41% for SemToken while Tables 4 and 5 report 'Compression Ratio' of 59% for the same condition; BPE is reported as 0% compression in some tables and 100% token count in others. Please unify the metrics.
  2. [Figures 1–7] Several figures are referenced and captioned but appear to be missing or are only placeholder captions in the manuscript. The 3D heatmaps, radar chart, and trajectory plots cannot be assessed, and no data underlying them are provided.
  3. [§5] The concluding paragraph states future work on 'joint training of tokenization and modeling,' which suggests the current pipeline is not trained end-to-end; this should be reconciled with the earlier claim that the output 'can be consumed directly by existing LLMs.' Also, the sentence 'SemToken brdiges tokenization with semantic compression, it offers a practical tool...' contains a typo and a grammar error.
  4. [References] The Sennrich et al. reference is duplicated as 2016a and 2016b, and several related-work citations are to workshop papers or unreferenced preprints (e.g., 'Quickmerge++', 'Liu and Yu, 2025a'). Please provide complete and verifiable bibliographic details.

Circularity Check

0 steps flagged

No significant circularity: efficiency formulas are arithmetic identities given the compression ratio, and quality claims are empirical rather than definitional.

full rationale

SemToken's derivation chain contains no step in which a claimed prediction is equivalent to its own input. The efficiency identities in Section 3.6 (Compute Gain = n/n' = 1/r, Memory Gain = 1/r) are arithmetic consequences of the definition of the compression ratio r and the stated linear attention-cost assumption; they are presented as theoretical analysis, and the experimental latency and KV-cache numbers in Tables 1, 3, 4, 5, and 6 are measured independently (e.g., measured speedup 2.0x with an r-derived theoretical value of about 2.4x), so the efficiency claims are not fitted parameters renamed as predictions. The quality-preservation claim rests on an empirical thresholding heuristic (cosine similarity > tau, semantic entropy H(C) > delta), not on a definition of 'semantic equivalence' that presupposes downstream performance. The self-citations (Quickmerge++, PiKV, MKA, Tinyserve, LLMEasyQuant, etc.) appear in related-work positioning or as comparisons, and none is load-bearing for the method's derivation; no uniqueness theorem or ansatz is imported from prior author work to force the design. The main weakness—the unspecified merge(C) operation in Algorithm 1 line 22 and the unsupported assertion that the merged stream can be consumed directly by pretrained LLMs—is an implementation and information-preservation gap, not a circularity: the paper would still be making an empirical claim, albeit an unvalidated one, if merge were fully specified. Therefore no circular step is exhibited, and the paper is not circular by the standards of this review.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The efficiency numbers are controlled by unreported thresholds and budgets; the semantic-equivalence assumption is untested; and the merged-token representation is not specified. What remains is a sketch of an existing token-pruning idea.

free parameters (4)
  • similarity threshold tau = Not reported
    Algorithm 1 uses tau to decide which adjacent tokens merge into a span; no value, grid, or sensitivity analysis is given, and the reported compression ratio depends directly on it.
  • entropy threshold delta = Not reported
    Algorithm 1 uses delta to assign Fine versus Coarse granularity; no value or sensitivity analysis is reported.
  • token budget B = Not reported
    Algorithm 1 constrains output to m <= B; B is never quantified, and top-B selection determines how many tokens survive.
  • fingerprint window half-width k = Not reported
    Algorithm 1 computes h_i over [x_{i-k},...,x_{i+k}]; k is not given, though it controls how contextual the embeddings are.
axioms (5)
  • domain assumption Frozen contextual embeddings from a lightweight encoder capture semantic equivalence and redundancy usable for token merging.
    Section 3.1 defines H(T) with a frozen encoder f_theta; no evidence is provided that this encoder's geometry aligns with the target LLM's information needs.
  • ad hoc to paper Adjacent tokens with cosine similarity above tau are semantically redundant and can be merged without loss.
    Algorithm 1 line 8 merges tokens into a span when cosine similarity exceeds tau; this is the core compression criterion and is asserted, not derived.
  • ad hoc to paper Trace of the covariance of contextual embeddings is a valid semantic-entropy measure.
    Section 3.1 defines H(T) as Tr(Cov(...)); no calibration against human judgments, downstream loss, or an independent entropy measure is provided.
  • ad hoc to paper A pretrained LLM can directly consume merged tokens x'_C = merge(C) without retraining or adaptation.
    Section 3.2 and Algorithm 1 line 22; merge(C) is undefined and no embedding mapping for merged units is described.
  • domain assumption Attention compute and memory costs scale linearly with sequence length.
    Section 3.6 uses this to state compute and memory gain = n/n' = 1/r. It is true for per-step decode KV cost, but not for full prefill attention, which is quadratic; the paper applies it without qualification.
invented entities (2)
  • Semantic density score H(T) no independent evidence
    purpose: Decides which spans receive fine-grained tokens versus coarse compression.
    Introduced in Section 3.1 as Tr(Cov(...)) over encoder embeddings; no external validation or falsifiable prediction outside the pipeline is given.
  • Merged token representation x'_C = merge(C) no independent evidence
    purpose: Represents a whole span C as one token so the sequence shortens.
    Defined only symbolically in Algorithm 1 line 22; no rule for how multiple vocab items collapse into one embedding is provided, making integration with a pretrained model unsupported.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling." pith.science (2026). https://pith.science/paper/FHE4BGGY

@misc{pith2026250815190,
  author       = {Pith},
  title        = {Pith review of: SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FHE4BGGY}},
  note         = {Machine review of arXiv:2508.15190}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Tokenization plays a critical role in language modeling, yet existing approaches such as Byte-Pair Encoding (BPE) or WordPiece operate purely on frequency statistics, ignoring the underlying semantic structure of text. This leads to over-tokenization of semantically redundant spans and underutilization of contextual coherence, particularly in long-context scenarios. In this work, we propose \textbf{SemToken}, a semantic-aware tokenization framework that jointly reduces token redundancy and improves computation efficiency. SemToken first extracts contextual semantic embeddings via lightweight encoders and performs local semantic clustering to merge semantically equivalent tokens. Then, it allocates heterogeneous token granularity based on semantic density, allowing finer-grained tokenization in content-rich regions and coarser compression in repetitive or low-entropy spans. SemToken can be seamlessly integrated with modern language models and attention acceleration methods. Experiments on long-context language modeling benchmarks such as WikiText-103 and LongBench show that SemToken achieves up to $2.4\times$ reduction in token count and $1.9\times$ speedup, with negligible or no degradation in perplexity and downstream accuracy. Our findings suggest that semantic structure offers a promising new axis for optimizing tokenization and computation in large language models.

Figures

Figures reproduced from arXiv: 2508.15190 by Dong Liu, Yanxuan Yu.

Figure 1
Figure 1. Figure 1: Semantic density visualization across text [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Token importance scores for different query [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: visualizes the theoretical efficiency gains across different compression ratios and shows how SemToken’s benefits compound with existing attention accelerators for multiplicative gains [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: 3D semantic density heatmap showing information richness across text regions. Red areas indicate high [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Radar chart comparing SemToken with base [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of semantic clusters showing [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages · 5 internal anchors

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. http://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding

    Pith/arXiv arXiv 2024

  5. [5]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. http://arxiv.org/abs/2004.05150 Longformer: The long-document transformer

    Pith/arXiv arXiv 2020

  6. [6]

    Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2022. http://arxiv.org/abs/2009.14794 Rethinking attention with performers

    Pith/arXiv arXiv 2022

  7. [7]

    NVIDIA Corporation. 2024. Cutlass: Cuda templates for linear algebra subroutines. https://github.com/NVIDIA/cutlass

    work page 2024

  8. [8]

    Tri Dao. 2023. https://arxiv.org/abs/2307.08691 Flashattention-2: Faster attention with better parallelism and work partitioning . arXiv preprint arXiv:2307.08691

    Pith/arXiv arXiv 2023

  9. [9]

    Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. 2021. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681

    Pith/arXiv arXiv 2021

  10. [10]

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. http://arxiv.org/abs/2107.00910 Learned token pruning for transformers

    Pith/arXiv arXiv 2022

  11. [11]

    Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. http://arxiv.org/abs/2105.08209 Booksum: A collection of datasets for long-form narrative summarization

    Pith/arXiv arXiv 2022

  12. [12]

    Taku Kudo. 2018. http://arxiv.org/abs/1804.10959 Subword regularization: Improving neural network translation models with multiple subword candidates

    Pith/arXiv arXiv 2018

  13. [13]

    https://openreview.net/pdf?id=dMdxHd0tRf Quickmerge++: Token merging with autoregressive prior

    Dong Liu and Yanxuan Yu. https://openreview.net/pdf?id=dMdxHd0tRf Quickmerge++: Token merging with autoregressive prior

    work page

  14. [14]

    Dong Liu and Yanxuan Yu. 2025 a . http://arxiv.org/abs/2406.19657 Llmeasyquant: Scalable quantization for parallel and distributed llm inference

    arXiv 2025

  15. [15]

    Dong Liu and Yanxuan Yu. 2025 b . https://openreview.net/forum?id=sOdtl4jLci Tinyserve: Query-aware cache selection for efficient LLM inference . In ICML 2025 Workshop on Methods and Opportunities at Small Scale

    work page 2025

  16. [16]

    Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, and Xuhong Wang. 2025 a . https://openreview.net/forum?id=hHoK1kBPd9 Pi KV : KV cache management system for moe architecture . In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models

    work page 2025

  17. [17]

    Dong Liu, Yanxuan Yu, Xuhong Wang, Ben Lengerich, and Ying Nian Wu. 2025 b . https://openreview.net/forum?id=r1GbqYMJys MKA : Memory-keyed attention for efficient long-context reasoning . In ICML 2025 Workshop on Long-Context Foundation Models

    work page 2025

  18. [18]

    Dong Liu, Yanxuan Yu, Yite Wang, Jing Wu, Zhongwei Wan, Sina Alinejad, Benjamin Lengerich, and Ying Nian Wu. 2025 c . http://arxiv.org/abs/2409.01990 Designing large foundation models for efficient training and inference: A survey

    Pith/arXiv arXiv 2025

  19. [19]

    Dong Liu, Jiayi Zhang, Yifan Li, Yanxuan Yu, Ben Lengerich, and Ying Nian Wu. 2025 d . http://arxiv.org/abs/2505.20353 Fastcache: Fast caching for diffusion transformer through learnable linear approximation

    arXiv 2025

  20. [20]

    Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. 2016. https://arxiv.org/abs/1609.04186 Neural machine translation with supervised attention . arXiv preprint arXiv:1609.04186

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  21. [21]

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024 a . Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38--56

    work page 2024

  22. [22]

    Zeyuan Liu, Ziyu Huan, Xiyao Wang, Jiafei Lyu, Jian Tao, Xiu Li, Furong Huang, and Huazhe Xu. 2024 b . http://arxiv.org/abs/2406.07381 World models with hints of large language models for goal achieving

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. http://arxiv.org/abs/2203.10244 Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Pith/arXiv arXiv 2022

  24. [24]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. https://arxiv.org/abs/2309.00071 Yarn: Efficient context window extension of large language models . arXiv preprint arXiv:2309.00071

    Pith/arXiv arXiv 2023

  25. [25]

    Rae, Anna Potapenko, Siddhant M

    Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2019. http://arxiv.org/abs/1911.05507 Compressive transformers for long-range sequence modelling

    Pith/arXiv arXiv 2019

  26. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. http://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer

    Pith/arXiv arXiv 2023

  27. [27]

    Tomer Ronen, Omer Levy, and Avram Golbert. 2023. http://arxiv.org/abs/2304.00287 Vision transformers with mixed-resolution tokenization

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [29]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 b . https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715--1725, Berlin, Germany. Association for Computational Linguistics

    doi:10.18653/v1/p16-1162 2016

  29. [30]

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.746 Dataset cartography: Mapping and diagnosing datasets with training dynamics . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275...

    doi:10.18653/v1/2020.emnlp-main.746 2020

  30. [31]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. http://arxiv.org/abs/2302.13971 Llama: Open and efficient foundation language models

    Pith/arXiv arXiv 2023

  31. [32]

    Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2023. http://arxiv.org/abs/2307.03170 Focused transformer: Contrastive training for context scaling

    Pith/arXiv arXiv 2023

  32. [33]

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason R...

    Pith/arXiv arXiv 2016

  33. [34]

    Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. http://arxiv.org/abs/1910.14142 Discourse-aware neural extractive text summarization

    work page internal anchor Pith review Pith/arXiv arXiv 2020

  34. [35]

    Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. Tr-bert: Dynamic token reduction for accelerating bert inference. arXiv preprint arXiv:2105.11618

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  35. [36]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. http://arxiv.org/abs/2306.14048 H _2 o: Heavy-hitter oracle for efficient generative inference of large language models

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.