OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference

Dokwan Oh; Jaehoon Oh; Seungjun Shin

REVIEW 4 major objections 3 minor 2 cited by

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that in transformer language models, tokens whose normalized hidden states are most orthogonal to the near-static first-token 'sink' are the ones that most need computing, and that selecting them at chosen layers yields…

desk verdict A practical training-free token-pruning method backed by a new hidden-state sink observation, but the motivating generality is thin and the LongBench claim oversells the data. read the letter →

arxiv 2507.03865 v2 pith:KKTEJASM submitted 2025-07-05 cs.CL cs.AIcs.LG

Seungjun Shin , Jaehoon Oh , Dokwan Oh This is my paper

classification cs.CLcs.AIcs.LG

keywords attentionsinktokenselectionefficientLLMinferencelayerpruningorthogonalityhiddenstatesimilaritydynamic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that token-level computation can be saved in large language models without any extra training, by reading a geometric signal that already exists in the hidden states. Its central claim is that, after an early 'attention sink' layer, every token's normalized hidden state drifts toward the first token — which itself barely moves — so the tokens that most need computing at a layer are precisely the ones still most orthogonal to that first token. On that basis it proposes OrthoRank, which at selected layers computes only the tokens ranked highest by orthogonality to the sink token, letting the rest pass through the residual path while still contributing their key and value states. The payoff the paper claims is that this beats whole-layer pruning at the same nominal sparsity: lower C4 perplexity and higher zero-shot accuracy across most tested models, with comparable throughput and no fine-tuning.

What carries the argument

The load-bearing object is the token–sink orthogonality score: the absolute inner product $|\bar{h}_0^\top \bar{h}_i|$ between the normalized pre-attention hidden states of the first ('sink') token and each other token, evaluated at the layers where selection is applied. The paper motivates it through the gradient identity $\|\partial \cos(\bar{h}_0,\bar{h}_i)/\partial \bar{h}_i\|^2 \propto 1-\cos^2(\bar{h}_0,\bar{h}_i)$, which holds once non-sink tokens are assumed to have roughly equal norms and converts 'speed of movement toward the sink' into 'orthogonality to the sink.' The selection step keeps the smallest-$|\bar{h}_0^\top \bar{h}_i|$ tokens for full computation in a subset of layers, which is what turns a static layer-pruning decision into a per-token, per-layer dynamic choice without any learned router.

What would settle it

A direct test is to run the same cosine-similarity measurement on contexts of thousands of tokens and on inputs with delimiter sinks ('.' or ' '). If the similarity between other tokens and the first token stops increasing with depth, or the first token's own state drifts, then OrthoRank's ranking no longer tracks how much a token still needs to change, and its perplexity advantage over random token selection at matched sparsity should shrink or invert; showing that the advantage disappears on long contexts would settle the claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a hidden-state counterpart to the attention-sink phenomenon plus a use for it. After the sink layer, the cosine similarity between the normalized hidden states of the first token and any other token rises steadily with depth, while the first token's own normalized state stays almost fixed; the paper concludes that other tokens are converging toward a stationary sink. Defining token importance as the speed of that convergence gives, under a near-equal-norm assumption, a quantity proportional to $1-\cos^2(\bar{h}_0,\bar{h}_i)$, so importance becomes proximity to orthogonality with the sink token, implemented as the smallest absolute inner products $|\bar{h}_0^\top \bar{h}_i|$. Applied to a subset of layers chosen with the layer-pruning procedure, with unselected tokens skipping query and feed-forward computation but keeping key–value participation, this selection is reported to improve perplexity and zero-shot accuracy over layer pruning at matched sparsity — for example, Llama-2-13B at 10% sparsity on C4 drops from 7.79 to 7.39 — while throughput gains stay nearly proportional to sparsity.

Load-bearing premise

The load-bearing premise is that the measured pattern — non-sink tokens drifting toward a nearly stationary first-token sink in normalized hidden-state space — is a general property of transformer language models, even though it is demonstrated on a single 101-token WikiText-2 prompt; if that pattern is an artifact of the short prompt, or does not hold for long contexts or for models with delimiter sinks, the orthogonality ranking loses its foundation.

Editorial extensions

If this is right

At matched 10% and 20% sparsity, OrthoRank reports lower C4 perplexity than SLEB layer pruning on nearly every model tested; Llama-2-13B at 10% sparsity goes from 7.79 to 7.39.
Mean zero-shot accuracy across PIQA, WinoGrande, HellaSwag, ARC-Challenge, and ARC-Easy is higher than layer pruning in most configurations, and LongBench accuracy improves in most settings.
Throughput scales nearly proportionally with sparsity because skipping a token's query and feed-forward work removes real FLOPs while the KV cache stays intact, giving a 1.18x gain at 20% sparsity in the direct comparison.
Dropping the key–value computation for unselected tokens degrades performance, so the scheme keeps KV for all tokens while omitting only their query and feed-forward updates, preserving inter-token interaction without paying for full computation.
The advantage is bounded: the paper reports parity with layer pruning at 40% sparsity and a sharp perplexity rise at 50%, and recommends keeping sparsity below 40%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the sink-convergence pattern is general, the orthogonality score is a free per-token 'remaining computation' signal: the same score could drive per-input choice of which layers to apply selection to, or flag tokens whose states have already fused toward the sink as safe to evict from the KV cache.
The paper deliberately restricts the sink to the first token; applying the same ranking against delimiter sinks ('.' or ' ') would test whether the mechanism is about the sink concept itself or about position 0 specifically.
The comparison with attention-based selection points to a general distinction: attention scores measure a token's influence on others, while sink-orthogonality measures a token's own unfinished evolution, and the latter is what predicts whether updating the token changes the output.
Long-context behavior is the open front: the motivating observation is measured on a 101-token prompt, so the strongest test is whether the perplexity advantage over random selection survives at the 2048–8192 token ranges where the paper's own LongBench results are least consistent.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives you a genuinely simple, training-free token-pruning method that improves over layer pruning at matched nominal sparsity, and the hidden-state sink analysis is new. The core criterion—keep tokens whose normalized hidden states are most orthogonal to the first token—is well-defined, and the ablations are persuasive: orthogonality beats random, reverse, attention-based, and norm-based selection across several model families. C4 perplexity and zero-shot accuracy gains over SLEB are consistent and non-trivial (e.g., Llama-2-13B at 10% sparsity, 7.79 to 7.39). The geometric derivation is sound under the equal-norm assumption, and the method is drop-in for existing LLMs, which is exactly what many practitioners want.

Now the soft spots. The motivating observations (Obs. 1 and 2) come from a single 101-token WikiText-2 prompt with newlines stripped, because newlines create competing sinks. Real inputs—and LongBench in particular—are long, delimiter-rich, and full of newlines. The paper's own Table 3 shows OrthoRank losing to SLEB at 8192 context at 10% sparsity (29.77 vs 30.42), which directly contradicts the abstract's unqualified claim of superior LongBench performance. That overclaim should be fixed. The leap from gradient magnitude to token importance is also heuristic; the equal-norm assumption is empirically plausible but only approximately verified, and the layer-selection step borrows the opaque calibration procedure from SLEB without much disclosure. None of this is fatal—the method clearly works on the evaluated corpora—but the stated generality rests on a thin empirical base.

Where does that leave us? This is a solid engineering contribution, not a new theoretical principle. The hidden-state sink dynamics are an interesting empirical observation, and the method is simple and reproducible enough that a practitioner could reimplement it from the paper. The main missing pieces are code or a detailed layer-selection protocol, error bars, and a toned-down LongBench claim. This deserves a serious referee: the method is useful, the experiments are broad, and the flaws are fixable. I'd send it to review and ask for conditional acceptance after those revisions.

For the reading group, it's a good case study in how empirical motivation can overreach its evidence, so I'd bring it up for discussion, though I wouldn't rank it among the year's most exciting papers.

Referee Report

4 major / 3 minor

Summary. The paper studies the attention-sink phenomenon in the hidden-state geometry of LLMs. It reports two observations (Section 2): after a certain early layer, the normalized hidden states of non-sink tokens become increasingly cosine-similar to a nearly static first-token sink state, while the sink state itself changes little across layers. Based on these observations, Section 3 proposes OrthoRank, a training-free token-selection method that, at a subset of layers, skips query and FFN computation for tokens whose normalized hidden states are least orthogonal to the first-token sink, while still computing their key and value states. The selected layers are obtained by adapting a layer-pruning calibration procedure. Experiments (Tables 1-4, Figures 4-8) compare OrthoRank against SLEB and Shortened LLaMA at 10% and 20% nominal sparsity, reporting better C4 perplexity, zero-shot accuracy, and TruthfulQA scores in most cases, with comparable or slightly lower throughput gains. LongBench results are reported for Llama-3-8B and are mixed across context lengths and sparsity levels.

Significance. If the results hold, OrthoRank is a simple, training-free token-selection method that improves over layer pruning at the same nominal sparsity across many model families, with careful ablations against random, reverse, norm-based, and attention-based criteria. The derivation from the gradient of cosine similarity to a monotone function of 1-cos^2 (Eqs. 1-3) is correct under the stated equal-norm assumption, and the paper evaluates on held-out C4 and LongBench rather than fitting to the test set. The main value is empirical: the method is easy to implement, spans seven model sizes/families, and the ablation study (Table 5) gives meaningful evidence that the orthogonality criterion is more useful than several natural alternatives. The principal weaknesses are the narrow empirical basis of the motivating geometric observation (one 101-token prompt with newlines removed), an under-specified layer-selection calibration, and a nominal-sparsity comparison that may overstate efficiency because unselected tokens still compute K and V states.

major comments (4)

[Section 2, Appendix F, Table 3] Observations (1) and (2) are the sole motivation for ranking tokens by orthogonality to the first token, yet they are demonstrated on a single 101-token WikiText-2 prompt with newlines removed, for a limited set of Llama and Mistral models. The paper itself notes that newlines are known to create additional attention sinks, and LongBench contains long documents, separators, and many newline or delimiter tokens. If the first-token sink is not the dominant, nearly static reference on such inputs, Eq. (4) ranks tokens against the wrong reference. This concern is not merely hypothetical: Table 3 shows that at 8192 context and 10% sparsity OrthoRank loses to SLEB (29.77 vs. 30.42), while the abstract states unqualified superior LongBench performance. Additionally, Appendix F says the context is limited to 101 tokens, but Appendix B figures refer to token positions 500 and 1000; the evidence for long-range behavior is therefore internally inconsistent. The authors should either demonstrate that Observations (1)-(2) hold on long and delimiter-rich inputs, or qualify the LongBench claims and the corresponding abstract sentence.
[Section 3.2, Section 4.1] The layer-selection component is under-specified. The text says the authors 'adopt the layer evaluation method from layer pruning' and 'evaluate them to identify the optimal token selection layers,' but it does not state the calibration dataset, the evaluation criterion, the order in which layers are evaluated, or how the reported 15% and 30% layer selections were obtained for each model. Since all main results depend on this selection, the method as described cannot be reproduced, and it is unclear whether the gains come from token selection itself or from a favorable choice of layers. Please provide a complete algorithmic description of the calibration procedure, ideally including the selected layer lists or a sensitivity analysis of the choice.
[Section 4.7.3, Algorithm 1, Figure 6] At the same nominal sparsity, OrthoRank does not reduce compute as much as layer pruning because unselected tokens still compute K and V projections and their key-value states are used in attention. The paper itself shows that computing KV for unselected tokens is needed for good performance (Table 5, rows 6 vs. 7). Thus a nominal 20% sparsity corresponds to a smaller true FLOP reduction for OrthoRank than for SLEB, which removes entire layers. The paper reports throughput in Section 4.1 and Figure 6, but it does not compare methods at matched measured throughput or report effective compute savings. Without this, the 'same sparsity ratio' comparisons in Tables 1-4 may be biased in favor of OrthoRank. Please report FLOP-matched or latency-matched comparisons, or at least quantify the KV overhead in the reported throughput numbers.
[Section 3.1, Eq. (3)] The step from gradient magnitude to token importance is an assumption, not a consequence of the derivation. Eq. (3) only shows that, under the equal-norm assumption, the squared gradient norm is proportional to (1 - cos^2); it does not show that a token with larger (1 - cos^2) benefits more from being updated. The ablations in Table 5 and Figure 4 provide meaningful empirical support for this design choice, and I do not regard the absence of a proof as disqualifying. However, the text in Sections 3.1 and 6 states that the criterion is 'validated' and 'optimized both theoretically and empirically,' which overstates what Eq. (3) establishes. Please either provide a derivation of why gradient magnitude should equal token importance, or explicitly label this as an empirical design choice justified by the ablations.

minor comments (3)

[Section 4.2] There is a typo: 'we used the the C4 validation set' should read 'we used the C4 validation set.'
[Section 4.7.2, Table 5] The text says that Row 3 uses hidden states and Row 5 uses normalized hidden states, but in Table 5 Row 3 is labeled with normalized hidden states (Norm↓ on ¯h_i) and Row 5 is labeled with unnormalized hidden states (Orthogonal↑ on h_i). The row references and the table labels are inconsistent and should be corrected.
[Appendix B, Appendix F] The captions of Figures 9-11 mention token positions 500 and 1000, while Appendix F states that the visualization used a context length of 101 tokens. Please clarify whether these figures were generated with a longer context or whether the captions are inaccurate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the orthogonality criterion is transparently derived from an empirical observation and validated against held-out perplexity, zero-shot, and LongBench targets; the narrow motivating data in Appendix F is a robustness risk, not a circular step.

full rationale

The derivation chain is self-contained rather than circular. Section 2 reports empirical observations (Obs. 1 and 2) about cosine similarity between normalized hidden states of the sink token and other tokens; these are measurements on model activations, not definitions of the target metric. Section 3.1 then defines token importance as the gradient magnitude of cosine similarity with the sink token and algebraically reduces it to 1 - cos^2, i.e., orthogonality; this is an explicit definitional derivation, not a hidden equivalence, and the paper tests it against random, reverse, norm-based, and attention-based selection on perplexity and accuracy. The evaluation uses WikiText-2 only for selecting layers and measures C4 and LongBench, so there is no fitted-input-called-prediction pattern. The only author-overlap citation is SLEB (Song et al., 2024, co-authored by Jaehoon Oh), which is used as a baseline and comparison protocol rather than as support for the orthogonality premise; SLEB is an externally published and reproducible method, so this self-citation is not load-bearing. Appendix F does state that the motivating Figures 2-3 use only a 101-token WikiText-2 prompt with newlines removed; that is a genuine generalization limitation for real-world inputs with delimiter sinks, but it weakens external validity rather than making the derivation circular. No step in the paper reduces by construction to its own input, and the central claim is supported by held-out empirical comparisons.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method's core criterion is parameter-free and validated on held-out tasks, but it rests on empirical observations about sink-token behavior, an equal-norm assumption for normalized hidden states, and a heuristic equivalence between gradient magnitude and computational importance. The token selection ratio (0.333) and the selected layers are calibration choices.

free parameters (2)

token_selection_ratio = 0.333
Fixed fraction of tokens computed in selected layers; chosen by hand and ablated in Section 4.7.4 over a range from 0 to 0.5.
selected_layers = not disclosed per model
Layer indices receiving token selection are determined per model by incremental evaluation on WikiText-2, following SLEB; the actual chosen layers are not listed in the paper.

assumptions (4)

domain assumption The first token acts as an attention sink after an early layer (l_sink), consistent across models.
Invoked in Sections 1-2; based on Xiao et al. 2024b and Sun et al. 2024, and confirmed by the paper's own figures.
domain assumption Obs. (1) and Obs. (2): cosine similarity between sink and other normalized hidden states increases after l_sink, and the sink token's normalized hidden state is nearly static across layers.
Section 2, Figures 2-3; based on a single 101-token WikiText-2 prompt and a few model families; used to motivate the orthogonality criterion.
domain assumption Non-sink normalized hidden states have approximately equal norms, so ranking by |h_0^T h_i| equals ranking by |cos|.
Appendix C, Eq. (7), and Figure 12; needed to replace the gradient norm expression with 1-cos^2 and the inner product proxy.
ad hoc to paper The magnitude of the gradient of cos(h0, hi) with respect to hi measures how much token i benefits from computation.
Section 3.1, Eq. (1)-(3); the paper calls it 'speed at which the token moves toward the sink token', but no network dynamics are derived; it is an interpretive heuristic validated only empirically.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference." pith.science (2026). https://pith.science/paper/KKTEJASM

@misc{pith2026250703865,
  author       = {Pith},
  title        = {Pith review of: OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KKTEJASM}},
  note         = {Machine review of arXiv:2507.03865}
}

read the original abstract

Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each token. Recent studies have revealed the sink token, which receives disproportionately high attention despite their limited semantic role. In this paper, we first expand the relationship between the sink token and other tokens, moving beyond attention to explore their similarity in hidden states, considering the layer depth. We observe that as the layers get deeper, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens consistently are directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Finally, through extensive experiments, we demonstrated that our method results in lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also achieving superior performance on LongBench.

Figures

Figures reproduced from arXiv: 2507.03865 by the authors.

**Figure 1.** Overview of our approach (OrthoRank). OrthoRank first determines the orthogonality of tokens to the sink token after normalization at each layer. Based on this, the top K tokens are selected for updates, while the remaining tokens bypass the layer except for KV calculations. To explore this, we analyze the internal workings of LLMs to determine whether each token requires an update within a layer. Our focus is on on… view at source ↗

**Figure 2.** (a, d) Cosine similarity between the normalized hidden states of the sink token (h¯0) and other tokens of Llama-2-13B and Mistral-7B. lsink is layer 4 and layer 2, respectively. (b-c, e-f) Conceptual representation of the relationship between the sink token (red line) and other tokens (blue lines) at layer right after lsink and the final layer. After the attention sink, as layers progress, the cosine similarity betw… view at source ↗

**Figure 3.** (a-b) Cosine similarity between the normalized hidden states of the sink token across layers. (c-d) Cosine similarity between the normalized hidden states of another token (postiton 50) across layers. The red boundary represents the layers after layer lsink. The sink token shows similar values not only with adjacent layers but also with distant layers, as confirmed through (a) and (b). In contrast, the another toke… view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: Layer-wise performance by token selection criteria. Our method achieves lower perplexity (PPL) across all layers except the final layer compared to both random token selection and the Reverse criteria, which selects tokens in the opposite manner to our approach. Thus, …

**Figure 5.** Figure 5: Comparison under the same sparsity. through the residual path. Specifically, unselected tokens are excluded from query computation, the query input to the attention mechanism, and the feed-forward network (FFN) computations. 4. Experiments 4.1. Implementation details W…

**Figure 6.** Figure 6: Performance vs. Throughput Trade-offs: (a) Log perplexity (↓) and (b) zero-shot accuracy (↑) under throughput improvements [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: compares OrthoRank’s performance under varying sparsity levels using log perplexity on the C4 dataset (Figure 8a) and mean accuracy on zero-shot tasks (Figure 8b). OrthoRank consistently achieves lower perplexity than SLEB across all sparsity levels except 40%, demon…

**Figure 9.** Figure 9: Cosine similarity across layers for different models and token positions. Each row represents a token position: 0 (sink), 50, and 100. The red boundary represents the layers after layer lsink. The sink token shows similar values not only with adjacent layers but also w…

**Figure 10.** Figure 10: Cosine similarity changes between tokens across layers. Each column corresponds to a different model: Llama-2-7B, 13B, 70B; Meta-Llama-3-8B; and Llama-3.1-70B. Left: Cosine similarity between tokens at position 0 and 91–100. Right: Cosine similarity between tokens at …

**Figure 11.** Figure 11: Cosine similarity changes between diverse tokens across layers. Each column corresponds to a different model: Llama-2-7B, 13B, 70B; and Meta-Llama-3-8B. Left: Cosine similarity between tokens at position 0 and {1, 10, 100, 1000}. Right: Cosine similarity between token…

**Figure 12.** Figure 12: Norms of the normalized hidden states. This figure plots the norms of the normalized hidden states ∥h¯i∥ for various token positions in the sequence. The red line represents the norm of the hidden state at position 0 (the sink token), while the blue lines correspond t…

**Figure 13.** Figure 13: shows the perplexity differences for layer-wise manner, comparing our orthogonal token selection method to both reverse (green) and random selection (blue). OrthoRank demonstrated superior performance compared to both Random and Reverse (opposite) approaches in most m…

**Figure 14.** Figure 14: The relationship between throughput improvements and perplexity (C4). Each point represents a different sparsity level, showing a sharp increase in perplexity at the 50% sparsity level 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
cs.LG 2025-08 conditional novelty 6.0 of 10

A learned router sends about 90% of tokens through a linear projection path instead of quadratic attention, keeps the MLP for every token, and matches dense Transformer accuracy at roughly 85% of the FLOPs.
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
cs.CL 2026-02 unverdicted novelty 5.0 of 10

Attention-sink weight is recast as an implicit MoE router per head, motivating a sink-aware head-balancing loss that yields small, consistent benchmark gains across three attention variants but rests on a definitional...

Reference graph

Works this paper leans on

40 extracted references · 19 canonical work pages · cited by 2 Pith papers

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

Bae, S., Ko, J., Song, H., and Yun, S.-Y. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 5910--5924, 2023

work page 2023
[3]

Longbench: A bilingual, multitask benchmark for long context understanding

Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3119--3137, 2024

work page 2024
[4]

Piqa: Reasoning about physical commonsense in natural language

Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

2020
[5]

Spectral filters, dark signals, and attention sinks

Cancedda, N. Spectral filters, dark signals, and attention sinks. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4792--4808, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.acl-long.263

doi:10.18653/v1/2024.acl-long.263 2024
[6]

Sepllm: Accelerate large language models by compressing one segment into one separator

Chen, G., Shi, H., Li, J., Gao, Y., Ren, X., Chen, Y., Jiang, X., Li, Z., Liu, W., and Huang, C. Sepllm: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412.12094, 2024 a

arXiv 2024
[7]

Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism

Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. In Forty-first International Conference on Machine Learning, 2024 b

work page 2024
[8]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018

Show all 40 references

[9]

Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference

Del Corro, L., Del Giorno, A., Agarwal, S., Yu, B., Awadallah, A., and Mukherjee, S. Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference. arXiv preprint arXiv:2307.02628, 2023

2023 arXiv
[10]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[11]

Layerskip: Enabling early exit inference and self-speculative decoding

Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computat...

2024
[12]

When attention sink emerges in language models: An empirical view

Gu, X., Pang, T., Du, C., Liu, Q., Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges in language models: An empirical view. In The Twelfth International Conference on Learning Representations, 2024

2024
[13]

J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

2022
[14]

Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

2023 arXiv
[15]

Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024 a

2024 arXiv
[16]

D-llm: A token adaptive computing resource allocation strategy for large language models

Jiang, Y., Wang, H., Xie, L., Zhao, H., Qian, H., Lui, J., et al. D-llm: A token adaptive computing resource allocation strategy for large language models. Advances in Neural Information Processing Systems, 37: 0 1725--1749, 2024 b

2024
[17]

Shortened llama: A simple depth pruning for large language models

Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 2024

2024 arXiv
[18]

B io M istral: A collection of open-source pretrained large language models for medical domains

Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rouvier, M., and Dufour, R. B io M istral: A collection of open-source pretrained large language models for medical domains. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Li...

2024
[19]

Sparser is faster and less is more: Efficient sparse attention for long-range transformers

Lou, C., Jia, Z., Zheng, Z., and Tu, K. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747, 2024

2024 arXiv
[20]

Shortgpt: Layers in large language models are more redundant than you expect

Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024

2024 arXiv
[21]

Pointer sentinel mixture models

Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022

2022
[22]

Using an llm to help with code understanding

Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., and Myers, B. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp.\ 1--13, 2024

2024
[23]

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

2020
[24]

C., and Santoro, A

Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., and Santoro, A. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

2024 arXiv
[25]

L., Bhagavatula, C., and Choi, Y

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

2021
[26]

Q., Tay, Y., and Metzler, D

Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V. Q., Tay, Y., and Metzler, D. Confident adaptive language modeling. In Advances in Neural Information Processing Systems, 2022

2022
[27]

A., Dong, X., Heinrich, G., Breuel, T., Kautz, J., Krueger, D., and Molchanov, P

Siddiqui, S. A., Dong, X., Heinrich, G., Breuel, T., Kautz, J., Krueger, D., and Molchanov, P. A deeper look at depth pruning of llms. arXiv preprint arXiv:2407.16286, 2024

2024 arXiv
[28]

Prefixing attention sinks can mitigate activation outliers for large language model quantization

Son, S., Park, W., Han, W., Kim, K., and Lee, J. Prefixing attention sinks can mitigate activation outliers for large language model quantization. arXiv preprint arXiv:2406.12016, 2024

2024 arXiv
[29]

Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks

Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., et al. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. In Forty-first International Conference on Machine Learning, 2024

2024
[30]

Z., and Liu, Z

Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024

2024
[31]

Razorattention: Efficient kv cache compression through retrieval heads

Tang, H., Lin, Y., Lin, J., Han, Q., Hong, S., Ke, D., Yao, Y., and Wang, G. Razorattention: Efficient kv cache compression through retrieval heads. In The Thirteenth International Conference on Learning Representations, 2025

2025
[32]

J., Ting, D

Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. Large language models in medicine. Nature medicine, 29 0 (8): 0 1930--1940, 2023

1930
[33]

Llama 2: Open foundation and fine-tuned chat models

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[34]

Bloomberggpt: A large language model for finance

Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023

2023 arXiv
[35]

T., Peng, R., Wu, Q., and Wang, C

Wu, Y., Jia, F., Zhang, S., Li, H., Zhu, E., Wang, Y., Lee, Y. T., Peng, R., Wu, Q., and Wang, C. Mathchat: Converse to tackle challenging math problems with llm agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024

2024
[36]

Duoattention: Efficient long-context llm inference with retrieval and streaming heads

Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. In The Twelfth International Conference on Learning Representations, 2024 a

2024
[37]

Efficient streaming language models with attention sinks

Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024 b

2024
[38]

Yu, Z., Wang, Z., Fu, Y., Shi, H., Shaikh, K., and Lin, Y. C. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In International Conference on Machine Learning, 2024

2024
[39]

Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4791--4800, 2019

2019
[40]

Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

Bae, S., Ko, J., Song, H., and Yun, S.-Y. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 5910--5924, 2023

work page 2023

[3] [3]

Longbench: A bilingual, multitask benchmark for long context understanding

Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3119--3137, 2024

work page 2024

[4] [4]

Piqa: Reasoning about physical commonsense in natural language

Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 7432--7439, 2020

2020

[5] [5]

Spectral filters, dark signals, and attention sinks

Cancedda, N. Spectral filters, dark signals, and attention sinks. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4792--4808, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.acl-long.263

doi:10.18653/v1/2024.acl-long.263 2024

[6] [6]

Sepllm: Accelerate large language models by compressing one segment into one separator

Chen, G., Shi, H., Li, J., Gao, Y., Ren, X., Chen, Y., Jiang, X., Li, Z., Liu, W., and Huang, C. Sepllm: Accelerate large language models by compressing one segment into one separator. arXiv preprint arXiv:2412.12094, 2024 a

arXiv 2024

[7] [7]

Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism

Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. In Forty-first International Conference on Machine Learning, 2024 b

work page 2024

[8] [8]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

arXiv 2018