{"id":"409d1c0b-1730-45e4-a7db-6b25aac316a2","arxiv_id":"2507.03865","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OrthoRank selects tokens for computation by their orthogonality to the sink token in normalized hidden states, yielding better perplexity and accuracy than layer pruning at matched sparsity without additional training.","lead":"This paper introduces OrthoRank, a training-free method that speeds up large language model inference by skipping computation for tokens that resemble the sink token, usually the first token, while updating tokens that are most orthogonal to it. Tests on Llama, Mistral, and Mixtral models show lower perplexity and higher zero-shot accuracy than layer pruning at the same nominal sparsity.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The motivating sink-convergence pattern is established from one 101-token WikiText sample with newlines removed; if it fails on long or delimiter-rich inputs, the orthogonality ranking loses its justification and the LongBench advantage is already mixed at 8192 context.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing premise: the entire method rests on the claim that, after l_sink, the first-token sink is nearly static and all other tokens steadily align with it, so tokens far from it are the ones that most need computation. That premise is supported only by a single 101-token WikiText-2 prompt with newlines deliberately removed, a setting far removed from the long, structured, delimiter-rich inputs used in LongBench and real applications. My read does not change the verdict: the paper has broad and mostly consistent empirical evidence, a simple and plausible mechanism, and useful ablations, but the external validity of the motivating observation is not established. The paper should remain conditional, with the additional requirement that the authors either demonstrate the pattern on long/structured contexts or soften the LongBench claim. I would not reject: the method may well work, and the ablation against random and reverse selection is genuine supporting evidence. The concern is not internal inconsistency but an under-tested generalization. I also note that Table 3's 2048 and 4096 rows are identical, which is likely a reporting error and should be corrected regardless. On the current evidence, a conditional acceptance with these checks is the right outcome.","tokens_in":20732,"tokens_out":9008,"duration_ms":108635,"concrete_test":"Run the Section 2 analysis on a diverse set of long contexts: 20 sequences of length 512-4096 sampled from C4, LongBench, and a chat-templated dataset, without stripping newlines, for Llama-2-13B and Llama-3-8B. Verify (i) cos(¯h^{l1}_0, ¯h^{l2}_0) remains near 1 after l_sink and (ii) cos(¯h^l_0, ¯h^l_i) increases at least generally for l after l_sink. Then rerun the 10%/20% perplexity and LongBench comparisons on these contexts. If the similarity pattern is absent or the gains over SLEB reverse on long/delimiter-rich inputs, the central premise is not general and the abstract's LongBench claim should be retracted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2's Obs. (1) and (2) are the entire motivation for ranking tokens by first-token orthogonality, yet they are demonstrated on a single 101-token WikiText-2 prompt (Appendix F) for Llama-2-13B and Mistral-7B, with Figures 9-11 extending the same short prompt to a few other Llama/Mistral models. Appendix F explicitly removes '\\n' because newlines are known to create additional attention sinks (Sun et al., 2024; Son et al., 2024). Real inference, and LongBench in particular, contains long documents, chat templates, separators, and many newline/delimiter tokens; prior work shows these can act as competing sinks. If the first token is not the dominant, nearly static sink on such inputs, the ranking in Eq. (4) has no stated justification: OrthoRank measures orthogonality to the wrong reference, and the empirical gains could shrink or reverse. Consistent with this risk, Table 3 shows OrthoRank already loses to SLEB on LongBench at 8192 context (29.77 vs 30.42 at 10% sparsity), and the abstract's unqualified claim of superior LongBench performance is not supported by the table. The ablations against random/reverse selection and against attention-based selection show that the criterion is useful on the evaluated corpora, but they do not test whether the first-token sink premise generalizes across input types; they only test whether, given that fixed reference, orthogonality beats alternatives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the attention-sink phenomenon in the hidden-state geometry of LLMs. It reports two observations (Section 2): after a certain early layer, the normalized hidden states of non-sink tokens become increasingly cosine-similar to a nearly static first-token sink state, while the sink state itself changes little across layers. Based on these observations, Section 3 proposes OrthoRank, a training-free token-selection method that, at a subset of layers, skips query and FFN computation for tokens whose normalized hidden states are least orthogonal to the first-token sink, while still computing their key and value states. The selected layers are obtained by adapting a layer-pruning calibration procedure. Experiments (Tables 1-4, Figures 4-8) compare OrthoRank against SLEB and Shortened LLaMA at 10% and 20% nominal sparsity, reporting better C4 perplexity, zero-shot accuracy, and TruthfulQA scores in most cases, with comparable or slightly lower throughput gains. LongBench results are reported for Llama-3-8B and are mixed across context lengths and sparsity levels.","tokens_in":21114,"tokens_out":8686,"duration_ms":96200,"significance":"If the results hold, OrthoRank is a simple, training-free token-selection method that improves over layer pruning at the same nominal sparsity across many model families, with careful ablations against random, reverse, norm-based, and attention-based criteria. The derivation from the gradient of cosine similarity to a monotone function of 1-cos^2 (Eqs. 1-3) is correct under the stated equal-norm assumption, and the paper evaluates on held-out C4 and LongBench rather than fitting to the test set. The main value is empirical: the method is easy to implement, spans seven model sizes/families, and the ablation study (Table 5) gives meaningful evidence that the orthogonality criterion is more useful than several natural alternatives. The principal weaknesses are the narrow empirical basis of the motivating geometric observation (one 101-token prompt with newlines removed), an under-specified layer-selection calibration, and a nominal-sparsity comparison that may overstate efficiency because unselected tokens still compute K and V states.","major_comments":[{"comment":"Observations (1) and (2) are the sole motivation for ranking tokens by orthogonality to the first token, yet they are demonstrated on a single 101-token WikiText-2 prompt with newlines removed, for a limited set of Llama and Mistral models. The paper itself notes that newlines are known to create additional attention sinks, and LongBench contains long documents, separators, and many newline or delimiter tokens. If the first-token sink is not the dominant, nearly static reference on such inputs, Eq. (4) ranks tokens against the wrong reference. This concern is not merely hypothetical: Table 3 shows that at 8192 context and 10% sparsity OrthoRank loses to SLEB (29.77 vs. 30.42), while the abstract states unqualified superior LongBench performance. Additionally, Appendix F says the context is limited to 101 tokens, but Appendix B figures refer to token positions 500 and 1000; the evidence for long-range behavior is therefore internally inconsistent. The authors should either demonstrate that Observations (1)-(2) hold on long and delimiter-rich inputs, or qualify the LongBench claims and the corresponding abstract sentence.","section":"Section 2, Appendix F, Table 3"},{"comment":"The layer-selection component is under-specified. The text says the authors 'adopt the layer evaluation method from layer pruning' and 'evaluate them to identify the optimal token selection layers,' but it does not state the calibration dataset, the evaluation criterion, the order in which layers are evaluated, or how the reported 15% and 30% layer selections were obtained for each model. Since all main results depend on this selection, the method as described cannot be reproduced, and it is unclear whether the gains come from token selection itself or from a favorable choice of layers. Please provide a complete algorithmic description of the calibration procedure, ideally including the selected layer lists or a sensitivity analysis of the choice.","section":"Section 3.2, Section 4.1"},{"comment":"At the same nominal sparsity, OrthoRank does not reduce compute as much as layer pruning because unselected tokens still compute K and V projections and their key-value states are used in attention. The paper itself shows that computing KV for unselected tokens is needed for good performance (Table 5, rows 6 vs. 7). Thus a nominal 20% sparsity corresponds to a smaller true FLOP reduction for OrthoRank than for SLEB, which removes entire layers. The paper reports throughput in Section 4.1 and Figure 6, but it does not compare methods at matched measured throughput or report effective compute savings. Without this, the 'same sparsity ratio' comparisons in Tables 1-4 may be biased in favor of OrthoRank. Please report FLOP-matched or latency-matched comparisons, or at least quantify the KV overhead in the reported throughput numbers.","section":"Section 4.7.3, Algorithm 1, Figure 6"},{"comment":"The step from gradient magnitude to token importance is an assumption, not a consequence of the derivation. Eq. (3) only shows that, under the equal-norm assumption, the squared gradient norm is proportional to (1 - cos^2); it does not show that a token with larger (1 - cos^2) benefits more from being updated. The ablations in Table 5 and Figure 4 provide meaningful empirical support for this design choice, and I do not regard the absence of a proof as disqualifying. However, the text in Sections 3.1 and 6 states that the criterion is 'validated' and 'optimized both theoretically and empirically,' which overstates what Eq. (3) establishes. Please either provide a derivation of why gradient magnitude should equal token importance, or explicitly label this as an empirical design choice justified by the ablations.","section":"Section 3.1, Eq. (3)"}],"minor_comments":[{"comment":"There is a typo: 'we used the the C4 validation set' should read 'we used the C4 validation set.'","section":"Section 4.2"},{"comment":"The text says that Row 3 uses hidden states and Row 5 uses normalized hidden states, but in Table 5 Row 3 is labeled with normalized hidden states (Norm↓ on ¯h_i) and Row 5 is labeled with unnormalized hidden states (Orthogonal↑ on h_i). The row references and the table labels are inconsistent and should be corrected.","section":"Section 4.7.2, Table 5"},{"comment":"The captions of Figures 9-11 mention token positions 500 and 1000, while Appendix F states that the visualization used a context length of 101 tokens. Please clarify whether these figures were generated with a longer context or whether the captions are inaccurate.","section":"Appendix B, Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution to efficient inference, and the central idea is simple enough to be practically useful. However, the LongBench claim in the abstract is contradicted by one of the reported settings, the layer-selection procedure is not reproducible as described, and the nominal-sparsity comparison may overstate the efficiency benefit given that unselected tokens still compute KV states. These are fixable with additional experiments and careful rewriting, but they currently prevent acceptance. The limited motivating evidence (one 101-token prompt) is also worth addressing with more diverse contexts, since it is the conceptual foundation of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives you a genuinely simple, training-free token-pruning method that improves over layer pruning at matched nominal sparsity, and the hidden-state sink analysis is new. The core criterion—keep tokens whose normalized hidden states are most orthogonal to the first token—is well-defined, and the ablations are persuasive: orthogonality beats random, reverse, attention-based, and norm-based selection across several model families. C4 perplexity and zero-shot accuracy gains over SLEB are consistent and non-trivial (e.g., Llama-2-13B at 10% sparsity, 7.79 to 7.39). The geometric derivation is sound under the equal-norm assumption, and the method is drop-in for existing LLMs, which is exactly what many practitioners want.\n\nNow the soft spots. The motivating observations (Obs. 1 and 2) come from a single 101-token WikiText-2 prompt with newlines stripped, because newlines create competing sinks. Real inputs—and LongBench in particular—are long, delimiter-rich, and full of newlines. The paper's own Table 3 shows OrthoRank losing to SLEB at 8192 context at 10% sparsity (29.77 vs 30.42), which directly contradicts the abstract's unqualified claim of superior LongBench performance. That overclaim should be fixed. The leap from gradient magnitude to token importance is also heuristic; the equal-norm assumption is empirically plausible but only approximately verified, and the layer-selection step borrows the opaque calibration procedure from SLEB without much disclosure. None of this is fatal—the method clearly works on the evaluated corpora—but the stated generality rests on a thin empirical base.\n\nWhere does that leave us? This is a solid engineering contribution, not a new theoretical principle. The hidden-state sink dynamics are an interesting empirical observation, and the method is simple and reproducible enough that a practitioner could reimplement it from the paper. The main missing pieces are code or a detailed layer-selection protocol, error bars, and a toned-down LongBench claim. This deserves a serious referee: the method is useful, the experiments are broad, and the flaws are fixable. I'd send it to review and ask for conditional acceptance after those revisions.\n\nFor the reading group, it's a good case study in how empirical motivation can overreach its evidence, so I'd bring it up for discussion, though I wouldn't rank it among the year's most exciting papers.","headline":"A practical training-free token-pruning method backed by a new hidden-state sink observation, but the motivating generality is thin and the LongBench claim oversells the data.","tokens_in":21577,"tokens_out":1919,"would_cite":true,"duration_ms":21903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in transformer language models, tokens whose normalized hidden states are most orthogonal to the near-static first-token 'sink' are the ones that most need computing, and that selecting them at chosen layers yields…","keywords":["attention sink","token selection","efficient LLM inference","layer pruning","sink token orthogonality","hidden state similarity","dynamic token selection"],"falsifier":"A direct test is to run the same cosine-similarity measurement on contexts of thousands of tokens and on inputs with delimiter sinks ('.' or '\n'). If the similarity between other tokens and the first token stops increasing with depth, or the first token's own state drifts, then OrthoRank's ranking no longer tracks how much a token still needs to change, and its perplexity advantage over random token selection at matched sparsity should shrink or invert; showing that the advantage disappears on long contexts would settle the claim.","tokens_in":20573,"feed_emoji":"⚡","tokens_out":12310,"duration_ms":113879,"temperature":0.7,"pith_summary":"The paper sets out to show that token-level computation can be saved in large language models without any extra training, by reading a geometric signal that already exists in the hidden states. Its central claim is that, after an early 'attention sink' layer, every token's normalized hidden state drifts toward the first token — which itself barely moves — so the tokens that most need computing at a layer are precisely the ones still most orthogonal to that first token. On that basis it proposes OrthoRank, which at selected layers computes only the tokens ranked highest by orthogonality to the sink token, letting the rest pass through the residual path while still contributing their key and value states. The payoff the paper claims is that this beats whole-layer pruning at the same nominal sparsity: lower C4 perplexity and higher zero-shot accuracy across most tested models, with comparable throughput and no fine-tuning.","feed_headline":"Orthogonality to the sink token beats whole-layer pruning","feed_subtitle":"Compute only tokens that still diverge from the near-fixed first-token state; same sparsity, lower perplexity, no retraining.","key_machinery":"The load-bearing object is the token–sink orthogonality score: the absolute inner product $|\\bar{h}_0^\\top \\bar{h}_i|$ between the normalized pre-attention hidden states of the first ('sink') token and each other token, evaluated at the layers where selection is applied. The paper motivates it through the gradient identity $\\|\\partial \\cos(\\bar{h}_0,\\bar{h}_i)/\\partial \\bar{h}_i\\|^2 \\propto 1-\\cos^2(\\bar{h}_0,\\bar{h}_i)$, which holds once non-sink tokens are assumed to have roughly equal norms and converts 'speed of movement toward the sink' into 'orthogonality to the sink.' The selection step keeps the smallest-$|\\bar{h}_0^\\top \\bar{h}_i|$ tokens for full computation in a subset of layers, which is what turns a static layer-pruning decision into a per-token, per-layer dynamic choice without any learned router.","core_discovery":"On the paper's own terms, the discovery is a hidden-state counterpart to the attention-sink phenomenon plus a use for it. After the sink layer, the cosine similarity between the normalized hidden states of the first token and any other token rises steadily with depth, while the first token's own normalized state stays almost fixed; the paper concludes that other tokens are converging toward a stationary sink. Defining token importance as the speed of that convergence gives, under a near-equal-norm assumption, a quantity proportional to $1-\\cos^2(\\bar{h}_0,\\bar{h}_i)$, so importance becomes proximity to orthogonality with the sink token, implemented as the smallest absolute inner products $|\\bar{h}_0^\\top \\bar{h}_i|$. Applied to a subset of layers chosen with the layer-pruning procedure, with unselected tokens skipping query and feed-forward computation but keeping key–value participation, this selection is reported to improve perplexity and zero-shot accuracy over layer pruning at matched sparsity — for example, Llama-2-13B at 10% sparsity on C4 drops from 7.79 to 7.39 — while throughput gains stay nearly proportional to sparsity.","pith_inferences":["If the sink-convergence pattern is general, the orthogonality score is a free per-token 'remaining computation' signal: the same score could drive per-input choice of which layers to apply selection to, or flag tokens whose states have already fused toward the sink as safe to evict from the KV cache.","The paper deliberately restricts the sink to the first token; applying the same ranking against delimiter sinks ('.' or '\n') would test whether the mechanism is about the sink concept itself or about position 0 specifically.","The comparison with attention-based selection points to a general distinction: attention scores measure a token's influence on others, while sink-orthogonality measures a token's own unfinished evolution, and the latter is what predicts whether updating the token changes the output.","Long-context behavior is the open front: the motivating observation is measured on a 101-token prompt, so the strongest test is whether the perplexity advantage over random selection survives at the 2048–8192 token ranges where the paper's own LongBench results are least consistent."],"forward_implications":["At matched 10% and 20% sparsity, OrthoRank reports lower C4 perplexity than SLEB layer pruning on nearly every model tested; Llama-2-13B at 10% sparsity goes from 7.79 to 7.39.","Mean zero-shot accuracy across PIQA, WinoGrande, HellaSwag, ARC-Challenge, and ARC-Easy is higher than layer pruning in most configurations, and LongBench accuracy improves in most settings.","Throughput scales nearly proportionally with sparsity because skipping a token's query and feed-forward work removes real FLOPs while the KV cache stays intact, giving a 1.18x gain at 20% sparsity in the direct comparison.","Dropping the key–value computation for unselected tokens degrades performance, so the scheme keeps KV for all tokens while omitting only their query and feed-forward updates, preserving inter-token interaction without paying for full computation.","The advantage is bounded: the paper reports parity with layer pruning at 40% sparsity and a sharp perplexity rise at 50%, and recommends keeping sparsity below 40%."],"supporting_citations":[{"why":"Names and characterizes the attention sink on the first token, the phenomenon this paper extends from attention maps to hidden-state geometry.","marker":"(Xiao et al., 2024b)"},{"why":"Explains sink emergence through massive activations and layer normalization and identifies delimiter sinks; the paper excludes '\n' tokens from its observation because of this work.","marker":"(Sun et al., 2024)"},{"why":"Provides SLEB, the iterative layer-pruning baseline, the layer-evaluation procedure, and the sparsity protocol that OrthoRank is compared against.","marker":"(Song et al., 2024)"},{"why":"Provides Shortened LLaMA, the one-shot depth-pruning baseline compared at 10% sparsity without fine-tuning.","marker":"(Kim et al., 2024)"},{"why":"Supplies WikiText-2, the source of the 101-token prompt behind the motivating observations and the calibration set for layer selection.","marker":"(Merity et al., 2022)"},{"why":"Supplies the C4 validation set on which the main perplexity comparisons against layer pruning are reported.","marker":"(Raffel et al., 2020)"},{"why":"Supplies LongBench, the long-context benchmark on which OrthoRank is compared with SLEB across calibration lengths.","marker":"(Bai et al., 2024)"},{"why":"Frames the contrast with trained-router computation allocation; Mixture-of-Depths is the approach OrthoRank claims to match without any training.","marker":"(Raposo et al., 2024)"}],"fun_headline_variants":["Sink-token orthogonality picks tokens, beats layer pruning","Compute only tokens orthogonal to the sink; better perplexity","Token selection via sink orthogonality outperforms layer pruning","OrthoRank: keep tokens that diverge from the sink token","Sink-token trick: prune tokens, not layers, for faster LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the measured pattern — non-sink tokens drifting toward a nearly stationary first-token sink in normalized hidden-state space — is a general property of transformer language models, even though it is demonstrated on a single 101-token WikiText-2 prompt; if that pattern is an artifact of the short prompt, or does not hold for long contexts or for models with delimiter sinks, the orthogonality ranking loses its foundation.","fun_headline_variants_meta":{"raw":{"variants":["Sink-token orthogonality picks tokens, beats layer pruning","Compute only tokens orthogonal to the sink; better perplexity","Token selection via sink orthogonality outperforms layer pruning","OrthoRank: keep tokens that diverge from the sink token","Sink-token trick: prune tokens, not layers, for faster LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1460,"prompt_tokens":1027,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":643,"tokens_out":433,"duration_ms":4560,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:01:17.752821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to run the same cosine-similarity measurement on contexts of thousands of tokens and on inputs with delimiter sinks ('.' or '\n'). If the similarity between other tokens and the first token stops increasing with depth, or the first token's own state drifts, then OrthoRank's ranking no longer tracks how much a token still needs to change, and its perplexity advantage over random token selection at matched sparsity should shrink or invert; showing that the advantage disappears on long contexts would settle the claim.","supporting_citations":[{"cited_title":"Z., and Liu, Z","cited_arxiv_id":null,"evidence_quote":"Explains sink emergence through massive activations and layer normalization and identifies delimiter sinks; the paper excludes '\n' tokens from its observation because of this work."},{"cited_title":"Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks","cited_arxiv_id":null,"evidence_quote":"Provides SLEB, the iterative layer-pruning baseline, the layer-evaluation procedure, and the sparsity protocol that OrthoRank is compared against."},{"cited_title":"Pointer sentinel mixture models","cited_arxiv_id":null,"evidence_quote":"Supplies WikiText-2, the source of the 101-token prompt behind the motivating observations and the calibration set for layer selection."},{"cited_title":"Longbench: A bilingual, multitask benchmark for long context understanding","cited_arxiv_id":null,"evidence_quote":"Supplies LongBench, the long-context benchmark on which OrthoRank is compared with SLEB across calibration lengths."}],"review_version":1}