Pith. sign in

REVIEW 3 major objections 5 minor 19 references

DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under KV-cached decoding, next-token decisions depend on a small subspace of the hidden state shared across tasks; removing it only at decode time collapses accuracy far beyond matched non-shared or prefill-estimated controls

desk verdict Good protocol, overclaimed headline: the decode-time causal effect is real and well-controlled, but 'shared across tasks' holds only for a majority threshold, not literally across tasks. read the letter →

arxiv 2607.20469 v1 pith:NGLWRPVJ submitted 2026-05-21 cs.AI

Zishan Shao , Lixun Zhang , Kangning Cui , Yixiao Wang , Ting Jiang , Hancheng Ye , Qinsi Wang , Zhixu Du
show 5 more authors
Yuzhe Fu Fan Yang Danyang Zhuo Yiran Chen Hai Helen Li
This is my paper
classification cs.AI
keywords KV-cacheddecodingdecode-timehiddenstatessharedsubspacepooledPCAcausalablationactivationsteeringprefill-decodemismatchLLMinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Under KV-cached decoding—the regime where a model has already read the whole prompt and then emits one token at a time over cached attention states—this paper claims that next-token decisions run through a small subspace of the hidden state that is shared across many tasks. It contributes a protocol that estimates this subspace from decode-time activations across 13 benchmarks, then ablates it only during decoding, leaving the prompt-processing prefill pass untouched. Removing the tiny shared set (a few percent of the layer width) collapses decision accuracy, while dimension- and energy-matched non-shared directions and prefill-estimated directions stay near baseline; restoring only the removed component rescues the lost decisions. The paper then draws practical consequences for activation steering: steering vectors overlap the shared channel, so projecting them off gives a robustness knob, and validating steering candidates at decode time ranks held-out utility better than prefill-based proxies. A reader should care because the claim, if right, replaces 'the decode-time state' with a concrete, compact, causally tested object that can be monitored or repaired without touching prefill.

What carries the argument

Three pieces carry the argument. First, the decode-aligned shared basis: pooled PCA of task-centered decode states gives a cross-task basis, and a per-task 'relative variance' threshold τ plus a 'used by at least m tasks' rule selects a compact shared set. Second, decode-only projection removal (h̃ = h − αQQᵀh) applied solely on single-token decode steps, prefill untouched, with dimension- and energy-matched controls that rule out a high-variance confound. Third, patchback: record the shared component during normal decoding, restore only that component in the ablated run, with random, energy-matched, and non-shared patches as specificity controls.

What would settle it

Hold the decode-only intervention and energy-matched controls fixed but swap the shared-set rule for the strict all-tasks intersection: for Llama-2-7B that set is empty (reported in the appendix), so the protocol predicts no causal collapse where the majority rule produced drops of tens of points—whichever behavior the experiment matches would separate a genuine cross-task channel from a tuned threshold. The complementary check replaces the PCA-threshold estimator with a threshold-free or nonlinear one (e.g., canonical correlations across task state matrices) and asks whether decode-only remov

Watch

Extended reading notes

Core claim

The central claim: during KV-cached decoding, next-token decisions causally depend on a small subspace S of the decode-time hidden state shared across tasks. S comes from pooled PCA of task-centered decode states, keeping directions whose per-task relative variance clears a threshold for most tasks. Though a few percent of layer width, decode-only removal of S collapses accuracy far beyond dimension- and energy-matched controls; leave-one-task-out re-estimation keeps the effect, and patching back only the removed component rescues flips, even with donors from other tasks. The channel is decode-specific: prefill-estimated bases are near-orthogonal to it and behave like random controls.

Load-bearing premise

The load-bearing premise is that the task-general decision channel is exactly the small set of directions a hand-set variance cutoff picks out inside a common coordinate system built from the model's decode-time states; if that structure is not linear, or the cutoff mostly reflects the particular tasks and model used to set it, the experiments describe the chosen directions rather than a naturally occurring shared channel.

Editorial extensions

If this is right

  • Removing the shared decode subspace during decoding collapses accuracy across tasks and model families, while dimension- and energy-matched controls stay near baseline—the effect is direction-specific, not a matter of how much energy is removed.
  • Prefill-estimated shared bases are near-orthogonal to the decode-time channel and fail the same decode-only causal test, so estimates must be collected from the decoding regime they are meant to explain.
  • Steering vectors carry a substantial component in the shared channel; removing that overlap is an offline knob that typically trades a small average-effect loss for lower template sensitivity and fewer worst-case failures.
  • Decode-stage validation of steering candidates ranks held-out KV-cached utility better than prefill or mixed prefill/decode validation, with higher utility, lower flip rate, and lower regret for the vectors actually deployed.
  • The channel is internally structured: a 29-dimensional residual core carries the causal weight and encodes procedural and symbolic tokens, while a 3-dimensional readout slice is near-inert when ablated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A threshold-free or nonlinear re-estimation of shared structure—e.g., canonical correlation between per-task state matrices—would show whether the PCA-plus-threshold step is essential or merely a convenient probe; the paper's own sensitivity tables show shared-set size swings sharply with τ, so the causal signature under alternative estimators is the natural next test.
  • If the channel is as task-general as claimed, projection energy onto it at decode time could serve as a lightweight serving-time monitor for distribution shift or decision-regime failures—a use the paper's machinery makes immediate but does not propose.
  • Because prefill and decode shared subspaces are near-orthogonal, KV-cache compression and pruning choices validated on prefill statistics may be silently removing exactly the directions that matter at decision time; re-testing those methods with decode-only causal ablations transfers this protocol to the systems setting the paper cites but does not run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DecodeShare, a protocol for identifying a low-dimensional subspace of LLM decode-time hidden states that is shared across tasks, and for testing its causal role under KV-cached inference. Decode-time hidden states are collected across tasks, pooled via PCA, and a 'shared' set S_l(τ,m) is selected as directions whose per-task relative variance exceeds τ for at least m of 13 tasks. The paper then performs decode-only projection removal of this subspace and compares against dimension- and energy-matched non-shared controls, with leave-one-task-out re-estimation and patchback. The main claims are (H1) such a shared structure exists beyond chance, (H2) removing it during decoding causes a large performance drop relative to controls, and (H3) prefill-estimated subspaces do not reproduce the decode-time causal effect. The paper also connects the decode-shared subspace to activation-steering interference and argues that decode-time validation ranks steering vectors better than prefill-based proxies.

Significance. If the central claim holds, the paper makes a useful protocol contribution: it provides a way to identify and causally test a compact, cross-task subspace directly in the KV-cached decode-time representation, rather than using prefill activations as a proxy. The empirical effort is substantial: energy- and dimension-matched controls, leave-one-task-out estimation, patchback with several specificity diagnostics, permutation/scramble nulls, multiple models and layers, and an open-source code link. The reported effects are often large (e.g., average LOTO generation accuracy 27.4 to 15.1 in Table 31) and patchback rescue rates are high with near-zero non-shared rescue, which is evidence in favor of direction-specific causal relevance. However, the significance is qualified by the mismatch between the hypothesis as stated ('shared across tasks') and the operationalization (majority of tasks under a hand-set threshold); the all-tasks intersection is empty for Llama-2, and a loosened threshold fails the scramble null. The paper is a strong candidate after the central claim is reframed or further supported.

major comments (3)
  1. [§2.3, Table 16, App. B.1.4] The central hypothesis states that next-token decisions depend on a subspace S 'shared across tasks.' The operational definition, however, is a majority threshold: S_l(τ,m) with m=8 of 13 tasks. Table 16 shows that under the strict all-tasks criterion (m=13) the shared set is empty for Llama-2-7B (|S|=0, p1=p2=1.0), and App. B.1.4 explicitly reframes the claim as a 'compact core shared by many tasks.' This is load-bearing: all causal H2 tests use the default m=8 set, so they do not establish a genuinely task-universal channel. Combined with Table 17, where loosening τ to 3×10^-4 produces large sets that fail the scramble null for two models, the evidence supports only a threshold-dependent majority-shared subspace. Please either (a) report H2 causal results under the strict all-tasks set where it is nonempty, or (b) revise the abstract, hypothesis, and title to say 'shared by a majority
  2. [Table 7, §3.2, Table 21] The paper claims a 'consistent performance collapse across tasks and models' (H2.1), but Table 7 shows StratQA with Dec-est 52.8 vs baseline 49.6, i.e., an improvement under decode-estimated shared-subspace removal, and Table 21 shows LogiQA 25.8 vs baseline 22.7, also an increase. These are not isolated rounding artifacts; they are per-task rows in the main evidence tables. The aggregate mean drops are real, but the word 'consistent' overstates the per-task pattern. Please report the number of tasks where the shared intervention is significantly harmful vs. unchanged/beneficial, and soften the claim accordingly or explain why these exceptions do not affect the central conclusion.
  3. [§2.3, §3.1, App. B.1.5] There is a self-referential element: the subspace is selected by a variance-based sharedness criterion in pooled PCA, and then the causal test removes exactly that subspace. The energy-matched controls and nulls address the alternative that the effect is merely 'high-energy' or 'low-rank,' but they do not establish that the selected directions correspond to a natural, pre-existing task-general channel. The all-tasks empty set and the threshold sensitivity in Table 17 make this more than a philosophical concern. I would like to see either an independent operationalization of 'shared' (e.g., based on intersection of per-task bases, or a task-general probe) or an explicit limitation statement that DecodeShare characterizes the causal import of the threshold-selected subspace, not necessarily the unique 'shared decode channel.'
minor comments (5)
  1. [Abstract/URL] The GitHub URL in the abstract is broken across a line: 'https://github.com/Zishan-Sha o/decodeshare.git' should be one continuous string.
  2. [Figures 5/16 and 6/17] Figures 5 and 16 are identical, and Figures 6 and 17 are identical. Please remove the duplicates in the appendix or cite them once.
  3. [§2.2 and §2.3] The text references 'Appendix 6' for formal definitions, but no such appendix is present in the visible manuscript; the definitions appear in unnumbered text after Table 8. Please fix the cross-reference.
  4. [Tables 10b and 11] The H1 p-values appear inconsistent across tables for the same model/setting: Table 10b reports p_perm <10^-3 and p_scr=0.0196 for Qwen2.5, while Table 11 reports p_perm ≤5.0×10^-4 and p_scr ≤5.0×10^-3. Clarify whether these are different null baselines or different runs.
  5. [Table 16/17 notation] p1 and p2 are used in Tables 16–17 without being defined in the main text; define them as permutation and scramble p-values at first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the shared subspace is defined by an unsupervised variance threshold and its causal role is tested on held-out examples against matched controls, so the central claim does not reduce to its inputs.

full rationale

DecodeShare's shared set S_l(τ,m) is built by pooling task-centered decode states, running PCA, and thresholding per-task relative variance contributions. This is an unsupervised construction: the selection rule uses variance statistics, not the intervention outcomes. The causal claim (H2) is then evaluated by removing that set only at decode time on held-out examples, with energy- and dimension-matched controls, LOTO re-estimation, and patchback specificity tests. A drop under removal is therefore an empirical prediction about previously unseen examples, not a restatement of the threshold rule. Patchback at full window trivially restores the removed component (100% rescue), but the paper's informative comparisons are Patched@0, transfer donors, time-shuffle, and non-shared/random controls, which show specificity beyond the by-construction inversion. H3's prefill-vs-decode estimator swap is also an independent 2x2 comparison. The paper does cite two prior works with overlapping authorship (Shao et al. 2026; Wu et al. 2026), but only in related-work/system context; no load-bearing uniqueness theorem or ansatz is imported from those citations. The main weaknesses—Table 16's empty all-tasks intersection for Llama-2 and Table 17's loosened-threshold sets failing the scramble null—are real construct-validity and robustness concerns about whether the operationalization captures a naturally all-task-shared channel; Appendix B.1.4 explicitly reframes the claim to a majority-shared core rather than an all-task intersection. These concerns affect how strongly the headline 'shared across tasks' is established, but they do not make the derivation circular: the causal tests against matched controls and held-out data are not forced by the subspace definition. No fitted parameter is renamed as a prediction, and no central result reduces to a self-citation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's central object Sℓ is constructed from PCA plus hand-set thresholds (τ, ρ, m). These are not fitted to maximize the causal effect, but the compactness and cross-task nature of the shared core depend on them; the empty all-tasks case for Llama-2 shows m=8 is load-bearing.

free parameters (6)
  • sharedness threshold τ = 1e-3
    Hand-set threshold on per-task relative variance; controls shared-set size. Swept in Fig. 3/Table 10a; looser τ fails the scramble null (Table 17).
  • PCA variance retention ρ = 0.95
    Determines pooled PCA rank k. Standard choice; shared-core size varies with ρ in Table 10a.
  • minimum task count m = 8 of 13
    'Majority-shared' criterion. The all-tasks m=13 intersection is empty for Llama-2 (Table 16), so m=8 is load-bearing for the existence claim.
  • intervention strength α = 1
    Full projection removal in the main causal tests; α-sweeps show monotonic degradation (Table 24).
  • layer ℓ = 10
    Main analyses reported at layer 10; layers 4/24 and other depths in the appendix show similar qualitative effects.
  • calibration prompts per task = 128
    Subspace-estimation sample size, balanced by subsampling to the minimum task count.
assumptions (5)
  • domain assumption Residual-stream hidden state h(s)_ℓ is the right causal locus: modifying only this state during cached decode steps isolates decision-time computation.
    Used in §2.4 Eq. (1); all causal tests and patchback rely on this.
  • domain assumption Task-general decision structure is linear and captured by pooled PCA directions after task-centering; per-task variance thresholding identifies the shared channel.
    §2.3 defines Sℓ via PCA+r; if false, the causal tests target a selected set rather than a natural shared channel.
  • domain assumption Decode states collected under greedy decoding are representative of the serving distribution for causal tests.
    Main estimates use greedy policy; sampling rotates the basis but H1 remains significant (Table 37).
  • domain assumption Teacher-forced conditional log-prob readout isolates decision quality from generation formatting failures.
    Used for forced-choice accuracy; generation metrics corroborate but show EOS/length collapse (Table 32).
  • domain assumption Orthogonal projection removal does not create off-manifold states whose generic damage drives the results.
    Supported by energy-matched controls and patchback, but generation diagnostics show large distribution shift (Table 32).

how reviews work

0 comments
Cite this review

Pith. "Pith review of DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions." pith.science (2026). https://pith.science/paper/NGLWRPVJ

@misc{pith2026260720469,
  author       = {Pith},
  title        = {Pith review of: DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NGLWRPVJ}},
  note         = {Machine review of arXiv:2607.20469}
}
read the original abstract

Large language models (LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time rather than during prefill. We propose DecodeShare, a protocol that identifies a low-dimensional subspace consistently shared across tasks in decode-time hidden states, and then tests its causal role by removing that subspace only during decoding. In our experiments, disturbing the discovered shared subspace degrades decision performance far more than disturbing either a prefill-derived or random subspace under the same intervention budget. We further show this decode-shared subspace has practical consequences for activation steering: common steering directions can overlap the task-general decode channel. Projecting out this shared subspace directly separates the functional roles of the two components, while evaluating steering vectors at decode-time yields more reliable signal for downstream deployment than prefill-based proxies. Despite its compactness, the shared subspace can serve as a high-leverage causal channel at decode time. Code is available at: https://github.com/Zishan-Shao/decodeshare.git.

Figures

Figures reproduced from arXiv: 2607.20469 by the authors.

Figure 1
Figure 1. DecodeShare Pipeline. (1) We collect KV-cached decode-time hidden states from prompts across multiple tasks and pool them into a matrix. (2) A pooled PCA yields a set of orthonormal directions (columns of V matrix); we then select a subset that is consistently shared across tasks (Q (s) ℓ ), rather than simply taking the top-variance components. (3) To test causality, we intervene only during decoding by removing th… view at source ↗
Figure 2
Figure 2. Sharedness heatmaps across layers. Rows are tasks and columns are pooled PCA components. Color shows the relative variance contribution rℓ,t,i. Components are sorted by the number of tasks with rℓ,t,i ≥ τ ; the vertical line marks the boundary of the shared set Sℓ(τ, m). task-center: X˜ ℓ,t = X′ ℓ,t − 1µ ⊤ ℓ,t, µℓ,t = 1 nℓ Xnℓ j=1 X′ ℓ,t[j, :]. Pooled PCA basis (computed via SVD). We pool (stack) all task-centered s… view at source ↗
Figure 3
Figure 3. Sensitivity to τ . Shared ratio |Sℓ(τ, m)|/k as a function of the sharedness threshold τ for several PCA retention levels ρ. The dashed line marks the default τ = 10−3 ; see Table 10a and §3.1 for the corresponding shared-core sizes and the τ k interpreta￾tion. Size- and energy-matched controls. Decode-time rep￾resentations are anisotropic, so removing high-energy sub￾space can be generically harmful. To isolate the… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: H1: a small but statistically significant shared sub￾space (layer ℓ=10). Scale of the shared core |Sℓ(τ, m)| relative to the full layer width d and pooled PCA dimension k. Although |Sℓ| is only a few percent of d (and of k), both permutation and scramble tests reject t…
Figure 6
Figure 6. Figure 6: Patchback case study (forced-choice). Patching back only the removed shared component over a narrow decode window W = {0, 1} restores the baseline answer, while energy/dimension￾matched controls do not. Two additional diagnostics address common patchback con￾founds (Ap…
Figure 7
Figure 7. Figure 7: Subspace intervention results. From left to right: (a) All-tasks Generation, (b) LOTO Generation, (c) LOTO Forced-Choice Generation. Rectangles show 95% CIs; vertical ticks indicate means; numbers are mean accuracies. More experimental details are available in Appendix…
Figure 8
Figure 8. Figure 8: Pooled PCA spectrum with shared components marked. Explained-variance ratios (log scale) for pooled PCA components at two layers. Components belonging to the identified shared set are visually distinguished, indicating where the shared directions sit in the overall var…
Figure 9
Figure 9. Figure 9: Energy-matched causal controls under strict decode-only interventions (Llama-2-7B-Chat, layer 10). We compare removing the shared subspace against two energy-matched controls (fixed-dimension α-match and expanded-dimension k-match), evaluated by cache-advanced forced-c…
Figure 10
Figure 10. Figure 10: Per-task α sweep (decode-only; energy-matched controls). Panels are ordered left-to-right, top-to-bottom as: AQuA, ARC-Challenge, BoolQ, CommonsenseQA, LogiQA, OpenBookQA, QASC, StrategyQA. Dashed line: baseline; shaded regions: 95% CIs. Therefore, ∥δ(h; Q, α)∥ 2 2 = …
Figure 11
Figure 11. Figure 11: shareness heatmaps across layers. Rows are tasks and columns are pooled PCA components. Color shows rℓ,t,i; columns are sorted by #tasks with rℓ,t,i ≥ τ ; vertical line marks Sℓ(τ, m). B. Additional Main Results. B.1. Additional Results for H1 : Diagnostics for Shared…
Figure 12
Figure 12. Figure 12: PCA visualization of component signatures for layer 10. Left: 3D projection onto the first three principal components (PC1–PC3). Right: top-down projection onto PC1–PC2. Each point corresponds to a component; color indicates the number of tasks whose component score e…
Figure 13
Figure 13. Figure 13: (Left) Within-category shareness vs. mixed-category baseline. (Right) Overlap to the full-task pooled subspace vs. number of tasks [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Phase-distinguished convergence trends across modes. Left: overlap to the full-task pooled subspace; right: cross dimension as the number of tasks increases [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Same prompt, same decoding, different vectors. vorig triggers pirate lexicon (success), while projecting out the shared PCA span at k= 128 suppresses the style (failure), consistent with v being entangled with the shared decode subspace [PITH_FULL_IMAGE:figures/full_…
Figure 16
Figure 16. Figure 16: Flip case study (forced-choice). The baseline forced-choice prediction matches the gold answer. Removing the shared decode subspace at layer ℓ = 10 during cached decoding flips the predicted choice demonstrating that the shared decode pathway can be causally necessary…
Figure 17
Figure 17. Figure 17: Patchback case study (forced-choice). Patching back only the removed shared component over a narrow decode window W = {0, 1} restores the baseline answer, while energy/dimension-matched controls do not [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 4 linked inside Pith

  1. [1]

    For eacht∈ T, run cached decoding on prompts inC t under policyπup toKsteps

  2. [2]

    Meng, K., Bau, D., Andonian, A., and Belinkov, Y

    URL https://openreview.net/forum ?id=Ebt7JgMHv1. Meng, K., Bau, D., Andonian, A., and Belinkov, Y . Locating and editing factual associations in gpt.Advances in neural information processing systems, 35:17359–17372, 2022. Merullo, J., Eickhoff, C., and Pavlick, E. Circuit component reuse across tasks in transformer language models. In The Twelfth Internat...

    arXiv 2022

  3. [3]

    Task-center: ˜Xt,ℓ ←X t,ℓ −1µ ⊤ t,ℓ

  4. [4]

    Zou, A., Phan, L., Chen, S

    Curran Associates Inc. Zou, A., Phan, L., Chen, S. L., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: A top-down approach to AI transparency.CoRR, abs/2310.01405,

    arXiv

  5. [5]

    stronger perturbation

    doi: 10.48550/ARXIV.2310.01405. URL https: //doi.org/10.48550/arXiv.2310.01405. 12 DECODESHARE: Tracing the Shared Subspace of LLM Decode-Time Decisions Table 8.Benchmark taxonomy used in our experiments. For discrete-choice tasks, we use decision-level forced-choice accuracy (FC-Acc; teacher-forced conditional logprob) as the primary readout; we also rep...

    arXiv doi:10.48550/arxiv.2310.01405

  6. [6]

    explained variance≥ρ; setQ ℓ ←V :,1:k

    PCA/SVD: ˜Xℓ =UΣV ⊤; choose smallestks.t. explained variance≥ρ; setQ ℓ ←V :,1:k

  7. [7]

    Collect decode last-token states{h (s) ℓ }where seq len= 1intoX t,ℓ ∈R nt,ℓ×d

  8. [8]

    Shared set:S ℓ(τ, m)← {i:|{t:r t,i ≥τ}| ≥m}

Show all 19 references
  1. [9]

    Balance ton ℓ = mint nt,ℓ by subsampling each ˜Xt,ℓ

  2. [10]

    Pool: ˜Xℓ ←concat( ˜Xt,ℓ :t∈ T)∈R (|T |nℓ)×d

  3. [12]

    For eacht, projectZ t,ℓ ← ˜Xt,ℓQℓ, computer t,i ←Var(Z t,ℓ[:, i]) P j Var(Zt,ℓ[:, j])

  4. [14]

    Algorithm 2Decode-time last-token subspace removal Require:basisQ ℓ (shared or control), strengthα, stage lengthK stage (optional)

    ReturnQ ℓ andQ (S) ℓ ←Q ℓ[:, Sℓ]. Algorithm 2Decode-time last-token subspace removal Require:basisQ ℓ (shared or control), strengthα, stage lengthK stage (optional)

  5. [15]

    : run cached forward pass withseq len = 1to obtainh (s) ℓ

    For decode steps= 1,2, . . .: run cached forward pass withseq len = 1to obtainh (s) ℓ

  6. [16]

    If full removal: ˜h(s) ℓ ←h (s) ℓ −αQ ℓQ⊤ ℓ h(s) ℓ

  7. [17]

    If staged: apply the same update only when the generated-token count is< K stage

  8. [18]

    any patch works

    Continue to logits; choose next token (greedy/sampling); update cache. dimensions). This highlights a key nuance:subspace stability alone is not decode-specific—prefill can also yield a stable low-rank pooled geometry. To connect stability to shareness, we evaluate thefully-sh...

    2000

  9. [19]

    using more dimensions

    still exhibits the same qualitative mismatch on generation-heavy benchmarks, indicating the effect is not explained by “using more dimensions”. C. Additional Experiments C.1.α-Sweep in Patch back Experiments α-sweep validates ablation strength as a continuous knob and transfer...

  10. [2023]

    Whitaker, S., Nasir, W., and Hart, E

    URL https://openreview.net/forum ?id=NpsVSN6o4ul. Whitaker, S., Nasir, W., and Hart, E. Identifying common semantics across modalities via contrastive latent align- ment.Preprints, July 2025. doi: 10.20944/preprints2025 07.0008.v1. URL https://doi.org/10.20944 /preprints202507...

    2025

  11. [2024]

    Bhaskar, A., Wettig, A., Friedman, D., and Chen, D

    URL https://arxiv.org/abs/2404.1 4082. Bhaskar, A., Wettig, A., Friedman, D., and Chen, D. Finding transformer circuits with edge pruning.Advances in Neu- ral Information Processing Systems, 37:18506–18534, 2024. Cai, Z., Zhang, Y ., Gao, B., Liu, Y ., Liu, T., Lu, K., Xiong, ...

    2024 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.