arxiv: 2602.21204 · v4 · submitted 2026-02-24 · 💻 cs.LG · cs.AI· cs.CV

Recognition: 1 theorem link

· Lean Theorem

Test-Time Training with KV Binding Is Secretly Linear Attention

Junchen Liu , Sven Elflein , Or Litany , Zan Gojcic , Ruilong Li

Authors on Pith no claims yet

Pith reviewed 2026-05-15 19:38 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CV

keywords test-time trainingKV bindinglinear attentionsequence modelingattention mechanismsmeta-learningonline learning

0 comments

The pith

Test-time training with KV binding can be rewritten exactly as learned linear attention.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper challenges the common view of TTT with KV binding as online meta-learning that memorizes key-value mappings at test time. It shows instead that a broad class of such architectures can be expressed exactly as learned linear attention operators. This equivalence explains model behaviors that contradict pure memorization and yields simplifications plus parallel implementations. Readers would care because the reframing unifies TTT with standard attention, replacing a memorization story with a capacity-enhanced attention story.

Core claim

Test-time training (TTT) with KV binding, commonly interpreted as memorizing a key-value mapping at test time, can be expressed as a form of learned linear attention operator. This holds for a broad class of TTT architectures, explains previously puzzling behaviors, enables principled simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and reduces diverse TTT variants to standard linear attention form.

What carries the argument

The KV-binding update rule together with the test-time gradient step, rewritten via exact linear algebra into a linear attention operator.

If this is right

Explains model behaviors that contradict a pure memorization interpretation
Enables principled architectural simplifications of TTT layers
Admits fully parallel formulations that preserve performance while improving efficiency
Reduces diverse TTT variants systematically to a standard linear attention form

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The equivalence suggests TTT could be treated as a drop-in capacity boost inside existing linear-attention stacks rather than a separate meta-learning module.
Optimization tricks developed for linear attention may transfer directly to TTT variants.
The parallel formulation opens the possibility of training larger TTT models on long sequences without sequential bottlenecks.

Load-bearing premise

The KV-binding update rule and test-time gradient step admit an exact linear-algebra rewriting that preserves the original computation for all practical sequence lengths and model sizes.

What would settle it

A concrete input sequence on which the original TTT-KV computation produces a numerically different output from its linear-attention rewrite.

Figures

Figures reproduced from arXiv: 2602.21204 by Junchen Liu, Or Litany, Ruilong Li, Sven Elflein, Zan Gojcic.

**Figure 1.** Figure 1: Inner-Loop Optimization vs. Performance. Increasing inner-loop iterations improves inner-loop loss but degrades task performance, contradicting the memorization-based interpretation of TTT. Experiments are based on LaCT (Zhang et al., 2025). memorization perspective, other works have explored advanced test-time optimizers (Behrouz et al., 2024; Zhang et al., 2025; Karami et al., 2025) and alternative regr… view at source ↗

**Figure 2.** Figure 2: Distributional Asymmetry Between Q and K. tSNE visualizations of (Q, K) and (V, O) features in a pretrained LaCT (Zhang et al., 2025) model on the NVS task, showing that the TTT inner loop is evaluated out of distribution and thus does not perform reliable retrieval. to, and in some cases even slightly better than, standard gradient descent ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Perplexity Metric for Ablation om LaCT-LLM. Evaluated on 2.5B tokens from the Book-3 dataset. GLU component. The GLU is defined as f(x) = silu(xW0) ⊙ (xW1), where W0 and W1 are fast weights updated via gradient descent. As in LaCT, the inner-loop loss is defined using a Frobenius inner product. Following a derivation analogous to previous sections (see Appendix F), evaluating the updated GLU on a query q… view at source ↗

**Figure 4.** Figure 4: Training loss vs. wall-clock time on LaCT-LLM. We compare the original LaCT-TTT with both parallel and recurrent form of Variant 2. The parallel form achieves a 1.19× end-to-end speedup while maintaining comparable convergence. ization improves performance on the LLM task. Overall, reducing the full TTT formulation to a basic linear attention operator (Variant 6) results in only minor performance degradat… view at source ↗

read the original abstract

Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity. Project page: https://research.nvidia.com/labs/sil/projects/tttla/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper shows TTT with KV binding rewrites as learned linear attention for single-step cases, which could simplify parallel inference but needs checks on multi-step variants.

read the letter

The main thing to know is that this work reframes test-time training with KV binding as a form of learned linear attention rather than online memorization. The authors identify behaviors that do not fit the memorization story and then derive an algebraic rewrite that turns the update rule plus a gradient step into a standard linear attention operator. If the identity is exact, it lets you drop the sequential test-time loop in favor of parallel primitives while keeping the same output.

Referee Report

2 major / 2 minor

Summary. The paper claims that test-time training (TTT) with KV binding, commonly interpreted as online meta-learning for memorizing key-value mappings, can instead be exactly reformulated as a learned linear attention operator for a broad class of architectures. This algebraic rewriting explains puzzling model behaviors, enables principled simplifications, fully parallel formulations that preserve performance, and reduces diverse TTT variants to standard linear attention, reframing TTT as enhanced linear attention rather than memorization.

Significance. If the claimed exact equivalence holds, the result would provide a unifying perspective on TTT and linear attention that could enable more efficient sequence modeling implementations and new architectural designs. The practical benefits of parallelization and simplification are high-value if verified, and the work would strengthen connections between test-time adaptation and attention mechanisms in large models.

major comments (2)

[Reformulation section (analysis of KV-binding update rule)] The central reformulation (analysis of KV-binding update plus test-time gradient step): the equivalence is derived under the assumption of a single gradient descent step with fixed scalar learning rate. The manuscript does not demonstrate that the closed-form linear attention expression remains exact for multi-step TTT, momentum, or per-parameter adaptive rates, which are common in practice and would make the rewriting approximate rather than exact.
[Experiments section] Experimental validation of parallel formulations: the claim that performance is preserved while improving efficiency lacks controls for long sequence lengths and quantitative measurement of any deviation from the original TTT computation when the single-step assumption is relaxed.

minor comments (2)

[Abstract] The abstract and introduction would benefit from a brief explicit statement of the key algebraic identity used in the rewriting to aid immediate verification.
[Figures] Figure captions and diagrams illustrating the KV-binding to linear attention mapping should include equation references for each transformation step.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's insightful comments on our work. We provide point-by-point responses to the major comments below, and we will incorporate revisions to address the concerns raised.

read point-by-point responses

Referee: The central reformulation (analysis of KV-binding update plus test-time gradient step): the equivalence is derived under the assumption of a single gradient descent step with fixed scalar learning rate. The manuscript does not demonstrate that the closed-form linear attention expression remains exact for multi-step TTT, momentum, or per-parameter adaptive rates, which are common in practice and would make the rewriting approximate rather than exact.

Authors: We agree that our exact equivalence is derived specifically for the single gradient descent step with a fixed scalar learning rate. This matches the standard TTT setup in the literature. For multi-step TTT, momentum-based optimizers, or adaptive per-parameter rates, the reformulation would indeed be approximate. We will revise the reformulation section to clearly delineate the conditions for exact equivalence and note that extensions to more general optimizers remain an open direction for future work. revision: yes
Referee: Experimental validation of parallel formulations: the claim that performance is preserved while improving efficiency lacks controls for long sequence lengths and quantitative measurement of any deviation from the original TTT computation when the single-step assumption is relaxed.

Authors: We thank the referee for this suggestion. Our current experiments demonstrate that the parallel formulation preserves performance on sequences up to length 4096, with negligible numerical differences from the sequential TTT implementation. To address the concern, we will extend the experiments to include longer sequences (up to 16384 tokens) and add quantitative measurements, such as the maximum absolute deviation in model outputs between the original TTT and the parallel linear attention version, for both single-step and relaxed multi-step settings. revision: yes

Circularity Check

0 steps flagged

Algebraic equivalence from TTT update rule; no load-bearing self-fit or self-citation

full rationale

The derivation rewrites the KV-binding update plus single test-time gradient step as linear attention via exact algebraic identities on the existing TTT formulation. This is not a fitted prediction, not a self-definitional loop, and does not rely on prior self-citations to force the result. The equivalence holds by construction under the single-step fixed-LR assumption stated in the paper, with independent content remaining in the practical simplifications and parallel formulations that follow from the rewrite.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the existence of an exact algebraic rewriting of the TTT update; no new free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption The test-time KV-binding update admits an exact rewriting as a linear attention operator for the class of architectures considered.
This identity is the load-bearing step that converts the online update into parallel linear attention.

pith-pipeline@v0.9.0 · 5462 in / 1085 out tokens · 34336 ms · 2026-05-15T19:38:35.338577+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Theorem 5.1: inner-loop GD update rewritten as o = ϕ_{t+1}(q) (W_t + ϕ_t(k)^T g_t(k))

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Exploration
cs.AI 2026-04 unverdicted novelty 6.0

LLM agents trained with a task-success reward on self-generated knowledge can spontaneously explore and adapt to new environments without any rewards or instructions at inference, yielding 20% gains on web tasks and a...
Fast Spatial Memory with Elastic Test-Time Training
cs.CV 2026-04 unverdicted novelty 6.0

Elastic Test-Time Training stabilizes test-time updates via an elastic prior and moving-average anchor, enabling Fast Spatial Memory for scalable long-sequence 4D reconstruction with reduced memory use and fewer shortcuts.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · cited by 2 Pith papers · 10 internal anchors

[1]

Titans: Learning to Memorize at Test Time

Behrouz, A., Zhong, P., and Mirrokni, V . Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663,

work page internal anchor Pith review Pith/arXiv arXiv
[2]

Atlas: Learning to optimally memorize the context at test time.arXiv preprint arXiv:2505.23735, 2025a

Behrouz, A., Li, Z., Kacham, P., Daliri, M., Deng, Y ., Zhong, P., Razaviyayn, M., and Mirrokni, V . Atlas: Learning to optimally memorize the context at test time.arXiv preprint arXiv:2505.23735, 2025a. Behrouz, A., Razaviyayn, M., Zhong, P., and Mirrokni, V . Nested learning: The illusion of deep learning architec- tures.arXiv preprint arXiv:2512.24695,...

work page arXiv
[3]

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

9 Test-Time Training with KV Binding Is Secretly Linear Attention Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Efficiently Modeling Long Sequences with Structured State Spaces

Gu, A., Goel, K., and R ´e, C. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

ViT$^3$: Unlocking Test-Time Training in Vision

Han, D., Li, Y ., Li, T., Cao, Z., Wang, Z., Song, J., Cheng, Y ., Zheng, B., and Huang, G. Vit3: Unlocking test-time train- ing in vision.arXiv preprint arXiv:2512.01643,

work page internal anchor Pith review Pith/arXiv arXiv
[6]

Karami, M., Pascanu, R., and Mirrokni, V

URL https: //kellerjordan.github.io/posts/muon/. Karami, M., Pascanu, R., and Mirrokni, V . Lattice: Learn- ing to efficiently compress the memory.arXiv preprint arXiv:2504.05646,

work page arXiv
[7]

Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics

Lei, J., Zhang, D., and Poria, S. Error-free linear attention is a free lunch: Exact solution from continuous-time dynamics.arXiv preprint arXiv:2512.12602,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

Longhorn: State space models are amortized online learners.arXiv preprint arXiv:2407.14207,

Liu, B., Wang, R., Wu, L., Feng, Y ., Stone, P., and Liu, Q. Longhorn: State space models are amortized online learners.arXiv preprint arXiv:2407.14207,

work page arXiv
[9]

Meta-Learning Update Rules for Unsupervised Representation Learning

Metz, L., Maheswaranathan, N., Cheung, B., and Sohl- Dickstein, J. Meta-learning update rules for unsupervised representation learning.arXiv preprint arXiv:1804.00222,

work page internal anchor Pith review Pith/arXiv arXiv
[10]

L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Cand `es, E., and Hashimoto, T

Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Cand `es, E., and Hashimoto, T. B. s1: Simple test-time scaling. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20286– 20332,

work page 2025
[11]

Rwkv: Reinventing rnns for the transformer era

Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcad- inho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. Rwkv: Reinventing rnns for the transformer era. InFindings of the Association for Com- putational Linguistics: EMNLP 2023, pp. 14048–14077,

work page 2023
[12]

Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence.arXiv preprint arXiv:2404.05892,

Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Al- caide, E., Biderman, S., Cheah, E., Du, X., Ferdinan, T., Hou, H., et al. Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence.arXiv preprint arXiv:2404.05892,

work page arXiv
[13]

Rwkv-7” goose” with expressive dynamic state evolution.arXiv preprint arXiv:2503.14456, 2025a

Peng, B., Zhang, R., Goldstein, D., Alcaide, E., Du, X., Hou, H., Lin, J., Liu, J., Lu, J., Merrill, W., et al. Rwkv-7” goose” with expressive dynamic state evolution.arXiv preprint arXiv:2503.14456, 2025a. Peng, L., Chattopadhyay, A., Zancato, L., Nunez, E., Xia, W., and Soatto, S. Gated kalmanet: A fading memory layer through test-time ridge regression....

work page arXiv
[14]

Hgrn2: Gated linear rnns with state expansion

Qin, Z., Yang, S., Sun, W., Shen, X., Li, D., Sun, W., and Zhong, Y . Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904,

work page arXiv
[15]

GLU Variants Improve Transformer

Shazeer, N. Glu variants improve transformer.arXiv preprint arXiv:2002.05202,

work page internal anchor Pith review Pith/arXiv arXiv 2002
[16]

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

10 Test-Time Training with KV Binding Is Secretly Linear Attention Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test- time compute optimally can be more effective than scal- ing model parameters.arXiv preprint arXiv:2408.03314,

work page internal anchor Pith review Pith/arXiv arXiv
[17]

Retentive Network: A Successor to Transformer for Large Language Models

Sun, Y ., Dong, L., Huang, S., Ma, S., Xia, Y ., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621,

work page internal anchor Pith review Pith/arXiv arXiv
[18]

End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675,

Tandon, A., Dalal, K., Li, X., Koceja, D., Rød, M., Buchanan, S., Wang, X., Leskovec, J., Koyejo, S., Hashimoto, T., et al. End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675,

work page arXiv
[19]

Kimi Linear: An Expressive, Efficient Attention Architecture

Team, K., Zhang, Y ., Lin, Z., Yao, X., Hu, J., Meng, F., Liu, C., Men, X., Yang, S., Li, Z., et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692,

work page internal anchor Pith review Pith/arXiv arXiv
[20]

A., Shi, J., and Fox, E

Wang, K. A., Shi, J., and Fox, E. B. Test-time regression: a unifying framework for designing sequence models with associative memory.arXiv preprint arXiv:2501.12352,

work page arXiv
[21]

T., and Tan, H

Zhang, T., Bi, S., Hong, Y ., Zhang, K., Luan, F., Yang, S., Sunkavalli, K., Freeman, W. T., and Tan, H. Test-time training done right.arXiv preprint arXiv:2505.23884,

work page arXiv
[22]

Understanding transformer from the perspective of associative memory

Zhong, S., Xu, M., Ao, T., and Shi, G. Understanding transformer from the perspective of associative memory. arXiv preprint arXiv:2505.19488,

work page arXiv
[23]

For evaluation, we report perplexity on 2.5B tokens from the Book-3 dataset (Gao et al., 2020)

configuration. For evaluation, we report perplexity on 2.5B tokens from the Book-3 dataset (Gao et al., 2020). All implementations are based on the Flame (Zhang & Yang,

work page 2020
[24]

Follow (Han et al., 2025), we train our model on the ImageNet-1K (Deng et al.,

as our baseline model, totaling 90M parameters. Follow (Han et al., 2025), we train our model on the ImageNet-1K (Deng et al.,

work page 2025