Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Test-Time Context Distillation claims that a teacher–student hidden-state discrepancy is the right self-supervised signal for deciding what a model's limited memory should keep, and that this objective beats prior recurrent and test-time tr

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:08 UTC pith:L7OG4TFH

load-bearing objection TTCD's context-distillation write signal is a real empirical step forward for TTT, but the theory is circular and the implementation's off-policy gap weakens the central attribution only partially. the 4 major comments →

arxiv 2608.01672 v1 pith:L7OG4TFH submitted 2026-08-03 cs.CL cs.AIcs.LG

Learning What to Remember: Test-Time Training via Context Distillation

Zixuan Wang , Xingyu Dang , Rui-Jie Zhu , Zixin Wen , Hengyu Fu , Wenhao Chai , Jason D. Lee This is my paper
classification cs.CL cs.AIcs.LG
keywords test-time trainingcontext distillationfast weightslong-context language modelingcontinual pre-trainingmemory allocationRULERsliding window attention
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces TTCD, a test-time training objective that decides what a model should retain by distilling the hidden states of a long-window teacher into a short-window student. The difference between the two views tells the model which distant information actually matters for future predictions, and that signal is used to update an in-place fast weight—the MLP down-projection—so the model's own parameters act as contextual memory. The claim is that this memory-allocation signal beats both reconstruction-based and next-token TTT objectives, and the paper shows on sliding-window perplexity, RULER, and BABILong that IP-TTCD outperforms DeltaNet, Gated DeltaNet, sliding-window attention, and In-Place TTT when trained from scratch. It also shows that the same objective can be grafted onto pretrained transformers via continual pre-training, extending their usable context with a lightweight modification. If true, the result is a practical step toward models that keep learning from each sequence as it unfolds.

Core claim

The central claim is that context distillation—not reconstruction or next-token prediction—is the right training signal for test-time memory. At each chunk, a long-window teacher and a short-window student process the same tokens with shared frozen weights; the squared difference between their MLP outputs, with the teacher detached, updates the student's MLP down-projection fast weight. The teacher–student residual marks exactly what the remote context contributes beyond the local window, so the fast weight stores future-useful information and writes it where the student's own activations can retrieve it later. Proposition 1 shows that, under alignment assumptions, the update moves a later r

What carries the argument

The central object is the context-distillation fast-weight update (Equations 4–6): a chunk-wise gradient step on the loss || student MLP output − stopgrad(teacher MLP output) ||², applied to the MLP down-projection W_down, with a learned projection and short convolutions added in the practical version. The teacher–student gap supplies the content to write and the student's activation supplies the retrieval key, making the down-projection a sequence-specific associative memory. Proposition 1 (Equations 7–9) is the theoretical backbone: it shows the one-step update transfers an in-context solution from an earlier query to a later aligned query when the student activations align.

Load-bearing premise

The implemented model computes every memory write with the frozen base down-projection and only later accumulates the updates, while the proof analyzes the on-policy recurrence where each chunk uses the fast weights from all previous chunks; the paper provides no bound on how much these two objects diverge.

What would settle it

Take the trained IP-TTCD checkpoints and re-run prefill with the true on-policy chunk recurrence, where chunk i's teacher and student activations are computed with the fast weight produced by chunks < i; if sliding-window perplexity and RULER NIAH do not match or beat the published off-policy numbers, the theoretical account of how the memory works is wrong. A second check: probe the fast-weight matrix for the stored teacher–student residual at q1 by measuring the cosine similarity between ΔW·z_q2 and the teacher residual y_q1^T − y_q1^S across many random sequences; Proposition 1 predicts str

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • IP-TTCD consistently beats DeltaNet, Gated DeltaNet, sliding-window attention, and IP-TTT on sliding-window perplexity from 1K to 32K context at 340M and 760M scale, with the gap widening at longer contexts.
  • At 760M, IP-TTCD more than doubles IP-TTT's RULER NIAH average at 32K (21.96 vs 9.29) and stays ahead at every tested length, without sacrificing common-sense accuracy.
  • Grafting IP-TTCD onto pretrained transformers (SmolLM2-360M/1.7B, LLaMA-3.1-8B) via 10B-token continual pre-training improves RULER average over attention-only tuning and IP-TTT, especially at 32K/64K.
  • The fused dual-window kernel keeps inference within −1.3% to +6.6% of IP-TTT prefill time, so the long-context gains are not bought with a large runtime penalty.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the distillation signal is genuinely about future utility, the same objective could be applied to choose which KV-cache entries to retain in standard attention models, turning cache eviction from a heuristic into a learned, online decision.
  • The off-policy implementation suggests a testable conjecture: that the frozen-projection approximation actually regularizes the fast weights, and that an on-policy version would be no better—this could be checked directly.
  • The teacher-window size itself becomes a hyperparameter of memory allocation; papers in this line could explore adapting the teacher window dynamically per layer or per chunk.
  • Because the fast weight lives in the MLP down-projection, IP-TTCD naturally composes with MoE layers or other parameter reuse; the memory capacity could be scaled by choosing how many layers participate, which the ablation shows improves performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Test-Time Context Distillation (TTCD), a test-time training objective in which a short-window student MLP is updated to match the hidden states of a long-window teacher. The in-place variant IP-TTCD uses the MLP down-projection as fast weights, with updates written as low-rank outer products of teacher–student residuals and student activations. The authors provide a theoretical proposition intended to show that this update stores a long-context signal at one query and replays it at a later aligned query. Empirically, they report that IP-TTCD outperforms DeltaNet, Gated DeltaNet, sliding-window attention, and IP-TTT on long-context language modeling, RULER, BABILong, and MAD, both when trained from scratch and when used as continual pretraining on SmolLM2, Qwen3, and LLaMA-3.1-8B. The paper also includes inference-kernel details and ablations of the added convolution/projection components.

Significance. If the empirical claims are sustained, the context-distillation objective is a plausible and useful alternative to reconstruction- or next-token-based TTT losses, with evidence across multiple benchmarks, model sizes, and training regimes. The paper ships code, reports detailed kernel-level validation, and includes a synthetic benchmark suite, which are strengths. The central theoretical argument, however, is currently not load-bearing: Proposition 1 is a restatement of its assumptions, and the analyzed on-policy recurrence is not the object implemented in Section 3.3. The empirical attribution to the distillation signal is also not isolated from the added learnable machinery. These gaps are fixable and do not, at this stage, invalidate the empirical trend, but they require substantial revision before the central claims can be accepted.

major comments (4)
  1. [§3.3 (Eqs. 10–13) vs. §3.2 (Eqs. 6–9)] The implemented IP-TTCD update is not the object analyzed in Proposition 1. Equation (6) is an on-policy recurrence where each chunk uses fast weights produced by previous chunks; the implementation instead computes all teacher/student activations with the base W_down^(0), applies conv_T/conv_S, W_proj, and L2 normalization, and then accumulates corrections causally. No bound or analysis is given for the difference between this off-policy update and the on-policy theory. Since the paper's theoretical guarantee is used to justify the fast-weight memory interpretation, this mismatch is load-bearing. The authors should either provide an analysis of the off-policy approximation or explicitly reframe the theoretical claim as applying only to an idealized variant.
  2. [§3.2, Proposition 1 and Assumption 1] Proposition 1 is essentially a restatement of Assumption 1. Equation (9) is simply the product of the assumed positive teacher–student gap c_gap and the assumed positive alignment c_align, after the orthogonality assumption removes all other terms. The orthogonality condition z_t^⊤ z_q2 = 0 for all t ≠ q1 is stated as an assumption that real token activations will not satisfy. Thus the proposition does not provide a mechanism or a transferable insight; it only repackages the assumptions. I recommend either removing the theorem or replacing it with a derivation that does not assume the conclusion.
  3. [§4.4 and Table 1 / Figure 2] The central attribution of the gains to the context-distillation objective is not isolated. The IP-TTCD vs. IP-TTT comparison changes both the loss and the architecture (conv_T/conv_S, W_proj, L2-normalized keys, β_i). The ablations in Figure 4 vary the added components but never replace the teacher–student residual with an alternative TTT objective (e.g., next-token prediction or reconstruction) while keeping the same machinery. Consequently, the reported improvements could stem from the learnable outer-product write rule rather than from the distillation signal. An ablation with the same update machinery trained under a different objective is necessary to support the abstract's claim that context distillation is a better memory-allocation signal.
  4. [Table 1 and Figure 2] All from-scratch pretraining results appear to be single runs without seeds or error bars. The claim of 'consistently outperforms' is supported by many data points, but the magnitude of some gaps (e.g., RULER NIAH 21.96 vs. 9.29 at 32K) would be more credible with at least a few seeds or confidence intervals, especially given the known variance of long-context retrieval metrics. Reporting seeds for at least the 340M configuration would strengthen the central empirical claim.
minor comments (5)
  1. [Table 2 / §4.3] For SmolLM2-360M at 64K, IP-TTCD (9.79) is worse than IP-TTT (10.35) and CPT (10.09), while the text says the gains are 'most visible at the longest contexts.' This is only true for the 1.7B model and the average; please qualify the statement.
  2. [Eq. (12)] β_i is used in the fast-weight correction but never defined as a learned scalar, a hyperparameter, or a schedule. Please clarify.
  3. [Appendix A.1] The teacher/student convolution initialization is described differently for pretraining ('teacher zero, student random') and continual pretraining ('teacher identity, student zero'). This inconsistency should be resolved, and the effect of initialization on stability should be stated.
  4. [§3.2 / Appendix C.1] The text says 'Detailed proof see Proposition 2' but Proposition 2 is a restatement of Proposition 1. The numbering is confusing; please unify or remove the duplicate.
  5. [§4.1 / Appendix A.1.1] The MAD benchmark reports best test accuracy after sweeping learning rate and weight decay. This is stated to follow the standard protocol, but it may overstate performance; consider also reporting the mean or median over the sweep.

Circularity Check

1 steps flagged

Proposition 1's transfer guarantee restates Assumption 1 by construction; the empirical claims are external and not circular.

specific steps
  1. self definitional [Section 3.2, Assumption 1 and Proposition 1 (Eq. 9)]
    "Assumption 1. We make the following two assumptions. • Non-trivial in-context signal: q1 − w_T < c1 ≤ c2 ≤ q1 − w_S and y(T)⊤_q1 (y(T)_q1 − y(S)_q1) ≥ c_gap > 0. • Chunk causality and related-query alignment: if chunk j contains q2, then q1 ∈ I_<j, z(S)⊤_q1 z(S)_q2 ≥ c_align > 0, and z(S)⊤_t z(S)_q2 = 0 for all t ∈ I_<j \ {q1}. Proposition 1 ... for any learning rate η > 0, the update induced by preceding chunks satisfies y(T)⊤_q1 Δy(S)_q2 ≥ η c_gap c_align."

    The proof expands Δy(S)_q2 = η Σ_t (y(T)_t−y(S)_t) z(S)⊤_t z(S)_q2. Assumption 1 zeroes every term except t=q1 and then asserts exactly that the remaining teacher–student gap is ≥ c_gap and the alignment is ≥ c_align. Therefore Eq. (9) is just η · (assumed gap) · (assumed alignment): the theorem's conclusion is the product of its own hypotheses. It provides no information about whether real token activations satisfy the required positivity or orthogonality, so the claimed 'guarantee' of future-utility memory transfer is already contained in the assumptions by construction.

full rationale

The paper's headline empirical claims—IP-TTCD outperforming DeltaNet, Gated DeltaNet, sliding-window attention, and IP-TTT on Books sliding-window perplexity and RULER, plus continual-pretraining gains on SmolLM2 and LLaMA—are evaluated against external benchmarks and are not fitted-then-predicted; those results are self-contained comparisons and do not reduce to the method's own definitions. No load-bearing self-citation was found: the in-place TTT formulation is attributed to Feng et al. [15], which has no author overlap with this paper. The one genuinely circular step is Proposition 1: under Assumption 1, the orthogonality condition leaves only the q1 term, and Eq. (9) is precisely η times the two positive quantities already assumed (c_gap and c_align). Separately, Section 3.3's off-policy implementation—all activations computed with W^(0)_down and updates accumulated causally—is not the recurrence analyzed by Proposition 1, and the paper asserts it 'preserves the update structure' without a bound; that is a theory-implementation gap and a threat to causal attribution, but not itself circularity. Because the central empirical results stand independently of the tautological theorem, the overall circularity is partial (score 4) rather than the 6+ reserved for claims that reduce entirely to their inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method is mostly empirical; its free parameters are standard training choices (learning rates, windows, layers, chunk size). The main theoretical support, Proposition 1, is an encoding of its own assumptions. No new physical or architectural entities are postulated; the fast weights reuse existing MLP parameters, and the teacher-student dual-window views are a design choice rather than an invented entity.

free parameters (5)
  • TTT learning rate eta = 0.3
    Hand-set learning rate for fast-weight updates (Table 3), used in Eq. 6 and the causal scan in Eq. 13. No sensitivity study is reported beyond layer-count and chunk-size ablations.
  • Update scale beta_i = unspecified
    Eq. 12 introduces beta_i as a per-step scale on the fast-weight correction; its value or schedule is not given in the main text or appendices.
  • Teacher and student window sizes = teacher 2048 from scratch, 16/32 on MAD; student half
    Window sizes are chosen per experiment and per MAD task. The teacher-student gap defines the distillation signal, so all results depend on this choice.
  • Number of TTT layers and chunk size = 4 layers; chunk 1024-2048
    Ablated in Figure 4 and chosen by hand; these control memory capacity and parallelism, and the paper shows both affect perplexity.
  • MAD hyperparameter selection = best test accuracy over LR and weight-decay grid
    Section A.1.1 reports the best test accuracy after sweeping 6 configurations per task, which selects on the test set. This follows the stated MAD protocol but is a form of fit-to-test.
axioms (4)
  • ad hoc to paper Assumption 1: nonzero teacher-student gap and related-query alignment, with all other student activations orthogonal to the future query
    Proposition 1's proof uses exactly these inequalities to reach its conclusion; the theorem is a restatement of the assumption rather than an independent guarantee.
  • domain assumption The teacher's extra history is predictive of future tokens
    The whole objective assumes the long-window hidden state encodes information worth memorizing for later predictions; no argument beyond the benchmark results establishes this.
  • domain assumption Off-policy updates computed with frozen base weights remain valid memory writes
    Section 3.3 replaces the on-policy recurrence of Eq. 6 with base-weight activations plus a causal scan; validity is checked only empirically, not bounded analytically.
  • domain assumption Shared backbone with teacher and student views generalizes across layers and scales
    The same MLP down-projection serves as fast memory in 4 layers at every model size tested; no analysis of layer-wise behavior beyond the layer-count ablation.

pith-pipeline@v1.3.0-daily-deepseek · 20780 in / 17063 out tokens · 160115 ms · 2026-08-04T23:08:21.854836+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Learning What to Remember: Test-Time Training via Context Distillation." pith.science (2026). https://pith.science/paper/L7OG4TFH

@misc{pith2026260801672,
  author       = {Pith},
  title        = {Pith review of: Learning What to Remember: Test-Time Training via Context Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7OG4TFH}},
  note         = {Machine review of arXiv:2608.01672}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later. Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling, yet existing TTT methods only optimize either reconstruction or online adaptation objectives without considering the future utility of retained information. In this work, we propose \textbf{T}est-\textbf{T}ime \textbf{C}ontext \textbf{D}istillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use. Specifically, TTCD uses a long-window teacher to supervise the fast weights of a short-window student, where the hidden-state discrepancy between them offers a dense, self-supervised signal guiding the model to memorize the contextual information crucial for future token predictions. We focus on an in-place variant: In-Place TTCD (IP-TTCD), which uses the existing MLP parameters as the fast weights. Experiments on long-context language modeling tasks show IP-TTCD consistently outperforms DeltaNet, Gated DeltaNet, sliding-window attention, and TTT when pre-trained from scratch. Furthermore, IP-TTCD allows pre-trained transformer models to adapt their parameters during inference through continual pre-training, gaining long-context capabilities with only a lightweight architectural augmentation. Our results position TTCD as a step toward architectural continual learning.

Figures

Figures reproduced from arXiv: 2608.01672 by Hengyu Fu, Jason D. Lee, Rui-Jie Zhu, Wenhao Chai, Xingyu Dang, Zixin Wen, Zixuan Wang.

Figure 1
Figure 1. Figure 1: Overview of TTCD. A shared and frozen backbone is run with two context lengths: a short-window student that observes only recent tokens and a long-window teacher that observes a longer history. TTCD treats the discrepancy between the teacher representation and the student’s projected hidden state as a context-distillation loss, and uses this loss to update the down-projection fast weight at test time. The … view at source ↗
Figure 2
Figure 2. Figure 2: Sliding window perplexity on the Books [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Table (a) Results on the MAD benchmark [36]. IP-TTCD attains the best overall score (82.1) and leads on the most memory-sensitive tasks (Fuzzy Recall, Compress, Memorize) that have not been saturated by prior baselines. Figure (b) Single-Needle-in-a-Haystack (S-NIAH-2) accuracy of 760M models from 4K to 32K. IP-TTCD consistently outperforms all baselines and degrades far more gracefully as the context grow… view at source ↗
Figure 4
Figure 4. Figure 4: Ablation studies on IP-TTCD design choices. We evaluate three key factors using 340M pre-training experiments on Long Data Collections. Left: ablating the causal convolutional feature maps and learned projection in the fast-weight update shows that combining both components yields the best long-context perplexity. Middle: increasing the number of IP-TTCD layers consistently improves sliding-window perplexi… view at source ↗
Figure 5
Figure 5. Figure 5: Prefill throughput with our fused inference kernels. FlashAttention-3 stack (IP-TTCD uses the LSE-combination kernel); mean of four repeats on H100. Across all three settings, IP￾TTCD’s overhead relative to IP-TTT is 1.3%–6.6% (Section B). Number of IP-TTCD layers We then consider how the performance scales with the number of IP-TTCD layers, which accounts for the state size of the implicit fast weight [P… view at source ↗
Figure 6
Figure 6. Figure 6: End-to-end prefill throughput on the FlashAttention-2 stack. Mean of four repeats on H100; IP-TTCD uses the fused dual-window and convolution kernels on its TTT layers. Across all settings the gap to IP-TTT is −1.3% to +3.2%. Numerical equivalence. Because IP-TTCD’s fast-weight recursion can amplify kernel-level round￾ing differences, we validate the kernels at three levels on the continually pre-trained c… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 8 canonical work pages

  1. [1]

    Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíˇcek, Agustín Piqueres Lajarín, Vaibhav Srivastav, et al. Smollm2: When smol goes big–data-centric training of a small language model.arXiv preprint arXiv:2502.02737, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Using fast weights to attend to the recent past.Advances in neural information processing systems, 29, 2016

    Jimmy Ba, Geoffrey E Hinton, V olodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past.Advances in neural information processing systems, 29, 2016

    2016

  3. [3]

    An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018

    Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.arXiv preprint arXiv:1803.01271, 2018

    Pith/arXiv arXiv 2018

  4. [4]

    Titans: Learning to memorize at test time

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Lessons from the trenches on reproducible evaluation of language models.arXiv preprint arXiv:2405.14782, 2024

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Ab- basi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models.arXiv preprint arXiv:2405.14782, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Kv-distill: Nearly lossless learnable context compression for llms.arXiv preprint arXiv:2503.10337, 2025

    Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms.arXiv preprint arXiv:2503.10337, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Learning to compress prompt in natural language formats

    Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7756–7767, 2024

    work page 2024

  8. [8]

    Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014

    Pith/arXiv arXiv 2014

  9. [9]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  10. [10]

    Finch: Prompt-guided key-value cache compression for large language models.Transactions of the Association for Computational Linguistics, 12: 1517–1532, 2024

    Giulio Corallo and Paolo Papotti. Finch: Prompt-guided key-value cache compression for large language models.Transactions of the Association for Computational Linguistics, 12: 1517–1532, 2024

    work page 2024

  11. [11]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024

    2024

  12. [12]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness.Advances in neural information processing systems, 35:16344–16359, 2022

    2022

  14. [14]

    Cartridges: Lightweight and general-purpose long context representations via self-study.arXiv, 2025

    Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re. Cartridges: Lightweight and general-purpose long context representations via self-study.arXiv, 2025. URLhttps://arxiv.org/abs/2506.06266

    Pith/arXiv arXiv 2025

  15. [15]

    In-place test-time training

    Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, and Tianle Cai. In-place test-time training. InThe Fourteenth International Conference on Learning Representations,

    work page

  16. [16]

    Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052, 2022

    Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models.arXiv preprint arXiv:2212.14052, 2022. 11

    Pith/arXiv arXiv 2022

  17. [17]

    The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling.arXiv preprint arXiv:2101.00027, 2020

    Pith/arXiv arXiv 2020

  18. [18]

    How to train long-context language models (effectively)

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7376–7399, 2025

    2025

  19. [19]

    Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms.arXiv preprint arXiv:2310.01801, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2021

    Pith/arXiv arXiv 2021

  22. [22]

    Log-linear attention.arXiv preprint arXiv:2506.04761, 2025

    Han Guo, Songlin Yang, Tarushii Goel, Eric P Xing, Tri Dao, and Yoon Kim. Log-linear attention.arXiv preprint arXiv:2506.04761, 2025

    arXiv 2025

  23. [23]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  24. [24]

    Long short-term memory.Neural computation, 9(8): 1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural computation, 9(8): 1735–1780, 1997

    1997

  25. [25]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395, 2024

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies.arXiv preprint arXiv:2404.06395, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Characteriz- ing prompt compression methods for long context inference.arXiv preprint arXiv:2407.08892, 2024

    Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, and Amir Gholami. Characteriz- ing prompt compression methods for long context inference.arXiv preprint arXiv:2407.08892, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Llmlingua: Compress- ing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compress- ing prompts for accelerated inference of large language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 13358–13376, 2023

    2023

  29. [29]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems, 37:106519–106554, 2024

    2024

  30. [30]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceedings of the 2021 conference on empirical methods in natural language processing, pages 3045–3059, 2021

    2021

  31. [31]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021

    2021

  32. [32]

    Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Llm knows what you are looking for before generation.Advances in Neural Information Processing Systems, 37:22947–22970, 2024

    2024

  33. [33]

    Leave no context behind: Efficient infinite context transformers with infini-attention.arXiv preprint arXiv:2404.07143, 101:15, 2024

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention.arXiv preprint arXiv:2404.07143, 101:15, 2024. 12

    Pith/arXiv arXiv 2024

  34. [34]

    Transformers are multi-state rnns

    Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi-state rnns. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18724–18741, 2024

    work page 2024

  35. [35]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. InFindings of the association for computational linguistics: EMNLP 2023, pages 14048–14077, 2023

    2023

  36. [36]

    Mechanistic design and scaling of hybrid architectures.arXiv preprint arXiv:2403.17844, 2024

    Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björn Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hybrid architectures.arXiv preprint arXiv:2403.17844, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018

    2018

  38. [38]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. InProceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, pages 9355–9366. PMLR, 2021. URLhttps://proceedings.mlr.press/v139/schlag21a/schlag21a.pdf

    work page 2021

  39. [39]

    FlashAttention-3: Fast and accurate attention with asynchrony and low-precision

    Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. InAdvances in Neural Information Processing Systems, 2024

    work page 2024

  40. [40]

    Learning by distilling context.arXiv preprint arXiv:2209.15189, 2022

    Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context.arXiv preprint arXiv:2209.15189, 2022

    Pith/arXiv arXiv 2022

  41. [41]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024

    2024

  42. [42]

    Test-time training with self-supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. InInternational conference on machine learning, pages 9229–9248. PMLR, 2020

    2020

  43. [43]

    Learning to (learn at test time): Rnns with expressive hidden states.arXiv preprint arXiv:2407.04620, 2024

    Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states.arXiv preprint arXiv:2407.04620, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675, 2025

    Arnuv Tandon, Karan Dalal, Xinhao Li, Daniel Koceja, Marcel Rød, Sam Buchanan, Xiaolong Wang, Jure Leskovec, Sanmi Koyejo, Tatsunori Hashimoto, et al. End-to-end test-time training for long context.arXiv preprint arXiv:2512.23675, 2025

    arXiv 2025

  45. [45]

    Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, et al. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Pith/arXiv arXiv 2025

  46. [46]

    Long data collections database, 2024

    TogetherAI. Long data collections database, 2024

    2024

  47. [47]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  48. [48]

    Rattention: Towards the minimal sliding window size in local-global attention models.arXiv preprint arXiv:2506.15545, 2025

    Bailin Wang, Chang Lan, Chong Wang, and Ruoming Pang. Rattention: Towards the minimal sliding window size in local-global attention models.arXiv preprint arXiv:2506.15545, 2025

    arXiv 2025

  49. [49]

    Fantastic pretraining optimizers and where to find them.arXiv preprint arXiv:2509.02046, 2025

    Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them.arXiv preprint arXiv:2509.02046, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule.arXiv preprint arXiv:2412.06464, 2024. 13

    Pith/arXiv arXiv 2024

  52. [52]

    Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491–115522, 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491–115522, 2024

    2024

  53. [53]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025

    2025

  54. [54]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th annual meeting of the association for computational linguistics, pages 4791–4800, 2019

    2019

  55. [55]

    Test-time training done right.arXiv preprint arXiv:2505.23884, 2025

    Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T Freeman, and Hao Tan. Test-time training done right.arXiv preprint arXiv:2505.23884, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    up to 1.3×

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023. 14 A Experiment Details A.1 Details of Training And Eva...

    work page arXiv 2023

  57. [2026]

    URLhttps://openreview.net/forum?id=dTWfCLSoyl

    work page