Pith. sign in

REVIEW 3 major objections 6 minor 21 references

SR-TTT does not learn retrieval: its reported gains were evaluation artifacts, and corrected evaluation yields 0% exact match across 2,250 trials.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:40 UTC pith:AQSP4FRY

load-bearing objection The abstract retracts the very paper in the body; the new negative result is real work but not verifiable from this submission, and the oracle-storage experiment needs scrutiny. the 3 major comments →

arxiv 2603.06642 v2 pith:AQSP4FRY submitted 2026-02-26 cs.LG cs.AIcs.CL

SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training

Swamynathan V P This is my paper
classification cs.LG cs.AIcs.CL MSC 68T07
keywords Test-Time TrainingSurprisal gatingHybrid memoryNeedle-in-a-HaystackEvaluation artifactCausal attentionExact recallNegative result
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper retracts the retrieval claims of an earlier version of SR-TTT, a hybrid test-time-training model that routes surprising tokens to an exact-attention cache. The original gains came from two evaluation errors: the loss and metric read logits at answer positions rather than one position earlier, and the cache attended non-causally over future tokens, including the answer itself. Under a corrected protocol, the model achieves 0% exact match in all 2,250 paired trials. The failure decomposes into a storage bottleneck (surprisal gating is position-biased and stores early needles at 0-1% containment) and an addressing bottleneck (content addressing extracts only about 0.06 nats of a 2.30-nat needle, and even oracle-perfect storage does not produce retrieval). The paper's value is a cautionary negative result and a reusable diagnostic protocol for surprise-gated memory architectures.

Core claim

The central claim is negative: SR-TTT does not learn retrieval, and the earlier positive results were measurement artifacts. With a corrected evaluation that reads logits one position earlier and enforces causality, exact match is 0% in all 2,250 paired trials across every condition. The paper decomposes the failure into two independent, separately measured bottlenecks: storage, where the surprisal filter's EMA threshold requires burn-in so early-context needles are stored at near-zero rates; and addressing, where even an oracle that perfectly stores needles supports content-based attention that extracts only ~0.06 of 2.30 nats, leaving token accuracy statistically unchanged after 2.5x more

What carries the argument

The paper's method for isolating the failure is the two-bottleneck decomposition. Storage is measured by containment: the fraction of a needle's tokens physically present in the residual cache. Addressing is measured by an oracle-storage experiment: feed perfectly stored needles to read-time attention and probe how much of the needle's information (in nats) is extracted, plus per-slot attention supervision. The original artifact is traced to two concrete mechanisms: logits read at answer positions instead of one position earlier, and non-causal cache attention that could attend to the answer token itself. The diagnostic protocol includes startup causality self-tests to catch both errors.

Load-bearing premise

The two-bottleneck decomposition assumes that the oracle-storage experiment cleanly isolates the addressing step; if the oracle leaks position information or interacts with the frozen TTT backbone, the measured addressing-mass increase would not cleanly prove that addressing is an independent failure.

What would settle it

Run the oracle-storage condition with position information removed from stored needles (shuffle or drop position encodings) while keeping the same read-time attention; if exact match rises above 0% or token accuracy changes materially, the addressing bottleneck is not independent of position. Conversely, store all early needles with a position-unbiased oracle; if exact match remains 0%, storage alone is not the barrier.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, surprisal-gated memory routing as implemented does not restore exact recall in TTT models; at best it leaves recall at zero.
  • Storage's position bias means the routing signal fails exactly where long-context memory is most needed: early-context needles require burn-in before they become relatively surprising.
  • Addressing over near-interchangeable ordered slots appears not to be solvable by content-based attention alone, even with perfectly stored keys and values.
  • The corrected evaluation protocol (causal self-tests, one-position-earlier logit reading) should be applied to any hybrid memory claim before reporting retrieval gains.
  • The released negative results provide a baseline for future attempts at surprise-gated memory: any proposed fix must clear the 0% bar in this protocol.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The two-bottleneck story suggests a deeper design principle: routing by reconstruction loss captures tokens that are hard to compress after burn-in, but recall-critical tokens in early context are not yet distinguishable; a position-aware or curriculum-based routing signal could be a testable fix.
  • The addressing bottleneck may generalize beyond TTT: any content-addressable memory faced with near-interchangeable, order-dependent items needs explicit slot indices or positional decoding; this is a hypothesis worth testing in linear-attention and state-space hybrids.
  • One can test the storage bottleneck directly by replacing the surprisal filter with an oracle that stores every token and checking whether exact match rises above 0%; if it does, the storage side is sufficient and addressing is the remaining wall.
  • The artifact decomposition (logit misalignment plus non-causal attention) may explain other reported 'massive gains' in small-scale memory-augmented models; scan prior work for the same two signatures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This manuscript is a correction paper for an earlier preprint. The corrected abstract claims that the previously reported SR-TTT gains on Needle-in-a-Haystack were evaluation artifacts: the loss and metric read logits at the answer positions rather than one position earlier, and the Residual Cache attended non-causally over future tokens including the answer. Under corrected evaluation, the abstract reports that exact match remains 0% in all 2,250 paired trials and that the failure decomposes into a storage bottleneck (surprisal gating stores early needles at 0–1% containment) and an addressing bottleneck (content addressing extracts only about 0.06 of 2.30 nats). The paper also releases a corrected implementation with startup causality self-tests and retracts the v1 claims. The body supplied in the manuscript, however, is the original v1 text, which reports large retrieval improvements and does not contain the corrected experimental protocols or results described in the abstract.

Significance. If the corrected negative result is substantiated, the paper would provide a valuable cautionary data point for surprise-gated memory architectures: routing high-surprisal tokens to an exact-attention cache does not, by itself, fix exact recall in TTT models. The artifact diagnosis—reading answer-position logits and non-causal cache attention—is plausible and important, and the proposed two-bottleneck decomposition, if cleanly measured, would be a useful mechanistic contribution. The paper also has strengths in releasing code, providing diagnostic self-tests, and testing against an external benchmark rather than against fitted parameters. However, the supplied manuscript body does not currently contain the corrected experiments, trial counts, containment measurements, or oracle-storage protocol; all load-bearing numbers appear only in the abstract. The significance of the paper therefore cannot be assessed from the submitted text as it stands.

major comments (3)
  1. [Abstract vs. §5] The central claim — "Exact match remains 0% in all 2,250 paired trials across all corrected conditions" — appears only in the abstract. The supplied body is the v1 manuscript: §5 reports "massive retrieval improvements" of +23% and +20% exact match, and Figure 4 is said to show gates opening to about 10%. There is no corrected methods section, no definition of containment, no trial protocol, no table of exact-match counts, and no description of how the two bottlenecks were measured. The abstract's numbers (0.06 → 0.15 addressing mass, ~0.06 of 2.30 nats, 0–1% containment at depth 0.1) cannot be audited. The manuscript must either include the full corrected experimental section or explicitly mark the v1 body as retracted and append the corrected results; otherwise the principal claim is unsupported by the submitted text.
  2. [Abstract / Eq. (1), §3.2, §5] The two-bottleneck decomposition depends on the oracle-storage experiment, but no implementation is reported. To isolate addressing, the oracle must give the read-time attention access to perfectly stored needles without leaking the answer through the fused output path in Eq. (1). The v1 body documents two leakage modes: the metric reads logits at answer positions, and the cache attends non-causally over future tokens including the answer itself. The corrected abstract does not state how these are controlled in the oracle experiment. The manuscript should report the causal-masking scheme, what exactly the oracle stores (e.g., keys/values versus raw token embeddings), and how α·CacheAttention is computed. A useful control would be to include a condition with oracle storage but scrambled or empty queries to estimate any direct-copy contribution. Without these details, the "addressing mass"
  3. [§5 and §5.1] The body's conclusions directly contradict the corrected abstract. Section 5 claims that SR-TTT substantially mitigates exact-recall failures, and §5.1 attributes the 4096-token collapse entirely to RoPE extrapolation. Both statements are incompatible with the corrected abstract's conclusion that the earlier gains were artifacts of the evaluation and that the method does not learn retrieval. The manuscript should state explicitly which figures, tables, and numerical claims come from v1, which have been superseded, and which are newly corrected. As written, readers cannot distinguish the retracted evidence from the evidence meant to replace it.
minor comments (6)
  1. [§3.1] The routing thresholds are not reported numerically. τ_EMA, the 0.8·τ_EMA chunk threshold, α_max, and the Stage 1/Stage 2 step split (7,000/3,000 steps) are free parameters; give their values and sensitivity, otherwise the surprising-token criterion is not reproducible.
  2. [Figures] Figures 1–4 are referenced but not present in the supplied text. At minimum, the corrected paper must include the actual figures or link to an accessible appendix; the current text's reliance on missing figures prevents verification of the v1 claims and any corrected versions.
  3. [Eq. (1)] The notation is unclear: α is called a "learned gate vector" and also a clamped scalar parameter. Specify its dimension, whether it is per-head or per-layer, and whether it depends on the current query hidden state.
  4. [§5] The v1 exact-match percentages are reported without trial counts, confidence intervals, or paired-trial details. The corrected abstract mentions 2,250 paired trials, but the body never ties those trials to the percentages.
  5. [References] Two references are listed as "Anonymous Authors" ([5] and [14]). If the companion papers have been revealed, update the citations; otherwise note that they are anonymized preprints.
  6. [Title/Abstract] The typeset title contains a spacing artifact: "AwareResidualTest-Time"; please fix. A few other spacing/grammar issues in the opening pages should also be cleaned up.

Circularity Check

0 steps flagged

No circularity: the corrected negative result is an external falsification, and the v1 self-reference is a retraction, not load-bearing.

full rationale

The paper's corrected central claim is that SR-TTT does not learn retrieval, with exact match at 0% in 2,250 paired trials under corrected evaluation. This is an externally measured negative result on the Needle-in-a-Haystack protocol, not a quantity derived from fitted parameters or definitions. The only self-reference is the v1 paper being corrected; the abstract explicitly retracts it ('We retract the claims of v1 and offer the corrected codebase...') and does not use v1 as evidence for the corrected conclusions. The two-bottleneck decomposition relies on an oracle-storage experiment whose implementation is not present in the supplied text, so the decomposition is under-verified and potentially confounded; however, that is an evidentiary gap, not a circular reduction, because the conclusion is not obtained by substituting the premise into itself. The v1 body does contain unverified 'massive improvements' claims, but the corrected abstract falsifies those using corrected metrics, and no equation in the supplied text is shown to be equivalent to its input by construction. No fitted parameter is relabeled as a prediction. Score 1 reflects the acknowledged self-citation to v1, which is not load-bearing.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 3 invented entities

The central corrected claims rest on the NIAH protocol, the oracle-storage decomposition, and the definition of correct evaluation. All are domain assumptions rather than derived mathematical facts; the manuscript provides no independent evidence beyond the author's own corrected experiments, and the supplied full text does not include those experiments.

free parameters (4)
  • τ_EMA = smoothed 95th percentile of recent reconstruction losses
    Routing threshold for the Surprisal Filter in the v1 architecture (Section 3.1); its value is a percentile threshold chosen by hand and affects which tokens are stored.
  • chunk loss threshold scale = 0.8 × τ_EMA
    Second condition in the dual-track routing (Section 3.1); a hand-set multiplier that gates storage.
  • α_max = not stated in provided text
    Upper clamp for the fusion gate α in Equation 1 (Section 3.2), set to avoid dead gradients; its value is a free design choice.
  • two-stage curriculum split = Stage 1 steps 1–7,000; Stage 2 steps 7,001–10,000
    Free training schedule that determines when the cache is enabled (Section 4.2); the corrected negative result may depend on this schedule.
axioms (4)
  • domain assumption Needle-in-a-Haystack with 8-character alphanumeric needles in TinyStories is a valid probe of exact-recall ability for TTT models.
    The corrected negative result and the v1 positive claims both treat this NIAH protocol as the ground truth for retrieval; if the probe is unsuitable, the conclusions about SR-TTT's failure do not transfer.
  • domain assumption The oracle-storage experiment isolates the addressing bottleneck from the storage bottleneck.
    Abstract states 'with storage solved by an oracle... per-slot attention supervision raises addressing mass 2.5x yet token accuracy is statistically unchanged'; this assumes the oracle adds no confounds.
  • domain assumption Reading logits one position earlier and enforcing causal cache access constitute the correct evaluation protocol.
    The entire retraction rests on defining the buggy vs. corrected metric; this assumes the corrected protocol is the right one for detecting retrieval rather than copying.
  • standard math TTT fast-weight gradient updates behave as a compression bottleneck that can be characterized by per-token reconstruction loss.
    Both v1 and the correction use per-token reconstruction loss as the surprisal signal (Section 3.1); this is a modeling choice, not an empirical fact.
invented entities (3)
  • Residual Cache no independent evidence
    purpose: Sparse exact-attention buffer that stores flagged high-surprisal tokens alongside the TTT fast weights.
    Introduced in v1 (Sections 1, 3.2); the corrected paper shows it provides no retrieval benefit (0% exact match). No falsifiable handle exists outside the paper's own benchmarks.
  • Surprisal Filter no independent evidence
    purpose: Loss-based gate that routes supposedly incompressible tokens to the cache.
    Introduced in v1 (Section 3.1); the correction shows its gating is position-biased, storing early needles at 0-1% containment. No external evidence.
  • read-time projections (new trainable projections) no independent evidence
    purpose: Attempt to improve addressing of stored slots in the corrected oracle-storage experiment.
    Mentioned in the abstract as 'new trainable read-time projections'; they raise addressing mass but not token accuracy, and no pre-registered external prediction is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 4475 in / 12831 out tokens · 116338 ms · 2026-08-02T20:40:19.455151+00:00 · methodology

0 comments
read the original abstract

Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks. Version 1 of this work proposed SR-TTT, which routes high-surprisal tokens to a sparse exact-attention Residual Cache, and reported large Needle-in-a-Haystack gains. We show those gains were evaluation artifacts: the loss and metric read logits at the answer positions rather than one position earlier, training both models to copy an answer already visible in their input (a model trained on retrieval-impossible data reaches 100% accuracy under the flawed metric); additionally, the cache attended non-causally over future tokens, including the answer itself. We release a corrected implementation with startup causality self-tests, then ask whether the SR-TTT hypothesis survives correction. It does not, and the failure decomposes into two independent, separately measured bottlenecks. Storage: surprisal gating is systematically position-biased - the TTT reconstruction loss requires burn-in before a needle becomes relatively surprising, so early-context needles are stored at near-zero rates (0-1% containment at depth 0.1) exactly where long-context memory matters most. Addressing: with storage solved by an oracle and with new trainable read-time projections, per-slot attention supervision raises addressing mass 2.5x (0.06 -> 0.15) yet token accuracy is statistically unchanged, and retrieval extracts only approx. 0.06 nats of the 2.30-nat needle; position-free content addressing cannot resolve ordered slots whose contents are near-interchangeable. Exact match remains 0% in all 2,250 paired trials across all corrected conditions. We retract the claims of v1 and offer the corrected codebase, diagnostic protocol, and negative results as a cautionary reference for surprise-gated memory architectures.

Figures

Figures reproduced from arXiv: 2603.06642 by Swamynathan V P.

Figure 1
Figure 1. Figure 1: Training loss comparison showing the Two-Stage Curriculum for SR-TTT. Graphs are stacked to display full [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overall loss trajectory comparison between the Pure TTT Baseline and Two-Stage SR-TTT methodologies. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Needle-in-a-Haystack Exact Match performance at sequence length 2048. SR-TTT demonstrates massive [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cache utilization statistics demonstrating saturation and Layer 1–3 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 7 linked inside Pith

  1. [1]

    (2017).Attention Is All You Need

    Vaswani, A., et al. (2017).Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS)

    2017

  2. [2]

    (2020).Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

    Sun, Y ., Wang, X., Liu, Z., Miller, J., Efros, A., & Hardt, M. (2020).Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. International Conference on Machine Learning (ICML)

    2020

  3. [3]

    (2024).Learning to (Learn at Test Time): RNNs with Expressive Hidden States

    Sun, Y ., et al. (2024).Learning to (Learn at Test Time): RNNs with Expressive Hidden States. International Conference on Machine Learning (ICML)

    2024

  4. [4]

    (2025).Test-Time Training Done Right

    Zhang, T., et al. (2025).Test-Time Training Done Right. arXiv preprint arXiv:2505.23884

    Pith/arXiv arXiv 2025

  5. [5]

    (2026).Gated Differentiable Working Memory (GDWM) for Long-Context Language Modeling

    Anonymous Authors. (2026).Gated Differentiable Working Memory (GDWM) for Long-Context Language Modeling. arXiv preprint arXiv:2601.12906

    arXiv 2026

  6. [6]

    F., et al

    Liu, N. F., et al. (2024).Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics (TACL)

    2024

  7. [7]

    (2023).Needle In A Haystack – Pressure Testing LLMs

    Kamradt, G. (2023).Needle In A Haystack – Pressure Testing LLMs. GitHub Repository

    2023

  8. [8]

    (2023).Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Gu, A., & Dao, T. (2023).Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint arXiv:2312.00752

    Pith/arXiv arXiv 2023

  9. [9]

    (2023).RWKV: Reinventing RNNs for the Transformer Era

    Peng, B., et al. (2023).RWKV: Reinventing RNNs for the Transformer Era. Findings of EMNLP

    2023

  10. [10]

    (2025).Titans: Learning to Memorize at Test Time

    Behrouz, A., et al. (2025).Titans: Learning to Memorize at Test Time. arXiv preprint arXiv:2501.00663

    Pith/arXiv arXiv 2025

  11. [11]

    (2024).Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff

    Arora, S., et al. (2024).Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff. International Conference on Machine Learning (ICML)

    2024

  12. [12]

    (2024).Zoology: Measuring and Improving Recall in Efficient Language Models

    Arora, S., et al. (2024).Zoology: Measuring and Improving Recall in Efficient Language Models. International Conference on Learning Representations (ICLR)

    2024

  13. [13]

    (2024).Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

    Dong, H., et al. (2024).Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference. International Conference on Machine Learning (ICML)

    2024

  14. [14]

    (2025).Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs

    Anonymous Authors. (2025).Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs. arXiv preprint arXiv:2512.03324

    arXiv 2025

  15. [15]

    (2024).MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

    Jiang, H., et al. (2024).MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention. Advances in Neural Information Processing Systems (NeurIPS)

    2024

  16. [16]

    (2024).KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantiza- tion

    Hooper, C., et al. (2024).KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantiza- tion. arXiv preprint arXiv:2401.18079

    Pith/arXiv arXiv 2024

  17. [17]

    (2024).H 2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

    Zhang, Z., et al. (2024).H 2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. Advances in Neural Information Processing Systems (NeurIPS). 6 SR-TTT: Surprisal-Aware Residual Test-Time Training

    2024

  18. [18]

    (2024).Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

    Nawrot, P., et al. (2024).Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference. Interna- tional Conference on Machine Learning (ICML)

    2024

  19. [19]

    (2023).TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

    Eldan, R., & Li, Y . (2023).TinyStories: How Small Can Language Models Be and Still Speak Coherent English?. arXiv preprint arXiv:2305.07759

    Pith/arXiv arXiv 2023

  20. [20]

    (2021).RoFormer: Enhanced Transformer with Rotary Position Embedding

    Su, J., et al. (2021).RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864

    Pith/arXiv arXiv 2021

  21. [21]

    (2023).YaRN: Efficient Context Window Extension of Large Language Models

    Peng, B., et al. (2023).YaRN: Efficient Context Window Extension of Large Language Models. arXiv preprint arXiv:2309.00071. 7

    Pith/arXiv arXiv 2023