Pith. sign in

REVIEW 2 major objections 2 minor 89 cited by

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

T0 review · 2 major / 2 minor · reviewed 2026-05-15 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read RNNs can match long-context performance by updating a learnable hidden-state model via self-supervised steps at test time.

desk verdict TTT layers turn the hidden state into a model trained on the fly at test time, and the scaling curves show continued perplexity gains past where Mamba flattens. read the letter →

arxiv 2407.04620 v4 pith:RCPETT5H submitted 2024-07-05 cs.LG cs.AIcs.CL

Yu Sun , Xinhao Li , Karan Dalal , Jiarui Xu , Arjun Vikram , Genghan Zhang , Yann Dubois , Xinlei Chen
show 4 more authors
Xiaolong Wang Sanmi Koyejo Tatsunori Hashimoto Carlos Guestrin
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords test-timetrainingRNNlongcontextsequencemodelinglinearcomplexityhiddenstateself-supervisedupdate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows how to build sequence layers that combine linear complexity with high expressive power. The hidden state is no longer a fixed vector but a small machine learning model whose parameters are adjusted by gradient descent on the incoming test sequence itself. This lets the layer adapt its representation to the specific data seen so far, so that perplexity keeps falling as more tokens arrive. A reader would care because the approach avoids both the quadratic cost of attention and the performance saturation of conventional RNNs after roughly 16k tokens.

What carries the argument

The TTT layer, whose hidden state is itself a small model updated by one or more gradient steps of self-supervised learning on the current test sequence.

What would settle it

A controlled run in which TTT-Linear or TTT-MLP stops improving perplexity after 16k tokens or begins to produce unstable outputs when the test-time updates are enabled.

Watch

Extended reading notes

Core claim

TTT layers instantiate the hidden state as a trainable model and replace the usual recurrence with a step of self-supervised learning performed on the test sequence. For the two concrete cases examined, TTT-Linear uses a linear model and TTT-MLP uses a two-layer network; both keep lowering perplexity when conditioned on longer contexts, while a strong Mamba baseline plateaus after 16k tokens. The evaluation covers models from 125M to 1.3B parameters and directly compares against a Transformer baseline.

Load-bearing premise

Gradient-based self-supervised updates performed on the hidden-state model during inference stay stable, cheap enough to run, and do not overfit or degrade the output.

Editorial extensions

If this is right

  • Linear-complexity layers can continue to benefit from additional context beyond the point where fixed-state RNNs saturate.
  • The same architecture family can be scaled from 125M to over a billion parameters while preserving the long-context scaling behavior.
  • Memory and compute trade-offs shift from attention's quadratic growth to the cost of storing and updating the internal model parameters.
  • Future layer designs can focus on improving the I/O efficiency of the gradient steps without changing the core recurrence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Dynamic adaptation of the hidden state could reduce reliance on extremely long fixed context windows if the model learns useful patterns from recent tokens alone.
  • The same mechanism might be applied to online settings where new data arrives continuously and the model must improve without a separate training phase.
  • If the internal model can be made lighter, TTT layers could serve as drop-in replacements for attention in resource-constrained inference environments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper proposes Test-Time Training (TTT) layers as a framework for sequence modeling with linear complexity but expressive hidden states. The hidden state is instantiated as a learnable model (linear regressor or 2-layer MLP) whose parameters are updated via self-supervised gradient steps on the input sequence at test time. Two variants, TTT-Linear and TTT-MLP, are evaluated at 125M–1.3B parameter scales against a strong Transformer baseline and Mamba; the key empirical result is that TTT models continue to reduce perplexity as context grows beyond 16k tokens while Mamba plateaus.

Significance. If the central empirical claim holds, the work supplies a concrete route to linear-complexity models whose hidden states adapt via test-time learning, yielding continued gains on long contexts where standard RNNs saturate. The scaling experiments to 1.3B parameters and direct head-to-head comparisons with Mamba and Transformer constitute reproducible empirical evidence that strengthens the case for test-time adaptation as a viable direction.

major comments (2)
  1. [§4 (Experiments)] §4 (Experiments): The claim that TTT-Linear/MLP continue reducing perplexity with >16k tokens while Mamba plateaus depends on the hidden-state model receiving stable, beneficial self-supervised gradient updates at inference. The section reports final perplexity numbers but provides no analysis of update stability (gradient norms, per-step loss trajectories, or divergence checks) or sensitivity to the number of gradient steps and learning-rate schedule used during test-time training. This is load-bearing for the scaling advantage.
  2. [§3 (Method)] §3 (Method): The update rule for the hidden-state parameters (linear or MLP) is defined as a self-supervised step, yet the manuscript does not specify the exact optimizer, step count per token/segment, or regularization used at test time. Without these details it is impossible to assess whether the reported linear-complexity advantage remains tractable and non-overfitting at 1.3B scale.
minor comments (2)
  1. [Abstract] Abstract and §4: The phrase 'memory I/O issues for TTT-MLP' is stated without any quantitative breakdown (e.g., peak memory vs. context length or wall-clock overhead relative to Mamba). Adding a short table or plot would clarify the practical limitation.
  2. [§3 (Method)] Notation in §3: The symbols for the hidden-state model parameters and the self-supervised loss are introduced without an explicit table of definitions, making cross-references to the update equations harder to follow.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognizing the potential of TTT layers for long-context scaling. We address each major comment below and will incorporate the requested details and analyses into the revised manuscript.

read point-by-point responses

  1. Referee: [§4 (Experiments)] The claim that TTT-Linear/MLP continue reducing perplexity with >16k tokens while Mamba plateaus depends on the hidden-state model receiving stable, beneficial self-supervised gradient updates at inference. The section reports final perplexity numbers but provides no analysis of update stability (gradient norms, per-step loss trajectories, or divergence checks) or sensitivity to the number of gradient steps and learning-rate schedule used during test-time training. This is load-bearing for the scaling advantage.

    Authors: We agree that stability analysis is necessary to support the central empirical claim. In the revised version we will add to §4 new figures and text reporting (i) gradient-norm trajectories during test-time updates on long sequences, (ii) per-step self-supervised loss curves on held-out segments, (iii) explicit checks for divergence or instability, and (iv) ablation tables showing sensitivity of final perplexity to the number of gradient steps and the learning-rate schedule used at test time. These additions will directly substantiate that the observed scaling advantage arises from stable, beneficial updates. revision: yes

  2. Referee: [§3 (Method)] The update rule for the hidden-state parameters (linear or MLP) is defined as a self-supervised step, yet the manuscript does not specify the exact optimizer, step count per token/segment, or regularization used at test time. Without these details it is impossible to assess whether the reported linear-complexity advantage remains tractable and non-overfitting at 1.3B scale.

    Authors: We acknowledge the omission of precise test-time hyperparameters. The revised §3 will explicitly state the optimizer (Adam with β1=0.9, β2=0.999), the exact number of gradient steps performed per token or per segment, the learning-rate value and any decay schedule, and the regularization applied (weight decay of 0.01 together with gradient clipping at norm 1.0). These details will be provided for both TTT-Linear and TTT-MLP so that readers can verify tractability and reproducibility at the 1.3B scale. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; architectural proposal with direct empirical validation

full rationale

The paper defines TTT layers by making the hidden state itself a learnable model (linear or 2-layer MLP) whose parameters are updated via a self-supervised gradient step on each test token or segment. This is an explicit architectural choice, not a mathematical derivation that reduces to prior equations or fitted inputs. No load-bearing self-citations, uniqueness theorems from the same authors, or ansatzes smuggled via prior work appear in the core construction. The central scaling claim (TTT continues reducing perplexity beyond 16k tokens while Mamba plateaus) rests on direct experimental comparisons at 125M–1.3B scale rather than any reduction of outputs to inputs by construction. The method is therefore self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

The central claim rests on the domain assumption that self-supervised test-time updates can meaningfully increase hidden-state expressiveness in linear-complexity layers; no free parameters or invented entities are quantified in the abstract.

assumptions (1)
  • domain assumption Self-supervised gradient updates on a small model serving as hidden state improve expressiveness without instability at test time
    This is the load-bearing premise that allows linear complexity to coexist with high capacity.
invented entities (1)
  • TTT layer
    purpose: Sequence modeling layer whose hidden state is a trainable model updated at test time
    New architectural primitive introduced to overcome limited expressiveness of standard RNN states.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to (Learn at Test Time): RNNs with Expressive Hidden States." pith.science (2026). https://pith.science/paper/RCPETT5H

@misc{pith2026240704620,
  author       = {Pith},
  title        = {Pith review of: Learning to (Learn at Test Time): RNNs with Expressive Hidden States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RCPETT5H}},
  note         = {Machine review of arXiv:2407.04620}
}
read the original abstract

Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 89 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 89 Pith citations

  1. Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

    cs.AI 2026-06 unverdicted novelty 8.0 of 10

    CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.

  2. WriteSAE: Sparse Autoencoders for Recurrent State

    cs.LG 2026-05 unverdicted novelty 8.0 of 10

    WriteSAE is the first sparse autoencoder that factors decoder atoms into the native d_k x d_v cache write shape of recurrent models and supplies a closed-form per-token logit shift for atom substitution.

  3. PhyCheck: Fine-Grained Evidence-Grounded Dataset for Physical Law Understanding in Video-LLMs

    cs.CV 2026-08 conditional novelty 7.0 of 10

    PhyCheck is a 69,825-pair video QA benchmark that tests and improves Video-LLMs' ability to judge whether events obey physical laws, with fine-grained evidence questions and a context-sensitivity pilot.

  4. RoboTTT: Context Scaling for Robot Policies

    cs.RO 2026-07 conditional novelty 7.0 of 10

    A robot policy that updates its own weights during deployment can use 8,000 steps of history, steadily improving as context grows and enabling one-shot imitation from human videos.

  5. Forget, Anticipate and Adapt: Test Time Training for Long Videos

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    FFN performs TTT on multi-hour videos by restricting updates to three frames and using a surprise metric for adaptive window sizing, plus a new EpicTours dataset.

  6. Tapered Language Models

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Tapered Language Models monotonically decrease MLP width across depth with a cosine schedule, yielding better perplexity and downstream performance than uniform-width baselines across multiple architectures and scales...

  7. RoboMME-Interference: Benchmarking Robot Memory Under Interference

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    RoboMME-Interference measures VLA memory under cross-session interference and finds that all tested systems decay to near their no-memory baseline as unrelated sessions accumulate.

  8. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    AURA-Mem uses an action-gated recurrent memory trained on closed-loop action error to deliver constant 4,224-byte state and 5-9x fewer writes than baselines while matching base policy success on LIBERO-Long.

  9. Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    TTRL-Guard mitigates the Correct-Answer Extinction Window in test-time RL via flip-rate-aware reward scaling, minority-preserving sampling, and risk-conditioned sparse updates, yielding best average pass@1 on Qwen mod...

  10. Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    Mamba-VGGT introduces a Sliding Window Mamba memory module and Zero-Init Spatial Memory Injector to enable persistent long-range geometric reasoning in VGGT for extended video sequences.

  11. Test-Time Learning with an Evolving Library

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    EvoLib improves black-box LLM test-time performance by maintaining an evolving, self-scored library of reusable skills and insights, without parameter updates or ground-truth feedback.

  12. Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning

    cs.LG 2026-03 conditional novelty 7.0 of 10

    A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.

  13. Test-time Offline Reinforcement Learning on Goal-related Experience

    cs.LG 2025-07 unverdicted novelty 7.0 of 10

    GC-TTT adapts goal-conditioned policies at test time by fine-tuning on self-supervised selected goal-related offline data, yielding performance gains in loco-navigation and manipulation tasks.

  14. pLSTM: parallelizable Linear Source Transition Mark networks

    cs.LG 2025-06 conditional novelty 7.0 of 10

    pLSTM extends linear recurrent networks to general directed acyclic graphs with a parallelizable scheme and two stabilization modes for long-range propagation.

  15. MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

    cs.LG 2025-06 conditional novelty 7.0 of 10

    MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.

  16. Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond

    cs.LG 2024-12 reject novelty 7.0 of 10

    A theoretical analysis claims attention networks fail to learn residual features when time series steps have opposite signs, giving a possible explanation for the known advantage of linear residual models.

  17. Marconi: Prefix Caching for the Era of Hybrid LLMs

    cs.DC 2024-11 conditional novelty 7.0 of 10

    A prefix cache for hybrid attention-SSM LLMs that selectively checkpoints SSM states and evicts by FLOP efficiency, improving token hit rates and time-to-first-token.

  18. Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues

    cs.LG 2024-11 conditional novelty 7.0 of 10

    Restricting state-transition eigenvalues to [0,1] is the root cause of linear RNNs' failure on parity and modular counting; extending to [-1,1] provably enables state-tracking and recognition of all regular languages.

  19. Metis: Memory Foundation Model

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Metis puts a trainable fixed-size memory matrix inside a frozen LLM backbone and learns to remember, update, forget, and reflect across turns without replaying original context.

  20. Rethinking Expressivity and Efficiency in Test-Time Training

    cs.LG 2026-08 conditional novelty 6.0 of 10

    E2-TTT derives closed-form scalar kernels that make the chunk-end fast-weight and momentum states of per-token TTT updates exactly parallelizable, improving long-context retrieval and extrapolation at up to 1.3B parameters.

  21. Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Proteus progressively expands the active fraction of a fixed-size memory state over the context, improving long-context performance in four recurrent architectures without adding parameters.

  22. Consolidator: Learning Persistent Routed Memory Across Context Boundaries

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A learned slot-local transform lets a frozen backbone update and retrieve a persistent memory after context reset, and using that memory to guide routing nearly doubles updated-mapping recall.

  23. Modular TTT: Rethinking Test-Time Training as Composable Modules

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Modular TTT expresses test-time training as a graph of primitives, ablates the components, and finds that simple shallow learners with small learning-rate initialization and scalar decay match Gated DeltaNet at 1.45B scale.

  24. Learning What to Remember: Test-Time Training via Context Distillation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    IP-TTCD distills the hidden-state gap between a long-window teacher and a short-window student into MLP fast weights, improving long-context language modeling and retrieval over DeltaNet, Gated DeltaNet, SWA, and IP-TTT.

  25. Raven: High-Recall Sequence Modeling with Sparse Memory Routing

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.

  26. Black-Mamba: Biologically-Inspired Leaky Accumulation for Conceptual Knowledge under Distribution Drift

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Gating test-time memory writes on leaky accumulated surprisal preserves most of the adaptation benefit while roughly halving the number of updates.

  27. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SDM sparsifies the Gated DeltaNet update rule to enable 1000x larger recurrent memory states at iso-FLOP, improving long-context recall and short-context reasoning over GDN and matching full attention at 8B scale.

  28. NoDrift3R: Raymap-Guided Coupling for Drift-Robust Unposed Feed-Forward 3D Reconstruction

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Anchoring Gaussian centers to predicted raymaps and jointly optimizing RGB, raymap, and camera losses with a dual-frequency curriculum suppresses pose drift and improves pose-free 3D reconstruction on long sequences.

  29. Learning the ARTS of Search for Automated Discovery

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    ARTS improves automated scientific discovery by using reasoning LMs with test-time training to separate hypothesis merit from execution quality in tree search, achieving 15.3% relative gains on 22 MLGym and MLEBench tasks.

  30. Test-Time Training with Next-Token Prediction

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    TTT-NTP adapts pretrained LLMs at test time by training fast weights to match next-position hidden states from the forward pass, yielding consistent gains on long-context benchmarks across Llama, Mistral, and Qwen models.

  31. Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    Reinforcement learning after SFT conversion narrows the performance gap between sliding-window attention and full self-attention on math reasoning benchmarks while preserving linear complexity.

  32. U-TTT: Towards Generalizable PET Image Denoising via Test-Time Training

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    U-TTT adds spatial and frequency-domain test-time training layers to a U-Net architecture to adapt PET denoising models at inference time for improved generalization across unseen dose levels and scanners.

  33. Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Sleep-time Knowledge Seeding plus Dreaming lets LLMs expand capacity, distill fragile in-context memories into stable parameters, and self-improve without human labels.

  34. AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    Introduces AgentOdyssey, a procedural generator of open-ended long-horizon text games, to evaluate test-time continual learning agents and diagnose limits in exploration, memory, and planning.

  35. Universal Time Series Generation with Neural Controlled Differential Equations

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Proves SLiCEs are universal time-series generators approximating path laws in W_∞ and proposes G-SLiCEs for path-space flow matching with benefits on irregular grids.

  36. Position: Modular Memory is the Key to Continual Learning Agents

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A modular memory combining in-context learning and in-weight learning is proposed as the key to continual learning agents.

  37. Listen to Rhythm, Choose Movements: Autoregressive Multimodal Dance Generation via Diffusion and Mamba with Decoupled Dance Dataset

    cs.GR 2026-01 unverdicted novelty 6.0 of 10

    LRCM is a new multimodal diffusion model with audio and text Conformers plus Motion Temporal Mamba for generating long, coherent dance sequences from rhythm and descriptions using a decoupled dataset.

  38. Higher-order Linear Attention

    cs.LG 2025-10 unverdicted novelty 6.0 of 10

    Higher-order Linear Attention realizes second-order and higher interactions in linear-time causal attention via constant-size state and associative scans.

  39. Short window attention enables long-term memorization

    cs.LG 2025-09 unverdicted novelty 6.0 of 10

    Short sliding windows in hybrid attention-xLSTM models boost long-context performance by encouraging long-term memory use, and stochastic window sizing improves both short and long tasks.

  40. ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution

    cs.CL 2025-09 unverdicted novelty 6.0 of 10

    ShinkaEvolve improves sample efficiency in LLM-driven program evolution via parent sampling, code novelty rejection-sampling, and bandit LLM ensemble selection, achieving new SOTA circle packing with 150 samples and g...

  41. Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A chunk-wise inference method called Smooth Reading lets recurrent LLMs match or exceed self-attention LLMs on LongBench and Needle-in-a-Haystack while keeping linear-time efficiency.

  42. Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Selecting the most confident 32-token prefix and completing only it gives better accuracy per compute than majority voting on five math reasoning datasets, using only the model's own confidence as a selector.

  43. Energy-Based Transformers are Scalable Learners and Thinkers

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...

  44. Cartridges: Lightweight and general-purpose long context representations via self-study

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.

  45. Test-Time Training Done Right

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Large-chunk online updates during inference let test-time training scale state capacity to 40% of model size and handle contexts up to 1M tokens without custom kernels.

  46. Long-Context State-Space Video World Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A hybrid state-space and local-attention architecture gives autoregressive video diffusion models long-term spatial memory with constant per-frame inference cost, demonstrated on Maze and Minecraft.

  47. Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    TTMM pre-trains many local LoRA experts on data clusters and merges the most relevant few at test time, approximating test-time training with a 100x speedup and near-TTT perplexity.

  48. UrbanMind: Urban Dynamics Prediction with Multifaceted Spatial-Temporal Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    UrbanMind combines a multifaceted masked autoencoder, semantic prompting, and test-time adaptation in an LLM to forecast traffic speed, inflow, and demand, reporting lower MAE and RMSE than baselines in three cities.

  49. Overflow Prevention Enhances Long-Context Recurrent LLMs

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OPRM, a training-free chunk-and-select inference method, improves recurrent LLMs by 14-51% on LongBench and sets a 7B-class LongBench v2 record, while raising doubts about whether recurrent models exploit long-range d...

  50. Rethinking Temporal Fusion with a Unified Gradient Descent View for 3D Semantic Occupancy Prediction

    cs.CV 2025-04 conditional novelty 6.0 of 10

    GDFusion fuses scene, motion, and geometry cues through gradient-descent-style RNN updates, improving mIoU by 1.4 to 4.8 points on Occ3D while cutting inference memory by 27 to 72 percent.

  51. LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning

    cs.CL 2025-02 unverdicted novelty 6.0 of 10

    LIFT fine-tunes short-context LLMs on long inputs with synthetic tasks to absorb information into parameters, enabling answers without the input present at inference.

  52. Test-time regression: a unifying framework for designing sequence models with associative memory

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Sequence models that perform associative recall are unified as 'test-time regression' layers, reducing to weighted, kernel, or online least-squares regression depending on three design choices.

  53. Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing

    cs.LG 2024-12 conditional novelty 6.0 of 10

    State space models have an inherent recency bias and over-smoothing in deep stacks, and a two-channel polarization fix improves long-range associative recall.

  54. Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognition

    eess.AS 2024-12 conditional novelty 6.0 of 10

    Structured speaker and deficiency adapters, applied during fine-tuning and test-time adaptation, lower word error rates for HuBERT and Wav2vec2-conformer on dysarthric and elderly speech and set new state-of-the-art results.

  55. Learning to Generate Gradients for Test-Time Adaptation via Test-Time Training Layers

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A learned gradient generator with a gradient memory layer improves test-time adaptation accuracy and speed on ImageNet-C/R/Sketch/A by refining noisy unsupervised gradients before model updates.

  56. LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LIFT adapts a short-context LLM by fine-tuning it on overlapping chunks of the test input, improving some long-context benchmarks when combined with in-context learning but degrading others.

  57. MobileMamba: Lightweight Multi-Receptive Visual Mamba Network

    cs.CV 2024-11 conditional novelty 6.0 of 10

    MobileMamba achieves 73.6 to 83.6% ImageNet top-1 accuracy across six model sizes, with GPU throughput up to roughly 21 times that of LocalVim, by combining wavelet-enhanced Mamba and multi-kernel depthwise convolutio...

  58. DATTA: Domain-Adversarial Test-Time Adaptation for Cross-Domain WiFi-Based Human Activity Recognition

    cs.CV 2024-11 conditional novelty 6.0 of 10

    DATTA merges domain-adversarial training, test-time feature alignment, and random weight resetting to reach 68.13 mean F1 on cross-domain WiFi HAR, beating a naive SotA combination by 8.1% relative.

  59. Preliminary Evaluation of the Test-Time Training Layers in Recommendation System (Student Abstract)

    cs.IR 2024-11 conditional novelty 6.0 of 10

    TTT4Rec, a sequential recommender using TTT-Linear as its feature extractor, performs comparably to or better than DIN, GRU4Rec, SASRec, and ComiRec on Beauty, Electronics, and MovieLens-1M.

  60. Maglev: Sliding Recurrent Memory

    cs.LG 2026-08 conditional novelty 5.0 of 10

    Using a full-context prefiller to write memory targets, Maglev trains a sliding-window recurrent transformer that keeps bounded memory at inference and improves language-modeling benchmarks.

See all 89 Pith citations

Reference graph

Works this paper leans on

85 extracted references · 85 canonical work pages · cited by 89 Pith papers (see all)

  1. [1]

    A disciplinary discourse perspective on university science learning: Achieving fluency in a critical constellation of modes

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    work page Pith review arXiv 2023

  2. [2]

    Learning to learn by gradient descent by gradient descent

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016

    work page 2016

  3. [3]

    You just found out your book was used to train ai

    Authors Guild. You just found out your book was used to train ai. now what?, 2023. Accessed: 2024-06-24

    work page 2023

  4. [4]

    xLSTM: Extended Long Short-Term Memory

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Ex- tended long short-term memory. arXiv preprint arXiv:2405.04517, 2024

    work page Pith review arXiv 2024

  5. [5]

    Learning a synaptic learning rule

    Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Citeseer, 1990

    work page 1990

  6. [6]

    The nadaraya-watson kernel regression function estimator

    Hermanus Josephus Bierens. The nadaraya-watson kernel regression function estimator. (Serie Research Memoranda; No. 1988-58). Faculty of Economics and Business Administration, Vrije Universiteit Amsterdam., 1988

    work page 1988

  7. [7]

    Pattern recognition and machine learning , volume 4

    Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006

    work page 2006

  8. [8]

    GPT-NeoX-20B: An Open-Source Autoregressive Language Model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    work page Pith review arXiv 2022

Show all 85 references
  1. [9]

    Local learning algorithms.Neural computation, 4(6):888–900, 1992

    Léon Bottou and Vladimir Vapnik. Local learning algorithms.Neural computation, 4(6):888–900, 1992

    1992

  2. [10]

    Variable kernel estimates of multivariate densities

    Leo Breiman, William Meisel, and Edward Purcell. Variable kernel estimates of multivariate densities. Technometrics, 19(2):135–144, 1977

    1977

  3. [11]

    Weighted nadaraya–watson regression estimation

    Zongwu Cai. Weighted nadaraya–watson regression estimation. Statistics & probability letters, 51(3):307–318, 2001

    2001

  4. [12]

    Training deep nets with sublinear memory cost, 2016

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016

    2016

  5. [13]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020

    2003 arXiv

  6. [14]

    A tutorial on kernel density estimation and recent advances

    Yen-Chi Chen. A tutorial on kernel density estimation and recent advances. Biostatistics & Epidemiology, 1(1):161–187, 2017

    2017

  7. [15]

    Meta-learning fast weight language models

    Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Moham- mad Norouzi. Meta-learning fast weight language models. arXiv preprint arXiv:2212.02475, 2022

    2022

  8. [16]

    Large scale transductive svms

    Ronan Collobert, Fabian Sinz, Jason Weston, Léon Bottou, and Thorsten Joachims. Large scale transductive svms. Journal of Machine Learning Research, 7(8), 2006

    2006

  9. [17]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 20

    2024 arXiv

  10. [18]

    Griffin: Mixing gated linear recurrences with local attention for e fficient language models

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for e fficient language models. arXiv preprint...

    2024

  11. [19]

    In the long (context) run, 2023

    Harm de Vries. In the long (context) run, 2023. Accessed: 2024-06-24

    2023

  12. [20]

    Dynamic connections in neural networks.Biological cybernetics, 46(1):27–39, 1982

    Jerome A Feldman. Dynamic connections in neural networks.Biological cybernetics, 46(1):27–39, 1982

    1982

  13. [21]

    Model-agnostic meta-learning for fast adapta- tion of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017

    2017

  14. [22]

    Gammerman, V

    A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. In In Uncertainty in Artificial Intelligence, pages 148–155. Morgan Kaufmann, 1998

    1998

  15. [23]

    Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 2022

    2022

  16. [24]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020

    2020

  17. [25]

    EasyLM: A Simple And Scalable Training Framework for Large Language Models

    Xinyang Geng. EasyLM: A Simple And Scalable Training Framework for Large Language Models. https://github.com/young-geng/EasyLM, mar 2023. https://github.com/ young-geng/EasyLM

    2023

  18. [26]

    Unlocking state-tracking in linear rnns through negative eigenvalues

    Riccardo Grazzi, Julien Siems, Arber Zela, Jörg KH Franke, Frank Hutter, and Massimiliano Pontil. Unlocking state-tracking in linear rnns through negative eigenvalues. International Conference on Learning Representations (ICLR), 2024

    2024

  19. [27]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  20. [28]

    Self-supervised policy adaptation during deployment

    Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. arXiv preprint arXiv:2007.04309, 2020

    2007

  21. [29]

    Test-time training on nearest neighbors for large language models

    Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv preprint arXiv:2305.18466, 2023

    2023

  22. [30]

    predictable

    Horace He. Strangely, matrix multiplications on gpus run faster when given "predictable" data! [short], 2024. Accessed: 2024-06-30

    2024

  23. [31]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  24. [32]

    Using fast weights to deblur old memories

    Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. InProceedings of the ninth annual conference of the Cognitive Science Society, pages 177–186, 1987

    1987

  25. [33]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997

    1997

  26. [34]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  27. [35]

    The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention

    Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639–9659. PMLR, 2022

    2022

  28. [36]

    Practical computational power of linear transformers and their recurrent and self-referential extensions

    Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber. Practical computational power of linear transformers and their recurrent and self-referential extensions. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  29. [37]

    Neural di fferential equations for learning to program neural nets through continuous learning rules

    Kazuki Irie, Francesco Faccio, and Jürgen Schmidhuber. Neural di fferential equations for learning to program neural nets through continuous learning rules. Advances in Neural Information Processing Systems, 35:38614–38628, 2022

    2022

  30. [38]

    Going beyond linear transformers with recurrent fast weight programmers.Advances in Neural Information Processing Systems, 34:7703–7717, 2021

    Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers.Advances in Neural Information Processing Systems, 34:7703–7717, 2021

    2021

  31. [39]

    A modern self-referential weight matrix that learns to modify itself

    Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In International Conference on Machine Learning , pages 9660–9677. PMLR, 2022

    2022

  32. [40]

    Images as weight matrices: Sequential image generation through synaptic learning rules

    Kazuki Irie and Jürgen Schmidhuber. Images as weight matrices: Sequential image generation through synaptic learning rules. International Conference on Learning Representations (ICLR), 2022

    2022

  33. [41]

    Online domain adaptation of a pre-trained cascade of classifiers

    Vidit Jain and Erik Learned-Miller. Online domain adaptation of a pre-trained cascade of classifiers. In CVPR 2011, pages 577–584. IEEE, 2011

    2011

  34. [42]

    Learning to classify text using support vector machines, volume 668

    Thorsten Joachims. Learning to classify text using support vector machines, volume 668. Springer Science & Business Media, 2002

    2002

  35. [43]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  36. [44]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  37. [45]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  38. [46]

    Meta learning backpropagation and improving it

    Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. Advances in Neural Information Processing Systems, 34:14122–14134, 2021

    2021

  39. [47]

    Dynamic evaluation of neural sequence models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In International Conference on Machine Learning, pages 2766–2775. PMLR, 2018

    2018

  40. [48]

    Dynamic evaluation of transformer language models

    Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of transformer language models. arXiv preprint arXiv:1904.08378, 2019

    1904 arXiv

  41. [49]

    E fficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. E fficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, ...

    2023

  42. [50]

    Building machines that learn and think like people

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017. 22

    2017

  43. [51]

    Building high-level features using large scale unsupervised learning

    Quoc V Le. Building high-level features using large scale unsupervised learning. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 8595–8598. IEEE, 2013

    2013

  44. [52]

    World model on million-length video and language with blockwise ringattention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08268, 2024

    2024

  45. [53]

    Consistent video depth estimation

    Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4):71–1, 2020

    2020

  46. [54]

    Gradient-based hyperparameter optimization through reversible learning

    Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International conference on machine learning, pages 2113–2122. PMLR, 2015

    2015

  47. [55]

    Meta-learning update rules for unsupervised representation learning

    Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. arXiv preprint arXiv:1804.00222, 2018

    2018 arXiv

  48. [56]

    Online model distillation for efficient video inference

    Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. Online model distillation for efficient video inference. arXiv preprint arXiv:1812.02699, 2018

    2018

  49. [57]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P . Pret- tenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learn...

    2011

  50. [58]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  51. [59]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 2024

    2024

  52. [60]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. arXiv preprint arXiv:2210.10340, 2022

    2022

  53. [61]

    The perceptron: a probabilistic model for information storage and organiza- tion in the brain

    Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386, 1958

    1958

  54. [62]

    Linear transformers are secretly fast weight programmers

    Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355–9366. PMLR, 2021

    2021

  55. [63]

    Learning associative inference using fast weight memory

    Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. arXiv preprint arXiv:2011.07831, 2020

    2011

  56. [64]

    Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-

    Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987

    1987

  57. [65]

    Learning to control fast-weight memories: An alternative to dynamic recurrent networks

    Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992

    1992

  58. [66]

    Glu variants improve transformer, 2020

    Noam Shazeer. Glu variants improve transformer, 2020

    2020

  59. [67]

    Normformer: Improved transformer pretraining with extra normalization

    Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456, 2021. 23

    2021

  60. [68]

    zero-shot

    Assaf Shocher, Nadav Cohen, and Michal Irani. “zero-shot” super-resolution using deep inter- nal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3118–3126, 2018

    2018

  61. [69]

    Roformer: Enhanced transformer with rotary position embedding, 2023

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023

    2023

  62. [70]

    Learning to (learn at test time)

    Yu Sun, Xinhao Li, Karan Dalal, Chloe Hsu, Sanmi Koyejo, Carlos Guestrin, Xiaolong Wang, Tatsunori Hashimoto, and Xinlei Chen. Learning to (learn at test time). arXiv preprint arXiv:2310.13807, 2023

    2023

  63. [71]

    Online learning of unknown dynamics for model-based controllers in legged locomotion

    Yu Sun, Wyatt L Ubellacker, Wen-Loong Ma, Xiang Zhang, Changhao Wang, Noel V Csomay- Shanklin, Masayoshi Tomizuka, Koushil Sreenath, and Aaron D Ames. Online learning of unknown dynamics for model-based controllers in legged locomotion. IEEE Robotics and Automation Letters, 6(...

    2021

  64. [72]

    Test-time training with self-supervision for generalization under distribution shifts

    Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, pages 9229–9248. PMLR, 2020

    2020

  65. [73]

    Learning to learn: Introduction and overview

    Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pages 3–17. Springer, 1998

    1998

  66. [74]

    Using fast weights to improve persistent contrastive divergence

    Tijmen Tieleman and Geoffrey Hinton. Using fast weights to improve persistent contrastive divergence. In Proceedings of the 26th annual international conference on machine learning, pages 1033–1040, 2009

    2009

  67. [75]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023

  68. [76]

    The nature of statistical learning theory

    Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013

    2013

  69. [77]

    Extracting and composing robust features with denoising autoencoders

    Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, page 1096–1103, 2008

    2008

  70. [78]

    The correlation theory of brain function

    Christoph Von Der Malsburg. The correlation theory of brain function. In Models of neural networks: Temporal aspects of coding and information processing in biological systems , pages 95–119. Springer, 1994

    1994

  71. [79]

    Test-time training on video streams

    Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang. Test-time training on video streams. arXiv preprint arXiv:2307.05014, 2023

    2023

  72. [80]

    Huggingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. 24

    1910 arXiv

  73. [81]

    Effective long-context scaling of foundation models, 2023

    Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, S...

    2023

  74. [82]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  75. [83]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484, 2024

    2024

  76. [84]

    Root mean square layer normalization, 2019

    Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019

    2019

  77. [85]

    improved recipe

    Hao Zhang, Alexander C Berg, Michael Maire, and Jitendra Malik. Svm-knn: Discriminative nearest neighbor classification for visual category recognition. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 2126–2136. IE...

    2006

Pith tools

Reviewed May 15, 2026 · model on record in the stance chip above.