Pith. sign in

REVIEW 19 cited by

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.07279 v2 pith:NDZ7CBXJ submitted 2024-11-11 cs.AI cs.CLcs.LG

Ekin Akyürek , Mehul Damani , Adam Zweiger , Linlu Qiu , Han Guo , Jyothish Pari , Yoon Kim , Jacob Andreas This is my paper · ORCID
classification cs.AIcs.CLcs.LG
keywords in-contexttrainingexamplesfew-shotlearningtaskstest-timeeffectiveness
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks even when given a small number of in-context task examples. We investigate the effectiveness of test-time training (TTT) -- temporarily updating model parameters during inference using a loss derived from input data -- as a mechanism for improving LMs' reasoning and few-shot learning capabilities. On the Abstraction and Reasoning Corpus (ARC), performing TTT with in-context examples yields up to $6\times$ higher accuracy compared to fine-tuned baselines -- reaching $53.0\%$ on the public validation set with an 8B-parameter LM and $61.9\%$ when ensembled with program-synthesis methods, matching average human performance. On BIG-Bench Hard (BBH), TTT on in-context examples surpasses standard few-shot prompting in the $10$-shot setting by $7.3$ percentage points ($50.5\%$ to $57.8\%$). Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 19 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Consensus as Privileged Context for Label-Free Self-Distillation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CANON turns a model's majority-vote consensus into dense per-token supervision by distilling a frozen teacher conditioned on a consensus solution, improving label-free LLM reasoning by up to about 12 points and transf...

  2. Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute

    cs.LG 2026-08 conditional novelty 6.0 of 10

    For mid-tier LLMs, generating paraphrased versions of an input and majority voting over answers converts inference budget into accuracy more efficiently than self-consistency sampling.

  3. Self-Guided Test-Time Training for Long-Context LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Model-selected evidence spans for test-time training beat random and full-context TTT on long-context reasoning benchmarks, with up to ~15% relative accuracy gains.

  4. No Time Like the Present: Agentic Test-Time Training for LLM Agents

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Continuous test-time training that downweights tokens in repeated n-grams improves multi-turn LLM agent success by up to about 5 points by limiting self-reinforcing drift.

  5. LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

    cs.AI 2025-07 conditional novelty 6.0 of 10

    LeMix co-locates LLM serving and retraining on shared GPUs with profiler-driven scheduling, reporting up to 3.53x throughput gains over separated deployments.

  6. Context Tuning for In-Context Optimization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Context Tuning initializes a trainable prompt or KV-cache prefix from the few-shot demonstrations and refines it by gradient descent, matching Test-Time Training accuracy on MMLU, BBH, and ARC at up to 2-3x less train...

  7. EasyARC: Evaluating Vision Language Models on True Visual Reasoning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EasyARC is a new procedurally generated visual reasoning benchmark where state-of-the-art vision-language models score below 20%, despite tasks designed to be easy.

  8. From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Frozen CogVideoX1.5, adapted with LoRA on 3 to 30 input-output videos, performs segmentation, pose estimation, and abstract reasoning (ARC-AGI 16.75%) with modest but real generalization.

  9. Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A length-penalized reinforcement learning method called TLDR reduces reasoning token use by about 50 percent in small math language models without hurting accuracy, and temperature scaling of the end-of-sequence token...

  10. Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

    cs.CL 2025-05 conditional novelty 6.0 of 10

    An open-source LLM pipeline with augmentation-aware training, probability-thresholded depth-first search, and product-of-experts scoring reaches 71.6% on the public ARC-AGI evaluation set.

  11. Test-time regression: a unifying framework for designing sequence models with associative memory

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Sequence models that perform associative recall are unified as 'test-time regression' layers, reducing to weighted, kernel, or online least-squares regression depending on three design choices.

  12. The broader spectrum of in-context learning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    The authors propose a broad definition of in-context learning as any sequence task where context reduces loss, and argue research should study this wider spectrum.

  13. Memoir: Should a Model Write to Its Memory While It Thinks?

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Writing to fast memory during pondering slows associative-recall learning at a fixed budget, but does not reduce final performance once training is long enough.

  14. RTTC: Reward-Guided Collaborative Test-Time Compute

    cs.CL 2025-08 reject novelty 5.0 of 10

    RTTC uses a reward model to route each query to direct inference, retrieval-augmented generation, or test-time training, plus caching to cut redundant work.

  15. Channel-Wise MLPs Improve the Generalization of Recurrent Convolutional Networks

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Adding a gated channel-wise MLP to a recurrent convolutional network raises median exact-match accuracy on 185 Re-ARC tasks from 78.75% to 92.19% in-distribution and from 2.34% to 14.58% on harder out-of-distribution tasks.

  16. Test-Time Learning for Large Language Models

    cs.CL 2025-05 reject novelty 5.0 of 10

    TLM adapts LLMs at test time by minimizing input perplexity on unlabeled data, using high-perplexity sample selection and LoRA updates.

  17. SLOT: Sample-specific Language Model Optimization at Test-time

    cs.CL 2025-05 conditional novelty 5.0 of 10

    SLOT adapts an LLM to each prompt by optimizing a lightweight final-layer vector to minimize prompt loss, boosting benchmark reasoning accuracy by a few points.

  18. Few-Shot Test-Time Optimization Without Retraining for Semiconductor Recipe Generation and Beyond

    cs.LG 2025-05 reject novelty 4.0 of 10

    A two-loop reverse-model method for test-time input optimization is claimed to find semiconductor etching recipes in five iterations, but the supporting evidence is simulation-only and not reproducible.

  19. Towards Contamination Resistant Benchmarks

    cs.CL 2025-05 conditional novelty 4.0 of 10

    The authors define contamination resistance as a benchmark property and show that most tested LLMs score near zero on Caesar-cipher encoding and decoding when the shift is not 3 and the text is random nonsense.

Pith tools