REVIEW 14 cited by
Learning by Distilling Context
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Language models significantly benefit from context tokens, such as prompts or scratchpads. They perform better when prompted with informative instructions, and they acquire new reasoning capabilities by generating a scratch-pad before predicting the final answers. However, they do not \textit{internalize} these performance gains, which disappear when the context tokens are gone. Our work proposes to apply context distillation so that a language model can improve itself by internalizing these gains. Concretely, given a synthetic unlabeled input for the target task, we condition the model on ``[instructions] + [task-input]'' to predict ``[scratch-pad] + [final answer]''; then we fine-tune the same model to predict its own ``[final answer]'' conditioned on the ``[task-input]'', without seeing the ``[instructions]'' or using the ``[scratch-pad]''. We show that context distillation is a general method to train language models, and it can effectively internalize 3 types of training signals. First, it can internalize abstract task instructions and explanations, so we can iteratively update the model parameters with new instructions and overwrite old ones. Second, it can internalize step-by-step reasoning for complex tasks (e.g., 8-digit addition), and such a newly acquired capability proves to be useful for other downstream tasks. Finally, it can internalize concrete training examples, and it outperforms directly learning with gradient descent by 9\% on the SPIDER Text-to-SQL dataset; furthermore, combining context distillation operations can internalize more training examples than the context window size allows.
Forward citations
Cited by 14 Pith papers
-
Can a Language Model Learn Facts Continually in Its Weights?
Weight writes create question-keyed knowledge that later sequential writes redirect, so context—not weights—is the reliable channel for facts that must survive or be composed.
-
Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
Training with appended behavioral instructions plus correctness-filtered self-distillation improves held-out math pass@1 over DAPO for a 1.7B model, but not for 4B at 4K context.
-
Learning What to Remember: Test-Time Training via Context Distillation
IP-TTCD distills the hidden-state gap between a long-window teacher and a short-window student into MLP fast weights, improving long-context language modeling and retrieval over DeltaNet, Gated DeltaNet, SWA, and IP-TTT.
-
Flux-OPD: On-Policy Distillation with Evolving Contexts
Evolving contexts improve open-ended on-policy distillation when injected as conflict-weighted corrections into a stable context-free teacher rather than used as direct targets.
-
Listen, Do Not Copy: Internalizing Audio-Grounded Scaffold Context for Robust Omni-Model Speech Understanding
Training three omnimodel speech systems with partial, audio-derived scaffold clues that are removed at test time cuts no-clue mpWER on overlapping noisy speech from 25–71% to 9–15%.
-
Sample-Efficient Learning from Agent Experience
Agents can consolidate in-context trial-and-error learning into their weights by distilling one-step teacher decisions at recorded histories, needing no additional environment interaction.
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems
RL-trained compound LLM systems can gain accuracy by having modules silently abandon their assigned roles, and a prompt-contrast regularizer can measure and limit that drift.
-
LiteOdyssey: A Lightweight Reasoning AI Agent for Interpretable Rare-Disease Diagnosis
A clinician-audited diagnostic policy plus public tools lets a single unmodified LLM reach high phenotype-first rare-disease Recall@1 and modestly beat baselines on real UDN patients.
-
Essential-Web v1.0: 24T tokens of organized web data
A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.
-
Cartridges: Lightweight and general-purpose long context representations via self-study
A per-corpus trained KV cache, called a Cartridge, matches full-context in-context learning quality on long-document benchmarks while using up to 38.6x less serving memory.
-
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
ADRS converts a frozen model's privileged skill-conditioned token scores into calibrated, return-gated reward shaping for agentic reinforcement learning, reporting consistent long-horizon gains.
-
Masked Distillation: Internalizing the Chain-of-Thought in Language Models
Masked distillation shows a student LLM can internalize its teacher's chain-of-thought on GSM8K but not on Countdown; emitting the last 30% of the trace as a 'suffix scaffold' recovers teacher-level accuracy with fewe...
-
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.
Discussion (0). Sign in to comment.