NITP augments next-token prediction with cosine alignment to stop-gradient shallow-layer features of the next token, improving geometry and downstream scores at ~2% extra training FLOPs.
Title resolution pending
4 Pith papers cite this work. Polarity classification is still indexing.
4
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 4years
2026 4roles
background 1polarities
background 1representative citing papers
Self-distillation turns pretrained autoregressive LMs into multi-token predictors that decode over 3x faster with under 5% accuracy drop on GSM8K.
Token-Superposition Training combines multiple tokens into bags for multi-hot cross-entropy pre-training followed by a recovery phase, yielding up to 2.5x reduction in training time at 10B scale under equal-loss conditions.
LLMs are reframed as a degenerate case of world models with a continuous spectrum of architectures from next-token prediction to joint-embedding predictive architectures.
citing papers explorer
-
NITP: Next Implicit Token Prediction for LLM Pre-training
NITP augments next-token prediction with cosine alignment to stop-gradient shallow-layer features of the next token, improving geometry and downstream scores at ~2% extra training FLOPs.
-
Multi-Token Prediction via Self-Distillation
Self-distillation turns pretrained autoregressive LMs into multi-token predictors that decode over 3x faster with under 5% accuracy drop on GSM8K.
-
Efficient Pre-Training with Token Superposition
Token-Superposition Training combines multiple tokens into bags for multi-hot cross-entropy pre-training followed by a recovery phase, yielding up to 2.5x reduction in training time at 10B scale under equal-loss conditions.
-
From Tokens to States: LLMs as a Special Case of World Models and the Continuous Path Beyond
LLMs are reframed as a degenerate case of world models with a continuous spectrum of architectures from next-token prediction to joint-embedding predictive architectures.