REVIEW 53 cited by
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReST$^{EM}$ scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.
Forward citations
Cited by 53 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers
An architecture that routes hidden states from higher to lower layers between consecutive tokens improves reasoning accuracy and length generalization in fine-tuned LLMs.
-
The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation
Strict-launch-gated rejection-sampling self-distillation compounds cross-family clean Godot generation from 8.8% to 42.2% and full best-of-K coverage, while gold duplication and a lenient BUILD filter erase the gain.
-
ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.
-
Recursive Synthesis for Long-Horizon Terminal Tasks
A recursive verified synthesis pipeline generates 37,484 harder terminal tasks from 639 seeds, and training on the collected trajectories improves terminal-agent benchmark scores.
-
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
Outcome-verified teacher continuations from student failure prefixes, plus divergence-local comparison and suffix distillation, raise skill-free agent success over scoring-based self-distillation.
-
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
OC-GRPO reweights GRPO gradients with an importance ratio so that hints used during rollout generation still optimize the original unguided objective, delivering a 13.8% relative Pass@1 gain over vanilla GRPO.
-
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
Answer-conditioned chains of thought are harmful supervision for reasoning distillation—causing up to ~27-point accuracy drops—and the damage is invisible to correctness filters.
-
Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
Post-training changes where confidence is trustworthy in reasoning traces: OPD best gauges difficulty before thinking, SFT best supports early stopping, RL best selects final answers, and confidence should be read onl...
-
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
A unified KG-plus-live-web data pipeline covering all three multimodal BrowseComp information-flow patterns, plus an exploration-degree filter, yields a 35B agent at 54.4 avg accuracy.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation
FormulaSPIN applies execution-aware self-play with an adaptive curriculum to spreadsheet formula generation, improving exact match from 68.2% to 74.9% on NL2Formula-70K without additional data.
-
A Task-Centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
Iterative self-improvement provably keeps improving only when initial performance lies in a moderate difficulty interval, and easy-to-hard curricula beat fixed mixtures under moderate difficulty separation and suffici...
-
rePIRL: Learn PRM with Inverse RL for LLM Reasoning
rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.
-
Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems
MOAT alternately aligns a planning and a grounding LLM agent via perplexity-based DPO and self-generated SFT, reporting small but consistent gains over independently tuned baselines.
-
Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online
A history-aware verifier that scores candidate actions using past interactions cuts failure rates in ambiguous robot manipulation tasks compared to using the generator alone.
-
Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding
LRTab retrieves error-avoiding prompt conditions learned from incorrect chain-of-thought traces on training tables to improve LLM tabular reasoning, achieving modest gains on WikiTQ and TabFact.
-
SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning
SyncLoop jointly evolves multimodal training data and model capability through alternating SFT and RL, selecting error-prone samples to improve geometry reasoning.
-
SPARQ: Synthetic Problem Generation for Reasoning via Quality-Diversity Algorithms
Filtering self-generated math problems by a model's own solve-rate improves that model's MATH accuracy from 38% to 47% and helps out-of-distribution generalization when data is diverse.
-
HardTests: Synthesizing High-Quality Test Cases for LLM Coding
HardTestGen generates higher-precision, higher-recall test suites for 47,136 competitive programming problems, improving test precision by 11.3 points and recall by 17.5 points over TACO and CodeContests when judging ...
-
ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution
ToolACE-DEV decomposes tool learning into documentation adaption, tool generation, and invocation, then lets the model self-train on its own majority-voted invocations, improving function-calling accuracy across three...
-
Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
GVM-RAFT allocates samples per prompt according to acceptance rate and gradient norm, reducing gradient variance and speeding up RAFT-style chain-of-thought training.
-
Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
A self-training method that makes an LLM write an abstract plan before each answer improves average test accuracy by about 3 points over three self-training baselines on four reasoning benchmarks.
-
LLMs can be easily Confused by Instructional Distractions
A new benchmark, DIM-Bench, shows that LLMs frequently follow instructions hidden inside the target input rather than the user's actual instruction, even when explicitly told to ignore them.
-
BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning
BRiTE is an EM-style algorithm that uses RL to sample high-likelihood reasoning chains and fine-tunes the LLM on them, with a theoretical 1/T convergence rate and mixed empirical gains.
-
B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners
B-STaR dynamically tunes sampling temperature and reward thresholds during iterative self-training, improving Pass@1 on math, code, and commonsense reasoning benchmarks versus static self-improvement baselines.
-
How to Synthesize Text Data without Model Collapse?
Token-level editing of human text with a high-confidence threshold yields modest performance gains over the original data and avoids the sharp degradation seen with purely synthetic data.
-
HARP: A challenging human-annotated math reasoning benchmark
HARP, a new benchmark of 5,409 US math competition problems with human solutions and choices, keeps frontier LLMs far from saturation: the best model scores 75.9% overall and only 41.1% on the hardest 197 problems.
-
From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
A single MLLM-based agent, finetuned with cross-domain supervision and online RL, achieves strong zero-shot generalization across manipulation, navigation, games, UI control, and planning.
-
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
A controlled, multi-family study shows the relative generation-verification gap grows with pretraining flops for stable verification methods, and iterative self-improvement saturates quickly.
-
GPO: Learning from Critical Steps to Improve LLM Reasoning
Guided Pivotal Optimization improves LLM reasoning post-training by resetting rollouts at an advantage-estimated critical step, beating each of five base optimizers on seven benchmarks in 34 of 35 comparisons.
-
ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding
ReST-RL pairs reward-variance-filtered self-training (ReST-GRPO) with value-model-guided MCTS decoding (VM-MCTS) to improve LLM code reasoning.
-
Spectra 1.1: Scaling Laws and Efficient Inference for Ternary Language Models
Ternary language models trained on 1.2 trillion tokens continue to improve, and a new GPU kernel speeds up their inference up to 5x end-to-end.
-
Maximizing Confidence Alone Improves Reasoning
Reinforcement learning with a reward equal to the negative entropy of the model's own token distribution improves benchmark accuracy without any labeled answers.
-
SLearnLLM: A Self-Learning Framework for Efficient Domain-Specific Adaptation of Large Language Models
Fine-tuning only on questions a model answers incorrectly reaches nearly the same performance as full-data fine-tuning while substantially reducing training time.
-
Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning
OREAL shows that outcome-reward RL with best-of-N positive behavior cloning, negative reward shaping, and token-level reweighting reaches state-of-the-art MATH-500 accuracy at 7B and 32B scale.
-
Policy Guided Tree Search for Enhanced LLM Reasoning
PGTS trains a graph-transformer policy via PPO to guide tree search over LLM reasoning steps, reporting improved accuracy and lower token use than chain-of-thought and MCTS baselines.
-
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
Iterative self-training on a model's own correct outputs, with simple length and voting filters, lets transformers generalize to far longer arithmetic and path-finding problems than they saw in training.
-
To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization
An EM-style training loop lets 7B math LLMs learn when to invoke code, improving MATH500 by 11 points and AIME by 9.4 points.
-
RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?
Long-CoT fine-tuning with QwQ-generated traces improves math and geometry benchmarks, and RedStar-DPO surpasses QwQ on AIME24, but several reported numbers conflict with the paper's own tables.
-
ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding
ReARTeR improves RAG multi-step reasoning by combining a process reward model, an aligned explanation model, temporal-difference lookahead, and MCTS-based iterative preference training.
-
Can Large Language Models Improve SE Active Learning via Warm-Starts?
LLM-generated warm starts improve active learning on low- and medium-dimensional software engineering tasks but underperform Gaussian process methods on high-dimensional tasks.
-
Diving into Self-Evolving Training for Multimodal Reasoning
M-STAR, a self-evolving training recipe combining continuous updates, a process-reward-model reranker, and adaptive sampling temperature, improves multimodal reasoning on several benchmarks across three vision-languag...
-
Offline Reinforcement Learning for LLM Multi-Step Reasoning
OREO jointly trains a policy and a value function with soft Bellman consistency, outperforming offline baselines on multi-step reasoning benchmarks and enabling value-guided search.
-
Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
DynSDPB fine-tunes small language models by self-distilling soft labels from the previous mini-batch, with dynamic per-sample temperature and loss weighting.
-
Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
A trained critique model providing step-level feedback improves an LLM actor's mathematical reasoning at test time and during self-improvement, with gains concentrated on hard problems.
-
CoTu at EXACT 2026: Neuro-Symbolic Reasoning for Transparent Educational QA
A 4B LLM that generates solver-runnable code (Z3 for logic, Python for physics) achieved a perfect physics score and the highest technical score (13.44/15) at EXACT 2026, with premise selection as the main remaining e...
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training
A multi-agent RAG framework trained by self-supervision on high-reward interaction trajectories outperforms a vanilla RAG baseline on DataMorgana-generated questions and places 7th in the LiveRAG 2025 competition.
-
Generalizing Large Language Model Usability Across Resource-Constrained
The dissertation shows that text-centric prompting, inference-time optimization, and correct-by-construction synthetic data can improve LLM robustness and Verilog code generation under resource constraints.
-
Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
Recursively fine-tuning language models on their own ground-truth-pruned reasoning traces improves GSM8K Pass@1, but the claimed GPT-4o-beating result rests on a nonstandard 500-question test subset.
-
QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
QLASS trains a stepwise Q-value model on self-explored trajectories and uses it to guide language agents action by action, outperforming baselines on WebShop, SciWorld, and ALFWorld.
-
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
This survey organizes LLM synthetic data research around quality, diversity, and complexity, claiming quality mainly helps in-distribution generalization, diversity mainly helps out-of-distribution generalization, and...
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.