REVIEW 4 cited by
Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30 percent reduction in iterations, all while achieving superior performance compared to prior work.
Forward citations
Cited by 4 Pith papers
-
Bridging Compute- and Data-Optimal Pretraining
Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.
-
Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics
Under Gaussian design with n ≍ d, the empirical distribution of leave-one-out influences for convex M-estimators converges to the pushforward of a four-dimensional Gaussian through an explicit nonlinear map built from...
-
MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling
Concentrating synthetic preference paraphrases on low-margin pairs gives consistent but small reward-model and alignment gains in single-run experiments, while the abstract's semantic-aware, multi-benchmark claims are...
-
The Anti-Ouroboros Effect: Emergent Resilience in Large Language Models from Recursive Selective Feedback
A recursive fine-tuning study claims quality filtering reverses model collapse, but the paper's own data show the filtered model only matched its starting score.
Discussion (0). Sign in to comment.