Synthetic pre-pre-training on structured data improves LLM robustness to noisy pre-training, matching baseline loss with up to 49% fewer natural tokens for a 1B model.
arXiv preprint arXiv:2309.17002 , year =
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
verdicts
UNVERDICTED 2representative citing papers
EchoAlign adjusts instances with controllable generative models to match noisy labels and selects reliable subsets, outperforming prior methods on benchmarks especially under 30% instance-dependent noise.
citing papers explorer
-
Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
Synthetic pre-pre-training on structured data improves LLM robustness to noisy pre-training, matching baseline loss with up to 49% fewer natural tokens for a 1B model.
-
EchoAlign: Bridging Generative and Discriminative Learning under Noisy Labels
EchoAlign adjusts instances with controllable generative models to match noisy labels and selects reliable subsets, outperforming prior methods on benchmarks especially under 30% instance-dependent noise.