REVIEW 6 cited by
Synthetic continued pretraining
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.
Forward citations
Cited by 6 Pith papers
-
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
A learned orchestrator builds per-example drop/untouch/clean pipelines over noise pruning and instruction-conditioned rewriting, improving from-scratch and math continued pretraining over fixed curation methods.
-
Learning Facts at Scale with Active Reading
Training LLMs on self-generated, diverse 'active reading' materials improves factual recall by 160-312% and scales to a 1T-token Wikipedia expert model.
-
PropMEND: Hypernetworks for Knowledge Propagation in LLMs
PropMEND meta-trains a gradient-editing hypernetwork with propagation questions as the outer-loop objective, improving multi-hop knowledge propagation in LLMs in-domain but generalizing poorly to unseen relations.
-
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
A knowledge-graph-guided method that scores an LLM's knowledge gaps and generates atomic, aggregated, and multi-hop QA pairs, improving closed-book QA after fine-tuning.
-
Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
For closed-book QA via LoRA adapters, once capacity is sufficient, training-data quality—especially answer brevity—outperforms rank, learning rate, and architecture changes, lifting 15-doc accuracy from 57.7% to 85.7%.
-
Generative Data Refinement: Just Ask for Better Data
A pretrained LLM can rewrite individual data samples to strip out PII or toxic content while preserving useful information, creating safer training data.
Discussion (0). Sign in to comment.