Pith. sign in

REVIEW 55 cited by

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2004.10964 v3 pith:QY4DZDTS submitted 2020-04-23 cs.CL cs.LG

Suchin Gururangan , Ana Marasović , Swabha Swayamdipta , Kyle Lo , Iz Beltagy , Doug Downey , Noah A. Smith This is my paper · ORCID
classification cs.CLcs.LG
keywords pretrainingtaskdomain-adaptivemodelsperformanceadaptingdatadomains
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 55 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TELLME: Test-Enhanced Learning for Language Model Enrichment

    cs.CL 2026-08 conditional novelty 6.0 of 10

    TELLME improves continual domain pre-training by training on plain text plus open-ended QA pairs while masking question tokens from the loss, with consistent finance gains and small medical gains.

  2. Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A ModernBERT-based encoder trained with masked language modeling on SMILES-annotated scientific documents plus a contrastive stage yields embeddings that are competitive on both molecular property prediction and scien...

  3. Improving Mental Health Screening and Early Risk Detection in Spanish

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Domain-adapted Spanish transformers plus ICE context-level relabeling beat prior systems on three MentalRisk early-detection benchmarks with lower latency.

  4. From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Unsupervised TSDAE and CSE adaptation of specialized Latin/Greek LMs yields corpus-specific sentence encoders that outperform multilingual, distilled, and supervised baselines on biblical reuse detection and retrieval.

  5. Test-Time Scaling via Error Localization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.

  6. RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Dense per-frame intermediate representations (traces, masks, grasp poses, subtasks) improve embodied VQA, VLA action generation, and world-model video prediction in the new 230k-episode RoboInter-Data suite.

  7. Adapter Merging Reactivates Latent Reasoning Traces: A Mechanism Analysis

    cs.CL 2026-01 reject novelty 6.0 of 10

    Merging domain-adaptation and instruction-tuning LoRA adapters can reactivate suppressed reasoning traces, localized to a low-dimensional late-layer subspace, and a fitted logit-direction intervention can modulate thi...

  8. Improving Community Detection in Academic Networks by Handling Publication Bias

    cs.IR 2025-07 reject novelty 6.0 of 10

    Clustering a researcher's publications into 'clones' produces overlapping academic communities in topic-based networks, but the reported evaluation is descriptive and lacks ground-truth validation.

  9. Energy-Based Transformers are Scalable Learners and Thinkers

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Energy-Based Transformers learn to predict by gradient-descent minimization of a learned energy function, and the paper reports faster pretraining scaling and inference-time thinking gains over Transformer++ and Diffu...

  10. BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A continued-pretrained ModernBERT encoder for biomedical and clinical text claims SOTA on several clinical NLP tasks, with caveats about data overlap between pretraining and evaluation.

  11. Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    CREPA fine-tunes video diffusion models with a loss that pulls each frame's hidden representation toward pretrained features of adjacent frames, improving semantic consistency in generated videos.

  12. PropMEND: Hypernetworks for Knowledge Propagation in LLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PropMEND meta-trains a gradient-editing hypernetwork with propagation questions as the outer-loop objective, improving multi-hop knowledge propagation in LLMs in-domain but generalizing poorly to unseen relations.

  13. ZeroVO: Visual Odometry with Minimal Assumptions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A two-frame visual odometry model using estimated depth, language priors, and semi-supervised pseudo-label filtering achieves zero-shot metric-scale pose estimation across multiple driving datasets.

  14. Structuring Radiology Reports: Challenging LLMs with Lightweight Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Fully finetuned T5 and BERT2BERT models match or beat prompt-adapted LLMs up to 70B parameters on radiology report structuring, at less than 1% of the inference cost.

  15. ScienceMeter: Tracking Scientific Knowledge Updates in Language Models

    cs.CL 2025-05 reject novelty 6.0 of 10

    ScienceMeter evaluates language model knowledge updates across three axes, preservation of old scientific claims, acquisition of new claims, and projection to future findings, and finds all current methods fall short.

  16. Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.

  17. Merge to Mix: Mixing Datasets via Model Merging

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.

  18. Training Strategies for Efficient Embodied Reasoning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Reasoning pre-training and dropout let robot policies benefit from chain-of-thought training without run-time reasoning, preserving most of the performance gains at standard VLA inference speeds.

  19. When Bad Data Leads to Good Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Pretraining on toxic data creates more linearly separable toxicity representations that are easier to steer away, yielding lower output toxicity after inference-time intervention, up to a sweet spot around 10%.

  20. BTS: Harmonizing Specialized Experts into a Generalist LLM

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Branch-Train-Stitch merges frozen domain-specialist LLMs into a generalist by training only lightweight stitch layers, and reports the best average benchmark performance among merging baselines.

  21. On the Impact of Requirements Smells in Prompts: The Case of Automated Traceability

    cs.SE 2025-01 conditional novelty 6.0 of 10

    Requirements smells in prompts caused a small significant decrease in LLM binary trace link accuracy, but no significant effect on line-level tracing, in a five-project study.

  22. Error-driven Data-efficient Large Multimodal Model Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    An error-driven teacher-student pipeline extracts a student LMM's missing skills from validation mistakes and retrieves targeted samples from a task-agnostic dataset to fine-tune it.

  23. Efficient Knowledge Injection in LLMs via Self-Distillation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Self-distillation from a model's own in-context answers injects factual knowledge into LLM weights more efficiently than supervised fine-tuning and is competitive with RAG.

  24. LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    LoRA3D specializes pretrained 3D foundation models to target scenes via confidence-calibrated pseudo-labels from multi-view robust optimization and LoRA fine-tuning, improving performance by up to 88%.

  25. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  26. A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.

  27. Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    A budget-aware framework (PU-ADKA) selects which domain expert an LLM should query under a fixed $100 budget, improving specialized-domain answers at low cost.

  28. Effortless Vision-Language Model Specialization in Histopathology without Annotation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Continued pretraining on keyword-matched image-caption pairs from Quilt1M improves zero-shot and few-shot performance of histopathology VLMs without manual annotations.

  29. AI-Driven Generation of Old English: A Framework for Low-Resource Languages

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A LoRA-fine-tuned 8B language model with backtranslation and a dual-agent pipeline generates Old English text, with reported English-to-Old English BLEU rising from 25.94 to 65.41.

  30. CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation

    cs.CL 2025-07 conditional novelty 5.0 of 10

    CCL-XCoT combines curriculum-based contrastive pretraining with cross-lingual chain-of-thought fine-tuning, lifting hallucination-free rates in low-resource QA from 1-18% to 55-74%.

  31. Text-to-LoRA: Instant Transformer Adaption

    cs.LG 2025-06 conditional novelty 5.0 of 10

    A hypernetwork can generate task-specific LoRA adapters from a text description, and when trained with supervised fine-tuning it zero-shot outperforms a multi-task LoRA baseline on ten benchmarks.

  32. Fine-Tune an SLM or Prompt an LLM? The Case of Generating Low-Code Workflows

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Fine-tuning a 12B SLM outperforms prompting larger LLMs by about 10% Flow Similarity when generating low-code enterprise workflows.

  33. AUTOLAW: Enhancing Legal Compliance in Large Language Models via Case Law Generation and Jury-Inspired Deliberation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    AutoLaw's verifier-ranked legal-role jury with a similar-case demonstration beats majority voting for violation detection on three law and policy benchmarks.

  34. Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?

    cs.CL 2025-01 reject novelty 5.0 of 10

    Continued pre-training with LoRA on simplified triple-only documents improves LLM performance on the paper's own insight-extraction benchmarks, but those benchmarks are built from the same documents used for training.

  35. TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

    cs.CL 2024-12 conditional novelty 5.0 of 10

    For Llama-2-7B on telecommunications tasks, instruction tuning on a telco-generated dataset suffices; continuing pretraining on raw telco text adds little (max +0.03 accuracy).

  36. Adapting Large Language Models to Log Analysis with Interpretable Domain Knowledge

    cs.CL 2024-12 reject novelty 5.0 of 10

    SuperLog, a LLaMA-2-7B model continually pre-trained on an LLM-generated natural-language QA corpus about logs, claims state-of-the-art results on four log analysis tasks.

  37. CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search

    cs.AI 2024-12 conditional novelty 5.0 of 10

    A continual pre-training framework combining query-item joint training, in-context pre-training on related queries/items, and teacher-generated reading comprehension data improves LLM relevance modeling in commercial search.

  38. Efficient Alignment of Large Language Models via Data Sampling

    cs.LG 2024-11 reject novelty 5.0 of 10

    Alignment performance plateaus quickly with data, and a GMM-plus-entropy sampling method (ISA) can allegedly match full-dataset alignment with under 10% of the data, though the supporting evidence is weak.

  39. Enhancing Financial Domain Adaptation of Language Models via Model Augmentation

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Composing a general Japanese instruction model with a finance-specialized model via CALM cross-attention improves Japanese financial benchmark scores beyond LoRA, even when trained on a different finance dataset.

  40. Boosting Data Utilization for Multilingual Dense Retrieval

    cs.IR 2025-09 conditional novelty 4.0 of 10

    A three-stage data-utilization pipeline for multilingual dense retrieval, combining ensemble hard-negative mining, LLM-based filtering/generation, and monolingual topic-diverse mini-batches, improves MIRACL nDCG@10 by...

  41. Measuring Time-Series Dataset Similarity using Wasserstein Distance

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Time-series dataset similarity is defined via the Wasserstein distance between fitted multivariate normal distributions, and the distance shows partial correlation with foundation model inference loss.

  42. Understanding Knowledge Transferability for Transfer Learning: A Survey

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A survey that classifies transferability metrics by knowledge modality (dataset vs. model) and granularity (task vs. instance), with a theoretical primer and applications to eight learning paradigms.

  43. Good Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels

    cs.LG 2025-07 conditional novelty 4.0 of 10

    A decoder-only LLM pre-trained on ECU log text and fine-tuned with an entropy regularizer detects cycle-time anomalies with 0.81 region recall despite noisy labels.

  44. GeistBERT: Breathing Life into German NLP

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A 126M-parameter German BERT, pretrained further on 1.3TB of mixed German text, beats other base models on most tested German NLP benchmarks.

  45. Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Using a BERT classifier's predicted hate-crime probabilities as an auxiliary sampling variable yields a Hansen-Hurwitz estimate of 6,051 hate crimes among 2022 Swedish police reports, with a design effect of 0.0068.

  46. Investigating the Impact of Data Selection Strategies on Language Model Performance

    cs.CL 2025-01 reject novelty 4.0 of 10

    Neural-embedding-based data selection underperforms DSIR on most GLUE tasks, and the paper's hybrid method is untested because only α=0 was run.

  47. On Adversarial Robustness of Language Models in Transfer Learning

    cs.CL 2024-12 reject novelty 4.0 of 10

    Sequential fine-tuning across related bias-detection tasks tends to raise adversarial attack success rates, but the size-resilience pattern the paper highlights is not borne out by its own data.

  48. ConfliBERT: A Language Model for Political Conflict

    cs.CL 2024-12 conditional novelty 4.0 of 10

    A conflict-specific BERT model beats much larger general-purpose LLMs on classifying political violence texts when the larger models are used off the shelf, and is hundreds of times faster.

  49. Adaptive Two-Phase Finetuning LLMs for Japanese Legal Text Retrieval

    cs.IR 2024-12 conditional novelty 4.0 of 10

    A two-phase fine-tuning pipeline (global contrastive pretraining, then domain-specific hard-negative training) improves LLaMA-based dense retrieval on a Gemini-generated Japanese legal dataset and on a subset of MS MARCO.

  50. Evaluating Hallucinations in Domain-Adapted Large Language Models

    cs.CL 2026-04 conditional novelty 3.0 of 10

    Fine-tuning Llama-2 on a small domain-specific dataset yields high memorization but near-zero reasoning on newly introduced entities, suggesting fine-tuning alone is insufficient for knowledge injection.

  51. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

    cs.CR 2025-06 conditional novelty 3.0 of 10

    A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.

  52. A Survey on Large Language Models with some Insights on their Capabilities and Limitations

    cs.CL 2025-01 unverdicted novelty 3.0 of 10

    A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.

  53. Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline

    cs.AI 2024-12 reject novelty 3.0 of 10

    A LoRA fine-tuned 7B model trained on AI-synthesized enterprise HR API data beat GPT-4 and GPT-4o on the authors' private benchmark.

  54. Challenges in Guardrailing Large Language Models for Science

    cs.AI 2024-11 conditional novelty 3.0 of 10

    A position paper proposing a guardrail framework with four dimensions (trustworthiness, ethics & bias, safety, legal) and implementation strategies for scientific LLM use.

  55. Dynamic and Parametric Retrieval-Augmented Generation

    cs.CL 2025-06 unverdicted novelty 2.0 of 10

    A tutorial outline that categorizes recent RAG work into Dynamic RAG and Parametric RAG, and explains why both are needed.

Pith tools