Continued pre-training with LoRA on simplified triple-only documents improves LLM performance on the paper's own insight-extraction benchmarks, but those benchmarks are built from the same documents used for training.
Adapting a Language Model While Preserving its General Knowledge
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach.
fields
cs.CL 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Learning Beyond the Surface: How Far Can Continual Pre-Training with LoRA Enhance LLMs' Domain-Specific Insight Learning?
Continued pre-training with LoRA on simplified triple-only documents improves LLM performance on the paper's own insight-extraction benchmarks, but those benchmarks are built from the same documents used for training.