Subject-object co-occurrence frequency in pretraining predicts whether a language model forms linear relational representations, and LRE quality can be used to estimate unknown pretraining term frequencies.
How do large language models acquire factual knowledge during pretraining? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
On Linear Representations and Pretraining Data Frequency in Language Models
Subject-object co-occurrence frequency in pretraining predicts whether a language model forms linear relational representations, and LRE quality can be used to estimate unknown pretraining term frequencies.