Subject-object co-occurrence frequency in pretraining predicts whether a language model forms linear relational representations, and LRE quality can be used to estimate unknown pretraining term frequencies.
To code or not to code? exploring impact of code in pre-training
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
On Linear Representations and Pretraining Data Frequency in Language Models
Subject-object co-occurrence frequency in pretraining predicts whether a language model forms linear relational representations, and LRE quality can be used to estimate unknown pretraining term frequencies.