{"id":"a7792314-521d-4cf9-99bc-feafb711da63","arxiv_id":"2504.12459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Subject-object co-occurrence frequency in pretraining predicts whether a language model forms linear relational representations, and LRE quality can be used to estimate unknown pretraining term frequencies.","lead":"This paper shows that language models form simple linear representations of factual relations mainly when the subject and object often appear together in pretraining data, with clear frequency thresholds. It then uses the strength of those linear representations to estimate how often terms appeared in a model's training data, even for a different model with a closed dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1k/2k/4k co-occurrence thresholds are fit post hoc to 24 relations with no held-out validation; a leave-one-relation-out test would determine whether the headline threshold claim survives.","rationale":"Reader's weakest assumption is the Elsahar co-occurrence proxy. I agree that proxy is a real limitation, and the paper itself hedges in Appendix G. But the proxy concerns what the counts mean; the empirical association between counted co-occurrences and LRE causality would still stand even if many co-occurrences are non-relational. The threshold numbers are more load-bearing because the abstract's central claim states specific frequency cutoffs, and those cutoffs are estimated post hoc on the same 24 relations. No cross-validation is reported for them; the regression experiment validates features, not thresholds. A LOOCV check is cheap given the released code and would settle whether thresholds transfer. If thresholds do not transfer, the paper still supports a frequency-linearity correlation but not the '1k/2k/4k consistently forms' conclusion; the correct verdict would remain CONDITIONAL rather than REJECT, because the correlation and cross-model prediction are independently supported. Hence UNCHANGED.","tokens_in":24889,"tokens_out":9510,"duration_ms":104203,"concrete_test":"Leave-one-relation-out validation of the threshold: for each of the 24 relations, fit the causality>0.9 crossing threshold to the other 23 relations (using the same pooled-checkpoint procedure as Figure 2), then check whether the held-out relation's mean causality exceeds 0.9 when its cumulative log co-occurrence is above the fitted threshold and falls below 0.9 when below. Report hit rate against chance, bootstrap confidence intervals on the three model thresholds, and repeat the fit using median rather than mean relation co-occurrence to test sensitivity to the relation-level aggregation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central threshold claim in Section 4.2 ('co-occurrence counts above which the average causality is above 0.9') is estimated from the same 24 relations plotted in Figure 2, with no confidence intervals, no held-out relations, and no correction for the fact that the eight OLMo checkpoints contribute autocorrelated points. The exact numbers (GPT-J 1,097; OLMo-7B 1,998; OLMo-1B 4,447) imply a precision the design cannot support. With n=24, a single relation such as star-constellation-name (low frequency, causality far below 0.9) can move the crossing point substantially. The abstract's headline 'consistently (but not exclusively) forms when ... co-occur at least 1k and 2k times' is exactly this fitted threshold; if it does not transfer to held-out relations, the strong version of the central claim collapses to a correlation that is already frequency-dependent. The proxy concern raised by the reader is real, but it weakens interpretation, not the reported co-occurrence-to-LRE association; threshold generalizability is the assumption on which the specific numeric claim rests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether the formation of linear representations of factual relations in transformer LMs is predictable from pretraining corpus statistics. Using Linear Relational Embeddings (LREs) and the causality metric on 24 factual relations, the authors report a strong correlation (r = 0.82) between subject-object co-occurrence frequency in pretraining data and LRE quality in OLMo-7B, OLMo-1B, and GPT-J. They further claim that LREs consistently form above co-occurrence thresholds of roughly 1k, 2k, and 4k for GPT-J, OLMo-7B, and OLMo-1B respectively, regardless of when in pretraining the threshold is reached. Finally, they train a random-forest regression on LRE features to predict term frequencies from linear representation quality and report that the regression transfers across models with different pretraining corpora. The paper also releases the Batch Search counting tool.","tokens_in":25139,"tokens_out":5456,"duration_ms":61775,"significance":"If the central claims hold, this is a valuable empirical contribution: it connects interpretability findings about linear representations to pretraining data statistics and suggests a new, representation-based signal for inferring properties of closed pretraining corpora. The paper has concrete strengths: it uses external corpus counts rather than constructing counts from model probabilities, it studies multiple checkpoints across pretraining, and the cross-model regression is evaluated on held-out relations in a genuinely different setting. The released counting tool is a useful artifact. However, the headline threshold numbers are estimated post hoc from the same scatterplots they describe, with a small number of relations and autocorrelated checkpoint observations, and the frequency proxy is noisy. These issues make the strong version of the threshold claim conditional rather than established.","major_comments":[{"comment":"The threshold values (GPT-J 1,097; OLMo-7B 1,998; OLMo-1B 4,447) are read off the same scatterplots that are used to assert them: the text defines the threshold as the co-occurrence count above which average causality exceeds 0.9, with no confidence intervals, no leave-one-relation-out analysis, and no held-out validation. With only 24 relations, a single low-frequency, low-causality relation such as star-constellation-name can move the crossing point substantially, so the abstract's claim that linear representations 'consistently ... form when ... co-occur at least 1k and 2k times' is not supported by the reported evidence. Please add bootstrap or leave-one-relation-out estimates of the threshold and report their variability; if the threshold does not transfer to held-out relations, the strong claim should be weakened to the correlation statement.","section":"Sec. 4.2, Fig. 2"},{"comment":"The load-bearing definition of a relation occurrence is 'any time two terms appear in the same sequence within a batch', which is used both for the threshold finding and for the frequency-prediction regression. The paper cites Elsahar et al. (2018) for this proxy but provides no validation within this study that same-sequence co-occurrence corresponds to a textual mention of the factual relation. Co-occurrences in lists or unrelated contexts are counted as relation occurrences, so the causal interpretation that 'frequency of factual relation mentions' drives LRE formation is conflated with generic co-occurrence frequency. Please quantify the share of counted co-occurrences that are true relation mentions (e.g., by template or dependency matching) and rerun the central analyses on the relation-mention subset; the threshold numbers may change.","section":"Sec. 3.2"},{"comment":"The reported Pearson correlation r = 0.82 pools observations across eight OLMo checkpoints per relation, but these checkpoints are not independent: they share the same architecture, the same corpus, and sequential training steps. Treating each (relation, checkpoint) pair as an independent observation inflates the effective sample size for both the correlation and the threshold estimation. Please report within-relation correlations, cluster the analysis by checkpoint or by relation, and include a non-parametric correlation (e.g., Spearman) to show that the headline association is not an artifact of pooling autocorrelated points.","section":"Sec. 4.1, Sec. 4.2"},{"comment":"The cross-model frequency prediction relies on scaling GPT-J counts by 'the ratio of total tokens trained between the two models', but the paper does not state how this ratio is computed, whether it is known a priori, or whether it was fitted to improve the results. Because the applied claim about predicting counts of closed-data models depends on this scaling, please specify the scaling, justify it independently of the evaluation data, and test sensitivity to the chosen ratio. In addition, Table 1 shows that for subject-object co-occurrence prediction evaluated on OLMo, LRE features (0.68) are essentially at the mean-frequency baseline (0.67); the paper's summary that LREs 'encode a rich signal' should be qualified for this setting.","section":"Sec. 5.3, Table 1"}],"minor_comments":[{"comment":"The manuscript contains unfinished editorial markers that should be removed before publication: 'TODO: What to do about awkward spacing' in the Figure 2 caption, 'TODO: I was going to define causality and faithfulness here' in Section 3, and 'TODO: make more concise, fill in other sections' in the Related Work heading. The main text also never actually defines causality and faithfulness despite promising to do so; please add the definitions or point to a precise appendix location.","section":"Throughout"},{"comment":"The caption text includes a stray incomplete sentence ('We highlight Even at very early training steps...'), and the figure label 'Frequency v . Efficacy' contains typographical errors. Please proofread the figure and caption carefully.","section":"Fig. 2 caption"},{"comment":"The sentence 'We fit 24 models such that each relation is held out once per random seed across 4 seeds' is ambiguous: if there are 24 relations and 4 seeds, the reader would expect 96 fits, not 24. Please clarify the exact protocol.","section":"Sec. 5.1"},{"comment":"The figure captions label the accuracy columns 'N Shot' instead of '5 Shot' in several places; please correct the label and explain whether 'N' denotes the number of demonstrations.","section":"Appendix F"},{"comment":"The notation in Equation (1) is not self-contained: the reader is told that F(s, c) = o is a forward pass, but the symbols s, c, o, and the Jacobian are not defined precisely enough to reproduce the computation without the Hernandez et al. paper. Please add a short formal definition of these objects.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict and the skeptic's concern are both on target, and I largely agree with them. The most serious issue is that the headline 1k/2k/4k thresholds are post hoc estimates from a 24-relation scatterplot with no out-of-sample check; that claim is presented in the abstract as a discovery, not as a descriptive fit. The paper is otherwise publishable in principle, and the released tool and cross-model regression are real strengths. I would ask for a leave-one-relation-out or bootstrap-based threshold analysis and a clearer statement of how the cross-model scaling is set. Note also that the manuscript text contains unfinished TODO markers; if this is the submitted version, it needs a full editorial pass before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the central claim holds up better than the stress-test note makes it sound. The correlation between subject-object co-occurrence counts and LRE causality is r=0.82 across checkpoints, and the checkpoint analysis is a genuine step beyond earlier LRE work. What does not hold up is the precision of the 1k/2k/4k thresholds. Those numbers are read off the same scatterplots they describe, with 24 relations, no held-out relations, and autocorrelated checkpoints from the same model runs. I would treat them as rough anchors, not as laws.\n\nWhat is new here: tracking LRE causality across eight OLMo checkpoints against exact batch-level co-occurrence counts, and showing the frequency-linearity link is present from early in training. The released batch-search tool is real engineering. The cross-model regression—fit LRE metrics on OLMo, predict counts on GPT-J—is a legitimate generalization test, and the finding that hard causality is the most informative feature is a useful signal for dataset-inference work.\n\nWhere the paper is soft, in order of softness:\n\n(1) The thresholds. The abstract's \"consistently (but not exclusively) forms when ... co-occur at least 1k and 2k times\" is exactly a post hoc fit. Leave-one-relation-out would give a fair estimate; the paper does not do it. A single low-frequency relation like star-constellation-name can move the crossing point a lot.\n\n(2) The co-occurrence proxy. Counting any subject-object co-occurrence in a sequence as a relation mention is inherited from Elsahar et al. and is a real limitation, but the authors flag it in Appendix G and their limitation section. It weakens the interpretation, not the measured association between co-occurrence and LRE quality.\n\n(3) The transfer prediction. It is above the log-prob baseline and the mean baseline for object frequency, but for subject-object co-occurrence it is within noise of baseline. The error analysis shows some relations transfer very poorly (Arcturus/Boötes off by 346x). The paper admits this; I would want the claim \"new method for estimating properties of closed-data models\" softened accordingly.\n\n(4) The arXiv v1 text is unfinished: TODO placeholders in Section 3 and a figure caption, and some citations are malformed. That is an editorial problem, not a scientific one.\n\nBottom line: the frequency-linearity association is real and worth building on. The exact threshold numbers should not be quoted as robust findings. I would send this to serious reviewers, and I would bring it to our reading group.","headline":"A credible, useful empirical link between LRE quality and pretraining co-occurrence frequency—but the headline thresholds are post hoc fits to 24 relations and the arXiv text is unfinished.","tokens_in":25692,"tokens_out":2154,"would_cite":true,"duration_ms":22637,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"How often subjects and objects co-occur in pretraining largely determines whether a language model forms a linear representation of a factual relation, and that structure can reveal hidden training-data frequencies.","keywords":["linear representations","pretraining data frequency","language model interpretability","Linear Relational Embeddings","subject-object co-occurrence","training data inference","in-context learning","factual relations"],"falsifier":"Re-run the main OLMo-7B analysis with co-occurrence counts restricted to sequences whose templates express the relation (e.g., 'X plays the Y'), then check whether the roughly 2,000-count threshold and the r = 0.82 correlation with causality survive; if they vanish, the reported frequency–linearity link is an artifact of the co-occurrence proxy.","tokens_in":1755,"feed_emoji":"📈","tokens_out":2610,"duration_ms":91302,"temperature":0.7,"pith_summary":"This paper argues that whether a language model encodes a factual relation linearly is largely determined by how often the subject and object of that relation appear together in the pretraining corpus. It reports strong correlations (r = 0.82) between co-occurrence counts and the quality of Linear Relational Embeddings, with thresholds around 1,000 co-occurrences for OLMo-7B, 2,000 for GPT-J, and 4,000 for OLMo-1B, above which linear structure consistently forms. The same relationship holds at every pretraining checkpoint, so total exposure matters more than when the exposure happened. It then reverses the direction of the link: a regression trained on linear-representation quality can estimate how often a term appeared in pretraining data, and this regression transfers from OLMo-7B to GPT-J, suggesting that closed-data models leak frequency information through their linear structure. If this is right, linear-representation quality becomes a predictable, corpus-driven property rather than an unexplained emergent one.","feed_headline":"Co-occurrence counts predict when linear representations form","feed_subtitle":"A regression on representation quality can estimate how often terms appeared in a model's hidden pretraining data.","key_machinery":"The load-bearing object is the Linear Relational Embedding (LRE), an affine map W and bias b fitted to approximate the model's computation F(s, c) = o from a subject representation s to an object representation o in a few-shot context, with W estimated as the expected Jacobian of the model with respect to s across examples. The paper measures LRE quality with causality (whether editing the subject representation changes the prediction to another object of the same relation) and faithfulness (whether the LRE reproduces the model's own prediction), and pairs those measurements with exact co-occurrence counts obtained by searching tokenized training batches with a released counting tool. The argument is carried by correlating the two: co-occurrence frequency predicts causality at every checkpoint, and causality then predicts frequency in the regression analyses.","core_discovery":"Across the 25 factual relations studied in OLMo-7B, OLMo-1B, and GPT-J, the quality of linear representations is strongly correlated with how often subjects and objects co-occur in pretraining data (Pearson r = 0.82). For each model there is a consistent co-occurrence threshold above which average causality exceeds 0.9: about 1,998 for OLMo-7B, 1,097 for GPT-J, and 4,447 for OLMo-1B, and this threshold holds regardless of which pretraining stage the co-occurrences fall in. The same linear-representation metrics, fed into a regression model, predict how often a term appeared in pretraining more accurately than log-probability features alone, and a regression fit on one model transfers to another model with different training data. The paper concludes that linear representational structure contains signal about the pretraining corpus and that frequency is a primary determinant of when such structure forms.","pith_inferences":["An untested corollary is that a low causality score for a relation may simply mean the subject-object pair was undertrained, not that the model stores that fact non-linearly; frequency-normalized interpretability comparisons would settle this.","The thresholds suggest a data-curation recipe the paper does not causally test: deliberately adding subject-object co-occurrences to pretraining data might create or strengthen steerable linear directions, and a controlled repetition experiment would be the direct test.","If the cross-model transfer is a general phenomenon, term-frequency estimation could eventually apply to entirely closed-weight models with no released data at all, though the paper only demonstrates transfer between two open-weight models.","Counting relation mentions by templated contexts rather than raw co-occurrence would sharpen or shift the reported thresholds, making template-filtered counts the natural next experiment."],"forward_implications":["A relation whose average subject-object co-occurrence exceeds the model-specific threshold will almost always show a high-quality linear representation, even at very early pretraining checkpoints; the exact training stage does not matter.","Causality correlates with subject-object co-occurrence (r = 0.82) more strongly than with subject frequency alone (r = 0.66) or object frequency alone (r = 0.59), making co-occurrence the operative statistic for linear-structure formation.","Regression models using LRE metrics predict how often an object appeared in pretraining within one order of magnitude about 70% of the time, a level the paper shows log-probability features alone do not reach.","A regression fit on one model's LREs transfers to another model trained on different data, providing a way to estimate term frequencies in open-weight models whose pretraining data is not public.","High few-shot accuracy can coexist with weak linear structure for low-frequency relations, so accuracy and linearity are related but not identical properties of a factual relation."],"supporting_citations":[{"why":"Defines LREs, the Relations dataset, and the faithfulness and causality metrics the paper uses to measure linear structure.","marker":"Hernandez et al. (2024)"},{"why":"Provides the subject-object co-occurrence proxy for relation mentions that all frequency counts and thresholds rely on.","marker":"Elsahar et al. (2018)"},{"why":"Supplies the OLMo-1B and OLMo-7B models and their intermediate checkpoints used to track LRE development over training.","marker":"Groeneveld et al. (2024)"},{"why":"Provides the Dolma corpus used to count OLMo subject-object co-occurrences via batch search.","marker":"Soldaini et al. (2024)"},{"why":"Supplies GPT-J, the second model used for the cross-model regression transfer.","marker":"Wang & Komatsuzaki (2021)"},{"why":"Provides the Pile corpus used for GPT-J co-occurrence counts.","marker":"Gao et al. (2020)"},{"why":"Provides the WIMBD search index used for GPT-J counts and to validate batch-search counts against whole-corpus counts.","marker":"Elazar et al. (2024)"},{"why":"Shows that linear structure in static embeddings depends on term frequency, motivating the paper's frequency hypothesis.","marker":"Ethayarajh et al. (2019)"}],"fun_headline_variants":["Co-occurrence frequency predicts linear representation strength","Term frequency shapes when LMs form linear representations","Linear representations leak pretraining data frequency","Data co-occurrence sets threshold for linear representations","Representation quality estimates pretraining term frequency"],"cache_read_input_tokens":27776,"weakest_assumption_plain":"All frequency counts treat a subject and object appearing in the same tokenized training sequence as a mention of the factual relation, so if most co-occurrences are incidental, the thresholds and the frequency regressions measure the counting proxy rather than actual knowledge of the fact.","fun_headline_variants_meta":{"raw":{"variants":["Co-occurrence frequency predicts linear representation strength","Term frequency shapes when LMs form linear representations","Linear representations leak pretraining data frequency","Data co-occurrence sets threshold for linear representations","Representation quality estimates pretraining term frequency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1448,"prompt_tokens":1041,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":657,"tokens_out":407,"duration_ms":4262,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:31:05.515922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the main OLMo-7B analysis with co-occurrence counts restricted to sequences whose templates express the relation (e.g., 'X plays the Y'), then check whether the roughly 2,000-count threshold and the r = 0.82 correlation with causality survive; if they vanish, the reported frequency–linearity link is an artifact of the co-occurrence proxy.","supporting_citations":[{"cited_title":"T - RE x: A large scale alignment of natural language with knowledge base triples","cited_arxiv_id":null,"evidence_quote":"Provides the subject-object co-occurrence proxy for relation mentions that all frequency counts and thresholds rely on."}],"review_version":1}