{"id":"183fb5ec-274f-48c5-a253-c8c3525cf790","arxiv_id":"2506.03434","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Over 40 checkpoints of OLMo-7B, attention heads and FFNs shift from general-purpose to specialized roles for factual recall, with location-based facts learned earlier and more stably than name-based facts.","lead":"This paper tracks how the attention heads and feed-forward networks of the 7-billion-parameter OLMo model change their jobs as the model is trained on more data. It finds that components start out general and become specialized, and that location facts are learned earlier than person-name facts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-specific head turnover may be threshold noise: §4.1.4 labels a head 'fact-answer' if IFR includes it for one answer token (160 facts), yet no null baseline is reported, so the high turnover driving the 'repurposing' claim may be spurious.","rationale":"The central scientific contribution is the temporal narrative about component evolution, especially the claim that attention heads are dynamically repurposed after facts are learned. The reader's weakest-assumption choice—IFR faithfulness at 7B scale and fixed thresholds—is defensible but broad; I narrow it to the specific link where a spurious result is most likely: the permissive fact-answer definition. The paper's strongest evidence for adaptive learning is the very low IoU for answer-specific heads even between adjacent checkpoints (0.2, §4.3). If that number reflects only that a head is labeled fact-answer whenever it appears in one answer token's circuit, then with 160 facts and thousands of tokens per snapshot, adjacent snapshots will share few such heads by chance, and high turnover follows from the taxonomy rather than from learning. The other findings (general heads increasing in count, LOC before NAME) are less affected by this artifact: the general-role threshold is microaveraged over all tokens and is much less permissive, and the accuracy curves are behavioral. The proposed permutation test is cheap, uses the authors' released data, and directly separates genuine role churn from classification noise. If the null test passes, the conditional acceptance is justified; if it fails, the dynamic-repurposing claims need revision or removal. I therefore retain the reader's CONDITIONAL verdict, with a sharper condition than the one stated in the original review.","tokens_in":22266,"tokens_out":10197,"duration_ms":114319,"concrete_test":"Run a permutation null test on the released circuit data: for each snapshot, preserve each component's marginal frequency of appearing in answer-token circuits but shuffle the mapping from facts to those answer-token circuit memberships 1000 times. Recompute the fact-answer role set, the S40-vs-Main IoU (reported as 0.2), and the accumulated switch and Markov transition counts. If the observed IoU and switch totals fall within the 95% null interval, the adaptive-repurposing claim collapses to a threshold artifact; if they exceed the interval, the turnover is real even under this conservative null. This check uses only the released code/data and costs minutes per snapshot.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central dynamic narrative—general components first, specialization later, and repurposing after reliable prediction—is inferred entirely from role labels obtained by thresholding IFR circuit membership. The weakest point is the fact-answer role in §4.1.4: a head c is classified fact-answer specific if cf_s > θ (θ=0.1), where cf_s is the circuit indicator averaged over the answer subtokens of a single fact. For one- or two-subtoken answers, cf_s is effectively binary, so any head that appears in the IFR circuit for even one answer token of one of the 160 facts qualifies. This is an extremely permissive and low-selectivity criterion. The paper reports that answer-specific heads have the highest turnover—IoU between the final checkpoint and the one just before it is 0.2 (§4.3, Fig. 3)—and uses this as evidence of adaptive reallocation. But with a per-token threshold and only 160 facts, a head's fact-answer label can change between adjacent snapshots merely because IFR's pruning threshold includes or excludes a single token among thousands. No comparison is made against a null distribution (e.g., shuffling fact labels or marginal activity counts) to show that the observed turnover exceeds what random assignment would produce. The same issue propagates into the Markov-chain transition probabilities (Fig. 6) and the accumulated switch counts (Fig. 4). If the fact-answer role is largely noise, the 'highest turnover' and 'repurposed components' conclusions are unsupported, even if the general-to-specialized trend in aggregate counts (which uses a more robust all-token average) survives. The authors' stated threshold-sensitivity check (Limitations) addresses the value of θ, not the absence of a null model for role identity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a time-course mechanistic interpretability study of OLMo-7B, tracking attention heads and FFNs across 40 pretraining snapshots. Using Information Flow Routes (IFR), it classifies components into four roles (general, entity, relation-answer, fact-answer) based on thresholded circuit membership, and reports that general components dominate early, specialization increases over training, answer-specific heads exhibit the highest turnover (supporting a 'repurposing' narrative), and FFNs are stable. Behavioral probing shows location-based relations are acquired earlier and more reliably than name-based relations, which the authors attribute to corpus frequency differences. The paper releases code and data.","tokens_in":22529,"tokens_out":5047,"duration_ms":47366,"significance":"If the central empirical claims are valid, this would be a valuable contribution: it is one of the first studies to trace component-level circuit evolution for factual recall at 7B scale across many checkpoints, and it provides a new probing dataset and open code. The behavioral LOC/NAME asymmetry and the Infini-gram frequency analysis are concrete and falsifiable. However, the component-level conclusions currently rest on thresholded IFR attributions without null baselines or uncertainty quantification, so the mechanistic narrative should be treated as provisional. The paper also explicitly acknowledges that it could not validate IFR against activation patching due to computational constraints.","major_comments":[{"comment":"The fact-answer specific role is defined by cf_s = c^f_{srf}(T_a) > θ with θ = 0.1, averaged over the answer subtokens of a single fact. For one- or two-subtoken answers, cf_s is effectively a binary indicator of whether the head appears in the IFR circuit for any answer token of that fact. Because the dataset contains only 160 facts, any head that appears in the circuit for one answer token of one fact is classified as fact-answer specific. This extremely permissive criterion is used to support the claim that answer-specific heads have the highest turnover (IoU = 0.2 between the final checkpoint and the one immediately before it) and that components are 'repurposed.' No null baseline is reported—for example, shuffling fact labels, permuting circuit membership, or comparing against marginal activity counts—so the observed turnover may be threshold jitter rather than functional reallocation. The Markov transition probabilities in Fig. 6 inherit this problem. This is load-bearing for the adaptive repurposing claim and must be addressed with a null model and/or a stricter role definition.","section":"§4.1.4, §4.3, Fig. 3"},{"comment":"The role labels and the temporal trends are both derived by thresholding the same IFR activation scores (θ = 0.1 for heads, 0.9 for FFNs). Consequently, the statements that specialized heads 'increase over time' and that general heads are 'stable' are, to a substantial degree, restatements of the classification rule rather than independent discoveries: any monotone increase in IFR scores will produce an increase in specialized counts. The only independent measurement is behavioral accuracy (Sec. 3.3). To break this circularity, the paper should validate the role assignments against an independent causal method (for example, activation patching on a subset of heads or facts) or at minimum show that the temporal trends are robust to a range of thresholds with confidence intervals. The Limitations section acknowledges the lack of patching validation, but the central claims depend on it.","section":"§4.1, §4.2"},{"comment":"The probing dataset was curated by selecting facts that the fully trained main model answers with high confidence (first-token probability above 75%, second-token below 10%, Appendix B step 3). Therefore the acquisition curves in Fig. 2 describe only facts that are known at the end of training. This selection can bias the relative timing of LOC vs. NAME acquisition: if the main model is more confident on the selected LOC facts than on the selected NAME facts, the earlier convergence of LOC may be an artifact of the selection criterion. The authors should report the distribution of main-model confidence for both groups, and ideally re-run the analysis on a held-out set of facts selected without reference to the final model's confidence.","section":"§3.1, Appendix B"},{"comment":"All reported counts, IoU values, and transition probabilities are point estimates without error bars or significance tests. For example, the count of relation-answer heads at S1 is 8 out of 1024, and the IoU = 0.2 for answer-specific heads between adjacent snapshots could easily arise from a handful of heads flipping in or out of the permissive fact-answer definition. The statement in Limitations that varying θ 'does not alter the overall trends' is not supported by any presented sensitivity analysis. Please provide bootstrap confidence intervals, a threshold sweep, and/or per-snapshot error estimates for the central metrics.","section":"§4.3, §4.4, Fig. 3"}],"minor_comments":[{"comment":"In the definition of IoU(Hg), the text says 'Hgs represents the set of entity components' but the equation and surrounding text refer to general components; this is a typo that should be fixed.","section":"§4.2"},{"comment":"The heatmap matrices are difficult to read because the numbers are not aligned with their row and column labels in the rendered figure; please reformat for clarity.","section":"Fig. 4"},{"comment":"The 'proper' sets use a hierarchical exclusion (Hf = Jf − Jr − Je − Jg). The authors should report the sizes of the non-proper intersections (for example, heads that are both relation-answer and fact-answer), because these are deliberately excluded and may be large; such reporting would help readers interpret the counts.","section":"§4.1.5"},{"comment":"The snapshot naming is inconsistent: the text says '40 snapshots ... S1-20B to S39-838B' but later sections refer to 'S40-838B'; clarify the final snapshot index and its token count.","section":"§3.2"},{"comment":"The template selection criterion mentions 'average probability of the facts where the first token is correct,' but the exact formula and the role of the 75% threshold could be stated more precisely.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the dataset/code release is a plus. The main risk is that the headline mechanistic narrative may be an artifact of thresholding the same IFR scores used for classification; requiring null baselines, uncertainty quantification, and an independent validation (even on a subset) is a reasonable bar for revision. I do not see grounds for rejection, but the revision needs to be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first study I know that tracks role-level evolution of attention heads and FFNs over 40 checkpoints of a 7B model using IFR, and the LOC-before-NAME acquisition ordering is a clean behavioral result. Second, the paper's most exciting dynamic claim—answer-specific heads are repurposed, with very high turnover—rests on a fragile per-token classification that has no null baseline, so I would not yet trust it.\n\nThe dataset is carefully curated (160 facts, templates validated against ambiguity), the snapshots span 20B to 838B tokens, and they release code and data. The aggregate trajectories—general heads dominating early, specialization increasing later, FFNs staying general—are consistent across the presented curves and survive the threshold sensitivity check they report in the Limitations. The Markov chain and switch-count analyses are a reasonable way to frame the dynamics. The related-work coverage is adequate; they cite Tigges et al. and Chang et al. and position against them. Credit is due: thoughtful study design, honest treatment of compute constraints.\n\nThe soft spots, in proportion. The fact-answer role (Sec. 4.1.4) is load-bearing for the repurposing claim, and it is defined too permissively: for one- or two-subtoken answers, cf_s > 0.1 is nearly binary, so a head that appears in one token's circuit for one of 160 facts gets labelled fact-answer. With no comparison against a null distribution, the observed turnover (IoU 0.2 between final and penultimate checkpoints) could be threshold noise rather than functional reallocation. The same issue propagates into the Markov transition probabilities and switch counts. The circularity concern is fair but only partial: role labels and counts both come from thresholding IFR, yet the accuracy curves are measured independently, and the aggregate specialization trend is not just a restatement of the taxonomy. The probe set is selected by main-model confidence, so acquisition curves only cover facts known at the end; that deserves more prominence. And they acknowledge not validating IFR against activation patching; a few patched heads would strengthen the causal story.\n\nWho is this for? Anyone working on circuit formation, continual learning, or factual recall in LLMs. It is a useful observational map, not a definitive mechanism paper. It deserves peer review; a serious referee should ask for a null model for fact-answer roles, error bars on counts, an analysis of the full fact set, and a small-scale causal check. I would not cite the turnover claim in its current form, but the dataset and the LOC/NAME acquisition result are citable.\n\nRecommendation: send to review, expecting substantial revision.","headline":"First time-course role map of a 7B model's factual recall circuitry, with a clean LOC-before-NAME result—but the high-turnover 'repurposing' claim needs a null model before it is credible.","tokens_in":23194,"tokens_out":3391,"would_cite":true,"duration_ms":32500,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"As OLMo-7B trains, factual recall shifts from broad, general-purpose attention heads to specialized ones; attention heads keep changing roles while feed-forward layers stay stable, and location facts are learned before name facts.","keywords":["mechanistic interpretability","factual recall","training dynamics","attention heads","feed-forward networks","circuit analysis","information flow routes","OLMo-7B"],"falsifier":"Run activation patching or causal tracing on a small set of snapshots (for example S5, S14, and the final model) on these same 160 facts: if the heads that IFR labels as answer-specific are not the ones whose intervention most changes the predicted answer, the role-classification timeline collapses. A cheaper check is to examine the raw distribution of IFR scores per snapshot: if the number of heads crossing the 0.10 threshold rises simply because the whole score distribution inflates as training proceeds, then per-snapshot rescaling of the thresholds should erase the observed growth in specialized heads.","tokens_in":22018,"feed_emoji":"🧠","tokens_out":11456,"duration_ms":109138,"temperature":0.7,"pith_summary":"This paper tries to show how factual knowledge takes shape inside a large language model while it is still training, by watching which internal components drive each answer at 40 successive checkpoints of OLMo-7B. The central claim is a general-to-specialized progression: early training leans on broad, multipurpose attention heads, and only later do specialized heads emerge for specific relations and facts, at which point some previously active components are repurposed, a sign the authors read as adaptive learning. The evidence also separates the two component types: attention heads show the highest role turnover, while feed-forward layers behave like a stable backbone that mostly sticks to general processing. A separate behavioral result is that location-based facts reach high accuracy earlier and with less volatility than name-based facts. If the picture holds, it gives a time-resolved mechanistic account of knowledge formation that could guide when and how to edit, prune, or fine-tune a model.","feed_headline":"LLM fact recall starts general, then specialists take over","feed_subtitle":"A 40-checkpoint trace of OLMo-7B shows attention heads repurposing while feed-forward layers hold steady","key_machinery":"The load-bearing objects are Information Flow Routes (IFRs), a method that walks the model's computational graph backward from the token being predicted and keeps only the edges whose modified ALTI (Aggregation of Layer-Wise Token-to-Token Interactions) contribution exceeds a threshold, and a five-way role taxonomy built on which token positions a component serves. A component is 'general' if it fires across all subtokens, 'entity' if it fires on subject and answer, 'relation-answer' if it fires on the answer for all facts of one relation, and 'fact-answer' if it fires on the answer for a single fact; 'proper' variants subtract the broader roles, and everything below threshold counts as 'deactivated'. Stability is quantified by Intersection-over-Union comparisons of each role's component set against the fully trained model, and role changes are modeled as a Markov chain over snapshots. This combination is what makes a 40-snapshot, 7-billion-parameter study feasible: the IFR extractor is cheap enough to run at every checkpoint, and the taxonomy converts raw circuit membership into a trackable claim about specialization over time.","core_discovery":"By tracing Information Flow Routes through OLMo-7B's 40 training snapshots and classifying every attention head and feed-forward layer into one of five roles (general, entity, relation-answer, fact-answer, deactivated), the paper claims that factual knowledge is assembled gradually rather than appearing all at once. The number of active attention heads grows from 113 to 423 across training, the sets of specialized heads keep turning over (answer-specific heads have an Intersection-over-Union of only 0.2 between the final model and the checkpoint just before it), and heads frequently pass through deactivated states; yet the total count of specialized heads rises because new specializations form faster than old ones dissolve. Feed-forward layers, by contrast, mostly remain general throughout, with a detectable shift into relation-answer roles around the point where accuracy passes 80%. The authors also find that location-based relations cross a top-1 accuracy of 0.8 by snapshot S5, whereas name-based relations only reach that level at S14, and that name-based relations recruit more heads and more role switching in the middle layers. They interpret the combined pattern as a hierarchical learning process in which stable general components build the groundwork and dynamic attention heads adapt to refine answers.","pith_inferences":["If the observed head turnover is genuine functional plasticity, then single-checkpoint circuit maps may be systematically misleading; averaging circuits across a training stage could give a more stable target for downstream interpretability work.","The authors' own caveat that a high threshold and coarse module granularity could make FFNs look general suggests a refinement: repeating the role classification at neuron or sublayer resolution might reveal specialization that the layer-level view hides.","A direct test of the task-complexity explanation is available: re-measure convergence order on the same ten relations after balancing corpus frequencies, or hold frequency fixed while varying the name-versus-location distinction, to separate data prevalence from task type.","The validation step the authors could not run, activation patching on even a few snapshots, is the decisive experiment: if patching the heads IFR identifies as answer-specific does not change the predicted answer, the role timeline is an artifact of the attribution method."],"forward_implications":["Factual-recall circuits keep reorganizing late into training, so the components behind a given answer at one checkpoint are not the components behind it a few hundred billion tokens later.","Attention heads are the adaptable part of the system: answer-specific heads repurpose most often, so head-level analyses of a single checkpoint are time-stamped statements, not permanent properties of the model.","Feed-forward layers stay mostly general, which marks them as the steadier substrate for knowledge-related interventions such as editing or pruning.","Task complexity shows up in acquisition order: location-based facts converge earlier and more stably than name-based facts, and the harder name-based relations engage more heads and more role switching in the stable middle layers.","Most heads never take part in factual recall: around 60% stay deactivated even in the fully trained model, and the Markov analysis gives a deactivated-to-deactivated transition probability of 0.96."],"supporting_citations":[{"why":"Supplies the Information Flow Routes method used to extract the circuits at every training snapshot.","marker":"Ferrando and Voita, 2024"},{"why":"Provides the ALTI score that IFR modifies to measure edge contributions, and the high threshold used for FFN roles.","marker":"Ferrando et al., 2022"},{"why":"Provides OLMo-7B and its pre-training checkpoints, the object of the time-course study.","marker":"Groeneveld et al., 2024"},{"why":"One of the resources the probing dataset is inspired by and the source of the relation-decoding framing for factual recall.","marker":"Hernandez et al., 2024"},{"why":"CounterFact is an inspiration for the dataset and supplies the locating-and-editing framing of factual associations.","marker":"Meng et al., 2022"},{"why":"ParaRel is an inspiration for the dataset and the template-consistency approach to probing.","marker":"Elazar et al., 2021"},{"why":"Summing Up The Facts is an inspiration for the dataset's fact selection.","marker":"Chughtai et al., 2024"},{"why":"Marks the attribution-patching alternative that motivates the choice of the cheaper IFR method across 40 snapshots.","marker":"Hanna et al., 2024"},{"why":"Supplies the background account of where factual knowledge is stored (FFNs enriching subjects, attention passing relations) that the role taxonomy builds on.","marker":"Geva et al., 2023"}],"fun_headline_variants":["Attention heads churn as LLM knowledge forms, FFNs stay put","OLMo-7B trace: attention heads turn over, FFNs stay stable","Fact knowledge builds gradually: dynamic heads, stable FFNs","Location facts learned earlier than name facts in LLMs","Attention heads repurpose, FFNs endure during LLM training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The timeline of specialization rests on the assumption that the Information Flow Route scores faithfully identify the components that actually cause each prediction, so the fixed thresholds that assign roles are separating real functional changes from noise in the attribution scores.","fun_headline_variants_meta":{"raw":{"variants":["Attention heads churn as LLM knowledge forms, FFNs stay put","OLMo-7B trace: attention heads turn over, FFNs stay stable","Fact knowledge builds gradually: dynamic heads, stable FFNs","Location facts learned earlier than name facts in LLMs","Attention heads repurpose, FFNs endure during LLM training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001023,"raw_usage":{"total_tokens":4330,"prompt_tokens":973,"completion_tokens":3357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":3268}},"tokens_in":589,"tokens_out":3357,"duration_ms":24079,"temperature":1.0,"reasoning_tokens":3268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:03:18.419064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run activation patching or causal tracing on a small set of snapshots (for example S5, S14, and the final model) on these same 160 facts: if the heads that IFR labels as answer-specific are not the ones whose intervention most changes the predicted answer, the role-classification timeline collapses. A cheaper check is to examine the raw distribution of IFR scores per snapshot: if the number of heads crossing the 0.10 threshold rises simply because the whole score distribution inflates as training proceeds, then per-snapshot rescaling of the thresholds should erase the observed growth in specialized heads.","supporting_citations":[],"review_version":1}