{"id":"4be0c33b-a0fb-49ce-b638-80a06388b37c","arxiv_id":"2602.22600","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Trained transformers contain low-dimensional causal subspaces — algorithmic cores — that recur across runs and scales and can be extracted, characterized, and steered.","lead":"This paper introduces Algorithmic Core Extraction (ACE), a method for finding the tiny subspaces inside transformers that are essential for a task, and shows these \"cores\" reappear in independently trained models. The method also locates a one-dimensional axis governing subject-verb agreement in GPT-2; flipping it inverts singular and plural language.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ACE's causal validation is self-referential: at GPT-2 Large's final-layer core, the extracted axis is a covariance-weighted readout direction, so the Table 5 AUC swings may be built into the estimator rather than evidence of a shared algorithmic core.","rationale":"The reader's weakest assumption — that the SVD of H J^T identifies the true algorithm rather than merely the output's linear sensitivity — is exactly the load-bearing issue. My stress-test sharpens it by noting that at GPT-2 Large's final-layer core, f is affine so J is constant, making the core a covariance-weighted readout direction and the ablations partially tautological. This does not invalidate the Markov and modular-addition results, which have ground truth and are less susceptible to this critique; the Markov spectral recovery is a genuinely nice independent check. But the paper's central, headline claim is about LLM algorithmic cores, and the GPT-2 evidence is currently the only LLM evidence. The missing LLaMA/Gemma/Qwen results in the abstract are another real support gap, but even adding them would not resolve the selection–evaluation coupling without random-subspace or alternative-functional controls. The reader's CONDITIONAL verdict already requests exactly these controls, so my analysis reinforces rather than changes that verdict.","tokens_in":21250,"tokens_out":13235,"duration_ms":133343,"concrete_test":"For GPT-2 Large at layer 36, draw 1,000 random unit vectors q in R^1280 and compute the Table 5 ablations (core-only, core-removed, core-flipped AUC) using the rank-1 projector P = q q^T, with all other analysis steps unchanged. If the median random q yields core-removed AUC < 0.3 or core-flipped AUC < 0.5, the reported effect is not unique to the ACE axis. Separately, rerun ACE with f replaced by a different linear functional of the same logits (e.g., logit('the') − logit('a')); if the extracted 'core' tracks the new f and reproduces the same necessity/sufficiency/steerability pattern, the core is readout-defined rather than algorithmically invariant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the selection–evaluation coupling in ACE as applied to GPT-2. At the core layer chosen for GPT-2 Large (layer 36, the final hidden state before unembedding), the target f(h) = (logit_are + logit_were) − (logit_is + logit_was) is affine in h, so every Jacobian row is the same vector v. Then H J^T = (H v) 1^T is rank 1, and the ACE core direction is proportional to (H^T H)v — a covariance-weighted readout direction, not an independently discovered algorithm. Core-only, core-removed, and core-flipped ablations therefore necessarily move f by construction, so Table 5's AUC values are partially self-fulfilling. The same coupling weakens the cross-scale alignment claim: aligning the extracted axes may amount to aligning the three GPT-2 unembeddings' number margins, not demonstrating a shared 1D computational core. The paper explicitly cautions that probe scores and subspace interventions can be deceptive (refs 31–33) but does not run the obvious control at the same layer: random or alternative 1D subspaces. This concern is more fundamental than the missing LLaMA/Gemma/Qwen results, because it questions the interpretation of the GPT-2 evidence that is actually present.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Algorithmic Core Extraction (ACE), a subspace method that selects directions in transformer activations that are both high-variance and output-relevant, validates the selected subspace through projection ablations, and interprets it by fitting linear dynamics. ACE is applied to three settings: single-layer transformers trained on a four-state Markov chain, two-layer transformers trained on modular addition, and pretrained GPT-2 Small/Medium/Large on subject-verb agreement. The paper claims that Markov-chain models converge to 3D cores recovering the transition spectrum, that modular-addition cores crystallize at grokking and later inflate under weight decay, and that GPT-2 agreement is governed by a one-dimensional late-layer axis shared across scales. The abstract additionally announces results on LLaMA-3.1, Gemma-2, and Qwen2.5 and a two-order-of-magnitude scale range, but the body contains no such experiments.","tokens_in":21604,"tokens_out":8551,"duration_ms":88175,"significance":"If valid, ACE would provide a genuinely useful invariant-based target for mechanistic interpretability, complementing circuit analysis and sparse-autoencoder approaches. The strongest evidence is in the controlled settings: the Markov-chain core spectra match ground truth to within 1%, and the automated recovery of rotational operators for modular addition is compelling because no algorithmic form is assumed. The attempt to derive and test a quantitative grokking law is also valuable. The GPT-2 cross-scale axis-alignment result, if confirmed with appropriate controls, would be a striking empirical finding. However, the GPT-2 causal evidence is weakened by selection-evaluation coupling, and the abstract overstates the evidence, so the significance is conditional pending revision.","major_comments":[{"comment":"The selection-evaluation coupling undermines the GPT-2 Large causal claim. At layer 36 (Table 4), the final hidden state before unembedding, the target f(h) = (logit_are + logit_were) - (logit_is + logit_was) is affine in h, so every Jacobian row is the same vector v; H J^T = (H v) 1^T is rank one and the ACE core reduces to span(H^T H v). Core-only/removed/flipped ablations therefore preferentially move f by construction, and the Table 5 AUC swings are not independent evidence of a unique algorithmic core. The manuscript cites refs. 31-33 on deceptive subspace interventions but does not run the obvious control: random or alternative one-dimensional subspaces at the same layer, or directions selected from a held-out target. Please add such controls and re-state the GPT-2 interpretation accordingly.","section":"Methods ('Algorithmic core extraction'; 'Subject-verb agreement experiment'); Table 5"},{"comment":"The abstract claims ACE is applied to six language models (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, Qwen2.5) and that subject-verb agreement is governed by a single axis 'that aligns across architectures.' The body contains no LLaMA, Gemma, or Qwen analysis; only GPT-2 Small/Medium/Large are reported. The claimed two-orders-of-magnitude scale range and the cross-architecture alignment claim are therefore unsupported by the manuscript. Either report the missing experiments or remove/qualify these claims in the abstract and introduction.","section":"Abstract; Results ('A Universal Agreement Core Across GPT-2 Scales')"},{"comment":"The inverse grokking law tau(p) = -Omega log(1 - p_crit/p) is validated by fitting Omega ~ 2770 and p_crit ~ 23 to the same p-sweep that is then compared with the curve; R^2 > 0.99 is an in-sample goodness-of-fit, not a predictive test. Since p_crit is near the smallest tested modulus (23 vs. p = 31), the claimed divergence near p_crit is heavily informed by the fit. Report parameter uncertainties, perform leave-one-out prediction across moduli or other out-of-sample checks, or fix p_crit from theory before fitting so that the announced scaling law is actually falsifiable.","section":"Box 1; Methods ('Grokking sweeps and scaling fits')"}],"minor_comments":[{"comment":"The spectral-gap entry for GPT-2 Large is typeset in a confusing way ('12.8×10 10'); clarify the value and units so the one-dimensionality claim can be checked.","section":"Table 4"},{"comment":"The fit quality R^2_h is used in several places but never explicitly defined; distinguish it from the oracle-relative R^2 used in the Markov experiment.","section":"Figure 2; Methods ('Modular Addition Experiment')"},{"comment":"The adaptive steering procedure has several thresholds (gating mass, target margin, cap); provide explicit default values and a sensitivity analysis, since the open-ended generation examples depend on these choices.","section":"Methods ('Subject-Verb Agreement Experiment')"},{"comment":"The text says code 'will be made available' at a URL; for reproducibility, please provide a fixed release version or archive alongside the submission rather than a future link.","section":"Code Availability"}],"recommendation":"major_revision","confidential_remarks":"The missing LLaMA-3.1/Gemma-2/Qwen2.5 results in the abstract are a serious integrity issue, and the GPT-2 Large core extraction is confounded by the affine readout. Both can be addressed in revision either by adding the promised experiments and controls or by substantially narrowing the claims. The Markov-chain and modular-addition sections are the strongest and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about, but the abstract is ahead of the evidence. The new thing is ACE: a simple SVD recipe that looks for directions in activation space that are both high-variance and sensitive in the output Jacobian. The Markov-chain experiment is the strongest part: three independent runs produce non-overlapping subspaces, yet the fitted operators recover the ground-truth transition spectrum to within 1%, and the statistical alignment via CCA is a nice illustration of functional equivalence without geometric overlap. The modular addition story is also interesting: the core crystallizes at grokking, the operator eigenvalues snap to the unit circle without anyone prespecifying Fourier structure, and the post-grokking core inflation under weight decay is a genuine observation. The theoretical grokking law is plausible but it is a two-constant fit to the data, not a prediction; treat it as suggestive.\n\nThe soft spot is the GPT-2 section, and the stress-test note lands. The target function for agreement is a scalar margin, which is affine in the final-layer hidden state. So every Jacobian row is the same vector v, H J^T is rank 1, and the extracted core direction is essentially the covariance-weighted readout direction. The ablations then move f by construction. That doesn't make the numbers wrong, but it makes 'causal validation' partly self-fulfilling and it undermines the claim that the method discovered a shared 1D computational core rather than simply a probe direction. The paper cites refs 31–33 about deceptive subspace interventions and still does not include a random one-dimensional subspace baseline at the same layer. That is the missing control. And the abstract claims LLaMA-3.1, Gemma-2, and Qwen2.5 results that do not appear anywhere in the body; either those experiments exist and were omitted, or the claim should be withdrawn. Post hoc layer selection is minor by comparison.\n\nWho is this for? People who work on mechanistic interpretability and grokking will want to read the synthetic experiments. The paper deserves a serious referee—not because the language-model claims are solid, but because ACE is a fresh method and the Markov/modular results are clean. My recommendation: send it to review with a request for a random-subspace control on the GPT-2 analyses, a correction of the abstract, and code release. If the authors cannot show that a random axis at the same layer fails the ablations, the agreement-core section should be reframed as a cautionary demonstration rather than evidence for universality.","headline":"ACE is a promising method with two clean toy experiments, but the GPT-2 'universal agreement core' is largely a re-readout of the probing objective, and the abstract promises more than the body delivers.","tokens_in":22062,"tokens_out":3679,"would_cite":false,"duration_ms":35715,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Independently trained transformers converge to compact shared algorithmic cores that are necessary and sufficient for their tasks.","keywords":["algorithmic cores","mechanistic interpretability","invariance across training runs","grokking","modular addition","subject-verb agreement","GPT-2","functional equivalence"],"falsifier":"Re-run the GPT-2 agreement ablations on the same prompts but with a random one-dimensional projection drawn from the same activation distribution; if random axes show core-only AUC near baseline and core-removed AUC near chance, the core's necessity and sufficiency are artifacts of the projection and readout. Also run the ACE protocol on LLaMA-3.1, Gemma-2, and Qwen2.5 at their late layers: if their agreement axes do not align with GPT-2's, the universal cross-architecture claim fails.","tokens_in":21121,"feed_emoji":"🧭","tokens_out":5546,"duration_ms":56014,"temperature":0.7,"pith_summary":"Training selects for behavior, not circuitry, so this paper tries to show that what persists across transformers is not individual weights but a low-dimensional algorithmic core. In a controlled Markov-chain task, three independently trained one-layer transformers have near-zero weight similarity yet embed nearly the same 3D core, and operators fit inside the cores recover the chain's spectrum. In modular addition, cores crystallize at grokking as cyclic operators and later inflate under continued weight decay, yielding a predicted inverse scaling law for grokking time. In GPT-2 Small, Medium, and Large, subject-verb agreement is claimed to reduce to a single steerable axis whose reflection inverts grammatical number throughout open-ended generation.","feed_headline":"GPT-2 agreement lives on a single flippable axis","feed_subtitle":"If the paper is right, subject-verb agreement reduces to one invariant core across model scales.","key_machinery":"Algorithmic Core Extraction (ACE): from a layer's hidden activations H and the Jacobians J of a task readout, ACE forms the interaction H J^T, takes its SVD, projects the top singular modes back into activation space, and defines the core as their span; core-only and core-removed ablations then certify sufficiency and necessity. The paper motivates this as the nonlinear analogue of selecting the jointly reachable and observable part of a linear system, and the central role of the operator spectrum is that fitted linear dynamics inside the core are used to identify the computation.","core_discovery":"The paper claims that independently trained transformers solving the same task contain a low-dimensional subspace that is simultaneously necessary and sufficient for performance and is conserved across realizations even when weights diverge. For the Markov task, three one-layer transformers yield 3D cores embedded in nearly orthogonal subspaces, yet least-squares operators fit in core coordinates recover the transition matrix's eigenvalues to within about 1%. For modular addition, cores are shown to crystallize at grokking as cyclic operators and to inflate under sustained weight decay, with grokking delay following a predicted inverse scaling law in weight decay and redundancy. For GPT-2 Sm","pith_inferences":["Editorial caveat: the abstract announces agreement-core results for LLaMA-3.1, Gemma-2, and Qwen2.5, but the manuscript body reports only GPT-2 Small, Medium, and Large; the cross-architecture universality claim is therefore asserted, not yet demonstrated in this text.","Editorial inference: the same extraction could be applied to other grammatical features, such as pronoun case, gender, or tense, to test whether agreement cores are verb-specific or instantiate a general number variable; the paper leaves this untested.","Editorial inference: the grokking ODE predicts a sharp dependence on mode count; a direct test on another group operation, such as multiplication modulo p, would show whether the inverse law is specific to the cyclic structure of addition or a general redundancy effect.","Editorial inference: because cores are statistically similar but geometrically orthogonal, aligning core coordinates before merging models could yield greater success than weight-space interpolation; the paper suggests this as a diagnostic but does not run a merge experiment."],"forward_implications":["Explanations that target cores rather than circuits should transfer across random seeds, checkpoints, and architectures, because they describe what is preserved rather than what is particular.","Necessary-and-sufficient cores give a principled intervention handle: the GPT-2 steering result shows a single reflective axis can flip a linguistic behavior in open-ended generation.","Because grokking time is modeled as inversely proportional to weight decay and mode count, training schedules can be tuned to shorten memorization-to-generalization delays; annealing weight decay after grokking preserves compact cores.","Core alignment, not weight similarity, may serve as a diagnostic of functional equivalence and as a basis for principled model merging; the paper argues naive weight interpolation fails precisely because cores are embedded in nearly orthogonal subspaces.","If core dimensionality stays low as models scale, mechanistic study of large models can focus on a handful of directions per task rather than on individual weights."],"fun_headline_variants":["Transformers share hidden algorithmic cores across runs","One flippable axis controls grammar in 6 language models","Grokking delay shrinks as cores inflate with weight decay","Markov tasks embed 3D cores that agree on transition spectra","Steering a single axis flips subject-verb agreement"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on ACE's assumption that the SVD of the activation-covariance times output-sensitivity product identifies the true computational subspace rather than merely the readout's linear sensitivity, and the paper does not compare the resulting core against random or alternative one-dimensional subspaces at the same layer, so the causality certificates stand or fall with that identification.","fun_headline_variants_meta":{"raw":{"variants":["Transformers share hidden algorithmic cores across runs","One flippable axis controls grammar in 6 language models","Grokking delay shrinks as cores inflate with weight decay","Markov tasks embed 3D cores that agree on transition spectra","Steering a single axis flips subject-verb agreement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1110,"prompt_tokens":799,"completion_tokens":311,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":228}},"tokens_in":543,"tokens_out":311,"duration_ms":3337,"temperature":1.0,"reasoning_tokens":228,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:39:48.110499+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GPT-2 agreement ablations on the same prompts but with a random one-dimensional projection drawn from the same activation distribution; if random axes show core-only AUC near baseline and core-removed AUC near chance, the core's necessity and sufficiency are artifacts of the projection and readout. Also run the ACE protocol on LLaMA-3.1, Gemma-2, and Qwen2.5 at their late layers: if their agreement axes do not align with GPT-2's, the universal cross-architecture claim fails.","supporting_citations":[],"review_version":1}