{"id":"38f3ffd2-e00b-4076-b430-1cc6f5167e8e","arxiv_id":"2501.18666","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In a list-sorting transformer, the mean and variance of gaps between adjacent sorted numbers determine whether attention heads split the vocabulary, suppress copying to calibrate, or switch off.","lead":"A one-layer transformer trained to sort numbers splits its attention heads into either vocabulary ranges or copy-suppression modes, and the gaps between numbers in the training data decide which mode appears. A generalist reader might care because it is a concrete example of how training data, not just architecture or regularization, shapes the internal organization of a transformer.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The δ-mean/variance causal claim is not isolated: within the fixed list-length/vocab family, variance is confounded with gap support and dataset entropy, and no matched-moment intervention is provided.","rationale":"The paper's central claim is causal: δ mean and variance are said to drive which head specialization emerges. For this to hold, datasets that differ in specialization should differ in δ mean/variance in a way that is not explained by other dataset properties. The paper varies datasets through four different generation procedures, and the authors themselves flag (Sec. 2.1) that equal mean δ does not imply equal distributions. The reader's weakest assumption captures exactly this: no intervention isolates δ statistics from confounds. My stress-test sharpens the point: even the D^{[2,k]} family—which fixes list length and vocabulary—confounds variance with the support/range of the allowed gaps and with the entropy of the resulting list distribution. Because no dataset in Table 2 is matched to another on mean and variance while differing only in higher-order structure, the data cannot distinguish 'variance matters' from 'range, support, or dataset entropy matters.' The proposed test constructs two datasets with identical mean and variance of δ but different gap supports; if specialization differs, the two-parameter causal claim fails. This does not overturn the paper's descriptive findings (the developmental stages and the circuits are carefully documented), and the authors' limitations section already acknowledges the missing interventions, so the CONDITIONAL verdict stands. The condition should explicitly include a matched-moment intervention of the kind proposed.","tokens_in":21656,"tokens_out":13344,"duration_ms":129876,"concrete_test":"Construct two datasets, both with list length 10 and vocabulary 52, using the authors' method 4: one draws δ uniformly from {2,10} (mean 6, variance 16); the other draws δ from {1,6,11} with probabilities 8/25, 9/25, 8/25 (also mean 6, variance 16), discarding lists with max>50 and shifting as in the paper. Train the baseline 2-head model on each with several seeds. If the emergent specializations (1-head sorting / copy-suppression / vocabulary-splitting) differ between the two, then mean and variance do not determine specialization and the paper's causal claim is unsupported; if they match, the two-parameter description survives this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the mean and variance of the gap distribution δ determine whether a 2-head model ends in vocabulary-splitting, copy-suppression, or 1-head sorting—rests on correlational evidence across heterogeneous dataset manipulations. Varying list length, vocabulary size, pruning, or allowed-gap range changes δ statistics together with token frequencies, the number of distinct lists, dataset entropy, the maximum gap, and higher moments of the gap distribution. The paper explicitly concedes in Sec. 2.1 that datasets with the same mean δ are not equivalent and in Sec. 6 that no intervention studies were performed. Even within the cleanest family (D^{[2,k]}, Table 2), where list length and vocabulary are fixed, increasing the upper bound k increases variance jointly with the support/range of δ and with the number of possible lists, so the observed ordering 1-head → copy-suppression → vocabulary-splitting cannot be attributed to variance per se. The load-bearing premise is therefore that the specialization differences are driven by δ mean and variance rather than by dataset shape; this premise is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a one-layer attention-only transformer trained to sort small lists of numbers and characterizes its developmental stages (initial learning, head-overlapping, vocabulary-splitting, copy-suppression) through circuit visualizations, the Local Learning Coefficient (LLC), and circuit rank. It identifies two end-of-training specializations—vocabulary-splitting and copy-suppression—and proposes that the mean and variance of the gap distribution δ between adjacent sorted list elements controls which specialization emerges: larger, more varied gaps favor vocabulary-splitting; smaller, less varied gaps favor copy-suppression; and very low variance leads to one-head sorting. The paper also reports a roughly constant product δ·∇QK across datasets, relates copy-suppression to a mechanism previously identified in GPT-2, and argues that vocabulary-splitting is a naturally preferred simpler solution even without weight decay.","tokens_in":21877,"tokens_out":4139,"duration_ms":38192,"significance":"If the main causal claim held, the paper would be a valuable and concrete demonstration that a simple, measurable property of training data—the gap distribution—shapes circuit-level organization in a transformer, with implications for interpretability and for the universality hypothesis. The study is unusually candid about its limitations (Sec. 6), includes extensive dataset variations and LLC-based complexity comparisons, and provides ablation evidence on copy-suppression. Its strengths include the breadth of controlled architecture variations (1–4 heads, LN/WD ablations) and the mechanistic connection to GPT-2 copy-suppression. However, the central data-to-specialization attribution currently rests on correlational evidence across heterogeneous dataset manipulations, and the quantitative 'constant product' claim is not yet supported by the reported fit.","major_comments":[{"comment":"The central claim that the mean and variance of δ determine which specialization emerges is not supported by the current experimental design. Within the cleanest family D^{[2,k]} (fixed list length and vocabulary), increasing k changes the variance together with the support/range of δ and the number of possible lists; across the other families, list length and vocabulary size also change. The paper explicitly concedes in Sec. 2.1 that datasets with the same mean δ are not equivalent and in Sec. 6 that no intervention studies were performed. The observed ordering 1-head sorting → copy-suppression → vocabulary-splitting with increasing variance can therefore not be attributed to variance per se. I ask for matched-moment or matched-support interventions—for example datasets with identical mean, support, and list/vocabulary cardinality but different variance—or at minimum a falsifiable prediction and additional controls, before the causal claim can be accepted.","section":"Sec. 3.2, Table 2, Fig. 16"},{"comment":"The claim that δ·∇QK is approximately constant is not quantified convincingly. The fitted line in Fig. 6 is δ·∇QK = 0.004δ + 0.468, which has a nonzero slope, and no error bars, per-seed spread, or residuals are shown. Over the displayed range of δ the product varies by a nontrivial amount if the slope is real, so the visual 'flatness' needs to be backed by an uncertainty estimate and a statement of the acceptable tolerance. In addition, the active-region definition relies on visually determined boundaries (Fig. 5, Tab. 1) and the partition is admitted to be somewhat arbitrary; a robustness check of the region-partition rule is needed before this quantitative relationship is treated as established.","section":"Sec. 3.2, Fig. 6"},{"comment":"The specialization labels are assigned by visual inspection of the circuits, and the same figure is used to show that the labels separate cleanly by relative head weight norm. Although the caption states that the labels were made without consulting the weight norms, a visual-only labeling protocol is hard to audit. Please provide a quantitative definition of each specialization class, or an inter-rater/robustness check, so that the reported clean separation in Fig. 16 is not dependent on the authors' subjective judgment. In addition, Table 2 includes an 'Other' class (for D^{ℓ=3}) and the perturbed-dataset model is described as a different specialization mode; these cases need to be reconciled with the three-way variance story or explicitly excluded with justification.","section":"Fig. 16 caption, Sec. 3.2"},{"comment":"The conclusion that copy-suppression acts by increasing model confidence is based on head ablation and a Shannon-entropy measure, but the reported support is thin. The paper states that ablating the copy-suppressing head leaves accuracy unchanged while increasing loss, and that entropy increases after ablating either head; however, the entropy computation is not specified (over which token distribution and context) and no magnitudes are given. Since the functional-role claim is a main contribution, please specify the entropy measure and report the actual loss and entropy changes, including variance across seeds, so the 'slightly increases confidence' conclusion can be evaluated.","section":"Sec. 4.3, Fig. 10"}],"minor_comments":[{"comment":"The symbol δ is used both for the per-list gap values and for the dataset mean, which is confusing; please introduce a different notation for the random variable (e.g., Δ) and the mean (e.g., δ̄) throughout.","section":"Sec. 2.1, Eq. (1)"},{"comment":"There is a typo: 'modular arithmetric' should be 'modular arithmetic'.","section":"Sec. 5"},{"comment":"The conclusion states that the toy model has 100% accuracy, but earlier appendices report a no-LN model achieving only 20% accuracy; please qualify the statement to refer to the specific baseline models and not to all trained models.","section":"Sec. 7"},{"comment":"The histograms in Fig. 4 would be easier to read with explicit bin widths and axis labels for the probability mass; currently the y-axis labeled 'Count' is on a log scale without stating the binning.","section":"Fig. 4"},{"comment":"The stage boundaries are described as chosen by eye from a combination of measures (Sec. 4.1). Please state whether the reported stage transitions, especially the vocabulary-splitting to copy-suppression transition in Fig. 3, are robust to an automated rule such as local extrema of the LLC or loss.","section":"Sec. 3.1"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the paper's empirical scope and to the authors' unusual candor about limitations. For a journal, however, the central causal claim needs either matched-moment interventions or substantially more controls; the current evidence is correlational across deliberately heterogeneous dataset manipulations. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. The paper may also be a better fit for a venue that explicitly welcomes exploratory empirical interpretability studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a careful, honest empirical study of a toy sorting transformer, and the main correlation it reports—between the gap statistics of the training data and the head specialization that emerges—is real and worth knowing about. The causal framing, however, is stronger than the evidence supports, and a good referee should push on that.\n\nWhat's new: a minimal instance of copy-suppression in a single-layer transformer, an LLC-based comparison showing vocabulary-splitting is a simpler state than overlapping heads, and a systematic map of how dataset variations move the final solution between vocabulary-splitting, copy-suppression, and 1-head sorting. The ablation and entropy experiments give a clean, reproducible result: the copy-suppressing head reduces loss by sharpening the logit distribution, not by fixing accuracy. The paper is also candid—it explicitly lists the lack of intervention studies, the limited seed checks, and the approximate nature of the LLC calculations.\n\nThe soft spots are proportionate. The biggest one is the gap-statistics claim. The paper varies list length, vocabulary size, pruning, and allowed gap ranges, and those manipulations change the mean and variance of δ, but they also change the support of the δ distribution, the number of distinct lists, and dataset entropy. Even in the cleanest family D^{[2,k]}, increasing k raises variance together with the allowed range and the space of possible lists. The paper itself concedes that datasets with the same mean δ are not equivalent. So the claim that mean and variance drive specialization is a correlation, not a demonstrated cause. The authors say this too, in Section 6, but the abstract and conclusion lean harder on the causal reading than the experiments justify. The constant δ·∇QK fit is also presented as flat, but the fitted line has a nonzero slope and no error bars; it's a mild trend, not a crisp invariant. Finally, specialization labels are set by visual inspection; the clean separation by weight norms in Fig. 16 is reassuring, but reporting the labeling rule explicitly would help.\n\nFor a mechanistically oriented reader, the paper is still worth reading. It extends a known toy model in a concrete direction and does the experiments carefully enough to be credible on the qualitative level. It deserves a serious referee, not a desk reject. My recommendation: send it out, but ask for (1) either softened causal language or matched-moment interventions—e.g., reweight datasets to fix mean and variance independently, or at least vary the support while holding moments fixed; (2) error bars on LLC and the δ·∇QK fit; (3) code and data release. Those are reasonable revisions, not a rewrite.","headline":"Solid, honest toy-model study showing gap statistics correlate with head specialization; the causal claim outstrips the data, but it deserves referee time.","tokens_in":22374,"tokens_out":2895,"would_cite":true,"duration_ms":26529,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dataset statistic—the mean and variance of gaps between adjacent sorted numbers—predicts how a sorting transformer organizes its attention heads.","keywords":["mechanistic interpretability","developmental interpretability","transformer circuits","list sorting","vocabulary-splitting","copy-suppression","local learning coefficient","training data distribution"],"falsifier":"Generate two datasets with identical mean and variance of $\\delta$ but different generation mechanisms, for example fixed allowed $\\delta$ values versus pruned lists, and train identical models on each; if the final specialization mode tracks the generation mechanism rather than the $\\delta$ statistics, the central claim is false. The paper also names the missing experiment: directly increasing or decreasing the QK gradient during training should move the model between the predicted specialization modes, and a null result would falsify the attribution.","tokens_in":21444,"feed_emoji":"🔢","tokens_out":9475,"duration_ms":78932,"temperature":0.7,"pith_summary":"This paper asks what in the training data makes a transformer organize its attention heads the way it does. Training a one-layer, attention-only transformer to sort lists of numbers, the authors identify three final circuit organizations: vocabulary-splitting, where heads take over disjoint ranges of the number vocabulary; copy-suppression, where one head copies tokens and the other fine-tunes that copy; and one-head sorting, where the second head is effectively switched off. The paper's central claim is that the deciding data feature is the distribution of gaps $\\delta$ between adjacent numbers in the sorted lists: larger, more varied gaps favor vocabulary-splitting, smaller and less varied gaps favor copy-suppression, and very low variance turns the second head off. It also reports that vocabulary-splitting scores as a simpler solution on the Local Learning Coefficient and emerges even without weight decay, which it reads as evidence for an inductive bias toward simpler solutions. If correct, this gives a concrete, measurable way in which training-data statistics sculpt the internal circuits of transformers, a step toward understanding larger language models.","feed_headline":"List gaps dictate how sorting transformers wire their heads","feed_subtitle":"Wider, varied gaps split vocabulary; tight gaps trigger copy-suppression; near-constant gaps switch a head off.","key_machinery":"The load-bearing object is the gap statistic $\\delta = l_{i+1} - l_i$, the difference between adjacent elements of a sorted list, summarized across the dataset by its mean and variance. The paper's argument runs through two interpretability tools: the OV/QK circuit decomposition, which represents each attention head as a copying circuit (diagonal of the OV matrix) and an attention-selection circuit (QK matrix), and the Local Learning Coefficient (LLC), a singular-learning-theory measure of solution complexity estimated from the local loss landscape. The mechanism is that smaller typical gaps force the QK circuit to distinguish nearby tokens more sharply, so the mean QK gradient inside each active region rises; with a roughly constant product $\\delta$ times the gradient, region sizes shrink and the model splits its vocabulary differently, while very small gap variance removes the need for the second head entirely.","core_discovery":"On its own terms, the paper establishes a developmental sequence for a one-layer, two-head transformer learning to sort lists: rapid initial learning, a stage of overlapping head roles, then head specialization. The final specialization comes in three modes characterized by the OV and QK circuits: vocabulary-splitting (contiguous non-overlapping diagonal regions across heads), copy-suppression (a negative OV diagonal in one head that calibrates the copying head), and one-head sorting (the sub-leading head's circuits become orders of magnitude smaller). The paper claims these modes are selected by the mean and variance of the gap distribution $\\delta$ between neighboring elements of sorted lists, and supports this with a sweep of datasets varying list length, vocabulary size, and sampling procedure. It further claims that vocabulary-splitting is a genuinely simpler solution, with lower LLC than the overlapping stage, and that it arises without weight decay, while copy-suppression is an intermediate regime that increases model confidence when the copying head is already accurate. The result is presented as a concrete demonstration that a measurable feature of training data determines the internal organization of a transformer.","pith_inferences":["A direct test the paper leaves open is whether the same $\\delta$ mean-and-variance rule holds in deeper transformer stacks; if it does, gap statistics could become a diagnostic for circuit regimes in real language-model data.","The observed ordering from one-head sorting to copy-suppression to vocabulary-splitting with increasing gap variance suggests a phase boundary that could be mapped precisely with a variance sweep at fixed mean $\\delta$.","Because the paper's datasets vary list length and vocabulary size alongside $\\delta$, an intervention that manipulates only the gap distribution, for example by re-weighting gaps within the same vocabulary and length, would isolate the causal direction.","If the LLC-based simplicity ordering is robust, training curricula that begin with compact gaps and then widen them could deliberately steer a model through copy-suppression into vocabulary-splitting."],"forward_implications":["If the central claim is right, the mean and variance of $\\delta$ become a practical predictor: before full training, one can estimate from the dataset which of the three head specializations a two-head sorting transformer will settle into.","Vocabulary-splitting forming without weight decay implies that simpler solutions can be a natural training outcome rather than an artifact of regularization.","Copy-suppression as a confidence-calibrating mechanism in a high-accuracy setting means the same circuit motif can have opposite effects on confidence depending on task accuracy.","The rough constancy of $\\delta$ times the QK gradient across datasets suggests a simple scaling law for how fine-grained the attention circuit must be to sort a given gap distribution.","The link between data statistics and circuit organization supports the universality hypothesis, suggesting that similar data-driven specialization could be searched for in larger transformer models."],"supporting_citations":[{"why":"Defines the list-sorting transformer setup that this paper trains and varies.","marker":"McDougall (2023a)"},{"why":"Provides the original mechanistic reading of the sorting solution in OV/QK circuits and the vocabulary-splitting observation.","marker":"McDougall (2023b)"},{"why":"Supplies the OV/QK circuit decomposition used to inspect attention head behavior.","marker":"Elhage et al. (2021)"},{"why":"Identifies copy-suppression in a large language model; this paper reproduces and refines that mechanism.","marker":"McDougall et al. (2023)"},{"why":"Defines the Local Learning Coefficient that the paper uses to measure solution complexity.","marker":"Lau et al. (2023)"},{"why":"Establishes the developmental-stage methodology with LLC drops that this paper applies to the sorting model.","marker":"Hoogland et al. (2024)"},{"why":"Provides the foundational singular learning theory behind the LLC estimates.","marker":"Watanabe (2009)"},{"why":"Defines the attention mechanism and transformer architecture used in all experiments.","marker":"Vaswani et al. (2017)"}],"fun_headline_variants":["Gap stats decide how sorting transformers specialize","List gaps select transformer head modes","Copy-suppression emerges from tight list gaps","Sorting gap distribution steers internal circuits","Developmental path set by gap variance in sorting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the assumption that the observed specializations are caused by the mean and variance of the gap distribution $\\delta$, not by other dataset properties that change at the same time (list length, vocabulary size, or distribution shape); the paper itself notes that datasets with the same mean $\\delta$ are not equivalent and that it performed no intervention studies.","fun_headline_variants_meta":{"raw":{"variants":["Gap stats decide how sorting transformers specialize","List gaps select transformer head modes","Copy-suppression emerges from tight list gaps","Sorting gap distribution steers internal circuits","Developmental path set by gap variance in sorting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1489,"prompt_tokens":915,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":523}},"tokens_in":531,"tokens_out":574,"duration_ms":5955,"temperature":1.0,"reasoning_tokens":523,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:26:52.354429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate two datasets with identical mean and variance of $\\delta$ but different generation mechanisms, for example fixed allowed $\\delta$ values versus pruned lists, and train identical models on each; if the final specialization mode tracks the generation mechanism rather than the $\\delta$ statistics, the central claim is false. The paper also names the missing experiment: directly increasing or decreasing the QK gradient during training should move the model between the predicted specialization modes, and a null result would falsify the attribution.","supporting_citations":[],"review_version":1}