{"id":"363cef7d-7607-4167-93c9-5cab587d3eaf","arxiv_id":"2501.12391","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper introduces Geometry, Resource, and Domino models that reproduce the sequential Domino effect in skill learning and link it to scaling laws, optimizers, and modularity.","lead":"This paper proposes three increasingly simple toy models to explain why neural networks often learn skills one at a time, with a new skill starting only after an earlier one finishes. The models are then used to connect skill learning to neural scaling laws, optimizer behavior, and potential speedups from modular networks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Resource model's allocation rule is fitted, not measured; a parameter-free conserved-quantity test on the paper's own real MLP would settle whether Eq. (1) explains the Domino effect.","rationale":"The reader's weakest assumption is the coarse-graining ansatz: independent linear skill directions with additive losses. My concern is a direct, testable consequence of that same gap: even granting the ansatz, the Resource model's central ODE is proposed and calibrated rather than measured, and its distinctive quantitative predictions are verified only on the toy models. The conservation law u_i^{1/p_i} = C is a sharp, N0-independent prediction of Eq. (1), and it can be checked on the paper's own real regression experiment from Section 6.3. This makes the concern concrete rather than a general appeal to realism, and it routes through correctness risk rather than disagreement with the field's consensus. The paper is honest about its limitations, including the missing data-to-skill recipe, which is why this is not a rejection: the framework is coherent and the proposed test could validate it. I therefore keep the reader's CONDITIONAL verdict, with the condition being the empirical test described above. Agreement is partial because the reader emphasizes the linear-representation premise, while I focus on the untested allocation rule that would still need verification even if linear representations were granted.","tokens_in":28278,"tokens_out":20986,"duration_ms":217997,"concrete_test":"Use the released code to retrain the two-output MLP from Section 6.3 (inputs x, y; targets x^2, y^2; y nonzero 1% of the time). At every checkpoint, record the per-task MSE losses l1 and l2 and form u_i = sqrt(l_i). Test the parameter-free collapse prediction of Eq. (1): u1^{1/p1} = u2^{1/p2} with p1 = 1 and p2 = 0.01 (equivalently l1^{1/2} = l2^{50}). If the two curves diverge beyond a small tolerance, then real per-task learning does not follow Eq. (1), and the resource-competition explanation of the Domino effect is not supported; if they collapse, the central mechanism gains direct empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the Domino effect is explained by resource competition via Eq. (1), du_i/dt = -eta_eff p_i u_i / (sum_j p_j u_j + N0). But Eq. (1) is asserted and then calibrated: the factor eta_eff and the only free parameter N0 are obtained by synchronizing to the Geometry model (Section 2.4), not by measuring resource allocation in a real network. The Geometry model itself rests on the linear, independent-skill ansatz of Section 2.1, which the paper's Limitations admit has no data-to-skill recipe. The paper's direct validations of Eq. (1) -- conserved quantities and learning-time scaling -- are carried out on the Resource model and, approximately, on the Geometry model (Appendix A, Figure 21), not on real networks. The motivating real observation is a two-task MLP trained with Adam (Section 1), whereas the Geometry model produces a robust Domino effect only with SignGD (Figure 4 caption); this makes it unclear that the toy mechanism transfers. What is missing is a falsifiable measurement of whether per-task updates in a real network obey the allocation rule in Eq. (1). The conserved quantity u_i^{1/p_i} = C is ideal for this: it is exact for Eq. (1), independent of N0, and directly checkable from per-task losses in the paper's own regression experiments. Without such a check, 'resource competition' is a plausible analogy rather than an established explanation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper begins from an observed 'Domino effect' in skill learning—skills appear to be acquired sequentially, often one completing just before the next starts—and proposes three increasingly abstract models to explain it. The Geometry model represents skills as linear directions in parameter space with a weighted sum of skill losses; the Resource model abstracts the dynamics into a single ODE for 'unskill' levels competing for a shared resource pool; the Domino model takes a strong-hierarchy limit in which skills are learned one by one. From these models the paper derives conserved quantities, learning-time scalings, and scaling-law exponents, and it connects the framework to neural scaling laws, optimizer behavior, compositional task structure, and modularity. The paper also reports real-network experiments on sparse parity, grokking, token reweighting, and modular MLPs.","tokens_in":28724,"tokens_out":3794,"duration_ms":41779,"significance":"If the central claims are accepted, the paper offers a useful coarse-grained language for thinking about skill learning dynamics, with clean analytical results: an exact conserved quantity for the Resource model, a closed-form learning-time expression, and a modularity speedup from O(ntask) to O(sqrt(ntask)). The paper is also commendable for shipping code, for making its toy models explicit, and for generating several falsifiable qualitative predictions about optimizers and data reweighting. However, the empirical validation is significantly weaker than the theoretical development. The key dynamical assumption—Eq. (1)—is proposed rather than derived or measured, and the central conserved-quantity prediction is tested only on the abstract models themselves. The paper's own Limitations section explicitly concedes that no recipe is given for mapping concrete data into 'skills', which limits the reach of the claimed explanations. As a toolbox of analogies, the paper is stimulating; as an established explanation of real skill-learning phenomena, it needs substantial additional empirical support.","major_comments":[{"comment":"The central resource-allocation rule du_i/dt = -eta_eff p_i u_i / (sum_j p_j u_j + N0) is asserted as a model rather than derived from the Geometry model or measured from real networks. The only free parameter N0 is tuned to reproduce the Geometry model (Figure 6), and the subsequent conserved-quantity test (Appendix A, Figure 21) is carried out on the Resource model and, approximately, on the Geometry model—not on real MLPs. The paper's motivating observation is a real two-task MLP trained with Adam (Section 1), yet the Geometry model exhibits the robust Domino effect mainly with SignGD (Figure 4 caption). The explanation of the real Domino effect therefore rests on an unmeasured analogy. I suggest adding a direct test: compute per-task unskill curves from the paper's own sparse-parity or two-output regression experiments and check whether u_i^{1/p_i} collapses across tasks; if it does not, the claim that resource competition explains the Domino effect should be correspondingly qualified.","section":"Section 2.4, Eq. (1)"},{"comment":"The compositional-task model is validated by fitting two ingredients to the target experiment: the soft-AND exponent gamma = 0.01 is hand-chosen, and in the independent-task ablation p3 is changed from 0.0045 to 0.08 after 2000 steps to match the observed speedup. This makes the agreement post hoc rather than predictive. I recommend reporting all fitted parameters in one table, adding a sensitivity analysis over gamma and the switching time, and ideally testing the model on a held-out task-dependence graph where no parameter is adjusted after seeing the data.","section":"Section 5.1, Eqs. (10)-(11)"},{"comment":"The paper explicitly states that it provides no recipe for mapping concrete data/tasks into 'skills' and assumes each data point belongs to exactly one skill. This limitation is load-bearing because every quantitative comparison to real tasks—sparse parity, compositional parities, the modular MLP—requires an implicit skill decomposition that the models themselves do not supply. Without such a mapping, the Geometry model's predictions cannot be applied to language data, so the Chinchilla-scaling match and the optimizer insights remain analogies rather than demonstrated mechanisms. The authors should either provide a concrete skill-extraction procedure for at least one real setting or restrict the paper's claims to the abstract models and present the real experiments as illustrative rather than as validation.","section":"Section 8, Limitations"},{"comment":"The claim that the Geometry model reproduces Chinchilla-like exponents (alpha_N ~ 0.34 and alpha_S ~ 0.28-0.37 at alpha = 1) is based on power-law fits over a finite range of ndim (roughly ndim < 250) in a model that also exhibits a critical point near ndim ~ 600. Power-law exponents extracted from short finite-size ranges can be strongly influenced by the fitting window and by proximity to the critical point. I ask the authors to report confidence intervals on the fitted exponents, to show the sensitivity of the exponents to the fitting range, and to test whether the apparent alpha dependence is stable when the fitting range is varied.","section":"Section 3.2, Figure 10"}],"minor_comments":[{"comment":"In the Quanta-model discussion, the sentence 'at step S proportional to n^{-alpha}, n tasks are learned' appears to have a sign error; given t_n proportional to n^alpha, the correct statement is S proportional to n^alpha, leading to the stated exponent (alpha-1)/alpha.","section":"Section 3.1"},{"comment":"The text contains a typo: 'thrid case' should read 'third case'.","section":"Section 5.2"},{"comment":"The caption says 'meachanistic model'; this should be 'mechanistic model'.","section":"Section 6.2, Figure 18 caption"},{"comment":"The figure caption appears to be incomplete: 'Figure 20: Instead of having a single model, it is better to have a spectrum of models...' is followed by an empty placeholder. Please replace it with the actual figure or remove the placeholder.","section":"Figure 20"},{"comment":"The horizontal axis of the right panel is labeled 'Focus on hard examples for how many steps', but the tick labels are not shown; please add numeric tick labels and state whether the axis is in steps or thousands of steps.","section":"Figure 14"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad, ambitious position piece with several internally consistent toy models. Its main weakness is not the algebra but the gap between the abstract models and the empirical claims: the key parameters N0 and p3 are fit to the very phenomena the paper claims to explain, and the paper admits it lacks a data-to-skill mapping. I believe this can be repaired by adding direct empirical tests of the conserved quantity and by explicitly reframing the real-network experiments as illustrations rather than validations. The paper may be a good fit for the journal if the authors accept this reframing and substantially strengthen the empirical section. There is also some novelty overlap with the authors' own Quanta model, but the Resource and Domino models add enough new structure that this is not disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ziming's paper is best read as conceptual scaffolding, not as an empirical claim about real networks. What's genuinely new: the Domino-effect observation on sparse parity, the Resource ODE with its conserved quantity u_i^{1/p_i}=C, the Domino model's scaling analysis, and the modularity speedup from O(n_task) to O(sqrt(n_task)). The derivations are clean and the conserved quantity is exact for Eq. (1); I checked the algebra, it holds. The paper also ships code and runs real experiments on sparse parity, GPT-2 reweighting, and modular MLPs, which is more than most theory papers do.\n\nThe soft spots are in the empirical validation. Eq. (1) is asserted, not derived from a real network, and its only free parameter N0 is tuned to match the Geometry model. The conserved quantity is verified on the Resource model and only approximately on the Geometry model (Appendix A), but never on a real network. In Section 5.1, the compositional-parity experiment changes p3 mid-training to match the data. And the Chinchilla exponent match in Figure 10 rests on a single no-error-bar simulation. These are genuine weaknesses, but they are proportionate: the paper frames these as toy models and its Limitations section already concedes there is no data-to-skill recipe. The stress-test suggestion is fair and cheap: compute per-task unskills in the Section 6.3 MLP regression and check whether u_i^{1/p_i} collapses. That would tell us whether Eq. (1) is a description or just an analogy.\n\nThe citation pattern looks fine; the Quanta model is discussed and extended honestly, and prior-work dependencies are clear. My main criticism is that the paper's reach exceeds its empirical grip in a few places—claims like 'we show how Chinchilla scaling laws can emerge' do a lot of work for a toy-model simulation. But that is fixable with hedging and one direct test.\n\nI'd send this to peer review. The right referee will push on the conserved-quantity measurement and on whether the Domino effect transfers beyond SignGD. The paper deserves that push; it will be better for it.","headline":"A conceptually rich toy-model paper whose analytic core (Resource ODE, conserved quantities, Domino scaling) is clean and new, but whose empirical validations lean on fitted parameters; worth refereeing with a push for a direct conserved-quantity test on real networks.","tokens_in":29174,"tokens_out":2114,"would_cite":true,"duration_ms":22080,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that skill acquisition in neural networks is sequential, driven by competition for a shared pool of parameters, and can be captured by a single differential equation.","keywords":["skill learning dynamics","Domino effect","resource competition","neural scaling laws","modularity","adaptive optimizers","coarse-grained models","task compositionality"],"falsifier":"Train a two-task network with SignGD at frequency ratios $p_1/p_2$ of 10, 100, and 1000 and measure the onset of skill 2: the resource model predicts the learning-time ratio $t_2/t_1$ saturates near 2 as the ratio grows, whereas a naive gradient-magnitude account predicts $t_2/t_1 \\propto p_1/p_2$. If skill 2 improves substantially before skill 1 saturates, the resource-competition mechanism is wrong.","tokens_in":28067,"feed_emoji":"🎲","tokens_out":7690,"duration_ms":73701,"temperature":0.7,"pith_summary":"This paper argues that the order in which a neural network learns different skills is not arbitrary: skills compete for a shared pool of model parameters, and a more frequent skill can block rarer ones until it finishes learning. The authors reproduce this 'Domino effect' in a deliberately simplified Geometry model, then distill it into a Resource model governed by one ordinary differential equation, and finally into a Domino model in which skills are mastered strictly one after another. If the picture is right, it explains why neural scaling exponents take the values they do, why adaptive optimizers change learning order, and why modular networks can cut total training time from linear to square-root in the number of tasks. The paper's stated goal is a physics-style abstraction that lets researchers reason about skill dynamics without tracking every parameter.","feed_headline":"Skills fall like dominoes, and one equation predicts their order","feed_subtitle":"A three-level model links sequential skill acquisition to scaling laws, optimizer choice, and faster modular networks.","key_machinery":"The load-bearing object is the Resource-model ODE, Eq. (1), along with its conserved quantities. The model coarse-grains a network into unskill levels $u_i$ for each task; each task's gradient strength is $p_i u_i$, and its share of training resources is that strength divided by the total over all tasks plus a waste term $N_0$. The conserved identity $u_i^{1/p_i} = C$ is what makes the model solvable: it collapses all skill learning curves onto one master curve and turns the learning-time calculation into a single integral. The Geometry model supplies the microscopic justification for this resource picture, and the Domino model is its strong-hierarchy limit.","core_discovery":"On its own terms, the paper claims that the Domino effect is driven by resource competition. In the overparameterized Geometry model, each skill is a direction $t_i$ in parameter space, the loss is $\\ell = \\sum_i p_i L(s_i)$ with $s_i = (\\theta-\\theta_0)\\cdot t_i$, and training under SignGD reveals that the number of gradient-aligned dimensions assigned to a skill behaves like a conserved resource. This motivates the Resource model, where the unskill level $u_i = 1-s_i$ obeys $$\\frac{du_i}{dt} = -\\eta_{\\text{eff}} \\frac{p_i u_i}{\\sum_j p_j u_j + N_0},$$ with $N_0$ the only phenomenological parameter. The ODE has $n_{\\text{task}}-1$ conserved quantities $u_i^{1/p_i} = C$, so learning curves collapse onto one another, and total learning time grows linearly in $n_{\\text{task}}$. When tasks are strongly hierarchical, the equations reduce to a Domino model in which each skill waits for the previous one. The paper's central discovery is that many observed phenomena—scaling-law exponents, optimizer differences, the benefit of modularity, even improved language-model training—follow from this single resource-competition mechanism.","pith_inferences":["If the conserved relation $u_i^{1/p_i}=C$ survives contact with real data, per-token loss curves become a measurement tool: fitting $C$ and the exponents $p_i$ would let one infer skill frequencies directly from training logs, an inverse problem the paper names but does not solve.","A curriculum prediction follows from the same mechanism: schedules that present tasks in decreasing frequency should coincide with the network's natural order and need no reweighting, while a curriculum that fronts rare skills should sharply slow frequent-skill learning—an experiment the paper does not run.","The modularity speedup, if it transfers to large models, suggests mixture-of-experts architectures should show their biggest training-time advantage on heavily imbalanced, heavy-tailed task mixes rather than on uniform tasks.","Because the Geometry model keeps the optimizer as a free knob, it can serve as a low-cost ranking testbed for new optimizers before large-scale pretraining, a use the paper explicitly advocates with its AdEMAMix and Lion case studies."],"forward_implications":["Under the Resource model, total training time for a monolithic network scales linearly with the number of tasks, and the Domino model predicts the same linear scaling.","Modular networks, where each skill owns its own parameters, change the scaling to $\\sqrt{n_{\\text{task}}}$, so modularity is predicted to speed up training on imbalanced task sets; experiments on a sparse regression task confirm the effect.","The Geometry model produces neural scaling exponents near the Chinchilla values ($\\alpha_N \\approx 0.34$, $\\alpha_S \\approx 0.28$ to $0.37$) when the task-frequency distribution is Zipfian with $\\alpha=1$, which the simpler Quanta and Domino models miss.","Loss-based data reweighting that emphasizes high-loss tokens speeds up GPT-2 training when applied early, consistent with the resource-competition view of rare skills.","Optimizer choice changes the effective waste parameter $N_0$ and therefore changes scaling exponents, implying that scaling-law analyses should report and control optimizer hyperparameters."],"supporting_citations":[{"why":"It supplies the power-law skill-frequency prior, the quanta model, and the multitask sparse-parity setup that the paper's scaling analysis builds on and compares against.","marker":"[3]"},{"why":"It provides the related skill-level abstraction of language models that the authors position their Geometry model as a higher-level, optimizer-agnostic alternative to.","marker":"[1]"},{"why":"It supports the linear-representation assumption used by the Geometry model, via task arithmetic in tangent space.","marker":"[5]"},{"why":"It provides the Chinchilla scaling-law exponents $\\alpha_N \\approx 0.34$ and $\\alpha_S \\approx 0.28$ used as the empirical benchmark.","marker":"[9]"},{"why":"It supplies the replication fit $\\alpha_S \\approx 0.37$ used to compare the Domino model's scaling prediction.","marker":"[10]"},{"why":"It defines the grokking benchmark on which the paper tests whether SignGD's geometry insights transfer to real networks.","marker":"[11]"},{"why":"It establishes that learning $k$-parity becomes exponentially slow with $k$, which motivates the compositional-dependency experiments on sparse parity.","marker":"[21]"},{"why":"It introduces AdEMAMix, whose behavior and failure modes the Geometry model is used to explain.","marker":"[15]"}],"fun_headline_variants":["Skill order emerges from resource competition in nets","One resource model predicts scaling laws and skill order","Domino effect in neural learning: skills fall sequentially","Physics of skill learning: one conserved resource"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the coarse-graining ansatz: that there is a compressor mapping parameters to skill levels, that total loss is a weighted sum of independent skill losses, and that each skill is a fixed linear direction $s_i = (\\theta-\\theta_0)\\cdot t_i$ in parameter space; the paper explicitly concedes it gives no recipe for mapping concrete data to skills.","fun_headline_variants_meta":{"raw":{"variants":["Skill order emerges from resource competition in nets","One resource model predicts scaling laws and skill order","Domino effect in neural learning: skills fall sequentially","Physics of skill learning: one conserved resource"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000828,"raw_usage":{"total_tokens":3667,"prompt_tokens":1043,"completion_tokens":2624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":2566}},"tokens_in":659,"tokens_out":2624,"duration_ms":18159,"temperature":1.0,"reasoning_tokens":2566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:13:22.307126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a two-task network with SignGD at frequency ratios $p_1/p_2$ of 10, 100, and 1000 and measure the onset of skill 2: the resource model predicts the learning-time ratio $t_2/t_1$ saturates near 2 as the ratio grows, whereas a naive gradient-magnitude account predicts $t_2/t_1 \\propto p_1/p_2$. If skill 2 improves substantially before skill 1 saturates, the resource-competition mechanism is wrong.","supporting_citations":[{"cited_title":"The quantization model of neural scaling","cited_arxiv_id":null,"evidence_quote":"It supplies the power-law skill-frequency prior, the quanta model, and the multitask sparse-parity setup that the paper's scaling analysis builds on and compares against."},{"cited_title":"Task arithmetic in the tangent space: Improved editing of pre-trained models","cited_arxiv_id":null,"evidence_quote":"It supports the linear-representation assumption used by the Geometry model, via task arithmetic in tangent space."},{"cited_title":"Failures of gradient-based deep learning","cited_arxiv_id":null,"evidence_quote":"It establishes that learning $k$-parity becomes exponentially slow with $k$, which motivates the compositional-dependency experiments on sparse parity."}],"review_version":1}