{"id":"14872167-4dec-467d-87fe-084c420f0f8d","arxiv_id":"2608.07110","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Modular TTT expresses test-time training as a graph of primitives, ablates the components, and finds that simple shallow learners with small learning-rate initialization and scalar decay match Gated DeltaNet at 1.45B scale.","lead":"The authors build a modular framework that turns test-time training (TTT) into composable graph components, and use it to ablate which design choices matter. The framework makes it easier to design and compare TTT sequence models, and the ablations point to simple linear learners with small learning rates and decay as the practical sweet spot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deep-memory finding is init-scale-sensitive: every deep run fixes both factors at the official std-0.02 Gaussian init, yet the paper's own Eqs.","rationale":"I read the central claim as two-part: (1) a compositional framework that expresses one-step dual-form TTT as a DAG of primitives and automatically composes the train-view forward, train-view backward, and causal query-view passes, and (2) an empirical map of that design space concluding that small-lr init, scalar decay, and a shallow SiLU-activated linear learner are best, while depth and normalization hurt. Part (1) is well supported: for chain graphs the composed rules reduce to the known TTT dual forms (Appendix A.4), the Tril-based chunkwise causality and the vector-decay state transition (Appendix A.5) are internally consistent, and the released code is concrete evidence. The reader's generality concern about Part (1) is valid but partly conceded in Appendix E (multi-step inner loops, momentum, and Muon-style normalization require specialized paths), so it mainly argues for a scope-calibrated rewrite of 'unified framework' rather than for a correction of any empirical result; that is why I rate my agreement as partial rather than full. My own stress-test locates the least-secure load-bearing point in Part (2): the negative deep-memory finding. The paper's own Eqs. (13)-(17) show the induced deep update is governed by the factor scale c and the random Wishart preconditioners W(1)W(1)^T and W(2)^T W(2), yet the deep-memory sweeps never vary that scale; under the default init with d_head = 128 the effective product fast weight starts about 4-5x smaller than the shallow init, so 'deep hurts' is entangled with an untested init-scale confound. The Abstract's attribution of the deep failure to 'excessively large activations' also contradicts the section's factor-coupling mechanism, and the pure-Linear-Linear row shows degradation without any activation amplification. This is load-bearing because the shallow-linear recipe is justified mainly by this negative result; if an orthogonal-balanced init with compensated inner LR closes the gap, the recipe survives only as a simplicity/efficiency choice, not as evidence that depth is intrinsically harmful. The reader's other points stand: the Abstract's 'normalization tends to hurt' conflicts with Table 5 (Norm improves at 160M under both losses and at 410M under inner-product), and several central tables lack error bars, though Appendix Table 17's five-seed runs do support the positive SiLU effect (0.010-0.018 with std 0.002-0.006). The proposed experiment settles the deep-memory question cleanly; since the framework claim and the positive findings remain supported either way, I keep the reader's CONDITIONAL verdict (UNCHANGED).","tokens_in":28830,"tokens_out":49111,"duration_ms":422539,"concrete_test":"Re-run the Table 6 deep-memory comparison at 160M with orthogonal-balanced fast factors: initialize W(1) and W(2) as independent random orthogonal matrices (d_head x d_head), so W(1)W(1)^T = W(2)^T W(2) = I and Eq. (14) predicts an induced update W - 2Δ + O(Δ^2), i.e., the shallow update at double the inner LR; therefore also run the deep variant at half the small-lr init (eta0 = 5e-4) plus the matched shallow controls (Linear at eta0 = 1e-3 and at 2e-3). Keep all other settings identical to Table 6 (MSE, scalar decay, chunk 256, 10B tokens, same seeds) and compare against the Table 6 entries (Linear-Linear 3.1265, Linear-SiLU-Linear-SiLU 3.1156, Linear-SiLU 3.0205). If the orthogonal deep variant tracks the shallow control and closes the gap, the 'deeper fast-weight networks hurt' claim is an init-scale artifact; if the gap persists, the finding is structural.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical package claims that deeper fast-weight networks hurt performance (Abstract; Section 4.1, Table 6). The paper's own explanation (Section 4.1, Eqs. (13)-(17)) shows that for a two-factor memory Y = XW(1)W(2) representing the same effective weight W = W(1)W(2), the induced one-step update on W is W - ΔW(2)^T W(2) - W(1)W(1)^T Δ + ΔW(2)^T W(1)^T Δ, and that the global rescaling W(1) -> cW(1), W(2) -> c^{-1}W(2) changes this update, with extreme c dominated by -c^2 W(1)W(1)^T Δ or -c^{-2} ΔW(2)^T W(2). The theory thus predicts that deep-variant behavior is controlled by the factor scale and the Wishart preconditioners W(1)W(1)^T and W(2)^T W(2). Yet every deep-memory experiment (Tables 6, 21-27; Appendix D.4) initializes both fast factors with the official Gaussian std-0.02 init (Appendix B.1); with d_head = 128 (Table 11), two such factors give an effective product W(1)W(2) with entry std about √128·0.02^2 ≈ 4.5e-3, roughly 4-5x smaller than the shallow init's 0.02, so the deep net starts with an under-scaled fast weight and a damped effective inner learning rate. The stabilization sweeps vary mean scaling, RMSNorm epsilon, chunk size, activation placement, and zero-vs-Gaussian init, but never the factor scale or balance that Eqs. (13)-(17) identify as the controlling variable. The reported gaps (up to +0.18 validation loss versus the Linear-SiLU reference of 3.0205 in Table 21) may therefore be an artifact of the default init scale rather than a structural property of deeper fast learners. Relatedly, the Abstract attributes the deep failure to 'excessively large activations,' but Section 4.1's own mechanism is factor-coupling and scale-symmetry; for the pure-Linear-Linear row of Table 6 that activation-size story does not even apply.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Modular TTT, a framework that represents test-time-training inner learners as DAGs of primitive operators (Linear, Gate, Norm, Act, Add, Mul) and automatically composes train-view forward, train-view backward, and causal query-view rules into a chunkwise TTT computation with fast-weight state transitions. The authors report controlled ablations at 160M and 410M scale over loss function, learning-rate initialization, decay, nonlinearity, depth, and normalization, finding that small-lr initialization, scalar decay, and a single SiLU activation are beneficial, MSE and inner-product losses are comparable, deeper fast-weight networks and normalization are fragile, and residual/gated structure gives little benefit. A selected shallow variant is scaled to 410M and 1.45B parameters for 100B tokens and is reported as comparable to Gated DeltaNet on training loss and multiple-choice benchmarks, while weaker on containment and long-context retrieval. The paper also reports 2.2-3.3x training-throughput gains over the official TTT implementation.","tokens_in":29186,"tokens_out":10960,"duration_ms":101960,"significance":"If the results hold, the primitive grammar is a genuinely useful engineering contribution: it removes the need for per-variant derivation of global TTT updates within a broad one-step dual-form family, and the ablations provide one of the most systematic component-level pictures of TTT to date. The released code, the five-seed paired runs for the key learning-rate/decay/activation comparisons (Appendix Table 17), and the explicit stability propositions are strengths. The scale-up comparison with GDN supports the practical relevance of the shallow recommendation. However, the central negative result on deep fast-weight memories currently rests on experiments that do not control the factor-scale variable identified by the paper's own theory, so the claimed structural conclusion is not yet established.","major_comments":[{"comment":"The conclusion that deeper fast-weight memories are structurally harder is underdetermined by the experiments. All deep variants initialize both fast factors with the official Gaussian std 0.02 (Appendix B.1), while the paper's own Eq. (15) shows that the induced update depends on the factor scale c. With d_head = 128, the effective product W^(1)W^(2) has entry std ≈ sqrt(128)·0.02^2 ≈ 4.5e-3, roughly 4-5x smaller than the shallow 0.02, and the terms W2^T W2 and W1 W1^T in the induced update scale as d_head·σ^2 ≈ 0.05, so the deep net starts with a damped effective update. The stabilization sweeps in Tables 22-24 vary mean scaling, epsilon, activation placement, and zero-vs-Gaussian initialization, but never the factor scale or balance that Eqs. (16)-(17) identify as controlling. Without a sweep that matches the product scale of the shallow learner, or that varies c directly, the observed +0.09 to +0.18 validation-loss gaps in Table 21 may be an artifact of the default initialization rather than evidence for the paper's factor-coupling explanation.","section":"§4.1, Table 6, Eqs. (13)-(17), Appendix D.4"},{"comment":"The abstract's causal explanation ('Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations') is inconsistent with the main-text analysis for the deep linear rows. The Linear-Linear row in Table 6 contains no nonlinearity, so an activation-magnitude mechanism cannot explain its 3.1265 loss; the paper's own §4.1 explanation is a factor-coupling/update-geometry argument, while for Norm the proposed mechanism is the 1/σ gradient amplification in Eq. (10). These are different mechanisms and should not be conflated. Please revise the abstract to state the factor-coupling explanation for depth and the gradient-amplification explanation for normalization separately, or provide activation-norm diagnostics for the deep variants.","section":"Abstract, §4.1, Table 6"},{"comment":"The paper repeatedly calls Modular TTT a 'unified framework' for the TTT design space, but Appendix E concedes that multi-step inner updates, momentum, and Muon-style update normalization (as used in LaCT) do not fit the current fused graph implementation without specialized update paths. Since the ablations and the deep-memory negative results are derived only under the one-step, dual-form update family, the framework is a subfamily of TTT rather than the full design space. The main text should carry the Appendix E scope statement forward, so that the 'unified framework' claim and the transferability of the ablations are not overstated.","section":"§1, §3.2, Appendix E"}],"minor_comments":[{"comment":"The validation set used for all final-loss tables is not described; please state its size, how it is split from the pretraining corpus, and how validation loss is computed.","section":"Appendix B.2"},{"comment":"Please clarify whether all external baselines in Table 9 (LLaMA, GDN, LaCT) were trained with the same tokenizer and corpus as the Modular TTT models; the LaCT check in Table 30 explicitly notes tokenizer-matched groups, and the main table should be equally explicit.","section":"Table 9, Appendix D.5.3"},{"comment":"Report seed variance for the deep-memory rows; the five-seed protocol is applied only to the shallow comparisons in Table 17, so the deep-memory gaps currently have no reported uncertainty.","section":"Table 6, Appendix D.4"},{"comment":"The 'validation loss' axis in Figure 1(c) is unlabeled in the caption; please add the metric name and note the configuration used.","section":"Figure 1"},{"comment":"The decay matrix notation M in Table 1 is defined only through the appendix's vector-decay formulas; a one-line pointer in Table 1 would help readers connect the primitive rules to the token-wise recurrence.","section":"Table 1, Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"The main technical issue is the deep-memory init-scale confound; if the authors add factor-scale/balance sweeps or soften the structural claim, I would support acceptance. The scope overclaim relative to Appendix E should also be fixed in the revision. The paper otherwise appears methodical, with code release and multi-seed checks for the key positive ablations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time: the DAG-based composition of TTT primitives into train-view forward, train-view backward, and causal query-view rules is a real engineering contribution, and the code is public. The ablations are the best controlled look yet at TTT design axes — small-lr init, scalar decay, single-layer SiLU, loss choice — and the five-seed paired runs support the main recipe. The 410M/1.45B runs against GDN are honest, and the efficiency numbers are credible. Props. 1–3 are simple but correct.\n\nThe soft spots, in order.\n\nFirst, the central negative finding — deeper fast-weight networks hurt — is under-tested. The authors' own Eqs. (13)–(17) show that two-factor updates are controlled by the scale of the factors and the Wishart preconditioners. Yet every deep run uses the official std-0.02 init for both factors, which makes the effective product roughly 4–5× smaller in entry std than the shallow init. That under-scaling damps the inner learning rate and could explain the gap. The paper never varies factor scale or balance, which is exactly the variable the theory says to vary. So 'deeper hurts' may be an artifact of default init. A factor-scale sweep, or at least a larger-init deep run, is needed before the claim is solid.\n\nSecond, the abstract says 'normalization tends to hurt,' but Table 5 shows Linear+Norm beating Linear at 160M under MSE (3.0300 vs 3.0380). The body text is careful — 'Norm is mixed' — but the abstract overstates. Fix the wording.\n\nThird, the framework is not as unified as advertised: it only handles one-step, dual-form updates. Multi-step inner loops, momentum, and Muon-style normalization are out of scope. Appendix E admits this, which is good, but it means the ablations are about a specific subfamily, not all TTT. That is a scope statement, not a fatal flaw.\n\nMinor: several central tables (loss, decay) have no error bars, though the key SiLU comparison has paired seeds. The theory section is standard but clean.\n\nWho this is for: anyone building TTT or linear-attention variants; the recipe and code save real time. It deserves a serious referee, but the deep-memory claim needs the init-scale control before it should be published as-is.","headline":"Useful modular framework and a mostly clean empirical recipe, but the headline 'deeper hurts' result is confounded by an untested init-scale variable that the authors' own math identifies.","tokens_in":29874,"tokens_out":2435,"would_cite":true,"duration_ms":21700,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Modular graphs turn test-time training into a composable design space","keywords":["test-time training","fast weights","sequence modeling","directed acyclic graph","composable modules","online learning","language modeling","linear attention"],"falsifier":"A concrete test: extend the modular primitive library with a two-step inner-update primitive and train the same 160M-scale models on 10B tokens under the same protocol; if the two-step variant lowers validation loss below Linear-SiLU, the paper's conclusion that one-step shallow learners are the strongest configuration fails. A second test would compare a momentum-augmented inner update against the same shallow frontier.","tokens_in":28537,"feed_emoji":"🧩","tokens_out":8684,"duration_ms":76471,"temperature":0.7,"pith_summary":"The paper tries to establish that test-time training (TTT) can be treated as a modular design space rather than a collection of hand-coded variants. It represents the inner learner as a directed acyclic graph of small primitives, so the full TTT computation, including the fast-weight update, follows automatically from three local rules. Using this framework, the paper isolates the effect of each design choice and finds that a shallow linear fast-weight network with small learning-rate initialization, scalar decay, and a SiLU activation is the strongest configuration. Deeper fast-weight networks, normalization, residual connections, and gating do not help under the tested one-step updates, and the best variant scales to 410M and 1.45B parameters with performance comparable to Gated DeltaNet.","feed_headline":"Composable test-time training favors shallow fast-weight learners","feed_subtitle":"A graph of primitives exposes each TTT design choice; the winning mix is linear plus SiLU with scalar decay.","key_machinery":"The central object is the learner DAG: a directed acyclic graph whose nodes are registered primitives (Linear, Gate, Norm, Act, Add, Mul) and whose edges are tensor dependencies, with a loss function attached at the output node. Each primitive carries three local rules, the train-view forward, the train-view backward, and the causal query-view forward; executing them in topological order, then reverse topological order, then topological order again produces the full TTT computation including the fast-weight state transition. The load-bearing identity is the chunkwise dual-form readout for a linear primitive, $O = QW_{\\text{start}} - \\operatorname{Tril}(Q\\hat{K}^\\top)d\\hat{V}$, where the primitive rules supply the learning-rate-scaled key $\\hat{K}$ and the decay matrix. Automatic differentiation supplies the local backward signals, while the query-view rules and fast-weight transition come from the registered primitives, removing the need to hand-derive a global update rule for each topology.","core_discovery":"The central claim is that every TTT variant can be expressed as a DAG of registered primitives, and that the full graph-level TTT computation composes automatically from each primitive's train-view forward, train-view backward, and causal query-view rules. The paper uses this to conduct controlled ablations at 160M and 410M scale on 10B tokens, finding that MSE and inner-product losses are equally competitive, that small learning-rate initialization stabilizes the update by keeping the spectrum of $I - K^\\top \\operatorname{diag}(\\eta) K$ controlled, that scalar decay recovers most of the gain of vector decay, and that a single SiLU layer gives a consistent improvement. Deeper product-form learners fail to beat the shallow frontier, and the paper offers a structural explanation: rescaling the factors of a product $W^{(1)}W^{(2)}$ leaves the represented memory function unchanged but changes the induced one-step update direction, so depth adds an extra optimization burden. Guided by these findings, the selected shallow variant trained on 100B tokens reaches training loss and benchmark performance comparable to Gated DeltaNet at 410M and 1.45B scale.","pith_inferences":["A consequence the paper leaves implicit is that the shallow-frontier result nudges TTT toward linear-attention-like memories with a bounded nonlinear write; if this holds, part of the reported gains in newer TTT variants may come from learning-rate and decay tuning rather than from a richer inner learner.","Because the framework's primitives assume a single gradient step, a natural testable extension is to register a two-step or momentum primitive; if multi-step inner updates beat Linear-SiLU, the depth penalty would be a limitation of the one-step family, not of TTT in general.","The finding that vector decay achieves the lowest loss but scalar decay captures most of the gain suggests an untested prediction: feature-selective decay should matter most on long heterogeneous sequences, where selective forgetting is more valuable.","The speedup of analytic backward operators over automatic differentiation implies that expanding the primitive library with analytic rules for gated and normalized paths could make deeper learners inexpensive enough to revisit the depth conclusion."],"forward_implications":["New TTT variants can be assembled by rearranging or replacing registered primitives, so the design space can be explored without hand-deriving a global update for each topology.","The ablation yields a concrete default recipe: a single linear fast weight, small learning-rate initialization, scalar decay, and a SiLU activation, with deeper, normalized, residual, or gated learners offering no measured benefit under one-step updates.","At 410M and 1.45B parameters trained on 100B tokens, the selected variant matches Gated DeltaNet on training loss and multiple-choice accuracy, while containment-style tasks and long-context retrieval remain weaker.","The analytic primitive back-ends give roughly 2-3x higher training throughput than the official hard-coded TTT implementation for the same topologies, and about 1.65-2.62x faster primitive-level backward computation than automatic-differentiation references."],"supporting_citations":[{"why":"Supplies the TTT formulation, the chunkwise dual-form causal update, and the official Linear/MLP implementations used as the throughput and reproduction baselines.","marker":"[44]"},{"why":"Gated DeltaNet is the recurrent baseline whose 410M and 1.45B training loss and downstream benchmarks the scaled-up Modular TTT variants are compared against.","marker":"[52]"},{"why":"LaCT provides the strong TTT variant used in the boundary-regime systems check and the SwiGLU-style fast learner examined in the deep-memory ablations.","marker":"[54]"},{"why":"Provides the sigmoid learning-rate parameterization $\\eta_t = 2\\,\\mathrm{Sigmoid}(\\beta_t + b)$ whose offset $b$ defines the standard and small learning-rate initializations.","marker":"[15]"},{"why":"Supplies the needle-in-a-haystack long-context retrieval tasks used to assess the precise-recall limits of the shallow variants.","marker":"[21]"}],"fun_headline_variants":["Composable TTT graph: depth hurts, SiLU helps, matches Gated DeltaNet","Modular TTT: compose primitives, avoid deep fast-weight nets","Modular TTT: shallow fast-weight learners beat deeper ones","Modular TTT: one DAG, shallow wins, SiLU helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's promise as a general design space rests on the assumption that every TTT variant that matters can be expressed as a directed graph of its one-step primitives; variants needing multi-step inner updates, momentum, or specialized update normalization lie outside it, and the ablations may not transfer to them.","fun_headline_variants_meta":{"raw":{"variants":["Composable TTT graph: depth hurts, SiLU helps, matches Gated DeltaNet","Modular TTT: compose primitives, avoid deep fast-weight nets","Modular TTT: shallow fast-weight learners beat deeper ones","Modular TTT: one DAG, shallow wins, SiLU helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":4011,"prompt_tokens":1019,"completion_tokens":2992,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2910}},"tokens_in":635,"tokens_out":2992,"duration_ms":21577,"temperature":1.0,"reasoning_tokens":2910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:37:56.589898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: extend the modular primitive library with a two-step inner-update primitive and train the same 160M-scale models on 10B tokens under the same protocol; if the two-step variant lowers validation loss below Linear-SiLU, the paper's conclusion that one-step shallow learners are the strongest configuration fails. A second test would compare a momentum-augmented inner update against the same shallow frontier.","supporting_citations":[{"cited_title":"Gated delta networks: Improving Mamba2 with delta rule","cited_arxiv_id":null,"evidence_quote":"Gated DeltaNet is the recurrent baseline whose 410M and 1.45B training loss and downstream benchmarks the scaled-up Modular TTT variants are compared against."},{"cited_title":"Franke, Arber Zela, Frank Hutter, and Massimiliano Pontil","cited_arxiv_id":null,"evidence_quote":"Provides the sigmoid learning-rate parameterization $\\eta_t = 2\\,\\mathrm{Sigmoid}(\\beta_t + b)$ whose offset $b$ defines the standard and small learning-rate initializations."}],"review_version":1}