{"id":"4e2f9841-4082-482f-943f-faa6c86ded9a","arxiv_id":"2608.01833","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Training trajectories during grokking show glass-like arrest and a swap-based optimizer plug-in can accelerate generalization on modular arithmetic.","lead":"This paper measures three properties of a neural network's training trajectory, mobility, replica correlation, and fractal dimension, and argues that grokking is like a glass freezing into place. It then introduces a parameter-swapping trick, borrowed from glass physics, that helps the network break free and generalize faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AdamW's decoupled weight decay can trivially produce FD≈1, constant RC, and collapsing PM once data gradients vanish; the purported glass signatures may be an optimizer artifact rather than evidence of kinetic arrest.","rationale":"The reader correctly identified the FD estimator as fragile, but the more load-bearing issue is that the entire set of observables may be structurally determined by the optimizer choice. AdamW with decoupled weight decay produces an exact radial contraction when data gradients vanish, and radial contraction mathematically implies FD≈1, constant Pearson correlation between scaled replicas, and decaying displacement. None of these require glassy dynamics or even grokking. The paper's claims therefore rest on an untested confound. This does not automatically falsify the glass analogy—the data-gradient term may still drive genuine non-radial dynamics on the timescale of generalization—but the current evidence cannot distinguish between the two. The proposed control experiments are inexpensive and would settle the issue: if Adam without weight decay also shows FD collapse, the glass interpretation gains support; if not, the paper's central empirical pillar fails. Because the paper already contains partial Adam comparisons but does not report these observables separately, I recommend a conditional verdict requiring this isolation check before the kinetic-arrest claim can be accepted.","tokens_in":11385,"tokens_out":8073,"duration_ms":97847,"concrete_test":"Run the same 1-layer transformer on (x²+y) mod 67 with Adam (no weight decay), tuned to still grok, and measure PM/RC/FD with the same T=20 window and probe noise; if FD still collapses to 1 and RC still plateaus, the signatures are not caused by weight decay. Additionally, on the existing AdamW run, recompute FD after removing the weight-decay component from each step, ΔR̃_t = ΔR_t + ηλθ_t; if D_f rises well above 1 during the memorization phase while the raw D_f is 1, the 'channel-like' motion is the AdamW contraction direction, not a glassy arrest. As a further null check, apply only θ_{t+1}=(1−ηλ)θ_t to a random initialization and verify whether it reproduces FD≈1 and constant RC.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central dynamical evidence (FD collapse to 1, RC plateau with aging, PM collapse) is extracted from a single AdamW run with learning rate 1e-2 and decoupled weight decay 1e-1 (Fig. 1, Methods). Under AdamW, once the data-loss gradient becomes negligible, the update is θ_{t+1} = (1 − ηλ)θ_t. This is a purely radial contraction: every increment is collinear with θ_t, so the trajectory segment is a straight line and the FD definition D_f = 2/α with ΔR² ∝ Δs² yields D_f ≈ 1 by construction. PM decays like ηλ||θ_t||, and any two replicas differing only by small noise δ are multiplied by the same positive scalar, so their Pearson correlation remains essentially constant. Thus all three 'glass' signatures—collapsed mobility, aging-like RC plateau, and channel-like FD≈1—would be reproduced by a trivial null model with no grokking and no glass physics. The paper never isolates this confound: Fig. 1 uses only AdamW, and although Fig. 2 includes an Adam (no weight decay) probe, its PM/RC/FD curves are not separately reported or compared against the AdamW case. Since FD≈1 is also the trigger for SAM-Swap, the acceleration result is tied to an observable that may simply be detecting the onset of weight-decay-dominated dynamics. This is distinct from the reader's window-sensitivity concern: even a perfect FD estimator would give 1 on a radial-contraction trajectory.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-observable framework—parameter mobility (PM), replica correlation (RC), and trajectory fractal dimension (FD)—to characterize grokking as a glass-like kinetic arrest in parameter space. Using full-batch AdamW on a one-layer transformer trained on (x^2+y) mod 67, the authors report that during the memorization phase PM collapses, RC shows two-step relaxation with aging, and FD decays to approximately 1, consistent with a channel-like, arrested trajectory. They then introduce State-Aware Monte Carlo Parameter Swapping (SAM-Swap), an FD-triggered intervention that randomly swaps parameters within layers, and report that it reduces grokking time from about 3000 to about 650 epochs. They compare this with weight decay and large Gaussian noise and conclude that accelerated generalization is associated with random exploration in parameter space.","tokens_in":11754,"tokens_out":7919,"duration_ms":84135,"significance":"If the central evidence were robust, the paper would provide a practically useful dynamical diagnostic for grokking and strengthen the analogy between grokking and glassy relaxation. The framework is intuitive, the paper is clearly written, and the authors are commendably candid about limitations in the Limitation section. However, the evidence is currently undermined by a plausible confound with AdamW's decoupled weight decay, the fragility of the FD estimator, the single-task/tuned-hyperparameter setting, and missing error bars. These issues are addressable, and a careful revision could make the paper a meaningful contribution, but as it stands the empirical support is not yet convincing.","major_comments":[{"comment":"All three glass signatures (PM collapse, RC plateau, FD≈1) are reported for AdamW with lr=1e-2, weight decay=1e-1. Once the training loss gradient becomes negligible after memorization, AdamW's decoupled weight decay dominates the update: θ_{t+1} ≈ (1 − ηλ)θ_t. This is a purely radial contraction toward the origin. Such a trajectory is a straight line in parameter space, so the FD estimate D_f = 2/α with ΔR² ∝ Δs^α gives α=2 and D_f≈1 by construction; the displacement decays as ηλ‖θ_t‖; and any two replicas that differ by small additive noise are multiplied by the same scalar, so their Pearson correlation remains essentially constant. The manuscript does not report PM/RC/FD curves for the Adam (no weight decay) baseline in Fig. 2, so this null model is not ruled out. Please add an Adam-without-weight-decay control, or subtract the radial-contraction component before computing the diagnos","section":"Observation (Fig. 1); Methods 'Small Additive Gaussian Noise as Probe'"},{"comment":"FD is simultaneously the evidence for glass arrest and the trigger for the SAM-Swap intervention, but the Limitation section states that the FD estimate is window-dependent, sensitive to sampling range and density, and 'requires multiscale refinement before it can serve as a robust order parameter.' The log-log regression lag range is unspecified, the window T=20 is chosen empirically, and FD values below 1 are dismissed as artifacts. The acceleration result depends on threshold df.glass=1.1 and swap ratio r=1e-2, both chosen empirically. Crucially, no control is shown with swaps applied at fixed intervals or random times unlinked to FD, so the 'state-aware' aspect of SAM-Swap is not demonstrated. Please specify the lag range, provide threshold and ratio sensitivity curves, include error bars, and add a non-triggered swap control to isolate the effect of state-awareness.","section":"Limitation; Algorithm 1; 'State-Aware Monte Carlo Parameter Swapping'"},{"comment":"The broad claim that 'standard optimization presents clear signatures of glass dynamics' is supported by a single task (one-layer transformer on (x^2+y) mod 67), and the caption of Fig. 2 states that all hyperparameters are selected to minimize grokking time. The comparison of SAM-Swap, weight decay, and Gaussian noise is therefore a post-hoc tuned demonstration, not a predictive test, and the paper's own conclusion concedes it is unclear whether FD-triggered perturbations generalize beyond modular arithmetic. Please add at least one additional algorithmic grokking task with pre-registered hyperparameters, or show that the qualitative results are robust across a reasonable hyperparameter grid without task-specific retuning.","section":"Conclusion and Future Directions; Fig. 2 caption"},{"comment":"Error bars are omitted for the key PM and FD curves in Fig. 1b/d and Fig. 2b/d, and no confidence intervals are given for the reported grokking times (~3000 vs ~650 epochs). The claims of 'collapse', 'plateau', and 'aging' are based on visual inspection, and the two-step RC relaxation is not quantified (e.g., no plateau-length or decorrelation-time metric). Without uncertainty quantification, the statistical strength of the central empirical claims cannot be assessed, especially given the potential weight-decay confound. Please include error bars or confidence bands, and report quantitative summary statistics for the observed dynamical regimes.","section":"Figures 1 and 2"}],"minor_comments":[{"comment":"The equation for dR has a formatting issue: it should read dR = (1/|l|) Σ_l ‖R_{t+T} − R_t‖_2. The notation R_t is used inconsistently (sometimes 'Rt'). Please clarify.","section":"Methods, 'Fractal Window Displacement'"},{"comment":"The notation 'τ=1, ....[t/2]' is informal. Use τ ∈ {1, ..., ⌊t/2⌋} and, importantly, specify the actual lag range used for the log-log regression. This is needed for reproducibility of the FD values.","section":"Methods, 'Fractal Dimension of the Training Trajectory'"},{"comment":"The pseudocode computes a per-layer fractal dimension d_f^ℓ and applies swaps per layer, while the text and Fig. 1 describe FD as 'average across layers.' Clarify whether swaps are triggered by layer-specific or averaged FD, and how the loss cap is evaluated after sequential layer-wise swaps.","section":"Algorithm 1"},{"comment":"Several references are dated 2026 (e.g., Ersoy and Wiesner 2026; Li et al. 2026; Pakman et al. 2026). The authors should verify publication status and ensure all citations conform to the journal's guidelines for forthcoming work.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The AdamW weight-decay confound is the most serious issue: if the radial-contraction null model reproduces the reported PM/RC/FD curves, the glass-arrest interpretation is not supported. The authors' own Limitation section is unusually candid but also reveals that the FD estimator is not yet a robust order parameter. I believe the paper can be made publishable with a focused revision that adds the Adam-no-decay control, a random-swap control, error bars, and at least one additional task. If the authors cannot rule out the weight-decay null model, the central claim would not survive. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to it: the new pieces here are real—three dynamical observables (PM, RC, FD) applied to grokking, and a swap-based optimizer plug-in that cuts grokking time by about 5x on modular arithmetic. That's worth a look. But the central claim—that grokking is a glass-like kinetic arrest evidenced by these observables—is not yet supported, and the stress-test got the main confound right: AdamW's decoupled weight decay can produce all three signatures for free once data gradients vanish.\n\nHere's the worry. With η=1e-2, λ=1e-1, once training loss hits zero, the AdamW update becomes θ_{t+1}=(1-ηλ)θ_t—pure radial contraction. A straight ray in parameter space gives FD≈1 by construction, PM decays because ||θ|| shrinks, and two replicas seeded with small noise are multiplied by the same scalar, so their Pearson correlation stays pinned. So the \"glass signatures\" in Fig 1 may be an artifact of the optimizer, not a dynamical phase. The paper doesn't address this. Fig 2 includes an Adam run without weight decay, but the PM/RC/FD curves for that baseline are never compared explicitly against the AdamW case. That comparison is the load-bearing experiment that's missing.\n\nThe other soft spots the reader flagged are real: one task/architecture, no error bars on the main curves, and the FD estimator is admitted to be window-dependent and sensitive to sampling. There's also circularity: the same FD collapse that is claimed as evidence is used to trigger the swap, and the threshold, swap ratio, and noise amplitudes are all tuned to minimize grokking time. So the acceleration result is not an independent prediction.\n\nTo be fair, what the paper does well: the observables are standard in glass physics and their adaptation to training dynamics is new; the SAM-Swap idea is clever; and the authors are unusually honest about limitations. But as it stands, the evidence doesn't distinguish a glass phase from weight-decay physics.\n\nI'd send it to a serious referee, but with a request for the Adam baseline, error bars, and ideally a second task. The reader's conditional verdict was about right; the stress-test concern makes it more conditional.","headline":"Interesting diagnostic trio and a swap trick, but the glass-arrest claim is confounded by AdamW's weight decay and the paper's own limitations concede the FD estimator isn't ready to carry the load.","tokens_in":12264,"tokens_out":3899,"would_cite":false,"duration_ms":40677,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grokking is a glass-like kinetic arrest in parameter space, and a swap-based Monte Carlo perturbation triggered by low fractal dimension reduces grokking time from about 3000 to about 650 epochs.","keywords":["grokking","glass dynamics","kinetic arrest","parameter mobility","replica correlation","fractal dimension","swap Monte Carlo","generalization delay"],"falsifier":"Train the same (x^2+y) mod 67 transformer with fractal dimension estimated over multiple window sizes (e.g., 20, 100, and 500 epochs). If the collapse to D_f near 1 during memorization disappears at larger windows, the channel-like arrest diagnosis is an artifact of window choice. Alternatively, if SAM-Swap triggered by FD below 1.1 fails to accelerate grokking on a different modular arithmetic task, the trigger is not general.","tokens_in":1321,"feed_emoji":"🔄","tokens_out":1566,"duration_ms":58798,"temperature":0.7,"pith_summary":"Grokking is the strange training regime where a network memorizes perfectly for thousands of epochs and then abruptly generalizes. This paper argues that the delay is not merely slow gradient flow but a glass-like kinetic arrest: during memorization, parameter mobility collapses, replicas branched from the same state stay highly correlated in a history-dependent way, and the trajectory becomes straight and channel-like with fractal dimension near one. It introduces SAM-Swap, a swap Monte Carlo-inspired intervention that randomly exchanges parameter values within layers when the fractal dimension drops below a threshold, rejecting changes that raise loss above the initial training value. In the paper's experiments, this reduces grokking time from about 3000 to about 650 epochs. The three observables jointly diagnose whether slow dynamics are just small updates or true confinement in parameter space.","feed_headline":"Swap Monte Carlo cuts grokking time from 3000 to 650 epochs","feed_subtitle":"Delayed generalization looks like kinetic arrest; random parameter swaps restore exploration.","key_machinery":"SAM-Swap is a swap Monte Carlo-inspired intervention: within each transformer layer, random pairs of parameter values are exchanged, preserving the parameter distribution while rearranging the configuration; a loss cap rejects swaps that push training loss above the initial value. It is triggered when the layer-level trajectory fractal dimension falls below 1.1. The supporting diagnostics are parameter mobility (Euclidean displacement over a 20-epoch window), replica correlation (Pearson correlation between independently perturbed replicas branched at waiting time), and fractal dimension (exponent relating squared displacement to arc length, estimated by log-log regression over a sliding 20-","core_discovery":"The central claim is that standard optimization on a grokking task leaves the network in a kinetically arrested, glass-like regime during memorization. Evidence comes from three observables: parameter mobility drops by orders of magnitude; replica correlation shows two-step relaxation and strong waiting-time dependence, a signature of aging; and trajectory fractal dimension collapses toward one, indicating low-dimensional, ballistic, channel-like motion. These signatures reverse near generalization. The same kinetic-arrest picture motivates SAM-Swap, a structured random parameter-swap perturbation triggered when fractal dimension falls below an empirical threshold, which restores mobility, d","pith_inferences":["If the kinetic-arrest picture is generic, similar state-triggered perturbations could be applied to non-algorithmic grokking tasks, but the paper only demonstrates the effect on modular arithmetic; testing on addition or multiplication mod different primes would clarify generality.","The claim that the FD threshold is not critical as long as it exceeds one suggests the arrest is a qualitative regime rather than a sharp boundary; a multiscale FD estimator could turn this into a practical order parameter for monitoring training.","SAM-Swap's loss-cap rejection is reminiscent of Metropolis acceptance; formalizing the swap rate and acceptance criterion as a Markov chain could connect the intervention to an effective temperature and predict optimal swap schedules.","The framework implies that memorization and generalization are not just different loss optima but different dynamical phases, so measuring replica correlation and fractal dimension during training could inform adaptive, state-dependent optimization strategies."],"forward_implications":["If grokking is kinetic arrest, interventions that restore parameter mobility and exploration can shorten generalization delay without changing architecture or data.","The three observables form a diagnostic: low mobility alone does not imply glassy confinement; it must coincide with high replica correlation and low fractal dimension.","Structured non-local parameter swaps can be as effective as large additive Gaussian noise in accelerating generalization, suggesting multiple routes to break the arrested regime.","The timing of generalization is trajectory-dependent: accessible generalizing solutions may exist in the landscape while the optimizer is dynamically unable to reach them.","Acceleration is consistently associated with random exploration in parameter space, connecting diffusion-like dynamics to entropy increase and generalization."],"fun_headline_variants":["Swap Monte Carlo reduces grokking from 3000 to 650 epochs","Grokking is glassy; swap Monte Carlo cuts epochs from 3000 to 650","Monte Carlo parameter swaps cut grokking epochs by 4.6x","Grokking as glass: kinetic arrest broken by swap Monte Carlo","Random parameter swaps accelerate grokking from 3000 to 650 epochs"],"cache_read_input_tokens":13440,"weakest_assumption_plain":"The central claim depends on the local fractal dimension estimate—log-log regression over a fixed 20-epoch window and a selected lag range—being a faithful trigger for the arrested regime; the paper concedes that this estimate is window-dependent and sensitive to sampling range and density.","fun_headline_variants_meta":{"raw":{"variants":["Swap Monte Carlo reduces grokking from 3000 to 650 epochs","Grokking is glassy; swap Monte Carlo cuts epochs from 3000 to 650","Monte Carlo parameter swaps cut grokking epochs by 4.6x","Grokking as glass: kinetic arrest broken by swap Monte Carlo","Random parameter swaps accelerate grokking from 3000 to 650 epochs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00139,"raw_usage":{"total_tokens":5477,"prompt_tokens":771,"completion_tokens":4706,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":4602}},"tokens_in":515,"tokens_out":4706,"duration_ms":39426,"temperature":1.0,"reasoning_tokens":4602,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:57:18.434176+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same (x^2+y) mod 67 transformer with fractal dimension estimated over multiple window sizes (e.g., 20, 100, and 500 epochs). If the collapse to D_f near 1 during memorization disappears at larger windows, the channel-like arrest diagnosis is an artifact of window choice. Alternatively, if SAM-Swap triggered by FD below 1.1 fails to accelerate grokking on a different modular arithmetic task, the trigger is not general.","supporting_citations":[],"review_version":1}