{"id":"d1643924-77e1-4733-ac5d-2eb88ef56d33","arxiv_id":"2502.04121","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors show that when unperturbed neural network training reaches a quasi-steady state, the mean time to a target test accuracy under periodic perturbations can be predicted from a single perturbation experiment.","lead":"Machine learning training is treated as a race to a target test accuracy, and the paper shows that measuring the effect of one rare training perturbation at a single time predicts speedups at many perturbation intervals. This matters because training perturbation protocols such as shrink-and-perturb are usually tuned by trial and error, and this offers a cheaper way to pick both the protocol and its frequency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The full-state quasi-steady-state factorization behind Eq. (9) is verified only via marginal test accuracy, so residual-time predictions can depend on unverified weight degrees of freedom; the P=3 speedup claim also extrapolates outside the formal validity range.","rationale":"The paper's central promise is that one measurement at a single perturbation epoch P* predicts E[T_P] across a range of P and ranks protocols. That promise is only guaranteed by the factorization in Eq. (6), which is precisely the premise checked only through the proxy G(A,t). The load-bearing nature of this premise is internal to the derivation, not a matter of prior disagreement: Eq. (4) weights E[tau_P(theta)] by G(theta,P); if G does not factorize, the weighted average at P* need not equal the weighted average at other P, and Eq. (9) is not a theorem. The paper's own Fig. 5 supplies direct empirical evidence that tau_P is roughly constant for P=10-100, which goes partway toward validating the operational consequence. The remaining soft spot is the extrapolation to P=3, where the formal condition P > max(t_r, tau) fails and the predicted speedup (16) is already below the brute-force value (21). This does not make the paper wrong; the ranking of protocols may still be robust. It means the headline quantitative claim is currently supported more by observed empirical regularity than by the stated QSS argument. The proposed stratification experiment would resolve whether the missing ingredient is a hidden state variable, which would be a correctable but essential modification, or whether the A-marginal happens to be sufficient in these settings. The code repository is a useful asset for running such a test, though no commit hash is provided. I therefore keep the reader's conditional verdict.","tokens_in":963,"tokens_out":1001,"duration_ms":105918,"concrete_test":"Using the existing 1000-trajectory ensemble, at P*=100 stratify the survivors into bins with matched test accuracy but different training loss and layer-wise weight norms; apply the S&P perturbation separately to each stratum and compare mean residual times. If E[tau] differs significantly across strata within an accuracy bin, the test-accuracy marginal is not a sufficient state variable for Eq. (7), and the full-state QSS factorization in Eq. (6) is not supported for the degrees of freedom that determine residual time.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (6) assumes G(theta,t) approximately equal to phi(theta) Psi_T(t) for t > t_r, a factorization of the full network-state propagator that makes the residual time tau_P independent of P in Eqs. (7)-(8) and justifies Eq. (9). The paper tests this hypothesis only through the marginal QSS of the test accuracy A (Eqs. (10)-(11), KS tests in Fig. 3D), which the authors explicitly call a proxy. The perturbations act on theta, not on A: S&P shrinks all weights and adds noise, while partial SR reinitializes the smallest-magnitude weights. Consequently E[tau(theta)] in Eq. (8) is an average over weight configurations, and unless the full theta-distribution among survivors factorizes as in Eq. (6), a single measurement at P* need not transfer to other P. The empirical constancy of tau_P in Fig. 5 over P=10-100 is real supporting evidence, but it is measured at only four P values and for two perturbations. The headline S&P prediction at P=3 lies outside the stated validity condition P > max(t_r, tau) (t_r approximately 28, tau approximately 3) and outside the measured range, so it rests on an extrapolation of the unverified factorization. The reported mismatch (predicted speedup about 16 vs brute-force about 21) is consistent with this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a first-passage framework for rationally choosing periodic perturbation protocols (shrink-and-perturb, partial stochastic resetting, and full stochastic resetting) in neural-network training. It defines a survival probability Psi_T(t) for unperturbed training to a target test accuracy, derives an exact expression for the mean first-passage time under perturbations, Eq. (3), and then argues that if the full network-state propagator reaches a quasi-steady state (QSS), Eq. (6), and perturbations are rare, the mean residual time bar_tau_P becomes P-independent, yielding the predictive formula Eq. (9). The authors test the QSS assumption on CIFAR-10 ResNet-18 via marginal test-accuracy distributions and Kolmogorov-Smirnov tests, measure bar_tau_P for S&P and partial SR over a range of P, and use Eq. (9) to predict speedups, reporting that S&P at P=3 gives a predicted speedup of about 16 versus a brute-force value of about 21. Transferability experiments across CIFAR-100, momentum, and a regression task appear in the supporting information.","tokens_in":15865,"tokens_out":4980,"duration_ms":53125,"significance":"If the central claim holds, the method is valuable: it promises a cheap, single-ensemble protocol for ranking perturbation schemes and choosing perturbation periods, based on a physically motivated QSS assumption. The paper has notable strengths: 1000 training trajectories for the main experiment, a publicly available code/data repository, a clean derivation of Eq. (3), and an explicit parameter-free speedup prediction for full stochastic resetting via Eq. (5). The transferability experiments, though less detailed, broaden the scope. However, the load-bearing QSS assumption is verified only through a marginal collective variable, and the headline P=3 prediction lies outside the formal validity range of Eq. (9). These issues are fixable but require additional evidence or a reframing of the claims.","major_comments":[{"comment":"The derivation of P-independent residual times rests entirely on the full-state quasi-steady-state factorization G(theta,t) ≈ phi(theta) Psi_T(t). The verification in Section III B (Eqs. (10)-(11), Fig. 3D) concerns only the marginal distribution of test accuracy A, which the authors explicitly call a proxy. Since S&P and partial SR act directly on theta (shrinking or reinitializing weights), the residual time E[tau(theta)] in Eq. (8) averages over weight configurations, and a marginal QSS in A does not imply the required factorization of the full state distribution. The constancy of bar_tau_P in Fig. 5 is supportive evidence, but it is established only for the specific perturbations and P values tested. Please either provide a more direct test of the full-state QSS (e.g., stationarity of layer-wise weight norms or gradient norms conditioned on survival) or clearly label Eq. (9) as an assumption-based prediction whose range of validity is itself part of the empirical claim.","section":"Section III A, Eq. (6)"},{"comment":"The headline S&P prediction at P=3 violates the stated validity condition P > max(t_r, bar_tau_P) given after Eq. (8) and repeated in the methodology paragraph: for the 72% target, t_r=28 and bar_tau≈3, so P=3 is smaller than t_r. The predicted speedup curves in Fig. 6 are plotted down to P=bar_tau_P*, not down to max(t_r, bar_tau_P*). The reported gap between the predicted speedup (~16) and the brute-force value (~21) is consistent with this extrapolation. The authors should either restrict their predictions and recommendations to P in [max(t_r, bar_tau_P*), P*], or provide an explicit empirical justification for extending the range to P<t_r.","section":"Section III D, Fig. 6"},{"comment":"The Kolmogorov-Smirnov test used to establish the QSS compares each epoch's CDF F_t(A) to the average CDF over epochs 20-100, and for t in that window the average includes F_t itself. This reference distribution is not independent of the tested sample, which biases the test toward accepting the QSS hypothesis and may lead to underestimated relaxation times t_r. A cleaner procedure would compare each F_t to an average over a disjoint hold-out set of epochs, or use pairwise tests between non-overlapping time windows.","section":"Section III B and SI B, Eqs. (S10)-(S11)"}],"minor_comments":[{"comment":"The phrase \"designed ad hocby intuition\" is missing a space and should read \"ad hoc by intuition.\"","section":"Introduction"},{"comment":"The caption and axis label would benefit from stating explicitly that the y-axis is on a logarithmic scale and how p-values equal to machine precision are handled.","section":"Fig. 3D"},{"comment":"The derivation of Eq. (3) in the SI is clear, but the main text could note more explicitly that the exact identity holds for arbitrary perturbation protocols and all P, with the approximation entering only later through the QSS assumption.","section":"Section II, Eq. (3)"},{"comment":"The repository link would be more useful with a version or commit identifier and a license statement.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The central identity Eq. (3) is taken from the authors' companion work [38] with overlapping authorship. This is not circular in a damaging sense because bar_tau at P* is measured independently and the subsequent prediction is not fitted to the target speedup, but the paper should make the overlap and derivation status explicit to avoid any appearance of self-reliance. The main revisions needed are the statistical test of the QSS assumption and careful restriction of the predictive range; once those are addressed, the paper would be a reasonable fit for a journal bridging statistical physics and machine learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a legitimate, well-executed application of first-passage response theory to neural network training. The authors show that if unperturbed training reaches a quasi-steady state, the mean residual time after a perturbation is roughly independent of when you apply it, so one measurement at a single P* plus the unperturbed survival curve predicts speedups across a wide range of perturbation periods. They verify this on CIFAR-10 with ResNet-18 and show transferability to CIFAR-100, MNIST, different optimizers, and a regression task. The code and data are public.\n\nWhat's actually new is the QSS reduction plus the single-P* prediction protocol. Eq. (3) itself is a law-of-total-expectation identity and is correctly derived in the SI; the predictive step is not circular because bar_tau is measured, not fitted. The empirical constancy of tau_P over P=10-100 in Fig. 5 is direct evidence for the practical claim, and the 1000-run ensemble gives the survival curves solid footing.\n\nThe soft spots are in proportion. Most importantly, the full-state QSS factorization (Eq. 6) is only tested through the marginal distribution of test accuracy, which is a proxy. The perturbation acts on weights, so in principle residual times could depend on the weight configuration that the marginal doesn't capture. The constancy of tau_P in Fig. 5 is reassuring, but it's measured at four P values for two perturbations. Second, the headline S&P speedup at P=3 is outside the formal validity condition P > max(t_r, tau_bar), since t_r ~ 28. The paper honestly reports the mismatch (predicted 16 vs brute-force 21) and the prediction lines are only plotted down to P = tau_bar, so the P=3 claim is an extrapolation. That said, the authors flag this themselves, and the discrepancy is a quantitative gap, not a qualitative failure.\n\nThis paper is for anyone working on perturbation training protocols—shrink-and-perturb, warm restarts, stochastic resetting—and for people who like statistical-physics-style reductions of training dynamics. It deserves a serious referee; the main revisions are to tighten the discussion of when the prediction is trustworthy at small P and to be clearer that the QSS is empirically supported but not proven at full state dimension. I'd engage with it.","headline":"A solid, honestly-reported prediction protocol for perturbation training; the main caveats (marginal QSS test, P=3 extrapolation) are real but acknowledged and do not sink the central idea.","tokens_in":16393,"tokens_out":3892,"would_cite":true,"duration_ms":35664,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that if unperturbed neural-network training reaches a quasi-steady state, the residual time to target accuracy after a perturbation becomes independent of when it is applied, so a single measurement predicts mean…","keywords":["first-passage processes","quasi-steady state","neural network training","stochastic resetting","shrink and perturb","perturbation optimization","CIFAR-10","ResNet-18"],"falsifier":"On a small network where all weights can be tracked, verify the full-state quasi-steady-state directly; then apply a perturbation that changes weights but leaves test accuracy nearly unchanged. If the residual time $\\bar\\tau_P$ varies with $P$ while the accuracy-marginal Kolmogorov–Smirnov test still says quasi-steady-state holds, the central claim is false.","tokens_in":15343,"feed_emoji":"⚡","tokens_out":5814,"duration_ms":57996,"temperature":0.7,"pith_summary":"The paper treats training a neural network up to a target test accuracy as a first-passage process, and asks how periodic perturbations—shrink-and-perturb, stochastic resetting, partial re-initialization—change the mean time to reach that target. Its central claim is that if the unperturbed training dynamics reach a quasi-steady state, then the average residual time after a perturbation becomes independent of when the perturbation is applied. That makes one measurement at a single perturbation interval $P^*$ sufficient to predict the mean first-passage time for every interval in a wide range. On CIFAR-10 with ResNet-18 the paper uses this to predict that shrink-and-perturb gives about a 16-fold speedup at $P=3$, close to the roughly 21-fold speedup found by brute force, and to select shrink-and-perturb over partial resetting and full stochastic resetting. The same recipe is shown to transfer to CIFAR-100, MNIST, momentum SGD, Adam, and a regression task.","feed_headline":"Measure one restart, predict every training speedup","feed_subtitle":"A first-passage identity maps a single perturbed run into a full frequency–speedup curve.","key_machinery":"The load-bearing object is the quasi-steady-state (QSS) factorization of the training propagator, $G(\\theta,t)\\simeq\\phi(\\theta)\\Psi_T(t)$, which says that after a relaxation time the relative population of network states freezes while the overall fraction of surviving models decays. Its role is to make the network state sampled at the perturbation time $P$ independent of $P$, converting the residual time $\\bar\\tau_P$ into a single number $\\bar\\tau$ that can be measured in one run. Eq. (9) is the resulting identity that turns that single measurement into a prediction curve for every perturbation frequency in the valid range.","core_discovery":"The central discovery is that under the quasi-steady-state factorization $G(\\theta,t)\\simeq\\phi(\\theta)\\Psi_T(t)$ for $t>t_r$, the conditional mean residual time after a perturbation becomes $\\bar\\tau_P\\simeq\\bar\\tau$, independent of the perturbation interval $P$, as long as $P$ is larger than both the relaxation time and the residual time itself. Substituting this into the first-passage response identity gives Eq. (9), $\\mathbb{E}[T_P]=\\sum_{t=0}^{P-1}\\Psi_T(t)+\\Psi_T(P)\\bar\\tau$, which is a parameter-free prediction: sample training once with the perturbation applied at a single large $P^*$, read off the survival function and the residual time, and predict the mean first-passage time for all $P$ in $[\\max(t_r,\\bar\\tau),P^*]$. The authors verify the quasi-steady-state proxy on test-accuracy marginals with Kolmogorov–Smirnov tests, show experimentally that $\\bar\\tau_P$ is flat over nearly two decades of $P$, and use the predicted speedup curves to rank perturbation protocols and identify an efficient interval.","pith_inferences":["The same one-shot response measurement could generalize to random or non-periodic perturbation schedules, such as exponentially distributed waiting times, by using the response-theory form in its full generality rather than fixed $P$.","If the quasi-steady-state assumption holds for other collective variables, the recipe could be used to select among data augmentation policies, learning-rate schedules, or architecture modifications, each treated as a perturbation with a measurable residual time.","For large-scale training where an ensemble is already run in parallel with early stopping, the survival function is available for free, so the only added cost of the method is one perturbed run at $P^*$.","The gap between the predicted speedup (about 16) and the brute-force speedup (about 21) indicates a weak residual dependence on $P$ even in the valid range; quantifying that dependence could sharpen the prediction or bound its error."],"forward_implications":["If correct, protocol selection reduces to comparing residual times $\\bar\\tau$ measured at one large $P^*$: the perturbation with the smallest residual time is the preferred one across the whole valid frequency range.","For stochastic resetting, Eq. (5) gives the speedup directly from the unperturbed survival function, so any candidate perturbation can be screened against a resetting baseline without extra training runs.","The predicted speedup curve lets a practitioner choose not only which perturbation to use but also the perturbation interval that maximizes acceleration, without brute-force sweeps.","The experimental observation that $\\bar\\tau_P$ stays flat even for $P$ below the relaxation time suggests the practical range of validity may be wider than the formal bound $P>\\max(t_r,\\bar\\tau_P)$."],"supporting_citations":[{"why":"Supplies the response-theory identity that decomposes the perturbed mean first-passage time into a survival sum plus a residual term.","marker":"[38]"},{"why":"Introduced the re-initialization/shrink-and-perturb idea that the paper treats as a perturbation protocol.","marker":"[24]"},{"why":"Provides the shrink-and-perturb hyperparameters used in the experiments and documents that no general theory of when re-initialization helps existed.","marker":"[25]"},{"why":"Demonstrates warm restarts as a perturbation that accelerates convergence, a baseline the paper's framework generalizes.","marker":"[15]"},{"why":"Applied stochastic resetting to neural-network training, the empirical counterpart against which the paper's resetting prediction is benchmarked.","marker":"[26]"},{"why":"First-passage under restart theory that underlies the derivation of Eq. (5), the exact stochastic-resetting prediction from the survival function.","marker":"[46]"},{"why":"Provides the CIFAR-10 dataset on which the main quasi-steady-state and speedup experiments are run.","marker":"[42]"},{"why":"Defines the ResNet-18 architecture used for the main experiments.","marker":"[44]"},{"why":"Supplies the quasi-steady-state approximation concept from chemical kinetics that the paper adapts to neural-network training.","marker":"[51]"}],"fun_headline_variants":["One restart predicts the full speedup curve","First-passage math tunes training perturbations","Sample once, predict every speedup","Single frequency maps all training speedups","Quasi-steady state yields a speedup law"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole prediction relies on the full network state, not just test accuracy, reaching a quasi-steady state; the paper only checks the accuracy marginal, so if weight configurations keep evolving while accuracy looks stationary, P-independence can fail.","fun_headline_variants_meta":{"raw":{"variants":["One restart predicts the full speedup curve","First-passage math tunes training perturbations","Sample once, predict every speedup","Single frequency maps all training speedups","Quasi-steady state yields a speedup law"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1447,"prompt_tokens":983,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":599,"tokens_out":464,"duration_ms":5206,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:26:12.532272+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small network where all weights can be tracked, verify the full-state quasi-steady-state directly; then apply a perturbation that changes weights but leaves test accuracy nearly unchanged. If the residual time $\\bar\\tau_P$ varies with $P$ while the accuracy-marginal Kolmogorov–Smirnov test still says quasi-steady-state holds, the central claim is false.","supporting_citations":[{"cited_title":"Universal Linear Response of First-Passage Kinetics: A Framework for Prediction and Inference","cited_arxiv_id":"2410.16129","evidence_quote":"Supplies the response-theory identity that decomposes the perturbed mean first-passage time into a survival sum plus a residual term."},{"cited_title":"Ash and R","cited_arxiv_id":null,"evidence_quote":"Introduced the re-initialization/shrink-and-perturb idea that the paper treats as a perturbation protocol."},{"cited_title":"Zaidi, T","cited_arxiv_id":null,"evidence_quote":"Provides the shrink-and-perturb hyperparameters used in the experiments and documents that no general theory of when re-initialization helps existed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Applied stochastic resetting to neural-network training, the empirical counterpart against which the paper's resetting prediction is benchmarked."},{"cited_title":"Pal and S","cited_arxiv_id":null,"evidence_quote":"First-passage under restart theory that underlies the derivation of Eq. (5), the exact stochastic-resetting prediction from the survival function."},{"cited_title":"Krizhevsky, G","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10 dataset on which the main quasi-steady-state and speedup experiments are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ResNet-18 architecture used for the main experiments."},{"cited_title":"Nitzan,Chemical dynamics in condensed phases: relaxation, transfer, and reactions in condensed molecular systems (Oxford university press, 2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the quasi-steady-state approximation concept from chemical kinetics that the paper adapts to neural-network training."}],"review_version":1}