{"id":"06092bb5-fbb7-4285-b5e0-1b212a9809aa","arxiv_id":"2505.13900","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Deep network training exhibits an early sensitive 'chaos' phase and a later stable 'cone' phase in function space, separated by an inflection point.","lead":"Neural network training appears to switch from an early chaotic phase, where tiny parameter changes cause large divergence, to a stable phase where the network's function-space trajectory stays inside a narrow cone. The paper offers an interval-wise way to look at training dynamics, which could help decide when to intervene during training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Chaos Effect experiment depends on an unverified 'same SGD noise' assumption; without a seed or code, divergence could be independent sampling noise rather than perturbation sensitivity.","rationale":"I agree with the reader that the synchronization assumption is the weakest link. Without it, the central 'chaotic phase' claim is unsecured, and the paper does not provide the mechanism or artifacts needed to verify it. I do not see an internal inconsistency that would warrant rejection; the conditional verdict is appropriate. I considered the time-window confound from varying t1−t0, but the paper's reported condition 't0≤2500 and t1>2500' is more specific than a pure horizon effect, so synchronization is the more fundamental threat. No change to the reader's verdict.","tokens_in":10246,"tokens_out":17115,"duration_ms":168754,"concrete_test":"Run a controlled replication: (1) fix a seed for initialization, minibatch ordering, and augmentation; (2) at t0, perturb one copy by ε; (3) train both to t1 and measure D, B, and C. Then repeat with two independent SGD noise streams but no perturbation (spawning baseline). If the no-perturbation baseline yields divergence comparable to the ε-synchronized run, the Chaos Effect is not evidence of sensitivity to ε. Releasing the code and exact seed would also settle whether the published runs were actually synchronized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4's Chaos Effect compares θt1 and θ′t1 after injecting ε at t0. The design requires that the two runs differ only by this ε: identical initialization, same minibatch order, same augmentation randomness. The paper states 'the same stochastic gradient noise are applied' (Figure 2 caption) but gives no mechanism, seed, or code. If the runs actually use different data orders or augmentation randomness, the observed divergence, loss barriers, and disagreement rates (Figure 3) can arise from independent SGD noise even with ε=0. This would sever the link between the perturbation and the divergence, removing the evidence for the claimed chaotic early phase. The two-phase claim has two pillars, and the Chaos Effect is the only direct probe of early-time sensitivity; the Cone Effect alone does not establish a chaotic regime.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an 'interval-wise' analysis of neural network training dynamics, comparing network states at two different times rather than at isolated checkpoints. It reports two empirical phenomena on VGG-16 and ResNet-20 for CIFAR-10: the 'Chaos Effect,' in which a tiny parameter perturbation injected early in training leads to large later divergence in parameters, loss barriers, and test disagreement, while the same perturbation injected later has little effect; and the 'Cone Effect,' in which the empirical Neural Tangent Kernel (eNTK) continues to evolve after an early inflection point but remains confined to a narrow angular region. The authors interpret these observations as evidence of a two-phase learning process: an early chaotic, highly sensitive phase followed by a stable but still nonlinear refinement phase. They also report a switching experiment intended to show that the second phase provides generalization advantages over a purely lazy/linearized regime. The paper is explicitly empirical and defers theory to future work.","tokens_in":10404,"tokens_out":5483,"duration_ms":54577,"significance":"If the observations are reproducible and robust, the interval-wise perspective is a useful addition to the literature on training dynamics, complementing existing point-wise analyses of sharpness, Hessian spectra, and linear mode connectivity. The use of several independent observables (parameter dissimilarity, loss barrier, disagreement rate, kernel distance) is a genuine strength, as is the explicit statement of the two-phase conjecture. However, the paper is entirely empirical and provides no code, no seeds, and no error bars, which is a serious reproducibility gap for a study whose central claims are quantitative (inflection points, plateau levels). The reported effects are descriptive rather than predictive; no parameter-free derivation or falsifiable quantitative prediction is offered. With the requested controls and repeated-seed evidence, the paper could make a solid empirical contribution.","major_comments":[{"comment":"The claim that the two runs use 'the same stochastic gradient noise' is not operationalized. The manuscript does not state how synchronization is achieved (e.g., identical random seed, fixed minibatch order, fixed augmentation sequence), and no zero-perturbation control is reported. If the two runs use different data orders or augmentation randomness, the divergence, loss barriers, and disagreement rates in Figure 3 could arise from independent SGD noise even when epsilon = 0, which would sever the link between the perturbation and the observed divergence. Please specify the synchronization mechanism, provide seeds or code, and report an epsilon = 0 baseline for the chaos-effect experiment.","section":"Section 4, Experimental Design and Figure 2"},{"comment":"All results appear to come from a single training run per architecture. No seed repetitions, confidence intervals, or error bars are reported, so the claimed inflection points (e.g., '2500 iteration' for VGG-16 and '100-500 iteration' for ResNet-20) and the cone-effect plateaus cannot be distinguished from run-to-run variability. Please repeat the experiments with multiple seeds and report means and ranges, or clearly state that the results are single-run and weaken the corresponding claims.","section":"Section 3, Main Experimental Setup, and Figures 3-6"},{"comment":"The switching experiment theta_std->lin(T; t) confounds the switching time t with the amount of standard training: later switching means more standard SGD steps and fewer linearized steps. The observed increase of test accuracy with t may simply reflect longer standard training rather than any benefit of the cone effect. A proper control is needed, such as comparing against standard training for t iterations only, or matching the loss or accuracy at the switching point, or using a fixed total budget with different splits. As presented, the claim that the cone effect provides advantages over the lazy regime is not established.","section":"Section 5, 'The Non-linear Advantages of the Cone Effect', Figure 6"},{"comment":"The term 'chaotic' is used as a dynamical-systems claim, but the evidence is limited to divergence after one fixed perturbation norm (||epsilon|| = 10^-7). Sensitive dependence on initial conditions is not quantified: there is no analysis of how divergence scales with epsilon, no divergence-rate or doubling-time measurement, and no saturation analysis. Since the paper itself labels the transition a 'conjecture,' please either add such a scaling analysis or consistently phrase the finding as 'sensitivity to perturbations' rather than 'chaotic dynamics,' so that the abstract's 'chaotic to stable' claim is precisely supported.","section":"Sections 1 and 4, 'The Chaos Effect'"}],"minor_comments":[{"comment":"The denominator should be ||vec(theta_i)||_2 * ||vec(theta_j)||_2; as written, the formula uses ||vec(theta_j)||_2 twice.","section":"Equation (1)"},{"comment":"The denominator should be ||H(theta_i)||_F * ||H(theta_j)||_F; as written, the subscript on the first norm is malformed.","section":"Equation (4)"},{"comment":"The notation f: D x R^p -> R with scalar output is inconsistent with a c-class classification problem and with the disagreement rate in Equation (6), which compares predicted labels. Please clarify whether f returns logits, probabilities, or argmax predictions, or introduce a separate prediction function.","section":"Section 3, Basic Notations"},{"comment":"The axes of Figure 3 are said to be in iterations, but the checkpoint interval k used to sample {theta_t} is not specified, nor is the total number of checkpoints. Please state these values.","section":"Figure 3 and Section 3"},{"comment":"The sentence 'the neural networks the neural network continues to evolve' contains a duplicated phrase; please correct.","section":"Section 5, Key Takeaways"},{"comment":"There are minor wording issues, including 'the inflection point severs as a hallmark' (should be 'serves'), and inconsistent use of 'non-chaotic' versus 'nonchaotic.' Please copyedit.","section":"Throughout"},{"comment":"No code or data availability statement is provided. For an empirical paper, a statement on whether code and training configurations will be released is important.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper is an extended version of a workshop paper, and the relationship to prior spawning and linear-mode-connectivity work is acknowledged but not crisply delineated in terms of novelty. The main technical risk is the unverified synchronization of SGD noise in the chaos-effect experiment; if the authors cannot provide the epsilon = 0 control and seed details, the central claim would be substantially weakened. The switching-experiment confound is also fixable but requires additional experiments. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, readable empirical paper that reframes an already documented two-phase picture into an interval-wise language and names two effects—Chaos and Cone. The Cone Effect, in particular, is a nice visual and quantitative description: after an early phase, the eNTK keeps changing but stays in a narrow angular cone, and the authors show that switching to linearized training later costs performance. That switching result, if it survives deconfounding, is a genuine point against the lazy-regime view of late training.\n\nThe new bits are real but modest. The fast-to-slow eNTK transition, critical periods, and spawning-induced divergence are all in the cited literature. What is new is the explicit interval-wise matrix view and the angular-cone characterization. The paper is honest about being empirical, and the limitations note is straightforward.\n\nWhere I worry: the Chaos Effect experiment depends on two runs sharing the same SGD noise (minibatch order, augmentation). The paper asserts this but gives no seed or code. Without that, the divergence, loss barriers, and disagreement rates could just be independent-run noise, and the 'chaotic sensitivity' conclusion loses its footing. This is the load-bearing assumption behind the paper's strongest claim, so it needs either code, seeds, or a control with ε=0. That said, the two-phase story does not rest solely on this experiment: the kernel-distance matrices in Figure 4 and the cone-effect plots are single-run observables that show a clear transition. So the paper's central qualitative claim is plausible and likely correct; the specific 'chaos' label is the weakest part.\n\nSecond soft spot: the switching experiment (Figure 6) compares standard-then-linearized training at different switching times. Test accuracy increases with t, but that could simply be because more epochs of standard training happen before the switch. There is no baseline of continued standard training to separate 'cone effect advantage' from 'more real training.' The authors should add that control.\n\nAlso, all measurements are single runs without error bars. For a paper making temporal-phase claims, that is a minor but real issue—the inflection points are read off one trajectory.\n\nNet: this is a useful paper for people who study training dynamics empirically. It deserves peer review, and a careful referee can push for the synchronization controls, a spawning/ε=0 baseline, and code. I wouldn't cite it in its current form, but I'd be interested in the revised version.","headline":"A plausible interval-wise reframing of a known two-phase picture, with two named effects that need tighter experimental evidence before they carry much weight.","tokens_in":10883,"tokens_out":1953,"would_cite":false,"duration_ms":18663,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural network training passes through an inflection point: before it, tiny parameter perturbations cause large functional divergence (the Chaos Effect), and after it, the network's kernel evolution is confined to a narrow cone (the Cone…","keywords":["two-phase learning dynamics","chaos effect","cone effect","empirical neural tangent kernel","phase transition in training","perturbation sensitivity","loss barriers","linear mode connectivity"],"falsifier":"Train two networks with identical initialization but deliberately different random seeds for mini-batch order and augmentation while keeping everything else fixed, and measure the loss barrier and disagreement rate at the same $t_0$ and $t_1$ as in the paper; if the pre-inflection divergence is similar in magnitude to the synchronized-perturbation case, the Chaos Effect cannot be attributed to perturbation sensitivity. Conversely, if the same experiment uses a perturbation far larger than $10^{-7}$ after the inflection point and divergence remains tiny, the claimed stability of the second phase would be supported.","tokens_in":10074,"feed_emoji":"🌀","tokens_out":7533,"duration_ms":60158,"temperature":0.7,"pith_summary":"The paper tries to establish that deep network training is not one continuous process but two qualitatively different phases separated by an inflection point. Using an interval-wise analysis that compares network states across time windows rather than at isolated checkpoints, it reports two phenomena: the Chaos Effect, in which a tiny parameter perturbation injected early in training produces large loss barriers and functional disagreement later, and the Cone Effect, in which the empirical neural tangent kernel keeps changing after the inflection point but only within a narrow angular cone in function space. If correct, training has a critical period after which the model enters a constrained refinement regime that is still nonlinear and that linear (lazy) training cannot reproduce. The paper is an empirical study on VGG-16 and ResNet-20 on CIFAR-10, and it offers this two-phase picture as a structural description of how exploration gives way to refinement.","feed_headline":"Neural nets train in two phases: chaos, then a cone","feed_subtitle":"Tiny early perturbations derail training; later, kernel evolution narrows to a cone that beats lazy training.","key_machinery":"The machinery is an interval-wise analysis framework: instead of measuring properties at single checkpoints, the paper compares pairs of time points along the trajectory using four pairwise matrices: parameter dissimilarity $C$ (cosine distance between flattened parameter vectors), kernel distance $S$ (cosine distance between empirical neural tangent kernel matrices $H(\\theta)$), loss barriers $B$ (maximized test-loss increase along the linear interpolation between two models), and the disagreement rate $D$ (fraction of test inputs classified differently). The eNTK matrix, with entries $H(\\theta)_{i,j}$ equal to the inner product of the gradients of the network outputs on inputs $i$ and $j$, is the central object for the Cone Effect, because the claim that 'the kernel continues to change but is trapped in a tight angular region' is exactly a statement about the trajectory of $H(\\theta)$ in matrix space. The perturbation experiment for the Chaos Effect rests on comparing two runs that are designed to be identical apart from the injected $\\epsilon$.","core_discovery":"The central discovery is a two-phase structure in learning dynamics, demonstrated by two interval-wise measurements. First, when two identically initialized networks are trained with the same stochastic gradient noise and one receives a parameter perturbation of norm $10^{-7}$ at time $t_0$, the divergence measured later at $t_1$ depends sharply on whether $t_0$ lies before or after an inflection point (around iteration 2500 for VGG-16 and roughly iterations 100–500 for ResNet-20): perturbations before this point lead to substantial loss barriers and high disagreement rates, while perturbations after it leave the trajectories close. Second, the kernel distance between eNTK matrices at different times shows that in the early phase the kernel evolves chaotically, while in the second phase the kernel trajectory is confined to a narrow cone around a reference kernel: the distance to the reference grows then stabilizes, and the distance between adjacent iterates is bounded by the same value for different step sizes $\\mathrm{d}t$. The paper also shows that switching from standard training to linearized (lazy) training at later times yields better test performance, indicating that the confined second phase still contributes nonlinear refinement beyond the lazy regime.","pith_inferences":["A testable extension: the plateau value of the kernel distance in the second phase may correlate with final generalization, so measuring this 'cone width' across datasets and hyperparameters could turn the cone effect into a predictive diagnostic.","The inflection point may coincide with the progressive-sharpening or edge-of-stability transition studied in other work; the paper does not test this alignment, but if it holds, the chaos-cone picture would unify with sharpness-based accounts of training.","Varying the perturbation norm $\\epsilon$ across orders of magnitude before the inflection point would reveal whether the early regime is truly chaotic (exponential divergence) or merely high-gain but linear, a distinction the paper does not address.","Running the perturbation study with explicitly independent SGD noise would isolate whether the effect reflects sensitivity to initial conditions or merely noise-induced divergence, directly testing the synchronization assumption."],"forward_implications":["Training has a critical period: interventions such as data corruption, learning-rate changes, or weight perturbations early in training can have outsized and possibly irreversible effects, while the same interventions later are comparatively benign.","The second phase is not a lazy regime: linearized training with a frozen kernel cannot reproduce the final model's performance, so analyses that treat late training as linear miss a real part of the learning.","The inflection point is a reproducible landmark: the same time point appears in the parameter dissimilarity, loss barrier, disagreement rate, and eNTK evolution, giving a concrete coordinate for where the regime change happens.","The two-phase pattern holds across architectures: both VGG-16 and ResNet-20 on CIFAR-10 show the same qualitative transition, suggesting the phenomenon is not an artifact of one model family.","The eNTK continues to evolve after the transition but only along a constrained angular direction, so tracking the kernel's angular position rather than its magnitude can indicate when the model has entered the refinement phase."],"supporting_citations":[{"why":"Supplies the kernel distance measure and the deep-versus-kernel learning perspective that the Cone Effect extends.","marker":"[11]"},{"why":"Supplies the loss barrier measure and the spawning method that the Chaos Effect builds on.","marker":"[12]"},{"why":"Supplies the parameter dissimilarity (cosine directionality) measure used to locate the inflection point.","marker":"[26]"},{"why":"Provides the critical learning period finding that motivates the two-phase claim.","marker":"[1]"},{"why":"Documents progressive sharpening and edge-of-stability, the two-phase sharpness evidence the paper aligns with.","marker":"[7]"},{"why":"Justifies the disagreement rate as an approximation of test error, validating the Chaos Effect's functional divergence measure.","marker":"[19]"},{"why":"Defines the neural tangent kernel framework underlying the eNTK and the lazy-regime baseline.","marker":"[17]"},{"why":"Provides the layerwise linear feature connectivity and spawning analysis behind the perturbation protocol.","marker":"[35]"}],"fun_headline_variants":["Neural nets: chaos first, then a cone","Two-phase learning: chaos, then a cone","Chaotic start, cone-shaped finish in neural nets","Early chaos, later cone: the two-phase training","Neural training shifts from chaos to a cone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The chaos-effect result assumes that the two training runs are identical except for the injected perturbation, meaning the same mini-batch order, data augmentation, and stochastic gradient noise, yet the paper gives no mechanism or seed that guarantees this synchronization; if the noise is not actually identical, the observed divergence could come from independent SGD noise rather than sensitivity to the perturbation.","fun_headline_variants_meta":{"raw":{"variants":["Neural nets: chaos first, then a cone","Two-phase learning: chaos, then a cone","Chaotic start, cone-shaped finish in neural nets","Early chaos, later cone: the two-phase training","Neural training shifts from chaos to a cone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1435,"prompt_tokens":972,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":588,"tokens_out":463,"duration_ms":4826,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:07:53.805166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two networks with identical initialization but deliberately different random seeds for mini-batch order and augmentation while keeping everything else fixed, and measure the loss barrier and disagreement rate at the same $t_0$ and $t_1$ as in the paper; if the pre-inflection divergence is similar in magnitude to the synchronized-perturbation case, the Chaos Effect cannot be attributed to perturbation sensitivity. Conversely, if the same experiment uses a perturbation far larger than $10^{-7}$ after the inflection point and divergence remains tiny, the claimed stability of the second phase would be supported.","supporting_citations":[{"cited_title":"The directionality of optimization trajectories in neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the parameter dissimilarity (cosine directionality) measure used to locate the inflection point."},{"cited_title":"Assessing generalization of SGD via disagreement","cited_arxiv_id":null,"evidence_quote":"Justifies the disagreement rate as an approximation of test error, validating the Chaos Effect's functional divergence measure."},{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the neural tangent kernel framework underlying the eNTK and the lazy-regime baseline."}],"review_version":1}