{"id":"7a355538-3749-4263-b87f-01a2b659e334","arxiv_id":"2605.29152","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Trained ResNets on CIFAR-10 retain measurable dependence on initialization scale under low-LR SGD (26.5 pp test accuracy spread) but not under Adam, indicating that practical inductive bias is shaped by the forgetting time scale of the optimizer and regularizers.","lead":"The paper defines initialization memory as the dependence of a trained network's validation performance on the scale of its random starting weights. Experiments on ResNets for CIFAR-10 show that low-learning-rate SGD preserves this memory while Adam largely erases it, so the effective inductive bias is the architecture prior after training dynamics filter it.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Test accuracy spread across init scales may stem from uncontrolled trajectory differences rather than retained initialization memory","rationale":"The reader's weakest assumption already isolates the precise point where the experimental design must do more work to support the mechanistic interpretation; no stronger internal inconsistency is visible from the given evidence.","tokens_in":1767,"tokens_out":327,"duration_ms":16236,"concrete_test":"For the low-LR SGD runs, compute final ||W - W_init||_2 for each init scale; if this distance varies monotonically with scale while test accuracy also varies, recompute the accuracy spread after re-normalizing each run to the same final norm (via post-hoc scaling or early stopping at matched norm); if the spread collapses, the effect is not memory but norm-dependent regularization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim equates observed test-accuracy variation (26.5 pp on ResNet-9 at ≥99.5 % train acc) with \"initialization memory\" that survives low-LR SGD but is erased by Adam or explicit L2. For this isolation to hold, high train accuracy plus fixed epoch count must suffice to rule out other systematic differences in the optimization trajectory (effective step-size distribution, implicit regularization strength, basin geometry, or final weight norms) that are merely correlated with init scale. The reported controls do not address these; extending epochs leaves the spread unchanged but does not constrain the other factors. If any of them covaries with init scale, the variation cannot be attributed specifically to retained memory of the random draw.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that trained deep networks retain a measurable dependence on the scale of their random initialization ('initialization memory'), which survives low-learning-rate SGD but is erased by Adam-family optimizers or explicit L2 regularization. This is demonstrated via controlled CIFAR-10 ResNet experiments showing a 26.5 percentage point spread in test accuracy across initialization scales at >=99.5% training accuracy; the spread persists even after extending training to 5000 epochs. The authors interpret the results as evidence that practical inductive bias is the architectural prior filtered by the forgetting dynamics of the training pipeline, with the same regularizers that aid generalization also erasing initialization memory.","tokens_in":1947,"tokens_out":508,"duration_ms":23065,"significance":"If the reported accuracy spreads can be isolated to retained initialization memory rather than correlated differences in optimization trajectories, the work offers a useful empirical lens on how training dynamics shape effective priors. The controlled regime comparisons (low-LR SGD vs. Adam vs. L2) and the observation that extended epochs do not close the gap provide concrete, falsifiable distinctions between regimes. The absence of free parameters or fitted models in the core measurements is a strength of the empirical design.","major_comments":[{"comment":"Abstract and the CIFAR-10 ResNet-9 experiments: the central attribution of the 26.5 pp test-accuracy spread to retained 'initialization memory' (rather than other init-scale-dependent trajectory factors such as effective step-size distribution, basin geometry, or final weight norms) is load-bearing for the forgetting-time interpretation, yet the reported controls (high training accuracy plus extension to 5000 epochs) do not constrain these alternatives. High train accuracy plus fixed epoch count is insufficient to isolate the claimed mechanism.","section":"Abstract"},{"comment":"The manuscript reports quantitative separation under controlled setups but provides no details on the number of independent runs, error bars, or exact hyperparameter sweep ranges for the 26.5 pp spread. Without these, it is impossible to assess whether the observed variation is statistically robust or sensitive to minor implementation choices.","section":"Abstract"}],"minor_comments":[{"comment":"Notation for 'initialization memory' is introduced without a formal definition or equation; a precise mathematical statement of the dependence being measured would improve clarity.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful reading and constructive feedback. We respond to each major comment below.","responses":[{"response":"We define initialization memory operationally as the dependence of the validation-selected predictor on initialization scale under a fixed training procedure. The reported experiments show that this dependence survives low-LR SGD even after ≥99.5% training accuracy is reached and after training is extended to 5000 epochs, while the same dependence is erased under Adam or explicit L2 regularization. These regime contrasts are the primary evidence for the forgetting-time view. We agree that additional measurements (e.g., final weight norms across scales or basin geometry) would further constrain alternative explanations and will add a limitations paragraph discussing this point in the revision.","revision_made":"partial","referee_comment":"[Abstract] Abstract and the CIFAR-10 ResNet-9 experiments: the central attribution of the 26.5 pp test-accuracy spread to retained 'initialization memory' (rather than other init-scale-dependent trajectory factors such as effective step-size distribution, basin geometry, or final weight norms) is load-bearing for the forgetting-time interpretation, yet the reported controls (high training accuracy plus extension to 5000 epochs) do not constrain these alternatives. High train accuracy plus fixed epoch count is insufficient to isolate the claimed mechanism."},{"response":"We agree that these experimental details are necessary. The 26.5 pp figure is computed from multiple independent runs that vary only the initialization scale (different random seeds), and we will include the exact number of runs, standard-error bars, and the precise hyperparameter ranges used for all reported regimes in the revised manuscript and supplementary material.","revision_made":"yes","referee_comment":"[Abstract] The manuscript reports quantitative separation under controlled setups but provides no details on the number of independent runs, error bars, or exact hyperparameter sweep ranges for the 26.5 pp spread. Without these, it is impossible to assess whether the observed variation is statistically robust or sensitive to minor implementation choices."}],"tokens_in":1525,"tokens_out":411,"duration_ms":27944,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The central observation is that low-learning-rate SGD on ResNet-9 reaches near-perfect training accuracy on CIFAR-10 yet still shows a 26.5-point test-accuracy gap across different initialization scales, and that gap stays put even after 5000 epochs. Adam-family optimizers and explicit L2 largely remove the dependence. The paper frames this as initialization memory surviving or being forgotten on different time scales set by the optimizer and regularizer.\n\nWhat is new is the explicit measurement of how much the final predictor still depends on the random draw at initialization, plus the forgetting-time lens on inductive bias. The experiments cleanly separate regimes: low-LR SGD preserves the spread, while larger steps plus norm control or adaptive methods do not. That separation is useful to see in controlled setups.\n\nThe soft spot is that the spread could come from systematic differences in the optimization trajectory—effective step-size distribution, implicit regularization strength, or final weight norms—that happen to correlate with init scale rather than from literal retention of the initial weights. High train accuracy and fixed epoch count do not rule those out, and the abstract gives no run counts or error bars. The interpretation that the same regularizers that help generalization are exactly those that erase initialization memory therefore rests on an assumption that needs tighter controls.\n\nThis paper is for researchers who care about how optimization choices shape what a network actually learns beyond the architecture. It is worth sending to peer review because the empirical pattern is sharp enough to merit checking, even if the causal story needs more work to hold up.","headline":"Low-LR SGD keeps a 26-point test-acc spread across init scales on CIFAR ResNets even at 99.5% train acc, while Adam and L2 erase it, but the spread may trace to other trajectory differences rather than retained initialization memory.","tokens_in":2414,"tokens_out":412,"would_cite":false,"duration_ms":19776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"The inductive bias of a trained neural network is its architectural prior filtered by the forgetting dynamics of the training pipeline.","keywords":["initialization memory","inductive bias","neural network training","SGD","forgetting","regularization","ResNet","CIFAR-10"],"falsifier":"An experiment in which the accuracy spread across init scales disappears when other optimization factors are controlled while maintaining the same training accuracy.","tokens_in":2694,"feed_emoji":"","tokens_out":584,"duration_ms":29973,"temperature":0.7,"pith_summary":"Neural networks start with a prior induced by random initialization, but training changes how much of that survives. Experiments on ResNets for CIFAR-10 reveal that low-learning-rate SGD keeps strong dependence on initialization scale, producing up to 26.5 percentage point swings in test accuracy even when training accuracy exceeds 99.5 percent. Adam optimizers and explicit regularization largely remove this dependence by accelerating forgetting of the initial conditions. The result is that effective inductive bias is not architecture alone but architecture after the training process has filtered the starting point.","feed_headline":"SGD remembers initialization while Adam erases it","feed_subtitle":"Test accuracy varies by 26 points across init scales at 99.5% training accuracy, showing that regularization erases initial bias","key_machinery":"Initialization memory, the dependence of the validation-selected predictor on the scale of the random initialization; it quantifies how much initial bias survives training.","core_discovery":"In controlled experiments, initialization memory—the dependence of the final predictor on random initialization scale—persists under gradient-flow-like dynamics such as low-LR SGD but is erased on timescales set by stochastic effects, norm decay, or adaptive preconditioning; therefore the practical bias equals the architectural prior after filtering by forgetting dynamics, and regularizers improve generalization precisely by erasing initialization memory.","pith_inferences":["Initialization scale may need to be tuned differently depending on the optimizer used.","This forgetting view could explain why certain training choices improve generalization beyond what architecture alone predicts.","Extending training time does not necessarily increase forgetting if the regime preserves memory."],"forward_implications":["Low-learning-rate SGD interpolates yet retains initialization memory, leading to large test accuracy variation.","Adam-family methods erase the dependence on initialization scale.","Pairing larger learning rates with L2 norm control causes SGD to forget initialization.","The time scale of forgetting is governed by the size of explicit or implicit regularization."],"fun_headline_variants":["Low LR SGD retains init memory on ResNets unlike Adam","ResNet accuracy varies 26 points across init scales at low LR","Forgetting dynamics erase init bias under Adam and norm decay","SGD remembers initialization scale while Adam family erases it","Initialization memory survives gradient flow but not preconditioning"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Variation in test accuracy across initialization scales after high training accuracy isolates retained initialization memory rather than other differences in optimization.","fun_headline_variants_meta":{"raw":{"variants":["Low LR SGD retains init memory on ResNets unlike Adam","ResNet accuracy varies 26 points across init scales at low LR","Forgetting dynamics erase init bias under Adam and norm decay","SGD remembers initialization scale while Adam family erases it","Initialization memory survives gradient flow but not preconditioning"]},"model":"grok-4.3","cost_usd":0.004235,"raw_usage":{"total_tokens":2081,"prompt_tokens":720,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":42353000,"prompt_tokens_details":{"text_tokens":720,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1281,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":720,"tokens_out":80,"duration_ms":14972,"temperature":1.0,"reasoning_tokens":1281,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T13:18:32.567648+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment in which the accuracy spread across init scales disappears when other optimization factors are controlled while maintaining the same training accuracy.","supporting_citations":[],"review_version":1}