{"id":"d464642c-38b2-4fda-a3b8-9edc148e3b26","arxiv_id":"2606.00340","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Derives exact two-step GD dynamics for linear networks and proves optimal layer-wise learning rates are unequal at step one but equal thereafter, with a tractable surrogate loss under gradient approximation.","lead":"The paper derives exact closed-form expressions for gradients and test loss after one and two gradient descent steps in linear neural networks learning linear targets. This identifies an early-training regime where unequal layer-wise learning rates minimize loss initially but equal rates become optimal afterward.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Optimality of unequal LRs at step 1 is derived under a gradient approximation whose error bound does not guarantee preservation of the argmin","rationale":"The reader's weakest assumption correctly flags the gradient approximation as the critical unverified link; the concrete test directly checks whether that approximation preserves the optimality statement that constitutes the paper's strongest claim. If the argmins differ, the headline regime does not follow even inside the linear setting.","tokens_in":1646,"tokens_out":331,"duration_ms":16296,"concrete_test":"For the two-layer linear case with a fixed linear target, symbolically or numerically compute both the true one-step test loss and the surrogate loss as functions of the two layer learning rates; locate the argmin of each over a grid or via gradient descent on the rates and report whether the minimizing rate pairs coincide within 5% relative tolerance and whether the loss gap at those points exceeds the claimed approximation error.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim states that test loss is minimized by unequal layer-wise learning rates at the initial step (under the gradient approximation) while equal rates are optimal subsequently, and that the resulting surrogate loss has a 'tight, small' approximation error. The exact closed-forms are given only for the first two steps; the optimality conclusion therefore rests on the unstated assumption that the approximation error, even if small in value, does not shift the location of the minimum with respect to the learning-rate vector. No explicit Lipschitz or sensitivity bound relating the error to the difference in argmin is supplied in the abstract or described derivations.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper derives exact closed-form expressions for gradients and test loss after one and two steps of gradient descent in two- and three-layer linear neural networks learning linear targets. It characterizes optimal layer-wise learning-rate scaling under a gradient approximation for the first two steps, proves that updates with this approximation yield a tractable surrogate loss with tight small approximation error, and shows that unequal learning rates minimize test loss at the initial step while equal rates become optimal subsequently. Numerical experiments validate the theory; code is provided.","tokens_in":1794,"tokens_out":317,"duration_ms":14782,"significance":"If the results hold, this supplies precise early-training dynamics for linear networks and highlights the value of unequal layer-wise rates at initialization. The reproducible code and numerical validation are explicit strengths that support the claims.","major_comments":[{"comment":"Abstract: the claim that the surrogate loss has a 'tight, small approximation error' underpins the optimality conclusion for unequal rates at step 1, yet no Lipschitz or sensitivity bound is supplied showing that the error does not shift the argmin over the learning-rate vector.","section":"Abstract"},{"comment":"Abstract: the exact closed-forms and optimality statements are derived only for the first two steps under linear networks and linear targets; the paper provides no argument that the identified early-training regime (unequal then equal rates) survives beyond these restrictions.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. The comments highlight two areas where the abstract claims can be strengthened with additional rigor and clearer scoping. We address each point below and will revise the manuscript accordingly.","responses":[{"response":"We agree that a formal sensitivity analysis would strengthen the link between the approximation error and the optimality of unequal rates. The current manuscript demonstrates small error numerically and shows that the surrogate preserves the qualitative ordering of test loss, but does not supply an explicit Lipschitz or perturbation bound on the argmin. In the revision we will add a short sensitivity lemma bounding the change in the optimal learning-rate vector as a function of the approximation error, using the fact that the surrogate loss is quadratic in the rates under the linear-network setting.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that the surrogate loss has a 'tight, small approximation error' underpins the optimality conclusion for unequal rates at step 1, yet no Lipschitz or sensitivity bound is supplied showing that the error does not shift the argmin over the learning-rate vector."},{"response":"The paper deliberately restricts attention to the first two gradient steps in linear networks with linear targets precisely because this regime admits exact closed forms. We do not claim that the unequal-then-equal pattern extends to deeper networks, nonlinear activations, or later training phases; the contribution is the exact characterization and the resulting insight that layer-wise rates should be balanced after the initial step. In the revision we will modify the abstract and introduction to state the scope more explicitly and add a brief paragraph on the limitations and possible extensions.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the exact closed-forms and optimality statements are derived only for the first two steps under linear networks and linear targets; the paper provides no argument that the identified early-training regime (unequal then equal rates) survives beyond these restrictions."}],"tokens_in":1250,"tokens_out":419,"duration_ms":14996,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper works out closed-form expressions for the gradients and test loss after exactly one and two steps of gradient descent on two- and three-layer linear networks with linear targets. Those expressions are new and let them track layer-wise behavior without the usual infinite-width or mean-field assumptions.\n\nThey then introduce a gradient approximation to build a surrogate loss, prove the approximation error is small, and use the surrogate to conclude that unequal layer learning rates minimize the loss at step one while equal rates are optimal afterward. The numerical checks line up with the formulas.\n\nThe gap is that a small pointwise error in the surrogate does not automatically mean the location of its minimum over the learning-rate vector stays the same. No Lipschitz or sensitivity bound is given to control how the argmin can move, so the central optimality statement is not yet fully supported by the error analysis.\n\nEverything is restricted to linear networks, linear targets, and the first two steps. That keeps the math tractable but limits how far the practical advice travels.\n\nThe work is for theorists who want exact early-training calculations in the simplest non-trivial setting. The derivations are explicit and the code is public, so it is worth sending to referees even though the approximation step needs tighter justification.","headline":"Exact closed forms for the first two GD steps are the real contribution; the optimality claim for unequal rates rests on an approximation whose effect on the argmin is not bounded.","tokens_in":2254,"tokens_out":331,"would_cite":false,"duration_ms":17753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Test loss in linear neural networks is minimized by unequal learning rates across layers in the first gradient step, with equal rates optimal afterward.","keywords":["learning rate scaling","linear neural networks","gradient descent dynamics","layer-wise learning rates","surrogate loss","early training"],"falsifier":"Training a two-layer linear network on a linear target for two gradient steps and measuring whether the test loss after the first step is lower with unequal per-layer rates than with equal rates would confirm or refute the central claim.","tokens_in":2578,"feed_emoji":"","tokens_out":633,"duration_ms":19113,"temperature":0.7,"pith_summary":"The paper derives exact closed-form expressions for the gradients and test loss after one and two steps of gradient descent in two-layer and three-layer linear networks trained on linear targets. It shows that a gradient approximation permits different per-layer learning rates in the first step to achieve lower loss, while equal rates become optimal in the second step. This approximation produces a surrogate loss whose error is provably small and tight. The result matters because it isolates a distinct early-training regime where balancing layer-wise rates improves outcomes, as the experiments confirm.","feed_headline":"Unequal rates across layers cut first-step test loss in linear nets","feed_subtitle":"Two-step closed-form analysis shows equal rates become optimal afterward with a tight surrogate loss.","key_machinery":"Exact closed-form expressions for the gradients and test loss after one and two steps of gradient descent, which support the characterization of learning-rate scaling under the approximation.","core_discovery":"In two- and three-layer linear neural networks trained to learn linear target functions, the exact closed-form expressions for gradients and test loss after one and two steps of gradient descent show that optimal learning rates are unequal across layers at the initial step but equal in subsequent steps. Performing updates with the gradient approximation yields a tractable surrogate loss with a tight, small approximation error, enabling analysis of layer-wise scaling.","pith_inferences":["The two-step exact dynamics could be checked for repetition in three or more steps to see if the unequal-to-equal transition pattern persists.","The initial unequal-rate regime might be tested as a practical heuristic in models that are approximately linear near initialization.","The surrogate loss construction could be applied to study scaling in wider linear networks without changing the core approximation."],"forward_implications":["Unequal learning rates across layers reduce test loss more than equal rates do in the first step.","Equal learning rates become optimal from the second step onward.","The surrogate loss approximation has provably small error and supports further theoretical analysis of layer-wise rates.","Numerical experiments on two- and three-layer networks confirm the early-training regime where layer balancing matters."],"fun_headline_variants":["Unequal layer rates at first step in linear neural nets","Equal rates optimal after first step in linear nets","Two-step dynamics show layer-wise rate scaling","Exact analysis of initial rates in linear networks"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The closed-form derivations and optimality claims hold only for linear networks with linear target functions and are restricted to the first two gradient steps using the gradient approximation.","fun_headline_variants_meta":{"raw":{"variants":["Unequal layer rates at first step in linear neural nets","Equal rates optimal after first step in linear nets","Two-step dynamics show layer-wise rate scaling","Exact analysis of initial rates in linear networks"]},"model":"grok-4.3","cost_usd":0.005452,"raw_usage":{"total_tokens":2593,"prompt_tokens":609,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":54524500,"prompt_tokens_details":{"text_tokens":609,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1927,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":609,"tokens_out":57,"duration_ms":14092,"temperature":1.0,"reasoning_tokens":1927,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T23:04:18.205563+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Training a two-layer linear network on a linear target for two gradient steps and measuring whether the test loss after the first step is lower with unequal per-layer rates than with equal rates would confirm or refute the central claim.","supporting_citations":[],"review_version":1}