{"id":"82e8c113-c8bd-4924-962f-fd0ec2c24752","arxiv_id":"1908.07607","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"AutoOpt computes per-layer learning rates and momentums that minimize a quadratic approximation of the expected loss one step ahead, and tests the idea on small CNN classifiers.","lead":"This paper introduces AutoOpt, an algorithm that claims to set the learning rate and momentum automatically at every step of SGD, Adam, and AdaGrad by estimating a one-step optimal update. Tests on two small image datasets show that the automatic settings sometimes match the best manually tuned results, but a key derivation step is mathematically incomplete and AutoAdam performs much worse on CIFAR10.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim fails as written: Section 4 collapses the 2-vector b_t to a scalar under '\\hat H_t=\\bar H_t', and Table 2 shows AutoAdam on CIFAR10 is 11.4 points worse than manual tuning.","rationale":"The reader's weakest_assumption and my independent reading point to the same load-bearing spot: the Section 4 simplification of b_t. I agree with that identification. The formula for \\hat \\gamma_{Ot} is the core of Algorithm 1, and the simplification is only valid under \\hat H_t = \\bar H_t; but the paper itself documents a case where that assumption fails and the result is far from the best manual setting. The CIFAR10 AutoAdam row is an internal counterexample to the strongest claim, so this is not a disagreement with current consensus or a matter of taste. I would keep the reader's REJECT verdict; the 'no change needed' code is UNCHANGED because my stress-test does not alter the verdict. If the authors can repair Eq. (23) and show the corrected AutoAdam recovers the manual error, a revised submission with a properly specified algorithm could be reconsidered.","tokens_in":10512,"tokens_out":11452,"duration_ms":110212,"concrete_test":"Re-rank the Table 2 CIFAR10 AutoAdam column after re-implementing Algorithm 1 in two variants: (i) exactly as written, with \\hat b_t the scalar from Eq. (23), and (ii) with \\hat b_t kept as the 2-vector obtained by substituting \\hat H_t = \\bar H_t into Eq. (18), i.e., \\hat b_t = (\\hat V, \\hat V)^T (up to the sign and factor that a clean re-derivation of Eq. (16) fixes). If variant (ii) still gives test error near 47.8%, the Hessian mismatch is the load-bearing failure and the central claim is contradicted; if it recovers the 36.4% manual result, the stated equations are wrong and the method may be partially salvageable after correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AutoOpt automatically matches the best manually tuned setting for SGD, Adam, and AdaGrad. Two connected problems put that claim at risk. First, Eq. (18) defines b_t as a 2x1 vector, but Section 4 assumes \\hat H_t = \\bar H_t and then replaces b_t with the scalar \\hat b_t = (1/2)\\hat V(g_t|\\hat H_t) in Eq. (23). Substituting \\hat H_t = \\bar H_t into Eq. (18) leaves b_t = E{G_t^T \\hat H_t^{-1}(g_t-\\bar g_t)}, whose two entries are both V(g_t|\\hat H_t) when the current mini-batch is independent of the previous gradient estimator; the extra factor 1/2 is also not obtained from the expansion in Eq. (16). Consequently Eq. (25), \\hat \\gamma_{Ot} = \\hat A_t^{-1}\\hat b_t, is dimensionally undefined as written, or, under a charitable reading \\hat b_t = (\\hat V/2)1_2, it scales the optimizer step incorrectly. Second, even the empirical results contradict the claim: Table 2 reports CIFAR10 AutoAdam test error 47.8% versus 36.4% for the best manually tuned Adam, an 11.4-point gap. The paper attributes this to the discrepancy between Adam's diagonal Hessian estimator and the true Hessian, which is exactly the assumption used in Section 4. Since that assumption is the only link between the oracle and the practical procedure, the method's stated scope is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AutoOpt, a method to automatically and simultaneously set the learning rate and momentum of stochastic gradient optimizers (SGD, Adam, AdaGrad) at each step and per layer. The authors derive an 'oracle' optimal parameter vector by minimizing the expected quadratic approximation of the loss after a step, then estimate that oracle from current mini-batch statistics. They report experiments on MNIST and CIFAR10 CNN classifiers, comparing AutoOpt against an exhaustive manual search, and claim that the automatic settings match or beat the best manually tuned configurations. The central theoretical step is in Section 4, where the 2x1 vector b_t is simplified to a scalar variance term under the assumption that the optimizer's Hessian estimator equals the true Hessian.","tokens_in":10905,"tokens_out":7524,"duration_ms":275902,"significance":"If the central derivation were sound, automatic per-layer adjustment of learning rate and momentum would be a practically valuable contribution, and the experimental comparison against thousands of manual configurations with multiple seeds is a genuine strength. The idea of using the second-order statistics of the unbiased gradient estimator is appealing, and the per-layer adaptation shown in Figures 2-4 is informative. However, the derivation in Section 4 contains a load-bearing dimensional inconsistency, and the paper's own Table 2 contradicts the headline claim for one of the three optimizers on CIFAR10. As a result, the method's theoretical basis and its stated empirical scope are both unsupported in the present form.","major_comments":[{"comment":"Equation (18) defines b_t as a 2x1 vector, but after the assumption \\hat H_t = \\bar H_t the paper replaces it by the scalar \\hat b_t = (1/2)\\hat V(g_t|\\hat H_t) in Eq. (23). Substituting \\hat H_t = \\bar H_t into Eq. (18) gives b_t = E{G_t^T \\hat H_t^{-1}(g_t - \\bar g_t)}, whose two entries are V(g_t|\\hat H_t) and V(g_t|\\hat H_t) - E{\\hat g_{t-1}^T \\hat H_t^{-1}(g_t - \\bar g_t)}; it is not a scalar. Consequently Eq. (25), \\hat \\gamma_{Ot} = \\hat A_t^{-1} \\hat b_t, is dimensionally undefined as written: \\hat A_t^{-1} is 2x2 and \\hat b_t is a scalar. This is not a presentational slip but the core step that connects the oracle to the practical estimator.","section":"§4, Eqs. (18)–(25)"},{"comment":"The simplification b_t = (1/2)V(g_t|\\hat H_t) is not derived from Eqs. (16)–(18). A direct expansion of the second component of b_t under \\hat H_t = \\bar H_t gives E{g_t^T \\hat H_t^{-1}(g_t - \\bar g_t)} - E{\\hat g_{t-1}^T \\hat H_t^{-1}(g_t - \\bar g_t)}, which equals V(g_t|\\hat H_t) only if E{\\hat g_{t-1}^T \\hat H_t^{-1}(g_t - \\bar g_t)} = 0; this independence condition is not stated or proved. The factor 1/2 also does not follow from the expansion in Eq. (16): the linear term in γ from Eq. (16) has coefficient -b_t, and minimizing 1/2 γ^T A_t γ - γ^T b_t gives γ = A_t^{-1} b_t, not A_t^{-1}(b_t/2). The factor 1/2 therefore appears without justification.","section":"§4, Eq. (20)"},{"comment":"The central claim that 'the results match the performance of the best settings obtained through an exhaustive search' is contradicted by the paper's own CIFAR10 results for AutoAdam: test error 47.8% versus 36.4% for manually tuned Adam, and train error 42.6% versus 27.2%. The text attributes this gap to the discrepancy between Adam's diagonal Hessian estimator and the true Hessian, but that discrepancy is exactly the assumption used in Section 4 to derive the oracle. Since the empirical failure occurs precisely where the derivation's key assumption fails, the method's stated scope ('in all cases the proposed method automatically attains the lowest, or comparable test and train errors') is unsupported.","section":"§5, Table 2 and Abstract/Conclusions"},{"comment":"The proposed method is described as automatic, but the exponentially weighted moving average in Eq. (26) introduces a smoothing factor υ that is never specified or tuned in the experiments. No value for υ is given in Algorithm 1 or in Section 5, and no sensitivity analysis is reported. Without a stated value or default, the method still depends on at least one hyperparameter, and the experiments are not fully reproducible as described.","section":"§4, Eq. (26) and §5"}],"minor_comments":[{"comment":"The sentence 'The results match the performance of the best settings obtained through an exhaustive search and therefore, removes the need for a tedious manual tuning' has a subject-verb agreement error ('results ... removes') that should be corrected.","section":"Abstract and §1"},{"comment":"The estimator \\hat A_t in Eq. (24) replaces the expectation in Eq. (17) by a single sample product G_t^T \\hat H_t^{-1} G_t and is then inverted at every step. The paper does not discuss what happens when the two columns of G_t are nearly collinear or when \\hat A_t is ill-conditioned; a regularization or averaging mechanism would make the algorithm more robust.","section":"§3, Eq. (16) and §4, Eq. (24)"},{"comment":"The caption states that the automatic counterparts 'achieve comparable, or better results' than manual tuning, but the CIFAR10 AutoAdam row in the same table shows a more than 10% degradation. The caption should be qualified to reflect the actual entries in the table.","section":"§5, Table 2 caption"},{"comment":"The complexity claim that Algorithm 1 has the same time complexity as back-propagation relies on the per-observation gradients being available at every layer. In standard deep-learning frameworks these per-sample gradients are not computed during a normal backward pass, so the practical overhead may be larger than stated unless the implementation accumulates them explicitly.","section":"Appendix B"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth a look: minimizing a quadratic surrogate to set learning rate and momentum simultaneously, then estimating the oracle from mini-batch statistics, is a legitimate extension of Schaul et al.'s learning-rate-only method. The paper does useful work in deriving an unbiased estimator of the variance term, providing a time-complexity analysis, and showing that AutoSGD and AutoAdaGrad roughly match manual tuning on small CNN tests. That is credit earned.\n\nThe soft spots are load-bearing, though. The stress-test note is correct: Eq. (18) defines b_t as a 2x1 vector, but Section 4 assumes \\hat H_t = \\bar H_t and then replaces it with the scalar \\hat b_t = (1/2) V(g_t | \\hat H_t) in Eq. (23). Eq. (25) is dimensionally undefined as written. Even a charitable reading that makes b_t a vector of two equal variance terms changes the step scaling, and the extra 1/2 factor does not follow from Eq. (16). This is not a typo; it is the bridge between the oracle and the practical algorithm.\n\nThe empirical claim also overreaches. Table 2 shows AutoAdam on CIFAR10 at 47.8% test error versus 36.4% for manual tuning. The authors' own explanation is exactly the assumption used in Section 4: Adam's diagonal Hessian estimator differs from the true Hessian. So the one case where the assumption is most doubtful is also the case where the method clearly fails. The paper's claim that \"in all cases\" the method matches manual tuning is false.\n\nNovelty is a bit overstated. The paper says there is no method to adjust momentum automatically, but YellowFin already tunes momentum and learning rate simultaneously; the reader flagged this, and it is accurate. Self-citation is not the issue here; the issue is claiming a first when there is relevant prior art.\n\nI agree with the reader's verdict. The derivation error is central, and the experimental evidence does not support the strongest claim. But this is not a crank paper. The framework is sensible, the estimation idea is reusable, and the SGD/AdaGrad results suggest the approach has value in restricted settings. I would send it to a serious referee, not desk-reject it, because the flaws are concrete and fixable, and the underlying idea deserves scrutiny. I would only accept after major revision, with Section 4 rewritten and the scope narrowed.","headline":"AutoOpt is a plausible extension of Schaul et al. to momentum, but the Section 4 derivation collapses a 2-vector into a scalar under the very assumption the experiments violate, and the CIFAR10 Adam gap is real.","tokens_in":11393,"tokens_out":1374,"would_cite":false,"duration_ms":525565,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper derives per-step optimal learning rate and momentum from mini-batch statistics and reports parity with exhaustive search.","keywords":["stochastic gradient descent","learning rate","momentum","hyperparameter tuning","automatic adjustment","CNN classification","gradient variance estimation","oracle estimator"],"falsifier":"Re-run the CIFAR10 experiment with AutoAdam using the paper's architecture: if the test error again lands near the reported 47.8% while the exhaustive-search optimum stays near 36.4%, the claim that AutoOpt matches the best manual settings fails for Adam.","tokens_in":10282,"feed_emoji":"🎯","tokens_out":8258,"duration_ms":75343,"temperature":0.7,"pith_summary":"AutoOpt is a method for automatically setting the learning rate and momentum of SGD-style optimizers on every update step, so that a practitioner runs one configuration instead of a search over hundreds. The paper derives the values that minimize the expected one-step loss of a quadratic model, estimates those values from the mini-batch's gradient statistics, and applies the scheme to SGD, Adam, and AdaGrad on CNN classifiers. Its central claim is that this automatic tuning matches the best train and test errors obtained by exhaustive manual search, eliminating the need for manual tuning. The paper reports one exception in its own tables: AutoAdam on CIFAR10 reaches 47.8% test error versus 36.4% for the best manually tuned Adam, which it attributes to the Hessian assumption the method relies on.","feed_headline":"AutoOpt matches exhaustive tuning without the manual search","feed_subtitle":"A single run adapts learning rate and momentum per layer and per step, replacing tedious grid search.","key_machinery":"The load-bearing identity is the reparameterization of the gradient estimator as $\\hat g_t = g_t - G_t\\gamma$, with $G_t = [g_t,\\ g_t-\\hat g_{t-1}]$ and $\\gamma = [\\gamma_1,\\gamma_2]^T$ linked to the learning rate and momentum by $\\alpha = 1-\\gamma_1$ and $\\beta = \\gamma_2/(1-\\gamma_1)$. Plugging this into a quadratic Taylor model of the loss produces an expected next-step loss whose minimizer is $\\gamma_t^O = A_t^{-1}b_t$, where $A_t$ is the expected Hessian-weighted Gram matrix of $G_t$ and $b_t$ is a vector of covariance and bias terms. The simplification $b_t = \\tfrac12 V(g_t|\\hat H_t)$—a scalar variance term—holds only when the optimizer's Hessian estimator equals the true Hessian; the practical estimator uses the unbiased per-example variance $\\hat V(g_t|\\hat H_t)$ and the per-step sample matrix $\\hat A_t = G_t^T\\hat H_t^{-1}G_t$, followed by exponential smoothing. These components convert an oracle quantity into a per-step, per-layer schedule.","core_discovery":"The paper's central claim is that the optimal learning rate and momentum at iteration $t$ can be written as $\\gamma_t^O = A_t^{-1} b_t$, where the $2\\times 2$ matrix $A_t$ and vector $b_t$ come from the Hessian-weighted covariance of the gradient estimator and its history. Because the true gradient is unknown, this is called the oracle solution; the paper's practical contribution is an estimator $\\hat\\gamma_t^O$ that replaces expectations with the current mini-batch, estimates the variance term $V(g_t|\\hat H_t)$ from per-example gradients, and smooths the result with an exponential moving average. The claim is that plugging this estimated optimal vector into the update rule for SGD, Adam, or AdaGrad yields the same or comparable train and test error as the best configuration found by exhaustive search, in one run instead of hundreds. The paper reports this for MNIST and CIFAR10 CNN experiments, with the notable exception that AutoAdam on CIFAR10 reaches 47.8% test error versus 36.4% for manually tuned Adam, a gap the paper attributes to the difference between the unknown Hessian and Adam's diagonal estimator.","pith_inferences":["The failure pattern on CIFAR10 AutoAdam suggests that the biggest gains from this approach will come from optimizers whose Hessian estimates track true curvature; a testable extension is to replace Adam's diagonal curvature estimate with a low-rank or block-diagonal correction and see whether the 10-point gap closes.","Because AutoOpt produces per-layer schedules, it implies that a single global learning rate is a real bottleneck; one could compare layer-wise AutoOpt schedules against the best global schedule and quantify the gain.","The method's estimator needs per-example gradients, so in models where storing those is prohibitive, an approximate variance estimator based on gradient chunks or running statistics would be needed; this follows from the method's mechanics, not from the paper.","The online and non-stationary extension the authors mention in their conclusion is a natural stress test: if the variance estimator reacts quickly enough, AutoOpt should re-adapt learning rate and momentum when the data distribution changes mid-training."],"forward_implications":["With AutoOpt, the learning rate and momentum are set anew each step and each layer; the paper observes larger mini-batches producing larger learning rates, followed by automatic decay, while momentum rises as the step size shrinks.","Using AutoOpt removes the exhaustive search: for SGD the 324 settings collapse to 1, and for Adam and AdaGrad the 54 settings collapse to 1, with comparable errors.","Because the method's per-step cost is the same order as backpropagation for diagonal Hessian estimators, it is practical for CNN training.","The method is generic across optimizers that fit the update rule (9), so it can be deployed on other SGD variants beyond SGD, Adam, and AdaGrad.","In the paper's runs, AutoSGD, AutoAdam, and AutoAdaGrad reach comparable performance to each other after automatic tuning, making it easy to switch optimizers without manual re-tuning."],"supporting_citations":[{"why":"Defines the Adam optimizer whose diagonal Hessian estimator AutoOpt reuses and whose manually tuned performance is the baseline it must match.","marker":"[10]"},{"why":"Defines the AdaGrad update and Hessian estimator, another deployment target and baseline.","marker":"[9]"},{"why":"The prior automatic-learning-rate method that AutoOpt extends to momentum and to non-diagonal covariance.","marker":"[13]"},{"why":"The Barzilai-Borwein-based automatic learning-rate method AutoOpt builds on and compares against.","marker":"[14]"},{"why":"Supplies the momentum gradient-estimator model used in Eq. (10), the foundation of the reparameterization.","marker":"[7]"},{"why":"Provides the two-point step-size method behind the automatic learning-rate baselines.","marker":"[15]"},{"why":"Backpropagation, the algorithm whose time complexity AutoOpt claims to match for diagonal Hessians.","marker":"[21]"},{"why":"The MNIST dataset used for the main comparison plots.","marker":"[22]"},{"why":"The CIFAR10 dataset used in the experiment that exposes the AutoAdam gap and tests the method's generality.","marker":"[23]"}],"fun_headline_variants":["Auto-tuned SGD matches exhaustive search","One run adjusts LR and momentum, no manual tuning","Learning rate and momentum auto-tuned to grid-search quality","AutoOpt matches best hyperparameters without the search","Hyperparameters set automatically: matches exhaustive tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the optimizer's approximation of the loss curvature is exact; when that approximation is wrong, the automatically chosen learning rate and momentum are no longer the best choices.","fun_headline_variants_meta":{"raw":{"variants":["Auto-tuned SGD matches exhaustive search","One run adjusts LR and momentum, no manual tuning","Learning rate and momentum auto-tuned to grid-search quality","AutoOpt matches best hyperparameters without the search","Hyperparameters set automatically: matches exhaustive tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2838,"prompt_tokens":917,"completion_tokens":1921,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1850}},"tokens_in":533,"tokens_out":1921,"duration_ms":13671,"temperature":1.0,"reasoning_tokens":1850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:02:03.737104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CIFAR10 experiment with AutoAdam using the paper's architecture: if the test error again lands near the reported 47.8% while the exhaustive-search optimum stays near 36.4%, the claim that AutoOpt matches the best manual settings fails for Adam.","supporting_citations":[{"cited_title":"No more pesky learning rates","cited_arxiv_id":null,"evidence_quote":"The prior automatic-learning-rate method that AutoOpt extends to momentum and to non-diagonal covariance."},{"cited_title":"Barzilai-borwein step size for stochastic gradient descent","cited_arxiv_id":null,"evidence_quote":"The Barzilai-Borwein-based automatic learning-rate method AutoOpt builds on and compares against."},{"cited_title":"Learning representations by back-propagating errors","cited_arxiv_id":null,"evidence_quote":"Backpropagation, the algorithm whose time complexity AutoOpt claims to match for diagonal Hessians."},{"cited_title":"The MNIST database of handwritten digits","cited_arxiv_id":null,"evidence_quote":"The MNIST dataset used for the main comparison plots."}],"review_version":1}