{"id":"4453d07e-e10e-4a16-9b21-4a47c2798574","arxiv_id":"2412.12473","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MIAdam adds an n-th order integral term to Adam's gradient update for the first ζ steps, then switches to Adam, and is reported to improve test accuracy and label-noise robustness.","lead":"The paper proposes MIAdam, a version of the Adam optimizer that temporarily adds repeated averaging of past gradients before switching back to ordinary Adam, and reports better test accuracy on image and text benchmarks. The idea is that this repeated averaging acts like a low-pass filter, steering training into wide, flat minima that generalize better.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central proof is not a valid model of MIAdam1: eliminating m_t and M_t from Eq. (18) yields a different continuum limit than Eq. (20), so Theorem 1's escape-time comparison is unsupported.","rationale":"MIAdam's central claim has two parts: an empirical claim that it improves generalization, and a mechanistic claim that multiple integration filters sharp minima and therefore reaches flat minima. The mechanistic claim rests entirely on Theorem 1. My check of the proof shows the discrete update in Eq. (18) reduces, in the continuum limit, to chi theta''' + theta'' = -g, whereas the paper's Eq. (20) is chi theta''' - theta'' = -g (taking delta chi = 1 and f = -g). This is not a matter of parameter tuning or of the diffusion-theory approximations; it is an algebraic mismatch in the motion law. Since the Fokker-Planck equation, the Smoluchowski-type reduction, and the escape-time expression in Eqs. (22)-(27) are all derived from Eq. (20), Theorem 1 cannot establish that MIAdam1 has the claimed escape behavior. The empirical results and the code link are useful, and the switch-to-Adam design is a reasonable way to obtain final convergence, but the experiments alone do not establish the flat-minimum mechanism; they are also weakened by missing error bars on ImageNet and by tuning of kappa and zeta on the target benchmarks, as the reader notes. Hence the rejection stands, and the concrete algebra check described above would make the failure of Theorem 1 unambiguous.","tokens_in":16456,"tokens_out":16755,"duration_ms":147184,"concrete_test":"Independently re-derive the continuous-time limit of Eq. (18) by symbolic elimination: substitute M_t = -(theta_{t+1}-theta_t)/alpha^2 and m_t = M_t - M_{t-1} into m_t = beta1 m_{t-1} + (1-beta1)g_t, expand in h=alpha, and compare with Eq. (20). If the resulting ODE is chi theta''' + theta'' = -g rather than chi theta''' - theta'' = -g, Eq. (20) is not the MIAdam1 dynamics and Theorem 1's proof fails. A numerical cross-check on a 1D quadratic loss L(theta) = H theta^2/2 with beta1 = 0.9, alpha = 0.01, kappa = 1 would compare the discrete trajectory from Eq. (18) with the numerical solution of Eq. (20) with the noise term set to zero; if the trajectories do not match, the escape-time calculation is not about the proposed optimizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 is the only theoretical support for the claim that the multiple integral term biases the optimizer toward flat minima, and its proof depends on the deformed motion equation (19)-(20). That equation is not implied by the MIAdam1 update (18). Eliminating m_t and M_t from (18), with M_t = sum_{j=0}^t m_j and theta_{t+1}=theta_t - alpha^2 M_t, gives the exact third-order difference theta_{t+1} - (2+beta1)theta_t + (1+2beta1)theta_{t-1} - beta1 theta_{t-2} = -alpha^2(1-beta1)g_t. Setting alpha=h, beta1=1-delta h and taking h->0 with delta fixed gives delta theta'' + theta''' = -delta g, i.e. chi theta''' + theta'' = -g with chi = h/(1-beta1). Equation (20) instead asserts chi theta''' = delta chi theta'' + f; with delta chi = 1 and f = -g this is chi theta''' - theta'' = -g, which has the opposite sign for theta''. The rescaling used later changes the noise/diffusion term rather than repairing this deterministic mismatch. Thus Eqs. (21)-(27) are not the dynamics of MIAdam1, and Eq. (14) is not MIAdam1's mean escape time. The comparison phi_MIAdam1 < phi_Adam, and with it the paper's theoretical reason that multiple integration finds flat minima, is therefore unsupported. The convergence section does not repair this: Theorem 2 only demonstrates non-convergence before the switch, and its proof of divergence appears to multiply the bounded sum sum_t kappa^{t-r} by an extra factor t; no regret bound is proven after the switch.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIAdam, an Adam variant that replaces the momentum update with an n-th order cumulative (multiple-integral) sum of past momentum for the first ζ iterations and then switches to standard Adam. The paper claims that the multiple integral acts as a low-pass filter that filters out sharp minima, guiding the optimizer to flat minima and thereby improving generalization and robustness to label noise while retaining Adam's fast convergence. The theoretical section uses a diffusion-theory mean-escape-time calculation to argue that MIAdam1 escapes sharp minima faster than Adam, together with a regret analysis showing non-convergence before the switch. The empirical section reports image classification (CIFAR and ImageNet), text classification (BERT/RoBERTa), and label-noise experiments, plus Hessian eigenvalue diagnostics, and the paper provides a code link.","tokens_in":16794,"tokens_out":8301,"duration_ms":72177,"significance":"If the theoretical claims were valid, MIAdam would be a simple, cheap drop-in optimizer with practical gains, and the paper's empirical scope is indeed substantial: multiple architectures on CIFAR, ImageNet results, text fine-tuning, label-noise experiments, and Hessian diagnostics, with code provided. The paper also honestly acknowledges that the multiple-integral phase is non-convergent and must be followed by a switch to Adam. However, the central theoretical justification is load-bearing and, on inspection, does not describe the algorithm's dynamics: the continuum limit of the MIAdam1 update is different from the equation used in Theorem 1, the escape-time formula contains unexplained scaling, and the convergence section neither covers the switched algorithm nor supports the claimed divergence. Because the main novel claims rest on these derivations, I cannot recommend publication in the current form.","major_comments":[{"comment":"The continuum limit of the MIAdam1 update in Eq. (18) is not the deformed motion equation in Eq. (20). Eliminating m_t and M_t from Eq. (18) gives θ_{t+1} - (2+β_1)θ_t + (1+2β_1)θ_{t-1} - β_1θ_{t-2} = -α^2(1-β_1)g_t. Setting α=h and β_1=1-δh and letting h→0 yields δθ'' + θ''' = -δg, i.e. χθ''' + θ'' = -g with χ=h/(1-β_1). Equation (20) instead asserts χθ''' = δχθ'' + f, which with δχ=1 and f=-g gives χθ''' - θ'' = -g, the opposite sign for θ''. The later rescaling in Eq. (21) changes the noise/diffusion term but does not repair this deterministic mismatch. Since Eq. (20) is the starting point of the Fokker-Planck reduction, the escape time in Eq. (27) is not a consequence of the MIAdam1 dynamics in Eq. (18).","section":"Appendix, 'Generalization Proof', Eqs. (18)-(20)"},{"comment":"Even if Eq. (20) were accepted, the comparison φ_MIAdam1 < φ_Adam is not supported by the displayed formulas. The factor 1/˜t appears in both the prefactor and the exponent of Eq. (14)/(27) without derivation, and the learning-rate scaling is inconsistent with the α^2 update in Eq. (18): Theorem 1 uses α as if it were the small step in the diffusion term, while the deterministic update already scales with α^2. The claimed inequality is therefore an artifact of an unexplained ˜t-dependent rescaling and the chosen effective learning-rate convention, rather than a property of the multiple integral term.","section":"Generalization Analyses, Eq. (14) and Eq. (27)"},{"comment":"The claimed divergence of term {3} in the proof of Theorem 2 rests on an extra factor t. For κ_1∈(0,1), the sum ∑_{r=1}^{t-1} κ_1^{t-r} is bounded by κ_1/(1-κ_1) and does not grow with t, whereas Eq. (36) as written multiplies this bounded sum by t and then concludes divergence; for κ_1=1 the bound with denominator 1-κ_1 is not valid. Moreover, even a correct divergence proof would only establish Theorem 2's statement of non-convergence before the switch, while the actual MIAdam algorithm switches to Adam at ζ; no regret bound after ζ is supplied. The conclusion that MIAdam provides 'a guarantee of convergence' is therefore unsupported.","section":"Convergence Analyses and Appendix, 'Convergence Proof', Eqs. (36)-(40)"}],"minor_comments":[{"comment":"The conclusion states that the paper has 'provided a guarantee of convergence,' but Theorem 2 only proves non-convergence before the switch and no post-switch regret bound is given; this wording should be corrected.","section":"Conclusion"},{"comment":"The dataset name is misspelled as 'CIAFR10' in the label-noise section, and Table 2's header says 'mean±std' while the table entries are single numbers; the reporting format should be made consistent.","section":"Experiments, 'Robustness Against Label Noises' and Table 2"},{"comment":"The proof of Theorem 1 sets κ=1 'without loss of generality,' but κ is a tuned hyperparameter in Algorithm 1 and in the grid search; either a justification of the WLOG reduction or an explicit restriction of the theorem to κ=1 is needed.","section":"Appendix, 'Generalization Proof'"},{"comment":"The text says MIAdam 'significantly outperforms' Adam on text classification, but many differences in Table 3 are within 0.1-0.3 percentage points and no significance tests are reported; the claim of significance should be softened or supported.","section":"Experiments, Table 3"}],"recommendation":"reject","confidential_remarks":"To the editor: the empirical section is reasonably broad and the authors provide code, but the central theoretical derivation is not a valid model of the proposed algorithm's dynamics, and the convergence theorem explicitly excludes the switched phase that the algorithm relies on. The manuscript could conceivably be reframed as an empirical study with the theory removed, but the current submission presents the theory as the primary justification for the title and the main claims, so I recommend rejection. I also note that the relationship to the authors' earlier integration-enhanced neural network methods is not discussed, which makes the novelty difficult to assess."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nMIAdam is a simple, inexpensive change to Adam — accumulate its momentum over time (multiple summations) before a scheduled switch back to plain Adam — with the intuition that this low-pass filters the trajectory and avoids sharp minima. The paper reports consistent gains over Adam and variants on CIFAR, ImageNet, text, and label-noise benchmarks, and it ships code. That part is real and worth a look.\n\nThe problem is the theory. The central theorem, meant to show MIAdam1 escapes sharp minima faster than Adam, is not about MIAdam1. Eliminating m_t and M_t from the discrete update (18) gives the exact difference equation θ_{t+1} - (2+β1)θ_t + (1+2β1)θ_{t-1} - β1θ_{t-2} = -α^2(1-β1)g_t. With α=h, β1=1-δh, this has the continuum limit δθ'' + θ''' = -δg, i.e. θ'' + χθ''' = -g. The paper's Eq. (20) instead reads χθ''' = δχθ'' + f, which with their substitutions becomes χθ''' - θ'' = -g — the θ'' term has the wrong sign. The subsequent rescaling changes the noise term but doesn't repair the deterministic dynamics. So Theorem 1's escape-time formula, and the φ_MIAdam1 < φ_Adam conclusion, are not established from the paper's own update rule.\n\nTheorem 2 is also shaky. It only shows non-convergence in the pre-switch phase, which is fine as a motivation for the switch, but the proof of that non-convergence appears to multiply a bounded sum by an extra factor t. And no regret bound is proven for the full algorithm after the switch, despite the conclusion claiming \"a guarantee of convergence.\"\n\nThe empirical section has weaknesses too: ImageNet and label-noise results have no error bars; κ and ζ are grid-searched on the target benchmarks; and a few baselines look off (the SW ATS numbers on PyramidNet have wild variance). These don't kill the algorithm, but they temper how much weight the gains can carry.\n\nHonestly, as a scientific claim the paper doesn't hold. As an empirical optimizer with a clear mechanism and public code, it's worth a reviewer's time — a referee could push the authors to fix the continuum limit or reframe the paper as purely empirical, which might be the honest framing anyway. I'd send it to peer review, with a note that the theory needs to be corrected or dropped.","headline":"A simple, plausible optimizer with substantial empirical results, but the central escape-time theorem is based on a continuum limit that doesn't match its own update rule.","tokens_in":17346,"tokens_out":7090,"would_cite":false,"duration_ms":55359,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding integrals to Adam's update finds flatter minima","keywords":["MIAdam","Adam optimizer","flat minima","generalization","multiple integration","label noise","diffusion theory","adaptive learning rate"],"falsifier":"A concrete test on the paper's two-parameter landscapes is to count, over many random initializations, how often MIAdam reaches the flat basin compared with Adam, and to compute the Hessian trace at each final point; if MIAdam does not reach the flat basin more often, or if its final Hessian trace is not smaller, the claim that the integral term filters sharp minima is unsupported.","tokens_in":16202,"feed_emoji":"📉","tokens_out":8960,"duration_ms":71442,"temperature":0.7,"pith_summary":"This paper proposes MIAdam, a version of the Adam optimizer whose parameter update includes a multiple integral term: several nested sums over the gradient history are added to Adam's first moment estimate. The authors argue that this term acts as a low-pass filter on the optimizer's trajectory, so sharp minima, viewed as high-frequency features, are smoothed away and the optimizer drifts toward flat minima. The paper's claim is that flat minima generalize better, and therefore MIAdam improves test accuracy and robustness to label noise while retaining Adam's fast convergence. The authors prove, under a diffusion-theory approximation, that the first-order integral version escapes sharp minima faster than Adam, and they prove that the integral phase alone does not converge, which motivates a switch back to Adam after a fixed number of epochs. A sympathetic reader would take the central claim to be that a simple summation added to Adam's update is enough to bias deep-network training toward flat, well-generalizing solutions.","feed_headline":"Adding integrals to Adam's update finds flatter minima","feed_subtitle":"MIAdam's nested gradient sums escape sharp valleys and beat Adam on vision, text, and noisy-label benchmarks.","key_machinery":"The carrying object is the $n$-th-order multiple summation term $m^{(n)}_{t,i}$, built by recursively accumulating the first moment $m_{t,i}$ $n$ times, each accumulation weighted by a rate $\\kappa$. In the continuous-time limit this term becomes an $n$-fold integral of the gradient signal, and integration acts as a low-pass filter: high-frequency fluctuations, identified with sharp minima, are attenuated. In the discrete algorithm the term is computed by $n$ extra running sums per parameter, so the added cost is small. Its role is to give an early exploration phase that gravitates toward flat basins; because the term alone prevents convergence (Theorem 2), the algorithm switches to standard Adam updates after $\\zeta$ steps, and the final phase converges quickly.","core_discovery":"The central claim is that adding an $n$-th-order multiple integral of past gradients to Adam's update, before switching back to Adam at step $\\zeta$, filters out sharp minima and guides the optimizer toward flat minima, improving generalization and label-noise robustness without sacrificing Adam's convergence speed. The update in Eq. (12) replaces the first moment $m_{t,i}$ with a nested summation $m^{(n)}_{t,i}=\\sum_{t_1=0}^{t}\\kappa^{t-t_1}\\sum_{t_2=0}^{t_1}\\cdots\\sum_{t_n=0}^{t_{n-1}}\\beta_1^{t_{n-1}-t_n}g_{t_n,i}$, where $\\kappa$ is the multiple-integration rate. Under the diffusion-theory approximation, Theorem 1 gives the mean escape time $\\phi_{\\mathrm{MIAdam1}}$ from a sharp minimum through a saddle point; it is smaller than the corresponding Adam escape time for $\\tilde{t}>1$, meaning the integrated optimizer is more likely to abandon sharp minima. Theorem 2 shows that before the switch the algorithm has non-vanishing average regret, so the integral phase is intentionally non-convergent, and the later switch to Adam restores convergence. Empirically, MIAdam1 achieves higher top-1 accuracy than Adam and its variants on CIFAR-10/100, ImageNet-1k, and text classification datasets, and higher accuracy under 20--80% label noise; Hessian eigenvalue densities show flatter final minima.","pith_inferences":["The paper's theory covers only the first-order integral (MIAdam1), while experiments include second- and third-order versions; a natural inference is that higher orders filter more aggressively but also cause larger oscillations near flat regions, consistent with the reported MIAdam3 instability and suggesting an optimal finite order.","The signal-processing analogy suggests the same integration trick could be applied to other adaptive optimizers, such as AdamW or AdaBound, or even to SGD with momentum; the mechanism is not obviously specific to Adam's moment estimates.","One testable extension is to make the integration rate $\\kappa$ or the switch time $\\zeta$ adaptive to gradient statistics, rather than fixed grid-searched values, which could remove the two new hyperparameters.","Because the theoretical escape-time comparison uses a locally quadratic loss, the practical claim is stronger than the proof: the empirical flatness gains on deep networks suggest the mechanism survives far outside the quadratic regime, but that generalization is not established by Theorem 1."],"forward_implications":["MIAdam is a drop-in replacement for Adam: it uses the same learning rate and decay hyperparameters, adds only a multiple-integration rate $\\kappa$ and a switch step $\\zeta$, and its per-iteration cost is $n$ extra summations per parameter.","If the central claim holds, training with MIAdam instead of Adam should consistently produce solutions with smaller top Hessian eigenvalues and traces, that is, flatter loss landscapes, across a wide range of architectures and datasets.","The label-noise experiments imply that the filtering effect is most valuable when gradients are corrupted, because the integral averages out spurious high-frequency gradient fluctuations; this extends naturally to noisy or small-sample training regimes.","Since the integral phase is provably non-convergent, the switch time $\\zeta$ is the key hyperparameter: too short a switch gives Adam-like behavior, too long delays convergence, so an automatic or scheduled switch could be a practical improvement."],"supporting_citations":[{"why":"Supplies the diffusion-theory framework and the mean-escape-time comparison that Theorem 1 extends to the integrated update.","marker":"(Xie, Sato, and Sugiyama 2020)"},{"why":"Provides the Adam escape-time formula (Eq. 15) that MIAdam1 is compared with, and the Adai baseline used in experiments.","marker":"(Xie et al. 2022)"},{"why":"Defines Adam and its moment updates, which MIAdam modifies by adding the multiple summation.","marker":"(Kingma and Ba 2015)"},{"why":"Gives the finite-inertia phase-space reduction used in the proof of Theorem 1 to obtain the escape-time formula.","marker":"(Kalinay and Percus 2012)"},{"why":"Establishes the flat-minima-to-generalization link on which the design rationale rests.","marker":"(Hochreiter and Schmidhuber 1997)"},{"why":"Introduces switching from Adam to another update rule (SWATS), the template for MIAdam's switch and a comparison baseline.","marker":"(Keskar and Socher 2017)"},{"why":"Supplies the two-parameter loss landscapes used in the simulations showing MIAdam escapes sharp minima.","marker":"(Yang 2020)"},{"why":"Provides the Hessian eigenvalue computation used to demonstrate that MIAdam's solutions are flatter.","marker":"(Yao et al. 2020)"}],"fun_headline_variants":["MIAdam's multiple integrals filter out sharp minima for flatter generalization","Nested gradient sums in Adam steer to flat minima","MIAdam integrates past gradients to escape sharp valleys","Integral-enhanced Adam finds flatter optima, not just faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole case rests on treating sharp minima as high-frequency components of the optimizer trajectory that a multiple integral can filter out, together with the diffusion-theory assumptions of a locally quadratic loss, quasi-equilibrium, and low gradient noise; if any of these fail, the escape-time comparison does not establish that MIAdam finds flatter minima.","fun_headline_variants_meta":{"raw":{"variants":["MIAdam's multiple integrals filter out sharp minima for flatter generalization","Nested gradient sums in Adam steer to flat minima","MIAdam integrates past gradients to escape sharp valleys","Integral-enhanced Adam finds flatter optima, not just faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3446,"prompt_tokens":989,"completion_tokens":2457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2389}},"tokens_in":605,"tokens_out":2457,"duration_ms":15385,"temperature":1.0,"reasoning_tokens":2389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:03:45.253542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test on the paper's two-parameter landscapes is to count, over many random initializations, how often MIAdam reaches the flat basin compared with Adam, and to compute the Hessian trace at each final point; if MIAdam does not reach the flat basin more often, or if its final Hessian trace is not smaller, the claim that the integral term filters sharp minima is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion-theory framework and the mean-escape-time comparison that Theorem 1 extends to the integrated update."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Adam escape-time formula (Eq. 15) that MIAdam1 is compared with, and the Adai baseline used in experiments."},{"cited_title":"P.; and Ba, J","cited_arxiv_id":null,"evidence_quote":"Defines Adam and its moment updates, which MIAdam modifies by adding the multiple summation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the finite-inertia phase-space reduction used in the proof of Theorem 1 to obtain the escape-time formula."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the flat-minima-to-generalization link on which the design rationale rests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Hessian eigenvalue computation used to demonstrate that MIAdam's solutions are flatter."}],"review_version":1}