{"id":"331f7fc8-dc97-4cea-bd19-23a04c4076a3","arxiv_id":"2505.22085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems without extra gradient evaluations.","lead":"This paper proposes PADAM, an optimizer that runs several averaged versions of Adam in parallel and picks the one with the smallest test error at the end. It reports lower optimization errors than standard Adam, SGD, and AdamW on most of 13 scientific machine learning benchmarks, at the same number of gradient evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported PADAM curves are the oracle best-of-K channel chosen on test loss, not the output of Algorithm 1; the claimed error reduction may be an artifact of selection.","rationale":"The paper has real strengths: the shared-gradient design means PADAM genuinely costs no extra gradient evaluations, the experiments cover 13 problems, and the code is public. However, the strongest claim is an empirical statement about the optimization error of the proposed method, and the manuscript's reporting protocol undermines that statement. The displayed curves are the minimum over K channels of test error, selected using the test labels, rather than the error of a single parameter sequence produced by a well-defined selection rule. This is not a disagreement with the optimizer community's consensus about averaging; it is an internal consistency problem between the plotted quantity, Algorithm 1, and Definition 2.2. Because the central claim depends on this quantity, the concern is load-bearing. If the rerun shows that the same-batch selection rule matches the oracle envelope, the paper's contribution would be substantially rehabilitated, which is why I recommend a conditional verdict rather than outright rejection. The reader's flagged schedule-tuning risk is real but secondary, and the reader also noted the test-loss selection issue in the rationale, hence partial agreement.","tokens_in":16183,"tokens_out":6818,"duration_ms":82741,"concrete_test":"Rerun one representative experiment (e.g., §3.3 heat DKM or §3.10 Burgers PINN) using the released code, and at each checkpoint record three quantities for PADAM: (a) the oracle best-channel test error as currently plotted; (b) the test error of the channel selected by evaluating all K channels on one shared held-out mini-batch; (c) the test error of the channel selected by Algorithm 1's current disjoint-block rule. If (b) or (c) lies noticeably above (a) at the final step, the headline claim is not supported by the figures. In parallel, check whether the released implementation follows Definition 2.2(i)-(iii) or Algorithm 1; the two prescribe different selection behavior.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that PADAM attains the smallest optimization error without extra gradient evaluations. The load-bearing condition is that the plotted PADAM errors are produced by the PADAM procedure as specified and implemented. This condition fails in the manuscript. After Algorithm 1, the authors state that they 'compute the test errors ... for the different channels ... and then plot the test error of the best performing channel'; that is oracle model selection on the test set, not a runnable optimizer. Algorithm 1 itself returns a single θ_{k*} selected at the final step, and lines 19-23 select it by comparing each channel's loss on a different mini-batch (block k J_N+1,...,(k+1)J_N for channel k). Separate noisy batches make this an unfair and high-variance comparison. Definition 2.2(iii) is also inconsistent with Algorithm 1: it selects a channel at every n and sets Θ_n to it, while Algorithm 1 updates all channels and selects once at the end. The plotted lower envelope of K candidates with test-set access can only improve over any fixed channel, so PADAM's apparent advantage may be an artifact of having K chances plus access to the evaluation labels. The schedule-tuning concern is real but secondary; even with good schedules, the reported evidence does not establish that the implementable selection rule achieves the plotted errors.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes PADAM (parallel averaged Adam), a stochastic optimizer that maintains K exponentially/moving-averaged channels of a single Adam trajectory and selects among those channels during or after training. Because all channels share the same underlying gradient evaluations, the method uses no more gradient evaluations than standard Adam. The paper reports experiments on 13 problems—polynomial regression, Gaussian density fitting, deep Kolmogorov, deep Ritz, PINN, deep BSDE, optimal control, and optimal stopping—comparing PADAM3/PADAM10 against SGD, momentum SGD, AdamW, and Adam with Ruppert–Polyak or EMA averaging. The central claim is that PADAM attains essentially the smallest optimization error in nearly all examples, with Python code made publicly available.","tokens_in":16502,"tokens_out":7690,"duration_ms":81008,"significance":"If the empirical claim were established, PADAM would be a practically attractive, nearly free modification of Adam: the user obtains a menu of averaged trajectories at no extra gradient cost and can pick a better final iterate. The paper's strengths are its breadth (13 problems, up to 50 independent runs per problem) and the provision of source code. However, the plotted PADAM curves are a test-loss oracle envelope over the K channels, not the output of the selection rule in Algorithm 1; together with a mismatch between Definition 2.2 and Algorithm 1, this means the advertised advantage is not demonstrated for the implementable method. The trial-and-error schedules and uneven baseline learning-rate choices further weaken the general recommendation.","major_comments":[{"comment":"Algorithm 1, line 10 updates the second moment as 'v ← βn m + (1 − βn)g⊗2'. To match Definition 2.1, Eq. (3), this should read 'v ← βn v + (1 − βn)g⊗2'; as printed, the second-moment estimate is driven by the first moment m rather than by its own previous value, so the pseudocode does not implement Adam and would not reproduce the experiments. Please correct the pseudocode and confirm that the released code implements the corrected update.","section":"§2.2, Algorithm 1"},{"comment":"Definition 2.2 does not describe Algorithm 1. In Definition 2.2(ii), each channel ϑ^k_n is updated using Θ_n, the PADAM output selected in (iii), whereas in Algorithm 1, line 15, channels are updated with the raw Adam iterate ϑ. Moreover, (iii) selects a channel at every step n and sets Θ_n to the selected channel, while Algorithm 1 updates all channels throughout and performs a single selection at the final step (lines 18–23). These are different methods, and the formal definition must be reconciled with the pseudocode or explicitly presented as a separate variant.","section":"§2.2, Definition 2.2 and Algorithm 1"},{"comment":"The curves labeled PADAM3 and PADAM10 are not produced by Algorithm 1's selection rule. The text states that 'we compute the test errors ... for the different channels ... and then plot the test error of the best performing channel'. This is a test-set oracle: at every evaluation point the plotted curve is the minimum over the K channels of their test errors. Algorithm 1 instead selects k* once at the end using a disjoint mini-batch loss. The plotted envelope cannot be worse than any fixed channel, so the claimed error reduction may be an artifact of selecting the best of K candidates with access to the evaluation labels. The central empirical claim should be demonstrated with the implementable selection rule, or the plots should be explicitly labeled as an idealized oracle-selection upper bound.","section":"§2.2, paragraph after Algorithm 1"},{"comment":"The averaging weights (i)–(x) for PADAM3 and PADAM10 are said to have been 'found by trial and error'. Because the same 13 problems are used both to tune these schedules and to demonstrate PADAM's advantage, the reported results may overstate performance on new problems. Please provide a protocol for selecting the schedules without test-set feedback (for example, using only training losses or a validation split), a sensitivity analysis over the δ_{n,k} values, or both, before drawing the general conclusion that PADAM should be considered for scientific machine learning.","section":"§2.2, averaging schedules"},{"comment":"Several baseline comparisons use substantially smaller learning rates for SGD and momentum SGD than for the Adam-based methods, with the phrase 'to avoid divergence' (e.g., 3·10^-6 vs 3·10^-4 in §3.6, 10^-4 vs 10^-2 in §3.11, and 3·10^-4 vs 3·10^-3 in §3.12). This makes the comparison uneven, as the baselines may be far from their best settings while PADAM inherits Adam's learning rate. The authors should either tune the baselines per problem and report the best performance, or explicitly restrict the conclusion to the fixed-learning-rate protocol used here.","section":"§3.6, §3.11, §3.12"}],"minor_comments":[{"comment":"There are several typos: 'movering' should be 'moving', 'parallely' should be 'in parallel', and the method name appears inconsistently as 'Padam' and 'PADAM'.","section":"Abstract and throughout"},{"comment":"'Moroever' should be 'Moreover'; please also unify the capitalization of PADAM/Padam throughout the paper.","section":"Introduction, first paragraph"},{"comment":"Typos: 'Furthemore' in §3.7, 'errror' in §3.8, 'Dacry' and 'momentun' in §3.12.","section":"§3.7, §3.8, §3.12"},{"comment":"Line 15 writes 'δ_{n,j}' but the schedule parameters are indexed as δ_{n,k}; please use a consistent subscript.","section":"Algorithm 1, line 15"},{"comment":"The paragraph defines n_T but does not explicitly state that the plotted quantity uses the Monte Carlo test-error approximations from the figure captions for channel selection; please make the distinction between the test loss used for plotting and the mini-batch loss used in Algorithm 1 explicit.","section":"§2.2, evaluation paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper's central numerical claim is not yet supported as written because the PADAM curves are a test-loss oracle envelope over K channels rather than the output of the selection rule in Algorithm 1. The mismatch between Definition 2.2, Algorithm 1, and the plotting protocol needs to be resolved, and the experiments should be re-run with the implementable selection rule or clearly presented as an oracle upper bound. The trial-and-error schedules and uneven baseline learning rates also need a validation or sensitivity analysis. If these issues are addressed, the paper could become a useful empirical contribution; I would not recommend acceptance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the central empirical claim is not backed by the experiments as reported. The paper plots, by its own description, the test error of the best performing EMA channel at each checkpoint. That is oracle selection on the test set, not the output of Algorithm 1, which picks one channel at the end using noisy mini-batch losses. So the PADAM curves are a lower envelope over K candidates plus access to test labels, and it is no surprise they beat any single fixed channel. The implementable algorithm may still work well, but this paper doesn't demonstrate it.\n\nWhat is actually new: running several EMA schedules in parallel on one Adam trajectory costs no extra gradient evaluations, and selecting among them is a natural, practical idea. The empirical scope is real: 13 problems, up to 50 runs, code on GitHub, with PINN, deep BSDE, optimal stopping, and other SciML benchmarks. The schedules are explicitly admitted to be found by trial and error, which is honest but means transferability is an open question.\n\nSoft spots, in order of severity. First, the test-loss oracle selection. The text in Section 2.2 says they compute test errors and plot the best channel. That is a different procedure from Algorithm 1's final selection on separate mini-batches, and also from Definition 2.2(iii), which selects at every step. This inconsistency means the reader cannot tell what PADAM actually outputs. Second, the schedules are hand-picked and may be tuned to these benchmarks. Third, some baselines get smaller learning rates 'to avoid divergence', which can flatter PADAM. Minor: line 15 has δn,j instead of δn,k.\n\nNone of this kills the idea. The method is cheap and plausible. But the evidence needs to be redone: report the test error of the channel selected by the algorithm, or do selection on a validation set and then evaluate. Fair baselines with matched learning rates would help.\n\nVerdict: worth engaging with, but not acceptable in current form. I'd send it to a serious referee because the method is simple, useful if true, and the flaw is fixable. I would not cite the current empirical claim.","headline":"PADAM's headline result is undercut by test-loss oracle selection; the method is worth a rerun with honest evaluation, not a citation yet.","tokens_in":16936,"tokens_out":2575,"would_cite":false,"duration_ms":29369,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","65K10","90C15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Parallel averaged Adam (PADAM) runs several exponential-moving-average channels of a single Adam trajectory in parallel, selects at checkpoints the channel with the smallest optimization error, and claims this reaches essentially the…","keywords":["parallel averaged Adam","PADAM","exponential moving average","stochastic optimization","scientific machine learning","physics-informed neural networks","deep BSDE","adaptive averaging"],"falsifier":"Take PADAM10 with the published schedules and run it on a new PDE or optimal-control problem where the best of the ten channels is known in hindsight; if the dynamically selected channel's error is consistently larger than the best channel's error, or if a single fixed schedule outside the bank beats all ten, the selection mechanism and schedule bank are not doing the claimed work.","tokens_in":16042,"feed_emoji":"🧮","tokens_out":6354,"duration_ms":64438,"temperature":0.7,"pith_summary":"Averaging tricks such as Ruppert–Polyak averaging and exponential moving averages help stochastic optimizers, but the best averaging type and parameters depend on the problem. This paper proposes parallel averaged Adam (PADAM), which runs several different EMA-averaged copies of the same Adam trajectory at once and, during training, keeps whichever copy currently has the smallest optimization error. Because all copies share one underlying Adam trajectory and therefore the same gradients, PADAM costs no extra gradient evaluations. In 13 stochastic optimization and deep learning problems, including polynomial regression, Gaussian density learning, and scientific machine learning solvers for PDEs, optimal control, and optimal stopping, PADAM achieves essentially the smallest error among the compared optimizers in nearly all cases. The paper therefore argues that PADAM is a strong default choice for scientific machine learning and motivates further research into adaptive averaging procedures.","feed_headline":"Parallel averaged Adam cuts error in 13 test problems","feed_subtitle":"Several exponential-averaged copies of one Adam run, and the best is chosen at no extra gradient cost.","key_machinery":"The central object is the PADAM process in Definition 2.2: one Adam process $\\vartheta^0$ whose gradients are shared by all channels, together with $K$ auxiliary EMA processes $\\vartheta^k_n = \\delta_{n,k}\\,\\vartheta^k_{n-1} + (1-\\delta_{n,k})\\,\\Theta_n$ that track the same Adam iterate under different averaging weights $\\delta_{n,k}$. At checkpoints the method evaluates the batch loss of each channel and returns the channel with the smallest loss, as specified in Algorithm 1. This carries the argument because it converts the problem of choosing an averaging schedule into a cheap online selection among schedules: each channel costs one vector recurrence per step, and all channels use the same gradients.","core_discovery":"The paper's central claim is that parallel averaged Adam (PADAM), a wrapper that runs several EMA-averaged versions of a single Adam trajectory side by side and at checkpoints switches to the version with the lowest current loss, reaches essentially the smallest optimization error in nearly all of the 13 stochastic optimization and DNN learning problems studied. The comparison includes standard SGD, momentum SGD, Adam, Adam with EMA, Adam with Ruppert–Polyak averaging, and AdamW, and PADAM does this with the same number of gradient evaluations as plain Adam. The claim is empirical, established through numerical experiments spanning physics-informed neural networks, deep Galerkin and deep Ritz methods, deep BSDE, deep Kolmogorov, optimal control, and optimal stopping problems.","pith_inferences":["The selection mechanism does not depend on Adam specifically, so a natural extension is to run parallel averaging over AdamW, SGD with momentum, or other base optimizers and apply the same best-channel rule.","Channel selection uses the loss on one fresh mini-batch; on problems with very small batches or very noisy losses, a reader could test whether selecting on a larger validation batch changes the outcome, since the paper's runs use batches of 256 and checkpoint thresholds of 500 or 5000 steps.","The hand-tuned schedules are the fragile part; an adaptive scheduler that adds, removes, or updates channels during training could extend the idea beyond the fixed PADAM3 and PADAM10 channel sets.","On problems where one averaging schedule dominates, PADAM should reduce to that schedule and match its error; the open empirical question is whether the selected channel tracks the best fixed schedule on problems outside the benchmark set."],"forward_implications":["PADAM3 and PADAM10 reach essentially the smallest optimization error in nearly all of the 13 problems, sometimes tied with other methods and sometimes exclusively, with no more gradient evaluations than plain Adam.","Because all channels share one Adam trajectory, PADAM is a drop-in addition to existing training loops for PINNs, deep Ritz, deep BSDE, deep Kolmogorov, optimal control, and optimal stopping solvers, with only minor extra computing time for large networks.","A small parallel bank of EMA schedules removes the need to commit in advance to a single averaging parameter; the selection rule picks among schedules online.","The strong empirical results support treating PADAM as a default optimizer for scientific machine learning problems and motivate further work on adaptive averaging procedures in deep neural network training."],"supporting_citations":[{"why":"Defines the Adam optimizer that PADAM wraps; the underlying trajectory and gradients are shared by all averaging channels.","marker":"[34]"},{"why":"Preliminary work showing that Adam with EMA reduces errors in PDE and optimal control training; PADAM extends this to a parallel bank of schedules.","marker":"[17]"},{"why":"Introduces the deep Kolmogorov method used in the heat and Black–Scholes experiments.","marker":"[6]"},{"why":"Provides the deep BSDE formulation and Cole–Hopf solution used in the HJB and related experiments.","marker":"[22]"},{"why":"Introduces the deep Ritz method used for the Poisson and p-Laplace experiments.","marker":"[24]"},{"why":"Introduces the deep optimal stopping method used in the American option experiment.","marker":"[8]"},{"why":"Studies Adam with model EMA, a key comparison baseline and motivation for averaging weights.","marker":"[1]"},{"why":"Introduces AdamW, one of the optimizers PADAM is compared against.","marker":"[39]"}],"fun_headline_variants":["Parallel Adam averages, best chosen, matches or beats all","PADAM: several Adam averages, pick the best, cut error","Run Adam averages in parallel, select the top performer","Dynamic Adam averaging wins across 13 optimization tests","Best of many Adam averages: same cost, lower error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"PADAM's advantage rests on the hand-picked averaging schedules, which the paper says were found by trial and error, being well matched to the problems; if those schedules were effectively tuned on the 13 benchmarks, the method may not beat a single well-chosen average on new problems.","fun_headline_variants_meta":{"raw":{"variants":["Parallel Adam averages, best chosen, matches or beats all","PADAM: several Adam averages, pick the best, cut error","Run Adam averages in parallel, select the top performer","Dynamic Adam averaging wins across 13 optimization tests","Best of many Adam averages: same cost, lower error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000275,"raw_usage":{"total_tokens":1675,"prompt_tokens":1007,"completion_tokens":668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":587}},"tokens_in":623,"tokens_out":668,"duration_ms":7441,"temperature":1.0,"reasoning_tokens":587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:14:55.852919+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take PADAM10 with the published schedules and run it on a new PDE or optimal-control problem where the best of the ten channels is known in hindsight; if the dynamically selected channel's error is consistently larger than the best channel's error, or if a single fixed schedule outside the bank beats all ten, the selection mechanism and schedule bank are not doing the claimed work.","supporting_citations":[{"cited_title":"The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems","cited_arxiv_id":null,"evidence_quote":"Introduces the deep Ritz method used for the Poisson and p-Laplace experiments."},{"cited_title":"Solving the Kolmogorov PDE by means of deep learning","cited_arxiv_id":null,"evidence_quote":"Introduces the deep Kolmogorov method used in the heat and Black–Scholes experiments."},{"cited_title":"Deep learning-based numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equations","cited_arxiv_id":null,"evidence_quote":"Provides the deep BSDE formulation and Cole–Hopf solution used in the HJB and related experiments."},{"cited_title":"Solving high-dimensional optimal stopping problems using deep learning","cited_arxiv_id":null,"evidence_quote":"Introduces the deep optimal stopping method used in the American option experiment."}],"review_version":1}