{"id":"57161478-d359-466f-ad37-6293974f6a66","arxiv_id":"2502.05133","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A data-parallel trust-region optimizer (APTS) achieves validation accuracy comparable to fine-tuned Adam on MNIST and CIFAR-10 with fixed hyperparameters.","lead":"This paper proposes a trust-region optimizer that trains several copies of a neural network in parallel on different data chunks and combines their updates. It reports accuracy close to tuned Adam on two small image datasets without needing learning-rate tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparable accuracy without tuning is asserted, not demonstrated: APTS runs lack error bars and hyperparameter values, SAPTS convergence is disclaimed in §2.2, and the SGD baseline promised in the abstract is absent.","rationale":"The reader's conditional verdict is appropriate. The paper proposes a plausible adaptation of APTS to data-parallel DNN training, and its deterministic variant may inherit convergence behavior from the APTS framework. However, the manuscript does not provide the evidence needed to verify the two most important claims. First, the global-convergence claim is made in the Introduction without qualification, while Section 2.2 concedes that the existing APTS convergence analysis does not extend to SAPTS; the minibatch experiments therefore operate outside the stated theory. No deterministic-only convergence theorem is supplied for the nonconvex DNN loss with L-SR1 approximations. Second, the empirical comparison is too thin to support 'comparable validation accuracy' and 'no hyperparameter tuning': APTS results have no error bars, the fixed hyperparameters are not listed, and the promised SGD baseline is absent. These are not evidence of fraud or even of an incorrect algorithm; they are missing checks that could be supplied by the authors. The central claim is therefore plausible but not yet established to the standard the abstract implies. A conditional verdict that asks for the missing numerical detail and for a deterministic/stochastic separation is the right outcome, so the reader's verdict should remain unchanged.","tokens_in":6964,"tokens_out":7086,"duration_ms":81491,"concrete_test":"Add a controlled experiment on MNIST: run deterministic APTS and SAPTS with published fixed hyperparameters (Δ0, η1, η2, α, β, ν, overlap) for 5 seeds and report mean±std final validation accuracy; compare against (a) SGD with the same fixed hyperparameter budget and (b) Adam best-of-13. Also record for each outer iteration the sign and magnitude of ρ in Eq. (6); if a substantial fraction of iterations have ρ<η1 or an undefined ratio, the acceptance mechanism assumed by the convergence argument is not operating as required.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—APTS matches SGD/Adam accuracy while needing no hyperparameter tuning—requires two conditions: (i) the deterministic APTS convergence theory from [11] applies to the DNN setting, and (ii) the reported empirical comparisons are strong enough to support 'comparable' and 'no tuning.' Condition (i) is only asserted: Section 2.2 explicitly states the existing convergence analysis does not extend to SAPTS, yet the experiments include minibatch (SAPTS) runs, and no theorem or proof is given for the deterministic case in the nonconvex DNN setting with L-SR1 Hessian approximations. Condition (ii) is underdetermined by the paper itself: no error bars are reported for APTS, no hyperparameter values are listed for the 'constant parameters,' and the abstract promises comparison with both SGD and Adam, but Section 3 compares only with Adam; SGD is dismissed as 'less optimal' without data. Because the claim 'no hyperparameter tuning' is the main differentiator of the method relative to SGD/Adam, an unsupported special-case treatment of this claim is material.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a data-parallel variant of the Additively Preconditioned Trust-Region Strategy (APTS) for training deep neural networks. The dataset is partitioned into minibatches or microbatches, and each subproblem trains a copy of the network with a local trust-region method on a modified local loss (3) that includes a first-order consistency term (4) matching the global gradient at the first local iterate. The local steps are summed, their quality is assessed via the ratio (6), and the global parameters are updated with a combined rule (7) that also updates the trust-region radius; one additional global trust-region step is then performed (Algorithm 1, line 9). A stochastic variant, SAPTS, is also described. Experiments on MNIST and CIFAR-10 compare APTS/SAPTS with Adam using fixed, unreported TR parameters, with first- and second-order local models. The paper claims comparable accuracy to SGD and Adam, global convergence, parallelizability, and no hyperparameter tuning.","tokens_in":7274,"tokens_out":6382,"duration_ms":62622,"significance":"The algorithmic idea is genuinely interesting: combining additive Schwarz-style nonlinear preconditioning with trust-region globalization for DNN training is a plausible way to reduce communication frequency and avoid learning-rate schedules. The paper makes a useful connection to the APTS/ASPIN literature and gives a concrete algorithm (Algorithm 1) with a software reference, which is a positive reproducibility practice. However, the strength of the claims is far ahead of the evidence: only two small datasets are considered, no error bars or hyperparameter values are reported for the new method, the SGD baseline named in the abstract is missing, and the convergence argument is cited rather than proved for the stochastic and nonconvex setting. With these gaps closed, the contribution could be valuable; as it stands, the significance is potential rather than demonstrated.","major_comments":[{"comment":"The abstract claims that the proposed APTS variant achieves validation accuracy comparable to SGD and Adam, but Section 3 states that APTS is compared only with the sequential Adam optimizer and that 'SGD and TR performed less optimally' without reporting any SGD experiments. The claim of parity with SGD is therefore not supported by the presented evidence.","section":"Abstract; §3"},{"comment":"The experimental section reports no error bars, confidence intervals, or repeated-trial statistics for APTS, and it does not list the values of the supposedly constant TR parameters (Δ0, η1, η2, α, β, ν, overlap, and L-SR1 memory m). Adam, by contrast, is reported as the best of 13 learning rates over 10 trials. Without the APTS parameter values and variance information, the claims 'comparable validation accuracy' and 'no hyperparameter tuning' cannot be verified or reproduced.","section":"§3"},{"comment":"The paper asserts in Section 1 that the method is globally convergent and inherently parallelizable, but the convergence statement is only a citation to [11]. No theorem or verification of the assumptions of that theory (nonconvex DNN loss, L-SR1 Hessian approximations, the local models fi in (3), the acceptance ratio (6), and the global TR step in Algorithm 1 line 9) is provided. Section 2.2 explicitly disclaims the extension of the convergence analysis to SAPTS, yet the reported minibatch experiments (minibatches of size 10,000 with 5% overlap) are SAPTS runs. Thus the evaluated stochastic variant has no convergence guarantee, and the deterministic variant's guarantee is not established in this setting.","section":"§1; §2.2"},{"comment":"The claim of reduced communication cost is not substantiated. In the deterministic APTS variant, the consistency term (4) requires the full gradient ∇f(θ_k, D), and Algorithm 1 line 9 performs a global TR iteration on f with the full dataset D; both incur synchronization and full-data evaluation costs every outer iteration. The paper provides no wall-clock communication comparison with data-parallel SGD/Adam and states that scalability studies are planned. The statement that 'communication overhead depends only on the connection speed' therefore overstates what is demonstrated.","section":"§2.2; §3"}],"minor_comments":[{"comment":"The dataset name is inconsistently typeset as 'CIF AR-10' in several places; it should be CIFAR-10.","section":"Throughout"},{"comment":"The phrase 'communication overheard' should be 'communication overhead'.","section":"§3"},{"comment":"The text says 'If ρk > η' but equation (7) uses η1 and η2; the acceptance threshold η is never defined. Please clarify the notation and align the text with the update rule.","section":"§2.2; Eq. (6)–(7)"},{"comment":"The notation 'D ∈ [D]' for APTS and 'D ∈ [D1, D2, ...]' for SAPTS is unclear; please specify how the full dataset, minibatches, and microbatches are enumerated and ordered.","section":"Algorithm 1, line 3"},{"comment":"The figures lack explicit captions and legends identifying which curves correspond to Adam versus APTS and to which number of subdomains; the text references 'first-order model' and 'second-order model' but the reader cannot infer all settings from the figures alone.","section":"Figures 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The core algorithmic idea is interesting and suitable for a methods paper, but the abstract and introduction overstate what the experiments establish. In revision, the authors should either add the missing experiments and convergence discussion or restrict the claims. I would not recommend rejection on novelty grounds, but the current empirical support is below the standard for the claims made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a reasonable incremental contribution—applying APTS, a nonlinear preconditioned trust-region method from domain decomposition, to data-parallel NN training—but its headline claims (comparable to Adam, no hyperparameter tuning) are not supported by the evidence as presented. The consistency term (3)-(4) is a clean idea: it makes each local loss gradient match the global gradient at the start of each outer iteration, which should reduce minibatch variance. The combined local/global acceptance ratio (6) is also sensible. The method is clearly described, the algorithm is explicit, and the authors point to a code repository, which is good practice.\n\nThe soft spots are material. First, the empirical comparison: Figures 1 and 2 show APTS curves against Adam, but there are no error bars for APTS and no hyperparameter values for the 'constant parameters.' The abstract promises comparison with SGD and Adam, but the experiments only show Adam; SGD is dismissed in one sentence without showing data. Second, the convergence claim: the paper says APTS is 'globally convergent' and cites Gross's thesis, but no theorem is stated for the nonconvex DNN setting with L-SR1 Hessian approximations. More importantly, Section 2.2 explicitly concedes that the existing convergence analysis does not extend to the stochastic SAPTS variant, yet the minibatch experiments (which are SAPTS runs) are presented without that caveat in the results. Third, the 'no tuning' claim is weakened by the fact that the method has at least six TR hyperparameters (Δ0, η1, η2, α, β, ν, overlap). Not tuning them is nice, but they still need to be chosen; the paper doesn't report what they were. The scaling claims are based on a single node type and a small number of subdomains, so the communication advantage is plausible but not measured carefully.\n\nThese are fixable problems. The core idea has merit and the paper is honest about the SAPTS limitation. What it needs before the claims can be accepted is a proper empirical study: error bars, a real SGD baseline, a sensitivity table for the TR parameters, and either a convergence proof for the deterministic case or a clear statement that only empirical convergence is claimed.\n\nWho this is for: researchers working on parallel nonlinear optimization for ML, and people interested in trust-region methods as an alternative to SGD/Adam. It deserves a serious referee, but the referee should push for the missing comparisons and a more careful claims-rewrite. My recommendation: send it to review, but condition acceptance on addressing the empirical gaps and softening the 'no tuning' claim.","headline":"A plausible APTS variant for data-parallel NN training, but the accuracy and no-tuning claims outrun the evidence: no error bars, no SGD baseline, and no convergence proof for the minibatch variant.","tokens_in":7763,"tokens_out":1994,"would_cite":false,"duration_ms":18917,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C53","65K05","68T07","68W10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A data-parallel trust-region method trains neural networks without learning-rate tuning and matches fine-tuned Adam on MNIST and CIFAR-10.","keywords":["data-parallel training","trust-region methods","nonlinear preconditioning","domain decomposition","neural network training","hyperparameter-free optimization","L-SR1","parallel deep learning"],"falsifier":"Run APTS with the paper's fixed trust-region settings on a standard image-classification task with many subdomains (e.g., a 16-way split of CIFAR-10) and check whether the full training loss decreases monotonically across accepted iterations and whether final accuracy stays within a couple of points of a well-tuned Adam baseline.","tokens_in":6801,"feed_emoji":"⚙️","tokens_out":5378,"duration_ms":49715,"temperature":0.7,"pith_summary":"The paper proposes a data-parallel trust-region method, APTS, for training deep neural networks without learning-rate tuning. It claims APTS reaches validation accuracy comparable to fine-tuned Adam and SGD on MNIST and CIFAR-10, while reducing communication compared with synchronized parallel SGD. The training data is split into overlapping minibatches, each processed by a local trust-region solver; a consistency term makes each local gradient agree with the global gradient, and step sizes are adapted automatically via trust-region radii. The authors argue this makes the method globally convergent and inherently parallelizable, unlike gradient-descent variants whose convergence depends on delicate hyperparameters.","feed_headline":"Parallel trust-region method trains neural nets without tuning","feed_subtitle":"APTS splits data across workers, adapts step sizes internally, and matches fine-tuned Adam on MNIST and CIFAR-10.","key_machinery":"The key object is the modified local loss (3) with its first-order consistency term (4), together with the acceptance ratio (6). The consistency term makes each local gradient equal to the full global gradient at the start of the outer iteration, so local trust-region steps approximate a global step; the ratio then measures whether the summed local reductions predict the actual global decrease, triggering trust-region radius updates (7). The global step bound (5) keeps the sum of local steps within the global trust-region radius. Local subproblems are solved with an L-SR1 Hessian approximation and the OBS trust-region solver.","core_discovery":"The central claim is that a trust-region method can be made data-parallel through additive nonlinear preconditioning: each worker solves a trust-region subproblem on a modified local loss $f_i(\\theta)=f(\\theta, D_i)+\\langle r_i, \\theta-\\theta_{i,0}\\rangle$, where $r_i$ is the difference between global and local gradients at the current iterate, so the local objective is first-order consistent with the global one. The local steps are summed, and the combined step is accepted only if the ratio $\\rho$ of actual global decrease to summed local predicted decrease exceeds $\\eta_1$. This mechanism transfers the global convergence properties of trust-region methods to the parallel setting and makes explicit step-size selection unnecessary. On MNIST and CIFAR-10 the method matches fine-tuned Adam accuracy with a single fixed set of trust-region parameters.","pith_inferences":["The consistency term is conceptually a variance-reduction device, and one could test whether replacing the full-gradient consistency term with a stochastic estimate would make the stochastic SAPTS variant theoretically tractable.","The observed accuracy degradation with more subdomains suggests a trade-off between parallelism and approximation quality; increasing overlap between minibatches might delay that degradation, a testable prediction.","The method currently applies trust region locally first and then once globally per outer loop; a natural extension would be to make the global trust-region step itself a distributed operation, potentially improving scaling further.","The claim of no hyperparameter tuning is limited to the tested architectures; verifying it on transformers or recurrent networks would be a direct next test."],"forward_implications":["Training deep networks would no longer require per-problem learning-rate schedules; a fixed trust-region configuration suffices across datasets and architectures.","Data-parallel training would communicate only after several local trust-region iterations, reducing synchronization overhead relative to gradient-synchronous SGD and Adam.","The deterministic APTS variant inherits a global convergence guarantee from trust-region and nonlinear-preconditioning theory, something SGD and Adam lack in the nonconvex regime.","The method should scale with the number of subdomains up to the point where local gradients stop being good approximations of the full gradient."],"supporting_citations":[{"why":"Supplies the APTS convergence theory that the proposed data-parallel variant builds on.","marker":"[11]"},{"why":"Provides the trust-region framework and global convergence machinery used by each local and global trust-region solve.","marker":"[5]"},{"why":"The Adam optimizer is the baseline that APTS is compared against; the best-tuned Adam results set the accuracy target.","marker":"[14]"},{"why":"Justifies the L-SR1 Hessian approximation used in the trust-region subproblem.","marker":"[15]"},{"why":"Provides the OBS method used to solve the trust-region subproblems.","marker":"[18]"},{"why":"The variance-reduction idea behind the first-order consistency term (4).","marker":"[23]"},{"why":"The authors' earlier layer-parallel APTS application that this data-parallel variant extends.","marker":"[2]"}],"fun_headline_variants":["Trust-region training goes parallel without hyperparameter tuning","Data-parallel APTS matches Adam, skips tuning entirely","Parallel trust-region approach eliminates tuning in DNN training","New parallel method trains nets with no step-size search","Trust-region meets data-parallelism: accuracy without tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's convergence depends on the assumption that the steps computed locally on each data subset, when added together, still reduce the full training loss reliably; the paper notes this is not proven for the stochastic version.","fun_headline_variants_meta":{"raw":{"variants":["Trust-region training goes parallel without hyperparameter tuning","Data-parallel APTS matches Adam, skips tuning entirely","Parallel trust-region approach eliminates tuning in DNN training","New parallel method trains nets with no step-size search","Trust-region meets data-parallelism: accuracy without tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1319,"prompt_tokens":896,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":342}},"tokens_in":512,"tokens_out":423,"duration_ms":4298,"temperature":1.0,"reasoning_tokens":342,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:06:26.190730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run APTS with the paper's fixed trust-region settings on a standard image-classification task with many subdomains (e.g., a 16-way split of CIFAR-10) and check whether the full training loss decreases monotonically across accepted iterations and whether final accuracy stays within a couple of points of a well-tuned Adam baseline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the APTS convergence theory that the proposed data-parallel variant builds on."},{"cited_title":"Society for Industrial and Applied Mathematics (2000)","cited_arxiv_id":null,"evidence_quote":"Provides the trust-region framework and global convergence machinery used by each local and global trust-region solve."},{"cited_title":"SIAM Journal on Matrix Analysis and Applications, 36, 1338–1359 (2015)","cited_arxiv_id":null,"evidence_quote":"Justifies the L-SR1 Hessian approximation used in the trust-region subproblem."},{"cited_title":"Computational Optimization and Applications, 66, 245–266 (2017)","cited_arxiv_id":null,"evidence_quote":"Provides the OBS method used to solve the trust-region subproblems."},{"cited_title":"A Multilevel Approach to Training","cited_arxiv_id":"2006.15602","evidence_quote":"The variance-reduction idea behind the first-order consistency term (4)."},{"cited_title":"Parallel Trust-Region Approaches in Neural Network Training: Beyond Traditional Methods","cited_arxiv_id":"2312.13677","evidence_quote":"The authors' earlier layer-parallel APTS application that this data-parallel variant extends."}],"review_version":1}