{"id":"30baeb8c-b435-49c4-96d6-7e413edee133","arxiv_id":"2508.15394","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid LS/GD training scheme for DeepONets, using a factored least-squares solve for the last layer, converges in far fewer work units than Adam-only training on advection, diffusion-reaction, and Poisson benchmarks.","lead":"The paper presents a hybrid least-squares and gradient-descent optimizer for DeepONet neural operators, solving for the last layer with a least-squares step made tractable by a Kronecker factorization. It reports faster convergence than standard Adam training on several PDE operator-learning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wall-clock acceleration claim rests on an unmeasured per-WU overhead estimate; WU comparisons do not establish actual speedup.","rationale":"The reader's weakest_assumption explicitly identified the unsupported 2–5% per-WU overhead claim as a second load-bearing assumption. My stress-test agrees this is the most consequential gap: the paper's practical value rests on actual wall-clock acceleration, yet all experimental comparisons are in WU and the conversion factor is asserted without evidence. The theoretical factorization (Theorem 3, eq. 3.12) appears mathematically sound, and the restrictive condition (3.5) is honestly disclosed in Section 4.3.1. The regularization confound and single-run results are additional weaknesses, but they do not undercut the qualitative optimization speedup shown in the training-loss curves; the overhead concern directly threatens the quantitative acceleration headline. The reader's CONDITIONAL verdict already accounts for this, so no verdict change is needed.","tokens_in":16170,"tokens_out":13587,"duration_ms":140081,"concrete_test":"Measure end-to-end wall-clock time for both Adam-only and LS+Adam on at least one benchmark (e.g., Section 4.1 advection, supervised) to reach a fixed validation L2 error, using the same hardware/software stack. Report per-WU times for each method, total training time to the target error, and repeat across at least 3 random seeds. If the measured per-WU overhead of the hybrid stage substantially exceeds 5%, the WU-based speedup claim must be requantified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LS+Adam accelerates DeepONet training. All experimental evidence is reported in Work Units (WU), where one hybrid WU is defined as five Adam epochs plus one full-batch LS solve (Algorithm 2, Section 4). The only link between WU and wall-clock time is the assertion in Section 4: 'the training time for a single WU in the hybrid stage is 2–5% longer than that of Adam's.' This is presented without timing data, measurement details, or variance. The LS step itself requires a full-batch forward pass through the branch network (P=1000 functions), formation of B^T B and Σ T_k^T T_k, and spectral decompositions; its true cost is not self-evident. If the per-WU overhead were, say, 20–50%, the claimed 10× WU advantage would shrink to roughly 6–8× or less in real time. Since the paper's headline benefit is 'accelerate DeepONet training' (Abstract), the unsupported 2–5% overhead estimate is load-bearing: without it, the experiments only demonstrate faster convergence per WU, not faster convergence per second.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid least-squares/gradient-descent (LSGD) training scheme for DeepONets. Because the DeepONet output is linear in the last-layer matrix C of the branch network, the authors fix the hidden parameters and update C by solving least-squares normal equations. Their main theoretical contribution (Theorem 3) is a factorization of the large LS matrix as A_k = K_{P,Q_k}(T_k ⊗ B) under a Cartesian-product data assumption and a linear-operator locality assumption (3.4)-(3.5). This reduces the normal equations to a Sylvester-type matrix equation whose solution is given in closed form by Proposition 6 via spectral decompositions. Algorithm 2 (LS+Adam) alternates blocks of Adam steps on hidden parameters with full-batch LS solves of the last layer. Experiments on advection, diffusion-reaction, and Poisson problems with supervised and physics-informed losses report that LS+Adam reaches at 10,000 work units the validation L2 error that Adam-only reaches at 100,000 work units.","tokens_in":16505,"tokens_out":11251,"duration_ms":116527,"significance":"The algebraic core is clean and appears correct; the factorization and closed-form solve are nontrivial and could be useful beyond this paper. The scope conditions (3.4)-(3.5) are stated explicitly, and the limitations for nonlinear operators and input-dependent linear operators are acknowledged in Sections 4.2 and 4.3.1. The paper honestly separates the exact LS step from the heuristic alternation with Adam. The main weakness is that the headline 'acceleration' claim rests on an unmeasured conversion from work units to wall-clock time. If the per-WU overhead is as stated, the 10x WU improvement is a substantial practical advance; without timing data, the paper demonstrates only faster convergence per WU.","major_comments":[{"comment":"The sentence 'the training time for a single WU in the hybrid stage is 2–5% longer than that of Adam's' is the only quantitative link between work units and wall-clock time, but no timing measurement is reported. One hybrid WU performs a full-batch LS solve (forward pass over all P=1000 branch inputs, formation of B^T B and T_k^T T_k, and two eigendecompositions), while an Adam-only WU uses batches of 50 functions; the 2–5% overhead is therefore not self-evident. Because the abstract promises accelerated DeepONet training, the experiments should report wall-clock runtimes (or at least measured overhead with variance) for the main benchmarks. Otherwise the 10x WU gap demonstrates faster convergence per WU, not necessarily per second.","section":"§4, Work Unit definition and Algorithm 2"},{"comment":"The convergence curves and the comparisons 'LS+Adam at 10,000 WU vs Adam-only at 100,000 WU' appear to come from single runs, with no number of random seeds, confidence intervals, or variance. Given the stochasticity of Adam and initialization, a few independent runs (e.g., 5) with mean and standard deviation of validation L2 error at the comparison checkpoints would make the empirical claim robust. This is secondary to the wall-clock issue but is needed for a quantitative evaluation.","section":"§4, Figures 2, 5, 7"}],"minor_comments":[{"comment":"The proposition states λ≥0, but the entrywise division in (3.14) requires dA,r dB,s + λ > 0; with λ=0 and singular A or B the formula is undefined. The text later sets λ>0, so the proposition should state that condition.","section":"§3, Proposition 6"},{"comment":"The sentence 'we can find the last layer parameter C ... that minimizes eq. (3.6)' should likely refer to the LS problem (3.11), not the factorization (3.6).","section":"§3, after Proposition 6"},{"comment":"Reference [17] is incomplete: the Adam reference should include J. Ba as a coauthor.","section":"References"},{"comment":"The phrase 'two smaller subproblems — one for the branch network and one for the trunk network' is a bit stronger than what is shown: the method solves a structured matrix equation using spectral decompositions of both Gram matrices, rather than literally decoupling the two networks.","section":"Abstract"},{"comment":"The paper says Adam momentums are maintained across the LS step; since θL is overwritten by the exact least-squares solution, the stale Adam moments for θL may be inconsistent with the new parameter value. A sentence explaining why this does not destabilize training would help.","section":"§4, Algorithm 2 and its description"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the mathematical contribution is sound and the paper is within scope. The main risk is that the central practical claim (wall-clock acceleration) is supported only by an unmeasured 2–5% overhead assumption. If the authors can add timing measurements, I would support acceptance; in the current form, the claim should be softened or the data added. The single-run nature of the experiments is a secondary but fixable weakness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, brief take: this is a real, modest advance for DeepONet training. The LSGD idea is not new (Cyr et al.), and ELM-DeepONet already froze hidden layers and solved an LS problem. What's new is the commutation-matrix factorization that splits the large least-squares system for the branch last-layer weights into two smaller subproblems, plus the closed-form solve via a generalized Sylvester equation. The extension to physics-informed losses for linear operators satisfying (3.5) is also new. The algebra in Section 3 checks out: Theorem 3's factorization and Proposition 6's spectral solve are derived cleanly, and the Tikhonov regularization for rank deficiency is sensible.\n\nThe experiments cover a reasonable set of PDE benchmarks (advection, diffusion-reaction, two Poisson variants, some supervised, some PI). The WU-based comparisons consistently show LS+Adam reaching the same validation error at 10,000 WU that Adam-only reaches at 100,000 WU. That's a real convergence-per-work-unit gain, and the error plots show it's not just training loss.\n\nThe soft spots are real but mostly addressable. The biggest one: the paper's own abstract promises \"accelerate DeepONet training,\" but every comparison is in work units. The only bridge to wall-clock time is the sentence in Section 4 claiming a single hybrid WU costs 2–5% more than an Adam WU. No timing data, no measurement details, no variance. If that overhead is actually, say, 20–50%, the 10x WU advantage shrinks to maybe 6–8x or less in real time. That's not fatal, but it's load-bearing because the headline benefit is speed. The authors should measure wall-clock time per WU for both methods or at least report end-to-end training times.\n\nOther issues: single runs, no error bars; several hyperparameters (J0, I0, λ schedule, ϵ2) appear tuned to the benchmarks; the variable-coefficient Poisson case doesn't satisfy (3.5), so the method can't be applied there — the authors disclose this, which is good. The claim that the method generalizes to any linear operator satisfying (3.5) is correctly scoped.\n\nOverall: the math is sound, the limitations are honestly stated, and the empirical evidence supports a WU-level acceleration. The wall-clock question is the one thing that needs to be nailed down. I'd send this to peer review — it deserves a serious referee — but the revision should include timing measurements and ideally multi-seed runs. Would I cite it? If I work on neural operator training, yes, mostly for the factorization.\n\nReading group? Maybe. It's not a field-changer, but the Kronecker trick is neat.","headline":"Solid math, honest limitations, but the headline acceleration claim needs wall-clock timings before I'd trust the 10x speedup.","tokens_in":16950,"tokens_out":1813,"would_cite":true,"duration_ms":18743,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["47-08","65F45","65Y10","68T07","68T20"],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid least-squares/gradient-descent method cuts DeepONet training compute by a factor of ten.","keywords":["DeepONet","hybrid least squares/gradient descent","operator learning","physics-informed loss","Kronecker product factorization","Sylvester matrix equation","Tikhonov regularization","training acceleration"],"falsifier":"Profile the per-work-unit wall-clock time of LS+Adam against Adam-only on the advection benchmark: if a single LS step (one spectral decomposition of B^T B and Σ ε_k T_k^T T_k plus the entrywise solve) costs more than about 5% of the five Adam epochs it replaces, the claimed speedup at equal error will not materialize in wall-clock time even though the work-unit count to a fixed error drops. Also, trying to apply the closed-form solve to a physics loss whose operator depends on the input function (the variable-coefficient Poisson residual) should fail condition (3.5), so the method's reported","tokens_in":16111,"feed_emoji":"⚡","tokens_out":8299,"duration_ms":83156,"temperature":0.7,"pith_summary":"This paper tries to show that the slowest part of DeepONet training — the last layer of the branch network — can be updated with an exact least-squares solve instead of gradient steps, and that doing so cuts the compute needed to reach a fixed validation error by roughly a factor of ten. The obstacle is that the least-squares matrix over all branch–trunk pairs is too large to form directly. The key move is a proof that under two structural conditions — data that is a Cartesian product of input functions and output points, and linear loss operators that act only on trunk outputs — the large matrix factors into a commutation matrix times a Kronecker product of two small matrices, turning the update into a closed-form matrix equation. Experiments on advection, diffusion-reaction, and Poisson benchmarks, with both supervised and physics-informed losses, support the tenfold acceleration claim.","feed_headline":"Tenfold faster DeepONet training via least-squares last layer","feed_subtitle":"Hybrid LS+Adam needs 10,000 work units to match Adam-only at 100,000.","key_machinery":"The factorization identity A_k = K_{P,Q_k}(T_k ⊗ B) (Theorem 3), where B is the branch pre-output matrix, T_k is the trunk output matrix under the loss operator L_k, and K_{P,Q_k} is a commutation matrix rearranging row order. This identity reduces the large least-squares problem eq. (3.11) to the generalized Sylvester equation (3.12), whose solution is given in closed form (Proposition 6) via spectral decompositions of the two Gram matrices, making the 'LS step' of hybrid training a small dense linear algebra solve.","core_discovery":"The central claim: for vanilla DeepONets, the branch network's last-layer weights can be optimized by an exact least-squares solve, not backpropagation, and this is feasible because the full LS matrix factors as A_k = K_{P,Q_k}(T_k ⊗ B) when the data is a Cartesian product and each loss operator acts only on trunk outputs. The normal equations then reduce to the matrix equation B^T B C^T (Σ ε_k T_k^T T_k) + λ C^T = B^T (Σ ε_k F_k^T T_k), a generalized Sylvester equation solved in closed form by spectral decomposition. The paper further claims LS+Adam—Adam on hidden layers plus this LS solve—matches or beats Adam-only's validation L2 error at 10,000 work units versus 100,000, across advection","pith_inferences":["This factorization is effectively an exact coordinate-descent step on the DeepONet loss; one could analyze hybrid training as alternating minimization and potentially prove convergence-rate statements under strong convexity of the last-layer subproblem — an analysis the paper does not attempt.","The same Kronecker-product trick should transfer to MIONet-style multi-input operators, where the tensor-product structure of the branch outputs would likely yield an analogous commutation-matrix factorization.","The advertised speedup is stated in work units; converting it into wall-clock speedup depends on the LS solve's overhead, so a lightweight implementation that reuses spectral decompositions across LS steps could make the tenfold claim hold in real time as well."],"forward_implications":["On the tested PDE benchmarks, LS+Adam reaches the same or lower validation L2 error at 10,000 work units that Adam-only reaches only at 100,000 work units.","The hybrid method extends to physics-informed losses for linear PDEs, where the residual and initial/boundary operators satisfy the factorization conditions for several tested problems.","The regularization term λ‖C‖² makes the LS system full-rank and can be annealed over training, for both supervised and unsupervised settings.","The factorization splits the LS problem into branch and trunk subproblems, so the cost of an LS step is governed by the small matrices B and T_k rather than the full P·Q_k × I·J system.","The scheme is limited to losses of the sum-of-squared-l2 form with linear operators acting on trunk outputs; the nonlinear reaction term and the variable-coefficient Poisson physics loss are explicitly outside its scope."],"supporting_citations":[{"why":"Defines the DeepONet architecture and its operator universal approximation property, the object the method optimizes.","marker":"[24]"},{"why":"Introduces the hybrid LSGD alternating scheme for neural networks that the paper adapts to DeepONets.","marker":"[9]"},{"why":"Proves the operator universal approximation theorem that justifies the branch–trunk inner-product structure with a linear last layer.","marker":"[6]"},{"why":"Supplies the commutation matrix properties used to factor A_k as a Kronecker-product times a permutation.","marker":"[28]"},{"why":"Provides the Kronecker product identities used to derive the normal-equation matrix form.","marker":"[29]"},{"why":"Classifies the generalized Sylvester equation type that the reduced matrix equation belongs to.","marker":"[7]"},{"why":"Establishes physics-informed DeepONets and supplies the PI-loss terms and FDM reference data used in the unsupervised benchmarks.","marker":"[35]"},{"why":"Prior extreme-learning-machine DeepONet work that the paper contrasts with, since it fixes hidden weights randomly rather than alternating with gradient descent.","marker":"[33]"},{"why":"Introduces physics-informed neural networks, the source of the PI-loss concept generalized to operators here.","marker":"[30]"}],"fun_headline_variants":["Least-squares last layer boosts DeepONet training 10x","Hybrid LS+Adam hits Adam accuracy at one-tenth the workload","DeepONet: exact last-layer LS solve gives 10x training speedup","Faster DeepONets: solve last layer via LS, use Adam for hidden","LS+Adam reduces DeepONet training work by 90%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Everything rests on the loss data being a Cartesian product of input functions and evaluation points with each linear loss operator acting only on trunk outputs; if either fails, the closed-form least-squares update cannot be formed, and the claimed speedup also assumes the LS step's wall-clock overhead stays at only a few percent per work unit.","fun_headline_variants_meta":{"raw":{"variants":["Least-squares last layer boosts DeepONet training 10x","Hybrid LS+Adam hits Adam accuracy at one-tenth the workload","DeepONet: exact last-layer LS solve gives 10x training speedup","Faster DeepONets: solve last layer via LS, use Adam for hidden","LS+Adam reduces DeepONet training work by 90%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1379,"prompt_tokens":725,"completion_tokens":654,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":469,"tokens_out":654,"duration_ms":6666,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:55:59.888585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Profile the per-work-unit wall-clock time of LS+Adam against Adam-only on the advection benchmark: if a single LS step (one spectral decomposition of B^T B and Σ ε_k T_k^T T_k plus the entrywise solve) costs more than about 5% of the five Adam epochs it replaces, the claimed speedup at equal error will not materialize in wall-clock time even though the work-unit count to a fixed error drops. Also, trying to apply the closed-form solve to a physics loss whose operator depends on the input function (the variable-coefficient Poisson residual) should fail condition (3.5), so the method's reported","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DeepONet architecture and its operator universal approximation property, the object the method optimizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the hybrid LSGD alternating scheme for neural networks that the paper adapts to DeepONets."},{"cited_title":"Chen and H","cited_arxiv_id":null,"evidence_quote":"Proves the operator universal approximation theorem that justifies the branch–trunk inner-product structure with a linear last layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the commutation matrix properties used to factor A_k as a Kronecker-product times a permutation."},{"cited_title":"Neudecker, The Kronecker matrix product and some of its applications in econometrics, Statistica Neerlandica, 22 (1968), pp","cited_arxiv_id":null,"evidence_quote":"Provides the Kronecker product identities used to derive the normal-equation matrix form."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classifies the generalized Sylvester equation type that the reduced matrix equation belongs to."},{"cited_title":"W ang, H","cited_arxiv_id":null,"evidence_quote":"Establishes physics-informed DeepONets and supplies the PI-loss terms and FDM reference data used in the unsupervised benchmarks."},{"cited_title":"Son, ELM-DeepONets: Backpropagation-free training of deep operator networks via extreme learning machines , IEEE Access, 13 (2025), pp","cited_arxiv_id":null,"evidence_quote":"Prior extreme-learning-machine DeepONet work that the paper contrasts with, since it fixes hidden weights randomly rather than alternating with gradient descent."},{"cited_title":"Raissi, P","cited_arxiv_id":null,"evidence_quote":"Introduces physics-informed neural networks, the source of the PI-loss concept generalized to operators here."}],"review_version":1}