{"id":"b9c7cd22-9362-469e-aeb4-9ac0d794170e","arxiv_id":"2501.02791","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Orthogonal greedy training of shallow ReLU networks is adapted to kernel estimation for linear operators, with stated convergence rates and large accuracy gains over neural operator baselines.","lead":"This paper adapts the orthogonal greedy algorithm, a training method for shallow neural networks, to learn the kernel or Green's function of a linear operator from input-output data. It reports large accuracy gains over neural operator baselines and gives a convergence analysis in a new data-dependent semi-inner product.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The key bound is for a semi-norm that ignores any kernel component orthogonal to the training forcing; Lemma 1's positivity holds only for the finite-dimensional surrogate, so the claimed kernel-estimation rate does not control the true Green's function.","rationale":"The paper's positive contribution, a greedy algorithm that fits responses without backpropagation, is plausible: the randomized-dictionary OGA rates are known, and the response-error plots in Section 4 show roughly the predicted n^{-1.25} behavior in several cases. However, the central advertised claim is kernel estimation, and the data-dependent inner product is intentionally degenerate. Lemma 1's proof has a logical gap: linear independence of the forcing functions does not imply injectivity of the map G -> (G*f_1, ..., G*f_N) when G is allowed to be an arbitrary continuous kernel. The kernel can be orthogonal, in the L2 sense on the y-variable, to all training forcing functions while being nonzero; only after forcing G into the finite-dimensional surrogate class of Eq. (22) does the map become injective. This is not a minor technicality: in the reported 2D/3D experiments, eps_G visibly diverges under a larger nullspace (Section 4.3.1), which is direct evidence that the semi-norm does not control the kernel. The reader's conditional verdict is therefore appropriate. Supporting issues include Lemma 5's chain of inequalities, which compares different minimizers without showing that the actual OGA iterate is near the optimal Sigma_n approximation in the L2 norm, and the numerical overclaim in Table 3, where direct OGA is worse than GL on several 2D cases despite the conclusion stating that both OGA and PW-OGA consistently outperform all baselines. A corrected version would need either a quantified modeling-error bound, a theorem stated for response prediction on span{f_j}, or both.","tokens_in":23874,"tokens_out":5319,"duration_ms":58241,"concrete_test":"Take Omega = (0,1), choose N = m_f forcing functions f_j(y) = sin(j pi y), and target kernel G(x,y) = sin((m_f+1) pi y). Then G*f_j = 0 for every j, so ||G||_H = 0 while ||G||_{L2} > 0; Lemma 1's hypotheses hold but positivity fails. Run OGA/PW-OGA on the resulting dataset (u_j = 0) and report ||G - G_n||_H and eps_G (Eq. 78). If the algorithm returns any nonzero kernel, or if eps_G stays O(1) while eps_u = 0, the theorem's interpretation as a kernel-estimation rate is disproved; if instead the claim is restricted to the surrogate class (22), that restriction must be stated in the theorem and the modeling error must be bounded.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theorem is rendered vacuous as a kernel-estimation result by the treatment of the data-dependent bilinear form. The paper defines ||G||_H^2 = (1/N) sum_j ||G*f_j||^2 (Eq. 20) and explicitly notes it is only a semi-norm. Lemma 1 then asserts positivity: if G*f_{j_l} = 0 for m_f linearly independent f_j, then G = 0. This inference is valid only after the identification G = G_hat in Eq. (22), i.e., after restricting to the finite-dimensional piecewise-linear surrogate generated by the discrete forcing samples. For a genuine Green's function, the kernel can be nonzero while lying in the nullspace of every observed forcing function; the paper itself concedes this in Eq. (21) and in the numerical discussion of Sec. 4.3.1, where a larger nullspace makes eps_G diverge while eps_u keeps decaying. The decomposition (25) separates out the modeling error ||G - G_hat||_H, but no quantitative bound on this term is supplied; it is merely declared negligible. Consequently, Theorem 1's rate controls only responses to forcing functions in span{f_j}, not the kernel itself, and the repeated claims about approximating the Green's function, as well as the reported eps_G rates, are not established for the true kernel.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two orthogonal greedy algorithms for linear operator learning with shallow ReLU networks: OGA, which approximates the Green's function directly in a data-dependent semi-inner product, and PW-OGA, which estimates the kernel pointwise as a collection of d-dimensional slice functions. The authors prove approximation-rate estimates for both algorithms (Theorems 1 and 2), claiming rates of order n^{-1/2-(2k+1)/(4d)} and n^{-1/2-(2k+1)/(2d)} respectively, and support these claims with numerical experiments on 1D, 2D, and 3D problems, comparing against GL, FNO, and DON baselines.","tokens_in":24121,"tokens_out":6531,"duration_ms":65150,"significance":"If the kernel-estimation rates were valid for the true Green's function, this would be a valuable contribution: it would supply the first greedy-algorithm-based method for operator learning with rigorous rates that avoid nonconvex optimization. The pointwise reduction from a 2d-dimensional to a d-dimensional approximation problem is a sensible and practically useful idea, and the numerical results for PW-OGA's response error ϵ_u are substantially better than the reported baselines in several 2D and 3D cases. The paper also deserves credit for explicitly acknowledging that its bilinear form is only a semi-inner product and for demonstrating the associated overfitting phenomenon numerically. However, as explained in the major comments, the central theoretical claim — that the method approximates the kernel itself — is not established: the rates control only the response error on the training forcing space, and the numerical claims about ϵ_G are not supported by the theory.","major_comments":[{"comment":"Lemma 1's positivity proof is invalid for a genuine kernel. The statement that G⋆f_{j_l}=0 for m_f linearly independent f_j implies G=0 is only true after identifying G with the finite-dimensional piecewise-linear surrogate Ĝ of Eq. (22), an identification that is made silently. For a true Green's function, any kernel whose y-section lies in the orthogonal complement of span{f_j} in L2(Ω) gives zero semi-norm without being zero; the paper itself concedes this in Eq. (21). Consequently, ⟨·,·⟩_H in Eq. (20) is only a semi-inner product on the kernel space, and Theorem 1's bound (54) controls the response error on the training forcing space, not the kernel error. The decomposition (25) separates out the modeling error ∥G−Ĝ∥_H but supplies no quantitative bound on it; declaring it negligible does not establish convergence of the kernel. This undermines the abstract's claim that the algorithm approximates the Green's function and invalidates the report of ϵ_G rates as consequences of Theorem 1.","section":"Sec. 3.3 and Sec. 3.6, Lemma 1 and Eq. (25)"},{"comment":"Lemma 5 is not a valid chain of inequalities. The middle term inf_{ψ∈N}∥G−Gn+ψ∥_{L2(Ω×Ω)} contains a free variable Gn that is not quantified by an infimum or supremum. Moreover, the second inequality, inf_ψ ∥G−Gn+ψ∥_{L2} ≤ inf_{μ_n∈Σ_{n,M}(D)} ∥G−μ_n∥_{L2}, does not follow from Lemma 2 and is generally false for a fixed suboptimal iterate Gn. Lemma 2 gives ∥G−μ_n∥_H ≤ ∥G−μ_n+ψ∥ for each μ_n and ψ, which shows only that the infimum over Gn of the semi-norm error is bounded by the quotient norm of a best L2 approximant. The intended conclusion that the OGA error is bounded by the L2 approximation rate requires a different argument, and the paper does not provide one. This is load-bearing because it is the step that connects the semi-norm rate to the standard shallow-network approximation rate.","section":"Sec. 3.6, Lemma 5, Eq. (64)"},{"comment":"The proof of Theorem 1 treats ⟨·,·⟩_H as a definite inner product, but the theorem claims validity for the semi-norm case. The recursion (58) divides by ∥Gn−Pn−1Gn∥^2_H; in a semi-inner product this quantity can be zero for a nonzero vector, since span{g_1,...,g_n} may contain elements with zero H-norm. Remark 1 states that one should interpret the projection as projection onto N⊥, but this is not implemented in the proof, and no argument shows that the dictionary elements selected by Algorithm 3 have positive H-norm after such a projection, nor that the orthogonal projection P_n is well-defined and contractive in the semi-norm. Thus the proof does not cover the advertised semi-norm setting.","section":"Sec. 3.6, proof of Theorem 1, Eq. (58)"},{"comment":"The numerical evidence does not support the paper's general claim that both OGA and PW-OGA outperform baselines by orders of magnitude. Table 3 shows that for the 2D Poisson and Helmholtz problems, direct OGA returns ϵ_u = 1.75e-1 and 3.79e-1 on the disk and H-domain, respectively, which is worse than GL (4.05e-2 and 7.13e-2) and sometimes worse than DON. The impressive improvements in Table 3 are specific to PW-OGA, not to OGA. Furthermore, Figs. 9(b) and 10 show that ϵ_G diverges or flattens as the number of neurons increases; the text acknowledges overfitting in Sec. 4.2.2 but still states that both ϵ_u and ϵ_G achieve the optimal rate. The rate claims should be restricted to PW-OGA's ϵ_u in the pre-overfitting regime, and the ϵ_G claims should be removed or substantially qualified.","section":"Sec. 4, Table 3 and Figs. 9–10"}],"minor_comments":[{"comment":"The definition of the surrogate class G is ambiguous: the condition G≈ f∼ = (...) should state that it holds for all f in the finite-dimensional sampling space, and the precise relation between piecewise-linear interpolation and the discrete matrix G≈ is not given.","section":"Sec. 3.3, Eq. (22)"},{"comment":"In the line after Eq. (56), 'sup_{G∈D}|⟨rn−1,G⟩_H|' should read 'sup_{g∈D}|⟨rn−1,g⟩_H|' to avoid confusion between the target kernel and dictionary elements.","section":"Sec. 3.6, proof of Theorem 1"},{"comment":"The notation ε_n(co(K))_H is not defined; the entropy number in Eq. (51) is introduced for a set A⊂H, but the subscript H in Lemma 4 is unexplained.","section":"Sec. 3.6, Lemma 4"},{"comment":"The loop 'for s = 1,2,...' should be 'for s = 1,...,m_u' with m_u defined as the number of response points; otherwise the algorithm appears to have an infinite outer loop.","section":"Sec. 3.8, Algorithm 4"},{"comment":"The captions say 'pair-wise kernel estimation' but the method is introduced as 'point-wise kernel estimation' (PW-OGA); the terminology should be made consistent.","section":"Fig. 9 and Fig. 10 captions"},{"comment":"The quotient norm ∥G∥_{L2(Ω×Ω)}/N is written with a division sign that is not standard notation; the authors should define it explicitly as inf_{ψ∈N}∥G+ψ∥_{L2(Ω×Ω)}.","section":"Sec. 3.6, Eq. (30)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central theoretical claim is not supported as stated: the data-dependent bilinear form is a genuine semi-norm on the true kernel space, so Theorem 1 is a statement about response error, not kernel estimation. The authors should either reframe the contribution as response-operator learning in a quotient space, or add explicit identifiability conditions and quantitative modeling-error bounds to justify the kernel-estimation interpretation. The numerical sections also need to be aligned with the theory: the ϵ_G rates are not covered by the theorems and the direct-OGA performance in Table 3 contradicts the claimed order-of-magnitude improvements. These are fixable within the manuscript's scope by honest reframing, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the point-wise variant is a genuinely useful addition, but the central kernel-estimation theorem doesn't control the true Green's function as stated. The paper is worth refereeing, not desk-rejecting, because the core idea is sound and the flaws are fixable.\n\nWhat's new: replacing the L2 inner product in OGA with the data-dependent semi-inner product (20) and adding the PW-OGA variant. The pointwise version approximates d-dimensional slices rather than a 2d kernel, giving a better rate, and the numerical results for PW-OGA are strong: sub-1e-3 relative errors in 2D, clearly beating GL, FNO, and DON. The authors also deserve credit for being explicit about the semi-norm and its nullspace; the overfitting discussion in Sec. 4.3.1 is honest and consistent with the semi-norm picture.\n\nWhere it breaks: Lemma 1's positivity proof only goes through after identifying G with the finite-dimensional piecewise-linear surrogate G-hat (22). For a true kernel, G can be nonzero and still be annihilated by all sampled forcing functions, so ||G||_H = 0 does not imply G = 0. The decomposition (25) separates out ||G - G-hat||_H, but no quantitative bound on it is given; it's declared negligible. So Theorem 1's rate controls responses to forcing in span{f_j}, not the kernel itself, and the repeated claims about approximating the Green's function outrun the proof. That's the load-bearing soft spot, and it's fixable: restate the theorem as a surrogate-kernel or response-error result, or supply a modeling-error bound.\n\nTwo smaller issues. Lemma 5's displayed inequality has a dangling G_n in the middle term; as printed it's invalid. And Table 3 shows direct OGA in 2D doing worse than GL and far worse than PW-OGA, which contradicts the abstract's \"orders of accuracy improvement\" claim for both algorithms. The conclusion should be narrowed to PW-OGA.\n\nWho's this for: people working on greedy training of shallow nets, and anyone fitting Green's functions with neural networks. The PW-OGA method is worth trying even if the theory is tightened. I'd send it to review with requests to fix Lemma 1/5, bound or explicitly scope the modeling error, and recalibrate the claims about direct OGA.","headline":"A useful point-wise greedy variant and a fixable but real gap in the kernel-estimation theory.","tokens_in":24647,"tokens_out":2344,"would_cite":false,"duration_ms":23020,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["41A65","65D15","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Orthogonal greedy algorithms train shallow neural networks to learn linear operators by estimating their Green's-function kernels, with proved convergence rates that numerical tests match.","keywords":["orthogonal greedy algorithm","Green's function","operator learning","kernel estimation","shallow neural network","semi-inner product","point-wise kernel estimation","weak greedy approximation"],"falsifier":"Train PW-OGA on forcing functions sampled from a low-dimensional subspace (e.g., only even functions or low-frequency modes), then evaluate the learned operator on a forcing function with a significant component outside that subspace; if the predicted response is wrong even though the training semi-norm error is at machine precision, the semi-norm certificate fails for out-of-span kernels. Concretely, one can check whether $\\epsilon_G$ remains large or diverges while $\\epsilon_u$ is small, using the SVD of the forcing data matrix to predict which kernel components are unobservable.","tokens_in":23667,"feed_emoji":"📐","tokens_out":8307,"duration_ms":75497,"temperature":0.7,"pith_summary":"This paper claims that orthogonal greedy algorithms (OGA) can train shallow neural networks to learn linear operators from data by estimating the operator's kernel—the Green's function—directly, without stochastic gradient descent. The key move is a new data-dependent semi-inner product defined by averages over the training pairs, which lets the greedy dictionary search measure how well a candidate kernel reproduces the observed responses. The authors prove that weak OGA in this product converges at rate $n^{-1/2-(2k+1)/(4d)}$ for directly estimating the $2d$-dimensional kernel, and that a point-wise variant (PW-OGA), which fits $d$-dimensional slices of the kernel, improves the rate to $n^{-1/2-(2k+1)/(2d)}$. In their numerical experiments the observed rates match these exponents and beat the deep operator-learning baselines by orders of magnitude in accuracy. If correct, the result turns operator learning into a dictionary approximation problem with certified convergence rates and a transparent dependence on data diversity.","feed_headline":"Greedy neural nets learn Green's functions at optimal rates","feed_subtitle":"Two OGA variants hit theoretical convergence rates and beat deep operator baselines by orders of magnitude in tests.","key_machinery":"The load-bearing object is the data-dependent semi-inner product of Eq. (20), built from the training forcing functions $\\{f_j\\}$ and the integral-transform pairing $G\\star f$. Together with a randomized discrete dictionary of $\\pm\\text{ReLU}^k$ ridge functions (Eq. (38)), it converts kernel estimation into a weak orthogonal greedy algorithm: at each step one searches the random dictionary for the neuron maximizing the semi-inner product against the current residual, then projects orthogonally onto the span of the selected neurons, solving the small linear system of Eq. (50) for the coefficients. The point-wise variant (PW-OGA) applies the same loop to each kernel slice $G_s(y)=G(y|x_s)$, so each subproblem is $d$-dimensional function approximation instead of $2d$-dimensional kernel approximation, which is why its rate has the factor-two improvement.","core_discovery":"The central claim is that linear operator learning is equivalent to estimating the kernel of an integral operator, and that this estimation can be performed by the orthogonal greedy algorithm with randomized dictionaries of ReLU$^k$ ridge functions, provided the Hilbert inner product is replaced by the data-dependent semi-inner product $$\\langle G_1, G_2\\rangle_H = \\frac{1}{N}\\sum_{j=1}^N (G_1\\star f_j, G_2\\star f_j)_{$L^{2}$(\\$\\Omega$)}.$$ In this product, a candidate kernel is judged by how well its integral transforms reproduce the training responses; the null space of the induced seminorm consists of kernels that are invisible to the training forcing functions. The paper proves that weak OGA converges in this seminorm (or norm, when the forcing data span the discrete forcing space) with the near-optimal exponent $n^{-1/2-(2k+1)/(4d)}$, and that the point-wise variant that learns each slice $G(y|x_s)$ separately attains the faster exponent $n^{-1/2-(2k+1)/(2d)}$. The same estimate transfers to the inference error for predicted responses, and the numerical sections report that both algorithms achieve the predicted rates while improving on deep operator-learning baselines by orders of magnitude.","pith_inferences":["Editorial inference: because positivity of the semi-inner product is guaranteed only when the training forcing functions span the discrete forcing space, a kernel component lying outside that span is invisible; a natural stress test is to evaluate the learned operator on forcing functions outside the training span, where the semi-norm certificates say nothing.","The paper's own SVD experiment on the forcing data matrix suggests a practical diagnostic: the rank profile of the data matrix predicts when $\\epsilon_G$ will diverge and could serve as an early-stopping or data-augmentation criterion for PW-OGA.","The faster point-wise rate hints that splitting the kernel into slices could be combined with mesh or domain decomposition, yielding a naturally parallel operator learner; such a combination is not studied in the paper.","A testable extension would be to add probing forcing functions (e.g., localized bumps or random projections) to the training set to shrink the null space and thereby convert semi-norm convergence into genuine kernel convergence; the paper does not propose this."],"forward_implications":["OGA-based kernel estimation gives a training procedure for shallow neural networks that avoids gradient descent and comes with explicit convergence-rate certificates in the data-dependent semi-norm.","Direct kernel OGA converges at $n^{-1/2-(2k+1)/(4d)}$ and PW-OGA at $n^{-1/2-(2k+1)/(2d)}$, so in every dimension the point-wise method is provably faster for the same neuron count (once the number of response points $m_u$ is below $n^{(2k+1)/(4d)}$).","The inference error for unseen responses obeys the same rate as the kernel estimation error, so accuracy on predicted solutions is controlled by the same theory.","A larger null space $\\mathcal{N}$ in the semi-norm speeds up convergence of the response error $\\epsilon_u$ but can cause kernel error $\\epsilon_G$ to diverge; the paper documents this trade-off and ties it to linear dependence of the forcing data.","On the paper's 1D-3D Poisson, Helmholtz, and cosine-kernel tests, both methods reach the predicted optimal rates and improve on the three deep operator-learning baselines by orders of magnitude."],"supporting_citations":[{"why":"Supplies the randomized-dictionary OGA framework and weak-greedy convergence machinery that the kernel algorithms are built on.","marker":"[27]"},{"why":"Gives the sharp $n^{-1/2-(2k+1)/(2d)}$ bound for shallow ReLU$^k$ networks that sets the target rates.","marker":"[28]"},{"why":"Establishes greedy training algorithms for neural networks and their application to PDEs, the direct antecedent of this extension.","marker":"[25]"},{"why":"Defines weak greedy algorithms and the weak-selection condition that Theorem 1 and Theorem 2 rely on.","marker":"[24]"},{"why":"Provides the entropy-based recursion that converts the greedy residual inequality into the $\\epsilon_n$ rate used in the proof of Theorem 1.","marker":"[66]"},{"why":"GreenLearning is the deep-network baseline that learns an empirical Green's function and is compared against in the numerical tests.","marker":"[5]"},{"why":"DeepONet is the operator-learning baseline compared in the numerical tests.","marker":"[6]"},{"why":"The Fourier neural operator is the spectral baseline compared in the numerical tests.","marker":"[7]"}],"fun_headline_variants":["OGA learns kernels with optimal convergence rates","Greedy kernel estimation hits theoretical limits","Shallow nets learn operators via orthogonal greedy","Two greedy algorithms beat deep baselines in operator learning","New semi-inner product unlocks optimal kernel learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true kernel $G$ can be replaced by its piecewise-linear discrete surrogate $\\hat{G}$ with negligible modeling error, and that the training forcing functions span the discrete forcing space so that the new 'inner product' is positive definite; if either fails, zero semi-norm error no longer implies the kernel is learned.","fun_headline_variants_meta":{"raw":{"variants":["OGA learns kernels with optimal convergence rates","Greedy kernel estimation hits theoretical limits","Shallow nets learn operators via orthogonal greedy","Two greedy algorithms beat deep baselines in operator learning","New semi-inner product unlocks optimal kernel learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1837,"prompt_tokens":954,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":814}},"tokens_in":570,"tokens_out":883,"duration_ms":8329,"temperature":1.0,"reasoning_tokens":814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:56.735551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PW-OGA on forcing functions sampled from a low-dimensional subspace (e.g., only even functions or low-frequency modes), then evaluate the learned operator on a forcing function with a significant component outside that subspace; if the predicted response is wrong even though the training semi-norm error is at machine precision, the semi-norm certificate fails for out-of-span kernels. Concretely, one can check whether $\\epsilon_G$ remains large or diverges while $\\epsilon_u$ is small, using the SVD of the forcing data matrix to predict which kernel components are unobservable.","supporting_citations":[{"cited_title":"Randomized Greedy Algorithms for Neural Network Optimization","cited_arxiv_id":"2407.17763","evidence_quote":"Supplies the randomized-dictionary OGA framework and weak-greedy convergence machinery that the kernel algorithms are built on."},{"cited_title":"Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks","cited_arxiv_id":null,"evidence_quote":"Gives the sharp $n^{-1/2-(2k+1)/(2d)}$ bound for shallow ReLU$^k$ networks that sets the target rates."},{"cited_title":"Greedy training algorithms for neural networks and applications to pdes","cited_arxiv_id":null,"evidence_quote":"Establishes greedy training algorithms for neural networks and their application to PDEs, the direct antecedent of this extension."},{"cited_title":"Weak greedy algorithms","cited_arxiv_id":null,"evidence_quote":"Defines weak greedy algorithms and the weak-selection condition that Theorem 1 and Theorem 2 rely on."},{"cited_title":"Entropy-based convergence rates of greedy algorithms","cited_arxiv_id":null,"evidence_quote":"Provides the entropy-based recursion that converts the greedy residual inequality into the $\\epsilon_n$ rate used in the proof of Theorem 1."},{"cited_title":"Data-driven discovery of green’s functions with human-understandable deep learning","cited_arxiv_id":null,"evidence_quote":"GreenLearning is the deep-network baseline that learns an empirical Green's function and is compared against in the numerical tests."},{"cited_title":"Stuart, and Anima Anandkumar","cited_arxiv_id":null,"evidence_quote":"The Fourier neural operator is the spectral baseline compared in the numerical tests."}],"review_version":1}