{"id":"18ec6f63-5f71-478b-9544-bdd39afa3d1c","arxiv_id":"2602.20555","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Standard Transformers attain the minimax optimal rate m^{-2γ/(2γ+dn)} (up to logs) for nonparametric regression of Hölder C^{s,λ} targets on [0,1]^{d×n}.","lead":"This paper proves that standard softmax Transformers can approximate smooth Hölder functions and, as a consequence, achieve the statistically optimal rate in nonparametric regression with such targets. The result is the first quantitative C^{s,λ} approximation rate for Transformers and extends prior work from merely continuous targets to higher-order smoothness.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1 invokes Lemma 9 on centered cells outside its stated [0,1] domain; the missing rescaling is load-bearing for Theorem 1 and hence Theorem 3.","rationale":"The reader's weakest_assumption isolates the same issue I would flag. The monomial approximation lemma domain mismatch is the place where the proof's central construction is least secure: unlike the typographical/cardinality issues in the combinatorial constants and the probability exponent, this is an actual unsupported invocation in the chain (9)→Step 6→Proposition 1→Theorems 1/2→Theorem 3. I checked whether a rescaling obviously repairs it: the cell width is (1−δ)/K, so K\\bar X is in [0,1]^{dn}; homogeneity of monomials then gives f_i(y) = K^{−|α|} g_i(Ky) with error K^{−|α|} ε_1. The only extra cost is an O(K) factor in the FNN's first-layer weights and no change in asymptotic width/depth; after Lemma 8 this lifts B_FF from O(1) to O(K), still below the K^{6dn+2} bound already present from the memorization step. Thus the concern probably does not invalidate the theorem, but it is a genuine gap in the written proof. Other flagged items—the d/(2s+d) exponent in Proposition 2, the C^{dn−1}_{s+dn−1} vs C^{dn}_{s+dn} notation, and the positional-encoding typo in Lemma 7—are local and evidently fixable; they do not affect the rate argument once corrected. Therefore I do not move the reader's verdict: CONDITIONAL is appropriate until the rescaling is written out and the induced weight bounds are verified.","tokens_in":53020,"tokens_out":26039,"duration_ms":237536,"concrete_test":"Analytically re-derive Proposition 1 Step 3 with the explicit rescaling z = K\\bar X/(1−δ) and f_i^{mnm}(y) = K^{−|α_i|} g_i(Ky), where g_i is the Lemma 9 approximant of z^{α_i}. Verify (i) |f_i^{mnm}(y) − y^{α_i}| ≤ C ε_1 on Ω_j, and (ii) after applying Lemma 8, the Transformer T_i^{(mnm)} has depth/width as stated and weight bound B_FF = O(K) (not O(1) as claimed in Step 3); then re-check that taking the max with T^{(prl−mmr)}'s weight bound C(f)d^2n^7K^{6dn+2} preserves Proposition 1's stated B_FF and M_FF for K = Θ(ε^{−1/γ}). If (i) or (ii) fails, Theorem 1's construction is invalid; if both pass, the gap is a missing detail and the conditional verdict can be upgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing gap is in Proposition 1, Step 3. For X ∈ Ω_j, the proof defines \\bar X = X − X^{(j)} and invokes Lemma 9 to claim |f_i^{mnm}(\\bar X^{flt}) − \\bar X^{α_i}| ≤ ε_1. But Lemma 9 is stated only for inputs in [0,1]^{dn}; the entries of \\bar X lie in [0,(1−δ)/K]^{dn}, and for K > 1 they are not in the lemma's domain. No rescaling or shift is described, and the proof does not track how the induced scaling of the FNN's weights and the monomial's homogeneity affect the width/depth/weight bounds later used in Steps 4–7. Inequality (9) is therefore unsupported. Step 6 uses (8)–(10) to conclude the cellwise error is ≤ ε; if (9) fails, the Taylor-term matching fails, so the cellwise approximation in Proposition 1—and hence Theorems 1, 2, and the approximation step of Theorem 3—is not established. This is fixable in all likelihood: taking z = K\\bar X/(1−δ) and f_i(y) = K^{−|α_i|} g_i(Ky) with g_i from Lemma 9 gives error O(ε_1) and scale factors that increase the FNN weight bound only by O(K), which is absorbed by the K^{6dn+2} term in B_FF. But this repair is absent, so as written the central proof has a missing load-bearing step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims the first quantitative approximation rates for standard (softmax) Transformers approximating Hölder functions in C^{s,λ}([0,1]^{d×n}) under L^t norms, and uses these rates to show that a least-squares estimator over a Transformer class achieves the minimax optimal rate m^{-2γ/(2γ+dn)} in nonparametric regression. The proof constructs a Transformer cellwise from Taylor polynomials of the target, with grid-point memorization (Lemma 7) and ReLU FNN monomial approximation (Lemma 9) as building blocks, then applies empirical-process covering-number bounds. Intermediate results include memorization capacity and a Lipschitz/covering estimate for standard Transformers. The overall strategy is coherent, but as written several load-bearing steps are not justified.","tokens_in":53336,"tokens_out":17918,"duration_ms":147169,"significance":"If repaired, this would be a substantial contribution: it moves Transformer approximation theory from C^{0,λ} to general C^{s,λ} targets and provides the first minimax-optimal regression-rate proof for standard softmax Transformers without attention bias. The modular proof — memorization, parallelization, Lipschitz bounds, and covering-number estimates — contains ingredients of independent interest. The paper also gives unusually detailed size/dimension/weight specifications. However, the listed gaps are in the central derivation, so significance is conditional on those being fixed.","major_comments":[{"comment":"For X∈Ω_j, the proof sets \\bar X = X-X^{(j)} and applies Lemma 9 directly. But the entries of \\bar X lie in [0,(1-δ)/K]^{dn}, not [0,1]^{dn}; Lemma 9 is stated only on [0,1]^d. No rescaling or shift is described, and the induced changes in the monomial and FNN weights are not tracked in Steps 4–7. Inequality (9) is therefore unsupported. A repair such as z=K\\bar X/(1-δ) with f_i(y)=K^{-|α_i|}g_i(Ky/(1-δ)) is plausible, but it is absent and the stated error and weight bounds do not follow as written.","section":"§3.1, Step 3 (Eq. (9))"},{"comment":"Lemma 7 defines the positional encoding as E=3r√d [1_{d×1},2_{d×1},…,n_{d×1}]. With r=√d this gives entries 3kd, whereas Step 2 invokes Lemma 7 on X^{(j)} + 1_{d×1}eBEB with eBEB=(3,6,…,3n), i.e. entries 3k. The proof of Lemma 7 itself in (14) uses ||E_{:,k}||≈3kr, which is consistent with the Step 2 encoding but not with the stated E (whose norm is 3krd). Thus the memorization claim used in Proposition 1 is not exactly covered by the lemma as stated; either the lemma’s E or the Step 2 encoding must be corrected.","section":"§3.1, Step 2; Lemma 7"},{"comment":"The proof defines the Taylor index set as {α:∑α_{uv}≤s} but states its cardinality as C_{s+dn-1}^{dn-1}. In dn variables, the number of multi-indices with |α|≤s is C_{dn+s}^{dn}, while C_{dn+s-1}^{dn-1} counts only |α|=s. If the intended polynomial is of order s−1, the count should be C_{dn+s-1}^{dn}. As written, the sums in Step 6 run over a set that omits all lower-degree monomials, so the identification with the Taylor polynomial in (8) and the Step 6 triangle bound are invalid. This affects Proposition 1 and Theorems 1–2, although the asymptotic ε-exponents would likely survive a corrected count.","section":"§3.1, multi-index set and Step 6"},{"comment":"The monomial approximation step does not cover the constant term or the low-smoothness cases. Lemma 9 requires \\bar α=|α|≥1 to define its depth/width, but the Taylor polynomial includes α=0. Moreover the proof uses 3^{⌈log2(s−1)⌉−1}, which is undefined for s=0 or 1, while Theorems 1–2 claim the result for s∈N_{\\ge0}. Thus C^{0,λ} and C^{1,λ} targets are not handled by the construction as written. A separate treatment of α=0 (identity/constant FNN) and of \\bar α=1 is required.","section":"§3.1, Step 3; Lemma 9"}],"minor_comments":[{"comment":"The proof mixes the sample size notation: several displays use n where m is the sample size, e.g. the definition of Z_g and the sup over 1/n ∑_{i=1}^n. Please correct to m throughout.","section":"§6, Proof of Proposition 2"},{"comment":"The phrase “s−1 order Taylor polynomial” is inconsistent with the stated index set |α|≤s and with the cardinality formula. Please clarify the intended order and align the notation.","section":"§3.1 / §3.3"},{"comment":"In the sensitivity bound for feedforward blocks, the sentence “From the derivation of Lemma 24” should refer to Lemma 28, not Lemma 24.","section":"§7, Lemma 29 proof"},{"comment":"There are numerous typos (“recursviely”, “defiend”, “resepctively”, “ahieve”, “contruct”, “appromants”) that should be corrected in a revision.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising and modular proof strategy, and the claimed results are likely repairable. However, the four gaps above are in the central chain from the cellwise approximation to the regression rate, so the manuscript is not ready for acceptance in its current form. I recommend major revision, with careful attention to the multi-index cardinality and the s=0,1 cases."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a real step forward for transformer theory. It gives the first quantitative approximation rates for standard softmax transformers on Hölder classes C^{s,λ} with s≥1, and this yields the minimax rate in nonparametric regression. The proof is long but the architecture is standard; the fine-grained size/dimension tracking is a genuinely useful contribution, and the memorization result for positionally-encoded inputs improves on earlier work. The empirical-process part of the regression proof is standard and, modulo a few typos, sound.\n\nThe stress-test's main objection, however, does not survive reading the paper. In Proposition 1, Step 3, the inputs \\bar X are X - X^{(j)} where X ∈ [β/K, (β+1-δ)/K]^{dn}. Since X^{(j)} = β/K, the components of \\bar X lie in [0, (1-δ)/K]^{dn}, which is a subset of [0,1]^{dn}. Lemma 9 is stated for all x ∈ [0,1]^d, so it applies directly. No shift or rescaling is missing. That part of the proof is fine.\n\nThe real soft spot is the positional encoding in Step 2. The paper uses e_{BEB} = [3,6,...,3n], so the added matrix is 1_{d×1} e_{BEB} = 3[1,...,n] ⊗ 1_{d×1}. Lemma 7's positional encoding E, with r=√d, is 3r√d [1,...,n] ⊗ 1_{d×1} = 3d [1,...,n] ⊗ 1_{d×1}. The factor d is missing. That matters: in the proof of Lemma 7, the separation between tokens at different positions relies on the norm of (k-l)p 1_{d×1}, which is |k-l| p √d. With p=3/d instead of 3, that becomes 3|k-l|/√d, and for d large it can fall below the required 1/K. The memorization lemma then may fail exactly in the regime where the approximation theorem needs it. This is fixable—by choosing e_{BEB} = 3d[1,...,n], or by adjusting the scaling in the FNN—but as written it's a genuine gap.\n\nThere are smaller issues: Proposition 2's probability bound has a typo (d/(2s+d) instead of d/(2γ+d)), and the constants in the entropy estimate deserve a careful check. These are minor. The central line of reasoning—Taylor expansions, memorization of grid data, monomial approximation, and covering-number entropy—is sound, and the result is likely correct after modest revisions.\n\nI would send this to serious peer review. The contribution is important enough that these fixable technical issues should not block it. A reviewer should verify the positional-encoding fix and the resulting bounds on B_FF.","headline":"Serious transformer approximation/regression paper with a real but fixable bug in the memorization step; the stress-test's domain objection does not hold, but the positional-encoding mismatch does.","tokens_in":53860,"tokens_out":5234,"would_cite":true,"duration_ms":46148,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["41A25","62G08","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard Transformers reach the minimax-optimal rate for smooth nonparametric regression.","keywords":["Transformer approximation","Hölder spaces","nonparametric regression","minimax optimal rate","softmax attention","memorization capacity","covering numbers","Lipschitz constant"],"falsifier":"For d=n=1, s=1 (a quadratic target), build the Transformer from Proposition 1 and evaluate it at a point X on the boundary of a grid cell, where X−X^{(j)} is negative and has magnitude (1−δ)/K; if the entrywise error is not O(ε), or if the monomial network from the lemma is tested on [−c,c] and its error grows with c, then the ε-approximation theorem and the m^{-2γ/(2γ+dn)} regression rate collapse.","tokens_in":52856,"feed_emoji":"📈","tokens_out":6597,"duration_ms":60001,"temperature":0.7,"pith_summary":"The paper tries to prove that a Transformer in its standard form—softmax self-attention, ReLU feedforward blocks, no task-specific redesign—can do two things. First, it can approximate any Hölder function C^{s,λ} on [0,1]^{d×n} entrywise in L^t to arbitrary precision, with a parameter count that improves as the smoothness γ=s+λ grows. Second, plugging that approximation into least-squares regression yields an estimator whose excess risk is O(m^{-2γ/(2γ+dn)}(log m)^2), matching the classical minimax lower bound up to log factors. A sympathetic reader would care because it gives a quantitative theoretical reason for why plain Transformers are expressive enough to compete with specialised nonparametric methods.","feed_headline":"Standard Transformers hit minimax-optimal regression rates","feed_subtitle":"A cell-by-cell Taylor construction shows plain Transformers learn Hölder-smooth functions as fast as classical estimators, up to log factors","key_machinery":"The load-bearing mechanism is a cell-wise Taylor approximation: a discretization feedforward block maps each grid cell to its grid point; a memorization Transformer built on contextual mapping outputs the Taylor coefficients at those points; and ReLU feedforward networks evaluate monomials and accumulate coefficient×monomial products. The size tuple and dimension vector are the bookkeeping devices that state exactly which widths, depths, heads, and dimensions are needed. For regression, the minimax rate follows by bounding the covering number of the Transformer class through a Lipschitz estimate for the whole architecture.","core_discovery":"Central claim: standard Transformers approximate any Hölder C^{s,λ} function on [0,1]^{d×n} to precision ε using O(ε^{-dn/γ}) parameters and O(log 1/ε) depth, in L^t and L∞ (Theorems 1–2); and a least-squares estimator over such Transformers has excess risk O(m^{-2γ/(2γ+dn)}(log m)^2), the minimax rate (Theorem 3). The construction partitions the cube into grid cells, approximates the target by Taylor polynomials on each cell, and uses softmax attention as a memorization mechanism to retrieve the needed Taylor coefficients.","pith_inferences":["Editorial: The proof suggests softmax attention is being used as an addressing/memorization primitive; if that is the true role, the approximation scheme may extend to anisotropic or manifold-structured targets by replacing the grid with a more efficient covering.","Editorial: Because the effective dimension is dn (the flattened token count), the rate does not exploit sequential structure of tokens; a target with smoothness defined along the sequence axis might yield faster rates than dn suggests.","Editorial: A concrete testable extension is to instantiate the constructed Transformer for small d, n, s and check whether the empirical L² approximation error tracks ε^{-dn/γ}; plateauing at cell boundaries would pinpoint the monomial-domain issue."],"forward_implications":["Approximation error scales as ε^{-dn/γ}: smoother targets (larger γ=s+λ) are learned with fewer parameters, and the exponent reveals the effective dimension dn of the flattened token space.","In regression, Transformers attain the same rate as classical kernel, spline, and neural estimators for Hölder classes, up to log factors, under i.i.d. Gaussian noise and bounded responses.","Pointwise (L∞) approximation is possible but costs an exponential-in-dn factor in width and depth compared with L^t approximation.","The Lipschitz and covering-number bounds provide quantitative control over the whole Transformer class, not just individual instances, so the same tools can be reused for generalization and optimization questions.","The memorization result improves on earlier memorization constructions by removing bias terms from the attention layers."],"fun_headline_variants":["Transformers hit minimax-optimal rates for smooth functions","Plain Transformers match optimal statistical rates","Minimax-optimal regression with standard Transformers","Transformers proven minimax-optimal for Hölder targets","Softmax attention yields minimax-optimal learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The approximation theorem assumes that a monomial-approximation lemma proved for inputs in [0,1]^{dn} still gives O(ε) accuracy on the shifted, possibly negative cell coordinates (X−X^{(j)}) used in the Taylor expansion; the paper does not spell out the rescaling, and if that bound fails the cell-wise error budget does not close.","fun_headline_variants_meta":{"raw":{"variants":["Transformers hit minimax-optimal rates for smooth functions","Plain Transformers match optimal statistical rates","Minimax-optimal regression with standard Transformers","Transformers proven minimax-optimal for Hölder targets","Softmax attention yields minimax-optimal learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0008,"raw_usage":{"total_tokens":3352,"prompt_tokens":738,"completion_tokens":2614,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":2550}},"tokens_in":482,"tokens_out":2614,"duration_ms":17180,"temperature":1.0,"reasoning_tokens":2550,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:19:24.976510+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For d=n=1, s=1 (a quadratic target), build the Transformer from Proposition 1 and evaluate it at a point X on the boundary of a grid cell, where X−X^{(j)} is negative and has magnitude (1−δ)/K; if the entrywise error is not O(ε), or if the monomial network from the lemma is tested on [−c,c] and its error grows with c, then the ε-approximation theorem and the m^{-2γ/(2γ+dn)} regression rate collapse.","supporting_citations":[],"review_version":1}