{"id":"a9b33179-a46e-4bec-ba96-c8c72ae96b94","arxiv_id":"2608.09558","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frozen random-weight attention transformers can emulate kernel regression and approximate Hölder functions at minimax-optimal rates, with soft prompts constructed by solving linear systems.","lead":"The paper proves that a single-layer transformer with random, untrained attention weights can approximate any smooth function on a compact manifold, as long as it is given a carefully constructed soft prompt. This matters because it suggests inference-time prompting, not pretraining, can carry the main computation in simplified settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Deferred kernel-estimation bounds (Lemmas 2-3) carry the minimax rate; if their hidden constants or assumptions fail on embedded manifolds with finite reach, the claimed O~(n^{-2alpha/(2alpha+dX)}) rate in Theorems 2, 4, and 6 is unsupported by this text.","rationale":"The reader's weakest_assumption exactly identifies Lemmas 2 and 3 as the unproven load-bearing input, and my pass found no other concern that lands as hard. I checked the algebra in Section 3.1: the linear system (3.5), the Moore-Penrose construction, the logit-shift cancellation (e^{beta} factoring), and the bound (3.7) are all internally consistent under the stated rank condition. The random-rank arguments in Theorems 3 and 5 are also sound: Gaussian-wide matrices have full row rank almost surely, the conditioning on W_K and the independence of w_V from the row space of W_{1:d_in+1} both hold, and the singular value lower bounds via Lemma 4 are standard; the block-diagonal covariance reparameterization in Theorem 5 is legitimate conditionally on (W_Q, M, u_D). The token-norm bounds (O(beta + h^{-2})/sqrt(d_h) and /d_h) follow from the pseudoinverse norm bound and are stated with high-probability caveats, so I do not see an internal inconsistency there. The experiments are plausible corroboration but do not settle the deferred-lemma question because they use only d_X = 2, alpha = 1 and do not independently validate Lemmas 2-3 in a regime where hidden assumptions would matter. One could also quibble that the prompt norm grows polynomially in n (for the rate-optimal tuning, ||p_i^*|| ~ n^{2/(2alpha+dX)}, or ~ n^{2/(2alpha+dX)}/sqrt(d_h) + 1), meaning the 'soft prompt' tokens diverge in norm as accuracy improves; this is a limitation for practice but not a flaw of the stated approximation theorem, since a large-norm prompt is still a legitimate input to the frozen network. Another possible concern is that the query-dependent attention logits after weighting by e^{beta} are assumed to cancel exactly when comparing against bK_h(x); the algebra shows the e^{beta} factor multiplies both numerator and denominator, so the cancellation is exact for the constructed prompt, not asymptotic. That is a point in favor of the construction. Hence the only verdict-movable concern is the deferred Lemmas 2-3, and the correct disposition is the same CONDITIONAL verdict as the reader: accept the prompt-to-kernel mechanism as proven, but require the cited bounds to be made self-contained and verified before the minimax-optimal rate claim can be treated as fully established.","tokens_in":27305,"tokens_out":4581,"duration_ms":34638,"concrete_test":"State and re-derive Lemmas 2 and 3 in full, with all constants and exact conditions on h relative to tau_X; verify (i) Lemma 3's O(h^alpha log(h^{-1})) bias bound holds pointwise and uniformly over X under Assumptions 1-2 without a small-tau_X factor that dominates, and (ii) Lemma 2's high-probability bound integrates with delta = 4h^2. Computationally, on X = S^2 with alpha = 1 and h = n^{-1/4}, compare measured integral-gap and bias to n^{-1/2}(log n)^4 and n^{-1/4}(log n)^2 for n = 64, 256, 1024, 4096, 16384; a clear exponent mismatch would invalidate the deferred lemmas.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's own chain of reasoning is transparent: Theorem 1 gives a deterministic, self-contained approximation of the empirical NW estimator with error O(1/(n^2 e^{2beta}) exp(3R^2/h^2)), and Theorem 3 extends it to random attention. Nothing in that part is suspect. The advertised minimax-optimal rate, however, comes entirely from the four-term decomposition in Theorem 2, where terms II, III, and IV are bounded by Lemmas 1-3. Lemma 1 is proved in the text and is fine. Lemmas 2 and 3 are imported without proof from Shen et al. [2026], with no statement of the dependence on the reach tau_X for Lemma 3, a factor d_X^{d_X/2} for Lemma 2, and a log(h^{-1}) factor in the bias. These are exactly the bounds that convert an O(n^{-2}) prompt-construction error and an O(n^{-2alpha/(2alpha+dX)}) noise/variance error into the headline rate. If the cited paper's manifold kernel bounds require, for example, the kernel to be localized with bandwidth below a constant times the reach, or if the uniformity of Lemma 3 over X fails at the cutoff distance tau_X/2, then the tuning h = n^{-1/(2alpha+dX)} and beta = (3R^2/2)n^{2/(2alpha+dX)} in Theorem 2 does not yield the stated rate. The reader is asked to take these bounds on faith; the text does not re-derive or even state them precisely. This is the single load-bearing concern because every headline claim about minimax optimality collapses if Lemmas 2 or 3 are false or have hidden assumptions, whereas the prompt construction itself is algebraically verified in the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies soft prompting of a single-layer softmax-attention transformer with frozen, randomly initialized weights. For any Hölder function on a compact manifold, the authors construct a query-independent soft prompt, of length equal to the number of support points, whose tokens are minimum-norm solutions of linear systems such as fW p_i = v_i in Eq. (3.5) or fWaff p_i = v'_i in Eq. (4.3). This makes the attention output match the Nadaraya-Watson kernel estimator up to a query self-attention term that is exponentially suppressed by a logit shift β. The construction requires only full row rank of fW, which is shown to hold almost surely for Gaussian attention weights; a further extension covers random affine embeddings and decoders. Building on standard kernel-regression bounds, Theorems 2, 4, and 6 state minimax-optimal squared-error rates O~(n^{-2α/(2α+d_X)}) and quantify prompt-token norms. Numerical experiments on the sphere S^2 illustrate the scaling of the construction and compare it with trained baselines.","tokens_in":27705,"tokens_out":16162,"duration_ms":144812,"significance":"If the stated results hold, the paper makes a striking conceptual contribution: a fixed random single-layer attention network can be steered to approximate a broad function class purely through an explicitly constructed prompt, without any training of the network weights. The core prompt construction is transparent and largely self-contained: the logit-matching equations, the pseudoinverse solution, and the logit-shift suppression are all checked in the text, and the random-matrix rank arguments are standard and sound. The paper also gives concrete prompt-norm tradeoffs in terms of n, d_h, and the Hölder/geometry parameters, and it provides experiments that support the predicted n^{-2} and β-scaling of the kernel-approximation error. The main caveat is that the advertised minimax rate is not derived in this manuscript: it is inherited from Lemmas 2 and 3, which are stated without proof and whose precise hypotheses are not fully specified.","major_comments":[{"comment":"The headline rates in Theorems 2, 4, and 6 rest entirely on the finite-sample variance bound (Lemma 2) and the manifold bias bound (Lemma 3), but neither lemma is proved in this manuscript and their statements are incomplete. Lemma 3 is stated for h > 0 with no restriction relative to the reach τ_X, and the O(h^α log(h^{-1})) bound is asserted uniformly over X without stating the required condition; Lemma 2 similarly does not state whether h ≤ τ_X/2 or whether the high-probability event and the constant are uniform in x. Since the proof of Theorem 2 invokes these lemmas with h = n^{-1/(2α+d_X)} and δ = 4h^2, the minimax claim cannot be verified from the text. The authors should include full proofs or quote the exact theorems from Shen et al. [2026] with all hypotheses, and confirm that the chosen h and δ satisfy them.","section":"Section 3.2, Lemmas 2 and 3"},{"comment":"The transition from Lemma 2's high-probability pointwise statement to the expectation bound (3.10) needs an explicit uniformity statement. As written, Lemma 2 is stated for a fixed x, while the theorem needs a bound on E_{x_i,ε_i,x} of the squared finite-sample gap; the proof uses the bounded-difference inequality with δ = 4h^2 but does not specify whether the O(·) constant in Lemma 2 is uniform over x ∈ X. If the constant depends on x through local reach or other geometric quantities, the expectation step is not justified as written. Please clarify or strengthen Lemma 2 accordingly.","section":"Section 3.2, proof of Theorem 2, Eq. (3.10)"},{"comment":"The paper presents the minimax-optimal rate as a main contribution, but the transformer-specific part of the proof establishes only the O(n^{-2}) prompt-construction error; the minimax rate itself comes from kernel-regression bounds imported from prior work. This division should be stated clearly in the introduction and in the statements of Theorems 2, 4, and 6, so that readers can distinguish the genuinely new prompt-construction mechanism from the inherited kernel-estimator guarantees.","section":"Section 1 and Theorem 2"}],"minor_comments":[{"comment":"The phrase 'with at most exponential dependence on d_X' is too vague for a theorem statement; at minimum, state whether the bound is uniform in x and whether it requires h ≤ c τ_X for a constant c.","section":"Section 3.2, Lemma 3 statement"},{"comment":"For h = 0.2, the text attributes the initial steeper drop to insufficient local support; a brief explanation of why this does not contradict the predicted n^{-2} bound would help the reader interpret the transient.","section":"Section 5, Fig. 2a"},{"comment":"The baselines are trained with Adam while the construction is not; the comparison is informative, but the reader should be reminded that the failure of the baselines in Architecture B may be due to initialization scale rather than fundamental capacity.","section":"Section 5, Fig. 4"},{"comment":"The statements of Theorems 4 and 6 combine an almost-sure rate statement with a high-probability token-norm statement; the proof should explicitly separate the probability space of the initialization from the probability space of the samples so that the constants in the O(·) notation are unambiguous.","section":"Theorem 4 and Theorem 6"}],"recommendation":"major_revision","confidential_remarks":"The core construction is sound and novel, and the deterministic prompt-approximation part is self-contained. My recommendation of major revision is driven by the fact that the advertised minimax rate depends on Lemmas 2 and 3, which are neither proved nor precisely stated in the manuscript; this is a fixable gap rather than a fundamental flaw. The self-citation to Shen et al. [2026] is heavy but not inappropriate, since the cited work concerns kernel estimators on manifolds rather than transformers. If the authors add an appendix with the missing statements and proofs, or explicitly verify the hypotheses needed for their tuning of h and β, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is genuine and worth refereeing: a single-layer softmax attention transformer with frozen Gaussian weights can be prompted, in closed form, to match the Nadaraya-Watson estimator on a compact manifold and thereby approximate any Hölder function at (up to logs) the minimax rate. The prompt construction is the real contribution. It is explicit — minimum-norm solutions of linear systems that align attention logits with Gaussian kernel exponents — and the logit-shifting trick to kill the query self-attention term is elegant and quantitatively tracked. The rank condition is mild and the Gaussian almost-sure proof is standard but correctly handled. The prompt-norm tradeoffs in Theorems 4 and 6 are useful: they say something concrete about how much \"control energy\" prompting costs as width grows. The numerical section is honest: it tests the construction against empirical NW and a few baselines, and the scalings line up. No code is shipped, but the experiments are simple enough to reproduce from the description.\n\nWhere I hesitate is the minimax-rate theorems, Theorems 2, 4, and 6. The four-term decomposition is sound, and Term I (the prompt-to-kernel approximation) is proven in the text with explicit constants. Terms II–IV, however, are handed over to Lemmas 2 and 3, which are stated but not proved, with only a pointer to Shen et al. [2026]. Those lemmas carry the noise/variance and bias bounds on manifolds, and the final rate n^{-2α/(2α+d_X)} is exactly as strong as they are. The stress-test note is right: if the cited bounds require extra conditions on the reach or a localized kernel regime that the paper's tuning h = n^{-1/(2α+d_X)} does not satisfy, the headline rate does not follow from this text. This is a load-bearing gap in terms of self-containedness, not a fatal flaw in the core idea. The prompt construction itself does not depend on those lemmas.\n\nI also read the abstract's \"pretraining is optional\" as an approximation-theoretic statement about a single random attention head, not about LLM pretraining. Anyone who reads it as a claim about real transformers will be misled, but the paper is careful enough in the body.\n\nBottom line: this deserves a serious referee. The right referee report will ask for the precise statements of Lemmas 2 and 3, their dependence on τ_X and d_X, and either proofs in the appendix or a verification that Shen et al. actually provides them under exactly these assumptions. If that check passes, the result is a solid, publishable contribution. If it fails, the construction still stands but the minimax-optimality claim collapses to whatever the cited bounds give. I would send it out, with the deferred lemmas flagged as the main thing to resolve.","headline":"Clean prompt-based universal approximation with random weights, but the advertised minimax rate leans on two imported kernel-regression lemmas that need a closer look.","tokens_in":28197,"tokens_out":3205,"would_cite":true,"duration_ms":28068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62G08","60B20","41A25"],"pacs":[],"model":"deepseek-v4-flash","headline":"A single-layer, never-trained random transformer can approximate any Hölder function on a compact manifold when steered by a soft prompt, at minimax-optimal rates.","keywords":["soft prompting","random transformers","universal approximation","Nadaraya-Watson estimator","kernel regression","Hölder functions on manifolds","minimax-optimal rates","training-free"],"falsifier":"Run the constructed prompt on a sphere with $d_X=2$, $\\alpha=1$, bandwidth $h=n^{-1/4}$ and the prescribed logit shift, measuring MSE over many test queries; if the error does not decay like $n^{-1/2}$ up to log factors as $n$ and $d_h$ grow, the claimed minimax-optimal rate is contradicted.","tokens_in":27086,"feed_emoji":"🎯","tokens_out":9148,"duration_ms":71822,"temperature":0.7,"pith_summary":"This paper tries to establish that pretraining is unnecessary for function regression with transformers: a single-layer softmax attention network whose weights are random and never trained can approximate any Hölder function on a compact manifold, provided it is given a soft prompt built for that function. The prompt is constructed explicitly, not learned: each prompt token solves a small linear system that aligns the attention logits with Gaussian kernel exponents, making the frozen network reproduce the Nadaraya-Watson kernel estimator in its forward pass. Because the network inherits kernel-regression guarantees, the approximation error reaches the minimax-optimal rate $\\tilde{O}(n^{-2\\alpha/(2\\alpha+d_X)})$ in the sample count $n$, where $\\alpha$ is the Hölder exponent and $d_X$ is the intrinsic manifold dimension. The construction needs only a rank condition on the random weights, which holds almost surely under Gaussian initialization, so the result applies to essentially every draw of the weights.","feed_headline":"Random transformer approximates any smooth function via prompt alone","feed_subtitle":"No weight training needed: the prompt encodes a kernel estimator that hits optimal regression rates on manifolds.","key_machinery":"The machinery is the explicit soft prompt as the minimum-norm solution of a linear system that matches attention logits to Gaussian kernel exponents. For each support point $x_i$ with label $y_i$, the token $p_i$ solves $\\widetilde{W}p_i = v_i$, where $\\widetilde{W}$ stacks the first $d_{\\rm in}+1$ rows of the query-key product $W = W_Q W_K^\\top/\\sqrt{d_h}$ with the readout vector $w_V^\\top$, and $v_i$ contains $x_i/h^2$, $-\\|x_i\\|^2/(2h^2)+\\beta$, and $y_i$; in the fully random architecture the analogous system uses the random embedding and decoding maps. A logit shift $\\beta$ added to the bias coordinate suppresses the query's self-attention term exponentially in $\\beta$, and a rank condition on $\\widetilde{W}$ (full row rank) guarantees solvability. This construction turns the attention layer into the Nadaraya-Watson kernel estimator, so the approximation theory of kernel regression on manifolds transfers to prompted transformers.","core_discovery":"The central claim is that a frozen, single-head softmax attention transformer with random Gaussian weights is a universal approximator for Hölder functions on a compact manifold, steered solely by a soft prompt. For any target function (given through noisy samples) the paper builds a query-independent prompt whose tokens are the minimum-norm solutions of the linear systems $\\widetilde{W} p_i = v_i$ and $\\widetilde{W}_{\\mathrm{aff}} p_i = v'_i$, forcing the attention output to match the Nadaraya-Watson kernel estimator up to an exponentially small self-attention term suppressed by a logit shift $\\beta$. This yields universal approximation theorems with rate $\\tilde{O}(n^{-2\\alpha/(2\\alpha+d_X)})$ — the minimax-optimal rate for nonparametric regression on a $d_X$-dimensional manifold — for deterministic attention parameters (Theorem 2), random attention parameters (Theorem 4), and fully random transformers with random affine embedding and decoding (Theorem 6). The prompt construction is explicit and depends on the realized weights, and the paper also tracks the price of prompting: token norms grow like $O(n^{2/(2\\alpha+d_X)})$ in the deterministic case, improving to $O(n^{2/(2\\alpha+d_X)}/\\sqrt{d_h}+1)$ with random attention and $O(n^{2/(2\\alpha+d_X)}/d_h+\\sqrt{d_h})$ for fully random transformers.","pith_inferences":["A testable extension: the same logit-shift construction should transfer to multi-head attention by assigning one kernel per head, which the paper does not prove.","The paper's explicit prompt solves regression with one token per support point; a natural follow-up is to ask whether a single prompt of length $m \\ll n$ can encode a coarser estimator, since the paper's bounds do not rule this out.","The experiments (a trained prompt beating the constructed one) suggest the construction is sufficient but not optimal; if that gap persists at larger $n$, the true cost of prompting may be below the paper's worst-case token norms.","If the mechanism is robust to finite precision, it offers a purely inference-time way to adapt frozen foundation models to new regression tasks without touching weights, which is the practical reading of 'pretraining is optional'."],"forward_implications":["A fixed, never-trained random transformer can perform nonparametric regression in-context: the task-specific information lives entirely in the soft prompt, not in the weights.","Approximation error decays at the minimax-optimal rate for Hölder functions, so prompting alone is not a weaker substitute for training in this setting — it matches the best possible statistical rate up to logarithms.","The construction is explicit and computable via the Moore-Penrose pseudoinverse, so no gradient-based prompt optimization is needed to realize the guarantee.","Prompt length $n$ doubles as the number of support points, so longer prompts give both more data and more steering capacity at the same time.","Fully random embeddings and decoding spread the prompt constraints across the hidden dimension, changing the prompt-norm cost: larger $d_h$ reduces token norms in the random-attention case but eventually increases them in the fully random case."],"supporting_citations":[{"why":"Defines the Nadaraya-Watson kernel estimator that the prompted transformer is constructed to emulate.","marker":"[Nadaraya, 1964, Watson, 1964]"},{"why":"Source of the manifold kernel-regression variance and bias bounds (Lemmas 2 and 3) that carry the minimax rate.","marker":"Shen et al. [2026]"},{"why":"Establishes minimax-optimal rates for local regression on unknown manifolds, the benchmark the paper's rates match.","marker":"[Bickel and Li, 2007]"},{"why":"Supplies the nonasymptotic singular-value tail bounds used to prove the generic rank condition and bound prompt norms.","marker":"[Davidson and Szarek, 2001]"},{"why":"Provides the small-ball probability estimate on manifolds used to bound the noise term of the kernel estimator.","marker":"Niyogi et al. [2008]"},{"why":"Supplies the Gaussian concentration and singular-value estimates used in the fully random setting.","marker":"[Vershynin, 2026]"}],"fun_headline_variants":["No training needed: prompt steers random transformer to universal approximation","Prompt-only transformer with random weights achieves minimax-optimal rates","Random single-layer transformer + prompt = universal function approximator","Prompted frozen transformer: universal approximation without training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pure kernel-regression bounds on manifolds (the variance and bias lemmas imported from the companion paper) are the load-bearing premise; if they are not reproducible as stated, the minimax rate does not follow from this paper's argument.","fun_headline_variants_meta":{"raw":{"variants":["No training needed: prompt steers random transformer to universal approximation","Prompt-only transformer with random weights achieves minimax-optimal rates","Random single-layer transformer + prompt = universal function approximator","Prompted frozen transformer: universal approximation without training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3986,"prompt_tokens":1052,"completion_tokens":2934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":2867}},"tokens_in":668,"tokens_out":2934,"duration_ms":16933,"temperature":1.0,"reasoning_tokens":2867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:08:04.175251+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the constructed prompt on a sphere with $d_X=2$, $\\alpha=1$, bandwidth $h=n^{-1/4}$ and the prescribed logit shift, measuring MSE over many test queries; if the error does not decay like $n^{-1/2}$ up to log factors as $n$ and $d_h$ grow, the claimed minimax-optimal rate is contradicted.","supporting_citations":[],"review_version":1}