{"id":"877b687f-2693-4739-b4dc-af9854b8ccc6","arxiv_id":"2411.19468","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Random feature models with activations parameterized as weighted sums of basis functions achieve lower test loss than fixed-activation random feature baselines, with theoretical guarantees on approximation and sample complexity.","lead":"This paper proposes random feature models whose activation function is a learnable weighted sum of basis functions such as radial basis functions, splines, or polynomials, and derives approximation and generalization bounds for the RBF version. The work matters because it offers a cheap way to make random feature models adapt to a task, with experiments showing consistent gains over fixed-activation random feature baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiments (N=16, h=0.25, M=1000-3000) sit an order of magnitude outside the regime of Prop. 4.1/Thm. 4.2 (h=Theta(epsilon), N=Theta(M*polylog)); at those hyperparameters the theory's bound is vacuous, so the empirical claims are neither covered by nor evidence for the theory.","rationale":"I read the paper in good faith. The theoretical core - the kernel formula (Thm. 3.1), the Taylor/feature-map analysis (Thm. 3.2, Cor. 3.3), the Gaussian-UAT activation approximation (Prop. 4.1), the Monte-Carlo concentration (Thms. 3.4/4.2), and the Rademacher generalization bound (Thm. 5.1) - is internally coherent. I spot-checked the kernel formula in limiting cases (c=0, x=x'=0, d=1) and the error decomposition inside Prop. 4.1's proof; no inconsistency surfaced. The e^{Theta(.)} notation is the main source of confusion: read literally, 'N=e^{Theta(M)}' and 'M=e^{Theta(sqrt(n))}' (Secs. 4-5) contradict the explicit bounds, which give N=Theta(M*polylog) and M=Theta(sqrt(n)*polylog) under the footnote's 'hides log factors' convention. The intended reading is recoverable, but it obscures the actual quantitative content of the rate claims. The load-bearing weakness is the theory-experiment link. The headline experiments use N=16 and h=0.25 while Prop. 4.1 requires N to grow with M and h to shrink with epsilon; plugging the experimental values into Eq. (29) yields a guaranteed error of order one. So Theorem 4.2 cannot explain the Table 1 results, and reading Figure 1 as 'substantiating Proposition 4.1' overreaches. The paper itself concedes the regime departure in Sec. 4. The synthetic recovery experiment (Sec. 6.3) is the closest to the theory's regime but uses non-Lipschitz targets, outside F_K. This matters because the central claim is joint: a worst-case theory plus an empirical superiority claim under the same model. My proposed check - recomputing the theoretical error envelope at the experimental hyperparameters - settles whether the gap is vacuity by an order of magnitude (I believe it is) or merely cosmetic. I found no reason to escalate to REJECT: the theorems are plausibly correct, the techniques are standard in the random-feature literature, and the gaps are addressable by matched experiments. The paper earns credit for the closed-form kernel computation and the unusually complete proofs. Secondary caveats (constraint set depending on unknown target constants; the implemented objective being unconstrained and nonconvex; checklist items asserting code and hyperparameters that are not present) reinforce CONDITIONAL without changing it. My concern is the same one the reader identified, sharpened with the quantitative computation.","tokens_in":46467,"tokens_out":39817,"duration_ms":317647,"concrete_test":"Recompute Prop. 4.1's two inequalities (Eq. 29) with the paper's own experimental hyperparameters (K=[-2,2], N=16, h=4/N=0.25) and O(1) class constants L, ||sigma||_inf, R, solving for the largest epsilon both conditions permit; then form Theorem 4.2's L1-error envelope 18(||sigma||_inf*R+epsilon)*sqrt(log(4/delta))/sqrt(M) + epsilon for M=1000 and M=3000 and compare it with Table 1's test losses on the same datasets. If the envelope exceeds the reported losses by an order of magnitude, the experimental regime lies outside the theory and the reader's conditional verdict is confirmed; if it is tight, the regime gap is smaller than claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim joins a worst-case theory (O(epsilon) approximation and excess risk with M=Theta(1/epsilon^2), N=Theta(M*polylog), h=Theta(epsilon)) to experiments at N=16, h=4/N=0.25, M=1000-3000; the link between the two is the weakest point. Substituting the experimental hyperparameters into Prop. 4.1's conditions (Eq. 29) - h <= epsilon/(4*sqrt(2)*L*sqrt(log(16||sigma||_inf*R/epsilon))) and |K|/N <= epsilon*h*sqrt(pi*e)/(16*sqrt(2)*||sigma||_inf*R*log(8||sigma||_inf*|K|*R/(sqrt(2*pi)*epsilon*h^2))) - with |K|=4 forces epsilon >= O(1) from the h-condition and epsilon >= 0.3 from the grid condition. Hence Theorem 4.2's bound is O(1), vacuous against the Table 1 losses. The paper concedes the departure ('a very humble number of basis functions N are sufficient', Sec. 4), yet Figure 1 is read as 'substantiating Proposition 4.1', which overreaches. Outside the theory's coverage, the observed gains are equally consistent with the RBF-mixture feature class per se, the (||a||^2-||v||^2)^2 regularizer's implicit bias, or asymmetric baseline tuning; learnable activation is not isolated as the cause. The only in-regime experiment (Sec. 6.3: N=400, h=0.005) uses non-Lipschitz targets (indicator functions), violating F_K's assumptions. Secondary: Thm. 5.1 bounds a constrained ERM whose feasible set (Eq. 9) uses target-dependent constants unknown to the learner; the implemented objective (Eq. 33) is unconstrained and nonconvex; the checklist asserts code and training details not present in the text. None of these refute the theorems - I checked the internal structure of Prop. 4.1's Riemann-sum argument, Thm. 3.4/4.2's concentration, and Lemma D.2's Rademacher bound, and they are coherent - but they split the central claim into an uninstantiated theory and an unexplained empirical finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Random Feature models with Learnable Activation Functions (RFLAF), where the activation is a weighted sum of basis functions (RBFs in the theory). It derives an analytic kernel for a single RBF activation, gives approximation bounds for multiple RBFs via the Gaussian universal approximation theorem, and proves excess-risk and sample-complexity guarantees showing that M = Theta(1/epsilon^2) random features and n scaling as eTheta(1/(epsilon^2 h^2)) suffice (with 1/h = eTheta(1/epsilon)). Experiments on seven datasets compare RFLAF with RBF, B-spline, and polynomial bases against fixed-activation random feature models, and also consider unfreezing the first layer for two-layer networks. The paper claims consistent empirical improvements and validates the learnable-activation mechanism.","tokens_in":46847,"tokens_out":6481,"duration_ms":59850,"significance":"The theoretical framework is a meaningful step: it provides a closed-form kernel for a single RBF activation (Theorem 3.1), a universal-approximation-based path from activation approximation to function approximation (Proposition 4.1 and Theorem 4.2), and Rademacher-complexity-based excess-risk bounds (Theorem 5.1). The proofs are detailed and the central derivation chain appears internally coherent. If the theory held for the demonstrated regime, the result would extend random feature methodology to learnable activations with quantitatively stated costs. The main weakness is that the experimental regime lies far outside the theory's validity conditions, and the implemented optimization problem is not the one analyzed in the generalization theorem, so the empirical claims are not currently backed by the paper's theoretical guarantees.","major_comments":[{"comment":"The experimental settings in Section 6 (grid range |K|=4, N=16, h=4/N=0.25, M=1000-3000) are far outside the regime required by Proposition 4.1. Substituting these values into the conditions of Eq. 29 forces epsilon to be O(1) (from the h-condition) and at least about 0.3 (from the grid-size condition), so Theorem 4.2's error bound is vacuous at those hyperparameters. Consequently, the statement in Section 6.2 that Figure 1 'substantiates Proposition 4.1' overreaches: in the tested regime the proposition's bound does not apply. The theoretical and empirical parts of the paper should be either reconciled by running in-regime experiments or explicitly separated, with claims of empirical support removed.","section":"Section 6 vs. Proposition 4.1 (Appendix C.1, Eq. 29)"},{"comment":"Theorem 5.1 analyzes the constrained ERM over the feasible set V defined in Eq. 9, which depends on target-related constants (||sigma||_inf, R, |K|) and epsilon that are unknown to the learner. The implemented objective in Eq. 33 is an unconstrained, nonconvex problem with the balancing regularizer lambda1(||a||_2^2 - ||v||_2^2)^2 and an L1 penalty on a. The paper provides no result connecting the solution of Eq. 33 to the solution of the constrained ERM of Eq. 10, so the sample-complexity guarantee of Theorem 5.1 is not established for the actual algorithm used in the experiments.","section":"Section 5 / Theorem 5.1 vs. Appendix E.2 (Eq. 33)"},{"comment":"The synthetic recovery experiments use target activations sigma1(x)=sin(pi x)1[-1,1], sigma2(x)=sin(pi x)1[0,1], and sigma3(x)=-sin(pi(x+0.5))1[-1.5,-0.5]+sin(pi(x-0.5))1[0.5,1.5]. These functions are discontinuous and are not in the class F_K of Eq. 1, which requires continuous, compactly supported, Lipschitz activations. Since Proposition 4.1 and Theorem 4.2 explicitly assume this target class, the experiments in Section 6.3 lie outside the assumptions of the theory and therefore cannot serve as a validation of those results. A smooth, compactly supported, Lipschitz activation should be used if the experiment is meant to substantiate the theory.","section":"Section 6.3 and Appendix E.1"},{"comment":"The empirical gains in Table 1 are presented as demonstrating the advantage of learnable activations, but the experimental design does not isolate that mechanism. In the tested regime (N=16, h=0.25), the theory does not apply, so the observed improvements could equally be caused by the RBF-mixture feature class, by the implicit bias of the regularizer (||a||_2^2 - ||v||_2^2)^2 in Eq. 33, or by differences in the optimization of the baseline models. A controlled ablation that fixes the feature class and regularizer while toggling only the learnability of the activation coefficients would be needed to support the causal claim; otherwise the paper should state that the gains are empirical observations without theoretical coverage.","section":"Section 6.1 / 6.2 and Table 1"},{"comment":"The checklist states that code, training details, and infrastructure are provided, but the manuscript and appendix do not include code, explicit optimizer hyperparameters (learning rate, epochs, batch size), data split details, or computing infrastructure. This makes the main empirical results hard to reproduce. The authors should either provide these materials or revise the checklist claims accordingly.","section":"Checklist and Appendix E"}],"minor_comments":[{"comment":"There are typos in the text: 'outperfom' in the abstract/contributions, 'empirial' risk, 'continous', 'techinique', and 'dicrete'. In Section 6.1, Observation (2) says 'RBF shows 2~4 times faster computational efficiency than RBF'; the second mention should clearly be 'BS' (B-spline).","section":"Section 1 and 6.1"},{"comment":"The sentence 'All models are compared within the same width M' is imprecise for RFLAF, which has N additional activation coefficients and different basis function evaluations. Please state explicitly whether the comparison matches width M only, or total parameter count, and clarify the effect on the time measurements in Table 2.","section":"Section 6.1"},{"comment":"Figure 1 plots test loss and inference/training time against grid number N. The statement that the figure 'substantiates Proposition 4.1' conflates test loss with the approximation error of the activation; the figure shows a performance trend, not the epsilon-dependent bound of Proposition 4.1. The wording should be adjusted to say the figure illustrates a trend consistent in direction, but not a quantitative validation.","section":"Figure 1 / Section 6.2"},{"comment":"The paper states that 'the number of grid points N should scale with approximately the same order of M' and then notes that N=16 suffices in practice. This distinction is useful, but the notation eTheta(1/(epsilon h)) in the sentence following Theorem 4.2 is not defined locally; please define it clearly at the first use.","section":"Section 4"},{"comment":"The phrase 'The second regularizer is the common L1 regularizer and is not compellent' appears to contain a typo; 'not compellent' should probably be 'not compulsory' or 'not essential'.","section":"Appendix E.2"}],"recommendation":"major_revision","confidential_remarks":"The theoretical results are likely sound in their own conditional framework, but the paper presents its empirical results as supporting the theory when the experimental hyperparameters fall outside the theory's validity conditions. The gap between Theorem 5.1's constrained ERM and the implemented unconstrained objective is a further correctness-relevant issue. If the authors can add experiments in the theoretically valid regime, or substantially weaken the claims linking theory and practice, and provide the missing code/training details, the paper could be acceptable. In its current form, the empirical-validation story overreaches."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look. The theoretical core is genuinely new and internally coherent: an exact kernel formula for a single RBF activation, a representation theorem, approximation guarantees via Gaussian UAT, and a Rademacher-based excess risk bound that recovers the sharp M = Theta(sqrt(n)) scaling for random features. I checked the structure of the proofs, including the Riemann-sum argument, the concentration steps, and the Rademacher bound, and they are sound. The authors also cite the prior work on optimal activations (Wang and Bento, Demir and Dogan) and are explicit that their contribution is the specific construction and its analysis. That is honest scholarship.\n\nThe soft spot is exactly where the stress-test puts it: the experiments live an order of magnitude outside the theoretical regime. The theory needs h = Theta(epsilon) and N = Theta(M polylog) with M = Theta(1/epsilon^2); the experiments use N = 16, h = 0.25, and M = 1000-3000, which makes the Proposition 4.1 conditions force epsilon to be O(1). So the bounds in Theorem 4.2 are vacuous at the experimental settings. The paper itself concedes that a humble N works, but then Figure 1 is described as substantiating Proposition 4.1, which is exactly the overreach the stress-test flags. The only in-regime experiment (Section 6.3) uses indicator-function targets, which are not Lipschitz and fail the paper's own assumptions. There are also secondary issues: the constrained set in Eq. (9) involves constants (||sigma||_inf, R, |K|) that the learner does not know, while the implemented objective is the unconstrained nonconvex problem in Eq. (33); and the checklist claims code and training details that are not in the text. None of this refutes the theorems, but it splits the paper into two parts: a valid theoretical contribution and an unexplained empirical effect.\n\nWhat the empirical gains actually come from is not isolated. The improvements are equally consistent with the RBF-mixture feature class, the implicit bias of the (||a||^2 - ||v||^2)^2 regularizer, or asymmetric baseline tuning. The paper does not claim more than 'consistently outperforms', and that claim is supported by the tables; the interpretation that learnable activation is the cause is not established.\n\nMy bottom line: the theory is a solid contribution to the random feature and kernel approximation literature and deserved a serious referee. The experimental section needs major revision: publish code, match compute, and either give in-regime experiments or explicitly state that the empirical gains are outside the theory's coverage. I would not cite it in my own work yet, but I would bring it to a reading group to discuss the kernel formula and the bounds.","headline":"Real theoretical contribution with a clean kernel formula and bounds, but the experimental section is outside the theory's regime and the paper overstates the connection.","tokens_in":47517,"tokens_out":1508,"would_cite":false,"duration_ms":17718,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68Q32"],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the fixed activation of a random feature model with a learned sum of basis functions enlarges the representable function class and reaches O(ε) error with M = Θ(1/ε²) random features.","keywords":["random features","learnable activation functions","radial basis functions","kernel approximation","excess risk","generalization bounds","two-layer neural networks"],"falsifier":"Train RFLAF with $N = 16$ and $M = 1000$ on a synthetic target whose activation $\\sigma$ is a narrow high-frequency oscillation on $[-2,2]$, and compare against the best fixed-activation random feature model; if RFLAF fails to match or beat that baseline, or if its excess risk does not decrease at the predicted $O(1/\\sqrt{M})$ rate as $M$ grows, the claimed consistent superiority and the $M = e^{\\Theta(\\sqrt{n})}$ trade-off would be contradicted.","tokens_in":46201,"feed_emoji":"🧠","tokens_out":9702,"duration_ms":78411,"temperature":0.7,"pith_summary":"Random feature models typically fix the activation function, which caps the functions they can represent. This paper introduces RFLAF, a random feature model whose activation is learned as a weighted sum of basis functions (radial basis functions, splines, or polynomials), and argues that this modest parametric change expands the represented function space without leaving the random feature framework. In the theory, a single RBF activation is shown to induce a kernel with an explicit closed form, and multiple RBFs are shown to approximate any Lipschitz, compactly supported activation. The paper proves that this model attains $O(\\epsilon)$ mean approximation error and $O(\\epsilon)$ excess risk with $M = \\Theta(1/\\epsilon^2)$ random features, a count that scales with the square root of the sample size up to logarithmic factors. On seven datasets, RFLAF with RBF or spline bases consistently beats fixed-activation random feature models, with RBFs about three times cheaper than splines.","feed_headline":"Learnable activations make random features more expressive","feed_subtitle":"A learned sum of basis functions replaces the fixed activation, cutting random-feature error on seven datasets at nearly no extra cost.","key_machinery":"The central object is the parametrized model $\\hat{f}(x) = \\frac{1}{M}\\sum_{m=1}^{M}\\sum_{i=1}^{N} a_i B_i(w_m^\\top x) v_m$, where the $B_i$ are basis functions with fixed centers and widths and the $a_i$ and $v_m$ are learned. The load-bearing identity is Theorem 3.1's closed form for the kernel induced by a single RBF activation, which yields a rotation-invariant Taylor expansion and an explicit feature map with rapidly decaying coefficients. The argument proceeds through the Gaussian universal approximation theorem to approximate any compactly supported Lipschitz activation by a weighted RBF sum, then through sub-Gaussian concentration bounds and Rademacher complexity to convert that approximation into finite-width and finite-sample guarantees.","core_discovery":"The central claim is that learnable activations, not the random first layer, are the source of expressivity in random feature models. The paper derives the first closed-form kernel for a single RBF activation, $K(x,x') = \\frac{h^2}{\\sqrt{(h^2+\\|x\\|^2)(h^2+\\|x'\\|^2)-\\langle x,x'\\rangle^2}} \\exp\\left(-\\frac{c^2}{2}\\frac{(h^2+\\|x\\|^2)+(h^2+\\|x'\\|^2)-2\\langle x,x'\\rangle}{(h^2+\\|x\\|^2)(h^2+\\|x'\\|^2)-\\langle x,x'\\rangle^2}\\right)$, and a rotation-invariant Taylor expansion whose coefficients are squares of explicitly defined polynomials. For the multi-RBF model, the approximation theorem gives $E_x|\\hat{f}(x)-f^*(x)| \\le 18(\\|\\sigma\\|_\\infty R+\\epsilon)\\sqrt{\\log(4/\\delta)}/\\sqrt{M} + \\epsilon$, and the generalization theorem bounds the excess risk by $\\rho C \\log(16/\\delta)/(h\\sqrt{n}) + \\rho C \\sqrt{\\log(8/\\delta)}/\\sqrt{M} + \\rho\\epsilon$. With the RBF width chosen as $h = e^{-\\Theta(1/\\epsilon)}$, the required number of random features is $M = e^{\\Theta(\\sqrt{n})}$, matching the known feature-count scaling for standard random feature models. The paper concludes that a learnable activation component buys the extra expressivity, and it validates this transfer by unfreezing the first layer, where the resulting two-layer networks generally outperform plain MLPs and a KAN baseline on the tested datasets.","pith_inferences":["The experiments use $N = 16$ basis functions with $M$ between 1000 and 3000, while Proposition 4.1 asks for $N = e^{\\Theta(1/\\epsilon^2)}$; the empirical gains are therefore better read as evidence for a low-complexity regime that the paper's bounds do not yet cover.","The closed-form single-RBF kernel could be used to compute a Mercer expansion numerically, which would let practitioners compare RFLAF kernels with standard kernels and predict when the learnable activation helps.","The two-layer results suggest that a single learnable activation may be the minimal ingredient of a Kolmogorov-Arnold-style network needed to beat plain MLPs on standard tasks, and it is much cheaper than KAN's many learned univariate functions."],"forward_implications":["RFLAF learns the data's effective activation by gradient descent, removing the need to hand-pick or grid-search the activation of a random feature model.","The theoretical feature count $M = e^{\\Theta(\\sqrt{n})}$ matches the known scaling results for standard random feature models, so the added expressivity does not worsen the sample-to-feature trade-off.","Because any universal approximation basis can replace RBFs, the learnable-activation mechanism transfers to spline, polynomial, and other random feature constructions.","Unfreezing the first layer turns RFLAF into a two-layer learnable activation network that the experiments show is competitive with or better than plain MLPs and a KAN baseline at a fraction of the cost.","The explicit single-RBF kernel formula invites spectral analysis of the induced kernel, which the paper identifies as the next step for understanding the model's inductive bias."],"supporting_citations":[{"why":"Defines the random feature model and the random kitchen sinks view that RFLAF extends.","marker":"(Rahimi and Recht, 2008b)"},{"why":"Introduces random features for scaling kernel machines, the foundational framework for the paper's feature-count comparisons.","marker":"(Rahimi and Recht, 2007)"},{"why":"Supplies the Gaussian universal approximation theorem used in Proposition 4.1 to justify RBF mixtures approximating compactly supported activations.","marker":"(Bacharoglou, 2010)"},{"why":"Provides the universal approximation background that lets the paper replace RBFs with splines or polynomials.","marker":"(Nestoridis and Stefanopoulos, 2007)"},{"why":"Gives the unified random Fourier feature analysis whose sharp feature count Theorem 5.1 is designed to match.","marker":"(Li et al., 2021)"},{"why":"Establishes generalization properties of random features and the feature-count scaling that RFLAF compares against.","marker":"(Rudi and Rosasco, 2017)"},{"why":"Sources the Rademacher-complexity excess-risk lemma used in the proof of Theorem 5.1.","marker":"(Shalev-Shwartz and Ben-David, 2014)"},{"why":"Supplies the sub-Gaussian and Bernstein concentration inequalities used throughout the approximation proofs.","marker":"(Vershynin, 2018)"}],"fun_headline_variants":["Learnable activations, not random weights, drive RF expressivity","Random feature models gain expressivity from learnable activations","The key to RF expressivity lies in learnable activations","Learnable activations expand random feature function space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true data-generating activation function is well approximated by a small weighted collection of basis functions; the paper's theorems actually require that collection to grow exponentially as the error tolerance shrinks, while the experiments use only sixteen RBFs.","fun_headline_variants_meta":{"raw":{"variants":["Learnable activations, not random weights, drive RF expressivity","Random feature models gain expressivity from learnable activations","The key to RF expressivity lies in learnable activations","Learnable activations expand random feature function space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3474,"prompt_tokens":1153,"completion_tokens":2321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":769,"completion_tokens_details":{"reasoning_tokens":2253}},"tokens_in":769,"tokens_out":2321,"duration_ms":15040,"temperature":1.0,"reasoning_tokens":2253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:10:20.152587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RFLAF with $N = 16$ and $M = 1000$ on a synthetic target whose activation $\\sigma$ is a narrow high-frequency oscillation on $[-2,2]$, and compare against the best fixed-activation random feature model; if RFLAF fails to match or beat that baseline, or if its excess risk does not decrease at the predicted $O(1/\\sqrt{M})$ rate as $M$ grows, the claimed consistent superiority and the $M = e^{\\Theta(\\sqrt{n})}$ trade-off would be contradicted.","supporting_citations":[],"review_version":1}