{"id":"f0546f1d-a29b-427d-a0f4-3687e4e29591","arxiv_id":"2502.00281","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper derives convergence rates for sigmoid gating mixture-of-experts with quadratic scores and uses them to argue sigmoid self-attention is more sample-efficient than softmax, but the link to attention is an unproven analogy.","lead":"This paper claims that sigmoid self-attention needs less data than softmax self-attention to reach the same approximation error, via a mixture-of-experts reformulation. The proof, however, analyzes a simplified regression model of sigmoid gating experts, not actual self-attention, so the central claim about attention is not established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dense-regime rate is for the wrong target: Theorem 3 bounds distance to an over-parameterized proxy, not to the true attention experts, so the O(epsilon^-2) claim does not compare with the softmax rates.","rationale":"The reader's verdict is REJECT, and I agree with that verdict, but my load-bearing concern is more specific and sits one step upstream of the transfer gap identified by the reader. The reader emphasized that the MoE regression rates are not proven to transfer to self-attention parameter learning. My concern is that even within the MoE analysis, the dense-regime result does not estimate the true expert parameters: it estimates distance to an over-parameterized proxy G. The paper's Section 4.2 and Table 1 present O(epsilon^-2) as the sample complexity for 'experts' to achieve approximation error epsilon, but the theorem only supports convergence to \\bar G, not to G*. Since the softmax comparison uses rates for estimating G* from [1], the central claim of superior sample efficiency for sigmoid attention is unsupported even before the attention-to-MoE transfer is addressed. This is not an external-consensus disagreement; it is an internal mismatch between the quantity being bounded and the quantity needed for the claim. The conclusion's explicit statement that sigmoid self-attention has 'higher sample complexity' than softmax in the dense regime directly contradicts the abstract, which further weakens confidence in the paper's internal consistency. I have not identified a flaw in the algebraic rewriting of attention as an MoE in Section 2; that rewriting is valid as an identity. The weakness is in interpreting rates for a misspecified MoE regression as rates for attention. My proposed test would settle whether the dense-regime proxy actually coincides with the true expert parameters, which is the key condition for the O(epsilon^-2) claim to have the meaning the paper assigns to it.","tokens_in":42882,"tokens_out":5965,"duration_ms":66003,"concrete_test":"Construct a dense-regime instance with N*=1 true expert and N=2 fitted experts, e.g., f_{G*}(x) = sigma(x^T A* x + b*^T x + c*) * lambda* ReLU(alpha*^T x + beta*) with (A*,b*) nonzero. Numerically or analytically compute \\bar G = argmin_{G in M_2(Theta) \\ M_1(Theta)} \\|f_G - f_{G*}\\|_{L2(mu)}. Then check whether \\bar eta_1 = \\bar eta_2 = eta* and whether \\|f_{\\bar G} - f_{G*}\\|_{L2(mu)} = 0. If the proxy experts differ from the true expert, Theorem 3's O(epsilon^-2) rate is a rate for estimating the wrong parameters and the dense-regime comparison in Table 1 does not follow. If the proxy does coincide with the ground truth, the next step is to verify that the O_P(sqrt(log n/n)) bound in Theorem 3 transfers to L3(\\hat G_n, G*) itself; Proposition 3 suggests this transfer fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim requires that, in the dense regime, sigmoid-attention expert estimators converge to the true expert parameters at rate O(sqrt(log n/n)), yielding O(epsilon^-2) samples to reach approximation error epsilon. The analysis does not establish this. Section 3 and Appendix B.5 (Proposition 3) show that when the gating parameters are dense (some (A*_i,b*_i) nonzero), an over-specified sum of sigmoid gates cannot converge to the true single-sigmoid gating function; the extra sigmoid can vanish only in the excluded sparse regime. Consequently, the least-squares estimator f_{\\hat G_n} converges to \\bar G := argmin_{G in M_N(Theta) \\ M_{N*}(Theta)} \\|f_G - f_{G*}\\|_{L2(mu)}, not to G*. Theorem 3 only bounds L3(\\hat G_n, \\bar G) = O_P(sqrt(log n/n)), where L3 measures distances to the proxy parameters (\\bar A_i, \\bar b_i, \\bar c_i, \\bar eta_i). The paper never shows that \\bar eta_i = eta*_i or that \\bar f_G = f_{G*}. In fact, Proposition 3 makes the opposite plausible: dense over-specified sigmoids cannot collapse to the true gating, so the proxy generally has different expert parameters. Thus the 'polynomial number of samples O(epsilon^-2)' in Section 4.2 is a rate for approximating \\bar G, while the softmax rates imported from [1] are rates for approximating the ground-truth experts G*. The comparison in Table 1 is therefore apples-to-oranges: it does not establish that sigmoid self-attention estimates the true attention parameters with fewer samples. The conclusion's sentence stating sigmoid has 'higher sample complexity' in the dense regime is also a direct contradiction of the abstract, but the target mismatch is the more fundamental logical gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to prove that sigmoid self-attention is more sample-efficient than softmax self-attention. It does so by representing each row of a self-attention matrix as a mixture of experts (MoE) with quadratic affinity scores, then analyzing the sample complexity of estimating the parameters of a sigmoid-gating MoE regression model (Eq. 5). The authors derive regression-function convergence rates under sparse and dense regimes for the gating parameters, translate these into parameter and expert convergence rates via Voronoi losses, and compare them with rates imported from a prior softmax-MoE analysis [1]. In the dense regime they claim a polynomial O(epsilon^-2) sample complexity for sigmoid-gating experts, versus O(epsilon^-4) or exponential rates for softmax, concluding that sigmoid self-attention is more sample-efficient. The paper also presents numerical experiments on the MoE regression models.","tokens_in":43165,"tokens_out":8872,"duration_ms":81104,"significance":"If the central claim were established, this would be a notable theoretical result giving a rigorous statistical justification for the empirically observed advantages of sigmoid self-attention. The technical machinery developed for sigmoid-gating MoE convergence—bracketing-entropy regression bounds, Voronoi-loss lower bounds, weak identifiability conditions, and a minimax lower bound—contains interesting components that could be of independent value. However, the paper does not provide a theorem connecting the MoE regression rates to the estimation of actual self-attention parameters, and the dense-regime analysis is carried out against a proxy target rather than the ground-truth experts. As a result, the headline claim about sigmoid versus softmax self-attention is not supported by the results as they stand.","major_comments":[{"comment":"No formal result connects the sample complexity of the MoE regression model in Eq. (5) to the sample complexity of learning self-attention parameters. In Eq. (5) the unknown quantities are fixed expert and gating parameters estimated from i.i.d. pairs (X_i, Y_i), whereas in the self-attention representation of Section 2 the 'experts' are the input token projections x_j W_V, which are random and shared across rows, and the learnable parameters are W_Q, W_K, W_V. The algebraic rewriting of one attention row as an MoE does not map one estimation problem onto the other, and no theorem in the paper states that the rates for the regression model transfer to attention parameter estimation. Therefore the abstract claim that 'sigmoid self-attention has lower sample complexity than softmax self-attention' does not follow from the MoE analysis.","section":"Section 2 and Section 4"},{"comment":"The dense-regime rate is for an over-parameterized proxy, not for the true experts. Corollary 1 states that inf_{G in M_N(Theta)\\M_{N*}(Theta)} ||f_{\\hat G_n} - f_G||_{L2(mu)} = O_P(sqrt(log n/n)), and Theorem 3 bounds L3(\\hat G_n, \\bar G) where \\bar G is the minimizer of ||f_G - f_{G*}|| over that same excluded set. The text then concludes that 'it takes those experts only a polynomial number of samples O(epsilon^-2) to achieve an approximation error of epsilon'. This conclusion is not established because the error is measured against \\bar G, not against the ground-truth expert parameters G*. In fact, Proposition 3 in Appendix B.5 shows that a dense over-specified sum of sigmoid gates cannot converge to the true single-sigmoid gating function, so there is no reason to expect \\bar eta_i = eta*_i or \\bar f_G = f_{G*}. The softmax rates imported from [1] are for estimation of the ground-truth experts, so the comparison in Table 1 compares distances to two different targets and does not support the claimed sample-efficiency advantage.","section":"Section 4.2, Theorem 3, Corollary 1, and Proposition 3"},{"comment":"The exponential sample-complexity claim for polynomial experts is not supported by Theorem 2. That theorem gives inf_{\\hat G_n} sup_{G} E[L_{2,r}(\\hat G_n, G)] \\gtrsim n^{-1/2} for every r \\ge 1. This is a minimax lower bound on the r-th-power Voronoi loss, and it implies at most that the corresponding parameter discrepancies cannot be estimated faster than a polynomial rate of order n^{-1/(2r)} for fixed r. The subsequent text claims that the parameter convergence rates are 'slower than any polynomial rates O_P(n^{-1/2r}) for any r \\ge 1, potentially as slow as O_P(1/log^tau(n))'. This is internally inconsistent because n^{-1/2r} is itself a polynomial rate, and the 'potentially as slow as n^{-1/log^tau(n)}' assertion is not a consequence of any proved statement. Consequently, the 'exponential number of data O(exp(epsilon^{-1/tau}))' for sigmoid-gating polynomial experts in the sparse regime, as listed in Table 1, is not proven, and the comparison with the softmax exponential rate from [1] is not established.","section":"Section 4.1.2, Theorem 2"}],"minor_comments":[{"comment":"The phrase 'ynthetic data' should be 'synthetic data'.","section":"Section 5, Setup"},{"comment":"The sentence 'Our results show that sigmoid self-attention has a higher sample complexity than the softmax version in the more common dense regime' contradicts the abstract, the title, and Table 1; it should read 'lower sample complexity'.","section":"Section 6, Conclusion"},{"comment":"The sentence 'Le et al. Furthermore, Akbarian et al. [1]...' contains an incomplete citation 'Le et al.' with no reference or title; please complete it or remove it.","section":"Appendix A, Related Works"},{"comment":"The statement 'If the expert function ... then the lower bound ... holds true ... then L1( bGn, G∗) = OP(...)' has a double-'then' construction that should be rephrased for clarity.","section":"Section 4.1.1, Theorem 1"},{"comment":"The covering numbers |\\Delta_\\tau| and |\\Omega_\\tau| are deterministic quantities but are written with OP(...); they should use O(...) notation.","section":"Appendix B.1, Step 4"},{"comment":"The Voronoi loss L3(\\hat G_n, G) is plotted for the sigmoid model fitted to data generated from a softmax-gating MoE, but the target measure G for the sigmoid fit is never defined; please specify how G is chosen in that setting.","section":"Section 5 and Figure 1"}],"recommendation":"reject","confidential_remarks":"The central claim of the paper does not follow from the results: there is no theorem linking the MoE regression analysis to self-attention parameter estimation, and the dense-regime comparison is against a proxy rather than the true experts. The polynomial-expert exponential-rate claim in the sparse regime is also unsupported by the stated lower bound. These are load-bearing issues that cannot be fixed by local editing of the presentation. If the authors were to reframe the contribution as a convergence analysis of sigmoid-gating MoE regression, the paper might merit reconsideration, but in its current form the attention-based significance claim is not justified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the sigmoid-gating MoE convergence analysis is a genuine contribution: the sparse/dense split, Proposition 3 on why over-specified sigmoids cannot collapse to a single sigmoid, and the minimax lower bound for polynomial experts are all new and technically substantive. Second, the paper's headline claim — that sigmoid self-attention is more sample-efficient than softmax — is not supported by the analysis. The dense-regime rate O(ε^-2) is for convergence to an over-parameterized proxy \\bar G, not to the true experts G*, so it cannot be compared with the softmax rates from [1], which are for G*.\n\nWhat the paper does well: the MoE proof strategy is carefully executed, and the authors are explicit about the identifiability obstruction in the dense regime. The observation that over-specified sigmoid gates can only vanish in the sparse regime is subtle and worth publishing on its own.\n\nWhere it falls apart: the jump from MoE regression to self-attention is an algebraic rewriting, not a reduction. In the regression model, expert parameters are fixed unknowns estimated from i.i.d. samples; in attention, the experts are input tokens and the learnable parameters are W_Q, W_K, W_V. No theorem bridges these. Section 4.2 concludes the O(ε^-2) rate for 'experts' but those experts are the proxy parameters, not the true value-matrix rows. Table 1 is therefore apples-to-oranges. The experiments use data generated from a softmax-gating model, which is misspecified for the sigmoid model, and the plots only show convergence of the Voronoi loss to some limit, not to ground truth. On top of that, the conclusion literally says sigmoid has 'higher sample complexity' in the dense regime, contradicting the abstract and Section 4.2. That looks like a typo, but it is a bad one to leave in.\n\nShould you read it? If you work on MoE convergence, the sigmoid analysis may be useful. If you care about the attention claim, it is not there. I would not cite it for self-attention. I would recommend reject, but with a clear message: the MoE content can be salvaged if the authors reframe the paper as a study of sigmoid-gating MoE and drop the unsupported attention interpretation. It deserves a serious referee because the technical work is non-trivial, but the current abstract overstates what is proven.","headline":"The sigmoid-gating MoE analysis is a real technical contribution, but the headline claim about self-attention sample complexity does not follow from it.","tokens_in":43804,"tokens_out":4029,"would_cite":false,"duration_ms":35856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F12","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Sigmoid self-attention needs fewer samples than softmax attention.","keywords":["sigmoid self-attention","softmax self-attention","sample complexity","mixture of experts","quadratic gating","convergence rates","Voronoi loss","Transformer theory"],"falsifier":"Fit a sigmoid-gated mixture of experts with polynomial experts to synthetic data from the same model in the dense over-specified regime and plot the Voronoi loss against $n$; the paper predicts decay of order $n^{-1/2}$, so observing decay of order $n^{-1/4}$ or slower, the softmax baseline rate, would refute the central claim.","tokens_in":42574,"feed_emoji":"📉","tokens_out":4901,"duration_ms":45285,"temperature":0.7,"pith_summary":"The paper sets out to prove that replacing the row-wise softmax in Transformer self-attention with an element-wise sigmoid makes the mechanism more sample-efficient: fewer training examples are needed to reach a given approximation accuracy. It does this by rewriting one row of the attention matrix as a mixture-of-experts model whose gate is quadratic, then analyzing how fast least-squares estimation recovers the expert parameters in that model. In the dense regime, where gating parameters are nonzero, the paper derives a polynomial sample complexity of $O(\\epsilon^{-2})$ for sigmoid attention, compared with $O(\\epsilon^{-4})$ or exponential rates for softmax attention from the comparison baseline. In the sparse regime, the two mechanisms have the same sample complexity. The paper concludes that sigmoid self-attention is at least as data-efficient as softmax, and strictly better in the regime the authors argue is common in practice.","feed_headline":"Sigmoid attention needs fewer samples than softmax attention","feed_subtitle":"An MoE analysis finds a quadratic sample advantage for sigmoid attention in the dense regime.","key_machinery":"The load-bearing object is the representation of one row of the attention matrix as a mixture of experts with quadratic affinity scores: $[\\mathrm{SigmoidAttn}(X)]_{i,:} = \\sum_j \\sigma(x_i B x_j^\\top)\\, x_j W_V$, with $B = W_Q W_K^\\top / \\sqrt{d_k}$. This rewrites learning attention weights as estimating the parameters of a sigmoid-gating mixture-of-experts regression. The proofs are carried by Voronoi loss functions, which measure parameter discrepancies cell by cell, together with strong and weak identifiability conditions expressed as linear independence of partial derivatives; those conditions determine whether Taylor-expanded parameter differences can be separated. The sigmoid's element-wise, unnormalized structure removes the softmax normalization constraint and, in the dense regime, lets first-order Taylor terms dominate, yielding the fast $O_P((\\log n/n)^{1/2})$ expert rate.","core_discovery":"The central claim is a concrete sample-complexity separation between sigmoid and softmax attention. Using the equivalence that each row of the attention output is a mixture of experts, with gate $\\sigma(x_i B x_j^\\top)$ for sigmoid and value rows $x_j W_V$ as experts, the authors analyze the sigmoid-gating mixture-of-experts regression model. They show that in the dense regime, weakly identifiable experts such as ReLU, GELU, and polynomial experts are estimated at rate $O_P((\\log n/n)^{1/2})$, so only $O(\\epsilon^{-2})$ samples are needed for approximation error $\\epsilon$. The comparable softmax analysis from the baseline they compare against yields $O(\\epsilon^{-4})$ for strongly identifiable experts and exponential $O(\\exp(\\epsilon^{-1/\\tau}))$ for polynomial experts. The paper therefore claims sigmoid attention is more sample-efficient than softmax attention in the dense regime and equally efficient in the sparse regime.","pith_inferences":["If the transfer assumption holds, the paper implies sigmoid attention is preferable in small-sample settings such as few-shot learning or low-resource modeling, where sample efficiency matters more than raw capacity.","The mixture-of-experts representation suggests a testable architectural prediction: attention heads whose value projections are well approximated by low-degree polynomials should benefit most from switching to sigmoid gating.","A rigorous extension to multi-head attention via hierarchical mixtures of experts, which the authors flag as future work, would likely preserve the dense-regime separation if the hierarchy inherits weak identifiability.","Because input-dependent gating weights are typical in trained models, the paper's argument predicts that practical gains from sigmoid attention should be widespread rather than confined to specially constructed cases."],"forward_implications":["In the dense regime, sigmoid self-attention reaches the same approximation error as softmax with quadratically fewer samples, $O(\\epsilon^{-2})$ versus $O(\\epsilon^{-4})$.","Polynomial experts, which are exponentially hard under softmax attention, become polynomially easy under sigmoid attention in the dense regime.","Under the sparse regime, sigmoid attention is not worse than softmax: both require $O(\\epsilon^{-4})$ samples for strongly identifiable experts.","The same separation holds under partially quadratic affinity scores, where linear experts also move from exponential to polynomial sample complexity.","The result provides a statistical justification for the empirical success of sigmoid attention: removing token competition also removes a statistical bottleneck in the dense regime."],"supporting_citations":[{"why":"Supplies the softmax quadratic-gating mixture-of-experts convergence rates, $O(\\epsilon^{-4})$ and exponential, that form the comparison baseline.","marker":"[1]"},{"why":"Provides the sigmoid self-attention formulation, the FlashAttention extension, and the empirical evidence the paper aims to explain.","marker":"[43]"},{"why":"Defines the Transformer and self-attention whose rows are represented as mixtures of experts.","marker":"[47]"},{"why":"Introduces the mixture-of-experts architecture with adaptive gating that the attention representation builds on.","marker":"[23]"},{"why":"Provides the empirical-process and bracketing-entropy machinery used to derive the least-squares convergence rates.","marker":"[46]"}],"fun_headline_variants":["Sigmoid attention: quadratic sample advantage over softmax","Sigmoid self-attention proven more sample-efficient than softmax","Sigmoid attention: lower sample complexity in dense regime","MoE analysis shows sigmoid attention needs fewer samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes the sample complexity of estimating a sigmoid-gated mixture-of-experts regression transfers directly to the sample complexity of learning self-attention parameters in a Transformer, even though in attention the experts are random input tokens shared across rows and the fitted parameters are $W_Q$, $W_K$, and $W_V$.","fun_headline_variants_meta":{"raw":{"variants":["Sigmoid attention: quadratic sample advantage over softmax","Sigmoid self-attention proven more sample-efficient than softmax","Sigmoid attention: lower sample complexity in dense regime","MoE analysis shows sigmoid attention needs fewer samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001043,"raw_usage":{"total_tokens":4387,"prompt_tokens":945,"completion_tokens":3442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":3383}},"tokens_in":561,"tokens_out":3442,"duration_ms":23414,"temperature":1.0,"reasoning_tokens":3383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:34:00.592734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit a sigmoid-gated mixture of experts with polynomial experts to synthetic data from the same model in the dense over-specified regime and plot the Voronoi loss against $n$; the paper predicts decay of order $n^{-1/2}$, so observing decay of order $n^{-1/4}$ or slower, the softmax baseline rate, would refute the central claim.","supporting_citations":[{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer and self-attention whose rows are represented as mixtures of experts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the mixture-of-experts architecture with adaptive gating that the attention representation builds on."},{"cited_title":"van de Geer.Empirical Processes in M-estimation","cited_arxiv_id":null,"evidence_quote":"Provides the empirical-process and bracketing-entropy machinery used to derive the least-squares convergence rates."}],"review_version":1}