{"id":"42cb4691-ff0d-490f-9a1b-5e16743d4ff3","arxiv_id":"2607.17624","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Learned replacement non-linearities show transformers are rarely optimal for algorithmic tasks, with benefits that are task-specific, while language/code gains are smaller and more transferable.","lead":"The paper shows that swapping a transformer's standard math functions (GeLU, softmax) for specially learned curves can make it learn much faster and generalize better on toy reasoning tasks, but the gains barely transfer to other tasks. On language and code modeling, the gains are small but transfer across datasets, suggesting standard transformers are not the unique best design for every job.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-task 'sacrifice universality' may reflect fixed-budget and baseline-hyperparameter artifacts rather than true inductive-bias incompatibility.","rationale":"The most load-bearing part of the claim is not merely that some activation tweak helps — same-task stage II results and Appendix D's fine-detail ablations already argue against a pure smoothness/regularizer artifact. What makes the result interesting is the trade-off: better task-specific architectures sacrifice universality. That trade-off is read directly from Figure 5. However, Figure 5's protocol (fixed step count and baseline-tuned hyperparameters, Appendix B) is not controlled for the transferred architectures. The diagonal may dominate simply because splines were selected on the diagonal task and because the LR was tuned for the baseline, not because off-diagonal inductive biases are intrinsically incompatible. The reader's weakest_assumption pointed at the stage II freeze generally; I am sharpening it to a particular, testable confound in the cross-task matrix. This does not invalidate the paper, but it keeps the universality claim conditional until the control is run. If the control shows the off-diagonal entries can be recovered by tuning, the conclusion weakens materially; if not, the conclusion is robust. The reader's CONDITIONAL verdict already captures this level of uncertainty, so no verdict change is needed.","tokens_in":24465,"tokens_out":10545,"duration_ms":117076,"concrete_test":"Pick 2-3 negative off-diagonal cells in Figure 5 (e.g., MEMORIZE-optimized spline on HAYSTACK and HAYSTACK-optimized spline on ADD). Re-run stage II for these transferred architectures with (a) a learning-rate sweep over at least 0.1x-10x the baseline LR and (b) training for up to 5x the fixed step budget used in Figure 5. If any off-diagonal cell becomes non-negative or competitive with the diagonal after tuning, the specialization conclusion is an artifact of budget/hyperparameters; if the diagonal still dominates under all tuned settings, the incompatibility is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The universality half of the central claim rests on Figure 5, where each architecture optimized for one algorithmic task is retrained on a target task for a fixed number of steps and with hyperparameters tuned only for the baseline transformer (Appendix B, Table 4). The paper reads the strong diagonal / weak off-diagonal pattern as evidence that algorithmic-task inductive biases are highly task-specific, hence that better architectures 'sacrifice universality' (Section 6). But a fixed-step, fixed-hyperparameter comparison conflates incompatible inductive biases with slower convergence or LR sensitivity of the transferred spline: the optimized splines in Figure 6 have very different shapes and scales, and the baseline-tuned LR and step budget may be strongly suboptimal for them. The sentence in Section 3.1 that re-tuning gave no further improvements was established only for same-task stage II training, not for cross-task transfers. Without evidence that off-diagonal entries remain negative after per-architecture LR re-tuning and longer training, the diagonal-dominates pattern could be an optimization artifact rather than a fundamental incompatibility. The 'rarely a local optimum' half of the claim survives, but the headline 'sacrifice universality' conclusion would be substantially weakened if this confound is real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage method to optimize the non-linear components of a transformer (GeLU MLP activations and softmax attention) by replacing them with learned linear splines. In stage I, the spline parameters are optimized on a held-out split while model weights train on the main split, using multiple parallel models to reduce co-adaptation. In stage II, the learned splines are frozen and models are retrained from scratch on the same or other datasets. The method is applied to eight algorithmic toy tasks and seven code/language modeling datasets. The main findings are: (i) on algorithmic tasks, optimized architectures dramatically improve learning speed, generalization, and seed stability, but the benefits are highly task-specific (strong diagonal in the cross-task transfer matrix); (ii) on code/language modeling, the improvements are small but consistent, and the optimized architectures transfer much better across datasets and domains; (iii) these results are used to argue that standard transformers are rarely a local optimum in the space of architectures, but that better task-specific architectures sacrifice universality.","tokens_in":24749,"tokens_out":3233,"duration_ms":32108,"significance":"If the findings hold, the paper makes a meaningful empirical contribution to the study of inductive biases in transformers. Its strengths include the use of multiple seeds (6 for algorithmic, 3 for language), a two-stage procedure that avoids a naive co-adaptation of weights and non-linearities, a tuned baseline, and a comparison against several existing alternative activations and attention variants. The paper also ships code and includes a careful discussion of limitations. The central empirical claims are reproducible in principle and the results on algorithmic tasks are striking. However, the interpretation of the cross-task transfer results as evidence of inductive-bias incompatibility, and the claim of consistent language-model improvements, rest on assumptions about hyperparameter transfer and statistical significance that are not fully demonstrated.","major_comments":[{"comment":"The claim that optimized architectures 'sacrifice universality' rests on the diagonal-dominant pattern in Figure 5, but the comparison is made after a fixed number of steps and with hyperparameters tuned only for the baseline transformer (Appendix B, Table 4). The optimized splines have very different shapes and scales (Fig. 6), so the baseline-tuned learning rate and step budget may be strongly suboptimal for them. The statement in §3.1 that 'we get no further improvements by re-tuning them' was established only for same-task stage II training, not for cross-task transfers. Without per-architecture LR re-tuning or a sensitivity analysis for the off-diagonal entries, the negative off-diagonal values could reflect slower convergence or LR mismatch rather than a fundamental incompatibility of inductive biases. This directly weakens the headline conclusion in the abstract and Section 6.","section":"§3.2, Fig. 5, Appendix B"},{"comment":"The language-modeling improvements are small (e.g. 0.1–0.4 percentage points in accuracy, differences of a few hundredths in perplexity), but the paper does not report confidence intervals, error bars, or seed-wise values for most of these results. Figure 9 presents a single scalar per cell, and Table 6 in Appendix E reports averages over 3 seeds with no variance. Without an indication of variability, the claims that the improvements are 'consistent' and that the architectures 'transfer much better' across language datasets cannot be evaluated statistically. This is load-bearing for the conclusion that standard transformers are 'not a local optimum' for language modeling, because the differences are close to the noise level one might expect from 3 seeds.","section":"§4, Fig. 9, Table 6, Table 7"}],"minor_comments":[{"comment":"Typo: 'GeLUs,;softmax' should be 'GeLUs and softmax'.","section":"Abstract"},{"comment":"The optimized MLP non-linearities are presented without axis labels or units. Adding x/y axis labels (e.g. input activation vs. output) would make the figure more informative.","section":"Figure 6"},{"comment":"The caption says 'relative difference in perplexity with the baseline in %, lower is better', but the sign convention is not self-evident. It would be helpful to state explicitly that negative values indicate improvement over the baseline.","section":"Figure 9 caption"},{"comment":"The sentence 'we get no further improvements by re-tuning them' is ambiguous. It should explicitly say that this was tested for same-task stage II training only, not for cross-task transfers, especially given the concern in the major comment.","section":"§3.1"},{"comment":"The number of parallel models M differs between algorithmic tasks (M=8) and language modeling (M=3). A brief justification of this choice would help readers assess the comparability of the two experimental settings.","section":"Appendix B, Table 4"},{"comment":"Typo: 'TorchsSscript' should be 'TorchScript'.","section":"Appendix F, Listing 2"},{"comment":"The abbreviations 'Tr. perplexity' and 'Val. acc.' are not defined in the caption. They appear to be training perplexity and validation token accuracy; please spell them out.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study with a clear two-stage methodology and a useful set of baselines. The main issue is that the strongest interpretive claims—'sacrifice universality' and 'consistent improvements in language modeling'—are not yet supported by the evidence as presented. The cross-task transfer results need a hyperparameter-sensitivity analysis or per-architecture re-tuning, and the language results need confidence intervals or seed-wise reporting. These are fixable with additional experiments or by softening the claims. I would support publication after these points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two genuinely useful things. First, it gives a clean two-stage procedure for optimizing transformer non-linearities (GeLUs, softmax) as splines on held-out data, with multi-model training to avoid co-adaptation. Second, it shows on eight algorithmic tasks that the resulting architectures dramatically improve speed, generalization, and seed stability, and on seven language/code datasets that they give small but consistent gains. The algorithmic results are the strongest part: multiple seeds, tuned baselines, and a real effect on out-of-distribution length generalization. The language results are honest about the effect being small, and the comparison against standard alternatives (ReLU, GLU, etc.) is useful.\n\nThe main soft spot is the universality claim. The conclusion that better architectures 'sacrifice universality' rests on Figure 5, where architectures optimized for one algorithmic task are retrained on another with a fixed number of steps and hyperparameters tuned only for the baseline. The stress-test note is right: the paper only shows that re-tuning gave no further improvements for same-task stage II training, not for cross-task transfers. The optimized splines have very different shapes and scales, so the off-diagonal entries could reflect slower convergence or learning-rate sensitivity rather than true inductive-bias incompatibility. That said, the diagonal gaps are large and the pattern is consistent, so some specialization is real. But the headline claim should be tempered.\n\nTwo smaller issues: the abstract says 'rarely a local optimum in the space of architectures,' but only non-linearities were searched—the paper's own Limitations section admits this, so it's a framing overstatement rather than a hidden flaw. And a few language-modeling tables lack error bars; the gains are consistent but the reader can't assess statistical significance.\n\nOverall, this is a serious empirical paper. The method is reproducible (code is public), the central affirmative finding—transformers are not locally optimal in the explored non-linearity subspace—holds up. The compatibility analysis is a useful first pass, but needs robustness checks before the 'sacrifice universality' conclusion is taken at face value. I'd send it to a careful reviewer, with the request to test whether off-diagonal transfer improves with per-architecture hyperparameter tuning and longer budgets.","headline":"Solid empirical study showing transformers can be improved by learned non-linearities, but the cross-task 'sacrifice universality' claim is not fully supported because the transfer comparisons use baseline-tuned budgets.","tokens_in":25241,"tokens_out":1456,"would_cite":true,"duration_ms":16605,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that standard transformers are rarely a local optimum in architecture space, and that replacing GeLU and softmax with splines optimized on held-out data yields large gains on algorithmic tasks and smaller but consistent ga","keywords":["inductive bias","transformers","learned activation functions","spline parametrization","task compatibility","length generalization","language modeling","algorithmic reasoning"],"falsifier":"Train stage II models with the learned spline replaced by a random spline of matched smoothness and the same training recipe; if the matched random spline reproduces the convergence and length-generalization gains, then the specific learned shapes are not the source of the improvement and the paper's compatibility story fails.","tokens_in":24346,"feed_emoji":"⚙️","tokens_out":4736,"duration_ms":43937,"temperature":0.7,"pith_summary":"The paper asks whether the transformer design that dominates modern AI is actually the best design for any given task. It answers no: by replacing the two main nonlinearities—the GeLU in MLP blocks and the softmax in attention—with piecewise-linear splines learned on held-out data, the authors find architectures that learn much faster, generalize better, and vary less across random seeds on algorithmic tasks such as addition, copying, and modular arithmetic. The same procedure gives small but consistent improvements on language and code modeling, and those improvements transfer across datasets and domains. The central message is that standard transformers are rarely a local optimum in architecture space: simple alternatives can do much better, but the best designs are often task-specific, especially for algorithmic reasoning.","feed_headline":"Learned splines beat transformer defaults - but only per task","feed_subtitle":"On algorithmic tasks, tiny tweaks bring large speed and generalization gains; language gains are small but transfer.","key_machinery":"The mechanism that carries the argument is a family of learnable nonlinearities: the elementwise MLP activation (normally GeLU) and the attention similarity kernel (normally softmax) are each replaced by a 1D linear spline whose knot values are learned in stage I on a held-out split of the target dataset. Stage I uses two losses—one for weights on training data, one for the spline parameters on held-out data—and trains several models in parallel sharing the splines, so the splines capture a bias that survives re-training rather than a co-adaptation to particular weights. In stage II the splines are frozen and models are trained from scratch, which is what turns an optimization procedure into","core_discovery":"The paper's central claim, stated on its own terms, is that the inductive biases of a standard transformer are rarely optimal for any single task. Replacing the GeLU nonlinearity and the softmax attention kernel with linear splines that are optimized on a held-out split, then frozen and reused to train fresh models, produces architectures that dramatically accelerate convergence, improve in- and out-of-distribution generalization, and stabilize seed-to-seed behavior on algorithmic toy tasks. These gains are highly task-specific: an architecture optimized for one algorithmic skill often underperforms even the baseline on another. For natural language and code, the same method yields smaller b","pith_inferences":["The paper leaves open whether the same spline optimization can satisfy several algorithmic tasks at once; a multi-task version of stage I would be a direct test of whether the observed task-specificity is fundamental or a consequence of optimizing one task at a time.","If the gains on algorithmic tasks truly come from inductive bias rather than optimization smoothing, then distilling the learned splines into cheap closed-form activations could make specialized models practical, since the paper finds high-degree polynomial approximations nearly as good.","The asymmetry—MLP nonlinearities matter while softmax is hard to beat—suggests future architecture search should concentrate on the MLP block, where the flexibility budget is largest."],"forward_implications":["If the central claim is right, the search for better LLMs should not stop at scaling: simple architectural tweaks to MLP nonlinearities can yield faster learning and better data efficiency at fixed parameter counts.","Because algorithmic skills demand highly task-specific designs, attempts to give LLMs robust arithmetic and reasoning may need to introduce explicit specialized modules or multi-task architecture optimization rather than one universal transformer.","The observed transfer of optimized non-linearities across English and code datasets implies that a single improved nonlinearity could serve multilingual and multimodal language modeling without retuning.","Better length generalization on COPY from spline-optimized linear-bias positional encodings suggests that part of the length-generalization problem is architectural, not just positional-encoding based."],"fun_headline_variants":["Customizing transformer nonlinearities yields task-specific boosts","Standard transformers rarely optimal for any single task","Task-specific spline designs beat defaults on algorithmic tasks","Big gains on toy tasks, small gains on language: transformer tweaks","Why transformers aren't the best fit for any given task"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the two-stage procedure isolates a task's inductive bias rather than an artifact of optimization: if the splines merely smoothed the loss landscape or acted as a regularizer, the dramatic gains on algorithmic tasks would not prove a better inductive bias and the cross-task incompatibility conclusion would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Customizing transformer nonlinearities yields task-specific boosts","Standard transformers rarely optimal for any single task","Task-specific spline designs beat defaults on algorithmic tasks","Big gains on toy tasks, small gains on language: transformer tweaks","Why transformers aren't the best fit for any given task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2783,"prompt_tokens":759,"completion_tokens":2024,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1946}},"tokens_in":503,"tokens_out":2024,"duration_ms":13689,"temperature":1.0,"reasoning_tokens":1946,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:27:16.241611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train stage II models with the learned spline replaced by a random spline of matched smoothness and the same training recipe; if the matched random spline reproduces the convergence and length-generalization gains, then the specific learned shapes are not the source of the improvement and the paper's compatibility story fails.","supporting_citations":[],"review_version":1}