{"id":"5e1a818b-7f1c-4ba5-ba78-d678816ac302","arxiv_id":"2605.25704","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"PowLU replaces SwiGLU with a rational-power activation to reduce outlier amplification and numerical instability during large-scale LLM pre-training while matching performance.","lead":"The paper introduces PowLU, an activation function based on a rational power that aims to deliver adaptive nonlinearity while avoiding the numerical instability caused by SwiGLU's quadratic-like growth on large inputs. A smart generalist might read it because stable activation functions are a practical bottleneck when scaling LLMs to hundreds of billions of parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"PowLU's scalability claim hinges on the rational power function providing stable nonlinearity without per-layer retuning or new failure modes at scales >124B.","rationale":"Reader's weakest_assumption matches the load-bearing empirical gap exactly. Full text availability does not remove the need to verify generalization of the power-function properties beyond the tested regime; the concern therefore stands as stated.","tokens_in":1733,"tokens_out":283,"duration_ms":23514,"concrete_test":"Re-train the 124B Ling model with PowLU but allow independent per-layer exponent tuning; compare final loss, outlier frequency, and training stability metrics against the untuned PowLU run. If tuned version shows >5% better loss or materially fewer spikes, the no-retuning assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (competitive results + improved scalability vs SwiGLU/SwiGLU-Clip) rests on scaling-law consistency and Ling-architecture runs at 7.9B/124B. For these to support generalization, the rational power must deliver sufficient representation capacity while avoiding new instabilities or requiring layer-specific adjustments. The abstract states theoretical justification exists, but if the proven properties do not bound behavior in the spike regions or extreme regimes encountered at larger scales, the stability and scalability conclusions do not follow from the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes the Power Linear Unit (PowLU) activation function based on a rational power function to provide adaptive nonlinearity and address numerical instability in SwiGLU (which approximates x² for large positive inputs) during large-scale, low-precision LLM pre-training. It supplies theoretical justification for key properties of PowLU, reports scaling-law experiments showing performance consistency across model sizes, and presents results on the Ling architecture at 7.9B and 124B total parameters claiming competitive performance versus SwiGLU and SwiGLU-Clip along with improved scalability.","tokens_in":1863,"tokens_out":481,"duration_ms":39088,"significance":"If the empirical claims hold with adequate controls, PowLU could meaningfully improve training stability for LLMs at and beyond current scales by mitigating outlier amplification without sacrificing representation capacity. The scaling-law consistency and large-model runs would be valuable if they include proper ablations; the rational-power approach is a plausible direction if its theoretical properties are shown to be robust rather than fitted.","major_comments":[{"comment":"Abstract: the central empirical claim that PowLU 'achieves competitive results against SwiGLU and SwiGLU-Clip' and 'effectively improves the scalability' on 7.9B/124B Ling models is asserted without any quantitative metrics, error bars, baseline details, data-exclusion rules, or per-layer retuning information, rendering the claim unevaluable.","section":"Abstract"},{"comment":"Theoretical justification and experimental sections: the paper states that theoretical properties justify stability, but does not demonstrate that these properties bound behavior in spike regions or extreme regimes at scales >124B; without such a bound the scalability conclusion does not follow from the reported Ling-architecture runs.","section":"Theoretical justification"}],"minor_comments":[{"comment":"The definition and exact form of the rational power function should be given as an explicit equation early in the method section rather than described only in prose.","section":null}],"recommendation":"major_revision","confidential_remarks":"The complete absence of numbers or tables in the abstract is atypical for an empirical ML paper and raises the possibility that the experimental section may also be light on controls; this should be checked against the full manuscript."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major comment below and indicate where revisions will be made.","responses":[{"response":"We agree that the abstract would benefit from greater specificity. In the revised manuscript we will expand the abstract to report the key quantitative metrics (e.g., relative perplexity or downstream scores) from the 7.9B and 124B Ling runs, identify the exact SwiGLU and SwiGLU-Clip baselines, and note the training configuration details that were used.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central empirical claim that PowLU 'achieves competitive results against SwiGLU and SwiGLU-Clip' and 'effectively improves the scalability' on 7.9B/124B Ling models is asserted without any quantitative metrics, error bars, baseline details, data-exclusion rules, or per-layer retuning information, rendering the claim unevaluable."},{"response":"We acknowledge that the current theoretical analysis supplies properties that reduce outlier amplification but does not derive explicit bounds that guarantee behavior for scales strictly larger than 124B. The scaling-law consistency and the 124B results provide supporting empirical evidence; we will add a limitations paragraph that clarifies the scope of the theoretical claims and the degree to which the observed trends can be extrapolated.","revision_made":"partial","referee_comment":"[Theoretical justification] Theoretical justification and experimental sections: the paper states that theoretical properties justify stability, but does not demonstrate that these properties bound behavior in spike regions or extreme regimes at scales >124B; without such a bound the scalability conclusion does not follow from the reported Ling-architecture runs."}],"tokens_in":1392,"tokens_out":405,"duration_ms":35473,"standing_objections":["Deriving explicit mathematical bounds on PowLU behavior in spike regions for model scales exceeding 124B parameters."]},"desk_editor":{"model":"grok-4.3","letter":"The paper introduces PowLU, an activation that uses a rational power instead of SwiGLU's near-quadratic behavior on large positive inputs. The goal is to keep useful nonlinearity while cutting the numerical instability that shows up in low-precision training of bigger models. That is a concrete engineering problem worth addressing.\n\nWhat the work does well is name the source of the instability clearly and propose a functional form that could adapt without needing per-layer fixes. The mention of theoretical justification for key properties is also positive if the derivations are actually carried through in the full text.\n\nThe soft spots sit in the empirical claims. The abstract states that PowLU matches or beats SwiGLU and SwiGLU-Clip on 7.9B and 124B Ling-architecture runs and improves scalability, with scaling laws holding across sizes. Yet it supplies no metrics, no error bars, no baseline details, and no description of how outliers or spike regions were handled. Without those, it is impossible to judge whether the rational power actually delivers the promised stability or simply shifts the failure modes. The stress-test note is on target here: the scalability conclusion only follows if the new function maintains representation capacity at larger scales without introducing fresh instabilities or requiring retuning.\n\nThis paper is for researchers who train large models and care about numerical behavior during pre-training. A reader already working on activation variants or low-precision scaling might extract a useful idea, but only if the full experiments include reproducible numbers and clear comparisons. The thinking is coherent on its own terms, so it counts as serious engagement even if the results turn out modest.\n\nI would send it to peer review. The scale of the reported runs is large enough that referees can check the missing details, and fixing a known source of blowup is the kind of incremental improvement that deserves scrutiny rather than desk rejection.","headline":"PowLU is a straightforward attempt to tame SwiGLU's quadratic blowup with a rational power, but the abstract gives no numbers to back the scalability claims.","tokens_in":2373,"tokens_out":453,"would_cite":false,"duration_ms":28798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"PowLU uses a rational power function to deliver stable nonlinearity in LLM activations, matching SwiGLU performance at scale.","keywords":["PowLU","activation function","LLM training","numerical stability","SwiGLU","scaling laws","pre-training"],"falsifier":"A training run at 100B parameters where PowLU produces higher validation loss or more divergence than SwiGLU-Clip under identical conditions.","tokens_in":2640,"feed_emoji":"","tokens_out":558,"duration_ms":24695,"temperature":0.7,"pith_summary":"The paper proposes PowLU as an activation function that uses a rational power function for adaptive nonlinearity in large language models. This addresses the numerical instability in SwiGLU, which approximates a quadratic for large inputs and causes outliers in low-precision training. Scaling experiments show consistent performance across different model sizes. Tests on the Ling architecture with up to 124 billion parameters demonstrate that PowLU achieves results competitive with SwiGLU and its clipped variant. If true, this would support more reliable training of very large models.","feed_headline":"PowLU stabilizes large LLM training with power-based activation","feed_subtitle":"The function matches SwiGLU results at 124B scale while avoiding quadratic-induced outliers in low precision.","key_machinery":"The rational power function, which supplies adaptive nonlinearity without quadratic amplification of large inputs.","core_discovery":"PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Theoretical justification is provided for several key properties. Scaling law experiments confirm that the performance is consistent across model sizes, and experimental results with the Ling architecture at 7.9B and 124B total parameters show that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip.","pith_inferences":["PowLU might allow training without per-layer hyperparameter tuning for activations.","Similar power-based functions could be tested in other neural network domains prone to outlier issues.","The approach could extend to multimodal models where activation stability is critical."],"forward_implications":["Performance stays consistent as models grow larger.","Training remains stable even with low precision and large inputs.","Scalability of LLM pre-training improves without additional clipping mechanisms.","Representation capacity matches or approaches that of SwiGLU in large architectures."],"fun_headline_variants":["PowLU power activation stabilizes large LLM pre-training","PowLU enables stable training for 124B parameter LLMs","New PowLU function avoids quadratic outliers in LLM training","PowLU matches SwiGLU at scale with better stability","Adaptive power function PowLU for consistent LLM scaling"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"A rational power function can supply enough nonlinearity and representation capacity for LLMs without creating new instability or requiring retuning.","fun_headline_variants_meta":{"raw":{"variants":["PowLU power activation stabilizes large LLM pre-training","PowLU enables stable training for 124B parameter LLMs","New PowLU function avoids quadratic outliers in LLM training","PowLU matches SwiGLU at scale with better stability","Adaptive power function PowLU for consistent LLM scaling"]},"model":"grok-4.3","cost_usd":0.007304,"raw_usage":{"total_tokens":3373,"prompt_tokens":688,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":73037000,"prompt_tokens_details":{"text_tokens":688,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2609,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":688,"tokens_out":76,"duration_ms":23222,"temperature":1.0,"reasoning_tokens":2609,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T21:45:40.320160+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A training run at 100B parameters where PowLU produces higher validation loss or more divergence than SwiGLU-Clip under identical conditions.","supporting_citations":[],"review_version":1}