{"id":"b9c90729-02ee-4388-932b-334da99c03d2","arxiv_id":"2606.17945","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Reducing parameter initialization scale in LLMs improves pretraining and reasoning by inducing a low-to-high complexity developmental trajectory in weights.","lead":"The paper reports that using smaller scales when initializing parameters in large language models improves pretraining results, especially on reasoning tasks, by creating a compression-then-expansion path in how parameters develop. A smart generalist might read it because this is presented as a nearly free change that could strengthen model capabilities across scales.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"The assumption that two specific empirical settings are the primary restraints on small-init advantages may not generalize without broader controls.","rationale":"The reader's weakest assumption directly matches the load-bearing point for the scaling and mechanistic claims. Because the full text is referenced but the concern centers on validation of those settings rather than new inconsistencies, the UNVERDICTED verdict stands.","tokens_in":1697,"tokens_out":276,"duration_ms":23687,"concrete_test":"From the sections defining the two settings, rerun the main scaling curves while relaxing only one setting at a time (holding the other fixed) across at least two model sizes; if the performance delta on reasoning tasks drops below 50% of the joint-relaxation gain, the primacy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that relaxing the two identified settings restores favorable scaling for small initialization. This rests on those settings being the dominant factors rather than correlated variables (e.g., effective learning-rate scale, gradient clipping thresholds, or data-ordering effects that interact with init variance). If the paper's ablations do not isolate them from optimizer hyperparameters or architecture-specific normalizations, the observed trajectory (condensation then expansion) and task-specific gains could be artifacts of the chosen regime rather than a general developmental mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that reducing the initialization scale for large language models consistently improves pretraining performance, with the largest gains on reasoning-demanding tasks. It identifies two widely used empirical settings that restrain this advantage and shows that relaxing them restores favorable scaling behavior. Mechanistically, small initialization induces a developmental trajectory in which parameters first condense into low-complexity structures and later expand into richer representations. Token-level analyses indicate that gains concentrate on non-trivial, context-constrained predictions. The work proposes a simple γ-initialization rule to expose initialization range as an explicit training knob.","tokens_in":1788,"tokens_out":406,"duration_ms":48290,"significance":"If the empirical results and mechanistic claims hold after addressing controls, the work would be significant for establishing initialization scale as a low-cost, high-impact determinant of LLM capacity and reasoning. The condensation-then-expansion trajectory supplies a concrete mechanism supporting the compression-is-intelligence hypothesis, and the task-specific token gains offer falsifiable predictions that could guide future training studies. The γ-initialization proposal is a practical contribution that could be adopted with minimal overhead if shown to be robust.","major_comments":[{"comment":"The central claim that relaxing the two identified empirical settings restores favorable scaling for small initialization is load-bearing. The ablations must isolate these settings from correlated variables such as effective learning-rate scale, gradient clipping thresholds, or data-ordering effects that interact with initialization variance; otherwise the observed trajectory and task-specific gains could be regime-specific artifacts rather than a general developmental mechanism (§ on identification of empirical settings and associated ablations).","section":"Identification of empirical settings and ablations"}],"minor_comments":[{"comment":"The abstract would be strengthened by including at least one key quantitative result (e.g., perplexity delta or reasoning benchmark improvement with error bars) to allow readers to gauge effect size immediately.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and for highlighting the potential significance of initialization scale as a determinant of LLM training dynamics. We address the single major comment below and will strengthen the ablations in revision to better isolate the claimed effects.","responses":[{"response":"We agree that isolating the two empirical settings from the listed confounders is essential for establishing the developmental mechanism as general rather than regime-specific. In the submitted manuscript we already held the learning-rate schedule fixed and verified that per-parameter gradient norms differ systematically with initialization scale; we also reported results across multiple random seeds. However, these controls are insufficient to fully rule out interactions. In the revised manuscript we will add targeted ablations that (i) explicitly rescale the base learning rate so that initial gradient norms are matched across initialization scales, (ii) sweep gradient-clipping thresholds while keeping all other hyperparameters constant, and (iii) fix the data order (identical seed and shuffling) while varying only the initialization scale. We will present these results in an expanded version of the section on empirical settings, together with statistical tests confirming that the condensation-then-expansion trajectory and the concentration of gains on non-trivial tokens remain statistically significant under the stricter controls. These additions directly address the load-bearing concern without altering the core claims.","revision_made":"yes","referee_comment":"[Identification of empirical settings and ablations] The central claim that relaxing the two identified empirical settings restores favorable scaling for small initialization is load-bearing. The ablations must isolate these settings from correlated variables such as effective learning-rate scale, gradient clipping thresholds, or data-ordering effects that interact with initialization variance; otherwise the observed trajectory and task-specific gains could be regime-specific artifacts rather than a general developmental mechanism (§ on identification of empirical settings and associated ablations)."}],"tokens_in":1308,"tokens_out":363,"duration_ms":24163,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that smaller initialization scales appear to boost LLM pretraining, with bigger lifts on reasoning tasks, by pushing parameters through a low-complexity condensation phase before they expand into richer structures. The gamma rule is presented as a cheap way to expose initialization as a tunable knob.\n\nWhat comes across as new is the explicit developmental trajectory and the token-level breakdown showing gains mostly on context-constrained predictions instead of blanket improvements. The paper does a reasonable job documenting consistent gains across scales once the two common settings are relaxed, and it frames initialization as more than a minor hyperparameter.\n\nThe soft spot is the claim that those two settings are the primary restraints. The experiments would need to demonstrate that the observed trajectory and task gains survive when learning-rate scaling, gradient clipping, or data ordering are varied independently; otherwise the advantage could be an artifact of the chosen regime rather than a general mechanism. The abstract gives no numbers or error bars, so effect sizes and robustness are hard to judge without the full tables.\n\nThis is aimed at people running pretraining runs or studying how capabilities emerge. A practitioner looking for low-cost tweaks or a researcher tracking scaling behavior would get something concrete to test.\n\nIt is worth sending to referees. The core empirical observation is straightforward to check and the intervention is cheap enough that even partial confirmation would matter.","headline":"Small initialization improves pretraining and reasoning via a condensation-then-expansion trajectory, but the paper needs tighter controls to show the two settings are the main drivers rather than correlated factors.","tokens_in":2294,"tokens_out":350,"would_cite":false,"duration_ms":23190,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Reducing the initialization scale improves pretraining of large language models with largest gains on reasoning tasks.","keywords":["parameter initialization","large language models","pretraining","reasoning tasks","developmental trajectory","model capacity","initialization scale"],"falsifier":"A controlled comparison in which small initialization is applied after the two settings are relaxed yet produces no gain or a loss on reasoning benchmarks would falsify the claim that the trajectory and scaling benefit are recovered.","tokens_in":2599,"feed_emoji":"","tokens_out":642,"duration_ms":34394,"temperature":0.7,"pith_summary":"The paper establishes that the scale at which parameters are randomly initialized functions as a gene-like control on how large language models develop during training. Smaller scales produce better final models, with the strongest improvements appearing on tasks that require reasoning rather than rote pattern matching. This advantage appears because parameters follow a two-phase path: they first collapse into simple low-complexity forms and only later expand into richer structures. Two common training choices have hidden this benefit in past work; relaxing them lets the improvement grow with model size. The authors therefore propose treating initialization scale as an explicit, almost cost-free control knob via a simple gamma rule.","feed_headline":"Reducing initialization scale improves LLM reasoning","feed_subtitle":"Smaller random starting values drive a condense-then-expand trajectory that strengthens performance on complex predictions.","key_machinery":"The condensation-then-expansion trajectory of parameters under small initialization, which supplies a concrete mechanism for the claim that compression precedes richer intelligence.","core_discovery":"Parameter initialization scale determines model capacity by setting a distinct developmental trajectory in which weights first condense into low-complexity structures and subsequently expand into richer representations; this path yields consistent pretraining gains that are largest on reasoning tasks and concentrated on non-trivial, context-constrained token predictions, while a critical scale balances reasoning performance against training stability.","pith_inferences":["The same condense-then-expand dynamic may appear in other neural architectures and training regimes beyond transformers.","Initialization scale could be scheduled or adapted during training rather than fixed at the start to further exploit the trajectory.","The finding reframes capacity as partly determined by starting conditions instead of emerging only from scale, data, and architecture.","Token-level diagnostics of the sort used here could be applied to diagnose whether other interventions also operate through non-uniform prediction improvements."],"forward_implications":["Small initialization produces consistent pretraining gains across model scales.","The largest improvements appear on reasoning-demanding tasks rather than uniform token prediction.","Gains concentrate on non-trivial predictions that depend on context rather than all tokens equally.","A critical initialization scale exists that trades off reasoning performance against training stability.","A gamma-initialization rule lets practitioners adopt small initialization by default at negligible cost."],"fun_headline_variants":["Small init sets LLM condense-expand trajectory","Initialization scale shapes LLM developmental path","Init scale determines LLM capacity via trajectory","Small initialization drives condense-then-expand path"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Two widely used empirical settings are the main factors that have hidden the benefit of small initialization, and relaxing those settings is what restores favorable scaling.","fun_headline_variants_meta":{"raw":{"variants":["Small init sets LLM condense-expand trajectory","Initialization scale shapes LLM developmental path","Init scale determines LLM capacity via trajectory","Small initialization drives condense-then-expand path"]},"model":"grok-4.3","cost_usd":0.005493,"raw_usage":{"total_tokens":2531,"prompt_tokens":614,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":54928000,"prompt_tokens_details":{"text_tokens":614,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1869,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":614,"tokens_out":48,"duration_ms":17504,"temperature":1.0,"reasoning_tokens":1869,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T00:55:57.005739+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled comparison in which small initialization is applied after the two settings are relaxed yet produces no gain or a loss on reasoning benchmarks would falsify the claim that the trajectory and scaling benefit are recovered.","supporting_citations":[],"review_version":1}