{"id":"3b2d568e-5292-4268-a17d-e531fd54ee0b","arxiv_id":"2501.10661","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Large model weights look Gaussian, random Gaussian updates with learned scales work well for fine-tuning, and the paper builds simple adaptation and merging methods on that view.","lead":"The authors report that the weights of large foundation models are approximately Gaussian, and that a randomly initialized Gaussian update scaled by one learned number per layer can replace learned fine-tuning updates with surprisingly good results on reasoning and language benchmarks. A generalist should read this because it suggests that adapting a huge model to a new task might be reducible to tuning a handful of scalars, not billions of weights.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The i.i.d. Gaussian-noise claim rests on an unverified asymptotic independence assumption (Sec. 3.1, Eq. 1); no correlation test is provided.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern that I see: the paper's leap from Gaussian marginals to i.i.d. Gaussian noise depends on an unverified independence premise. I considered alternatives—the 3σ filtering in Sec. 2.2, the unfalsifiable W* conjectures in Appendix B, and the variance arithmetic in Appendix A.2—but these weaken the presentation rather than undermine the core derivation. The independence assumption is the keystone: without it, Eq. (1) does not follow, the claim that weights share i.i.d. Gaussian-noise properties lacks support, and the interpretation of transformation weights as scaled Gaussian noise collapses. The proposed row-correlation test would directly settle the issue. Because the reader already flags this and asks for conditions, the verdict remains CONDITIONAL; no change is needed.","tokens_in":83867,"tokens_out":9163,"duration_ms":95297,"concrete_test":"Take LLaMA-7B layer 0 self_attn.q_proj (4096×4096). Compute the Pearson correlation between every pair of distinct rows, and average the absolute values. Under the i.i.d. null, this average should be near zero with standard error about 1/√4096 ≈ 0.016; if the observed average exceeds 0.05, the independence assumption in Sec. 3.1 is rejected, and Eq. (1) cannot be used to infer i.i.d. Gaussian weights. Repeat on 5 matrices (e.g., q/k/v/up/down, and a ConvNeXt pwconv) to confirm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's transition from 'weight marginals are approximately Gaussian' to 'weights share the i.i.d. properties of Gaussian noise' (Sec. 3.1) depends entirely on the borrowed premise that individual weight elements are asymptotically independent. Eq. (1) writes the joint density as a product of Gaussian marginals and concludes each element is i.i.d. N(μ, σ²). But the observed Gaussianity is only a property of the pooled empirical histogram of all entries, which cannot distinguish i.i.d. entries from strongly correlated entries with non-Gaussian marginals. For structured matrices such as attention Q/K/V projections or MLP layers, correlations are expected (e.g., row/column structure from training dynamics). If weight elements are correlated, Eq. (1) is not justified, and the subsequent derivation that transformation weights are Gaussian noise with a scaled standard deviation (Eq. 8) lacks a foundation. The paper provides no direct evidence of independence—no correlation statistics, no autocorrelation analysis, no test of exchangeability. The cited works (Thamm et al., Yang, Lee et al.) address initialization or random matrix limits, not trained weights. Thus the i.i.d. conclusion is load-bearing for the central Gaussian-noise interpretation, and it is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that pretrained and fine-tuned weight matrices of large foundation models are predominantly Gaussian, that weight elements are i.i.d. like Gaussian noise, that transformation weights can be replaced by Gaussian noise with a learned scalar standard deviation, and that the underlying optimal weights are zero-mean, symmetric, and sparse with truncated Gaussian entries plus outliers. It then proposes two applications: LoRA+Ours, which augments LoRA by also learning a scalar multiplier of the pretrained weight matrix, and a model-merging method that amplifies outliers outside a per-layer threshold. The empirical sections report consistent improvements in commonsense reasoning (up to +7.8 points over LoRA) and in multimodal model merging (+2.0 points over averaging).","tokens_in":84112,"tokens_out":5231,"duration_ms":56059,"significance":"The empirical portion of the paper is valuable: the observation that a randomly initialized Gaussian Delta-W with a single learned scalar per layer recovers a large fraction of full fine-tuning performance on GLUE (Table 2) is a striking and falsifiable result, and the proposed LoRA+Ours and merging methods give clear gains. The paper also documents weight histograms across nine models in NLP, CV, and MM, which could serve as a useful reference. However, the theoretical framing goes well beyond what the evidence supports. The i.i.d. claim rests on an unverified independence assumption, the Gaussianity validation uses post hoc filtering, and the M-estimator derivation of W* contains unjustified structural assumptions and a mathematical error in the variance algebra. The paper would be significantly improved by reframing the theoretical claims as conjectures and adding direct tests of independence and distributional fit.","major_comments":[{"comment":"The derivation of the i.i.d. property assumes independence rather than establishing it. Eq. (1) writes the joint density as a product of Gaussians after noting that the observed pooled histogram is approximately Gaussian, but a Gaussian marginal for the pooled entries does not imply that the entries are independent or identically distributed. The cited references address initialization or random-matrix limits, not trained transformer weights. Since the subsequent claim that transformation weights can be 'directly derived from Gaussian noise' depends on this i.i.d. interpretation, the paper needs either a direct test of independence (e.g., correlation or autocorrelation statistics on the weight matrices) or a reformulation that does not claim i.i.d. properties.","section":"Sec. 3.1, Eq. (1)"},{"comment":"The variance algebra in Eq. (8) is incorrect as written. If W and W' are independent samples from distributions with variances sigma_1^2/n and sigma_2^2/m, then the variance of their difference Delta-W is the sum of the two variances, not the difference. If they are correlated because both share W*, then a covariance term is required. The expression 'sigma_2^2/m - sigma_1^2/n' can even be negative, which is impossible for a variance. This error undermines the derivation that Delta-W is zero-mean Gaussian with variance equal to the difference of the two noise variances, which is then used in Sec. 3.3 to explain why adding Gaussian noise helps.","section":"Appendix A.2, Eq. (8)"},{"comment":"The Gaussianity claim is weakened by post hoc filtering. The main validation computes skewness and kurtosis only after a 3-sigma filter, and Appendix A.1 shows that for layers with high kurtosis (e.g., 3.7-4.5 in ConvNeXt-xlarge stage 2), kurtosis drops to about 3 only after additionally removing all values with magnitude below 10^-3. The ratio reported in Table 1 (99%+) corresponds to the 3-sigma filter only, so the reader has no sense of how much data is removed by the second filter. This does not support the blanket statement that 'nearly all pre-trained weights conform to a Gaussian distribution.' The authors should report the full filtering procedure and the retained fraction for both filters, or use a distributional test on unfiltered weights.","section":"Sec. 2.2 and Appendix A.1, Table 5"},{"comment":"The M-estimator argument treats the entire pretrained weight vector as a finite-dimensional parameter estimated from n i.i.d. samples and concludes that W - W* is asymptotically normal with isotropic covariance sigma^2 I. This is an assumption, not a derivation. Neural network weights are high-dimensional and trained with SGD on non-i.i.d. data, and no justification is given for the spherical covariance or for the applicability of classical M-estimator asymptotics to this setting. The subsequent interpretation of W as W* plus Gaussian noise, and the properties of W* derived in Sec. 4, inherit this unverified assumption. The authors should either provide a rigorous argument or explicitly label this as a heuristic model.","section":"Sec. 3.3, Eqs. (4)-(5)"},{"comment":"The simulation in Appendix B.2 is a curve-fitting exercise, not an independent validation of the W* hypothesis. The standard deviations of the added Gaussian noise (0.001 to 0.3), the truncation bounds (0.001 and 0.5), the sparsity (2 million nonzero of 10 million), and the outlier fraction (0.5% in [0.6, 1]) are all selected after observing the target histograms. The paper even acknowledges in Appendix B that W* is unverifiable. The conclusion that the observed distributions 'can be derived' from the hypothesized W* therefore has no falsifiable content. The authors should soften the claim to a demonstration of plausibility, or propose a concrete, testable implication of the W* model that goes beyond reproducing the already-observed shapes.","section":"Appendix B.2"}],"minor_comments":[{"comment":"The title in the PDF body reads 'Gaussian Distribution Never F ades' with an erroneous space; this should be corrected to 'Fades'.","section":"Title"},{"comment":"There is a typo in the sentence 'kurtosis describes the \"tailedness\" of a it'; it should read 'of a distribution'.","section":"Sec. 2.2"},{"comment":"The narrative moves from 'carefully crafted Delta-W matrices' that use auxiliary information from fully fine-tuned weights to 'randomly initialized with standard Gaussian noise' without clearly distinguishing the two experimental protocols. The transition should be made explicit so the reader understands that the Gaussian-noise result is a separate ablation.","section":"Sec. 3.2"},{"comment":"The phrase 'This comes to the conflict' should be 'This leads to a conflict' or 'This creates a conflict'.","section":"Sec. 4"},{"comment":"The table formatting is broken: the line '36.9Math-LLaV A' concatenates the average score for LLaVA-1.6-13B with the model name for Math-LLaVA. The table should be reformatted for clarity.","section":"Table 4"},{"comment":"The final sentence 'For experimental details, please refer to Appendix. B.2.' refers to the section in which it appears; the intended cross-reference is presumably Appendix C.6.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The empirical contributions are real and the random-Gaussian Delta-W experiment is a notable result, but the paper currently overstates its theoretical conclusions. The i.i.d. claim and the W* story need to be reframed as empirically motivated conjectures, Eq. (8) must be corrected, and the Gaussianity validation needs to address the post hoc filtering. I would recommend a major revision rather than rejection because the core experiments and applications are defensible and the requested changes are within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One genuinely interesting empirical result sits inside a lot of overclaimed theory. Training DeBERTaV3-base on GLUE with 72 parameters, where each layer's delta-W is just random Gaussian noise scaled by a learned scalar, reaches 78.6 average versus 88.2 for full fine-tuning. That is a real finding and it is not in the prior work they cite. The sigma-depth pattern and the sigma-performance conjecture are also new, though the paper is honest enough to label that last one a hypothesis.\n\nWhat the paper does well: the random-substitution experiment stands on its own, because its value does not depend on whether real delta-W matrices are actually i.i.d. I also credit the row-5 control, where initializing the whole model as Gaussian and training only the standard deviation collapses to 45.1. That is the authors testing their own hypothesis against a disconfirming case and then revising the story to include outliers. The appendix explicitly concedes that W* is unverifiable, which is the right posture.\n\nNow the soft spots, in rough order of seriousness. First, the i.i.d. claim in Sec. 3.1 is not supported. A pooled histogram cannot distinguish i.i.d. entries from strongly correlated ones, and the paper provides no correlation statistics, no autocorrelation analysis, and no exchangeability test. The cited works are about initialization or random-matrix limits, not trained weights. The stress-test note is right about this, and the paper should either test it directly or drop the claim. Second, the validation has a selection-bias problem: kurtosis values only settle near 3 after the extra |w| < 1e-3 filter in Appendix A.1, and comparing Table 5 columns shows the filter was chosen to fit the hypothesis. The 3-sigma filter alone is defensible; the second one is cherry-picking. Third, Eq. 8 is circular: it assumes both W and W' are Gaussian, which is the very observation being explained, and the M-estimator step assumes isotropic covariance without justification. These are interpretive glosses, not derivations. Fourth, the B.2 simulation fits the noise levels and truncation bounds to reproduce observed shapes; it demonstrates possibility, not necessity. Minor: no error bars on Tables 3 and 4, and no code release.\n\nThe paper is for people working on PEFT and model merging, who may get usable ideas from the method, and for anyone who needs a reminder that visual Gaussianity is not statistical Gaussianity. It deserves peer review with major revision: scale back the abstract, test or remove the i.i.d. claim, and separate the solid empirical result from the speculative framework.","headline":"Genuinely interesting random-Gaussian-delta-W result inside an overclaimed theoretical wrapper; the i.i.d. claim is unsupported but testable, and the paper deserves a serious referee.","tokens_in":84675,"tokens_out":2989,"would_cite":false,"duration_ms":33903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62E20"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the weights of large foundation models are, statistically, near-i.i.d.","keywords":["gaussian weight distribution","large foundation models","weight analysis","parameter-efficient fine-tuning","model merging","transformation weights","optimal weights","i.i.d. noise"],"falsifier":"Compute the pairwise correlations among weight elements within a single attention or MLP projection matrix (say Q, K, or V of any LLaMA layer): if the average absolute off-diagonal correlation is clearly nonzero across many element pairs, Eq. (1) cannot hold and the i.i.d. claim is refuted. A second settlement test: fine-tune a model, take its real $\\Delta W$, and replace it layer by layer with fresh Gaussian noise rescaled to the same standard deviations; the paper's claim predicts nearly equal performance, so a large accuracy gap would refute the 'transformation weights are Gaussian noise' thesis.","tokens_in":83596,"feed_emoji":"🎲","tokens_out":8823,"duration_ms":77224,"temperature":0.7,"pith_summary":"This paper sets out to show that the weights of large foundation models are, statistically, nearly indistinguishable from Gaussian noise: across nine models spanning language, vision, and multimodal tasks, weight histograms are zero-mean and bell-shaped regardless of initialization, and the difference between fine-tuned and pre-trained weights (the paper's 'transformation weights') behaves like i.i.d. Gaussian noise. On that basis the authors claim that fine-tuning a pretrained layer is mostly a matter of rescaling its weight noise — one learned scalar per layer — plus updating a few outliers, and they build two methods on the idea: a LoRA augmentation that adds a Gaussian term with learned standard deviation, and a model-merging rule that amplifies outliers before averaging. If the picture is right, it would simplify adaptation, editing, and compression into operations on a distribution's standard deviation rather than on millions of parameters. The paper's own experiments report gains of up to 8 points on commonsense reasoning and 2 points on multimodal merging, and it states explicitly that its strongest theoretical premise — asymptotic independence of individual weight elements — is borrowed from earlier work rather than verified here.","feed_headline":"Fine-tuning may just rescale Gaussian weight noise","feed_subtitle":"Paper: pretrained weights are near-i.i.d. Gaussian, so each layer needs only one learned sigma.","key_machinery":"The load-bearing object is the scalar-rescaled Gaussian update $W' = W + s\\Delta W$, in which $\\Delta W$ is frozen standard Gaussian noise and the single trainable parameter $s$ per layer stands in for the standard deviation of the correction. The argument that this works rests on two identities: Eq. (1), which factors the joint distribution of independent weight elements into a product of marginals and forces a single shared Gaussian, and Eq. (5), $W = W^* + \\mathcal{N}(0, \\sigma^2/n\\, I) + o(1/\\sqrt{n})$, which treats the pretrained weights as a statistical estimate of the optimal weights plus Gaussian noise whose variance shrinks with training-set size. The paper validates Gaussianity with skewness and kurtosis after a 3σ filter, and it converts the picture into algorithms: LoRA+Ours updates $W' = (s+1)W + AB$, and the merging rule keeps elements beyond $t\\sigma$ unshrunk while averaging the rest.","core_discovery":"On the paper's own terms, the central discovery is that the observed Gaussian shape of pretrained weights is not incidental but structural: the paper reports that in LLaMA, Vicuna, Qwen2.5, SAM, ConvNeXt, SigLip, Idefics3, LLaVA-NeXT, and Ovis1.6, on average over 99 percent of weight elements survive a 3σ filter, skewness is 0, and kurtosis sits near 3. Combining that marginal Gaussian with an assumed asymptotic independence of weight elements, the paper concludes the weights are i.i.d. Gaussian noise, so the difference between pretrained and fine-tuned weights should itself be pure noise with an adjustable standard deviation. It tests this by freezing a randomly initialized Gaussian matrix per layer and training only one scalar $s$ in $W' = W + s\\Delta W$ on GLUE with DeBERTaV3-base, reaching a 78.62 average with 72 parameters, about 7 points above a head-only baseline. The paper further claims that this noise's standard deviation grows with layer depth, that optimal weights $W^*$ are zero-mean, symmetric, and sparse with truncated-Gaussian values and outliers, and that adaptation and merging succeed by respectively increasing σ while updating outliers and by amplifying outliers while averaging the Gaussian part; the LoRA variant and merging rule built from these claims beat their baselines by up to 7.8 and 2.0 points.","pith_inferences":["If per-layer σ truly encodes adaptation, then hyperparameters like rank and learning rate should be predictable from σ alone; a cheap test is to train any PEFT method, record its layer-wise σ, and check whether it predicts that method's accuracy across a grid of configurations.","The sharp, inverted-T, and linear weight patterns in shallow layers are read by the paper as small-noise windows onto the sparse $W^*$; a testable extension is to use per-layer kurtosis as a proxy for training quality or layer difficulty.","The i.i.d. assumption is the natural place to probe: measuring the correlation spectrum of attention projections would either harden the foundation or, if correlations are strong, reduce the paper's contribution to an empirical heuristic that still works.","The sigma-gap versus performance-gap hypothesis could be stress-tested across many methods and seeds on one task, plotting $|\\sigma_1 - \\sigma_2|$ against accuracy difference; the paper itself calls this only preliminarily validated."],"forward_implications":["Fine-tuning a pretrained layer can be approximated per layer by a scalar: augmenting LoRA with a Gaussian term and learning only the scale lifts commonsense-reasoning accuracy by up to 7.8 points over LoRA alone on LLaMA2-7B.","Because the standard deviation of the transformation weights grows with layer depth, adaptation effort should concentrate on deeper layers, and per-layer σ becomes a natural budget for where to spend parameters.","Model merging should preserve outliers beyond $t\\sigma$ and average the Gaussian remainder; this rule beats plain weight averaging by 2.0 points on a multimodal benchmark.","The learned scalar $s$ in the Gaussian-noise experiment is stable across random seeds (average difference 0.009), implying the standard deviation, not the specific noise draw, carries the adaptation signal.","The paper's sigma-based evaluation conjecture, if confirmed, would let practitioners rank fine-tuning methods by comparing the standard deviations of their weight changes without running a test set."],"supporting_citations":[{"why":"Supplies the asymptotic-independence premise that lets the observed Gaussian marginals be read as i.i.d. weight elements.","marker":"(Thamm et al., 2022; Anonymous, 2024; Yang, 2020; Lee et al., 2017)"},{"why":"Asymptotic normality of M-estimators, the source of Eq. (5) linking training-set size to weight noise.","marker":"(Van der Vaart, 2000)"},{"why":"LoRA, both the baseline that LoRA+Ours augments and a source of transformation weights.","marker":"(Hu et al., 2021)"},{"why":"Task arithmetic, the weight-difference framework the merging rule builds on.","marker":"(Ilharco et al., 2022)"},{"why":"Evidence that outlier weights alone carry over 70 percent of model performance, motivating outlier-preserving merging.","marker":"(Yadav et al., 2024)"},{"why":"DeBERTaV3-base, the model on which the scalar-only Gaussian-noise experiment is tested.","marker":"(He et al., 2021)"},{"why":"GLUE benchmark used to validate that frozen Gaussian $\\Delta W$ with a learned scalar approaches full fine-tuning.","marker":"(Wang et al., 2018)"},{"why":"Kaiming initialization, the default scheme the paper checks against Xavier under the Gaussian distribution claim.","marker":"(He et al., 2015)"}],"fun_headline_variants":["Pretrained weights are mostly Gaussian noise—fine-tuning adds more","One scalar per layer: fine-tuning is just adding Gaussian noise","99% of weights are background noise; fine-tuning tunes its scale","Gaussian never fades: fine-tuning merely resizes the noise","Zero-mean, symmetric, sparse: the optimal weight profile"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's Gaussian-noise interpretation rests on the borrowed premise that individual weight elements inside a trained matrix are asymptotically independent; if elements are actually correlated, the step from 'the histogram is Gaussian' to 'the weights are i.i.d. Gaussian noise' fails and the derivation behind the scalar-only method collapses.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained weights are mostly Gaussian noise—fine-tuning adds more","One scalar per layer: fine-tuning is just adding Gaussian noise","99% of weights are background noise; fine-tuning tunes its scale","Gaussian never fades: fine-tuning merely resizes the noise","Zero-mean, symmetric, sparse: the optimal weight profile"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1594,"prompt_tokens":1043,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":659,"tokens_out":551,"duration_ms":5506,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:06:06.858284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the pairwise correlations among weight elements within a single attention or MLP projection matrix (say Q, K, or V of any LLaMA layer): if the average absolute off-diagonal correlation is clearly nonzero across many element pairs, Eq. (1) cannot hold and the i.i.d. claim is refuted. A second settlement test: fine-tune a model, take its real $\\Delta W$, and replace it layer by layer with fresh Gaussian noise rescaled to the same standard deviations; the paper's claim predicts nearly equal performance, so a large accuracy gap would refute the 'transformation weights are Gaussian noise' thesis.","supporting_citations":[],"review_version":1}