Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that the weights of large foundation models are, statistically, near-i.i.d.

desk verdict Genuinely interesting random-Gaussian-delta-W result inside an overclaimed theoretical wrapper; the i.i.d. claim is unsupported but testable, and the paper deserves a serious referee. read the letter →

arxiv 2501.10661 v1 pith:MJZLP5RZ submitted 2025-01-18 cs.LG cs.AIcs.CL

Chongjie Si , Jingjing Jiang , Wei Shen This is my paper · ORCID
classification cs.LGcs.AIcs.CL MSC 68T0762E20
keywords gaussianweightdistributionlargefoundationmodelsanalysisparameter-efficientfine-tuningmodelmergingtransformationweightsoptimali.i.d.noise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the weights of large foundation models are, statistically, nearly indistinguishable from Gaussian noise: across nine models spanning language, vision, and multimodal tasks, weight histograms are zero-mean and bell-shaped regardless of initialization, and the difference between fine-tuned and pre-trained weights (the paper's 'transformation weights') behaves like i.i.d. Gaussian noise. On that basis the authors claim that fine-tuning a pretrained layer is mostly a matter of rescaling its weight noise — one learned scalar per layer — plus updating a few outliers, and they build two methods on the idea: a LoRA augmentation that adds a Gaussian term with learned standard deviation, and a model-merging rule that amplifies outliers before averaging. If the picture is right, it would simplify adaptation, editing, and compression into operations on a distribution's standard deviation rather than on millions of parameters. The paper's own experiments report gains of up to 8 points on commonsense reasoning and 2 points on multimodal merging, and it states explicitly that its strongest theoretical premise — asymptotic independence of individual weight elements — is borrowed from earlier work rather than verified here.

What carries the argument

The load-bearing object is the scalar-rescaled Gaussian update $W' = W + s\Delta W$, in which $\Delta W$ is frozen standard Gaussian noise and the single trainable parameter $s$ per layer stands in for the standard deviation of the correction. The argument that this works rests on two identities: Eq. (1), which factors the joint distribution of independent weight elements into a product of marginals and forces a single shared Gaussian, and Eq. (5), $W = W^* + \mathcal{N}(0, \sigma^2/n\, I) + o(1/\sqrt{n})$, which treats the pretrained weights as a statistical estimate of the optimal weights plus Gaussian noise whose variance shrinks with training-set size. The paper validates Gaussianity with skewness and kurtosis after a 3σ filter, and it converts the picture into algorithms: LoRA+Ours updates $W' = (s+1)W + AB$, and the merging rule keeps elements beyond $t\sigma$ unshrunk while averaging the rest.

What would settle it

Compute the pairwise correlations among weight elements within a single attention or MLP projection matrix (say Q, K, or V of any LLaMA layer): if the average absolute off-diagonal correlation is clearly nonzero across many element pairs, Eq. (1) cannot hold and the i.i.d. claim is refuted. A second settlement test: fine-tune a model, take its real $\Delta W$, and replace it layer by layer with fresh Gaussian noise rescaled to the same standard deviations; the paper's claim predicts nearly equal performance, so a large accuracy gap would refute the 'transformation weights are Gaussian noise' thesis.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the observed Gaussian shape of pretrained weights is not incidental but structural: the paper reports that in LLaMA, Vicuna, Qwen2.5, SAM, ConvNeXt, SigLip, Idefics3, LLaVA-NeXT, and Ovis1.6, on average over 99 percent of weight elements survive a 3σ filter, skewness is 0, and kurtosis sits near 3. Combining that marginal Gaussian with an assumed asymptotic independence of weight elements, the paper concludes the weights are i.i.d. Gaussian noise, so the difference between pretrained and fine-tuned weights should itself be pure noise with an adjustable standard deviation. It tests this by freezing a randomly initialized Gaussian matrix per layer and training only one scalar $s$ in $W' = W + s\Delta W$ on GLUE with DeBERTaV3-base, reaching a 78.62 average with 72 parameters, about 7 points above a head-only baseline. The paper further claims that this noise's standard deviation grows with layer depth, that optimal weights $W^*$ are zero-mean, symmetric, and sparse with truncated-Gaussian values and outliers, and that adaptation and merging succeed by respectively increasing σ while updating outliers and by amplifying outliers while averaging the Gaussian part; the LoRA variant and merging rule built from these claims beat their baselines by up to 7.8 and 2.0 points.

Load-bearing premise

The paper's Gaussian-noise interpretation rests on the borrowed premise that individual weight elements inside a trained matrix are asymptotically independent; if elements are actually correlated, the step from 'the histogram is Gaussian' to 'the weights are i.i.d. Gaussian noise' fails and the derivation behind the scalar-only method collapses.

Editorial extensions

If this is right

  • Fine-tuning a pretrained layer can be approximated per layer by a scalar: augmenting LoRA with a Gaussian term and learning only the scale lifts commonsense-reasoning accuracy by up to 7.8 points over LoRA alone on LLaMA2-7B.
  • Because the standard deviation of the transformation weights grows with layer depth, adaptation effort should concentrate on deeper layers, and per-layer σ becomes a natural budget for where to spend parameters.
  • Model merging should preserve outliers beyond $t\sigma$ and average the Gaussian remainder; this rule beats plain weight averaging by 2.0 points on a multimodal benchmark.
  • The learned scalar $s$ in the Gaussian-noise experiment is stable across random seeds (average difference 0.009), implying the standard deviation, not the specific noise draw, carries the adaptation signal.
  • The paper's sigma-based evaluation conjecture, if confirmed, would let practitioners rank fine-tuning methods by comparing the standard deviations of their weight changes without running a test set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If per-layer σ truly encodes adaptation, then hyperparameters like rank and learning rate should be predictable from σ alone; a cheap test is to train any PEFT method, record its layer-wise σ, and check whether it predicts that method's accuracy across a grid of configurations.
  • The sharp, inverted-T, and linear weight patterns in shallow layers are read by the paper as small-noise windows onto the sparse $W^*$; a testable extension is to use per-layer kurtosis as a proxy for training quality or layer difficulty.
  • The i.i.d. assumption is the natural place to probe: measuring the correlation spectrum of attention projections would either harden the foundation or, if correlations are strong, reduce the paper's contribution to an empirical heuristic that still works.
  • The sigma-gap versus performance-gap hypothesis could be stress-tested across many methods and seeds on one task, plotting $|\sigma_1 - \sigma_2|$ against accuracy difference; the paper itself calls this only preliminarily validated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper claims that pretrained and fine-tuned weight matrices of large foundation models are predominantly Gaussian, that weight elements are i.i.d. like Gaussian noise, that transformation weights can be replaced by Gaussian noise with a learned scalar standard deviation, and that the underlying optimal weights are zero-mean, symmetric, and sparse with truncated Gaussian entries plus outliers. It then proposes two applications: LoRA+Ours, which augments LoRA by also learning a scalar multiplier of the pretrained weight matrix, and a model-merging method that amplifies outliers outside a per-layer threshold. The empirical sections report consistent improvements in commonsense reasoning (up to +7.8 points over LoRA) and in multimodal model merging (+2.0 points over averaging).

Significance. The empirical portion of the paper is valuable: the observation that a randomly initialized Gaussian Delta-W with a single learned scalar per layer recovers a large fraction of full fine-tuning performance on GLUE (Table 2) is a striking and falsifiable result, and the proposed LoRA+Ours and merging methods give clear gains. The paper also documents weight histograms across nine models in NLP, CV, and MM, which could serve as a useful reference. However, the theoretical framing goes well beyond what the evidence supports. The i.i.d. claim rests on an unverified independence assumption, the Gaussianity validation uses post hoc filtering, and the M-estimator derivation of W* contains unjustified structural assumptions and a mathematical error in the variance algebra. The paper would be significantly improved by reframing the theoretical claims as conjectures and adding direct tests of independence and distributional fit.

major comments (5)
  1. [Sec. 3.1, Eq. (1)] The derivation of the i.i.d. property assumes independence rather than establishing it. Eq. (1) writes the joint density as a product of Gaussians after noting that the observed pooled histogram is approximately Gaussian, but a Gaussian marginal for the pooled entries does not imply that the entries are independent or identically distributed. The cited references address initialization or random-matrix limits, not trained transformer weights. Since the subsequent claim that transformation weights can be 'directly derived from Gaussian noise' depends on this i.i.d. interpretation, the paper needs either a direct test of independence (e.g., correlation or autocorrelation statistics on the weight matrices) or a reformulation that does not claim i.i.d. properties.
  2. [Appendix A.2, Eq. (8)] The variance algebra in Eq. (8) is incorrect as written. If W and W' are independent samples from distributions with variances sigma_1^2/n and sigma_2^2/m, then the variance of their difference Delta-W is the sum of the two variances, not the difference. If they are correlated because both share W*, then a covariance term is required. The expression 'sigma_2^2/m - sigma_1^2/n' can even be negative, which is impossible for a variance. This error undermines the derivation that Delta-W is zero-mean Gaussian with variance equal to the difference of the two noise variances, which is then used in Sec. 3.3 to explain why adding Gaussian noise helps.
  3. [Sec. 2.2 and Appendix A.1, Table 5] The Gaussianity claim is weakened by post hoc filtering. The main validation computes skewness and kurtosis only after a 3-sigma filter, and Appendix A.1 shows that for layers with high kurtosis (e.g., 3.7-4.5 in ConvNeXt-xlarge stage 2), kurtosis drops to about 3 only after additionally removing all values with magnitude below 10^-3. The ratio reported in Table 1 (99%+) corresponds to the 3-sigma filter only, so the reader has no sense of how much data is removed by the second filter. This does not support the blanket statement that 'nearly all pre-trained weights conform to a Gaussian distribution.' The authors should report the full filtering procedure and the retained fraction for both filters, or use a distributional test on unfiltered weights.
  4. [Sec. 3.3, Eqs. (4)-(5)] The M-estimator argument treats the entire pretrained weight vector as a finite-dimensional parameter estimated from n i.i.d. samples and concludes that W - W* is asymptotically normal with isotropic covariance sigma^2 I. This is an assumption, not a derivation. Neural network weights are high-dimensional and trained with SGD on non-i.i.d. data, and no justification is given for the spherical covariance or for the applicability of classical M-estimator asymptotics to this setting. The subsequent interpretation of W as W* plus Gaussian noise, and the properties of W* derived in Sec. 4, inherit this unverified assumption. The authors should either provide a rigorous argument or explicitly label this as a heuristic model.
  5. [Appendix B.2] The simulation in Appendix B.2 is a curve-fitting exercise, not an independent validation of the W* hypothesis. The standard deviations of the added Gaussian noise (0.001 to 0.3), the truncation bounds (0.001 and 0.5), the sparsity (2 million nonzero of 10 million), and the outlier fraction (0.5% in [0.6, 1]) are all selected after observing the target histograms. The paper even acknowledges in Appendix B that W* is unverifiable. The conclusion that the observed distributions 'can be derived' from the hypothesized W* therefore has no falsifiable content. The authors should soften the claim to a demonstration of plausibility, or propose a concrete, testable implication of the W* model that goes beyond reproducing the already-observed shapes.
minor comments (6)
  1. [Title] The title in the PDF body reads 'Gaussian Distribution Never F ades' with an erroneous space; this should be corrected to 'Fades'.
  2. [Sec. 2.2] There is a typo in the sentence 'kurtosis describes the "tailedness" of a it'; it should read 'of a distribution'.
  3. [Sec. 3.2] The narrative moves from 'carefully crafted Delta-W matrices' that use auxiliary information from fully fine-tuned weights to 'randomly initialized with standard Gaussian noise' without clearly distinguishing the two experimental protocols. The transition should be made explicit so the reader understands that the Gaussian-noise result is a separate ablation.
  4. [Sec. 4] The phrase 'This comes to the conflict' should be 'This leads to a conflict' or 'This creates a conflict'.
  5. [Table 4] The table formatting is broken: the line '36.9Math-LLaV A' concatenates the average score for LLaVA-1.6-13B with the model name for Math-LLaVA. The table should be reformatted for clarity.
  6. [Appendix B.2] The final sentence 'For experimental details, please refer to Appendix. B.2.' refers to the section in which it appears; the intended cross-reference is presumably Appendix C.6.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: the W*-hypothesis 'validation' and the ΔW-Gaussianity derivation reduce to their own assumptions, while the direct histogram observations remain independent.

  1. fitted input called prediction [Appendix B.2, Fig. 7; simulation parameters in Appendix C.6]
    "We first simulate W∗. We randomly fit a sparse matrix and assign its values a truncated Gaussian distribution. Specifically, we set the standard deviation of the Gaussian distribution to 0.1, and then filter out values whose absolute values are less than 0.001 or greater than 0.5, resulting in a truncated Gaussian distribution. Next, we simulate the training process by adding different levels of Gaussian noise to W∗. ... Therefore, based on the properties of W∗, we have derived the weight distributions under various conditions, which correspond well to the observed cases."

    The observed distribution shapes (linear, inverted T, sharp, Gaussian) are produced by explicitly assuming the hypothesis under test: W∗ is a sparse truncated Gaussian plus outliers, and observed weights are W∗ plus Gaussian noise. The noise levels (0.001 to 0.3), truncation bounds (0.001 and 0.5), sparsity ratio (20%), outlier fraction (0.5%), and outlier magnitudes ([0.6, 1]) are hand-chosen in Appendix C.6 to reproduce the target shapes. Figure 7 is therefore a fit of the hypothesis to the data, not an independent derivation or prediction. The paper labels this 'derivation' and 'validation,' but the match is by construction because the simulation's inputs already encode the conclusion it claims to support.

  2. other [Appendix A.2, Eqs. (7)-(8)]
    "Through observation, we know that both the pretrained weights and the fine-tuned weights follow a Gaussian distribution. Naturally, since the transformation weights are the difference between these two sets of weights, they too should follow a Gaussian distribution. ... This implies that ∆W follows a Gaussian distribution with a mean of 0."

    Equation (7) begins by assuming W = W∗ + N(0, σ1²/n I) and W′ = W∗ + N(0, σ2²/m I), i.e., it assumes exactly the Gaussian-noise characterization of weights that the paper claims to establish. The conclusion that ΔW is Gaussian then follows formally as the difference of two Gaussian terms, so Eq. (8) restates the model rather than independently deriving the observed Gaussianity of transformation weights. The paper presents this as explaining why 'transformation weights can be directly derived from Gaussian noise,' but the premise already contains that conclusion; the empirical observation is only the histogram shape, which the derivation does not independently produce.

full rationale

The paper's primary observational claims — that pretrained and fine-tuned weight histograms are approximately Gaussian, with near-zero skewness and kurtosis near 3 — are direct measurements and are not circular. The Sec. 3.2 experiment, in which a random Gaussian ΔW with a trained per-layer scalar s achieves nontrivial GLUE performance, is also genuine empirical evidence with real predictive content. The circularity is partial and located in two places. First, the Appendix B.2 simulation purports to 'derive' the observed distributions from the hypothesized optimal weights W∗, but it constructs W∗ and the additive Gaussian noise using parameters chosen to reproduce those same distributions, making the agreement a fit rather than a validation. Second, the derivation of ΔW's Gaussianity in Appendix A.2 assumes Eq. (7), which already encodes the Gaussian-noise model for W and W′, so Eq. (8) is a tautological consequence, not an independent first-principles derivation. Additionally, the i.i.d. conclusion in Sec. 3.1 depends on a borrowed and unverified premise — asymptotic independence of individual weight elements, cited to initialization and random-matrix literature — and the paper provides no correlation tests; this is a load-bearing missing-support issue rather than a circularity per se. Self-citations (Si et al. 2024a,b,c) are used as method baselines and PEFT references, not to establish the Gaussian claim, so they do not increase the circularity score. Overall, the direct histogram observations remain independent, but two claimed derivations/validations reduce to their own inputs, giving a partial circularity score of 6.

Assumptions & free parameters 6 free parameters · 7 assumptions · 1 invented entities

The central empirical observation (Gaussian weights) is plausible, but the theoretical explanation rests on assumptions about independence, M-estimator normality with isotropic covariance, and an unverifiable optimal weight matrix W*. The fitted simulation in Appendix B.2 uses hand-picked noise levels and truncation bounds to reproduce observed shapes, so it should be read as an illustration rather than a derivation.

free parameters (6)
  • W* simulation Gaussian sigma = 0.1
    In Appendix B.2, the hypothetical optimal weights are simulated from a truncated Gaussian with sigma=0.1, chosen by hand to make simulated distributions resemble observed ones.
  • W* truncation bounds = [-0.5, -0.001] union [0.001, 0.5]
    Values with absolute magnitude below 0.001 or above 0.5 are zeroed to create the truncated Gaussian producing the observed sharp and linear shapes.
  • Simulated sparsity = 20% (2M of 10M nonzero)
    Chosen to satisfy the sparsity condition of the hypothesis, not derived from data.
  • Simulated outlier fraction and range = 0.5% of nonzero in [0.6, 1]
    Chosen by hand to designate outliers in the simulation; no empirical basis is given for these values.
  • Validation filter thresholds = 3-sigma filter plus 10^-3 small-value filter
    Skewness and kurtosis are computed after removing points outside [mu-3sigma, mu+3sigma], and Appendix A.1 removes values below 10^-3 when kurtosis is far from 3; this is a post hoc data selection that affects the Gaussian claim.
  • Merging threshold multiplier t = 2 and 3 (best reported t=2)
    In Algorithm 2, outliers are defined as |delta_w| > t*sigma; t is a hand-chosen hyperparameter and the better t=2 result is emphasized without error bars.
assumptions (7)
  • domain assumption Individual weight elements are asymptotically independent
    Sec. 3.1 cites Thamm et al. 2022, Anonymous 2024, Yang 2020, and Lee et al. 2017 for asymptotic independence; this is required to infer i.i.d. from the observed Gaussian marginal.
  • standard math The trained weight matrix W is an M-estimator of an optimal W* with sqrt(n)(W-W*) converging to N(0, sigma^2 I)
    Sec. 3.3 Eqs. (4)-(5) invoke asymptotic normality of M-estimators (Yohai and Maronna; van der Vaart). The theorem is standard, but its application to non-convex neural network weights is an unproven step.
  • ad hoc to paper The asymptotic covariance of W is isotropic (sigma^2 I)
    Eq. (5) treats each weight element as having equal independent variance; this is a strong assumption for correlated weight matrices and is needed for the Gaussian delta-W variance-difference formula (Eq. 8).
  • domain assumption Pre-training and fine-tuning datasets are subsets of the dataset defining W*
    Sec. 3.3 and Appendix A.2 assume both W and W' are samples from the same W* with different sample sizes n and m, which presumes a fixed optimal weight matrix exists.
  • domain assumption Training injects zero-mean Gaussian noise with layer-dependent variance
    Appendix B.1 states that training inevitably introduces zero-mean Gaussian noise, and this is used to explain sigma growth with depth; no evidence is given that the noise is Gaussian rather than some other distribution.
  • domain assumption Shallow layers are easier to train and thus closer to W*
    Appendix A.4 uses this claim to explain why delta-W standard deviation increases with depth; it is argued qualitatively, not demonstrated.
  • ad hoc to paper Occam's razor and simplicity principles constrain the form of W*
    The hypothesis that W* is sparse, zero-mean, and symmetric is justified by appeals to Newton and Occam in Sec. 3.3 and Appendix B.1, which is a philosophical preference rather than a mathematical theorem.
invented entities (1)
  • Optimal weight matrix W*
    purpose: Postulated true weights from which observed pretrained and fine-tuned weights are noisy samples; used to derive the Gaussian distribution of observed weights, sparsity, and outlier structure.
    The paper admits in Appendix B that W* is unknown, unverifiable, and that 'obtaining and even verifying W* is impossible'. No falsifiable prediction is attached to W* that could be tested independently of the paper's own simulation and filtering choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades." pith.science (2026). https://pith.science/paper/MJZLP5RZ

@misc{pith2026250110661,
  author       = {Pith},
  title        = {Pith review of: Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fades},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJZLP5RZ}},
  note         = {Machine review of arXiv:2501.10661}
}
read the original abstract

This paper presents a pioneering exploration of the mechanisms underlying large foundation models' (LFMs) weights, aiming to simplify AI research. Through extensive observation and analysis on prevailing LFMs, we find that regardless of initialization strategies, their weights predominantly follow a Gaussian distribution, with occasional sharp, inverted T-shaped, or linear patterns. We further discover that the weights share the i.i.d. properties of Gaussian noise, and explore their direct relationship. We find that transformation weights can be derived from Gaussian noise, and they primarily serve to increase the standard deviation of pre-trained weights, with their standard deviation growing with layer depth. In other words, transformation weights broaden the acceptable deviation from the optimal weights, facilitating adaptation to downstream tasks. Building upon the above conclusions, we thoroughly discussed the nature of optimal weights, ultimately concluding that they should exhibit zero-mean, symmetry, and sparsity, with the sparse values being a truncated Gaussian distribution and a few outliers. Our experiments in LFM adaptation and editing demonstrate the effectiveness of these insights. We hope these findings can provide a foundational understanding to pave the way for future advancements in the LFM community.

Figures

Figures reproduced from arXiv: 2501.10661 by the authors.

Figure 1
Figure 1. The distribution of pre-trained weights of prevailing large foundation models across NLP, CV, and MM. We show the weight distribution in different layers and modules. The distribution of these weights exhibits a remarkable resemblance to a Gaussian distribution. We randomly selected and showcased the distribution in several layers and modules. Additionally, we provide the weight distribution plots for each layer in … view at source ↗
Figure 2
Figure 2. The distribution of the transformation matrices with different forms learned by two adaptation methods when fine-tuning LLaMA-7B on commonsense reasoning tasks, including different settings and initialization strategies. Clearly, regardless of different settings, initializations, or computation methods, the transformation weights closely resemble a Gaussian distribution. We randomly selected and showcased the distri… view at source ↗
Figure 3
Figure 3. The distribution of the elements if they are independent but not identically distributed. The subfigures represent the overall distributions derived under the assumption that each element follows a different Gaussian distribution. the impact of outliers, we filter the elements by retaining the values within the range [µ − 3σ, µ + 3σ] 8 . Indeed, we observed that almost all µ of the weight elements is approx￾imately … view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Weight distribution of ConvNeXt-xlarge, Stage 2, Layer 0-17, after 3σ and extremely small value filter. Indeed, similar sharp weight distributions can be observed in other models, such as Qwen2.5-32B. Upon verification, these sharp distributions are primarily caused by…
Figure 5
Figure 5. Figure 5: The relationship between σ difference and performance gap. Through these experiments, and considering that the role of ∆W is to complement the Gaussian noise standard deviation, we believe our hypothesis is preliminarily validated. Furthermore, the implications of this…
Figure 6
Figure 6. Figure 6: (a). The standard deviation of the Gaussian noise increases as the depth of the layers increases and the training data size decreases. Besides, as the training data size decreases, the increase in standard deviation becomes more pronounced in the deeper layers. (b). Th…
Figure 7
Figure 7. Figure 7: Derivation of Weight Distribution. been well-trained. As a result, they are usually few in number and tend to appear in the shallower layers of the model. Moreover, it is important to note that these sharp distributions are not the true optimal weight distributions. Th…
Figure 8
Figure 8. Figure 8: Weight Distribution of LLaMA-7B. 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 100 Layer 0, self_attn.q_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 50 Layer 0, self_attn.k_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 50 Layer …
Figure 9
Figure 9. Figure 9: Weight Distribution of Vicuna-13B. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_9.png]
Figure 10
Figure 10. Figure 10: Weight Distribution of Qwen2.5-32B. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_10.png]
Figure 11
Figure 11. Figure 11: Weight Distribution of SAM-h. 0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.15 0.20 0 10 Stage 0, Layer 0, pwconv1 0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.15 0.20 0 10 Stage 0, Layer 0, pwconv2 0.20 0.15 0.10 0.05 0.00 0.05 0.10 0.15 0.20 0 10 Stage 0, Layer 1, pwconv1 0.20 0.15 …
Figure 12
Figure 12. Figure 12: Weight Distribution of ConvNeXt-xlarge. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]
Figure 13
Figure 13. Figure 13: Weight Distribution of SigLip. 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 100 Layer 0, self_attn.q_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 100 Layer 0, self_attn.k_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 100 Layer…
Figure 14
Figure 14. Figure 14: Weight Distribution of Idefics3-8B. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]
Figure 15
Figure 15. Figure 15: Weight Distribution of LLaVA-NeXT. 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 20 Layer 0, self_attn.q_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 20 Layer 0, self_attn.k_proj 0.100 0.075 0.050 0.025 0.000 0.025 0.050 0.075 0.100 0 50 Laye…
Figure 16
Figure 16. Figure 16: Weight Distribution of Ovis1.6. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]
Figure 17
Figure 17. Figure 17: Weight Distribution of LoRA-Dash (∆Wr=4) when fine-tuning LLaMA-7B. 0.04 0.02 0.00 0.02 0.04 0 50 Layer 0, self_attn.q_proj 0.04 0.02 0.00 0.02 0.04 0 50 Layer 0, self_attn.k_proj 0.04 0.02 0.00 0.02 0.04 0 100 Layer 0, self_attn.v_proj 0.04 0.02 0.00 0.02 0.04 0 50 L…
Figure 18
Figure 18. Figure 18: Weight Distribution of LoRA-Dash (∆Wr=8) when fine-tuning LLaMA-7B. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_18.png]
Figure 19
Figure 19. Figure 19: Weight Distribution of LoRA-Dash (∆Wr=16) when fine-tuning LLaMA-7B. 0.04 0.02 0.00 0.02 0.04 0 200 Layer 0, self_attn.q_proj 0.04 0.02 0.00 0.02 0.04 0 200 Layer 0, self_attn.k_proj 0.04 0.02 0.00 0.02 0.04 0 200 Layer 0, self_attn.v_proj 0.04 0.02 0.00 0.02 0.04 0 1…
Figure 20
Figure 20. Figure 20: Weight Distribution of LoRA-Dash (∆Wr=32) when fine-tuning LLaMA-7B. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_20.png]
Figure 21
Figure 21. Figure 21: Weight Distribution of LoRA-Dash (∆Wr=64) with Xavier initialization when fine-tuning LLaMA-7B. 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 200 Layer 0, self_attn.q_proj 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 200 Layer 0, self_attn.k_proj …
Figure 22
Figure 22. Figure 22: Weight Distribution of DoRA (∆Wr=4) when fine-tuning LLaMA-7B. 39 [PITH_FULL_IMAGE:figures/full_fig_p039_22.png]
Figure 23
Figure 23. Figure 23: Weight Distribution of DoRA (∆Wr=8) when fine-tuning LLaMA-7B. 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 100 Layer 0, self_attn.q_proj 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 100 Layer 0, self_attn.k_proj 0.020 0.015 0.010 0.005 0.000 0.0…
Figure 24
Figure 24. Figure 24: Weight Distribution of DoRA (∆Wr=16) when fine-tuning LLaMA-7B. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_24.png]
Figure 25
Figure 25. Figure 25: Weight Distribution of DoRA (∆Wr=32) when fine-tuning LLaMA-7B. 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 50 Layer 0, self_attn.q_proj 0.020 0.015 0.010 0.005 0.000 0.005 0.010 0.015 0.020 0 50 Layer 0, self_attn.k_proj 0.020 0.015 0.010 0.005 0.000 0.00…
Figure 26
Figure 26. Figure 26: Weight Distribution of DoRA (∆Wr=64) when fine-tuning LLaMA-7B. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_26.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NAN: A Training-Free Solution to Coefficient Estimation in Model Merging

    cs.LG 2025-05 reject novelty 4.0 of 10

    NAN sets merging coefficients inversely proportional to each model's parameter norm and claims a least-squares justification, but the derivation yields a different formula and performance gains are inconsistent.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    A): We provide a more in-depth exploration of weight distribution

    Exploration of Weight Distribution (Appendix. A): We provide a more in-depth exploration of weight distribution

    work page

  2. [2]

    B): Here, we delve deeper into the discussion of W∗

    Open Discussion (Appendix. B): Here, we delve deeper into the discussion of W∗. We strongly encourage readers to review this section for a more profound understanding of our work

    work page

  3. [3]

    C): We include all experimental details presented in the main text

    Experimental Details (Appendix. C): We include all experimental details presented in the main text

    work page

  4. [4]

    D): Other supplementary information, such as algorithm tables, will also be provided

    Additional Details (Appendix. D): Other supplementary information, such as algorithm tables, will also be provided

    work page

  5. [5]

    E): We will introduce relevant literature related to our work

    Related Work (Appendix. E): We will introduce relevant literature related to our work

    work page

  6. [6]

    We hope that the researchers could have a happy journey, and we hope that after reviewing the appendix, they will gain a deeper understanding of our entire work

    Weight Distribution Plots: We will supplement the appendix with weight distribution plots for all layers of the model. We hope that the researchers could have a happy journey, and we hope that after reviewing the appendix, they will gain a deeper understanding of our entire work. Contributions The contributions of this paper are as follows: • We conducted...

    work page

  7. [12]

    Through observation, we know that both the pretrained weights and the fine-tuned weights follow a Gaussian distribution

    Transformation Weights Follow a Gaussian Distribution: 17 Unveiling the Mystery of Weight in Large Foundation Models We have not dedicated much effort to elaborating on the fact that the transformation follows a Gaussian distribution, since we think it is natural for the following reasons. Through observation, we know that both the pretrained weights and ...

    work page

  8. [13]

    Here, we also validate ∆W obtained from LoRA-Dash and DoRA

    Gaussian Noise in Transformation Weights Complements the Standard Deviation of That in Pre-training: Additionally, in the paper, we mention that the Gaussian noise in ∆W serves to complement the standard deviation of the Gaussian noise in pre-training. Here, we also validate ∆W obtained from LoRA-Dash and DoRA. We first present the results of these two me...

    work page 2019

Show all 16 references
  1. [14]

    The results are shown in the Table

    The Closer the Standard Deviations of Two ∆W, the More Similar the Corresponding Model Performance Becomes: Furthermore, we compared the differences in the standard deviations of ∆W obtained from LoRA-Dash and DoRA for the same rank. The results are shown in the Table. 7, row ...

  2. [15]

    physical experiment

    The Standard Deviation of ∆W Increases with the Layer Depth: As will be shown in Appendix. B, the Gaussian noise introduced during the training of each layer increases as the layer depth 17Given that the performance of DoRA at r = 4 is significantly lower, we exclude this sett...

    2002

  3. [204]

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K

    PMLR, 2015. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, ...

    2015 arXiv

  4. [645]

    He, M., Liu, Y ., Wu, B., Yuan, J., Wang, Y ., Huang, T., and Zhao, B

    Springer, 2016b. He, M., Liu, Y ., Wu, B., Yuan, J., Wang, Y ., Huang, T., and Zhao, B. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024. He, P., Gao, J., and Chen, W. Debertav3: Im- proving deberta using electra-style pre-trai...

    2024 arXiv

  5. [1126]

    Feng, T., Jin, C., Liu, J., Zhu, K., Tu, H., Cheng, Z., Lin, G., and You, J

    IOS Press, 2020. Feng, T., Jin, C., Liu, J., Zhu, K., Tu, H., Cheng, Z., Lin, G., and You, J. How far are we from agi. arXiv preprint arXiv:2405.10313, 2024. Fischer, M., Bartler, A., and Yang, B. Prompt tuning for parameter-efficient medical image segmentation.Medical Image A...

    2020 arXiv

  6. [2023]

    However, existing studies have rarely validated or analyzed these observations in depth

    noted that trained weights often exhibit Gaussian-like patterns. However, existing studies have rarely validated or analyzed these observations in depth. Most works have either mentioned Gaussian properties in passing or treated them as incidental findings without further expl...

    2023

  7. [2799]

    PMLR, 2019. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. Hu, Z., Wang, L., Lan, Y ., Xu, W., Lim, E.-P., Bing, L., Xu, X., Poria, S., and Le...

    2019 arXiv

  8. [3938]

    10 Unveiling the Mystery of Weight in Large Foundation Models Han, K., Wang, Y ., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y ., Xiao, A., Xu, C., Xu, Y ., et al

    PMLR, 2020. 10 Unveiling the Mystery of Weight in Large Foundation Models Han, K., Wang, Y ., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y ., Xiao, A., Xu, C., Xu, Y ., et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1...

    2020 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.