Pith. sign in

REVIEW 3 major objections 5 minor 41 references

DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces the DOCS index for comparing transformer weight matrices, proves it can tell orthogonal matrices apart where CKA, CCA, and SVCCA cannot, and uses it to show that neighboring layers of large language models tend to…

desk verdict A cheap and reasonable weight-similarity index with a broad empirical survey, but the central theorem is unproven because the Gumbel MLE degenerates on the constructed example. read the letter →

arxiv 2501.16650 v1 pith:GGXGU2OA submitted 2025-01-28 cs.CL cs.AI

Zeping Min , Xinshang Wang This is my paper · ORCID
classification cs.CLcs.AI
keywords weightmatrixsimilaritylargelanguagemodelscosineorthogonalmatricesGumbeldistributionlayerclusteringtransformerinterpretabilitymixtureofexperts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DOCS (Distribution of Cosine Similarity), an index for comparing the weight matrices of transformer layers by looking at how well each neuron-weight column aligns with some column in the other matrix. Its central claim is that DOCS is genuinely discriminative where standard indices are blind: for orthogonal weight matrices, CCA, SVCCA, and linear CKA all return constant scores, while DOCS provably returns different values for genuinely different matrices. The paper further claims that applying DOCS to open LLMs reveals consistent structure: adjacent layers have unusually similar weights, and groups of nearby layers form clusters, which supports the idea that transformer layers specialize by depth. If these claims hold, DOCS is a cheap, weight-only way to inspect trained models and could guide layer sharing, pruning, distillation, and fine-tuning strategies.

What carries the argument

The object carrying the argument is the per-column maximum absolute cosine similarity. For two weight matrices X and Y, DOCS computes the absolute cosine between every column of X and every column of Y, records, for each column of X, the largest value it attains against columns of Y, and does the same in the reverse direction; it then fits a Gumbel distribution to each of the two lists of maxima by maximum likelihood and takes the average of the two fitted location parameters as the similarity score. The location parameter acts as a summary of how closely the best-matching columns align. The theoretical example that proves discriminative power pairs a matrix of standard basis vectors with a normalized Hadamard matrix, so every absolute cosine is exactly 1/sqrt(m), giving the stated small DOCS value despite the large Frobenius difference.

What would settle it

Run the algorithm described in the paper on the X and Y constructed in Theorem 1; every per-column maximum absolute cosine similarity is 1/sqrt(m), a constant vector, and maximum-likelihood fitting of a Gumbel distribution to a constant sample is degenerate, so Algorithm 1 will not output the claimed 1/sqrt(m) without an added tie-breaking or smoothing step. A reproduction that yields the claimed value would therefore have to use a different estimation procedure than the one defined.

Watch

Extended reading notes

Core claim

The paper's central claim is that DOCS (Distribution of Cosine Similarity) is a valid similarity index for weight matrices, and one that is discriminative where established indices are blind. Existing measures such as CCA, SVCCA, and linear CKA are shown, with proofs, to return a constant score for any pair of orthogonal matrices; DOCS instead assigns different scores to different orthogonal pairs. Theorem 1 constructs column-orthogonal matrices X and Y with Frobenius difference $\Omega$($\sqrt$(m)) and DOCS score 1/$\sqrt$(m), demonstrating that the index can register large structural differences that standard indices collapse to a constant. On real models, DOCS heatmaps show strong similarity between adjacent layers, decreasing similarity with layer distance, and clusters of mutually similar layers, which the paper reads as evidence of depth-wise functional specialization.

Load-bearing premise

The index assumes that fitting a Gumbel curve to the largest per-column cosine similarities is always well-defined and that the fitted location parameter reflects genuine functional overlap, rather than shared input statistics or chance alignments.

Editorial extensions

If this is right

  • Adjacent transformer layers in the studied decoder-only LLMs share unusually similar weights, and weight similarity decreases as layer distance grows, with a secondary rise at the first and last layers.
  • Groups of mutually similar consecutive layers exist at consistent depths across model families and sizes, suggesting a universal structural pattern from training.
  • Base and instruction-tuned versions of the same model keep DOCS scores above 0.7 for every matrix, and the matrices split into three families: MLP up/down, query/key, and value/output.
  • In the MoE model studied, most layers have a single expert whose weights stand apart from the others, consistent with expert specialization or routing imbalance.
  • DOCS has a higher Gini coefficient on layer-similarity heatmaps than the seven other indices tested, meaning its similarity signal is concentrated in fewer, stronger layer pairs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves no null distribution for DOCS, so a natural next step is a permutation test that shuffles columns within layers and asks whether adjacent-layer scores exceed chance; this would calibrate the reported clusters as statistically significant or not.
  • If the clusters genuinely mark redundant computation, sharing or dropping weights inside a detected cluster should degrade downstream performance less than the same intervention across clusters, a prediction the paper motivates but does not test.
  • Because DOCS is orientation-sensitive, rerunning the pipeline on row vectors instead of the transposed column vectors used here would show how much of the reported layer structure depends on the chosen neuron ordering convention.
  • The same analysis applied to vision transformers or diffusion models offers a direct test of the paper's implicit claim that adjacent-layer similarity is a generic outcome of training residual stacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DOCS, a weight-matrix similarity index that computes per-column maximum absolute cosine similarities between two matrices, fits a Gumbel distribution to each of the two resulting vectors, and averages the fitted location parameters. The authors claim that DOCS is discriminative on orthogonal matrices, where CCA, SVCCA, and linear CKA are constant or dimension-dependent, and they present a theorem (Theorem 1) intended to show this discriminative power. Empirically, the paper applies DOCS to weight matrices of several open-source LLMs and reports that adjacent layers are highly similar, that clusters of similar layers exist, that base and instruction-tuned models share high DOCS values, and that Mixture-of-Experts models often contain a distinctive expert. The paper also proves several invariance properties (permutation, symmetry, isotropic scaling, reflexivity) and provides a comparison of DOCS with other indices on a Llama model.

Significance. If the DOCS index is valid, it offers a simple and computationally lightweight tool for analyzing weight-space structure in LLMs, potentially informing layer-level pruning, parameter-efficient fine-tuning, and knowledge distillation. The paper is commendable for giving explicit definitions, a clear algorithm, and concrete numerical examples, and for identifying a real limitation of existing representation-similarity indices on orthogonal matrices. The discriminative example in Appendix A.5 supports the weaker but meaningful claim that DOCS can distinguish some orthogonal pairs. However, the central theoretical result, Theorem 1, rests on a degenerate maximum-likelihood fit, and the main empirical findings are supported only by visual inspection without statistical baselines. The contribution is therefore promising but not yet rigorously established; the issues are local and repairable.

major comments (3)
  1. [Section 3.1, Theorem 1, Appendix C, and Algorithm 1] The proof of Theorem 1 constructs matrices X and Y for which every absolute cosine similarity equals 1/sqrt(m), so the vectors sX and sY are constant. Algorithm 1 then specifies fitting a Gumbel distribution by maximum likelihood. For a constant vector, the Gumbel log-likelihood has no finite maximizer: the likelihood increases without bound as the scale parameter tends to 0, and the location parameter is unidentified. Consequently, Algorithm 1 as written does not produce the stated value SDOCS(X,Y) = 1/sqrt(m), and the theorem is not established by the given proof. The same degeneracy appears in the reflexivity proof in Appendix A.4, where sX is a vector of all ones and the claim that the Gumbel MLE yields uX = 1 is not valid. The weaker Definition 3 claim is separately supported by the explicit orthogonal pair in Appendix A.5, so the issue is repairable; for example, the index could be redefined using a well-defined functional of the max-cosine vectors (e.g., the empirical mean or median), or the algorithm could specify a regularized or limiting Gumbel estimator whose value on constant data is explicitly derived and proven.
  2. [Sections 4.2-4.5 and Table 2] The central empirical claims that adjacent layers exhibit similar weights, that clusters of similar layers exist, and that some MoE experts are uniquely specialized are supported only by visual inspection of heatmaps and scatter plots. The Gini coefficient in Table 2 is a descriptive statistic without confidence intervals or a null baseline. Without a null model (e.g., random orthogonal matrices matched for dimensionality, column-permuted weights, or layer-shuffled matrices), the observed diagonal and block patterns could arise from the index's intrinsic dependence on column norms, dimensionality, or shared input statistics rather than from functional specialization. Please provide quantitative validation, such as permutation tests, bootstrap confidence intervals, or cluster validity indices, and report effect sizes.
  3. [Appendix G] The choice of the Gumbel distribution is justified solely by histograms of the max-cosine vectors from one pair of layers (layers 4 and 8) of a single model, and no goodness-of-fit test or theoretical extreme-value argument is given. This is a post-hoc model selection on the same data that the method then fits, and it does not address the degenerate constant-data case that arises in Theorem 1 and Appendix A.4. Since the interpretation of the DOCS value depends on the location parameter of a well-specified Gumbel model, please either supply a theoretical justification (e.g., from extreme-value theory under explicit independence assumptions) or demonstrate robustness to distributional misspecification by comparing DOCS with nonparametric summaries of the max-cosine vectors.
minor comments (5)
  1. [Section 4, first paragraph] The transposition convention is stated for Wv, Wk, Wq, and MLP-UP, and for W1 and W3 in the MoE experiment, but not for Wo; since Wo heatmaps appear in Appendix I, please clarify whether Wo is also transposed and why.
  2. [Appendix A.2, proof of Lemma 2] The proof states that Cjk = Ckj, but the cosine similarity matrix is not symmetric in general; Cjk uses X_j and Y_k, while C_kj uses X_k and Y_j. The symmetry of SDOCS actually holds because the index averages the fits from both directions, so the proof should be revised to derive symmetry from that averaging rather than from a false symmetry of C.
  3. [Appendix A.4, proof of Lemma 4] The statement 'Fitting a Gumbel distribution to sX consisting of all ones yields a location parameter uX = 1' is not justified; as noted in the major comments, the Gumbel MLE is degenerate for constant data. A separate argument is needed for the reflexivity property if the algorithm is changed.
  4. [Theorem 1 statement] The notation 'm = Ω( n)' appears with a stray space; it should read 'm = Omega(n)'. The proof uses m = 2^{floor(log2 n)}, which indeed gives m = Omega(n), but the construction should be stated more cleanly and the constant in the Omega notation should be explicit.
  5. [Appendix A.5] The discriminative example reports numerical values SDOCS(X,Y) = 0.88 and SDOCS(X',Y') = 0.76, but the intermediate max-cosine vectors and Gumbel fits are not shown. Providing these intermediate computations, or the code used to generate them, would improve reproducibility and allow readers to verify the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the derivation chain is self-contained, though the Theorem 1 proof has a Gumbel-MLE degeneracy gap that is a correctness issue, not a circular one.

full rationale

The paper does not fit DOCS to the empirical conclusions it reports, and it contains no load-bearing self-citations or imported uniqueness claims. The index definition (Algorithm 1), the mathematical properties (Lemmas 1-5), and the empirical heatmaps are all computed from the proposed index and are not used as inputs to define the index. The Gumbel choice is justified in Appendix G by histograms of the same max-cosine statistics the algorithm uses, which is a self-referential model check, but it is not circular: the final similarity value is not tuned to make adjacent layers look similar or to make the theorem true. The one substantive defect is in the proof of Theorem 1 (Appendix C): the constructed matrices give all |cos(theta_jk)| = 1/sqrt(m), so the max-cosine vectors are constant, and the Gumbel maximum-likelihood fit has no finite maximizer; the proof writes SDOCS(X,Y)=1/sqrt(m) directly from the constant max-cosine value rather than from Algorithm 1's MLE. That is a correctness or well-posedness gap in a central theoretical claim, not a circularity: the value is not assumed as a premise, and the discriminative-on-orthogonal definition is also supported independently by the explicit numeric orthogonal pairs in Appendix A.5 (SDOCS 0.88 vs 0.76). The empirical claims about adjacent-layer similarity and clusters are outputs of the method, not fitted inputs, and no prediction is equivalent to its inputs by construction. Hence the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on two fitted Gumbel parameters per matrix pair and on domain assumptions about orthogonality and functional correspondence. No physical entities are introduced; DOCS is a statistical index rather than a new particle, mediator, force, or dimension.

free parameters (3)
  • Gumbel location parameters u_X and u_Y = not reported; estimated per matrix pair
    Algorithm 1 fits Gumbel distributions to max-cosine vectors and uses u_X and u_Y as the index; these are estimated from the data being compared, so DOCS is a fitted statistic rather than a parameter-free measure.
  • Gumbel scale parameter beta = not reported
    The MLE fit also estimates a scale for each pair, but the scale is discarded and no standard errors are reported, so the uncertainty in DOCS values is unknown.
  • Diagonal block sizes for cluster analysis = 3x3 to 7x7
    Section 4.3 chooses these block sizes to identify clusters; the choice affects which clusters are seen and is not derived from the data or from a theory of cluster scale.
assumptions (4)
  • standard math Hadamard matrices of size m = 2^k exist for the construction
    Appendix C uses H_m with H_m H_m^T = m I for m a power of two; this is a standard result.
  • domain assumption Gumbel distribution models the per-column maximum absolute cosine similarities
    Algorithm 1 fits Gumbels without a proof; Appendix G only shows two empirical histograms, and the maxima are dependent and bounded, so extreme value theory does not guarantee a Gumbel limit.
  • domain assumption LLM weight matrices are approximately orthogonal
    Section 2 motivates the orthogonality focus by citing Tian et al. and by measuring a proxy, the off-diagonal average absolute cosine similarity, in Appendix E.2; the proxy is not a rigorous orthogonality test.
  • domain assumption Maximum absolute cosine alignment between columns indicates functional correspondence
    Section 3 states that DOCS captures significant parameter alignments; this interpretability claim is assumed rather than tested against function, for example through ablations or task probes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models." pith.science (2026). https://pith.science/paper/GGXGU2OA

@misc{pith2026250116650,
  author       = {Pith},
  title        = {Pith review of: DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGXGU2OA}},
  note         = {Machine review of arXiv:2501.16650}
}
read the original abstract

We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs), aiming to facilitate the analysis of their complex architectures. Leveraging DOCS, our analysis uncovers intriguing patterns in the latest open-source LLMs: adjacent layers frequently exhibit high weight similarity and tend to form clusters, suggesting depth-wise functional specialization. Additionally, we prove that DOCS is theoretically effective in quantifying similarity for orthogonal matrices, a crucial aspect given the prevalence of orthogonal initializations in LLMs. This research contributes to a deeper understanding of LLM architecture and behavior, offering tools with potential implications for developing more efficient and interpretable models.

Figures

Figures reproduced from arXiv: 2501.16650 by the authors.

Figure 1
Figure 1. Comparison of similarity indices applied to representation similarities ((a) and (b)) and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of similarity indices on the MLP-U [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The top row displays heatmaps of DOCS scores between layers for different weight ma [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Analysis of Wv matrices across various LLMs. Top row: Heatmaps visualize DOCS similarity scores between transformer layers. Bottom row: Average DOCS scores are computed for diagonal blocks (sizes 3x3 to 7x7) within each heatmap. (a) MLP-DOWN (b) MLP-UP (c) Wq (d) Wv […
Figure 5
Figure 5. Figure 5: Average DOCS scores for diagonal blocks of varying sizes (3x3 to 7x7) within heatmaps [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: DOCS similarity scores between the base and instruction fine-tuned weight matrices for [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: DOCS similarity scores between the MoE experts. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Comparison between DOCS and Linear CKA on the MLP-U [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: Average cosine similarity for Meta-Llama-3.1-8B-Instruct, with the Q matrix on the left [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Comparison of similarity ratios between models (A) and (B) relative to models (A) and [PITH_FULL_IMAGE:figures/full_fig_p027_10.png]
Figure 11
Figure 11. Figure 11: Comparison of similarity indices on the MLP-U [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]
Figure 12
Figure 12. Figure 12: Histograms of the maximum cosine similarity vectors [PITH_FULL_IMAGE:figures/full_fig_p028_12.png]
Figure 13
Figure 13. Figure 13: Comparison of similarity heatmaps for the MLP-U [PITH_FULL_IMAGE:figures/full_fig_p029_13.png]
Figure 14
Figure 14. Figure 14: DOCS scores between transformer layers in gemma-2-9b. [PITH_FULL_IMAGE:figures/full_fig_p030_14.png]
Figure 15
Figure 15. Figure 15: DOCS scores between transformer layers in gemma-2-27b. [PITH_FULL_IMAGE:figures/full_fig_p030_15.png]
Figure 16
Figure 16. Figure 16: DOCS scores between transformer layers in Llama-3.1-3B. [PITH_FULL_IMAGE:figures/full_fig_p031_16.png]
Figure 17
Figure 17. Figure 17: DOCS scores between transformer layers in Llama-3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p031_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 17 canonical work pages

  1. [1]

    Representation topol- ogy divergence: A method for comparing neural network representations

    Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topol- ogy divergence: A method for comparing neural network representations. arXiv preprint arXiv:2201.00058,

    arXiv

  2. [2]

    Define: • X ∈ Rn×m as X = [e1, e2,

    Let m = 2⌊log2 n⌋, so m = Ω(n). Define: • X ∈ Rn×m as X = [e1, e2, . . . , em], where ei is the i-th standard basis vector in Rn. • Y ∈ Rn×m as: Y = 1√m Hm 0(n−m)×m , where Hm is the m × m Hadamard matrix, and 0 is a zero matrix. Orthogonality of X and Y : 21 Published as a conference paper at ICLR 2025 • The columns of X are orthonormal since they are st...

    work page 2025

  3. [5]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  4. [7]

    Training compute- optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Aurelia Guy, Jacob Menick, Rapha¨el Milliere, Scott Reed, et al. Training compute- optimal large language models. arXiv preprint arXiv:2203.15556,

    arXiv

  5. [8]

    The results underscore the strengths of DOCS in revealing the weight structure. Specifically, DOCS reveals an intricate pattern in Figure 8a, indicating that layer 9 is highly similar to layer 25 , layer 10 is highly similar to layer 26, layer 11 is highly similar to layer 27, and so on. This suggests a repetition of a section of layers within 01-ai/Yi-1....

    work page 1984

  6. [9]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088,

    arXiv

  7. [11]

    For any orthogonal matrices X, Y∈ Rn×n, SLinear HSIC(X, Y) = Y TX 2 F (n − 1)2 is a constant that depends solely on n. Proof. Let X and Y be any orthogonal matrices in Rn×n. Since both X and Y are orthogonal, their transpose inverses satisfy X TX = Y TY = In, where In is the n × n identity matrix. Consider the product Y TX. Since the product of two orthog...

    work page 2025

  8. [12]

    Towards measuring representational similarity of large language models

    11 Published as a conference paper at ICLR 2025 Max Klabunde, Mehdi Ben Amor, Michael Granitzer, and Florian Lemmerich. Towards measuring representational similarity of large language models. InUniReps: the First Workshop on Unifying Representations in Neural Models, 2023a. Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similar...

    arXiv 2025

Show all 41 references
  1. [14]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, Tamara Duke, et al. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668,

    2006 arXiv

  2. [16]

    Beyond kv caching: Shared attention for efficient llms

    Bingli Liao and Danilo Vasconcellos Vargas. Beyond kv caching: Shared attention for efficient llms. arXiv preprint arXiv:2407.12866,

    arXiv

  3. [18]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155,

    arXiv

  4. [21]

    Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks

    Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025,

    arXiv

  5. [22]

    Similarity of neural networks with gradients

    12 Published as a conference paper at ICLR 2025 Shuai Tang, Wesley J Maddox, Charlie Dickens, Tom Diethe, and Andreas Damianou. Similarity of neural networks with gradients. arXiv preprint arXiv:2003.11498,

    2025 arXiv

  6. [23]

    Joma: Demystifying mul- tilayer transformers via joint dynamics of mlp and attention

    Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du. Joma: Demystifying mul- tilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.00535,

    arXiv

  7. [24]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  8. [25]

    Simi- larity analysis of contextual word representation models.arXiv preprint arXiv:2005.01172,

    John M Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Simi- larity analysis of contextual word representation models.arXiv preprint arXiv:2005.01172,

    2005 arXiv

  9. [26]

    Glm-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414,

    arXiv

  10. [27]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068,

    arXiv

  11. [28]

    Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568,

    arXiv

  12. [29]

    Taming sparsely activated transformer with stochastic experts

    Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint arXiv:2110.04260,

    arXiv

  13. [30]

    Let X, Y∈ Rn×m and let PX , PY ∈ Rm×m be permutation matrices

    13 Published as a conference paper at ICLR 2025 A P ROOFS OF MATHEMATICAL PROPERTIES OF DOCS S IMILARITY INDICES A.1 P ROOF OF PERMUTATION TRANSFORMATION INVARIANCE Lemma 1 (Permutation Transformation Invariance). Let X, Y∈ Rn×m and let PX , PY ∈ Rm×m be permutation matrices. ...

    2025

  14. [31]

    −0.6676 0 .5171 −0.5357 −0.7310 −0.5917 0 .3399 −0.1412 0 .6185 0 .7730 # , Y =

    A.5 P ROOF OF DISCRIMINATIVE ON ORTHOGONAL MATRICES Lemma 5 (Discriminative on Orthogonal Matrices). There exist orthogonal matrices X, Y, X′, Y′ such that: SDOCS(X, Y) ̸= SDOCS(X ′, Y′). Proof. Consider the following orthogonal matrices: First pair: X = "−0.6676 0 .5171 −0.53...

    2025

  15. [32]

    17 Published as a conference paper at ICLR 2025 B.2 C ANONICAL CORRELATION ANALYSIS (CCA) WITH R2 CCA CCA R2 CCA is defined as: SCCA(R2 CCA)(X, Y) = QT Y QX 2 F n , where QX and QY represent orthonormal bases corresponding to the columns of X and Y , respec- tively. Lemma

    2025

  16. [33]

    Meanwhile, TX and TY denote truncated identity matrices that retain the left singular vectors, ensuring that the accumulated variance meets a predefined limit

    18 Published as a conference paper at ICLR 2025 B.4 S INGULAR VECTOR CCA (SVCCA) WITH R2 SVCCA SVCCA R2 SVCCA is defined as: SR2 SVCCA = (UY TY )T UX TX 2 F min ∥TX ∥2 F , ∥TY ∥2 F , where UX and UY represent the left singular vectors of X and Y , respectively, arranged in de-...

    2025

  17. [34]

    Meanwhile, TX and TY denote truncated identity matrices that retain the left singular vectors, ensuring that the accumulated variance meets a predefined limit

    B.5 S INGULAR VECTOR CCA (SVCCA) WITH ¯ρSVCCA SVCCA (¯ρSVCCA ) is defined as: S ¯ρSVCCA = (UY TY )T UX TX ∗ min ∥TX ∥2 F , ∥TY ∥2 F , where UX and UY represent the left singular vectors of X and Y , respectively, arranged in de- scending order based on their associated singula...

    2025

  18. [37]

    ∥X − Y ∥F = √ 2m = Ω(√n)

    = 2m. ∥X − Y ∥F = √ 2m = Ω(√n). Computing SDOCS(X, Y): For columns Xj and Yk, the cosine similarity is: cos(θjk ) = X ⊤ j Yk ∥Xj∥∥Yk∥ = Yjk ∥Yk∥ . 22 Published as a conference paper at ICLR 2025 Since ∥Xj∥ = 1 and ∥Yk∥ = 1, and Yjk = hjk√m, we have: |cos(θjk )| = 1√m . Thus, t...

    2025

  19. [39]

    This formulation indicates that a larger Off-Diagonal Average Cosine Similarity value corresponds to a lower degree of orthogonality in the matrix

    nX i=1 nX j=1 j̸=i xi · xj ∥xi∥ ∥xj∥ , where X is a matrix with n columns, and xi represents the i-th column of X. This formulation indicates that a larger Off-Diagonal Average Cosine Similarity value corresponds to a lower degree of orthogonality in the matrix. To further und...

    2025

  20. [40]

    To quantify this, we define the similarity ratio as the ratio of the similarity scores between models (A) and (B) to those between models (A) and (C)

    To further compare the effectiveness of the DOCS method with other approaches, we conducted additional experiments using three models: (A) meta-llama/Meta-Llama-3.1-8B (the base model) (B) meta-llama/Meta-Llama-3.1-8B-Instruct (an instruction-tuned version of the base model) (...

    2025

  21. [41]

    Importance of the Maximization Function The maximization operation in the M AXCOSSIM function plays a crucial role in the DOCS algo- rithm

    The overlaid curves represent the fitted Gumbel distributions. Importance of the Maximization Function The maximization operation in the M AXCOSSIM function plays a crucial role in the DOCS algo- rithm. To demonstrate this, we compare the results of the standard DOCS algorithm...

    2025

  22. [1984]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Kelsey Maziarz, Andy Davis, Quoc V Le, Geoffrey E Hinton, and Jeffrey Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,

    arXiv

  23. [2005]

    Diachronic word embeddings reveal statis- tical laws of semantic change

    William L Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statis- tical laws of semantic change. arXiv preprint arXiv:1605.09096,

    arXiv

  24. [2008]

    The remarkable robustness of llms: Stages of infer- ence? arXiv preprint arXiv:2406.19384,

    Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of infer- ence? arXiv preprint arXiv:2406.19384,

    arXiv

  25. [2016]

    Stable- moe: Stable routing strategy for mixture of experts

    Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stable- moe: Stable routing strategy for mixture of experts. arXiv preprint arXiv:2204.08396,

    arXiv

  26. [2017]

    Contrasim–analyzing neural representations based on con- trastive learning

    Adir Rahamim and Yonatan Belinkov. Contrasim–analyzing neural representations based on con- trastive learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape...

    2024

  27. [2018]

    Cross-layer attention sharing for large language models

    Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Qiaozhi He, Murun Yang, Tong Xiao, and Jingbo Zhu. Cross-layer attention sharing for large language models. arXiv preprint arXiv:2408.01890,

  28. [2019]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  29. [2020]

    Flm-101b: An open llm and how to train it with 100 k budget

    Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al. Flm-101b: An open llm and how to train it with 100 k budget. arXiv preprint arXiv:2309.03852,

    arXiv

  30. [2021]

    Gpt-neox-20b: An open-source autore- gressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autore- gressive language model. arXiv preprint arXiv:2204.06745,

    arXiv

  31. [2022]

    Language models are few-shot learners

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and others. Language models are few-shot learners. In Advances in Neural Information Processing Systems , volume 33, pp. 1877–1901,

    1901

  32. [2023]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    2001 arXiv

  33. [2024]

    Mistral 7b: A compact and effective transformer model

    Xinyu Jiang, John Smith, Jane Doe, et al. Mistral 7b: A compact and effective transformer model. arXiv preprint arXiv:2309.05500,

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.