Pith. sign in

REVIEW 2 major objections 6 minor 62 references

The heavy-tailed shape of an LLM's weight spectra is a compact, data-free signature that tracks lineage, clusters families, and proxies performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 02:54 UTC pith:6AGZU3GF

load-bearing objection Solid empirical packaging of HT-SR shape metrics into a practical, data-free LLM fingerprint that works well for lineage and clustering; the performance-proxy half is correlational and weaker. the 2 major comments →

arxiv 2607.03377 v1 pith:6AGZU3GF submitted 2026-07-03 cs.CL cs.AI

Spectral Signatures of Large Language Models

Zhuoying Zhang , Ishan V. Prasad , Yuanzhe Hu , Zihang Liu , Hengrui Luo , Pu Ren , Yaoqing Yang This is my paper
classification cs.CL cs.AI
keywords large language modelsmodel lineagespectral signaturesheavy-tailed self-regularizationweight-space learningempirical spectral densitymodel rankingPL_Alpha_Hill
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Public LLM collections are too large and heterogeneous for task-by-task benchmarks or prompt-based fingerprints. This paper argues that the shape of each weight matrix's eigenvalue spectrum, summarized by a single heavy-tail index (PL_Alpha_Hill), already encodes the model's pre-training identity and remains stable under fine-tuning, merging, pruning, and output-preserving reparameterizations. Because the signature is computed once from weights alone, needs no data, and is scale-invariant, it can be stored and compared cheaply across hundreds of models. On a curated corpus the authors show that the same signature correctly routes derived models to their base families, separates families in unsupervised clustering, and supports nearest-neighbor interpolation of benchmark scores with statistically significant rank correlation. The practical claim is that spectral shape supplies a lightweight, theory-backed proxy for organizing and ranking modern open-model ecosystems.

Core claim

The layer-wise power-law tail index of weight empirical spectral densities (PL_Alpha_Hill) forms a compact spectral signature of an LLM that is fixed mainly during pre-training, robust to post-training and output-invariant transforms, and sufficiently informative to support accurate lineage classification, unsupervised family clustering, and nearest-neighbor performance prediction without any task data.

What carries the argument

PL_Alpha_Hill — the Hill estimator of the power-law tail index of the eigenvalues of W^T W for each weight matrix — aggregated into a module-by-layer tensor that serves as the model's spectral signature; similarity is then measured by average Spearman rank correlation of these tensors.

Load-bearing premise

That models whose weight spectra have similar heavy-tailed shapes also have comparable generalization ability, so that nearest-neighbor interpolation in spectral space can stand in for actual benchmark scores.

What would settle it

Take a large set of models whose architectures or training regimes lie outside the curated corpus (for example, extreme sparsity, novel mixture-of-experts designs, or heavy continual pre-training) and check whether spectral-signature nearest neighbors still recover statistically significant rank correlation on held-out benchmarks; a systematic failure would falsify the performance-proxy claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Model repositories can maintain a compact spectral index that supports lineage queries and family clustering without storing activations or running inference.
  • Output-invariant reparameterizations and moderate noise no longer break weight-space similarity measures, enabling more reliable provenance checks.
  • Performance trends across hundreds of open models can be estimated from a few reference scores via spectral nearest-neighbor interpolation.
  • The same signature can be adapted to mixture-of-experts and depth-mismatched pairs by averaging experts or dynamic-programming layer matching.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If spectral signatures remain stable under quantization and low-rank adapters, they could serve as a lightweight integrity check for model marketplaces and licensing audits.
  • The method may expose which layers change most under different post-training regimes, offering a diagnostic for targeted fine-tuning or pruning without task data.
  • A natural next stress test is whether the geometric-to-performance link survives when models are trained on radically different data mixtures or objectives not represented in the current leaderboard corpus.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes spectral signatures for large language models based on Heavy-Tailed Self-Regularization theory: the layer-wise PL_Alpha_Hill shape statistic of weight empirical spectral densities is aggregated into a compact tensor Z ∈ R^{N_mod × L}. This signature is claimed to be data-free, scale-invariant, and robust under post-training and output-invariant reparameterizations. On curated corpora (128 models for lineage; 499 Open LLM Leaderboard models for ranking), the authors show nearest-neighbor lineage classification at 98.44% accuracy, unsupervised family clustering with Silhouette 0.91, and distance-weighted k-NN prediction of ARC/HellaSwag/MMLU/TruthfulQA scores with low MAE and statistically significant Kendall Tau in all 100 trials, outperforming or matching data-aware baselines at far lower cost.

Significance. If the claims hold, the work supplies a practical, theory-grounded, weight-only kernel for organizing large open-model repositories—lineage tracing, clustering, and coarse performance proxying—without prompts or inference. Strengths include a large multi-family corpus, systematic comparison to PCS, REEF, Logits, GhostSpec, EmbedLLM and LLMDNA, explicit robustness ablations under scaling, permutation and noise (Sec. 5.5, Tabs. 8–9, Fig. 5), MoE and cross-depth generalization tests (Sec. 5.4), and released code. The lineage and clustering results are especially actionable for IP/safety and model-zoo management; the performance-proxy result, if deconfounded, would further reduce evaluation cost at scale.

major comments (2)
  1. [Sec. 4.2, Sec. 5.3, Tab. 5, Fig. 1] Sec. 4.2 (“Predicting LLM Performance”) and Sec. 5.3 / Tab. 5 rest on the assumption that similar HT-SR spectral patterns imply comparable generalization, so that d = 1 − Sim and distance-weighted k-NN yield a reliable proxy. The reported MAE and 100/100 significant Kendall Tau are purely correlational on a multi-family pool; there is no leave-one-family-out, scale-matched, or family-identity-controlled ablation. Because the same signature is shown to be nearly invariant under post-training (Fig. 1, Sec. 3) while benchmark scores can move substantially, family/architecture co-occurrence may drive much of the predictive signal. Appendix 7.5’s limited intra-family checks help but do not close the gap. A leave-one-family-out (or within-family residual) protocol, or an explicit caveat that the proxy is largely family-level, is needed for the quantification half of the central claim.
  2. [Tab. 2, Sec. 5.1] Tab. 2 (Unrelated Models): spectral-signature similarities to architecturally distinct bases remain high (e.g., 0.92 vs llama-7b, 0.77 vs Mistral-7B, 0.66 vs Llama-3.1-8B), while PCS collapses near zero. Classification still reaches 98.44% because same-family scores are higher, but absolute similarity is poorly calibrated as a “lineage vs independent” decision statistic. The paper should report ROC/threshold analysis or calibrated decision rules, and clarify when high inter-family scores are expected (shared transformer inductive bias vs true ancestry).
minor comments (6)
  1. [Eq. (1), Sec. 3] Eq. (1): the Hill cutoff is fixed at k = n/2 with a brief citation; a short sensitivity plot (or appendix table) over k would strengthen the “free-parameter” discussion.
  2. [Fig. 4, Tab. 5] Fig. 4 shows only MMLU and ARC; adding HellaSwag and TruthfulQA scatter plots (or residual plots) would match Tab. 5 more completely.
  3. [Sec. 5.2, Sec. 4.2] Sec. 5.2: UMAP is used before HDBSCAN/BGMM/K-Means while silhouette is computed in original space—state the UMAP dimension and distance metric explicitly for reproducibility.
  4. [Table 1] Table 1 marks Logits/REEF as non-scalable and non-theory-driven; a one-sentence justification in the caption would avoid appearing categorical.
  5. [Sec. 4.1] Minor notation: Z vs 𝑍 / 𝑍^{(k)} and PL_Alpha_Hill vs PL_Alpha_Hill / α are used interchangeably; unify in Sec. 4.1.
  6. [Appendix 7.1, Sec. 3] Appendix 7.1 Fig. 8 training-step evolution is informative; a pointer from the main-text robustness paragraph would help readers find it.

Circularity Check

1 steps flagged

Minor non-load-bearing self-citations to prior HT-SR applications by overlapping authors; no derivation reduces to its inputs by construction.

specific steps
  1. self citation load bearing [Sec. 3, paragraph after Eq. 1]
    "For the selection of k, we adopt the approach from previous works [50, 57], where we set k = n/2."

    References [50] and [57] share co-authors with the present paper. The choice of the Hill-estimator cutoff is therefore taken from the authors’ own prior empirical practice rather than re-derived. The step is minor: it affects only a hyper-parameter of a standard estimator and does not make any reported accuracy, silhouette, or MAE number true by construction.

full rationale

The paper is an empirical methods paper that applies the standard Hill estimator (Eq. 1) of the power-law tail of weight-matrix ESDs, aggregates the resulting PL_Alpha_Hill values into a model-level tensor signature, and then uses ordinary Spearman similarity / k-NN for three external tasks. Lineage labels are taken from public training provenance, clustering is unsupervised and scored by silhouette/DBI on the signatures themselves, and performance prediction is evaluated against independent Open LLM Leaderboard scores (and against activation-based baselines EmbedLLM/LLMDNA). None of these quantities is defined in terms of the signature, fitted from the same data that is later “predicted,” or forced by a uniqueness theorem. The only self-citations are to earlier HT-SR applications that supply the conventional choice k = n/2 and the general claim that ESD shape correlates with training quality; those citations are not required for the algebraic construction of the signature nor for the external validation numbers. Consequently the central claims remain independently falsifiable and the circularity score stays low.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims rest on the empirical validity of HT-SR for modern LLMs, a conventional choice of Hill-estimator cutoff, and the modeling assumption that spectral-shape proximity implies performance proximity. No new physical entities are postulated; free parameters are limited to standard estimator and algorithm settings.

free parameters (3)
  • Hill estimator cutoff k = n/2
    Fixed to k = n/2 following prior HT-SR papers; controls which eigenvalues enter the tail-index estimate and is not derived from first principles.
  • k-NN neighborhood size = 3
    Set to k=3 for performance interpolation; chosen by the authors and affects predicted scores.
  • UMAP / clustering hyperparameters
    Selected by maximizing silhouette score; influence unsupervised cluster recovery.
axioms (4)
  • domain assumption Well-trained neural networks exhibit heavy-tailed empirical spectral densities whose power-law tail index (PL_Alpha_Hill) encodes training quality and capacity.
    Taken from HT-SR theory (Martin & Mahoney et al.); invoked throughout Sec. 3 and used as the foundation for the spectral signature.
  • domain assumption The spectral signature is predominantly fixed during pre-training and remains essentially invariant under typical post-training (SFT, RL, light pruning).
    Stated and illustrated in Sec. 3 and Fig. 1; load-bearing for lineage robustness claims.
  • ad hoc to paper Models with similar HT-SR spectral patterns possess comparable generalization capabilities, justifying distance-weighted k-NN performance interpolation.
    Explicit modeling assumption in Sec. 4.2; not independently derived.
  • ad hoc to paper Spearman rank correlation of layer-wise PL_Alpha_Hill vectors is a suitable similarity for lineage and clustering.
    Chosen in Eq. 2–3 for scale-invariance; alternatives (RBF, cosine) are also tested but Spearman is primary.
invented entities (1)
  • Spectral signature tensor Z ∈ R^{N_mod × L} independent evidence
    purpose: Compact model-level representation obtained by stacking PL_Alpha_Hill values of each module type across layers.
    A convenient packaging of existing per-layer statistics rather than a new ontological entity; independent evidence is the empirical utility shown in the experiments.

pith-pipeline@v1.1.0-grok45 · 23954 in / 3041 out tokens · 27686 ms · 2026-07-12T02:54:37.678456+00:00 · methodology

0 comments
read the original abstract

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.

Figures

Figures reproduced from arXiv: 2607.03377 by Hengrui Luo, Ishan V. Prasad, Pu Ren, Yaoqing Yang, Yuanzhe Hu, Zhuoying Zhang, Zihang Liu.

Figure 1
Figure 1. Figure 1: Eigenspectrum shape serves as a distinct signature [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our proposed shape-based spectral signature framework. We analyze the eigenspectrum of model weights [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Classification accuracy of the 8-way classifier for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Predicted versus ground-truth benchmark scores on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Robustness of different similarity metrics under [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Heatmap of spectral signature similarity across [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The evolution of layer-wise PL_Alpha_Hill through [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: t-SNE 2D embedding for spectral clustering on PL_ [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 1 canonical work pages

  1. [1]

    Jeff Alstott, Ed Bullmore, and Dietmar Plenz. 2014. powerlaw: a Python package for analysis of heavy-tailed distributions.PloS one9, 1 (2014), e85777

    2014

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, et al . 2023. Qwen Technical Report. arXiv:2309.16609 [cs.CL] https://arxiv.org/abs/2309.16609

    Pith/arXiv arXiv 2023

  3. [3]

    Richard Bellman. 1958. Dynamic Programming.Science127, 3304 (1958), 976–976

    1958

  4. [4]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question An- swering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457 [cs.AI] https://arxiv.org/abs/1803.05457

    Pith/arXiv arXiv 2018

  5. [5]

    Aaron Clauset, Cosma Rohilla Shalizi, and Mark EJ Newman. 2009. Power-law distributions in empirical data.SIAM review51, 4 (2009), 661–703

    2009

  6. [6]

    2022.Random matrix methods for machine learning

    Romain Couillet and Zhenyu Liao. 2022.Random matrix methods for machine learning. Cambridge University Press, Cambridge, UK

    2022

  7. [7]

    David L Davies and Donald W Bouldin. 2009. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligencePAMI-1, 2 (2009), 224–227

    2009

  8. [8]

    2023.OpenLLaMA: An Open Reproduction of LLaMA

    Xinyang Geng and Hao Liu. 2023.OpenLLaMA: An Open Reproduction of LLaMA. OpenLM Research. https://github.com/openlm-research/open_llama

    2023

  9. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsieh. 2022. Watermarking Pre-trained Language Models with Backdooring. https://api.semanticscholar.org/CorpusID:252907247

    2022

  11. [11]

    Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023. Knowledge is a Region in Weight Space for Finetuned Language Models

    2023

  12. [12]

    Mert Gurbuzbalaban, Umut Simsekli, and Lingjiong Zhu. 2021. The heavy-tail phenomenon in SGD. InInternational Conference on Machine Learning. PMLR, Virtual, 3964–3975

    2021

  13. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. arXiv:2009.03300 [cs.CY] https://arxiv.org/abs/2009.03300

    Pith/arXiv arXiv 2021

  14. [14]

    Bruce M Hill. 1975. A simple general approach to inference about the tail of a distribution.The annals of statistics3, 5 (1975), 1163–1174

    1975

  15. [15]

    Liam Hodgkinson and Michael Mahoney. 2021. Multiplicative noise and heavy tails in stochastic optimization. InInternational Conference on Machine Learning. PMLR, Virtual, 4262–4274

    2021

  16. [16]

    Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. 2022. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning. PMLR, Baltimore, Maryland, USA, 8774–8795

    2022

  17. [17]

    Liam Hodgkinson, Zhichao Wang, and Michael W Mahoney. 2025. Models of heavy-tailed mechanistic universality

    2025

  18. [18]

    Eliahu Horwitz, Nitzan Kurer, Jonathan Kahana, Liel Amar, and Yedid Hoshen

  19. [19]

    We Should Chart an Atlas of All the World’s Models

  20. [20]

    Eliahu Horwitz, Asaf Shul, and Yedid Hoshen. 2025. Unsupervised Model Tree Heritage Recovery. InProceedings of the 13th International Conference on Learning Representations. OpenReview.net, Singapore, 47900–47918

    2025

  21. [21]

    Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, and Yaoqing Yang. 2025. Eigenspec- trum Analysis of Neural Networks without Aspect Ratio Bias. InProceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research). PMLR, Vancouver, Canada, 24290–24313

    2025

  22. [22]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. 2023. Mistral 7B. arXiv:2310.06825 [cs.CL] https://arxiv.org/abs/2310.06825

    Pith/arXiv arXiv 2023

  23. [23]

    Ya Jiang, Chuxiong Wu, Massieh Kordi Boroujeny, Brian Mark, and Kai Zeng. 2025. StealthInk: A Multi-bit and Stealthy Watermark for Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research). PMLR, Vancouver, Canada, 27685–27709

    2025

  24. [24]

    Heng Jin, Chaoyu Zhang, Shanghao Shi, Wenjing Lou, and Y Thomas Hou. 2024. Proflingo: A fingerprinting-based intellectual property protection scheme for large language models. In2024 IEEE Conference on Communications and Network Security (CNS). IEEE, Taipei, Taiwan, 1–9

    2024

  25. [25]

    1948.Rank Correlation Methods

    Maurice George Kendall. 1948.Rank Correlation Methods. Charles Griffin & Company, London, UK

    1948

  26. [26]

    Zhenyu Liao and Michael W Mahoney. 2025. Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models

    2025

  27. [27]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Dublin, Ireland, 3214–3252

    2022

  28. [28]

    Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang

  29. [29]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.)

    Model Balancing Helps Low-data Training and Fine-tuning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 1311–1331. doi:10.18653/v1/ 2024.emnlp-main.78

    doi:10.18653/v1/ 2024

  30. [30]

    Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. 2024. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models.Advances in neural information processing systems37 (2024), 9117–9152

    2024

  31. [31]

    Michael Mahoney and Charles Martin. 2019. Traditional and heavy tailed self regularization in neural network models. InInternational Conference on Machine Learning. PMLR, Long Beach, CA, USA, 4284–4293

    2019

  32. [32]

    Vladimir A Marčenko and Leonid Andreevich Pastur. 1967. Distribution of eigenvalues for some sets of random matrices.Mathematics of the USSR-Sbornik 1, 4 (1967), 457–483

    1967

  33. [33]

    Charles H Martin and Christopher Hinrichs. 2025. SETOL: A Semi-Empirical Theory of (Deep) Learning

    2025

  34. [34]

    Charles H Martin and Michael W Mahoney. 2021. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research22, 165 (2021), 1–73

    2021

  35. [35]

    Charles H Martin, Tongsu Serena Peng, and Michael W Mahoney. 2021. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.Nature Communications12, 1 (2021), 1–13

    2021

  36. [36]

    Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open- LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evalua- tion, Benchmark, and Arena

    2024

  37. [37]

    Ivica Nikolic, Teodora Baluta, and Prateek Saxena. 2025. Model Provenance Testing for Large Language Models. InAdvances in Neural Information Processing Systems. Curran Associates, Inc., Sydney, Australia, 34126–34153

    2025

  38. [38]

    Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.Journal of computational and applied mathematics 20 (1987), 53–65

    1987

  39. [39]

    Konstantin Schürholt, Léo Meynent, Yefan Zhou, Haiquan Lu, Yaoqing Yang, and Damian Borth. 2025. A Model Zoo on Phase Transitions in Neural Networks

    2025

  40. [40]

    Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. 2019. A tail-index analysis of stochastic gradient noise in deep neural networks. InInternational Conference on Machine Learning. PMLR, Long Beach, CA, USA, 5827–5837

    2019

  41. [41]

    Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. 2020. Hausdorff dimension, heavy tails, and generalization in neural networks.Ad- vances in Neural Information Processing Systems33 (2020), 5138–5151

    2020

  42. [42]

    Mingjie Sun, Yida Yin, Zhiqiu Xu, J Zico Kolter, and Zhuang Liu. 2025. Idiosyn- crasies in large language models

    2025

  43. [43]

    Yehui Tang, Xiaosong Li, Fangcheng Liu, Wei Guo, Hang Zhou, Yaoyuan Wang, Kai Han, Xianzhi Yu, Jinpeng Li, Hui Zang, Fei Mi, Xiaojun Meng, Zhicheng Liu, Hanting Chen, Binfan Zheng, Can Chen, Youliang Yan, Ruiming Tang, Peifeng Qin, Xinghao Chen, Dacheng Tao, and Yunhe Wang. 2025. Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity. arXiv:2505...

    Pith/arXiv arXiv 2025

  44. [44]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv. org/abs/2302.13971

    Pith/arXiv arXiv 2023

  45. [45]

    Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 [cs.CL] https://arxiv.org/abs/ 2307.09288

    Pith/arXiv arXiv 2023

  46. [46]

    Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin’ichi Satoh. 2017. Embedding Watermarks into Deep Neural Networks. InProceedings of the 2017 ACM on International Conference on Multimedia Retrieval(Bucharest, Romania) (ICMR ’17). Association for Computing Machinery, New York, NY, USA, 269–277. doi:10.1145/3078971.3078974

    work page doi:10.1145/3078971.3078974 2017

  47. [47]

    Suqing Wang, Ziyang Ma, Xinyi Li, and Zuchao Li. 2025. Ghost in the Transformer: Tracing LLM Lineage with SVD-Fingerprint

    2025

  48. [48]

    Tianhao Wang and Florian Kerschbaum. 2019. Attacks on Digital Watermarks for Deep Neural Networks. In2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Brighton, UK, 2622–2626. doi:10. 1109/ICASSP.2019.8682202

    arXiv 2019

  49. [49]

    Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023. LLMDet: A Third Party Large Language Models Generated Text Detection Tool. InFindings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Lin- guistics, Singapore, 2113–2133. doi:10.18653/v1/20...

    doi:10.18653/v1/2023.findings-emnlp.139 2023

  50. [50]

    Zhaomin Wu, Haodong Zhao, Ziyang Wang, Jizhou Guo, Qian Wang, and Bing- sheng He. 2025. LLM DNA: Tracing Model Evolution via Functional Representa- tions

    2025

  51. [51]

    Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. 2021. Taxonomizing local versus global structure in neural network loss landscapes. InAdvances in Neural Information Processing Systems, Vol. 34. Curran Associates, Inc., Virtual, 18722– 18733

    2021

  52. [52]

    Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. 2023. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, Long Beach, CA, USA, 3011–30...

    2023

  53. [53]

    Zhiguang Yang and Hanzhou Wu. 2024. A Fingerprint for Large Language Models. https://api.semanticscholar.org/CorpusID:270869924

    2024

  54. [54]

    Do-hyeon Yoon, Minsoo Chun, Thomas Allen, Hans Müller, Min Wang, and Rajesh Sharma. 2025. Intrinsic Fingerprint of LLMs: Continue Training is NOT All You Need to Steal A Model!

    2025

  55. [55]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence?. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Ko- rhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 4791–4800. doi:10...

    doi:10.18653/v1/p19-1472 2019

  56. [56]

    Boyi Zeng, Lin Chen, Ziwei He, Xinbing Wang, and Zhouhan Lin. 2025. AWM: Accurate Weight-Matrix Fingerprint for Large Language Models

    2025

  57. [57]

    Boyi Zeng, Lizheng Wang, Yuncong Hu, Yi Xu, Chenghu Zhou, Xinbing Wang, Yu Yu, and Zhouhan Lin. 2024. HuRef: HUman-REadable Fingerprint for Large Language Models. InAdvances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., Vancouver, Canada, 126332–126362

    2024

  58. [58]

    Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. 2025. REEF: Representation Encoding Fingerprints for Large Language Models. InInternational Conference on Learning Representations. OpenReview.net, Singapore, 48092–48117

    2025

  59. [59]

    Yefan Zhou, Tianyu Pang, Keqin Liu, Michael W Mahoney, Yaoqing Yang, et al

  60. [60]

    Temperature balancing, layer-wise weight analysis, and neural network training.Advances in Neural Information Processing Systems36 (2023), 63542– 63572

    2023

  61. [61]

    Ahmed, Rohith Kuditipudi, and Percy Liang

    Sally Zhu, Ahmed M. Ahmed, Rohith Kuditipudi, and Percy Liang. 2025. Inde- pendence Tests for Language Models. InProceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research). PMLR, Vancouver, Canada, 79673–79698

    2025

  62. [62]

    locks in

    Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. 2024. EmbedLLM: Learning Compact Representations of Large Language Models. arXiv:2410.02223 [cs.CL] https://arxiv.org/abs/2410.02223 7 Appendix 7.1 Robustness of ESD Shape Metrics Pre-training and post-training reshape the correlations within weight matrices, as refl...

    Pith/arXiv arXiv 2024