Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

HessFormer: Hessians at Foundation Scale

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HessFormer brings Hessian spectral estimation to 70-billion-parameter models by distributing Lanczos quadrature across eight GPUs.

desk verdict The engineering is real and the small-scale validation is convincing, but the 70B negative-outlier claim is not yet separated from Lanczos ghosts. read the letter →

arxiv 2505.11564 v1 pith:KITSL3RI submitted 2025-05-16 cs.LG stat.ML

Diego Granziol This is my paper · ORCID
classification cs.LGstat.ML
keywords distributedHessian-vectorproductsstochasticLanczosquadratureHessianspectraldensityfoundationmodelslargelanguageeigenvalueoutliersmodelcompressioninfluencefunctions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces HessFormer, a software framework that computes Hessian-vector products and Hessian spectral densities for very large language models across multiple GPUs without requiring users to rewrite model code. It claims to be the first such framework that works with standard PyTorch and Huggingface Transformers pipelines, using layerwise device placement so a second backward pass can flow through the distributed graph. The authors use it to report, for what they say is the first time, spectral density plots of the Hessian of a 70-billion-parameter DeepSeek-Llama distilled model, finding large negative eigenvalues that persist as the data subsample becomes less aggressive. The paper frames this as closing a tooling gap between million-parameter theory and billion-parameter practice, opening curvature analysis to influence functions, unlearning, compression, and optimiser design at foundation scale.

What carries the argument

The load-bearing object is the distributed Hessian-vector product, computed by Pearlmutter's double-backward trick on a model split across GPUs with Huggingface's inference-style automatic device placement, which preserves the autograd graph that FSDP-style sharding severs. On top of this sits a Lanczos iteration without reorthogonalization, plus stochastic trace estimation over a few random starting vectors, to produce a moment-matched spectral density from the tridiagonal Ritz values and weights. Because orthogonality is lost quickly in finite precision, the method deliberately caps iterations at about 10; the paper's numerical analysis, using a standard floating-point error model, argues the squared Ritz-vector weights still retain roughly six accurate decimal digits at that iteration count.

What would settle it

Run the 70B spectral estimate with several independent random starting vectors at the same 10-iteration budget: if the large negative outliers do not reproduce across seeds, or if a full-reorthogonalization control at smaller scale shows the same outliers vanishing when orthogonality is enforced, the central empirical claim is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that Hessian information is not fundamentally out of reach for models with tens of billions of parameters. By keeping gradients in graph form instead of letting sharded-data-parallel hooks destroy them, HessFormer runs Pearlmutter's double-backward trick across GPUs, combined with a distributed stochastic Lanczos quadrature estimator, and obtains spectra for DeepSeek-LLM-70B on a single eight-GPU node. The spectra show large negative eigenvalues, which the paper interprets as directions in the loss landscape where descent would decrease loss; it reports that these outliers persist under less aggressive subsampling. The paper also reports near-total sparsity of Hessian columns within float32 precision on 12B and 32B models, suggesting structured curvature that could be banded within machine precision.

Load-bearing premise

That ten Lanczos iterations without reorthogonalization, starting from a single random vector in float32, faithfully represent the 70B Hessian spectrum, so the large negative eigenvalues are real curvature rather than numerical ghosts.

Editorial extensions

If this is right

  • Practitioners can now estimate hundreds of leading Hessian eigenvalues of models beyond 7B parameters on a single multi-GPU node, removing the single-GPU memory ceiling that previously capped empirical curvature studies.
  • The measured spectra provide the first empirical check on whether diagonal, block-diagonal, Kronecker, or heavy-tailed Hessian assumptions survive at foundation scale.
  • If the negative outliers are genuine, trained frontier models retain directions of descending loss, with consequences for fine-tuning stability, flatness, and mode connectivity.
  • Near-machine-precision sparsity of Hessian columns means structured approximations and banded or sparse curvature solvers may be viable for very large models.
  • Distributed Hessian-vector products make inverse-Hessian-vector-product techniques such as influence functions and unlearning available at the 10-100B parameter scale, at least within the memory budget of an 8-GPU H200 node.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: A direct test of the negative-outlier claim would be to descend along the corresponding Ritz vectors and measure whether the loss actually decreases; if it does not, the outliers are numerical artifacts rather than real landscape structure.
  • Inference: The same distributed Hessian-vector-product primitive could be adapted to compute Hessian diagonals or (H+lambda*I)^(-1) v solves for unlearning and data valuation at scale, which the paper motivates but does not demonstrate.
  • Inference: If near-sparsity of Hessian columns is generic, random-column sampling could map the band structure of large-model Hessians without O(P^2) cost, connecting directly to randomized numerical linear algebra.
  • Inference: The negative outliers might be a signature of distillation, since a distilled student model can retain residual curvature toward its teacher; the paper does not compare against a non-distilled 70B model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents HessFormer, a software framework that computes distributed Hessian-vector products for PyTorch/HuggingFace models via double backward through device_map='auto', combined with stochastic Lanczos quadrature. The authors validate the distributed setup against single-GPU runs on a 1.5B model, report nearly identical loss curves and similar spectra, and then apply the method to DeepSeek-LLM-70B, reporting Hessian spectral densities with large negative Ritz values that they interpret as negative-curvature outliers. The paper also reports Hessian-column sparsity statistics and a compute accounting for the experiments.

Significance. If the software claim is correct, HessFormer is a useful engineering contribution: it makes distributed HVPs accessible without user code modification and provides the first public spectral estimates at 70B scale. The paper ships a concrete package, gives honest compute accounting, and includes a small-scale validation showing close agreement between auto and single-GPU setups (Figures 1 and 2). However, the central scientific observation, the negative outliers in the 70B spectrum, is not yet separated from Lanczos ghosts; the paper itself documents ghost formation without orthogonalization, and the 70B runs use exactly the regime where that risk is highest.

major comments (4)
  1. [Section 4, Figure 7; Section 3.2, Figure 3d] The headline 70B claim is under-supported. Section 4 and Table 2 use 10 Lanczos iterations without reorthogonalization on a single random vector in float32, while Section 3.2 (Figure 3d) shows that this no-orthogonalization regime produces ghost eigenvalues within 25 iterations on a 1.5B model. The paper's statement that fewer than 20 iterations are safe is not demonstrated, and the negative Ritz values in Table 2 have weights between 1e-11 and 1e-13, which is the small-weight regime where numerical ghosts are expected. Without a full-orthogonalization baseline, a second seed, or an independent moment check, the negative outliers in Figure 7 cannot be attributed to true Hessian structure.
  2. [Section 3.4] The error analysis in Section 3.4 does not establish the reliability of the Ritz pairs. The bound |bvi - vi| <= ku|vi| concerns Ritz-vector components, and the relative-error bound on bwi is a bound on squared components, not on the Ritz values or the weights used in the quadrature sum. The text explicitly defers a rigorous Lanczos error analysis to future work, so the manuscript provides neither a mathematical nor an empirical control against the ghost mechanism it documents in Section 3.2.
  3. [Section 4, Figure 7] The robustness claim for the negative outliers is not quantified. Figure 7 varies dataset and subsampling fraction across panels without error bars, multiple seeds, or a matched-eigenvalue comparison; the text's conclusion that the trend 'persists' is therefore not supported by the presented evidence. A quantitative stability measure, such as overlap of large Ritz values across fractions and seeds, is needed.
  4. [Section 3.1] The validation setup is internally inconsistent: the text states that device='auto' is inference-only and does not support training, yet it then describes training comparisons with auto and reports differences in training and validation losses in Figures 1a-b. Please clarify whether the comparison used auto for training or only for inference/HVPs, and adjust the claims and wording accordingly.
minor comments (5)
  1. [Section 3.2] The phrase 'Pearlmutters trick' should be 'Pearlmutter's trick' (Pearlmutter, 1994).
  2. [Section 5] The reference 'as shown in the extensive Tables ??' is unresolved; point to the specific tables in Section 4.1 or elsewhere.
  3. [Section 4] The dataset labels 'realnewslike (1%)' in the text and '1% RealNews' in Figure 7d should be made consistent.
  4. [Throughout] Numerous typos should be corrected, including 'inteded', 'atpyical', 'none the less', 'identitcal', and 'inherrently'.
  5. [Algorithm 2] State the distribution of the starting vector q0 and specify whether H(qk) is the exact Hessian-vector product or a stochastic/batched approximation, since this affects interpretation of the resulting Ritz values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical software contribution whose spectral estimates are outputs, not fitted inputs.

full rationale

The paper's central claims are (1) a distributed stochastic Lanczos quadrature framework for Hessian-vector products on PyTorch/HuggingFace models and (2) spectral density estimates for a 70B model. Neither claim reduces to its inputs by construction. The method is validated against independent baselines: single-GPU training loss differences at the 1e-6 level, spectral stem comparisons between device='auto' and single-GPU execution, and a full-orthogonalization Lanczos baseline versus the no-reorthogonalization variant. These are benchmark controls, not fitted parameters renamed as predictions. The 70B negative-outlier finding is an output of the measurement pipeline; its weakness is numerical soundness (few Lanczos iterations, one seed, float32, no multi-seed control), which is a correctness/reliability concern, not circularity. The only self-citation in the paper (Granziol et al. 2022, used when comparing outlier behavior on vision models) is illustrative and not load-bearing for the central derivation. No equation is defined in terms of the quantity it is said to predict, no fitted constant is relabelled as a discovery, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. The paper is therefore self-contained as an empirical study, and the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No free constants are fitted to produce the spectral plots, but the resolution and stability of the estimates depend on hand-chosen Lanczos iteration counts, a single fixed seed, and chosen data subsampling fractions. The derivation assumes standard Lanczos error bounds are pessimistic and relies on unverified self-averaging for single-seed estimates. No new physical entities are introduced.

free parameters (3)
  • Lanczos iteration count m = 10 for 70B runs; 20 to 30 for smaller models
    Chosen by hand to balance spectral resolution against ghost eigenvalues from lost orthogonality. With m=10 only 10 moments of the spectral density are matched, so the 70B spectrum is a low-resolution estimate and the negative outliers could be Lanczos artifacts.
  • Random starting vector seed = seed = 42, single draw
    The Lanczos initial vector is drawn once with a fixed seed and no seed averaging. The conclusion explicitly says no multiple-seed experiments were run, so the stability of the 70B spectrum across random vectors is untested.
  • Dataset subsampling fractions = Wikitext 5% and 25%, RealNews 0.1% and 1%, Pile 1% and 100%
    Subsample sizes are chosen by compute budget, not by a stated criterion, and the paper claims robustness to subsampling without quantifying variance across subsamples. These choices affect whether the large negative eigenvalues persist.
assumptions (3)
  • domain assumption HuggingFace auto device placement preserves the autograd graph so a double backward pass computes exact Hessian-vector products across devices.
    Section 3.1 uses this as the basis for Algorithm 1. The validation is done only on 1.5B models, and the text notes auto is inference-only and leaves n-1 GPUs idle, which is why it is not standard for training.
  • standard math The Lanczos recurrence in float32 without reorthogonalization still produces a useful moment-matched approximation of the true Hessian spectrum.
    Paige's error analysis is cited to bound loss of orthogonality, but the authors then decide those bounds are too conservative and make an 'artistic decision' not to correct the float32 numerical plots. This assumption is load-bearing for interpreting the 70B spectrum.
  • domain assumption Self-averaging of large random vectors makes a single-seed Lanczos estimate representative at 70B scale.
    The conclusion states that because of the self-averaging property of large random vectors and matrices, the lack of seed averaging should not have major impact. No empirical evidence is provided for Hessian spectral densities of LLMs, so this is an unverified assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HessFormer: Hessians at Foundation Scale." pith.science (2026). https://pith.science/paper/KITSL3RI

@misc{pith2026250511564,
  author       = {Pith},
  title        = {Pith review of: HessFormer: Hessians at Foundation Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KITSL3RI}},
  note         = {Machine review of arXiv:2505.11564}
}
abstract

Whilst there have been major advancements in the field of first order optimisation of deep learning models, where state of the art open source mixture of expert models go into the hundreds of billions of parameters, methods that rely on Hessian vector products, are still limited to run on a single GPU and thus cannot even work for models in the billion parameter range. We release a software package \textbf{HessFormer}, which integrates nicely with the well known Transformers package and allows for distributed hessian vector computation across a single node with multiple GPUs. Underpinning our implementation is a distributed stochastic lanczos quadrature algorithm, which we release for public consumption. Using this package we investigate the Hessian spectral density of the recent Deepseek $70$bn parameter model.

Figures

Figures reproduced from arXiv: 2505.11564 by the authors.

Figure 1
Figure 1. Comparison between Training a 1.5Bn parameter Qwen distilled Deepseek model on wikitrain for 4 epochs using auto from huggingface on multiple GPUs against using a single large enough GPU training on cuda. further plot the differences in spectra when using auto or not. As shown in Figure 1c, for a smoothed density, we see barely perceptible difference. Note that we are only using a single n = 1 seed vector. So we exp… view at source ↗
Figure 2
Figure 2. Stem comparison of using Auto (blue) vs. not using Auto (red) for Hessian of the DeepSeek [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. DeepSeek Qwen 1.5 Bn: spectral effects of data subsampling and orthogonalization on the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Random Data with Various (n=num_samples, l=seq_length) Hessian with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Real News Data (subsampled to 1%) with various amounts of further subsampling Hessia for Qwen 1.5Bn parameter DeepSeek distilled 100 0 100 200 300 Eigenvalues of 12b Model 10 10 10 8 10 6 10 4 10 2 10 0 min = 1.34e + 02 max = 3.10e + 02 (a) 1% Pile 0 50 100 150 200 250…
Figure 6
Figure 6. Figure 6: Pythia 12B (20 Lanczos iterations) spectral widths for the Pile and Wikipedia subsets. The [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: 70 Bn DeepSeek Llama distillation: spectral widths under varying subsampling of Wikitext and RealNews. Given the ubiquity of the Pile dataset and the scraping wars to get as much data as possible to train large language models, a question of interest may be as to wheth…
Figure 8
Figure 8. Figure 8: 50-bin absolute-magnitude histograms of Hessian-column elements for DeepSeek 32B (1 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Defense of the Quadratic Model

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373,

    arXiv

  2. [8]

    Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Sijia Liu, Bharat Runwal, James Diffender- fer, and Bhavya Kailkhura

    doi: 10.14778/3611540.3611569. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Sijia Liu, Bharat Runwal, James Diffender- fer, and Bhavya Kailkhura. Soul: Unlocking the power of second-order optimization for llm un- learning. In EMNLP 2024,

    arXiv

  3. [11]

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma

    URL https://arxiv.org/abs/2203.07259. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342,

    arXiv

  4. [12]

    James Martens

    URL https://arxiv.org/abs/2305.14342. James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning,

    arXiv

  5. [17]

    Youran Sun and Babak Haghighat

    URL https://arxiv.org/abs/2004.14340. Youran Sun and Babak Haghighat. Phase transitions in large language models and theo(n) model. arXiv preprint arXiv:2501.16241,

    arXiv 2004

  6. [18]

    On the power-law hessian spectrums in deep learning

    Zeke Xie, Qian-Yuan Tang, Yunfeng Cai, Mingming Sun, and Ping Li. On the power-law hessian spectrums in deep learning. arXiv preprint arXiv:2201.13011,

    arXiv

  7. [19]

    Gspmd: General and scalable parallelization for ML computation graphs

    Sheng Xu, Noam Shazeer, Dehao Chen, et al. Gspmd: General and scalable parallelization for ML computation graphs. arXiv preprint arXiv:2105.13290,

    arXiv

  8. [20]

    Adadelta: an adaptive learning rate method

    Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701,

    arXiv

Show all 21 references
  1. [23]

    URL https://arxiv.org/abs/2306.17835. 11

    arXiv

  2. [1976]

    Barak A Pearlmutter

    doi: 10.1093/imamat/18.3.341. Barak A Pearlmutter. Fast exact multiplication by the Hessian. Neural computation, 6(1):147–160,

    doi

  3. [1994]

    A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization

    Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. arXiv preprint arXiv:2012.03801,

    2012 arXiv

  4. [2002]

    doi: 10.1137/1

    ISBN 9780898715217. doi: 10.1137/1. 9780898718027. Yanli Huang, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, and Zachary DeVito. Pytorch fsdp: Experiences on scaling fully sharded data par...

    doi

  5. [2010]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,

    arXiv

  6. [2012]

    Why trans- formers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why trans- formers need adam: A hessian perspective. arXiv preprint arXiv:2402.16788,

    arXiv

  7. [2017]

    The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models

    Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference on Empirical Methods in ...

    2022

  8. [2020]

    What does it mean to be a transformer? insights from a theoretical hessian analysis

    Weronika Ormaniec, Felix Dangel, and Sidak Pal Singh. What does it mean to be a transformer? insights from a theoretical hessian analysis. arXiv preprint arXiv:2410.10986,

    arXiv

  9. [2021]

    URL https://arxiv.org/abs/2103.00065. Poster. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    arXiv

  10. [2022]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sidney Black, Leo Golding, Laurence King, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027,

    arXiv

  11. [2023]

    Babak Hassibi and David G

    URL https://arxiv.org/abs/ 2308.03296. Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems ,

    arXiv

  12. [2024]

    Pang Wei Koh and Percy Liang

    URL https://aclanthology.org/2024.emnlp-main.245. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning ,

    2024

  13. [2025]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer

    URL https://arxiv.org/abs/2501.12948. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems ,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.