{"id":"6d4b3e20-3a1a-4a14-aee7-08e28b689f97","arxiv_id":"2505.11564","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"HessFormer delivers distributed Hessian-vector products for 70B-parameter models and reports the first spectral-density plots, including large negative eigenvalues, for DeepSeek-LLM-70B.","lead":"HessFormer is software that lets researchers measure the curvature of the error landscape of a 70-billion-parameter AI model using several GPUs at once. It reports the first published Hessian spectrum at that scale, which matters for tuning optimizers, compressing models, and removing data from trained models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 70B negative-outlier claim is not yet separated from Lanczos ghosts: the protocol uses 10 no-reorthogonalization iterations, one seed, and float32, with no control chain, so the headline empirical result is under-supported.","rationale":"The reader's weakest assumption and my stress-test converge on the same load-bearing concern: the reliability of 10-iteration, non-orthogonalized, single-seed Lanczos spectra at 70B scale. The paper's own ablation work shows that ghost eigenvalues appear without reorthogonalization, and the authors explicitly state that multi-seed experiments were not run. The 70B negative outliers carry very small Ritz weights, exactly the regime where finite-precision Lanczos artifacts appear. This does not mean the outliers are certainly ghosts; it means the current evidence cannot distinguish them from ghosts. The software contribution, by contrast, has independent support in the 1.5B single-GPU versus multi-GPU comparisons, so the appropriate verdict remains conditional rather than rejection. If the proposed multi-seed check fails, the empirical claim should be downgraded or withdrawn, but the tooling claim could still stand with code release and further validation. My read does not change the reader's verdict: CONDITIONAL with requirements for code availability, multi-seed runs, and ghost-control evidence.","tokens_in":11690,"tokens_out":10441,"duration_ms":113148,"concrete_test":"Run ten additional 10-iteration Lanczos chains on DeepSeek-LLM-70B with independently drawn random starting vectors, using the same float32 no-reorthogonalization protocol as Section 4, and record the negative Ritz pairs. If the large negative Ritz values (−8e4, −7e4, …) and their 1e−11-scale weights reappear in at least, say, 7 of 10 chains with values agreeing to within a few percent, the outliers are likely real; if the negative tail is seed-dependent or absent, the claim is falsified as Lanczos ghosts. This is the standard ghost test for Lanczos without reorthogonalization and is directly feasible on the reported 8×H200 node.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim—large negative Hessian eigenvalue outliers in DeepSeek-LLM-70B—requires that 10 finite-precision Lanczos iterations without reorthogonalization, one random starting vector, and float32 arithmetic yield reliable Ritz pairs. The paper does not establish this. Section 3.2 shows that omitting orthogonalization produces ghost eigenvalues on a 1.5B model (Figure 3d, 25 iterations), and the authors only assume that fewer than 20 iterations are safe. Section 4 uses 10 iterations; Table 2 lists negative Ritz values (−8.2e4, −7.2e4, …) with weights near 1e−11 to 1e−13. There is no full-orthogonalization baseline at 70B, no second seed, and no independent moment check. The conclusion concedes that multi-seed runs were not performed, and the self-averaging argument applies to stochastic trace estimates, not to individual Ritz pairs from a single starting vector. If these negative Ritz pairs are Lanczos ghosts—a mechanism the paper itself documents—the central scientific novelty collapses, even though the distributed-HVP tooling claim retains its smaller-scale validation. Section 3.4 also defers a rigorous Lanczos error analysis to future work, so the paper currently has no mathematical or empirical control separating true negative outliers from numerical artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HessFormer, a software framework that computes distributed Hessian-vector products for PyTorch/HuggingFace models via double backward through device_map='auto', combined with stochastic Lanczos quadrature. The authors validate the distributed setup against single-GPU runs on a 1.5B model, report nearly identical loss curves and similar spectra, and then apply the method to DeepSeek-LLM-70B, reporting Hessian spectral densities with large negative Ritz values that they interpret as negative-curvature outliers. The paper also reports Hessian-column sparsity statistics and a compute accounting for the experiments.","tokens_in":11887,"tokens_out":5775,"duration_ms":55883,"significance":"If the software claim is correct, HessFormer is a useful engineering contribution: it makes distributed HVPs accessible without user code modification and provides the first public spectral estimates at 70B scale. The paper ships a concrete package, gives honest compute accounting, and includes a small-scale validation showing close agreement between auto and single-GPU setups (Figures 1 and 2). However, the central scientific observation, the negative outliers in the 70B spectrum, is not yet separated from Lanczos ghosts; the paper itself documents ghost formation without orthogonalization, and the 70B runs use exactly the regime where that risk is highest.","major_comments":[{"comment":"The headline 70B claim is under-supported. Section 4 and Table 2 use 10 Lanczos iterations without reorthogonalization on a single random vector in float32, while Section 3.2 (Figure 3d) shows that this no-orthogonalization regime produces ghost eigenvalues within 25 iterations on a 1.5B model. The paper's statement that fewer than 20 iterations are safe is not demonstrated, and the negative Ritz values in Table 2 have weights between 1e-11 and 1e-13, which is the small-weight regime where numerical ghosts are expected. Without a full-orthogonalization baseline, a second seed, or an independent moment check, the negative outliers in Figure 7 cannot be attributed to true Hessian structure.","section":"Section 4, Figure 7; Section 3.2, Figure 3d"},{"comment":"The error analysis in Section 3.4 does not establish the reliability of the Ritz pairs. The bound |bvi - vi| <= ku|vi| concerns Ritz-vector components, and the relative-error bound on bwi is a bound on squared components, not on the Ritz values or the weights used in the quadrature sum. The text explicitly defers a rigorous Lanczos error analysis to future work, so the manuscript provides neither a mathematical nor an empirical control against the ghost mechanism it documents in Section 3.2.","section":"Section 3.4"},{"comment":"The robustness claim for the negative outliers is not quantified. Figure 7 varies dataset and subsampling fraction across panels without error bars, multiple seeds, or a matched-eigenvalue comparison; the text's conclusion that the trend 'persists' is therefore not supported by the presented evidence. A quantitative stability measure, such as overlap of large Ritz values across fractions and seeds, is needed.","section":"Section 4, Figure 7"},{"comment":"The validation setup is internally inconsistent: the text states that device='auto' is inference-only and does not support training, yet it then describes training comparisons with auto and reports differences in training and validation losses in Figures 1a-b. Please clarify whether the comparison used auto for training or only for inference/HVPs, and adjust the claims and wording accordingly.","section":"Section 3.1"}],"minor_comments":[{"comment":"The phrase 'Pearlmutters trick' should be 'Pearlmutter's trick' (Pearlmutter, 1994).","section":"Section 3.2"},{"comment":"The reference 'as shown in the extensive Tables ??' is unresolved; point to the specific tables in Section 4.1 or elsewhere.","section":"Section 5"},{"comment":"The dataset labels 'realnewslike (1%)' in the text and '1% RealNews' in Figure 7d should be made consistent.","section":"Section 4"},{"comment":"Numerous typos should be corrected, including 'inteded', 'atpyical', 'none the less', 'identitcal', and 'inherrently'.","section":"Throughout"},{"comment":"State the distribution of the starting vector q0 and specify whether H(qk) is the exact Hessian-vector product or a stochastic/batched approximation, since this affects interpretation of the resulting Ritz values.","section":"Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nothing wrong with the engineering here, and the small-scale validation is real. HessFormer is a plausible way to do distributed HVPs on unmodified HuggingFace models, and the 1.5B comparison against single-GPU training is convincing: losses match to ~1e-6 and the spectra overlap except for one ghost the paper itself flags. That part deserves a serious look.\n\nWhat is new is the scale: first reported Hessian spectral density of a 70B transformer. That is a legitimate extension, and the authors are honest about many of the caveats. But the headline empirical finding — large negative outliers in the 70B spectrum — is not yet supported. The protocol uses 10 Lanczos iterations with no reorthogonalization, one seed, and float32. Section 3.2 shows ghosts appear with 25 iterations without orthogonalization on a 1.5B model. Ten iterations is below that threshold, but there is no control: no second seed, no full-orthogonalization baseline at 70B, no independent check on the Ritz pairs. The paper's own Section 3.4 derives a bound on the weights, not on the Ritz values, so it doesn't close the gap. The self-averaging argument in the conclusion applies to stochastic trace estimates, not to individual Ritz pairs from a single starting vector. So the central scientific claim — 'very interesting large negative outliers' — is plausible but not established.\n\nSome smaller soft spots: the code release is mentioned but no link or commit hash appears, which matters for a software paper. The compute accounting is fine. The banded-Hessian section is speculative and the table references are broken (Tables ??), but that is a presentation issue, not a research flaw.\n\nIf the tooling claim is the paper, it's in good shape. If the 70B negatives are the paper, it needs multi-seed runs, a reorthogonalized or otherwise verified baseline at smaller scale, and possibly higher precision. The reader's conditional verdict is right, and my recommendation: send it to review, but flag the numerical control question as the main issue. A good referee could either push the authors to run the missing controls or trim the claim.","headline":"The engineering is real and the small-scale validation is convincing, but the 70B negative-outlier claim is not yet separated from Lanczos ghosts.","tokens_in":12478,"tokens_out":1777,"would_cite":false,"duration_ms":17187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HessFormer brings Hessian spectral estimation to 70-billion-parameter models by distributing Lanczos quadrature across eight GPUs.","keywords":["distributed Hessian-vector products","stochastic Lanczos quadrature","Hessian spectral density","foundation models","large language models","eigenvalue outliers","model compression","influence functions"],"falsifier":"Run the 70B spectral estimate with several independent random starting vectors at the same 10-iteration budget: if the large negative outliers do not reproduce across seeds, or if a full-reorthogonalization control at smaller scale shows the same outliers vanishing when orthogonality is enforced, the central empirical claim is refuted.","tokens_in":11394,"feed_emoji":"📊","tokens_out":7016,"duration_ms":63603,"temperature":0.7,"pith_summary":"This paper introduces HessFormer, a software framework that computes Hessian-vector products and Hessian spectral densities for very large language models across multiple GPUs without requiring users to rewrite model code. It claims to be the first such framework that works with standard PyTorch and Huggingface Transformers pipelines, using layerwise device placement so a second backward pass can flow through the distributed graph. The authors use it to report, for what they say is the first time, spectral density plots of the Hessian of a 70-billion-parameter DeepSeek-Llama distilled model, finding large negative eigenvalues that persist as the data subsample becomes less aggressive. The paper frames this as closing a tooling gap between million-parameter theory and billion-parameter practice, opening curvature analysis to influence functions, unlearning, compression, and optimiser design at foundation scale.","feed_headline":"Distributed Hessians reach 70-billion-parameter LLMs","feed_subtitle":"HessFormer runs Lanczos quadrature across eight GPUs and reports large negative eigenvalues in DeepSeek-70B spectra.","key_machinery":"The load-bearing object is the distributed Hessian-vector product, computed by Pearlmutter's double-backward trick on a model split across GPUs with Huggingface's inference-style automatic device placement, which preserves the autograd graph that FSDP-style sharding severs. On top of this sits a Lanczos iteration without reorthogonalization, plus stochastic trace estimation over a few random starting vectors, to produce a moment-matched spectral density from the tridiagonal Ritz values and weights. Because orthogonality is lost quickly in finite precision, the method deliberately caps iterations at about 10; the paper's numerical analysis, using a standard floating-point error model, argues the squared Ritz-vector weights still retain roughly six accurate decimal digits at that iteration count.","core_discovery":"On the paper's own terms, the central discovery is that Hessian information is not fundamentally out of reach for models with tens of billions of parameters. By keeping gradients in graph form instead of letting sharded-data-parallel hooks destroy them, HessFormer runs Pearlmutter's double-backward trick across GPUs, combined with a distributed stochastic Lanczos quadrature estimator, and obtains spectra for DeepSeek-LLM-70B on a single eight-GPU node. The spectra show large negative eigenvalues, which the paper interprets as directions in the loss landscape where descent would decrease loss; it reports that these outliers persist under less aggressive subsampling. The paper also reports near-total sparsity of Hessian columns within float32 precision on 12B and 32B models, suggesting structured curvature that could be banded within machine precision.","pith_inferences":["Inference: A direct test of the negative-outlier claim would be to descend along the corresponding Ritz vectors and measure whether the loss actually decreases; if it does not, the outliers are numerical artifacts rather than real landscape structure.","Inference: The same distributed Hessian-vector-product primitive could be adapted to compute Hessian diagonals or (H+lambda*I)^(-1) v solves for unlearning and data valuation at scale, which the paper motivates but does not demonstrate.","Inference: If near-sparsity of Hessian columns is generic, random-column sampling could map the band structure of large-model Hessians without O(P^2) cost, connecting directly to randomized numerical linear algebra.","Inference: The negative outliers might be a signature of distillation, since a distilled student model can retain residual curvature toward its teacher; the paper does not compare against a non-distilled 70B model."],"forward_implications":["Practitioners can now estimate hundreds of leading Hessian eigenvalues of models beyond 7B parameters on a single multi-GPU node, removing the single-GPU memory ceiling that previously capped empirical curvature studies.","The measured spectra provide the first empirical check on whether diagonal, block-diagonal, Kronecker, or heavy-tailed Hessian assumptions survive at foundation scale.","If the negative outliers are genuine, trained frontier models retain directions of descending loss, with consequences for fine-tuning stability, flatness, and mode connectivity.","Near-machine-precision sparsity of Hessian columns means structured approximations and banded or sparse curvature solvers may be viable for very large models.","Distributed Hessian-vector products make inverse-Hessian-vector-product techniques such as influence functions and unlearning available at the 10-100B parameter scale, at least within the memory budget of an 8-GPU H200 node."],"supporting_citations":[{"why":"Supplies the double-backward trick that computes Hessian-vector products without forming the Hessian.","marker":"[Pearlmutter 1994]"},{"why":"Documents that FSDP's gradient hooks sever the autograd graph, the obstacle HessFormer's automatic device placement avoids.","marker":"[Huang et al., 2023]"},{"why":"Releases the DeepSeek-LLM-70B model whose Hessian spectrum the paper estimates.","marker":"[DeepSeek-AI 2025]"},{"why":"Provides the loss-of-orthogonality bound used to justify capping Lanczos iterations.","marker":"[Paige 1976]"},{"why":"Supplies the floating-point error model used to bound the accuracy of squared Ritz-vector weights.","marker":"[Higham 2002]"},{"why":"Motivates the search for extreme outlier structure beyond 6B parameters.","marker":"[Dettmers et al., 2022]"},{"why":"Provides Pythia models with known training data for the subsampling and negative-eigenvalue comparison.","marker":"[Biderman et al., 2023]"},{"why":"Provides the Pile dataset used to test the model on its training distribution.","marker":"[Gao et al., 2020]"},{"why":"Provides the WikiText dataset used in the spectral and ablation experiments.","marker":"[Merity et al., 2016]"}],"fun_headline_variants":["Distributed Hessians reach 70B models on one node","HessFormer computes Hessians for 70B-parameter LLMs","Lanczos quadrature scales Hessians to DeepSeek-70B","Hessian spectra of DeepSeek-70B from 8 GPUs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That ten Lanczos iterations without reorthogonalization, starting from a single random vector in float32, faithfully represent the 70B Hessian spectrum, so the large negative eigenvalues are real curvature rather than numerical ghosts.","fun_headline_variants_meta":{"raw":{"variants":["Distributed Hessians reach 70B models on one node","HessFormer computes Hessians for 70B-parameter LLMs","Lanczos quadrature scales Hessians to DeepSeek-70B","Hessian spectra of DeepSeek-70B from 8 GPUs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000504,"raw_usage":{"total_tokens":2404,"prompt_tokens":833,"completion_tokens":1571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":1500}},"tokens_in":449,"tokens_out":1571,"duration_ms":11295,"temperature":1.0,"reasoning_tokens":1500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:59:53.344771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 70B spectral estimate with several independent random starting vectors at the same 10-iteration budget: if the large negative outliers do not reproduce across seeds, or if a full-reorthogonalization control at smaller scale shows the same outliers vanishing when orthogonality is enforced, the central empirical claim is refuted.","supporting_citations":[],"review_version":1}