Pith. sign in

REVIEW 3 major objections 5 minor 12 references

Visualizing Loss Functions as Topological Landscape Profiles

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that compressing a loss function's minima and saddle points into a two-dimensional topological landscape profile makes 3D and 4D loss landscapes visible and reveals that better-performing models have simpler…

desk verdict Useful visualization method with a suggestive but under-supported empirical claim; the kNN hyperparameter deserves sensitivity analysis before the 'simpler topology for better models' conclusion is accepted. read the letter →

arxiv 2411.12136 v1 pith:VI7OC5S5 submitted 2024-11-19 cs.LG cs.AI

Caleb Geniesse , Jiaqing Chen , Tiankai Xie , Ge Shi , Yaoqing Yang , Dmitriy Morozov , Talita Perciano , Michael W. Mahoney
show 2 more authors
Ross Maciejewski Gunther H. Weber
This is my paper · ORCID
classification cs.LGcs.AI
keywords topologicallandscapeprofilelossmergetreedataanalysisHessianeigenvectorsphysics-informedneuralnetworksmodelperformancediagnosis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a two-dimensional topological landscape profile built from the merge tree of a loss function, and argues that this representation makes three- and four-dimensional Hessian-based loss landscapes visible in a way that one- or two-direction slices cannot. It applies the profile to physics-informed neural networks solving a convection problem and to a segmentation network with a CRF layer, and reports two qualitative findings: loss landscapes are topologically simpler for better-performing models, and landscape shape varies most near transitions between low and high performance. If these findings hold, the loss landscape becomes a practical visual diagnostic for model quality and training dynamics rather than an object that can only be inspected in one or two random directions.

What carries the argument

The load-bearing construction is the merge tree computed from loss values sampled on a grid of resolution $r=41$ spanning the top $n$ Hessian eigenvectors, with connectivity defined by a symmetric $k$-nearest-neighbor graph using $k=4n$. As the loss threshold rises, the merge tree records when connected components of sublevel sets appear at local minima and when they merge at saddle points, so each branch ending in a minimum becomes a basin in the resulting topological landscape profile. Each basin is drawn as a stack of rectangles whose cumulative width encodes the number of sampled points along the corresponding merge-tree branch, with basins colored by average loss and annotated by their minima and saddle points. The profile construction adapts an earlier landscape representation for high-dimensional point clouds from density hills to loss basins.

What would settle it

Resample the same models on a finer grid, say $r=81$ or $r=121$, and with different values of $k$, then recompute the merge tree and landscape profiles; if the ranking of better versus worse models by topological simplicity changes, or if spurious basins appear and vanish with resolution, the reported simplicity claim is an artifact of the discretization rather than a property of the loss landscape. A complementary check is to train models with matched training loss but different test accuracy and see whether the simpler-profile association follows test performance.

Watch

Extended reading notes

Core claim

The paper's central claim is that the critical-point structure of an n-dimensional loss landscape—its local minima, the saddle points connecting them, and the loss thresholds at which basins merge—can be encoded in a merge tree and redrawn as a two-dimensional landscape of basins, and that this representation is faithful enough to expose properties of the underlying model. In the examples studied, models with lower error or higher test accuracy show simpler topology: fewer and more funnel-like basins, with the low-loss region dominating the profile. Near the hyperparameter settings where a model crosses from performing well to performing poorly, the paper reports the largest variation in landscape shape across random seeds, with some runs showing funnel-like profiles and others bowl-like ones. The paper also reports that moving from three sampled directions to four preserves the global shape while adding finer spiky structure, which it presents as evidence that the profile captures information beyond two-dimensional slices.

Load-bearing premise

The argument depends on the assumption that sampling the loss at 41 evenly spaced points along each direction and connecting nearby samples with $k=4n$ neighbors faithfully captures the true shape of the loss function rather than creating artificial bumps and valleys.

Editorial extensions

If this is right

  • Landscape profiles computed in three or four Hessian directions give a visual summary that can be compared across models, random seeds, and hyperparameter settings without collapsing the landscape to one or two dimensions.
  • If the reported pattern is general, better-performing models should consistently show fewer dominant basins and more funnel-like profiles in the subspaces spanned by their top Hessian eigenvectors.
  • At hyperparameter values where performance begins to degrade, landscape shape across random seeds should be less consistent, reflecting a mixture of funnel-like and bowl-like profiles.
  • Because the profile is annotated with the average loss of each basin, it encodes optimization state as well as topology, making the transition from a shallow high-loss basin to a deep stable basin visible over training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's examples, the same representation could be used as a live training monitor: the observed deepening and flattening of the main basin over epochs suggests that a scalar summary such as basin count or basin depth could track learning progress.
  • The merge-tree construction does not depend on Hessian directions, so the profile could equally summarize random or filter-normalized subspaces; testing the simplicity claim across subspace choices would show whether the finding is tied to Hessian geometry.
  • A natural experiment the paper does not run is comparing models with equal training loss but different test accuracy; if the simpler-topology pattern tracks test performance, the profile would be a generalization diagnostic rather than just an optimization diagnostic.
  • The resolution dependence is easy to probe: recomputing one PINN profile at $r=81$ and comparing it with $r=41$ would quantify how much of the reported basin structure is an artifact of the grid.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a topological landscape profile representation for loss landscapes, built on a merge tree computed over a k-nearest-neighbor graph of a grid-sampled loss function in the subspace spanned by the top Hessian eigenvectors. The authors claim this representation enables visualization of three- and four-dimensional loss landscapes, and they report two empirical findings from PINN and UNet models: (1) better-performing models have topologically simpler loss landscapes (e.g., funnel-like versus bowl-like), and (2) there is greater landscape-shape variability near transitions from low to high model performance. The method is a standard TDA pipeline (sample grid, construct neighborhood graph, compute merge tree, render landscape profile), adapted from Oesterling et al. (2013) with a valley-oriented interpretation.

Significance. If validated, the proposed representation would be a useful diagnostic tool for comparing higher-dimensional loss landscapes, going beyond one- and two-dimensional slices. A clear strength is that the paper uses established TDA machinery (merge trees, landscape profiles) on genuinely trained models (PINNs and UNets), and it identifies concrete, falsifiable-looking qualitative contrasts between low- and high-error regimes. However, the central empirical claims are not yet supported at the level expected for a rigorous contribution: 'simpler topology' is never measured, the graph-construction choice is not validated with sensitivity analysis, and the second use case (UNet training) does not actually use the proposed profile representation. The paper also does not provide code or data, which limits reproducibility. With quantitative simplicity metrics, a sensitivity analysis of k and grid resolution, and a demonstration of the actual profile on the UNet setting, the contribution could be solid; in its current form the evidence is mostly visual and the claims outrun the analysis.

major comments (3)
  1. [3.1] The merge tree is computed on a symmetric k-nearest-neighbor graph with k=4n (k=12 for the 3D PINN landscapes, k=16 for 4D). On a regular 3D grid the full cubical neighborhood has 26 neighbors, so k=12 omits most of the diagonal near-neighbor edges. Merge-tree critical points and component merges are defined by graph adjacency; omitted edges can create spurious local minima and postpone legitimate merges, directly changing the resulting landscape profile. Since the 'simpler topology for better models' finding is drawn from these profiles, the result may be an artifact of the graph-construction choice rather than a property of the loss function. The paper reports no sensitivity analysis over k or the grid resolution r=41, and provides no code or data to rule out this alternative explanation. Please validate the pipeline using, for example, the full cubical complex or a range of k values, and report how the profiles and the qualitative conclusions change.
  2. [Abstract; Section 5] The central claim that 'the topology of the loss landscape is simpler for better-performing models' is not operationalized. 'Simpler' is never defined or measured; the paper provides no quantity such as number of persistent minima, total persistence, merge-tree leaf count, or any statistical comparison across random seeds and hyperparameters. The visual evidence in Figure 3 is suggestive but does not support a general conclusion absent a quantitative summary. Please define a simplicity metric, compute it across the hyperparameter grid, and test whether differences between low- and high-error regimes are statistically significant.
  3. [4.2; Figure 4] The UNet study is presented as a use case for the proposed method, but Section 4.2 states that two-dimensional loss landscapes were computed using the top two Hessian eigenvectors, and Figure 4 shows standard 2D loss heat maps, not topological landscape profiles. Consequently, this section does not demonstrate the new representation's ability to visualize higher-dimensional landscapes or to extract topology information for this model. Please either apply the proposed profile construction to the UNet setting (even at 2D or 3D resolution) to show the versatility claim, or explicitly reframe Section 4.2 as a baseline comparison using classical visualization.
minor comments (5)
  1. [3.1] The text states that each α_i is an equally spaced integer between 0 and r, with r=41 and the center corresponding to the original model, i.e., Σ α_i δ_i = 0. If α_i ranges from 0 to 41, the center is 20.5, not 0; please clarify the coordinate centering (e.g., α_i ∈ [−20, 20] or a shifted indexing).
  2. [3.1] Since the samples lie on a regular grid, the connectivity is already known; the use of an approximate k-NN graph and its symmetric pruning is unusual. Please justify why a direct cubical complex or structured grid adjacency was not used, and note that the algorithm of Dong et al. (2011) is approximate, introducing another potential source of variability.
  3. [4.1] The discussion of Figure 3 states that landscapes are 'stable across seeds' for low and high β, but this is asserted from five visualizations with no quantitative measure of similarity. Consider reporting a simple metric (e.g., persistence, branch count, or profile distance) across seeds to substantiate the stability claim.
  4. [5] The concluding sentence says 'our approach opens up the door to visualizing higher-dimensional loss landscapes,' but the experiments stop at four dimensions and the sampling cost is acknowledged as a limitation. This is fine, but the wording could be softened to avoid overclaiming practical scalability.
  5. [Global] The paper does not include a data or code availability statement. Given the empirical and visual nature of the work, making the sampling and topological pipeline available would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the topological landscape profiles and the qualitative simplicity/variation findings are empirical observations from a merge-tree representation, not quantities forced by construction or by self-citation.

full rationale

The paper's derivation chain is: sample a Hessian-based n-dimensional subspace, build a k-nearest-neighbor grid graph, compute a merge tree with an external TDA library, and draw a landscape profile from that merge tree. The central claims—'the topology of the loss landscape is simpler for better-performing models' and 'greater variation in the shape of loss landscapes near transitions from low to high model performance'—are correlations between the extracted merge-tree structure and measured error/accuracy. Neither claim is defined in terms of the other, and no parameter is fitted to the reported outcome and then renamed a prediction: k = 4n and r = 41 are fixed a priori, not tuned to reproduce the simplicity or variation findings. The landscape-profile construction is credited to Oesterling et al. (2013), a prior publication, and the merge tree computation uses the external Topology ToolKit; these are independent methodological inputs rather than unverified self-citations. The paper's reference to the authors' previous TDA work (Xie et al., 2024) is contextual and not load-bearing for the new representation or the qualitative conclusions. The main methodological vulnerability—that a sparse symmetric k-NN graph with k = 4n may not faithfully represent the sampled grid's topology, with no sensitivity analysis over k or r—is a correctness and sampling concern, not a circularity: even if the graph introduces spurious critical points, the reported correlations would be artifacts of the approximation rather than equivalences forced by definition. Under the review rule, graph-fidelity concerns belong to correctness risk, not to circularity. Therefore no circular step is exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the adequacy of the sampling and graph construction for representing the true loss landscape topology, as well as on the choice of Hessian directions. These are not derived or benchmarked, so the conclusions inherit their uncertainty.

free parameters (4)
  • k (kNN graph neighborhood size) = 4n (12 for 3D, 16 for 4D)
    Chosen heuristically to mimic pixel adjacency; no sensitivity analysis. Entered in Section 3.1.
  • r (grid resolution per dimension) = 41
    Fixed grid resolution; no sensitivity analysis. Entered in Section 3.1.
  • n (subspace dimension) = 3 or 4
    Limit chosen due to sampling cost; claims about higher-dimensional landscapes are only for these dimensions. Entered in Section 3.1.
  • perturbation distance (UNet) = 0.01
    Used for layerwise normalization when perturbing the model; entered in Section 4.2.
assumptions (3)
  • domain assumption The merge tree of the kNN graph approximates the topology of the continuous loss landscape.
    The paper computes the merge tree on a finite graph (Section 3.2) but draws conclusions about the true loss landscape topology.
  • domain assumption Top Hessian eigenvectors are the most informative directions for understanding loss landscape geometry.
    The paper limits sampling to the subspace spanned by the top n Hessian eigenvectors (Section 3.1), following prior work (Yao et al. 2020, Li et al. 2018).
  • ad hoc to paper The grid resolution r=41 and neighborhood size k=4n are sufficient to resolve the relevant topological features.
    These values are chosen without a convergence study; the central claim depends on them being adequate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Visualizing Loss Functions as Topological Landscape Profiles." pith.science (2026). https://pith.science/paper/VI7OC5S5

@misc{pith2026241112136,
  author       = {Pith},
  title        = {Pith review of: Visualizing Loss Functions as Topological Landscape Profiles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VI7OC5S5}},
  note         = {Machine review of arXiv:2411.12136}
}
read the original abstract

In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. For neural network models, visualizing how this loss changes as model parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) as well as global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher-dimensional loss landscapes. After describing this new topological landscape profile representation, we show how the shape of loss landscapes can reveal new details about model performance and learning dynamics, highlighting several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks). Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces: we find that the topology of the loss landscape is simpler for better-performing models; and we observe greater variation in the shape of loss landscapes near transitions from low to high model performance.

Figures

Figures reproduced from arXiv: 2411.12136 by the authors.

Figure 1
Figure 1. Our topological landscape profiles enable the visualization of higher-dimensional loss landscapes by capturing their underlying shape (or topology). Here we show loss landscapes based on the top n Hessian eigenvectors. See Section 3 for details. significant local loss fluctuations for a given model. Given the n orthogonal directions, we generalize the approach taken by Li et al. (2018) by expanding the subspace beyo… view at source ↗
Figure 2
Figure 2. Representing the merge tree as a topological landscape profile. In (A) we show a single basin corresponding to a merge tree with a single branch, and in (B) we show multiple basins corresponding to multiple branches. In (C) we color the basins based on their average loss. We also use a symmetric version of this graph, where points are only considered neighbors if each point is a neighbor of the other. In this case, … view at source ↗
Figure 3
Figure 3. Analyzing the loss function of a physics-informed neural network (PINN) trained to solve simple physical convection problems. See Section 4.1 for details. In other words, since the solution is constrained by the physical parameter, perturbing the model results in a faster increase in the loss, given that the physical problem is no longer satisfied. In contrast, the more bowl-like landscapes correspond to the failure… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: , we consider how the loss landscape changes over training. When looking at the loss landscapes for three different random seeds, after zooming in, we observe an initially shallow loss landscape but with the global minimum at a much higher loss compared to the end of t…
Figure 5
Figure 5. Figure 5: Comparing topological landscape profiles based on (A) three-dimensional and (B) four-dimensional loss landscapes. See Section 4.1 for details. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Loss landscapes across learning rates for UNet models with a CRF layer trained on the Oxford-IIIT Pet dataset. See Section 4.2 for details. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages

  1. [10]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 ,

    arXiv

  2. [11]

    Evaluating Loss Landscapes from a Topology Perspective

    12 Visualizing Loss Functions as Topological Landscape Profiles Tiankai Xie, Caleb Geniesse, Jiaqing Chen, Yaoqing Yang, Dmitriy Morozov, Michael W Mahoney, Ross Maciejewski, and Gunther H Weber. Evaluating loss landscapes from a topology perspective. arXiv preprint arXiv:2411.09807 ,

    work page Pith review arXiv

  3. [12]

    Evaluating natural language process- ing models with generalization metrics that do not need access to any training or testing data

    Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchan- dran, Charles H Martin, and Michael W Mahoney. Evaluating natural language process- ing models with generalization metrics that do not need access to any training or testing data. arXiv preprint arXiv:2202.02842 , 2022b. Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Ma...

    arXiv 2020

  4. [1992]

    Sok: On finding common ground in 11 Geniesse et al

    Arham Khan, Todd Nief, Nathaniel Hudson, Mansi Sakarvadia, Daniel Grzenda, Aswathy Ajith, Jordan Pettyjohn, Kyle Chard, and Ian Foster. Sok: On finding common ground in 11 Geniesse et al. loss landscapes using deep model merging techniques. arXiv preprint arXiv:2410.12927 ,

    arXiv

  5. [2011]

    Qualitatively characterizing neural network optimization problems

    Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544 ,

    arXiv

  6. [2012]

    Challenges in training pinns: A loss landscape perspective

    Pratik Rathore, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. Challenges in training pinns: A loss landscape perspective. arXiv preprint arXiv:2402.01868 ,

    arXiv

  7. [2013]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on Computer Vision and Pattern Recognition , pages 3498–3505. IEEE,

    work page 2012

  8. [2016]

    An empirical analysis of the opti- mization of deep network loss surfaces

    Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the opti- mization of deep network loss surfaces. arXiv preprint arXiv:1612.04010 ,

    arXiv

Show all 12 references
  1. [2017]

    Adversarial machine learning at scale

    Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.01236 ,

    arXiv

  2. [2018]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 ,

    1907 arXiv

  3. [2021]

    BERT: Pre- training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    arXiv

  4. [2024]

    Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson, Caleb Geniesse, Kyle Chard, Yaoqing Yang, Ian Foster, and Michael W. Mahoney. Mitigating memorization in language models. arXiv preprint arXiv:2410.02159 ,

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.