REVIEW 4 major objections 3 minor 21 references
Understanding Mode Connectivity via Parameter Space Symmetry
T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Symmetry groups determine how many disconnected pieces the minima of a linear network split into, and skip connections can merge those pieces.
desk verdict Worth a round of revisions: the symmetry-based component count is a real result, but the paper needs to fix a broken proof, tighten its scope claims, and correct a sign error before I'd trust it. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the symmetry group acting on parameters by $(g_1,\dots,g_{l-1})\cdot(W_1,\dots,W_l)=(g_1W_1,\ g_2W_2g_1^{-1},\dots,W_l g_{l-1}^{-1})$, whose orbits are the zero-loss minima; the component count is inherited from the two components of $GL_h(\mathbb{R})$ under the homeomorphism with $(GL_h)^{l-1}$. For path construction, the mechanism is the Lie-group exponential: for a symmetry element $g$, the curve $\gamma(t)=\exp(t\log g)\cdot w$ stays inside the level set of $w$, and its curvature bounds the distance from the chord $w_1w_2$ to the level set, yielding the loss-barrier bound via Lipschitz continuity.
What would settle it
Run gradient descent to zero loss on a three-layer linear network with $h=2$, full-rank fixed $X,Y\in\mathbb{R}^{2\times 2}$, and many random initializations; record the sign pattern of the determinants of the two gauge factors at each converged solution. If a single initialization can reach two different sign patterns at zero loss, or if fewer or more than four disconnected basins appear, Corollary 4.2 is wrong. For the residual claim, repeat the scalar $W_3(W_2W_1X+\varepsilon X)$ experiment with $\varepsilon\neq 0$; finding four rather than three disconnected components of the minimum would falsify Proposition 4.3.
Extended reading notes
Core claim
The paper's central result is a symmetry-topology dictionary for minima. For $L(W_1,\dots,W_l)=\|Y-W_l\cdots W_1 X\|_2^2$ with $X,Y\in \mathbb{R}^{h\times h}$ of full rank, the zero-loss minimum is homeomorphic to $(GL_h)^{l-1}$ via $(g_1,\dots,g_{l-1})\mapsto (g_1 X^{-1}, g_2,\dots,g_{l-1}, Y\prod_i g_i^{-1})$. Since $GL_h(\mathbb{R})$ has two connected components, the minimum has exactly $2^{l-1}$ components. Adding a residual term $\varepsilon X$ to a three-layer scalar linear network merges two of the four components into one, so the minimum has three components. For hidden dimension $h\ge 2$, any two zero-loss points become connected after conjugation by permutation matrices. Using the rescaling symmetry $\sigma(cz)=c^k\sigma(z)$, the paper proves that two minima in the same connected component can have unbounded loss on their linear interpolation, and gives a boundedness and curvature counterpart ensuring that linear mode connectivity holds approximately when the symmetry-induced curve is nearly straight.
Load-bearing premise
The counting argument assumes the input and output matrices are full rank and that every zero-loss solution can be reached from any other by the network's layer-wise invertible transformations; if the data are rank-deficient, the minimum has a different shape and the $2^{l-1}$ count is not established.
Editorial extensions
If this is right
- For full-rank linear networks of depth $l$, zero-loss training can settle into exactly $2^{l-1}$ separate basins, counted by determinant signs; width does not change this number.
- Permutation alignment, the standard trick in model merging, is sufficient to connect any two zero-loss solutions of a full-rank linear network once the hidden dimension is at least 2.
- Skip connections are a topological intervention: a residual term can merge connected components of the minimum, giving a new mechanistic reason ResNets and DenseNets may be easier to train and merge.
- Linear interpolation of minima is not safe in general: for homogeneous activations, the loss barrier between two minima on the same component can be made arbitrarily large by rescaling one of them.
- Approximate linear mode connectivity holds when the symmetry-induced curve has bounded curvature and the loss is Lipschitz: the loss on the chord is at most $C_L d_{\max}$, with $d_{\max}$ given by the curvature bound.
Reading between the lines
- Extension: if the $2^{l-1}$ count persists for positive-loss sublevel sets, then SGD's observed basin of attraction would be predictable from the determinant signs of the initialization; a testable prediction is that runs initialized in different sign sectors are never connected without permutation or a skip connection.
- Extension: the residual-connection example suggests a design principle: architectures whose symmetry group has fewer connected components than the unmodified product should have more connected minima; quantifying this for normalization layers or attention would be a natural next step.
- Extension: the unbounded-barrier construction relies on unbounded rescaling, so the practical takeaway is that implicit bias toward bounded weights, rather than connectivity per se, may explain why linear mode connectivity is seen in practice; this can be tested by deliberately inflating weights and measuring barrier growth.
- Extension: Theorem 6.2 yields a cheap pre-merging check: estimate the curvature of the symmetry path between two models, and if $\kappa_{\max}\|w_2-w_1\|$ is small, direct averaging should stay near the minimum manifold.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes parameter-space symmetry as a tool to analyze the topology of the loss landscape. For a linear network with square full-rank input and output matrices, it proves a homeomorphism between the zero-loss set and (GL_h)^{l-1}, yielding 2^{l-1} connected components of the minimum. It then shows that adding a scalar skip connection reduces this count, proves mode connectivity up to permutation for the same linear family, constructs explicit symmetry-induced curves on the minimum, and studies linear mode connectivity, including unbounded barriers in homogeneous networks and a curvature-based sufficient condition for approximate linear connectivity.
Significance. The connection between symmetry-group topology and the number of connected components of the minimum is a clean and potentially useful idea. The paper's Section 4 is largely correct under the stated square-invertible assumptions, and the skip-connection example in Proposition 4.3 is a nice illustration. The failure-case results in Section 5.2 are also plausible and relevant to the empirical literature on linear mode connectivity. However, the paper overstates the scope of the component-counting result as 'full-rank linear regression' when the proof actually requires square invertible data, and the central curvature-based sufficient condition in Section 6.2 rests on a theorem that is false as stated. These issues are load-bearing and require major revision.
major comments (4)
- [Abstract and Section 4.1] The abstract and contributions describe the main counting result as applying to 'full-rank linear regression,' but Proposition 4.1 and Corollary 4.2 require X, Y in R^{h x h} to be invertible square matrices, not merely full-rank. The homeomorphism in (10) uses X^{-1} and Y, and its inverse (W1,...,Wl) -> (W1 X, W2,...,W_{l-1}) requires X to be invertible. For rank-deficient or rectangular data the construction breaks; for example, with h=2, l=2, X=Y=diag(1,0), the set L^{-1}(0) is connected: for every nonzero u, W1=[u,0], W2=e1 u^T/||u||^2 gives W2 W1 X = Y, and varying u connects det(W1)>0 to det(W1)<0 within the minimum. The statements in the abstract and contributions should be restricted to square invertible data.
- [Appendix D, proof of Proposition 5.3] The curve gamma(t) = ((1-t)W_l + t W_l m^{-k}, (1-t)W_{l-1} + t m W_{l-1}, W_{l-2}, ..., W_1) used in the proof does not lie in L^{-1}(0) for intermediate t. Writing a = 1-t+t m^{-k} and b = 1-t+t m, the network output on this curve is a b^k Y, which equals Y only when a b^k = 1, i.e., at t=0 and t=1. Thus the claim that W and W' belong to the same connected component of L^{-1}(0) is not established by the displayed curve, although the statement is true via the level-set path s -> (W_l s^{-k}, W_{l-1} s, W_{l-2}, ..., W_1). The proof should be corrected to use a path that actually stays in the minimum.
- [Section 6.2, Theorem 6.2] Theorem 6.2 is false as stated. The proof compares an arbitrary smooth curve with the osculating circle and concludes that the curve cannot deviate from its chord more than the circular arc, but this is not valid for curves with bounded curvature that are not convex or that wind around. A concrete counterexample: let L^{-1}(c) contain the curve formed by a straight segment from (0,0) to (100,0), a semicircle of radius 1 turning back to (100,2), and a straight segment to (0,1.5), with w1=(0,0), w2=(0,1.5). This C^1 curve has curvature at most 1, chord length L=1.5, so kappa_max=1 and dmax = 1 - sqrt(1 - 0.75^2) ≈ 0.339. The midpoint (0,0.75) of the chord is at distance 0.75 from the curve, violating inequality (8). Consequently, the claimed sufficient condition for approximate linear mode connectivity from bounded curvature is invalid without additional hypotheses on the curve (e.g., bounded total curvature or geodesic convexity of the level set).
- [Section 6.1, Eq. (5)] The explicit curve gamma(t) = exp(t log(g)) · w is defined only when log(g) exists. For GL_h(R), the matrix logarithm is not defined for all g: matrices with negative determinant have no real logarithm, and so do some positive-determinant matrices with negative eigenvalues. For a general g in the orbit, exp(t log(g)) is undefined, so the paper's claim of an explicit symmetry-induced curve between any two points in the same orbit is too strong. The construction should be restricted to the identity component (or to elements in the image of the exponential map), or replaced by a path in the group obtained from polar decomposition.
minor comments (3)
- [Section 6.1, Figure 3(a)] The text says 'All points are above the line y = x, as predicted by Proposition 6.1,' but inequality (7) states ||U sigma(V X) - U' sigma(V' X)|| <= ||U sigma(V X)||, which predicts points on or below the line y = x. The axes or the accompanying statement should be corrected to match the inequality.
- [Proposition 5.2] There is a typo in the statement: 'these exists' should be 'there exists'.
- [Appendix D, proof of Proposition 5.3] The inequality in (17) is written with a strict '>' but the preceding choices yield equality: (1 - (sqrt(b)/||Y|| + 1))^2 ||Y||^2 = b. The conclusion 'at least b' is still valid if the strictness is relaxed to '>='.
Circularity Check
No significant circularity: the component counts and connectivity claims are proved from explicit homeomorphisms and standard topological facts, with self-citations only contextual.
full rationale
The central derivation chain is self-contained. Proposition 4.1 constructs an explicit homeomorphism f : (GL_h)^{l-1} -> L^{-1}(0) by f(g_1,...,g_{l-1}) = (g_1 X^{-1}, g_2, ..., g_{l-1}, Y (g_{l-1}...g_1)^{-1}) and gives an explicit inverse, so the 2^{l-1} component count in Corollary 4.2 follows directly from the known two-component structure of GL_h(R) and standard product-topology counting, not from any fitted quantity or self-citation. Proposition 4.3 similarly proves the ResNet component count by an explicit decomposition into S_1 and S_0 and explicit homeomorphisms. The mode-connectivity results in Section 5 use explicit group actions and direct inequalities, and Theorem 6.2 is a geometric curvature-chord bound proved in the appendix. The citations to the authors' prior work (Zhao et al. 2023, 2024) are contextual remarks about broader symmetry frameworks and are not load-bearing for the new counting or connectivity theorems; those theorems are proved from the loss function's own symmetry structure and standard general-linear-group topology. The full-rank square-data limitation on Corollary 4.2 is a scope restriction on the theorem, not a circularity, because the proof explicitly states and uses the invertibility of X and Y. No parameter is fitted and then renamed a prediction, and no uniqueness claim is imported from a self-citation. Accordingly, the paper receives a circularity score of 0.
Assumptions & free parameters
assumptions (6)
- standard math Continuous images of connected spaces are connected (Theorem 3.1).
- standard math GL_n(R) has exactly two connected components, distinguished by the sign of the determinant.
- domain assumption The loss for the linear network is invariant under the GL_h^{l-1} action, and X and Y are full rank so the minimum is exactly a product of GL groups.
- domain assumption The activation sigma is positively homogeneous of degree k (sigma(c z) = c^k sigma(z)), and the scale-invariance of the last two layers is used to construct the second minimum.
- domain assumption The loss L is Lipschitz continuous in Theorem 6.2.
- domain assumption The connecting curve is smooth with curvature bounded by kappa_max in Theorem 6.2.
Cite this review
Pith. "Pith review of Understanding Mode Connectivity via Parameter Space Symmetry." pith.science (2026). https://pith.science/paper/SR3EGYUO
@misc{pith2026250523681,
author = {Pith},
title = {Pith review of: Understanding Mode Connectivity via Parameter Space Symmetry},
year = {2026},
howpublished = {\url{https://pith.science/paper/SR3EGYUO}},
note = {Machine review of arXiv:2505.23681}
}
read the original abstract
Neural network minima are often connected by curves along which train and test loss remain nearly constant, a phenomenon known as mode connectivity. While this property has enabled applications such as model merging and fine-tuning, its theoretical explanation remains unclear. We propose a new approach to exploring the connectedness of minima using parameter space symmetry. By linking the topology of symmetry groups to that of the minima, we derive the number of connected components of the minima of linear networks and show that skip connections reduce this number. We then examine when mode connectivity and linear mode connectivity hold or fail, using parameter symmetries which account for a significant part of the minimum. Finally, we provide explicit expressions for connecting curves in the minima induced by symmetry. Using the curvature of these curves, we derive conditions under which linear mode connectivity approximately holds. Our findings highlight the role of continuous symmetries in understanding the neural network loss landscape.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
There exists a space X and a group G with an action on X, such that each orbit for the group action is connected and X is not connected
-
[2]
There exists a space X and a group G with an action on X, such that each orbit for the group action is disconnected and X is connected. Proof. For part (a), consider a subspace of R2, X = X1 ∪ X2 where X1 = {(x, y) :x = 0, y >0} and X2 = {(x, y) : x = 1, y >0}. The space X is not connected. Let G be the multiplicative group of positive real numbers and ac...
work page 2021
-
[5]
For any g ∈ GL(h) such that det(g) < 0, g · (W1, W2) and (W ′ 1, W′
∈ L−1(0) that are not connected in L−1(0). For any g ∈ GL(h) such that det(g) < 0, g · (W1, W2) and (W ′ 1, W′
-
[6]
are connected in L−1(0). Proof. Consider the map f and its inverse f −1 defined in (10) in the proof of Proposition 4.1. Let g = f −1(W1, W2) and g′ = f −1(W ′ 1, W′ 2). By Corollary A.2, since (W1, W2) and (W ′ 1, W′
-
[7]
are not in the same connected component of L−1(0), g and g′ are not in the same connected component of GLh. Equivalently, det(gg ′) < 0. Consider a g1 ∈ GLh such that det(g) < 0. Then det(g1gg ′) > 0, which means that g1g and g′ belong to the same connected component of GLh. Therefore, according to Corollary A.2, g1 · (W1, W2) =f (g1g) and (W ′ 1, W′
-
[8]
=f (g′) belong to the same connected component of L−1(0). Example. Suppose W1 = 1 0 0 1 , W2 = −1 0 0 1 is a point in L−1(0) for some loss function L. Then W ′ 1 = −1 0 0 1 , W′ 2 = 1 0 0 1 is also a point in L−1(0). However, (W1, W2) and (W ′ 1, W′
-
[9]
By Lemma 5.1, any g ∈ GL(h) with det(g) < 0 can bring (W1, W2) and (W ′ 1, W′
are not on the same connected component of the minimum, since their determinants have different signs. By Lemma 5.1, any g ∈ GL(h) with det(g) < 0 can bring (W1, W2) and (W ′ 1, W′
-
[10]
Let g be the permutation matrix 0 1 1 0
to the same connected component in L−1(0). Let g be the permutation matrix 0 1 1 0 . Then g · (W1, W2) = 0 1 1 0 , 0 −1 1 0 , which is in the same connected component as (W ′ 1, W′ 2). Proposition 5.2. Assume that h ≥ 2. For all (W1, ..., Wl), (W ′ 1, ..., W′ l ) ∈ L−1(0), these exists a list of permutation matri- ces P1, ..., Pl−1 such that (W1P1, P−1 1 ...
Show all 21 references
-
[11]
Then W, W′ belong to the same connected component of L−1(0), connected by curve γ : R → Param, γ(t) = ((1− t)Wl + tWlm−k, (1 − t)Wl−1 + tmWl−1, Wl−2, ..., W1)
= (Wlm−k, mWl−1, Wl−2, ..., W1). Then W, W′ belong to the same connected component of L−1(0), connected by curve γ : R → Param, γ(t) = ((1− t)Wl + tWlm−k, (1 − t)Wl−1 + tmWl−1, Wl−2, ..., W1). Since W ∈ L−1(0), we have Wlσ [Wl−1f (Wl−2, ..., W1, X)] =Y . The loss on the linear...
-
[12]
Then L ((1 − α)W + αW ′) = 1 − 1 2 + 1 2 m−k 1 2 + 1 2 m k!2 ||Y ||2 2 = 1 − 2−(k+1)(1 +m−k)(1 +m)k 2 ||Y ||2 2 (15) Let m = 2k+1 √ b ||Y ||2 + 1 − 1 k
(14) Let α = 0.5. Then L ((1 − α)W + αW ′) = 1 − 1 2 + 1 2 m−k 1 2 + 1 2 m k!2 ||Y ||2 2 = 1 − 2−(k+1)(1 +m−k)(1 +m)k 2 ||Y ||2 2 (15) Let m = 2k+1 √ b ||Y ||2 + 1 − 1 k . Recall that k >0. Then m >0, (1 +m)k > 1, and 2−(k+1)(1 +m−k)(1 +m)k > 2−(k+1)(1 +m−k) = √ b ||Y ||2 + 1>...
-
[13]
At large m, the two minima are farther apart, and the loss evaluated at the middle point of their linear interpolation grows unboundedly as predicted by Proposition 5.3
by SGD, and the second (W ′′ 1 , W′′ 2 , W′′ 3 , W′′ 4 , W′′ 5 ) = (W ′ 1, W′ 2, W′ 3, mW′ 4, W′ 5m−1) by rescaling the last two layers with m ∈ R+. At large m, the two minima are farther apart, and the loss evaluated at the middle point of their linear interpolation grows unb...
-
[14]
Since W ∈ L−1(0), we have Wlσ [Wl−1f (Wl−2, ..., W1, X)] =Y
= (Wlm−k, mWl−1, Wl−2, ..., W1). Since W ∈ L−1(0), we have Wlσ [Wl−1f (Wl−2, ..., W1, X)] =Y . The loss on the linear interpolation of W, W′ is L ((1 − α)W + αW ′) =||Y − ((1 − α)Wl + αW ′ l P )σ ((1 − α)Wl−1 + αP −1W ′ l−1)f (Wl−2, ..., W1, X) ||2
-
[15]
Then L ((1 − α)W + αW ′) =||Y − 1 4 Wl(I + m−kP )σ (I + mP −1)Wl−1f (Wl−2, ..., W1, X) ||2
(18) Let α = 0.5. Then L ((1 − α)W + αW ′) =||Y − 1 4 Wl(I + m−kP )σ (I + mP −1)Wl−1f (Wl−2, ..., W1, X) ||2
-
[16]
(19) When m → ∞, lim m→∞ σ (I + mP −1)Wl−1f (Wl−2, ..., W1, X) = lim m→∞ mkσ (m−1I + P −1)Wl−1f (Wl−2, ..., W1, X) = lim m→∞ mkP −1σ [Wl−1f (Wl−2, ..., W1, X)] . (20) Therefore, lim m→∞ L ((1 − α)W + αW ′) = lim m→∞ ||Y − 1 4 Wl(I + m−kP )mkP −1σ [Wl−1f (Wl−2, ..., W1, X)] ||2...
-
[17]
Therefore, L ((1 − α)W + αW ′) is unbounded for any P
(21) Since we assumed that there does not exist a permutationP such that every column ofP σ(Wl−1f (Wl−2, Wl−3, ..., W1, X)) is in the null space of Wl, at least one element in the second term is unbounded for any permutation P . Therefore, L ((1 − α)W + αW ′) is unbounded for ...
-
[18]
and W ′′ = (W ′′ 1 , W′′ 2 ) are both in S. Then, min ˆW ∈S ∥ ((1 − α)W ′ + αW ′′) − ˆW ∥2 2 = min ˆW ∈S ∥(1 − α)W1g−1 1 + αW1g−1 2 − ˆW1∥2 2 + ∥(1 − α)g1W2 + αg2W2 − ˆW2∥2 2 = min g∈GL(n) ∥W1((1 − α)g−1 1 + αg−1 2 − g−1)∥2 2 + ∥W2((1 − α)g1 + αg2 − g)∥2
-
[19]
Let α = 1
(22) 18 Understanding Mode Connectivity via Parameter Space Symmetry Let g1 = βI and g2 = β−1I for some β >0. Let α = 1
-
[20]
(23) As β → ∞, g and g−1 cannot approach β+β−1 2 I simultaneously
Then, in the limit of a large β, we have lim β→∞ min ˆW ∈S ∥ ((1 − α)W + αW ′) − ˆW ∥2 2 = lim β→∞ min g∈GL(n) W1 β + β−1 2 I − g−1 2 2 + W2 β + β−1 2 I − g 2 2 . (23) As β → ∞, g and g−1 cannot approach β+β−1 2 I simultaneously. Therefore, (23) is not bounded. Proposition 5.6...
-
[21]
The connectedness results derived from symmetry raise several interesting questions about mode connectivity
(24) As m, m−k, and α are all bounded, the loss value for points on the linear interpolation L ((1 − α)W + αW ′) is also bounded. The connectedness results derived from symmetry raise several interesting questions about mode connectivity. For example, it would be interesting t...
2018
-
[3269]
Freeman, C
PMLR, 2020. Freeman, C. D. and Bruna, J. Topology and geometry of half-rectified network optimization. In 5th International Conference on Learning Representations, ICLR, 2017. Gabrielsson, R. B. and Carlsson, G. Exposition and interpre- tation of the topology of neural network...
2020
-
[4799]
Nguyen, Q
PMLR, 2019. Nguyen, Q. A note on connectivity of sublevel sets in deep learning. arXiv preprint arXiv:2101.08576, 2021. Nguyen, Q. N., Br´echet, P., and Mondelli, M. When are so- lutions connected in deep networks? Advances in Neural Information Processing Systems, 34:20956–20...
2019 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.