REVIEW 5 major objections 4 minor 12 references
Understand the Effectiveness of Shortcuts through the Lens of DCA
T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Applying DCA to a vanilla network yields the ResNet gradient, making shortcut connections a consequence of difference-of-convex optimization rather than an ad hoc design.
desk verdict Interesting idea, unsupported central claim: the DCA-shortcut equivalence rests on an unproved and likely false orthogonality assumption; NegNet is a nice by-product. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is a specific DC decomposition of the training objective, $G_2 = L + \tfrac{1}{2} L'_{-1} + \tfrac{\rho}{2}\|w\|^2$ and $H_2 = \tfrac{1}{2} L'_{-1} + \tfrac{\rho}{2}\|w\|^2$, where $L'_{-1}$ is the 'counter loss' evaluated at $-h_L + 2h_{L-1}$, chosen to absorb the negative eigenvalues of the Hessian's off-diagonal block. Solving the DCA subproblem by Newton's method produces a Hessian whose essential part is the rank-2 matrix $a a^\top + \tfrac{1}{2}(2c-a)(2c-a)^\top$, and inverting this matrix in the high-dimensional regime gives the closed-form update of Eq. (19). The argument leans on the high-dimensional orthogonality of random unit vectors (Eq. (18)) to discard the off-diagonal block $*$; the paper calls this the 'blessing from the high dimension.'
What would settle it
Train a small network with the exact update in Eq. (19) and compare the resulting parameter trajectory, step by step, with the gradient of the corresponding ResNet; a disagreement larger than floating-point tolerance would falsify the identity. A more targeted check is to measure, during real training, the projection of the off-diagonal Hessian block onto the $a$ and $c$ directions—the paper's claim predicts these projections stay near zero at realistic widths.
Extended reading notes
Core claim
The central claim is the identity DCA + vanilla network = SGD + shortcut network. Concretely, the paper derives Eq. (19): the DCA update for a vanilla network is $\Delta w \propto a + \frac{\|a\|^2}{2\|c\|^2 + \rho/4} c$, where $a$ is the ordinary vanilla gradient and $c$ is the gradient of the same loss through the shortcut path. Because the update contains a component along $c$, the parameter trajectory matches that of a residual network trained by SGD. The derivation holds for mean-square loss and, with a stated approximation whose error decays during training, for cross-entropy loss. The paper also introduces NegNet, which replaces the identity shortcut with a negative identity; despite contradicting the usual residual intuition, it performs comparably to ResNet18 on CIFAR10, which the authors read as support for the DCA account rather than the information-flow account.
Load-bearing premise
The derivation collapses if the neglected off-diagonal Hessian block is not confined to directions perpendicular to both the vanilla gradient $a$ and the shortcut gradient $c$; the paper justifies this by treating $a$ and $c$ as effectively random, although they are deterministic and data-dependent.
Editorial extensions
If this is right
- Residual connections can be replaced by a DC decomposition of the loss, turning architectural design into an optimization-design problem: pick a decomposition, derive the update, and read off the architecture needed to compute it with automatic differentiation.
- The DCA convergence theorem becomes available for training shortcut networks by SGD, giving a route to global-convergence guarantees under bounded iterates.
- NegNet's parity with ResNet on CIFAR10 indicates the information-flow interpretation of shortcuts is not the whole story, since a negative shortcut trains just as well.
- For cross-entropy loss the shortcut-gradient identity holds approximately, with an error of order $e^{E_{i\neq t}}/\sum_k e^{E_k}$ that shrinks as the model fits.
Reading between the lines
- Treating the counter-loss coefficient $\alpha$ as a free parameter generates a one-parameter family of architectures whose updates interpolate between vanilla, ResNet, and NegNet; the paper's derivation suggests all should train similarly in high dimension.
- The same derivation could be rerun for transformer blocks, where the residual stream is the analog of $h_{l-1} + F^{l-1}$; if high-dimensional orthogonality holds there, it would give an optimization-based explanation for why attention architectures need residual connections.
- Even if the orthogonality claim fails at moderate widths, the qualitative conclusion survives: DCA injects a shortcut-direction component into the update with a modified coefficient, so the recipe of inventing architectures from DC decompositions does not hinge on the strongest form of the assumption.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that applying the Difference-of-Convex Algorithm to a vanilla neural network produces the gradient structure of a shortcut (ResNet) network, summarized by the slogan 'DCA + vanilla network = SGD + shortcut network.' The main derivation in Section 3 introduces an auxiliary loss L'_alpha, proposes the DC decomposition G2 = L + 1/2 L'_{-1} + rho/2 L2 and H2 = 1/2 L'_{-1} + rho/2 L2, and, under a high-dimensional orthogonality assumption, obtains the update Delta w proportional to a + (||a||^2/(2||c||^2 + rho/4)) c, where a is the vanilla gradient and c is the shortcut-path gradient. The paper also introduces NegNet, a network with negative shortcuts, and reports a CIFAR10 experiment in which it performs comparably to ResNet18.
Significance. If the central claim were rigorously established, the paper would offer a genuinely new interpretation of residual connections: shortcuts would be seen as implicit carriers of second-order curvature information inside first-order training, and DCA would become a constructive recipe for generating new architectures. The paper has tangible strengths: the DC decomposition is explicitly written down, Eq. (19) is a concrete falsifiable formula, and the NegNet experiment is a creative test of the framework. The Section 2 survey of DC decompositions is also useful. However, the load-bearing derivation is not currently sound: the central orthogonality step is unproved, and the derived update is not actually the SGD update of a shortcut network. The significance as a theoretical explanation is therefore not yet established.
major comments (5)
- [§3, Eq. (18) and footnote 3] The 'blessing from the high dimension' is load-bearing and unproved. Equation (18) is an expectation over independent random unit vectors, but the vectors a and c defined in Eq. (17) are deterministic functions of the weights, activations, and labels. No argument shows that they behave like independent isotropic random vectors; near initialization, or when features are correlated, their inner product can be large. The same objection applies to the claim that the eigenvectors of the indefinite block * in Eqs. (7)-(8) are perpendicular to a and c; * is built from the same d_h F and d_w F blocks as A and C, so there is no statistical independence to appeal to. If * has nonzero projection onto span{a, c}, the exact solution of Eq. (14) contains additional terms and Eq. (19) is not the DCA update. Since Eq. (19) is the only evidence for the paper's central claim, this gap is decisive.
- [§3, Eq. (19)] Even if a and c are orthogonal and the * block can be neglected, Eq. (19) gives Delta w proportional to a + beta c with beta = ||a||^2/(2||c||^2 + rho/4), not to a + c. The ResNet gradient in Eq. (6) is a + c, so the derived update is not the shortcut-network SGD gradient unless beta = 1, which is not shown and is generically false because beta depends on the norms of a and c and on rho. Moreover, Eq. (14) is a Newton step for a convex subproblem, not an SGD iteration. The abstract's equality 'DCA + vanilla network = SGD + shortcut network' therefore overstates what is derived; at most one can say that the DCA update has an extra component along c, a qualitative statement rather than the claimed equivalence.
- [§3, Eq. (12)] The paper never proves that G2 and H2 are convex, which is required for the DCA framework. The assertion that L'_{-1} can absorb the negative eigenvalues of the * block is made without calculation, and the subsequent instruction that rho should be larger than the largest negative eigenvalue of * is not enough by itself. One must verify that the full Hessian of H2 = 1/2 L'_{-1} + (rho/2)L2 is positive semidefinite and that the Hessian of G2 = L + 1/2 L'_{-1} + (rho/2)L2 is positive definite. The Hessian of L'_{-1} contains both the positive semidefinite term (2C-A)^T d^2_hL'(2C-A) and the indefinite alpha = -1 block, so the needed bound involves singular values of * and the interplay with A and C. Without this verification, the algorithm analyzed is not known to be a DCA and Theorem 2.1 cannot be invoked.
- [§3, CE loss case, Eq. (22)] The cross-entropy case, promised in the abstract, rests on the approximation in Eq. (22), but for C > 2 the exact Hessian in Eq. (21) is a sum of pairwise difference terms (w_k-w_l)(w_k-w_l)^T. The outer product d_hL(d_hL)^T is not proportional to that sum in general; the claim that the approximation error is of order e^{E_{i != t}}/sum_k e^{E_k} is asserted without proof. Since this approximation is the basis for reducing the CE case to the rank-two MSE argument, the CE part of the central claim is not established.
- [§4, Eq. (23) and Figure 2] The NegNet experiment is presented as empirical support but is a single CIFAR10 comparison with no training hyperparameters, number of runs, or error bars, so the claim that it performs 'on par' with ResNet is not quantified. In addition, Eq. (23) is not a DC decomposition because H3 is admitted to be nonconvex ('quasi'), so the DCA convergence guarantees do not apply. This does not invalidate the architectural idea, but as reported it cannot serve as evidence for the DCA-shortcut equivalence.
minor comments (4)
- [Section 5] The word 'Morden' should be 'Modern'.
- [Section 3] The phrase 'Euclidian' should be 'Euclidean'.
- [§3, after Eq. (9)] The claim that a sufficiently large lambda always exists because 'parameters are often constrained within a limited range' is not a proof, since the training objective is defined over unbounded parameter space.
- [§3, Eq. (10)-(11)] The notation for the Hessian of L'_alpha would be clearer if the dimension of d^2_hL' and the meaning of the block matrix in Eq. (11) were specified explicitly.
Circularity Check
No significant circularity: the DCA-to-shortcut derivation is a constructive conditional argument; the main gap is an unproved high-dimensional orthogonality premise, not a definitional loop.
full rationale
The paper claims to derive the shortcut-network gradient from a particular DC decomposition of the vanilla loss. The construction is explicit: G2 = L + 1/2 L'_{-1} + ρ/2 L2 and H2 = 1/2 L'_{-1} + ρ/2 L2, with L'_α defined in Eq. (10). Since G2 − H2 = L, the decomposition exactly represents the vanilla loss; the DCA update is then computed, not assumed. The appearance of the shortcut block C in the Hessian of G2 follows from differentiating L'_{-1} with respect to h_{L-1}, not from inserting the final ResNet update into the algorithm. Eq. (19) is obtained algebraically from the rank-2 Hessian in Eq. (16) together with the explicit orthogonality assumption that a and c are orthogonal. Thus the derivation is self-contained as a conditional statement: under that geometric premise, the Newton step on the chosen subproblem has the stated form. This is not circular; it is a constructive existence result with an unproved premise. The high-dimensional orthogonality premise is asserted in the footnote 'blessing from the high dimension' without proof and is a correctness risk, not a circularity. The paper contains several self-citations (e.g., Niu 2022; Niu et al. 2024; Pham & Niu 2011), but they support background DCA convergence and applications rather than the shortcut derivation; Theorem 2.1 is also attributed to the classical Pham & Le Thi 1997. No fitted parameters are relabeled as predictions, no uniqueness theorem is imported from the authors, and no known result is merely renamed. The NegNet experiment is offered as a novel architectural consequence rather than as evidence that the derivation was circular. Overall circularity is minimal.
Assumptions & free parameters
free parameters (3)
- rho (ρ in Eq. 12) =
not specified; paper requires it to be larger than the largest negative eigenvalue of the * matrix
- alpha (α in counter loss L'_α, Eq. 10) =
-1 for G2, 1/2 for G3, -1/2 for G4
- lambda (λ in Eq. 9) =
not specified; 'large enough' for convexity
assumptions (4)
- domain assumption ReLU has zero second derivative: ∂²_h F^l = 0 and ∂²_w F^l = 0
- domain assumption The loss Hessian is positive definite: ∂²_h L ≻ 0
- ad hoc to paper a and c behave like random unit vectors in high dimension
- ad hoc to paper Existence of ρ such that G2 and H2 are convex
invented entities (2)
-
Counter loss L'_α = L(α h_L + (1 - α) h_{L-1})
-
NegNet (h_l = -h_{l-1} + F^{l-1}(h_{l-1}))
Cite this review
Pith. "Pith review of Understand the Effectiveness of Shortcuts through the Lens of DCA." pith.science (2026). https://pith.science/paper/ST3BI45V
@misc{pith2026241209853,
author = {Pith},
title = {Pith review of: Understand the Effectiveness of Shortcuts through the Lens of DCA},
year = {2026},
howpublished = {\url{https://pith.science/paper/ST3BI45V}},
note = {Machine review of arXiv:2412.09853}
}
read the original abstract
Difference-of-Convex Algorithm (DCA) is a well-known nonconvex optimization algorithm for minimizing a nonconvex function that can be expressed as the difference of two convex ones. Many famous existing optimization algorithms, such as SGD and proximal point methods, can be viewed as special DCAs with specific DC decompositions, making it a powerful framework for optimization. On the other hand, shortcuts are a key architectural feature in modern deep neural networks, facilitating both training and optimization. We showed that the shortcut neural network gradient can be obtained by applying DCA to vanilla neural networks, networks without shortcut connections. Therefore, from the perspective of DCA, we can better understand the effectiveness of networks with shortcuts. Moreover, we proposed a new architecture called NegNet that does not fit the previous interpretation but performs on par with ResNet and can be included in the DCA framework.
Figures
Reference graph
Works this paper leans on
-
[2]
URL https: //arxiv.org/abs/1803.01719. He, F., Liu, T., and Tao, D. Why resnet works? residuals generalize. IEEE Transactions on Neural Networks and Learning Systems , 31(12):5349–5362,
-
[7]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A
URL https://arxiv.org/abs/1505.04597. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need,
-
[9]
Wang, C.-Y ., Yeh, I.-H., and Liao, H.-Y
URL https://arxiv.org/abs/1605.06431. Wang, C.-Y ., Yeh, I.-H., and Liao, H.-Y . M. Yolov9: Learn- ing what you want to learn using programmable gradi- ent information,
-
[10]
He, K., Zhang, X., Ren, S., and Sun, J
1109/TNNLS.2020.2966319. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. CoRR, abs/1512.03385,
arXiv 2020
-
[12]
URL https://arxiv.org/abs/ 1811.09885. 8
-
[2015]
He, K., Zhang, X., Ren, S., and Sun, J
URL http://arxiv.org/abs/1512.03385. He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. CoRR, abs/1603.05027,
-
[2016]
URL http://arxiv.org/abs/1603.05027. Le Thi, H. A., Huynh, V . N., Pham, D. T., and Hau Luu, H. P. Stochastic difference-of-convex-functions algo- rithms for nonconvex programming. SIAM Journal on Optimization, 32(3):2263–2293, 2022a. doi: 10.1137/ 20M1385706. URL https://doi.org/10.1137/ 20M1385706. Le Thi, H. A., Luu, H. P. H., Le, H. M., and Pham, D. T...
-
[2017]
Mean Field Residual Networks: On the Edge of Chaos
URL https: //arxiv.org/abs/1712.08969. Zhang, L. and Schaeffer, H. Forward stability of resnet and its variants,
Show all 12 references
-
[2018]
Hanin, B
URL https://arxiv.org/abs/1702.08591. Hanin, B. and Rolnick, D. How to start training: The effect of initialization and architecture,
-
[2022]
org/abs/2211.10942
URL https://arxiv. org/abs/2211.10942. Niu, Y .-S. and Zhang, H. Power-product matrix: nonsingu- larity, sparsity and determinant. Linear and Multilinear Algebra, 72(7):1170–1187,
-
[2023]
7 Understand the Effectiveness of Shortcuts through the Lens of DCA Veit, A., Wilber, M., and Belongie, S
URL https://arxiv.org/ abs/1706.03762. 7 Understand the Effectiveness of Shortcuts through the Lens of DCA Veit, A., Wilber, M., and Belongie, S. Residual networks be- have like ensembles of relatively shallow networks,
-
[2024]
URL https://arxiv.org/ abs/2402.13616. Yang, G. and Schoenholz, S. S. Mean field residual net- works: On the edge of chaos,
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.