{"id":"3c05310d-28d9-461b-a30e-1fa8040b15dd","arxiv_id":"2412.09853","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims that DCA, when applied to a vanilla network, yields residual-network-like updates, and introduces a new architecture NegNet as evidence.","lead":"This paper argues that applying the Difference-of-Convex Algorithm to a plain neural network can reproduce the gradient updates of a residual network. It also introduces NegNet, a network with negative shortcuts, and reports it matches ResNet on CIFAR10.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'blessing from the high dimension' is an unproved and likely false application of random-vector orthogonality to deterministic gradient vectors; without it Eq. (19) and the claimed DCA-shortcut equivalence do not follow.","rationale":"The paper's algebra from Eq. (14) to Eq. (19) is internally consistent once one grants a perpendicular to c and the eigenvectors of * perpendicular to span{a,c}; I verified the 2x2 solve reproduces the stated coefficient. The load-bearing weakness is exactly that grant. The paper asserts it as a 'blessing from the high dimension' but applies a probabilistic fact about independent random vectors to deterministic gradient vectors that share a common computational graph. No empirical or theoretical support is given, and the NegNet experiment (Fig. 2) lacks training details (epochs, learning rate, seeds, hyperparameters) and does not test the orthogonality directly. The reader's weakest_assumption matches this concern. A single concrete numerical test of the orthogonality and of the exact versus approximate step would settle whether Eq. (19) holds; in the absence of such evidence, the central claim is not established. Therefore the reader's REJECT verdict is appropriate; no adjustment is needed.","tokens_in":9349,"tokens_out":8144,"duration_ms":74764,"concrete_test":"On a small vanilla CNN (e.g., 2-3 layers, width 64-512) trained on CIFAR-10, compute at initialization and after 1, 10, 100 SGD steps: a = A^T w_L, c = C^T w_L, and the eigenvectors of the block matrix [0, *; *^T, 0] from Eq. (7-8). Measure |cos(a,c)| and the norm of the projection of a and c onto the top eigenspace of *, normalized by ||a|| and ||c||. Also compute the exact Newton step from Eq. (14) with * included and compare its direction to Eq. (19) via cosine similarity. If the cosines are not ~1/sqrt(#w), or the projection is non-negligible, or the step direction differs substantially, the orthogonality premise fails and the claimed equivalence does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central derivation of Eq. (19) depends on two unproved orthogonality claims: (i) the deterministic, data-dependent vectors a and c are nearly orthogonal because Eq. (18) holds for random unit vectors; and (ii) the eigenvectors of the indefinite block * (Eqs. 7-8) are perpendicular to a and c, so ignoring * does not change the Newton step. Neither claim follows. a and c are functions of the weights, activations, and labels; they can be strongly correlated (e.g., for shallow or wide layers, or near initialization, gradients often align). Eq. (18) is an expectation over independent random vectors, not a property of arbitrary fixed vectors. Likewise, * has no statistical independence from a and c; it is built from the same d_h F and d_w F blocks. If * has nonzero projection onto span{a, c}, the exact solution of Eq. (14) includes additional terms and Eq. (19) is not the DCA update. Since Eq. (19) is the only evidence for 'DCA + vanilla = SGD + shortcut', the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that applying the Difference-of-Convex Algorithm to a vanilla neural network produces the gradient structure of a shortcut (ResNet) network, summarized by the slogan 'DCA + vanilla network = SGD + shortcut network.' The main derivation in Section 3 introduces an auxiliary loss L'_alpha, proposes the DC decomposition G2 = L + 1/2 L'_{-1} + rho/2 L2 and H2 = 1/2 L'_{-1} + rho/2 L2, and, under a high-dimensional orthogonality assumption, obtains the update Delta w proportional to a + (||a||^2/(2||c||^2 + rho/4)) c, where a is the vanilla gradient and c is the shortcut-path gradient. The paper also introduces NegNet, a network with negative shortcuts, and reports a CIFAR10 experiment in which it performs comparably to ResNet18.","tokens_in":9666,"tokens_out":13208,"duration_ms":131260,"significance":"If the central claim were rigorously established, the paper would offer a genuinely new interpretation of residual connections: shortcuts would be seen as implicit carriers of second-order curvature information inside first-order training, and DCA would become a constructive recipe for generating new architectures. The paper has tangible strengths: the DC decomposition is explicitly written down, Eq. (19) is a concrete falsifiable formula, and the NegNet experiment is a creative test of the framework. The Section 2 survey of DC decompositions is also useful. However, the load-bearing derivation is not currently sound: the central orthogonality step is unproved, and the derived update is not actually the SGD update of a shortcut network. The significance as a theoretical explanation is therefore not yet established.","major_comments":[{"comment":"The 'blessing from the high dimension' is load-bearing and unproved. Equation (18) is an expectation over independent random unit vectors, but the vectors a and c defined in Eq. (17) are deterministic functions of the weights, activations, and labels. No argument shows that they behave like independent isotropic random vectors; near initialization, or when features are correlated, their inner product can be large. The same objection applies to the claim that the eigenvectors of the indefinite block * in Eqs. (7)-(8) are perpendicular to a and c; * is built from the same d_h F and d_w F blocks as A and C, so there is no statistical independence to appeal to. If * has nonzero projection onto span{a, c}, the exact solution of Eq. (14) contains additional terms and Eq. (19) is not the DCA update. Since Eq. (19) is the only evidence for the paper's central claim, this gap is decisive.","section":"§3, Eq. (18) and footnote 3"},{"comment":"Even if a and c are orthogonal and the * block can be neglected, Eq. (19) gives Delta w proportional to a + beta c with beta = ||a||^2/(2||c||^2 + rho/4), not to a + c. The ResNet gradient in Eq. (6) is a + c, so the derived update is not the shortcut-network SGD gradient unless beta = 1, which is not shown and is generically false because beta depends on the norms of a and c and on rho. Moreover, Eq. (14) is a Newton step for a convex subproblem, not an SGD iteration. The abstract's equality 'DCA + vanilla network = SGD + shortcut network' therefore overstates what is derived; at most one can say that the DCA update has an extra component along c, a qualitative statement rather than the claimed equivalence.","section":"§3, Eq. (19)"},{"comment":"The paper never proves that G2 and H2 are convex, which is required for the DCA framework. The assertion that L'_{-1} can absorb the negative eigenvalues of the * block is made without calculation, and the subsequent instruction that rho should be larger than the largest negative eigenvalue of * is not enough by itself. One must verify that the full Hessian of H2 = 1/2 L'_{-1} + (rho/2)L2 is positive semidefinite and that the Hessian of G2 = L + 1/2 L'_{-1} + (rho/2)L2 is positive definite. The Hessian of L'_{-1} contains both the positive semidefinite term (2C-A)^T d^2_hL'(2C-A) and the indefinite alpha = -1 block, so the needed bound involves singular values of * and the interplay with A and C. Without this verification, the algorithm analyzed is not known to be a DCA and Theorem 2.1 cannot be invoked.","section":"§3, Eq. (12)"},{"comment":"The cross-entropy case, promised in the abstract, rests on the approximation in Eq. (22), but for C > 2 the exact Hessian in Eq. (21) is a sum of pairwise difference terms (w_k-w_l)(w_k-w_l)^T. The outer product d_hL(d_hL)^T is not proportional to that sum in general; the claim that the approximation error is of order e^{E_{i != t}}/sum_k e^{E_k} is asserted without proof. Since this approximation is the basis for reducing the CE case to the rank-two MSE argument, the CE part of the central claim is not established.","section":"§3, CE loss case, Eq. (22)"},{"comment":"The NegNet experiment is presented as empirical support but is a single CIFAR10 comparison with no training hyperparameters, number of runs, or error bars, so the claim that it performs 'on par' with ResNet is not quantified. In addition, Eq. (23) is not a DC decomposition because H3 is admitted to be nonconvex ('quasi'), so the DCA convergence guarantees do not apply. This does not invalidate the architectural idea, but as reported it cannot serve as evidence for the DCA-shortcut equivalence.","section":"§4, Eq. (23) and Figure 2"}],"minor_comments":[{"comment":"The word 'Morden' should be 'Modern'.","section":"Section 5"},{"comment":"The phrase 'Euclidian' should be 'Euclidean'.","section":"Section 3"},{"comment":"The claim that a sufficiently large lambda always exists because 'parameters are often constrained within a limited range' is not a proof, since the training objective is defined over unbounded parameter space.","section":"§3, after Eq. (9)"},{"comment":"The notation for the Hessian of L'_alpha would be clearer if the dimension of d^2_hL' and the meaning of the block matrix in Eq. (11) were specified explicitly.","section":"§3, Eq. (10)-(11)"}],"recommendation":"reject","confidential_remarks":"The paper has a provocative central idea and a worthwhile Section 2 introduction to DCA, but the core derivation depends on an unproved orthogonality assumption, and the derived update does not equal the ResNet SGD gradient. The mismatch with the abstract's slogan is not a presentation issue but a mathematical gap. I would not rule out a future version that reframes the claim as an approximate or empirical connection and provides rigorous conditions, but the current manuscript does not support the stated equivalence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paper has a genuinely interesting idea: a DC decomposition of a vanilla network whose DCA step looks like the gradient of a residual network. The NegNet architecture is a nice by-product. But the central derivation is held together by an unproved high-dimensional orthogonality assumption, so the headline claim 'DCA + vanilla = SGD + shortcut' is not established.\n\nWhat is new: the specific decomposition in Eq. (12) with the counter loss L'_α, the NegNet recurrence, and the framing that shortcuts encode second-order information. The DCA background is solid and the related work is fair. The NegNet experiment, while minimal, is a good idea: it tests the architectural principle rather than the theory.\n\nThe soft spots are load-bearing. The 'blessing from the high dimension' applies random-vector orthogonality (Eq. 18) to deterministic, data-dependent gradient blocks a and c. These are functions of the weights and activations and can be strongly correlated; no argument justifies treating them as independent random vectors. The same leap is made for the eigenvectors of the indefinite block *, which are asserted to be perpendicular to a and c. That is not only unproved, it is implausible, since * is built from the same derivatives. Without those orthogonality claims, the Newton step in Eq. (14) has extra terms and Eq. (19) does not follow. Convexity of G2 and H2 is also asserted rather than proven; the claim that L'_−1 cancels negative eigenvalues is sketched but not computed. The CE case is explicitly approximate, which may be fine, but the error bound is vague. The NegNet experiment has no hyperparameters or training details shown; one clean curve is not enough to demonstrate parity.\n\nI don't think the authors are being sloppy on purpose; the paper reads like an early-stage research idea. But as it stands, the central claim is a conjecture, not a result. The flexibility of DC decompositions makes the existence outcome unsurprising: one can often craft a decomposition to yield a desired update. The value is in the possibility of using DCA to generate architectures, and NegNet is a cute demonstration.\n\nWho is this for? People working on ResNet theory or DCA who want a provocative pointer. I would send it to reviewers, but I'd expect them to ask for a major rewrite, either proving the orthogonality under concrete assumptions or reframing the contribution as a conjecture with a sound empirical study. It is not citable in its current form.","headline":"Interesting idea, unsupported central claim: the DCA-shortcut equivalence rests on an unproved and likely false orthogonality assumption; NegNet is a nice by-product.","tokens_in":10159,"tokens_out":6167,"would_cite":false,"duration_ms":58573,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","90C53"],"pacs":[],"model":"deepseek-v4-flash","headline":"Applying DCA to a vanilla network yields the ResNet gradient, making shortcut connections a consequence of difference-of-convex optimization rather than an ad hoc design.","keywords":["Difference-of-Convex Algorithm","DCA","ResNet","shortcut connections","residual networks","second-order optimization","neural network architecture","NegNet"],"falsifier":"Train a small network with the exact update in Eq. (19) and compare the resulting parameter trajectory, step by step, with the gradient of the corresponding ResNet; a disagreement larger than floating-point tolerance would falsify the identity. A more targeted check is to measure, during real training, the projection of the off-diagonal Hessian block onto the $a$ and $c$ directions—the paper's claim predicts these projections stay near zero at realistic widths.","tokens_in":9138,"feed_emoji":"🧠","tokens_out":8979,"duration_ms":80564,"temperature":0.7,"pith_summary":"This paper aims to show that residual (shortcut) connections in modern neural networks are not merely an information-flow convenience: they are what you get when a plain, shortcut-free network is trained with the Difference-of-Convex Algorithm (DCA). For a specific DC decomposition of the loss, one DCA step produces the same gradient direction that SGD would produce on a ResNet, summarized by the paper as 'DCA + vanilla network = SGD + shortcut network.' The consequence is that residual connections embed second-order curvature information into a first-order update rule, which would explain their stabilizing effect on training. The same DCA viewpoint yields NegNet, an architecture that replaces the residual plus sign with a minus sign and still performs on par with ResNet18 on CIFAR10.","feed_headline":"DCA on shortcut-free networks reproduces ResNet's gradient","feed_subtitle":"Residual connections may be carrying second-order curvature information into a first-order training rule.","key_machinery":"The carrying object is a specific DC decomposition of the training objective, $G_2 = L + \\tfrac{1}{2} L'_{-1} + \\tfrac{\\rho}{2}\\|w\\|^2$ and $H_2 = \\tfrac{1}{2} L'_{-1} + \\tfrac{\\rho}{2}\\|w\\|^2$, where $L'_{-1}$ is the 'counter loss' evaluated at $-h_L + 2h_{L-1}$, chosen to absorb the negative eigenvalues of the Hessian's off-diagonal block. Solving the DCA subproblem by Newton's method produces a Hessian whose essential part is the rank-2 matrix $a a^\\top + \\tfrac{1}{2}(2c-a)(2c-a)^\\top$, and inverting this matrix in the high-dimensional regime gives the closed-form update of Eq. (19). The argument leans on the high-dimensional orthogonality of random unit vectors (Eq. (18)) to discard the off-diagonal block $*$; the paper calls this the 'blessing from the high dimension.'","core_discovery":"The central claim is the identity DCA + vanilla network = SGD + shortcut network. Concretely, the paper derives Eq. (19): the DCA update for a vanilla network is $\\Delta w \\propto a + \\frac{\\|a\\|^2}{2\\|c\\|^2 + \\rho/4} c$, where $a$ is the ordinary vanilla gradient and $c$ is the gradient of the same loss through the shortcut path. Because the update contains a component along $c$, the parameter trajectory matches that of a residual network trained by SGD. The derivation holds for mean-square loss and, with a stated approximation whose error decays during training, for cross-entropy loss. The paper also introduces NegNet, which replaces the identity shortcut with a negative identity; despite contradicting the usual residual intuition, it performs comparably to ResNet18 on CIFAR10, which the authors read as support for the DCA account rather than the information-flow account.","pith_inferences":["Treating the counter-loss coefficient $\\alpha$ as a free parameter generates a one-parameter family of architectures whose updates interpolate between vanilla, ResNet, and NegNet; the paper's derivation suggests all should train similarly in high dimension.","The same derivation could be rerun for transformer blocks, where the residual stream is the analog of $h_{l-1} + F^{l-1}$; if high-dimensional orthogonality holds there, it would give an optimization-based explanation for why attention architectures need residual connections.","Even if the orthogonality claim fails at moderate widths, the qualitative conclusion survives: DCA injects a shortcut-direction component into the update with a modified coefficient, so the recipe of inventing architectures from DC decompositions does not hinge on the strongest form of the assumption."],"forward_implications":["Residual connections can be replaced by a DC decomposition of the loss, turning architectural design into an optimization-design problem: pick a decomposition, derive the update, and read off the architecture needed to compute it with automatic differentiation.","The DCA convergence theorem becomes available for training shortcut networks by SGD, giving a route to global-convergence guarantees under bounded iterates.","NegNet's parity with ResNet on CIFAR10 indicates the information-flow interpretation of shortcuts is not the whole story, since a negative shortcut trains just as well.","For cross-entropy loss the shortcut-gradient identity holds approximately, with an error of order $e^{E_{i\\neq t}}/\\sum_k e^{E_k}$ that shrinks as the model fits."],"supporting_citations":[{"why":"Supplies the DCA algorithm and its convergence theorem, the optimization backbone of the whole derivation.","marker":"Pham & Le Thi, 1997"},{"why":"Provides the DCA convergence analysis the paper cites for the strong DC critical point result.","marker":"Niu, 2022"},{"why":"Establishes the stochastic DCA variants that reduce to SGD, making the vanilla-network-plus-DCA claim concrete.","marker":"Le Thi et al., 2022a;b"},{"why":"Introduces the ResNet/shortcut architecture whose gradient the paper reproduces from DCA on a vanilla network.","marker":"He et al., 2015"},{"why":"Analyzes identity mappings and shortcut gradient flow, the prior interpretation the paper reinterprets through second-order information.","marker":"He et al., 2016"}],"fun_headline_variants":["DCA on plain nets equals SGD on residual nets","Why shortcuts work: DCA on vanilla nets replicates ResNet","DCA with vanilla nets yields ResNet's training dynamics","Shortcut gradient from DCA on vanilla nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The derivation collapses if the neglected off-diagonal Hessian block is not confined to directions perpendicular to both the vanilla gradient $a$ and the shortcut gradient $c$; the paper justifies this by treating $a$ and $c$ as effectively random, although they are deterministic and data-dependent.","fun_headline_variants_meta":{"raw":{"variants":["DCA on plain nets equals SGD on residual nets","Why shortcuts work: DCA on vanilla nets replicates ResNet","DCA with vanilla nets yields ResNet's training dynamics","Shortcut gradient from DCA on vanilla nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000856,"raw_usage":{"total_tokens":3685,"prompt_tokens":878,"completion_tokens":2807,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2742}},"tokens_in":494,"tokens_out":2807,"duration_ms":23845,"temperature":1.0,"reasoning_tokens":2742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:39:13.111284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small network with the exact update in Eq. (19) and compare the resulting parameter trajectory, step by step, with the gradient of the corresponding ResNet; a disagreement larger than floating-point tolerance would falsify the identity. A more targeted check is to measure, during real training, the projection of the off-diagonal Hessian block onto the $a$ and $c$ directions—the paper's claim predicts these projections stay near zero at realistic widths.","supporting_citations":[],"review_version":1}