REVIEW 4 major objections 3 minor 3 cited by
Automatic Differentiation for Complex Valued SVD
T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper derives the reverse-mode automatic differentiation formula for complex-valued SVD, adding a fourth term with no real-valued counterpart that couples U and V in the gradient.
desk verdict A correct and useful complex-SVD backprop formula, but the paper overstates its domain by omitting the distinct-nonzero-singular-value condition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the Wirtinger calculus setup (derivatives with respect to z and z*, with gradient \bar A = 2 ∂L/∂A*), combined with the differential relations dC = U†dU, dD = V†dV, and the algebraic identities (8)-(9) for elementwise products. The proof resolves dA into four pieces — diagonal of S, off-diagonal of C and D, and the pure-imaginary diagonals of C and D — and uses gauge invariance, equation (31), to show that the naive diagonal contributions cancel, leaving the new $S^{{-1}}$ term. The matrix F = 1/($s_j^{2}$ - $s_i^{2}$) carries the coupling between different singular vectors and is the same object that appears in the real-valued SVD gradient.
What would settle it
Take a random 2x2 complex matrix with two distinct nonzero singular values and the gauge-invariant loss L = Re(U00 V*_00), compute the central finite-difference gradient of L with respect to the real and imaginary parts of A, and compare with Eq. (1). Any mismatch for such a matrix would refute the formula; this is the kind of test the paper itself recommends.
Extended reading notes
Core claim
The paper's claim is that reverse-mode differentiation of a complex-valued SVD, A = U S V†, for a real gauge-invariant loss L is given by a closed-form expression with four contributions. The first three contributions are the complex analogues of the real SVD gradient, built from F_ij = 1/($s_j^{2}$ - $s_i^{2}$) for i≠j and zero on the diagonal. The genuinely complex addition is \bar A_d = (1/2) U $S^{{-1}}$(L† - L) V†, where L = I ∘ (V† \bar V), a term required because the diagonal blocks of the differentials dC = U†dU and dD = V†dV are pure imaginary rather than zero. The derivation uses gauge invariance of the loss under phase transformations of U and V — equation (31) — to cancel the ambiguous diagonal contributions, and it extends to rectangular matrices by adding two standard projection terms. The upshot: for any real loss that is invariant under the SVD phase gauge and depends on both U and V, the gradient of the loss with respect to the input matrix can be computed exactly.
Load-bearing premise
The formula assumes the loss is real and invariant under the phase-gauge freedom of the SVD (multiplying U and V by the same diagonal phase matrix), and that all singular values are distinct and nonzero; if any of these fails, the gradient either depends on the arbitrary gauge choice or the expression contains divisions by zero.
Editorial extensions
If this is right
- Complex SVD joins the list of primitives that can be differentiated in reverse mode, closing the explicit gap noted in prior work on complex-valued automatic differentiation.
- Tensor network algorithms that use complex SVD for truncation or canonicalization can now be optimized end-to-end by gradient descent on complex tensors.
- Any automatic differentiation library can implement Eq. (1) directly; the new fourth term is zero when the loss depends on only U or only V, but indispensable when the loss couples both.
- The rectangular extension in Eq. (36) makes the formula applicable to non-square matrices, covering typical uses of SVD in machine learning and physics.
- The derivation's gauge-freedom argument shows why a correct complex SVD backprop must respect phase gauge invariance, not merely holomorphic derivatives.
Reading between the lines
- The appearance of 1/(s_j^2 - s_i^2) means gradients amplify strongly when singular values are close; near-degenerate spectra in tensor network optimization could produce large or unstable gradient updates, so practitioners may need to regularize or avoid SVD layers at near-degenerate points.
- Because the new term vanishes for losses that depend only on U or only on V, existing AD libraries with a complex SVD might have passed correctness checks without exercising the true complex contribution; a correct test must couple U and V, as the paper notes.
- The same differential-geometric treatment of the SVD fiber (the phase gauge) could be adapted to derive reverse-mode formulas for other factorizations with gauge freedom, such as the eigenvalue decomposition of non-Hermitian matrices or polar decomposition.
- The gauge-invariance condition (31) effectively tells us that the gradient of a gauge-invariant loss is a well-defined function on the quotient space of SVD factors, which may matter for optimization landscapes of tensor networks.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper derives a reverse-mode backpropagation formula for the complex-valued singular value decomposition (SVD), for the case of a real, gauge-invariant scalar loss function L depending on the SVD factors. The main result, Eq. (1), expresses the conjugate gradient of L with respect to the input matrix A in terms of U, S, V and the reverse-mode signals for U, S, V. The first three terms parallel the real SVD derivation; the new contribution, Eq. (2), is a term proportional to U S^{-1}(L^\dagger - L) V^\dagger that arises from the imaginary diagonal parts of dC = U^\dagger dU and dD = V^\dagger dV. The derivation is a direct chain-rule computation using the gauge invariance of the loss under diagonal phase transformations, and the paper also sketches the rectangular extension via two additional terms.
Significance. If valid, the result fills a genuine gap in complex-valued automatic differentiation, with direct applications to tensor-network algorithms where complex SVD is a primitive operation. The paper is explicit and does not rely on fitted parameters or curve fitting: the missing term is obtained from the stated gauge-invariance condition, and the derivation is internally consistent under the implicit assumptions of distinct nonzero singular values. A GitHub implementation and a TensorFlow merge are mentioned, which strengthens the practical relevance. The main caveat is that the domain of validity is narrower than the unqualified statement of the result, and the paper would benefit from stating that restriction clearly.
major comments (4)
- [Results, Eq. (1)] The central formula is stated without the domain restriction required by its own ingredients: F_{ij}=1/(s_j^2-s_i^2) and S^{-1} are undefined when any two singular values coincide or when any singular value is zero. This is not merely a numerical caveat; the derivation's perturbation equation (13) has no first-order solution at a repeated singular value. For example, with A=diag(1,1,2), U=V=I, and an infinitesimal dA with dA_{01}=1 and dA_{10}=0, Eq. (13) gives dP_{01}=dC_{01}-dD_{01}=1 and dP_{10}=-conj(dC_{01})+conj(dD_{01})=0, which are contradictory because dC and dD are skew-Hermitian. The paper must state explicitly that Eq. (1) is valid only for distinct, nonzero singular values, and it should discuss the near-degenerate case where the gradient is large or undefined.
- [Discussions, Eq. (36)] The claimed extension to rectangular matrices is not fully specified. In a full SVD of an m by n matrix A, the matrix S is m by n and S^{-1} is undefined; the extension (36) is only meaningful if one adopts a thin SVD in which S is square, or if S^{-1} is defined as a pseudo-inverse. The paper should state the dimensions of U, S, V in Eq. (36) and justify how the square formula (1) is applied in that case.
- [Notation, Eqs. (7), (35), and (1)] The notation in Eq. (7) appears to identify \bar{A}=∂L/∂A with \bar{A}=2∂L/∂A^* = ∇_A L, but these are different objects. The derivation up to Eq. (35) is carried out for derivatives, while the final formula (1) is stated for gradients, and the conversion is only alluded to with 'Note how J,K,L changed accordingly'. Please write out this conversion explicitly so that the relation between Eq. (35) and Eq. (1) can be verified directly.
- [Gauge freedom, Eq. (31)] The gauge-invariance condition (31) is derived for diagonal phase transformations Λ only. If degenerate singular values were included in the domain, the gauge group would be larger (unitary rotations mixing the degenerate subspace), so Eq. (31) would not be the full condition. Since the formula is only valid for simple singular values, this issue is already covered by the domain restriction, but the paper should state that the derivation assumes simple singular values and that the diagonal-phase gauge is the only relevant gauge freedom in that case.
minor comments (3)
- [Throughout] There are several typos: 'Namley' after Eq. (21), 'imiginary' after Eq. (12), and 'Beijin g' in the affiliation. These should be corrected.
- [Eq. (1)] The symbol L is overloaded: it denotes the loss function and also the matrix I∘(V^† \bar{V}) in the definition after Eq. (1). Rename the matrix (e.g., M) to avoid confusion.
- [Abstract and Introduction] The paper would benefit from a short numerical verification, such as a finite-difference check of Eq. (1) on a small non-diagonal example with a gauge-invariant loss depending on both U and V; this would guard against sign and transpose convention errors.
Circularity Check
No circularity: Eq. (1) is derived from a self-contained chain-rule computation with no fitted inputs and no load-bearing self-citation.
full rationale
The manuscript derives Eq. (1) rather than assuming it. The chain-rule expression (18) is expanded via dP = U†dAV and the algebraic identities (14)–(17), which follow from unitarity and the skew-Hermitian character of dC and dD. Each partial gradient in (22), (25), and (26) is obtained by direct manipulation of traces with the explicitly defined matrix F, whose entries are 1/(s_j^2 − s_i^2) off-diagonal and 0 on the diagonal. The new term (35) is obtained from Eq. (15) and the gauge-invariance identity (31). Eq. (31) is itself derived, not postulated as the target: differentiating L(UΛ,S,VΛ) = L(U,S,V) with respect to the diagonal phase Λ at Λ = I, and using dΛ† = −dΛ, forces (31) as a consequence of the stated assumption that L is phase-gauge invariant. This is an input assumption about the loss function, not a restatement of the fourth term in the backpropagation formula. No parameter is fitted, no external uniqueness theorem is invoked, and no prior work by the authors is load-bearing; the cited works are used only as context or as an implementation link. The practical restriction to distinct nonzero singular values, visible through F_ij and S^{-1}, is a domain-of-validity concern rather than a circularity: at degeneracy the perturbation algebra itself fails, but that does not make the derivation of Eq. (1) from the stated assumptions circular.
Assumptions & free parameters
assumptions (5)
- standard math SVD A=USV† exists with unitary U,V and positive diagonal S.
- domain assumption Singular values are distinct and nonzero, so F_ij = 1/(s_j^2 − s_i^2) and S−1 are finite.
- domain assumption The loss L is real-valued and invariant under the SVD phase gauge U → UΛ, V → VΛ.
- standard math Standard Wirtinger calculus applies: for real L, the gradient is 2∂L/∂z*, and dL = Tr[ĵ^T dA + c.c.].
- standard math dC = U†dU and dD = V†dV are anti-Hermitian: dC = −dC† and dD = −dD†.
Cite this review
Pith. "Pith review of Automatic Differentiation for Complex Valued SVD." pith.science (2026). https://pith.science/paper/VNVDOKCW
@misc{pith2026190902659,
author = {Pith},
title = {Pith review of: Automatic Differentiation for Complex Valued SVD},
year = {2026},
howpublished = {\url{https://pith.science/paper/VNVDOKCW}},
note = {Machine review of arXiv:1909.02659}
}
read the original abstract
In this note, we report the back propagation formula for complex valued singular value decompositions (SVD). This formula is an important ingredient for a complete automatic differentiation(AD) infrastructure in terms of complex numbers, and it is also the key to understand and utilize AD in tensor networks.
Forward citations
Cited by 3 Pith papers
-
Optimizing Quantum Photonic Integrated Circuits using Differentiable Tensor Networks
Gradient-based optimization of quantum photonic circuits is achieved via differentiable tensor networks that model nonlinear unitary gates and stochastic losses at low photon numbers.
-
Variational Quantum Simulation of Anyonic Chains
RSOS anyonic chains can be encoded into qubit registers with ceil(log2 p) qubits per site, and variational Euler-Cartan circuits prepare their critical ground states with energy errors below 0.5% for A_p chains with p...
-
Enhancing Robotic System Robustness via Lyapunov Exponent-Based Optimization
A differentiable 'sum of Lyapunov exponents' score is used as a robustness objective to co-optimize robot hardware and control policies in simulation.
Reference graph
Works this paper leans on
-
[8]
Jin-Guo Liu, Linear Algebra Autodiff (complex valued) (2019)
work page 2019
-
[1]
(2) This term is new and has no counterpart in real valued SVD
as ¯Ad = 1 2 U S−1(L† − L)V †. (2) This term is new and has no counterpart in real valued SVD. DERIV ATION Basics on complex calculus and some notations: 2 The partial derivatives for function f (z) where z = x + iy is defined as: ∂f ∂z = 1 2 ( ∂f ∂x − i ∂f ∂y ), ∂f ∂z ∗ = 1 2 ( ∂f ∂x + i ∂f ∂y ). (3) Such form of derivative is justified as d f= ∂f ∂z dz + ...
-
[2]
Hai-Jun Liao, Jin-Guo Liu, Lei Wang, and Tao Xiang, arXiv :1903.09650
arXiv 1903
-
[3]
Michael Bartholomew-Biggs, Steven Brown, Bruce Christ ianson, and Laurence Dixon, J. Comput. Appl. Math. 124, 1711 90 (2000)
work page 2000
-
[4]
Mike Giles, An extended collection of matrix derivative results for forward and reverse mode algorithmic differenti ation, Tech. Rep. (2008)
work page 2008
-
[5]
Wavefunction positivization via automatic differentiation
Giacomo Torlai, Juan Carrasquilla, Matthew T. Fishman, Roger G. Melko, and Matthew P. A. Fisher, arXiv:1906.04654
work page Pith review arXiv 1906
-
[6]
Matthias Seeger, Asmus Hetzel, Zhenwen Dai, Eric Meissn er, and Neil D. Lawrence, arXiv:1710.08717
-
[7]
James Townsend, Differentiating the Singular Value Deco mposition, Tech. Rep. (2016)
work page 2016
Show all 12 references
-
[9]
Claudius Hubig, arXiv:1907.13422
1907 arXiv
-
[10]
See https://github.com/Zhouquan-Wan/SVD_autodiff for a possible implementation in TensorFlow and the correct back propagation formula for complex valued SVD is merged by TensorFlow at https://github.com/tensorflow/tensorflow/pull/32226
-
[11]
Hunger, An introduction to complex differentials and c omplex differentiability, Tech
R. Hunger, An introduction to complex differentials and c omplex differentiability, Tech. Rep. (2007)
2007
-
[31]
] = Tr[( ¯U T U + ¯V T V − c.c
by noting I ◦ dC is pure imaginary, Tr[(¯U T U + ¯V T V )(I ◦ dC) + c.c. ] = Tr[( ¯U T U + ¯V T V − c.c. )(I ◦ dC)] = Tr[( ¯U T U + ¯V T V − c.c. ) ◦ I)dC] = 0 . (33) The remaining one can be replaced by ( 15): Tr[( ¯Aud + ¯Avd)T dA + c.c. ] = 1 2 Tr[¯V T V (I ◦ (dP † − dP )S−...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.