REVIEW 2 major objections 5 minor 5 references
Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?
T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Even with every data point on the unit sphere, gradient descent can converge to a cycle instead of the minimizer for non-separable logistic regression in sufficiently high dimensions.
desk verdict The 1D theorem is solid and new; the high-dimensional counterexample is probably right but the proof skips a stability-transfer step that needs a short repair. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two mechanisms carry the argument. The first is the 'schmearing' embedding: each base example x_i is replaced by 2(d−2) unit-norm copies of the form (x_i, ±s_i e_j) with s_i = sqrt(1−∥x_i∥²); this pads the data onto the unit sphere while leaving the loss symmetric in the extra coordinates, so the Hessian at the solution is block diagonal with top block ∇²L_b(w*_b) and bottom block (c_b/(d−2))I, and taking d large enough keeps λ equal to the base curvature and preserves the base two-dimensional trajectory. The second is the average-derivative ratio R(w) = (σ(w)−σ(w*))/(w−w*), which the one-dimensional proof uses to show one-step contraction from the right, a double-crossing property, and two-step contraction with rate factor 1−γ(2−γ)R(w_t)²/σ′(w*)²; the inequality tanh(ε) < ε supplies the positivity needed for the final bound.
What would settle it
Compute the full Jacobian of the period-13 return map for the d = 5000, γ = 0.95 construction reported in the paper; if any eigenvalue of that Jacobian lies outside the unit circle, the embedded cycle is not the stable attractor the proof requires, and the asserted stability transfer fails. A complementary check is to initialize the last d − 2 coordinates at small nonzero values and observe whether they decay to zero over many periods; any growth would indicate the transverse direction is not contracting.
Extended reading notes
Core claim
The central claim is Theorem 2: normalization to the unit sphere does not prevent gradient descent from cycling in higher dimensions. For any γ < 2, the paper constructs a d-dimensional non-separable logistic regression problem with all ∥x_i∥ = 1 and a trajectory that converges to a cycle of period k > 1 under η = γ/λ. The construction starts from a two-dimensional cycling dataset and embeds it into d dimensions by duplicating every example with extra coordinates chosen so that each copy has norm exactly one; the Hessian at the new minimizer is block diagonal, and taking d large enough keeps the top block's largest eigenvalue equal to λ, so the original cycle is reproduced in the first two coordinates. Alongside this, Theorem 1 shows that in d = 1 the spherical restriction is genuinely useful: it extends the guaranteed convergence range from γ ≤ 1 to all γ < 2, with oscillatory convergence for γ > 1. The paper also proves a scaling invariance: bounded data, ∥x_i∥ ≤ 1, inherits all cycling counterexamples, so boundedness alone is not sufficient.
Load-bearing premise
The proof of the high-dimensional counterexample assumes, without a fully demonstrated argument, that the stable cycle of the two-dimensional base dataset remains attracting after the 'schmearing' embedding into the sphere; the last step of Section 3 asserts the contraction in the padded coordinates from the block-diagonal Hessian rather than proving it.
Editorial extensions
If this is right
- In one dimension with unit-norm features, global convergence is guaranteed for every η = γ/λ with γ < 2, extending the earlier γ ≤ 1 guarantee; convergence is oscillatory for γ > 1 and slows as γ approaches 2.
- In high enough dimensions, normalizing data to the unit sphere is not a sufficient condition for global convergence: for every γ < 2 some unit-sphere dataset admits a stable cycle.
- Because uniform scaling preserves gradient-descent trajectories at the same γ, restricting data to a ball rather than the sphere cannot remove cycling counterexamples.
- The dimension of the constructed counterexample is finite but can be large; the paper's construction needs d large enough that λ_b ≥ c_b/(d−2).
- Any sufficient condition for all step sizes below the stability threshold must constrain the dataset as a whole, not just the magnitude of each example.
Reading between the lines
- The paper leaves open whether cycling occurs in every dimension d > 1; a natural next test is the minimal dimension of the construction and whether d = 2 or d = 3 already admit unit-sphere cycles for every γ < 2.
- Because the one-dimensional two-step rate is bounded by 1 − (2−γ)/γ, the practical benefit of pushing γ toward 2 may be offset by arbitrarily slow convergence; one could optimize the step size for a fixed accuracy target.
- The same spherical-embedding mechanism could transport stable cycles from other base losses or base dynamics, suggesting that the obstruction is geometric rather than specific to logistic regression.
- If the asserted stability transfer under the embedding is made fully rigorous, the construction would yield explicit high-dimensional unit-sphere datasets with prescribed cycle periods, which could serve as benchmarks for detecting cycling behavior in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies gradient descent (GD) on logistic regression with step size eta = gamma/lambda, where lambda is the largest eigenvalue of the Hessian at the minimizer, under the restriction that all data points lie on the unit sphere. The authors prove that in one dimension, for data points in {-1,1}, GD converges globally for every gamma < 2, extending the previously known gamma <= 1 threshold; the proof gives an explicit two-step linear contraction rate in the oscillatory regime. In higher dimensions they construct, for every gamma < 2, a unit-sphere dataset by 'schmearing' a two-dimensional counterexample from Meng et al. (2024), and they claim that GD has a periodic orbit of period greater than one. A numerical experiment in d = 5000 illustrates a period-13 cycle. The paper concludes that the unit-sphere restriction guarantees global convergence only in one dimension, not in general.
Significance. If the results are correct, they settle a natural question: equal-magnitude data does not restore global convergence below the stability threshold in dimension greater than one, while in one dimension it does. The 1D theorem is the main self-contained contribution; its contraction lemmas are carefully proved and the rate estimate is explicit. The high-dimensional counterexample is an elegant reduction to prior work and is supported by a numerical demonstration. The main weakness is a genuine gap in the proof of Theorem 2: the transfer of stability from the base cycle to the embedded cycle is asserted rather than demonstrated, and the proof as written is incorrect in a degenerate case. However, the central construction can be repaired with a short invariant-subspace argument, so the paper's main claim is defensible.
major comments (2)
- [Section 3 (Theorem 2)] The proof asserts that the embedded process follows the same cycle as the base dataset in the first two coordinates and has zeros in the remaining coordinates, but the only computation performed is at the minimizer w*. One needs to verify at an arbitrary point (u,0) that the full gradient decomposes as nabla_{1:2}L(u,0) = nabla L_b(u) and nabla_{3:d}L(u,0) = 0; this follows by pairing the +/- s_i e_j copies, but it is not written. Without this step, the existence of the cycle in the full map is not established.
- [Section 3, last paragraph] The stability transfer is asserted rather than proved. At a cycle point (u_t,0) the Jacobian of the full GD map is block diagonal, with transverse block (1 - eta c(u_t)/(d-2))I, where c(u_t) = (1/n_b) sum_i ell''(-u_t^T x_i) s_i^2. Equation (15) only controls c_b/(d-2) at the minimizer and gives no control over the product prod_t (1 - eta c(u_t)/(d-2)); if all s_i = 0, the transverse multiplier is exactly 1, so a nearby initialization with nonzero last d-2 coordinates does not converge to the cycle. The theorem's conclusion can nevertheless be repaired: either use the invariant subspace {w_{3..d}=0}, on which the restricted update is exactly the base 2D GD map and a base-basin initialization converges, or, when at least one s_i > 0, choose d large enough so that sup_t eta c(u_t)/(d-2) < 2 (possible because c(u_t) <= 1/4) to obtain genuine transverse contraction. This repair should replace the final paragraph.
minor comments (5)
- [Section 2, Lemma 1] The statement 'for any w in [w*, infinity)' includes w = w*, where the claimed strict inequality |T(w) - w*| < |w - w*| reads 0 < 0; the lemma should be restricted to w > w*.
- [Appendix A, Lemma 3] In the final displayed inequality of Lemma 3, the reference to 'R(2w* - wt) > R(wt) from Lemma 2' uses the wrong index; the factor needing a lower bound is R(2w* - w_{t+1}), and the bound follows from monotonicity of R on the positive axis rather than directly from Lemma 2.
- [Table 1] The row 'd>1 Counterexamples exist' can be read as asserting a counterexample in every dimension d>1; since Theorem 2 only constructs some dimension d>1 (and the Discussion acknowledges this), the table should say 'for some d>1'.
- [Discussion] The sentence 'Notably, this dimension may need to be fairly large, as lambda_b.' is incomplete; the intended dependence on lambda_b should be spelled out.
- [Fact 1 / Section 3] Fact 1 is stated for datasets with all labels equal to 1; since logistic-regression labels can be absorbed into the feature vectors by replacing x_i with y_i x_i, this reduction should be mentioned when Fact 1 is invoked for the base counterexample.
Circularity Check
No significant circularity: the 1D theorem is self-contained and the high-dimensional counterexample is a transparent, legitimate reduction to the same authors' prior 2D counterexample; the only flagged issue is an asserted stability transfer in Theorem 2's proof, an omitted proof (repairable via the invariant zero-transverse subspace), not a circular step.
full rationale
Derivation-chain walk. (1) Theorem 1 (global convergence for gamma<2 when d=1 and ||x_i||=1) is self-contained: the gamma<=1 case cites Meng et al. (2024, Theorem 1), a standard prior result proved for unconstrained data, and the gamma in (1,2) case is proved from scratch in Appendix A through Lemmas 1-3 (one-step contraction, double crossing, two-step contraction, Eq. (8)) with no quantity fitted or defined in terms of the conclusion. (2) Fact 1 (scale-invariance of cycles under uniform scaling) is proved in Appendix A and is not circular. (3) Theorem 2 is a reduction: 'Theorem 3 in Meng et al. (2024) showed that without any data restrictions, for any gamma<2, one can construct a 2-dimensional counterexample logistic regression problem on which a cycle exists.' This base result is an independent proved theorem, parameter-free, with assumptions that do not include the sphere target, so per the reviewing rules it is real evidence and not a circular premise; the schmearing embedding (11) is a new construction that enforces the unit-sphere condition, and Eq. (15) is a legitimate existence choice of dimension so that lambda=lambda_b, not a fitted input called a prediction. (4) Flagged soft spot, Section 3, last paragraph of the proof of Theorem 2: 'Since the cycle in the base dataset is stable (proven in Meng et al. (2024)), as long as we initialize close enough to the cycle in the first two dimensions and close enough to 0 in the remaining d-2 dimensions, we can converge to it.' This stability transfer is asserted, not proven: Eq. (13) gives the transverse block (c_b/(d-2))I only at w*, and Eq. (15) does not control the transverse multipliers prod_t(1-eta c(u_t)/(d-2)) along the cycle, which equal 1 when all s_i=0. This is an omitted proof, not a circular reduction: the conclusion survives because {w_{3..d}=0} is invariant and the restricted map coincides with the base GD map, so a base-basin initialization yields the embedded cycle. No fitted inputs renamed as predictions, no uniqueness theorems imported, no ansatz smuggled via citation. Verdict: no significant circularity; score 2 registers the load-bearing but legitimate self-citation and the unproven stability-transfer claim.
Assumptions & free parameters
assumptions (3)
- standard math The sigmoid function sigma has sigma' symmetric about 0, sigma' decreasing on positive reals, and tanh(epsilon) < epsilon for epsilon > 0.
- domain assumption For every gamma < 2 there exists a 2D non-separable logistic regression dataset, scaled to norm at most 1, with a stable cycle under GD at step size eta = gamma / lambda (Theorem 3 of Meng et al. 2024).
- ad hoc to paper The stable cycle of the base dataset remains stable after the schmearing embedding, with transverse coordinates contracting.
Cite this review
Pith. "Pith review of Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?." pith.science (2026). https://pith.science/paper/PVBBSGCR
@misc{pith2026250711228,
author = {Pith},
title = {Pith review of: Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?},
year = {2026},
howpublished = {\url{https://pith.science/paper/PVBBSGCR}},
note = {Machine review of arXiv:2507.11228}
}
abstract
Gradient descent (GD) on logistic regression has many fascinating properties. When the dataset is linearly separable, it is known that the iterates converge in direction to the maximum-margin separator regardless of how large the step size is. In the non-separable case, however, it has been shown that GD can exhibit a cycling behaviour even when the step sizes is still below the stability threshold $2/\lambda$, where $\lambda$ is the largest eigenvalue of the Hessian at the solution. This short paper explores whether restricting the data to have equal magnitude is a sufficient condition for global convergence, under any step size below the stability threshold. We prove that this is true in a one dimensional space, but in higher dimensions cycling behaviour can still occur. We hope to inspire further studies on quantifying how common these cycles are in realistic datasets, as well as finding sufficient conditions to guarantee global convergence with large step sizes.
Figures
Reference graph
Works this paper leans on
-
[1]
Gradient descent on logistic regression with non-separable data and large step sizes
Si Yi Meng, Antonio Orvieto, Daniel Yiming Cao, and Christopher De Sa. Gradient descent on logistic regression with non-separable data and large step sizes. arXiv:2406.05033, 2024
arXiv 2024
-
[2]
Lectures on Convex Optimization , volume 137
Yurii Nesterov. Lectures on Convex Optimization , volume 137. Springer, 2018
work page 2018
-
[3]
The Implicit Bias of Gradient Descent on Separable Data
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The Implicit Bias of Gradient Descent on Separable Data . Journal of Machine Learning Research, 19: 0 70:1--70:57, 2018
work page 2018
-
[4]
Jingfeng Wu, Vladimir Braverman, and Jason D. Lee. Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability . arXiv:2305.11788, 2023
work page Pith review arXiv 2023
-
[5]
Bartlett, Matus Telgarsky, and Bin Yu
Jingfeng Wu, Peter L. Bartlett, Matus Telgarsky, and Bin Yu. Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency . In The Thirty Seventh Annual Conference on Learning Theory, COLT , volume 247 of Proceedings of Machine Learning Research, pages 5019--5073. PMLR , 2024
work page 2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.