{"id":"e1fa9b15-61e6-4cf0-a34d-084b8bbd9084","arxiv_id":"2607.25299","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A retraction-free Stiefel manifold optimization algorithm with a fixed penalty parameter is proposed and applied to LoRA fine-tuning, claiming faster convergence and better downstream performance.","lead":"The authors propose a new retraction-free optimization method for the Stiefel manifold with a fixed penalty parameter, and apply it to LoRA fine-tuning of large language models. The method reportedly accelerates training and improves performance, but the theoretical proof contains a suspicious step and the practical algorithm relies on tuned hyperparameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cited false identity in Lemma 5 is actually true (tangent η at Q satisfies ⟨η,QN⟩=0 for symmetric N); the real load-bearing gap is the unstated almost-sure bound on ∥g_k∥ in Theorem 1's stochastic invariance proof, and Algorithm 2 uses a non-delayed AdamW that Appendix B explicitly excludes.","rationale":"The reader's central objection—that Lemma 5 relies on a false equality ⟨η,Xbar N⟩=0—is itself incorrect: for a tangent vector η at Q∈St(d,r), Q^Tη+η^TQ=0, and for symmetric N, ⟨η,QN⟩=tr(Nη^TQ)=tr(N sym(η^TQ))=0. Thus the proof element singled out by the reader actually holds. However, the stochastic convergence theorem has a genuine unstated assumption: the invariance argument uses a pointwise bound ∥g_k∥≤\\hat D_f, whereas Assumption 1 only provides bounded variance. A simple example shows δ can exceed 1/8 under the allowed step size, so Lemma 5 is not applicable and the O(logK/√K) stochastic guarantee is not established as written. In addition, the paper itself (Appendix B, Remark B.1) excludes the non-delayed Adam preconditioner used in Algorithm 2, and the experiments use tuned μ values, so the Manifold-LoRA algorithm actually evaluated is not covered by the provided convergence theory. These issues are fixable but require revision, so the appropriate verdict is conditional rather than an outright reject based on the alleged false identity.","tokens_in":30595,"tokens_out":30511,"duration_ms":273027,"concrete_test":"Test the invariance step: construct two smooth functions with ∇f1=10e and ∇f2=−8e on \\bar U(1/8) (so ∇f=e and \\hat D_f=1), set μ=1/3, α=1/120, δ_1=1/8, and take g_1=10e (the one-sample mini-batch case). Lemma 4 gives δ_2≤√(2/3)/8 + (5/2)(1/120)(10) ≈ 0.310 > 1/8, so X_2 can leave the tube; if this happens, Lemma 5's bound cannot be used and Theorem 1's stochastic claim is unproven.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Reader's alleged Lemma 5 error does not land. For Q∈St(d,r), η∈T_Q St means Q^Tη+η^TQ=0. For symmetric N, ⟨η,QN⟩=tr(Nη^TQ)=tr(N sym(η^TQ))=0. The same argument validates the normal-space orthogonality used in Lemma 5 and Appendix B. Hence the main REJECT rationale is not supported.\n\nThe genuine concern is in the proof of Theorem 1 (Appendix A). To keep all iterates in \\bar U(1/8), the proof uses the induction δ_{k+1}≤√(2/3)δ_k+(5/2)α_k∥g_k∥ and then asserts this is ≤1/8 with α_k≤1/(120 \\hat D_f). This step silently replaces ∥g_k∥ by \\hat D_f. Assumption 1 only gives E∥g_k−∇f(X_k)∥²≤σ²; it does not give an almost-sure bound. Example: let ∇f_1=10e, ∇f_2=−8e, f=(f1+f2)/2, so ∇f=e and \\hat D_f=1; a one-sample mini-batch chooses g_k=10e with probability 1/2. Then (5/2)α∥g∥=(5/2)(1/120)(10)=0.208, so δ_2 can be ≈0.310>1/8. Thus the iterates can leave the tube where Lemma 5's constants apply, and the stochastic convergence theorem is not proven as stated.\n\nA second, self-identified gap: Appendix B's Adam-style theorem uses a delayed preconditioner and Remark B.1 states the non-delayed version 'is not covered by the conditional-independence argument above.' Algorithm 2, however, uses standard non-delayed v_k and clips α(C_k), and the experiments use tuned μ (0.8–1.2), not μ=1/3. So the convergence guarantees do not apply to the Manifold-LoRA variant actually evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retraction-free, penalty-parameter-free landing method for optimization over the Stiefel manifold. The key theoretical claims are: (i) an explicit penalty parameter mu=1/3 is sufficient; (ii) under a tube initialization dist(X_1,St)<=1/8, the iterates of Algorithm 1 with projected stochastic gradients and a constant penalty step mu=1/3 converge in deterministic and stochastic settings with the best-known iteration complexities (Theorem 1); (iii) a two-scale step-size scheme yields O(log K/K) constraint-violation decay. The paper then reformulates LoRA fine-tuning as optimization over a product manifold and Euclidean space (with B constrained to Stiefel or Oblique manifolds), proposes Algorithm 2 ('Manifold-LoRA') with SGD or AdamW updates and per-iteration clipping of the step size, and reports experiments on GLUE, SQuAD, E2E, and LLaMA-scale benchmarks. Appendices provide proofs, including a delayed-preconditioner Adam-style convergence theorem (Theorem B.1).","tokens_in":31112,"tokens_out":3167,"duration_ms":27566,"significance":"If the convergence theory were fully sound, the explicit penalty parameter and two-scale step-size analysis would be a useful advance in retraction-free optimization, and the LoRA reformulation is a reasonable and interesting idea with supporting experimental gains. The paper also makes explicit comparisons to prior landing methods and identifies a self-contained gap in the Adam analysis (Remark B.1), which is commendable. However, the central stochastic convergence proof contains a load-bearing gap that is not fixed within the current manuscript, and the experiments use tuned hyperparameters and a non-delayed AdamW variant that the theory explicitly excludes. The theoretical contribution is therefore not established as stated, and the experimental evaluation does not directly validate the theoretical claims.","major_comments":[{"comment":"The initialization condition ||X_1 - bar X_1|| <= 1/8 is essential for the RSI and all subsequent lemmas, but the paper does not explain how LoRA initializations (e.g., random A and B, or B fixed at initialization as in some LoRA implementations) guarantee this condition. For the practical algorithm, B_0 is typically initialized randomly and the constraint is not satisfied initially. This is an unaddressed gap between theory and practice. The paper should either provide a practical initialization scheme or state the limitation explicitly.","section":"Section IV.A, Lemmas 1-2"}],"minor_comments":[{"comment":"Assumption 1 states gradients are Lipschitz over the convex hull of bar U(1/8), but Lemma 3 uses L and D_f over the manifold; the relation between these constants should be stated more carefully. Also, the notation X_1 vs X_0 is inconsistent between Algorithm 1 and Theorem 1 (Algorithm 1 initializes with X_0, Theorem 1 assumes ||X_1 - bar X_1|| <= 1/8).","section":"Notation and Assumption 1"},{"comment":"The proof of Lemma 5 in the manuscript is compressed; in particular the constant 7 in front of the variance term and the step where the term involving mu D_f is absorbed into the constant 4 hat D_f + 9 hat L^2 + 8L + 3 are not shown. Expanding the algebra would help verifiability.","section":"Proof of Lemma 5"},{"comment":"The experimental section reports results on many tasks, but the number of random seeds for the GLUE tables and the LLaMA-scale tables is not stated; Figure 3 mentions five seeds. Also, Tables I-II compare against 'LoRA + Stiefel' from [27]; the corresponding baseline setup should be described precisely to allow fair comparison.","section":"Section V"},{"comment":"The abstract claims 'best-known iteration complexities' — this should be qualified relative to the specific setting (e.g., stochastic nonconvex with bounded variance) and compared with the most relevant prior results beyond [11,12].","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The reader's proposed false identity in Lemma 5 is not the real problem; the identity <eta, Q N> = 0 for tangent eta and symmetric N is correct. The genuine load-bearing gap is the unproven almost-sure bound on ||g_k|| in the induction inside Theorem 1's proof, plus the self-identified mismatch between the Adam theory (delayed preconditioner) and Algorithm 2's non-delayed AdamW. An experienced optimization referee would spot these. The paper has considerable merit but the central stochastic convergence theorem is not established as written. If the authors can fix the almost-sure issue (e.g., by adding a bounded-gradient assumption or a restart/truncation argument) and either extend the Adam analysis to Algorithm 2 or rescope the claims, the paper could be publishable. The experiments are strong but not a substitute for closing the theory-practice gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is better than the reader's report suggests, but not as clean as the authors claim. The alleged false identity in Lemma 5 is actually true, so the REJECT rationale misses. The real problems are elsewhere.\n\nWhat's genuinely new: the explicit choice μ=1/3 for retraction-free Stiefel optimization, with deterministic convergence to the manifold and a better complexity than [11]'s neighborhood result. That is a solid theoretical contribution, and the proof structure via RSI and proximal smoothness is reasonable. The LoRA reformulation is a natural extension, and the experiments are thorough: multiple tasks, backbones, and consistent gains in convergence speed and final metrics. They also correctly cite concurrent Riemannian LoRA work.\n\nWhere the soft spots are:\n\n1. The stochastic convergence theorem (Theorem 1) has an unstated almost-sure bound. The induction uses δ_{k+1} ≤ ... + (5/2)α_k ||g_k|| and then bounds the last term by a constant, which effectively assumes ||g_k|| ≤ \\hat D_f almost surely. Assumption 1 only gives variance. With a one-sample mini-batch the bound can fail and the iterates can leave the 1/8 tube. This is load-bearing for the stochastic claim; the deterministic case is likely fine.\n\n2. The theory/algorithm mismatch is explicit. Appendix B proves convergence for a delayed Adam-style preconditioner and Remark B.1 says the non-delayed version is not covered. Algorithm 2 uses the non-delayed version, and the experiments use tuned μ not 1/3. So the practical Manifold-LoRA is not a verification of the theorem, and the \"penalty parameter-free\" label overstates what is shown.\n\nCredit where earned: the paper ships no code, but the experiments are detailed enough to be assessed, and the theoretical discussion is honest in places. The deterministic result alone is worth refereeing.\n\nBottom line: this deserves a serious referee, not a desk reject. The main theorem should be checked carefully and the stochastic proof either fixed or its assumptions strengthened. The authors should also either change Algorithm 2 to the delayed version or extend the theory.","headline":"The explicit-penalty result is genuinely new, but the stochastic convergence proof has a real gap and the evaluated algorithm escapes the theory.","tokens_in":31600,"tokens_out":2895,"would_cite":true,"duration_ms":31106,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C26","90C30","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A retraction-free optimizer for Stiefel constraints converges with a fixed penalty parameter and makes LoRA fine-tuning faster.","keywords":["Stiefel manifold","retraction-free optimization","penalty-parameter-free","landing method","restricted secant inequality","LoRA","parameter-efficient fine-tuning","constraint violation"],"falsifier":"Run the deterministic update with μ=1/3 on a tiny Stiefel problem starting from an X with singular values 0.90 and 0.95 (distance about 0.112, inside the 1/8 tube) and zero loss gradient. The theory predicts the squared distance to the manifold contracts by factor 2/3 at every step; any iterate that fails to contract or leaves the 1/8-neighborhood refutes Lemma 2 and the deterministic half of Theorem 1.","tokens_in":30486,"feed_emoji":"🧮","tokens_out":10661,"duration_ms":91268,"temperature":0.7,"pith_summary":"The paper tries to show that a simple projected-gradient-plus-penalty update can solve optimization problems on the Stiefel manifold without any retraction operator and without tuning the penalty strength. It proves that with the penalty coefficient fixed at 1/3, the iterates both fall onto the manifold and reach a stationary point, at the same O(1/K) rate as retraction-based methods in the deterministic setting. The same mechanism is then applied to LoRA: rewriting the low-rank update as a product of a manifold-constrained basis B and a free coordinate A removes the redundancy of the BA factorization. In experiments, this Manifold-LoRA reaches the same loss in roughly half the epochs and matches or beats vanilla LoRA with half the trainable parameters. If correct, the work removes two practical headaches—retraction cost and penalty tuning—from a widely used fine-tuning method.","feed_headline":"No-retraction optimizer converges and speeds LoRA","feed_subtitle":"Landing update with μ=1/3 reaches stationary points and fine-tunes LoRA in about half the epochs.","key_machinery":"Restricted secant inequality (RSI): a strong-convexity substitute for the nonconvex penalty φ, asserting that for X within distance 1/8 of the manifold, ⟨∇φ(X), X − P_St(X)⟩ ≥ ||X − P_St(X)||². This is what fixes the penalty parameter: gradient descent on φ with step 1/3 contracts the distance to the manifold by a factor 2/3 each step. Proximal smoothness of the Stiefel manifold supplies the Lipschitz behavior of the projection and of the tangent-space projection, which connects feasibility to loss descent. The landing update itself—a projected loss gradient plus a constant-multiple penalty gradient—is the mechanism that lets the iterates remain near the manifold while decreasing f.","core_discovery":"The quadratic penalty φ(X)=¼||X^T X−I||² satisfies a restricted secant inequality near the Stiefel manifold, so gradient descent on φ with step 1/3 converges linearly to the manifold. Combined with proximal smoothness, this yields the landing update X_{k+1}=X_k − α_k P_{T_{X_k}St}(g_k) − (1/3)∇φ(X_k): the penalty step controls feasibility, the projected loss step drives optimality. The paper proves exact convergence to a stationary point under constant step size deterministically, and O(log K/√K) stationarity plus O(log K/K) constraint violation stochastically, improving on landing methods that converged only to a neighborhood or needed tuned penalties. Manifold-LoRA constrains B to the Stie","pith_inferences":["The RSI-plus-proximal-smoothness engine is probably not specific to the Stiefel manifold; any proximally smooth matrix manifold whose constraint penalty satisfies a similar secant inequality should admit an analogous landing algorithm.","The geometric constraint may act as an implicit regularizer; a controlled comparison of Manifold-LoRA with ordinary LoRA at matched final feasibility (e.g., by projecting B post-hoc) would tell whether the gains come from faster optimization or from the constraint itself.","The heatmap evidence that B^T B ≈ I at checkpoints suggests a testable prediction: with the fixed penalty step, the iterates should stay close to the manifold throughout training, not just at the end.","Since the theory is proven for SGD-style updates while the reported experiments use AdamW, the delayed-preconditioner Adam-style analysis in the appendix can be empirically checked for the same O(log K / √K) rate."],"forward_implications":["Deterministic Stiefel-constrained problems can be solved with retraction-free updates at the same O(1/K) iteration rate as retraction-based methods, with no penalty parameter to tune.","Stochastic mini-batch versions retain global convergence; the two-scale step-size schedule (decaying loss step, constant penalty step) improves the constraint-violation rate to O(log K/K).","LoRA fine-tuning can be reparameterized to remove the BA representation redundancy; a manifold-constrained B converges faster and can match or beat vanilla LoRA at half the rank or parameter count.","The fixed penalty strength 1/3 transfers across tasks, so the only hyperparameter needing scheduling is the loss step size.","The Oblique-manifold variant, which only normalizes the columns of B, gives similar gains, indicating the benefit does not depend on strict orthogonality."],"fun_headline_variants":["No-retraction landing converges for LoRA tuning","Penalty-free Stiefel optimizer proves global convergence","Landing update with step 1/3 speeds LoRA fine-tuning","Manifold-LoRA: retraction-free tuning in half the epochs","Retraction-free Stiefel convergence plus LoRA speed"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The convergence theorems assume the initial point lies within distance 1/8 of the Stiefel manifold and that step sizes keep every iterate inside that neighborhood; if a practical LoRA initialization violates this, the fixed-penalty landing guarantees may not hold.","fun_headline_variants_meta":{"raw":{"variants":["No-retraction landing converges for LoRA tuning","Penalty-free Stiefel optimizer proves global convergence","Landing update with step 1/3 speeds LoRA fine-tuning","Manifold-LoRA: retraction-free tuning in half the epochs","Retraction-free Stiefel convergence plus LoRA speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000286,"raw_usage":{"total_tokens":1515,"prompt_tokens":733,"completion_tokens":782,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":477,"tokens_out":782,"duration_ms":8213,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:51:21.589973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the deterministic update with μ=1/3 on a tiny Stiefel problem starting from an X with singular values 0.90 and 0.95 (distance about 0.112, inside the 1/8 tube) and zero loss gradient. The theory predicts the squared distance to the manifold contracts by factor 2/3 at every step; any iterate that fails to contract or leaves the 1/8-neighborhood refutes Lemma 2 and the deterministic half of Theorem 1.","supporting_citations":[],"review_version":1}