{"id":"804a47e1-46b3-4135-9c47-84edf2d3f582","arxiv_id":"2602.14656","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"POGO uses a two-step tangent-plus-normal update with lambda = 1/2 to keep iterates near the Stiefel manifold at the cost of five matrix multiplications, making large-scale orthogonality constraints practical.","lead":"POGO is a new optimizer that keeps weight matrices orthogonal by taking one cheap tangent step followed by an approximate projection back to the orthogonal set. It trains models with over 200,000 orthogonal kernels in minutes rather than hours, while matching the accuracy of unconstrained Adam.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.5's feasibility guarantee depends on an unverified uniform bound ||G|| ≤ L; the VAdam experiments don't establish ξ<1, so the 'always on manifold' claim is empirical, not theorem-backed.","rationale":"POGO's headline theoretical result, Theorem 3.5, is a conditional statement: if ||G|| ≤ L uniformly and ξ=ηL<1, then all iterates stay o(ξ^7)-close to the Stiefel manifold. This is the core of the paper's D1 (feasibility) claim. The proof of Prop. 3.3 uses Assumption 1 to bound ||S|| via Prop. A.4, and the induction in Thm. 3.5 relies on Thm. 3.4, which also invokes the same bound. Neither POGO nor the recommended base optimizer VAdam is designed to enforce ||G|| ≤ L; VAdam's vector-wise normalization produces outputs whose norms can grow with the gradient history and the loss landscape. The paper's own ablation (Fig. C.2) shows that for SGD without a small enough η the iterates leave the manifold, demonstrating that the assumption is not automatically satisfied. In the benchmark experiments where POGO is claimed to 'remain on the manifold at all times' (Figs. 5-6), η and the base optimizer are chosen by grid search (App. C), but the resulting ||G_t|| values are never reported, so there is no evidence that ξ < 1 actually holds in the successful runs. Thus the theoretical guarantee does not currently cover the empirical successes, and the practical claim rests on robustness that is not explained. The reader's CONDITIONAL verdict captures this appropriately; the paper should either add a mechanism (e.g., gradient clipping or normalization) that provably enforces ||G|| ≤ L, or report the empirical ξ and the corresponding bound from Thm. 3.5 for each benchmark.","tokens_in":26486,"tokens_out":9945,"duration_ms":88498,"concrete_test":"Instrument the POGO+VAdam runs from Section 5.3 (squared unitary PC) to log the norm of the VAdam output G_t before the update in Algorithm 1. Compute max_t ||G_t|| and check whether η·max_t ||G_t|| < 1 for the reported η=0.5. If the inequality fails at any iteration, Assumption 1 is violated in the headline experiment and Theorem 3.5 cannot be invoked; the feasibility claim would then rest on unproven empirical robustness rather than the theorem. As a complementary check, repeat Thm. 3.4's bound numerically and see whether the observed manifold distance actually scales as o(ξ^7) when ξ is computed from logged ||G_t||.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is Assumption 1 (§3.3): the theorem's feasibility guarantee requires a uniform bound ||G|| ≤ L on the base optimizer's output, with ξ = ηL < 1. This assumption enters directly into Prop. 3.3 (via Prop. A.4) and again into the induction step through Thm. 3.4. Yet nothing in POGO or VAdam enforces it. VAdam's vector-wise normalization bounds individual vector components relative to their RMS history, but the resulting G can have arbitrarily large norm; no proposition in the paper establishes a global L for any base optimizer. The claim in §3.3 that 'we can keep ∥G∥ under control' is an appeal to the user, not a guarantee. The paper's own ablation (Fig. C.2) shows that if η is too large (with SGD), POGO's iterates diverge from the manifold, so the condition is doing real work. Crucially, the headline experiments with VAdam (Sections 5.2-5.3) never report the observed ||G_t||, so there is no evidence that ξ < 1 actually holds in the runs used to support the 'maintains orthogonality at all times' claim. Without such verification, Thm. 3.5 cannot be invoked to explain the empirical feasibility; the D1 claim is supported only by anecdotal observation. This is exactly the gap the reader identified, and it warrants a conditional verdict requiring the authors either prove a bound for VAdam or report ξ empirically.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces POGO, an optimizer for problems with orthogonality constraints of the form X X^T = I. POGO builds on the Landing algorithm, but (i) decouples the tangent-space step from the normal correction and (ii) shows that a fixed correction strength λ = 1/2 keeps iterates provably close to the Stiefel manifold under a uniform bound on the base optimizer's output norm. The algorithm wraps any base optimizer (SGD, VAdam, etc.), uses only matrix products, and is evaluated on PCA, Procrustes, O-ViT, CNN, and squared unitary PC benchmarks, where it matches unconstrained Adam accuracy while being order-of-magnitude faster than retraction-based methods.","tokens_in":26882,"tokens_out":11222,"duration_ms":104350,"significance":"If the theoretical and empirical claims hold, POGO is a significant practical advance: it scales to hundreds of thousands of orthogonal constraints on a GPU, removes several hyperparameters from Landing, and achieves state-of-the-art downstream performance while staying on the manifold. The explicit bound in Prop. 3.3 is elegant and the empirical results are strong. However, several load-bearing assumptions and claims need clarification or verification before the results can be accepted as stated.","major_comments":[{"comment":"The paper repeatedly states that POGO 'can compute exactly the optimal step size to land back on the manifold' and 'calculate the exact λ to stay within the manifold' (Fig. 3). This is not supported: P(λ) is a real quartic and for p>1 with unequal singular values it need not have a real root. The heuristic 'take the real part of the root with the least imaginary part' does not guarantee P(λ)=0 or even a local minimum of P(λ). The ablation in Fig. C.2 (left, η=0.025) shows iterates escaping the manifold even with root-based λ. Please revise the text to claim instead that POGO minimizes (or approximately minimizes) P over real λ, and remove the 'exact landing' wording unless a real-root existence result is proved. The default λ=1/2 variant is unaffected, but the §3.2 contribution as written is inaccurate.","section":"§3.2, Lemma 3.1 and 'Choosing a step size'"},{"comment":"Assumption 1 (∥G∥≤L, ξ=ηL<1) is the key condition in Prop. 3.3 and the induction in Thm. 3.5, but no mechanism in POGO or VAdam enforces it. The paper never reports the observed ∥G_t∥ or ξ for the headline VAdam experiments in §5.2–5.3. The ablation in Fig. C.2 shows that with SGD and η=0.025 the manifold distance grows, so the condition is not vacuous. The statements in §3.3 ('we can keep ∥G∥ under control') and App. C.6 ('gradient normalization ... helps us to adaptively control ∥G∥') are not proofs. Please either prove a uniform bound for VAdam or another base optimizer, or empirically report ∥G_t∥ and ξ for the runs used to support the 'maintains orthogonality at all times' claim. Without this, Thm. 3.5 cannot be invoked to explain the empirical feasibility of the main experiments.","section":"§3.3, Assumption 1 and Theorem 3.5"},{"comment":"The paper claims VAdam 'satisfies Def. 1' because it uses vector-wise normalization m/∥v∥_2. This is not a linear function of the input gradient: the denominator ∥v∥_2 is computed from squared past gradients, so the equivariance in Eq. (8) does not hold exactly. If VAdam is used as the base optimizer in §5, the theoretical rationale for replacing ∇f by G in the relative-gradient construction is weakened. Please either state the precise equivariance property that VAdam actually satisfies (e.g., rotation equivariance per vector) or clearly mark the use of VAdam as an empirical contribution. The λ=1/2 theory for SGD remains valid, but the current text overstates the theoretical support for adaptive base optimizers.","section":"§3.1, Definition 1 and Eq. (8)"}],"minor_comments":[{"comment":"Typos: 'hyperparemeters' (abstract), 'dowstream' (Section 6), and inconsistent use of 'o(ξ7/2)' vs 'o(ξ^{7/2})' in several places (e.g., Thm. 3.5). Consider using the explicit bound P(1/2) ≤ (3/4+ξ²/4)² ξ⁸ from Prop. A.7 instead of only writing o(ξ⁷) in the main text; the explicit bound is stronger and more informative.","section":"Throughout"},{"comment":"The phrase 'finishes 5 minutes quicker than RSDM' could be replaced with 'is 5 minutes faster than RSDM' for clarity.","section":"Fig. 5 caption"},{"comment":"The notation 'arg min_{i} min_{λ_i∈R} (λ_i − bar λ_i)²' is confusing. Clarify that you choose the root with the smallest imaginary part and then take its real component.","section":"§3.2, root selection"},{"comment":"The caption of Fig. C.3 appears to be copied from Fig. C.2 (it mentions 'Left plots solve the landing polynomial' but the figure shows combined curves). Please correct.","section":"App. C.6 / Fig. C.3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a substantial empirical contribution and the λ=1/2 theory is sound conditional on Assumption 1. The main revisions needed are: (1) fix the overstated 'exact landing' claim in §3.2, (2) verify or explicitly condition the feasibility guarantee on Assumption 1 for the experiments, and (3) correct the linearity claim for VAdam. These are addressable within the manuscript's scope and do not require new algorithmic ideas. The benchmark suite is strong, but the reproducibility statement says code will be released 'upon acceptance'; I would ask the authors to commit to a public release before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"POGO is a genuinely useful optimizer for orthogonally constrained problems. The recipe — a tangent step, then a normal step with lambda = 1/2, wrapping a per-vector adaptive optimizer like VAdam — is simple, GPU-friendly, and delivers large speedups over retraction-based methods while matching Adam's accuracy on CIFAR-10 and MNIST. The scaling to 218k orthogonal kernels in a few minutes is real and worth having. The paper is honest about its relation to SLPG (lambda = 1/2 is their polar-retraction approximation) and to Landing; the incremental contribution is the packaging with adaptive base optimizers, the quartic landing-polynomial solve, and the o(xi^7) iterate bound.\n\nThe soft spots are real but fixable. First, 'exact landing' is not what the landing polynomial gives you for real matrices: a quartic with real coefficients need not have a real root, so there is no guaranteed lambda that puts X_{t+1} back on the manifold. The heuristic of taking the real part of the least-imaginary root is a heuristic, not an exact solution. The abstract and Section 3.2 oversell this. Second, the main-text coefficients of the landing polynomial differ from the appendix derivation (the lambda and lambda^2 terms are wrong); anyone implementing from the paper will get a different polynomial. That should be corrected. Third, Theorem 3.5's guarantee rests on Assumption 1, a uniform bound ||G|| <= L with xi = eta L < 1, and the paper never verifies that this holds in the VAdam experiments. Figure C.2 shows the bound is doing real work — when eta is too large, the iterates leave the manifold. The empirical claim of staying close is supported by the plots, but the theoretical explanation is conditional, and the paper should either prove or empirically estimate xi for the actual runs. This is a gap, not a fatal flaw.\n\nThe empirical work is on solid ground: multiple problems, multiple baselines, confidence intervals, and the comparisons look fair, apart from RSDM being dropped in the PC experiment (with explanation). The code appears to be available, though the reproducibility statement says 'will release upon acceptance' — that should be clarified.\n\nVerdict: send it to review. A careful referee will want the theorem's assumption verified, the landing polynomial coefficients fixed, and the exact-landing claim toned down. But the method itself is simple, fast, and competitive, and the paper's practical contribution is clear. I'd cite it if I worked on orthogonal constraints.","headline":"A practical, fast orthogonal-constraint optimizer that deserves a serious look; the exact-landing claim is overstated and the theorem's key assumption is unverified in practice, but the empirical package is strong.","tokens_in":27333,"tokens_out":3691,"would_cite":true,"duration_ms":36006,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An optimizer that lands on the orthogonal manifold every iteration—using just five matrix products—makes thousands of orthogonality constraints practical at deep-learning scale.","keywords":["orthogonal optimization","Stiefel manifold","Landing algorithm","adaptive optimizers","retraction-free optimization","VectorAdam","CIFAR-10","probabilistic circuits"],"falsifier":"Run the released code on the CIFAR-10 orthogonal-kernel benchmark (218,624 3×3 kernels) with VectorAdam, keeping ηL < 1; if POGO fails to stay within, say, 1e−4 manifold distance while matching Adam's final accuracy in under ~10 minutes on a modern GPU, the central practical claim is refuted. For the theory, the same run could record ξ_t = η‖G_t‖ at every iteration and check whether the observed ‖X_t X_t^T − I‖² is consistently below the o(ξ_t^7) rate on a small-scale quadratic.","tokens_in":26387,"feed_emoji":"⚡","tokens_out":5941,"duration_ms":56019,"temperature":0.7,"pith_summary":"Orthogonality constraints improve stability and robustness, but existing optimizers are either slow (retraction-based) or let matrices wander off the manifold before eventually landing. This paper argues that the Landing update already contains the ingredients for an exact one-step return: take a tangent step to an intermediate point, then choose a step along the normal direction that puts the iterate back on the orthogonal matrices. The authors show that a fixed normal step of 1/2 achieves this whenever the base optimizer's output is uniformly bounded, giving squared distance to the manifold of order ξ^7, and that the step can be wrapped around adaptive optimizers such as VectorAdam. The result is POGO: five GPU-friendly matrix multiplications, no retractions, and empirically the same accuracy as unconstrained Adam on CIFAR-10 with 218,624 orthogonal 3×3 kernels trained in minutes. If right, this removes a key bottleneck and lets orthogonality constraints be used at deep-learning scale.","feed_headline":"POGO trains 218k orthogonal kernels in minutes","feed_subtitle":"A one-step landing rule keeps every weight matrix on the Stiefel manifold while matching unconstrained Adam on CIFAR-10.","key_machinery":"The landing polynomial P(λ) = ‖(M − λ∇N(M))(M − λ∇N(M))^T − I‖², a quartic in λ whose coefficients can be computed in O(p²n); the paper proves that its root gives the exact step to return to the manifold. The key simplification is the reformulated update M = X − ηXS (a step in the tangent space) followed by the normal step with λ: the bound in Proposition 3.2 shows ‖MM^T − I‖ ≤ η²‖S²‖, so that λ = 1/2 yields P(1/2) = o(ξ^7). The constant 1/2 is the first-order Taylor expansion of (MM^T)^{-1/2}, revealing POGO as an approximate polar retraction that uses only matrix products.","core_discovery":"The central discovery is that optimizing under orthogonality constraints can be decomposed into a tangent step followed by a normal step whose optimal size is both computable in closed form (the landing polynomial, a quartic) and, under a uniform gradient-norm bound, well approximated by the constant 1/2. Theorem 3.5 states that starting on the Stiefel manifold and iterating X_{t+1} = M_t + (1/2)(I - M_t M_t^T) M_t with M_t = X_t - η X_t S_t keeps every iterate within o(ξ^7) of the manifold, where ξ = ηL and L bounds the base optimizer's output. Because the update uses only matrix multiplications and because the skew-symmetric relative gradient is preserved by linear base optimizers, standar","pith_inferences":["If the uniform bound ξ < 1 could be enforced algorithmically (e.g., via gradient clipping or a schedule), POGO's feasibility guarantee would become unconditional; the paper leaves that enforcement mechanism implicit.","The λ = 1/2 update is a low-order Taylor approximation of a polar retraction, so one could derive higher-order corrections (e.g., λ as a function of ‖MM^T − I‖) to trade one extra matrix product for an even tighter manifold bound.","The same two-step leap-and-land pattern could be adapted to other constraint sets with a quadratic normal potential, such as doubly stochastic matrices or low-rank subspaces, whenever the normal step has a closed-form root.","Because the base optimizer only needs to be linear, future momentum variants that remain linear in the gradient can be dropped into POGO; Adam's elementwise normalization should be avoided in favor of vector-wise normalization."],"forward_implications":["POGO can wrap modern adaptive optimizers (e.g., VectorAdam) and still keep every iterate close to the orthogonal manifold, closing the gap between constrained and unconstrained optimization in practice.","Because it needs only five matrix products and no QR/SVD retractions, POGO scales to problems with hundreds of thousands of orthogonal matrices (218,624 3×3 kernels) in minutes on a GPU, where retraction methods take hours.","The constant λ = 1/2 reduces hyperparameters: users only set the base optimizer and learning rate, and observe manifold distance ~ξ^7.","The complex-Stiefel extension means POGO applies to squared unitary probabilistic circuits, matching RGD performance in half the time.","POGO maintains feasibility throughout training, enabling early stopping and reliable constraint satisfaction without a post-hoc correction step."],"fun_headline_variants":["POGO keeps orthogonality with just 5 matrix products","Scale orthogonal optimization to thousands of constraints in minutes","A quartic landing step makes orthogonal training fast","POGO: closed-form landing for large-scale orthogonal ML","One-step landing rule scales orthogonality to 218k kernels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole guarantee rests on a single uniform bound: the base optimizer's output G must satisfy ‖G‖ ≤ L across all iterations with ηL < 1; the paper gives no mechanism that provably enforces this, and the ablation shows that when the learning rate is too large POGO escapes the manifold.","fun_headline_variants_meta":{"raw":{"variants":["POGO keeps orthogonality with just 5 matrix products","Scale orthogonal optimization to thousands of constraints in minutes","A quartic landing step makes orthogonal training fast","POGO: closed-form landing for large-scale orthogonal ML","One-step landing rule scales orthogonality to 218k kernels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2450,"prompt_tokens":761,"completion_tokens":1689,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1607}},"tokens_in":505,"tokens_out":1689,"duration_ms":13428,"temperature":1.0,"reasoning_tokens":1607,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:07:47.931018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released code on the CIFAR-10 orthogonal-kernel benchmark (218,624 3×3 kernels) with VectorAdam, keeping ηL < 1; if POGO fails to stay within, say, 1e−4 manifold distance while matching Adam's final accuracy in under ~10 minutes on a modern GPU, the central practical claim is refuted. For the theory, the same run could record ξ_t = η‖G_t‖ at every iteration and check whether the observed ‖X_t X_t^T − I‖² is consistently below the o(ξ_t^7) rate on a small-scale quadratic.","supporting_citations":[],"review_version":1}