{"id":"d80f092a-afd3-420b-bd38-b23edb91b50a","arxiv_id":"2507.08091","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.","lead":"A new optimizer called MoFaSGD fine-tunes large language models with much less GPU memory by keeping a low-rank compressed version of the optimizer momentum. It matches the memory savings of LoRA while often beating the competing low-rank method GaLore, but its proof of fast convergence contains a serious gap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.5 is unproven: Lemma D.5's monotone-subspace step asserts Range([U_{t-1},G_{t-1}V_{t-1}])⊆Range(U_t), but Eq. 7–8 only give the reverse containment, and the inclusion fails already in a rank-1 example.","rationale":"The reader's weakest_assumption identifies the same load-bearing flaw: Lemma D.5 requires a monotone subspace containment that the MoFaSGD update does not provide. I agree with that diagnosis. The update in Eq. 7–8 produces Range(U_t) inside the expanded 2r-dimensional span of the previous factors and the projected gradient, not the other way around. The rank-r truncation makes the reverse containment impossible in general, and the weaker containment needed to remove the tangent-space part is also false. The proof's subsequent recursive bound on the momentum compression error therefore has no valid basis, and Theorem 4.5's claimed optimal rate is unsupported. The empirical results may still be honest and interesting, and a corrected proof could rescue the method, but the paper as submitted does not establish its central formal claim. I also note the second gap concerning the 1/√B noise scaling: Assumption 4.2 bounds the first moment of the stochastic-gradient error by σ, not a variance, so the √B denominators in Lemmas D.4 and D.5 are not justified; this is independent of the subspace monotonicity issue and makes the theoretical concern stronger. Minor issues such as the mismatched GitHub URLs between the abstract and Appendix C.1 and the absence of error bars are secondary. Overall, the reader's REJECT verdict is appropriate, and I would not change it.","tokens_in":30882,"tokens_out":16865,"duration_ms":172007,"concrete_test":"Run the following deterministic check with Eq. 7–8 exactly. Take m=n=3, r=1, β=0.5, U0=V0=e1, Σ0=1, G1=e2e1^T. Then U'0=QR([e1,e2]), V'0=QR([e1,0]), S0=[[β,0],[1,0]], so U1∝βe1+e2 and V1=e1. This gives Range([U0,G1V0])=span(e1,e2), which is not contained in Range(U1), and Range(U0)=span(e1) is also not contained in Range(U1). Thus the monotone-subspace assertion used in Lemma D.5 is false. As a follow-up, compute R0=∥(I−U0U0^T)G1(I−V0V0^T)∥_* and P1=∥(I−U1U1^T)G1(I−V1V1^T)∥_*; searching over rank-1 G1 (e.g., G1=e2(e1+e2)^T and random combinations) determines whether the key inequality P1≤R0 itself, rather than only its stated justification, is violated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the optimal O(1/sqrt(T)) rate in Theorem 4.5. Its proof depends on Lemma D.5, specifically on bounding ∥(I−U_tU_t^T)G_{t−1}(I−V_tV_t^T)∥_* by the previous tangent-space residual R_{t−1}=∥(I−U_{t−1}U_{t−1}^T)G_{t−1}(I−V_{t−1}V_{t−1}^T)∥_*. The proof justifies this by asserting, from Eq. 7, that Range([U_{t−1},G_{t−1}V_{t−1}])⊆Range(U_t), and similarly for V. This inclusion is backwards: the update U_t=U'_{t−1}U''_{t−1} with U'_{t−1}=QR([U_{t−1},G_{t−1}V_{t−1}]) gives Range(U_t)⊆Range([U_{t−1},G_{t−1}V_{t−1}]), not the reverse. Since U_t has only r columns while the bracketed matrix generically spans up to 2r directions, the asserted inclusion is dimensionally impossible in general; the weaker inclusion Range(U_{t−1})⊆Range(U_t), which is what the argument actually needs to kill the tangent-space component, also fails. Without this monotonicity, the recursion that converts the projection residual into the claimed ηLt+2tσ/√B error bound is unjustified, so the convergence theorem does not follow from the submitted proof. Separately, Assumption 4.2 only bounds E∥G−∇L∥_* by σ, so the 1/√B noise scaling introduced in Lemmas D.4 and D.5 is not derived; this is an independent gap that reinforces the same conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoFaSGD, a memory-efficient optimizer that maintains a low-rank SVD factorization of the first-order momentum and updates it online via tangent-space projections. The factors U_{t+1}, V_{t+1} are used directly in the parameter update W_{t+1}=W_t-η U_{t+1}V_{t+1}^T, yielding a LoRA-level memory footprint with per-iteration online subspace adaptation. The central theoretical claim is Theorem 4.5, which asserts an optimal O(1/√T) convergence rate for non-convex stochastic optimization under nuclear-norm smoothness and a first-moment noise bound. Empirically, the paper evaluates MoFaSGD on NanoGPT pre-training, GLUE fine-tuning, and Tulu3 instruction tuning, reporting competitive or better performance than GaLore and LoRA with comparable memory usage. The paper includes a memory profiling study, an ablation on GaLore's subspace update frequency, and a spectral analysis of AdamW momentum buffers that supports the low-rank momentum conjecture.","tokens_in":31312,"tokens_out":11208,"duration_ms":122063,"significance":"If the theoretical result were valid, the paper would make a significant contribution: it combines online low-rank momentum factorization with spectrally normalized updates, achieving a memory footprint comparable to LoRA while providing a provably optimal stochastic non-convex rate. The algorithmic idea is original and the empirical evaluation is unusually thorough for a memory-optimizer paper: it includes per-category memory breakdowns, wall-clock convergence curves, rank ablations, and a direct test of the low-rank momentum conjecture. The code is provided. However, the main theoretical guarantee is not established by the submitted proof. The proof of Lemma D.5 relies on a false subspace inclusion, and the stochastic noise scaling used throughout the analysis is not supported by the stated assumptions. These are load-bearing gaps, not presentation issues. The empirical results remain interesting, but the paper's central claim of an optimal convergence rate is unproven in the current form.","major_comments":[{"comment":"The proof asserts that, from the update rule of Eq. (7)-(8), Range([U_{t-1}, G_{t-1}V_{t-1}]) is contained in Range(U_t). This inclusion is reversed. The update defines U_t = U'_{t-1}U''_{t-1} with U'_{t-1} = QR([U_{t-1}, G_{t-1}V_{t-1}]), so only Range(U_t) ⊆ Range([U_{t-1}, G_{t-1}V_{t-1}]) holds. The bracketed matrix spans up to 2r directions while U_t has only r columns, so the asserted reverse inclusion is dimensionally impossible unless the new gradient columns lie in the old range. This inclusion is exactly what lets Eq. (46) bound the current tangent-space residual by the previous residual; without it, the recursion in Eq. (47) and the bound in Eq. (50) do not follow, and Theorem 4.5 is unproven.","section":"Appendix D.3.1, Lemma D.5, Eq. (46)"},{"comment":"The proof replaces M̂_t with Ĝ_t + βM̂_{t-1}, but M̂_t is the rank-r SVD of that 2r-rank matrix, not the matrix itself. The step from Eq. (43) to Eq. (44) omits the rank-r truncation error ∥M̂_t - (Ĝ_t + βM̂_{t-1})∥_*, which is not generally zero and is not bounded by the projection residual in the argument. Even if the subspace inclusion in Eq. (46) were corrected, the compression-error recursion would still need an additional term; the current proof does not account for it.","section":"Appendix D.3.1, Lemma D.5, Eqs. (43)-(44)"},{"comment":"Assumption 4.2 only bounds the first moment of the stochastic noise, E[∥∇L(W,ξ)-∇L(W)∥_*] ≤ σ, yet the proofs of Lemma D.4 (Eq. 39) and Lemma D.5 (Eq. 50) introduce a √B scaling, e.g., Tσ/((1-β)√B). With only a first-moment bound, summing T noise terms gives Tσ, not Tσ/√B; the √B factor requires a second-moment or sub-Gaussian condition that is neither stated nor derived. This is an independent gap that prevents the claimed O(1/√T) rate under the stated assumptions.","section":"Assumption 4.2 and Lemmas D.4-D.5"}],"minor_comments":[{"comment":"The lemma says 'Under Assumptions 4.1 and 4.1'; the second reference should be Assumption 4.2.","section":"Lemma D.4 statement"},{"comment":"The text says 'the inner matrix has rank at most r'; the 2r×2r matrix [βΣ_t-U_t^T G_t V_t, I; I, 0] generically has rank up to 2r, so this statement is unclear and should be corrected or clarified.","section":"Equation (7) and following text"},{"comment":"The notation U_{L,R} is used for different block matrices in Eq. (18), Eq. (24), and Eq. (27), which makes the proof hard to follow; the minimization over L and R is stated only implicitly. A clearer presentation of the SVD decomposition of L and R and the achievability of the lower bound would improve readability.","section":"Theorem 4.3 proof"},{"comment":"The caption reads 'GaLore Update Frequency (τ) Ablation' but the legend and axis labels contain stray symbols (e.g., '= 300', '= 75'); these should be cleaned up.","section":"Figure 6b"},{"comment":"The phrase 'the 0.73B token budget, optimized for Muon's convergence speed' is a bit ambiguous; consider rewording to clarify that the budget is the standard NanoGPT speedrun budget.","section":"Section 5.1"}],"recommendation":"reject","confidential_remarks":"The main theorem is the paper's key theoretical selling point, and its proof contains a false and load-bearing subspace inclusion. The noise-scaling gap is also fundamental. These are not merely local presentation problems; they call into question whether the claimed optimal convergence rate can be established by the proposed algorithm or under the stated assumptions. The empirical study is solid and could support a revised paper that either fixes the theory with additional assumptions and a different proof strategy, or reframes the contribution as an empirical method without the theoretical claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2507.08091. First, the algorithm is a genuine new composition: MoFaSGD keeps an online low-rank SVD of the first-order momentum, updates it through tangent-space projections, and uses U_{t+1} V_{t+1}^T directly as the optimizer step. That is a clean middle ground between GaLore's offline subspace resampling and Muon's full-rank momentum. Second, the central theory does not hold as written. The claimed optimal O(1/sqrt(T)) bound in Theorem 4.5 rests on Lemma D.5, and the proof of that lemma uses a subspace inclusion that is backwards. Equation 7–8 imply Range(U_t) is a subset of Range([U_{t-1}, G_{t-1}V_{t-1}]), but the proof asserts the reverse containment. Since U_t has r columns and the bracketed matrix can span 2r directions, the asserted monotone subspace property is dimensionally impossible in general. Without it, the recursion bounding the momentum compression error collapses, so the main theorem is not established. There is also a separate gap: Assumption 4.2 only bounds E||G - grad L||_* by sigma, but the proof uses a 1/sqrt(B) noise scaling that would require second-moment or sub-Gaussian control.\n\nThe paper does a number of things well. The empirical evaluation is more thorough than most: GPT-2 pretraining, GLUE, Tulu3 instruction tuning, plus ablations on rank, GaLore update frequency, and a spectral analysis of AdamW first-moment states showing low-rank concentration. The memory profiling is detailed and believable. Minor issues: no error bars, and the code URL in the abstract does not match the appendix. The authors also state their limitations clearly, which I appreciate.\n\nI would send this to peer review, because the method is plausible and the empirical story deserves scrutiny. But I would expect major revision: either produce a correct proof of the convergence claim, or present the paper as an empirical technique with heuristic motivation. I would not cite the convergence guarantee in my own work until the proof is fixed; I might cite the algorithm as an empirical method. Serious referee: yes.","headline":"MoFaSGD is a genuinely new low-rank momentum optimizer with a plausible empirical story, but the central O(1/sqrt(T)) convergence proof has a load-bearing subspace-inclusion error and an unjustified noise scaling, so the theory does not hold as submitted.","tokens_in":31859,"tokens_out":3140,"would_cite":false,"duration_ms":32466,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","90C06"],"pacs":[],"model":"deepseek-v4-flash","headline":"MoFaSGD compresses the optimizer's first-order momentum into a low-rank SVD that is updated every step, and proves this costs nothing in asymptotic convergence while cutting memory to LoRA levels.","keywords":["low-rank momentum factorization","memory-efficient training","tangent space projection","spectrally normalized updates","non-convex stochastic optimization","LLM fine-tuning","low-rank SVD","optimizer state compression"],"falsifier":"Run MoFaSGD on a small model and, at several consecutive iterations, compute the largest principal angle between the subspace $\\mathrm{Range}([U_{t-1}, G_{t-1} V_{t-1}])$ and $\\mathrm{Range}(U_t)$. The proof of Lemma D.5 requires the former to be a subset of the latter; any iteration where the containment fails, which the rank-$2r$ SVD update does not prevent, invalidates the monotone-residual step. Alternatively, track $\\|(I - U_t U_t^T) G_t (I - V_t V_t^T)\\|_*$ across iterations: if it ever increases, the recursive compression bound that Theorem 4.5 depends on does not hold as written.","tokens_in":30615,"feed_emoji":"🧠","tokens_out":10547,"duration_ms":97856,"temperature":0.7,"pith_summary":"MoFaSGD is an optimizer designed for fine-tuning large models with scarce GPU memory. Its core idea is to store the first-order momentum, the running average of gradients that Adam-style optimizers keep in full, as a low-rank factored object $(U, \\Sigma, V)$, updated online at every iteration by projecting the current gradient onto the tangent space of the previous factorization. The update direction itself is the product $U_{t+1} V_{t+1}^T$, which gives the spectral normalization effect of methods like Muon while keeping optimizer-state memory at LoRA-like levels. The paper proves an optimal $O(1/\\sqrt{T})$ non-convex stochastic convergence rate and reports competitive fine-tuning results on LLM pre-training, GLUE, and instruction-tuning benchmarks.","feed_headline":"Low-rank momentum optimizer hits optimal rate at LoRA memory","feed_subtitle":"Per-step factored momentum reaches full-rank fine-tuning quality at LoRA-level memory.","key_machinery":"The load-bearing object is the tangent space $T_t$ of the low-rank momentum manifold at $(U_t, \\Sigma_t, V_t)$: the collection of matrices $U_t M V_t^T + U_p V_t^T + U_t V_p^T$ with $U_p$ and $V_p$ orthogonal to the current factors. Projecting the gradient onto this tangent space yields the closed form $\\hat{G}_t = U_t U_t^T G_t + G_t V_t V_t^T - U_t U_t^T G_t V_t V_t^T$, which Theorem 4.3 shows is the minimal-residual two-sided sketch. Because $\\hat{G}_t$ is rank $r$, the sum $\\hat{G}_t + \\beta \\hat{M}_{t-1}$ has rank at most $2r$; after a QR factorization of $[U_t, G_t V_t]$ and $[V_t, G_t^T U_t]$, a rank-$r$ SVD of a $2r \\times 2r$ matrix updates the factors in $O((m+n)r^2 + r^3)$ time. This machinery carries the argument by letting the optimization subspace adapt continuously at low cost, while Theorem 4.5 uses the controlled projection residual to keep the momentum-factorization error from spoiling the descent rate.","core_discovery":"The paper claims that the full-rank first-order momentum of an optimizer can be replaced by a rank-$r$ SVD factorization without sacrificing the asymptotic convergence rate. The key move is to project each incoming gradient onto the tangent space of the current momentum factors; because that projection is optimal among a natural class of two-sided sketches (Theorem 4.3), the residual of the low-rank momentum approximation stays controlled. The update of the factors themselves is cheap: since the projected gradient and the old momentum are each rank $r$, their sum lies in a $2r$-dimensional space and a small SVD of a $2r \\times 2r$ matrix produces the new $U, \\Sigma, V$. The same factors define the parameter update $W_{t+1} = W_t - \\eta U_{t+1} V_{t+1}^T$, bypassing the subspace moment accumulation used by GaLore and making MoFaSGD a low-rank, memory-efficient variant of Muon. Theorem 4.5 then bounds the averaged nuclear-norm gradient by $O(\\Delta/(\\eta T) + \\eta L + \\sigma/\\sqrt{T})$, which is optimal for smooth non-convex stochastic optimization.","pith_inferences":["Applying the same tangent-space factorization to second-moment statistics would turn MoFaSGD into a fully adaptive (Adam-like) preconditioner at similar memory; the paper leaves this extension open.","The per-layer projection residual that the analysis controls suggests a concrete rule for adaptive rank allocation: give more rank to layers where $\\|G_t - \\hat{G}_t\\|_F$ stays large.","Because the optimizer state is already a short list of factors, the low-rank gradient-buffer trick used for gradient accumulation could be extended to shard factors across devices in distributed fine-tuning, potentially cutting communication volume as well as memory.","The method's per-step subspace adaptation could also serve as a drop-in momentum module inside other low-rank subspace or PEFT pipelines, since it only consumes gradients and produces $(U, V)$ directions."],"forward_implications":["Optimizer-state memory for a weight matrix of size $m \\times n$ shrinks from $O(mn)$ (AdamW) or $O(mr + nr)$ with full gradients (GaLore) to $O((m+n)r + r^2)$ for the stored factors, the same order as LoRA, while parameters are still updated in full.","Because the subspace is refined every iteration, there is no periodic full-matrix SVD and no staleness window; the ablation shows that increasing GaLore's update frequency degrades its performance, while MoFaSGD's per-step adaptation does not suffer this effect.","The method inherits the spectral-normalization behavior of Muon- and Shampoo-type preconditioners without storing second moments or computing matrix roots, so the per-iteration cost stays at the level of a standard optimizer.","The convergence guarantee is not degraded by the factorization: under nuclear-norm smoothness and a bounded-variance oracle, the averaged gradient norm reaches $O(1/\\sqrt{T})$, matching the known lower bound for non-convex stochastic optimization.","As a low-rank variant of Muon, MoFaSGD offers a drop-in replacement for hidden-layer optimizers in transformer training with comparable quality to full-rank methods in the tested regimes."],"supporting_citations":[{"why":"Defines GaLore, the subspace-projecting baseline MoFaSGD is compared against and whose offline SVD resampling and subspace moment accumulation the paper argues against.","marker":"Zhao et al. (2024a)"},{"why":"Introduces Muon, the full-rank spectral-normalization optimizer MoFaSGD is positioned as a low-rank variant of, and supplies the benchmark's default hyperparameters.","marker":"Jordan et al. (2024b)"},{"why":"Provides the observation that the EMA of gradient covariance has fast spectral decay, the empirical motivation for conjecturing that the first momentum is low-rank.","marker":"Feinberg et al. (2024)"},{"why":"Supplies the modular-norm and nuclear-norm smoothness framework used in the paper's descent lemma and Assumption 4.1.","marker":"Large et al. (2025)"},{"why":"Gives the lower bound for non-convex stochastic optimization that establishes the $O(1/\\sqrt{T})$ rate as optimal.","marker":"Arjevani et al. (2023)"},{"why":"Introduces Shampoo's Kronecker preconditioning, the lineage for spectrally normalized updates and the connection used to motivate the $U V^T$ update.","marker":"Gupta et al. (2018)"},{"why":"LoRA, the parameter-efficient fine-tuning competitor whose memory footprint defines the LoRA-level memory claim.","marker":"Hu et al. (2021)"},{"why":"Provides the Tulu3 instruction-tuning dataset, setup, and hyperparameters used in the main post-training experiments.","marker":"Lambert et al. (2024)"}],"fun_headline_variants":["Low-rank momentum factorization achieves optimal rate","MoFaSGD: memory-efficient training with factored momentum","Factored momentum optimizer matches full-rank quality at LoRA memory","Optimal-rate low-rank optimizer for memory-efficient fine-tuning","Dynamic low-rank SVD of momentum cuts memory without slowing convergence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof of the convergence theorem assumes that the optimization subspace only grows at each step, meaning the new subspace must contain the previous subspace together with the projected gradient, so that the momentum compression error never increases. The algorithm's actual update rule guarantees the opposite inclusion, so the monotonicity the proof relies on is not assured.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank momentum factorization achieves optimal rate","MoFaSGD: memory-efficient training with factored momentum","Factored momentum optimizer matches full-rank quality at LoRA memory","Optimal-rate low-rank optimizer for memory-efficient fine-tuning","Dynamic low-rank SVD of momentum cuts memory without slowing convergence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1376,"prompt_tokens":1030,"completion_tokens":346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":267}},"tokens_in":646,"tokens_out":346,"duration_ms":4000,"temperature":1.0,"reasoning_tokens":267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:28:01.650658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MoFaSGD on a small model and, at several consecutive iterations, compute the largest principal angle between the subspace $\\mathrm{Range}([U_{t-1}, G_{t-1} V_{t-1}])$ and $\\mathrm{Range}(U_t)$. The proof of Lemma D.5 requires the former to be a subset of the latter; any iteration where the containment fails, which the rank-$2r$ SVD update does not prevent, invalidates the monotone-residual step. Alternatively, track $\\|(I - U_t U_t^T) G_t (I - V_t V_t^T)\\|_*$ across iterations: if it ever increases, the recursive compression bound that Theorem 4.5 depends on does not hold as written.","supporting_citations":[{"cited_title":"Sketchy: Memory-efficient adaptive regularization with frequent directions","cited_arxiv_id":null,"evidence_quote":"Provides the observation that the EMA of gradient covariance has fast spectral decay, the empirical motivation for conjecturing that the first momentum is low-rank."},{"cited_title":"Scalable optimization in the modular norm","cited_arxiv_id":null,"evidence_quote":"Supplies the modular-norm and nuclear-norm smoothness framework used in the paper's descent lemma and Assumption 4.1."},{"cited_title":"Shampoo: Preconditioned stochastic tensor optimization","cited_arxiv_id":null,"evidence_quote":"Introduces Shampoo's Kronecker preconditioning, the lineage for spectrally normalized updates and the connection used to motivate the $U V^T$ update."}],"review_version":1}