{"id":"a9b4c220-4c10-43cf-be05-01be6943d1ed","arxiv_id":"2607.05872","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The top-r gradient subspace GaLore refreshes is largely non-identifiable noise beyond ~39 of 128 directions; the apparent rotation is estimator noise, and optimizer state should be transported rather than carried blindly.","lead":"A study of memory-saving low-rank optimizers like GaLore finds that the top-r gradient subspace they refresh is not a stable object to track: two estimates from the same step disagree about as much as estimates from refreshes apart. The paper offers a diagnostic (the reproducible rank k*) and shows that transported optimizer state plus a shorter second-moment memory helps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Batch-size mismatch between same-step and across-time comparisons could invalidate the 'no subspace to track' claim; missing batch reporting in drift measurements is load-bearing.","rationale":"The reader's weakest assumption concerned the N^{-1/4} averaging scaling extrapolated from N≤64. That is a real concern, but the more load-bearing and more directly testable issue is the possibility that the same-step and across-time measurements are not matched in effective batch size. The paper's core claim that 'the apparent rotation is dominated by estimator noise' rests entirely on the quantitative closeness of 0.725 and 0.742. Appendix F specifies the same-step protocol in detail (B=8, seq len 512, N=1), but the across-time rotation measurement is reported without any batch size or sequence length, making it impossible to verify that the two are comparable. If the training batch used for refresh SVDs is larger, Table 4's own N=64 row suggests the same-step floor would be substantially lower (≈0.55), and the across-time rotation would then indicate real subspace motion rather than noise. This would undercut the central negative result. The paper is otherwise unusually careful about limitations (one-seed cells, measured-only scaling, unseparated transport vs error feedback), and the split-batch probe design is a good idea. But the missing batch-size detail is a concrete reproducibility gap that can be closed with a simple matched-batch re-run. The conditional verdict remains appropriate: the paper should not be accepted until this control is reported. I do not see evidence of internal inconsistency; the issue is a missing control, not a circular argument.","tokens_in":23782,"tokens_out":10173,"duration_ms":103104,"concrete_test":"Run a matched-batch control at Pythia-160M: (1) compute the across-time rotation using refresh gradients with B=8, seq len 512, and compare it to the same-step floor with the same B=8; (2) separately compute the same-step floor using the actual training batch size used in the original across-time measurement (if known). If the matched same-step disagreement differs from the across-time rotation by more than ~10% of sqrt(2r), the 'dominated by estimator noise' conclusion fails. Also report batch sizes and sequence lengths for all drift measurements in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in §3.2 (same-step split-batch disagreement 0.725 ≈ across-time rotation 0.742, Figure 3a) is controlled only if both measurements use the same effective gradient sample size. Appendix F specifies the same-step probe: B=8 sequences of length 512, N=1. The across-time rotation is described as 'across one refresh interval of 160M training' (Appendix F) with no batch size or sequence length stated anywhere in the paper (Sections 3 and 5, Appendices E and F). If the training/refresh gradients that define the across-time rotation are computed on a larger batch — e.g., B=512, common for Pythia-scale training — then the estimator-noise floor at that batch size is much lower than 0.725. Table 4 already shows that N=64 averaging (equivalent to B=512 under the probe's B=8 unit) reduces the same-step disagreement to ~0.55 at 160M. The matched floor would then be ~0.55, not 0.725, and the across-time rotation of 0.742 would exceed it by a large margin, indicating substantial genuine rotation rather than pure estimator noise. The paper's trajectory control (Appendix F) reruns the same-step probe with the same B=8, so it does not resolve this mismatch. Thus the headline claim — that the apparent rotation is dominated by estimator noise — is not established unless the batch sizes are matched. This is a load-bearing gap because it is the paper's primary evidence that 'there is no subspace to track.'","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper measures the geometric premise of GaLore-family optimizers: that the top-r gradient subspace is a slowly drifting, trackable object. Using a same-step control at Pythia-160M, it reports that two top-r subspace estimates from disjoint minibatches at the same step disagree by 0.725·sqrt(2r), nearly equal to the 0.742·sqrt(2r) rotation observed across a full refresh interval, and concludes that the apparent rotation is dominated by estimator noise. It further reports that the gradient spectrum is gapless at r, that only k*≈39 of r=128 directions are reproducible, that N-fold averaging shrinks the deep spectral tail only as N^{-1/4}, and that this non-identifiability holds across scales and architectures. The paper then derives optimality statements for transporting Adam state across the refresh (Theorem 1, Proposition 3) and presents a 2×2 experiment at Pythia-1B comparing carry vs. transport and β2=0.999 vs. 0.99, plus a β2 sweep across optimizers and a full-rank control.","tokens_in":24147,"tokens_out":3382,"duration_ms":39794,"significance":"If the central same-step claim holds, this is an important and largely novel diagnosis of why GaLore-family optimizers work: not by tracking a persistent subspace but by capturing gradient energy in a re-drawn frame. The paper's strengths are genuine: the same-step split-batch control is a clean, time-removed measurement that is independent of the paper's own theory; Theorem 1 and Proposition 3 are parameter-free derivations with stated assumptions; the cross-architecture scaling (70M to 6.9B, plus ViT) is a valuable generality check; and the paper repeatedly and explicitly labels its own limitations, including the inability to separate transport from error feedback and the recipe-dependence of the β2 effect. The main empirical claims are falsifiable and the proposed k* probe is actionable. However, the headline conclusion depends on a matched-batch comparison that is not fully documented, and the empirical transport result is confounded by error feedback; these issues are load-bearing and require revision.","major_comments":[{"comment":"The central same-step versus across-time comparison is not fully controlled, because the batch size of the across-time rotation measurement is not reported. Appendix F specifies the same-step probe as B=8 sequences of length 512, N=1, but the 'across-time reference' 0.742·sqrt(2r) is only described as 'the sign-corrected rotation across one refresh interval of 160M training' with no batch size, sequence length, or averaging fold. If the refresh-subspace gradient during training is computed on a larger effective batch — for example the equivalent of N=64 averaging — then Table 4 implies the matched same-step floor at 160M is 0.725×0.76 ≈ 0.55·sqrt(2r), not 0.725. In that case the across-time rotation of 0.742 would exceed the estimator-noise floor by a large margin, indicating substantial genuine rotation. Because this comparison is the primary evidence for 'no subspace to track,' the man","section":"§3.2, Appendix F"},{"comment":"The empirical claim that 'transporting the state works' is not isolated from error feedback. The transport row of Table 1 is the full LDAdam update, which combines change-of-basis transport of both moments with error feedback. The paper explicitly acknowledges this in §5.2 and in the Limitations, but the contribution list in §1 and the conclusion still present 'transport beats the best untransported configuration' as a supported empirical result. The 18.73 versus 19.28 gap could be driven entirely by LDAdam's error-feedback mechanism. Since the paper's own theory (Proposition 3, Theorem 1) makes a sharp prediction about transport alone, the experiment should include an ablation that transports the moments without error feedback, or the empirical claims in the abstract and conclusion should be explicitly scoped to 'the full LDAdam update (transport plus error feedback).'","section":"§5.2, Table 1, §7 Limitations"},{"comment":"The abstract's claim that 'no averaging budget makes the subspace well defined' is an extrapolation beyond the data. The N^{-1/4} scaling is measured for N ≤ 64 at 160M and confirmed only qualitatively at larger scales (Appendix F); the paper explicitly disclaims a closed-form law and reports that the spiked-covariance prediction fails out of sample. It is possible that the apparent power-law tail is finite-sample noise that would cross over to N^{-1/2} at larger N, which the reader's own 'weakest assumption' note identifies. The measured scaling supports the weaker claim that averaging does not rescue identifiability at any tested budget; the stronger 'no finite averaging budget' claim requires either a derived scaling argument or an explicit statement that it is an extrapolation. I recommend softening the abstract and §3.5 conclusion to match the evidence.","section":"§3.5, Abstract"}],"minor_comments":[{"comment":"Please add the batch size, sequence length, and averaging fold (if any) for the across-time rotation measurement. This is needed not only for the scientific control but also for reproducibility.","section":"Appendix F"},{"comment":"The axis label 'deep-tail s500 N' is unclear; it should define the rank index (e.g., singular value at index 500) and the normalization explicitly in the caption.","section":"Figure 3b"},{"comment":"The entries are ratios of N=64 to N=1 disagreement. This is fine, but the caption should also give the absolute N=1 baseline (or point to Table 5) so a reader can reconstruct the absolute floor at each scale.","section":"Table 4"},{"comment":"The sentence 'Both readings derive from the same measurement, so this is a consistency check between two estimators rather than an independent prediction' is honest and useful; consider moving it into the main text of §3.4 rather than the appendix-like discussion, as it preempts a natural circularity objection.","section":"§3.4"},{"comment":"The full-rank AdamW control is one seed. This is stated, but the figure caption calls it 'directional'; consider adding a second seed or explicitly marking the margin as untested for seed variation, since it is already reversed by learning rate.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is conceptually strong and unusually honest about its limitations, but the batch-size mismatch in the central comparison is a genuine load-bearing gap that the authors did not close. The transport-versus-error-feedback confound is acknowledged but not fixed in the presentation. Both are fixable within the manuscript's scope, so I am not recommending rejection; however, the current text overstates the strength of the evidence in the abstract and conclusion relative to what is actually measured."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nOn first read, the paper deserves attention: the same-step split-batch probe is a genuinely new way to ask whether a low-rank gradient subspace is identifiable, and the k* reproducible-rank count is a practical, memorable diagnostic. The paper is also unusually honest about its limitations, and the theory in Appendix B is clean and parameter-free.\n\nThe problem is the central comparison. The headline claim — that GaLore's per-refresh rotation is estimator noise, not motion — rests on the 0.725 vs 0.742 chordal distances in Figure 3a. The same-step floor is measured with minibatches of B=8 sequences. The across-time rotation is 'across one refresh interval of 160M training' with no batch size reported. If the training gradients are computed on larger batches, the estimator-noise floor at that batch size is lower: the paper's own Table 4 shows N=64 averaging (equivalent to B=512 under the probe's B=8 unit) drops the floor to ~0.55. Then the across-time rotation would exceed the floor by a wide margin, and there is real subspace motion. The paper's trajectory control reruns the same-step probe at the same B=8, so it does not resolve this mismatch. The stress-test note is right: the central evidence as written does not establish the noise-dominance claim.\n\nThat said, the paper has real value beyond that comparison. The spectral analysis shows a gapless tail with N^{-1/4} scaling, which is interesting and correctly labeled as measured, not derived. The k* probe is a useful heuristic for practitioners. The energy-capture reframing of GaLore (§4) and the second-moment staleness analysis are plausible and clearly argued. The author also flags the important confounds: transport is tested only as the full LDAdam update, beta2 effects are recipe-sensitive, and some cells are single-seed.\n\nThe missing artifacts are a genuine nuisance: no code or data means the load-bearing numbers are hard to verify. The extrapolation from N≤64 to 'no averaging budget' is also a stretch, though the paper states it as a measured scaling.\n\nWho is this for? Anyone building on low-rank gradient optimizers: the k* probe is worth stealing, and the non-identifiability argument might reframe the design space even if the stronger claim needs qualification.\n\nMy take for review: send it out. The experiment is clever, the write-up is honest, and the central question matters. But the referee reports should insist on a matched-batch-size control; if the authors can show the same-step floor at the training batch size is still close to the across-time rotation, the paper's main claim stands. If not, they need to substantially weaken it. The paper is not ready as is, but it's a legitimate candidate for a solid venue after revision.","headline":"Clever same-step probe and a useful k* diagnostic, but the headline 0.725 vs 0.742 comparison likely mismatches batch sizes; load-bearing gap.","tokens_in":24596,"tokens_out":4856,"would_cite":true,"duration_ms":48316,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the top-r gradient subspace used by GaLore-type optimizers is not a trackable object: two same-step minibatch estimates disagree as much as estimates taken a refresh apart, so the apparent rotation is estimator noise.","keywords":["gradient subspace non-identifiability","GaLore","low-rank training","Adam second moment","chordal distance","reproducible rank","optimizer memory","spectral tail scaling"],"falsifier":"Take a pretrained Pythia-1B checkpoint, run the same split-batch probe with N=1024 or larger averaged minibatches instead of stopping at N=64, and measure the same-step chordal distance. If the disagreement starts decaying as N^(-1/2) beyond N=64 and the adjacent singular-value ratio at r=128 develops a clear gap, the no-averaging-budget claim fails. A second falsifier: if two same-step estimates from disjoint minibatches agree well below the across-time rotation (for example, below 0.5 sqrt(2r)) at some layer during training, the subspace is identifiable at that layer.","tokens_in":1568,"feed_emoji":"🧮","tokens_out":2027,"duration_ms":60791,"temperature":0.7,"pith_summary":"The paper tests the premise behind memory-saving optimizers like GaLore: that the top-r gradient subspace they recompute every T steps is a slowly drifting object worth tracking. It finds that premise false. At the same training step, two estimates of the top-128 subspace from disjoint minibatches disagree almost exactly as much as estimates taken a full refresh interval apart (0.725 versus 0.742 of the maximum chordal distance), so the refresh-to-refresh rotation is dominated by sampling noise rather than motion. Only about 39 of 128 directions survive resampling, and N-fold gradient averaging does not open the spectral gap that would make the subspace identifiable: the deep spectral tail shrinks as N^(-1/4), not the N^(-1/2) of pure noise. The paper turns this into a concrete prescription: do not average the basis, transport Adam's state through each refresh's rotation, and shorten the second-moment memory to beta2=0.99, with controlled perplexity gains at 1B parameters.","feed_headline":"GaLore's rank-128 gradient subspace is mostly noise, same-step probe shows","feed_subtitle":"Two same-step estimates disagree as much as estimates a refresh apart; only ~39 of 128 directions reproduce.","key_machinery":"The load-bearing tool is the sign-corrected chordal distance between orthonormal frames, sqrt(2r - 2*sum cos(theta_i)), which removes the per-column sign artifact and saturates toward the geometric maximum sqrt(2r) when frames are nearly orthogonal. Applied as a same-step split-batch probe, it separates estimator noise from true motion; combined with the gradient singular spectrum and N-fold averaging scaling, it yields the reproducible rank k*. The theoretical cost analysis uses a uniform random rotation model to bound the blind second-moment carry, plus an optimality argument showing that first-moment transport by the rotation matrix R = U_new^T U_old is the best linear map under isotropic","core_discovery":"Beyond a reproducible core of about 39 of 128 directions, the gradient's top-r subspace is statistically non-identifiable at the ranks GaLore-family optimizers use. The paper demonstrates this with a same-step control: two disjoint split-batch estimates disagree as much as two time-separated estimates. It explains the cause as a gapless gradient singular spectrum with a power-law tail that averaging shrinks only as N^(-1/4), so no finite averaging budget makes the top-r frame well defined. The practical consequence is that Adam's second moment, carried blindly across a refresh, is provably about (r-k*)/2 worse than the best rotation-blind estimator, while first-moment transport through the r","pith_inferences":["If the N^(-1/4) tail scaling extends beyond N=64, then any tracking algorithm relying on subspace persistence has an identifiability floor set by k*, suggesting incremental subspace trackers cannot recover more than k* directions from fresh SVDs.","A cheap pre-flight check for other optimizers and modalities would be the same split-batch probe: comparing r to k* predicts when state transport or a lower beta2 matters and when a random projection would suffice.","The paper's energy-capture account suggests a testable extension: a random projection that still includes the dominant spike should match exact-SVD GaLore whenever k* is much smaller than r, and should diverge as k* approaches r.","Because the full-rank control reverses the beta2 preference, the sign of the beta2 effect could serve as a diagnostic for whether an optimizer is implicitly refreshing a subspace, even when the refresh is hidden."],"forward_implications":["At each refresh, the GaLore top-r frame is redrawn nearly orthogonal to the previous one; the apparent rotation is mostly estimator noise, not drift of a stable object.","No N-fold gradient averaging budget opens a spectral gap at rank r, because the deep tail is power-law signal shrinking as N^(-1/4), not random noise shrinking as N^(-1/2).","Averaging the basis over time does not help: at 1B parameters, even a 10,000-step covariance average leaves the per-refresh rotation at 84% of its geometric maximum and leaves final perplexity essentially flat.","Transporting Adam state through the rotation (first moment by R, second moment by the squared entries of R) and lowering beta2 toward 0.99 improves final perplexity: 18.73 versus 19.28 for the best untransported configuration at the default beta2.","Before trusting any low-rank assumption, measure the reproducible rank k* by splitting a batch and counting the directions that agree; if r exceeds k*, the excess directions are noise being refreshed."],"fun_headline_variants":["Only 39 of 128 gradient directions reproduce; the rest is noise","Same-step subspace estimates disagree as much as refresh-apart","GaLore's subspace is mostly noise—averaging can't save it","Low-rank training: subspace non-identifiable beyond a small core"],"cache_read_input_tokens":25856,"weakest_assumption_plain":"The conclusion that no averaging budget can make the top-r subspace identifiable rests on extrapolating the measured N^(-1/4) spectral-tail decay from N up to 64 to all N and from finite samples to a permanently gapless population spectrum; if the tail is finite-sample noise that would decay as N^(-1/2) at larger N, a large enough budget could open a gap.","fun_headline_variants_meta":{"raw":{"variants":["Only 39 of 128 gradient directions reproduce; the rest is noise","Same-step subspace estimates disagree as much as refresh-apart","GaLore's subspace is mostly noise—averaging can't save it","Low-rank training: subspace non-identifiable beyond a small core"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000846,"raw_usage":{"total_tokens":3611,"prompt_tokens":928,"completion_tokens":2683,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":2607}},"tokens_in":672,"tokens_out":2683,"duration_ms":17927,"temperature":1.0,"reasoning_tokens":2607,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:21:57.659850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a pretrained Pythia-1B checkpoint, run the same split-batch probe with N=1024 or larger averaged minibatches instead of stopping at N=64, and measure the same-step chordal distance. If the disagreement starts decaying as N^(-1/2) beyond N=64 and the adjacent singular-value ratio at r=128 develops a clear gap, the no-averaging-budget claim fails. A second falsifier: if two same-step estimates from disjoint minibatches agree well below the across-time rotation (for example, below 0.5 sqrt(2r)) at some layer during training, the subspace is identifiable at that layer.","supporting_citations":[],"review_version":2}