{"id":"15f45a54-24b3-4f3b-969a-876ef6541d63","arxiv_id":"2505.01744","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Adding a 'granularity' reshape to low-rank gradient projection improves memory efficiency and, in most tested settings, accuracy at a fixed memory cost.","lead":"The authors propose reshaping the gradient matrix before applying low-rank projection during LLM fine-tuning, which lets them adjust the size of the projection unit. This new 'granularity' knob appears to save memory and sometimes improve accuracy under a fixed memory budget.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-granularity advantage may be a bfloat16 rounding artifact; Prop. 3.2 predicts the opposite in exact arithmetic, and no float32/64 experiments are provided.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing concern: the fine-granularity benefit is partly attributed to bfloat16 rounding error, and no higher-precision experiments rule out that the headline claim is an artifact of low-precision arithmetic. This is indeed the most load-bearing issue because Proposition 3.2 shows that in exact arithmetic, finer granularity slightly increases estimator variance under fixed M, so the paper's own theory cannot explain the claimed advantage; the only proposed mechanism is numerical error in bf16. The concern is concrete, testable, and directly targets the central claim. I also verified the Theorem 3.3 algebra error noted by the reader: substituting eta=C into the denominator yields 2/(C T) rather than 2C/T, so the theorem as stated is incorrect, though the O(1/T) rate survives with the corrected constant. The LLaMA3.2-3B GSM8K result further weakens the unqualified 'finer is better' claim, but the authors already phrase it as a general trend. Because the reader's CONDITIONAL verdict already accounts for this and the concern is addressable by a higher-precision experiment, no verdict change is needed; the correct action is to require that experiment before accepting the general claim.","tokens_in":34129,"tokens_out":7944,"duration_ms":87450,"concrete_test":"Re-run the GPT2-XL Commonsense sweep of Table 4 (memory budget M=64) in float32 instead of bfloat16, keeping all hyperparameters and configurations identical, and compare the average accuracy ordering across c=2^6, 2^4, ..., 2^-6. If the monotonic fine-granularity advantage flattens or reverses, the central claim is a bfloat16 rounding artifact; if the ordering persists, the granularity effect is not purely numerical. In the same run, also record the c=2^6 vs c=1 difference in float64 for a single checkpoint to quantify the exact-arithmetic effect predicted by Proposition 3.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that, under a fixed memory budget M=c*r, finer projection granularity (larger c) is generally preferable. This claim is not supported by the paper's own estimator analysis: Proposition 3.2 gives E||Go-G||^2 = ((m+c)/M)||G||^2, so in exact arithmetic a larger c slightly increases, not decreases, gradient-estimation variance. The empirical advantage is therefore attributed by the authors to reduced bfloat16 rounding error (Section 3.3, 'Numerical Error'; Appendix D.3), but no experiments in float32 or float64 are reported. If the fine-granularity gain disappears in higher precision, the headline result reduces to a low-precision numerical tip rather than a property of projection granularity. This is the most load-bearing weakness because the paper's theoretical framework (Proposition 3.2) cannot explain the claimed benefit; the only proposed mechanism is explicitly tied to bf16 arithmetic. A secondary correctness issue is in Theorem 3.3/Appendix C.2: with eta=C=M/((m+c+M)L), the derived factor is 2/(C T), not 2C/T as stated. This does not destroy the O(1/T) rate but means the theorem as written is algebraically wrong. Additionally, the LLaMA3.2-3B GSM8K result (Figure 12) shows the finest configuration (c=2^8, r=1) is not best; c=2^4 attains the highest score, contradicting the unqualified 'finer is better' phrasing, though the authors soften this to a 'general trend'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VLoRP, a framework that generalizes low-rank gradient projection (LoRP) by introducing a granularity factor c that reshapes each gradient matrix from R^{n x m} to R^{nc x (m/c)} before projecting with rank r = M/c under a fixed memory budget M = cr. The central empirical claim is that, for a fixed M, finer projection granularity (larger c, smaller r) generally improves performance and stability. The paper also introduces ProjFactor, an Adam-style optimizer that stores first moments in the subspace and uses an Adafactor-style factorization for second moments, and it provides convergence guarantees: an O(1/T) bound for SGD (Theorem 3.3) and a monotone-descent/stationary-convergence result for ProjFactor via Hamiltonian descent (Theorem 4.1). Experiments compare VLoRP with Adam, Adafactor, LoRA, Galore, fira, and APOLLO on LLaMA2-7B, GPT2-XL, and LLaMA3.2-3B across commonsense reasoning, MMLU, and GSM8K.","tokens_in":34511,"tokens_out":8874,"duration_ms":86616,"significance":"If the fine-granularity advantage holds beyond the specific low-precision setting tested, VLoRP adds a simple and useful memory/performance trade-off knob to the LoRP family, and ProjFactor is a practical optimizer that reduces memory below Galore while remaining competitive. The paper's strengths include a clean formulation, extensive experiments across multiple models and memory budgets, and a falsifiable central claim: higher-precision experiments would directly test whether the benefit is a property of projection granularity or an artifact of bfloat16 arithmetic. The theoretical framework is conventional rather than machine-checked, but the estimator derivation is explicit and reproducible from the stated assumptions. The main weakness is that the paper's own Proposition 3.2 predicts a slight variance increase with finer granularity in exact arithmetic, so the only proposed mechanism for the empirical gain is reduced bfloat16 rounding error, a mechanism that is not isolated experimentally.","major_comments":[{"comment":"The headline claim that finer granularity is preferable under a fixed memory budget is not supported by the paper's own estimator analysis. Proposition 3.2 states E||G_o - G||^2 = (m+c)/M ||G||^2, so for fixed M the gradient-estimation variance slightly increases with c in exact arithmetic, not decreases. The empirical advantage is attributed in Section 3.3 ('Numerical Error') and Appendix D.3 to reduced bfloat16 rounding error, but no float32 or float64 experiments are reported. If the gain disappears in higher precision, the headline result reduces to a low-precision numerical tip rather than a property of projection granularity. Please add experiments that isolate the precision variable, and state clearly which part of the observed gain survives in float32/float64.","section":"Section 3.3 and Proposition 3.2"},{"comment":"The stated convergence bound is algebraically inconsistent with the proof. With eta = C = M/((m+c+M)L), the denominator in inequality (10) is eta - (m+c+M)L eta^2/(2M) = C/2, so the resulting bound is 2/(C T) (L(W0)-L(W*)), not 2C/T as printed in Theorem 3.3. The proof's final equality '2(m+c+M)L/M = 2C' is also wrong; the left-hand side equals 2/C. This does not destroy the O(1/T) rate, but the theorem as stated and proved must be corrected.","section":"Theorem 3.3 and Appendix C.2"},{"comment":"The statement in Section 5 that 'the finest-grained VLoRP configuration (c=2^8, r=2^0) achieves the highest scores ... on ... GSM8k' is contradicted by the LLaMA3.2-3B results in Figure 12 and Appendix D.8, where c=2^4 (r=16) reaches the highest GSM8K score of 39.88 while c=2^8 (r=1) scores 36.77. The appendix softens this to a 'general trend,' but this exception should be reported in the main text and the unqualified claim amended, since it bears directly on the paper's central empirical conclusion.","section":"Section 5 and Figure 12"},{"comment":"MMLU and GSM8K results are reported without any measure of variability, and in Table 1 many adjacent configurations differ by less than the reported standard errors (e.g., 60.38 vs 60.39 for the two coarsest rows). Because the central empirical claim is a monotone trend in c, the absence of repeated runs or seed-dependent variability for two of the three benchmarks makes it difficult to assess whether the observed ordering is significant. Please report error bars or variance information for all headline results.","section":"Table 2 and Figure 3"}],"minor_comments":[{"comment":"The claimed memory complexity O(mn + 2nM + n + m) for VLoRP+ProjFactor does not reflect the dependence on c: the factored second-moment states have sizes nc and m/c, giving O(mn + 2nM + nc + m/c). The stated simplified form is acceptable only under an additional restriction on c that is not made explicit.","section":"Section 5, memory analysis"},{"comment":"The condition stated before (16) as 'setting a >= (R+1)b/4a' appears to be a typo; for the displayed inequality to hold the correct condition is a >= (R+1)b/4. Please correct it.","section":"Appendix C.3, inequality (16)"},{"comment":"The asterisk on 'VLoRP*' in the left panels of Figures 3 and 12 is not explained in the caption or text; please define it.","section":"Figures 3 and 12"},{"comment":"The symbol L is used both for the loss function and for the smoothness constant in Theorem 3.3, which is confusing; consider using L(.) for the loss and a different constant for smoothness.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core is potentially valuable, but the discrepancy between Proposition 3.2 and the claimed benefit, together with the absence of higher-precision experiments, is the key risk. If the authors can show that the fine-granularity advantage persists in float32/64 or otherwise demonstrate a mechanism independent of bfloat16 rounding, the paper would be much stronger. The Theorem 3.3 algebraic error should be corrected before publication. No concerns about novelty overlap beyond the already acknowledged FLoRA relation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"VLoRP deserves a serious referee, but the central empirical claim has a load-bearing loose end. What's new: the paper treats projection granularity c as a tunable degree of freedom by reshaping the gradient before low-rank projection, and it shows a mostly consistent pattern that, at fixed memory budget M=c*r, larger c (finer granularity) beats larger rank on commonsense, MMLU, and GSM8K. That systematic sweep is the contribution, not the individual gains, which are modest. The ProjFactor optimizer, storing subspace momentum and factored second moments, is a sensible engineering addition and the memory numbers look right. I also credit the authors for being explicit that FLoRA is VLoRP at c=1, so the novelty is properly scoped.\n\nThe soft spots are real but mostly addressable. Proposition 3.2 gives E||Go-G||^2 = (m+c)/M ||G||^2, so in exact arithmetic finer granularity slightly increases estimation variance. The paper's own explanation for why fine granularity helps is reduced bfloat16 rounding error (Section 3.3 and Appendix D.3), but there are no float32 or float64 experiments. If the benefit vanishes in higher precision, the headline reduces to a numerical-precision tip. This is the key missing experiment. Also, Theorem 3.3 as stated has a factor error: the proof actually yields 2/(C T), not 2C/T, after substituting eta. The O(1/T) rate survives, but the constant is wrong as written. One experiment (LLaMA3.2-3B GSM8K, Figure 12) has c=2^4 best, not the finest c=2^8; the authors soften this to a general trend, which is the right thing to do but means the unqualified 'finer is better' claim should be adjusted. MMLU and GSM8K results lack error bars, and code is promised but not released.\n\nI do not think the central idea is undermined beyond repair. The variance formula and the empirical trend point in opposite directions, and the numerical-error mechanism is plausible but untested; a revision with fp32/fp64 runs, a corrected theorem constant, and error bars would be convincing. The paper is for people working on memory-efficient fine-tuning who care about the memory-performance frontier, and it gives them a new knob worth testing. It deserves a serious referee.","headline":"A useful new knob for low-rank gradient projection, but the headline 'finer is better' rests on an untested bfloat16-numerics explanation and a theorem with a factor error.","tokens_in":35048,"tokens_out":2684,"would_cite":true,"duration_ms":27684,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-rank gradient projection can be tuned by projection granularity, and finer granularity beats larger rank at the same memory cost.","keywords":["low-rank gradient projection","projection granularity","memory-efficient fine-tuning","large language models","ProjFactor","adaptive optimizers","forward gradient estimation","gradient accumulation"],"falsifier":"Run the same VLoRP configurations, for example $(c=256, r=1)$ versus $(c=1, r=256)$ with $M=256$, on LLaMA2-7B under the same tasks in float32 or float64; if the fine-grained configuration no longer outperforms the coarse one, the central claim reduces to a numerical-precision effect rather than a property of projection granularity.","tokens_in":33918,"feed_emoji":"🧠","tokens_out":8828,"duration_ms":86397,"temperature":0.7,"pith_summary":"The paper sets out to show that low-rank gradient projection (LoRP) has a second tuning knob: how finely the gradient matrix is sliced before projection. VLoRP reshapes each gradient matrix before applying a random projection, so a granularity factor $c$ and a rank $r$ jointly determine the memory cost $M=c r$. The authors claim that for any fixed $M$, finer-grained projection (larger $c$, smaller $r$) generally performs better than coarser projection with higher rank, across commonsense reasoning, MMLU, and GSM8K on LLaMA2-7B. They also introduce ProjFactor, an Adam-style optimizer that keeps the first moment in the projected subspace and stores a factored second moment, matching Adam-like dynamics while using less memory and supporting gradient accumulation.","feed_headline":"Finer gradient projection wins at fixed memory budget","feed_subtitle":"Splitting gradients into finer slices improves accuracy at no extra memory cost.","key_machinery":"The load-bearing object is the projection granularity factor $c$. VLoRP reshapes a gradient $G\\in\\mathbb{R}^{n\\times m}$ into $\\tilde{G}\\in\\mathbb{R}^{nc\\times (m/c)}$, applies row-wise random projection $\\tilde{G}\\tilde{P}$ with $\\tilde{P}\\in\\mathbb{R}^{(m/c)\\times r}$, then reshapes back; with entries i.i.d. from $N(0,1/r)$, the estimator is $G_o = \\tilde{G}\\tilde{P}\\tilde{P}^\\top$ reshaped to $n\\times m$. The memory budget $M=c r$ fixes the stored size $ncr=nM$, and Proposition 3.2 gives $\\mathbb{E}[G_o]=G$ and $\\mathbb{E}\\|G_o-G\\|^2 = \\frac{m+c}{M}\\|G\\|^2$, making $c$ and $r$ interchangeable to first order in the variance. ProjFactor keeps the first-moment state $\\tilde{m}_s$ in the subspace, projects it back as $\\tilde{m}_s\\tilde{P}^\\top$ for the update, and stores the second moment as a rank-1 row/column factorization in the Adafactor style; its continuous-time dynamics have Lyapunov function $H = L(W) + \\langle \\tilde{m}_s, \\tilde{m}_s/\\sqrt{\\hat{v}_o}\\rangle/(2a)$ and satisfy $dH/dt \\leq 0$.","core_discovery":"The paper's central claim is that projection granularity is a meaningful new degree of freedom in low-rank gradient projection. Concretely, instead of always projecting the $n\\times m$ gradient matrix $G$ row by row into rank $r$, VLoRP reshapes $G$ into $\\tilde{G}$ of shape $nc\\times (m/c)$, projects rows of $\\tilde{G}$ with a random Gaussian matrix $\\tilde{P}$ of size $(m/c)\\times r$, and reshapes the result back. Because only $\\tilde{G}_s = \\tilde{G}\\tilde{P}$ is stored, its size is $ncr = nM$, so the memory budget $M=cr$ can be held fixed while $c$ and $r$ vary. The paper argues, and demonstrates on LLaMA2-7B across Commonsense170k, MMLU, and GSM8K, that configurations with finer granularity (larger $c$, smaller $r$) consistently outperform coarser ones at the same $M$, with the finest tested configuration $(c=256, r=1)$ best on all three benchmarks. It also proves the projected estimator stays unbiased with variance $O((m+c)/M)\\|G\\|^2$ and achieves $O(1/T)$ SGD convergence, and it gives ProjFactor a monotone-descent guarantee through a Hamiltonian-descent Lyapunov argument.","pith_inferences":["Editorial inference: the paper's numerical-error experiment gives a clean way to separate causes; rerunning the same configuration grid in float32 or float64 would show whether finer granularity remains best once bfloat16 rounding is removed.","Editorial inference: the variance identity suggests the granularity benefit may be layer- and scale-dependent, so an adaptive per-layer choice of $c$ could outperform any single global configuration.","Editorial inference: the forward-gradient viewpoint places VLoRP on a spectrum between whole-model gradient estimation and coordinate-wise estimation, so the same reshaping trick could be applied to zeroth-order optimizers or to non-Gaussian projection distributions.","Editorial inference: because numerical error compounds over repeated accumulation, coarse configurations might be rescued by higher-precision accumulators rather than by finer projections, which is a directly testable variant of ProjFactor."],"forward_implications":["Under a fixed memory budget, choosing the finest granularity tested, such as $c=256$ with $r=1$, improves average accuracy over coarser configurations at equal memory on commonsense reasoning, MMLU, and GSM8K.","VLoRP with ProjFactor stores $O(mn + 2nM + n + m)$ per parameter matrix and keeps the gradient itself in projected form, so gradient accumulation does not require storing full-rank gradients.","The gradient estimator remains unbiased and its variance depends only on $(m+c)/M$, so changing granularity at fixed memory does not change the $O(1/T)$ SGD convergence rate.","ProjFactor's update dynamics have a Lyapunov function that decreases monotonically, guaranteeing convergence to a stationary point under the stated assumptions.","If the granularity claim holds, memory-limited fine-tuning can trade rank for granularity rather than simply lowering rank, which adds a new calibration axis for low-rank optimizers."],"supporting_citations":[{"why":"Defines low-rank gradient projection (Galore) with SVD projection; it is the c=1 baseline that VLoRP generalizes and the main memory comparison target.","marker":"Zhao et al., 2024"},{"why":"Shows LoRA is secretly a random Gaussian gradient projector and supplies the stochastic-approximation interpretation plus the FLoRA c=1 baseline.","marker":"Hao et al., 2024"},{"why":"LoRA is the parameter-efficient baseline whose gradient-compression equivalence VLoRP builds on and must beat.","marker":"Hu et al., 2022"},{"why":"Adafactor's rank-1 factored second moment is the memory-saving ingredient that ProjFactor adapts to projected gradients.","marker":"Shazeer & Stern, 2018"},{"why":"Adam is the optimizer whose dynamics the Original Space scheme and ProjFactor aim to match while cutting optimizer-state memory.","marker":"Kingma & Ba, 2014"},{"why":"Forward gradient estimation is the row-wise form that Observation 3.1 identifies LoRP with, and its Monte Carlo view motivates granularity.","marker":"Baydin et al., 2022"},{"why":"The Hamiltonian descent framework supplies the Lyapunov function used to prove ProjFactor's monotone descent.","marker":"Maddison et al., 2018"},{"why":"APOLLO is a random-projection LoRP baseline compared in the experiments and in the memory analysis.","marker":"Zhu et al., 2024"},{"why":"fira is a LoRP method that keeps residual full-rank gradients, used as a baseline illustrating the memory cost of full-rank gradient storage under accumulation.","marker":"Chen et al., 2024b"}],"fun_headline_variants":["Finer gradient slices at same memory: LLM wins","VLoRP: Granularity beats rank in fixed-budget fine-tuning","Same memory, finer grads: accuracy up on three benchmarks","Gradient chunking: new degree of freedom for memory-saving LLMs","ProjFactor: adaptive optimizer, finer projection, less memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that finer granularity is always preferable at a fixed memory budget rests on the expectation that the gain is not just a bfloat16 rounding artifact, since the paper's numerical-error experiments are the only direct evidence for the mechanism and no float32 or float64 runs are shown.","fun_headline_variants_meta":{"raw":{"variants":["Finer gradient slices at same memory: LLM wins","VLoRP: Granularity beats rank in fixed-budget fine-tuning","Same memory, finer grads: accuracy up on three benchmarks","Gradient chunking: new degree of freedom for memory-saving LLMs","ProjFactor: adaptive optimizer, finer projection, less memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1601,"prompt_tokens":1028,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":644,"tokens_out":573,"duration_ms":6349,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:13:26.830156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same VLoRP configurations, for example $(c=256, r=1)$ versus $(c=1, r=256)$ with $M=256$, on LLaMA2-7B under the same tasks in float32 or float64; if the fine-grained configuration no longer outperforms the coarse one, the central claim reduces to a numerical-precision effect rather than a property of projection granularity.","supporting_citations":[{"cited_title":"Galore: Memory-efficient LLM training by gradient low-rank projection","cited_arxiv_id":null,"evidence_quote":"Defines low-rank gradient projection (Galore) with SVD projection; it is the c=1 baseline that VLoRP generalizes and the main memory comparison target."},{"cited_title":"and Stern, M","cited_arxiv_id":null,"evidence_quote":"Adafactor's rank-1 factored second moment is the memory-saving ingredient that ProjFactor adapts to projected gradients."}],"review_version":1}