{"id":"c59f22e0-ffe0-419d-94b2-d8fe5a9ea1b6","arxiv_id":"2504.12436","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A randomly sparsified Adam optimizer with importance-based moment pruning improves few-shot CLIP adaptation accuracy and memory efficiency over low-rank projection methods.","lead":"This paper presents Sparse Optimization (SO), an Adam-style optimizer that randomly prunes most gradient entries and prunes the first moment by importance, for few-shot adaptation of CLIP image-text models. On 11 datasets, it reports higher average accuracy than low-rank fine-tuning baselines while storing fewer optimizer variables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an asymmetric comparison: low-rank baselines are limited to 500 iterations and fixed rank=2, while SO gets 2000 iterations and tuned sparsity hyperparameters; a fair comparison is needed.","rationale":"The reader's weakest_assumption correctly identifies the evaluation protocol's differential iteration caps and stopping rule as the pivotal issue. My reading of the full text confirms this: the paper's own Fig. 2 and Appendix F provide direct evidence that low-rank methods are sensitive to rank, with the optimal rank varying by dataset, yet the baselines are all evaluated at rank=2. The 500-iteration cap for baselines versus the 2000-iteration cap for SO introduces another asymmetry. Although the paper motivates the cap by arguing that low-rank methods overfit, the comparison should at least allow baselines to demonstrate their best achievable accuracy under the same rule, or the claim should be framed as 'under a fixed, untuned protocol' rather than as state-of-the-art. The absence of error bars over only 3 seeds compounds the problem, since the reported margins are small. These issues are addressable and do not invalidate the method's potential, so a CONDITIONAL verdict remains appropriate. My concern does not move the reader's verdict; hence UNCHANGED.","tokens_in":42837,"tokens_out":6001,"duration_ms":60544,"concrete_test":"Re-run all low-rank baselines on the 11 datasets with the same loss threshold 0.01 and the same 2000-iteration budget as SO, and for each dataset and shot test ranks 2, 3, 4, and 5. Record the test accuracy at the first iteration where the loss crosses 0.01, and also the best test accuracy along the trajectory. Compare these numbers against SO under the identical threshold and budget, using at least 5 seeds and reporting standard errors. If SO no longer has a consistent average advantage over the best-rank baseline, the headline claim fails; if it does, the claim survives this concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SO achieves state-of-the-art few-shot adaptation is supported by Table 1, but the comparison is asymmetric. Low-rank baselines are run with rank fixed to 2 and a 500-iteration cap; SO is run with κ=0.05%, T=10, and up to 2000 iterations. The paper itself demonstrates in Fig. 2 and Appendix F that LoRA's accuracy depends strongly on rank (the best rank varies from 2 to 5 across DTD, Pets, UCF101), so rank=2 systematically understates the low-rank methods on at least some of the 11 datasets. Similarly, the 500-iteration cap is an arbitrary budget: when a baseline's loss has not reached 0.01 by iteration 500, it is truncated, while SO is allowed to continue to 2000. Although the paper argues that low-rank methods overfit after many iterations, a fair SOTA comparison should report the best performance each method can achieve under the same stopping rule (e.g., by evaluating accuracy at the point where the loss threshold is actually crossed, or by tuning rank on a validation split). In addition, reported averages are over 3 seeds with no error bars; the observed 1–2 point average gaps between SO and ReLoRA could be within run-to-run noise. Because the paper's own evidence of rank and iteration sensitivity makes the baseline configuration look deliberately handicapped, the SOTA claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sparse Optimization (SO), an Adam-type optimizer that randomly selects a small fraction of gradient entries at each iteration, refreshes the support every T iterations, and prunes the first and second moments by magnitude, and applies it to few-shot adaptation of CLIP. The central claim is that this sparsity-based approach outperforms low-rank projection methods such as LoRA, ReLoRA, VeRA, DoRA, PiSSA, and GaLore on 11 datasets at 1, 2, and 4 shots, while reducing memory overhead. The paper also presents extensive ablations supporting the internal design choices: random gradient pruning, importance-based moment pruning, dynamic sparsity support, and the use of moments.","tokens_in":43161,"tokens_out":3790,"duration_ms":40730,"significance":"If the comparative claim were established, the result would be a useful contribution to parameter-efficient transfer learning, showing that a dynamically supported sparse optimizer can be a competitive alternative to low-rank reparameterization for few-shot VLM adaptation. The paper is strong in its internal validation: the ablations (Tables 2-5) are systematic, the method is clearly specified in Algorithm 1, code is provided, and the memory analysis in Appendix C is a useful reference. The main weakness is that the headline state-of-the-art claim rests on a comparison that is not yet fair: the low-rank baselines are run with fixed rank and a smaller iteration budget than SO, and the main results table contains no variance information.","major_comments":[{"comment":"The central claim that SO achieves state-of-the-art performance is not yet supported because the comparison in Table 1 is asymmetric. Low-rank baselines are run with rank fixed to 2 and a maximum of 500 iterations, while SO is run with κ=0.05%, T=10, and up to 2000 iterations. The paper's own Fig. 2 and Appendix F show that LoRA's accuracy depends strongly on rank, with the best rank varying across DTD, Pets, and UCF101, so rank=2 systematically understates at least some baselines on some datasets. Similarly, the 500-iteration cap truncates a baseline whenever its training loss has not reached 0.01 by iteration 500, while SO is permitted to continue to 2000 iterations. Please rerun the baselines under an equal budget or with per-method rank/budget tuning on a validation split, and report best-over-rank results or per-rank curves for all methods.","section":"§4(a), Table 1, Fig. 2, Appendix F"},{"comment":"Table 1 reports averages over only 3 random seeds with no standard deviations, error bars, or significance tests. The observed average advantages of SO over the strongest baseline (ReLoRA) are 1.3 points at 1 shot, 1.5 points at 2 shots, and 1.8 points at 4 shots; with three seeds these gaps could plausibly be within run-to-run noise. Please provide per-seed numbers or confidence intervals, and state whether the differences are statistically meaningful.","section":"Table 1"},{"comment":"The hyperparameters κ and T appear to be selected using the same benchmark datasets (Pets and DTD are shown in Fig. 3), and no validation split is described. Because the paper motivates the method by arguing that hyperparameter tuning is impractical in few-shot settings, the selection protocol must be clarified. If κ and T were chosen by inspecting test accuracy on the evaluation datasets, the comparison is optimistic; if they were chosen on a held-out validation split, that split should be described. This point is load-bearing for the generality claim that SO 'remains robust' without validation-based tuning.","section":"§4(a), Appendix D"}],"minor_comments":[{"comment":"The caption of Figure 1 states that LoRA uses rank fixed at 3, while Section 4(a) states that rank is set to 2 for all low-rank baselines; please reconcile this discrepancy.","section":"Figure 1"},{"comment":"In Algorithm 1, the while condition uses |L(Θ_{t-1})| > τ, but for a cross-entropy loss the absolute value is not meaningful and the iteration index appears off by one; please correct the stopping condition and indexing.","section":"Algorithm 1"},{"comment":"The results subsections are labeled (a), (b), (d), (e), (f), (g); subsection (c) is missing and should be renumbered.","section":"Section 4"},{"comment":"The header of Table 6 concatenates method names without clear separators (e.g., 'SO GaLoRE LoRA'); please format the header as separate columns for readability.","section":"Table 6"},{"comment":"References [20] and [54] are duplicates of the same VeRA paper; please cite it once.","section":"References"},{"comment":"Some standard deviations in Appendix G appear to be typos, such as '64.74±77.8' in Table 19 and '29.86±30.8' in Table 20; please check all reported variances.","section":"Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The paper's internal optimization story is convincing, but the published version should not claim state-of-the-art until the baseline comparison is made fair. I would ask the authors to either (i) rerun low-rank baselines with multiple ranks and the same iteration budget/stopping rule, or (ii) substantially soften the claim to 'SO is competitive with low-rank methods under a fixed small budget'. The absence of error bars in Table 1 is also a blocking issue for the comparative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes SO, a sparse optimizer that randomly prunes gradients, prunes first moments by magnitude, and refreshes the sparsity support every T steps. The ablations are the best part: Tables 2-5 cleanly show that random gradient selection beats importance-based selection, that importance-based moment pruning beats random moment pruning, and that a dynamic support beats a static one. These results are consistent, and the two-layer network experiments in the appendix add independent evidence for the mechanism. The algorithm is simple, clearly specified, and the code is public.\n\nThe problem is the headline claim. Table 1 compares SO against low-rank baselines under an asymmetric protocol: LoRA and friends run at rank 2 for at most 500 iterations, while SO runs with kappa=0.05%, T=10, and up to 2000 iterations. The paper's own Figure 2 shows LoRA's peak accuracy depends strongly on rank and can shift past 500 iterations, so the baseline settings do not let the low-rank methods show their best. Perhaps the authors would defend this as a fixed-budget, no-validation comparison, but then they should apply the same budget and tuning discipline to SO's hyperparameters, which were chosen on the benchmark. On top of that, the averages in Table 1 have no error bars; the 1-2 point average gap between SO and ReLoRA could easily shrink under seed noise.\n\nThe core idea is worth taking seriously. The paper is honest in its ablations and does not hide the sensitivity of LoRA. I would send it to peer review, but the reviewers should ask for a fair comparison: either tune rank and budget on a validation split for both sides, or report the best accuracy over iterations for each method with the same stopping rule, and include error bars. As it stands, the SOTA claim is not established, but the method and its ablation methodology are a solid contribution to few-shot VLM adaptation.\n\nI'd bring this to a reading group precisely because the comparison protocol is a great teaching example. I'd cite it for the sparse optimizer idea, not for the SOTA claim.","headline":"Useful sparse optimizer with excellent ablations, but the SOTA claim rests on an unfair baseline comparison and missing error bars.","tokens_in":43643,"tokens_out":4994,"would_cite":true,"duration_ms":49896,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A sparse optimizer that randomly updates a tiny parameter subset outperforms low-rank projection methods in few-shot CLIP adaptation.","keywords":["few-shot adaptation","vision-language models","sparse optimization","low-rank adaptation","CLIP","gradient sparsity","moment pruning","parameter-efficient fine-tuning"],"falsifier":"Run LoRA or ReLoRA on the same 11 datasets with a per-dataset early-stopping budget chosen on a held-out split (or with 2000 iterations and a tuned rank), and check whether the average accuracy still trails SO; if the low-rank methods match or exceed 73.8, 76.6, and 78.9 at 1, 2, and 4 shots, the paper's central claim would no longer hold.","tokens_in":1704,"feed_emoji":"⚡","tokens_out":2126,"duration_ms":67489,"temperature":0.7,"pith_summary":"Few-shot adaptation of vision-language models overfits when all parameters are updated, and low-rank projections constrain updates to a fixed subspace that still overfits and is hard to tune. This paper proposes a Sparse Optimization (SO) optimizer, an Adam-style method that keeps a very small density ratio (0.05%) of gradient entries per step, resamples the sparsity support every few iterations, and prunes the first moment by importance. On 11 datasets with 1, 2, and 4 shots, SO reports the highest average accuracy, surpassing LoRA, ReLoRA, DoRA, VeRA, PiSSA, and GaLore while storing only a fraction of their gradient and optimizer state. The point is that carefully chosen sparsity, not low-rank structure, is the better protective mechanism against few-shot overfitting.","feed_headline":"Sparse updates beat low-rank projections in few-shot CLIP","feed_subtitle":"A tiny random parameter subset plus importance-pruned moments tops 11 few-shot datasets.","key_machinery":"The engine is a sparse variant of Adam with two selection rules. At each step the optimizer retains $M = \\lfloor \\kappa d \\rfloor$ gradient entries chosen uniformly at random (Random-M), then combines them with the previous sparse first moment; the union can hold up to $2M$ values, from which the Top-$M$ largest-magnitude entries are kept for the first moment, and the second moment is kept on the same indices. The sparsity support is refreshed every $T$ iterations. These mechanisms implement the paper's two paradigms, local sparsity with global density and local randomness with global importance, and they are what prevent the optimizer from rushing into an overfit subspace while keeping the update path full-rank over time.","core_discovery":"The central claim is that restricting updates to a low-rank subspace is the wrong inductive bias for few-shot vision-language adaptation: the rank is a discrete hyperparameter that must be tuned per dataset, and fixed-subspace updates still overfit. The paper replaces this with a sparse optimizer, SO, that enforces local sparsity (only $M = \\lfloor \\kappa d \\rfloor$ entries are updated at a time) and global density (the support resamples every $T$ iterations, so many parameters receive updates over training), plus local randomness (gradient entries are selected uniformly at random) and global importance (the first moment is pruned by Top-$M$ magnitude, with the second moment aligned to the same indices). With $\\kappa = 0.05\\%$ and $T = 10$, SO achieves the highest average accuracy at every shot count, including 73.8, 76.6, and 78.9 for 1, 2, and 4 shots on the 11 datasets, with the largest gains on EuroSAT, Stanford-Cars, Flowers, and UCF101, while keeping gradient and optimizer memory near 0.47 MB and 0.70 MB for the full CLIP model.","pith_inferences":["If the support-resampling mechanism is the active ingredient, the principle should transfer to full fine-tuning of large language models, where memory is a bottleneck; the paper lists this as future work, and one direct test is to replace Adam in low-rank LLM training with SO and compare convergence.","The supplementary rank-evolution plots suggest random gradient selection keeps the gradient rank near full while importance-based selection collapses it, implying that SO's sparsity acts as a stochastic exploration mechanism; one could test whether SO's advantage disappears on nearly convex loss landscapes where random updates are pure noise.","The two paradigms are separable: importance-based moment pruning could be bolted onto existing adapters such as LoRA or VeRA, producing a hybrid low-rank-plus-sparse method that might outperform either family alone.","A testable extension is to vary the refresh interval $T$ as a schedule rather than a constant, since the paper's sensitivity study already shows smaller $T$ values improve accuracy at the cost of slower convergence."],"forward_implications":["Extreme sparsity ($\\kappa = 0.05\\%$) is sufficient to adapt a full CLIP backbone competitively, with 1, 2, and 4-shot averages of 73.8, 76.6, and 78.9 across 11 datasets.","The performance ranking no longer hinges on a discrete rank choice: the ablations show dynamic sparsity support beats a fixed support, and random gradient selection beats importance-based selection at every shot count.","Memory overhead drops to about 0.47 MB for gradients and 0.70 MB for optimizer states on CLIP, compared with 468 MB for full Adam gradients and 936 MB for its optimizer states.","Low-rank baselines need an early-stopping threshold to avoid overfitting, while SO trains to the same loss threshold without rank tuning, reducing the number of hyperparameters that must be set in few-shot settings.","Because SO reduces to plain Adam when the density ratio is 1, it can serve as a drop-in optimizer on top of existing architectures rather than a reparameterization method."],"supporting_citations":[{"why":"Defines the low-rank reparameterization family (LoRA) that SO is compared against and whose fixed-subspace updates the paper critiques.","marker":"[16]"},{"why":"Supplies the few-shot adaptation protocol, hyperparameters, and the rank-2 with 500-iteration baseline setting used in the main comparison.","marker":"[46]"},{"why":"Provides the pretrained CLIP model that all methods adapt.","marker":"[33]"},{"why":"Provides the Adam optimizer whose gradient and moment recursions SO sparsifies.","marker":"[19]"},{"why":"Introduces GaLore, the gradient low-rank projection baseline, and the periodic update-interval idea that $T$ resembles.","marker":"[48]"},{"why":"Introduces ReLoRA, the merge-and-reinitialize low-rank baseline that is closest in spirit to SO's dynamic support.","marker":"[23]"},{"why":"Introduces DoRA, a weight-decomposed low-rank baseline in the comparison.","marker":"[25]"},{"why":"Introduces VeRA, a vector-based low-rank baseline in the comparison.","marker":"[20]"},{"why":"Introduces PiSSA, a singular-value-initialized low-rank baseline in the comparison.","marker":"[29]"}],"fun_headline_variants":["Sparse optimization beats low-rank in few-shot CLIP","Sparse updates top low-rank for few-shot CLIP","Few-shot CLIP: sparse beats low-rank","Random sparse updates beat low-rank in few-shot CLIP","Sparsity outperforms low-rank projections in few-shot CLIP"],"cache_read_input_tokens":45824,"weakest_assumption_plain":"The comparison assumes that stopping every method at the same training-loss threshold (0.01), with 500 iterations for low-rank baselines and 2000 for SO, is fair; if a longer or per-dataset-tuned budget for the baselines lifts their accuracy, SO's reported advantage could shrink.","fun_headline_variants_meta":{"raw":{"variants":["Sparse optimization beats low-rank in few-shot CLIP","Sparse updates top low-rank for few-shot CLIP","Few-shot CLIP: sparse beats low-rank","Random sparse updates beat low-rank in few-shot CLIP","Sparsity outperforms low-rank projections in few-shot CLIP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001472,"raw_usage":{"total_tokens":5933,"prompt_tokens":977,"completion_tokens":4956,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":4874}},"tokens_in":593,"tokens_out":4956,"duration_ms":36211,"temperature":1.0,"reasoning_tokens":4874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:31:20.232477+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LoRA or ReLoRA on the same 11 datasets with a per-dataset early-stopping budget chosen on a held-out split (or with 2000 iterations and a tuned rank), and check whether the average accuracy still trails SO; if the low-rank methods match or exceed 73.8, 76.6, and 78.9 at 1, 2, and 4 shots, the paper's central claim would no longer hold.","supporting_citations":[{"cited_title":"Low-rank few-shot adaptation of vision-language models","cited_arxiv_id":null,"evidence_quote":"Supplies the few-shot adaptation protocol, hyperparameters, and the rank-2 with 500-iteration baseline setting used in the main comparison."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained CLIP model that all methods adapt."},{"cited_title":"Adam: A method for stochastic optimization","cited_arxiv_id":null,"evidence_quote":"Provides the Adam optimizer whose gradient and moment recursions SO sparsifies."},{"cited_title":"Galore: Memory- efficient llm training by gradient low-rank projection","cited_arxiv_id":null,"evidence_quote":"Introduces GaLore, the gradient low-rank projection baseline, and the periodic update-interval idea that $T$ resembles."},{"cited_title":"Relora: High-rank training through low-rank updates","cited_arxiv_id":null,"evidence_quote":"Introduces ReLoRA, the merge-and-reinitialize low-rank baseline that is closest in spirit to SO's dynamic support."},{"cited_title":"DoRA: Weight-decomposed low-rank adaptation","cited_arxiv_id":null,"evidence_quote":"Introduces DoRA, a weight-decomposed low-rank baseline in the comparison."},{"cited_title":"VeRA: Vector-based random matrix adaptation","cited_arxiv_id":null,"evidence_quote":"Introduces VeRA, a vector-based low-rank baseline in the comparison."},{"cited_title":"PiSSA: Principal singular values and singular vectors adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Introduces PiSSA, a singular-value-initialized low-rank baseline in the comparison."}],"review_version":1}