Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Taming LLMs by Scaling Learning Rates with Gradient Grouping

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A wrapper that clusters gradients and scales learning rates per cluster improves LLM training and lets low-rank pretraining match full-rank results.

desk verdict SGG is a plausible new optimizer wrapper with broad but uneven evidence; the headline gains rely on borrowed baselines and one hidden regression, so treat the 'consistent gains' claim with caution. read the letter →

arxiv 2506.01049 v1 pith:LW4VRSKD submitted 2025-06-01 cs.LG cs.AI

Siyuan Li , Juanxi Tian , Zedong Wang , Xin Jin , Zicheng Liu , Wentao Zhang , Dan Xu This is my paper
classification cs.LGcs.AI
keywords largelanguagemodelsadaptiveoptimizerslearningratescalinggradientclusteringlow-rankpretrainingparameter-efficientfine-tuningAdamWoptimizerwrapper
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that adaptive optimizers for large language models leave performance on the table because they treat every parameter's learning rate independently, and that a cheap wrapper can fix this by grouping gradient statistics and rescaling learning rates per group. The wrapper, SGG, clusters momentum vectors inside each layer and multiplies each parameter's adaptive learning rate by a cluster-specific factor. Across pretraining, fine-tuning, preference alignment, and multimodal training, SGG reports lower validation perplexity or higher accuracy than the same optimizer without the wrapper. The strongest headline claim is that LoRA-based low-rank pretraining with SGG becomes comparable to full-rank Adam training at the same token budget, which prior low-rank optimizers did not achieve.

What carries the argument

The central object is SGG, an optimizer wrapper that operates after the base optimizer computes momentum and adaptive learning rates. For each layer, SGG flattens the momentum vector, assigns its entries to K clusters using mini-batch K-means, and computes a scaling factor per cluster. The factor is the Median-of-Deviation-to-Average: the ratio of the model-wide median absolute deviation to the cluster's median absolute deviation from its layer mean, smoothed by an EMA and clamped to [0.1, 10]. This imposes group-level constraints while preserving per-parameter adaptation, and it is what carries the claimed gains.

What would settle it

Rerun the Table 4 comparisons with every baseline trained in-house under an identical protocol: same 131K-token batch size, same warmup and cosine schedule, same learning-rate grid, and same token budgets. If Adam or LoRA at the 1B scale matches or beats SGG's perplexity, the claimed benefit is a protocol artifact rather than an effect of SGG.

Watch

Extended reading notes

Core claim

The paper claims that applying SGG on top of an existing optimizer consistently improves LLM and multimodal LLM training, with faster convergence and no architectural changes. In pretraining on C4, Adam+SGG lowers validation perplexity by 1.26 to 3.75 points depending on model size, and LoRA+SGG brings low-rank pretraining to within striking distance of full-rank training, for example 14.73 versus 14.30 perplexity at the 1B scale. The same wrapper improves supervised fine-tuning on GLUE, LoRA-based commonsense reasoning, DPO preference alignment, and VQA benchmarks, and it keeps training stable across batch sizes from 128 to 4096 and learning rates up to 0.1.

Load-bearing premise

The reported gains over Adam, GaLore, and LoRA depend on the borrowed baseline numbers having been produced under exactly the same token budgets, batch size, schedule, and learning-rate search as the SGG runs.

Editorial extensions

If this is right

  • Wrapping AdamW, CAME, or APOLLO with SGG lowers validation perplexity on C4 pretraining across model sizes from 60M to 1B.
  • LoRA+SGG reaches perplexity close to full-rank Adam, making low-rank pretraining a viable memory-saving alternative.
  • SGG improves SFT on GLUE, PEFT on commonsense reasoning, DPO accuracy, and MLLM benchmarks when attached to AdamW, LAMB, or Adafactor.
  • SGG stabilizes training across batch sizes from 128 to 4096 and learning rates up to 0.1, a regime where plain Adam becomes unstable.
  • SGG adds little training overhead: roughly 1.8% to 8.2% extra time, with no extra GPU memory in the CPU-offloaded version.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The gains may come more from per-cluster learning-rate homogenization than from discovering semantically meaningful clusters; a natural test is replacing mini-batch K-means with fixed random partitions of the same size and comparing perplexity.
  • If the low-rank pretraining result survives a fully matched baseline comparison, it implies that memory-constrained labs could adopt LoRA+SGG as a default pretraining setup for smaller models.
  • The stability across extreme learning rates and batch sizes hints that SGG acts as an automatic stabilizer against the known surge phenomenon, though the paper offers no theoretical account of why.
  • Because the extra state lives on the CPU, SGG's marginal memory cost is near zero, which suggests it could be combined with quantized or memory-efficient optimizers without erasing their savings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Scaling with Gradient Grouping (SGG), an optimizer wrapper that clusters momentum vectors within each layer into K clusters, computes a cluster-specific scaling factor from the ratio of a global median-deviation statistic to the cluster's median absolute deviation, and multiplies the base optimizer's per-parameter learning rates by these factors (with EMA smoothing and clamping). The method is presented as plug-and-play for Adam, LAMB, CAME, APOLLO, and LoRA-style training. Experiments cover C4 pre-training at 60M-1B scale, GLUE SFT, commonsense-reasoning PEFT on LLaMA-7B, DPO on Qwen2.5-0.5B, and MLLM SFT/PEFT/Q-LoRA with LLaVA variants. The central claims are that SGG gives consistent perplexity/accuracy gains over baselines and, for the first time, enables LoRA-based low-rank pre-training to match full-rank performance (e.g., LoRA+SGG 14.73 vs Adam 14.30 at 1B on C4).

Significance. If the empirical claims hold, SGG is a practically attractive contribution: it is a simple wrapper requiring no architecture changes, it applies across full-rank and low-rank training, and it reports modest training-time and memory overheads (Table 3). The paper's breadth is a genuine strength: results span pre-training, SFT, PEFT, DPO, and multimodal models, with ablations on the clustering algorithm and hyperparameters. However, the central evidence is currently fragile. The headline C4 gains are computed against baselines borrowed from GaLore and a separate low-rank paper without demonstrated protocol identity, no error bars or significance tests are reported despite a claim of three independent runs, and the key global statistic D_t in Eq. (6) is never defined. These are load-bearing issues for the paper's main conclusions, so the contribution is significant but not yet established.

major comments (4)
  1. [Sec. 3.2, Table 4 and App. B.1] The headline ΔGain values, and especially the claim that LoRA+SGG matches full-rank Adam, are computed against baselines marked with a dagger (Adam, Adam-mini, Adafactor, APOLLO, LoRA, ReLoRA, GaLore, Low-Rank) that are borrowed from GaLore and from a separate low-rank training paper. The manuscript states that 'consistent hyper-parameters' were kept and learning rates tuned within a fixed budget, but it does not establish that the borrowed runs used the same token budgets, sequence length, batch size, warmup fraction, cosine decay minimum, or LoRA rank/alpha. This concern is concrete: Table A1 reports Data Amount as 1.3B/2.6B/7.8B/13.1B for the four model sizes, while Table 4's 'Training Tokens' row reports 1.1B/2.2B/6.4B/13.1B, so the paper's own numbers do not agree about the training budget. Because the ΔGain columns and the low-rank-matches-full-rank signature are the central evidence for the paper's claims, the authors must either rerun all baselines under exactly the protocol described in App. B.1 or provide per-method protocol tables showing that the borrowed numbers are directly comparable.
  2. [Sec. 3.1] The text states that all experiments were conducted with 'three independent runs,' yet no table or figure reports variance, error bars, or significance tests. This matters because many of the claimed gains are small: for example, APOLLO+SGG at 1B is only -0.25 PPL relative to APOLLO, and AdamW+SGG on SciVQA I in Table 8 is -0.8, i.e., worse than the baseline. Without run-to-run variance, the reader cannot tell whether the consistent-gains claim is real or within noise. Please report mean±std (or at least the individual runs) for the headline results in Tables 4, 5, 6, and 8.
  3. [Sec. 2.3, Eqs. (5)-(7)] The scaling factor in Eq. (6) is defined as D_t / (D^t_{l,c} + epsilon), where D_t is called the 'global MDA' and is said to characterize the typical parameter-wise deviation throughout the model. However, no formula or algorithmic description is given for how D_t is aggregated from the per-cluster quantities across layers and clusters. This is not a minor omission: the value of every scaling factor depends directly on D_t, so without this definition the method cannot be reproduced or tested. In addition, the notation m_t^l · C_t^l[c] in Eq. (5) is unclear (it presumably denotes a mask-selected subset, but this is not stated), and the apparent use of mean(m_t^l) inside a median over cluster elements should be written out precisely.
  4. [Sec. 2.3, Table 2, Fig. 6, Table 9] The central SGG hyperparameters (cluster number K, recluster interval T, EMA decay beta3, clamp range, and even the choice of the MDA statistic itself) are selected by validation perplexity on C4, and the paper's main pre-training claims are then evaluated on C4. This means Table 4 should be treated as a tuned-result table, not as evidence that SGG is a zero-tuning improvement. The authors should either tune on a different development set and then evaluate on C4, or explicitly present C4 as a development benchmark and rely on the other tasks (GLUE, commonsense, DPO, MLLM) as the independent evidence for the method's generality.
minor comments (6)
  1. [App. A vs Sec. 2.3, Eq. (7)] Appendix A says that 'scaling factors Sl are updated using an EMA of the median gradient magnitudes within each cluster,' which contradicts Eq. (7), where the EMA is applied to the ratio D_t/D^t_{l,c}. Please align the appendix description with the actual algorithm.
  2. [Table 4 / Table A1] The training-token counts are inconsistent between Table 4 (1.1B/2.2B/6.4B/13.1B) and Table A1 (1.3B/2.6B/7.8B/13.1B). Please correct this discrepancy; it is directly relevant to the protocol-match issue raised above.
  3. [Figure 2] The caption contains the typo 'Gradiant Distribution'; it should be 'Gradient Distribution'.
  4. [Table 4 / References] The NAdam row cites ICLR'18 and the reference list attributes NAdam to Reddi et al. (2018), but the standard reference for NAdam is Dozat (2016); Reddi et al. is a different optimizer. Please fix this citation.
  5. [Table 2] The column header 'Var. Sign(Var.)' is difficult to read and the duplicated 'Grad. Grad.' header is confusing. Please clarify what statistics were actually evaluated in each column.
  6. [App. B.1] The text says experiments were scaled 'up to 7 billion parameters,' but no 7B result appears anywhere in the paper. Please either add such an experiment or revise the sentence to reflect the actual maximum scale of 1B.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SGG's scaling is a data-dependent transformation of base-optimizer states; the main caveats are experimental-validity concerns, not derivation-level circularity.

full rationale

Walking the paper's derivation chain, SGG's scaling factors are defined by Eqs. (5)-(7) as functions of the momentum vectors m_t^l that the base optimizer already computes; the update step α_t^l ← α_t^l · S_t^l[C_t^l] in Algorithm 1 is a data-dependent reweighting, not a quantity whose definition already contains the target result. There is no equation in which the reported validation perplexity or the 'comparable to full-rank' conclusion appears as an input. The empirical claims are comparisons against external baselines (GaLore, APOLLO, etc.) and against reproduced optimizers; no baseline number is generated from SGG's own formulas. The one self-citation (Li et al., 2024c) is used only to motivate intra-layer grouping and is independently supported by the paper's own pilot study (Figure 2), so it is not load-bearing. The main caveats — Table 4 baselines borrowed from GaLore without a demonstrated protocol match, and K/T/β3/MDA chosen on C4 validation before reporting C4 — are threats to experimental validity and generalizability, not circularity: they do not make any reported result equal to its input by construction. Hence no circular step meets the quoted-evidence bar; score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's gains rest on several tuned hyperparameters (K, T, beta3, clamp range) and on an assumed global statistic D_t whose formula is not fully specified. No new physical entities are introduced.

free parameters (4)
  • cluster number K = 2 or 3 depending on task
    K is selected per task from validation performance (Table 9): K=2 best for C4 60M/130M and MNLI, K=3 best for MLLM. The headline C4 results use K=2 for 60M/130M even though the text calls K=3 default.
  • recluster interval T = 500 (or 1-5% of total iterations)
    Selected via ablation on C4 LLaMA-60M/130M (Figure 6a), balancing cost and validation PPL. This value is used for all reported results.
  • EMA decay beta3 = 0.99
    Chosen from ablation over {0.9, 0.99, 0.999, 0.9999} (Figure 6b); authors state it is robust, but the value still tunes the scaling dynamics.
  • clamp range [0.1, 10] = 0.1 to 10
    Imposed on St_l[c] (Eq. 7) with no ablation; the range directly bounds how much a cluster can scale the learning rate.
assumptions (5)
  • domain assumption Different layers and parameters in LLMs exhibit distinct yet internally consistent optimization behaviors, so clustering momentum vectors yields meaningful groups.
    Invoked in Sec 2.2 to motivate clustering, citing Li et al. 2024c and Zhang et al. 2025b; SGG's effectiveness depends on this structural property.
  • ad hoc to paper The global MDA D_t is a meaningful reference statistic for homogenizing cluster scales.
    No formula is provided for D_t; it is selected because Table 2 shows MDA outperforms alternatives on C4. If D_t is defined differently, scaling factors change.
  • domain assumption Mini-batch K-means on flattened 1D momentum vectors converges to stable clusters with the default hyperparameters of the sklearn implementation.
    The clustering implementation is not fully specified; the paper relies on the algorithm's default behavior (Sec. A).
  • ad hoc to paper Scaling learning rates by cluster-wise median deviations, with EMA and clamping, preserves or improves convergence.
    No convergence analysis is given; the benefit is established only empirically on the selected benchmarks.
  • domain assumption Baseline results borrowed from GaLore (marked with a dagger) were produced under the same token budgets, data, and LR tuning protocol as the SGG runs.
    Needed for the Table 4 deltas to be meaningful; if setups differed, the gains could be artifacts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming LLMs by Scaling Learning Rates with Gradient Grouping." pith.science (2026). https://pith.science/paper/LW4VRSKD

@misc{pith2026250601049,
  author       = {Pith},
  title        = {Pith review of: Taming LLMs by Scaling Learning Rates with Gradient Grouping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LW4VRSKD}},
  note         = {Machine review of arXiv:2506.01049}
}
read the original abstract

Training large language models (LLMs) poses challenges due to their massive scale and heterogeneous architectures. While adaptive optimizers like AdamW help address gradient variations, they still struggle with efficient and effective parameter-wise learning rate estimation, resulting in training instability, slow convergence, and poor compatibility with parameter-efficient fine-tuning (PEFT) techniques. This work introduces Scaling with Gradient Grouping (SGG), an optimizer wrapper that improves adaptive learning rate estimation by dynamic grouping and group-specific scaling. SGG first groups gradient statistics in each layer into clusters and then applies cluster-specific scaling to calibrate learning rates for each parameter, thus imposing collective group-wise constraints while maintaining precise per-parameter adaptation. Experiments on diverse (M)LLM benchmarks show that SGG integrates seamlessly with existing optimizers, and offers consistent gains and faster convergence over baselines, with various model sizes. Its stability across varying batch sizes and learning rates establishes SGG as a robust choice for LLM optimization.

Figures

Figures reproduced from arXiv: 2506.01049 by the authors.

Figure 1
Figure 1. Scaling with Gradient Grouping. Illustra￾tion of SGG with online grouping and group-specific learning rate (LR) scaling upon adaptive LR optimizers. To address this, parameter-efficient fine-tuning (PEFT) (Hu et al., 2021; Dettmers et al., 2024) has garnered increasing attention, which reduces trainable parameters via low-rank updates. While memory-efficient, PEFT incurs performance degra￾dation compared to full-ran… view at source ↗
Figure 2
Figure 2. Clusters of gradient statistics with LLaMA-1B pre-training on C4. Distributions of (a) parameter-wise gradients g t and (b) learning rates α t for 12-th FFN layer at 5k iterations. SGG identifies diverse clusters compared to Adam (in gray), introducing group constraints while maintaining parameter-wise adaptation. (c) Distribution of gradient L2-norms across layers, showcasing SGG’s ability to adapt to LLMs’ heterog… view at source ↗
Figure 3
Figure 3. Grouping Methods PPL-efficiency trade￾off with LLaMA-1B on C4. Blue bars show validation Perplexity (PPL↓), and pink bars show training time. Mini-batch K-means achieves the best trade-off. where β3 is the EMA decay rate. Subsequently, per￾parameter adaptive learning rates are multiplied by their corresponding group-wise scaling factors as α t l · St l [C t l ] in Algorithm 1 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Convergence and Scaling-up on C4 Pre-training. Validation Perplexity (PPL%↓: lower is better) vs Training Tokens/Parameters. (a) LLaMA-130M and (b) LLaMA-1B training curves demonstrate faster convergence and lower PPL of SGG compared to baselines in both low-rank (Adam…
Figure 5
Figure 5. Figure 5: Learning Rate and Batch Size Scaling-up with Qwen2.5-0.5B SFT on Alpaca. Validation loss↓ vs SFT Learning Rate for Adam and Adam+SGG across various batch sizes (128 to 4096). SGG offers consistent robustness over a wider range of hyper-parameters. 30.0 30.5 31.0 31.5 3…
Figure 6
Figure 6. Figure 6: Ablation of Hyperparameters with LLaMA￾60M and LLaMA-130M pre-training on C4. Validation Perplexity (PPL %↓: lower is better) vs. (a) Recluster Interval T (% total iterations) and (b) EMA Decay β3. The results demonstrate that T ≈ 500 and β3 = 0.99 are the most favorab…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

Reference graph

Works this paper leans on

92 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  4. [4]

    Klaudia Ba azy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. 2024. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604

    arXiv 2024

  5. [5]

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar. 2018. signsgd: compressed optimisation for non-convex problems. In International Conference on Machine Learning

    2018

  6. [6]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, pages 7432--7439

    2020

  7. [7]

    Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024 a . A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering

    2024

  8. [8]

    Yuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He, Ao Tong, Zhenye Gan, Chengjie Wang, and Xiang Bai. 2024 b . Llava-kd: A framework of distilling multimodal large language models. arXiv preprint arXiv:2410.16236

    arXiv 2024

Show all 92 references
  1. [9]

    Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. 2024. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623

    2024

  2. [10]

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. 2023. Symbolic discovery of optimization algorithms. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  3. [11]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  4. [12]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  5. [13]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    2019 arXiv

  6. [14]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

    2018 arXiv

  7. [15]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500

    2023 arXiv

  8. [16]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

    2023

  9. [17]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36

    2024

  10. [18]

    Martin Ester, Hans-Peter Kriegel, J \"o rg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Knowledge Discovery and Data Mining

    1996

  11. [19]

    Boris Ginsburg, Igor Gitman, and Yang You. 2018. Large batch training of convolutional networks with layer-wise adaptive rate scaling. In International Conference on Learning Representations (ICLR)

    2018

  12. [20]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904--6913

    2017

  13. [21]

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018. Vizwiz grand challenge: Answering visual questions from blind people. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3608--3617

    2018

  14. [22]

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354

    2024 arXiv

  15. [23]

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. International Conference on Learning Representations (ICLR)

    2021

  16. [24]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. ArXiv

    2019

  17. [25]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  18. [26]

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933

    2023 arXiv

  19. [27]

    Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wang, and Shiwei Liu. 2025. Spam: Spike-aware adam with momentum reset for stable llm training. arXiv preprint arXiv:2501.06842

    2025 arXiv

  20. [28]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  21. [29]

    Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N Gomez. 2022. Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209.13569

    2022 arXiv

  22. [30]

    Nanda Kambhatla and Todd K. Leen. 1994. Classifying with gaussian mixtures and clusters. In Advances in Neural Information Processing Systems (NeurIPS), page 681–688, Cambridge, MA, USA. MIT Press

    1994

  23. [31]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR)

    2015

  24. [32]

    o pf, Yannic Kilcher, Dimitri Von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri Von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, et al. 2023. Openassistant conversations-democratizing large language model alignment. Advances in Neura...

    2023

  25. [33]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  26. [34]

    Guangyan Li, Yongqiang Tang, and Wensheng Zhang. 2024 a . Lorap: Transformer sub-layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695

    2024 arXiv

  27. [35]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  28. [36]

    Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. 2024 b . Surge phenomenon in optimal learning rate and batch size scaling. arXiv preprint arXiv:2405.14578

    2024 arXiv

  29. [37]

    Siyuan Li, Juanxi Tian, Zedong Wang, Luyuan Zhang, Zicheng Liu, Weiyang Jin, Yang Liu, Baigui Sun, and Stan Z Li. 2024 c . Unveiling the backbone-optimizer coupling bias in visual representation learning. arXiv preprint arXiv:2410.06373

    2024 arXiv

  30. [38]

    Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. 2024 d . Moganet: Multi-order gated aggregation network. In International Conference on Learning Representations (ICLR)

    2024

  31. [39]

    Siyuan Li, Luyuan Zhang, Zedong Wang, Juanxi Tian, Cheng Tan, Zicheng Liu, Chang Yu, Qingsong Xie, Haonan Lu, Haoqian Wang, and Zhen Lei. 2025. Mergevq: A unified framework for visual generation and representation with disentangled token merging and quantization. In Conference...

    2025

  32. [40]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4582--4597

    2021

  33. [41]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023 b . Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  34. [42]

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. 2023. Relora: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Representations

    2023

  35. [43]

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. 2024. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947

    2024 arXiv

  36. [44]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  37. [45]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  38. [46]

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. 2025 a . Muon is scalable for llm training. arXiv preprint arXiv:2502.16982

    2025 arXiv

  39. [47]

    Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020 a . On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations

    2020

  40. [48]

    Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. 2020 b . Understanding the difficulty of training transformers. In Conference on Empirical Methods in Natural Language Processing

    2020

  41. [49]

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024 c . Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

    2024 arXiv

  42. [50]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025 b . Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2025

  43. [51]

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022. A convnet for the 2020s. In Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  44. [52]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR)

    2019

  45. [53]

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022 a . Unified-io: A unified model for vision, language, and multi-modal tasks. In International Conference on Learning Representations (ICLR)

    2022

  46. [54]

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 b . Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 3...

    2022

  47. [55]

    Qijun Luo, Hengxu Yu, and Xiao Li. 2025. Badam: A memory efficient full parameter optimization method for large language models. Advances in Neural Information Processing Systems, 37:24926--24958

    2025

  48. [56]

    Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. 2023. Came: Confidence-guided adaptive memory efficient optimization. arXiv preprint arXiv:2307.02047

    2023 arXiv

  49. [57]

    Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. 2024. Visual perception by large language model's weights. arXiv preprint arXiv:2405.20339

    2024 arXiv

  50. [58]

    James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, pages 281--297

    1967

  51. [59]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    2018 arXiv

  52. [60]

    Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, Binh Tang, Diana Liskovich, Puxin Xu, Yuchen Zhang, Melissa Hall Melanie Kambadur, Stephen Roller, and Susan Zhang. 2023. A theory on...

    2023 arXiv

  53. [61]

    Hamid Nasiri and Peter Garraghan. 2025. Edora: Efficient weight-decomposed low-rank adaptation via singular value decomposition. arXiv preprint arXiv:2501.12067

    2025 arXiv

  54. [62]

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2025. Lisa: layerwise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 37:57018--57049

    2025

  55. [63]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  56. [64]

    Reddi, Satyen Kale, and Surinder Kumar

    Sashank J. Reddi, Satyen Kale, and Surinder Kumar. 2018. On the convergence of adam and beyond. In International Conference on Learning Representations

    2018

  57. [65]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  58. [66]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728

    2019 arXiv

  59. [67]

    D. Sculley. 2010. Web-scale k-means clustering. In International Conference on World Wide Web

    2010

  60. [68]

    Shazeer and Mitchell Stern

    Noam M. Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. ArXiv, abs/1804.04235

    2018 arXiv

  61. [69]

    Fangxun Shu, Yue Liao, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, et al. 2024. Llava-mod: Making llava tiny via moe knowledge distillation. arXiv preprint arXiv:2408.15881

    2024 arXiv

  62. [70]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317--8326

    2019

  63. [71]

    Sinha and Michael P

    Naresh K. Sinha and Michael P. Griscik. 1971. A stochastic approximation method. IEEE Transactions on Systems, Man, and Cybernetics, SMC-1(4):338--344

    1971

  64. [72]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  65. [73]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  66. [74]

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. ArXiv, abs/2409.11321

    2024 arXiv

  67. [75]

    Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  68. [76]

    Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2024. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence

    2024

  69. [77]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. ArXiv, abs/2503.20215

    2025 arXiv

  70. [78]

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  71. [79]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549

    2023 arXiv

  72. [80]

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representati...

    2020

  73. [81]

    Lijun Yu, Jos \'e Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. 2023. Language model beats diffusion--tokenizer is key to visual generation. In International Conference on Learning Representa...

    2023

  74. [82]

    Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. 2025. Mars: Unleashing the power of variance reduction for training large models. In International Conference on Machine Learning (ICML)

    2025

  75. [83]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  76. [84]

    Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, and Jie Tang. 2025 a . Parameter-efficient fine-tuning for foundation models. arXiv preprint arXiv:2501.13787

    2025 arXiv

  77. [85]

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhiquan Luo. 2025 b . Why transformers need adam: A hessian perspective. Advances in Neural Information Processing Systems, 37:131786--131823

    2025

  78. [86]

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. 2024. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793

    2024 arXiv

  79. [87]

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024 a . Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507

    2024 arXiv

  80. [88]

    Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. 2024 b . Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972

    2024 arXiv

  81. [89]

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289

    2024 arXiv

  82. [90]

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang, and Jinwon Lee. 2024 a . Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270

    2024 arXiv

  83. [91]

    Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. 2025. Transformers without normalization. arXiv preprint arXiv:2503.10622

    2025 arXiv

  84. [92]

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhiyuan Xu, and Yaxin Peng. 2024 b . Llava-phi: Efficient multi-modal assistant with small language model. In Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited, pages 18--22

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.