REVIEW 8 cited by
BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This work presents BAdam, an optimization method that leverages the block coordinate descent (BCD) framework with Adam's update rule. BAdam offers a memory efficient approach to the full parameter finetuning of large language models. We conduct a theoretical convergence analysis for BAdam in the deterministic case. Experimentally, we apply BAdam to finetune the Llama 3-8B and Llama 3-70B models using a single RTX3090-24GB GPU and 4 A100-80GB GPUs, respectively. The results confirm BAdam's efficiency in terms of memory usage, running time, and optimization capability. Furthermore, the downstream performance evaluation based on MT-bench and math benchmarks shows that BAdam outperforms existing memory efficient baselines such as LoRA. It also demonstrates that BAdam can achieve comparable or even superior performance compared to Adam. Finally, the ablation study using SGD's update rule illustrates the suitability of BCD for finetuning LLMs. Our code can be easily integrated into any PyTorch-based codebase and is available at https://github.com/Ledzy/BAdam.
Forward citations
Cited by 8 Pith papers
-
Low-rank Momentum Factorization for Memory Efficient Training
MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.
-
Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale
A unified analysis of subspace perturbations in zero-order optimization identifies subspace alignment as the key driver of convergence, and leads to MeZO-BCD, a block-coordinate method with up to 2.77x wall-clock spee...
-
SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generation
SABER poisons a chain-of-thought generator so that emphasized trigger words in a prompt make AI coding assistants write subtly incorrect code.
-
ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models
ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.
-
GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
GaLore 2 pre-trains a Llama 7B model on 500B tokens using low-rank gradient projection, matching an 8-bit Adam baseline while using less optimizer memory.
-
Geometrically Principled Randomized Optimization for Efficient LLM Training
Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.
-
Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
A benchmark and two low-cost tricks (weight refactorization and momentum reset) that make low-rank LLM pre-training competitive with GaLore and Fira at about 25% lower memory.
-
Systematic Optimization of Open Source Large Language Models for Mathematical Reasoning
A hyperparameter search for LLM math reasoning that reports simulated, not measured, performance gains.
Discussion (0). Continue with ORCID to comment.