REVIEW 5 cited by
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. During finetuning, the quantized component remains fixed and only the low-rank component is updated. We present an integer linear programming formulation of the quantization component which enables dynamic configuration of quantization parameters (e.g., bit-width, block size) for each matrix given an overall target memory budget. We further explore a data-aware version of the algorithm which uses an approximation of the Fisher information matrix to weight the reconstruction objective during matrix decomposition. Experiments on finetuning RoBERTa and LLaMA-2 (7B and 70B) demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines and enables aggressive quantization to sub-3 bits with only minor performance degradations. When finetuned on a language modeling calibration dataset, LQ-LoRA can also be used for model compression; in this setting our 2.75-bit LLaMA-2-70B model (which has 2.85 bits on average when including the low-rank components and requires 27GB of GPU memory) performs respectably compared to the 16-bit baseline.
Forward citations
Cited by 5 Pith papers
-
Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
Q-resafe restores much of the safety lost in quantized LLMs by distilling the original model's responses through DPO while selectively updating only safety-critical weights.
-
Efficient Reasoning on the Edge
LoRA adapters, budget-forced GRPO, dynamic switching, parallel verification and FPTQuant enable practical chain-of-thought reasoning on quantized Qwen2.5-7B for edge devices.
-
Quantized but Deceptive? A Multi-Dimensional Truthfulness Evaluation of Quantized LLMs
The study introduces TruthfulnessEval and reports that 4-bit quantization preserves simple true/false accuracy, but explicit 'lie' prompts make quantized and full-precision LLMs output falsehoods even when internal pr...
-
Quaff: Quantized Parameter-Efficient Fine-Tuning under Outlier Spatial Stability Hypothesis
Quaff shows that activation outlier channels keep their spatial positions during LLM fine-tuning, and exploits this stability to cut fine-tuning memory and latency with INT8 quantization while matching or beating full...
-
Efficient Large Language Models with Zero-Shot Adjustable Acceleration
A fine-tuning policy that samples a token-preservation rate from a uniform distribution lets a model be accelerated at inference by setting a pruning knob, without retraining.
Discussion (0). Sign in to comment.