REVIEW 3 cited by
Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we present an efficient method for storing fine-tuned models by leveraging the low-rank properties of weight residuals. Our key observation is that weight residuals in large overparameterized models exhibit even stronger low-rank characteristics. Based on this insight, we propose Efficient Residual Encoding (ERE), a novel approach that achieves efficient storage of fine-tuned model weights by approximating the low-rank weight residuals. Furthermore, we analyze the robustness of weight residuals and push the limit of storage efficiency by utilizing additional quantization and layer-wise rank allocation. Our experimental results demonstrate that our method significantly reduces memory footprint while preserving performance in various tasks and modalities. We release our code.
Forward citations
Cited by 3 Pith papers
-
Multi-objective Large Language Model Alignment with Hierarchical Experts
HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.
-
RanDeS: Randomized Delta Superposition for Multi-Model Compression
Randomized layer shuffling and column-wise sign flips make interfering task deltas nearly orthogonal, allowing one compressed superset to retrieve near-fine-tuned accuracy for many tasks.
-
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
ImPart sparsifies delta weights in SVD space by assigning lower drop rates to high-importance singular vectors, reporting better accuracy than DARE and LowRank at high compression ratios.
Discussion (0). Continue with ORCID to comment.