Pith. sign in

REVIEW 21 cited by

One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2306.07967 v2 pith:5ITIMXHN submitted 2023-06-13 cs.LG cs.AIcs.CV

Arnav Chavan , Zhuang Liu , Deepak Gupta , Eric Xing , Zhiqiang Shen This is my paper · ORCID
classification cs.LGcs.AIcs.CV
keywords gloralorageneralizedtasksactivationsadaptationdomainfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured vision benchmarks, achieving superior accuracy with fewer parameters and computations. The proposed method on LLaMA-1 and LLaMA-2 also show considerable enhancements compared to the original LoRA in the language domain. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications. Code and models are available at: https://github.com/Arnav0400/ViT-Slim/tree/master/GLoRA.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A rehearsal-free open-world detector using collaborative LoRA adapters and dual-stage objectness modeling outperforms exemplar-replay OWOD methods on standard benchmarks.

  2. TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    TARA adds token-focused masking and a token alignment loss to LoRA adapters, allowing several independently trained personalized adapters to be composed with less identity loss and feature leakage.

  3. ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.

  4. Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SR-LoRA sets each LoRA layer's rank to the stable rank of that layer's pretrained weight matrix, improving few-shot transfer on large domain gaps without rank search.

  5. DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DA-VPT guides visual prompts with a proxy-anchor metric loss and dynamic class-to-prompt clustering, reporting consistent gains over VPT baselines across classification and segmentation.

  6. MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance

    cs.CV 2025-05 conditional novelty 6.0 of 10

    MultLFG merges multiple LoRA adapters by adaptively weighting them in wavelet frequency subbands per denoising timestep, improving multi-concept composition on the ComposLoRA benchmark compared to prior training-free methods.

  7. Cached Multi-Lora Composition for Multi-Concept Image Generation

    cs.CV 2025-02 conditional novelty 6.0 of 10

    CMLoRA schedules adapter activation by high- and low-frequency content and caches non-dominant adapters, improving multi-LoRA composition scores while not consistently reducing compute versus all baselines.

  8. FineGates: LLMs Finetuning with Compression using Stochastic Gates

    cs.LG 2024-12 conditional novelty 6.0 of 10

    FineGates uses stochastic gates to prune structured rows and columns of a transformer during fine-tuning, matching LoRA accuracy while removing up to 20-40% of weights.

  9. A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Search

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLM embeddings plus Bayesian optimization find better LoRA hyperparameters in ~30 proxy trials than standard published settings.

  10. Language-Aware Information Maximization for Transductive Few-Shot CLIP

    cs.CV 2025-08 conditional novelty 5.0 of 10

    LIMO, a transductive loss combining mutual information, zero-shot KL regularization, and LoRA, sets new state-of-the-art few-shot accuracy for CLIP on 11 datasets.

  11. EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices

    cs.DC 2025-07 conditional novelty 5.0 of 10

    EdgeLoRA combines automatic adapter routing, LRU caching with a memory pool, and grouped LoRA batching to serve thousands of LoRA adapters on edge devices with up to 4x higher throughput than llama.cpp.

  12. Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergence

    cs.LG 2025-06 conditional novelty 5.0 of 10

    CorDA++ uses data-driven SVD to initialize LoRA adapters, adding per-layer covariance selection and rank allocation that reduce forgetting and speed convergence compared to LoRA, PiSSA, QLoRA, and other baselines.

  13. EfficientLLM: Efficiency in Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.

  14. SuperCode: Sustainability PER AI-driven CO-DEsign

    astro-ph.IM 2024-12 unverdicted novelty 5.0 of 10

    The paper proposes an AI-driven hardware-software-science co-design methodology for radio astronomy, using sustainability as the key performance indicator, with no empirical results yet.

  15. ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

    cs.CV 2024-12 conditional novelty 5.0 of 10

    ALoRE aggregates multiple low-rank experts in a Kronecker-product space and merges them into the frozen backbone, reporting top accuracy on FGVC and VTAB-1k with only 0.15M trainable parameters.

  16. PETALface: Parameter Efficient Transfer Learning for Low-resolution Face Recognition

    cs.CV 2024-12 conditional novelty 5.0 of 10

    PETALface adapts pre-trained face-recognition models to low-resolution images by blending two image-quality-weighted LoRA modules, improving low-res benchmarks while preserving high-res accuracy with 0.48% trainable p...

  17. Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networks

    eess.IV 2025-07 conditional novelty 4.0 of 10

    TenVOO represents 3D convolution weight updates as tensor networks, fine-tuning a brain MRI DDPM with only 0.3% of full trainable parameters while achieving competitive or better structural similarity on ADNI, PPMI, a...

  18. Regularized Low-Rank Adaptation for Few-Shot Organ Segmentation

    cs.CV 2025-07 reject novelty 4.0 of 10

    ARENA, a low-rank adaptation method with l1-regularized singular values, reportedly improves few-shot organ segmentation, but its described update rule is internally inconsistent.

  19. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

  20. KARST: Multi-Kernel Kronecker Adaptation with Re-Scaling Transmission for Visual Classification

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A sum of low-rank Kronecker adapters plus channel-wise re-scaling gives small average accuracy gains over prior PEFT methods on visual classification benchmarks.

  21. Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning

    cs.CV 2025-01 conditional novelty 4.0 of 10

    EMTAL turns pretrained ViT feed-forward networks into low-rank LoRA-tuned experts, stabilizes converged tasks with an EMA-logit distillation loss, and merges everything back into the original network for fast inference.

Pith tools