Pith. sign in

REVIEW 2 major objections 4 minor 297 references

SOS-LoRA claims that splitting a fixed-rank LoRA update into several static, orthogonal low-rank experts improves fine-tuning accuracy without changing the parameter budget or inference cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:45 UTC pith:G4VMTBZJ

load-bearing objection Useful and honestly framed reparameterization of LoRA whose empirical claim currently rests on an unverified assumption: that the baselines ran under the same LoRA+ optimizer protocol. the 2 major comments →

arxiv 2607.16252 v1 pith:G4VMTBZJ submitted 2026-06-27 cs.LG cs.AI

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

Yupeng Chang , Yuan Wu , Yi Chang This is my paper
classification cs.LG cs.AI
keywords Low-Rank Adaptationparameter-efficient fine-tuningmulti-scale scalingorthogonal subspacestatic expertsLLM fine-tuningexpressivity equivalence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SOS-LoRA claims that the standard LoRA recipe leaves accuracy on the table: under a fixed rank budget, a single low-rank pathway forces heterogeneous adaptation behaviors to share the same input directions and interfere with one another. The paper reparameterizes the same rank-rtot update as a sum of K always-on low-rank experts, each with a fixed scale and input directions explicitly encouraged to stay orthogonal. It argues that this changes only the optimization geometry, not the set of representable updates, and that everything can be merged back into the base weights after training. Across Llama 2/3 reasoning tasks, RoBERTa on GLUE, and GSM8K/MATH, the paper reports consistent gains over matched-budget LoRA and recent variants, with no added inference latency after merging.

Core claim

The paper's central claim is that the effectiveness of low-rank fine-tuning depends not only on the rank budget but on how that budget is organized across input-side directions. SOS-LoRA replaces a single adapter AB with K adapters AkBk, assigns each expert a fixed scale sk chosen so the average matches standard LoRA scaling, initializes the Ak factors to mutually orthogonal blocks with Bk=0 (so the initial update is zero), and adds a regularizer that penalizes cross-expert overlap of input directions. Proposition 1 shows that the merged update sum_k skAkBk equals a single rank-rtot product, so the method does not enlarge the hypothesis class. The paper concludes from its experiments that th

What carries the argument

The central mechanism is the reparameterization Delta W = sum_{k=1}^{K} s_k A_k B_k, with A_k in R^{m x r'}, B_k in R^{r' x n}, and r' = rtot/K. It combines: fixed multi-scale scales s_k = (alpha / rtot) * (gamma_k / gamma_bar) with linearly spaced gamma_k in [1, gamma_max]; cross-expert orthogonal initialization via a thin QR decomposition of a random Gaussian matrix, partitioned into K blocks, with B_k initialized to zero; and an orthogonality regularizer L_orth = (lambda / r'^2) * sum_{k<ell} ||A_tilde_k^T A_tilde_ell||_F^2 on column-wise normalized input-side factors. The concatenation identity A_cat B_cat proves expressivity equivalence with standard LoRA, while the fixed scales linearl

Load-bearing premise

The central empirical claim assumes that every comparison isolates the SOS-LoRA parameterization, in particular that the same optimizer-side settings, including the LoRA+ asymmetric learning-rate treatment mentioned in the method section, were applied to all baselines and not only to SOS-LoRA.

What would settle it

Re-run the main Llama 2-7B benchmark grid with two additional control arms: (a) all baselines using the same LoRA+ learning-rate asymmetry that SOS-LoRA reports, and (b) SOS-LoRA trained with plain AdamW without that asymmetry. If the claimed average gain over LoRA collapses to noise in either arm, the SOS reparameterization itself would not be the cause of the reported improvements.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At identical total rank and parameter count, the merged SOS-LoRA update is a single matrix added to the pretrained weights, so inference latency, FLOPs, and memory after merging match standard LoRA.
  • If the reported numbers hold, practitioners can swap a LoRA adapter for an SOS-LoRA adapter without changing model size, training data, or deployment pipeline.
  • The decomposition produces faster early training-loss reduction than LoRA at both rtot=32 and rtot=128 on GSM8K, suggesting improved optimization dynamics under matched budgets.
  • The reported gains replicate across decoder-only Llama 2/3 backbones, encoder-only RoBERTa on GLUE, and math reasoning, with the largest margin on GSM8K (57.03 vs. 42.68 with LoRA).
  • The method deliberately does not enlarge the set of representable rank-rtot updates; all gains are attributed to the optimization-oriented inductive bias, not to extra capacity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is to combine the SOS-LoRA expert decomposition with SVD-based initialization, such as PiSSA-style initialization, to see whether orthogonal static experts and singular-value initialization compound or whether their benefits overlap.
  • If the optimization-bias explanation is correct, the method's advantage should be largest at small ranks and shrink as rtot approaches the full-rank capacity; a dedicated scaling-law study across ranks could pin this down more sharply than the paper's sweeps.
  • The fixed scales are chosen linearly; the scale-separated gradient mechanism suggests that the schedule shape (geometric, learned, or optimizer-normalized) may be a productive search space beyond the gamma_max=2.5 default.
  • Because the method is expressed purely as a sum of low-rank products, it can plausibly be composed with other PEFT techniques, such as layer-wise scaling or different adapters for attention versus MLP, but the paper does not test those combinations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces SOS-LoRA, which replaces a rank-r_tot LoRA adapter with K static low-rank experts (r'=r_tot/K), fixed multi-scale scales s_k, orthogonal initialization, and a cross-expert orthogonality regularizer. It proves that the reparameterization is expressively equivalent to standard LoRA (Proposition 1), analyzes gradient scaling (Eq. 4), and reports experiments on Llama 2/3, RoBERTa-base/GLUE, and GSM8K/MATH, claiming consistent gains over matched-budget LoRA baselines at equal inference cost. The mathematical development is elementary and correct; the key open issue is whether the experimental protocol controlled the optimizer setting across baselines.

Significance. If confirmed, the contribution is practically useful: a drop-in, mergeable PEFT method with no inference overhead and a clear inductive bias. The paper is commendable for releasing code, for the honest Limitations section (geometric proxies, untested larger models), and for evaluating against standard external benchmarks. The internal analyses in Figs. 2-3 and the ablations in Fig. 5 support the mechanism. The main significance is conditional on the baseline optimizer protocol being matched; otherwise the core empirical claim is not established.

major comments (2)
  1. [Section 3.2.1, Section 4, Appendix A.5] The paper states in Section 3.2.1 that 'unless stated otherwise, we adopt a LoRA+ optimizer setting that uses different effective learning rates for the LoRA factors,' but Section 4 reports only 'We optimize with AdamW.' Appendix A.5 lists the matched conditions (backbone, data, steps, adapted modules, rank) but does not say whether LoRA, PiSSA, DoRA, Delta-LoRA, MELoRA, or the other baselines also used the LoRA+ asymmetric learning rates. Since Hayou et al. (2024) show that LoRA+ alone improves LoRA, a comparison in which only SOS-LoRA receives LoRA+ cannot isolate the proposed parameterization. This is the load-bearing point for the paper's central empirical claim. Please state explicitly, for every method and every table, whether LoRA+ was used, and include the optimizer configuration in the released configs; if the baselines did not use LoRA+, rerun the comparisons under a common opt
  2. [Section 4, Tables 1 and 2] The text says reported results are averaged over three seeds and 'mean ± standard deviation where applicable,' but Tables 1 and 2 contain no variance information. The smallest claimed advantage over the strongest baseline is 0.7 points on Llama 3-8B (Table 1) and 0.8 points on the GLUE average (Table 2); without standard deviations, confidence intervals, or a significance test, these specific gains are not assessable. Please add per-task standard deviations (or place them in an appendix) and state whether the main conclusions are robust to seed variability.
minor comments (4)
  1. [Figure 5] In the supplied version, the caption and axis labels of panels (c) and (d) contain encoding artifacts; please ensure the final PDF renders correctly and the curves are legible.
  2. [Figure 5(c) and Table 3] The statement 'matches or surpasses Full FT at higher ranks in our setup' is stronger than the evidence, because Full FT used a modified recipe to avoid OOM and is marked as not directly comparable (Appendix A.7). Please qualify this sentence.
  3. [Section 3.2.1] The optional input-independent calibration is described but it is not stated whether it was used in any reported experiment. Please specify explicitly, even if it was not used.
  4. [Section 4.5] The ablation conclusions in Fig. 5(a-b) are stated without numeric values; a small table with the exact stepwise ablation scores would make the contribution of each component quantitative.

Circularity Check

0 steps flagged

No significant circularity: the theoretical claim is an algebraic identity and the empirical claims are evaluated on external benchmarks.

full rationale

The paper's only formal claim, Proposition 1, states that sum_{k=1}^K s_k A_k B_k = A_cat B_cat after concatenating the A factors and row-scaling the B factors. This is a definitional matrix identity, explicitly framed as expressivity equivalence, so it does not derive the method's success from its own construction; it simply establishes that the reparameterization lies in the same hypothesis class as LoRA. The central empirical contributions are benchmark comparisons against external datasets (BoolQ, GLUE, GSM8K, MATH, etc.) and matched-budget PEFT baselines; these results are not functions of any parameter fitted to the target metric. Hyperparameters (gamma_max, lambda, K) are declared defaults with ablations, and the analyses in Figs. 2-4 measure the exact quantities the regularizer and scaling scheme target (cross-expert squared-cosine similarity and relative update magnitudes); these are internal consistency checks rather than predictions that reduce to fitted values. The self-citations (Chang et al., 2025, 2026) appear only as related-work context and are not load-bearing. The limitations section honestly notes the added hyperparameters and the use of geometric proxies. The only substantive concern is an experimental-protocol gap: Section 3.2.1 states that 'unless stated otherwise, we adopt a LoRA+ optimizer setting' while Section 4 reports only 'We optimize with AdamW', and Appendix A.5 does not explicitly confirm that all baselines received LoRA+ asymmetric learning rates. That is a potential fairness/confound issue about whether the comparisons isolate the SOS parameterization, but it is not a circularity step under the definitions used here: no prediction is equivalent by construction to its input, and no load-bearing claim rests on a self-citation. Accordingly, no circular step is identified.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper's central claims are empirical; the math (Proposition 1 and the gradient-scaling identity in Eq. 4) is elementary and correct. Its load-bearing premises are hyperparameter choices (γ_max, λ, K) and domain assumptions about how AdamW and geometric similarity relate to optimization outcomes. No invented entities are introduced: the 'static expert' is a reparameterization of the existing LoRA matrices with no falsifiable handle outside the benchmark results themselves.

free parameters (5)
  • γ_max (multi-scale range) = 2.5
    Hand-chosen default for the fixed scale schedule in Eq. 3; ablated (uniform vs. full) but not derived from any principle. The paper's mechanism story depends on the spread.
  • λ (orthogonality regularizer weight) = 0.01
    Hand-chosen weight for L_orth in Eq. 5; no sensitivity sweep for λ is shown, yet the Fig. 2 interpretation relies on it being large enough to act.
  • K (expert count) = 4
    Chosen via the Fig. 5(d) sweep on GSM8K (r=128), where performance 'peaks at K=4', then fixed as default everywhere else. Hyperparameter selection on one benchmark that is also used for headline reporting.
  • LoRA+ learning-rate asymmetry = not specified
    §3.2.1 adopts 'different effective learning rates for the LoRA factors' per Hayou et al. 2024, but the ratio and its application to baselines are never stated.
  • r_tot default rank = 8
    Standard rank budget chosen for main results; not fitted, but the choice interacts with K=4 giving r'=2, and the cost analysis (Table 3) uses r=128, so the two headline regimes differ in rank.
axioms (5)
  • standard math LoRA expressivity and mergeability: any rank-r update (α/r)AB can be added to W0 with no inference overhead (Hu et al. 2022)
    Foundational premise for the whole method, invoked in §1 and §3; accepted.
  • domain assumption A single shared input-side projection couples heterogeneous behaviors and induces optimization interference under a fixed rank budget
    The motivating problem statement (§1, §3.1). Plausible but unproven; the paper offers no direct measurement of 'interference' beyond the geometric proxies it later admits are indirect (Limitations section).
  • domain assumption With AdamW, fixed multi-scale scales produce meaningfully separated expert dynamics
    Eq. 4 shows s_k rescales raw gradients linearly, but the paper itself notes adaptive optimizers make updates 'not strictly linear in s_k' (§3.2.1). The benefit of the scaling rests on an empirical assumption tested only by the FULL vs UNIFORM ablation.
  • domain assumption Cross-expert squared-cosine similarity of input directions is a valid proxy for harmful interference
    The regularizer (Eq. 5), the analysis (Fig. 2), and the interpretation all use this proxy; the Limitations section concedes it 'does not directly measure functional specialization'.
  • domain assumption Matched training budget and unified evaluation pipeline give comparable results for all baselines
    The entire comparison rests on this; it is where the LoRA+ ambiguity (§3.2.1) lives. If baselines were not given the same optimizer protocol, the benchmark comparisons are confounded.

pith-pipeline@v1.3.0-alltime-deepseek · 16939 in / 22202 out tokens · 189835 ms · 2026-08-02T09:45:37.311551+00:00 · methodology

0 comments
read the original abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. Under a fixed rank budget, LoRA parameterizes each adapted weight through a single low-dimensional input-side pathway, which may couple heterogeneous behaviors through shared input directions and induce interference during optimization. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that reparameterizes a rank-rtot update as a sum of K static (always-on, non-routed) low-rank experts. SOS-LoRA (i) decomposes the total rank across experts, (ii) applies a fixed multi-scale scaling scheme to encourage scale-separated optimization dynamics, and (iii) promotes diverse input-side directions via cross-expert orthogonal initialization and a lightweight regularizer. SOS-LoRA remains fully mergeable, adding no inference-time parameters or latency after merging. Experiments on reasoning and knowledge-intensive benchmarks (Llama 2/3), encoder-based NLU (GLUE), and math reasoning (GSM8K/MATH) show consistent gains over matched-budget LoRA baselines and recent variants. Code is available at https://github.com/llm172/sos-lora.

Figures

Figures reproduced from arXiv: 2607.16252 by Yi Chang, Yuan Wu, Yupeng Chang.

Figure 1
Figure 1. Figure 1: Overview of Static Orthogonal Subspace LoRA (SOS-LoRA) with row-vector convention [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Orthogonal regularization reduces cross [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Optimization dynamics on GSM8K. Training [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Analysis of SOS-LoRA on Llama 2-7B. (a) Ablation on reasoning and knowledge-intensive benchmarks [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

297 extracted references · 122 linked inside Pith

  1. [1]

    2019 , eprint=

    Parameter-Efficient Transfer Learning for NLP , author=. 2019 , eprint=

    2019

  2. [2]

    2026 , url=

    Yupeng Chang and Yi Chang and Yuan Wu , booktitle=. 2026 , url=

    2026

  3. [3]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization , author=. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages=

    2025

  4. [4]

    Advances in Neural Information Processing Systems , year=

    AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping , author=. Advances in Neural Information Processing Systems , year=

  5. [5]

    2026 , eprint=

    NeuReasoner: Towards Explainable, Controllable, and Unified Reasoning via Mixture-of-Neurons , author=. 2026 , eprint=

    2026

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning , author=. Advances in Neural Information Processing Systems , volume=

  7. [7]

    arXiv preprint arXiv:2309.12284 , year=

    Metamath: Bootstrap your own mathematical questions for large language models , author=. arXiv preprint arXiv:2309.12284 , year=

    Pith/arXiv arXiv

  8. [9]

    arXiv preprint arXiv:2312.03732 , year=

    A rank stabilization scaling factor for fine-tuning with lora , author=. arXiv preprint arXiv:2312.03732 , year=

    Pith/arXiv arXiv

  9. [10]

    2025 , eprint=

    TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts , author=. 2025 , eprint=

    2025

  10. [11]

    , author=

    Lora: Low-rank adaptation of large language models. , author=. ICLR , volume=

  11. [12]

    arXiv preprint arXiv:1701.06538 , year=

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer , author=. arXiv preprint arXiv:1701.06538 , year=

    Pith/arXiv arXiv

  12. [13]

    Advances in neural information processing systems , volume=

    Qlora: Efficient finetuning of quantized llms , author=. Advances in neural information processing systems , volume=

  13. [14]

    Advances in neural information processing systems , volume=

    Fourier features let networks learn high frequency functions in low dimensional domains , author=. Advances in neural information processing systems , volume=

  14. [15]

    Bloom: A 176b-parameter open-access multilingual language model , author=

  15. [16]

    arXiv preprint arXiv:2108.12409 , year=

    Train short, test long: Attention with linear biases enables input length extrapolation , author=. arXiv preprint arXiv:2108.12409 , year=

    Pith/arXiv arXiv

  16. [17]

    2016 , eprint=

    Pointer Sentinel Mixture Models , author=. 2016 , eprint=

    2016

  17. [18]

    OpenWebText Corpus , author=

  18. [19]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  19. [20]

    Anonymous , booktitle=. Hi. 2025 , url=

    2025

  20. [21]

    arXiv preprint arXiv:2307.01952 , year=

    Sdxl: Improving latent diffusion models for high-resolution image synthesis , author=. arXiv preprint arXiv:2307.01952 , year=

    Pith/arXiv arXiv

  21. [22]

    arXiv preprint arXiv:2212.07143 , year=

    Reproducible scaling laws for contrastive language-image learning , author=. arXiv preprint arXiv:2212.07143 , year=

    Pith/arXiv arXiv

  22. [23]

    Improving language understanding by generative pre-training , author=

  23. [24]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  24. [25]

    Advances in neural information processing systems , volume=

    Llm-pruner: On the structural pruning of large language models , author=. Advances in neural information processing systems , volume=

  25. [26]

    arXiv preprint arXiv:2310.05015 , year=

    Compresso: Structured pruning with collaborative prompting learns compact large language models , author=. arXiv preprint arXiv:2310.05015 , year=

    Pith/arXiv arXiv

  26. [27]

    arXiv preprint arXiv:2405.05254 , year=

    You only cache once: Decoder-decoder architectures for language models , author=. arXiv preprint arXiv:2405.05254 , year=

    Pith/arXiv arXiv

  27. [28]

    arXiv preprint arXiv:2405.12981 , year=

    Reducing Transformer Key-Value Cache Size with Cross-Layer Attention , author=. arXiv preprint arXiv:2405.12981 , year=

    Pith/arXiv arXiv

  28. [29]

    arXiv preprint arXiv:2405.14366 , year=

    MiniCache: KV Cache Compression in Depth Dimension for Large Language Models , author=. arXiv preprint arXiv:2405.14366 , year=

    Pith/arXiv arXiv

  29. [30]

    arXiv preprint arXiv:2406.09297 , year=

    MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding , author=. arXiv preprint arXiv:2406.09297 , year=

    Pith/arXiv arXiv

  30. [31]

    arXiv preprint arXiv:2210.17323 , year=

    Gptq: Accurate post-training quantization for generative pre-trained transformers , author=. arXiv preprint arXiv:2210.17323 , year=

    Pith/arXiv arXiv

  31. [32]

    International Conference on Machine Learning , pages=

    Smoothquant: Accurate and efficient post-training quantization for large language models , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  32. [33]

    int8 (): 8-bit matrix multiplication for transformers at scale , author=

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale , author=. Advances in Neural Information Processing Systems , volume=

  33. [34]

    arXiv preprint arXiv:2402.02750 , year=

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache , author=. arXiv preprint arXiv:2402.02750 , year=

    Pith/arXiv arXiv

  34. [35]

    arXiv preprint arXiv:2401.18079 , year=

    Kvquant: Towards 10 million context length llm inference with kv cache quantization , author=. arXiv preprint arXiv:2401.18079 , year=

    Pith/arXiv arXiv

  35. [36]

    International conference on machine learning , pages=

    Transformers are rnns: Fast autoregressive transformers with linear attention , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  36. [37]

    arXiv preprint arXiv:2006.04768 , year=

    Linformer: Self-attention with linear complexity , author=. arXiv preprint arXiv:2006.04768 , year=

    Pith/arXiv arXiv 2006

  37. [38]

    arXiv preprint arXiv:2305.13048 , year=

    Rwkv: Reinventing rnns for the transformer era , author=. arXiv preprint arXiv:2305.13048 , year=

    Pith/arXiv arXiv

  38. [39]

    arXiv preprint arXiv:2312.00752 , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    Pith/arXiv arXiv

  39. [40]

    arXiv preprint arXiv:2402.19427 , year=

    Griffin: Mixing gated linear recurrences with local attention for efficient language models , author=. arXiv preprint arXiv:2402.19427 , year=

    Pith/arXiv arXiv

  40. [41]

    arXiv preprint arXiv:2407.14057 , year=

    Lazyllm: Dynamic token pruning for efficient long context llm inference , author=. arXiv preprint arXiv:2407.14057 , year=

    Pith/arXiv arXiv

  41. [42]

    arXiv preprint arXiv:2407.20485 , year=

    A2sf: Accumulative attention scoring with forgetting factor for token pruning in transformer decoder , author=. arXiv preprint arXiv:2407.20485 , year=

    Pith/arXiv arXiv

  42. [43]

    arXiv preprint arXiv:2404.14469 , year=

    Snapkv: Llm knows what you are looking for before generation , author=. arXiv preprint arXiv:2404.14469 , year=

    Pith/arXiv arXiv

  43. [44]

    Advances in Neural Information Processing Systems , volume=

    H2o: Heavy-hitter oracle for efficient generative inference of large language models , author=. Advances in Neural Information Processing Systems , volume=

  44. [45]

    arXiv preprint arXiv:1911.02150 , year=

    Fast transformer decoding: One write-head is all you need , author=. arXiv preprint arXiv:1911.02150 , year=

    Pith/arXiv arXiv 1911

  45. [46]

    arXiv preprint arXiv:2405.04434 , year=

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model , author=. arXiv preprint arXiv:2405.04434 , year=

    Pith/arXiv arXiv

  46. [47]

    arXiv preprint arXiv:2406.07056 , year=

    Effectively Compress KV Heads for LLM , author=. arXiv preprint arXiv:2406.07056 , year=

    Pith/arXiv arXiv

  47. [48]

    arXiv preprint arXiv:2401.15024 , year=

    Slicegpt: Compress large language models by deleting rows and columns , author=. arXiv preprint arXiv:2401.15024 , year=

    Pith/arXiv arXiv

  48. [49]

    arXiv preprint arXiv:2310.06694 , year=

    Sheared llama: Accelerating language model pre-training via structured pruning , author=. arXiv preprint arXiv:2310.06694 , year=

    Pith/arXiv arXiv

  49. [50]

    arXiv preprint arXiv:2306.11695 , year=

    A simple and effective pruning approach for large language models , author=. arXiv preprint arXiv:2306.11695 , year=

    Pith/arXiv arXiv

  50. [51]

    arXiv preprint arXiv:2106.10199 , year=

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models , author=. arXiv preprint arXiv:2106.10199 , year=

    arXiv

  51. [52]

    arXiv preprint arXiv:2305.16597 , year=

    Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models , author=. arXiv preprint arXiv:2305.16597 , year=

    Pith/arXiv arXiv

  52. [53]

    arXiv preprint arXiv:2004.12406 , year=

    Masking as an efficient alternative to finetuning for pretrained language models , author=. arXiv preprint arXiv:2004.12406 , year=

    Pith/arXiv arXiv 2004

  53. [54]

    Advances in Neural Information Processing Systems , volume=

    Training neural networks with fixed sparse masks , author=. Advances in Neural Information Processing Systems , volume=

  54. [55]

    arXiv preprint arXiv:2110.07560 , year=

    Composable sparse fine-tuning for cross-lingual transfer , author=. arXiv preprint arXiv:2110.07560 , year=

    Pith/arXiv arXiv

  55. [56]

    arXiv preprint arXiv:2109.05687 , year=

    Raise a child in large language model: Towards effective and generalizable fine-tuning , author=. arXiv preprint arXiv:2109.05687 , year=

    Pith/arXiv arXiv

  56. [57]

    arXiv preprint arXiv:2012.07463 , year=

    Parameter-efficient transfer learning with diff pruning , author=. arXiv preprint arXiv:2012.07463 , year=

    Pith/arXiv arXiv 2012

  57. [58]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    On the effectiveness of parameter-efficient fine-tuning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  58. [59]

    arXiv preprint arXiv:2101.00121 , year=

    Warp: Word-level adversarial reprogramming , author=. arXiv preprint arXiv:2101.00121 , year=

    Pith/arXiv arXiv

  59. [60]

    arXiv preprint arXiv:2101.00190 , year=

    Prefix-tuning: Optimizing continuous prompts for generation , author=. arXiv preprint arXiv:2101.00190 , year=

    Pith/arXiv arXiv

  60. [61]

    AI Open , year=

    GPT understands, too , author=. AI Open , year=

  61. [62]

    arXiv preprint arXiv:2110.07904 , year=

    Spot: Better frozen model adaptation through soft prompt transfer , author=. arXiv preprint arXiv:2110.07904 , year=

    Pith/arXiv arXiv

  62. [63]

    arXiv preprint arXiv:2205.11961 , year=

    ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts , author=. arXiv preprint arXiv:2205.11961 , year=

    Pith/arXiv arXiv

  63. [64]

    arXiv preprint arXiv:2303.02861 , year=

    Multitask prompt tuning enables parameter-efficient transfer learning , author=. arXiv preprint arXiv:2303.02861 , year=

    Pith/arXiv arXiv

  64. [65]

    Advances in Neural Information Processing Systems , volume=

    Controlling text-to-image diffusion by orthogonal finetuning , author=. Advances in Neural Information Processing Systems , volume=

  65. [66]

    arXiv preprint arXiv:2311.06243 , year=

    Parameter-efficient orthogonal finetuning via butterfly factorization , author=. arXiv preprint arXiv:2311.06243 , year=

    Pith/arXiv arXiv

  66. [67]

    arXiv preprint arXiv:2405.17484 , year=

    Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation , author=. arXiv preprint arXiv:2405.17484 , year=

    Pith/arXiv arXiv

  67. [68]

    arXiv preprint arXiv:2305.13245 , year=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    Pith/arXiv arXiv

  68. [69]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  69. [70]

    Q-Tuning: Queue-based prompt tuning for lifelong few-shot language learning , author=

  70. [71]

    Findings of ACL 2022 , pages=

    Plug-and-Play Adaptation for Continuously-updated QA , author=. Findings of ACL 2022 , pages=

    2022

  71. [72]

    arXiv preprint arXiv:2306.11250 , year=

    Inrank: Incremental low-rank learning , author=. arXiv preprint arXiv:2306.11250 , year=

    Pith/arXiv arXiv

  72. [73]

    NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models , year=

    Continual Learning with Low Rank Adaptation , author=. NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models , year=

    2023

  73. [74]

    arXiv preprint arXiv:2404.00228 , year=

    InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning , author=. arXiv preprint arXiv:2404.00228 , year=

    Pith/arXiv arXiv

  74. [75]

    arXiv preprint arXiv:2402.11260 , year=

    MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning , author=. arXiv preprint arXiv:2402.11260 , year=

    Pith/arXiv arXiv

  75. [76]

    arXiv preprint arXiv:2402.12220 , year=

    Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting , author=. arXiv preprint arXiv:2402.12220 , year=

    Pith/arXiv arXiv

  76. [77]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=

  77. [78]

    Proceedings of the European conference on computer vision , pages=

    Memory aware synapses: Learning what (not) to forget , author=. Proceedings of the European conference on computer vision , pages=

  78. [79]

    Proceedings of Conference on Empirical Methods in Natural Language Processing , pages=

    Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting , author=. Proceedings of Conference on Empirical Methods in Natural Language Processing , pages=

  79. [80]

    Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Continual Learning for Text Classification with Information Disentanglement Based Regularization , author=. Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

  80. [81]

    Proceedings of Conference on Empirical Methods in Natural Language Processing , pages=

    Continual Named Entity Recognition without Catastrophic Forgetting , author=. Proceedings of Conference on Empirical Methods in Natural Language Processing , pages=

Showing first 80 references.