Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

Huazheng Wang; Pengcheng Wang; Saurabh Bagchi; Xiang Li

REVIEW 4 major objections 5 minor 45 references

Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that rank-1 task residuals trained on a pinned domain centroid can match the accuracy of rank-16 LoRA adapters while cutting per-adapter memory up to 16x.

desk verdict SALT tackles a real serving bottleneck with a genuinely new training-time decomposition, but the causal story for why alignment helps is not fully proven; still worth refereeing. read the letter →

arxiv 2608.03579 v1 pith:ZKCGTBUF submitted 2026-08-04 cs.LG cs.AI

Xiang Li , Pengcheng Wang , Huazheng Wang , Saurabh Bagchi This is my paper · ORCID

classification cs.LGcs.AI

keywords LoRAservingmulti-tenantinferencelow-rankadaptationsubspacealignmentcentroid-residualdecompositionadaptercompressionPCIeswappingultra-low-rankadapters

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multi-tenant LLM serving faces a trade-off: high-rank LoRA adapters perform better but eat GPU memory and PCIe bandwidth when swapped in and out, while ultra-low-rank adapters are cheap but lose accuracy. The paper claims that this trade-off is not inevitable. It introduces SALT, a three-phase training scheme in which a provider trains a domain centroid on public data, users fine-tune only rank-1 or rank-2 residuals on top of that frozen centroid, and the serving engine pins the centroid in GPU memory while swapping only the tiny residuals. Across three model families and math and coding benchmarks, SALT reports that these residuals recover most of the accuracy of standard rank-16 LoRA, with per-adapter memory reduced by up to 16x and serving throughput gains of up to 51% under PCIe pressure. If correct, this turns adapter compression from a post-hoc lossy step into a training-time design that separates representational capacity from per-tenant memory cost.

What carries the argument

The load-bearing object is the centroid-residual decomposition $\Delta W_i = \bar W + \delta_i$ together with the epsilon-stabilized matrix cosine regularizer $\mathcal{L}_{\text{align}} = 1 - \frac{\operatorname{Tr}(\Delta W_i^\top \bar W)}{\|\Delta W_i\|_F \|\bar W\|_F + \epsilon}$. This regularizer is what forces independently trained in-domain adapters to converge into the same geometric subspace as the centroid, giving the rank-1 residual a structurally aligned anchor to correct rather than a random basis to compensate. Around this core, SALT adds a two-stage activation-profiling router that selects the right centroid for unlabelled user data and rejects out-of-distribution tasks, plus a discretized inference-time scaling factor $\gamma$ following task arithmetic to dampen multi-task interference. The machinery is what decouples representational capacity from physical per-adapter memory.

What would settle it

Take a domain where one task is multiple-choice and the rest are free-form generation, train the centroid on the free-form tasks, then train a rank-1 residual on the multiple-choice task while sweeping the alignment weight. If raising the alignment weight raises adapter-centroid cosine similarity but leaves multiple-choice accuracy unchanged or lower, the parameter-space-to-function-space transfer premise is broken; a task whose accuracy improves monotonically with alignment would support the claim.

Watch

Extended reading notes

Core claim

The central claim is that a frozen, high-capacity domain centroid trained with a matrix-cosine alignment regularizer creates a shared geometric subspace, so that an ultra-low-rank task residual ($r \le 2$) trained on top of it behaves like a well-trained high-rank LoRA adapter. Concretely, SALT writes each adapter's weight update as $\Delta W_i = \bar W + \delta_i$, where $\bar W$ is the pinned centroid and $\delta_i = B_i' A_i'$ is the rank-1 or rank-2 residual. The joint Phase-1 objective trains in-domain adapters and the centroid together with an alignment penalty that maximizes cosine similarity between each adapter update and the centroid in parameter space. Phase 2 freezes the centroid and trains only the residual on private data, so tenants never touch shared weights. The paper reports that on Mistral-7B-v0.3, Llama-3.2-3B, and Pythia-12B, SALT at $r=1$ largely recovers the accuracy of standard rank-16 LoRA, outperforms VeRA and Compress-then-Serve, and does so with a 0.85 MB swap footprint per adapter.

Load-bearing premise

The result depends on the premise that making adapter weight matrices point in the same direction as a shared centroid, in the raw numeric sense, also makes them work together well when actually answering questions; if that numeric alignment does not transfer to behaviour, the tiny rank-1 correction cannot overcome the centroid's bias on tasks with a different format.

Editorial extensions

If this is right

Per-adapter memory for dynamic swapping drops to 0.85 MB at $r=1$, a 16x reduction versus rank-16 LoRA, so a GPU can hold far more concurrent tenants before KV-cache starvation.
Throughput under PCIe bandwidth pressure improves by up to 51% for Llama-3.2-3B, and by up to 28% under VRAM constraints, without custom kernels.
SALT's accuracy is stable across $r \in \{1,2,4\}$, unlike standard LoRA and Compress-then-Serve, whose accuracy is volatile from rank to rank.
Centroid routing generalizes: held-out math and coding tasks are routed to the correct centroid, while out-of-distribution tasks are rejected and fall back to standard LoRA training.
The stability of the centroid scale factor $\gamma$ in the 0.6–0.9 band means providers can adopt a few shared $\gamma$ bins and pre-fuse scaled centroids into base weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If parameter-space alignment transfers to function space, the same centroid-residual recipe should work for other parameter-efficient methods and other modalities, with the caveat that the centroid must be trained on public, domain-coherent data.
The admitted AQuA failure suggests a function-space or output-distribution alignment loss as a natural variant that might keep the serving benefits while removing the structural penalty for tasks whose format is orthogonal to the centroid's.
The routing stage's threshold-based OOD rejection could be stress-tested on adversarial or blended-domain data; the paper does not say whether residuals remain composable if the test-time distribution shifts after Phase 2.
A direct extension implied by the MoE limitation is a router-conditional centroid set, one centroid per expert group, rather than a single global centroid.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read SALT—it's worth your time. The core idea is genuinely useful: instead of post-hoc compressing LoRA adapters, train a domain centroid on public data with a matrix-cosine regularizer that pulls in-domain tasks into a shared subspace, then fine-tune ultra-low-rank residuals on top. The serving story is coherent—pin the centroid, swap residuals—and the throughput numbers in vLLM are plausible. This is the first paper I've seen that trains for the serving decomposition rather than compressing after the fact, so it clears the bar for novelty.

The paper does a lot right. It evaluates across three model families and multiple datasets, includes ablations on the alignment weight, centroid rank, and scale factor, and is honest about failures: Section 6 admits AQuA degradation, and Appendix A.5 has a concrete SPIDER failure mode where an unaligned centroid plus residual produces a hallucinated JOIN. The routing mechanism with OOD rejection is a practical addition, not just a toy.

The soft spots are real but not fatal. The central mechanistic claim—that Phase 1 alignment is what lets a rank-1 residual correct the centroid—is asserted more than shown. Eq. 2 is applied only in Phase 1; Phase 2 residuals are trained without any alignment objective, so there is no formal reason delta_i should lie in the centroid's subspace. The paper calls this a 'structural shortcut,' but that's a heuristic. The key comparison to a concatenated centroid is confounded: the two training procedures differ in more than the regularizer (multi-task co-training versus single merged-task LoRA), so the ablation does not isolate the alignment term. The AQuA results remind us the mechanism can fail. Additionally, all results appear to be single runs with no error bars; the paired t-test over nine paired configurations is not a test over independent seeds, so the statistical claim is weaker than it reads. The 16x memory reduction is specifically the swap payload, not total VRAM including the shared centroid—which is a fair metric, but the abstract's phrasing could mislead. And no code is released, though the method is simple enough to reimplement.

Verdict: this paper deserves a serious referee. The idea is new, the system evaluation is substantial, and the limitations are acknowledged. A reviewer should push for an ablation that isolates the alignment regularizer (e.g., multi-task co-training with and without Eq. 2) and for error bars. But this is not a desk-reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes SALT, a three-phase fine-tuning and serving framework for multi-tenant LoRA systems. In Phase 1, a provider jointly trains a high-rank domain centroid and per-task adapters on public data with a matrix-cosine alignment regularizer (Eqs. 1–2). In Phase 2, users fine-tune ultra-low-rank residuals (r ≤ 2) on top of the frozen centroid (Eq. 3). In Phase 3, the centroid is pinned in GPU memory and only the residuals are swapped during inference. The authors claim that SALT recovers high-rank LoRA accuracy with r=1 residuals, reduces per-adapter swap memory by up to 16×, and improves vLLM throughput by up to 51% under PCIe bandwidth pressure and 28% under VRAM constraints. Experiments cover three model families and math/coding tasks, with ablations on the alignment penalty λ, centroid scale γ, centroid rank, and an alignment-vs-concatenation comparison.

Significance. If the central claims hold, SALT would be a practical contribution to multi-tenant LoRA serving: decoupling adapter capacity from the dynamic swap payload could reduce PCIe and VRAM pressure without giving up accuracy. The systems side of the paper—pinning a shared centroid and swapping only tiny residuals—is plausible and the throughput gains are expected given the reduced footprint. The alignment-based training methodology is, however, the linchpin of the accuracy claims, and the current evidence for it is weakened by experimental confounds and missing baselines. The paper also includes useful engineering details such as the activation-based routing mechanism and a careful treatment of the inference-time scale factor γ. The manuscript is clearly written and the experiments are extensive; the main open question is whether the accuracy recovery is actually caused by the proposed subspace alignment or by more mundane factors such as multi-task pre-training on the same data.

major comments (4)

[Section 4.1, Table 1] For tasks marked with ‡ in Table 1 (GSM8K, SVAMP, MBPP, SPIDER), the same training examples are used in both Phase 1 centroid training and Phase 2 residual fine-tuning. This contradicts the paper's stated privacy separation between the provider-owned public corpus and user-private data, and it makes the comparison to standard LoRA (which sees each example only once) unfair. The large gains on these seen tasks (e.g., GSM8K: 56.88 vs 53.88 for LoRA r=16) may be attributable to the centroid's prior exposure to the exact training examples rather than to the proposed alignment mechanism. The authors should either split each dataset into disjoint public and private portions for Phase 1 and Phase 2, or restrict the headline accuracy claims to the 'unseen in-domain' tasks and report the centroid-only accuracy to disentangle the effects.
[Section 5.4, Appendix A.5] The ablation in Table 6 and Table 14 does not isolate the alignment regularizer (Eq. 2). The 'concatenated centroid' condition is a single standard LoRA trained on the concatenation of the datasets, whereas the 'subspace-aligned' condition uses the full Phase 1 objective (Eq. 1), which includes both the alignment loss and the multi-adapter co-training architecture with a separate centroid. These two conditions differ in at least two factors, so the observed improvements cannot be attributed specifically to the alignment term. The SPIDER failure case (Listing 1) is therefore consistent with an alternative explanation: that multi-adapter co-training itself, not the cosine alignment, is what makes residual composition safe. A proper control would train the centroid under Eq. 1 with λ=0, keeping all other elements fixed.
[Section 3.1, Eq. (3)] The central mechanistic claim—that the ultra-low-rank residual is 'forced' to use the aligned centroid as a structural shortcut—is not supported by the analysis. The Phase 2 objective in Eq. (3) contains no term that encourages δ_i to lie in the subspace spanned by the centroid, no initialization constraint tying δ_i to the centroid, and no regularization on the relation between δ_i and the centroid. The empirical evidence for the mechanism is limited to a single SPIDER failure case (Appendix A.5) and is undercut by the AQuA results in Table 1, where SALT r=1 (34.31) does not recover LoRA r=16 accuracy (34.80) despite the alignment regularizer. The paper should either provide a theoretical argument for the transfer from parameter-space cosine similarity to functional composability, or measure directly whether the trained δ_i remain aligned with the centroid subspace, and test the conditions under which this alignment holds.
[Section 4.1, Tables 1, 8, 9] The paper never reports the accuracy of the pinned centroid alone (W̄ without any residual) on the evaluation tasks. Since the centroid is jointly trained on multiple tasks, it may already achieve high accuracy on unseen in-domain tasks such as MultiArith (where SALT r=1 yields 86.21 vs. 30.35 for LoRA r=16). If the centroid alone already performs near the reported SALT numbers, the residual adds little, and the 'recovering high-rank accuracy' claim reduces to a property of multi-task centroid training rather than of centroid–residual composition. Reporting centroid-only accuracy for all tasks and all models is necessary to substantiate the contribution.

minor comments (5)

[Table 1 and Figure 1] The 16× memory reduction applies to the dynamic swap payload per adapter (0.85 MB residual vs. 13.61 MB LoRA r=16), not to the total per-adapter memory. For a single tenant the total footprint (centroid + residual) is slightly larger than the LoRA baseline; the reduction is amortized over many tenants. Please clarify this distinction in the text and figure.
[Tables 2, 10, 11] The throughput experiments do not report the downstream task accuracy of the models served. If SALT's accuracy differs from that of the standard LoRA baseline, the throughput comparison is not quality-controlled. Please state the accuracy of the served models or provide a throughput-vs-quality trade-off curve.
[Section 3.2, Table 3] The routing thresholds τ_N=2.0 and τ_R=2.0 are described as 'standard 95% CI threshold' without empirical validation of the Gaussian assumption on activation norms. Please provide a sensitivity analysis over a range of thresholds, or calibrate them on a validation set.
[Abstract, Section 4.2] The abstract states 'up to 18.5% absolute accuracy gains over state-of-the-art compression baselines,' but Table 1 does not display this value; the gains in parentheses are relative to standard LoRA at the same rank, not to compression baselines. Please clarify the exact comparison underlying this number.
[Appendix A.1, Table 7] Table 7 lists AQuA with 5000 training examples, but the text says AQuA was subsampled; please be explicit about the final training-set size and whether the full test set was used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SALT's accuracy and serving claims are empirical, with gamma selected on a disjoint validation split and the central alignment effect tested by ablation rather than defined into existence.

full rationale

The paper's central claim—that ultra-low-rank residuals atop a pinned, subspace-aligned centroid recover high-rank LoRA accuracy—is not equivalent to any fitted input or self-citation. Test accuracies in Tables 1, 8, and 9 are reported on test splits, and the inference-time scale gamma is explicitly selected on a disjoint validation split ('All SALT results use the scaling factor gamma selected via a small hold-out validation set, disjoint from the test sets in Table 1'). The routing thresholds are presented as conventional 95% CI z-scores, and the routing evaluation in Table 3 uses datasets held out from Phase 1. The alignment regularizer (Eq. 2) directly optimizes the cosine metric that Figure 5 reports, but the downstream benefit is not read off that metric; it is tested in the Section 5.4 ablation against a concatenated-data centroid and in the Appendix A.5 SPIDER failure case, which are empirical comparisons. The claim that a rank-constrained residual is 'forced' to use the centroid as a structural shortcut is a heuristic, not a derivation, and the paper's own Section 6 limitation (AQuA degradation) shows the claim is falsifiable rather than tautological. Self-citations (Li et al. 2026; Zhang et al. 2026) appear only in background/motivation and are not load-bearing for the serving-accuracy result. Concerns about the missing mechanistic link between parameter-space cosine similarity and functional composability, or about confounds in the concatenated-centroid ablation, are experimental-design and correctness issues and do not make the derivation circular.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several unproven assumptions, the most important being that matrix-cosine alignment in parameter space transfers to functional composability at inference time. The empirical design also leaves the main results dependent on hand-set thresholds and an undeclared lambda for the main tables.

free parameters (7)

lambda (alignment penalty) = not specified for main tables; figures sweep 0.5 to 2.5
Controls the alignment regularizer in Eq. (1). The main results (Table 1) do not state the exact lambda used, so the result depends on an undisclosed hand-set or early-stopped value.
gamma (centroid scale) = discretized to {0.6, 0.8, 1.0}; selected via hold-out validation
Inference-time scaling of the centroid in W_final = W_base + gamma W_bar + residual. The optimal value varies by task (AQuA needs near 1.3), so the shared bins are a fitted approximation.
tau_N (OOD rejection threshold) = 2.0
Hand-set threshold for OOD detection in routing; assumes a 95% CI on the displacement norm distribution.
tau_R (routing threshold) = 2.0
Hand-set threshold for centroid routing decision in Eq. (4).
centroid rank r_centroid = 16
Chosen from sensitivity analysis in Table 5; no principled selection rule for new domains is given.
Phase 1 dataset composition per domain = 2 tasks per domain (math: GSM8K, SVAMP; code: MBPP, SPIDER)
A modeling choice; the paper motivates tens to hundreds of tasks but only trains centroids on two task datasets per domain.
task weight w_i = proportional to |D_i|
Reweighting in Eq. (1); not ablated, so its effect on the reported results is unknown.

assumptions (6)

domain assumption The base model weights W_base are frozen and shared; the centroid and residuals are additive updates.
Standard LoRA assumption used throughout Sections 3.1 and 3.3.
domain assumption The matrix-cosine similarity in Eq. (2) is a meaningful measure of functional alignment between adapters and the centroid.
The whole SALT framework relies on this premise; no theory connects parameter-space cosine similarity to functional composability.
domain assumption The displacement norm N_k = ||h_k - h_base|| is a sufficient statistic for domain relevance in routing.
Used in Section 3.2 for OOD rejection and centroid selection; only empirically tested on a few datasets.
domain assumption The baseline activation norms are approximately Gaussian, so Z-scores with a threshold of 2.0 give 95% confidence.
Section 3.2 routing thresholds assume normality; the paper does not test this on the actual activation norm distributions.
domain assumption Public Phase 1 data are representative of private Phase 2 tasks within a domain.
Required for the centroid to transfer to user tasks; the paper evaluates only two in-domain held-out tasks per domain.
domain assumption A single scalar gamma per centroid can correct multi-task interference across all tasks in the domain.
Section 3.3 discretizes gamma into shared bins; Figure 11 shows AQuA requires gamma near 1.3, violating this assumption for some tasks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving." pith.science (2026). https://pith.science/paper/ZKCGTBUF

@misc{pith2026260803579,
  author       = {Pith},
  title        = {Pith review of: Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKCGTBUF}},
  note         = {Machine review of arXiv:2608.03579}
}

abstract

Modern multi-tenant Low-Rank Adapters (LoRAs) serving systems concurrently host tens to hundreds of LoRA adapters. Though powerful, this introduces a critical system dilemma between serving efficiency and task performance: higher-rank adapters generally achieve better downstream task performance, but their GPU VRAM footprint and Host-to-Device PCIe swapping overhead severely constrain scalability. Conversely, ultra-low-rank adapters ($r \le 2$) minimize both VRAM footprint and PCIe transfer overhead, but suffer from downstream task performance degradation. To solve this problem, we propose Subspace-Aligned LoRA Training (SALT), a serving efficiency-aware hierarchical fine-tuning framework. Our solution operates in three phases. First, a provider jointly trains high-capacity domain centroids on public data within the domain using a novel alignment regularizer that coheres in-domain task subspaces into a unified basis. Next, users fine-tune ultra-low-rank task residual adapters on private data atop those frozen centroids. Finally, during inference, the provider pins the centroid in GPU VRAM and dynamically swaps in each user's task residual on demand. Across LLMs of varying scales, SALT recovers high-rank accuracy using $r \le 2$ residuals, achieving up to 18.5% absolute accuracy gains over state-of-the-art compression baselines and reducing per-adapter memory by up to 16x. When integrated into vLLM, SALT improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints for Llama-3.2-3B.

Figures

Figures reproduced from arXiv: 2608.03579 by the authors.

**Figure 2.** The three phases of sub-space aligned fine-tuning for efficient LoRA serving. Phase 1: Training sub-space align [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Average task performance recovery and adapter [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Left: Cosine similarity between GSM8K and [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 4.** Figure 4: Mistral: (Left) Swap footprint versus adapter for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Effect of alignment penalty λ. regularization, average downstream accuracy within the domain remains stable, demonstrating that alignment incurs no significant performance penalty for in-domain tasks. 5.2 Centroid Scale γ Sensitivity Scale Factor (γ) Dataset 0.4 0.6 0…

**Figure 9.** Figure 9: Llama-3.2-3B: (Left) Swap footprint versus adapter [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

**Figure 10.** Figure 10: Pythia-12B: (Left) Swap footprint versus adapter [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]

**Figure 7.** Figure 7: The prompt template used for code generation. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: The prompt template used for the SQL generation. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

**Figure 11.** Figure 11: Normalized downstream performance across a [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 18 canonical work pages

[1]

Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo. 2022 , url=

2022
[2]

arXiv preprint arXiv:2311.03285 , year=

S-lora: Serving thousands of concurrent lora adapters , author=. arXiv preprint arXiv:2311.03285 , year=

arXiv
[3]

Compress then Serve: Serving Thousands of Lo

Rickard Br. Compress then Serve: Serving Thousands of Lo. Forty-second International Conference on Machine Learning , year=

work page
[4]

2025 USENIX Annual Technical Conference (USENIX ATC 25) , year =

Suyi Li and Hanfeng Lu and Tianyuan Wu and Minchen Yu and Qizhen Weng and Xusheng Chen and Yizhou Shan and Binhang Yuan and Wei Wang , title =. 2025 USENIX Annual Technical Conference (USENIX ATC 25) , year =

work page 2025
[5]

Proceedings of Machine Learning and Systems , volume=

Punica: Multi-tenant lora serving , author=. Proceedings of Machine Learning and Systems , volume=

work page
[6]

ICML 2024 Workshop on LLMs and Cognition , year=

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning , author=. ICML 2024 Workshop on LLMs and Cognition , year=

work page 2024
[7]

Dawid Jan Kopiczko and Tijmen Blankevoort and Yuki M Asano , booktitle=. Ve. 2024 , url=

2024
[8]

The Thirteenth International Conference on Learning Representations , year=

Federated Residual Low-Rank Adaption of Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=

work page

Show all 45 references

[9]

arXiv preprint arXiv:2602.07263 , year=

tlora: Efficient multi-lora training with elastic shared super-models , author=. arXiv preprint arXiv:2602.07263 , year=
[10]

arXiv preprint arXiv:2602.06043 , year=

Shared LoRA Subspaces for almost Strict Continual Learning , author=. arXiv preprint arXiv:2602.06043 , year=
[11]

Customizing Language Models with Instance-wise Lo

Xiaoyu Kong and Jiancan Wu and An Zhang and Leheng Sheng and Hui Lin and Xiang Wang and Xiangnan He , booktitle=. Customizing Language Models with Instance-wise Lo. 2024 , url=

2024
[12]

arXiv preprint arXiv:2411.00027 , year=

Personalization of large language models: A survey , author=. arXiv preprint arXiv:2411.00027 , year=

arXiv
[13]

18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year =

Bingyang Wu and Ruidong Zhu and Zili Zhang and Peng Sun and Xuanzhe Liu and Xin Jin , title =. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , year =
[14]

arXiv preprint arXiv:2603.05520 , year=

Information-theoretic privacy control for sequential multi-agent llm systems , author=. arXiv preprint arXiv:2603.05520 , year=
[15]

International Conference on Learning Representations , volume=

Longlora: Efficient fine-tuning of long-context large language models , author=. International Conference on Learning Representations , volume=
[16]

Advances in neural information processing systems , volume=

Qlora: Efficient finetuning of quantized llms , author=. Advances in neural information processing systems , volume=
[17]

arXiv preprint arXiv:2110.14168 , year=

Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

arXiv
[18]

Are NLP Models really able to Solve Simple Math Word Problems?

Patel, Arkil and Bhattamishra, Satwik and Goyal, Navin. Are NLP Models really able to Solve Simple Math Word Problems?. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021. doi:10....

2021 doi
[19]

NeurIPS , year=

Measuring Mathematical Problem Solving With the MATH Dataset , author=. NeurIPS , year=
[20]

Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems

Ling, Wang and Yogatama, Dani and Dyer, Chris and Blunsom, Phil. Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2...

2017 doi
[21]

arXiv preprint arXiv:2108.07732 , year=

Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

arXiv
[22]

S pider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to- SQL Task

Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir. S pider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semanti...

2018 doi
[23]

arXiv preprint arXiv:2107.03374 , year=

Evaluating large language models trained on code , author=. arXiv preprint arXiv:2107.03374 , year=

arXiv
[24]

2026 , month = mar, day =

Mohammed, Bashir and others , title =. 2026 , month = mar, day =

2026
[25]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

LoRA in LoRA: Towards parameter-efficient architecture expansion for continual visual instruction tuning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[26]

Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=
[27]

arXiv preprint arXiv:2508.05078 , year=

Align, Don't Divide: Revisiting the LoRA Architecture in Multi-Task Learning , author=. arXiv preprint arXiv:2508.05078 , year=

arXiv
[28]

Advances in Neural Information Processing Systems , volume=

Hydralora: An asymmetric lora architecture for efficient fine-tuning , author=. Advances in Neural Information Processing Systems , volume=
[29]

Forty-third International Conference on Machine Learning , year=

Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning , author=. Forty-third International Conference on Machine Learning , year=
[30]

2026 , url=

Haoran Zhang and Zhiyu Liang and ZUO Decheng and Hongzhi Wang , booktitle=. 2026 , url=

2026
[31]

Fanxu Meng and Zhaohui Wang and Muhan Zhang , booktitle=. Pi. 2024 , url=

2024
[32]

European Conference on Computer Vision , pages=

Ziplora: Any subject in any style by effectively merging loras , author=. European Conference on Computer Vision , pages=. 2024 , organization=

2024
[33]

arXiv preprint arXiv:2307.13269 , year=

Lorahub: Efficient cross-task generalization via dynamic lora composition , author=. arXiv preprint arXiv:2307.13269 , year=

arXiv
[34]

The Eleventh International Conference on Learning Representations , year=

Editing models with task arithmetic , author=. The Eleventh International Conference on Learning Representations , year=
[35]

Advances in neural information processing systems , volume=

React: Out-of-distribution detection with rectified activations , author=. Advances in neural information processing systems , volume=
[36]

arXiv preprint arXiv:2106.09022 , year=

A simple fix to mahalanobis distance for improving near-ood detection , author=. arXiv preprint arXiv:2106.09022 , year=

arXiv
[37]

arXiv preprint arXiv:1909.09436 , year=

Codesearchnet challenge: Evaluating the state of semantic code search , author=. arXiv preprint arXiv:1909.09436 , year=

1909 arXiv
[38]

and Schwenk, Holger and Stoyanov, Veselin , title =

Conneau, Alexis and Rinott, Ruty and Lample, Guillaume and Williams, Adina and Bowman, Samuel R. and Schwenk, Holger and Stoyanov, Veselin , title =. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , year =

2018
[39]

Solving General Arithmetic Word Problems

Roy, Subhro and Roth, Dan. Solving General Arithmetic Word Problems. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015. doi:10.18653/v1/D15-1202

2015 doi
[40]

NeurIPS , year=

Measuring Coding Challenge Competence With APPS , author=. NeurIPS , year=
[41]

arXiv:1803.05457v1 , year =

Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord , title =. arXiv:1803.05457v1 , year =

arXiv
[42]

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=

HellaSwag: Can a Machine Really Finish Your Sentence? , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year=
[43]

EMNLP , year=

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. EMNLP , year=
[44]

2019 , booktitle=

Social IQa: Commonsense Reasoning about Social Interactions , author=. 2019 , booktitle=

2019
[45]

Forty-third International Conference on Machine Learning , year=

Embodied Task Planning via Graph-Informed Action Generation with Large Language Models , author=. Forty-third International Conference on Machine Learning , year=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo. 2022 , url=

2022

[2] [2]

arXiv preprint arXiv:2311.03285 , year=

S-lora: Serving thousands of concurrent lora adapters , author=. arXiv preprint arXiv:2311.03285 , year=

arXiv

[3] [3]

Compress then Serve: Serving Thousands of Lo

Rickard Br. Compress then Serve: Serving Thousands of Lo. Forty-second International Conference on Machine Learning , year=

work page

[4] [4]

2025 USENIX Annual Technical Conference (USENIX ATC 25) , year =

Suyi Li and Hanfeng Lu and Tianyuan Wu and Minchen Yu and Qizhen Weng and Xusheng Chen and Yizhou Shan and Binhang Yuan and Wei Wang , title =. 2025 USENIX Annual Technical Conference (USENIX ATC 25) , year =

work page 2025

[5] [5]

Proceedings of Machine Learning and Systems , volume=

Punica: Multi-tenant lora serving , author=. Proceedings of Machine Learning and Systems , volume=

work page

[6] [6]

ICML 2024 Workshop on LLMs and Cognition , year=

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning , author=. ICML 2024 Workshop on LLMs and Cognition , year=

work page 2024

[7] [7]

Dawid Jan Kopiczko and Tijmen Blankevoort and Yuki M Asano , booktitle=. Ve. 2024 , url=

2024

[8] [8]

The Thirteenth International Conference on Learning Representations , year=

Federated Residual Low-Rank Adaption of Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=

work page