Pith. sign in

REVIEW 5 minor 86 references

Heterogeneous replica deployment and per-step workload-balanced dispatch cut joint LoRA fine-tuning GPU seconds by 45–60%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LobRA reduces GPU seconds for multi-tenant LoRA fine-tuning by 45.03%-60.67% through heterogeneous FT replicas and per-step workload-balanced dispatching.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid systems paper: heterogeneous LoRA fine-tuning replicas with per-step workload-balanced dispatch cut GPU seconds 45-60% under a padding baseline; a packing baseline would shrink but not erase the gain.

arxiv 2509.01193 v1 pith:NYKNB5KL submitted 2025-09-01 cs.DC

LobRA: Multi-tenant Fine-tuning over Heterogeneous Data

Sheng Lin , Fangcheng Fu , Haoyang Li , Hao Ge , Xuanyu Wang , Jiawen Niu , Yaofeng Tu , Bin Cui This is my paper
classification cs.DC
keywords LoRA fine-tuningmulti-tenant fine-tuningheterogeneous model parallelismworkload-balanced data dispatchingvariable-length sequencessequence packing and paddingGPU schedulingparameter-efficient fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the cost of jointly fine-tuning many LoRA adapters on one shared base model is dominated by two data-side mismatches, not by model size alone. First, different fine-tuning tasks have very different sequence lengths, so a single homogeneous replica configuration forces every replica to use more GPUs than short sequences need. Second, sequence-length distributions are skewed toward short texts, so dispatching each sequence to the cheapest capable replica leaves high-parallelism replicas idle. LobRA deploys heterogeneous fine-tuning replicas with different parallel configurations and, for every training step, dispatches data to balance their workloads, with bucket boundaries chosen dynamically to minimize padding. Experiments on 7B, 32B, and 70B models over 16–64 GPUs show GPU seconds for joint fine-tuning fall by 45.03%–60.67% relative to the homogeneous fused baseline.

Core claim

On the paper's own terms, the central discovery is that joint LoRA fine-tuning should be modeled as a min-max optimization over two coupled decisions: how many replicas to deploy with each parallel configuration, and which sequences to send to each replica at each step. Because the joint problem is too slow to solve every step, LobRA decomposes it: a deployment plan that is optimal in expectation is chosen once from the sequence-length distribution, and a fast integer-linear dispatch is solved per step for the actual batch. The empirical payoff is the GPU-second reduction; the authors also report that the decomposition tracks the joint optimum within 15% and the cost model tracks actual runt

What carries the argument

The key object is the heterogeneous FT replica: a copy of the shared base model plus its LoRA adapters, instantiated with a specific tensor- and pipeline-parallel configuration and therefore a specific maximum supportable sequence length and per-GPU throughput. LobRA combines these replicas with a per-step workload-balanced data dispatching scheme, formulated as minimizing the slowest replica's running time subject to every sequence being processed. A dynamic bucketing routine picks sequence-length bucket boundaries per batch by dynamic programming, so that padding is minimized and the dispatch problem has few variables. The two-stage decomposition of deployment planning and step-wise dispat

Load-bearing premise

The reported savings assume that variable-length batches are padded to a common length rather than packed; the authors state the designs also apply to packing but do not evaluate that setting, so a packing-based baseline could reduce the 45–60% advantage.

What would settle it

Run the same 7B and 70B workloads with a homogeneous fused baseline that uses sequence packing (block-diagonal causal masks) instead of padding, keeping batch sizes, models, and hardware fixed. If the packed baseline's GPU seconds fall within 20% of LobRA's, the headline speedup is an artifact of the padding baseline rather than of heterogeneous deployment itself.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuning service providers can serve concurrent LoRA requests on a shared base model with 45–60% fewer GPU seconds than the current fused-batch practice.
  • Larger base models benefit more: the savings grow from 45.03% on 7B to 60.67% on 70B, because heterogeneous replicas avoid the cross-server communication that a homogeneous high-parallel setting pays for all sequences.
  • Per-step dispatch adapts to the random composition of each batch, so the deployment stays fixed while the data routing changes; the planning overhead is hidden behind the previous step's training.
  • The same framework applies when sequence packing replaces padding, since the optimization only depends on the time-cost function being linear in the number of assigned sequences.
  • The solution quality remains close to the intractable joint optimum: the two-stage decomposition is within 15% of the original problem in estimated time, and the cost model within 10% of actual runtime.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The headline speedup is measured against a padding-based homogeneous baseline; a packing-based baseline would likely narrow the gap, so the 45–60% range should be read as conditional on the padding assumption.
  • The workload-balancing principle should transfer to other frozen-backbone fine-tuning methods and to co-serving inference, wherever compute per sample varies.
  • A natural next test is whether the deployment plan can be re-optimized incrementally as tasks arrive and leave, without checkpoint-restart, by warm-starting the deployment solver from the current plan.
  • Dynamic bucketing could be reused in single-task fine-tuning and pre-training, where variable-length micro-batches create pipeline bubbles; the paper's own sequential experiments show those gains are task-dependent and sometimes negative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. LobRA is a multi-tenant fine-tuning framework for jointly training multiple LoRA adapters over a shared base model. The paper identifies two data-heterogeneity problems in this setting: sequence-length variation across tasks and skewness in the length distribution. Its two main ideas are (i) deploying heterogeneous FT replicas with different TP/PP configurations and GPU footprints, and (ii) per-step workload-balanced dispatch of sequences among these replicas, augmented by dynamic bucketing to reduce padding. The problem is decomposed into a one-time deployment-planning stage and a per-step ILP dispatch stage. The system is implemented on Hetu and evaluated on Llama2-7B, Qwen2.5-32B, and Llama2-70B over 16-64 GPUs, reporting a 45.03%-60.67% reduction in GPU seconds relative to a homogeneous Task-Fused baseline, with ablations and a planner-quality check.

Significance. If the reported results hold, LobRA addresses a practically important cost in model-as-a-service fine-tuning. The strengths of the paper are substantial: end-to-end measurements at multiple scales, ablations that isolate the contributions of heterogeneous replicas, workload balancing, and dynamic bucketing, a planner-quality experiment (Figure 10), an artifact link, and an unusually honest discussion of cases where the method does not help (e.g., the LobRA-Sequential degradation in Appendix B.2). The paper also clearly states its scope, including the padding assumption in §2.1 and the heuristic nature of configuration pruning. The packing-baseline concern raised in review is a legitimate external-validity limitation, but it is not an internal inconsistency; the central comparative claim is measured, and the core designs are not logically tied to padding alone.

minor comments (5)
  1. [Abstract; §2.1; §5.2, Figure 7] The headline 45.03%-60.67% reduction should be explicitly qualified as being measured in the padding regime. The paper states in §2.1 that sequence packing can be more efficient and that the proposed designs apply to packing, but no packing-based baseline is measured. Since dynamic bucketing (Eq. 4) is explicitly a padding-minimization device and the ablation attributes a nontrivial fraction of the gain to it, the current unqualified abstract and Section 5.2 wording overstates generality. Please add a qualifier and, ideally, a packing-based Task-Fused comparison or a discussion of the expected effect.
  2. [Appendix D, Eq. (12)] The ILP/MINLP statements in §4 rely on T being linear in the dispatch counts. Eq. (12) contains a max term over t(b_j,s_j) and t(r_j,s_j). The text should state explicitly that t(r_j,s_j) ≤ t(b_j,s_j), so the max term is a constant independent of d_j, making T linear. Without this clarification, the reader cannot verify the claimed ILP formulation of Eq. (3).
  3. [§4.2-4.3, Eqs. (1)-(3)] The objective uses d_{i,j}/p_i, which is generally fractional, while the per-replica dispatch counts must be integers. The paper does not describe how the implementation divides d_{i,j} sequences among the p_i replicas when d_{i,j} is not divisible by p_i. Please clarify the tie-breaking or rounding used in the implementation, since it affects the workload-balance claim.
  4. [Appendix A, Theorem 1] The proof of Theorem 1 depends on Assumption 1 and Observation 1, both of which are empirical assumptions rather than proven statements. The lower-bound filtering should be described more explicitly as a heuristic; Table 5 validates it on the tested workloads, but the current text does not flag the assumption's limited generality enough.
  5. [§5.2, Competitors] The evaluation compares against self-implemented Task-Fused and Task-Sequential baselines. Since related work includes an existing joint-FT system (mLoRA, [67]), a direct comparison with such a system, or at least a comment on why Task-Fused is representative, would strengthen external validity. The NeMo comparison in Appendix C is useful but only covers homogeneous training.

Circularity Check

0 steps flagged

No significant circularity: the headline speedup is a measured result; the fitted cost model is an optimizer, not the evidence.

full rationale

The paper's central claim is an empirical measurement: Figure 7 reports actual GPU seconds for LobRA versus Task-Fused/Task-Sequential, with reductions of 45.03%-60.67% over 100 training steps. The cost model in Appendix D is fitted to offline profiling data and used inside the optimizer to choose replica deployment and per-step dispatch, but the reported speedups are measured, not derived from the model. Dynamic bucketing (Eq. 4) explicitly minimizes padding under the paper's stated padding assumption; this is a genuine optimization objective rather than a predicted conclusion. The paper openly assumes padding and cites external work (Bai et al. [5]) for the padding/packing trade-off, so the choice is not smuggled in via self-citation. Self-citations to Hetu [33,38] and prior group works are implementation/background references and are not the evidence for the efficiency gain; Appendix C independently benchmarks LobRA against NVIDIA NeMo under identical homogeneous configurations, providing external grounding. The paper itself flags limitations: Assumption 1 in Appendix A is acknowledged not to always hold and is used only as a pruning heuristic, and footnote 16 notes that pipeline-bubble reduction is orthogonal. These are scope/robustness caveats, not circularity. The possible shrinkage of the reported gains under a packing-based baseline is an external-validity/benchmark-choice concern, not an internal circularity. No load-bearing step equates an input to an output by definition or by a self-citation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claimed efficiency gain is an empirical benchmark, not a theoretical derivation. The planning components depend on a fitted cost model and several hand-chosen hyperparameters (R, U, threshold); these are disclosed but not derived from first principles. No new physical/mathematical entities are introduced.

free parameters (4)
  • Cost model coefficients (per-configuration time fitting) = Not disclosed in paper (curve fitted to offline profiling data, Appendix D)
    The time cost function T used in Equations (1)-(3) is fitted to profiling measurements for each parallel configuration. Deployment and dispatch plans depend on these fitted values.
  • Number of sequence buckets R = 16 (default, sensitivity tested 4-32)
    Chosen by hand and evaluated; affects granularity of length dispatch and dynamic bucketing.
  • Lower-bound filtering threshold = 15% (default)
    Hand-chosen threshold in configuration pruning; deployment plans whose estimated lower bound exceeds the current minimum by >15% are discarded.
  • Initial interval boundaries U for dynamic bucketing = Equal-length division starting at 256 (e.g., 256, 512, ...)
    Pre-defined by the authors; the DP in Eq. (4) selects R boundaries from these U intervals.
axioms (4)
  • domain assumption Memory consumption of Transformer training is linear in the summed sequence length per chunk
    Used in §2.2 to justify why each parallel configuration supports a maximum sequence length and to determine r_i; cites [8,9,73].
  • ad hoc to paper The time cost function T is linear with respect to the number of dispatched sequences d_{i,j}
    Stated in the §2.2 footnote and Appendix D as a requirement for the ILP/MINLP formulations; the explicit formulas (Eq. 10-12) contain floor/remainder terms, so exact linearity only holds approximately for large counts.
  • ad hoc to paper Assumption 1 (Appendix A): ATB ordering of configurations at one sequence length reflects throughput ordering at all shorter lengths
    Used to prove Theorem 1 and justify the lower-bound filtering heuristic. The authors state it does not always hold in real-world scenarios, but use it only as a relative filter.
  • domain assumption Sequence padding rather than packing is used for variable-length data
    Adopted in §2.1 based on prior work showing comparable efficiency of padding and packing; the quantitative claim is evaluated under padding.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LobRA: Multi-tenant Fine-tuning over Heterogeneous Data." pith.science (2026). https://pith.science/paper/NYKNB5KL

@misc{pith2026250901193,
  author       = {Pith},
  title        = {Pith review of: LobRA: Multi-tenant Fine-tuning over Heterogeneous Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYKNB5KL}},
  note         = {Machine review of arXiv:2509.01193}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the breakthrough of Transformer-based pre-trained models, the demand for fine-tuning (FT) to adapt the base pre-trained models to downstream applications continues to grow, so it is essential for service providers to reduce the cost of processing FT requests. Low-rank adaption (LoRA) is a widely used FT technique that only trains small-scale adapters and keeps the base model unaltered, conveying the possibility of processing multiple FT tasks by jointly training different LoRA adapters with a shared base model. Nevertheless, through in-depth analysis, we reveal the efficiency of joint FT is dampened by two heterogeneity issues in the training data -- the sequence length variation and skewness. To tackle these issues, we develop LobRA, a brand new framework that supports processing multiple FT tasks by jointly training LoRA adapters. Two innovative designs are introduced. Firstly, LobRA deploys the FT replicas (i.e., model replicas for FT) with heterogeneous resource usages and parallel configurations, matching the diverse workloads caused by the sequence length variation. Secondly, for each training step, LobRA takes account of the sequence length skewness and dispatches the training data among the heterogeneous FT replicas to achieve workload balance. We conduct experiments to assess the performance of LobRA, validating that it significantly reduces the GPU seconds required for joint FT by 45.03%-60.67%.

Figures

Figures reproduced from arXiv: 2509.01193 by Bin Cui, Fangcheng Fu, Hao Ge, Haoyang Li, Jiawen Niu, Sheng Lin, Xuanyu Wang, Yaofeng Tu.

Figure 1
Figure 1. Figure 1: An illustration of the fusion of different LoRA adapters. conveys the opportunity to share the same base model across mul￾tiple FT tasks rather than maintaining individual model replica(s) for each one. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of applying sequence padding and packing to variable-length data of one batch. Sequence padding uses the special token ⟨PAD⟩ to ensure sequences within the same chunk are the same length. Sequence packing concatenates sequences together and uses the block-diagonal casual masks to avoid cross-contamination. [76] considered fusing multiple LoRA adapters for joint FT. Nev￾ertheless, they carry ou… view at source ↗
Figure 4
Figure 4. Figure 4: An example of 4 FT tasks with four different approaches, where (a) denotes fine-tuning the 4 tasks sequentially, whilst (b)-(d) present three different designs discussed in §3. We focus on the total GPU seconds required to run one training step for each task. (e) illustrates the inputs and decision variables of Equation (1) based on (d). example of four FT tasks in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of LobRA. To start the joint FT task, given the base model and the sequence length distribution of the FT datasets, LobRA determines the deployment plan of FT replicas that minimizes the running time in expectation. During the FT process, for each training step, LobRA analyzes how the corresponding batch of data should be dispatched among the FT replicas in order to achieve workload balance. First… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of dynamic bucketing. a notorious combinatorial optimization problem. Although libraries like SCIP [3] support solving MINLP problems, it is very time￾consuming, especially when there are many decision variables (i.e., 𝑑𝑖,𝑗 and 𝑝𝑖 ). Specifically, the number of decision variables of Equation (2) is 𝑆 + Í𝑆 𝑖=1 𝑟𝑖 ≤ 𝑆 + 𝑆 × 𝑅. Thus, the solving cost of Equation (2) is highly related to 𝑆, as 𝑅 i… view at source ↗
Figure 8
Figure 8. Figure 8: Ablation Studies (7B model, 16 A100-40GB GPUs). task. The detailed descriptions of the FT datasets and the batch size settings are provided in Appendix B.1 [4]. By default, we consider 6 tasks for the 7B model and 12 tasks for the other two models. We use the Adam optimizer [27, 36] for all experiments. Protocols. Since our goal is to improve the efficiency of joint FT, we focus on the GPU seconds required… view at source ↗
Figure 9
Figure 9. Figure 9: Case studies (7B model, 16 A100-40GB GPUs). Each bar represents one kind of FT replica(s). Top: The per-step time of each kind of FT replica(s). Bottom: The organization of dispatched data in terms of their sequence lengths for each kind of FT replica(s). 20 40 60 80 100 Step 0 20 40 60 Per-step Time (s) Overlappable Origin Solving Two-stage Planning 20 40 60 80 100 Step 1.0 1.1 Relative Time Tactual/Torig… view at source ↗
Figure 10
Figure 10. Figure 10: Left: Time cost of solving the original problem (Equa￾tion (1)) vs. the two-stage planning (dynamic bucketing + solving Equation (3)). The horizontal dashed line indicates the average per￾step time. Right: Comparison of estimated running time for solv￾ing the original problem (𝑇𝑜𝑟 𝑖𝑔𝑖𝑛), the two-stage decomposition (𝑇𝑑𝑒𝑐𝑜𝑚𝑝 ), and the actual running time (𝑇𝑎𝑐𝑡𝑢𝑎𝑙 ). 5.3 Effectiveness of the Proposed Techn… view at source ↗
Figure 12
Figure 12. Figure 12: Impact of number of buckets (i.e., 𝑅) in dy￾namic bucketing to the per-step time and padding ratios (7B model, 16 A100-40GB GPUs). The per-step time is scaled by that with 4 buckets. Effectiveness of Configuration Planning. To assess the scalability of configuration planning (i.e., solving Equation (2)) as well as the effectiveness of the two configuration pruning techniques, we conduct experiments to mea… view at source ↗
Figure 13
Figure 13. Figure 13: Illustration of the 1F1B pipeline parallel execution with variable-length inputs. Cells in light background color and dark background color represent forward and backward processes, respectively. The blue and green indicate two kinds of micro-batches that are different in length. The critical path of the execution is highlighted in red. To reduce fragmented bubble time caused by variable-length inputs and… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 64 canonical work pages · 2 internal anchors

  1. [1]

    Optimization with PuLP

    2009. Optimization with PuLP. https://coin-or.github.io/pulp/

    2009

  2. [2]

    The Ymir Proejct: Dataset and Workload

    2023. The Ymir Proejct: Dataset and Workload. https://sites.google.com/view/ ymir-project#h.dw77b5uw44tb

    2023

  3. [3]

    SCIP: Solving Constraint Integer Programs

    2024. SCIP: Solving Constraint Integer Programs. https://www.scipopt.org/

    2024

  4. [4]

    Full Version (with Appendix) of LobRA

    2025. Full Version (with Appendix) of LobRA. https://github.com/ccchengff/ LobRA/blob/main/LobRA_Full_Version_with_Appendix.pdf

    2025

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. LongAlign: A Recipe for Long Context Alignment of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024 (EMNLP Findings, 2024) . 1376–1395

    2024

  6. [6]

    Weibo Cai, Shulin Yang, Gang Sun, Qiming Zhang, and Hongfang Yu. 2023. Adaptive load balancing for parameter servers in distributed machine learning over heterogeneous networks. ZTE Communications 21, 1 (2023), 72

    2023

  7. [7]

    Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krish- namurthy. 2024. Punica: Multi-Tenant LoRA Serving. In Proceedings of Machine Learning and Systems 2024 (MLSys 2024)

    work page 2024

  8. [8]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations 2024 (ICLR 2024)

    work page 2024

  9. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Annual Conference on Neural Information Processing Systems 2022 (NeurIPS 2022)

    work page 2022

  10. [10]

    DataBricks. 2025. Documents for Foundation Model Fine-tuning. https://docs. databricks.com/aws/en/large-language-models/foundation-model-training

    work page 2025

  11. [11]

    Harm de Vries. 2023. In the long (context) run. https://www.harmdevries.com/ post/context-length/

    work page 2023

  12. [12]

    Letian Deng and Yanru Zhao. 2023. Deep learning-based semantic feature extraction: A literature review and future directions. ZTE communications 21, 2 (2023), 11

    work page 2023

  13. [13]

    Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. Fewer Truncations Improve Language Modeling. In International Conference on Machine Learning 2024 (ICML 2024)

    work page 2024

  14. [14]

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. CoRR abs/2002.06305 (2020)

    Pith/arXiv arXiv 2020

  15. [15]

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayi- heng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). 177–198

    work page 2024

  16. [16]

    Fei Du, Xin-Jian Ma, Jing-Ru Yang, Yi Liu, Chao-Ran Luo, Xue-Bin Wang, Hai-Ou Jiang, and Xiang Jing. 2024. A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot. J. Comput. Sci. Technol. (2024)

    work page 2024

  17. [17]

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018) . 489–500

    work page 2018

  18. [18]

    Shaoduo Gan, Xiangru Lian, Rui Wang, Jianbin Chang, Chengjun Liu, Hongmei Shi, Shengzhuo Zhang, Xianghong Li, Tengxu Sun, Jiawei Jiang, Binhang Yuan, Sen Yang, Ji Liu, and Ce Zhang. 2021. BAGUA: Scaling up Distributed Learning with System Relaxations. Proc. VLDB Endow. 15, 4 (2021), 804–813

    work page 2021

  19. [19]

    Wensheng Gan, Shicheng Wan, and Philip S. Yu. 2023. Model-as-a-Service (MaaS): A Survey. CoRR abs/2311.05804 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  20. [20]

    Lei Guan, Dong-Sheng Li, Jiye Liang, Wen-Jian Wang, Ke-shi Ge, and Xicheng Lu. 2024. Advances of Pipeline Model Parallelism for Deep Learning Training: An Overview. J. Comput. Sci. Technol. (2024)

    work page 2024

  21. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations 2022 (ICLR 2022)

    work page 2022

  22. [22]

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2024. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. In First Conference on Language Modeling (COLM 2024)

    work page 2024

  23. [23]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Annual Conference on Neural Information Processing Systems 2019 (NeurIPS 2019). 103–112

    work page 2019

  24. [24]

    Insu Jang, Zhenning Yang, Zhen Zhang, Xin Jin, and Mosharaf Chowdhury

    work page

  25. [25]

    Zhihao Jia, Matei Zaharia, and Alex Aiken. 2019. Beyond Data and Model Parallelism for Deep Neural Networks. In Proceedings of Machine Learning and Systems 2019 (MLSys 2019)

    work page 2019

  26. [26]

    Youhe Jiang, Fangcheng Fu, Xupeng Miao, Xiaonan Nie, and Bin Cui. 2023. OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI 2023). 2142–2150

    work page 2023

  27. [27]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. In 3rd International Conference on Learning Representations 2015 (ICLR 2015)

    work page 2015

  28. [28]

    Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael An- dersch, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Reducing Activation Recomputation in Large Transformer Models. InProceedings of Machine Learning and Systems 2023 (MLSys 2023)

    work page 2023

  29. [29]

    Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. 2021. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. CoRR abs/2107.02027 (2021)

    Pith/arXiv arXiv 2021

  30. [30]

    Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. 2024. Enhancing Training Efficiency Using Packing with Flash Attention. CoRR abs/2407.09105 (2024)

    Pith/arXiv arXiv 2024

  31. [31]

    Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2024. GPTuner: A Manual- Reading Database Tuning System via GPT-Guided Bayesian Optimization. Proc. VLDB Endow. 17, 8 (2024), 1939–1952

    work page 2024

  32. [32]

    Daiyi Li, Yaofeng Tu, Xiangsheng Zhou, Yangming Zhang, and Zongmin Ma

    work page

  33. [33]

    Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Xupeng Miao, and Bin Cui. 2025. Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations. CoRR abs/2504.20490 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala

  35. [35]

    Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP 2019). 3728–3738

    work page 2019

  36. [36]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations 2019 (ICLR 2019)

    work page 2019

  37. [37]

    Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. 2024. A Survey on LoRA of Large Language Models. CoRR abs/2407.11046 (2024)

    Pith/arXiv arXiv 2024

  38. [38]

    Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. 2023. Hetu: a highly efficient automatic parallel distributed deep learning system. Sci. China Inf. Sci. 66 (2023)

    work page 2023

  39. [39]

    Xupeng Miao, Gabriele Oliaro, Xinhao Cheng, Mengdi Wu, Colin Unger, and Zhihao Jia. 2024. FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning. CoRR abs/2402.18789 (2024)

    arXiv 2024

  40. [40]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism. Proc. VLDB Endow. 16, 3 (2022), 470–479

    work page 2022

  41. [41]

    Devanur, Gregory R

    Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. 2019. PipeDream: generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles (SOSP 2019) . 1–15

    work page 2019

  42. [42]

    Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Za- haria. 2021. Memory-Efficient Pipeline-Parallel DNN Training. In International Conference on Machine Learning 2021 (ICML 2021) , Vol. 139. 7937–7947

    work page 2021

  43. [43]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient large-scale language model training on GPU clusters using megatron- LM. In International Conference for High Performance Comput...

    work page 2021

  44. [44]

    Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, and Bin Cui. 2023. Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent. Proc. VLDB Endow. 16, 12 (2023), 3781–3794

    work page 2023

  45. [45]

    NVIDIA. 2024. NeMo. https://github.com/NVIDIA/NeMo

    work page 2024

  46. [46]

    NVIDIA. 2024. NVIDIA Collective Communications Library (NCCL). https: //developer.nvidia.com/nccl

    work page 2024

  47. [47]

    OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. https: //openai.com/blog/chatgpt

    work page 2022

  48. [48]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023)

    Pith/arXiv arXiv 2023

  49. [49]

    OpenAI. 2024. OpenAI Platform: Fine-tuning. https://platform.openai.com/ docs/guides/fine-tuning/

    work page 2024

  50. [50]

    Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Iqbal khan, and Arsalan Shahid. 2024. The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities. CoRR abs/2408.13296 (2024)

    Pith/arXiv arXiv 2024

  51. [51]

    Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot.CoRR abs/2004.13637 (2020)

    Pith/arXiv arXiv 2020

  52. [52]

    Gon- zalez, and Ion Stoica

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gon- zalez, and Ion Stoica. 2024. SLoRA: Scalable Serving of Thousands of LoRA Adapters. In Proceedings of Machine Learning and Systems 2024 (MLSys 2024)

    work page 2024

  53. [53]

    Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. 2024. In-Context Pretraining: Language Modeling Beyond Document Boundaries. In International Conference on Learning Representations 2024 (ICLR 2024)

    work page 2024

  54. [54]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs/1909.08053 (2019)

    Pith/arXiv arXiv 2019

  55. [55]

    Arjun Singh, Nikhil Pandey, Anup Shirgaonkar, Pavan Manoj, and Vijay Aski

    work page

  56. [56]

    Snowflake. 2025. Fine-tuning (Snowflake Cortex). https://docs.snowflake.com/ en/user-guide/snowflake-cortex/cortex-finetuning

    work page 2025

  57. [57]

    Zijian Song, Wenhan Zhang, Lifang Deng, Jiandong Zhang, Kaigui Bian, and Bin Cui. 2024. MultiLoRA: Multi-Directional Low Rank Adaptation for Multi-Domain Recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024) . 2148–2157

    work page 2024

  58. [58]

    Jakub Tarnawski, Deepak Narayanan, and Amar Phanishayee. 2021. Piper: Multidimensional Planner for DNN Parallelization. In Annual Conference on Neural Information Processing Systems 2021 (NeurIPS 2021) . 24829–24840

    work page 2021

  59. [59]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony H...

    Pith/arXiv arXiv 2023

  60. [60]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Annual Conference on Neural Information Processing Systems 2017 (NeurIPS 2017)

    work page 2017

  61. [61]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khy- athi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. In Annual Conference on Neural Information Processing Systems 2023 (NeurIPS 2023)

    work page 2023

  62. [62]

    Yujie Wang, Youhe Jiang, Xupeng Miao, Fangcheng Fu, Shenhan Zhu, Xiaonan Nie, Yaofeng Tu, and Bin Cui. 2024. Improving Automatic Parallel Training via Balanced Memory Workload Optimization. IEEE Trans. Knowl. Data Eng. 36, 8 (2024), 3906–3920

    work page 2024

  63. [63]

    Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. 2023. MultiLoRA: Democratizing LoRA for Better Multi-Task Learning. CoRR abs/2311.11501 (2023)

    Pith/arXiv arXiv 2023

  64. [64]

    Xun Wu, Shaohan Huang, and Furu Wei. 2024. Mixture of LoRA Experts. In The Twelfth International Conference on Learning Representations 2024 (ICLR 2024)

    work page 2024

  65. [65]

    Yifei Xia, Fangcheng Fu, Wentao Zhang Jiawei Jiang, and Bin Cui. 2024. Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters. InAnnual Conference on Neural Information Processing Systems 2024 (NeurIPS 2024)

    work page 2024

  66. [66]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, M...

    Pith/arXiv arXiv 2024

  67. [67]

    Zhengmao Ye, Dengchun Li, Zetao Hu, Tingfeng Lan, Jian Sha, Sicong Zhang, Lei Duan, Jie Zuo, Hui Lu, Yuanchun Zhou, and Mingjie Tang. 2023. mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs. CoRR abs/2312.02515 (2023)

    Pith/arXiv arXiv 2023

  68. [68]

    Huangzhao Zhang, Kechi Zhang, Zhuo Li, Jia Li, Jia Li, Yongmin Li, Yunfei Zhao, Yuqi Zhu, Fang Liu, Ge Li, and Zhi Jin. 2024. Deep learning for code generation: a survey. Sci. China Inf. Sci. 67 (2024)

    work page 2024

  69. [69]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In International conference on machine learning (ICML 2020) , Vol. 119. 11328–11339

    work page 2020

  70. [70]

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL 2020 Demo). 270–278

    work page 2020

  71. [71]

    Zhongping Zhang, Yin Jia, Yuehan Hou, and Xinlu Yu. 2024. Explicit Behavior Interaction with Heterogeneous Graph for Multi-behavior Recommendation. Data Sci. Eng. 9 (2024)

    work page 2024

  72. [72]

    Zhen Zhang, Shuai Zheng, Yida Wang, Justin Chiu, George Karypis, Trishul Chilimbi, Mu Li, and Xin Jin. 2022. MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud. Proc. VLDB Endow. 16, 1 (2022), 37–50

    work page 2022

  73. [73]

    Pinxue Zhao, Hailin Zhang, Fangcheng Fu, Xiaonan Nie, Qibin Liu, Fang Yang, Yuanbo Peng, Dian Jiao, Shuaipeng Li, Jinbao Xue, Yangyu Tao, and Bin Cui

    work page

  74. [74]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proc. VLDB Endow. 16, 12 (2023), 3848–3860

    work page 2023

  75. [75]

    Xing, Joseph E

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yan- ping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Par- allelism for Distributed Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2022) . 559–578

    work page 2022

  76. [76]

    Ying Zheng, Lei Jiao, Han Yang, Lulu Chen, Ying Liu, Yuxiao Wang, Yuedong Xu, Xin Wang, and Zongpeng Li. 2024. Online Scheduling and Pricing for Multi-LoRA Fine-Tuning Tasks. In Proceedings of the 53rd International Conference on Parallel Processing, (ICPP 2024). 357–366

    work page 2024

  77. [77]

    Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. 2024. Multi-LoRA Composition for Image Generation. CoRR abs/2402.16843 (2024)

    Pith/arXiv arXiv 2024

  78. [78]

    Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2024. D-Bot: Database Diagnosis System using Large Language Models. Proc. VLDB Endow. 17, 10 (2024), 2514– 2527

    work page 2024

  79. [79]

    Xuanhe Zhou, Zhaoyan Sun, and Guoliang Li. 2024. DB-GPT: Large Language Model Meets Database. Data Sci. Eng. 9 (2024)

    work page 2024

  80. [80]

    Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. 2022. PetS: A Unified Framework for Parameter-Efficient Transformers Serving. In2022 USENIX Annual Technical Conference (ATC 2022). 489–504

    work page 2022

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.