Pith. sign in

REVIEW 3 major objections 4 minor 42 references

Topology-aware Preemptive Scheduling for Co-located LLM Workloads

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that preempting co-located LLM jobs with the victim's hardware topology in view — freeing GPUs near the same NUMA node or socket the high-priority service needs — raises topology-affinity hit rate from 44.5% to 100% in a…

desk verdict A sensible, well-built scheduling system that overclaims its 55% performance number, since the evaluation only measures the same topology score the scheduler already optimizes. read the letter →

arxiv 2411.11560 v1 pith:JOXOJ3SD submitted 2024-11-18 cs.DC cs.AI

Ping Zhang , Lei Su , Jinjie Yang , Xin Chen This is my paper · ORCID
classification cs.DCcs.AI
keywords LLMservingco-locationtopology-awareschedulingpreemptionNUMAaffinityGPUclusterKubernetesscheduler
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clusters that co-locate LLM workloads fill their GPUs with a mix of latency-sensitive services and preemptible batch jobs, but when demand spikes, the scheduler evicts low-priority jobs without checking whether the hardware they release actually suits the high-priority service that needs to scale up. The paper argues this is the root cause of failed or degraded placements of online services, and that preemption should be topology-aware: the resources freed by victims should match the preemptor's NUMA and socket affinity. To that end it introduces FlexTopo, a graph-based representation of each server's real-time hardware topology, and a victim-selection policy that scores candidate preemptions by topology fitness while still preferring minimal, low-priority victim sets. The central empirical claim is a jump in topology-affinity hit rate from 44.5% to 100% across 5000 simulated preemptions, with the authors translating this into a 55% reduction in scheduling failures for guaranteed-QoS services and a 55% improvement in scheduled performance for best-effort ones. If true, co-location becomes much more attractive: clusters can stay fully packed and still reclaim topology-suitable resources on demand.

What carries the argument

The load-bearing objects are three. (1) FlexTopo, a graph-based, server-generic representation of hardware topology and live allocation: sockets, NUMA nodes, CPU core groups, and GPU devices are nodes; host, contain, localized, and nearby edges encode hierarchy and proximity; status and used-by attributes link components to running instances. (2) A scoring function for preemption candidates, $S(C) = \alpha \times \frac{1}{\text{sum of priority}(C)} + (1-\alpha)\, T(C_{\mathrm{flextopo}})$, where $T$ is a piecewise affinity score: high for same-NUMA victim resources, medium for same-socket but different-NUMA, low for cross-socket; $\alpha \in [0,1]$ trades victim priority against topology fitness. (3) Incremental Minimal Preemption (IMP), a greedy search that tests victim subsets from size 1 upward and stops at the first feasible size, keeping the average-case cost close to polynomial while preserving minimal victim sets. An in-cluster FlexTopo agent keeps the representation fresh with periodic hardware checks and event-driven allocation updates, and a two-stage pipeline applies guaranteed filtering and best-effort sorting according to the preemptor's QoS.

What would settle it

A controlled latency experiment on one of the paper's own server types (for example, an 8-GPU 4090 or A100 node): place the same LLM inference workload in three conditions — same-NUMA, same-socket different-NUMA, and cross-socket — and measure end-to-end throughput and tail latency. If the cross-socket placement performs within noise of the same-NUMA placement for the workloads the scheduler targets, the scoring function's ordering is miscalibrated and the claimed 55% scheduled-performance improvement would not appear; conversely, a direct before/after comparison of tail latency for a scaled-up service, with and without FlexTopo preemption, would settle whether the hit-rate gain translates into the claimed performance gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that preemption quality is a topology problem, not just a priority problem. A scheduler that knows, at victim-selection time, which sockets, NUMA nodes, CPU core groups, and GPU devices each candidate victim occupies can free resources that line up with the preemptor's requirements, and it can do so without an exponential search. The paper demonstrates this with FlexTopo — a unified graph whose nodes are sockets, NUMA nodes, CPU core groups, and GPUs, with edges for hosting, containment, and proximity — and a piecewise scoring of candidates that rates same-NUMA allocation highest, same-socket allocation medium, and cross-socket allocation lowest, balanced against victim priority by a weight $\alpha$. The reported outcome is that in 100 simulation cycles with 50 preemptions each on a 100-node simulated 4090 cluster, the standard scheduler met topology requirements in 2225 of 5000 preemptions (44.5%), while FlexTopo-based preemption met them in all 5000 (100%); the paper reads the gap as a 55% reduction in scheduling failures for guaranteed QoS and a 55% improvement in scheduled performance for best-effort QoS. A greedy Incremental Minimal Preemption procedure keeps the victim search near-polynomial in the average case, cutting candidate-sourcing latency by 7.3% to 76.5% relative to exhaustive evaluation.

Load-bearing premise

The load-bearing premise is that a hand-assigned scoring rule — same NUMA node is best, same socket is second, cross-socket is worst — captures how much LLM inference performance actually depends on hardware topology, so that optimizing this score genuinely improves real scheduled performance.

Editorial extensions

If this is right

  • Preemption can be made topology-reliable: in the paper's simulation, all 5000 preemptions met the preemptor's affinity requirements, versus 2225 (44.5%) for the standard scheduler.
  • The Kubernetes TopologyAffinityError failure mode is reduced by 55% for guaranteed-QoS preemptors, and scheduled performance improves by 55% for best-effort preemptors.
  • Saturation allocation becomes practical: clusters can run fully packed with mixed LLM workloads and still reclaim resources that satisfy latency-sensitive services when auto-scaling fires.
  • The overhead of topology-aware victim selection is manageable: IMP cuts candidate-sourcing latency by 7.3% to 76.5% versus exhaustive FlexTopo evaluation, with the smallest overhead for small GPU requests.
  • Because FlexTopo abstracts away server-specific layouts, the same policy transfers across hardware generations (for example, 4090 and A100 servers) without per-server adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct calibration test is missing from the paper: the score assumes same-NUMA beats same-socket beats cross-socket, but the paper itself reports communication penalties of 1.2× within a socket and 3.2× across sockets on 4090 servers (2× on A100 servers); plugging those measured ratios into the topology score would be a natural, testable tightening.
  • The 55% scheduled-performance gain is inferred from the affinity-hit-rate gap rather than measured end-to-end; a direct before/after measurement of tail inference latency for the scaled-up service would confirm whether the hit-rate improvement actually buys the latency win the paper claims.
  • The FlexTopo representation and scoring could serve normal placement and defragmentation, not only preemption; the paper leaves those applications implicit.
  • The benefit is bounded by how topology-sensitive the workload actually is: for inference engines or quantization schemes that tolerate memory-bandwidth imbalance, the same scoring may over-rank or under-rank candidates, so the win would vary by engine.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes FlexTopo, a graph-based representation of per-node hardware topology and allocation state, and integrates it into the Gödel scheduler as a preemption plugin. The scheduler uses a weighted scoring function combining victim priority and topology affinity, plus an incremental minimal preemption algorithm to limit search cost. The evaluation, conducted in a 100-node KWOK simulation and a near-production cluster, reports an increase in topology-affinity hit rate from 44.5% to 100% and claims a 55% improvement in scheduled performance for LLM workloads.

Significance. If the claimed performance improvement were supported by end-to-end measurements, the work would be a useful practical addition to topology-aware scheduling: victim selection during preemption is a genuine gap in existing Kubernetes/Gödel schedulers, and the open-source implementation is a concrete artifact. The proposed unified graph representation and the incremental search heuristic are reasonable engineering contributions. However, the current evidence does not substantiate the headline performance claim, because the evaluation metric is essentially the same topology score that the scheduler optimizes.

major comments (3)
  1. [Section 5, Table 4] The only quantitative success metric, topology-affinity hit rate, is defined by same-NUMA/same-socket placement, which is exactly the objective T(C_flextopo) in Eq. (1). A scheduler that maximizes this score will naturally achieve a high hit rate, so the 100%-vs-44.5% result demonstrates self-consistency rather than an improvement in LLM scheduled performance. The paper contains no measurement of end-to-end latency, throughput, TTFT, or goodput for any LLM workload, and the phrase "55% improvement in Scheduled Performance" in Section 5 is therefore unsupported. Re-run the comparison with actual serving metrics (e.g., p50/p99 token-generation latency, throughput, or the number of placement failures observed by the kubelet) or restrict the claims to hit-rate improvement.
  2. [Section 3.4, Eq. (1)] The scoring function uses α, whose value is never reported, and a piecewise T(C_flextopo) whose high/medium/low numeric values are never specified. This underdetermines the reported comparison: depending on α and the numeric mapping, the tie-breaking between priority and topology can change the selected victims, so the 100% hit rate cannot be reproduced by a reader. Please report the exact values, justify them, and include a sensitivity analysis over α and the topology-score values.
  3. [Section 5, Simulation Configuration and Table 3] The simulation uses one fixed workload mix, one GPU server type (8×RTX 4090), and one saturation state, and no replication or confidence interval is reported for the 5000-preemption hit-rate numbers. The "100%" result especially needs to be tested across load levels, workload mixes, and hardware topologies (e.g., A100-style 2-socket/2-NUMA servers) before claiming that cross-socket allocations are eliminated. At minimum, report run-to-run variation or worst-case outcomes over multiple seeds and configurations.
minor comments (4)
  1. [Section 2.2 and Section 3.4] There are typos in the prose: "Readers are refereed" should be "Readers are referred," and "as detailed in the bellow" should be "as detailed below."
  2. [Section 5, Figure 8] The authors acknowledge that the before/after production snapshots may not be strictly comparable; please label this figure as illustrative, since it is not a controlled comparison and should not be cited as quantitative evidence.
  3. [Section 5, text after Table 4] The sentence claiming that the scheduling failure rate is "reduced by 55%" conflates a 55.5-percentage-point absolute increase in hit rate with a relative reduction or a performance improvement; please state the arithmetic precisely.
  4. [Abstract and Section 5] The title and abstract emphasize LLM workloads, but no LLM inference engine is exercised in the evaluation; clarify that the results currently concern topology-based victim selection in a simulated Kubernetes cluster, with LLM relevance argued from hardware topology rather than measured inference performance.

Circularity Check

1 steps flagged · score 6.0 of 10

The 55% Scheduled Performance claim is the optimizer's own topology score counted as a hit rate, making the headline result self-referential.

  1. self definitional [Section 3.4, Eq. (1); Section 5, 'Topology Affinity Hit Rate']
    "T (Cflextopo) = high, if vN ∈ the same aligned NUMA of N; medium, if vN ∈ the same aligned socket but different NUMA of N; low, if vN is located across sockets ... While for workloads with best-effort QoS, FlexTopo-based preemption provides a 55% improvement in Scheduled Performance."

    The scheduler maximizes S(C) = α × 1/sum of priority(C) + (1−α) × T(C_flextopo), where T assigns high/medium/low to same-NUMA, same-socket, and cross-socket victim placements. Section 5's 'topology affinity hit rate' counts preemptions that 'satisfied topology requirements' — the same same-NUMA/same-socket/cross-socket categories that define T. A scheduler that maximizes T therefore achieves a 100% hit rate on T by construction; the claimed '55% improvement in Scheduled Performance' is the 100%-versus-45% hit-rate gap relabeled as performance, not an independent LLM performance measurement. No end-to-end latency, throughput, or goodput result is reported.

full rationale

The underlying system pieces — FlexTopo as a graph representation, the event-driven agent, and the IMP greedy search with its overhead measurements — are self-contained and not circular. The circularity is confined to the headline performance claim: the evaluation metric (topology affinity hit rate) is identical to the scoring function in Eq. (1) that the scheduler optimizes. Because the central '55% Scheduled Performance' claim reduces to self-consistency with Eq. (1), the score is 6. The overhead and production snapshot analyses are independent, but they do not validate the headline performance number.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central engineering claim rests on the assumption that topology affinity matters for LLM performance (not measured), on a saturation-allocation deployment model, and on an uncalibrated scoring function. The only explicit free parameter shown in the paper is alpha in Eq. 1, but no value is given. The CoreGroup size is configurable and also unspecified. No new physical entities are introduced.

free parameters (3)
  • Alpha (alpha) in scoring function = not reported
    Controls trade-off between priority and topology in Eq. 1; no value given, but results depend on it.
  • CoreGroup size = not reported (configurable)
    FlexTopo groups CPU cores; the grouping size changes graph resolution and scheduling decisions, but is not specified for experiments.
  • Topology score thresholds = high/medium/low not quantified
    Piecewise scores for same-NUMA, same-socket, cross-socket allocation are hand-assigned and never calibrated against measured performance.
assumptions (4)
  • domain assumption Topology affinity significantly determines LLM inference performance
    Section 2.2 asserts NUMA/socket affinity improves scheduled performance but cites no LLM-specific measurements.
  • domain assumption Saturation allocation with preemption is the right operational model
    Section 3.1 assumes cluster should be fully allocated and preemption is the primary mechanism for auto-scaling, which may not hold in all production clusters.
  • domain assumption Draining all victims hypothetically and then scoring is sufficient to find a feasible placement
    Filtering in Section 3.4 assumes max draining captures feasibility; actual victim selection may still produce unsatisfying configurations. The scoring function is supposed to handle this, but correctness is not proven.
  • domain assumption Graph edges omit PCIe and NVSwitch details without affecting decisions
    Section 3.2 explicitly omits PCIe/NVSwitch; this may be fine for NUMA-level decisions but is unverified for multi-GPU communication.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Topology-aware Preemptive Scheduling for Co-located LLM Workloads." pith.science (2026). https://pith.science/paper/JOXOJ3SD

@misc{pith2026241111560,
  author       = {Pith},
  title        = {Pith review of: Topology-aware Preemptive Scheduling for Co-located LLM Workloads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOXOJ3SD}},
  note         = {Machine review of arXiv:2411.11560}
}
abstract

Hosting diverse large language model workloads in a unified resource pool through co-location is cost-effective. For example, long-running chat services generally follow diurnal traffic patterns, which inspire co-location of batch jobs to fulfill resource valleys between successive peaks, and thus to saturate resource allocation in cluster-wide scope. These heterogeneous workloads often have different business priorities, and therefore preemption can be leveraged for resource elasticity. However, workloads often have distinct topology preferences as well. The resources released by lower-priority instances may fail to meet the requirements of high-priority online services which are usually latency-sensitive. The root cause behind such mis-match is a lack of topology awareness of resource scheduler, especially during preemption. To bridge this gap, we develop a fine-grained topology-aware method for preemptive scheduling of hybrid workloads. The method ensures that the resources freed by preempted tasks adhere to the topological affinity needs of high-priority preemptors in a guaranteed or best-effort manner. This dynamic alignment significantly increases the efficiency of preemption and improves overall scheduled performance for LLM workloads by $55\%$.

Figures

Figures reproduced from arXiv: 2411.11560 by the authors.

Figure 1
Figure 1. Distributed LLM serving with workload co-location from the cluster view. Hereby, we take an online LLM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two hardware topology examples of NVIDIA 4090 server and A100 server. The configurations are 2Sockets [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A snapshot of resource allocation in a cluster of 4090 Server for three co-located workloads [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Overview architecture of topology-aware preemption [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Two illustrative examples of FlexTopo and resource allocation states [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Topology-aware candidates sourcing and scoring [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Scheduling and preemption workflow in Gödel[17] [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Snapshot of GPU allocation distribution before and after deploying topology-aware scheduling [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Number of instances during preemption Topology Affinity Hit Rate To evaluate the improvements achieved with FlexTopo-based preemption, we analyze the topology affinity hit rate for Workloads B and C (shown in [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Overhead of candidate sourcing across various workloads [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Distribution of candidate sourcing overhead in microseconds [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 32 canonical work pages

  1. [1]

    A survey on large language models: Applications, challenges, limitations, and practical usage

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023

    2023

  2. [2]

    Efficient training of large language models on distributed infrastructures: A survey

    Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infrastructures: A survey. arXiv preprint arXiv:2407.20018, 2024

    arXiv 2024

  3. [3]

    Towards efficient and reliable llm serving: A real-world workload study

    Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Towards efficient and reliable llm serving: A real-world workload study. arXiv preprint arXiv:2401.17644, 2024

    arXiv 2024

  4. [4]

    Gödel: Unified large-scale resource management and scheduling at bytedance

    Wu Xiang, Yakun Li, Yuquan Ren, Fan Jiang, Chaohui Xin, Varun Gupta, Chao Xiang, Xinyi Song, Meng Liu, Bing Li, et al. Gödel: Unified large-scale resource management and scheduling at bytedance. In Proceedings of the 2023 ACM Symposium on Cloud Computing, pages 308–323, 2023. 15 Topology-aware Preemption

    work page 2023

  5. [5]

    Topology-aware gpu scheduling for learning workloads in cloud environments

    Marcelo Amaral, Jordà Polo, David Carrera, Seetharami Seelam, and Malgorzata Steinder. Topology-aware gpu scheduling for learning workloads in cloud environments. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12, 2017

    work page 2017

  6. [6]

    Numa (non-uniform memory access): An overview: Numa becomes more common because memory controllers get close to execution units on microprocessors

    Christoph Lameter. Numa (non-uniform memory access): An overview: Numa becomes more common because memory controllers get close to execution units on microprocessors. Queue, 11(7):40–51, 2013

    work page 2013

  7. [7]

    M\’elange: Cost efficient large language model serving by exploiting gpu heterogeneity

    Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu, Doyoung Kim, Wei-Lin Chiang, Alvin Cheung, and Ion Stoica. M\’elange: Cost efficient large language model serving by exploiting gpu heterogeneity. arXiv preprint arXiv:2404.14527, 2024

    arXiv 2024

  8. [8]

    Fastertransformer

    NVIDIA. Fastertransformer. https://github.com/NVIDIA/FasterTransformer, 2021. Commit: df4a753, Accessed on: 2023-11-25

    work page 2021

Show all 42 references
  1. [9]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  2. [10]

    FMInference. Flexgen. https://github.com/FMInference/FlexGen, 2023. Commit: d34f7b4, Accessed on: 2023-11-25

    2023

  3. [11]

    Huggingface text generation inference

    Huggingface. Huggingface text generation inference. https://github.com/huggingface/ text-generation-inference, 2023. Commit: 3c02262, Accessed on: 2023-11-25

    2023

  4. [12]

    Deepspeed inference

    Microsoft. Deepspeed inference. https://github.com/microsoft/DeepSpeed, 2022. Commit: 2afa1c7, Accessed on: 2023-11-25

    2022

  5. [13]

    Tensorrt-llm

    NVIDIA. Tensorrt-llm. https://github.com/NVIDIA/TensorRT-LLM, 2023. Commit: 6837c81, Accessed on: 2023-11-25

    2023

  6. [14]

    To- wards efficient generative large language model serving: A survey from algorithms to systems

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. To- wards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234, 2023

    2023 arXiv

  7. [15]

    Kubernetes topology manager moves to beta

    Kubernetes.io. Kubernetes topology manager moves to beta. https://kubernetes.io/blog/2020/04/01/ kubernetes-1-18-feature-topoloy-manager-beta/ , 2023. Published on: 2020-04-01

    2020

  8. [16]

    Pod priority and preemption

    Kubernetes.io. Pod priority and preemption. https://kubernetes.io/docs/concepts/ scheduling-eviction/pod-priority-preemption , 2024. Accessed on: 2024-11-04

    2024

  9. [17]

    Godel scheduler: a unified scheduler for online and offline tasks

    Kubewharf. Godel scheduler: a unified scheduler for online and offline tasks. https://github.com/ kubewharf/godel-scheduler, 2023

    2023

  10. [18]

    Daemonset

    Kubernetes.io. Daemonset. https://kubernetes.io/docs/concepts/workloads/controllers/ daemonset/, 2024. Accessed on: 2024-11-14

    2024

  11. [19]

    Kubernetes without kubelet

    Kwok.sigs.k8s.io. Kubernetes without kubelet. https://kwok.sigs.k8s.io/, 2024. Accessed on: 2024-11-13

    2024

  12. [20]

    Control topology management policies on a node

    Kubernetes.io. Control topology management policies on a node. https://kubernetes.io/docs/tasks/ administer-cluster/topology-manager/policy-restricted , 2024. Accessed on: 2024-11-14

    2024

  13. [21]

    Towards {GPU} utilization prediction for cloud deep learning

    Gingfung Yeung, Damian Borowiec, Adrian Friday, Richard Harper, and Peter Garraghan. Towards {GPU} utilization prediction for cloud deep learning. In 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20), 2020

    2020

  14. [22]

    Horus: Interference-aware and prediction-based scheduling in deep learning systems

    Gingfung Yeung, Damian Borowiec, Renyu Yang, Adrian Friday, Richard Harper, and Peter Garraghan. Horus: Interference-aware and prediction-based scheduling in deep learning systems. IEEE Transactions on Parallel and Distributed Systems, 33(1):88–100, 2021

    2021

  15. [23]

    Beware of fragmentation: Scheduling {GPU-Sharing} workloads with fragmentation gradient descent

    Qizhen Weng, Lingyun Yang, Yinghao Yu, Wei Wang, Xiaochuan Tang, Guodong Yang, and Liping Zhang. Beware of fragmentation: Scheduling {GPU-Sharing} workloads with fragmentation gradient descent. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 995–1008, 2023

    2023

  16. [24]

    {HiveD}: Sharing a {GPU} cluster for deep learning with guarantees

    Hanyu Zhao, Zhenhua Han, Zhi Yang, Quanlu Zhang, Fan Yang, Lidong Zhou, Mao Yang, Francis CM Lau, Yuqi Wang, Yifan Xiong, et al. {HiveD}: Sharing a {GPU} cluster for deep learning with guarantees. In 14th USENIX symposium on operating systems design and implementation (OSDI 20...

    2020

  17. [25]

    Supporting gpu sharing in cloud environments with a transparent runtime consolidation framework

    Vignesh T Ravi, Michela Becchi, Gagan Agrawal, and Srimat Chakradhar. Supporting gpu sharing in cloud environments with a transparent runtime consolidation framework. In Proceedings of the 20th international symposium on High performance distributed computing, pages 217–228, 2011

    2011

  18. [26]

    Fine-grained gpu sharing primitives for deep learning applications

    Peifeng Yu and Mosharaf Chowdhury. Fine-grained gpu sharing primitives for deep learning applications. Proceedings of Machine Learning and Systems, 2:98–111, 2020. 16 Topology-aware Preemption

    2020

  19. [27]

    Gpushare: Fair-sharing middleware for gpu clouds

    Anshuman Goswami, Jeffrey Young, Karsten Schwan, Naila Farooqui, Ada Gavrilovska, Matthew Wolf, and Greg Eisenhauer. Gpushare: Fair-sharing middleware for gpu clouds. In 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1769–1776. ...

    2016

  20. [28]

    Nvidia cloud native technologies: Gpu sharing

    NVIDIA. Nvidia cloud native technologies: Gpu sharing. https://docs.nvidia.com/datacenter/ cloud-native/gpu-operator/latest/gpu-sharing.html , 2024. Accessed on: 2024-11-04

    2024

  21. [29]

    Advanced features in ibm power8 systems

    Balaram Sinharoy, Randy Swanberg, Naresh Nayar, B Mealey, Jeff Stuecheli, Berni Schiefer, Jens Leenstra, Joefon Jann, Philipp Oehler, David Levitan, et al. Advanced features in ibm power8 systems. IBM Journal of Research and Development, 59(1):1–1, 2015

    2015

  22. [30]

    Performance evaluation of the nvidia tesla p100: Our directive-based partitioning and pipelining vs

    Xuewen Cui, Thomas RW Scogland, Bronis R de Supinski, and Wu-chun Feng. Performance evaluation of the nvidia tesla p100: Our directive-based partitioning and pipelining vs. nvidia’s unified memory. Matrix, 40:50, 2017

    2017

  23. [31]

    Topology-aware scheduling framework for microservice applications in cloud

    Xin Li, Junsong Zhou, Xin Wei, Dawei Li, Zhuzhong Qian, Jie Wu, Xiaolin Qin, and Sanglu Lu. Topology-aware scheduling framework for microservice applications in cloud. IEEE Transactions on Parallel and Distributed Systems, 34(5):1635–1649, 2023

    2023

  24. [32]

    Katalyst core

    Kubewharf. Katalyst core. https://github.com/kubewharf/katalyst-core, 2023. Accessed on: 2024-11- 04

    2023

  25. [33]

    Topology-aware resource allocation for data-intensive workloads

    Gunho Lee, Niraj Tolia, Parthasarathy Ranganathan, and Randy H Katz. Topology-aware resource allocation for data-intensive workloads. In Proceedings of the first ACM asia-pacific workshop on Workshop on systems, pages 1–6, 2010

    2010

  26. [34]

    Towards topology aware pre-emptive job scheduling with deep reinforcement learning

    Bon Ryu, Aijun An, Zana Rashidi, Junfeng Liu, and Yonggang Hu. Towards topology aware pre-emptive job scheduling with deep reinforcement learning. In Proceedings of the 30th Annual International Conference on Computer Science and Software Engineering, pages 83–92, 2020

    2020

  27. [35]

    Microsecond-scale preemption for concurrent {GPU- accelerated}{DNN} inferences

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent {GPU- accelerated}{DNN} inferences. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, 2022

    2022

  28. [36]

    Efficiently programming large language models using sglang

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. arXiv e-prints, pages arXiv–2312, 2023

    2023

  29. [37]

    Orca: A distributed serving system for {Transformer-Based} generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for {Transformer-Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    2022

  30. [38]

    Fast distributed inference serving for large language models

    Bingyang Wu, Yinmin Zhong, Zili Zhang, Gang Huang, Xuanzhe Liu, and Xin Jin. Fast distributed inference serving for large language models. arXiv preprint arXiv:2305.05920, 2023

    2023 arXiv

  31. [39]

    Bert loses patience: Fast and robust inference with early exit

    Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330–18341, 2020

    2020

  32. [40]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022

    2022

  33. [41]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  34. [42]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100,...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.