Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Adapting parallelism between denoising steps lifts SLO attainment by up to 32% in mixed DiT serving.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:51 UTC pith:SDYWGEYX

load-bearing objection TetriServe is a genuine step forward for DiT serving—step-level elastic sequence parallelism is new and the evaluation mostly backs it—but the headline SAR gains rest on an unmeasured assumption that step-transition overhead (latent transfer and NCCL warmup) is negligible. the 3 major comments →

arxiv 2510.01565 v4 pith:SDYWGEYX submitted 2025-10-02 cs.LG cs.DC

TetriServe: Efficiently Serving Mixed DiT Workloads

Runyu Lu , Shiqi He , Wenxuan Tan , Shenggui Li , Ruofan Wu , Jeff J. Ma , Ang Chen , Mosharaf Chowdhury This is my paper
classification cs.LG cs.DC
keywords DiT servingsequence parallelismSLO attainmentdeadline-aware schedulinground-based schedulingdiffusion transformersrequest packingGPU scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that fixed-degree sequence parallelism — the standard way to parallelize diffusion transformer (DiT) inference — is fundamentally mismatched to mixed-resolution workloads, where the best GPU count for a small image is wasteful or slow for a large one. It proposes step-level sequence parallelism: within a single request's denoising loop, the scheduler can change how many GPUs execute each step, allocating more GPUs to urgent or high-resolution steps and fewer to the rest. To make this tractable online, TetriServe divides time into fixed rounds; each round it computes the minimal GPU allocation that lets each pending request still meet its deadline, then packs requests onto the GPU pool with a dynamic program that maximizes how many requests survive to the next round without becoming 'definitely late.' On two open DiT models and two GPU platforms, the paper reports up to 32% higher SLO attainment than the best fixed-degree baseline at tight deadlines, with the largest gains in skewed, large-image-dominant workloads.

Core claim

The central claim: for deadline-constrained DiT serving, the degree of sequence parallelism should be a per-step decision, not a per-request constant. The paper formalizes the offline problem of scheduling dependent steps on a fixed GPU pool to maximize deadline satisfaction and proves it NP-hard even with single-step requests, then shows a round-based heuristic is practical. The heuristic pre-profiles step latency as a function of GPU count; per round, it chooses for each request the minimal GPU allocation that satisfies its deadline while minimizing GPU-hours, and packs requests using a group-knapsack dynamic program that maximizes the number of requests not becoming 'definitely late' (i.e

What carries the argument

A round-based, deadline-aware scheduler. A 'round' is a fixed-duration time window that discretizes continuous time so scheduling decisions become group-knapsack problems. The scheduling loop combines: (1) a cost model built from offline profiling of per-step latency versus GPU count, which identifies the minimal feasible GPU allocation per request; (2) a dynamic program that packs the chosen allocations into the N-GPU pool, maximizing the number of requests that will not be 'definitely late' at the next round boundary; and (3) placement preservation and elastic scale-up to avoid idle GPU bubbles. The NP-hardness reduction from a single-step integer program both motivates the round heuristic

Load-bearing premise

The scheduler treats the time to move intermediate latents between differently sized GPU groups as negligible and excludes it from deadline accounting; if that transfer or the communication-group reconfiguration at round boundaries is not actually negligible, the chosen step allocations will miss the deadlines they were computed to meet.

What would settle it

Take a request that the scheduler would change from one GPU to four mid-run; measure the wall-clock time of the step that follows the change, including latent handoff and communication-group warm-up, at realistic load. If the measured step time exceeds the profiled per-step time by more than about 10%, the deadline accounting in the dynamic program is optimistic and the claimed SLO attainment gains would shrink accordingly. A simpler check: rerun the evaluation with latent-transfer time included in deadline accounting and compare the SLO attainment ratios.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If TetriServe is right, serving systems for image DiTs should treat parallelism as an elastic per-step resource, not a fixed per-request configuration.
  • The 10–15% average (up to 32% at tight SLOs) improvement over the best fixed strategy implies that even modest flexibility in GPU allocation can recover most of the lost goodput in heterogeneous workloads.
  • Because the approach relies only on predictable per-step times and a discrete set of resolutions, it could transfer to other iterative generative models (e.g., video DiTs) with comparable step profiles.
  • The paper's claim that image quality is unaffected follows because parallelism changes only the compute layout, not the arithmetic per step; if true, adaptive scheduling is free in the sense of not trading quality for latency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The deadline accounting explicitly ignores latent-transfer time between differently sized GPU groups; if reconfiguration costs on real clusters (communication-channel setup, residual tensor movement) exceed the sub-millisecond assumption even occasionally, the DP's 'definitely late' classification and the 32% figure would need recomputation under actual round-boundary overhead. This is an editoria
  • The step-granularity sensitivity shown in §6.4 suggests the round length τ could itself become a load-adaptive parameter rather than a fixed constant; at high arrival rates a shorter round with adaptive granularity might outperform the fixed 5-step sweet spot.
  • The group-knapsack DP scales as O(R·N) per round, which makes the approach plausible for clusters of hundreds of GPUs; the paper's experiments at 8 and 4 GPUs do not demonstrate that scale, but the complexity argument points there.
  • Because the scheduler only needs a discrete resolution profile (e.g., 256/512/1024/2048), it can be combined with latent-caching systems by folding cached-step counts into the remaining-step accounting, a connection the paper mentions but does not quantify.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents TetriServe, a serving system for Diffusion Transformer (DiT) workloads that dynamically changes the sequence-parallelism (SP) degree of individual requests at the granularity of individual denoising steps. The scheduler operates in fixed-duration rounds: for each pending request it computes a minimal GPU allocation from an offline profiled cost model, then uses a dynamic program to pack requests into the current round while maximizing the number of requests that are not 'definitely late' at the next round boundary. The system also includes placement preservation, work-conserving elastic scale-up, and selective continuous batching. The evaluation compares TetriServe against four fixed-SP xDiT baselines on FLUX.1-dev (8×H100) and SD3 (4×A40), reporting up to 32% higher SLO attainment (SAR) with average improvements of 10% on a Uniform mix and 15% on a Skewed mix at 12 req/min. The authors claim NP-hardness of the offline step-level scheduling problem and motivate their round-based heuristic accordingly.

Significance. If the system works as described, it is a useful contribution to the emerging area of DiT serving. The paper identifies a real inefficiency of fixed-parallelism serving, proposes a concrete and implementable step-level scheduling mechanism, and validates it on two model families and two hardware platforms. The implementation details (latent transfer abstraction, process group warmup, DP scheduler) are practical and likely to be of interest to systems researchers. However, the work's headline claim is not yet fully supported: the exclusion of latent-transfer/NCCL warmup overhead from deadline accounting is untested, the formal hardness argument is asserted rather than proven, and the evaluation omits the number of denoising steps, error bars, and a comparison to a dynamic-parallelism baseline. These issues are load-bearing for the central claim and require further work, although they appear fixable within the scope of the paper.

major comments (3)
  1. [§5 'Latent Transfer' and Algorithm 1 (line 10)] The scheduler explicitly excludes latent-transfer time from deadline accounting because 'latent tensors are compact, so transfer overhead is negligible.' This is an unmeasured assumption. When TetriServe changes a request's SP degree between steps, the latent must be redistributed across a new GPU group, and if that group has not been pre-warmed, NCCL channel initialization occurs on first use (see §5 'Communication Process Groups Warmup'). The per-step times T_i(k) profiled offline are for a fixed, warmed group. Consequently LB_i(o) in Algorithm 1 is not a true lower bound on residual time; a request classified as 'not definitely late' can become late solely because of reconfiguration costs at round boundaries. Fixed-SP baselines never pay these costs, so the reported 10–15% average and 32% peak SAR improvements could shrink at tight SLO scales, where slack is small. The paper needs to
  2. [§4.1 'NP-hardness'] The paper claims 'we prove that finding a globally optimal step-level schedule ... is NP-hard,' but the argument is: (i) formulate a single-step special case as a Zero-one Integer Linear Program, and (ii) state that solving such formulations is NP-hard [15,24,35]. This is not a proof. The hardness of a ZILP formulation does not imply hardness of the specific problem; a reduction from a known NP-hard problem (e.g., bin packing or multiprocessor scheduling) is needed. The claim appears in the contribution list and in Section 4.1, so it should either be replaced by a real reduction or weakened to 'the problem admits a natural ILP formulation whose general instance class is NP-hard.' As written, the statement overreaches.
  3. [§6.1 and §6.2 'Methodology' and 'End-to-End Performance'] The evaluation omits three pieces of information that are essential for interpreting the central claim. First, the number of denoising steps per request is never stated; the step-level scheduler's granularity and the DP's per-round progress depend directly on this number. Second, no error bars, confidence intervals, or number of random seeds are reported for the SAR numbers; the workload is generated by a Poisson process and the results could vary substantially across runs. Third, the comparison is exclusively against fixed-SP xDiT variants. The paper itself cites DDiT [13] as a dynamic-resource-allocation DiT serving system; a comparison against a dynamic-parallelism baseline (or a clear explanation of why it is not applicable to image generation) is necessary before claiming 'up to 32% higher SLO attainment compared to existing solutions.' At minimum, the abstract and contributions sho
minor comments (5)
  1. [§5 'Latent Transfer'] The term 'FluidSP' is used without definition or prior introduction. If it is the name of the runtime mechanism for step-level SP, define it in Section 3 or 4.
  2. [§4.2.2 'Request Packing'] The text says the DP 'maximizes the number of surviving requests under the round capacity N,' but the DP state dp[c] is described as 'exactly capacity c consumed.' Since the DP transition allows any capacity ≤ N and the final selection is arg max over c, the wording should be 'at most capacity c' to avoid confusion.
  3. [§2.3 / Figure 1] The toy example in Figure 1 is useful but the labels 'R1', 'R2', 'R3' are not visible in the printed figure, making it hard to map the textual description to the diagrams. Please increase font size and add a legend or caption explaining the color/shape encoding.
  4. [§6.4 'Step Granularity'] The sensitivity analysis varies 'step granularity' from 1 to 10 steps, but the scheduler is described as round-based with a fixed round duration τ. The relationship between step granularity and round duration is not explained; is the round duration adjusted to match the granularity, or is granularity a separate knob? This should be clarified.
  5. [Global] There are minor typographical issues, e.g., 'Workloads' capitalized mid-sentence in §6.2, and the references to Nirvana [2,3] appear twice in the reference list. A final proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity found; the central claim is validated empirically against external fixed-SP baselines.

full rationale

TetriServe's derivation chain is not circular. The scheduler's offline cost model profiles per-step execution times T_ij(k) and uses them to select minimal GPU allocations and round-level packings; the reported SLO attainment ratios are then measured empirically against xDiT fixed-parallelism configurations, not fitted from those profiled values. The claimed 10-15% average and 32% peak improvements are external comparisons, not outputs of the scheduler's objective. The DP planner uses T_i^min as a lower bound on remaining work, but SAR itself is an independently measured outcome, so no prediction is forced by construction. The paper's self-citations (MuxServe, Sequence Parallelism) appear only as background or implementation reuse, and they do not carry the central load-bearing claim. One non-circular but noteworthy risk exists in Section 5: the statement that 'latent-transfer time' is excluded from deadline accounting because 'transfer overhead is negligible' is an unmeasured assumption, and if actual SP-degree transitions incur non-negligible reconfiguration cost, the scheduler's survival decisions could be optimistic. This is a correctness or robustness concern about an assumption, not a circularity, because the evaluation does not define success in terms of that assumed-away cost. All other load-bearing components—the round-based scheduling, NP-hardness reduction, and empirical evaluation—are self-contained and externally checked against fixed-degree baselines.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The system depends on empirically profiled step times, several hand-set evaluation parameters, and the assumption that step-level SP reconfiguration is cheap. No new physical entities or fitted theory constants are introduced; the main audit items are the unspecified round duration, step granularity, and the unmeasured latent-transfer overhead.

free parameters (6)
  • Round duration tau
    Fixed round length divides scheduling; tau is a tunable system parameter with no stated default, affecting packing granularity and queueing delay (§4.2.2).
  • Step granularity = moderate 5/10 steps suggested; no default given
    Figure 13 varies granularity over 1,2,5,10,15 steps; the value used in the main results is not specified (§6.4).
  • Per-resolution baseline SLOs = 1.5s, 2.0s, 3.0s, 5.0s
    Hand-set latency targets for 256, 512, 1024, and 2048 resolutions; these define the metric being optimized and directly shape SAR results (§6.1).
  • Skewed workload parameter alpha = 1.0
    Controls how strongly the Skewed workload biases toward large resolutions (§6.1).
  • Offline execution-time profile T_ij(k) = lookup table from measurements
    The cost model is populated by profiling and drives all allocation decisions; it is an empirical input rather than a derived quantity (§4.2.1).
  • Number of denoising steps per request = not stated
    SLO feasibility for 2048px/5s depends on the number of steps; the paper never specifies this in the evaluation setup (§6.1).
axioms (5)
  • domain assumption DiT per-step execution time is highly predictable, with CV below 0.7% across 100 runs (§2.2, Table 1).
    The entire deadline-aware scheduler relies on stable per-step latencies; if real contention breaks this stability, the cost model allocations become inaccurate.
  • ad hoc to paper Latent transfer and SP reconfiguration overheads are negligible and can be excluded from deadline accounting (§5 'Latent Transfer').
    The scheduler explicitly omits latent-transfer time; no measurement is provided for this overhead, which is load-bearing for step-level SP.
  • domain assumption The VAE decoder is off the critical path, so sequential per-request decoding does not increase end-to-end latency (§5).
    Serializing VAE decoding bounds memory but assumes the decoder is not a bottleneck at the tested resolutions and batch sizes.
  • domain assumption Step-level preemption and changing SP degree at step boundaries are semantically safe for diffusion inference.
    The system transfers intermediate latents between GPU groups; the paper asserts this is correct and cheap, but does not verify numerical equivalence across configurations.
  • domain assumption The offline profile remains valid online under varying batch composition and load (§4.2.1, §6.1).
    Profiling is done in isolation; the evaluation assumes these times hold when requests are packed and batched together.

pith-pipeline@v1.3.0-alltime-deepseek · 17159 in / 14202 out tokens · 133705 ms · 2026-08-04T12:51:27.922140+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of TetriServe: Efficiently Serving Mixed DiT Workloads." pith.science (2026). https://pith.science/paper/SDYWGEYX

@misc{pith2026251001565,
  author       = {Pith},
  title        = {Pith review of: TetriServe: Efficiently Serving Mixed DiT Workloads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDYWGEYX}},
  note         = {Machine review of arXiv:2510.01565}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Diffusion Transformer (DiT) models excel at generating high-quality images through iterative denoising steps, but serving them under strict Service Level Objectives (SLOs) is challenging due to their high computational cost, particularly at larger resolutions. Existing serving systems use fixed-degree sequence parallelism, which is inefficient for heterogeneous workloads with mixed resolutions and deadlines, leading to poor GPU utilization and low SLO attainment. In this paper, we propose step-level sequence parallelism to dynamically adjust the degree of parallelism of individual requests according to their deadlines. We present TetriServe, a DiT serving system that implements this strategy for highly efficient image generation. Specifically, TetriServe introduces a novel round-based scheduling mechanism that improves SLO attainment by (1) discretizing time into fixed rounds to make deadline-aware scheduling tractable, (2) adapting parallelism at the step level and minimizing GPU hour consumption, and (3) jointly packing requests to minimize late completions. Extensive evaluation on state-of-the-art DiT models shows that TetriServe achieves up to 32% higher SLO attainment compared to existing solutions without degrading image quality.

Figures

Figures reproduced from arXiv: 2510.01565 by Ang Chen, Jeff J. Ma, Mosharaf Chowdhury, Runyu Lu, Ruofan Wu, Shenggui Li, Shiqi He, Wenxuan Tan.

Figure 1
Figure 1. Figure 1: Three DiT serving requests—each with 5 denoising steps—arrive over time with different SLOs and output reso￾lutions. DiT serving solutions using static parallelism cannot adapt and fail to meet multiple SLOs. TetriServe meets more SLOs via SLO-aware scheduling and packing. latent representation over a sequence of discrete denoising steps, setting a new standard for generation quality. As DiT models move in… view at source ↗
Figure 2
Figure 2. Figure 2: End-to-end scaling efficiency of FLUX.1-dev [17] for four resolutions on an 8×H100 server for different batch size (BS). Efficiency scales sublinearly. Larger resolutions benefit more from increased parallelism, while smaller resolu￾tions exhibit limited scalability. Note different Y-axes scales. accurate performance modeling and effective deadline-aware scheduling. Insight 1: DiT workloads consist of hete… view at source ↗
Figure 4
Figure 4. Figure 4: System overview of TetriServe. step execution times and heterogeneous scaling behavior, this approach enables finer-grained resource shaping and better SLO attainment than conventional fixed-SP policies. Insight 3: Step-level parallelism adapts GPU allocation to request deadlines, avoiding the resource waste of fixed parallelism and improving SLO satisfaction. 3 TetriServe Overview TetriServe allows more D… view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of TetriServe’s scheduling process. The progression is shown from top to bottom: each row represents an intermediate scheduling step, while the final row shows the actual GPU allocation decision. Time is fixed across rows. their deadlines. To make the problem tractable, we approxi￾mate it by minimizing the number of requests that become definitely late—those that cannot meet their deadlines ev… view at source ↗
Figure 6
Figure 6. Figure 6: End-to-end performance on the Uniform workload at 12 req/min. (Top) TetriServe achieves the highest SLO Attainment Ratio (SAR) across all SLO scales. (Bottom) The spider plots show that xDiT variants only perform well for specific resolutions, TetriServe delivers high SAR across all resolutions no matter tight or loose SLO Setting. 1.0x 1.1x 1.2x 1.3x 1.4x 1.5x SLO Scale 0.0 0.2 0.4 0.6 0.8 1.0 SLO Attainm… view at source ↗
Figure 7
Figure 7. Figure 7: End-to-end performance on the Skewed workload at 12 req/min. (Top) TetriServe again achieves the highest SLO Attainment Ratio (SAR) across all SLO scales. (Bottom) The spider plots confirm that TetriServe’s adaptive paral￾lelism provides robust performance across all resolutions, even in a workload dominated by large images TetriServe Benefits All Resolutions. TetriServe’s strength lies in its ability to d… view at source ↗
Figure 9
Figure 9. Figure 9: Average parallel degree of TetriServe during serv￾ing under the Uniform workload (1.5× SLO Scale). TetriServe dynamically adjusts sequence parallelism (SP) per request, assigning more GPUs to intensive requests (longer bars) to meet deadlines. 1.0x 1.1x 1.3x 1.5x SLO Scale 0.0 0.2 0.4 0.6 0.8 1.0 SLO Attainment Ratio TetriServe (ours) xDiT (SP=1) xDiT (SP=2) xDiT (SP=4) (a) SAR vs. SLO Scale (SD3, Uniform … view at source ↗
Figure 10
Figure 10. Figure 10: TetriServe’s performance on the Stable Diffusion 3 (SD3) model. The plots show the SLO Attainment Ratio (SAR) as a function of SLO Scale for the Uniform mix (left) and Skewed mix (right) on A40 GPUs. In both workloads, TetriServe consistently outperforms all xDiT variants 6.4 Sensitivity Analysis Different GPU Settings and Models. On SD3, trends align with FLUX. In both the Uniform mix (Figure 10a) and Sk… view at source ↗
Figure 11
Figure 11. Figure 11: SLO Attainment Ratio vs. arrival rate under the Uniform mix (SLO Scale=1.0x). TetriServe gracefully handles increasing load, maintaining a high SAR. 256×256 512×512 1024×1024 2048×2048 Shape Distribution 0.0 0.2 0.4 0.6 0.8 1.0 SLO Attainment Ratio TetriServe (ours) xDiT (SP=1) xDiT (SP=2) xDiT (SP=4) xDiT (SP=8) [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: SLO Attainment Ratio for homogeneous work￾loads at 12 req/min with a 1.5x SLO Scale. Each group of bars represents a workload with only one resolution type. TetriServe consistently achieves the highest SAR across all resolutions. of 1.0× as the arrival rate increases from 6 to 18 req/min. TetriServe demonstrates superior performance across the full range of arrival rates. At low-to-medium rates, TetriServ… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving

    cs.DC 2026-07 conditional novelty 6.0

    FlashDiff reduces diffusion serving latency by 30–97% and raises throughput 1.2–2.2× by adaptively skipping refinement of latent regions that no longer need it.

  2. Xema: Efficient Diffusion Serving through Fine-Grained Memory Management and Auto-Configuration

    cs.DC 2026-07 conditional novelty 6.0

    Trace-guided fine-grained memory control and offline joint planning raise diffusion serving SLO attainment by up to 3.7× while cutting configuration search from hours to minutes.

  3. GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving

    cs.DC 2026-06 unverdicted novelty 6.0

    GF-DiT introduces elastic GPU parallelism scheduling for DiT serving via asynchronous trajectory tasks and group-free collectives, reporting up to 6.01x throughput gains over static configurations.

  4. GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving

    cs.DC 2026-06 unverdicted novelty 6.0

    GF-DiT dynamically adapts parallelism during DiT serving via trajectory tasks and group-free collectives, reporting up to 6x throughput and 95% latency reduction versus static configurations.

  5. FlashDiff: Efficient Regional Execution and Scheduling for Diffusion Model Serving

    cs.DC 2026-07 conditional novelty 5.0

    FlashDiff cuts diffusion serving latency 30–97% and raises throughput 1.2–2.2× by selectively executing only active latent regions and rescheduling the reclaimed compute.

Reference graph

Works this paper leans on

39 extracted references · 12 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Tahir Abbas, Ujwal Gadiraju, Vassilis-Javed Khan, and Panos Markopoulos. 2022. Understanding user perceptions of response delays in crowd-powered conversational systems.Proceedings of the ACM on Human-Computer Interaction6, CSCW2 (2022), 1–42

    2022

  2. [3]

    Shubham Agarwal, Subrata Mitra, Sarthak Chakraborty, Srikrishna Karanam, Koyel Mukherjee, and Shiv Kumar Saini. 2024. Approximate Caching for Efficiently Serving Text-to-Image Diffusion Models. In21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 1173–1189.https:// www.usenix.org/conference...

    2024

  3. [4]

    Stability AI. 2024. Stable Diffusion 3 Medium.https://huggingface.co/ stabilityai/stable-diffusion-3-medium

    2024

  4. [5]

    Stability AI. 2024. Stable Diffusion 3.5 Large.https://huggingface.co/ stabilityai/stable-diffusion-3.5-large

    2024

  5. [6]

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. 2024. Video generation models as world simulators. (2024).https://openai.com/research/video- generation-models-as-world-simulators

    2024

  6. [7]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis- senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Trans- formers for Image Recognition at Scale. InInternational Conference on Learning Representations (ICLR)

    2021

  7. [8]

    Kuntai Du, Bowen Wang, Chen Zhang, Yiming Cheng, Qing Lan, Hejian Sang, Yihua Cheng, Jiayi Yao, Xiaoxuan Liu, Yifan Qiao, Ion Stoica, and Junchen Jiang. 2025. PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications. arXiv:2505.07203 [cs.DC]https://arxiv.org/abs/2505.07203

    Pith/arXiv arXiv 2025

  8. [9]

    Jiangfei Duan, Runyu Lu, Haojie Duanmu, Xiuhong Li, Xingcheng Zhang, Dahua Lin, Ion Stoica, and Hao Zhang. 2024. MuxServe: flexible spatial-temporal multiplexing for multiple LLM serving. InProceed- ings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org

    2024

  9. [10]

    Jiarui Fang, Jinzhe Pan, Xibo Sun, Aoyu Li, and Jiannan Wang. 2024. xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism. arXiv:2411.01738 [cs.DC]https://arxiv.org/abs/ 2411.01738

    Pith/arXiv arXiv 2024

  10. [11]

    2025.Flux.1 AI Image Generator.https://flux1.ai/create

    Flux.1 AI. 2025.Flux.1 AI Image Generator.https://flux1.ai/create

    2025

  11. [12]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. InAdvances in Neural Information Processing Systems (NeurIPS)

    2020

  12. [13]

    Heyang Huang, Cunchen Hu, Jiaqi Zhu, Ziyuan Gao, Liangliang Xu, Yizhou Shan, Yungang Bao, Sun Ninghui, Tianwei Zhang, and Sa Wang

  13. [14]

    Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Ex- treme Long Sequence Transformer Models. arXiv:2309.14509 [cs.LG] https://arxiv.org/abs/2309.14509

    Pith/arXiv arXiv 2023

  14. [15]

    Alind Khare, Dhruv Garg, Sukrit Kalra, Snigdha Grandhi, Ion Sto- ica, and Alexey Tumanov. 2025. SuperServe:Fine-Grained Inference Serving for Unpredictable Workloads. In22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25). 739–758

    2025

  15. [16]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica

  16. [17]

    2024.FLUX.1-dev: Text-to-Image Generation Model

    Black Forest Labs. 2024.FLUX.1-dev: Text-to-Image Generation Model

    2024

  17. [18]

    Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. 2022. Sequence Parallelism: Long Sequence Training from System Perspective. arXiv:2105.13120 [cs.LG]https://arxiv.org/abs/2105.13120

    Pith/arXiv arXiv 2022

  18. [19]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring At- tention with Blockwise Transformers for Near-Infinite Context. arXiv:2310.01889 [cs.CL]https://arxiv.org/abs/2310.01889

    Pith/arXiv arXiv 2023

  19. [20]

    Yixuan Mei, Yonghao Zhuang, Xupeng Miao, Juncheng Yang, Zhihao Jia, and Rashmi Vinayak. 2025. Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, Rotterdam, Netherlands. doi:10.11...

    arXiv 2025

  20. [21]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2023. SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification. CoRRabs/2305.09781 (2023).https://arxiv.org/abs/2305.09781

    Pith/arXiv arXiv 2023

  21. [22]

    Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, and Zhihao Jia. 2024. SpotServe: Serving Generative Large Lan- guage Models on Preemptible Instances. InProceedings of the 29th ACM International Conference on Architectural Support for Programming Lan- guages and Operating Systems (ASPLOS), Vol. 2. ACM, 1112–1127

    2024

  22. [23]

    NVIDIA. 2022. NVIDIA Collective Communication Library (NCCL) Documentation.https://docs.nvidia.com/deeplearning/nccl/user- guide/docs/index.html

    2022

  23. [24]

    1998.Combinatorial optimization: algorithms and complexity

    Christos H Papadimitriou and Kenneth Steiglitz. 1998.Combinatorial optimization: algorithms and complexity. Courier Corporation

    1998

  24. [25]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. InProceedings of the International Symposium on Computer Architecture (ISCA). ACM, Buenos Aires, Argentina

    2024

  25. [26]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Transformers. InInternational Conference on Computer Vision (ICCV). arXiv:2212.09748 [cs.CV]https://arxiv.org/abs/2212.09748 12 TetriServe : Efficient DiT Serving for Heterogeneous Image Generation

    Pith/arXiv arXiv 2023

  26. [27]

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. InIn- ternational Conference on Medical Image Computing and Computer- Assisted Intervention (MICCAI). Springer, 234–241

    2015

  27. [28]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilib- rium thermodynamics.arXiv preprint arXiv:1503.03585(2015)

    Pith/arXiv arXiv 2015

  28. [29]

    Yang Song and Stefano Ermon. 2021. Score-Based Generative Modeling through Stochastic Differential Equations. InInternational Conference on Learning Representations (ICLR)

    2021

  29. [30]

    Stability AI. 2024. Stability AI Platform API Reference.https:// platform.stability.ai/docs/api-referenceAccessed: 2024-11-26

    2024

  30. [31]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. InAdvances in Neural Information Processing Systems (NeurIPS)

    2017

  31. [32]

    Wang, Evan Montoya, David Munechika, Haoyang Yang, Ben- jamin Hoover, and Duen Horng Chau

    Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Ben- jamin Hoover, and Duen Horng Chau. 2023. DiffusionDB: A Large- scale Prompt Gallery Dataset for Text-to-Image Generative Models. arXiv:2210.14896 [cs.CV]https://arxiv.org/abs/2210.14896

    Pith/arXiv arXiv 2023

  32. [33]

    Bingyang Wu, Shengyu Liu, Yinmin Zhong, Peng Sun, Xuanzhe Liu, and Xin Jin. 2024. LoongServe: Efficiently Serving Long- Context Large Language Models with Elastic Sequence Parallelism. arXiv:22405.09526 [cs.DC]https://arxiv.org/abs/22405.09526

    arXiv 2024

  33. [34]

    Yuchen Xia, Divyam Sharma, Yichao Yuan, Souvik Kundu, and Nishil Talati. 2025. MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models. arXiv:2503.11972 [cs.DC]https://arxiv. org/abs/2503.11972

    Pith/arXiv arXiv 2025

  34. [35]

    Hong Zhang, Yupeng Tang, Anurag Khandelwal, and Ion Stoica. 2023. SHEPHERD: Serving DNNs in the wild. In20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 787–808

    2023

  35. [36]

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al

  36. [37]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. InThe Thirty-eighth Annual Conference on Neural Information Processing Sys- tems.https://openreview.net/foru...

    2024

  37. [39]

    arXiv preprint arXiv:2304.11277(2023)

    Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277(2023)

    Pith/arXiv arXiv 2023

  38. [2023]

    InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles

  39. [2025]

    arXiv:2506.13497 [cs.DC]https://arxiv.org/abs/2506

    DDiT: Dynamic Resource Allocation for Diffusion Transformer Model Serving. arXiv:2506.13497 [cs.DC]https://arxiv.org/abs/2506. 13497

    Pith/arXiv arXiv