Pith. sign in

REVIEW 3 major objections 7 minor 22 references

Efficiently Serving Large Multimodal Models Using EPD Disaggregation

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-way split of encode, prefill, and decode onto dedicated GPUs lets multimodal servers handle 10x more images per request and cut first-token latency by up to 71%.

desk verdict Useful disaggregated serving system for LMMs, but the patch-parallelism claim needs an output-equivalence check before the headline gains can be trusted. read the letter →

arxiv 2501.05460 v4 pith:7ZJKKDJ3 submitted 2024-12-25 cs.DC cs.AIcs.CVcs.LG

Gursimran Singh , Xinglu Wang , Yifan Hu , Timothy Yu , Linzi Xing , Wei Jiang , Zhefeng Wang , Xiaolong Bai
show 4 more authors
Yi Li Ying Xiong Yong Zhang Zhenan Fan
This is my paper · ORCID
classification cs.DCcs.AIcs.CVcs.LG
keywords largemultimodalmodelsdisaggregatedservingencode-prefill-decodeintra-requestparallelizationtimetofirsttokenmemoryefficiencydynamicroleswitchingcache
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Serving a large multimodal model means encoding images or video into tokens before the language model can generate text, and that extra stage collides with prefill on the same GPUs, inflating memory and delaying the first token. This paper claims that separating encode, prefill, and decode onto dedicated resources removes the collision, and that the split is practical because multimodal tokens can be cached and moved cheaply, encoding work within one request can be spread across GPUs, and instance roles can be reassigned as demand shifts. If the claim holds, multimodal serving can support more images per request, far larger batches, and tighter latency targets on the same hardware. The paper reports up to 15x lower peak memory, 22x larger batch sizes, 10x more images per request, and up to 71% lower time to first token against baselines that bundle or partially aggregate these stages.

What carries the argument

The load-bearing object is the EPD pipeline itself: encode, EP-migration, prefill, PD-migration, and decode, each on dedicated GPU instances. Three mechanisms carry the argument: the MM cache and asynchronous token transfer that decouple encode from prefill; intra-request parallelization (IRP), which shards one request's image patches across encoders and merges their tokens at prefill; and the dynamic role switcher plus black-box optimizer that keep the stage-wise resource split matched to the workload. IRP is the mechanism that directly converts the disaggregation into lower TTFT, while the MM cache is what makes the extra transfer step affordable.

What would settle it

Run identical prompts through the monolithic encoder and the IRP-sharded encoder and compare generated outputs on a benchmark such as Video-MME; if the sharded pipeline changes answers or accuracy, or if end-to-end TTFT at high concurrency no longer beats the monolithic baseline once token-transfer costs are included, the paper's central claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the encoding stage of an LMM can be disaggregated from prefill and decode, and that this three-way split is the right systems abstraction for multimodal serving. The paper argues that bundling encoding with prefill creates interference and forces the LLM's weights and KV cache to share memory with the vision encoder, which caps batch sizes, image counts, and cache sizes. EPD assigns each stage its own GPU instances, moves multimodal tokens through an MM cache with asynchronous transfer, shards a single request's image patches across encoders via intra-request parallelization, picks per-stage configurations with a black-box optimizer, and switches instance roles dynamically when the workload shifts. Measured against monolithic and prefill-decode-disaggregated baselines on MiniCPM-V 2.6 and InternVL2-8B and InternVL2-26B, the paper reports up to 15x lower peak memory, 22x larger batches, 10x more images per request, 2.2x larger KV caches, 90-100% SLO attainment gains, and up to 71% lower TTFT.

Load-bearing premise

The load-bearing premise is that a request's image patches can be encoded independently across GPUs and then merged without changing the token representations or the model's output, but the paper reports no accuracy or output-equivalence check.

Editorial extensions

If this is right

  • Encoding becomes independently scalable: cluster operators can add encoder GPUs without touching LLM weights or KV-cache memory.
  • Intra-request parallelization converts encoding latency into parallel work, shrinking TTFT by up to 71% on the models tested.
  • Freeing encoding workers from LLM weights and KV caches raises the ceiling on batch size (up to 22x), images per request (up to 10x), and KV-cache allocation (up to 2.2x).
  • Dynamic role switching lets a deployed cluster reallocate instances among encode, prefill, and decode in under 0.7 seconds, absorbing workload shifts without restarting in-flight requests.
  • The framework extends to audio inputs and to NPU hardware, with larger gains when encoding is a greater share of end-to-end latency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the decisive test the paper leaves undone is an output-quality comparison between monolithic and IRP-sharded encoding; until that exists, the speedups are for a pipeline whose equivalence to the original model is unverified.
  • Beyond the paper: the memory savings suggest a heterogeneous deployment where low-memory high-compute devices handle encoding and high-memory devices handle prefill, turning the 15x memory reduction into a cost saving rather than only a capacity gain.
  • Beyond the paper: in edge-cloud settings the same split could keep raw images on edge devices and transmit only multimodal tokens, extending the paper's privacy direction into a concrete architecture.
  • Beyond the paper: a natural stress test is whether IRP's TTFT gains hold when the bottleneck moves from encode compute to token-transfer bandwidth, such as many small images spread across distant nodes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes EPD Disaggregation, a serving framework that separates the multimodal encoding, prefill, and decode stages of an LMM onto dedicated GPU resources, and couples this with asynchronous token transfer, intra-request parallelization (IRP) that shards image patches across encoding workers, a black-box optimizer for resource allocation, and dynamic role switching. The claims are evaluated on MiniCPM-V 2.6, InternVL2-8B, and InternVL2-26B with synthetic workloads, NextQA, and Video-MME, reporting up to 15x lower peak memory, 22x larger batch sizes, 10x more images per request, 2.2x larger KV caches, and substantial TTFT and SLO-attainment improvements over vLLM and a multimodal extension of DistServe.

Significance. If the results hold, EPD is a valuable contribution to LMM serving: it treats the encoding stage as a first-class pipeline component, ships a released implementation, and reports broad improvements across three models and multiple workloads. The inclusion of a limitations section, an NPU deployment, and ablations of the optimizer and role-switching strengthens the empirical package. However, the significance is conditional on two evaluation properties the paper does not currently establish: that IRP's sharded encoding is semantically equivalent to monolithic encoding for the evaluated ViT-based encoders, and that the SLO thresholds used to compare goodput are not chosen post hoc in a way that favors EPD. The evidence is suggestive but not yet conclusive.

major comments (3)
  1. [§3.2.2, §4.1] The IRP mechanism shards a request's image patches across encoding workers and merges patch-level tokens only at prefill, justified by the assertion that 'patches are encoded independently.' This property is not established for MiniCPM-V 2.6 (SigLIP) or InternVL2-8B/26B (InternViT), which are ViT-style encoders with self-attention across patches of an image or tile. If cross-patch attention is removed, the merged tokens differ from the monolithic encoder's output, and the measured TTFT and SLO gains apply to a modified vision encoder whose output quality is unverified. No experiment compares EPD's token representations or downstream accuracy (e.g., on NextQA or Video-MME) against monolithic encoding. Please add an equivalence check (task accuracy, logit-level comparison, or an explicit proof of patch independence for each encoder) and report results; if equivalence fails, the IRP gains should be reframed as serving a modified model.
  2. [Appendix E.3, §4.1] Table 9 states that the SLO thresholds are 'empirically derived' in part from 'what is realistically achievable by both our method and the baselines on a fixed number (8 GPUs) used in experiments.' This makes the SLO-attainment comparison circular: the headline '90–100% improvement' is partly determined by selecting thresholds that EPD meets while baselines miss. Derive the thresholds from an independent, pre-specified rule (e.g., a target percentile of the monolithic single-request TTFT/TPOT distribution, or a fixed multiple of the no-load latency) and re-report SLO attainment and goodput under that rule. The NPU experiment (Appendix F.3) inherits this issue and should be treated the same way.
  3. [§4.3, Table 2] The text states: 'at 4032×3024 resolution, 7× more for InternVL2-26B and 10× more for InternVL2-8B.' Table 2 shows the opposite: InternVL2-26B goes from 1 to 10 images (10×), while InternVL2-8B stays at 19 for both DistServe and EPD (no gain). The abstract's '10x more images per request' claim is therefore not supported by the table for InternVL2-8B. Please correct this mismatch and qualify the headline claim per model (e.g., '10× more for InternVL2-26B at 4K resolution').
minor comments (7)
  1. [Appendix E.3, Table 9] The TPOT entry for InternVL-26B at 6 images per request (0.95 s) is an order of magnitude larger than neighboring entries (0.08 and 0.15 s) and appears to be a typo (likely 0.095 s); please check.
  2. [§4.4, Table 5] The first data row is labeled 'EDP' rather than 'EPD', and the table shows TPOT improving (0.031 to 0.025 s) when the optimizer is disabled; the text explains the goodput and TTFT degradation but does not comment on this TPOT difference, so please add an explanation or correct the table.
  3. [§4.2] The statement that 'the vLLM baseline is equivalent to DistServe and is thus omitted' when decoding is excluded is not self-evident, since vLLM and DistServe may use different numbers of GPUs for the encode+prefill phases; please state the GPU allocation or include vLLM's TTFT result.
  4. [§4.1, §4.4, Appendix E.1] Each condition appears to be a single 100-request run with no repeated trials or confidence intervals; for a goodput comparison, reporting run-to-run variance (e.g., across seeds or bootstrap resamples) would materially strengthen the SLO-attainment claims.
  5. [§4.3, Table 2] For InternVL2-8B, the maximum images per request is reported as 19 in both DistServe and EPD because of a context-length limit; the sentence 'Without this constraint, a larger number of images per request could be supported' is speculative and should be labeled as such.
  6. [Abstract] The phrase 'up to 90-100% improvement' is ambiguous; using 'up to 100 percentage points improvement in SLO attainment' would be clearer.
  7. [Appendix B] The limitations section acknowledges pipeline bubbles and throughput trade-offs but does not discuss the risk that IRP's sharded encoding may change the encoder's output semantics; please add a sentence addressing the equivalence concern raised in the major comments.

Circularity Check

1 steps flagged · score 4.0 of 10

SLO-attainment headline is partly threshold-fitted by construction; core latency/memory results remain independent.

  1. fitted input called prediction [Appendix E.3 (SLO Criteria), used in Section 4.1 and Figures 5, 7, 8]
    "These criteria are empirically derived based on the characteristics of the underlying models, such as the computational complexity of the MME and LLM. We also consider what is realistically achievable by both our method and the baselines on a fixed number (8 GPUs) used in experiments."

    The paper's headline claim of up to 90-100% SLO-attainment improvement is measured against TTFT/TPOT thresholds that the authors explicitly admit were chosen after considering what both EPD and the baselines can achieve. Attainment is defined as the fraction of requests meeting these pre-set thresholds, so placing thresholds where EPD passes and DistServe/vLLM fail makes the SLO-attainment gap partly an artifact of threshold selection rather than an independent measurement of the architecture. The thresholds in Table 9 are not justified by user requirements or external SLO definitions; they are fitted to the tested systems. This is a form of fitting an input (the SLO criteria) to the known outcomes and then reporting the resulting attainment as evidence.

full rationale

EPD is an empirical systems paper: the core contributions are a disaggregation architecture, asynchronous token transfer, intra-request parallelization, a black-box resource optimizer, and role switching. No equation in the paper derives a predicted quantity from a fitted parameter, and the TTFT, memory, batch-size, and KV-cache numbers in Sections 4.2 and 4.3 are direct measurements that stand independently of any tuned threshold. The one genuinely circular element is the SLO-attainment headline: Appendix E.3 states that the TTFT/TPOT SLO criteria were chosen partly based on 'what is realistically achievable by both our method and the baselines,' so the 90-100% SLO-attainment improvement is to a significant degree guaranteed by construction rather than demonstrated by an externally fixed requirement. This is evaluation circularity, not derivation circularity. The IRP assumption that image patches can be encoded independently and merged losslessly is an unverified correctness assumption for standard ViT-style vision encoders, and the paper reports no output-equivalence experiments; however, this is not a circular step because the paper never defines 'independent' in terms of the claimed results and no equation reduces its latency gains to that assumption. Self-citations (DivPrune, IPProtect) appear only in future-work suggestions and are not load-bearing. Overall, the central engineering claims are self-contained, but the SLO-attainment comparison is partially threshold-fitted, warranting a moderate score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. The MM cache, block manager, and role-switching are software components of the proposed system. The central claims rest on three domain assumptions about encoder patch independence, simulator fidelity, and lossless migration, plus calibrated SLO thresholds and engine hyperparameters. The ledger is dominated by measurement and engineering assumptions rather than first-principles derivation.

free parameters (3)
  • SLO thresholds (TTFT/TPOT per model and images-per-request) = Table 9: e.g., MiniCPM-V 2.6 at 2 images has TTFT 1.40 s and TPOT 0.04 s; InternVL2-26B at 8 images has TTFT 15.00 s…
    Defined in Appendix E.3 as empirically derived and chosen considering what is realistically achievable by both the method and the baselines, so they are calibrated to the systems under test rather than independent external SLOs.
  • Beta (cost penalty in optimizer objective) = not stated
    Equation (1) optimizes f(p,b,s) minus beta times cost(p); no value or range for beta is reported, and it directly affects the resource allocations selected.
  • Memory and engine settings = KV cache GPU utilization 50%, MM cache size 3000, block size 16, max blocks 2048, context cap 49152, decode cap 81920
    Appendix E.1 fixes these for all experiments; they influence the reported memory savings and capacity numbers and are chosen by the authors rather than by a stated external standard.
assumptions (3)
  • domain assumption Patches within a request can be encoded independently and merged losslessly in prefill (IRP, Section 3.2.2).
    The validity of intra-request parallelism rests on the vision encoder treating image tiles or patches independently. The paper does not verify output-token equivalence for MiniCPM-V 2.6, InternVL2-8B, or InternVL2-26B.
  • domain assumption The simulator extended from DistServe faithfully models EPD behavior for configuration search (Section 3.2.3).
    The black-box optimizer selects configurations using simulator outputs; no simulator-versus-real-system validation is reported.
  • domain assumption Transferring multimodal tokens and KV caches between stages is lossless and does not change model outputs (Section 3.1, EP and PD migration).
    The paper assumes token and cache migration preserves semantics, but reports no accuracy or output-equivalence comparison against monolithic serving.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficiently Serving Large Multimodal Models Using EPD Disaggregation." pith.science (2026). https://pith.science/paper/7ZJKKDJ3

@misc{pith2026250105460,
  author       = {Pith},
  title        = {Pith review of: Efficiently Serving Large Multimodal Models Using EPD Disaggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ZJKKDJ3}},
  note         = {Machine review of arXiv:2501.05460}
}
read the original abstract

Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such as images, audio, and video, but at the cost of adding a multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first token (TTFT) and time per output token (TPOT). We introduce Encode-Prefill-Decode (EPD) Disaggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedicated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportunities and optimizations. These include a mechanism to cache multimedia tokens for efficient transfer, a novel way to parallelize the encoding load within a request, a module for optimal resource allocation for disaggregated serving, and a novel role-switching method to handle changing workload characteristics. Experimental evaluations with popular LMMs show substantial gains in memory efficiency (up to 15x lower peak memory utilization), batch sizes (up to 22x larger), 10x more images per request, and 2.2x larger KV caches. Furthermore, it leads to significant improvements in SLO attainment (up to 90-100% improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at https://github.com/vbdi/epdserve.

Figures

Figures reproduced from arXiv: 2501.05460 by the authors.

Figure 1
Figure 1. Aggregated (top) vs. disaggregated (bottom) sys [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Impact of disaggregation on supported batch [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The inference pipeline of EPD Disaggregation. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: System architecture of the proposed EPD Disaggregated Inference. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: SLO attainment (↑) for end-to-end inference across multiple models and image counts per request. Subfigures (a), (b), and (c) correspond to MiniCPM-V 2.6, InternVL2-8B, and InternVL2-26B, respectively. The top and bottom rows show results for 2 and 4 images per request…
Figure 6
Figure 6. Figure 6: Distribution of TTFT (Y-axis) across varying numbers of images per request (X-axis) for (a) MiniCPM-V 2.6, (b) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: SLO attainment (↑) versus request rate on the Video-MME dataset using the MiniCPM-V 2.6 model. EPD significantly outperforms competing baselines across all request rates. 4.2. First Token Generation Latency Multimodal requests often impose a heavy load on both the E an…
Figure 9
Figure 9. Figure 9: As shown, EPD is the only configuration that achieves the SLO requirements, while the other baselines fail to meet the SLOs entirely, even at low request rates [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Left: Impact of varying the number of encoding workers in the EPD method. The notation xEyP denotes a configuration with x encoder and y prefill workers. The DistServe method uses a fixed 7P configuration, assigning 7 workers to handle both encoding and prefill steps.…
Figure 11
Figure 11. Figure 11: SLO attainment (↑) for end-to-end inference across multiple models and image counts per request. Subfigures (a), (b), and (c) correspond to MiniCPM-V 2.6, InternVL2-8B, and InternVL2-26B, respectively. The top and bottom rows show results for 6 and 8 images per reques…
Figure 12
Figure 12. Figure 12: Breakdown of latency for encode and prefill stages using the InternVL2-8B model across varying numbers of [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Gulavani, and Ramachandran Ramjee

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills. arXiv:2308.16369, 2023

    arXiv 2023

  4. [4]

    Bayesian performance analysis for black-box optimization benchmarking

    Borja Calvo, Ofer M Shir, Josu Ceberio, Carola Doerr, Hao Wang, Thomas B \"a ck, and Jose A Lozano. Bayesian performance analysis for black-box optimization benchmarking. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1789--1797, 2019

    work page 2019

  5. [5]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

    2024

  6. [6]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of CVPR, pages 24185--24198, 2024

    work page 2024

  7. [7]

    Inference without interference: Disaggregate llm inference for mixed downstream workloads

    Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Jiang Xu, Shuang Chen, Hao Feng, Chenxi Wang, Sa Wang, Yungang Bao, et al. Inference without interference: Disaggregate llm inference for mixed downstream workloads. arXiv:2401.11181, 2024

    arXiv 2024

  8. [8]

    P/d-serve: Serving disaggregated large language model at scale

    Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, et al. P/d-serve: Serving disaggregated large language model at scale. arXiv:2408.08147, 2024

    arXiv 2024

Show all 22 references
  1. [9]

    Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm

    Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. arXiv:2403.05527, 2024

    2024 arXiv

  2. [10]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  3. [11]

    Snapkv: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: LLM knows what you are looking for before generation. arXiv:2404.14469, 2024

    2024 arXiv

  4. [12]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of NeurIPS, 2023

    2023

  5. [13]

    A survey of resource-efficient llm and multimodal foundation models

    Huan Liu, Yujie Zhang, Xue Yang, Xiaowei Li, Huan Chen, Jure Zhu, and Yu Gong. A survey of resource-efficient llm and multimodal foundation models. arXiv:2401.08092, 2024

    2024 arXiv

  6. [14]

    Inf-mllm: Efficient streaming inference of multimodal large language models on a single gpu

    Zhenyu Ning, Jieru Zhao, Qihao Jin, Wenchao Ding, and Minyi Guo. Inf-mllm: Efficient streaming inference of multimodal large language models on a single gpu. arXiv:2409.09086, 2024

    2024 arXiv

  7. [15]

    Splitwise: Efficient generative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Inigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 118--132, 2024

    2024

  8. [16]

    Mooncake: A kvcache-centric disaggregated architecture for llm serving

    Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kvcache-centric disaggregated architecture for llm serving. arXiv:2407.00079, 2024

    2024 arXiv

  9. [17]

    Déjàvu: Kv-cache streaming for fast, fault-tolerant generative llm serving

    Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjàvu: Kv-cache streaming for fast, fault-tolerant generative llm serving. In ICML, 2024

    2024

  10. [18]

    Multimodal large language models: A survey

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247--2256. IEEE, 2023

    2023

  11. [19]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777--9786, 2021

    2021

  12. [20]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv:2408.01800, 2024

    2024 arXiv

  13. [21]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521--538, 2022

    2022

  14. [22]

    Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In OSDI , pages 193--210. USENIX Association, 2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.