Pith. sign in

REVIEW 3 major objections 4 minor 45 references

The paper claims that on multi-partition NUMA GPUs, the performance penalty of LLM serving kernels is governed by how workgroups share each operand, and that the right placement strategy — pinning, replication, or subgroup co-scheduling — c

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:34 UTC pith:JY6E7UAB

load-bearing objection Useful sharing taxonomy for NUMA-aware LLM kernels, but the speedup numbers are idealized upper bounds, not measured optimizations. the 3 major comments →

arxiv 2607.28824 v1 pith:JY6E7UAB submitted 2026-07-30 cs.AR cs.DCcs.PF

Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs

Donghyeon Joo , Sooraj Puthoor , Nuwan Jayasena , Bahar Asgari This is my paper
classification cs.AR cs.DCcs.PF
keywords LLM inferencemulti-partition GPUNUMA localityworkgroup-level analysismemory placementattention kernelsmixture-of-expertsperformance characterization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language model serving kernels run on GPUs composed of several memory partitions, and the default round-robin way pages and workgroups are spread across those partitions makes most accesses remote and slow. The paper sets out to show that every kernel operand falls into one of three inter-workgroup sharing patterns — private, global, or partial — and that this pattern alone determines both how much latency is lost and which optimization could recover it. Using traces from real serving kernels and cycle-level simulation, it finds that removing inter-partition overhead would speed up memory-bound decode kernels by roughly 1.6 to 1.8 times, while a compute-bound prefill kernel barely benefits. The point of the work is that no single memory-placement policy works everywhere: private operands need page pinning, globally shared operands need replication, and partially shared operands need co-scheduling of the workgroup subgroup that shares them. A sympathetic reader would care because multi-partition GPUs are becoming the standard way to scale memory and compute, and this gives a concrete map of where NUMA-aware runtime and hardware support would pay off.

Core claim

The paper's central claim is that LLM kernel operands can be categorized by how workgroups access them — private, global, or partial — and this category predicts the NUMA penalty and the required mitigation. Under default round-robin page interleaving and workgroup dispatch, every large operand has a baseline partition locality of about 1/N, meaning roughly three quarters of accesses are remote on a four-partition GPU. The distinguishing factor is the ceiling reachable by optimized placement: private operands can reach full locality by pinning each workgroup's pages locally, partial operands require co-scheduling the sharing subgroup on one partition before pinning, and globally shared opera

What carries the argument

The argument is carried by a workgroup-level memory trace analysis pipeline that extracts per-workgroup memory chunks from real kernel executions, computes byte-level overlap between workgroup pairs, and labels each operand as private, global, or partial sharing. This taxonomy is paired with a Partition Locality metric that decomposes locality into kernel access footprint, page placement policy, and workgroup scheduling policy, giving baseline locality of about 1/N under default round-robin policies and per-category achievable ceilings. The performance numbers come from a cycle-level simulator configured as a four-partition GPU, comparing round-robin page placement and round-robin workgroup

Load-bearing premise

The reported speedups assume an idealized placement and co-scheduling capability — zero-cost page pinning, no partition capacity limits, no interference from co-located workgroups — that the paper never simulates, so the real achievable gains could be lower.

What would settle it

Run weight-projection and grouped-query attention decode kernels on a multi-partition GPU with pages pinned to the executing partition and the sharing workgroups co-scheduled, then measure whether latency approaches the simulated 1.72x and 1.79x improvements; if the measured speedup is much smaller, the idealized ceiling does not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the paper is right, memory-bound decode kernels in LLM serving can be sped up by roughly 60 to 80 percent through placement-aware page pinning and subgroup co-scheduling, without changing the kernel math or the model.
  • Compute-bound prefill is largely insensitive to NUMA placement, so runtime designers should prioritize decode-phase and MoE kernels when investing in placement support.
  • No single placement policy suffices across operators; systems need to classify operands by sharing pattern and apply pinning, replication, or co-scheduling accordingly.
  • MoE's routing-dependent sharing makes static placement insufficient — activation pages must be placed dynamically per batch, which calls for runtime support beyond compiler-time analysis.
  • Kernel developers can already improve locality by reordering workgroup scheduling so that sharing subgroups land on the same partition, even without new MMU features.
  • The same trace-based categorization can expose cross-kernel producer-consumer reuse, as in MLA's two-stage pipeline, where co-location enables L2 reuse across kernel boundaries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The baseline-locality formula implies the penalty grows with the number of partitions; on a GPU with more than four partitions, the same kernels would show even larger gaps, strengthening the incentive for placement-aware runtimes.
  • A cheap intermediate step — such as coarser page interleaving or large pages with careful alignment — might capture a meaningful fraction of the 1.6-1.8x headroom for private operands without the need for full per-page pinning or subgroup co-scheduling.
  • The sharing-pattern taxonomy is not specific to LLMs; other memory-bound workloads with private or partially shared operands, such as embedding tables or recommendation models, should exhibit the same category-dependent sensitivity on multi-partition GPUs.
  • A natural test of the paper's central claim is to implement only the private-operand pinning strategy on real hardware and measure how much of the simulated 1.72x projection gap is recovered; if pinning overhead or partition capacity limits eat most of it, the idealized ceiling is optimistic.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper characterizes workgroup-level memory access patterns of LLM serving kernels (weight projection, MHA/GQA/MLA decode, FlashAttention prefill, and MoE) on multi-partition NUMA GPUs. It collects memory traces from real kernels via Omniprobe, derives inter-workgroup sharing categories (private, global, partial), defines Partition Locality under round-robin page placement and round-robin workgroup scheduling, and uses an extended MGPUSIM cycle-level simulator to compare default placement with a zero-latency ideal interconnect. It reports latency ratios from 1.05x to 1.79x and uses these to argue that the sharing category of the dominant operand determines both the NUMA penalty and the required optimization, from per-workgroup page pinning to subgroup-aware co-scheduling.

Significance. The taxonomy and trace-analysis methodology are a useful step toward reasoning about intra-package NUMA effects in multi-partition GPUs, an area where prior characterization work has largely focused on monolithic GPUs or system-level inference phases. The case studies are drawn from real kernels in vLLM/SGLang, and the sharing categories are based on a clearly described byte-level overlap analysis rather than fitted to the speedup results, so I do not see a circularity problem. The value of the work, however, is currently limited by a gap between what is measured and what is claimed: the simulator only compares default round-robin placement with a zero-latency ideal, and the proposed pinning/co-scheduling optimizations are not executed, modeled, or costed. With an explicit upper-bound framing and appropriate caveats, the contribution would be a solid characterization; as written, the quantitative 'achievable speedup' claims overstate what is demonstrated.

major comments (3)
  1. [§5.1, Figure 4, Table 3] The speedup numbers are produced by comparing 'default' round-robin placement with a 'zero-latency direct connection' ideal interconnect (Section 5.1), not by executing the pinning, co-scheduling, or replication strategies listed in Table 3. Figure 4 labels this 'Achievable kernel speedup' and Table 3 attaches the numbers to specific 'Required Optimization' strategies. These results are upper bounds on removing all inter-partition traffic, not demonstrated speedups of the proposed optimizations. Please either relabel these as ideal upper bounds throughout the abstract, Section 5.4, Figure 4, and Table 3, or add a modeled/implemented representative optimization path that includes pinning, migration, and contention costs.
  2. [§4.2] The 'Achievable locality under optimized placement' ceilings assume idealized capabilities: per-workgroup pinning can place each WG's pages locally without capacity imbalance; a partial-sharing subgroup of G workgroups can always be co-scheduled on one partition without interference from other subgroups; and global-shared operands can be N-way replicated without memory cost. None of these capabilities, costs, or constraints are represented in the simulator or in the locality analysis. Since Section 5.4's optimization-complexity conclusions rest on these ceilings, the assumptions should be stated explicitly and, ideally, stress-tested (e.g., partition capacity limits, subgroup co-location conflicts, page-granularity mismatch for small slices).
  3. [§5.4 vs. §5.3.4] The central claim that 'the sharing category of the dominant operand determines both the severity of the penalty and the complexity of the solution' is too strong as stated. FlashAttention prefill (Section 5.3.4) exhibits partial KV sharing yet shows only a 1.09x penalty because it is compute-bound, while GQA decode shows 1.79x with partial sharing. Thus severity is jointly determined by sharing category, memory-boundness, and the volume of remote traffic, not by category alone. Please qualify the claim (e.g., 'for memory-bound kernels') or provide a combined metric that accounts for compute intensity.
minor comments (4)
  1. [§5.3.2] The statement that GQA 'increas[es] inter-chiplet traffic by 23.7%' and raises vector memory stall cycles from 1.96 to 2.44 is not supported by any table or methodology description. Please report where these numbers come from or remove them.
  2. [§3.1.5] The chunk gap tolerance δ is configurable, but no value is given in the trace analysis or in the later case studies. Since chunk merging could affect the reported byte-level overlaps and thus the sharing categories, please state the value used and, if feasible, its sensitivity.
  3. [Table 2] The simulation parameters are described as 'chosen to constitute a viable simulation setup' without a source or validation for several values (e.g., interconnect 32-cycle one-way, 768 GB/s; DRAM 95-cycle row-miss). As the absolute speedup depends on these parameters, a sensitivity analysis or a reference to measured MI300X values would strengthen the quantitative conclusions.
  4. [§4.2, Eq. (1)] The baseline locality approximation ≈1/N is stated as valid when the per-WG footprint is large relative to the page size. Some operands listed in Table 3 are very small (e.g., 256B query); it would be helpful to explicitly mark such operands as negligible in the locality analysis, as the text sometimes does, to avoid confusion about Eq. (1)'s applicability.

Circularity Check

0 steps flagged

No significant circularity: sharing taxonomy comes from traces, speedups from an independent simulator; same-author citations are background only.

full rationale

The paper's derivation chain is self-contained for its main claims. The three sharing categories (WG-private, global shared, partial sharing) are defined from byte-overlap intervals computed by the trace-analysis pipeline in Section 3.1.6 and then assigned to operands from real kernel traces; no parameter is fitted to make the taxonomy match the speedup results. The speedup numbers in Figure 4 and Table 3 come from an independent cycle-level simulator (MGPUSIM extended with NVGIM/MCM-GPU parameters) that compares the default round-robin configuration against a zero-latency interconnect, as described in Section 5.1. This is a labeled counterfactual upper bound, not a prediction derived from the taxonomy itself. The per-category locality ceilings in Section 4.2 are analytic consequences of the definitions of private/global/partial sharing under the stated pinning/co-scheduling assumptions; they are not empirical predictions fitted to data. The same-author citations (Chasoň, CODA, MemPod, Coruscant) appear only in background and related-work discussions and are not load-bearing for the paper's central characterization. The main caveat is rhetorical rather than circular: Figure 4 labels zero-interconnect-overhead results as 'Achievable kernel speedup' and Table 3 lists 'Required Optimization' strategies that are not executed, with pinning/co-scheduling costs and partition-capacity limits unmodeled. That is an over-claim about support or generality, not a circular derivation; the simulated default-vs-ideal ratio is transparently defined by the simulation configuration, not a fitted parameter renamed as a result.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The taxonomy rests on virtual-address trace overlap plus an assumed default page/WG round-robin policy; the speedup numbers rest on a hand-configured, unvalidated simulator and an ideal zero-latency configuration. No new physical entities are introduced.

free parameters (3)
  • Chunk gap tolerance delta
    Section 3.1.5: intervals within a configurable gap tolerance delta are merged, directly affecting measured overlap and the resulting sharing category; the value is never reported.
  • Simulator interconnect parameters = 32-cycle one-way, 768 GB/s
    Table 2: latency/bandwidth hand-chosen from MCM-GPU [6]; not calibrated to MI300X. All speedup magnitudes depend on them.
  • Simulator DRAM/L2 latencies = 95-cycle DRAM row-miss, L2 14+2 cycles
    Table 2: taken from MCM-GPU/NVGIM, not measured on the target; these values set the default-vs-ideal speedup ratios.
axioms (6)
  • domain assumption Default policies: 4KB round-robin page interleaving and round-robin workgroup scheduling on MI300X (AMD docs [1] and [5]).
    Baseline Partition Locality of ~1/N and all 'default' speedups depend on this being the actual default behavior.
  • domain assumption Virtual-address overlap between workgroup chunk sets determines physical sharing and NUMA behavior, invariant to physical mapping.
    Section 3 'Virtual Address to Physical Address Mapping' asserts this invariance; if physical page placement can split shared virtual regions across partitions, the derived categories may not match actual remote traffic.
  • domain assumption Simulator parameters (MCM-GPU/NVGIM latencies) represent a 4-partition MI300X-like GPU without calibration.
    Section 5.1 and Table 2: 'chosen to constitute a viable simulation setup'; no validation against real MI300X measurements.
  • domain assumption Zero-latency ideal interconnect gives the achievable speedup bound.
    Section 5.1: every inter-partition request completes in one cycle; the paper treats the default-vs-ideal ratio as optimization headroom but does not model pinning or co-scheduling costs.
  • ad hoc to paper Per-WG page pinning and subgroup co-scheduling can reach locality ~1.0 without capacity, replication, or contention costs for private/partial operands.
    Section 4.2 'Achievable locality under optimized placement'; this ceiling is asserted analytically, never simulated, and is the basis of the Required Optimization column in Table 3.
  • domain assumption Omniprobe instrumentation produces traces representative of uninstrumented execution.
    All sharing categories are extracted from these traces; instrumentation overhead and perturbation are not discussed or validated.

pith-pipeline@v1.3.0-alltime-deepseek · 17803 in / 13566 out tokens · 143763 ms · 2026-08-03T01:34:18.819814+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs." pith.science (2026). https://pith.science/paper/JY6E7UAB

@misc{pith2026260728824,
  author       = {Pith},
  title        = {Pith review of: Characterizing LLM Kernel Access and Memory Interaction in Multi-Partition NUMA GPUs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JY6E7UAB}},
  note         = {Machine review of arXiv:2607.28824}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language model (LLM) workloads motivate multi-partition GPUs as a path to scaling compute and memory capacity, but their non-uniform memory access characteristics and inter-partition communication can amplify contention and degrade locality, leading to suboptimal kernel latency. To address this, we analyze performance-critical LLM kernel implementations spanning weight projection, mixture-of-experts, and attention variants of state-of-the-art serving engines to present a characterization of data access patterns in multi-partition GPUs. First, we introduce memory trace analysis methodology to derive workgroup-level data access and sharing behavior, then evaluate the locality implications on latency using a cycle-level simulator. Using these tools, we categorize LLM kernel operands into three inter-workgroup sharing patterns (global, partial, or private) and show that the required optimization strategies differ across categories, from simple per-workgroup pinning to subgroup-aware co-scheduling. Our findings highlight the need for placement-aware kernel programming and smarter architectural support for work and data locality in multi-partition GPUs.

Figures

Figures reproduced from arXiv: 2607.28824 by Bahar Asgari, Donghyeon Joo, Nuwan Jayasena, Sooraj Puthoor.

Figure 1
Figure 1. Figure 1: Multi-Partition Architecture of AMD Instinct [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example of Omniprobe Memory Trace. 3.1.2. Kernel and Workgroup Selection. From the fil￾tered trace, we enumerate all unique kernel dispatches (identified by dispatch ID and kernel name) and allow the user to select a target dispatch. Within that dispatch, we enumerate all unique workgroup coordinates and extract the trace subset belonging to a chosen reference WG for single-WG analysis. 3.1.3. Intra-Lane S… view at source ↗
Figure 3
Figure 3. Figure 3: Categorization of WG-level Data Structure Access. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Achievable kernel speedup when inter-partition [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: NUMA sensitivity of weight projection across [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: NUMA sensitivity of attention decode variants [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 7 linked inside Pith

  1. [1]

    AMD CDNA 3 Architecture: Powering the Next Generation of AI and HPC Accelerators,

    Advanced Micro Devices, Inc., “AMD CDNA 3 Architecture: Powering the Next Generation of AI and HPC Accelerators, ” AMD, Tech. Rep., 2023, documents 4 KiB page-granularity interleaving across HBM stacks: “switch stack every 4KiB through physical memory space” (partitioning diagram, p. 13). [Online]. Avail- able: https://www.amd.com/content/dam/amd/en/docum...

    2023

  2. [2]

    Omniprobe,

    ——, “Omniprobe, ” github.com/AMDResearch/omniprobe, 2025, gitHub repository

    2025

  3. [3]

    Using Origami and Stream-K with hipBLASLt,

    ——, “Using Origami and Stream-K with hipBLASLt, ” ROCm Documentation, hipBLASLt, 2025, describes the Origami with Stream-K kernel selection strategy for GEMM on AMD Instinct accelerators. [Online]. Avail- able: https://rocm.docs.amd.com/projects/hipBLASLt/en/develop/how- to/how-to-use-streamk.html

    2025

  4. [4]

    AMD MI300X GPU Performance Analysis,

    C. Ambati and T. Diep, “AMD MI300X GPU Performance Analysis, ”arXiv preprint arXiv:2510.27583, 2025. [Online]. Available: https://arxiv.org/abs/2510.27583

    arXiv 2025

  5. [5]

    Workgroups are automatically distributed across all XCDs (round-robin)

    AMD ROCm Software Team. (2024) AMD Instinct MI300X GPU Partitioning Overview. SPX mode: “Workgroups are automatically distributed across all XCDs (round-robin)”. [Online]. Available: https://instinct.docs.amd.com/projects/amdgpu- docs/en/latest/gpu-partitioning/mi300x/overview.html

    2024

  6. [6]

    MCM-GPU: Multi-Chip-Module GPUs for Continued Performance Scalability,

    A. Arunkumar, E. Bolotin, B. Cho, U. Milic, E. Ebrahimi, O. Villa, A. Jaleel, C.-J. Wu, and D. Nellans, “MCM-GPU: Multi-Chip-Module GPUs for Continued Performance Scalability, ” inProceedings of the 44th Annual International Symposium on Computer Architecture. Toronto ON Canada: ACM, Jun. 2017, pp. 320–332

    2017

  7. [7]

    Chasoň: Supporting cross- HBM channel data migration to enable efficient sparse algebraic acceleration,

    U. Bakhtiar, A. Namjoo, and B. Asgari, “Chasoň: Supporting cross- HBM channel data migration to enable efficient sparse algebraic acceleration, ” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2025, pp. 778–794

    2025

  8. [8]

    Gemini: Mapping and architecture co-exploration for large-scale DNN chiplet accelerators,

    J. Cai, Z. Wu, S. Peng, Y. Wei, Z. Tan, G. Shi, M. Gao, and K. Ma, “Gemini: Mapping and architecture co-exploration for large-scale DNN chiplet accelerators, ” in2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024, pp. 156–171

    2024

  9. [9]

    Optimizing attention on GPUs by exploiting GPU architectural NUMA effects,

    M. Choudhary, K. Sangaiah, S. Singh, M. Osama, L. W. Wills, and G. Dasika, “Optimizing attention on GPUs by exploiting GPU architectural NUMA effects, ”arXiv preprint arXiv:2511.02132, 2025

    arXiv 2025

  10. [10]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning, ” inInternational Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://arxiv.org/abs/2307.08691

    Pith/arXiv arXiv 2024

  11. [11]

    Flash-decoding for long- context inference,

    T. Dao, D. Haziza, F. Massa, and G. Sizov, “Flash-decoding for long- context inference, ” https://pytorch.org/blog/flash-decoding/, 2023, describes a two-phase attention kernel with parallel per-split com- putation followed by a reduction over splits

    2023

  12. [12]

    DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model,

    DeepSeek-AI, “DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model, ”arXiv preprint arXiv:2405.04434, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Barre Chord: Efficient Virtual Memory Translation for Multi-Chip-Module GPUs,

    Y. Feng, S. Na, H. Kim, and H. Jeon, “Barre Chord: Efficient Virtual Memory Translation for Multi-Chip-Module GPUs, ” in2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). Buenos Aires, Argentina: IEEE, Jun. 2024, pp. 834–847

    2024

  14. [14]

    Inter-kernel reuse-aware thread block scheduling,

    M. Huzaifa, J. Alsop, A. Mahmoud, G. Salvador, M. D. Sinclair, and S. V. Adve, “Inter-kernel reuse-aware thread block scheduling, ”ACM Transactions on Architecture and Code Optimization (TACO), vol. 17, no. 3, pp. 1–27, 2020

    2020

  15. [15]

    Microbenchmarking nvidia’s blackwell architecture: An in-depth architectural analysis,

    A. Jarmusch and S. Chandrasekaran, “Microbenchmarking nvidia’s blackwell architecture: An in-depth architectural analysis, ”arXiv preprint arXiv:2512.02189, 2025. [Online]. Available: https://arxiv.org/abs/2512.02189

    arXiv 2025

  16. [16]

    Coruscant: Co- designing GPU kernel and sparse tensor core to advocate unstruc- tured sparsity in efficient LLM inference,

    D. Joo, H. Hosseini, R. Hadidi, and B. Asgari, “Coruscant: Co- designing GPU kernel and sparse tensor core to advocate unstruc- tured sparsity in efficient LLM inference, ” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2025, pp. 232–245

    2025

  17. [17]

    Gemma 3 technical report,

    A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J.-b. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, and G. Liu, “Gemma 3 technical report, ”arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Locality-Centric Data and Threadblock Management for Massive GPUs,

    M. Khairy, V. Nikiforov, D. Nellans, and T. G. Rogers, “Locality-Centric Data and Threadblock Management for Massive GPUs, ” in2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). Athens, Greece: IEEE, Oct. 2020, pp. 1022–1036. [Online]. Available: https://ieeexplore.ieee.org/document/9251964/

    arXiv 2020

  19. [19]

    CODA: Enabling Co-location of Computation and Data for Multiple GPU Systems,

    H. Kim, R. Hadidi, L. Nai, H. Kim, N. Jayasena, Y. Eckert, O. Kayiran, and G. Loh, “CODA: Enabling Co-location of Computation and Data for Multiple GPU Systems, ”ACM Transactions on Architecture and Code Optimization, vol. 15, no. 3, pp. 1–23, Sep. 2018

    2018

  20. [20]

    Efficient memory management for large language model serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with PagedAttention, ” inProceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023

    2023

  21. [21]

    Locality-aware CTA clustering for modern GPUs,

    A. Li, S. L. Song, W. Liu, X. Liu, A. Kumar, and H. Corporaal, “Locality-aware CTA clustering for modern GPUs, ” inProceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM, 2017, pp. 297–311

    2017

  22. [22]

    The Llama 3 herd of models,

    Llama Team, AI@Meta, “The Llama 3 herd of models, ”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    The Next Era for Chiplet Innovation,

    G. H. Loh and R. Swaminathan, “The Next Era for Chiplet Innovation, ” in2023 Design, Automation & Test in Europe Conference & Exhibition (DATE). Antwerp, Belgium: IEEE, Apr. 2023, pp. 1–6

    2023

  24. [24]

    Beyond the socket: NUMA-aware GPUs,

    U. Milic, O. Villa, E. Bolotin, A. Arunkumar, E. Ebrahimi, A. Jaleel, A. Ramirez, and D. Nellans, “Beyond the socket: NUMA-aware GPUs, ” inProceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture. Cambridge Massachusetts: ACM, Oct. 2017, pp. 123–135

    2017

  25. [25]

    Communi- cation characterization of AI workloads for large-scale multi-chiplet accelerators,

    M. Musavi, E. Irabor, A. Das, E. Alarcón, and S. Abadal, “Communi- cation characterization of AI workloads for large-scale multi-chiplet accelerators, ” in2025 IEEE International Symposium on Circuits and Systems (ISCAS), 2025, pp. 1–5

    2025

  26. [26]

    Leveraging Chiplet- Locality for Efficient Memory Mapping in Multi-Chip Module GPUs,

    J. Park, S. Jang, O. Kwon, Y. Lee, and S. Hong, “Leveraging Chiplet- Locality for Efficient Memory Mapping in Multi-Chip Module GPUs, ” inProceedings of the 2025 58th IEEE/ACM International Symposium on Microarchitecture. Seoul Korea: ACM, Oct. 2025, pp. 1040–1057. 11

    2025

  27. [27]

    Efficient generative llm inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, Íñigo Goiri, S. Maleki, and R. Bianchini, “Efficient generative llm inference using phase splitting, ” inProceedings of the 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024. [Online]. Available: https://www.microsoft.com/en-us/research/publication/splitwise- efficient-generative-llm-i...

    2024

  28. [28]

    Heterogeneous system coherence for integrated CPU-GPU systems,

    J. Power, A. Basu, J. Gu, S. Puthoor, B. M. Beckmann, M. D. Hill, S. K. Reinhardt, and D. A. Wood, “Heterogeneous system coherence for integrated CPU-GPU systems, ” inProceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2013, pp. 457–467

    2013

  29. [29]

    Designing Virtual Memory System of MCM GPUs,

    B. Pratheek, N. Jawalkar, and A. Basu, “Designing Virtual Memory System of MCM GPUs, ” in2022 55th IEEE/ACM International Sympo- sium on Microarchitecture (MICRO). Chicago, IL, USA: IEEE, Oct. 2022, pp. 404–422

    2022

  30. [30]

    MemPod: A clustered architecture for efficient and scalable migration in flat address space multi-level memories,

    A. Prodromou, M. R. Meswani, N. Jayasena, G. H. Loh, and D. M. Tullsen, “MemPod: A clustered architecture for efficient and scalable migration in flat address space multi-level memories, ” in2017 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2017, pp. 433–444

    2017

  31. [31]

    Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference,

    P. G. Recasens, F. Agulló, Y. Zhu, C. Wang, E. K. Lee, O. Tardieu, J. Torres, and J. L. Berral, “Mind the memory gap: Unveiling gpu bottlenecks in large-batch llm inference, ”arXiv preprint arXiv:2503.08311, 2025. [Online]. Available: https://arxiv.org/abs/2503.08311

    Pith/arXiv arXiv 2025

  32. [32]

    Simba: Scaling deep-learning inference with multi-chip-module-based architecture,

    Y. S. Shao, J. Clemons, R. Venkatesan, B. Zimmer, M. Fojtik, N. Jiang, B. Keller, A. Klinefelter, N. Pinckney, P. Raina, S. G. Tell, Y. Zhang, W. J. Dally, J. S. Emer, C. T. Gray, S. W. Keckler, and B. Khailany, “Simba: Scaling deep-learning inference with multi-chip-module-based architecture, ” inProceedings of the 52nd IEEE/ACM International Symposium o...

    2019

  33. [33]

    Realizing the AMD Exascale Heterogeneous Processor Vision : Industry Product,

    A. Smith, G. H. Loh, M. J. Schulte, M. Ignatowski, S. Naffziger, M. Mantor, M. F. N. Kalyanasundharam, V. Alla, N. Malaya, J. L. Greathouse, E. Chapman, and R. Swaminathan, “Realizing the AMD Exascale Heterogeneous Processor Vision : Industry Product, ” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). Buenos Aires, Arg...

    2024

  34. [34]

    MGPUSim: enabling multi-GPU performance modeling and optimization,

    Y. Sun, T. Baruah, S. A. Mojumder, S. Dong, X. Gong, S. Treadway, Y. Bao, S. Hance, C. McCardwell, V. Zhao, H. Barclay, A. K. Ziabari, Z. Chen, R. Ubal, J. L. Abellán, J. Kim, A. Joshi, and D. Kaeli, “MGPUSim: enabling multi-GPU performance modeling and optimization, ” inProceedings of the 46th International Symposium on Computer Architecture. Phoenix Ari...

    arXiv 2019

  35. [35]

    tritonBLAS: Triton-based analytical approach for GEMM kernel parameter selection,

    R. Swann, M. Osama, X. Guo, B. Nelson, L. Zhang, A. Brown, Y. Ong, A. Yazdani, S. Siddens, G. Dasika, and A. Underwood, “tritonBLAS: Triton-based analytical approach for GEMM kernel parameter selection, ”arXiv preprint arXiv:2512.04226, 2024

    arXiv 2024

  36. [36]

    ACOPT: Adaptive continuity-aware address translation for performance optimization of MCM-GPU architectures,

    J. Tan, Z. Li, W. Wang, J. Wang, K. Yan, and X. Wei, “ACOPT: Adaptive continuity-aware address translation for performance optimization of MCM-GPU architectures, ”Future Generation Computer Systems, vol. 175, p. 108048, Feb. 2026

    2026

  37. [37]

    NN-Baton: DNN workload orchestration and chiplet granularity exploration for multichip accelerators,

    Z. Tan, H. Cai, R. Dong, and K. Ma, “NN-Baton: DNN workload orchestration and chiplet granularity exploration for multichip accelerators, ” in2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 2021, pp. 1013–1026

    2021

  38. [38]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev,...

    Pith/arXiv arXiv 2023

  39. [39]

    PAVER: Locality graph-based thread block scheduling for GPUs,

    D. Tripathy, A. Abdolrashidi, L. N. Bhuyan, L. Zhou, and D. Wong, “PAVER: Locality graph-based thread block scheduling for GPUs, ” ACM Transactions on Architecture and Code Optimization (TACO), vol. 18, no. 3, pp. 1–26, 2021

    2021

  40. [40]

    Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures,

    P. Vellaisamy, T. Labonte, S. Chakraborty, M. Turner, S. Sury, and J. P. Shen, “Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures, ” in2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). Ghent, Belgium: IEEE, May 2025, pp. 49–61

    2025

  41. [41]

    A Systematic Characterization of LLM Inference on GPUs,

    H. Wang, X. Xiao, M. Yan, Z. Zhu, D. Han, D. Wang, W. Li, X. Ye, C. Hu, H. Chen, and G. Sun, “A Systematic Characterization of LLM Inference on GPUs, ”arXiv preprint arXiv:2512.01644, Dec. 2025

    arXiv 2025

  42. [42]

    Exploring the Wafer-Scale GPUs,

    D. Xu, L. Xu, J. Ren, and Y. Sun, “Exploring the Wafer-Scale GPUs, ” inProceedings of the 17th Workshop on General Purpose Processing Using GPU. Las Vegas NV USA: ACM, Mar. 2025, pp. 8–13

    2025

  43. [43]

    Qwen3-omni technical report,

    J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin, “Qwen3-omni technical report, ”arXiv preprint arXiv:2509.17765, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    SGLang: Efficient execution of structured language model programs,

    L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng, “SGLang: Efficient execution of structured language model programs, ” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  45. [45]

    LRM-GPU: Alleviating synchronization overhead for multi-chiplet GPU architecture,

    B. Zhong, Z. Ye, X. Li, P. Wang, H. Huang, Z. Li, Z. Yu, and M. Wang, “LRM-GPU: Alleviating synchronization overhead for multi-chiplet GPU architecture, ” in2026 IEEE International Symposium on High- Performance Computer Architecture (HPCA), 2026. 12

    2026