Pith. sign in

REVIEW 3 major objections 5 minor 42 references

A hybrid scheduler for GPU-NDP systems speeds up MoE inference 2.41x on average by balancing experts across memory-side processors and the GPU.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 12:07 UTC pith:T2FKSMZ2

load-bearing objection Real scheduling ideas buried under a headline number that is not the measured 2.41x average end-to-end speedup claimed. the 3 major comments →

arxiv 2601.03992 v1 pith:T2FKSMZ2 submitted 2026-01-07 cs.DC cs.AI

A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems

Qi Wu , Chao Fang , Jiayuan Chen , Ye Lin , Yueqi Zhang , Yichuan Bai , Yuan Du , Li Du This is my paper
classification cs.DC cs.AI
keywords Mixture-of-Expertsnear-data processingedge inferencetensor parallelismload balancingexpert pre-fetchingGPU-NDP systemsMoE scheduling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Mixture-of-Experts (MoE) models keep edge AI affordable by activating only a few expert subnetworks per token, but their huge weight matrices outgrow GPU memory, pushing experts into near-data processing (NDP) memory units. The paper argues the real bottleneck in this setting is scheduling: expert selection is unbalanced across NDP units, the GPU idles while NDPs compute, and pre-fetching depends on offline calibration data. It proposes a three-part framework — tensor parallelism that splits every activated expert across all NDP units, a load-balancing equation that divides experts between NDPs and the GPU, and a pre-fetching rule that learns expert popularity from the prefill phase — and reports 2.41x average (up to 2.56x) end-to-end latency speedup over a state-of-the-art NDP baseline across four MoE models at batch size 1. If the simulation-based results hold on real hardware, the framework would make commodity GPU-plus-NDP-DIMM systems a practical edge deployment target for large sparse models.

Core claim

The paper's central claim is that tensor parallelism—assigning each activated expert's weight matrices in column/row partitions across all NDP-DIMMs—converts the irregular expert-parallel workload into a uniform one, so every NDP unit is kept busy regardless of routing. On top of this, two closed-form balance conditions select how many experts run on the GPU versus the NDPs: one for the decode stage (weight-transfer time plus GPU compute equals NDP compute plus activation transfer) and one for the prefill stage that adds a sequence-length term. A three-stage, dataset-free pre-fetching pipeline (prefill statistics → intermediate transfer → decode reuse) then hides weight-transfer latency with

What carries the argument

The load-balancing equations are the core mechanism: Equation (1) for decode and Equation (2) for prefill equate the time to transfer E_g experts' weights to the GPU plus the unhidden GPU compute against the NDP compute time plus activation transmission, yielding a per-layer count of GPU-resident experts that removes the need for a historical scaling factor. Tensor parallelism is implemented as a two-stage partition—column-wise for the first two GEMV layers, row-wise for the final linear layer—producing partial sums that are reduced on the GPU. Equation (3) caps the number of pre-fetched experts that can run on the GPU while still keeping NDPs busy. These formulas turn MoE scheduling into a

Load-bearing premise

The entire quantitative result rests on the fidelity of the simulation stack—the GPU simulator and the modified DRAM simulator for NDP-DIMM behavior—and on the balance equations' assumption that weight transmission and NDP computation overlap cleanly on real PCIe and NDP hardware.

What would settle it

Run the same four MoE models at batch size 1 on a physical GPU-NDP DIMM system (or a cycle-accurate simulator with measured PCIe contention and NDP compute throughput) and compare end-to-end latency against the state-of-the-art NDP baseline; if the overlap assumptions in Equations (1)-(3) fail, the 2.41x average speedup will not appear.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-batch edge inference no longer needs multi-batch amortization: tensor parallelism spreads the expert-weight read across all NDP units, so even one token can engage all of them.
  • Consumer GPUs paired with NDP-DIMMs can serve MoE models well beyond GPU VRAM (e.g., 30B-60B) without the per-request weight transfers that dominate today's offloading schemes.
  • Pre-fetching driven by the current prompt's prefill statistics can beat calibration-dataset pre-fetching, improving robustness to distribution shift in dynamic edge workloads.
  • The per-layer balance equations make scheduling adaptive to sequence length and TopK, enabling the same framework to shift allocation between prefill and decode automatically.
  • The reported 2.41x average speedup implies MoE latency on a fixed edge hardware budget can be cut by more than half relative to prior NDP-based schedulers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The balance-equation form suggests a direct extension to larger batches: adding a batch-size factor to the activation-transfer and compute terms would likely produce a family of scheduling policies for multi-request edge servers, though the paper only evaluates batch size 1.
  • If real NDP-DIMM compute throughput falls below the simulation assumption, the relative benefit of tensor parallelism over expert parallelism would shrink, because communication overhead grows relative to compute; a hardware prototype measuring PCIe contention is the natural test.
  • The pre-fetching mechanism's success depends on an implicit assumption that expert activation during prefill predicts activation during decode; conversations that switch topic mid-stream should degrade pre-fetch hit rate, a testable prediction.
  • A system-level implication the paper leaves implicit: the scheduler's workload is tiny (per-layer counts), so it could run on the host CPU at negligible overhead, enabling fine-grained, per-layer adaptation without separate profiling passes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a scheduling framework for MoE inference on an edge GPU-NDP-DIMM system, targeting single-batch decode-heavy workloads. Three optimizations are presented: (1) tensor parallelism across NDP units for expert computations, (2) a load-balancing condition that allocates some activated experts to the GPU while others run on NDPs, and (3) a dataset-free pre-fetching strategy that learns expert activation patterns from the prefill stage and uses them to preload experts for the decode stage. The framework is evaluated in simulation using AttAcc! and a modified Ramulator 2.0 on four MoE models against three baselines (MoE-OnDemand, Fiddler, MoNDE). The reported gains are up to 5.49x over MoE-OnDemand, 4.05x over Fiddler, and up to 2.56x over MoNDE in end-to-end latency, with an average 2.41x claimed for the full framework.

Significance. If the results hold, the framework makes a useful contribution to edge inference of MoE models, where weight movement and load imbalance are known bottlenecks. The tensor-parallelism idea for multi-NDP MoE scheduling is well motivated and, to my knowledge, not explored in the cited prior systems. The dataset-free pre-fetching is an improvement over calibration-based methods, and the paper evaluates it as a prediction from prefill to decode rather than as a fit to the final speedup, which is a positive sign. The ablation study across four models and several NDP counts gives useful evidence about the contribution of each component. However, the paper in its current form overstates the headline result: the 2.41x average is reported for the MoE decode stage only, while the abstract and conclusion present it as an end-to-end speedup. The performance model underlying load balancing is also asserted without derivation, and the simulation results are point estimates without error bars or released artifacts.

major comments (3)
  1. [Abstract; Section IV-C; Section IV-D; Fig. 11] The central quantitative claim is overstated. The abstract and conclusion state that the framework achieves 2.41x on average and up to 2.56x speedup in end-to-end latency. However, the 2.41x average is reported in Section IV-D for the MoE decode stage only, as shown in Fig. 11 whose title explicitly says 'impact on MoE computation in the decode stage.' Section IV-C, which reports end-to-end latency in Fig. 9, gives only a maximum speedup of 2.56x over MoNDE and does not report an average. Since the prefill MoE speedup is only about 1.36x and non-MoE computation (e.g., attention) is included in end-to-end latency, the average end-to-end speedup over MoNDE is expected to be lower than 2.41x. The paper never computes the quantity claimed in the abstract. This is not a simulator-fidelity issue: even with perfect simulators, the headline number as stated is unsupported. Please compute and rep
  2. [III-C, Eqs. (1)-(3)] The load-balancing model is load-bearing for the paper's central contribution, but Equations (1), (2), and (3) are asserted without derivation. In particular, E_g' is defined as 'one N-th of the fractional part of E_g' with a nominal 1/N when E_g is an integer multiple of N. This term is ad hoc and not justified by any latency or overlap analysis. The balance condition determines which experts are assigned to the GPU and is therefore critical to the reported speedups. If the model is inaccurate, the load-balancing and prefetching benefits could change substantially. Please provide a derivation of Eqs. (1)-(3) from the underlying compute/transfer timeline, or an empirical validation of the predicted balance point within the simulator, and add sensitivity analysis over the key parameters (e.g., PCIe bandwidth, NDP throughput, activation sizes) to show the conclusions are robust.
  3. [IV-A, IV-D] The entire quantitative evaluation is simulation-based, using AttAcc! for the GPU and a modified Ramulator 2.0 for NDP-DIMMs. No error bars, confidence intervals, or run-to-run variability are reported, and no artifacts are released. The paper's point estimates (e.g., 1.36x, 1.69x, 2.41x) are presented as exact values, making it difficult to assess their reliability. Since the speedup numbers are the main evidence for the framework's claims, please include variability measures (e.g., across input prompts and simulator seeds), and consider a sensitivity study of the modified Ramulator 2.0 parameters, especially the NDP compute throughput and PCIe contention. Releasing the simulation configuration would also help reproducibility.
minor comments (5)
  1. [IV-B] The text says the results demonstrate 'an average speedup of 1.36x in the prefill stage and 1.69x in the decode stage' but Fig. 8 appears to show results for Qwen3-30B-A3B only. Please clarify whether the average is over the number of NDP-DIMMs or over models, and make the caption consistent.
  2. [IV-D] In the sentence 'average speedups of 2.19x and 2.23x are achieved with NDP+TP+LD and NDP+TP+PRE', the abbreviation 'LD' should be 'LB' for consistency with the figure and the rest of the text.
  3. [Fig. 6] The caption and labels in Fig. 6 are hard to parse. It is not immediately clear which bars correspond to 'prefill-stage data' and which to 'dataset data', and the text says the prefetch utilization is 'even higher' with prefill data while the numbers in the figure appear to show the opposite. Please clarify the color coding, the definition of 'Proportion of Experts Capable of Using Pre-fetched Weights', and the direction of the comparison.
  4. [Table II] The table uses footnotes (triangledelta and asterisk) that do not render in the text; the definitions '6/2' and '27/28' should be explained directly in the table or caption.
  5. [Abstract] The phrase 'compared to state-of-the-art approaches' is ambiguous. The paper compares against MoE-OnDemand, Fiddler, and MoNDE, and the maximum speedup over MoNDE is 2.56x while maximum speedups over the other baselines are different. Please specify which baseline each headline number refers to.

Circularity Check

0 steps flagged

No significant circularity: the scheduling equations are design heuristics and the reported speedups are empirical simulation outcomes, not derived from the paper's own inputs.

full rationale

The paper's derivation chain is not circular. The load-balance conditions (Eqs. 1-3) are design heuristics: they determine the number of GPU-assigned experts from hardware timing parameters (Tw, Tg, Tn, Ta) and are used as a scheduling policy; the reported speedups are measured with AttAcc! and a modified Ramulator 2.0 by comparing the full policy against baselines, not calculated from these equations. The dataset-free pre-fetching strategy is an empirical predictor: prefill activation frequencies are used to select experts, and the utilization and speedup in the decode stage are measured as external outcomes rather than fitted. No fitted parameter is renamed as a prediction, and no calibration dataset is used to fit the prefetch set to the decode outputs. The only same-author citation (“C. Fang et al., Anda”, [7]) appears in a list of dense-model references and is not load-bearing. The observation that the abstract's “2.41x average” actually corresponds to the decode MoE-stage ablation (Fig. 11) rather than average end-to-end latency (Fig. 9) is a reporting-precision issue, not circularity; similarly, simulator fidelity is an external-validity concern, not a self-referential derivation.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

No new physical entities or fitted empirical constants are introduced. The framework is a scheduling heuristic over existing hardware; the load-bearing assumptions are simulator fidelity, prefill-to-decode expert predictability, and the concurrency model behind the balance equations.

free parameters (1)
  • E_g' (unhidden GPU expert computation fraction) = fractional part of E_g divided by N; nominal 1/N when E_g is a multiple of N
    Introduced 'for analytical tractability' in Section III-C to close the load-balance equations; it is a hand-chosen modeling term, not derived from measurements or independent evidence.
axioms (4)
  • domain assumption Expert activation patterns observed during prefill are representative of decode-stage activations for the same prompt.
    Section III-D motivates dataset-free pre-fetching; Fig. 6 provides evidence on only 20 GSM8K samples, so the transferability is assumed beyond that.
  • domain assumption The AttAcc!/Ramulator 2.0 simulation stack faithfully models GPU and NDP-DIMM performance, including PCIe and memory timing.
    Section IV-A: all latency numbers come from simulators; accuracy is inherited from [40] rather than demonstrated for this configuration.
  • domain assumption Weight transfers and computation overlap perfectly except for the final unhidden transmission.
    Section III-C: Equations (1)-(2) balance Tw*E_g + T_g*E_g' against NDP time under this concurrency model.
  • domain assumption The NDP-DIMM hardware parameters (64 multipliers/DIMM, 102.4 GB/s internal bandwidth, DDR4 3200) are representative of real commodity NDP-DIMMs.
    Table I defines the only hardware configuration evaluated; speedups are not shown to be robust to weaker NDP compute throughput.

pith-pipeline@v1.3.0-alltime-deepseek · 29319 in / 13806 out tokens · 129634 ms · 2026-08-03T12:07:44.988642+00:00 · methodology

0 comments
read the original abstract

Mixture-of-Experts (MoE) models facilitate edge deployment by decoupling model capacity from active computation, yet their large memory footprint drives the need for GPU systems with near-data processing (NDP) capabilities that offload experts to dedicated processing units. However, deploying MoE models on such edge-based GPU-NDP systems faces three critical challenges: 1) severe load imbalance across NDP units due to non-uniform expert selection and expert parallelism, 2) insufficient GPU utilization during expert computation within NDP units, and 3) extensive data pre-profiling necessitated by unpredictable expert activation patterns for pre-fetching. To address these challenges, this paper proposes an efficient inference framework featuring three key optimizations. First, the underexplored tensor parallelism in MoE inference is exploited to partition and compute large expert parameters across multiple NDP units simultaneously towards edge low-batch scenarios. Second, a load-balancing-aware scheduling algorithm distributes expert computations across NDP units and GPU to maximize resource utilization. Third, a dataset-free pre-fetching strategy proactively loads frequently accessed experts to minimize activation delays. Experimental results show that our framework enables GPU-NDP systems to achieve 2.41x on average and up to 2.56x speedup in end-to-end latency compared to state-of-the-art approaches, significantly enhancing MoE inference efficiency in resource-constrained environments.

Figures

Figures reproduced from arXiv: 2601.03992 by Chao Fang, Jiayuan Chen, Li Du, Qi Wu, Ye Lin, Yichuan Bai, Yuan Du, Yueqi Zhang.

Figure 1
Figure 1. Figure 1: The rising memory gap between consumer-grade GPU VRAM and large MoE models motivates GPU-NDP systems as a promising cost-effective edge solution. Our scheduling framework enables efficient MoE inference on such systems via tensor parallelism, load balancing, and dataset-free pre-fetching. To address this challenge of memory capacity, current re￾search follows mainly three different technical paths. Two of … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of GPU-NDP DIMM System. models, such as Qwen3-30B-A3B [10] which exceeds 60GB, significantly impedes edge deployment under tight memory and I/O budgets. Empirical evidence reveals that expert activation follows highly uneven patterns: only a small fraction of experts are frequently activated while the majority remain idle for most tokens. Prior systems [12], [15] exploit this sparsity through expe… view at source ↗
Figure 4
Figure 4. Figure 4: The computing process at the MoE layer with our introduced tensor [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Utilization of prefetched weights in Qwen3-30B-A3B [10] is compared between analyses derived from the full calibration datasets (GSM8K, MMLU) and from dataset-free prefill-stage data. The comparison is conducted using 20 randomly sampled GSM8K inputs. among activated experts and oversimplify balance conditions by only equating NDP computation time with weight transmission time, neglecting differences in th… view at source ↗
Figure 8
Figure 8. Figure 8: The normalized MoE latency of the prefill and the decode stage in [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: End-to-end model performance with varying number of NDP-DIMMs [PITH_FULL_IMAGE:figures/full_fig_p005_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The impact of tensor parallelism and load balancing on MoE [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 7 linked inside Pith

  1. [1]

    Attention is all you need,

    A. Vaswaniet al., “Attention is all you need,”Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017

    2017

  2. [2]

    Adaptive mixtures of local experts,

    R. A. Jacobset al., “Adaptive mixtures of local experts,”Neural compu- tation, vol. 3, no. 1, pp. 79–87, 1991

    1991

  3. [3]

    Outrageously large neural networks: The sparsely- gated mixture-of-experts layer,

    N. Shazeeret al., “Outrageously large neural networks: The sparsely- gated mixture-of-experts layer,” inInternational Conference on Learning Representations (ICLR), 2017

    2017

  4. [4]

    Gpt-4 technical report,

    J. Achiamet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Llama: Open and efficient foundation language models,

    H. Touvronet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Opt: Open pre-trained transformer language models,

    S. Zhanget al., “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,

    C. Fanget al., “Anda: Unlocking efficient llm inference with a variable- length grouped activation data format,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1467–1481

    2025

  8. [8]

    A survey on mixture of experts in large language models,

    W. Caiet al., “A survey on mixture of experts in large language models,” IEEE Transactions on Knowledge and Data Engineering (TKDE), 2025

    2025

  9. [9]

    (2025) GeForce RTX 5080

    NVIDIA Corporation. (2025) GeForce RTX 5080. [Online]. Available: https://www.nvidia.com/zh-cn/geforce/graphics-cards/50-series/rtx-5080/

    2025

  10. [10]

    Qwen3 technical report,

    A. Yanget al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,

    R. Y . Aminabadiet al., “Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale,” inInternational Confer- ence for High Performance Computing, Networking, Storage and Analysis (SC), 2022, pp. 1–15

    2022

  12. [12]

    Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,

    Z. Fanget al., “Klotski: Efficient mixture-of-expert inference via expert- aware multi-batch pipeline,” inProceedings of the 30th ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Volume 2, 2025, pp. 574–588

    2025

  13. [13]

    Fast inference of mixture-of-experts language models with offloading,

    A. Eliseevet al., “Fast inference of mixture-of-experts language models with offloading,”arXiv preprint arXiv:2312.17238, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    DAOP: Data-aware offloading and predictive pre- calculation for efficient moe inference,

    Y . Zhanget al., “DAOP: Data-aware offloading and predictive pre- calculation for efficient moe inference,” inDesign, Automation & Test in Europe Conference (DATE), 2025, pp. 1–7

    2025

  15. [15]

    Moe-lightning: High-throughput moe inference on memory-constrained gpus,

    S. Caoet al., “Moe-lightning: High-throughput moe inference on memory-constrained gpus,” inProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Volume 1, 2025, pp. 715–730

    2025

  16. [16]

    Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models,

    K. Kamahoriet al., “Fiddler: CPU-GPU orchestration for fast inference of mixture-of-experts models,” inInternational Conference on Learning Representation (ICLR), 2025

    2025

  17. [17]

    Monde: Mixture of near-data experts for large-scale sparse models,

    T. Kimet al., “Monde: Mixture of near-data experts for large-scale sparse models,” inProceedings of the 61st ACM/IEEE Design Automation Conference (DAC), 2024, pp. 1–6

    2024

  18. [18]

    Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,

    S. Yunet al., “Duplex: A device for large language models with mixture of experts, grouped query attention, and continuous batching,” in57th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2024, pp. 1429–1443

    2024

  19. [19]

    Co-designing binarized transformer and hardware accel- erator for efficient end-to-end edge deployment,

    Y . Jiet al., “Co-designing binarized transformer and hardware accel- erator for efficient end-to-end edge deployment,” inProceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2024, pp. 1–9

    2024

  20. [20]

    Language models at the edge: A survey on techniques, challenges, and applications,

    S. Hadishet al., “Language models at the edge: A survey on techniques, challenges, and applications,” in2024 2nd International Conference on Foundation and Large Language Models (FLLM). IEEE, 2024, pp. 262– 271

    2024

  21. [21]

    A precision-scalable risc-v dnn processor with on-device learning capability at the extreme edge,

    L. Huanget al., “A precision-scalable risc-v dnn processor with on-device learning capability at the extreme edge,” in29th Asia and South Pacific Design Automation Conference (ASP-DAC), 2024, pp. 927–932

    2024

  22. [22]

    Spark: Scalable and precision-aware acceleration of neural networks via efficient encoding,

    F. Liuet al., “Spark: Scalable and precision-aware acceleration of neural networks via efficient encoding,” in2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024, pp. 1029–1042

    2024

  23. [23]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads,

    T. Caiet al., “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” inInternational Conference on Machine Learning (ICML), 2024, pp. 5209–5235

    2024

  24. [24]

    Recnmp: Accelerating personalized recommendation with near-memory processing,

    L. Keet al., “Recnmp: Accelerating personalized recommendation with near-memory processing,” inACM/IEEE 47th Annual International Sym- posium on Computer Architecture (ISCA). IEEE, 2020, pp. 790–803

    2020

  25. [25]

    Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning,

    Y . Kwonet al., “Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2019, pp. 740–753

    2019

  26. [26]

    Attacc! unleashing the power of pim for batched transformer-based generative model inference,

    J. Parket al., “Attacc! unleashing the power of pim for batched transformer-based generative model inference,” inProceedings of the 29th ACM International Conference on Architectural Support for Program- ming Languages and Operating Systems (ASPLOS), Volume 2, 2024, pp. 103–119

    2024

  27. [27]

    PIMoE: Towards efficient moe transformer deployment on npu-pim system through throttle-aware task offloading,

    L. Wuet al., “PIMoE: Towards efficient moe transformer deployment on npu-pim system through throttle-aware task offloading,” inProceedings of the 62st ACM/IEEE Design Automation Conference (DAC), 2025

    2025

  28. [28]

    Make llm inference affordable to everyone: Augmenting gpu memory with ndp-dimm,

    L. Liuet al., “Make llm inference affordable to everyone: Augmenting gpu memory with ndp-dimm,” inIEEE International Symposium on High Performance Computer Architecture (HPCA), 2025, pp. 1751–1765

    2025

  29. [29]

    The true processing in memory accelerator,

    F. Devaux, “The true processing in memory accelerator,” in2019 IEEE Hot Chips 31 Symposium (HCS). IEEE Computer Society, 2019, pp. 1–24

    2019

  30. [30]

    LP-Spec: Leveraging lpddr pim for efficient llm mobile speculative inference with architecture-dataflow co-optimization,

    S. Heet al., “LP-Spec: Leveraging lpddr pim for efficient llm mobile speculative inference with architecture-dataflow co-optimization,” inPro- ceedings of the 44rd IEEE/ACM International Conference on Computer- Aided Design (ICCAD), 2025

    2025

  31. [31]

    Ndpage: Efficient address translation for near-data pro- cessing architectures via tailored page table,

    Q. Jianget al., “Ndpage: Efficient address translation for near-data pro- cessing architectures via tailored page table,” in2025 Design, Automation & Test in Europe Conference (DATE), 2025, pp. 1–7

    2025

  32. [32]

    Hyqa: Hybrid near-data processing platform for embed- ding based question answering system,

    S. Lianget al., “Hyqa: Hybrid near-data processing platform for embed- ding based question answering system,” in2024 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2024, pp. 1–6

    2024

  33. [33]

    Near-memory parallel indexing and coalescing: Enabling highly efficient indirect access for spmv,

    C. Zhanget al., “Near-memory parallel indexing and coalescing: Enabling highly efficient indirect access for spmv,” in2024 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2024, pp. 1–6

    2024

  34. [34]

    Um-pim: Dram-based pim with uniform & shared memory space,

    Y . Zhaoet al., “Um-pim: Dram-based pim with uniform & shared memory space,” inACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 2024, pp. 644–659

    2024

  35. [35]

    Bramac: Compute-in-bram architectures for multiply- accumulate on fpgas,

    Y . Chenet al., “Bramac: Compute-in-bram architectures for multiply- accumulate on fpgas,” inIEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2023, pp. 52–62

    2023

  36. [36]

    An overview of processing-in-memory circuits for artificial intelligence and machine learning,

    D. Kimet al., “An overview of processing-in-memory circuits for artificial intelligence and machine learning,”IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS), vol. 12, no. 2, pp. 338–353, 2022

    2022

  37. [37]

    Mixtral of experts,

    A. Q. Jianget al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Aim: accelerating computational genomics through scalable and noninvasive accelerator-interposed memory,

    J. Conget al., “Aim: accelerating computational genomics through scalable and noninvasive accelerator-interposed memory,” inProceedings of the International Symposium on Memory Systems (MEMSYS), 2017, pp. 3–14

    2017

  39. [39]

    (2025) intel-core-i7-14700

    Intel Corporation. (2025) intel-core-i7-14700. [Online]. Available: https://www.intel.cn/content/www/cn/zh/products/sku/ 236781/intel-core-i7-processor-14700-33m-cache-up-to-5-40-ghz/ specifications.html

    2025

  40. [40]

    Ramulator 2.0: A modern, modular, and extensible dram simulator,

    H. Luoet al., “Ramulator 2.0: A modern, modular, and extensible dram simulator,”IEEE Computer Architecture Letters (RAL), vol. 23, no. 1, pp. 112–116, 2023

    2023

  41. [41]

    DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models,

    D. Daiet al., “DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 1280–1297

    2024

  42. [42]

    Phi-3 safety post-training: Aligning language models with a ‘break-fix’ cycle,

    E. Haideret al., “Phi-3 safety post-training: Aligning language models with a ‘break-fix’ cycle,”arXiv preprint arXiv:2407.13833, 2024

    Pith/arXiv arXiv 2024