Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two Mac Studios can serve a 132-billion-parameter mixture-of-experts model at 5.9 tokens per second, claiming 1.15x the throughput per dollar of an 8xH100 NVIDIA server.

desk verdict Solid engineering with a real finding about Apple's driver processing, but the 1.15x cost-efficiency headline rests on a price mismatch and an informal baseline—fixable before publication. read the letter →

arxiv 2506.23635 v1 pith:N324HERG submitted 2025-06-30 cs.DC cs.AIcs.PF

Mu-Chi Chen , Po-Hsuan Huang , Xiangrui Ke , Chia-Heng Tu , Chun Jason Xue , Shih-Hao Hung This is my paper
classification cs.DCcs.AIcs.PF
keywords largelanguagemodelmixture-of-expertsexpertparallelismmulti-nodeinferenceAppleSiliconcostefficiencyloadbalancingperformancemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a private, small-group LLM service can be built without a datacenter GPU budget. The authors run the unquantized 132-billion-parameter mixture-of-experts model DBRX on a cluster of two Apple Mac Studio workstations with M2 Ultra chips, and report 5.9 tokens/sec of generation on a 2000-input/256-output single-user workload, which they compute is 1.15 times more cost-efficient than Databricks' 8xH100 NVIDIA server measured on the same workload. The paper's insight is that the obstacles are not compute but Apple's memory-management overhead and network latency, and it shows three optimizations, weight prestacking, router-aided warmup loading, and decentralized attention/router, lift generation throughput from 1.2 to 6.1 tokens/sec. A performance model then lets a designer predict throughput for different node counts and network cards.

What carries the argument

The design that carries the argument is expert parallelism organized as a fork-join across nodes, with the 16 experts split evenly so the router's selected experts run concurrently and a weighted-sum all-reduce merges their outputs. Named in the paper, the optimized scheme is P-LR-D: expert-wise weight prestacking (P), multi-node compute load balancing (LB for busy full loading, LR for router-aided dynamic LRU loading), and decentralized self-attention/router with an envoy process (D). The central identity that predicts performance is Equation (1), which states the per-token lower bound as the sum of the max of GPU load time and GPU compute time per node plus communication latency times layers plus data transfer time; the measured variables are the mean number of experts executed per node per layer, communication latency (1 ms), and the sizes of weights and intermediate data. This equation both explains why latency, not bandwidth, dominates and lets the authors extrapolate to 3, 4, 6, and 8 nodes and to InfiniBand and RoCEv2 networks.

What would settle it

Look up the list price of the exact tested Mac Studio node (M2 Ultra, 76-core GPU, 192 GB) and divide the measured 5.9 tokens/sec by that price times two nodes; do the same for the 8xH100 system at $289,000 with 112.5 tokens/sec. If the Apple ratio is not above 0.000389 tokens/sec/USD, the claimed 1.15x cost-efficiency advantage does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that expert parallelism on a small Apple Silicon cluster makes private serving of a full-size MoE LLM practical. Distributing the 16 experts of the unquantized DBRX Instruct model across two Mac Studios, each with an M2 Ultra and 192 GB of unified memory, cuts per-layer expert compute time, and the whole optimized pipeline (P-LR-D) reaches 6.1 tokens/sec during token generation (5.9 tokens/sec on the same 2000-input/256-output workload Databricks used). The paper finds two bottlenecks that dominate: Apple's on-demand memory-management 'driver processing' overhead, which repeatedly re-wires expert weights as layers run, and the per-message latency of the 10 GbE network, which matters more than bandwidth because each all-reduce transfers only about 24,576 bytes. Its optimizations, prestacking each expert's 40 layers of weights into one tensor, keeping unselected experts warm with LRU-chosen spare computation, and replicating self-attention and the router on every node so only one all-reduce per layer remains, together produce the 5.9-6.1 tokens/sec figure. The paper also derives a lower-bound performance model, max of GPU load time and compute time plus communication latency and transfer time, which matches the measured two-to-four-node data and projects scaling to eight nodes.

Load-bearing premise

Cost parity rests on the assumption that the list prices reflect the hardware actually benchmarked: the $6,599 per-node Apple price points to a 64 GB / 60-core Mac Studio, while the tested node has 192 GB and a 76-core GPU, and the Databricks H100 throughput and $289,000 price come from a GitHub comment and an archived store page rather than a same-workload benchmark.

Editorial extensions

If this is right

  • A two-node Mac Studio cluster with the P-LR-D optimizations serves the unquantized DBRX Instruct model at 5.9 tokens/sec on Databricks' 2000-input/256-output workload, giving 0.000447 tokens/sec per USD versus 0.000389 for the 8xH100 system.
  • Scaling from two to four nodes raises generation throughput only from 6.1 to 7.0 tokens/sec, and communication grows from 23% to 33% of per-token time, so network latency, not bandwidth, caps scalability.
  • Swapping 10 GbE for RoCEv2 or InfiniBand NICs is projected to raise the two-node throughput bound from 9.7 to 16.3 tokens/sec for a 5-20% cost increase, improving cost efficiency.
  • The performance model, validated against two-to-four-node measurements, estimates bounds of 9.7 to 14.2 tokens/sec for clusters of two to eight Mac Studios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Apple driver-processing behavior is a general property of macOS unified-memory management, then prestacking and LRU-touch warmup should transfer to other large MoE models and other unified-memory Apple chips, not just DBRX on M2 Ultra, a testable extension the paper does not run.
  • The cost comparison would shrink if priced at the tested configuration: the $6,599 list price cited corresponds to a 64 GB / 60-core Mac Studio, not the 192 GB / 76-core node in Table 2, so a reader should recompute the 1.15x ratio with the actual current price of the tested SKU.
  • The router-aided LRU loading is effectively a software cache policy for GPU-visible weights; the same policy could be applied to dense models that are too large for one node, where layer weights must be streamed.
  • Because the throughput target is a single user, the 5.9 tokens/sec figure does not directly answer multi-user or batch serving; an extension would measure how the LRU warmup and decentralized design behave under concurrent requests.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents an engineering study of running the unquantized DBRX 132B Mixture-of-Experts model on a small cluster of Apple Mac Studio M2 Ultra machines connected by 10 GbE. The authors measure the performance of a naive expert-parallel implementation, identify a driver-side memory-management overhead in the MLX/Metal stack, and propose three optimizations (expert-wise weight prestacking, router-aided load balancing, and decentralized self-attention/router with an envoy process). The optimized two-node system reaches 6.1 tokens/sec on a 128-token workload and 5.9 tokens/sec on a 2000-input/256-output workload, and the paper reports a 1.15x throughput-per-dollar advantage over a Databricks 8xH100 TensorRT-LLM system. A performance model is introduced to estimate bounds for 2-8 nodes and for RDMA networks.

Significance. If the measured speedups and the cost comparison held, the paper would demonstrate that a low-cost private LLM inference appliance is feasible for the DBRX class of models, and the characterization of driver processing overhead would be useful to the MLX/Metal systems community. The strengths of the paper are the direct per-token breakdowns in Tables 3-4, the carefully constructed microbenchmark in Section 3.2 (Algorithms 1-2, Figure 4), and the explicit identification of the network-latency bottleneck. The main weakness is that the headline cost-efficiency ratio rests on a price anchor that does not match the tested hardware and on an informal baseline; the performance model additionally overstates its validation. The paper is thus rich in engineering content but its central quantitative claim needs correction.

major comments (3)
  1. [Section 5.4, Table 5, footnote 6] The price anchor for the Apple node is inconsistent with the testbed. Table 2 specifies an M2 Ultra with 76 GPU cores and 192 GB unified memory, but footnote 6 cites an Apple Store URL for a 60-core GPU, 64 GB configuration. The 64 GB SKU cannot host the two-node DBRX layout of Section 3.2, which requires 384 GB total. The $6,599 per-node price is therefore not the price of the evaluated system. Recomputing Table 5 with a conservative list price for the tested configuration is necessary; if the node costs $7,600 or more, the 1.15x ratio falls below parity with the Databricks baseline. The abstract and contribution (4) must be revised accordingly.
  2. [Section 5.4, footnote 5] The Databricks baseline is not reproducible from the cited sources. The 112.5 tokens/sec figure is taken from a GitHub-issue comment, and the $289,000 system price comes from a web-archived vendor page. The manuscript does not report the baseline's batch size, input/output lengths, precision, TensorRT-LLM version, or GPU clocks, and it does not establish that the 'same workload' claim is accurate. For a comparison that drives the paper's headline result, the authors should either run the baseline themselves under the stated workload or clearly report the exact conditions and cite a primary source that includes them.
  3. [Section 5.5, Figure 8, Table 6] The claim that the performance model is validated by the 2-4 node measurements is not supported by the data. The model's predicted throughputs (9.7, 10.4, and 12.3 tokens/sec for 2, 3, and 4 nodes) exceed the realized values (6.1, 6.5, and 7.0 tokens/sec) by 59-76%. Moreover, the model's inputs E[#exec. experts/node/layer] and comm. latency are measured on the same 2-4 node systems, so the comparison is not an independent validation. The model is better described as a loose upper bound; the paper should avoid 'validating' and should temper the extrapolations to 6 and 8 nodes.
minor comments (6)
  1. [Abstract and Section 5.4] The abstract and contribution (4) state that the system achieves 6.1 tokens/sec and is 1.15x more cost-efficient, but the 1.15x ratio in Table 5 uses 5.9 tokens/sec (the 2000-input/256-output workload) rather than the 6.1 tokens/sec figure (the 128-token workload). Please clarify which workload each number refers to.
  2. [Section 4.2] For router-aided dynamic loading, the text says spare computation goes to LRU experts but does not explicitly state that these experts' outputs are zero-weighted in the weighted sum. While the router probability for unselected experts is zero, the masking should be stated explicitly to avoid ambiguity about whether the model output is unchanged.
  3. [Table 1] The measured variable 'comm. latency' (1e-3 sec) is not accompanied by a description of how it was measured. Please specify the measurement method (e.g., ping/round-trip of a payload of the relevant size).
  4. [Figure 4] The y-axis labels (102, 103, 104) and the plotted data below 100 ms are inconsistent unless the axis is logarithmic with a lower bound; please add an explicit axis scale and tick labels.
  5. [Section 5.4] Calling the 8xH100 system a 'state-of-the-art AI supercomputer' is an overstatement; it is a single server. Adjust the wording to avoid overselling the baseline.
  6. [Section 5.3] Adding nodes increases throughput but also decreases throughput per dollar (from about 0.000447 at two nodes to about 0.000265 at four nodes at the nominal price). A brief discussion of this trade-off would strengthen the cost-efficiency analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline throughput and cost-efficiency results are direct measurements or external comparisons, and the performance model is not fitted to the values it predicts.

full rationale

The paper's central results are empirical measurements of a real Mac Studio cluster running the unquantized DBRX model, not quantities derived from the claims themselves. The token-generation throughputs in Tables 3 and 4 (1.2 to 7.0 tokens/sec) come from profiled runs, and the 1.15x cost-efficiency ratio in Table 5 divides a measured Mac throughput (5.9 tokens/sec) by an externally reported Databricks baseline (112.5 tokens/sec). No fitted parameter is renamed as a prediction: the performance model in Equation (1) uses independently stated hardware parameters (memory bandwidth, GPU BF16 FLOPS, #layers), model parameter counts, and measured workload statistics such as E[#exec. experts/node/layer] and comm. latency. Crucially, the model's outputs are upper-bound estimates (Table 6: 9.7, 10.4, 12.3 tokens/sec) that are not equal to the realized measurements (6.1, 6.5, 7.0 tokens/sec), so the model is not forced to reproduce its inputs by construction. The in-sample validation against the two-to-four-node configurations is a legitimate consistency check rather than a circular derivation, since the predicted values do not reduce to the measured throughputs. The only self-reference is the footnote identifying this as a revised Best Paper Award version, which is not load-bearing. Concerns about the Mac Studio list price not matching the tested 192 GB/76-core configuration and about the informal GitHub-issue baseline are correctness and reproducibility risks, not circularity. Therefore, no circular step meeting the evidence standard is present.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The model's predictions depend on measured router statistics and a fixed latency assumption, while the entire measurement stack assumes the MLX implementation is correct. The introduced entity (envoy) is a software component with no independent artifact. Weight and hardware specifications are inputs from prior art/vendor specs, not counted as free parameters.

free parameters (2)
  • E[#exec. experts/node/layer] = 2.65 (2 nodes), 2.32 (3 nodes), 1.57 (4 nodes)
    Measured router expert-selection statistics from the 128-token workload (Table 1); the performance model (Eq. 1) uses them as inputs and is then validated against throughput measurements from the same configurations, so the 2 to 4 node predictions are calibrated, not independent.
  • comm. latency = 1e-3 sec
    Measured per-layer communication latency (Table 1) used in Eq. 1b; it is held constant for 2 to 8 nodes in Table 6, assuming added nodes do not increase all-reduce latency, which is an unverified modeling choice.
assumptions (5)
  • standard math An MoE decoder layer decomposes into independent expert computations followed by a weighted-sum all-reduce
    The distributed design rests on this decomposition (Sections 2.2, 3.1) and correctly cites GShard as prior art.
  • domain assumption The MLX-based DBRX implementation faithfully reproduces the pretrained DBRX Instruct model's outputs
    All throughput measurements assume the distributed system computes the same function as the reference implementation; no output-quality or correctness check is reported (Sections 4.2-4.3, 5.2).
  • domain assumption The synthetic benchmark (Algorithms 1-2) with 40 layers of 8192 by 8192 matrix multiplications emulates the compute and memory behavior of a real DBRX expert
    The driver processing findings in Section 3.2 are established with this proxy workload, then applied to DBRX; the transfer is asserted, not measured.
  • domain assumption GPU weight-loading time and compute time overlap, so Eq. 1a takes their maximum
    Stated in Section 4.4 ('as these two can overlap'); if load and compute serialize, the model's lower bound would understate time.
  • ad hoc to paper Communication latency per layer is independent of the number of nodes in the cluster
    Table 6 keeps Lat. = 0.040 sec for 2, 3, 4, 6, and 8 nodes; an all-reduce over more participants normally adds latency, so this assumption is specific to the paper's model.
invented entities (1)
  • envoy process
    purpose: An isolated asynchronous gRPC server on each node that dispatches intermediate data during the all-reduce phase, so network traffic does not disturb GPU compute (Section 4.3, Figure 7).
    Introduced by this paper; no code is shipped, and its benefit is asserted via the P-LR-D end-to-end numbers rather than an ablation isolating the envoy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model." pith.science (2026). https://pith.science/paper/N324HERG

@misc{pith2026250623635,
  author       = {Pith},
  title        = {Pith review of: Towards Building Private LLMs: Exploring Multi-Node Expert Parallelism on Apple Silicon for Mixture-of-Experts Large Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N324HERG}},
  note         = {Machine review of arXiv:2506.23635}
}
read the original abstract

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) with significant advancements such as OpenAI's ChatGPT, Meta's Llama, and Databricks' DBRX. This paper addresses the cost and scalability challenges encountered when constructing private LLM systems for personal or small group services, as aimed by Apple Intelligence. A Mac Studio cluster with Apple's M2 Ultra chips is established as a cost-efficient solution to host and accelerate the pretrained DBRX model with the Mixture-of-Experts (MoE) architecture. Our performance analysis reveal that parallel execution of the model's experts across two to four machine nodes significantly reduces inference time. We find that computation time for the experts is comparable to the communication time for exchanging their outputs, emphasizing the importance of network latency over bandwidth. We also observe significant management overhead due to Apple software stack's memory management logic. Based on these findings, we develop optimization schemes to eliminate the memory management overhead. As a result, the Mac Studio cluster is 1.15 times more cost-efficient than the state-of-the-art AI supercomputer with NVIDIA H100 GPUs. In addition, we construct a performance model to estimate system performance under varying configurations, and the model provides valuable insights for designing private LLM systems.

Figures

Figures reproduced from arXiv: 2506.23635 by the authors.

Figure 1
Figure 1. Transformer-based decoder-only LLM architectures: dense [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A parallel computing system organization with four nodes [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A two-node parallel system for the inference of the MoE [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Illustration of the runtime behaviors when running a layer [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Different Weights Warmup Strategies. 4.1 Expert-wise Weights Prestacking With our naive implementation, each of the expert’s weight matri￾ces is loaded as a separate array. When not selected by the router, each layer’s weights can sit idle for multiple tokens. Therefor…
Figure 4
Figure 4. Figure 4: Analysis of the performance impact of the weight-packing [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 7
Figure 7. Figure 7: Architecture of one decoder layer with decentralized self [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Realized results and theoretical performance bounds. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

  1. [1]

    TensorRT-LLM Authors. 2024. TensorRT-LLM. https://github.com/NVIDIA/ TensorRT-LLM

    work page 2024

  2. [2]

    Brown, Benjamin Mann, Nick Ryder, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language Models are Few- Shot Learners. In Proceedings of International Conference on Neural Information Processing Systems

    work page 2020

  3. [3]

    Scott Cheng, Jun-Liang Lin, Murali Emani, et al. 2024. Thorough Characterization and Analysis of Large Transformer Model Training At-Scale. Proceedings of the ACM on Measurement and Analysis of Computing Systems 8, 1 (2024), 1–25

    work page 2024

  4. [4]

    gRPC Authors. 2016. gRPC. https://grpc.io/

    work page 2016

  5. [5]

    Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. 2023. mlx. https://github.com/ml-explore

    work page 2023

  6. [6]

    Yanping Huang, Youlong Cheng, Ankur Bapna, et al . 2019. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In Proceedings of International Conference on Neural Information Processing Systems . 103–112

    work page 2019

  7. [7]

    Apple Inc. 2023. Apple introduces M2 Ultra. www.apple.com/newsroom/2023/ 06/apple-introduces-m2-ultra/

    work page 2023

  8. [8]

    Jacobs, Michael I

    Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton

Show all 18 references
  1. [9]

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, et al . 2021. GShard: Scal- ing Giant Models with Conditional Computation and Automatic Sharding. In Proceedings of International Conference on Learning Representations

    2021

  2. [10]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, et al . 2023. Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems. arXiv:2312.15234

    2023 arXiv

  3. [11]

    OpenAI. 2022. OpenAI: Introducing ChatGPT. https://openai.com/blog/chatgpt

    2022

  4. [12]

    James Reed, Pavel Belevich, and Ke Wen. 2022. PiPPy: Pipeline Parallelism for PyTorch. https://github.com/pytorch/PiPPy

    2022

  5. [13]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, et al. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In Pro- ceedings of International Conference on Learning Representations

    2017

  6. [14]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al . 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053

    2020 arXiv

  7. [15]

    The Mosaic Research Team. 2024. Introducing dbrx: A New State-of-the-Art Open LLM. www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

    2024

  8. [16]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971

    2023 arXiv

  9. [17]

    Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 ns; bandwidth: 200 Gbps; price: 1,267 USD/NIC (https://web.archive.org/web/20240416112730/https: //www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=36...

    2023 arXiv

  10. [1991]

    Neural Computation 3, 1 (1991), 79–87

    Adaptive Mixtures of Local Experts. Neural Computation 3, 1 (1991), 79–87

    1991

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.