Pith. sign in

REVIEW 3 major objections 5 minor 78 references

ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The deep learning compiler Elk claims it can bring inter-core connected AI chips to 94.84% of their ideal roofline performance by jointly optimizing per-core execution, inter-core data exchange, and HBM preloading.

desk verdict A genuinely new joint scheduling framework for inter-core AI chips, whose headline number rests on an HBM emulation that needs validation or caveating. read the letter →

arxiv 2507.11506 v2 pith:7FZ464TJ submitted 2025-07-15 cs.AR cs.DCcs.LG

Yiqi Liu , Yuqi Xue , Noelle Crawford , Jilong Xue , Jian Huang This is my paper
classification cs.ARcs.DCcs.LG
keywords deeplearningcompilerinter-coreconnectedAIchipHBMpreloadoperatorschedulingon-chipmemoryallocationLLMinferencedesignspaceexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a deep learning compiler can bring inter-core connected AI chips (chips with direct links between cores forming a large distributed on-chip memory, plus HBM) to within about 5% of their theoretical roofline performance. The claim is that the three factors that decide end-to-end speed, per-core execution, inter-core data exchange, and off-chip HBM loading, can be treated as configurable compiler parameters and searched jointly instead of optimized one at a time. The compiler, Elk, chooses for each operator how many future operators to preload, how much of each core's SRAM goes to execution versus preload, and in what order preloads happen. On an emulator built from a real ICCA chip with modeled HBM, Elk is reported to reach 94.84% of ideal roofline performance on average, improving on a basic next-operator-only preloader by 1.87x and on a static-allocation scheme by 1.37x. If this holds, the efficiency ceiling of this chip class is mostly a software scheduling problem.

What carries the argument

The load-bearing mechanism is a two-level search over a global trade-off space. The outer level is an inductive operator scheduling policy: working backward through the model, Elk fixes each operator's preload count (the number of future operators whose HBM loads overlap its execution) by minimizing the current-to-end time, with a base case at the last operator and a theorem showing the induction step is exact given optimal per-preload-number execution times. The inner level is a cost-aware on-chip memory allocation algorithm that enumerates Pareto-optimal partition plans for the running operator and each preloaded operator, then greedily trades memory for time using the ratio of reduced space to increased time. A third component, preload order permutation, reorders the sequence of HBM loads within a bounded edit distance, pruning by HBM-heavy operators per transformer layer, to reduce interconnect contention and shorten the on-chip lifetimes of large tensors.

What would settle it

Replace the emulator's single-controller broadcast approximation with a cycle-accurate multi-controller HBM model that includes per-controller queues and bank conflicts, and re-run Elk's optimized plan for Llama-2-70B; if the reported preload/execution overlap or the 94.84% roofline fraction drops materially, the headline result depends on a broadcast model that real HBM controllers would not produce.

Watch

Extended reading notes

Core claim

Elk's central discovery is that the apparent tussle among compute, communication, and I/O on an ICCA chip can be represented as a small global trade-off space in the compiler, with each factor mapped to a knob: execution speed grows with the per-core SRAM allocated to the running operator, HBM utilization grows with the number of operators preloaded ahead of time, and per-operator preload space sizes determine how much shared data is broadcast at preload time instead of fetched from other cores during execution. Elk searches this space with a two-level inductive scheduler and a cost-aware memory allocator. Scheduling works backward from the last operator: for each operator, Elk tries every feasible preload count and keeps the one minimizing the time from the current operator to the end of the model. The allocator walks each operator down its Pareto-optimal curve of partition plans, repeatedly downgrading the plan with the largest ratio of SRAM saved to time added, until everything fits on-chip. A final pass reorders preloads, within a bounded edit distance, to shrink large tensors' SRAM lifespans and to move heavy HBM traffic away from interconnect rush hours. The full design reaches 94.84% of the ideal roofline on average, eliminates 87.65% of the interconnect congestion overhead compared to the same design without reordering, and raises average HBM bandwidth utilization from 34.7% (basic) to 62.40%, close to the 64.38% of the ideal configuration.

Load-bearing premise

The result stands or falls on whether emulating HBM as one controller core that broadcasts data to all cores, with added delays that saturate the interconnect, faithfully captures how real multi-controller HBM traffic contends with inter-core data exchange.

Editorial extensions

If this is right

  • LLM inference on ICCA chips can run within about 5% of the roofline, meaning near-ideal overlap of HBM preloading and on-chip execution is achievable in software.
  • Preload reordering alone removes 87.65% of the interconnect congestion overhead that remains when scheduling and allocation are optimized without reordering, at an average edit distance of 2.9 preload steps.
  • HBM bandwidth utilization on the tested models rises from 34.7% with a next-operator-only preloader to 62.40% with Elk, close to the 64.38% ideal, so existing HBM bandwidth can be used substantially harder.
  • Design-space exploration built on Elk indicates that HBM and interconnect bandwidth should scale together, and that mesh-interconnect ICCA chips are more sensitive to NoC bandwidth than all-to-all chips.
  • Large GQA models such as Gemma2-27B and Llama2-70B can serve tokens at latencies close to smaller models, because grouped-query attention cuts the KV-cache traffic Elk must schedule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-controller broadcast model of HBM in the emulator is the main place where reality could diverge from the 94.84% number; on real multi-controller HBM, per-controller queues and bank conflicts may create contention patterns a single broadcast cannot reproduce.
  • The same three-way trade-off space should transfer to other chips with direct inter-core links, such as GPU SM clusters, where the paper's own discussion notes HBM and inter-SM bandwidth are comparable; Elk's preload reordering could be tested there.
  • Because Elk prunes to HBM-heavy operators and identical transformer layers, the same machinery should extend to mixture-of-experts models by scheduling expert preloads after routing, which the paper proposes as future work but does not implement.
  • The Pareto-ratio greedy allocation could be reused as a general resource-budgeting heuristic for any accelerator whose per-operator memory and time costs lie on measured Pareto curves, independent of inter-core connectivity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Elk is a DL compiler framework for inter-core connected AI (ICCA) chips with off-chip HBM. It structures the trade-off among per-core execution, inter-core communication, and off-chip data loading into configurable compiler parameters, and searches over the number of preloaded operators, the per-operator execution/preload space sizes, and the preload order. The three main algorithmic components are a two-level inductive operator scheduler, a cost-aware greedy memory allocator, and preload-order permutation. The paper evaluates Elk on an emulator built on real IPU-POD4 hardware, with HBM accesses emulated by a controller core that broadcasts data and applies DRAMsim3 latencies, and also on an event-driven simulator supporting all-to-all and mesh topologies. End-to-end LLM decoding results report that Elk-Full reaches 94.84% of an ideal roofline, achieves 89.52% interconnect utilization, and outperforms the Basic and Static baselines by 1.87x and 1.37x on average. Design-space exploration with varied HBM bandwidth, interconnect bandwidth, topology, and core count is also presented.

Significance. If the quantitative claims hold, this is a useful contribution: it demonstrates a systematic compiler-level search space for ICCA chips that couples on-chip scheduling with HBM preloading, and it provides an open-source artifact, a real-IPU-based emulator, and a simulator that supports multiple topologies. The workload breadth (four LLMs plus DiT-XL), the multiple batch sizes and sequence lengths, and the two network topologies strengthen the empirical study. The headline 94.84% result, however, rests on an HBM emulation that is not cross-validated against the paper's own multi-controller simulator or real HBM hardware, and the formal optimality claim for the inductive scheduler is not fully supported by the presented proof. These issues do not invalidate the overall system design, but they must be addressed before the quantitative results can be taken at face value.

major comments (3)
  1. [§5 and §6.2] The end-to-end results in Figure 17, including the headline '94.84% of ideal,' are produced on an emulator whose HBM is realized by a single controller core that broadcasts all preload data to all cores and applies DRAMsim3 latencies as fixed delays. This is a single-serialized-source approximation, whereas the architecture described in §2.1 and modeled in the simulator has multiple HBM controllers with independent request streams. A full broadcast to every receiver also places every preload byte on every inbound link, which is not the traffic shape of per-tile unicast or multicast preloads issued by real HBM controllers with queues, bank conflicts, and row-buffer state. The paper reports no quantitative cross-validation of this HBM emulation against the simulator's multi-controller model or against real HBM-equipped hardware. Because the headline number and the HBM/utilization values in Figure 18(b) depend directly on this approximation, the revision should either validate the emulation (for example, compare emulator and simulator on the same plan and traffic) or report sensitivity of the headline to HBM emulation parameters such as controller count, queueing, and burstiness.
  2. [§4.2, Theorem 4.2] The proof of the inductive step is incomplete. The contradiction argument is asserted rather than derived: it does not formally show that enumerating all preload numbers for operator i must find a schedule minimizing T_end - T^i_s-exe, and the equality T^i_e-exe = min(T^{i+1}_s-exe, T^{i+p+1}_s-pre) omits the on-chip memory capacity constraints from §4.3 and the dependence of L^i_exe on the preload-state plans of the p future operators. Consequently, the sentence 'The algorithm provably finds the end-to-end plan with the shortest total time' is not supported by the presented proof. Please supply a complete induction proof with explicit invariants, or reclassify the scheduling algorithm as a heuristic and justify its quality empirically (for example, against exhaustive search on small models).
  3. [Abstract and §4.3] The abstract states that Elk 'generates globally optimized execution plans,' but the allocation step in §4.3 is an iterative greedy heuristic that repeatedly moves an operator to the next Pareto-optimal plan with the largest ratio Delta = (reduced space)/(increased time) and stops when memory fits. Greedy selection does not guarantee a global optimum, and no bound on the gap to the optimal allocation is given. This overstates the formal status of the contribution. The wording should be qualified, or the paper should include a proof of optimality under the stated cost model, or a small-scale experiment comparing the greedy result to the exhaustive optimum.
minor comments (5)
  1. [§6.1] The Ideal roofline is defined with separate interconnects for preload and execution and zero-latency data distribution; this is a well-defined upper bound, but the paper should state more explicitly that it is an abstraction rather than a hardware-achievable configuration when discussing '94.84% of ideal.'
  2. [§4.4] The pruning rule restricts reordering to HBM-heavy operators with above-average tensor size and to HBM-heavy operators per layer; the paper should quantify how sensitive the final results are to this threshold and justify it beyond the OPT-30B example.
  3. [Figure 18(a)] The legend entry 'interconnect' is ambiguous; it should clarify whether this component represents preload stalled by interconnect, execution stalled by interconnect, or both.
  4. [§5] The sentence 'We also use our real IPU-based emulator to validate our simulator' is not accompanied by a quantitative comparison; please add a figure or table with error bounds for predicted versus measured per-kernel or end-to-end latencies.
  5. [§4.2] The notation T^{i+p+1}_s-pre in Theorem 4.2 is introduced without a definition that links p to the number of overlapped preloads; align this notation with Lemma 4.1 and the surrounding text.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the headline speedups are measured on a real-IPU emulator and the roofline ratio is against an explicit upper bound; the T10 self-citation is a baseline/plan source, not a construction that forces the result.

full rationale

The derivation chain is not circular. Elk's search procedure enumerates partition plans, preload counts, memory allocations, and preload orders, and the final end-to-end latencies in Figure 17 are measured on the IPU-POD4 emulator, not computed from the fitted cost model. The cost model of Section 4.3 is fitted from profiled tile executions and transfer volumes and validated in Figure 12, but it only guides plan selection; the reported speedups and utilizations are emulator measurements. The '94.84% of ideal' claim is a ratio against an explicitly stated upper bound (Section 6.1: 'each of preload and execution has its own interconnect ... and the data distribution phase has zero latency'), not a quantity derived from Elk's own fitted parameters. The only notable self-citation is T10 [34]: the paper uses T10's compute-shift execution-model plans as per-operator inputs and builds the Static baseline on T10. This is a normal use of prior work by overlapping authors; T10 is a peer-reviewed system, and the paper does not invoke a T10 uniqueness theorem to forbid alternatives or define its objective. The inductive scheduling proof (Lemma 4.1, Theorem 4.2) is conditional on obtaining optimal per-preload-number execution times, while Section 4.3 uses a greedy heuristic; that is a correctness/over-claim concern, not circularity. The single-controller HBM broadcast emulation in Section 5 is a fidelity approximation and a validation risk, but it is not a self-referential reduction of the prediction to its inputs. Overall, no load-bearing step reduces by construction to a fitted parameter or a self-citation chain.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The free parameters are the fitted cost model and the reorder threshold. The axioms are the sequential execution model, the HBM emulation approximation, cost model accuracy, and the completeness of the plan enumerator.

free parameters (2)
  • Linear tree cost model coefficients = not reported
    Elk fits linear tree models to profiled execution times per operator type and transfer times per network link (§4.3, Figure 12). These fitted parameters drive plan selection and scheduling.
  • HBM-heavy operator threshold = tensor size above model_size / operator_count
    In §4.4, Elk reorders preloads only for operators whose tensor size is above average. This heuristic, set by hand, prunes the preload order search space.
assumptions (4)
  • domain assumption Operators in a DL model execute sequentially in dependency order.
    Used in §4.2 to justify the inductive scheduling algorithm and its optimality proof. Modern transformer layers are mostly sequential, but arbitrary ONNX DAGs with parallel branches would break the proof.
  • domain assumption A single controller core broadcasting data over the interconnect faithfully emulates HBM controller-to-core preload traffic and contention.
    The emulation framework in §5 treats one core as the HBM controller and assumes broadcast saturates the interconnect and inbound links like real HBM modules.
  • domain assumption The fitted linear tree cost models accurately predict execution and transfer times across the plan space.
    Elk's search uses the cost model to estimate times and choose plans (§4.3). Figure 12 shows good agreement, but the model is an approximation.
  • domain assumption The partition plans enumerated by existing compilers such as T10 cover the relevant plan space.
    Elk relies on partition plans generated by T10 and other compilers (§4.3, §5). If a better plan shape exists outside this set, Elk cannot find it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques." pith.science (2026). https://pith.science/paper/7FZ464TJ

@misc{pith2026250711506,
  author       = {Pith},
  title        = {Pith review of: ELK: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7FZ464TJ}},
  note         = {Machine review of arXiv:2507.11506}
}
read the original abstract

To meet the increasing demand of deep learning (DL) models, AI chips are employing both off-chip memory (e.g., HBM) and high-bandwidth low-latency interconnect for direct inter-core data exchange. However, it is not easy to explore the efficiency of these inter-core connected AI (ICCA) chips, due to a fundamental tussle among compute (per-core execution), communication (inter-core data exchange), and I/O (off-chip data access). In this paper, we develop Elk, a DL compiler framework to maximize the efficiency of ICCA chips by jointly trading off all the three performance factors discussed above. Elk structures these performance factors into configurable parameters and forms a global trade-off space in the DL compiler. To systematically explore this space and maximize overall efficiency, Elk employs a new inductive operator scheduling policy and a cost-aware on-chip memory allocation algorithm. It generates globally optimized execution plans that best overlap off-chip data loading and on-chip execution. To examine the efficiency of Elk, we build a full-fledged emulator based on a real ICCA chip IPU-POD4, and an ICCA chip simulator for sensitivity analysis with different interconnect network topologies. Elk achieves 94% of the ideal roofline performance of ICCA chips on average, showing the benefits of supporting large DL models on ICCA chips. We also show Elk's capability of enabling architecture design space exploration for new ICCA chip development.

Figures

Figures reproduced from arXiv: 2507.11506 by the authors.

Figure 1
Figure 1. Architecture of inter-core connected AI (ICCA) chip. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Resource contentions on ICCA chip with HBM. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Operator partitioning and inter-core data sharing. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (16 more)
Figure 6
Figure 6. Figure 6: HBM bandwidth demands of models across time, [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 5
Figure 5. Figure 5: The execution times of representative operators [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: The inter-core bandwidth demand of each core [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: The total per-core interconnect bandwidth demand. [PITH_FULL_IMAGE:figures/full_fig_p005_8.png]
Figure 9
Figure 9. Figure 9: Overview of our Elk framework. 4 Design and Implementation We design Elk, a compiler framework for exploring the efficiency of ICCA chips. Elk automatically trades-off performance factors by configuring the number of preloaded operators, the per-core execution space si…
Figure 10
Figure 10. Figure 10: Select the preload number that minimizes the [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 11
Figure 11. Figure 11: Tradeoff between time overhead & memory usage. [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]
Figure 13
Figure 13. Figure 13: Reorder preloads to allow larger execution space. [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]
Figure 14
Figure 14. Figure 14: The generation of candidate preload orders. [PITH_FULL_IMAGE:figures/full_fig_p008_14.png]
Figure 16
Figure 16. Figure 16: Elk compile time for varied model/batch sizes. (3) The code generation in Elk generates the kernel code for com￾puting each tile and the inter-core data transfer operations, based on the target hardware and selected partition plans. For compute, Elk uses code template…
Figure 17
Figure 17. Figure 17: The per-token serving latency of various models and batch sizes on 4 ICCA chips with 16TB/s HBM. [PITH_FULL_IMAGE:figures/full_fig_p010_17.png]
Figure 20
Figure 20. Figure 20: Breakdown of LLama2-13B per-token latency with [PITH_FULL_IMAGE:figures/full_fig_p011_20.png]
Figure 19
Figure 19. Figure 19: Per-token latency at varied HBM bandwidths. [PITH_FULL_IMAGE:figures/full_fig_p011_19.png]
Figure 21
Figure 21. Figure 21: Interconnect utilization at varied HBM bandwidths. 8TB/s HBM 10 20 30 All-to-All 10TB/s HBM 12TB/s HBM 14TB/s HBM Basic Static ELK-Dyn ELK-Full Ideal 30 40 10 20 30 Mesh 30 40 30 40 30 40 Total Interconnect Bandwidth (TB/s) Latency (ms) [PITH_FULL_IMAGE:figures/full_…
Figure 22
Figure 22. Figure 22: Llama2-70B latency of at varied NoC bandwidths. [PITH_FULL_IMAGE:figures/full_fig_p012_22.png]
Figure 23
Figure 23. Figure 23: Per-token latency at varied core counts. [PITH_FULL_IMAGE:figures/full_fig_p012_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 63 canonical work pages

  1. [1]

    Open Neural Network Exchange format

    2017. Open Neural Network Exchange format. https://onnx.ai/

    work page 2017

  2. [2]

    NVIDIA Hopper Architecture In-Depth

    2022. NVIDIA Hopper Architecture In-Depth. https://developer.nvidia.com/ blog/nvidia-hopper-architecture-in-depth/

    work page 2022

  3. [3]

    Abdelhafez, Christopher Zimmer, Sudharshan S

    Hazem A. Abdelhafez, Christopher Zimmer, Sudharshan S. Vazhkudai, and Matei Ripeanu. 2019. AHEAD: A Tool for Projecting Next-Generation Hardware En- hancements on GPU-Accelerated Systems. In2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW’19)

    work page 2019

  4. [4]

    Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, and Alexey Tumanov. 2024. Vidur: A Large-Scale Simulation Framework For LLM Inference. InProceedings of Machine Learning and Systems (MLSys’24)

    work page 2024

  5. [5]

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training Generalized Multi-Query Trans- former Models from Multi-Head Checkpoints.arXiv preprint arXiv:2305.13245 (2023)

    arXiv 2023

  6. [6]

    Ionescu, Klaus E

    Albert Alexandrov, Mihai F. Ionescu, Klaus E. Schauser, and Chris Scheiman

    work page

  7. [7]

    Mohamed Bahnas. 2024. Tenstorrent Overview: Products and Software. https: //icl.utk.edu/newsletter/presentations/2024/mohamed-bahnas-2024-03-22.pdf

    work page 2024

  8. [8]

    Jingwei Cai, Xuan Wang, Mingyu Gao, Sen Peng, Zijian Zhu, Yuchen Wei, Zuo- tong Wu, and Kaisheng Ma. 2025. SoMa: Identifying, Exploring, and Understand- ing the DRAM Communication Scheduling Space for DNN Accelerators.arXiv preprint arXiv:2501.12634(2025)

    work page arXiv 2025

Show all 78 references
  1. [9]

    Jingwei Cai, Yuchen Wei, Zuotong Wu, Sen Peng, and Kaisheng Ma. 2023. Inter- layer Scheduling Space Definition and Exploration for Tiled Accelerators. In Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA’23)

    2023

  2. [10]

    Marco Cerliani. 2022. Linear-Tree. https://github.com/cerlymarco/linear-tree

    2022

  3. [11]

    Chang Chen, Xiuhong Li, Qianchao Zhu, Jiangfei Duan, Peng Sun, Xingcheng Zhang, and Chao Yang. 2024. Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communi- cation Partitioning. InProceedings of the 29th ACM Internatio...

    2024

  4. [12]

    Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In13th USENIX Symposium on Op...

    2018

  5. [13]

    Peizhuang Cong, Aomufei Yuan, Shimao Chen, Yuxuan Tian, Bowen Ye, and Tong Yang. 2024. Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing.arXiv preprint arXiv:2404.16914(2024)

    2024 arXiv

  6. [14]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv preprint arXiv:2205.14135(2022)

    2022 arXiv

  7. [15]

    Dahu Feng, Erhu Feng, Dong Du, Pinjie Xu, Yubin Xia, Haibo Chen, and Rong Zhao. 2025. Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units. InProceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA’25)

    2025

  8. [16]

    Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, and Tianqi Chen. 2023. TensorIR: An Abstraction for Automatic Tensorized Program Optimization. InProceedings of the 28th ACM International Conference on Archi...

    2023

  9. [17]

    Weiwei Gong and Xu Zhou. 2017. A survey of SAT solver. InAIP Conference Proceedings. https://doi.org/10.1063/1.4981999

    2017 doi

  10. [18]

    Google. 2023. XLA. https://www.tensorflow.org/xla

    2023

  11. [19]

    Graphcore. 2022. Tile Vertex ISA. https://docs.graphcore.ai/projects/isa/en/ latest/_static/Tile-Vertex-ISA_1.2.3.pdf

    2022

  12. [20]

    Graphcore. 2024. Next Generation IPU Systems: IPU-M2000 + IPU-POD4. https: //www.graphcore.ai/products/mk2/ipu-m2000-ipu-pod4

    2024

  13. [21]

    Graphcore. 2024. PopLibs API reference. https://docs.graphcore.ai/projects/ poplar-api/en/latest/poplibs_api.html

    2024

  14. [22]

    Congjie He, Yeqi Huang, Pei Mu, Ziming Miao, Jilong Xue, Lingxiao Ma, Fan Yang, and Luo Mai. 2025. WaferLLM: A Wafer-Scale LLM Inference System.arXiv preprint arXiv:2502.04563(2025)

    2025 arXiv

  15. [23]

    Emmanuel Hebrard. 2012. Scheduling and SAT.Nantes(2012). https://homepages. laas.fr/ehebrard/papers/prescpaior2012.pdf

    2012

  16. [24]

    Charles Hong, Sahil Bhatia, Altan Haan, Shengjun Kris Dong, Dima Nikiforov, Alvin Cheung, and Yakun Sophia Shao. 2024. LLM-Aided Compilation for Tensor Accelerators.arXiv preprint arXiv:2408.03408(2024)

    2024 arXiv

  17. [25]

    Muyan Hu, Ashwin Venkatram, Shreyashri Biswas, Balamurugan Marimuthu, Bo- han Hou, Gabriele Oliaro, Haojie Wang, Liyan Zheng, Xupeng Miao, Jidong Zhai, and Zhihao Jia. 2024. Optimal Kernel Orchestration for Tensor Programs with Korch. InProceedings of the 29th ACM Internationa...

    2024

  18. [26]

    Abhinav Jangda, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi

  19. [27]

    Zhe Jia, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza. 2020. Dis- secting the Graphcore IPU Architecture via Microbenchmarking.arXiv preprint arXiv:1912.03413(2020)

    2020 arXiv

  20. [28]

    Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson. 2023. TPU v4: An Optically Reconfigurable Supercomputer for Machine Learn...

    2023

  21. [29]

    Simon Knowles. 2021. Graphcore Colossus Mk2 IPU. In2021 IEEE Hot Chips 33 Symposium (HCS’21)

    2021

  22. [30]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

    2023

  23. [31]

    Dacheng Li, Hongyi Wang, Eric Xing, and Hao Zhang. 2022. AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness. InAdvances in Neural Information Processing Systems (NeurIPS’22)

    2022

  24. [32]

    Shang Li, Zhiyuan Yang, Dhiraj Reddy, Ankur Srivastava, and Bruce Jacob. 2020. DRAMsim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator.IEEE Com- puter Architecture Letters(2020)

    2020

  25. [33]

    Sean Lie. 2021. Multi-Million Core, Multi-Wafer AI Cluster. In2021 IEEE Hot Chips 33 Symposium (HCS’21)

    2021

  26. [34]

    Yiqi Liu, Yuqi Xue, Yu Cheng, Lingxiao Ma, Ziming Miao, Jilong Xue, and Jian Huang. 2024. Scaling Deep Learning Computation over the Inter-Core Con- nected Intelligence Processor with T10. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles (SOSP’24)

    2024

  27. [35]

    Wenyan Lu, Guihai Yan, Jiajun Li, Shijun Gong, Yinhe Han, and Xiaowei Li

  28. [36]

    Xinhao Luo, Zihan Liu, Yangjie Zhou, Shihan Fang, Ziyu Huang, Yu Feng, Chen Zhang, Shixuan Sun, Zhenzhe Zheng, Jingwen Leng, and Minyi Guo. 2025. Clus- terFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive.arXiv preprint arXiv:2508...

    2025 arXiv

  29. [37]

    Meta. 2024. Our next-generation Meta Training and Inference Accelera- tor. https://ai.meta.com/blog/next-generation-meta-training-inference- accelerator-AI-MTIA/

    2024

  30. [38]

    Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. 2022. Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism.Proceedings of the VLDB Endowment(2022)

    2022

  31. [39]

    Micron. 2024. HBM3E. https://www.micron.com/products/memory/hbm/hbm3e

    2024

  32. [40]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. Efficient Large-Scale Language Model Training on GPU Clus...

    2021

  33. [41]

    NVIDIA. 2024. Blackwell Architecture for Generative AI. https://www.nvidia. com/en-us/data-center/technologies/blackwell-architecture/

    2024

  34. [42]

    NVIDIA H100 Tensor Core GPU. 2024. https://www.nvidia.com/en-us/data- center/h100/

    2024

  35. [43]

    Dylan Patel and Daniel Nishball. 2023. Groq Inference Tokenomics: Speed, But At What Cost. https://www.semianalysis.com/p/groq-inference-tokenomics- speed-but

    2023

  36. [44]

    William Peebles and Saining Xie. 2023. Scalable Diffusion Models with Trans- formers.arXiv preprint arXiv:2212.09748(2023)

    2023 arXiv

  37. [45]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Brad- bury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently Scaling Transformer Inference.arXiv preprint arXiv:2211.05102(2022)

    2022 arXiv

  38. [46]

    Shah, Edison Chen, Kaizhao Liang, Swayambhoo Jain, Urmish Thakker, Dawei Huang, Sumti Jairath, Kevin J

    Raghu Prabhakar, Ram Sivaramakrishnan, Darshan Gandhi, Yun Du, Mingran Wang, Xiangyu Song, Kejie Zhang, Tianren Gao, Angela Wang, Karen Li, Yongn- ing Sheng, Joshua Brot, Denis Sokolov, Apurv Vivek, Calvin Leung, Arjun Sabnis, Jiayu Bai, Tuowen Zhao, Mark Gottscho, David Jacks...

    2024 arXiv

  39. [47]

    Raghu Prabhakar, Yaqi Zhang, David Koeplinger, Matt Feldman, Tian Zhao, Stefan Hadjis, Ardavan Pedram, Christos Kozyrakis, and Kunle Olukotun. 2017. Elk: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques MICRO’25, October 18-22, 2...

    2017

  40. [48]

    PyTorch. 2024. Building Models with PyTorch. https://pytorch.org/tutorials/ beginner/introyt/modelsyt_tutorial.html

    2024

  41. [49]

    Saeed Rashidi, William Won, Sudarshan Srinivasan, Srinivas Sridharan, and Tushar Krishna. 2022. Themis: a network bandwidth-aware collective scheduling policy for distributed training of DL models. InProceedings of the 49th Annual International Symposium on Computer Architectu...

    2022

  42. [50]

    Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. 2023. Welder: Scheduling Deep Learning Memory Access via Tile-graph. In17th USENIX Symposium on Operating Systems Design and Implementation (OSDI’23)

    2023

  43. [51]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juli- ette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, ...

  44. [52]

    Tenstorrent. 2023. Meet Grayskull. https://tenstorrent.com/grayskull/

    2023

  45. [53]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucu- rull, David Esiobu, Jude Fernandes, Jeremy...

    2023 arXiv

  46. [54]

    Taegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee, Goeun Kim, Dongseob Kim, Youngtaek Kim, Mohd Muzzammil, and Myeongjae Jeon. 2024. Metis: Fast Automatic Distributed Training on Heterogeneous GPUs. In2024 USENIX Annual Technical Conference (USENIX ATC’24)

    2024

  47. [55]

    Mario Vanhoucke and José Coelho. 2016. An approach using SAT solvers for the RCPSP with logical constraints.European Journal of Operational Research(2016)

    2016

  48. [56]

    Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen

    Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S. Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018. Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions.arXiv preprint arXiv:180...

    2018 arXiv

  49. [57]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need.arXiv preprint arXiv:1706.03762(2023)

    2023 arXiv

  50. [58]

    Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, and Mao Yang. 2024. Ladder: Enabling Efficient Low-Precision Deep Learning Computing through Hardware-aware Tensor Transformation. In18th USENI...

    2024

  51. [59]

    Shibo Wang, Jinliang Wei, Amit Sabne, Andy Davis, Berkin Ilbeyi, Blake Hecht- man, Dehao Chen, Karthik Srinivasa Murthy, Marcello Maggioni, Qiao Zhang, Sameer Kumar, Tongfei Guo, Yuanzhong Xu, and Zongwei Zhou. 2022. Over- lap Communication with Dependent Computation via Decom...

    2022

  52. [60]

    Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: An In- sightful Visual Performance Model for Multicore Architectures.Communications of the ACM(2009)

    2009

  53. [61]

    Yuqi Xue and Jian Huang. 2025. ReGate: Enabling Power Gating in Neural Processing Units.arXiv preprint arXiv:2508.02536(2025)

    2025

  54. [62]

    Yuqi Xue, Yiqi Liu, and Jian Huang. 2023. System Virtualization for Neural Processing Units. InProceedings of the 19th Workshop on Hot Topics in Operating Systems (HotOS’23)

    2023

  55. [63]

    Yuqi Xue, Yiqi Liu, Lifeng Nai, and Jian Huang. 2023. V10: Hardware-Assisted NPU Multi-tenancy for Improved Resource Utilization and Fairness. InProceedings of the 50th Annual International Symposium on Computer Architecture (ISCA’23)

    2023

  56. [64]

    Yuqi Xue, Yiqi Liu, Lifeng Nai, and Jian Huang. 2024. Hardware-Assisted Virtual- ization of Neural Processing Units for Cloud Platforms. In2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO’24)

    2024

  57. [65]

    Haoyang Zhang, Yirui Zhou, Yuqi Xue, Yiqi Liu, and Jian Huang. 2023. G10: Enabling An Efficient Unified GPU Memory and Storage Architecture with Smart Tensor Migrations. InProceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO’23)

    2023

  58. [66]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  59. [67]

    Yaqi Zhang, Alexander Rucker, Matthew Vilim, Raghu Prabhakar, William Hwang, and Kunle Olukotun. 2019. Scalable interconnects for reconfigurable spatial architectures. InProceedings of the 46th International Symposium on Computer Architecture (ISCA’19)

    2019

  60. [68]

    Yaqi Zhang, Nathan Zhang, Tian Zhao, Matt Vilim, Muhammad Shahbaz, and Kunle Olukotun. 2021. SARA: Scaling a Reconfigurable Dataflow Accelerator. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA’21)

    2021

  61. [69]

    Jie Zhao, Siyuan Feng, Xiaoqiang Dan, Fei Liu, Chengke Wang, Sheng Yuan, Wenyuan Lv, and Qikai Xie. 2023. Effectively Scheduling Computational Graphs of Deep Neural Networks toward Their Domain-Specific Accelerators. In17th USENIX Symposium on Operating Systems Design and Impl...

    2023

  62. [70]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020. Ansor: generating high-performance tensor programs for deep learning. InProceedings of the 14th USENIX Confer...

    2020

  63. [71]

    Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yan- ping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al. 2022. Alpa: Automating Inter-and Intra-Operator Parallelism for Distributed Deep Learning. In16th USENIX Symposium on Operating Syste...

    2022

  64. [72]

    Size Zheng, Renze Chen, Anjiang Wei, Yicheng Jin, Qin Han, Liqiang Lu, Bingyang Wu, Xiuhong Li, Shengen Yan, and Yun Liang. 2022. AMOS: Enabling automatic mapping for Tensor Computations on spatial Accelerators with Hardware Ab- straction. InProceedings of the 49th Annual Inte...

    2022

  65. [73]

    Zhen Zheng, Xuanda Yang, Pengzhan Zhao, Guoping Long, Kai Zhu, Feiwen Zhu, Wenyi Zhao, Xiaoyong Liu, Jun Yang, Jidong Zhai, Shuaiwen Leon Song, and Wei Lin. 2022. AStitch: Enabling a New Multi-Dimensional Optimization Space for Memory-Intensive ML Training and Inference on Mod...

    2022

  66. [74]

    benchmark_scripts/generate_data_from_sim.py

    Hongyu Zhu, Ruofan Wu, Yijia Diao, Shanbin Ke, Haoyu Li, Chen Zhang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Wei Cui, Fan Yang, Mao Yang, Lidong Zhou, Asaf Cidon, and Gennady Pekhimenko. 2022. ROLLER: Fast and Efficient Tensor Compilation for Deep Learning. In16th USENIX Symposiu...

    2022

  67. [1995]

    InProceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA’95)

    LogGP: incorporating long messages into the LogP model—one step closer towards a realistic model for parallel computation. InProceedings of the Seventh Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA’95)

  68. [2017]

    InProceedings of the 23rd IEEE Symposium on High Performance Computer Architecture (HPCA’17)

    FlexFlow: A Flexible Dataflow Accelerator Architecture for Convolutional Neural Networks. InProceedings of the 23rd IEEE Symposium on High Performance Computer Architecture (HPCA’17)

  69. [2022]

    InProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’22)

    Breaking the computation and communication abstraction barrier in dis- tributed machine learning workloads. InProceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’22)

  70. [2024]

    Gemma: Open Models Based on Gemini Research and Technology.arXiv preprint arXiv:2403.08295(2024)

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.