PIM-AI: A Novel Architecture for High-Efficiency LLM Inference

Cristobal Ortega; Renaud Ayrignac; Yann Falevoz

REVIEW 4 major objections 5 minor 62 references

PIM-AI: A Novel Architecture for High-Efficiency LLM Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that embedding RISC-V compute inside DDR5/LPDDR5 memory chips, in a design named PIM-AI, can cut the energy cost of mobile LLM inference by 10-20x per token and reduce cloud serving cost per query by up to 6.94x, based…

desk verdict The simulator and cloud design are worth engaging, but the mobile results are built on a 2 GB chip that can't fit a 7B model, so the headline energy numbers don't hold as presented. read the letter →

arxiv 2411.17309 v1 pith:M2I6JEJK submitted 2024-11-26 cs.AR cs.AIcs.DCcs.ET

Cristobal Ortega , Yann Falevoz , Renaud Ayrignac This is my paper · ORCID

classification cs.ARcs.AIcs.DCcs.ET

keywords processing-in-memoryLLMinferenceenergyefficiencymemorywallRISC-VDDR5LPDDR5performancesimulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces PIM-AI, a processing-in-memory architecture that places RISC-V processors with tensor/vector units on a logic die stacked under a DRAM die, allowing LLM weights and activations to be read at 102.4 GB/s and 0.95 pJ/bit instead of crossing a chip boundary. Its central claim is that this eliminates the energy and latency penalty of the 'memory wall' in LLM decoding, which is memory-bound rather than compute-bound. Simulating Llama-2 and Mixtral models against an NVIDIA H100 server and three flagship mobile SoCs, the authors report that four PIM-AI servers match or beat a DGX-H100 on queries per second while cutting 3-year cost per query by up to 6.94x, and that on mobile the design reduces energy per token by 10 to 20x and energy per query by 6.9 to 13.4x. These numbers make PIM-AI a potentially disruptive approach to LLM serving and on-device inference, if the assumed hardware parameters hold up in a real chip.

What carries the argument

The load-bearing object is the PIM-AI chip profile: a DDR5/LPDDR5 memory chip with an embedded logic die containing four Linux-capable RISC-V processors, each with tensor and vector units, accessing DRAM banks at 102.4 GB/s and 0.95 pJ/bit. The argument's arithmetic is carried by a simulator that counts operations and data transfers from actual PyTorch model graphs and multiplies them by per-TOPS and per-bit energy constants. The most important constant is the memory-access energy ratio: the simulator treats PIM-AI's internal access as 0.95 pJ/bit versus 20 pJ/bit for the A17 Pro's LPDDR5 and 7 pJ/bit for the H100's HBM, and the headline energy reductions are almost exactly those ratios.

What would settle it

Fabricate or field-accurately model the PIM-AI chip and measure the average energy per bit for DRAM bank reads and writes, plus sustained bandwidth and TOPS, then run Llama-2-7B decoding with the same quantization and KV-cache settings as the simulation; compare measured energy per token against the predicted 10-20x improvement over a phone SoC. The paper itself states that a prototype is the planned validation, with a goal by the end of 2025.

Watch

Extended reading notes

Core claim

The central claim is that placing arithmetic near the data eliminates the memory-wall energy penalty of moving weights and activations across a chip boundary, and that this pays off most in the decoding phase, where each generated token streams the entire model's weights from memory. PIM-AI's architecture uses a stacked die: a standard DRAM die with an added logic die containing four RISC-V cores with tensor/vector units, connected at 102.4 GB/s and 0.95 pJ/bit. The authors simulate this design against the NVIDIA H100 and three mobile SoCs (A17 Pro, Snapdragon 8 Gen 3, Dimensity 9300) on Llama-2 and Mixtral models. They find that PIM-AI is slower to first token (about 3x with GQA, 75% with MHA) but decodes tokens 2.23 to 2.75x faster than the H100-based server, and on mobile decodes tokens at 10 to 20x lower energy per token, enough to offset the encoding lag. The paper frames the result as making PIM-AI a viable, socket-compatible accelerator that needs no changes to the host memory controller or DDR PHY.

Load-bearing premise

The claimed gains are computed from assumed per-bit memory-access energies (0.95 pJ/bit for PIM-AI versus 7-20 pJ/bit for alternatives) extrapolated from a DDR4 PIM product to an unbuilt DDR5/LPDDR5 logic die; if real memory access energy is higher or bandwidth lower, the headline improvements shrink or vanish.

Editorial extensions

If this is right

Because PIM-AI is socket-compatible with standard DDR5/LPDDR5 slots, servers and phones could adopt it without redesigning the host memory controller or PHY.
In cloud serving, four PIM-AI servers (96 DIMMs in 8U) process roughly 55% more queries per second than one DGX-H100 server, cutting the 3-year total cost of ownership per query per second by 6.2x to 6.94x.
On mobile, a single PIM-AI chip delivers 25% to 45% more queries per second than the A17 Pro, Snapdragon 8 Gen 3, or Dimensity 9300 while consuming 6.9x to 13.4x less energy per query.
The advantage grows when generated output is long: with 1,000 output tokens, the mobile energy-per-query advantage rises to 9.8x to 19.5x, directly extending battery life.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The headline energy reductions are essentially the ratio of the assumed internal-access energy (0.95 pJ/bit) to the competing memory interfaces (7-20 pJ/bit); a real logic die with interconnect overhead could easily land at 2-3 pJ/bit, shrinking the mobile advantage to the single digits.
The paper's own finding that PIM-AI is slower on the encoding phase suggests the strongest design is heterogeneous: a conventional accelerator for the prompt GEMMs plus PIM-AI for the decoding GEMVs, an idea the authors flag as future work.
A testable extension would measure the TCO advantage under a realistic cloud workload mix, since the advantage concentrates on long-output decoding; heavy prompt traffic would narrow the gap versus GPUs.
If the per-chip 102.4 GB/s bandwidth scales as described to a 1.6 TB/s DIMM, PIM-AI would also be attractive for small-batch and latency-sensitive edge serving, where GPUs are over-provisioned.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the simulator is real, the architecture description is clear, and the paper is an honest attempt to push PIM for LLM inference. But the mobile headline numbers rest on a configuration that cannot physically hold the model, so those results are wrong as stated.

The genuinely useful piece here is the open-source simulator (upmem_llm_framework) and the transparent linear energy/bandwidth model. The paper clearly lays out how it counts operations, transfers, and energy. That is reproducible and worth building on. The cloud design, with multiple PIM-AI DIMMs for 70B-class models, is also a reasonable configuration to study, and the paper's own limitations section is candid about constant-peak-performance assumptions and the need for a prototype.

The soft spots, in order:

1. Capacity. Section 2.1 fixes each PIM-AI chip at 2 GB. Section 3.4 applies 4-bit quantization to Llama2-7B and Mistral-7B. Those models need at least 3.5 GB for weights alone, plus roughly 0.5 GB for the KV cache and activations in the standard 1000+100 token setup. So a single 2 GB chip cannot hold the workload. The simulator has no memory-capacity parameter, so it happily reports results for an unbuildable configuration. If the weights were stored off-chip, the 20 pJ/bit transfer energy (Table 1) would apply, and the claimed 10-20x mobile energy advantage collapses. This is not a parameter-uncertainty issue; it is an internal inconsistency.

2. The hardware profile parameters. The 0.95 pJ/bit internal memory energy and 5 TOPS per chip are extrapolated from UPMEM DDR4 PIM products to an unbuilt DDR5/LPDDR5 design. Because the simulator computes energy linearly in pJ/bit, the headline gains are essentially the ratio of assumed memory energies (20 vs 0.95 pJ/bit). No sensitivity analysis is provided, and there is no validation against measured baselines. The paper cites its own previous work for UPMEM energy numbers, which is fine, but the step to 0.95 pJ/bit for a new stacked-logic design is not justified in the text.

3. TCO. The 6.2-6.94x cloud TCO improvement is driven heavily by assumed server prices: $15k for a PIM-AI server vs $300k for a DGX-H100. These numbers are stated without sources, and no sensitivity to electricity price or utilization is given. The QPS improvement itself is modest (55%) and the first-token latency is 3x worse.

Who is this for? Hardware architects looking at PIM for LLMs will find the simulator useful and the design questions relevant. It deserves a serious referee — the toolkit and the cloud analysis are worth engaging — but the mobile section needs a full rewrite with capacity-aware modeling and either real measurements or a well-justified parameter range. I would not take the headliners at face value.

Referee Report

4 major / 5 minor

Summary. The paper proposes PIM-AI, a DDR5/LPDDR5 processing-in-memory architecture for LLM inference, and describes a PyTorch-based simulator that estimates execution time, energy, and throughput for cloud and mobile workloads. Using hand-tuned hardware profiles, the authors report substantial advantages over an NVIDIA DGX-H100 server (up to 6.94x lower 3-year TCO per QPS) and over three mobile SoCs (10-20x lower energy per token, 25-45% higher QPS, and 6.9-13.4x lower energy per query). The simulator is open-sourced on GitHub, and the evaluation covers Llama2-70B and Mixtral-8x22B in the cloud and Llama2-7B and Mistral-7B on mobile.

Significance. The paper targets an important real problem: the memory-bound nature of LLM decoding and the energy cost of moving weights and KV caches between memory and compute. Its open-source simulator and the transparent decomposition of energy into compute, memory-access, and transfer terms are strengths that make the analysis easy to reproduce and extend. However, the headline quantitative results are not independent findings; they are direct consequences of the assumed hardware profile parameters (especially pJ/bit values and server cost). Moreover, the mobile scenario appears to exceed the capacity of the described 2 GB PIM-AI chip. With revisions that add capacity-aware mapping, sensitivity analysis, and measured parameter anchors, the manuscript could be a valuable architecture study; in its current form, the results are best regarded as idealized projections rather than validated efficiency gains.

major comments (4)

[Section 3.2, Table 1] The claimed mobile energy reductions and cloud TCO improvements are algebraically determined by the hand-set hardware profile parameters, not by independent measurements. For example, the 20x energy-per-token advantage over the A17 Pro is essentially the ratio of the assumed main-memory pJ/bit values (20 vs. 0.95), and the 6.94x TCO ratio is largely the assumed $15k vs. $300k server cost. Because the simulator is a linear model, these numbers should be presented as consequences of the assumptions, and the paper needs a sensitivity analysis that sweeps the uncertain parameters (e.g., PIM-AI memory energy 0.5-3 pJ/bit, bandwidth 50-150 GB/s, server price $10k-$30k). I also request a calibration exercise against any public measurements of UPMEM DDR4 PIM energy per bit to anchor the 0.95 pJ/bit value.
[Section 2.1, 3.4, and 4.2] The mobile scenario is internally inconsistent. Section 2.1 fixes the PIM-AI chip at 2 GB of DRAM, yet the mobile simulation runs Llama2-7B and Mistral-7B with 4-bit weights, which require at least 3.5 GB for weights alone, plus a 16-bit KV cache of roughly 0.5 GB for the 1000-token prompt and 100 generated tokens. The simulator has no memory-capacity parameter, so the reported mobile energy per token and QPS numbers correspond to a configuration that cannot be built with a single 2 GB chip. The paper must either specify a multi-chip mapping (with inter-chip transfer energy and capacity-aware scheduling) or reduce the mobile models to sizes that fit in the described chip; otherwise the reported gains are not physically meaningful.
[Section 5.1] The 3-year TCO per QPS claim is not substantiated with a model. The text mentions a PIM-AI production server cost of approximately $15k and a DGX-H100 server cost of $300k, and an electricity price of $0.153/kWh, but the TCO formula (including server amortization, power draw, cooling, and utilization) is not given. Since the cost ratio is a load-bearing input, the paper should present the full TCO calculation and provide a sensitivity range for the PIM-AI server price, as a small change in the assumed cost could materially change the 6.2-6.94x claim.
[Section 3.1 and 5.2] The simulator's fidelity is not established. The execution-time model assumes constant peak performance and does not model DRAM bank contention, refresh, or the scheduling of the four RISC-V processors on the PIM-AI chip. The references cited in Section 5.2 as supporting evidence ([62], [63]) are vendor blog posts rather than peer-reviewed measurements of the relevant hardware. I recommend either demonstrating that the simulator reproduces measured UPMEM PIM performance and energy on a small workload, or explicitly characterizing all reported percentages (e.g., 49.6% more tokens/s) as idealized, first-order estimates.

minor comments (5)

[Section 3.4] The mobile benchmark list says 'Mistral-8x7B' while the abstract and Section 4.2 use 'Mistral-7B'; similarly, the cloud benchmark 'Mixtral-22x7B' should be 'Mixtral-8x22B' to agree with Section 4.1.
[Section 2.1 vs. Table 1] Section 2.1 states that the tensor units perform up to 8 TOPS, but Table 1 lists the PIM-AI chip at 5 TOPS; please reconcile the numeric discrepancy and ensure the simulator uses a single consistent value.
[Figures 4 and 5] Several axis labels and captions are corrupted or missing text (e.g., 'eries/s' and 'alcomm Snapdragon' in Figure 5, and misaligned subplot labels in Figure 4); please regenerate the figures with clean, complete labels.
[Section 5.2] The statement that 'recent research shows that mobile NPUs and high-end GPUs can achieve similar performance to our simulations' cites Apple and NVIDIA blog posts; these are not adequate to validate the simulator's energy-per-bit parameters, so either replace them with peer-reviewed benchmarks or soften the claim.
[References] Reference [55] is listed as 'Upmem tech paper' without a full bibliographic entry; please provide the complete technical report or paper citation.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline mobile energy advantage is the input pJ/bit ratio; the PIM-AI profile rests on a UPMEM self-citation, and the 7B mobile scenario exceeds the 2 GB chip capacity.

other [Section 3.1 (simulator model), Table 1 (hardware profiles), Section 4.2.2 (mobile decoding results)]
"PIM-AI chip 5 0.4 102.4 0.95 12.8 20 ... A17 Pro 17 0.4 51.2 20 51.2 20 ... The PIM-AI chip is also significantly more energy efficient, 20 times more efficient per token than the A17 Pro and 10 times more efficient than the other profiles."

Section 3.1 says data-transfer energy is computed from 'bandwidth and energy per bit' and GEMM/GEMV energy from 'energy per TOP'. Decoding is memory-bound, so for the same model, token count, and bit width, per-token energy is proportional to the memory pJ/bit parameter. Table 1 sets PIM-AI main-memory pJ/bit to 0.95 and A17 Pro to 20, a ratio of about 21x; Section 4.2.2 then reports '20 times more efficient per token'. The claimed 10-20x mobile energy reduction is exactly the ratio of the hand-set pJ/bit inputs, so the 'prediction' reduces to the input by construction.
self citation load bearing [Section 3.2, with reference [55]]
"We parameterize the PIM-AI chip and DIMM to create hardware profiles for the simulator based on DDR4 PIM products such as UPMEM [55]."

The paper's authors are UPMEM personnel (contact@upmem.com) and [55] is listed only as 'Upmem tech paper'. The PIM-AI profile (0.95 pJ/bit, 102.4 GB/s, 5 TOPS) that drives the headline energy and QPS results is sourced from this self-citation without independent measurement or external benchmark; Section 5.3 says validation on a real PIM-AI chip is future work. Thus the central premise of the efficiency advantage is load-bearing self-citation.

full rationale

Most of the paper is a standard parameterized simulation study, and the cloud QPS/TCO calculations are transparent arithmetic. However, the mobile energy result is not an independent finding: the simulator's linear energy model makes per-token energy proportional to the chosen pJ/bit parameter, and Table 1 chooses a ~21x ratio between A17 Pro and PIM-AI, which Section 4.2.2 then reports as the 20x improvement. The profile values themselves are justified by a UPMEM tech-paper self-citation, and the paper concedes no real chip has been built. Separately, the mobile scenario is internally inconsistent: Section 2.1 fixes each PIM-AI chip at 2 GB, while Section 3.4 runs 7B models with 4-bit weights (>=3.5 GB plus KV cache); because the simulator has no capacity parameter, it reports energy as if all weight reads hit the 0.95 pJ/bit PIM memory. This is a feasibility/correctness flaw rather than a circular reduction, but it reinforces that the mobile numbers are not grounded in the architecture as defined. The TCO comparison also depends heavily on the assumed $15k versus $300k server costs, but that dependence is explicit and not hidden. Overall, one central 'prediction' reduces by construction and is built on a self-cited parameter source, so score 6.

Assumptions & free parameters 8 free parameters · 5 assumptions · 2 invented entities

The central claims depend on a substantial set of hand-set hardware parameters (energy per bit, TOPS, costs) and on the simulator's fidelity, none of which is independently verified. The invented PIM-AI entities are architectural proposals without measured evidence. The free parameters, especially memory energy per bit and server cost, are the primary drivers of the headline results.

free parameters (8)

PIM-AI main-memory energy per bit = 0.95 pJ/bit
Assumed for the proposed DDR5/LPDDR5 PIM chip based on UPMEM DDR4 PIM products; directly determines the claimed 10-20x energy reduction in mobile.
PIM-AI compute energy per operation = 0.4 pJ/OP
Assumed for tensor/vector units; within the range used for all compared profiles, but not measured.
PIM-AI peak throughput per chip = 5 TOPS (Table 1), 8 TFLOPs (footnote)
Assumed; the paper is inconsistent between 5 and 8, and the value affects all performance results.
Mobile SoC main-memory energy per bit = 20 pJ/bit (A17 Pro), 10 pJ/bit (Snapdragon, Dimensity)
Derived from memory technologies with a 2x interface factor; these values anchor the mobile energy comparison.
H100 main-memory energy per bit = 7 pJ/bit
Assumed HBM energy; used in cloud comparison and affects the relative energy efficiency of PIM-AI.
PIM-AI server cost = $15k per server, $60k for 4 servers
Used for the 3-year TCO calculation; no source or bill-of-materials is provided.
DGX-H100 server cost = $300k
Used for TCO calculation; no source provided, and this assumption strongly influences the 6.94x TCO ratio.
Batch sizes = Llama2-70B GQA=8: 200 H100 vs 80 PIM-AI; MHA: 46 vs 10; Mixtral similar
Chosen per hardware/model; the selection rationale is not given, and different batch sizes affect QPS comparisons.

assumptions (5)

domain assumption The LLM hardware simulator accurately estimates time, energy, and power for LLM layers
The entire analysis relies on this simulator, which is not validated against measured hardware.
domain assumption Hand-set hardware profile parameters represent real systems
PIM-AI parameters are extrapolated from UPMEM DDR4 PIM; SoC parameters are from memory specifications. No sensitivity analysis is provided.
domain assumption Constant peak performance is a sufficient model for compute and memory
Stated in Section 5.2 as a simplification that ignores real-world variations such as thermal throttling and memory contention.
domain assumption The benchmark models are representative of LLM workloads and correctly implemented in the simulator
Llama2 and Mixtral models are used with quantization schemes for mobile, but no accuracy or runtime verification of the simulated execution is given.
domain assumption Rack-unit normalization is the correct comparison basis for cloud TCO
The cloud comparison uses four PIM-AI servers (8U total) versus one DGX-H100 (8U), which is reasonable only if the goal is density-normalized cost.

invented entities (2)

PIM-AI chip
purpose: Performs GEMM/GEMV operations inside the memory chip to reduce data transfer energy and latency
No silicon exists; specifications (4 RISC-V processors, 8 TOPS, 102.4 GB/s) are design goals, not measured properties.
PIM-AI DIMM
purpose: Scales PIM-AI to 32GB modules with 1.6TB/s bandwidth and 128 TFLOPs for server deployments
Defined in Section 2.2 but not fabricated or measured; system-level attributes are assumptions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PIM-AI: A Novel Architecture for High-Efficiency LLM Inference." pith.science (2026). https://pith.science/paper/M2I6JEJK

@misc{pith2026241117309,
  author       = {Pith},
  title        = {Pith review of: PIM-AI: A Novel Architecture for High-Efficiency LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2I6JEJK}},
  note         = {Machine review of arXiv:2411.17309}
}

read the original abstract

Large Language Models (LLMs) have become essential in a variety of applications due to their advanced language understanding and generation capabilities. However, their computational and memory requirements pose significant challenges to traditional hardware architectures. Processing-in-Memory (PIM), which integrates computational units directly into memory chips, offers several advantages for LLM inference, including reduced data transfer bottlenecks and improved power efficiency. This paper introduces PIM-AI, a novel DDR5/LPDDR5 PIM architecture designed for LLM inference without modifying the memory controller or DDR/LPDDR memory PHY. We have developed a simulator to evaluate the performance of PIM-AI in various scenarios and demonstrate its significant advantages over conventional architectures. In cloud-based scenarios, PIM-AI reduces the 3-year TCO per queries-per-second by up to 6.94x compared to state-of-the-art GPUs, depending on the LLM model used. In mobile scenarios, PIM-AI achieves a 10- to 20-fold reduction in energy per token compared to state-of-the-art mobile SoCs, resulting in 25 to 45~\% more queries per second and 6.9x to 13.4x less energy per query, extending battery life and enabling more inferences per charge. These results highlight PIM-AI's potential to revolutionize LLM deployments, making them more efficient, scalable, and sustainable.

Figures

Figures reproduced from arXiv: 2411.17309 by the authors.

**Figure 2.** PIM-AI chip architecture: The logic die houses 4 RI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Execution flow of the LLM hardware simulator. The LL [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparative performance of one DGX-H100 server an [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Comparative Performance of PIM-AI on Mobile Scena [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 58 canonical work pages

[62]

Introducing Apple’s On-Device and Server Found ation Models

Apple. Introducing Apple’s On-Device and Server Found ation Models. https://machinelearning.apple.com/research/introducing-apple-foundation-models. Ac- cessed: 2024-07-03

work page 2024
[63]

Achieving High Mixtral 8x7B Performance with N VIDIA H100 Tensor Core GPUs and TensorRT- LLM

NVIDIA. Achieving High Mixtral 8x7B Performance with N VIDIA H100 Tensor Core GPUs and TensorRT- LLM. https://developer.nvidia.com/blog/achieving-high-mi xtral-8x7b-performance-with-nvidia-h100-ten Accessed: 2024-07-03. 14

work page 2024
[1]

A comprehensive overvi ew of large language models, 2024

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, S aeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overvi ew of large language models, 2024

work page 2024
[2]

A survey of large language models, 2023

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei W ang, Y upeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Y ang, Y ushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Y un Nie, and Ji-R ong Wen. A survey of large language models, 2023

work page 2023
[3]

Saddam Hossain Mukta, K aniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eu nus Ali, and Sami Azam

Mohaimenul Azam Khan Raiaan, Md. Saddam Hossain Mukta, K aniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eu nus Ali, and Sami Azam. A review on large language models: Architectures, applications, taxonomie s, open issues and challenges. IEEE Access, 12:26839– 26874, 2024

work page 2024
[4]

C. E. Shannon. Prediction and entropy of printed english . The Bell System T echnical Journal, 30(1):50–64, 1951

work page 1951
[5]

Recurrent neural net- work based language model

Tomáš Mikolov, Martin Karaﬁát, Lukáš Burget, Jan ˇCernocký, and Sanjeev Khudanpur. Recurrent neural net- work based language model. In Proc. Interspeech 2010, pages 1045–1048, 2010

work page 2010
[6]

Attention is all you need

Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkor eit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 10 PIM-AI: A Novel Architecture for High-Efﬁciency LLM Infere nce

work page 2017

Show all 62 references

[7]

Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina T outanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

2019
[8]

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Su tskever, et al. Improving language understanding by generative pre-training. 2018

2018
[9]

Exploring the limits of transfer learnin g with a uniﬁed text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee , Sharan Narang, Michael Matena, Y anqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learnin g with a uniﬁed text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

2020
[10]

Bart: Denoising sequence-to-s equence pre-training for natural language generation, translation, and comprehension, 2019

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine jad, Abdelrahman Mohamed, Omer Levy, V es Stoy- anov, and Luke Zettlemoyer. Bart: Denoising sequence-to-s equence pre-training for natural language generation, translation, and comprehension, 2019

2019
[11]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al . Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

1901
[12]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Y uan, Henri que Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Y uri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, S co...

2021
[13]

Webgpt: Browser-assisted question-answering with human feedback, 2022

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu , Long Ouyang, Christina Kim, Christopher Hesse, Shan- tanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Kar l Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. ...

2022
[14]

Llama: Open and efﬁcient foundation l anguage models, 2023

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aure lien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efﬁcient foundation l angu...

2023
[15]

Llama 2: Open foundation and ﬁne-tuned chat models, 2023

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Y asmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, ...

2023
[16]

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maa rten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Geh rmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes , Yi Tay, Noam Shazeer, V...

2023
[17]

Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shak- eri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, J onathan H

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shak- eri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, J onathan H. Clark, Laurent El Shafey, Y anping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreir a, Mark ...

2023
[18]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensc h, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample , Lucile Saulnier, Lélio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, T ho...

2023
[19]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahma d, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

2023 arXiv
[20]

Cheaper, better, faster, stronger continuing to push t he frontier of ai and making it accessible to all
[21]

Webster and Chunyu Kit

Jonathan J. Webster and Chunyu Kit. Tokenization as the initial phase in nlp. In Proceedings of the 14th Conference on Computational Linguistics - V olume 4 , COLING ’92, page 1106–1110, USA, 1992. Association for Computational Linguistics

1992
[22]

Distributed representations of words and phrases and their compositionality, 2013

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality, 2013

2013
[23]

Glove: Global vectors for word representation

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in n atural language processing (EMNLP) , pages 1532–1543, 2014

2014
[24]

Root mean square layer normalization

Biao Zhang and Rico Sennrich. Root mean square layer normalization . Curran Associates Inc., Red Hook, NY , USA, 2019

2019
[25]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. L ayer normalization, 2016

2016
[26]

vllm: Easy, fast, and cheap llm serving with p agedattention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lia nmin Zheng, Cody Y u, Joey Gonzalez, Hao Zhang, and Ion Stoica. vllm: Easy, fast, and cheap llm serving with p agedattention. 2023

2023
[27]

Dissecting batching effects in gpt inferen ce, 2023

Lequn Chen. Dissecting batching effects in gpt inferen ce, 2023

2023
[28]

Mobilellm: Optimizing sub-billion parameter language models for on-device use ca ses, 2024

Zechun Liu, Changsheng Zhao, Forrest Iandola, Chen Lai , Y uandong Tian, Igor Fedorov, Y unyang Xiong, Ernie Chang, Y angyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. Mobilellm: Optimizing sub-billion parameter language models for on-device use ca ses, 2024

2024
[29]

Octopus v2: On-device language model for super agent, 2024

Wei Chen and Zhiyuan Li. Octopus v2: On-device language model for super agent, 2024

2024
[30]

A survey on hardware accelerato rs for large language models, 2024

Christoforos Kachris. A survey on hardware accelerato rs for large language models, 2024

2024
[31]

Ene rgy and policy considerations for deep learning in nlp, 2019

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Ene rgy and policy considerations for deep learning in nlp, 2019

2019
[32]

Wm. A. Wulf and Sally A. McKee. Hitting the memory wall: i mplications of the obvious. SIGARCH Comput. Archit. News, 23(1):20–24, mar 1995

1995
[33]

Mahoney, and Kurt Keutzer

Amir Gholami, Zhewei Y ao, Sehoon Kim, Coleman Hooper, M ichael W . Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, pages 1–5, 2024

2024
[34]

From wor ds to watts: Benchmarking the energy costs of large language model inference

Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From wor ds to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extre...

2023
[35]

Me asuring and improving the energy efﬁciency of large language models inference

Mauricio Fadel Argerich and Marta Patiño-Martínez. Me asuring and improving the energy efﬁciency of large language models inference. IEEE Access, 12:80194–80207, 2024

2024
[36]

Risks and beneﬁts of large language models for the environment

Matthias C Rillig, Marlene Ågerstrand, Mohan Bi, Kenne th A Gould, and Uli Sauerland. Risks and beneﬁts of large language models for the environment. Environmental Science & T echnology, 57(9):3464–3466, 2023

2023
[37]

A short survey of viewing large languag e models in legal aspect, 2023

Zhongxiang Sun. A short survey of viewing large languag e models in legal aspect, 2023

2023
[38]

What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 2280–2292, 2022

Hannah Brown, Katherine Lee, Fatemehsadat Mireshghal lah, Reza Shokri, and Florian Tramèr. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pages 2280–2292, 2022

2022
[39]

Y ou are what you write: Preserving privacy in the era of large language models, 2022

Richard Plant, V alerio Giuffrida, and Dimitra Gkatzia . Y ou are what you write: Preserving privacy in the era of large language models, 2022

2022
[40]

Deli ver high performance ml inference with aws inferentia

Gadi Hutt, Vibhav Viswanathan, and Adam Nadolski. Deli ver high performance ml inference with aws inferentia. 2019

2019
[41]

Mm1: Methods, analysis & insights from multimo dal llm pre-training, 2024

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhan g, ...

2024
[42]

SmoothQuant: Accurate and efﬁcient post-training quantization for large language mo dels

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien D emouth, and Song Han. SmoothQuant: Accurate and efﬁcient post-training quantization for large language mo dels. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarle tt, ...

2023
[43]

Compression of generative pre-trained language models via quantizatio n, 2022

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantizatio n, 2022

2022
[44]

Mahoney, and Kurt Keutzer

Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Y ao, Michae l W . Mahoney, and Kurt Keutzer. A survey of quantization methods for efﬁcient neural network inferenc e, 2021

2021
[45]

Onebit: Towards extremely low-bit large language models, 2 024

Y uzhuang Xu, Xu Han, Zonghan Y ang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models, 2 024
[46]

The era of 1-bit llms: All large lang uage models are in 1.58 bits, 2024

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui W ang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large lang uage models are in 1.58 bits, 2024

2024
[47]

Llm-pruner: On the structural pruning of large language mod- els

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language mod- els. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, a nd S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 21702–21720. Curra...
[48]

From dense to sparse: Contrastive pruning for better pre-trained lang uage model compression

Runxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Hua ng, Songfang Huang, and Fei Huang. From dense to sparse: Contrastive pruning for better pre-trained lang uage model compression. Proceedings of the AAAI Conference on Artiﬁcial Intelligence , 36(10):11547–11555, Jun. 2022

2022
[49]

A s urvey on model compression for large language models, 2023

Xunyu Zhu, Jian Li, Y ong Liu, Can Ma, and Weiping Wang. A s urvey on model compression for large language models, 2023

2023
[50]

Distill ing the knowledge in a neural network, 2015

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill ing the knowledge in a neural network, 2015

2015
[51]

Knowledge distillation: A survey

Jianping Gou, Baosheng Y u, Stephen J Maybank, and Dache ng Tao. Knowledge distillation: A survey. Interna- tional Journal of Computer Vision , 129(6):1789–1819, 2021

2021
[52]

De Lima, Hamid Farzaneh, and Jeronimo Castrillon

Asif Ali Khan, João Paulo C. De Lima, Hamid Farzaneh, and Jeronimo Castrillon. The landscape of compute- near-memory and compute-in-memory: A research and commerc ial overview, 2024

2024
[53]

A Modern Primer on Processing in Memory, pages 171–243

Onur Mutlu, Saugata Ghose, Juan Gómez-Luna, and Rachat a Ausavarungnirun. A Modern Primer on Processing in Memory, pages 171–243. Springer Nature Singapore, Singapore, 202 3
[54]

High-speed emerging memories for ai hardwar e accelerators

Anni Lu, Junmo Lee, Tae-Hyeon Kim, Muhammed Ahosan Ul Ka rim, Rebecca Sejung Park, Harsono Simka, and Shimeng Y u. High-speed emerging memories for ai hardwar e accelerators. Nature Reviews Electrical Engineering, 1(1):24–34, 2024

2024
[56]

The breakthrough memory solutions for improved perfo rmance on llm inference

Byeongho Kim, Sanghoon Cha, Sangsoo Park, Jieun Lee, Su khan Lee, Shin-haeng Kang, Jinin So, Kyungsoo Kim, Jin Jung, Jong-Geon Lee, Sunjung Lee, Y oonah Paik, Hyeonsu Kim, Jin-Seong Kim, Won-Jo Lee, Y uhwan Ro, Y eonGon Cho, Jin Hyun Kim, JoonHo Song, Jaehoon Y u, Seung won Le...

2024
[57]

Oliveira, and Onur Mutlu

Juan Gómez-Luna, Izzat El Hajj, Ivan Fernandez, Christ ina Giannoula, Geraldo F. Oliveira, and Onur Mutlu. Benchmarking memory-centric computing systems: Analysis of real processing-in-memory hardware. In 2021 12th International Green and Sustainable Computing Confer ence (IGS...

2021
[58]

Energy efﬁciency impa ct of processing in memory: A comprehensive review of workloads on the upmem architecture

Y ann Falevoz and Julien Legriel. Energy efﬁciency impa ct of processing in memory: A comprehensive review of workloads on the upmem architecture. In Demetris Zeinali pour, Dora Blanco Heras, George Pallis, Herodotos Herodotou, Demetris Trihinas, Daniel Balouek, Patrick Diehl,...

2023
[59]

Technical report, Qualcomm, 2024

Unlocking on-device generative AI with an NPU and heter ogeneous computing. Technical report, Qualcomm, 2024

2024
[60]

Huggingface’s transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaum ond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019

1910 arXiv
[61]

Accessed: 2024-07-11

Electricity prices. Accessed: 2024-07-11

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [62]

Introducing Apple’s On-Device and Server Found ation Models

Apple. Introducing Apple’s On-Device and Server Found ation Models. https://machinelearning.apple.com/research/introducing-apple-foundation-models. Ac- cessed: 2024-07-03

work page 2024

[2] [63]

Achieving High Mixtral 8x7B Performance with N VIDIA H100 Tensor Core GPUs and TensorRT- LLM

NVIDIA. Achieving High Mixtral 8x7B Performance with N VIDIA H100 Tensor Core GPUs and TensorRT- LLM. https://developer.nvidia.com/blog/achieving-high-mi xtral-8x7b-performance-with-nvidia-h100-ten Accessed: 2024-07-03. 14

work page 2024

[3] [1]

A comprehensive overvi ew of large language models, 2024

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, S aeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overvi ew of large language models, 2024

work page 2024

[4] [2]

A survey of large language models, 2023

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei W ang, Y upeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Y ang, Y ushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Y un Nie, and Ji-R ong Wen. A survey of large language models, 2023

work page 2023

[5] [3]

Saddam Hossain Mukta, K aniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eu nus Ali, and Sami Azam

Mohaimenul Azam Khan Raiaan, Md. Saddam Hossain Mukta, K aniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eu nus Ali, and Sami Azam. A review on large language models: Architectures, applications, taxonomie s, open issues and challenges. IEEE Access, 12:26839– 26874, 2024

work page 2024

[6] [4]

C. E. Shannon. Prediction and entropy of printed english . The Bell System T echnical Journal, 30(1):50–64, 1951

work page 1951

[7] [5]

Recurrent neural net- work based language model

Tomáš Mikolov, Martin Karaﬁát, Lukáš Burget, Jan ˇCernocký, and Sanjeev Khudanpur. Recurrent neural net- work based language model. In Proc. Interspeech 2010, pages 1045–1048, 2010

work page 2010

[8] [6]

Attention is all you need

Ashish V aswani, Noam Shazeer, Niki Parmar, Jakob Uszkor eit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 10 PIM-AI: A Novel Architecture for High-Efﬁciency LLM Infere nce

work page 2017