REVIEW 3 major objections 5 minor 25 references
Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that LLM decoding on a CPU repeatedly walks the model's vocabulary, weights, and logits once per generated token, and that a cache policy tuned to that long re-reference interval could significantly speed up inference.
desk verdict Useful simulation data, but the decoder trace only covers token-data and logits, not the full forward pass over weights, so the paper's central claim about LLM memory behavior does not follow from its own data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the per-token memory-access cycle: a histogram showing that almost every address is touched exactly once per generated token, with a stride of about 8 million cycles between visits to the same address. That cycle defines the working set (token data, vocabulary, weights, logits) and its re-reference interval, the time between two uses of the same cache line. The argument connects this observed cycle to simulated cache behavior: the L2 cache cannot hold the working set and is helped by next-line prefetching, while the last-level cache holds it and is improved by DRRIP, a replacement policy that predicts each line's re-reference interval and preferentially evicts lines predicted to be needed soon rather than far in the future.
What would settle it
Run the same 0.5B model on a real CPU with hardware performance counters and compare the measured L2/LLC miss rates and the distribution of per-address access counts against the simulation; if the real L2 miss rate is not near 70% under LRU-like replacement, or if roughly 98% of addresses are not accessed exactly once per generated token, the paper's central characterization is falsified.
Extended reading notes
Core claim
The paper's central discovery is that the memory footprint of LLM decoding on a CPU is dominated by a slow, regular recycle of the model's vocabulary, weights, and logits. Almost every address (98.06%) is accessed exactly once per generated token, so with 128 tokens it is accessed 128 times; the minority of single-access and high-frequency addresses make up the rest. The token-data array alone spans about 3.65 MB, and the stride between successive accesses to the same address is roughly 8 million cycles, defining a token-generation period. At the L2 cache this working set overflows capacity, producing a 69.884% miss rate that a simple next-line prefetcher reduces to 30.786%; at the 4 MB LLC the working set mostly fits, with LRU missing only 0.065%, and DRRIP, which protects lines predicted to have distant re-references, cuts that to 0.018%. The paper concludes that a replacement policy exploiting the long re-reference interval, together with a stride-aware prefetcher for the L2, would greatly improve CPU inference performance.
Load-bearing premise
The load-bearing premise is that the traced memory addresses and the simulated cache behavior on one small model, one prompt, and 128 tokens faithfully represent how real CPU hardware would execute LLM inference; if the mapping from addresses to program variables or the simulator is wrong, the working-set and prefetch conclusions would not transfer to actual CPUs.
Editorial extensions
If this is right
- A replacement policy that keeps lines with long re-reference intervals, such as DRRIP, can cut last-level cache misses by about 3.6x (from 0.065% to 0.018%) on this workload.
- The L2 cache, not the last-level cache, is the main capacity bottleneck in the memory-bound decode phase; next-line prefetching halves its miss rate (from 69.9% to 30.8%), so even a low-cost prefetcher at L2 is an immediate win.
- The roughly 8-million-cycle re-reference period gives a prefetcher or replacement policy a natural time base: it can time evictions and prefetches to the token-generation cadence.
- The low LLC miss rate is specific to a 0.5B model and 128 tokens; as model size or token count grows, the working set will exceed the 4MB LLC and miss rates will rise sharply.
- Stride prefetchers that track page-by-page deltas outperform a single next-line stride at the L1D cache, indicating that the spatial access pattern differs across cache levels.
Reading between the lines
- The paper does not test whether the once-per-token access pattern persists across model families, prompt lengths, batch sizes, or thread counts; a multi-model trace would show whether the 98% peak is a general LLM property or an artifact of this setup.
- Because the access-count peak is keyed to the number of tokens, a replacement policy that counts token cycles rather than raw cache cycles may generalize better than DRRIP; this is an extension the paper leaves implicit.
- The heuristic address-to-variable mapping is the most fragile link; compiler-based or binary-instrumentation-based variable annotation could confirm which addresses are vocabulary, weights, and logits.
- A real-hardware validation using performance counters would test whether the simulated L2 miss-rate gap (69.9% to 30.8%) appears on actual CPUs, which would determine whether the proposed cache changes are practical.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper characterizes the memory access patterns of a 0.5B-parameter Qwen model running under llama.cpp on a CPU. Using an Intel Pin/ChampSim trace of a single prompt generating 128 tokens with one thread, the author simulates several prefetchers (Next Line, Berti, Bingo, IPCP, IP Stride, SPP) and replacement policies (LRU, SRRIP, DRRIP, SHiP), and analyzes L1D/L2C/LLC address frequency and stride histograms. The headline findings are that 98.06% of addresses are accessed exactly 128 times, that the majority of memory accesses come from vocabulary, weight, and logit accesses with long re-reference intervals, and that cache prefetchers or replacement policies exploiting this behavior could greatly improve CPU inference performance.
Significance. CPU-only LLM inference is a practically relevant problem, and a careful trace-based characterization could inform future cache designs. The paper describes its methodology in enough detail to be replicable in principle, uses public baseline prefetchers and replacement policies, and reports simulator parameters and commands. However, the significance is heavily qualified by the narrow experimental basis (one model, one prompt, 128 tokens, one thread, no hardware validation) and by a serious mismatch between the reported address statistics and the model's weight footprint; as presented, the central claim is not supported by the data.
major comments (3)
- [§2.2, §3.2 (Table 3.5)] The trace statistics are inconsistent with a full decoder pass over a 0.5B-parameter fp16 model. The model contains roughly 1 GB of weights, i.e., about 16 million 64-byte cache lines; reading every weight once per generated token would yield at least 2 billion L1D accesses over 128 tokens and more than 16 million unique addresses for weights alone. Table 3.5 reports only 309,999 unique addresses and Table 3.1 reports 477,321,401 total L1D accesses. The 303,875 addresses accessed exactly 128 times are close to the size of the token-data array (151,936 elements) plus a logits array described in §4, suggesting that the traced window or the address analysis captured the output/logit stage rather than the full transformer forward pass. Consequently, the statement in §5.2 that 'the majority of memory accesses in an LLM are due to accessing the vocabulary of the model, updating its weights and logits' is not supported by the reported data.
- [§5.2 vs §3.1 (Tables 3.1–3.4)] The claim that a replacement policy or prefetcher 'could greatly improve performance' is not supported by the paper's own simulation results. In Table 3.1, the best L1D prefetcher raises IPC from 1.47 to 1.622 and reduces misses from 4.056% to 3.176%. In Table 3.2, next-line prefetching cuts the L2C miss rate from 69.884% to 30.786% but IPC moves only from 1.47 to 1.58. In Tables 3.3 and 3.4, DRRIP lowers the LLC miss rate from 0.065% to 0.018% with IPC unchanged at 1.47. These are modest improvements, not a demonstration of large inference-speed gains, so the conclusion overstates the impact even for the trace that was simulated.
- [§3.2, §5.2] The '98.06%' statistic is computed over unique addresses, not over memory accesses. The 303,875 addresses accessed exactly 128 times generate approximately 38.9 million accesses, which is only about 8% of the 477.3 million total L1D accesses in Table 3.1; the remaining majority of accesses go to a small number of stack variables. Thus the phrasing 'the majority of memory accesses ... are due to accessing the vocabulary ...' conflates address population with traffic volume. The authors should report access-count shares and rephrase the conclusion accordingly.
minor comments (5)
- [Throughout] There are numerous typos and formatting slips: 'tranlsation', 'recieved', 'comperable', 'ALSR' for ASLR, 'formant' for format, 'prefeched' for prefetched, and the missing comma in Table 3.6 ('86563339').
- [§3.2] The text says 'As shown in Figure 3.5, 98.06% addresses are accessed exactly 128 times,' but Figure 3.5 is a stride histogram; the frequency statistic is in Table 3.5.
- [§4] The statement that 'Figure 3.7 and 3.8 are identical' is confusing, since they are different plots; the intended comparison needs a clearer explanation or a corrected reference.
- [§2.1.3, §5.1] The mapping of traced addresses to source-level variables relies on a heuristic GDB/heap-band analysis; the authors acknowledge this in §5.1, but an error analysis or a validation on a known allocation would strengthen the address-to-variable conclusions.
- [§2.2.1] Table 2.3 reports the decoder phase as 1,544,850,000 instructions, while the text says 1,344,850,000 simulated instructions after warmup; the relationship between these numbers should be stated consistently.
Circularity Check
No circularity: empirical trace study with external baselines; conclusions are interpretation, not derived from inputs by construction.
full rationale
The paper's chain is empirical rather than derivational. A real llama.cpp/Qwen execution is traced with Intel Pin, and the resulting trace is simulated in ChampSim under public prefetcher and replacement-policy baselines. No parameter is fitted to the trace and then renamed as a prediction: the 98.06% figure is an observed count of unique addresses accessed exactly 128 times, and the prefetcher/replacement results are external algorithms evaluated on that trace. The proposed cache-policy implication is a qualitative interpretation of the measured working-set and re-reference behavior, not a quantity that reduces to its own input by construction. The only mild in-sample aspect is that the same trace both reveals the pattern and motivates the optimization, but that is interpretation, not circular derivation. Self-citations (e.g., ChampSim [25] and SPP [21]) are tool/baseline citations and are not load-bearing for the central claim. The skeptic's concern that the trace may not cover the full weight footprint is a validity and completeness issue, not a circularity issue. No circular step is present.
Assumptions & free parameters
assumptions (4)
- domain assumption ChampSim's simulated hierarchy (Table 2.1) approximates real CPU cache behavior.
- domain assumption Intel Pin instrumentation plus single-thread execution does not change the memory access pattern.
- ad hoc to paper The GDB and heap-band mapping assigns traced addresses to token data and logits correctly.
- ad hoc to paper One 0.5B model, one prompt, 128 tokens, and one thread represent LLM CPU inference.
Cite this review
Pith. "Pith review of Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts." pith.science (2026). https://pith.science/paper/JKZ7YYWQ
@misc{pith2026250601827,
author = {Pith},
title = {Pith review of: Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts},
year = {2026},
howpublished = {\url{https://pith.science/paper/JKZ7YYWQ}},
note = {Machine review of arXiv:2506.01827}
}
read the original abstract
As machine learning algorithms are shown to be an increasingly valuable tool, the demand for their access has grown accordingly. Oftentimes, it is infeasible to run inference with larger models without an accelerator, which may be unavailable in environments that have constraints such as energy consumption, security, or cost. To increase the availability of these models, we aim to improve the LLM inference speed on a CPU-only environment by modifying the cache architecture. To determine what improvements could be made, we conducted two experiments using Llama.cpp and the QWEN model: running various cache configurations and evaluating their performance, and outputting a trace of the memory footprint. Using these experiments, we investigate the memory access patterns and performance characteristics to identify potential optimizations.
Figures
Figures from the paper (12 more)
Reference graph
Works this paper leans on
-
[1]
Applying deep learning to the cache replacement problem,
Z. Shi, X. Huang, A. Jain, and C. Lin, “Applying deep learning to the cache replacement problem,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Mi- croarchitecture, MICRO ’52, (New York, NY , USA), p. 413–425, Association for Computing Machinery, 2019
work page 2019
-
[2]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...
work page 2020
-
[3]
Bert: Pre-training of deep bidirec- tional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. N. Toutanova, “Bert: Pre-training of deep bidirec- tional transformers for language understanding,” 2018
work page 2018
- [4]
-
[5]
J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y . Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y . Zhang, ...
work page 2023
-
[6]
Understanding Performance Implications of LLM Inference on CPUs ,
S. Na, G. Jeong, B. H. Ahn, J. Young, T. Krishna, and H. Kim, “ Understanding Performance Implications of LLM Inference on CPUs ,” in 2024 IEEE International Symposium on Work- load Characterization (IISWC) , (Los Alamitos, CA, USA), pp. 169–180, IEEE Computer Society, Sept. 2024
work page 2024
-
[7]
Nomad-attention: Efficient llm inference on cpus through multiply-add-free attention,
T. Zhang, J. W. Yi, B. Yao, Z. Xu, and A. Shrivastava, “Nomad-attention: Efficient llm inference on cpus through multiply-add-free attention,” 2024. 32
work page 2024
-
[8]
Fiddler: Cpu-gpu orchestration for fast infer- ence of mixture-of-experts models,
K. Kamahori, Y . Gu, K. Zhu, and B. Kasikci, “Fiddler: Cpu-gpu orchestration for fast infer- ence of mixture-of-experts models,” 2024
work page 2024
Show all 25 references
-
[9]
Ai and memory wall,
A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “Ai and memory wall,” 2024
2024
-
[10]
Intel SGX explained
V . Costan and S. Devadas, “Intel SGX explained.” Cryptology ePrint Archive, Paper 2016/086, 2016
2016
-
[11]
A review on tinyml: State-of-the-art and prospects,
P. P. Ray, “A review on tinyml: State-of-the-art and prospects,”Journal of King Saud Univer- sity - Computer and Information Sciences , vol. 34, no. 4, pp. 1595–1623, 2022
2022
-
[12]
Evaluating the energy efficiency of deep convo- lutional neural networks on cpus and gpus,
D. Li, X. Chen, M. Becchi, and Z. Zong, “Evaluating the energy efficiency of deep convo- lutional neural networks on cpus and gpus,” in 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (Social- Com), Sustainable Com...
2016
-
[13]
A comprehensive survey on tinyml,
Y . Abadade, A. Temouden, H. Bamoumen, N. Benamar, Y . Chtouki, and A. S. Hafid, “A comprehensive survey on tinyml,”IEEE Access, vol. 11, pp. 96892–96922, 2023
2023
-
[14]
Performance analysis and cpu vs gpu comparison for deep learn- ing,
E. BUBER and B. DIRI, “Performance analysis and cpu vs gpu comparison for deep learn- ing,” in 2018 6th International Conference on Control Engineering & Information Technol- ogy (CEIT), pp. 1–6, 2018
2018
-
[15]
Benchmarking tinyml systems: Challenges and direction,
C. R. Banbury, V . J. Reddi, M. Lam, W. Fu, A. Fazel, J. Holleman, X. Huang, R. Hurtado, D. Kanter, A. Lokhmotov, D. Patterson, D. Pau, J. sun Seo, J. Sieracki, U. Thakker, M. Ver- helst, and P. Yadav, “Benchmarking tinyml systems: Challenges and direction,” 2021
2021
-
[16]
Temporal prefetching without the off-chip metadata,
H. Wu, K. Nathella, J. Pusdesris, D. Sunwoo, A. Jain, and C. Lin, “Temporal prefetching without the off-chip metadata,” in Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’52, (New York, NY , USA), p. 996–1008, Associ- ation for Comp...
2019
-
[17]
Stride directed prefetching in scalar processors,
J. W. Fu, J. H. Patel, and B. L. Janssens, “Stride directed prefetching in scalar processors,” ACM SIGMICRO Newsletter, vol. 23, no. 1-2, pp. 102–110, 1992
1992
-
[18]
Berti: A per-page best-request-time delta prefetcher,
A. Ros, “Berti: A per-page best-request-time delta prefetcher,” 2019
2019
-
[19]
Bouquet of instruction pointers: Instruction pointer classifier- based spatial hardware prefetching,
S. Pakalapati and B. Panda, “Bouquet of instruction pointers: Instruction pointer classifier- based spatial hardware prefetching,” in 2020 ACM/IEEE 47th Annual International Sympo- sium on Computer Architecture (ISCA), pp. 118–131, 2020
2020
-
[20]
Accurately and maximally prefetching spatial data access patterns with bingo,
M. Bakhshalipour, M. Shakerinava, P. Lotfi-Kamran, and H. Sarbazi-Azad, “Accurately and maximally prefetching spatial data access patterns with bingo,” 2019
2019
-
[21]
Path confi- dence based lookahead prefetching,
J. Kim, S. H. Pugsley, P. V . Gratz, A. N. Reddy, C. Wilkerson, and Z. Chishti, “Path confi- dence based lookahead prefetching,” in 2016 49th Annual IEEE/ACM International Sympo- sium on Microarchitecture (MICRO), pp. 1–12, 2016
2016
-
[22]
High performance cache replacement using re-reference interval prediction (rrip),
A. Jaleel, K. B. Theobald, S. C. Steely, and J. Emer, “High performance cache replacement using re-reference interval prediction (rrip),” inProceedings of the 37th Annual International Symposium on Computer Architecture , ISCA ’10, (New York, NY , USA), p. 60–71, Associ- ation...
2010
-
[23]
Ship: signature-based hit predictor for high performance caching,
C.-J. Wu, A. Jaleel, W. Hasenplaugh, M. Martonosi, S. C. Steely, and J. Emer, “Ship: signature-based hit predictor for high performance caching,” in Proceedings of the 44th An- nual IEEE/ACM International Symposium on Microarchitecture , MICRO-44, (New York, NY , USA), p. 430–...
2011
-
[24]
A survey of cache bypassing techniques,
S. Mittal, “A survey of cache bypassing techniques,” Journal of Low Power Electronics and Applications, vol. 6, no. 2, 2016
2016
-
[25]
The championship simulator: Architectural simulation for education and compe- tition,
N. Gober, G. Chacon, L. Wang, P. V . Gratz, D. A. Jimenez, E. Teran, S. Pugsley, and J. Kim, “The championship simulator: Architectural simulation for education and compe- tition,” 2022. 34
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.