Anatomy of the gem5 Simulator: AtomicSimpleCPU, TimingSimpleCPU, O3CPU, and Their Interaction with the Ruby Memory System

Johan S\"oderstr\"om (1); Yuan Yao (1) ((1) Uppsala University)

REVIEW 3 major objections 4 minor 14 references

Anatomy of the gem5 Simulator: AtomicSimpleCPU, TimingSimpleCPU, O3CPU, and Their Interaction with the Ruby Memory System

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that gem5's Ruby memory subsystem is the dominant execution-time cost in the AtomicSimpleCPU and TimingSimpleCPU models, mainly on the instruction-fetch path, whereas the O3 CPU spends most of its time constructing dynamic

desk verdict Worth a serious look for the gem5 community, but the sampling-count inconsistency and missing artifacts keep it from being a usable baseline yet. read the letter →

arxiv 2508.18043 v1 pith:KSUTQ3TJ submitted 2025-08-25 cs.AR

Johan Söderström (1) , Yuan Yao (1) ((1) Uppsala University) This is my paper · ORCID

classification cs.AR

keywords gem5computerarchitecturesimulationCPUmodelsRubymemorysystemprofilingperf_eventout-of-orderperformance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show where gem5 actually spends its simulation time across its three major CPU models: AtomicSimpleCPU, TimingSimpleCPU, and the out-of-order O3CPU. Using a lightweight sampling profiler built on Linux's perf_event interface, the authors partition each model's function-call tree into hardware layers and benchmark suites. Their central finding is that the Ruby memory subsystem is the largest single cost in both simple CPU models, and that this cost is concentrated in the instruction-fetch path. In the O3 CPU, Ruby takes a comparatively smaller share; the dominant costs are instead dynamic instruction construction and the pipeline stages themselves. If true, this gives gem5 developers a concrete target: optimizing Ruby's fetch-side handling for simple CPUs, and instruction-object construction for the O3 CPU.

What carries the argument

The central mechanism is the profiler itself: a Python-configurable sampler that uses the Linux perf_event_open syscall to collect callchains every 1000 ms, stores them as a nested JSON tree of function calls with sample counts, and then parses this tree using whitelist and blacklist rules to assign time to named gem5 components. The callchain tree, with inclusive sample counting, is what carries every time-partition figure in the paper. Secondary machinery is gem5's own CPU tick structure: AtomicSimpleCPU's single tick, TimingSimpleCPU's separate I-tick and D-tick events, and O3CPU's five ordered stage ticks (Fetch, Decode, Rename, IEW, Commit).

What would settle it

Run the same gem5 configuration twice with a 100 ms sampling interval instead of 1000 ms: if the per-component percentages do not reproduce, or if the total sample count does not scale roughly tenfold while the figures' y-axis values stay near one million, the sampling premise fails and the fine-grained attributions are not reliable.

Watch

Extended reading notes

Core claim

The paper establishes a component-level time budget for gem5 24.0 running x86_64 full-system simulations with the Ruby MESI_Two_Level memory system and Garnet network. By sampling callchains every 1000 ms over one-hour runs and counting each sample inclusively up the call stack, the authors show that for the AtomicSimpleCPU and TimingSimpleCPU, Ruby accounts for the largest share of execution time, primarily while serving instruction-fetch requests. In contrast, the O3 CPU spends a relatively smaller fraction of its time inside Ruby, with most time going to constructing dynamic instruction instances in the Fetch stage and to the IEW pipeline stage. The paper frames this as identifying a 'kil

Load-bearing premise

That sampling the gem5 process once every 1000 milliseconds over a single one-hour run, counting each sample inclusively up the callchain, produces unbiased time partitions at every layer the paper reports.

Editorial extensions

If this is right

Optimization efforts for gem5 should target Ruby's instruction-fetch response path when the goal is speeding up AtomicSimpleCPU and TimingSimpleCPU configurations.
For the O3 CPU, reducing the cost of dynamic instruction construction and the IEW stage's bookkeeping would have more impact than reducing Ruby time.
Increasing simulated memory from 3GB to 8GB or 16GB does not relieve the Ruby bottleneck in simple CPUs; it shifts time within Ruby from address validation to recvAtomic handling.
The profiling method, with its per-component callchain partitioning, can be applied to other gem5 components or to new CPU models to establish where their simulation time goes.
The reported breakdowns provide a baseline for verifying whether a future gem5 change actually moves time out of one layer and into another.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorially, the hottest leaves the paper identifies, such as the decoder's Reset state checking its internal cache and Ruby's h_ifetch_hit action, suggest that a software-side instruction-fetch cache for the simple CPU models could bypass a large share of Ruby overhead.
Editorially, because all runs were stopped at one wall-clock hour, workloads that slow down over time may be over-represented in the sample; repeating the same configurations with shorter sampling intervals and multiple seeds would test whether the reported component percentages are stable.
Editorially, the O3 CPU's high time in the DynInst constructor points toward allocation pooling or trimming per-instruction metadata as a testable optimization that the paper does not itself evaluate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this is a genuinely useful measurement paper, not a modeling exercise. The authors profile gem5's three CPU models against three benchmark suites and produce a component-internal time breakdown that nobody has published before. Their central finding - Ruby dominates AS and TS, mostly on the instruction-fetch path, while O3 spends most time constructing dynamic instructions and moving them through pipeline stages - is consistent across GAPBS, PARSEC, and SPEC2017. I find that claim credible, and it is exactly the kind of target an optimization effort needs.

The new part is real: prior work (Umeike et al.) asked what host hardware gem5 likes; this paper asks where gem5's own cycles go. The anatomical diagrams and the source-reading needed to categorize functions are a solid contribution. The authors also state their limitations honestly: fixed 1-hour windows, relative-time-only comparisons, no claim about total execution time. That is the right framing for a profiling study.

The soft spots are real too. The stated 1000 ms sampling interval cannot produce the 1e6-1e7 sample counts shown in the figures. With inclusive callchain counting, the root count should equal the number of samples, so a 1-hour run at 1000 ms gives about 3600, not millions. Either the interval is misreported or the counts mean something else. That is a load-bearing detail, because the deep-layer percentages (decoder FSM states, L1 MESI actions, Garnet internals) rest on the counting semantics. This is fixable, but it needs to be fixed before anyone treats the per-layer numbers as a baseline. There are also no repeat runs and no variance analysis, and the profiler, parser, and raw JSON files are not shipped, so the quantitative claims are not independently checkable right now.

None of this overturns the main qualitative picture. The Ruby-dominance for AS/TS and the buildInst/DynInst cost for O3 are visible at the coarsest level and survive the counting objection. But the fine-grained attributions should be read as provisional.

Who is this for? gem5 developers and architecture researchers who want a map of where the simulator actually spends time, plus educators who want to show students how the models fit together. It deserves a serious referee, not a desk reject, but the referee should demand artifact release and a corrected statement of the sampling semantics before the numbers are used.

Referee Report

3 major / 4 minor

Summary. The paper profiles gem5 24.0 for the AtomicSimpleCPU, TimingSimpleCPU, and O3 CPU models running with the Ruby memory subsystem (MESI_Two_Level) under benchmark workloads from GAPBS, PARSEC-3.0, and SPEC2017. Using a custom profiler built on the Linux perf_event interface, the authors collect call-chain samples over 1-hour runs and parse them into layered time-breakdown figures. The central claim is that Ruby dominates execution time in the sequential AS and TS CPUs, especially on the instruction-fetch path, while the O3 CPU spends a relatively smaller fraction of time in Ruby and most of its time in instruction-object construction and pipeline bookkeeping. The paper also provides detailed anatomical call-flow diagrams and discusses the effects of core and memory scaling on the observed breakdowns.

Significance. If the measurements are reliable, the paper delivers a useful and actionable result for the gem5 community: the simple CPU models are bottlenecked by the Ruby memory machinery on the instruction-fetch path, whereas the O3 model pays more for simulator-side instruction construction. The study's strengths are its breadth (three benchmark suites, 458 configuration/application runs, multiple core and memory settings), its source-level attribution of call chains to gem5 components, and the extensive anatomical diagrams, which have genuine educational value. The work is an external measurement study without fitted parameters, so the qualitative conclusions are not circular. However, the quantitative precision of the paper currently rests on a profiling methodology that is internally inconsistent in its reported sample counts and lacks repeat runs and raw data. The headline qualitative result may survive, but the fine-grained layer-by-layer percentages need to be re-established before the paper's quantitative claims can be accepted.

major comments (3)

[§2.2, Figs. 3.2, 4.2, 5.1] The sampling interval is stated as 1000 ms and each run as 1 hour. With inclusive call-chain counting, the root count equals the number of samples, i.e. about 3600 per run. Yet the figures report counts on the order of 1e6–1e7 per bar. This is inconsistent by roughly three orders of magnitude. Either the effective sampling interval is much shorter than stated, or the plotted values are not the sample counts defined in §2.2. Since every percentage is derived from these counts, the quantitative layer-by-layer attributions require respecification or re-execution before they can be checked.
[§2.2–2.3, Figs. 3.12–3.14, 4.44–4.46] Each configuration was run once, with no repeat runs, variance estimates, or error bars. The paper reports fine-grained percentages down to decoder FSM states and individual L1 MESI actions. At the stated 3600 samples per hour, many leaf categories would be represented by very few samples, and the figures' large counts contradict that sampling rate. Without repeat runs, confidence intervals, or the raw JSON files and parser code, the reader cannot distinguish genuine component-level differences from sampling noise. The central qualitative claim might survive, but the precise fine-grained numbers are not independently checkable.
[§3.5, §4.4] The paper interprets changes in relative percentages as effects of memory/core scaling, e.g. 'increasing memory will increase the proportional time spent in fetch'. Because all runs are limited to 1 wall-clock hour, different configurations may simulate different amounts of work, so a change in a component's percentage can reflect a change in the denominator rather than in that component's own cost. The authors acknowledge in §2.2 that total execution time effects cannot be assessed, but the scaling discussion in §3.5 and §4.4 still makes causal claims. The cross-configuration comparisons should be framed as relative-share observations only, or supplemented with simulated-instruction counts.

minor comments (4)

[Figure captions 3.36–3.41] The figures describing probeInstCommit are captioned 'Division of countCommitInst'; the captions should be corrected to name the profiled function.
[Throughout] There are a number of typos and grammatical slips, e.g. 'constast' in §1.1.1, 'exeuction' in Figure 5.86, 'inidicate' in the Chapter 4 introduction, and 'A Score' in the §2.3 label explanation.
[Figures throughout] Most stacked-bar figures lack explicit y-axis units. The labels such as '1e6' are ambiguous without stating whether the values are raw sample counts, cumulative counts, or normalized counts. This should be clarified in the methodology or in the figure axes.
[Overall presentation] The paper contains a very large number of similar stacked-bar figures. Several groups (e.g. 3.2–3.4 versus 3.5–3.7) are highly repetitive. Moving some of these to an appendix or supplement would improve readability without losing the anatomical detail.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an external measurement study; no prediction reduces to its inputs.

full rationale

The paper's claimed results are attributions of observed perf_event sample counts to gem5 components based on source-code inspection of call chains. The central claims—Ruby dominance in AS/TS instruction fetch and O3 cost concentrated in instruction construction/pipeline stages—are empirical observations of an independently built simulator, not derivations from a fitted parameter or from a self-citation. §2.2 describes a profiler built on Linux perf_event_open and a parser; §2.3 explains that categorization is based on studying gem5 source code. There is no equation in which an output is defined to be an input, no parameter fitted to a subset and then 'predicted' on a closely related quantity, and no load-bearing self-citation: the only prior profiling work discussed (Umeike et al. [13], §1.2) is external and is used as motivation/contrast, not as justification for the results. The paper's own caveat that it can only observe relative time and cannot assess total execution time, and the possible internal inconsistency in sampling counts between §2.2 (~3600 samples/hour at 1000 ms) and figure magnitudes (1e6–1e7), are methodological/validity concerns about sampling bias or reporting semantics, not circularity. They do not make the derivation equivalent to its inputs. Accordingly no circular step is identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This is a measurement study, not a derivation: there are no fitted constants, no tuned parameters, and no new entities. All numbers rest on the sampling methodology, the source-code-based categorization of functions, and the representativeness of the benchmark and configuration selection.

assumptions (5)

domain assumption perf_event callchain sampling at the stated interval yields a representative, unbiased sample of gem5's execution time distribution
Invoked in §2.2-2.3 as the basis for all percentage attributions; no validation against known workloads or ground-truth instrumentation is provided.
domain assumption The fixed 1-hour window starting at each benchmark's ROI is representative enough for cross-configuration relative-time comparisons
Stated in §2.2; the paper acknowledges it 'cannot assess the effect of total execution time'. SPEC2017 is moreover run only at 1 core / 3 GB, a selection shown in Tables 2.2-2.3.
domain assumption Categorization of gem5 functions into hardware-layer semantics (Ruby, decoder, TLB, Garnet) based on source study is correct
The parser's whitelist/blacklist configs in §2.3 encode the authors' reading of gem5 source; mis-grouping would propagate into every figure.
domain assumption The gem5 fast binary compiled with debug symbols is behaviorally and performance-wise comparable to the stock fast binary
§2.1 states they modified the fast build to include function symbols; symbol presence does not alter code generation here, but the assumption is unverified.
domain assumption Standard simulator stack: gem5 24.0 x86-64 FS mode, Ruby MESI_Two_Level, Garnet 2.0, and the benchmark ROI definitions reproduce the intended workloads
Defined in §2.1; results are specific to this configuration family, a scope the paper mostly respects but occasionally over-generalizes (e.g., 'killer component' phrasing).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Anatomy of the gem5 Simulator: AtomicSimpleCPU, TimingSimpleCPU, O3CPU, and Their Interaction with the Ruby Memory System." pith.science (2026). https://pith.science/paper/KSUTQ3TJ

@misc{pith2026250818043,
  author       = {Pith},
  title        = {Pith review of: Anatomy of the gem5 Simulator: AtomicSimpleCPU, TimingSimpleCPU, O3CPU, and Their Interaction with the Ruby Memory System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSUTQ3TJ}},
  note         = {Machine review of arXiv:2508.18043}
}

read the original abstract

gem5 is a popular modular-based computer system simulator, widely used in computer architecture research and known for its long simulation time and steep learning curve. This report examines its three major CPU models: the AtomicSimpleCPU (AS CPU), the TimingSimpleCPU (TS CPU), the Out-of-order (O3) CPU, and their interactions with the memory subsystem. We provide a detailed anatomical overview of each CPU's function call-chains and present how gem5 partitions its execution time for each simulated hardware layer. We perform our analysis using a lightweight profiler built on Linux's perf_event interface, with user-configurable options to target specific functions and examine their interactions in detail. By profiling each CPU across a wide selection of benchmarks, we identify their software bottlenecks. Our results show that the Ruby memory subsystem consistently accounts for the largest share of execution time in the sequential AS and TS CPUs, primarily during the instruction fetch stage. In contrast, the O3 CPU spends a relatively smaller fraction of time in Ruby, with most of its time devoted to constructing instruction instances and the various pipeline stages of the CPU. We believe that the anatomical view of each CPU's execution flow is valuable for educational purposes, as it clearly illustrates the interactions among simulated components. These insights form a foundation for optimizing gem5's performance, particularly for the AS, TS, and O3 CPUs. Moreover, our framework can be readily applied to analyze other gem5 components or to develop and evaluate new models.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

[1]

Garnet: A detailed on- chip network model inside a full-system simulator

Niket Agarwal, Tushar Krishna, Li-Shiuan Peh, and Niraj K Jha. Garnet: A detailed on- chip network model inside a full-system simulator. InPerformance Analysis of Systems and Software, 2009. ISPASS 2009. IEEE International Symposium on, pages 33–42. IEEE, 2009

work page 2009
[2]

The GAP benchmark suite.arXiv preprint arXiv:1508.03619, 2015

Scott Beamer, Krste Asanović, and David Patterson. The GAP benchmark suite.arXiv preprint arXiv:1508.03619, 2015

arXiv 2015
[3]

The gem5 simulator.ACM SIGARCH computer architecture news, 39(2):1–7, 2011

Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. The gem5 simulator.ACM SIGARCH computer architecture news, 39(2):1–7, 2011

work page 2011
[4]

Fix Performance Bottlenecks with Intel® VTune™ Profiler

Intel Corporation. Fix Performance Bottlenecks with Intel® VTune™ Profiler. https://www. intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html. Ac- cessed: September 11, 2024

work page 2024
[5]

gprof2dot

José Fonseca. gprof2dot. https://github.com/jrfonseca/gprof2dot, 2024. Accessed: October 14, 2024

work page 2024
[6]

Elsevier, 2011

John L Hennessy and David A Patterson.Computer architecture: a quantitative approach. Elsevier, 2011

2011
[7]

gem5 documentation.https://www.gem5.org/documentation/, 2025

Jason Lowe-Power. gem5 documentation.https://www.gem5.org/documentation/, 2025. Accessed: 2025-02-10

work page 2025
[8]

The gem5 simulator: Version 20.0+.arXiv preprint arXiv:2007.03152, 2020

Jason Lowe-Power, Abdul Mutaal Ahmad, Ayaz Akram, Mohammad Alian, Rico Amslinger, Matteo Andreozzi, Adrià Armejach, Nils Asmussen, Brad Beckmann, Srikant Bharadwaj, et al. The gem5 simulator: Version 20.0+.arXiv preprint arXiv:2007.03152, 2020

arXiv 2007

Show all 14 references

[9]

URLhttps://man7.org/linux/ man-pages/man2/syscalls.2.html

Michael Kerrisk.Linux System Calls Manual Page, 2024. URLhttps://man7.org/linux/ man-pages/man2/syscalls.2.html. Accessed: 2025-01-13

2024
[10]

Automatically characterizing large scale program behavior

Sherwood,TimothyandPerelman,ErezandHamerly,Greg andCalder,Brad. Automatically characterizing large scale program behavior. InInternational Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 45—57, 2002

2002
[11]

SPEC 2017 documentation, 2017

SPEC. SPEC 2017 documentation, 2017. URLhttps://www.spec.org/cpu2017/Docs/. Accessed: 2024-10-29

2017
[12]

An efficient algorithm for exploiting multiple arithmetic units.IBM Journal of research and Development, 11(1):25–33, 1967

Robert M Tomasulo. An efficient algorithm for exploiting multiple arithmetic units.IBM Journal of research and Development, 11(1):25–33, 1967

1967
[13]

Profiling gem5 simulator

Johnson Umeike, Neel Patel, Alex Manley, Amin Mamandipoor, Heechul Yun, and Mo- hammad Alian. Profiling gem5 simulator. In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 103–113. IEEE, 2023. 89

2023
[14]

Xusheng Zhan, Yungang Bao, Christian Bienia, and Kai Li. PARSEC3. 0: A multicore benchmark suite with network stacks and SPLASH-2X.ACM SIGARCH Computer Archi- tecture News, 44(5):1–16, 2017. 90

2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Garnet: A detailed on- chip network model inside a full-system simulator

Niket Agarwal, Tushar Krishna, Li-Shiuan Peh, and Niraj K Jha. Garnet: A detailed on- chip network model inside a full-system simulator. InPerformance Analysis of Systems and Software, 2009. ISPASS 2009. IEEE International Symposium on, pages 33–42. IEEE, 2009

work page 2009

[2] [2]

The GAP benchmark suite.arXiv preprint arXiv:1508.03619, 2015

Scott Beamer, Krste Asanović, and David Patterson. The GAP benchmark suite.arXiv preprint arXiv:1508.03619, 2015

arXiv 2015

[3] [3]

The gem5 simulator.ACM SIGARCH computer architecture news, 39(2):1–7, 2011

Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. The gem5 simulator.ACM SIGARCH computer architecture news, 39(2):1–7, 2011

work page 2011

[4] [4]

Fix Performance Bottlenecks with Intel® VTune™ Profiler

Intel Corporation. Fix Performance Bottlenecks with Intel® VTune™ Profiler. https://www. intel.com/content/www/us/en/developer/tools/oneapi/vtune-profiler.html. Ac- cessed: September 11, 2024

work page 2024

[5] [5]

gprof2dot

José Fonseca. gprof2dot. https://github.com/jrfonseca/gprof2dot, 2024. Accessed: October 14, 2024

work page 2024

[6] [6]

Elsevier, 2011

John L Hennessy and David A Patterson.Computer architecture: a quantitative approach. Elsevier, 2011

2011

[7] [7]

gem5 documentation.https://www.gem5.org/documentation/, 2025

Jason Lowe-Power. gem5 documentation.https://www.gem5.org/documentation/, 2025. Accessed: 2025-02-10

work page 2025

[8] [8]

The gem5 simulator: Version 20.0+.arXiv preprint arXiv:2007.03152, 2020

Jason Lowe-Power, Abdul Mutaal Ahmad, Ayaz Akram, Mohammad Alian, Rico Amslinger, Matteo Andreozzi, Adrià Armejach, Nils Asmussen, Brad Beckmann, Srikant Bharadwaj, et al. The gem5 simulator: Version 20.0+.arXiv preprint arXiv:2007.03152, 2020

arXiv 2007