Scalable Software Testing in Fast Virtual Platforms: Leveraging SystemC, QEMU and Containerization

Jan Henrik Weinstock; Lukas J\"unger; Tim Kraus

REVIEW 5 major objections 5 minor 9 references

Scalable Software Testing in Fast Virtual Platforms: Leveraging SystemC, QEMU and Containerization

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Containerized QEMU/SystemC virtual platforms can cut pre-silicon test setup from days to hours.

desk verdict A useful experience report on containerizing a QEMU/SystemC virtual platform; the TTE improvement claim is plausible but rests on an anecdotal comparison, not measured data. read the letter →

arxiv 2506.10624 v1 pith:JP56PQ4D submitted 2025-06-12 cs.SE cs.AR

Lukas Jünger , Jan Henrik Weinstock , Tim Kraus This is my paper · ORCID

classification cs.SEcs.AR

keywords virtualplatformsSystemCTLM-2.0QEMUQBoxVCMLcontainerizationDockerpre-silicontesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the real bottleneck in pre-silicon software testing is often environment setup rather than simulation speed: tools, models, and dependencies have to be assembled, versions must match, and commercial licenses limit how many developers can run simulations at once. The proposed answer is to package a virtual platform as a container image, with the platform itself built from open-source components: QEMU supplies the CPU model, QBox connects it to the SystemC TLM-2.0 standard, and VCML provides peripheral and interconnect building blocks. In an AI accelerator case study, the authors report that this containerized platform reached significantly reduced time-to-execution: initial setup took hours instead of days, repeated per-user setup was minimal, and simulation ran faster than half of real time. The broader claim is that containerization plus license-free simulation makes pre-silicon testing scalable in the same way cloud-based continuous integration scales ordinary software testing.

What carries the argument

The machine that carries the argument is a three-layer integration. QBox is the bridge that exposes QEMU CPU models as SystemC TLM-2.0 modules, so the fast, dynamically translated instruction set simulator can talk to standard transaction-level hardware models. VCML supplies reusable SystemC components, including register interfaces, interconnect protocols, UART and trace output, so the rest of the platform does not have to be built from scratch. Docker is the packaging layer: the whole VP, with all dependencies, is captured in an image built from a Dockerfile, and SUNRISE provides a RESTful API that configures, launches, and collects results from these containers. The named piece that makes the approach work is therefore the containerized QBox-VCML-QEMU combination, with SUNRISE as the orchestration layer that turns it into a scalable test farm.

What would settle it

Reproduce the AI accelerator workload on a fresh machine using both a commercial simulation tool and the containerized QBox-VCML-QEMU platform, with setup performed by someone new to each environment, and record the time from start to first successful test run. If the containerized path is not faster, or if the commercial path matches it, the paper's TTE claim is falsified for that environment.

Watch

Extended reading notes

Core claim

The authors claim that a virtual platform assembled from open-source pieces and delivered as a Docker image can replace a commercial, seat-licensed simulation environment for early automotive software development without sacrificing usefulness. Their case study wraps a QEMU ARM core, integrated into SystemC TLM-2.0 through QBox, together with VCML peripherals and a custom AI accelerator model, and runs it under the SUNRISE container orchestration framework. The reported result is that the containerized VP is portable across platforms, runs headlessly and in parallel, and achieves a significant reduction in time-to-execution compared with the authors' prior experience using off-the-shelf commercial simulation tools, with the additional benefit that no seat licenses are needed. The paper frames this as a recipe: simulation providers own the environment inside the container, users build or pull the image, and the same image feeds automated regression runs and interactive debugging alike.

Load-bearing premise

The central TTE comparison rests on the authors' undocumented prior experience with commercial simulation setups, so the headline improvement is only as solid as that unreported baseline.

Editorial extensions

If this is right

Pre-silicon regression tests can run headlessly and in parallel, with each containerized VP isolated from the others, so scaling is limited by compute capacity rather than by license seats.
A VP can be distributed as source plus a Dockerfile, letting every user build the identical environment and inspect what goes into it, instead of receiving a black-box binary.
Per-user setup shrinks to near zero because the built image is reused from a local registry; the paper reports repeated effort as minimal after the one-time image build.
Interactive debugging remains available inside the container through QEMU's GDB server over TCP, so the containerized flow does not force developers into batch-only work.
The same containerized workflow is the intended basis for continuous integration and build/test chains in automotive software development.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension the paper does not report: benchmarking throughput versus number of parallel containers on a fixed host would separate the containerization benefit from the underlying simulator's speed.
The paper's own observation that timing accuracy is concentrated in the peripheral, interconnect, and memory models suggests the recipe should transfer to other CPU-centric workloads, but the QEMU model's lack of detailed timing remains the ceiling for timing-sensitive analyses.
If the time-to-execution result generalizes, virtual platforms could be provisioned like cloud CI runners: teams would spin up simulator containers per test job and tear them down afterward, removing the hardware-provisioning bottleneck from pre-silicon development.
A controlled comparison with a documented commercial-tool baseline would tell whether the TTE gain comes from containerization, from the open-source models, or from the specific workload; the paper's comparison point is the authors' prior experience.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's my take on arXiv:2506.10624.

What's actually new: a specific case study of an AI-accelerator virtual platform built from QEMU + QBox + VCML, wrapped in Docker, and managed through the authors' SUNRISE framework. The combination of these open-source pieces into a containerized, headless test flow for pre-silicon automotive software is a practical contribution. The paper describes the architecture clearly and honestly notes where the QEMU model's abstraction is adequate and where it isn't.

It does several things well. It spells out the workflow: Docker image build on user side, configuration via REST API, GDB access into the container, artifact retrieval. The authors explicitly acknowledge that the low-level software bring-up effort could be counted into TTE and that they exclude it. That kind of honesty is rare and should be credited. The related-work section covers QEMU/SystemC integration attempts and gives proper credit to QBox and VCML.

The soft spots are in the evaluation. The central claim in the conclusion—that TTE was 'significantly reduced'—is supported only by a qualitative comparison to the authors' 'previous experience' with commercial tools. There's no baseline data, no measurement protocol, no actual numbers for the 'few hours' vs 'days' comparison. The setup cost of the Docker engine itself is not counted (the paper mentions it was already present). The scalability claim is asserted but never measured: no parallel speedup, no container overhead, no cloud deployment results. The paper is also self-referential: SUNRISE, VCML, and QBox all come from the authors' own work or their company. That's not a flaw by itself, but it does mean the reader is being asked to trust an evaluation in which the authors are also the vendors.

The paper's own framing is as an experience report, and it is honest about that. If the venue treats it as such, it deserves peer review—the case study is real, the engineering is sensible, and practitioners would find the description useful. But a referee should insist either on measured data (even a simple table of setup times and parallel scaling) or on toned-down claims. As written, the headline quantitative conclusion is not established.

I'd bring this to a reading group only if someone is specifically interested in containerized VP workflows. I wouldn't cite it in my own work in the next year. That said, I'd send it to review: the topic is relevant, the paper is coherent, and a good reviewer could push it into a useful form.

Best.

Referee Report

5 major / 5 minor

Summary. The paper proposes an open-source, containerized virtual platform (VP) approach for pre-silicon embedded software testing: QEMU CPU models are integrated into SystemC TLM-2.0 via QBox, peripheral models are taken from VCML, and the whole simulation environment is encapsulated in Docker containers managed by the SUNRISE framework. A case study on an automotive AI accelerator is presented, and the authors claim that this approach significantly reduces Time to Execution (TTE), improves portability, and enables scalable, parallel, headless test execution. The technical architecture is described at the block-diagram level, and the evaluation is based on the authors' practical experience in Section V.C rather than on controlled measurements.

Significance. If the TTE and scalability claims were substantiated with quantitative evidence, the paper would interest practitioners in embedded software verification: it combines well-known open-source components (QEMU, SystemC TLM-2.0, VCML) with containerization and a REST-based management framework, potentially eliminating seat-license constraints and enabling cloud-based parallel test runs. The concrete recipe using a Dockerfile and publicly available models has practical reuse value. However, the central quantitative conclusion is currently supported only by anecdotal recollection, and the scalability claims lack any measured data. The significance of the contribution is therefore conditional on the evidence being upgraded to a level that can be independently checked.

major comments (5)

[Section V.C and Section VI] The claim that "Time to Execution (TTE) was significantly reduced" (Section VI) is based on an undocumented comparison to the authors' "previous experience" with off-the-shelf commercial simulation tools (Section V.C). No baseline setup time is defined, no measurement protocol is described, and no data points are reported. To make this load-bearing conclusion supportable, the paper must replace the recollection with a reproducible measurement, for example, measuring the time from a clean environment to the first executed regression for both the containerized open-source flow and a defined baseline flow, with a stated project scope, machine specifications, and several repetitions.
[Section V.C] The paper itself acknowledges that "The significant time spent on this could be counted into the TTE" for creating the low-level software project, boot code, and toolchain settings, yet this cost is excluded from the reported "few hours" initial setup. This exclusion biases the comparison in favor of the proposed approach. The authors should either include this cost in the reported TTE figures or report separate numbers with and without that cost, and state explicitly which costs are included for both the containerized approach and the commercial baseline.
[Section V.C and Abstract] The scalability claim -- that containerization "enables cloud deployment for fast, parallelized test execution" (Abstract) -- is asserted without experimental support. No measurements of parallel speedup, container overhead, resource usage, or number of concurrent instances are reported. If scalability is a central contribution, the paper should include experiments with, for example, 1, 2, 4, 8, and 16 concurrent simulation containers, reporting wall-clock time and throughput. If such experiments are outside the scope of the current study, the claims should be softened to state that the architecture "supports" parallel deployment rather than that scalability was demonstrated.
[Section V.C] The performance statement that "The simulation performance was faster than half of real-time for the workloads tested" is too vague to verify or reproduce. The paper does not report the simulated workload size (e.g., simulated seconds or instruction count), the host hardware, the container runtime, or the variance across runs. This should be replaced with a small benchmark table showing wall-clock time versus simulated time, host configuration, and ideally a comparison against a baseline VP configuration.
[Sections III and V] The evaluation is entirely self-referential: the SUNRISE framework (reference [11]), the VCML model library (reference [2]), and the QBox integration are the authors' own components, and the positive TTE results are reported by the same team from their own use of these components. Self-use is not itself a flaw, but it limits the generalizability of the conclusions. Independent validation, or at least a clearly documented reproduction by a user who did not co-develop the framework, would substantially strengthen the paper's claims about portability and ease of use.

minor comments (5)

[Section IV] QBox is introduced with the sentence "QBox enables QEMU/SystemC integration" but the reader is not pointed to reference [8] until later; cite [8] at the first mention of QBox.
[Section V.C] The text contains several typos, including "which than can be re-used" and "the QEMU model as proven to be a good choice", and reference [3] has a malformed author entry ("G Bellard, Fabrice."). These should be corrected in revision.
[Section V.C] The phrase "few hours" for initial setup is imprecise; if the metric is reported, provide a range or a more granular breakdown (e.g., dependency download, network build, user-side build) so that the claim is interpretable.
[Section VI] The conclusion states that the environment "ensuring compatibility across various platforms", but the paper does not test cross-platform compatibility; either add a compatibility test (e.g., Linux distributions, container runtimes) or soften the wording to "aims to ensure".
[Figures 1 and 2] The figure captions are present but the figures themselves are not included in the text provided to the reviewer; in a published version, ensure that Figure 1 and Figure 2 clearly depict the containerized runtime architecture and the VP block diagram, since the written description depends on them.

Circularity Check

2 steps flagged · score 6.0 of 10

TTE-reduction claim is a self-referential comparison to an undocumented 'previous experience' baseline; SUNRISE/VCML self-citations are load-bearing for the scalability claim.

fitted input called prediction [Section V.C 'Practical Experience' and Section VI 'Conclusion']
"With the conventional approach in the past, setup times were in the range of days for initial setups of a new simulation technology in a project and repeated efforts in the region of half a working day for every new user. Through the containerized approach and because in the context of this work a docker engine was already set up in the development environment, the initial preparation and build took only few hours and the repeated effort is minimal."

The claimed TTE reduction ('TTE was significantly reduced') is the output of comparing two self-reported recollections: the authors' 'previous experience' baseline of days and their current 'few hours'. No measurement protocol, dataset, or independent source defines the baseline, and the paper itself concedes that the low-level software setup effort 'could be counted into the TTE' yet is excluded. The conclusion is therefore not an independent empirical finding; it is a direct restatement of the authors' chosen baseline, making the output determined by the input recollection rather than by measured evidence.
self citation load bearing [Section III 'Scalable Execution Framework' and Section V.B 'Software Development Flow']
"In this work, we are making use of the SUNRISE (Scalable Unified RESTful Infrastructure for System Evaluation) framework [11]."

SUNRISE is the central infrastructure of the proposed approach and of the case-study demonstration; its API and orchestration capabilities are asserted only through reference [11], whose authors (T. Kraus, A. Sauer) include a co-author of the present paper. The paper provides no independent description, code, or benchmark of SUNRISE's behavior. The 'highly portable and scalable simulation environment' conclusion thus rests in part on a self-citation chain for the framework that produced the reported results, rather than on an external validation within this paper.

full rationale

The paper's central quantitative result, 'Time to Execution (TTE) was significantly reduced' (Section VI), is supported only by Section V.C's comparison against the authors' 'previous experience' with commercial tools. The baseline setup times (days; half-day per user) are not measured, not described, and not independently verifiable; the treatment times (few hours) were obtained in an environment where a Docker engine was already installed and an archived registry existed. The paper itself notes that low-level software setup effort 'could be counted into the TTE' yet is excluded. Thus the reported reduction is a direct function of the authors' chosen baseline and is not an independent finding. Additionally, the scalable framework SUNRISE used in the case study is introduced solely through reference [11] by co-author T. Kraus; its functionality is load-bearing for the claim of a 'highly portable and scalable simulation environment' but is not independently validated in this paper. These issues make the evaluation self-referential: the components (VCML, SUNRISE) and the baseline are all supplied by the authors. The paper nevertheless contains independent content — QEMU is external, and the containerization approach is standard — so the circularity is partial rather than total, warranting a score of 6.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new fitted parameters or theoretical entities. Its central claims rest on several domain assumptions about timing accuracy and container overhead, and on the validity of the authors' own SUNRISE framework and prior experience as baselines.

assumptions (5)

domain assumption QEMU's CPU simulation is sufficiently accurate for the AI accelerator workload, where the core mainly configures the accelerator.
Stated in Section V.A: 'the ARM core is mainly responsible for configuring the accelerator peripheral, rendering the rather high level of abstraction of QEMU inconsequential.' If this is false, the simulation results may be inaccurate.
domain assumption Timing accuracy of the VP depends mainly on peripheral, interconnect, and memory models, not the CPU model.
Section V.A: 'the timing accuracy of the VP for the observed workload mainly depends on the accuracy of the peripheral, the interconnect and the memory models.' This justifies using QEMU.
domain assumption Containerization introduces negligible overhead and enables interference-free parallel replication.
Section III claims 'efficient and interference-free replication of the same VP multiple times in parallel', but no measurements are given.
ad hoc to paper The SUNRISE framework provides a correct and usable management layer for VP containers.
The paper relies on [11], the authors' own prior work, as the runtime manager, with no independent evaluation in this paper.
ad hoc to paper The earlier commercial-tool experience is a valid baseline for TTE comparison.
Section V.C: 'the proposed methodology was put to test with a concrete AI workload which was executed on other virtual platforms in off-the-shelf commercial simulation tools before.' The comparison is qualitative.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scalable Software Testing in Fast Virtual Platforms: Leveraging SystemC, QEMU and Containerization." pith.science (2026). https://pith.science/paper/JP56PQ4D

@misc{pith2026250610624,
  author       = {Pith},
  title        = {Pith review of: Scalable Software Testing in Fast Virtual Platforms: Leveraging SystemC, QEMU and Containerization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JP56PQ4D}},
  note         = {Machine review of arXiv:2506.10624}
}

read the original abstract

The ever-increasing complexity of HW/SW systems presents a persistent challenge, particularly in safety-critical domains like automotive, where extensive testing is imperative. However, the availability of hardware often lags behind, hindering early-stage software development. To address this, Virtual Platforms (VPs) based on the SystemC TLM-2.0 standard have emerged as a pivotal solution, enabling pre-silicon execution and testing of unmodified target software. In this study, we propose an approach leveraging containerization to encapsulate VPs in order to reduce environment dependencies and enable cloud deployment for fast, parallelized test execution, as well as open-source VP technologies such as QEMU and VCML to obviate the need for seat licenses. To demonstrate the efficacy of our approach, we present an Artificial Intelligence (AI) accelerator VP case study. Through our research, we offer a robust solution to address the challenges posed by the complexity of HW/SW systems, with practical implications for accelerating HW/SW co-development.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 9 canonical work pages

[11]

Deployment of containerized simulations in an API-driven distributed infrastructure

T. Kraus, A. Sauer. “Deployment of containerized simulations in an API-driven distributed infrastructure”. DVCon Europe. 2024

work page 2024
[2]

Virtual Components Modeling Library (vcml), MachineWare GmbH, accessed 2024-04-12, https://github.com/machineware-gmbh/vcml

work page 2024
[1]

Open-Source Virtual Platforms for Industry and Research

N. Bosbach, L. Jünger, and R. Leupers, "Open-Source Virtual Platforms for Industry and Research", Design and Verification Conference & Exhibition Europe (DVCon Europe), Munich, Germany, 2023

work page 2023
[4]

A Survey of QEMU-based Fault Injection Tools & Techniques for Emulating Physical Faults

Bekele, Yohannes B., Daniel B. Limbrick, and John C. Kelly. "A Survey of QEMU-based Fault Injection Tools & Techniques for Emulating Physical Faults." IEEE Access (2023)

work page 2023
[5]

NQC²: A Non-Intrusive QEMU Code Coverage Plugin

N. Bosbach, et al. "NQC²: A Non-Intrusive QEMU Code Coverage Plugin." Proceedings of the 16th Workshop on Rapid Simulation and Performance Evaluation for Design. 2024

work page 2024
[6]

LibAFL QEMU: A Library for Fuzzing-oriented Emulation

Romain Malmain, Andrea Fioraldi, Francillon Aurélien. LibAFL QEMU: A Library for Fuzzing-oriented Emulation. BAR 2024, Workshop on Binary Analysis Research, colocated with NDSS 2024, Mar 2024, San Diego (CA), United States

work page 2024
[8]

QBox: an industrial solution for virtual platform simulation using QEMU and SystemC TLM-2.0

Guillaume Delbergue, Mark Burton, Frederic Konrad, Bertrand Le Gal, Christophe Jego. QBox: an industrial solution for virtual platform simulation using QEMU and SystemC TLM-2.0. 8th European Congress on Embedded Real Time Software and Systems (ERTS 2016), Jan 2016, TOULOUSE, France

work page 2016
[9]

Virtual ECUs with QEMU and SystemC TLM-2.0

L. Jünger, J. Weinstock M. Jassi, M. Yoshinaga, H. Hamao, and K. Sato, "Virtual ECUs with QEMU and SystemC TLM-2.0", Design and Verification Conference & Exhibition Europe (DVCon Europe), Munich, Germany, 2023

work page 2023

Show all 9 references

[12]

Vehicle-centralized, zone-oriented E/E architecture with vehicle computers

Robert Bosch GmbH. “Vehicle-centralized, zone-oriented E/E architecture with vehicle computers”. accessed 2024-06-12. https://www.bosch-mobility.com/en/mobility-topics/ee-architecture/

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [11]

Deployment of containerized simulations in an API-driven distributed infrastructure

T. Kraus, A. Sauer. “Deployment of containerized simulations in an API-driven distributed infrastructure”. DVCon Europe. 2024

work page 2024

[2] [2]

Virtual Components Modeling Library (vcml), MachineWare GmbH, accessed 2024-04-12, https://github.com/machineware-gmbh/vcml

work page 2024

[3] [1]

Open-Source Virtual Platforms for Industry and Research

N. Bosbach, L. Jünger, and R. Leupers, "Open-Source Virtual Platforms for Industry and Research", Design and Verification Conference & Exhibition Europe (DVCon Europe), Munich, Germany, 2023

work page 2023

[4] [4]

A Survey of QEMU-based Fault Injection Tools & Techniques for Emulating Physical Faults

Bekele, Yohannes B., Daniel B. Limbrick, and John C. Kelly. "A Survey of QEMU-based Fault Injection Tools & Techniques for Emulating Physical Faults." IEEE Access (2023)

work page 2023

[5] [5]

NQC²: A Non-Intrusive QEMU Code Coverage Plugin

N. Bosbach, et al. "NQC²: A Non-Intrusive QEMU Code Coverage Plugin." Proceedings of the 16th Workshop on Rapid Simulation and Performance Evaluation for Design. 2024

work page 2024

[6] [6]

LibAFL QEMU: A Library for Fuzzing-oriented Emulation

Romain Malmain, Andrea Fioraldi, Francillon Aurélien. LibAFL QEMU: A Library for Fuzzing-oriented Emulation. BAR 2024, Workshop on Binary Analysis Research, colocated with NDSS 2024, Mar 2024, San Diego (CA), United States

work page 2024

[7] [8]

QBox: an industrial solution for virtual platform simulation using QEMU and SystemC TLM-2.0

Guillaume Delbergue, Mark Burton, Frederic Konrad, Bertrand Le Gal, Christophe Jego. QBox: an industrial solution for virtual platform simulation using QEMU and SystemC TLM-2.0. 8th European Congress on Embedded Real Time Software and Systems (ERTS 2016), Jan 2016, TOULOUSE, France

work page 2016

[8] [9]

Virtual ECUs with QEMU and SystemC TLM-2.0

L. Jünger, J. Weinstock M. Jassi, M. Yoshinaga, H. Hamao, and K. Sato, "Virtual ECUs with QEMU and SystemC TLM-2.0", Design and Verification Conference & Exhibition Europe (DVCon Europe), Munich, Germany, 2023

work page 2023