Evaluating the Overhead of the Performance Profiler Cloudprofiler With MooBench

David Georg Reichelt; Shinhyung Yang; Wilhelm Hasselbring

REVIEW 3 major objections 4 minor 16 references

Evaluating the Overhead of the Performance Profiler Cloudprofiler With MooBench

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In the MooBench benchmark, Cloudprofiler's buffered and Zstandard-compressed logging handler adds about 2.28 microseconds per instrumented call, 6.15 times less than its unbuffered non-compressed handler.

desk verdict A reproducible benchmark report that gives Cloudprofiler users useful overhead numbers, but the headline 2.28 µs excludes asynchronous compression and I/O cost, and the paper lacks error bars. read the letter →

arxiv 2411.17413 v1 pith:HI6MWLRT submitted 2024-11-26 cs.DC cs.SE

Shinhyung Yang , David Georg Reichelt , Wilhelm Hasselbring This is my paper · ORCID

classification cs.DCcs.SE

keywords CloudprofilerMooBenchobservabilityoverheadperformanceprofilingbufferedloggingZstandardcompressionJVMinstrumentationcloud-nativemonitoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates the runtime overhead of Cloudprofiler, a C++ performance profiler for native and JVM processes, using the MooBench benchmark. The central result is that the buffered ID handler with Zstandard compression logs an instrumented call in about 2.28 microseconds on average, which the authors report as 6.15 times faster than the non-buffered, non-compressing handler. The speedup comes from moving disk writes off the critical path: log entries go to in-memory buffer blocks, and parallel compression threads drain those blocks before the I/O thread writes them out. If the measurement holds beyond this synthetic workload, it suggests that continuous observability can be added to cloud-native services at a cost of roughly two microseconds per instrumented call.

What carries the argument

The central mechanism is the buffered and compressed ID handler, Cloudprofiler's logging path that writes each log entry to in-memory buffer blocks instead of performing disk I/O per entry. Full blocks are dequeued from a non-blocking multi-producer/multi-consumer queue by four compression threads, which compress with either the Zstandard (ZSTD) or LZO1X codec and enqueue the compressed block to a second queue drained by the I/O thread. This design keeps compression and disk writing off the critical path of the instrumented method; the non-buffered ID handler, which does disk I/O per log entry, is the baseline the 6.15-times speedup is measured against.

What would settle it

A concrete check would be to run MooBench with nonzero method durations and variable call depths, or with a tight loop that fills buffer blocks faster than compression threads drain them, and compare the resulting per-call overhead against 2.28 microseconds; if the buffered compressed handler loses its 6.15-times advantage or becomes slower than the unbuffered handler, the central claim would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that Cloudprofiler's buffered and compressed ID handler delivers the best overhead profile in MooBench: 2.28 microseconds per call with either the ZSTD or LZO1X codec, against a null-handler cost of 0.461 microseconds and a non-buffered ID handler that is 6.15 times slower. The authors also report that Cloudprofiler's memory footprint is more stable than the comparison frameworks, with a maximum coefficient of variation of 11.78 percent versus 50.39 percent for Kieker and 54.35 percent for OpenTelemetry. These numbers are offered as evidence that locating the profiler outside the target process and using buffer blocks plus compression threads removes disk I/O from the critical path.

Load-bearing premise

The load-bearing assumption is that MooBench's synthetic workload, a depth-10 recursion with zero execution time, is representative enough that the measured 2.28 microsecond overhead predicts Cloudprofiler's cost in real applications; if real call patterns or buffer fill rates differ, the absolute number and the 6.15-times speedup may change.

Editorial extensions

If this is right

Cloudprofiler's buffered ZSTD ID handler adds about 2.28 microseconds per instrumented call in the MooBench workload, versus roughly 14 microseconds for the unbuffered non-compressing handler.
The null JNI handler costs only 0.461 microseconds, so the dominant part of the overhead is the logging and disk path, not the native-call boundary.
ZSTD and LZO1X compression perform almost identically in this workload, so codec choice does not change the measured overhead.
Cloudprofiler's heap-memory variation is much lower than Kieker's and OpenTelemetry's, implying more predictable memory behavior during instrumented execution.
MooBench can now be used for continuous regression benchmarking of Cloudprofiler, matching its design goal of catching performance regressions in CI pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: on hosts where all CPU cores are already saturated, the 2.28 microsecond wall-clock figure may understate total cost, because four compression threads consume cores that the application could otherwise use.
Inference: the synthetic workload's zero-duration recursive calls never fill a buffer block mid-call, so the measurement reflects a low-pressure case; a workload that fills blocks rapidly could show a different balance between buffering and compression cost.
Inference: the same buffering and compression design could be tested on native C/C++ targets, where the JNI boundary disappears and the per-call floor may drop below 0.461 microseconds.
Inference: a direct comparison of Cloudprofiler against distributed tracing frameworks in a realistic microservice trace, rather than a single-process recursion, would test whether the 2.28 microsecond advantage survives multi-service coordination.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a small empirical paper that adds Cloudprofiler to the existing MooBench benchmark and reports overhead numbers—most notably 2.28 µs average execution time for the buffered ZSTD-compressed ID handler, 6.15x faster than the non-buffered handler. The new thing is the data, not the method. Code and datasets are linked, which is real and makes the work reproducible.

What it does well: the measurement design is straightforward, the architecture description matches the implementation (buffer blocks, MPMC queues, compression threads), and the numbers are internally consistent. The authors are explicit that disk writing is moved off the critical path. That is honest, and it explains why the buffered handlers look so cheap.

The soft spots are real but not fatal. First, the reported 2.28 µs is per-iteration latency on the instrumented method, not a full accounting of what Cloudprofiler consumes. The compression threads and I/O thread do work asynchronously, so the measured number reflects only the enqueue side. In a sustained workload where those threads become the bottleneck, the application-facing cost could be higher. The paper should state this limitation clearly rather than let the abstract imply 2.28 µs is the total cost of the logging path. Second, execution times are point estimates without standard deviations or confidence intervals. The figure has no error bars, and the text gives only averages. For a benchmark paper, that is a significant omission. Third, the paper says MooBench is designed for regression benchmarking within a single framework, then immediately compares Cloudprofiler to Kieker and OpenTelemetry. That is a stated-purpose mismatch, though it does not invalidate the Cloudprofiler internal comparison.

Who is this for: practitioners using Cloudprofiler who want a rough overhead baseline, and researchers working on observability tooling who can reuse the artifact. It does not change the field, but it is a decent, honest measurement report.

My recommendation: send it to peer review. The reproducibility and the new measurement are worth referee time, but the authors should be asked to add variance information, clarify that the 2.28 µs excludes asynchronous compression/I/O cost, and either drop the cross-framework comparison or justify it against their own stated scope.

Referee Report

3 major / 4 minor

Summary. This paper integrates the Cloudprofiler profiler into the MooBench benchmark harness and measures the per-iteration execution time of five Cloudprofiler handler configurations (null, non-buffered ID, buffered binary-encoded, buffered ZSTD-compressed, and buffered LZO1X-compressed), together with non-instrumentation, Kieker, and OpenTelemetry configurations for context. The headline result is that the buffered, ZSTD-compressed ID handler averages 2.28 microseconds and is reported to be 6.15 times faster than the non-buffered, uncompressed ID handler. The authors describe the experimental setup, report garbage collection and heap variability, relate the results to prior overhead studies, and provide public code and dataset links.

Significance. If the headline result is accepted as stated, the paper provides a useful, reproducible data point for Cloudprofiler's instrumentation overhead within MooBench and supports the project's continuous-integration regression-benchmarking goal. The manuscript's strengths are its use of a standard benchmark harness, its direct measurement rather than model fitting, and its public code and data. Its significance is limited by the fact that the measured interval excludes the asynchronous compression and I/O threads, so the result is best read as producer-side enqueue latency rather than total observability overhead, and by the absence of dispersion statistics. The comparison with Kieker and OpenTelemetry is useful context but is not the main contribution.

major comments (3)

[Abstract and Section 3 (Experimental Setup)] The abstract presents the 2.28 microsecond figure as the cost of the buffered and compressed logging path, but the measurement interval covers only the producer-side enqueue operation. Section 3 describes four compression threads draining a non-blocking MPMC queue and an I/O thread writing compressed blocks; MooBench's per-iteration elapsed time does not include the CPU time of those threads, queue backpressure, or the final flush of buffered data. The non-buffered ID handler performs disk I/O synchronously per entry, so the 6.15x speedup is partly a comparison of work shifted off the measured critical path against work eliminated. Please either include total CPU time, drain and flush costs, and queue occupancy, or explicitly define and consistently phrase the claim as producer-side latency rather than total observability overhead.
[Section 4 (Evaluation), Figure 2] All execution times in Section 4 and Figure 2 are point estimates. The manuscript reports 10 repetitions of 2M iterations and says MooBench selects 10M execution results for statistics, but no standard deviation, confidence interval, or numeric table is provided. The central speedup of 6.15x appears only in the abstract and cannot be reconstructed from the text because Figure 2 is log-scaled and values for configurations (1), (3), (11), and (12) are not stated. Since several buffered variants are within 0.014 microseconds of one another, the claim that compression variants differ from the binary-encoded variant needs variance information. Please add a numeric table with means, dispersion measures, and sample counts.
[Section 3, Listing 1] The synthetic workload (recursive method with depth 10 and zero duration) is an empty recursive call, and the paper does not characterize the event production rate relative to the capacity of the four compression threads. Under this workload, buffered handlers may report near-best-case enqueue latency because the consumers are never saturated; in a sustained or bursty workload, producer blocking or flush-at-shutdown costs would appear in end-to-end overhead. Please report the achieved event rate, buffer block fill and drain rates, observed queue lengths, or add a sustained and bursty workload so that the reader can judge how representative the 2.28 microsecond figure is.

minor comments (4)

[Section 2, Listing 1 caption and Figure 1] The caption contains a duplicated word: 'measures the span of of extractedMethod' should read 'measures the span of extractedMethod'; also, 'instrumentated' in Figure 1 should be 'instrumented'.
[Section 3] 'MooBenchs default configurations' should be 'MooBench's default configurations'. More substantively, the sentence 'MooBench selects 10 M execution results to create statistics for each configuration' is unclear and should state how the 20M collected results are reduced to 10M per configuration.
[Section 4] Garbage collection counts and heap-size coefficients of variation are reported but never interpreted; the paper should explain whether these metrics matter for comparing the handlers or whether they are merely auxiliary.
[Reproducibility] The artifact links are welcome, but the paper would be easier to evaluate if the repository contained a version-stamped README listing the exact commit, benchmark configuration, and host environment used for the reported run.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central 2.28 µs result is a direct MooBench measurement, not an equation-derived prediction or a fitted-parameter retrofit.

full rationale

The paper's central claim is an empirical measurement: Cloudprofiler's buffered ZSTD-compressed ID handler averaged 2.28 µs per instrumented call under MooBench's synthetic recursive workload, 6.15x faster than the non-buffered, non-compressed handler. This is a measured execution-time statistic, not a quantity derived from a model whose parameters were fitted to the same data, and no equation in the paper reduces the output to an input. The only self-citations are contextual: [15] identifies Cloudprofiler itself, while [1,10,16] identify MooBench and prior instrumentation-overhead comparisons by the same group. These citations describe the benchmark and tooling but are not load-bearing for the numerical result, since the 2.28 µs figure comes from the new MooBench integration and the published dataset, not from the cited papers. The paper itself notes that MooBench is designed for regression benchmarking within continuous integration pipelines of individual monitoring frameworks, not for comparing such frameworks against each other (Section 1); this is a validity caveat about the cross-framework comparisons in Figure 2, not a circularity. Similarly, the skepticism about asynchronous compression threads not being fully charged to the measured per-iteration elapsed time is a correctness or interpretation concern about what 'overhead' includes, not a case of the result being equivalent to its assumptions by construction. No fitted-input-called-prediction, imported uniqueness theorem, or ansatz-via-citation pattern is present. Overall, the derivation chain is a straightforward benchmark measurement, so circularity is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted: the benchmark configuration (MooBench defaults, 2M iterations, 10 repeats) and Cloudprofiler settings (32 buffer blocks, 1M entries per block, 4 compression threads) are stated experimental inputs, not fitted constants. The central claim depends on domain assumptions about workload representativeness, clock resolution, environment stability, and instrumentation path. No new entities are introduced.

assumptions (4)

domain assumption MooBench's synthetic workload is representative of performance-profiler overhead.
The overhead numbers are measured on a tight recursive busy-wait loop (depth 10, time 0 ns) from Listing 1; real workloads may have different call shapes and allocation patterns.
domain assumption System.nanoTime() provides sufficient resolution and low enough overhead for microsecond-scale measurements.
It is used both to measure the target method and as the synthetic workload in Listing 1; biased or high-latency clocks would shift all numbers.
domain assumption The measured environment is stable enough that 10 repeated MooBench runs characterize the framework.
One bare-metal server with 2M iterations per run and no reported execution-time variance is used; single-machine measurements are not evidence across hardware.
domain assumption Source-level manual instrumentation via JNI is a representative way to use Cloudprofiler.
Cloudprofiler's interface is instrumented manually into MooBench's monitoredMethod; other instrumentation paths such as C/C++ are not evaluated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating the Overhead of the Performance Profiler Cloudprofiler With MooBench." pith.science (2026). https://pith.science/paper/HI6MWLRT

@misc{pith2026241117413,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Overhead of the Performance Profiler Cloudprofiler With MooBench},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HI6MWLRT}},
  note         = {Machine review of arXiv:2411.17413}
}

read the original abstract

Performance engineering has become crucial for the cloud-native architecture. This architecture deploys multiple services, with each service representing an orchestration of containerized processes. OpenTelemetry is growing popular in the cloud-native industry for observing the software's behaviour, and Kieker provides the necessary tools to monitor and analyze the performance of target architectures. Observability overhead is an important aspect of performance engineering and MooBench is designed to compare different observability frameworks, including OpenTelemetry and Kieker. In this work, we measure the overhead of Cloudprofiler, a performance profiler implemented in C++ to measure native and JVM processes. It minimizes the profiling overhead by locating the profiler process outside the target process and moving the disk writing overhead off the critical path with buffer blocks and compression threads. Using MooBench, Cloudprofiler's buffered ID handler with the Zstandard lossless data compression ZSTD showed an average execution time of 2.28 microseconds. It is 6.15 times faster than the non-buffered and non-compression handler.

Figures

Figures reproduced from arXiv: 2411.17413 by the authors.

**Figure 1.** Cloudprofiler Deployment in MooBench picts the Cloudprofiler modules within the MooBench architecture running as a JVM process on the Host OS. The MooBench workload is instrumented at the source code level with the Cloudprofiler interface. We incorporated the instrumentation structure from [16], which evaluated different instrumentation technologies for JVM applications, including source code-level, and bytecode-le… view at source ↗

**Figure 2.** MooBench’s evaluation comparison of Cloudprofiler, Kieker [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

[1]

Including Performance Benchmarks into Con- tinuous Integration to Enable DevOps

J. Waller, N. C. Ehmke, and W. Hasselbring. “Including Performance Benchmarks into Con- tinuous Integration to Enable DevOps”. In: SIGSOFT Softw. Eng. Notes 40.2 (Mar. 2015)

work page 2015
[2]

Borg, Omega, and Kubernetes

B. Burns et al. “Borg, Omega, and Kubernetes”. In: Commun. ACM 59.5 (Apr. 2016), pp. 50–57

work page 2016
[3]

Microservices for Scalability

W. Hasselbring. “Microservices for Scalability”. In: Proceedings of the 7th ACM/SPEC on Inter- national Conference on Performance Engineer- ing. 2016, pp. 133–134

work page 2016
[4]

Ef- ﬁcient and Viable Handling of Large Object Traces

P. Lengauer, V. Bitto, and H. M¨ ossenb¨ ock. “Ef- ﬁcient and Viable Handling of Large Object Traces”. In: ICPE. ICPE ’16. Delft, The Nether- lands: Association for Computing Machinery, 2016, pp. 249–260

work page 2016
[5]

Cloud-Native Applications

D. Gannon, R. Barga, and N. Sundaresan. “Cloud-Native Applications”. In: IEEE Cloud Computing 4.5 (2017), pp. 16–21

work page 2017
[6]

Application Performance Man- agement: State of the Art and Challenges for the Future

C. Heger et al. “Application Performance Man- agement: State of the Art and Challenges for the Future”. In: ICPE. 2017, pp. 429–432

work page 2017
[7]

Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead

M. Gebai and M. R. Dagenais. “Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead”. In: ACM Comput. Surv. 51.2 (Mar. 2018)

work page 2018
[8]

Kieker: A monitoring framework for software engineering research

W. Hasselbring and A. van Hoorn. “Kieker: A monitoring framework for software engineering research”. In: Software Impacts 5 (2020)

work page 2020

Show all 16 references

[9]

Benchmarking as Empirical Standard in Software Engineering Research

W. Hasselbring. “Benchmarking as Empirical Standard in Software Engineering Research”. In: International Conference on Evaluation and Assessment in Software Engineering (EASE 2021). ACM, June 2021, pp. 365–372

2021
[10]

Overhead Comparison of OpenTelemetry, in- spectIT and Kieker

D. G. Reichelt, S. K¨ uhne, and W. Hasselbring. “Overhead Comparison of OpenTelemetry, in- spectIT and Kieker”. In: SSP. Gesellschaft f¨ ur Informatik eV, 2021

2021
[11]

D. G. Blanco. Practical OpenTelemetry: Adopt- ing Open Observability Standards Across Your Organization. APress, 2023

2023
[12]

A Comparison of Distributed Tracing Tools in Serverless Applications

C. Eder, S. Winzinger, and R. Lichtenth¨ aler. “A Comparison of Distributed Tracing Tools in Serverless Applications”. In: 2023 IEEE SOSE . IEEE. 2023, pp. 98–105

2023
[13]

Julia Cloud Matrix Ma- chine: Dynamic Matrix Language Accelera- tion on Multicore Clusters in the Cloud

J. H. Lee et al. “Julia Cloud Matrix Ma- chine: Dynamic Matrix Language Accelera- tion on Multicore Clusters in the Cloud”. In: PMAM’23. ACM, 2023

2023
[14]

Towards Solving the Challenge of Minimal Overhead Monitoring

D. G. Reichelt, S. K¨ uhne, and W. Hasselbring. “Towards Solving the Challenge of Minimal Overhead Monitoring”. In: Companion of the 2023 ACM/SPEC ICPE . 2023, pp. 381–388

2023
[15]

Cloudproﬁler: TSC-based inter- node proﬁling and high-throughput data in- gestion for cloud streaming workloads

S. Yang et al. “Cloudproﬁler: TSC-based inter- node proﬁling and high-throughput data in- gestion for cloud streaming workloads”. In: arXiv:2205.09325 (2023)

2023 arXiv
[16]

Overhead Comparison of Instrumentation Frameworks

D. G. Reichelt et al. “Overhead Comparison of Instrumentation Frameworks”. In: Companion of the 15th ACM/SPEC ICPE . 2024

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Including Performance Benchmarks into Con- tinuous Integration to Enable DevOps

J. Waller, N. C. Ehmke, and W. Hasselbring. “Including Performance Benchmarks into Con- tinuous Integration to Enable DevOps”. In: SIGSOFT Softw. Eng. Notes 40.2 (Mar. 2015)

work page 2015

[2] [2]

Borg, Omega, and Kubernetes

B. Burns et al. “Borg, Omega, and Kubernetes”. In: Commun. ACM 59.5 (Apr. 2016), pp. 50–57

work page 2016

[3] [3]

Microservices for Scalability

W. Hasselbring. “Microservices for Scalability”. In: Proceedings of the 7th ACM/SPEC on Inter- national Conference on Performance Engineer- ing. 2016, pp. 133–134

work page 2016

[4] [4]

Ef- ﬁcient and Viable Handling of Large Object Traces

P. Lengauer, V. Bitto, and H. M¨ ossenb¨ ock. “Ef- ﬁcient and Viable Handling of Large Object Traces”. In: ICPE. ICPE ’16. Delft, The Nether- lands: Association for Computing Machinery, 2016, pp. 249–260

work page 2016

[5] [5]

Cloud-Native Applications

D. Gannon, R. Barga, and N. Sundaresan. “Cloud-Native Applications”. In: IEEE Cloud Computing 4.5 (2017), pp. 16–21

work page 2017

[6] [6]

Application Performance Man- agement: State of the Art and Challenges for the Future

C. Heger et al. “Application Performance Man- agement: State of the Art and Challenges for the Future”. In: ICPE. 2017, pp. 429–432

work page 2017

[7] [7]

Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead

M. Gebai and M. R. Dagenais. “Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead”. In: ACM Comput. Surv. 51.2 (Mar. 2018)

work page 2018

[8] [8]

Kieker: A monitoring framework for software engineering research

W. Hasselbring and A. van Hoorn. “Kieker: A monitoring framework for software engineering research”. In: Software Impacts 5 (2020)

work page 2020