RECORD
Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud
Software Engineering · Submitted 20 Mar 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim There is no clear superior framework, but the ranking of the frameworks depends on the use case.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Context: The combination of distributed stream processing with microservice architectures is an emerging pattern for building data-intensive software systems. In such systems, stream processing frameworks such as Apache Flink, Apache Kafka Streams, Apache Samza, Hazelcast Jet, or the Apache Beam SDK are used inside microservices to continuously process massive amounts of data in a distributed fashion. While all of these frameworks promote scalability as a core feature, there is only little empirical research evaluating and comparing their scalability. Objective: The goal of this study to obtain evidence about the scalability of state-of-the-art stream processing framework in different execution environments and regarding different scalability dimensions. Method: We benchmark five modern stream processing frameworks regarding their scalability using a systematic method. We conduct over 740 hours of experiments on Kubernetes clusters in the Google cloud and in a private cloud, where we deploy up to 110 simultaneously running microservice instances, which process up to one million messages per second. Results: All benchmarked frameworks exhibit approximately linear scalability as long as sufficient cloud resources are provisioned. However, the frameworks show considerable differences in the rate at which resources have to be added to cope with increasing load. There is no clear superior framework, but the ranking of the frameworks depends on the use case. Using Apache Beam as an abstraction layer still comes at the cost of significantly higher resource requirements regardless of the use case. We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, and the selected cloud environment. Moreover, vertical scaling can be a complementary measure to achieve scalability of stream processing frameworks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Context: The combination of distributed stream processing with microservice architectures is an emerging pattern for building data-intensive software systems. | background 1.00 | 0.04 | 0.01 | asserted 0.99 |
| s1 | In such systems, stream processing frameworks such as Apache Flink, Apache Kafka Streams, Apache Samza, Hazelcast Jet, or the Apache Beam SDK are used inside microservices to continuously process massive amounts of data in a distributed fashion. | background 1.00 | 0.04 | 0.01 | asserted 1.00 |
| s2 | While all of these frameworks promote scalability as a core feature, there is only little empirical research evaluating and comparing their scalability. | gap 1.00 | 0.08 | 0.02 | asserted 0.99 |
| s3 | Objective: The goal of this study to obtain evidence about the scalability of state-of-the-art stream processing framework in different execution environments and regarding different scalability dimensions. | contribution 0.95 | 0.22 | 0.02 | asserted 0.60 |
| s4 | Method: We benchmark five modern stream processing frameworks regarding their scalability using a systematic method. | method 0.95 | 0.96 | 0.02 | asserted 0.99 |
| s5 | We conduct over 740 hours of experiments on Kubernetes clusters in the Google cloud and in a private cloud, where we deploy up to 110 simultaneously running microservice instances, which process up to one million messages per second. | method 1.00 | 0.97 | 0.03 | asserted 1.00 |
| s6 | Results: All benchmarked frameworks exhibit approximately linear scalability as long as sufficient cloud resources are provisioned. | result 1.00 | 0.95 | 0.38 | asserted 1.00 |
| s7 | However, the frameworks show considerable differences in the rate at which resources have to be added to cope with increasing load. | result 0.99 | 0.93 | 0.15 | asserted 1.00 |
| s8 | There is no clear superior framework, but the ranking of the frameworks depends on the use case. | result 0.54 | 0.93 | 0.47 | asserted 1.00 |
| s9 | Using Apache Beam as an abstraction layer still comes at the cost of significantly higher resource requirements regardless of the use case. | result 0.98 | 0.90 | 0.12 | asserted 1.00 |
| s10 | We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, and the selected cloud environment. | result 0.93 | 0.94 | 0.10 | asserted 1.00 |
| s11 | Moreover, vertical scaling can be a complementary measure to achieve scalability of stream processing frameworks. | result 0.83 | 0.86 | 0.08 | hedged possibility 0.59 |
What was examined, and what was not
- arXiv listing abstract v2
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Context: The combination of distributed stream processing with microservice architectures is an emerging pattern for building data-intensive software systems. In such systems, stream processing frameworks such as Apache Flink, Apache Kafka Streams, Apache Samza, Hazelcast Jet, or the Apache Beam SDK are used inside microservices to continuously process massive amounts of data in a distributed fashion. While all of these frameworks promote scalability as a core feature, there is only little empirical research evaluating and comparing their scalability. Objective: The goal of this study to obtain evidence about the scalability of state-of-the-art stream processing framework in different execution environments and regarding different scalability dimensions. Method: We benchmark five modern stream processing frameworks regarding their scalability using a systematic method. We conduct over 740 hours of experiments on Kubernetes clusters in the Google cloud and in a private cloud, where we deploy up to 110 simultaneously running microservice instances, which process up to one million messages per second. Results: All benchmarked frameworks exhibit approximately linear scalability as long as sufficient cloud resources are provisioned. However, the frameworks show considerable differences in the rate at which resources have to be added to cope with increasing load. There is no clear superior framework, but the ranking of the frameworks depends on the use case. Using Apache Beam as an abstraction layer still comes at the cost of significantly higher resource requirements regardless of the use case. We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, and the selected cloud environment. Moreover, vertical scaling can be a complementary measure to achieve scalability of stream processing frameworks.
Human review
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.