REVIEW 5 major objections 5 minor 38 references
Runtime-optimized Multi-way Stream Join Operator for Large-scale Streaming data
T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read On a four-table TPC-DS stream, re-optimizing probe order each cycle cut runtime by 31.2% on average versus a fixed order.
desk verdict The adaptive reoptimization idea is sound, but the experiments never actually test adaptation—randomly shuffled static data is stationary, so the claimed gains likely just reflect better static order selection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the probe-order optimizer dpPick combined with its cost model. dpPick treats a join as a graph whose edges are probe pairs $\langle l_i, r_i\rangle$, enumerates all complete probe sequences with depth-first search, and memoizes subsequence costs so each subsequence is evaluated once. Costs come from the recursive formula that adds a query cost $\alpha_q f(\kappa_{r_i})$ to a probabilistic match cost $\gamma_{l_i}^{r_i}(\alpha_m \mu_{l_i}^{r_i} + C_{o_{i+1}})$; the statistics $\gamma$, $\mu$, and $\kappa$ are collected per cycle and projected to the next cycle with quadratic exponential smoothing (Holt's linear trend for counts, damping trend for match rates). This machinery is what lets the operator adapt its probe order without prior knowledge of stream characteristics.
What would settle it
Run dpPick on a synthetic stream where each cycle a different join becomes selective, or where selectivities shift abruptly, and compare its chosen order's measured cost with the predicted cost from the recursive formula. If the predicted and measured per-cycle costs have near-zero rank correlation, or if dpPick's runtime is worse than a fixed order for more than a small fraction of cycles, the central claim fails. A simpler version: sweep $\alpha_q$ and $\alpha_m$ over a grid on the same TPC-DS workload; if the reported average advantage over fixedOrder disappears or reverses for reasonable constant values, the model is miscalibrated.
Extended reading notes
Core claim
The paper's central claim is that cyclic, statistics-driven reordering of probe pairs inside a multi-way stream join operator yields materially faster processing than any static or greedily chosen order. The operator stores one state backend per input stream; on each arriving tuple it iteratively probes the other backends in a chosen order, stopping early when a probe fails. dpPick takes a join graph, estimates each probe's query cost and match cost from predicted statistics, recursively evaluates every complete probe sequence with memoization, and installs the cheapest sequence for the next cycle. The cost estimate for a probe pair is $C_{o_i}\approx \alpha_q f(\kappa_{r_i}) + \gamma_{l_i}^{r_i}(\alpha_m \mu_{l_i}^{r_i} + C_{o_{i+1}})$, where $\gamma$ is the predicted match-success probability, $\mu$ the average number of matched records, and $\kappa$ the predicted number of keys in the probed backend. The authors support the claim with TPC-DS experiments: dpPick beat a fixed initial order in 22 of 24 orderings (6.0%–53.7% lower runtime, 31.2% average), beat a cost-greedy baseline in all 24 (1.5%–42.8% lower, 20.4% average), and beat a selectivity-first baseline in all 24 (43.6%–75.3% lower, 56.1% average).
Load-bearing premise
The argument rests on the cost model ranking probe orders correctly; that ranking depends on uncalibrated constants in the cost formulas, and on the assumption that smoothed historical statistics predict the next cycle in streams that are changing.
Editorial extensions
If this is right
- On the tested four-table TPC-DS join, changing probe order every cycle instead of keeping it fixed cuts runtime by 31.2% on average, with reductions up to 53.7%.
- A cost model that includes both query cost and match cost outperforms either cost alone, by 19.5% and 6.3% on average respectively.
- Predicting next-cycle statistics with quadratic exponential smoothing beats using the previous cycle's raw statistics in 22 of 24 orderings.
- Optimization cycle periods of roughly 3–13 seconds and prediction history lengths near 60 give the best stability; much shorter or longer periods increase runtime.
- Adaptive reordering removes the need to know stream statistics in advance, so the operator can track evolving workloads automatically.
Reading between the lines
- The experiments use one dataset and one four-table join shape; a natural test is whether the same gains appear on higher-degree joins or real-world streams with sudden regime changes, and whether the smoothing horizon needs to adapt.
- The cost model constants $\alpha_q$, $\alpha_m$, $c$, and $m$ are left unspecified and uncalibrated, so the method's portability to other backends likely hinges on tuning them; a sensitivity analysis over a constant grid would reveal how robust the claimed gains are.
- The full enumeration of probe sequences in dpPick could become a bottleneck when the join graph is dense; an extension that prunes by branch-and-bound would make the approach scale beyond small graphs.
- A direct check of the prediction layer would be to compare predicted $\gamma$, $\mu$, and $\kappa$ against observed values cycle by cycle; if prediction error is large, the runtime gains come from the cost model rather than the smoothing, which would change how one would improve the operator.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes a runtime-optimized multi-way stream join operator for state-based stream processing on Flink. The operator divides execution into cycles, collects per-cycle statistics (match success probability, average number of matches, and key counts), predicts next-cycle statistics using exponential smoothing, and invokes a dynamic-programming algorithm, dpPick, that uses an approximate cost model to choose a probe order for each input stream. Experiments on a four-table TPC-DS join compare dpPick with fixed-order, selectivity-first, and greedy-cost baselines across 24 initial orders, together with ablations that remove query cost, match cost, or smoothing. The paper reports large average runtime reductions over the baselines and concludes that the adaptive strategy improves processing efficiency for unknown, changing data streams.
Significance. If the central claim were established, runtime reoptimization of probe order for multi-way stream joins would be a useful and practical contribution to stream processing engines, especially for workloads with drifting statistics. The paper's strengths include a clean recursive cost formulation, a memoized dynamic-programming optimizer (Algorithm 3), a clearly described operator architecture, and an ablation design that separates query-cost, match-cost, and smoothing contributions. The main limitation is evidential: the experiments use a static dataset and therefore do not exercise the adaptation mechanism that motivates the work, and the cost model's free parameters are left unspecified. The contribution is plausible but not yet demonstrated at the level claimed.
major comments (5)
- [Section 5.1, Section 5.2] The experimental design does not test adaptation to changing streams. Randomizing the order in which a static TPC-DS dataset is read from Kafka changes tuple interleaving but leaves the marginal distributions of keys, match counts, and match rates stationary; successive optimization cycles see sampling noise, not drift. The reported 6.0% to 53.7% reductions versus fixedOrder can therefore be explained by dpPick finding a better static order rather than by runtime tracking of non-stationary statistics. The statement in Section 5.1 that random order 'simulate[s] real-time data streams that change dynamically' is not justified. Please add experiments with explicit drift or bursts (e.g., changing key distributions, arrival rates, or selectivities over cycles) and compare against a no-reoptimization control and an adaptive baseline such as A-Greedy.
- [Sections 4.1-4.3, Eqs. (4)-(8)] The cost model contains unspecified constants alpha_q, alpha_m, c, and m, and the smoothing method has unspecified parameters (Holt's alpha, beta, and damping phi). The paper reports no values, no calibration procedure, and no sensitivity analysis. Because dpPick selects probe orders by minimizing this approximate cost, the ranking of orders could depend strongly on these constants; without evidence of robustness, the claim that dpPick identifies a near-optimal order is not well supported. Please specify parameter values, state how they were chosen, and report sensitivity of the selected order and runtime to these parameters.
- [Section 5.2] The performance comparison rests on a single four-way join query with no repeated runs, error bars, or statistical tests. The statement that dpPick 'significantly outperforms' the comparative methods is not supported by the evidence as reported. Please provide multiple runs with variance estimates, additional query shapes with different selectivities and numbers of streams, and ideally additional datasets.
- [Section 4.2, Algorithms 3-4] dpPick enumerates all possible probe sequences via depth-first search, and the paper gives no complexity analysis. For a complete join graph with n input streams, the number of candidate sequences is O(n!), which is prohibitive for the 'large-scale' multi-way joins claimed in the title and for per-cycle reoptimization beyond small n. Please report the time complexity of dpPick and include experiments with more than four streams to substantiate the scalability claim.
- [Section 5.3] The smoothing ablation is not conclusive for the same reason as the first major comment: on randomized static data, the previous cycle's statistics are nearly exchangeable with the smoothed prediction, so the 2.5% to 11.6% differences may reflect noise or one-workload effects rather than prediction quality. Please report prediction error (e.g., MAPE) of the smoothing method against held-out cycles and evaluate the ablation on drifting streams.
minor comments (5)
- [Section 5.2] The text cites 'MJoin[35]' and 'GrubJoin[19]', but references [35] and [19] are unrelated papers; the correct citations appear to be [6] and [7]. Similarly, 'MultiStream[42]' does not exist in the reference list; the MultiStream operator is cited as [8] and [32].
- [Algorithm 3] Line 2 initializes subMemo with the key '(null, 0)', which does not match the subsequence keys used in calculateCost; line 10 also contains a typo, 'subsequence' for 'subSequence'. Please make the memoization key type consistent.
- [Algorithm 4] Line 21 contains the typo 'allSequencces', and line 3 omits a separator between 'allSequences.add(...)' and 'return'.
- [Section 4.3] The text mentions Holt's linear trend method and damping trend method but gives no update equations. Please add the smoothing equations or a precise citation to the definitions used.
- [Eq. (5)] The explanation of kappa/(2m) as the average linked-list traversal length is imprecise; under uniform hashing with separate chaining, the expected number of probes for a successful search is 1 + kappa/(2m), while the average chain length is kappa/m. Please clarify the intended interpretation.
Circularity Check
No circularity found: dpPick's cost-based order selection is validated by external wall-clock runtime, not by its own cost estimates.
full rationale
The claimed derivation chain is: collect per-cycle statistics (γ, µ, κ), smooth them to predict the next cycle, evaluate every admissible probe order with a recursive cost model (Eqs. 4–8), and pick the minimum-cost order; the paper then validates dpPick against fixedOrder, selectivityFirst, and greedy_MSJ using measured wall-clock processing time on TPC-DS (Section 5.2). The optimization objective (estimated cost) is not the same as the measured outcome (runtime), so the central result is not equivalent to its inputs by construction. The cost model is an ansatz with unspecified constants (αq, αm, c, m) rather than a reported fitted parameter; without evidence that those constants or the smoothing parameters were tuned on the same TPC-DS workload, this is a calibration/robustness gap, not a demonstrated circularity. The paper cites prior work for the iterative-probing operator and for baseline algorithms, but none of those citations are by the present authors or load-bearing in a way that forbids alternatives. The experimental design (random order of a static dataset) weakens the claim that the method adapts to changing streams, but that is an external-validity concern, not a reduction of the result to its inputs. No circular step can be exhibited with a specific equation-to-equation or fitted-parameter-to-prediction reduction.
Assumptions & free parameters
free parameters (6)
- alpha_q (query cost coefficient) =
Not reported
- alpha_m (match cost coefficient) =
Not reported
- c (basic hash query cost) =
Not reported
- m (hash table slot count) =
Not reported
- Exponential smoothing parameters (alpha, beta, phi) =
Not reported
- Optimization cycle T and history length L =
T tested from 1 to 20 seconds; L tested up to 200
assumptions (5)
- domain assumption Expected total cost of a probe sequence factorizes as query cost plus success probability times match cost plus recursive cost, as in Eq. 1 and Eq. 8.
- ad hoc to paper The hash-table query cost is well approximated by c + kappa/(2m) and ordered-array cost by log(kappa), as in Eqs. 5 and 6.
- domain assumption Historical statistics smoothed by quadratic exponential smoothing predict next-cycle statistics.
- domain assumption The observed matching rate gamma and average match count mu are properties of the streams and independent of the probe order used to measure them.
- domain assumption The number of input streams is small enough that exhaustive DFS enumeration of all probe orders is feasible at runtime.
Cite this review
Pith. "Pith review of Runtime-optimized Multi-way Stream Join Operator for Large-scale Streaming data." pith.science (2026). https://pith.science/paper/4JRIFPNU
@misc{pith2026241115827,
author = {Pith},
title = {Pith review of: Runtime-optimized Multi-way Stream Join Operator for Large-scale Streaming data},
year = {2026},
howpublished = {\url{https://pith.science/paper/4JRIFPNU}},
note = {Machine review of arXiv:2411.15827}
}
read the original abstract
Streaming computing enables the real-time processing of large volumes of data and offers significant advantages for various applications, including real-time recommendations, anomaly detection, and monitoring. The multi-way stream join operator facilitates the integration of multiple data streams into a single operator, allowing for a more comprehensive understanding by consolidating information from diverse sources. Although this operator is valuable in stream processing systems, its current probe order is determined prior to execution, making it challenging to adapt to real-time and unpredictable data streams, which can potentially diminish its operational efficiency. In this paper, we introduce a runtime-optimized multi-way stream join operator that incorporates various adaptive strategies to enhance the probe order during the joining of multi-way data streams. The operator's runtime operation is divided into cycles, during which relevant statistical information from the data streams is collected and updated. Historical statistical data is then utilized to predict the characteristics of the data streams in the current cycle using a quadratic exponential smoothing prediction method. An adaptive optimization algorithm based on a cost model, namely dpPick, is subsequently designed to refine the probe order, enabling better adaptation to real-time, unknown data streams and improving the operator's processing efficiency. Experiments conducted on the TPC-DS dataset demonstrate that the proposed multi-way stream join method significantly outperforms the comparative method in terms of processing efficiency.
Figures
Reference graph
Works this paper leans on
-
[1]
B. Halstead, Y . S. Koh, P. Riddle, et al. Recurring concept memory management in data streams: exploiting data stream concept evolution to improve performance and transparency. Data Mining and Knowledge Discovery , 35(3):796–836, 2021
work page 2021
-
[2]
M. Zaki, A. J. Lee, and P. K. Chrysanthis. Effective access control in shared-operator multi-tenant data stream management systems. In Data and Applications Security and Privacy XXXIV , pages 118–136, Cham, 2020. Springer International Publishing
work page 2020
- [3]
- [4]
-
[5]
Q. Wang, D. Zuo, Z. Zhang, et al. An adaptive non-migrating load-balanced distributed stream window join system. The Journal of Supercomputing, 79(8):8236–8264, 2023
work page 2023
-
[6]
Maximizing the output rate of multi-way join queries over streaming information sources
Viglas S D, Naughton J F, and Burger J. Maximizing the output rate of multi-way join queries over streaming information sources. In Proceedings 2003 VLDB Conference , pages 285–296, San Francisco, 2003. Morgan Kaufmann
work page 2003
-
[7]
Gedik B, Wu K-L, Yu P S, et al. Grubjoin: An adaptive, multi-way, windowed stream join with time correlation- aware cpu load shedding. IEEE Transactions on Knowledge and Data Engineering , 19(10):1363–1380, 2007
work page 2007
-
[8]
Optimizing Multi-Way Joins for Adaptive, Scale-out Stream Processing
Manuel D. Optimizing Multi-Way Joins for Adaptive, Scale-out Stream Processing . PhD thesis, Rheinland- Pfälzische Technische Universität Kaiserslautern-Landau, Germany, 2023
work page 2023
Show all 38 references
-
[9]
Toward fast theta-join: A prefiltering and amalgamated partitioning approach
Wu J, Wang Y , Fan X, et al. Toward fast theta-join: A prefiltering and amalgamated partitioning approach. Concurrency and Computation: Practice and Experience , 34(17):e6996, 2022
2022
-
[10]
Slidingwindow based multi-join algorithms over distributed data streams
Dongdong Zhang, Jianzhong Li, Kimeli K, et al. Slidingwindow based multi-join algorithms over distributed data streams. In 22nd International Conference on Data Engineering (ICDE’06) , pages 139–139, 2006
2006
-
[11]
Load shedding for multi-way stream joins based on arrival order patterns
Kwon T-H, Lee K Y , and Kim M H. Load shedding for multi-way stream joins based on arrival order patterns. Journal of Intelligent Information Systems , 37(2):245–265, 2011
2011
-
[12]
Scalable stream join processing with expensive predicates: workload distribution and adaptation by time-slicing
Wang S and Rundensteiner E. Scalable stream join processing with expensive predicates: workload distribution and adaptation by time-slicing. In Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, pages 299–310, 2009
2009
-
[13]
Optimizing multi-way theta join for data skew in sub-second stream computing
Fan X, Liu X, Wang Y , et al. Optimizing multi-way theta join for data skew in sub-second stream computing. In 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS) , pages 476–485, 2020
2020
-
[14]
Multi-query optimization in wide-area streaming analytics
Jonathan A, Chandra A, and Weissman J. Multi-query optimization in wide-area streaming analytics. In Proceedings of the ACM Symposium on Cloud Computing , pages 412–425, 2018
2018
-
[15]
Astream: Ad-hoc shared stream processing
Karimov J, Rabl T, and Markl V . Astream: Ad-hoc shared stream processing. In Proceedings of the 2019 International Conference on Management of Data , pages 607–622, 2019
2019
-
[16]
Ajoin: ad-hoc stream joins at scale
Karimov J, Rabl T, and Markl V . Ajoin: ad-hoc stream joins at scale. Proceedings of the VLDB Endowment , 13(4):435–448, 2019
2019
-
[17]
Adaptive optimisation for continuous multi-way joins over rdf streams
Le-Phuoc D. Adaptive optimisation for continuous multi-way joins over rdf streams. In Companion Proceedings of the The Web Conference 2018, pages 1857–1865, 2018. 14
2018
-
[18]
Optimizing multiple multi-way stream joins
Dossinger M and Michel S. Optimizing multiple multi-way stream joins. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) , pages 1985–1990, 2021
2021
-
[19]
Zero-shot cost models for distributed stream processing
Heinrich R, Luthra M, Kornmayer H, et al. Zero-shot cost models for distributed stream processing. InProceedings of the 16th ACM International Conference on Distributed and Event-Based Systems , pages 85–90, 2022
2022
-
[20]
Rate-based query optimization for streaming information sources
Viglas S D and Naughton J F. Rate-based query optimization for streaming information sources. In Proceedings of the 2002 ACM SIGMOD international conference on Management of data , pages 37–48, 2002
2002
-
[21]
Rapid bushy join-order optimization with cartesian products
Vance B and Maier D. Rapid bushy join-order optimization with cartesian products. ACM SIGMOD Record, 25(2):35–46, 1996
1996
-
[22]
Towards multi-way join aware optimizer in sap hana
Wi S, Han W-S, Chang C, et al. Towards multi-way join aware optimizer in sap hana. Proceedings of the VLDB Endowment, 13(12):3019–3031, 2020
2020
-
[23]
jointree: A novel join-oriented multivariate operator for spatio-temporal data management in flink
Ji H, Wu G, Zhao Y , et al. jointree: A novel join-oriented multivariate operator for spatio-temporal data management in flink. GeoInformatica, 27(1):107–132, 2023
2023
-
[24]
Accelerating multi-way joins on the gpu
Lai Z, Sun X, Luo Q, et al. Accelerating multi-way joins on the gpu. The VLDB Journal, 31(3):529–553, 2022
2022
-
[25]
Adaptive optimization of join trees for multi-join queries over sensor streams.Information Fusion, 9(3):412–424, 2008
Gomes J and Choi H-A. Adaptive optimization of join trees for multi-join queries over sensor streams.Information Fusion, 9(3):412–424, 2008
2008
-
[26]
Adaptive continuous query reoptimization over data streams
Park H K and Lee W S. Adaptive continuous query reoptimization over data streams. IEICE Transactions on Information and Systems, E92-D(7):1421–1428, 2009
2009
-
[27]
Pmjoin: Optimizing distributed multi-way stream joins by stream partitioning
Zhou Y , Yan Y , Yu F, et al. Pmjoin: Optimizing distributed multi-way stream joins by stream partitioning. In Database Systems for Advanced Applications , pages 325–341, Berlin, Heidelberg, 2006. Springer
2006
-
[28]
Adaptive ordering of pipelined stream filters
Babu S, Motwani R, Munagala K, et al. Adaptive ordering of pipelined stream filters. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data , pages 407–418, Paris, France, 2004. ACM
2004
-
[29]
Streamon: an adaptive engine for stream query processing
Babu S and Widom J. Streamon: an adaptive engine for stream query processing. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data , pages 931–932, Paris, France, 2004. ACM
2004
-
[30]
Processing sliding window multi-joins in continuous queries over data streams
Golab L and Tamer Özsu M. Processing sliding window multi-joins in continuous queries over data streams. In Proceedings 2003 VLDB Conference, pages 500–511, San Francisco, 2003. Morgan Kaufmann
2003
-
[31]
Scalable distributed stream join processing
Lin Q, Ooi B C, Wang Z, et al. Scalable distributed stream join processing. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data , pages 811–825, Melbourne, Victoria, Australia,
2015
-
[32]
Scaling out multi-way stream joins using optimized, iterative probing
Dossinger M and Michel S. Scaling out multi-way stream joins using optimized, iterative probing. In 2019 IEEE International Conference on Big Data (Big Data) , pages 449–456, Los Angeles, CA, USA, 2019. IEEE
2019
-
[33]
Trijoin: A time-efficient and scalable three-way distributed stream join system
Yu S, Zheng Y , Zhang F, et al. Trijoin: A time-efficient and scalable three-way distributed stream join system. Journal of Internet Technology, 24(2):475–485, 2023
2023
-
[34]
Cai and L
K. Cai and L. Ma. User behavior data analysis of taobao online based on flink-based k-means algorithm. In 2020 International Conference on Applications and Techniques in Cyber Intelligence , pages 852–859, Cham, 2021. Springer International Publishing
2020
-
[35]
L. Liu, H. Zhang, Y . Jing, et al. Learned optimizer for online approximate query processing in data exploration. IEEE Transactions on Knowledge and Data Engineering , 2024(1):1–14, 2024
2024
-
[36]
G. Chen, T. Johnson, and M. Cilimdzic. Quantifying cloud data analytic platform scalability with extended tpc-ds benchmark. In Performance Evaluation and Benchmarking, pages 135–150, Cham, 2022. Springer International Publishing
2022
-
[37]
Y . Hong, S. Du, and J. Leng. Evaluating presto and sparksql with tpc-ds. In Database Systems for Advanced Applications. DASF AA 2022 International Workshops , pages 319–329, Cham, 2022. Springer International Publishing
2022
-
[38]
van Dongen and D
G. van Dongen and D. V . D. Poel. A performance analysis of fault recovery in stream processing frameworks. IEEE Access, 9:93745–93763, 2021. 15
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.