REVIEW 3 major objections 5 minor 47 references
Evaluation for Regression Analyses on Evolving Data Streams
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper introduces a CTGAN-based drift simulator that converts real regression datasets into streams with abrupt, gradual, or incremental concept drift, and a standardized evaluation protocol for them.
desk verdict Novel GAN-based drift simulator for streaming regression, useful but under-validated: the incremental drift claims rest on an assumption that is never tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the chunk-wise CTGAN: the real dataset is sorted by the most target-correlated numeric feature, split into chunks, and each chunk trains its own conditional tabular generative adversarial network, so each chunk defines a concept. The load-bearing step for incremental drift is sorting the transition segment by the drifting feature, then deleting that feature from the stream; this creates a smooth, hard-to-detect transition without letting the model see the feature that orders it.
What would settle it
On a generated incremental stream, estimate the conditional distribution of the target given the remaining features on the first stable segment and on the second stable segment; if the two conditional distributions are statistically indistinguishable, the simulated 'concept drift' is only covariate shift.
Extended reading notes
Core claim
The paper claims that concept drift in regression streams can be synthesized from real data by selecting the numeric feature most correlated with the target, sorting the data by that feature, splitting it into chunks, and training a separate generative model on each chunk so the chunks act as distinct concepts. Abrupt drift is produced by concatenating concept chunks in random order; gradual drift by interleaving the tail of one concept with the head of the next; incremental drift by sorting the transition segment by the drifting feature and then discarding that feature to avoid leakage. The paper further claims that the resulting incremental streams are genuinely harder, since state-of-the-art regressors show lower adjusted R-squared and more turbulent prequential error than on the original data, and that the proposed evaluation protocol reveals how adaptive prediction intervals recover coverage after drift.
Load-bearing premise
The simulation assumes that sorting real data by the most target-correlated feature and training a separate generative model on each chunk produces genuinely different feature–target relationships, but since the drifting feature is later discarded, the observed change may only be a shift in the distribution of an unobserved input.
Editorial extensions
If this is right
- Researchers can generate regression streams with abrupt, gradual, or incremental drift of specified length from public real datasets, with all code and scripts released.
- The proposed evaluation stack—cumulative and prequential versions of RMSE, adjusted R-squared, coverage, and NMPIW—gives a common yardstick for comparing streaming regressors and prediction-interval methods.
- On the synthesized streams, all tested algorithms perform worse than on the original data, and incremental-drift streams produce the most unstable prequential errors, indicating the simulator creates genuinely harder problems.
- Adaptive prediction intervals (AdaPI) recover target coverage faster after drift than static mean-and-variance intervals, at the cost of wider intervals.
- SOKNL and ARF-Reg outperform FIMT-DD and KNN on these streams, suggesting ensemble methods are better suited to drifting regression data.
Reading between the lines
- Editorial extension: because the drifting feature is discarded, the incremental streams may be better described as covariate shift on a hidden driver; the procedure's claim to produce concept drift could be tested by checking whether the conditional target distribution changes across segments.
- Editorial extension: the same chunk-and-CTGAN recipe could be reused to simulate recurrent or mixed drift by cycling or randomly permuting concept order, which would make the benchmark more general.
- Editorial extension: the per-chunk generative modeling step means the fidelity of the simulated drift depends on how well the GAN captures joint feature dependencies; a failure there would weaken the realism of the drift regardless of the sorting procedure.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses stream regression evaluation, proposing a standardized protocol (cumulative and prequential metrics, including RMSE, adjusted R2, coverage, and NMPIW) and a drift simulation strategy based on CTGAN. Real datasets are sorted by a target-correlated feature, split into chunks, and a separate CTGAN is trained on each chunk; concepts are then concatenated (abrupt), interleaved (gradual), or sorted across a transition (incremental) to produce 18 synthetic streams. Four streaming regressors and two prediction-interval methods are evaluated on these streams. The central claim is that the simulator produces genuine concept drifts, especially incremental drift, and that the proposed evaluation process is a useful standard for the community.
Significance. If the drift simulation is valid, the paper offers a practical and much-needed resource: it augments real regression datasets to stream lengths, generates three drift types including incremental drift, and provides open-source code, which promotes reproducibility in a field that lacks dedicated regression stream benchmarks. The evaluation protocol itself is not conceptually new (it combines existing streaming metrics), but the drift simulator is the main contribution. However, the load-bearing claim that the simulated streams contain concept drift is not substantiated; the paper does not verify that P(Y|X) changes across the synthesized concepts, and it does not compare against existing simulators. If the required verification were added, the paper could serve as a useful benchmarking contribution.
major comments (3)
- [5.2.3] For incremental drift, the drifting feature is discarded from the generated data to avoid information leakage, so the visible stream has no dimension along which the transition is ordered. The statement in §5.2.3, "This ensures that the simulated concept drift indeed exists," is therefore unsupported: the paper provides no measurement of a change in P(Y|X) across the transition; it verifies only a change in the marginal distribution of a feature that is then removed. Without direct evidence (e.g., comparing conditional distributions on held-out instances from adjacent concepts, or applying a drift detector to model residuals), the incremental streams may constitute covariate shift on an unobserved driver rather than concept drift, which is exactly the central claim of the paper.
- [5.2, Tables 3 and 4] The comparison between the original datasets (Table 4) and the synthesized datasets (Table 3) shows that all algorithms perform worse on the latter, but this is not evidence that the difference is caused by concept drift. The comparison confounds drift simulation with CTGAN artifacts and with the removal of the drifting feature. The paper does not include a control stream (e.g., a random-order concatenation of chunks from different concepts without a transition, or a stream where the concept order is shuffled) that would isolate the effect of the drift itself. Additionally, no comparison against existing incremental drift simulators (e.g., Hyperplane or RBF) is reported, so the novelty and effectiveness of the GAN-based approach relative to the state of the art are not demonstrated.
- [6.2, Figure 4c] The prequential RMSE plot for the incremental Abalone dataset shows more "turbulence" than the abrupt and gradual cases, and the text interprets this as affirming the effectiveness of the incremental simulation. This inference is not valid: turbulence can result from the independent CTGANs generating incompatible random noise, from a shift in the target marginal distribution, or from covariate shift. The paper does not check whether the performance drops coincide with the known drift boundaries (around 20k and 80k instances), nor whether the change corresponds to a change in the conditional distribution of the target given the observed features. Without such analysis, the visual evidence does not establish that the intended drift type was generated.
minor comments (5)
- [Title] The title "Evaluation for Regressive Analyses on Evolving Data Streams" appears to contain a grammatical error; "Regressive Analyses" should likely be "Regression Analyses".
- [3.2.1, Equation (3)] The typeset of Equation (3) is broken: the square root symbol is rendered as "vt" in the provided text, obscuring the RMSE formula.
- [5.1] The CTGAN hyperparameters (epochs, batch size, learning rate) are stated, but no sensitivity analysis is provided; since CTGAN is the backbone of the simulator, the robustness of the drift simulation to these choices is unknown.
- [Table 4] The "Overflowed" entries for FIMT-DD on several original datasets prevent a complete comparison between the original and synthetic results for those datasets; the paper should either report alternative robust metrics (e.g., median absolute error) or explicitly state the limitation.
- [Figure 1] The caption "Simple Showcase of Feature and Target Values with Simulated Incremental Drifts" is vague; it does not state which feature is plotted, how the target is displayed, or how the figure relates to the simulation procedure described in §5.2.3.
Circularity Check
No significant circularity: the drift simulation and evaluation framework are self-contained; self-citations are not load-bearing.
full rationale
The paper's claimed derivation chain is: select a target-correlated feature, sort and split real data into chunks, train a CTGAN per chunk, concatenate generated streams to form drift types, and evaluate streaming regression and prediction-interval algorithms with standard metrics (RMSE, Adjusted R2, Coverage, NMPIW). No step defines an output in terms of the claim being validated. The drift simulation is an input to the experiments, not a fitted quantity, and the metrics are external to the simulation. The authors evaluate their own previously proposed algorithms, SOKNL and AdaPI, but the benchmark results are not used to derive the simulation method; the central contribution does not depend on those algorithms being state of the art. The potential limitation that discarding the drifting feature in incremental streams may yield only a shift in an unobserved variable's marginal distribution is a validity concern, not a circularity: the paper asserts drift exists without verifying P(Y|X) changes, but that assertion is not made true by definition or by fitting. No self-citation is load-bearing for the main claim, and no equation reduces to its own inputs. Thus the derivation is self-contained and no circular step can be exhibited.
Assumptions & free parameters
free parameters (3)
- CTGAN training hyperparameters =
epochs=300, batch_size=500, lr=0.001
- Drift transition length =
10,000 instances for gradual drifts; 20,000 instances for incremental drifts
- Number of concepts per stream =
4 for abrupt/gradual (3 drifts); 3 stable + 2 drift segments for incremental (2 drifts)
assumptions (4)
- domain assumption CTGAN-generated synthetic data preserves the feature-target relationships of the original real datasets.
- domain assumption Sorting real data on the most target-correlated feature and splitting it into chunks yields meaningful, distinct concepts suitable for drift simulation.
- ad hoc to paper A stream generated by concatenating CTGAN samples from different chunks, possibly with a sorted transition segment, is a faithful analog of a naturally evolving data stream with concept drift.
- domain assumption Pearson correlation is a sufficient criterion for selecting the drifting feature.
Cite this review
Pith. "Pith review of Evaluation for Regression Analyses on Evolving Data Streams." pith.science (2026). https://pith.science/paper/LC4BN7FO
@misc{pith2026250207213,
author = {Pith},
title = {Pith review of: Evaluation for Regression Analyses on Evolving Data Streams},
year = {2026},
howpublished = {\url{https://pith.science/paper/LC4BN7FO}},
note = {Machine review of arXiv:2502.07213}
}
read the original abstract
The paper explores the challenges of regression analysis in evolving data streams, an area that remains relatively underexplored compared to classification. We propose a standardized evaluation process for regression and prediction interval tasks in streaming contexts. Additionally, we introduce an innovative drift simulation strategy capable of synthesizing various drift types, including the less-studied incremental drift. Comprehensive experiments with state-of-the-art methods, conducted under the proposed process, validate the effectiveness and robustness of our approach.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Albert Bifet, Geoff Holmes, Bernhard Pfahringer, Philipp Kranen, Hardy Kremer, Timm Jansen, and Thomas Seidl. 2010. Moa: Massive online analysis, a framework for stream classification and clustering. In Proceedings of the first workshop on applications of pattern analysis . PMLR, 44–50
2010
-
[2]
Albert Bifet, Jesse Read, Indr˙e Žliobait˙e, Bernhard Pfahringer, and Geoff Holmes
-
[3]
Davide Chicco, Matthijs J Warrens, and Giuseppe Jurman. 2021. The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. Peerj computer science 7 (2021), e623
work page 2021
-
[4]
Ajay Choudhary, Preeti Jha, Aruna Tiwari, and Neha Bharill. 2021. A brief survey on concept drifted data stream regression. Soft Computing for Problem Solving: Proceedings of SocProS 2020, Volume 2 (2021), 733–744
work page 2021
-
[5]
Pedro Domingos and Geoff Hulten. 2000. Mining high-speed data streams. In Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining . 71–80
work page 2000
-
[6]
Hadi Fanaee-T and Joao Gama. 2014. Event labeling combining ensemble de- tectors and background knowledge. Progress in Artificial Intelligence 2 (2014), 113–127
work page 2014
-
[7]
Joao Gama. 2012. A survey on learning from data streams: current and future trends. Progress in Artificial Intelligence 1 (2012), 45–55
work page 2012
-
[8]
João Gama, Indr˙e Žliobait˙e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation. ACM computing surveys (CSUR) 46, 4 (2014), 1–37
2014
Show all 47 references
-
[9]
Heitor Murilo Gomes, Jean Paul Barddal, Luis Eduardo Boiko Ferreira, and Albert Bifet. 2018. Adaptive random forests for data stream regression.. In ESANN
2018
-
[10]
Heitor Murilo Gomes and Albert Bifet. 2024. Practical machine learning for streaming data. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 6418–6419
2024
-
[11]
Heitor Murilo Gomes, Maciej Grzenda, Rodrigo Mello, Jesse Read, Minh Huong Le Nguyen, and Albert Bifet. 2022. A survey on semi-supervised learning for delayed partially labelled data streams. Comput. Surveys 55, 4 (2022), 1–42
2022
-
[12]
Heitor Murilo Gomes, Anton Lee, Nuwan Gunasekara, Yibin Sun, Guil- herme Weigert Cassales, Justin Liu, Marco Heyden, Vitor Cerqueira, Maroua Bahri, Yun Sing Koh, Bernhard Pfahringer, and Albert Bifet. 2025. CapyMOA: Efficient Machine Learning for Data Streams in Python. arXiv:...
2025 arXiv
-
[13]
Heitor Murilo Gomes, Jesse Read, and Albert Bifet. 2019. Streaming random patches for evolving data stream classification. In 2019 IEEE international confer- ence on data mining (ICDM) . IEEE, 240–249
2019
-
[14]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems 27 (2014)
2014
-
[15]
Nuwan Gunasekara, Bernhard Pfahringer, Heitor Gomes, and Albert Bifet. 2024. Gradient boosted trees for evolving data streams. Machine Learning 113, 5 (2024), 3325–3352
2024
-
[16]
Nuwan Gunasekara, Bernhard Pfahringer, Heitor Murilo Gomes, and Albert Bifet
-
[17]
Nuwan Gunasekara, Bernhard Pfahringer, Heitor Murilo Gomes, Albert Bifet, and Yun Sing. 2024. Recurrent concept drifts on data streams. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 . 8029–8037
2024
-
[18]
Nuwan Amila Gunasekara. 2023. Advanced Adaptive Classifier Methods for Data Streams. Ph. D. Dissertation. The University of Waikato
2023
-
[19]
Myrianthi Hadjicharalambous, Marios M Polycarpou, and Christos G Panayiotou
-
[20]
Kam Hamidieh. 2018. A data-driven statistical model for predicting the critical temperature of a superconductor. Computational Materials Science 154 (2018), 346–354
2018
-
[21]
Rolland L Hardy. 1971. Multiquadric equations of topography and other irregular surfaces. Journal of geophysical research 76, 8 (1971), 1905–1915
1971
-
[22]
Geoff Hulten, Laurie Spencer, and Pedro Domingos. 2001. Mining time-changing data streams. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining . 97–106
2001
-
[23]
Elena Ikonomovska, Joao Gama, and Sašo Džeroski. 2011. Learning model trees from evolving data streams. Data mining and knowledge discovery 23 (2011), 128–168
2011
-
[24]
Jonathan Jakob, André Artelt, Martina Hasenjäger, and Barbara Hammer. 2022. SAM-kNN regressor for online learning in water distribution networks. In Inter- national Conference on Artificial Neural Networks . Springer, 752–762
2022
-
[25]
Michał Koziarski, Bartosz Krawczyk, and Michał Woźniak. 2017. Radial-based approach to imbalanced data oversampling. In Hybrid Artificial Intelligent Sys- tems: 12th International Conference, HAIS 2017, La Rioja, Spain, June 21-23, 2017, Proceedings 12. Springer, 318–327
2017
-
[26]
Shanmugavelayutham Muthukrishnan et al. 2005. Data streams: Algorithms and applications. Foundations and Trends® in Theoretical Computer Science 1, 2 (2005), 117–236
2005
-
[27]
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. 1995. Abalone. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C55C7W
1995 doi
-
[28]
Burcu Ozek, Zhenyuan Lu, Srinivasan Radhakrishnan, and Sagar Kamarthi. 2024. Uncertainty quantification in neural-network based pain intensity estimation. PLoS One 19, 8 (2024), e0307970
2024
-
[29]
Jingyue Pang, Datong Liu, Yu Peng, and Xiyuan Peng. 2018. Optimize the coverage probability of prediction interval for anomaly detection of sensor-based monitoring series. Sensors 18, 4 (2018), 967
2018
-
[30]
Neha Patki, Roy Wedge, and Kalyan Veeramachaneni. 2016. The Synthetic data vault. In IEEE International Conference on Data Science and Advanced Analytics (DSAA). 399–410. https://doi.org/10.1109/DSAA.2016.49
2016 doi
-
[31]
Karl Pearson. 1896. VII. Mathematical contributions to the theory of evolu- tion.—III. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character 187 (1896), 253–318
-
[32]
Jesse Read and Indre Zliobaite. 2023. Learning from data streams: An overview and update. A vailable at SSRN 4326595 (2023)
2023
-
[33]
Glenn Shafer and Vladimir Vovk. 2008. A Tutorial on Conformal Prediction. Journal of Machine Learning Research 9, 3 (2008)
2008
-
[34]
Vinicius MA Souza, Denis M dos Reis, Andre G Maletzke, and Gustavo EAPA Batista. 2020. Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery 34, 6 (2020), 1805–1858
2020
-
[35]
Charles Spearman. 1961. The proof and measurement of association between two things. (1961)
1961
-
[36]
Yibin Sun, Heitor Murilo Gomes, Bernhard Pfahringer, and Albert Bifet. 2024. Real-Time Energy Pricing in New Zealand: An Evolving Stream Analysis. In Pacific Rim International Conference on Artificial Intelligence . Springer, 91–97
2024
-
[37]
Yibin Sun, Bernhard Pfahringer, Heitor Murilo Gomes, and Albert Bifet. 2022. SOKNL: A novel way of integrating K-nearest neighbours with adaptive random forest regression for data streams. Data Mining and Knowledge Discovery 36, 5 (2022), 2006–2032
2022
-
[38]
Yibin Sun, Bernhard Pfahringer, Heitor Murilo Gomes, and Albert Bifet. 2024. Adaptive Prediction Interval for Data Stream Regression. In Pacific-Asia Confer- ence on Knowledge Discovery and Data Mining . Springer, 130–141
2024
-
[39]
Luis Torgo. 1990. House8L Dataset. Derived from the U.S. Census, 1990. Available at https://www.dcc.fc.up.pt/~ltorgo/Regression/census.html
1990
-
[40]
Scott Wares, John Isaacs, and Eyad Elyan. 2019. Data stream mining: methods and challenges for handling concept drift. SN Applied Sciences 1 (2019), 1–19
2019
-
[41]
Maksymilian Wojtas and Ke Chen. 2020. Feature importance ranking for deep learning. Advances in neural information processing systems 33 (2020), 5105–5114
2020
-
[42]
Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni
-
[43]
Overflowed
Hua Zhong and Li Xu. 2021. An all-batch loss for constructing prediction intervals. Applied Sciences 11, 4 (2021), 1728. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Sun et al. A Appendix We provide supplementary information to the main contents of the paper. A.1 Re...
2021
-
[2013]
In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I 13
Pitfalls in benchmarking data stream classification and how to avoid them. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I 13. Springer, 465–479
2013
-
[2019]
Modeling tabular data using conditional gan.Advances in neural information processing systems 32 (2019)
2019
-
[2020]
Neural Computing and Applications 32, 11 (2020), 6715–6733
Neural network-based construction of online prediction intervals. Neural Computing and Applications 32, 11 (2020), 6715–6733
2020
-
[2023]
In IJCAI
Survey on Online Streaming Continual Learning.. In IJCAI. 6628–6637
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.