REVIEW 4 major objections 8 minor 1 cited by
Variable Selection for Comparing High-dimensional Time-Series Data
T0 review · 4 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Time-Slicing Variable Selection finds which variables and time intervals differ between two multivariate time series when only one realization of each is available.
desk verdict Useful meta-algorithm for comparing single realizations of high-dimensional time series, honestly evaluated but with unaddressed autocorrelation in its p-values. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the argument is the train/test split inside each window: the training columns are used for variable selection, and the test columns are used for the permutation test, so the reported p-value is computed on variables chosen without access to the test data. The wrapper is deliberately generic; the paper's implementations use sparse maximum-mean-discrepancy weights and marginal Wasserstein distances as selectors, and sliced Wasserstein distance as the test statistic. Time-Slicing Variable Selection is the procedure of turning each pair of $D\times T$ series into $B$ pairs of $D\times m_b$ data matrices, inside which time order is discarded and the observations are treated as independent samples.
What would settle it
Run the method on two long autocorrelated time series that are identical everywhere except for a brief change in one variable, slicing coarsely enough that the changed window is mixed with unchanged observations; if p-values become small on slices far from the change at a rate far above the nominal level, the independence approximation is falsified.
Extended reading notes
Core claim
The central claim is that variable selection for two multivariate time series can be reduced to repeated two-sample variable selection on time-localized submatrices, making it unnecessary to observe many independent trajectories of each series. The paper proposes to ignore the order of time points within each subinterval, select variables on a training split, and use a permutation test on the test split of the selected variables. In synthetic data with a known planted difference, the approach recovers the correct variable on the subintervals that overlap the change, and it fails in a constructed case where time-slicing makes the marginal distributions coincide. In applications, it flags discrepancies between a deep-network fluid emulator and a particle-based simulator and tracks the spatial pattern of traffic rerouting after road closures.
Load-bearing premise
The assumption that after randomizing time order the observations inside each subinterval behave like independent draws is the load-bearing premise; if consecutive observations are autocorrelated, the permutation p-values are only approximate and the selected variable sets carry uncontrolled uncertainty.
Editorial extensions
If this is right
- A simulator run can be checked against a single observed run, reporting which variables and which time windows carry the discrepancy.
- An emulator can be validated against its simulator, with failures localized in both space and time rather than summarized by one global score.
- Changing a simulator parameter can be compared at the output level, showing when the effect first appears and which sensors or road segments react.
- Because the selector is a plug-in component, the same time-slicing wrapper works with other two-sample variable-selection algorithms.
- The number of slices controls the resolution-sample-size trade-off, so the user can tune the slicing to the expected duration of differences.
Reading between the lines
- Beyond the paper, a natural correction for autocorrelated outputs would replace the within-window permutation test with a block-permutation version; the paper does not develop this.
- The same time-slicing wrapper could be paired with frequency-domain or spectral two-sample statistics to detect differences in dynamics that marginal distributions miss, a direction the paper leaves open.
- Seen as a diagnostic, the per-window selected variables form a time-varying attention map that could point a simulator developer to the mechanism behind a discrepancy, extending the paper's validation framing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes Time-Slicing Variable Selection, a meta-algorithm for comparing a single pair of multivariate time series. The method splits each series into B subintervals, ignores time order within each subinterval, applies an existing two-sample variable selection method on a training portion, and performs a permutation test on the held-out test portion using the selected variables. Synthetic experiments with two settings, a fluid-emulator validation, and a traffic simulation comparison are presented. The central claim is that the approach identifies subintervals and variables where the two series differ significantly using only one realization per series.
Significance. If valid, the approach would be practically useful for simulator validation, especially in settings where multiple runs are too expensive. The train/test split within each subinterval is a sensible way to avoid selection bias in the final permutation test, and the paper is transparent about the independence assumption. However, the statistical validity of the reported p-values under temporal dependence is not established, and the empirical evidence is currently too limited to support the strength of the claimed conclusions. The two real-data demonstrations are illustrative but lack ground truth, so they cannot validate the method by themselves.
major comments (4)
- [Section 2.3; Algorithm 1 lines 3-6] The method explicitly ignores time order within each subinterval, stating in Section 2.3 that "we essentially treat data vectors in X_b ... as independent," which makes the permutation test p-values exact only when observations are exchangeable under the null. In the fluid and traffic applications, consecutive time steps are strongly autocorrelated, so this assumption is violated. The synthetic experiments in Section 4 (Eqs. 4.1-4.2) use iid Gaussian noise and therefore never test the autocorrelated regime. The reported p-values in Sections 5-6 are thus heuristic rather than statistically calibrated. Please add experiments with autocorrelated noise (e.g., AR(1)) and report type I error rates, or use dependence-robust resampling (e.g., block bootstrap) and justify the validity of the resulting p-values.
- [Section 4.3] The synthetic validation uses only three independent realizations per setting. With n=3, the means and standard deviations shown in Figures 5-6 give no reliable estimate of p-value calibration or power. There is also no null scenario where X and Y are generated from the same process, so the observation that "producing large p-values outside intervals 3,4 and 5 is correct" is not substantiated by a false-positive rate. Please increase the number of replications (at least tens), include a null case, and report the empirical type I error at the nominal 0.05 level.
- [Section 4.2] The MsKernel-Gaussian and MsKernel-IMQ baselines are given the true number of ground-truth variables (1) as input, which is an oracle advantage not available to the other methods. This makes the comparison in Figures 5-6 unfair. Please report their performance with a non-oracle selection rule (e.g., choosing the number of variables by a data-driven rule) or clearly state the oracle setting and its implications for the conclusions.
- [Sections 4 and 5; Figure 9] The paper does not address multiple testing across subintervals: with B=10, the probability of at least one small p-value under the global null is much larger than 0.05. Since the method's output includes p_b for each b, a multiple-testing correction (e.g., Benjamini-Hochberg) or an explicit exploratory interpretation is needed to support claims of significant differences on particular subintervals.
minor comments (8)
- [Section 4.3] Typo: "identiy" should be "identify".
- [Section 5.1] Typo: "Wassertstein" should be "Wasserstein".
- [Section 2.3] Typo: "top tow figures" should be "top two figures".
- [Eq. (4.1)] The phrase "independence zero-mean Gaussian noises" should be "independent zero-mean Gaussian noises".
- [Section 2.2, Algorithm 1 line 6] The omitted variable name makes the statement incomplete: "the marginal distributions of X_b^(te) and Y_b^(te) on are the same" should specify "on Ŝ_b".
- [Figures 5 and 6 captions] The caption text contains "recallsores", which appears to be a typo for "recall scores".
- [Throughout (e.g., Sections 3.2, 4.2)] The naming of the marginal-MMD method is inconsistent: "Mskernel" and "MsKernel" both appear; please standardize.
- [Section 7] The conclusion lists the subinterval-length trade-off as a limitation but does not mention the independence/exchangeability assumption, which is at least as important; please add it to the limitations discussion.
Circularity Check
No circular derivation: p-values come from held-out test slices and synthetic evaluation is anchored to external ground truth; self-cited MMD modules add an inheritance burden but not a circular step.
full rationale
The central claim is a meta-algorithm, not a derived first-principles identity. Algorithm 1 splits each series into subintervals, randomly partitions each subinterval into training and test portions (lines 3-4), performs variable selection on training only (line 5), and computes the permutation-test p-value on the held-out test portion restricted to the selected variables (line 6). The test portion was not used for selection, so the p-value is not equal by construction to the selection criterion; under the paper's explicit within-slice exchangeability assumption it is a standard conditional permutation test. The synthetic experiments (Section 4, Eqs. 4.1-4.2) use known ground-truth variables and intervals, so the evaluation is externally anchored rather than defined by the method's output. The main inherited element is the MMD-based variable-selection machinery (ARD-weight optimisation, thresholding, MMD-Selection, MMD-CV-AGG), cited from Mitsuzawa et al. [18], which shares authors with this paper; no independent implementation or formal verification is provided. This is a self-citation and an inheritance burden, but it is not load-bearing circularity: the framework does not invoke that work to prove its own validity, and its conclusions are tested against synthetic ground truth and two external simulators. The limitations stated in Sections 2.3, 4, and 7 (within-slice independence, trade-off in subinterval length, and the p-value-1.0 convention when no variable is selected) are substantive assumptions and implementation conventions; they weaken the statistical guarantees under autocorrelation but do not reduce any reported prediction to the method's inputs. Overall, no circular step is exhibited; the score reflects the mild self-citation burden.
Assumptions & free parameters
free parameters (5)
- Number of subintervals B =
10 or 2 in synthetic experiments; 6 in fluid; 8 in traffic
- Train-test ratio rho_train =
0.8
- MMD regularization parameter lambda =
Optimized by Optuna in ranges such as [1e-6, 2] and aggregated in MMD-CV-AGG
- Mskernel number of selected variables =
Set to 1, the true number of ground-truth variables, in synthetic experiments
- Kernel bandwidths gamma_d =
Median or mean heuristic per dimension
assumptions (3)
- domain assumption Observations within each subinterval are i.i.d. or exchangeable after randomization
- domain assumption The two-sample variable selection methods import reliable power from prior work
- domain assumption Grid counts of particle positions or road vehicle counts are informative representations
Cite this review
Pith. "Pith review of Variable Selection for Comparing High-dimensional Time-Series Data." pith.science (2026). https://pith.science/paper/Z34ZONPB
@misc{pith2026241206870,
author = {Pith},
title = {Pith review of: Variable Selection for Comparing High-dimensional Time-Series Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z34ZONPB}},
note = {Machine review of arXiv:2412.06870}
}
read the original abstract
Given a pair of multivariate time-series data of the same length and dimensions, an approach is proposed to select variables and time intervals where the two series are significantly different. In applications where one time series is an output from a computationally expensive simulator, the approach may be used for validating the simulator against real data, for comparing the outputs of two simulators, and for validating a machine learning-based emulator against the simulator. With the proposed approach, the entire time interval is split into multiple subintervals, and on each subinterval, the two sample sets are compared to select variables that distinguish their distributions and a two-sample test is performed. The validity and limitations of the proposed approach are investigated in synthetic data experiments. Its usefulness is demonstrated in an application with a particle-based fluid simulator, where a deep neural network model is compared against the simulator, and in an application with a microscopic traffic simulator, where the effects of changing the simulator's parameters on traffic flows are analysed.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
Post Hoc Inference for Component Attribution in Multivariate Change-Point Detection
After a multivariate change-point is detected, a grid-based or sample-splitting two-sample test determines whether a pre-specified block of coordinates changed, with Type I error bounded by α0+α1.
Reference graph
Works this paper leans on
-
[1]
Optuna: A next-generation hyperparameter optimization framework
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. InThe 25th ACM SIGKDD Inter- national Conference on Knowledge Discovery & Data Mining, pages 2623–2631, 2019. doi: 10.1145/3292500.3330701
arXiv 2019
-
[2]
Jan Bender et al. SPlisHSPlasH Library. N.A. URL https://github.com/ InteractiveComputerGraphics/SPlisHSPlasH
-
[3]
Sliced and Radon Wasser- steinBarycentersofMeasures
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasser- steinBarycentersofMeasures. JournalofMathematicalImagingandVision ,1(51):22–45,2015. doi: 10.1007/s10851-014-0506-3. URL https://hal.science/hal-00881872
-
[4]
Katherine Campbell, Michael D McKay, and Brian J Williams. Sensitivity analysis when model outputs are functions.Reliability Engineering & System Safety, 91(10-11):1468–1472, 2006
work page 2006
-
[5]
Representation learning for a generalized, quantitative comparison of complex model outputs
Colin G Cess and Stacey D Finley. Representation learning for a generalized, quantitative comparison of complex model outputs.arXiv preprint arXiv:2208.06530, 2022
work page Pith review arXiv 2022
-
[6]
Lara Codeca and Jérôme Härri. Monaco SUMO Traffic (MoST) Scenario: A 3D Mobility ScenarioforCooperativeITS.In SUMO2018-SimulatingAutonomousandIntermodalTransport Systems,volume2of EPiCSeriesinEngineering ,pages43–55.EasyChair,2018. doi: 10.29007/ 1zt5. URL https://easychair.org/publications/paper/x3nd
work page 2018
-
[7]
Bradley Efron and RJ Tibshirani.An Introduction to the Bootstrap. CRC Press, 1994
work page 1994
-
[8]
Trafficgen: Learning to generate diverse and realistic traffic scenarios
Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and realistic traffic scenarios. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3567–3575. IEEE, 2023
work page 2023
Show all 36 references
-
[9]
Large sample analysis of the median heuristic.arXiv preprint arXiv:1707.07269, 2018
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic.arXiv preprint arXiv:1707.07269, 2018
2018 arXiv
-
[10]
Borgwardt, Malte J
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773,
-
[11]
Un- supervised change analysis using supervised learning
Shohei Hido, Tsuyoshi Idé, Hisashi Kashima, Harunobu Kubo, and Hirofumi Matsuzawa. Un- supervised change analysis using supervised learning. InProceedings of the 12th Pacific-Asia ConferenceonAdvancesinKnowledgeDiscoveryandDataMining ,PAKDD’08,pages148–159, 2008
2008
-
[12]
URLhttp://arxiv.org/abs/1412.6980
DiederikP.KingmaandJimmyBa.Adam: Amethodforstochasticoptimization.In International Conference on Learning Representations, 2015. URLhttp://arxiv.org/abs/1412.6980
2015 arXiv
-
[13]
Mcgraw-hill New York, 5th edition, 2014
Averill M Law.Simulation Modeling and Analysis. Mcgraw-hill New York, 5th edition, 2014
2014
-
[14]
Morepowerfulselectivekerneltestsforfeatureselection
Jen Ning Lim, Makoto Yamada, Wittawat Jitkrittum, Yoshikazu Terada, Shigeyuki Matsui, and HidetoshiShimodaira. Morepowerfulselectivekerneltestsforfeatureselection. In Proceedings 24 of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume ...
2020
-
[15]
Microscopictrafficsimulationusingsumo
Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flöt- teröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopictrafficsimulationusingsumo. In The21stIEEEInternationalConference on Intelligent Tra...
2018
-
[16]
Revisiting classifier two-sample tests
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. InInternational Conference on Learning Representations, 2017
2017
-
[17]
Digital twin- driven smart manufacturing: Connotation, reference model, applications and research issues
Yuqian Lu, Chao Liu, Kevin I-Kai Wang, Huiyue Huang, and Xun Xu. Digital twin- driven smart manufacturing: Connotation, reference model, applications and research issues. Robotics and Computer-Integrated Manufacturing, 61:101837, 2020. ISSN 0736-5845. doi: 10.1016/j.rcim.2019....
2020
-
[18]
arXiv preprint arXiv:2311.01537, 2023
Kensuke Mitsuzawa, Motonobu Kanagawa, Stefano Bortoli, Margherita Grossi, and Paolo Pa- potti.Variableselectioninmaximummeandiscrepancyforinterpretabledistributioncomparison. arXiv preprint arXiv:2311.01537, 2023
2023
-
[19]
Principal differences analysis: Interpretable characteriza- tionofdifferencesbetweendistributions
Jonas W Müller and Tommi Jaakkola. Principal differences analysis: Interpretable characteriza- tionofdifferencesbetweendistributions. NeuralInformationProcessingSystems(NeurIPS) ,28, 2015
2015
-
[20]
PyTorch: an imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...
2019
-
[21]
Guaranteed con- servation of momentum for learning particle-based fluid dynamics
Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guaranteed con- servation of momentum for learning particle-based fluid dynamics. InConference on Neural Information Processing Systems, 2022
2022
-
[22]
Construction with digital twin information systems.Data-Centric Engineering, 1:e14, 2020
Rafael Sacks, Ioannis Brilakis, Ergo Pikas, Haiyan Sally Xie, and Mark Girolami. Construction with digital twin information systems.Data-Centric Engineering, 1:e14, 2020. doi: 10.1017/ dce.2020.16
2020
-
[23]
Battaglia
AlvaroSanchez-Gonzalez,JonathanGodwin,TobiasPfaff,RexYing,JureLeskovec,andPeterW. Battaglia. Learningtosimulatecomplexphysicswithgraphnetworks. In Proceedingsofthe37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020
2020
-
[24]
B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R.G. Lanckriet. Hilbert spaceembeddingsandmetricsonprobabilitymeasures. JournalofMachineLearningResearch , 11:1517–1561, 2010
2010
-
[25]
Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton
Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum 25 mean discrepancy. In International Conference on Learning Representations, 2017. URL https://open...
2017
-
[26]
A method for comparing multivariate time series with different dimensions.PloS One, 8(2):e54201, 2013
Avraam Tapinos and Pedro Mendes. A method for comparing multivariate time series with different dimensions.PloS One, 8(2):e54201, 2013
2013
-
[27]
Regression shrinkage and selection via the lasso
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267–288, 1996. ISSN 00359246. URL http://www.jstor.org/stable/2346178
1996
-
[28]
Lagrangian fluid sim- ulationwithcontinuousconvolutions
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid sim- ulationwithcontinuousconvolutions. In InternationalConferenceonLearningRepresentations , 2020
2020
-
[29]
Springer, 2009
Cédric Villani.Optimal Transport: Old and New. Springer, 2009
2009
-
[30]
Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt,MatthewBrett,JoshuaWilson,K.JarrodMillman,NikolayMayorov,AndrewR.J.Nelson, Eric Jones, ...
2020
-
[31]
Variable selection for kernel two-sample tests.arXiv preprint arXiv:2302.07415, 2023
Jie Wang, Santanu S Dey, and Yao Xie. Variable selection for kernel two-sample tests.arXiv preprint arXiv:2302.07415, 2023
2023 arXiv
-
[32]
Akerneltwo-sampletestforfunctionaldata
GeorgeWynneandAndrewB.Duncan. Akerneltwo-sampletestforfunctionaldata. Journalof Machine Learning Research, 23(1), jan 2022. ISSN 1532-4435
2022
-
[33]
Post selection inference with incomplete maximum mean discrepancy estimator
Makoto Yamada, Denny Wu, Yao-Hung Hubert Tsai, Hirofumi Ohta, Ruslan Salakhutdinov, Ichiro Takeuchi, and Kenji Fukumizu. Post selection inference with incomplete maximum mean discrepancy estimator. InInternational Conference on Learning Representations, 2019. 26 A Details of M...
2019
-
[35]
It stops the optimisation when the objective value does not change significantly for the past100 epochs
Convergence-basedstopper: Thisstoppingcriterionstartsmonitoringtheobjectivevalue(3.3) after the initial 200 epochs. It stops the optimisation when the objective value does not change significantly for the past100 epochs. More specifically, the optimisation is stopped if the mi...
-
[36]
training
Variable-selection-based stopper:After the initial 400 epochs, the stopper starts performing variable selection every 10 epochs (by the thresholding algorithm applied to the ARD weights). It stops the optimisation if the selected variables have not changed for the past100 epoc...
-
[2012]
URL http://jmlr.org/papers/v13/gretton12a.html
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.