{"id":"307d00b8-68d8-47d6-b4d3-94a210acc28d","arxiv_id":"2412.06870","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Time-Slicing Variable Selection splits two time series into subintervals, selects distinguishing variables on each subinterval with a two-sample method, and returns a permutation-test p-value per subinterval.","lead":"A new framework, Time-Slicing Variable Selection, compares two multivariate time series by slicing time into subintervals and running two-sample variable selection on each slice. It targets simulator validation and emulator checking when only one run of each system is available, avoiding the need for many repeated simulations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Within-slice exchangeability underlies every reported p-value; autocorrelated series, the paper's target applications, violate it, and the synthetic experiments never test this regime.","rationale":"I read the paper as proposing a practical meta-algorithm that is clearly described and honestly evaluated on synthetic data, and the authors explicitly acknowledge the subinterval-length trade-off in Section 7. However, the central deliverable is a set of selected variables and p-values per subinterval, and every p-value is computed under the assumption that observations within a subinterval are exchangeable. Section 2.3 says the method 'essentially treat[s] data vectors in X_b ... as independent.' This assumption is not an internal inconsistency; it is an unverified condition that is directly load-bearing for the paper's motivating regime. The fluid and traffic applications produce counts at consecutive time steps that are highly autocorrelated, yet the synthetic experiments generate independent Gaussian noise, so the validity of the permutation p-values in the target regime is untested. A calibration experiment with autocorrelated null data would settle whether the concern lands: if null p-values are not approximately uniform, then the reported significance statements in Sections 5 and 6 are not formally justified. The reader's weakest assumption identifies exactly this issue, and I agree with it. Because this is a condition on validity rather than a demonstrated false result, the appropriate verdict remains CONDITIONAL: the paper should either restrict its claims to weakly dependent slices, provide a block-permutation or bootstrap correction, or demonstrate via the autocorrelated-null experiment that the approximation is acceptable.","tokens_in":20442,"tokens_out":4695,"duration_ms":53314,"concrete_test":"Run a null-calibration experiment: generate X and Y as two independent realizations of the same D=5 VAR(1) (or AR(1) per variable) process with identical stationary marginals, T=1000, B=10 subintervals, rho_train=0.8, and apply Algorithm 1 exactly as in Section 4, using MMD-CV-AGG for selection and sliced Wasserstein for the permutation test. Repeat at least 200 times and record p_b on subintervals where the null holds. If the empirical frequency of p_b <= 0.05 is substantially above 5% (e.g., >10%), the permutation test is not controlling type I error under autocorrelation and the central claim needs qualification; a control condition with i.i.d. data should show approximately 5%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (lines 3-6) randomly splits and implicitly permutes time order within each subinterval, and Section 2.3 states: 'we essentially treat data vectors in X_b ... as independent.' For each reported p-value p_b to be valid for the stated null hypothesis, the test-set observations must be exchangeable under the null. In the fluid and traffic applications, counts on adjacent time steps are strongly autocorrelated. Even when the marginal distributions of X_b and Y_b coincide, the joint distribution is not invariant under arbitrary time permutations, so the permutation null distribution is not the true null distribution. The random train/test split in lines 3-4 also fails to make training and test sets independent under autocorrelation, so variable selection on the training portion can be informative about the test portion even under the null. The synthetic validation (Section 4) uses i.i.d. Gaussian noise (Eqs. 4.1-4.2), which is exactly the regime where exchangeability holds; thus it provides no evidence about the autocorrelated regime that motivates the paper. Until this is addressed, the reported p-values and selected intervals in Sections 5-6 are heuristic rather than statistically calibrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Time-Slicing Variable Selection, a meta-algorithm for comparing a single pair of multivariate time series. The method splits each series into B subintervals, ignores time order within each subinterval, applies an existing two-sample variable selection method on a training portion, and performs a permutation test on the held-out test portion using the selected variables. Synthetic experiments with two settings, a fluid-emulator validation, and a traffic simulation comparison are presented. The central claim is that the approach identifies subintervals and variables where the two series differ significantly using only one realization per series.","tokens_in":20669,"tokens_out":5277,"duration_ms":54533,"significance":"If valid, the approach would be practically useful for simulator validation, especially in settings where multiple runs are too expensive. The train/test split within each subinterval is a sensible way to avoid selection bias in the final permutation test, and the paper is transparent about the independence assumption. However, the statistical validity of the reported p-values under temporal dependence is not established, and the empirical evidence is currently too limited to support the strength of the claimed conclusions. The two real-data demonstrations are illustrative but lack ground truth, so they cannot validate the method by themselves.","major_comments":[{"comment":"The method explicitly ignores time order within each subinterval, stating in Section 2.3 that \"we essentially treat data vectors in X_b ... as independent,\" which makes the permutation test p-values exact only when observations are exchangeable under the null. In the fluid and traffic applications, consecutive time steps are strongly autocorrelated, so this assumption is violated. The synthetic experiments in Section 4 (Eqs. 4.1-4.2) use iid Gaussian noise and therefore never test the autocorrelated regime. The reported p-values in Sections 5-6 are thus heuristic rather than statistically calibrated. Please add experiments with autocorrelated noise (e.g., AR(1)) and report type I error rates, or use dependence-robust resampling (e.g., block bootstrap) and justify the validity of the resulting p-values.","section":"Section 2.3; Algorithm 1 lines 3-6"},{"comment":"The synthetic validation uses only three independent realizations per setting. With n=3, the means and standard deviations shown in Figures 5-6 give no reliable estimate of p-value calibration or power. There is also no null scenario where X and Y are generated from the same process, so the observation that \"producing large p-values outside intervals 3,4 and 5 is correct\" is not substantiated by a false-positive rate. Please increase the number of replications (at least tens), include a null case, and report the empirical type I error at the nominal 0.05 level.","section":"Section 4.3"},{"comment":"The MsKernel-Gaussian and MsKernel-IMQ baselines are given the true number of ground-truth variables (1) as input, which is an oracle advantage not available to the other methods. This makes the comparison in Figures 5-6 unfair. Please report their performance with a non-oracle selection rule (e.g., choosing the number of variables by a data-driven rule) or clearly state the oracle setting and its implications for the conclusions.","section":"Section 4.2"},{"comment":"The paper does not address multiple testing across subintervals: with B=10, the probability of at least one small p-value under the global null is much larger than 0.05. Since the method's output includes p_b for each b, a multiple-testing correction (e.g., Benjamini-Hochberg) or an explicit exploratory interpretation is needed to support claims of significant differences on particular subintervals.","section":"Sections 4 and 5; Figure 9"}],"minor_comments":[{"comment":"Typo: \"identiy\" should be \"identify\".","section":"Section 4.3"},{"comment":"Typo: \"Wassertstein\" should be \"Wasserstein\".","section":"Section 5.1"},{"comment":"Typo: \"top tow figures\" should be \"top two figures\".","section":"Section 2.3"},{"comment":"The phrase \"independence zero-mean Gaussian noises\" should be \"independent zero-mean Gaussian noises\".","section":"Eq. (4.1)"},{"comment":"The omitted variable name makes the statement incomplete: \"the marginal distributions of X_b^(te) and Y_b^(te) on are the same\" should specify \"on Ŝ_b\".","section":"Section 2.2, Algorithm 1 line 6"},{"comment":"The caption text contains \"recallsores\", which appears to be a typo for \"recall scores\".","section":"Figures 5 and 6 captions"},{"comment":"The naming of the marginal-MMD method is inconsistent: \"Mskernel\" and \"MsKernel\" both appear; please standardize.","section":"Throughout (e.g., Sections 3.2, 4.2)"},{"comment":"The conclusion lists the subinterval-length trade-off as a limitation but does not mention the independence/exchangeability assumption, which is at least as important; please add it to the limitations discussion.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a framework/application paper rather than a fully validated statistical method. The core idea is reasonable and the use of a held-out test split to reduce selection effects is commendable. However, the absence of any treatment of temporal dependence is a serious gap given the target applications, and the synthetic evidence is very thin. The authors may be able to address the major comments with additional experiments and a clear caveat, so I do not recommend rejection, but the current version should not be accepted without these changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest but real contribution. The time-slicing idea is simple: split each series into subintervals, treat each subinterval's columns as i.i.d. samples, run any two-sample variable selector on a training split, then compute a permutation p-value on a test split using the selected variables. To my knowledge, the problem formulation—identifying variables and time intervals from a single pair of multivariate time series—is new, and the meta-algorithm cleanly separates the slicing from the choice of selector. The paper does this well: the synthetic setup has clear ground truth, the train/test split is a nice touch that protects the final p-values from selection effects, and the authors are upfront about the trade-off between subinterval length and sample size in Section 7. The two applications (fluid emulator validation and traffic scenario comparison) are real and the results are interpretable.\n\nThe soft spots are real but not fatal. The within-slice independence assumption is acknowledged in Section 2.3, but the consequence for autocorrelated series—which are the target applications—is not discussed. The permutation p-values are only exact if the test-set vectors are exchangeable; for traffic counts and fluid states, consecutive time steps are strongly dependent. The synthetic experiments use i.i.d. Gaussian noise, so they never probe this regime. The paper also provides no type I error study, only three realizations per setting, and no code or data for the core method. These are all fixable in revision.\n\nI agree with the reader's conditional verdict. The central claim—that the framework can be a useful diagnostic—holds up, but the statistical calibration of the p-values is overstated for autocorrelated data. The authors should either weaken the interpretation (call the p-values heuristic) or provide evidence under autocorrelation (e.g., block-permutation or pre-whitening).\n\nWho this is for: practitioners doing simulator or emulator validation who want a cheap, interpretable diagnostic. A statistical referee would benefit from engaging; the paper deserves a serious review, not a desk reject. I'd ask for code, a simulation study with autocorrelated data, and a discussion of the exchangeability violation before publication.\n\nRecommendation: send to peer review, but the revision should address the dependence issue and reproducibility.","headline":"Useful meta-algorithm for comparing single realizations of high-dimensional time series, honestly evaluated but with unaddressed autocorrelation in its p-values.","tokens_in":21174,"tokens_out":2629,"would_cite":false,"duration_ms":26608,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H15","62M10","62G09"],"pacs":[],"model":"deepseek-v4-flash","headline":"Time-Slicing Variable Selection finds which variables and time intervals differ between two multivariate time series when only one realization of each is available.","keywords":["multivariate time series","variable selection","two-sample testing","permutation test","maximum mean discrepancy","simulator validation","time slicing","high-dimensional data"],"falsifier":"Run the method on two long autocorrelated time series that are identical everywhere except for a brief change in one variable, slicing coarsely enough that the changed window is mixed with unchanged observations; if p-values become small on slices far from the change at a rate far above the nominal level, the independence approximation is falsified.","tokens_in":20250,"feed_emoji":"📈","tokens_out":7340,"duration_ms":74476,"temperature":0.7,"pith_summary":"The paper tackles a practical problem: given one recorded trajectory from each of two multivariate time series, identify which variables and which time intervals differ, so that a simulator can be checked against data or against a learned emulator without running many expensive simulations. It proposes Time-Slicing Variable Selection, a meta-algorithm that cuts both series into aligned subintervals, carries out a two-sample variable-selection task inside each subinterval, and then performs a permutation test on the selected variables using a held-out part of that subinterval. The output is, for every time window, a set of variables and a p-value that together say where and when the two series are significantly different. The paper's experiments support the usefulness of the approach for validating a neural fluid emulator and for locating the traffic effects of closing roads.","feed_headline":"Time-slicing finds where and when two series diverge","feed_subtitle":"A meta-algorithm applies two-sample variable selection inside each window, so one pair of trajectories yields per-window p-values.","key_machinery":"The object that carries the argument is the train/test split inside each window: the training columns are used for variable selection, and the test columns are used for the permutation test, so the reported p-value is computed on variables chosen without access to the test data. The wrapper is deliberately generic; the paper's implementations use sparse maximum-mean-discrepancy weights and marginal Wasserstein distances as selectors, and sliced Wasserstein distance as the test statistic. Time-Slicing Variable Selection is the procedure of turning each pair of $D\\times T$ series into $B$ pairs of $D\\times m_b$ data matrices, inside which time order is discarded and the observations are treated as independent samples.","core_discovery":"The central claim is that variable selection for two multivariate time series can be reduced to repeated two-sample variable selection on time-localized submatrices, making it unnecessary to observe many independent trajectories of each series. The paper proposes to ignore the order of time points within each subinterval, select variables on a training split, and use a permutation test on the test split of the selected variables. In synthetic data with a known planted difference, the approach recovers the correct variable on the subintervals that overlap the change, and it fails in a constructed case where time-slicing makes the marginal distributions coincide. In applications, it flags discrepancies between a deep-network fluid emulator and a particle-based simulator and tracks the spatial pattern of traffic rerouting after road closures.","pith_inferences":["Beyond the paper, a natural correction for autocorrelated outputs would replace the within-window permutation test with a block-permutation version; the paper does not develop this.","The same time-slicing wrapper could be paired with frequency-domain or spectral two-sample statistics to detect differences in dynamics that marginal distributions miss, a direction the paper leaves open.","Seen as a diagnostic, the per-window selected variables form a time-varying attention map that could point a simulator developer to the mechanism behind a discrepancy, extending the paper's validation framing."],"forward_implications":["A simulator run can be checked against a single observed run, reporting which variables and which time windows carry the discrepancy.","An emulator can be validated against its simulator, with failures localized in both space and time rather than summarized by one global score.","Changing a simulator parameter can be compared at the output level, showing when the effect first appears and which sensors or road segments react.","Because the selector is a plug-in component, the same time-slicing wrapper works with other two-sample variable-selection algorithms.","The number of slices controls the resolution-sample-size trade-off, so the user can tune the slicing to the expected duration of differences."],"supporting_citations":[{"why":"Supplies the sparse ARD-weight MMD variable-selection algorithm and the histogram thresholding used inside each subinterval.","marker":"[18]"},{"why":"Defines the MMD estimator that the sparse selector optimizes.","marker":"[10]"},{"why":"Introduces optimized ARD weights for MMD test power, which the sparse version extends.","marker":"[25]"},{"why":"Provides the sliced Wasserstein distance used as the permutation test statistic on each subinterval.","marker":"[3]"},{"why":"Justifies the permutation two-sample test used to obtain per-window p-values.","marker":"[7]"},{"why":"Supplies the particle-fluid simulation benchmark whose simulator output is compared with a learned emulator.","marker":"[23]"},{"why":"Provides the deep-network fluid emulator that is validated against the simulator.","marker":"[21]"},{"why":"Supplies the microscopic traffic simulator used in the road-blocking model-comparison experiment.","marker":"[15]"},{"why":"Defines the large-scale traffic scenario whose original and modified versions are compared.","marker":"[6]"}],"fun_headline_variants":["Compare two time series without repeated trials","Time-window trick pins down source of divergence","Slicing time reveals where two series truly differ","One pair of trajectories enough to spot differences","Per-window variable selection localizes differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that after randomizing time order the observations inside each subinterval behave like independent draws is the load-bearing premise; if consecutive observations are autocorrelated, the permutation p-values are only approximate and the selected variable sets carry uncontrolled uncertainty.","fun_headline_variants_meta":{"raw":{"variants":["Compare two time series without repeated trials","Time-window trick pins down source of divergence","Slicing time reveals where two series truly differ","One pair of trajectories enough to spot differences","Per-window variable selection localizes differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1259,"prompt_tokens":856,"completion_tokens":403,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":472,"tokens_out":403,"duration_ms":4505,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:39:38.021450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the method on two long autocorrelated time series that are identical everywhere except for a brief change in one variable, slicing coarsely enough that the changed window is mixed with unchanged observations; if p-values become small on slices far from the change at a rate far above the nominal level, the independence approximation is falsified.","supporting_citations":[{"cited_title":"Guaranteed con- servation of momentum for learning particle-based fluid dynamics","cited_arxiv_id":null,"evidence_quote":"Provides the deep-network fluid emulator that is validated against the simulator."},{"cited_title":"Borgwardt, Malte J","cited_arxiv_id":null,"evidence_quote":"Defines the MMD estimator that the sparse selector optimizes."},{"cited_title":"Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton","cited_arxiv_id":null,"evidence_quote":"Introduces optimized ARD weights for MMD test power, which the sparse version extends."},{"cited_title":"CRC Press, 1994","cited_arxiv_id":null,"evidence_quote":"Justifies the permutation two-sample test used to obtain per-window p-values."},{"cited_title":"Battaglia","cited_arxiv_id":null,"evidence_quote":"Supplies the particle-fluid simulation benchmark whose simulator output is compared with a learned emulator."},{"cited_title":"Microscopictrafficsimulationusingsumo","cited_arxiv_id":null,"evidence_quote":"Supplies the microscopic traffic simulator used in the road-blocking model-comparison experiment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the large-scale traffic scenario whose original and modified versions are compared."}],"review_version":1}