{"id":"8260db05-2bdf-4ac7-b8b4-98a1fac92d33","arxiv_id":"2501.04009","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Multi-SpaCE uses NSGA-II to generate Pareto-optimal, subsequence-based counterfactual explanations for multivariate time series, achieving perfect validity on all tested datasets.","lead":"A new explainable AI method, Multi-SpaCE, creates counterfactual explanations for multivariate time series by swapping in values from the nearest different-class example and optimizing four goals with a genetic algorithm. It reports perfect validity on 25 benchmark datasets and returns a menu of trade-off solutions instead of a single explanation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'perfect validity' guarantee is only a penalty-induced dominance claim; it presumes a valid mask exists, which the single-NUN search space does not ensure.","rationale":"In good faith, the paper does three things: it extends Sub-SpaCE to multivariate time series, it introduces an NSGA-II-based multi-objective formulation, and it reports strong empirical validity across many datasets. The central novelty claim is the guarantee of perfect validity. For that claim to hold, two conditions are needed: a valid solution must exist for every instance, and the optimizer must be able to find one. The paper shows neither. Eq. (5)'s penalty term only ensures that invalid solutions are dominated if valid solutions exist; NSGA-II is a heuristic with no convergence certificate. More fundamentally, the search space is restricted to masks whose non-zero entries copy values from a single fixed NUN. Algorithm 1 can raise the activation percentage h up to 100%, at which point the candidate is exactly xnun; if the classifier misclassifies the NUN, even full substitution is invalid. The dataset filter F1 > 80% does not remove this possibility, because it is an aggregate accuracy threshold, not a per-instance guarantee about NUN classification. The reported validity of 1.0 on 100 random test instances per dataset is meaningful empirical evidence, but it is not a guarantee; claiming 'ensures' overreaches the evidence. This is the same weakest assumption identified by the reader. The concern does not require rejecting the paper; it requires rephrasing the guarantee and adding a verification step on NUN-misclassified instances. Therefore the reader's CONDITIONAL verdict remains appropriate.","tokens_in":23015,"tokens_out":4260,"duration_ms":42179,"concrete_test":"Using the released code, compute for each dataset and each of the 100 evaluated test instances whether b(xnun) = ynun. Restrict to instances where the NUN is misclassified by the explained classifier, and run Multi-SpaCE on those instances, reporting validity on that subset. If any such instance returns no valid Pareto solution, Eq. (5)'s penalty does not ensure perfect validity, and the abstract/conclusions should be reworded to a finite-sample empirical claim. A complementary synthetic check is to train a deliberately imperfect classifier on a two-class dataset where at least one test instance has a misclassified NUN, then verify whether Multi-SpaCE still attains validity 1.0.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption behind the headline claim is existential: for each explained instance x there must exist at least one mask M in the searched space such that b(find(x|M, xnun)) = ynun. Section 3.2.3 states that the penalty term -nu * 1class in Eq. (5) ensures that non-valid solutions are always dominated by valid counterfactuals. This only imposes an ordering on the population; it cannot create a valid member of the search space. Because the counterfactual is always a subsequence-wise copy of a single nearest unlike neighbor, validity is possible only if some subsequence combination of the NUN changes the classifier's output to the desired class. Algorithm 1's reinitialization raises h toward 100%, so the strongest candidate is the full substitution x' = xnun. If the InceptionTime classifier misclassifies xnun (readily possible despite the F1 > 80% dataset filter), even this candidate need not be valid, and no penalty value nu can repair the absence. The paper reports validity 1.0 on all selected datasets, but that is an empirical observation on 100 test samples per dataset, not a guarantee. A single adversarial or misclassified-NUN instance would violate the claim as worded. Thus 'ensures perfect validity' is unsupported as a formal property; the defensible claim is 'achieves perfect validity on all evaluated instances.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Multi-SpaCE is a counterfactual explanation method for multivariate time series classification. It uses NSGA-II to optimize several objectives: adversarial class probability, sparsity, contiguity, and autoencoder-based plausibility, while searching over binary masks that substitute values from the nearest unlike neighbor (NUN). The method returns a Pareto front of candidate counterfactuals rather than a single solution. The paper reports experiments on UCR/UEA datasets, comparing against NG, Glacier, GlacierAE, AB-CF, DiscoX, and COMTE, and claims that Multi-SpaCE is the first multi-objective method that ensures perfectly valid counterfactuals for multivariate time series.","tokens_in":23313,"tokens_out":3165,"duration_ms":30477,"significance":"If the central claim were fully supported, Multi-SpaCE would fill a clear gap: existing multivariate counterfactual methods for time series either enforce rigid subsequence structures, treat validity as a soft objective, or lack multivariate support. The paper's strengths include publicly available code, a broad benchmark across 25 datasets, comparison with several recent baselines, and an ablation study of mutation and mask design choices. The inclusion of Isolation Forest and Local Outlier Factor as independent plausibility evaluators is a good safeguard. However, the perfect-validity guarantee is overstated relative to what a penalty-based ranking can deliver, and several evaluation choices create in-sample optimism. The empirical results are still useful, but the headline formal claim and the interpretation of some reported metrics need substantial revision.","major_comments":[{"comment":"The claim that the penalty term -nu * 1class 'ensures that non-valid solutions are always dominated by valid counterfactuals' only imposes an ordering on the population. It cannot create a valid solution if none exists in the search space. Since the counterfactual is always a subsequence-wise copy of a single NUN, validity is possible only if some mask makes b(x') = y_nun. The reinitialization schedule in Algorithm 1 approaches the full substitution x' = x_nun, but if the classifier misclassifies x_nun (not excluded by the F1 > 80% dataset filter), even this extreme candidate can be invalid, and no value of nu repairs the absence. Thus 'ensures perfect validity' in Sections 1 and 5 is unsupported as a formal property; the defensible statement is that Multi-SpaCE achieved perfect validity on all evaluated instances. Please either add an explicit existence assumption and prove the guarantee under it, or soften the claim throughout.","section":"Section 3.2.3, Eq. (5), Algorithm 1"},{"comment":"The abstract and introduction state that Multi-SpaCE 'balances proximity, sparsity, plausibility, and contiguity', but Eq. (5) contains no proximity objective: o1 is class probability, o2 sparsity, o3 contiguity, and o4 AE-based plausibility. Proximity appears only as an evaluation metric in Section 4.4 and Table 3. Sparsity of NUN substitutions is a rough proxy for proximity but is not identical to the reported l2 distance. Please either add an explicit proximity objective to Eq. (5) or revise the claims to say that proximity is evaluated but not directly optimized.","section":"Abstract and Section 1 vs. Eq. (5)"},{"comment":"The hyperparameters of Multi-SpaCE are selected using the same multivariate datasets on which the final results are reported. Tables A1-A4 and Figures A1-A2 use A WR, BasicMotions, Cricket, Epilepsy, NATOPS, RacketSports, SR-SCP1, and UWave to choose pe, pc, pp, and the common/independent mask schedule; Section 4.5 then reports the performance of the selected configuration on those same datasets. This creates an in-sample selection loop that can inflate the reported average ranks. Please either use a separate validation split for the ablation, or clearly disclose the selection loop and quantify its potential bias, for example by comparing the selected configuration with a default configuration on held-out datasets.","section":"Appendix A and Section 4.5"},{"comment":"The plausibility objective o4(x, x') = -e(x,x')/e_max is computed with the autoencoder f_AE, and the evaluation metric OS(AE) in Tables 4 and 5 uses the same f_AE. Consequently, the OS(AE) rankings partly measure how well the method optimizes the evaluation quantity itself. The OS(IF) and OS(LOF) columns provide independent evidence and partly mitigate this concern, but the text should state this circularity explicitly and give the OS(IF)/OS(LOF) results at least equal weight when drawing plausibility conclusions.","section":"Eq. (5) and Tables 4-5"}],"minor_comments":[{"comment":"The word 'beggining' should be 'beginning'.","section":"Section 3.2.1"},{"comment":"The word 'obtimization' should be 'optimization'.","section":"Section 1"},{"comment":"The term 'Bernouilli' should be 'Bernoulli' in the descriptions of delta_e, delta_c, and delta_p.","section":"Equations (2)-(4)"},{"comment":"The typeset 'F itnessn' should be 'Fitness_n'.","section":"Algorithm 2, line 14"},{"comment":"The sentence 'Every objective is an adaptation of the terms in (5)' should refer to Eq. (1) rather than Eq. (5), since Eq. (5) itself defines the objectives being adapted.","section":"Section 3.2.3"},{"comment":"The dataset 'Phoneme' appears in Table B6 but is not listed in Table 1 or described in Section 4.2; either add it to the dataset selection or remove the row.","section":"Appendix B, Table B6"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within the scope of the journal and the code release is a positive feature. The main risk is that the 'ensures perfect validity' phrasing, repeated in the abstract and conclusions, promises a formal guarantee that the method does not actually deliver; this should be fixed before publication. The in-sample hyperparameter selection is also a typical reviewer concern and should be addressed head-on rather than buried in an appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a referee's time, but the headline claim is wrong as stated. Multi-SpaCE empirically achieves validity 1.0 on every dataset it tests, and that is a real result. But 'ensures perfect validity' (abstract, Section 1, Section 5) is not supported: the strict penalty in Eq. (5) only orders candidates within the NSGA-II population. It cannot manufacture a valid counterfactual if no mask on the single NUN flips the classifier. That existential assumption is never stated or checked. The reinitialization schedule drives h toward 100%, so the strongest possible candidate is full substitution x' = xnun; if InceptionTime misclassifies the NUN, perfect validity is unattainable no matter how large nu is. What the evidence actually supports is 'achieves perfect validity on all evaluated instances,' which is still competitive with or better than the baselines. What is genuinely new: the multivariate extension (common-then-independent mask schedule), the subsequence pruning mutation, and the NSGA-II formulation with four objectives over subsequence masks. The paper is also unusually honest in places: it admits proximity is not directly optimized (Section 4.5), and it discloses the parallelization overhead in Appendix D. Code and data are public, the baselines use the original implementations, the dataset filter (F1 > 80%) is sensible, and the ablation study in Appendix A is thorough. Soft spots, in order of severity. First, the perfect-validity claim is load-bearing; it appears in the abstract, intro, and conclusions, and it is the stated novelty over TSEvo and TX-Gen. Rewording is mandatory, not cosmetic. Second, the abstract says the method balances proximity, but Eq. (5) has no proximity objective; the method optimizes adversarial probability, sparsity, contiguity, and plausibility. Minor fix. Third, hyperparameters (mask schedule, mutation probabilities, pruning) were selected in Appendix A on the same ten multivariate datasets used for the reported benchmarks, and the final configuration was confirmed on that same data. That is an in-sample selection loop, and the paper should disclose it in the main text rather than bury it in an appendix. Fourth, the plausibility objective uses the autoencoder, and one of the three plausibility evaluation metrics is that same autoencoder's outlier score; the IF and LOF evaluations mitigate this only partially. Who this is for: researchers working on time-series XAI, especially anyone who needs valid counterfactuals in multivariate settings. The comparison against original implementations of COMTE, AB-CF, DiscoX, NG, and Glacier is useful and fair. Recommendation: send to peer review. The method is solid and reproducible; the claims need tempering and the existence assumption needs a stated limitation.","headline":"A solid, reproducible multivariate extension of Sub-SpaCE that achieves validity 1.0 on every tested dataset, but the 'ensures perfect validity' claim is a penalty-induced ordering claim, not a guarantee.","tokens_in":23849,"tokens_out":3796,"would_cite":true,"duration_ms":32458,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-SpaCE claims to be the first multi-objective counterfactual method that guarantees perfect validity for multivariate time-series classification.","keywords":["eXplainable Artificial Intelligence (XAI)","Counterfactual Explanations","Genetic Algorithm Optimization","Time Series Classification","Multivariate Time Series","Sparsity","Validity","Pareto front"],"falsifier":"For a single test instance, enumerate all possible binary masks (or run Multi-SpaCE many times with different random seeds) and check whether any mask's nearest-unlike-neighbor substitution changes the classifier's output. If a single instance admits no valid mask, the method cannot return a valid counterfactual for it regardless of the penalty; finding such an instance among the test set would disprove the perfect-validity claim. Similarly, measuring validity on a dataset with many misclassified instances would expose whether the 100 percent scores reflect the algorithm or the easy datasets.","tokens_in":22803,"feed_emoji":"🎯","tokens_out":9580,"duration_ms":79410,"temperature":0.7,"pith_summary":"Multi-SpaCE aims to solve a practical problem in explainable AI: when a time-series classifier makes a decision, how can we produce a \"what-if\" version of the input that reliably changes the decision while keeping the change small, interpretable, and realistic? The paper's central claim is that by treating validity—the requirement that the counterfactual actually flips the predicted class—as a strict, uniformly penalized constraint inside a multi-objective evolutionary search, it is possible to guarantee perfect validity for multivariate time series, not just for univariate ones. The method returns a whole Pareto front of alternative counterfactuals, so different users can choose among sparser or more plausible explanations without re-tuning weights. If true, this removes the need to discard invalid explanations after the fact and makes counterfactual explanations usable in high-stakes settings such as healthcare or finance.","feed_headline":"Every returned counterfactual flips a time-series model's prediction","feed_subtitle":"Every explanation on the Pareto front is guaranteed to change the predicted class, so users can pick without filtering out failures.","key_machinery":"The central object is the binary mask $M \\in \\{0,1\\}^{L \\times C}$ (a shared mask $m \\in \\{0,1\\}^L$ is used first) that marks which time-channel cells are replaced by the corresponding values of the nearest unlike neighbor. The search engine is NSGA-II, a non-dominated sorting genetic algorithm, augmented with subsequence-level mutation operators that extend, compress, or prune whole contiguous blocks of changes. The load-bearing idea is that each of the four objectives in Eq.\\ (5) carries the same strict penalty $\\nu \\cdot \\mathbb{1}_{\\text{class}}$ with $\\nu = 100$; because any candidate with the wrong output class is dominated by every candidate with the correct class, the final non-dominated front can only contain valid counterfactuals. Plausibility is scored by an autoencoder's reconstruction-error outlier score, and the method combines a common-mask phase with an independent per-channel-mask phase to balance search difficulty and flexibility.","core_discovery":"The paper claims that Multi-SpaCE is the first multi-objective optimization method that ensures perfectly valid counterfactual explanations for multivariate time-series classification. A counterfactual is built by choosing a binary mask of positions and copying the values of the nearest unlike neighbor (the closest training instance from a different class) at those positions; the search over masks is done with NSGA-II over four objectives: adversarial probability for the desired class, sparsity, contiguity of subsequences, and plausibility measured by an autoencoder. In every objective the same large penalty term penalizes candidates that do not change the predicted class, so invalid candidates are dominated by valid ones and drop out of the Pareto front. On the paper's experiments, Multi-SpaCE attains 100 percent validity on every tested dataset, is the only method to do so in the multivariate benchmarks, and is competitive or best on proximity, sparsity, plausibility, and the joint sparsity-contiguity measure.","pith_inferences":["Beyond the paper: the perfect-validity guarantee is conditional on the existence of a valid mask for each instance; if the classifier already misclassifies the nearest unlike neighbor or no subsequence combination flips the output, the penalty term can only rank candidates, not create a valid one.","Beyond the paper: a natural test is to apply Multi-SpaCE to harder settings—lower classifier accuracy, noisy labels, or out-of-distribution test points—and check whether validity stays at 100 percent.","Beyond the paper: replacing nearest-neighbor copying with generative substitutes (the paper's own future direction) would change the nature of the guarantee, since generated values must also stay on the data manifold for the counterfactual to remain plausible."],"forward_implications":["Users of Multi-SpaCE can take any counterfactual from the returned Pareto front and know it changes the classifier's predicted class, without post-hoc filtering.","Different stakeholders can choose among explanations that favor sparsity, contiguity, or plausibility, because the Pareto front preserves those trade-offs instead of collapsing them into one tuned solution.","The method is model-agnostic: it needs only the classifier's class probabilities, so it applies to any multivariate time-series classifier that provides them.","On the reported benchmarks, Multi-SpaCE matches the perfect validity of the single-subsequence univariate method it extends and becomes the only tested method with perfect validity in the multivariate datasets.","The method ranks first or near-first on proximity, sparsity, and the joint sparsity-contiguity metric, with execution times in the middle of the baseline spectrum."],"supporting_citations":[{"why":"The predecessor method; supplies the mask-and-NUN formulation, the custom initialization and mutation ideas, and the utility function used to pick one solution from the Pareto front.","marker":"[25]"},{"why":"Defines the nearest unlike neighbor and the subsequence-substitution strategy that Multi-SpaCE adopts.","marker":"[17]"},{"why":"NSGA-II is the non-dominated sorting genetic algorithm that Multi-SpaCE customizes and uses to compute the Pareto front.","marker":"[26]"},{"why":"A recent multi-objective baseline that claims perfect validity in univariate settings only; the comparison motivates the multivariate guarantee.","marker":"[24]"},{"why":"A fixed-length attention-based baseline for multivariate time series that Multi-SpaCE must outperform in validity and sparsity.","marker":"[22]"},{"why":"A fixed-length discord-based baseline for multivariate time series used in the experimental comparison.","marker":"[23]"},{"why":"The deep convolutional classifier used in all experiments; the validity results depend on its predictions.","marker":"[35]"},{"why":"Source of the univariate benchmark datasets used in the experiments.","marker":"[36]"},{"why":"Source of the multivariate benchmark datasets used in the experiments.","marker":"[37]"}],"fun_headline_variants":["Perfectly valid counterfactuals for multivariate time series","Guaranteed-valid counterfactuals for multivariate time-series models","Every Pareto counterfactual flips the time-series prediction","Multi-objective counterfactuals: 100% validity, multivariate time series","First counterfactual method with perfect validity on time series"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that for every instance explained there exists some binary mask whose nearest-unlike-neighbor substitution flips the classifier's output to the desired class; the strict penalty can only prefer valid masks over invalid ones, it cannot produce a valid mask where none exists.","fun_headline_variants_meta":{"raw":{"variants":["Perfectly valid counterfactuals for multivariate time series","Guaranteed-valid counterfactuals for multivariate time-series models","Every Pareto counterfactual flips the time-series prediction","Multi-objective counterfactuals: 100% validity, multivariate time series","First counterfactual method with perfect validity on time series"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1255,"prompt_tokens":911,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":258}},"tokens_in":527,"tokens_out":344,"duration_ms":3504,"temperature":1.0,"reasoning_tokens":258,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:37:45.818257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a single test instance, enumerate all possible binary masks (or run Multi-SpaCE many times with different random seeds) and check whether any mask's nearest-unlike-neighbor substitution changes the classifier's output. If a single instance admits no valid mask, the method cannot return a valid counterfactual for it regardless of the penalty; finding such an instance among the test set would disprove the perfect-validity claim. Similarly, measuring validity on a dataset with many misclassified instances would expose whether the 100 percent scores reflect the algorithm or the easy datasets.","supporting_citations":[{"cited_title":"In: World Conference on Explainable Artificial Intelligence (XAI), pp","cited_arxiv_id":null,"evidence_quote":"The predecessor method; supplies the mask-and-NUN formulation, the custom initialization and mutation ideas, and the utility function used to pick one solution from the Pareto front."},{"cited_title":"In: 29th International Conference on Case-Based Reasoning Research and Development (ICCBR), pp","cited_arxiv_id":null,"evidence_quote":"Defines the nearest unlike neighbor and the subsequence-substitution strategy that Multi-SpaCE adopts."},{"cited_title":"IEEE Transactions on Evolutionary Computation 6(2), 182–197 (2002)","cited_arxiv_id":null,"evidence_quote":"NSGA-II is the non-dominated sorting genetic algorithm that Multi-SpaCE customizes and uses to compute the Pareto front."},{"cited_title":"TX-Gen: Multi-Objective Optimization for Sparse Counterfactual Explanations for Time-Series Classification","cited_arxiv_id":"2409.09461","evidence_quote":"A recent multi-objective baseline that claims perfect validity in univariate settings only; the comparison motivates the multivariate guarantee."},{"cited_title":"In: Wrembel, R., Gamper, J., Kotsis, G., Tjoa, A.M., Khalil, I","cited_arxiv_id":null,"evidence_quote":"A fixed-length attention-based baseline for multivariate time series that Multi-SpaCE must outperform in validity and sparsity."},{"cited_title":"Data Mining and Knowledge Discovery 38(6), 3347–3371 (2024)","cited_arxiv_id":null,"evidence_quote":"A fixed-length discord-based baseline for multivariate time series used in the experimental comparison."},{"cited_title":"Data Mining and Knowledge Discovery 34(6), 1936–1962 (2020)","cited_arxiv_id":null,"evidence_quote":"The deep convolutional classifier used in all experiments; the validity results depend on its predictions."},{"cited_title":"IEEE/CAA Journal of Automatica Sinica 6(6), 1293–1305 (2019)","cited_arxiv_id":null,"evidence_quote":"Source of the univariate benchmark datasets used in the experiments."}],"review_version":1}