pith. sign in

arxiv: 2504.11159 · v2 · submitted 2025-04-15 · 💻 cs.AI

C-SHAP for time series: An approach to high-level temporal explanations

Annemarie Jutte , Faizan Ahmed , Jeroen Linssen , Maurice van Keulen This is my paper

Pith reviewed 2026-05-22 20:13 UTC · model grok-4.3

classification 💻 cs.AI
keywords explainable AItime seriesconcept-based explanationsSHAPhuman activity recognitionpredictive maintenanceXAI for temporal data
0
0 comments X

The pith

C-SHAP extracts high-level patterns from time series and applies SHAP to quantify their influence on model predictions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper notes that current XAI methods for time series rely on points or subsequences, which often miss broader structures and produce explanations that are hard for humans to interpret. It introduces C-SHAP to close this gap by treating high-level temporal patterns as concepts and using SHAP to measure how much each concept affects the model's output. The method is shown on human activity recognition data from healthcare and on predictive maintenance tasks from industry. If the approach holds, decision makers in these domains would gain insight into AI reasoning at the level of recognizable patterns rather than raw data points. This directly addresses the reliability concern raised for high-stakes time series applications.

Core claim

C-SHAP defines concepts as high-level patterns extracted from the time series data and leverages the SHAP method to determine the influence of these concepts on predictions, thereby delivering explanations that capture relevant temporal structures more effectively than point- or subsequence-based techniques.

What carries the argument

C-SHAP, the framework that extracts high-level temporal patterns as concepts from time series and computes their SHAP attribution values for model predictions.

If this is right

  • Explanations in healthcare time series tasks shift from sensor values to recognizable activity patterns.
  • Predictive maintenance models reveal the contribution of specific operational patterns to failure forecasts.
  • XAI development for time series moves beyond low-level features to concept-level attribution.
  • Model trust in regulated domains increases when explanations align with domain-expert vocabulary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same concept-extraction step could be paired with other attribution methods beyond SHAP for comparison.
  • Domains with sequential data such as finance or sensor networks could adopt similar pattern-based explanations.
  • Automated concept discovery might be tested for consistency across different time series lengths or sampling rates.

Load-bearing premise

High-level patterns can be extracted from time series data such that the resulting explanations are reliably more human-interpretable than those based on points or subsequences.

What would settle it

A controlled user study on the HAR or predictive maintenance datasets in which experts find C-SHAP explanations no more useful or interpretable than subsequence-based explanations would falsify the claimed advantage.

Figures

Figures reproduced from arXiv: 2504.11159 by Annemarie Jutte, Faizan Ahmed, Jeroen Linssen, Maurice van Keulen.

Figure 1
Figure 1. Figure 1: Whereas point-based attribution determines the attribution of individual points, concept-based [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mean absolute SHAP values over all test data. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Local explanation of a forecast for a sample from January 2016. The “Growth” and “Weekly” [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Local explanation of the sample with the lowest SHAP value for “Other”. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relation between the SHAP value of each concept and the last value of the component before model [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
read the original abstract

In high-stakes domains, such as healthcare and industry, the explainability of AI-based decision-making has become crucial. Without insight into model reasoning, the reliability of these models cannot be ensured. Applications often rely on the time series data type which, unlike the image data type, is underexplored with respect to the development of explainable AI (XAI) techniques. Most existing XAI techniques for time series are focused on point- or subsequence-based explanations. This limits their usability since points and subsequences do not capture all relevant patterns and may not result in human-interpretable explainability. In this paper, we close this gap and propose a concept-based XAI approach (C-SHAP), where concepts are defined as high-level patterns extracted from the time series data. C-SHAP leverages the SHAP method to determine the influence of these concepts on predictions. The effectiveness of the developed framework is illustrated for use cases from healthcare and industry, in the form of Human Activity Recognition (HAR) and predictive maintenance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper proposes C-SHAP, a concept-based XAI method extending SHAP to time series data. Concepts are defined as high-level temporal patterns extracted from the input; SHAP is then applied to attribute the influence of these concepts on model predictions. The framework is illustrated on Human Activity Recognition (HAR) and predictive-maintenance tasks.

Significance. If the central claim holds, the work would address a recognized gap by moving beyond point- and subsequence-based explanations toward more human-interpretable, high-level temporal attributions in high-stakes domains. The reuse of the established SHAP framework is a methodological strength that could facilitate adoption.

major comments (1)
  1. [Method description (concept extraction)] The concept-extraction step is load-bearing yet underspecified: the manuscript defines concepts as “high-level patterns” but supplies no formal algorithm, loss function, clustering criterion, or stability analysis across seeds/hyper-parameters. Without this, it is impossible to verify that the reported attributions on HAR and predictive-maintenance tasks are faithful to the model or generalizable beyond manual feature engineering.
minor comments (1)
  1. [Abstract] The abstract states that effectiveness is “illustrated” on two use cases but provides no quantitative metrics, baselines, or ablation results; these details should be added to the main text or supplementary material.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive feedback and positive evaluation of the work's potential significance. We address the single major comment below and will incorporate the requested clarifications in a revised manuscript.

read point-by-point responses

  1. Referee: [Method description (concept extraction)] The concept-extraction step is load-bearing yet underspecified: the manuscript defines concepts as “high-level patterns” but supplies no formal algorithm, loss function, clustering criterion, or stability analysis across seeds/hyper-parameters. Without this, it is impossible to verify that the reported attributions on HAR and predictive-maintenance tasks are faithful to the model or generalizable beyond manual feature engineering.

    Authors: We agree that the concept-extraction procedure is central to the framework and that the current manuscript description is insufficiently detailed for full reproducibility and verification. In the revised manuscript we will add a dedicated subsection that formally specifies the extraction algorithm, including the precise clustering method (or alternative technique) used to identify high-level temporal patterns, any objective or loss function guiding the extraction, the criterion for selecting the number of concepts, and quantitative stability results across random seeds and hyperparameter choices. These additions will enable readers to assess the faithfulness of the resulting attributions on the reported tasks and to evaluate generalizability. revision: yes

Circularity Check

0 steps flagged

No circularity; derivation extends SHAP independently to new inputs.

full rationale

The paper defines C-SHAP as an application of the established SHAP method to high-level temporal patterns treated as concepts. No equations, self-citations, or definitions are provided in the abstract or described structure that reduce any claimed prediction or result back to the inputs by construction. The central step is an extension to a new input type (concepts) rather than a renaming, fit, or self-referential loop, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no details on free parameters, axioms, or invented entities; the method is described as building directly on the existing SHAP framework.

pith-pipeline@v0.9.0 · 5720 in / 1038 out tokens · 83158 ms · 2026-05-22T20:13:42.247055+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Explainable AI to Improve Machine Learning Reliability for Industrial Cyber-Physical Systems

    cs.LG 2026-01 unverdicted novelty 3.0

    XAI analysis of time-series decomposition reveals insufficient context in CPS ML training, and enlarging data windows improves model performance.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)

    Amina Adadi and Mohammed Berrada. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6:52138–52160, 2018

    work page 2018

  2. [2]

    TimeSHAP: Explaining Recurrent Models through Sequence Perturbations

    João Bento, Pedro Saleiro, André F Cruz, Mário AT Figueiredo, and Pedro Bizarro. TimeSHAP: Explaining Recurrent Models through Sequence Perturbations. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2565–2573, 2021

    work page 2021

  3. [3]

    Concept-based AI interpretability in physiologi- cal time-series data: Example of abnormality detection in electroencephalography

    Alexander Brenner, Felix Knispel, Florian P Fischer, Peter Rossmanith, Yvonne Weber, Henner Koch, Rainer Röhrig, Julian Varghese, and Ekaterina Kutafina. Concept-based AI interpretability in physiologi- cal time-series data: Example of abnormality detection in electroencephalography. Computer Methods and Programs in Biomedicine, 257:108448, 2024

    work page 2024

  4. [4]

    Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

    Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734. Association for Computa...

    work page 2014

  5. [5]

    Concept Activation Regions: A Generalized Framework For Concept-Based Explanations

    Jonathan Crabbé and Mihaela van der Schaar. Concept Activation Regions: A Generalized Framework For Concept-Based Explanations. Advances in Neural Information Processing Systems , 35:2590–2607, 2022

    work page 2022

  6. [6]

    Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey, 2020

    Arun Das and Paul Rad. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey, 2020

    work page 2020

  7. [7]

    A review on time series forecasting techniques for building energy consumption

    Chirag Deb, Fan Zhang, Junjing Yang, Siew Eang Lee, and Kwok Wei Shah. A review on time series forecasting techniques for building energy consumption. Renewable and Sustainable Energy Reviews, 74:902–924, 2017. 9 C-SHAP for time series

    work page 2017

  8. [8]

    Towards automatic concept-based explanations

    Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in neural information processing systems , 32, 2019

    work page 2019

  9. [9]

    Explaining classifiers with causal concept effect (cace), 2020

    Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining Classifiers with Causal Concept Effect (CaCE). arXiv preprint arXiv:1907.07165, 2019

    work page arXiv 1907

  10. [10]

    Artificial intelligence for industry 4.0: Systematic review of applications, challenges, and opportunities

    Zohaib Jan, Farhad Ahamed, Wolfgang Mayer, Niki Patel, Georg Grossmann, Markus Stumptner, and Ana Kuusk. Artificial intelligence for industry 4.0: Systematic review of applications, challenges, and opportunities. Expert Systems with Applications , 216:119456, 2023

    work page 2023

  11. [11]

    Inter- pretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCA V)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCA V). In International conference on machine learning , pages 2668–2677. PMLR, 2018

    work page 2018

  12. [12]

    Conceptual Explanations of Neural Network Prediction for Time Series

    Ferdinand Küsters, Peter Schichtel, Sheraz Ahmed, and Andreas Dengel. Conceptual Explanations of Neural Network Prediction for Time Series. In 2020 International joint conference on neural networks (IJCNN), pages 1–6. IEEE, 2020

    work page 2020

  13. [13]

    A Unified Approach to Interpreting Model Predictions

    Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 4768–4777, 2017

    work page 2017

  14. [14]

    Concept-based model explanations for electronic health records

    Diana Mincu, Eric Loreaux, Shaobo Hou, Sebastien Baur, Ivan Protsyuk, Martin Seneviratne, Anne Mot- tram, Nenad Tomasev, Alan Karthikesalingam, and Jessica Schrouff. Concept-based model explanations for electronic health records. In Proceedings of the Conference on Health, Inference, and Learning , pages 36–46, 2021

    work page 2021

  15. [15]

    Interpreting financial time series with SHAP values

    Karim El Mokhtari, Ben Peachey Higdon, and Ay¸ se Ba¸ sar. Interpreting financial time series with SHAP values. In Proceedings of the 29th annual international conference on computer science and software engineering, pages 166–172, 2019

    work page 2019

  16. [16]

    Time Series Prediction Using Deep Learning Methods in Healthcare

    Mohammad Amin Morid, Olivia R Liu Sheng, and Joseph Dunbar. Time Series Prediction Using Deep Learning Methods in Healthcare. ACM Transactions on Management Information Systems , 14(1):1–29, 2023

    work page 2023

  17. [17]

    Reddy, Brandon Foreman, and Vignesh Subbian

    Amin Nayebi, Sindhu Tipirneni, Chandan K. Reddy, Brandon Foreman, and Vignesh Subbian. Window- SHAP: An efficient framework for explaining time-series classifiers based on Shapley values. Journal of Biomedical Informatics, 144:104438, 2023

    work page 2023

  18. [18]

    Concept-based Explainable Artificial Intelligence: A Survey, 2023

    Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept-based Explainable Artificial Intelligence: A Survey, 2023

    work page 2023

  19. [19]

    A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts

    Gesina Schwalbe and Bettina Finzel. A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts. Data Mining and Knowledge Discovery , 38(5):3043–3101, 2024

    work page 2024

  20. [20]

    Lloyd S. Shapley. A Value for N-Person Games. Contribution to the Theory of Games , 2, 1952

    work page 1952

  21. [21]

    Explain any concept: Segment anything meets concept-based explanation

    Ao Sun, Pingchuan Ma, Yuanyuan Yuan, and Shuai Wang. Explain any concept: Segment anything meets concept-based explanation. Advances in Neural Information Processing Systems , 36, 2024

    work page 2024

  22. [22]

    Forecasting at scale

    Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37–45, 2018

    work page 2018

  23. [23]

    Explainable AI for time series classification: a review, taxonomy and research directions

    Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti. Explainable AI for time series classification: a review, taxonomy and research directions. IEEE Access, 10:100700–100724, 2022

    work page 2022

  24. [24]

    On Completeness-aware Concept-Based Explanations in Deep Neural Networks

    Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On Completeness-aware Concept-Based Explanations in Deep Neural Networks. Advances in neural information processing systems, 33:20554–20565, 2020. 10

    work page 2020