Interpretability in Deep Time Series Models Demands Semantic Alignment

Alberto Termine; Giovanni De Felice; Giuseppe Marra; Pietro Barbiero; Riccardo D'Elia; Silvia Santini

REVIEW 2 major objections 6 minor 2 references

Interpretability in Deep Time Series Models Demands Semantic Alignment

T0 review · 2 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Time series AI is interpretable only when it thinks in human concepts

desk verdict A genuinely useful position paper on semantic alignment for time series, but Eq. (5) is secretly a determinism assumption that the authors never flag. read the letter →

arxiv 2602.02239 v2 pith:WWZJFNVV submitted 2026-02-02 cs.LG

Giovanni De Felice , Riccardo D'Elia , Alberto Termine , Pietro Barbiero , Giuseppe Marra , Silvia Santini This is my paper · ORCID

classification cs.LG

keywords semanticalignmenttimeseriesinterpretabilityconcept-basedmodelstemporalpropagationmechanismconceptdrifttrustworthyAIexplainability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deep time series models are accurate but opaque, and this position paper argues that the dominant interpretability toolkit—saliency maps, attention weights, surrogates, counterfactuals—answers a different question than the one practitioners care about. The authors claim that interpretability in this setting is semantic alignment: a model's internal variables must equal the concepts a domain expert actually reasons with (for example, 'thermal stress accumulated' instead of 'hidden unit 47'), and this equality must survive the model's own temporal propagation. They formalize this as two probabilistic equalities, one for instantaneous concepts and one for dynamic concepts, plus a separate requirement that the mechanisms relating those concepts honor human constraints such as monotonicity or known physical laws. If accepted, the definition turns interpretability into a structural, verifiable property rather than a post-hoc story, and gives model designers a concrete blueprint: predict through a bottleneck of expert-meaningful concepts whose evolution is explicitly supervised. The reader should care because the paper is setting a new evaluation target for a field where 'explainability' currently means many incompatible things.

What carries the argument

Definition 1 (semantic alignment of concepts) is the load-bearing object: it requires P(Ut=CUt|X≤t)=1 and P(Zt+1=CZt+1|X≤t)=1, turning 'interpretable' into a probabilistic equality between model variables and human concepts rather than a human judgment about explanations. The temporal clause—that alignment must be preserved one step ahead under propagation—is what distinguishes the time-series setting from static concept-bottleneck models. Definition 2 (semantic alignment of mechanisms) makes mechanism interpretability a constraint-satisfaction problem: a mechanism is aligned if its conditional distribution belongs to the user-specified admissible set. Together these definitions provide the

What would settle it

In a real forecasting application, ask domain experts to label the intended concept (say, 'thermal stress') at many timesteps and test whether experts agree with each other and with their own past labels, and whether a model whose states align at time t continues to align at t+1. If experts' labels drift or agree poorly, the probabilistic equality in Definition 1 cannot be satisfied; if a model that fails the equality is still judged interpretable by its users, the definition is too strong.

Watch

Extended reading notes

Core claim

The paper's central claim is a definition. A deep time series model is semantically aligned when (i) the encoder's representations Ut equal instantaneous human concepts CUt with probability one given the observed past, and (ii) the propagated state Zt+1 equals the dynamic concept CZt+1 with probability one given the observed past. The second condition—preservation of alignment under temporal evolution—is the constraint with no analogue in static settings; without it, a representation that matches the intended concept at time t can silently become an unrelated concept at time t+1. The paper pairs this with a definition of mechanism alignment: a mechanism is aligned when the conditional distri

Load-bearing premise

The load-bearing premise is that human reasoning about a domain can be modeled as random variables whose semantics remain stable over time, so that 'does the model's state equal the human concept?' is a well-defined yes/no question at every step.

Editorial extensions

If this is right

Users can intervene on aligned models in concept space: correcting a concept value at time t updates all future concept estimates, enabling temporally consistent counterfactuals and targeted mechanism repair without retraining.
Aligned concepts enable formal verification (model checking) and fairness analysis, because internal states correspond to explicit domain-relevant properties that can be checked and traced over time.
Omitting the propagation supervision term allows concept drift: per-step concept accuracy is not enough to keep alignment from decaying as the model steps forward.
Residual pathways that process unconstrained representations in parallel with the concept space let semantically aligned models retain the expressivity of black boxes, so interpretability and accuracy need not trade off.
By the paper's own comparison table, existing paradigms—attention, saliency, surrogates, counterfactuals, prototypes, linearized dynamics, and most physics-informed or symbolic models—fail at least one of the two alignment conditions, so none of them qualifies as making time series models semantically interpretable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical corollary the paper does not spell out: alignment can be measured continuously, for example as the expected distance between model states and expert-labelled concept trajectories, so the definition could be turned into a benchmark metric rather than a yes/no criterion.
The assumption of temporally stable concept semantics is likely the practical bottleneck; in domains with regime shifts, concepts like 'recession' or 'overheating' change meaning, so Definition 1 would require explicitly modelling concept drift, not just model dynamics.
The intervention story (correct a concept value and roll the dynamics forward) presupposes that learned propagation mechanisms are causal, not merely predictive; enforcing alignment of conditional distributions may not be enough for genuine counterfactual reasoning.
By making interpretability a property of the model rather than the explanation, the definition shifts research effort from post-hoc analysis to annotation: scalable collection of expert concept labels over time becomes the enabling resource, possibly via automated or language-model-assisted annotation workflows.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: this is a better-than-usual position paper. It gives time series interpretability something it lacked — a formal definition of semantic alignment that includes a temporal persistence condition (Eq. 5), not just static concept matching. The distinction between instantaneous and dynamic concepts is real, and the blueprint with the extra temporal loss term is concrete enough to build on. The taxonomy in the appendix and Table 1 are handy. The authors are honest that the blueprint is speculative; that is fine for the genre.

The main soft spot is the formal core. In Definition 1, they require C^Z_{t+1} to be σ(X≤t)-measurable and demand P(Z_{t+1}=C^Z_{t+1}|X≤t)=1. Since Z_{t+1} is a function of X≤t through the model class, this forces the future dynamic concept to be deterministically determined by the observed past. Any concept with irreducible aleatoric uncertainty — vital signs with measurement noise, financial series with unobserved drivers, any latent-state process — violates the condition for every model in the class. The paper never flags this. Observation 3.1 discusses the temporal stability of concept semantics, which is a different issue. So the headline claim that alignment must be preserved under temporal evolution is stated much more broadly than the definition actually supports. The stress-test note lands.

That said, the flaw is fixable and not fatal to the paper's purpose. A distributional version of alignment, or an explicit scoping to deterministic concepts, would patch it. But as written, the central definition does not cover the probabilistic forecasting problems the paper uses as motivation.

Other soft spots are minor. Definition 2 is a placeholder: a mechanism is aligned if it lies in a set of human-admissible distributions, but how that set is built or verified is left open. The accuracy-interpretability rebuttal leans on prior CBM results, which is acceptable in a position piece but not evidence. No experiments, but none are promised.

Who this is for: people working on concept bottleneck models, interpretable time series, and anyone tired of saliency maps as the default answer. It is worth a serious referee. I would send it out, with a request that the authors either relax Eq. (5) or explicitly restrict the definition to deterministic concepts and say what that costs them.

Referee Report

2 major / 6 minor

Summary. The paper is a position paper arguing that interpretability of deep time series models should be defined by semantic alignment (SA): internal variables and mechanisms should correspond to human-meaningful concepts, and this correspondence must be preserved under temporal evolution. It formalizes SA in Definition 1 (concepts) and Definition 2 (mechanisms), proposes a blueprint extending concept bottleneck models to time series via source concepts, temporal and spatio-temporal propagation mechanisms, and a three-term training objective, and discusses implications for trust, intervention, counterfactuals, and interaction with symbolic knowledge. An extended appendix reviews existing post-hoc and intrinsic interpretability paradigms and classifies them against SA.

Significance. If the central formal notion were fully operational, the paper would make a useful contribution: it isolates semantic from structural opacity in a time-series context, introduces a temporal invariance requirement absent from static concept-based models, and offers a concrete architectural template. The writing is clear and the literature review is broad; Table 1 is a useful synthesis. As a position paper, it does not provide empirical demonstrations, and several statements are explicitly speculative; that is appropriate for the genre. The value of the paper currently hinges on whether Definition 1 can be made to apply to the probabilistic settings the paper itself motivates. With that repair, the paper would be a good anchor for future work.

major comments (2)

[Sec. 3.2, Definition 1, Eq. (5)] The definition requires C^Z_{t+1} to be σ(X≤t)-measurable and P(Z_{t+1}=C^Z_{t+1}|X≤t)=1 a.s. Combined with the model class in Eq. (2), where Z_{t+1} is computed from X≤t, this forces every dynamic concept to be an a.s. deterministic function of the observed past. But Sec. 3.1 defines concepts as general stochastic processes, and the motivating applications (physiological or financial signals with noise or unobserved drivers) have dynamic concepts with irreducible aleatoric uncertainty given X≤t. For such concepts, P(Z_{t+1}=C^Z_{t+1}|X≤t)<1 for every model in the class, so Definition 1 is not operational for the very probabilistic forecasting problems the paper sets up in Sec. 2.1. Observation 3.1 addresses only temporal stability of semantics, not σ(X≤t)-measurability. Please either restate alignment distributionally (e.g., P(Z_{t+1}|X≤t)=P(C^Z_{t+1}|X≤t)) or explicitly restrict Defini
[Sec. 3.2, Definition 2] Definition 2 is a placeholder: it asserts P(V|V') ∈ M^h_{V|V'}, but M^h_{V|V'} is not constructed, verified, or exemplified. As written, any mechanism can be declared aligned for a suitable choice of M^h, so the definition carries no formal content and cannot be falsified. The surrounding text says human knowledge often specifies only constraints, but it does not say how those constraints become M^h. The paper should either specify a concrete construction (e.g., M^h as the set of distributions satisfying a list of monotonicity, sparsity, or physics constraints) or explicitly state that Definition 2 is a schema to be instantiated per domain. As it stands, the claimed separation of variable alignment and mechanism alignment is not yet a definition.

minor comments (6)

[Sec. 4.3] The loss equation uses L_spatial and L_temporal, but the bullets define L_concept and L_prop. The notation should be harmonized.
[Observation 3.1] The sentence 'This is not a limitation' is too strong; stable semantics is a substantive assumption. It should be phrased as an explicit assumption, and the relaxation mentioned should be described as a limitation.
[Sec. 2.2] The text says U_t and Z_t are realizations of stochastic processes, but Eqs. (1)-(2) present them as deterministic functions of x≤t. Please clarify whether stochasticity comes only from the input process or also from model-internal randomness.
[Sec. 3.2] The assertion that omitting Eq. (5) 'could lead the SA to decay exponentially over time' is unsubstantiated. If it is a conjecture, mark it as such; if it is a theorem, provide a statement and proof sketch.
[Table 1] The legend uses ✓/∼/✗ but the criteria for 'partially satisfies' are not defined. Add a sentence or footnote explaining how partial satisfaction is judged.
[Definition 1] The phrase 'aligned with C^U_t and C^Z_t, respectively' could be ambiguous because condition (ii) uses C^Z_{t+1}; consider aligning the notation consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: semantic alignment is defined relative to external human concepts and the blueprint is supervised from those concepts; self-citations are background only.

full rationale

This is a position paper, not a derivation with fitted parameters or empirical predictions. The central Definition 1 (Eqs. 4–5) ties interpretability to equality with externally specified human concepts C^U_t and C^Z_{t+1}; the subsequent blueprint (Sec. 4.3) enforces this via supervision L_concept and L_prop, which is the opposite of internal circularity. The paper does not claim to predict anything from its own definitions: the temporal condition (Eq. 5) is introduced as a normative requirement ('must be preserved under temporal evolution'), not derived from Eq. (4). The cited prior work by the same authors (e.g., Barbiero et al. 2025, De Felice et al. 2025, Dominici et al. 2025) appears only as context or as independent support for concept-based models and is not load-bearing for the definition. There is a genuine formal-scope concern, correctly identified by the skeptic: Definition 1 assumes C^Z_{t+1} is σ(X≤t)-measurable and demands P(Z_{t+1}=C^Z_{t+1}|X≤t)=1 a.s., which is unsatisfiable for stochastic dynamic concepts. But that is a limitation and operationality issue, not circularity: it does not make the definition's output equivalent to its input. Observation 3.1's claim that stable semantics 'is not a limitation' is an acknowledged assumption, not a hidden circular step. Therefore no circular step can be quoted and exhibited; score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters and no new physical/empirical entities. Its central definitions rest on the assumption that human concepts are stable stochastic processes and that concept supervision is available; these are explicit domain assumptions rather than fitted parameters.

assumptions (4)

domain assumption Human reasoning can be formalized as operating over random variables (concepts).
Sec. 3.1 states 'we adopt a probabilistic perspective and assume human reasoning can be formalized as if it operated over random variables'. This is load-bearing for Definition 1, which equates model variables to these concept random variables.
domain assumption The semantics of human concepts remain consistent over time.
Observation 3.1 explicitly says 'we implicitly assume the semantics of human concepts remain consistent over time'. This is required for the temporal preservation condition (Eq. 5) to be meaningful.
domain assumption The model's latent variables (Ut, Zt) exist as stochastic processes.
Sec. 2.2 assumes ut and zt 'can be modeled as realizations of stochastic processes {Ut} and {Zt}', which is necessary to state probabilistic alignment conditions.
domain assumption Concept annotations can be obtained at sufficient scale.
Sec. 6.4 acknowledges the cost objection and responds by citing LLM annotators and annotation workflows, but no concrete evidence is given that such annotations are available for arbitrary time series domains.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Interpretability in Deep Time Series Models Demands Semantic Alignment." pith.science (2026). https://pith.science/paper/WWZJFNVV

@misc{pith2026260202239,
  author       = {Pith},
  title        = {Pith review of: Interpretability in Deep Time Series Models Demands Semantic Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WWZJFNVV}},
  note         = {Machine review of arXiv:2602.02239}
}

read the original abstract

Deep time series models continue to improve predictive performance, yet their deployment remains limited by their black-box nature. In response, existing interpretability approaches in the field keep focusing on explaining the internal model computations, without addressing whether they align or not with how a human would reason about the studied phenomenon. Instead, we state interpretability in deep time series models should pursue semantic alignment: predictions should be expressed in terms of variables that are meaningful to the end user, mediated by spatial and temporal mechanisms that admit user-dependent constraints. In this paper, we formalize this requirement and state that, once established, semantic alignment must be preserved under temporal evolution: a constraint with no analog in static settings. Provided with this definition, we outline a blueprint for semantically aligned deep time series models, identify properties that support trust, and discuss implications for model design.

Figures

Figures reproduced from arXiv: 2602.02239 by the authors.

**Figure 1.** Template architecture for time series tasks, characterizing the class of models considered in this work. Observations within a window are first mapped by an Encoder into latent representations, which are transformed by a task-specific Propagation module and subsequently mapped by a Decoder to the target output. Modules operating on latent representations are highlighted in gray. 2.2. Considered class of models We co… view at source ↗

**Figure 2.** Computational graph showing the inference pathway for a model from the class described in Sec. 2.2. The symbol ∼ indicates SA of representations with concepts. change. Relaxing this assumption could open interesting research directions. Observation 3.2. When the representation and concept domains differ (Z ̸= C), alignment could be defined via a translation function τ : Z → C, and the equalities above are replaced b… view at source ↗

**Figure 3.** Examples of possible instantiations of the blueprint for different time series tasks. Es. (a) Concept-based forecasting: some of the input variables (green and orange) are chosen as the forecasting target, while all available input information is used to encode new dynamic concepts that contribute to the forecasting. Es. (b) Concept-based window classification: instantaneous concepts are extracted from different inp… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

[2024]

From a semantic alignment perspective, this introduces an additional failure mode compared to feature attribution methods

fall into this category. From a semantic alignment perspective, this introduces an additional failure mode compared to feature attribution methods. Not only are explanations expressed in terms of input-level features or latent representations rather than domain-level concepts, but they are also mediated by an approximate model that effectively adds an add...

2021
[2025]

interpretability

Accessed: 2026-01-29. Guidotti, R., Monreale, A., Spinnato, F., Pedreschi, D., and Giannotti, F. Explaining any time series classifier. In 2020 IEEE second international conference on cognitive machine intelligence (CogMI), pp. 167–176. IEEE, 2020. Hauser, M., Gunn, S., Saab Jr, S., and Ray, A. State-space representations of deep neural networks.Neural co...

arXiv 2026

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2024]

From a semantic alignment perspective, this introduces an additional failure mode compared to feature attribution methods

fall into this category. From a semantic alignment perspective, this introduces an additional failure mode compared to feature attribution methods. Not only are explanations expressed in terms of input-level features or latent representations rather than domain-level concepts, but they are also mediated by an approximate model that effectively adds an add...

2021

[2] [2025]

interpretability

Accessed: 2026-01-29. Guidotti, R., Monreale, A., Spinnato, F., Pedreschi, D., and Giannotti, F. Explaining any time series classifier. In 2020 IEEE second international conference on cognitive machine intelligence (CogMI), pp. 167–176. IEEE, 2020. Hauser, M., Gunn, S., Saab Jr, S., and Ray, A. State-space representations of deep neural networks.Neural co...

arXiv 2026