pith. sign in

arxiv: 2605.31580 · v1 · pith:A3G4EHLXnew · submitted 2026-05-29 · 💻 cs.LG

Giving Sensors a Voice: Multimodal JEPA for Semantic Time-Series Embeddings

Utsav Dutta , Gerardo Pastrana , Sina Khoshfetrat Pakazad , Henrik Ohlsson This is my paper

Pith reviewed 2026-06-28 23:07 UTC · model grok-4.3

classification 💻 cs.LG
keywords time series embeddingsJEPAmultimodal learninganomaly detectionforecastingtransformerrepresentation learningsensor data
0
0 comments X

The pith

Textual descriptions of sensor channels combined with joint embedding prediction let a transformer learn time-series embeddings that perform strongly on multiple tasks via linear probes alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a transformer can learn reusable representations for heterogeneous multivariate time series by feeding textual descriptions of each channel into an order-equivariant encoder and training it with latent-space prediction plus a stability loss. A sympathetic reader would care because such representations could let practitioners apply the same embedding to anomaly detection, classification, and both short- and long-term forecasting without retraining the encoder for each new task or dataset. The work argues that the quality comes chiefly from the prediction objective and the description-aware gating rather than from the text inputs themselves, which mainly serve to identify channels across datasets.

Core claim

CHARM incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order. It is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings. Latent-space prediction encourages robustness to sensor noise while description-aware gating provides interpretability through learned inter-channel relationships. Across anomaly detection, classification, and short- and long-term forecasting, the learned embeddings achieve strong performance using only a linear probe. Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel ide

What carries the argument

CHARM, the Channel-Aware Representation Model: a channel-order-equivariant transformer that uses description-aware gating and is trained by joint embedding prediction in latent space.

If this is right

  • The embeddings support anomaly detection, classification, short-term forecasting and long-term forecasting when paired with a linear probe.
  • Performance is driven mainly by the joint embedding prediction objective and the conditioning architecture.
  • Text descriptions function chiefly as channel identifiers that support generalization across different datasets.
  • Latent-space prediction builds robustness to sensor noise.
  • Description-aware gating learns interpretable inter-channel relationships.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Standardized channel descriptions could let the same embeddings transfer to entirely new sensor configurations without retraining.
  • The same conditioning pattern might be applied to other data types where metadata can be expressed in short text.
  • Because the embeddings already capture long-horizon structure, they could serve as drop-in features for control or planning loops that require future-state estimates.
  • Pairing the embeddings with a language model might allow natural-language queries over raw sensor streams.

Load-bearing premise

The quality of the embeddings is produced by the joint embedding prediction objective together with description-aware gating rather than by dataset-specific fitting or choices made after training.

What would settle it

A controlled experiment in which a plain transformer without the JEPA loss or the text-based gating reaches comparable linear-probe accuracy on the same anomaly detection, classification, and forecasting benchmarks would falsify the claim that those two components drive performance.

Figures

Figures reproduced from arXiv: 2605.31580 by Gerardo Pastrana, Henrik Ohlsson, Sina Khoshfetrat Pakazad, Utsav Dutta.

Figure 1
Figure 1. Figure 1: Overview of the model architecture, featuring a context-aware temporal convolutional network and a series of contextual attention layers, each guided by textual descriptions of the input time series channels [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic of the context-aware temporal convolutional network, performing initial featurization of multivariate time series inputs guided by granular textual descriptions of each channel. anism. An overview of the full architecture is provided in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Gating mechanisms used to control cross-channel and temporal interactions. ilarity threshold matrix, Z, as S = EdE⊤ d , Z[i, j] = sigmoid Ed[i, :]Wb Ed[j, :]⊤  . The similarity threshold matrix governs our inter-channel gating mechanism. Specif￾ically, this layer outputs the gating matrix given as Gd = ReLU(Z − S). This process, illustrated in Figure 3a, allows the model to selectively suppress cross-atte… view at source ↗
Figure 4
Figure 4. Figure 4: JEPA architecture with three encoders processing aug￾mented views. 2.1.3. PUTTING IT ALL TOGETHER This completes the integration of the various components within our multimodal time-series embedding architecture. For a given input tuple t = (T, D, pos), we first generate the initial embeddings X ∈ R T ×C×H from our contextual TCN layer. These embeddings pass through a stack of N contextual attention layers… view at source ↗
Figure 5
Figure 5. Figure 5: Evolution of Channel Gates for the ETT Dataset. A causal structure evolves over training, where the target causal variable Oil Temperature attends to all other independent channels but not vice versa. Extended discussion on evolution of channel gates can be found in Section L.2. the text-based attention mechanism. Additional details on ablations experiments—covering description quality, choice of textual e… view at source ↗
Figure 6
Figure 6. Figure 6: Visualized representation of our data structure. Note, that T ∈ R T ×C , |D| = C, and pos ∈ R T considered in our framework as Tmax. C. Implementation Details We attempt to follow the general set of best practices developed in the field of self-supervised learning, specifically those applicable to the Self-Distillation (Balestriero et al., 2023) family of algorithms. We outline the key details here; 1. Opt… view at source ↗
Figure 7
Figure 7. Figure 7: Na¨ıve attention-weight matrix construction. C.3. Model Sizing For the given hyperparameter set N = 8, d = 128,ffdim = 4d, our pretrained model is ∼7.1M parameters. C.4. Additional Modifications to the transformer layers In line with recent developments in large scale pretraining of transformer based architectures, we implement several modifications that diverge from the original transformer architecture. … view at source ↗
Figure 8
Figure 8. Figure 8: Fast attention-weight matrix construction. D. JEPA D.1. Dataset Generation The core principle of JEPA-based self-superived training involves producing representations for two augmented views originating from the same data instance. JEPA training aims to minimize a discrepancy measure (e.g., ℓ1 or ℓ2) between these representations. In vision, these views commonly result from image augmentations such as jitt… view at source ↗
Figure 9
Figure 9. Figure 9: JEPA Tasks Visualized : Causal Prediction (left) Smoothing (right) [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Context and Target Network 1 class Predictor: 2 def forward(self, ctx_embeds, ctx_idx, target_idx): 3 """ 4 embeds : [..., T, C, H] 5 target_pos : [..., T2] 6 """ 7 x = self.downproj(ctx_embeds) # [..., T, C, H1] 8 mask_tokens = broadcast(self.mask_token, target_idx) # [..., T2, C, H1] 9 10 x = concat([x, mask_tokens]) # [..., T+T2, C, H1] 11 pos = concat([ctx_idx, target_idx]) 12 13 for layer in self.enc… view at source ↗
Figure 11
Figure 11. Figure 11: Predictor Network On the other hand our encoder ingests the embedded time series and returns a contextually embedded time series while maintaining the same output dimensions i.e.; E : R T ×C×H → R T ×C×H Given this notation, our 3 JEPA networks (Context, Target, Predictor) can be represented as: Context ⇒ [F → E1] (7) Target ⇒ [F → E1] (8) Predictor ⇒ [DownProj → E2 → UpProj] (9) Now, with this featurizat… view at source ↗
Figure 12
Figure 12. Figure 12: JEPA 1. Limited context lengths Given our model’s architecture, we are required to compute attention scores over the entire C × T input. As we do not rely on downsampling/patching, we compute the full O(C 2T 2 ) attention matrix, which can be prohibitively large, especially for datasets with a large number of unrelated channels, or extremely long time horizons. Potential workarounds to this are computing … view at source ↗
Figure 13
Figure 13. Figure 13: Sizes of different UEA Datasets Number of Wins. For M models {hm}M m=1, define accuracy of model m on dataset i as Acci,m = 1 ni Xni j=1 1[hm(xij ) = yij ] . The number of wins for model m is Wins(m) = X D i=1 1 h Acci,m = max m′ Acci,m′ i . We report average accuracy and number of wins as they are standard measures used in other papers that use the UEA benchmark, however, as noted in (Fleming & Wallace, … view at source ↗
Figure 14
Figure 14. Figure 14: UCR Anomaly test set reconstructions visualized, with anomalous regions highlighted for 1sddb40, CIMIS44AirTemperature3 and CIMIS44AirTemperature5. true refers to the ground truth values, while pred refers to our reconstruction head’s predictions. Baselines. To compare ourselves to a diverse set of models, we include all baselines available in SKAB leaderboard, as well as T-Rep, TS2Vec, MOMENT. The baseli… view at source ↗
Figure 15
Figure 15. Figure 15: Illness Forecasts 38 [PITH_FULL_IMAGE:figures/full_fig_p038_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Illness Forecasts [PITH_FULL_IMAGE:figures/full_fig_p039_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: ETTh1 Forecasts 39 [PITH_FULL_IMAGE:figures/full_fig_p039_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: ETTh1 Forecasts [PITH_FULL_IMAGE:figures/full_fig_p040_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: ETTh2 Forecasts 40 [PITH_FULL_IMAGE:figures/full_fig_p040_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: ETTh2 Forecasts [PITH_FULL_IMAGE:figures/full_fig_p041_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: ETTm1 Forecasts 41 [PITH_FULL_IMAGE:figures/full_fig_p041_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: ETTm1 Forecasts Linear Probing and Pooling Ablations Experiments related to pooling strategies, shown in [PITH_FULL_IMAGE:figures/full_fig_p042_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Evolution of BasicMotions similarity heatmaps over training epochs (a) Epoch 0 (b) Epoch 3 (c) Epoch 6 (d) Epoch 9 [PITH_FULL_IMAGE:figures/full_fig_p046_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Evolution of Skoltech Anomaly Benchmark similarity heatmaps over training epochs (a) Epoch 0 (b) Epoch 3 (c) Epoch 6 (d) Epoch 9 [PITH_FULL_IMAGE:figures/full_fig_p046_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Evolution of Epilepsy similarity heatmaps over training epochs L.2. Evolution of Channel Gates In this section we aim to visualize how channel gates, as defined in Paragraph Section 2.1.1, evolve over the course of training our model. We plot the gating matrix, Gd, for each dataset for different checkpoints. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p046_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Evolution of Channel Gates for the ETT Dataset 46 [PITH_FULL_IMAGE:figures/full_fig_p046_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Evolution of inter channel gates during training. Checkpoints extracted at epoch=0;step=49, epoch=0;step=499, epoch=0;step=999, epoch=2;step=49, epoch=6;step=49, epoch=8;step=49. Each row represents a particular dataset. Each column represents a sampled checkpoint as training progresses. Each heatmap represents Gd for a particular dataset, which is a C × C matrix with values in [0, 1]. Brighter colors on … view at source ↗
read the original abstract

Transformer-based architectures have advanced sequence modeling in language and vision, yet general-purpose representation learning for heterogeneous multivariate time series remains underexplored. We introduce CHARM (Channel-Aware Representation Model), which incorporates channel-level textual descriptions into a Transformer encoder equivariant to channel order. CHARM is trained with a Joint Embedding Predictive Architecture (JEPA) and a novel loss promoting informative, temporally stable embeddings; latent-space prediction encourages robustness to sensor noise while description-aware gating provides interpretability through learned inter-channel relationships. Across anomaly detection, classification, and short- and long-term forecasting, the learned embeddings achieve strong performance using only a linear probe. Performance is driven primarily by the JEPA objective and conditioning architecture, with text descriptions serving as channel identifiers for cross-dataset generalization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper introduces CHARM (Channel-Aware Representation Model), a Transformer encoder for heterogeneous multivariate time series made equivariant to channel order via incorporation of channel-level textual descriptions. The model is trained with a Joint Embedding Predictive Architecture (JEPA) objective plus a novel loss promoting informative and temporally stable embeddings; latent prediction is intended to confer robustness to sensor noise while description-aware gating supplies interpretability. Embeddings are evaluated on anomaly detection, classification, and short- and long-term forecasting tasks using only linear probes, with the central claim that performance is driven primarily by the JEPA objective and conditioning architecture rather than dataset-specific fitting, and that text descriptions function as channel identifiers enabling cross-dataset generalization.

Significance. If the experimental controls confirm that the reported gains are attributable to the JEPA training objective and multimodal conditioning rather than post-hoc fitting or evaluation choices, the work would constitute a meaningful contribution to general-purpose time-series representation learning by demonstrating how JEPA-style latent prediction combined with textual channel conditioning can yield robust, interpretable embeddings across heterogeneous sensor datasets.

major comments (1)
  1. [Abstract and §4] Abstract and §4 (experimental claims): the assertion that 'performance is driven primarily by the JEPA objective and conditioning architecture' is load-bearing for the central contribution yet is not accompanied by the ablations or controls needed to rule out dataset-specific fitting or post-hoc evaluation artifacts; without these, the claim that text descriptions serve only as channel identifiers for generalization cannot be assessed.
minor comments (2)
  1. The abstract would benefit from naming the specific datasets and reporting at least one quantitative metric (e.g., AUC or MSE improvement) to allow readers to gauge the strength of the 'strong performance' claim.
  2. [§3] Notation for the description-aware gating mechanism should be introduced with an equation or diagram in §3 to clarify how textual embeddings modulate the Transformer layers.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for stronger controls to support the central claims. We will revise the manuscript to include the requested ablations and cross-dataset experiments.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (experimental claims): the assertion that 'performance is driven primarily by the JEPA objective and conditioning architecture' is load-bearing for the central contribution yet is not accompanied by the ablations or controls needed to rule out dataset-specific fitting or post-hoc evaluation artifacts; without these, the claim that text descriptions serve only as channel identifiers for generalization cannot be assessed.

    Authors: We agree that the load-bearing claim requires explicit ablations to isolate the JEPA objective, channel conditioning, and text descriptions from potential dataset-specific fitting or evaluation artifacts. In the revised manuscript we will add: (1) ablation replacing the JEPA latent-prediction loss with standard reconstruction and contrastive baselines while keeping the architecture fixed; (2) ablation removing description-aware gating (replacing channel text with random or positional identifiers); (3) cross-dataset transfer experiments in which embeddings trained on one sensor collection are linearly probed on another to test whether text descriptions function as general channel identifiers. These controls will directly address concerns about post-hoc artifacts and will be reported in an expanded §4 with quantitative tables. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The provided abstract and context describe an empirical model (CHARM) trained via JEPA objective on time-series data, with performance evaluated via linear probes on downstream tasks. No derivation chain, equations, fitted parameters renamed as predictions, or self-citation load-bearing steps are present that would reduce any claimed result to its inputs by construction. The central claims rest on training objectives and empirical results rather than definitional equivalence or imported uniqueness theorems, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; all modeling choices are described at a high level without equations or implementation specifics.

pith-pipeline@v0.9.1-grok · 5672 in / 1107 out tokens · 17316 ms · 2026-06-28T23:07:32.913751+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 5 canonical work pages

  1. [1]

    URL https://openreview.net/pdf? id=UVF1AMBj9u. poster. Chen, M., Shen, L., Li, Z., Wang, X. J., Sun, J., and Liu, C. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters, 2025. URL https: //arxiv.org/abs/2408.17253. Chen, S., Gong, C., Li, J., Yang, J., Niu, G., and Sugiyama, M. Learning contrastive embedding in low-dimens...

    work page doi:10.1109/icassp.2018.8462506 2025

  2. [2]

    Gong, Y.-A

    URL https://doi.org/10.48550/ arXiv.2412.10925. Edwards, T. D. P., Alvey, J., Alsing, J., Nguyen, N. H., and Wandelt, B. D. Scaling-laws for large time-series mod- els, 2025. URL https://arxiv.org/abs/2405. 13867. Falcon, W. and The PyTorch Lightning team. PyTorch Light- ning, March 2019. URL https://github.com/ Lightning-AI/lightning. Fleming, P. J. and ...

    work page doi:10.21437/interspeech.2021-698 2025

  3. [3]

    The learning rate follows a linear warmup followed by a cosine decay

    Optimization ScheduleWe use an AdamW optimizer to optimize our model. The learning rate follows a linear warmup followed by a cosine decay

  4. [4]

    Weight InitializationWe use a fixed N(0,0.02) initialization which is commonly used in pretraining large transformer models (OLMo et al., 2025)

    2025

  5. [5]

    Weight Decay SchedulingWe use a cosine schedule for increasing the optimizer’s weight decay over the course of training which has been shown to be crucial for training stability

  6. [6]

    "" 4Block-wise assembly via tensor indexing and reshape. 5

    EMA Schedule for Target EncoderWe use an exponentially moving average with a momentum schedule that is increased gradually over the course of training. The weight decay scheduling and EMA schedule are identical to IJEPA (Assran et al., 2023). Besides sweeping over a few learning rates, we perform no additional hyperparameter tuning on the rest of the hype...

    2023

  7. [7]

    Time Series Classification Models:MiniRocket

  8. [8]

    Semantic Representation Learning Models :T-Rep,TS2Vec,T-Loss,TS-TCCetc

  9. [9]

    Reconstruction Based Representation Learning Models :MOMENT Given our limited compute availability, all baseline results reported in the results table are drawn from prior published work. We restrict our comparison to models with results on the majority of the UEA datasets, and therefore exclude models with incomplete or missing UEA coverage (e.g., UniTS)...

    work page arXiv 1986

  10. [10]

    ||xtest −ˆxtest|| lie outside the UCL limit

    Then, for the reconstructed test data ˆxtest, we classify anomalies if the absolute values of the residuals, i.e. ||xtest −ˆxtest|| lie outside the UCL limit. This exact anomaly detection setup is commonly applied to all baseline models in the test suite. Downstream Setup.Similar to J.2.1, we rely on training a linear head to reconstruct“clean” training d...

    2021

  11. [11]

    CHARM+NLH:A common non-linear prediction head is trained across all datasets, channels, and horizons, with the encoder kept frozen

  12. [12]

    CHARM+NLH FT:The full model (encoder + non-linear prediction head) is trained end-to-end, shared across datasets, channels, and horizons. Non-Linear Head (NLH)The head is designed to first mix information across both time and channels, then refine within each channel, and finally project to the forecasting horizon: • Transformer across time & channels (nh...

    work page arXiv 2024

  13. [13]

    Z:R T×H →Z mean ∈R H 3.Last Time Step: The embedding from the last time step of each channel is taken as the representative embedding

    Mean Pooling: The embeddings are averaged over the time dimension (but not across channels), yielding an H- dimensional representation per channel. Z:R T×H →Z mean ∈R H 3.Last Time Step: The embedding from the last time step of each channel is taken as the representative embedding. Z:R T×H →Z −1 ∈R H In addition, we experimented with: • Frozen Encoder + 2...

    2024

  14. [14]

    Total number of correct classifications for UEA

  15. [15]

    Average accuracy for UEA

  16. [16]

    Mean Squared Error for ETTh1 (T f = 168)

  17. [17]

    Mean Absolute Error for ETTh1 (T f = 168)

  18. [18]

    Mean Squared Error for ETTh2 (T f = 168)

  19. [19]

    Classification Evaluations.The hyperparameters used for measuring classification performance are listed in Table 25

    Mean Absolute Error for ETTh2 (T f = 168) For the ablation study, we train and probe the model with the following protocols: Pretraining.The base model is pretrained with a subset of datasets (UEA, ETTh, ETTm, Weather, Illness), for 50 epochs, with a learning rate of 5e-4. Classification Evaluations.The hyperparameters used for measuring classification pe...

  20. [20]

    These are obtained through either manual human annotation obtained by parsing the accompanying dataset metadata files, or are natively provided by the dataset provider

    Annotated descriptions: manually curated sensor descriptions obtained from the official dataset metadata. These are obtained through either manual human annotation obtained by parsing the accompanying dataset metadata files, or are natively provided by the dataset provider

  21. [21]

    Noisy descriptions: high quality annotated descriptions, but with words dropped at random (with p= 0.2 ) during both training and evaluation

  22. [22]

    unknown sensor

    Ordinal descriptions: replace the annotated descriptions with structured, placeholder descriptions: [ Sensor1, Sensor2,Sensor3...] for all datasets. •(iv) Effect of text embedding model We investigate the usage of different embedding models to assess the effect on downstream performance. We use 1) nomic (Nussbaum et al., 2025), 2) minilm (Wang et al., 202...

    work page arXiv 2025