pith. sign in

arxiv: 2410.23222 · v3 · submitted 2024-10-30 · 💻 cs.LG · cs.AI· stat.ML

Dataset-Driven Channel Masks in Transformers for Multivariate Time Series

Seunghan Lee , Taeyoung Park , Kibok Lee This is my paper

Pith reviewed 2026-05-23 18:36 UTC · model grok-4.3

classification 💻 cs.LG cs.AIstat.ML
keywords channel maskspartial channel dependencemultivariate time seriesTransformer attentionchannel dependencydataset-specific parameterssimilarity matrix
0
0 comments X

The pith

Channel masks from similarity matrices and learnable domain parameters enable partial channel dependence in Transformer attention for multivariate time series.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that attention mechanisms in Transformer models for multivariate time series typically enforce either complete or absent channel dependence, which overlooks dataset-specific patterns. It introduces partial channel dependence by constructing channel masks that combine a data-derived similarity matrix with a small set of learnable dataset-specific parameters. These masks are multiplied element-wise into the attention matrices, allowing the model to capture only the relevant dependencies. The approach is shown to integrate into existing Transformer backbones and improve results across forecasting, classification, and imputation tasks on multiple datasets.

Core claim

Channel masks consisting of a similarity matrix that captures relationships between channels and dataset-specific learnable domain parameters that refine it are integrated into attention matrices via element-wise multiplication, thereby achieving partial channel dependence and enhancing channel dependency modeling in Transformer-based multivariate time series models.

What carries the argument

Channel masks (CMs) formed by a similarity matrix and dataset-specific learnable domain parameters, multiplied element-wise into the attention matrices to realize partial channel dependence.

If this is right

  • The method improves performance on diverse multivariate time series tasks including forecasting and classification without altering core Transformer architecture.
  • It applies across multiple existing Transformer backbones by only modifying the attention computation step.
  • Dataset-specific parameters allow adaptation to varying degrees of channel dependence present in different data collections.
  • Validation across tasks shows gains from balancing full and zero channel dependence through the masks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The masks could reduce reliance on manual per-dataset architecture choices by letting parameters adjust dependence automatically.
  • The partial dependence idea might transfer to other attention-based sequence models where not all variables interact equally.
  • Further work could test whether the similarity matrix can be learned end-to-end without a separate pre-computation step.

Load-bearing premise

A similarity matrix derived from the data plus a modest number of learnable domain parameters will reliably isolate the relevant partial dependencies without introducing harmful bias.

What would settle it

An experiment on a synthetic multivariate time series dataset engineered with known full channel dependence, where applying the learned masks reduces accuracy compared to unmodified attention.

Figures

Figures reproduced from arXiv: 2410.23222 by Kibok Lee, Seunghan Lee, Taeyoung Park.

Figure 1
Figure 1. Figure 1: PCD aims to capture the varying dependencies between chan￾nels across datasets. Multivariate time series (MTS) forecasting has been explored with two different strategies: the channel-dependent (CD) strategy and the channel-independent (CI) strategy, with the former emphasizing inter-channel dependencies, while the latter ignoring these dependencies and dealing with channels individually. However, most pre… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Domain parameters to adjust correlation matrix. As correlation is a relative measure depending on the dataset, we refine the correlation matrix using the domain parameters. First, we normalize |R| by subtracting its mean, resulting in R¯ . We then scale and shift R¯ using domain parameters α and β, respectively, and apply a sigmoid function, resulting in M = σ(α · R¯ + β). 3.1 COMPONENTS OF CHANNEL MASK As… view at source ↗
Figure 4
Figure 4. Figure 4: Global and local dependencies. (a) shows a CM and an attention matrix, which capture the global and local dependencies between channels, respectively. (b) illustrates the global and local correlations between two channels of ETTh1 (Zhou et al., 2021), revealing that local correlations can vary by input TS even with the same global correlation. Global and local CD. As a correlation matrix is calculated base… view at source ↗
Figure 5
Figure 5. Figure 5: CD ratio. CD ratio of IC×C for CI, σ(α · R¯ + β) for PCD, and 1C×C for CD. To quantify the degree of CD for each dataset, we propose to measure the channel dependence ratio (CD ratio), a metric based on a CM. The CD ra￾tio of M, denoted as r(M), is the average of the off-diagonal elements of M, excluding the autocor￾relations of their respective channels. This metric yields a value of 0 for CI cases and 1 … view at source ↗
Figure 6
Figure 6. Figure 6: TS visualization by r(M). + Domain Params [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance gain by CD vs. CD ratio. Method Dataset MSE MAE UniTS 1.006 0.701 + CM FCST + CLS 0.995 0.684 FCST 0.993 0.683 Closest 0.993 0.683 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of CMs w/ and w/o domain parameters. The figure shows the correlation matrices and the CMs of two datasets, with each color scaled from 0 (light) to 1 (dark). In the inference stage ( f: pretrained model w/o masking ) Channel 1 Masked Input Channel 2~170 Loss f( , ,.., )= 1 2 170 Input [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Masked channel prediction. H PEMS04 (C = 307) PEMS08 (C = 170) iTrans. + CM Impr. iTrans. + CM Impr. 12 0.549 0.300 45.4% 0.628 0.200 68.1% 24 0.718 0.351 51.1% 0.678 0.241 64.5% 48 0.750 0.409 45.5% 1.197 1.059 11.5% 96 0.758 0.513 32.3% 1.375 1.217 11.5% Avg. 0.694 0.393 43.3% 0.970 0.679 29.9% [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Robustness to missingness [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
read the original abstract

Recent advancements in foundation models have been successfully extended to the time series (TS) domain, facilitated by the emergence of large-scale TS datasets. However, previous efforts have primarily Capturing channel dependency (CD) is essential for modeling multivariate time series (TS), and attention-based methods have been widely employed for this purpose. Nonetheless, these methods primarily focus on modifying the architecture, often neglecting the importance of dataset-specific characteristics. In this work, we introduce the concept of partial channel dependence (PCD) to enhance CD modeling in Transformer-based models by leveraging dataset-specific information to refine the CD captured by the model. To achieve PCD, we propose channel masks (CMs), which are integrated into the attention matrices of Transformers via element-wise multiplication. CMs consist of two components: 1) a similarity matrix that captures relationships between the channels, and 2) dataset-specific and learnable domain parameters that refine the similarity matrix. We validate the effectiveness of PCD across diverse tasks and datasets with various backbones. Code is available at this repository: https://github.com/YonseiML/pcd.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces partial channel dependence (PCD) as a refinement to channel dependency (CD) modeling in Transformer-based multivariate time series. It proposes channel masks (CMs) formed from a similarity matrix of channel relationships plus dataset-specific learnable domain parameters; these masks are applied via element-wise multiplication to the attention matrices. The authors claim this lightweight, dataset-driven addition improves CD modeling and validate it across tasks, datasets, and backbones.

Significance. If the empirical results support the claim, the approach would offer a simple mechanism to inject dataset-specific partial dependencies into existing attention mechanisms without architectural overhaul, addressing a gap in how standard Transformers handle multivariate channel interactions.

major comments (3)
  1. [Abstract] Abstract: The central claim of effectiveness rests on validation 'across diverse tasks and datasets with various backbones,' yet the abstract (and the method description) supplies no quantitative results, ablation studies, error bars, baseline comparisons, or even pseudocode for mask construction and integration. This absence makes the effectiveness claim impossible to assess.
  2. [§3] §3 (Method): The domain parameters are explicitly dataset-specific and learnable; no experiment demonstrates that the resulting masks produce improvements independent of these fitted values or that the similarity matrix isolates relevant partial dependencies without introducing dataset-specific bias. This creates a circularity risk between the claimed PCD benefit and the per-dataset training procedure.
  3. [§4] §4 (Experiments): No details are provided on how many domain parameters are introduced per dataset, whether hyper-parameter search is required, or how performance compares to simply adding equivalent free parameters without the similarity-matrix structure; these omissions are load-bearing for the claim that CMs are a general, lightweight enhancement.
minor comments (2)
  1. [Abstract] Abstract contains a capitalization error: 'primarily Capturing channel dependency' should be lowercase.
  2. The repository link is given but no details on reproducibility (e.g., exact hyper-parameters for the domain parameters or seed settings) are mentioned in the text.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our work introducing partial channel dependence through channel masks in Transformer models for multivariate time series. We respond point-by-point to the major comments below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim of effectiveness rests on validation 'across diverse tasks and datasets with various backbones,' yet the abstract (and the method description) supplies no quantitative results, ablation studies, error bars, baseline comparisons, or even pseudocode for mask construction and integration. This absence makes the effectiveness claim impossible to assess.

    Authors: We agree that the abstract would be strengthened by including quantitative highlights. The full manuscript reports detailed results in Section 4, including performance gains over baselines across tasks, datasets, and backbones. We will revise the abstract to summarize key quantitative improvements and add pseudocode for mask construction and integration to Section 3. revision: yes

  2. Referee: [§3] §3 (Method): The domain parameters are explicitly dataset-specific and learnable; no experiment demonstrates that the resulting masks produce improvements independent of these fitted values or that the similarity matrix isolates relevant partial dependencies without introducing dataset-specific bias. This creates a circularity risk between the claimed PCD benefit and the per-dataset training procedure.

    Authors: The similarity matrix is computed from channel-wise relationships in the input data to capture structural dependencies, with domain parameters providing dataset-specific scaling within that fixed structure. End-to-end task performance validates the PCD approach. We acknowledge that dedicated ablations isolating the similarity matrix from the learnable parameters would further address potential circularity concerns, and we will incorporate such experiments in the revision. revision: yes

  3. Referee: [§4] §4 (Experiments): No details are provided on how many domain parameters are introduced per dataset, whether hyper-parameter search is required, or how performance compares to simply adding equivalent free parameters without the similarity-matrix structure; these omissions are load-bearing for the claim that CMs are a general, lightweight enhancement.

    Authors: We will add explicit details on the number of domain parameters per dataset and clarify that they are learned jointly without separate hyper-parameter search. We will also include a new ablation comparing against a variant with an equivalent number of unstructured free parameters added to attention, to demonstrate the benefit of the similarity-matrix structure. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper proposes channel masks (similarity matrix + dataset-specific learnable domain parameters) as an architectural addition to Transformer attention for partial channel dependence. No equations or derivation chain are shown in the abstract that reduce any claimed result to its inputs by construction. Learnable parameters are explicitly part of the model definition and trained on data, which is standard ML practice rather than a 'prediction' or self-definition. No self-citations, uniqueness theorems, or ansatzes are referenced. The method is presented as a design choice validated empirically across datasets, making the derivation self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 2 invented entities

The approach rests on the domain assumption that channel relationships are only partially shared across datasets and can be isolated by a similarity matrix plus a small set of learnable scalars; the learnable scalars themselves are free parameters fitted per dataset.

free parameters (1)
  • dataset-specific and learnable domain parameters
    These scalars refine the similarity matrix and are optimized during training on each dataset.
axioms (1)
  • domain assumption Channel dependencies in multivariate time series are partial and dataset-specific rather than fully shared or fully independent.
    This premise directly motivates the introduction of PCD and the mask construction.
invented entities (2)
  • Partial channel dependence (PCD) no independent evidence
    purpose: Conceptual framing that dependencies are neither full nor absent but partial and data-dependent.
    New term introduced to justify the mask design.
  • Channel masks (CMs) no independent evidence
    purpose: Element-wise multiplier applied to attention matrices.
    Core technical artifact of the paper.

pith-pipeline@v0.9.0 · 5725 in / 1076 out tokens · 35135 ms · 2026-05-23T18:36:12.791850+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 2 internal anchors

  1. [1]

    Timemachine: A time series is worth 4 mambas for long-term forecasting

    Md Atik Ahamed and Qiang Cheng. Timemachine: A time series is worth 4 mambas for long-term forecasting. arXiv preprint arXiv:2403.09898,

    work page arXiv

  2. [2]

    arXiv preprint arXiv:1811.00075,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Monash time series forecasting archive

    Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman, and Pablo Montero- Manso. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643,

    work page arXiv

  4. [4]

    itransformer: Inverted transformers are effective for time series forecasting

    Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR, 2024a. 10 Yong Liu, Haoran Zhang, Chenyu Li, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. Timer: Generative pre-trained transformers are large time series models. In ICML, 202...

    work page 2016

  5. [5]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Vcformer: Variable correlation transformer with in- herent lagged correlation for multivariate time series forecasting

    Yingnan Yang, Qingling Zhu, and Jianyong Chen. Vcformer: Variable correlation transformer with in- herent lagged correlation for multivariate time series forecasting. arXiv preprint arXiv:2405.11470,

    work page arXiv

  7. [7]

    Table A.1: Single-task forecasting datasets. 13 A.2 D ATASET FOR MULTI-TASK MODEL : U NITS The datasets used in the experiment are aggregated from the Monash Forecasting Repository (Goda- hewa et al., 2021), the Time Series Classification Website (Middlehurst et al., 2024), and the Time Series Library (Wu et al., 2023). The combined training set includes ...

    work page 2021

  8. [8]

    9 270 29 12SpokenArabicDigits (Bagnall et al., 2018)10 6599 93 13Heartbeat (Bagnall et al.,

    work page 2018

  9. [9]

    5 500 140 1NonInvasiveFetalECGThorax1 (Dau et al., 2019)52 1800 750 1 EEG Blink (Bagnall et al.,

    work page 2019

  10. [10]

    2 5890 62 144SelfRegulationSCP2 (Bagnall et al., 2018)2 200 1152 7 Sensors ElectricDevices (Dau et al.,

    work page 2018

  11. [11]

    2 3636 500 1 Human ActivityMotionSenseHAR (Bagnall et al., 2018)6 966 200 12EMOPain (Bagnall et al.,

    work page 2018

  12. [12]

    3 968 180 30UWaveGestureLibrary (Bagnall et al., 2018)8 120 315 3 Traffic Chinatown (Dau et al.,

    work page 2018

  13. [13]

    2 20 24 1MelbournePedestrian (Dau et al., 2019)10 1194 24 1PEMS-SF (Bagnall et al.,

    work page 2019

  14. [14]

    Category Dataset L C Electricity ETTm1 (Zhou et al., 2021)96 7 ETTh1 (Zhou et al., 2021)96 7 ECL (Wu et al.,

    3 1000 1024 1 Table A.5: Few-shot classification datasets. Category Dataset L C Electricity ETTm1 (Zhou et al., 2021)96 7 ETTh1 (Zhou et al., 2021)96 7 ECL (Wu et al.,

    work page 2021

  15. [15]

    Category Dataset L C Machine SMD (Su et al.,

    96 321 Weather Weather (Wu et al., 2021)96 21 Table A.6: Few-shot imputation datasets. Category Dataset L C Machine SMD (Su et al.,

    work page 2021

  16. [16]

    15 A.2.3 Z ERO-SHOT LEARNING For TS forecasting in a zero-shot setting, we evaluate the effectiveness of our proposed method using six datasets

    96 25 SpacecraftMSL (Hundman et al., 2018)96 55SMAP (Hundman et al., 2018)96 25 InfrastructureSWaT (Mathur & Tippenhauer, 2016)96 51 Table A.7: Few-shot anomaly detection datasets. 15 A.2.3 Z ERO-SHOT LEARNING For TS forecasting in a zero-shot setting, we evaluate the effectiveness of our proposed method using six datasets. Three of these datasets are use...

    work page 2018

  17. [17]

    All experimental settings follow those outlined in UniTS (Gao et al., 2024)

    on datasets from various domains, under multiple settings, including multi-task, few-shot, and zero-shot settings. All experimental settings follow those outlined in UniTS (Gao et al., 2024). The sections and tables outlining the full experiment results are listed in Table D.1. Settings Section TS downstream tasks FCST CLS IMP AD Multi-task D.1 Table 3 Ta...

    work page 2024

  18. [18]

    0.175 0.266 0.168 0.263 0.168 0.263 0.168 0.262 Table E.1: Results of TS forecasting with TimeSiam

    96 0.147 0.239 0.140 0.236 0.140 0.236 0.141 0.237 192 0.162 0.253 0.157 0.251 0.157 0.251 0.157 0.250 336 0.175 0.269 0.173 0.268 0.173 0.268 0.172 0.267 720 0.215 0.304 0.203 0.297 0.203 0.297 0.203 0.296 Avg. 0.175 0.266 0.168 0.263 0.168 0.263 0.168 0.262 Table E.1: Results of TS forecasting with TimeSiam. 22 F M ASKED CHANNEL PREDICTION Tables F.1 an...

    work page 2021