Pith. sign in

REVIEW 4 major objections 6 minor 35 references

Chart design can hijack what CNNs learn from time-series images, so models often latch onto rendering style instead of the signal itself.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 04:28 UTC pith:PHU35Z5A

load-bearing objection Clean multi-metric evidence that chart encodings reshape CNN features on UCR data; the “hijacking” label still outruns what the diagnostics can separate. the 4 major comments →

arxiv 2607.05641 v1 pith:PHU35Z5A submitted 2026-07-06 cs.CV

VEIL: How Visual Encoding Hijacking Induces Bias In Vision Models

Suranjana Sooraj , Xuyang Chen , Madhumitha Venkatesan , Dongyu Liu This is my paper
classification cs.CV
keywords time-series classificationchart-based representationsCNNsvisual encoding hijackingrepresentation similarityCKAGrad-CAMencoding sensitivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When time series are drawn as line, area, bar, or scatter charts and fed to CNNs, the networks frequently learn features that depend on how the chart was drawn rather than on the underlying temporal class structure. The authors call this visual encoding hijacking and diagnose it with a joint suite of tests: representation similarity (CKA), linear-probe transfer across encodings, and Grad-CAM attribution plus controlled rendering perturbations. Across 31 UCR datasets the same series produce divergent feature spaces and weak cross-encoding transfer for many complex multi-class problems, while simple binary problems remain more invariant. Attention guidance (HINT) can reduce the sensitivity when the diagnostics agree that hijacking is present, but it helps little or hurts when they do not. The practical claim is that chart-based time-series classification is a representation and measurement choice, not a neutral modeling detail.

Core claim

Visual encoding hijacking occurs: CNN representations of the same time series become encoding-dependent (low cross-encoding CKA, weak linear-probe transfer, attribution shifts toward rendering cues) rather than encoding-invariant temporal structure, so chart design must be treated as a representation and measurement decision rather than a simple modeling choice.

What carries the argument

VEIL: a joint diagnostic framework that combines Centered Kernel Alignment (CKA) for representation similarity, cross-encoding linear probing for transferability, Grad-CAM attribution and controlled chart perturbations for evidence localization, plus scalar Encoding Sensitivity Indices (ESI) that summarize off-diagonal divergence.

Load-bearing premise

The joint pattern of low cross-encoding similarity, weak transfer, and attribution or perturbation sensitivity is enough to label the behavior as illegitimate hijacking rather than legitimate encoding-specific evidence that still reflects true signal structure.

What would settle it

A controlled experiment in which two encodings that differ only in task-irrelevant visual cues (for example stroke style or bar edge thickness) still produce high CKA, high probe transfer, and Grad-CAM maps focused on the same temporal regions would falsify the claim that those cues systematically hijack the representation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether CNNs trained on chart renderings of time series learn encoding-invariant temporal structure or encoding-dependent visual cues. It introduces “visual encoding hijacking,” diagnosed jointly via low cross-encoding CKA, weak linear-probe transfer, Grad-CAM attribution shifts, and perturbation sensitivity, and evaluates four chart encodings (line, area, bar, scatter) on 31 UCR datasets under a fixed CNN/training protocol. An Encoding Sensitivity Index (ESI) summarizes dataset-level dependence; HINT-style attention guidance is tested as a mitigation. The main claim is that chart design acts as an inductive bias that shapes learned representations, so chart-based TSC should be treated as a representation/measurement problem rather than a neutral modeling choice.

Significance. If the encoding-dependence results hold, the paper makes a useful contribution to image-based TSC and to machine graphical perception: it shows that visualization design is not a free parameter and supplies a reusable multi-metric diagnostic suite (CKA, cross-probe, ESI, Grad-CAM, controlled perturbations) under a clean, fixed-protocol design across 31 datasets. Strengths include the complementary metrics, the transparent caveats that perturbations/attributions alone do not prove task-irrelevance, the nuanced (not universal) HINT results, and the explicit link to human graphical perception. The work is significant as a diagnostic and framing paper even if the stronger “hijacking = illegitimate bias” reading is softened.

major comments (4)
  1. [Abstract, §1, §3.2, §4.2, Discussion] Central framing vs. evidence (Abstract, §1 definition of hijacking, §3.2 Chart Perturbations, §4.2, Discussion): The paper defines hijacking as encoding-dependent behavior diagnosed by the joint suite (low off-diagonal CKA, weak transfer, attribution/perturbation sensitivity) and titles/frames the work as encoding-induced bias. Yet the same sections repeatedly state that this pattern is also produced by legitimate encoding-specific evidence that faithfully reflects signal structure (line continuity, bar heights, scatter density), and that perturbations do not establish task-irrelevance. Without a control that separates the two cases (e.g., synthetic series with known temporal-only labels, or a temporal-only baseline that forces the metrics to diverge), the joint suite remains definitional rather than discriminative. Either add such a control or align title/abstract/claims with the weaker
  2. [§3.2, §4] §3.2 “Generalization Across Encoding Families” promises a robustness check on five mathematical transforms (GASF, MTF, RP, CWT, STFT) under the same protocol, but §4 reports no corresponding CKA, probe, ESI, or accuracy results. This is a load-bearing gap for the claim that the effect is not an artifact of chart-style rendering alone. Report the results or remove the claim.
  3. [§4.2, Fig. 6a] Fig. 6a reports max–min accuracy Δ = 0.023 across four encodings over 31 datasets, while the narrative emphasizes substantial encoding effects on what models learn. Representational divergence (CKA/UMAP/ESI) can matter even when accuracy is similar, but the paper should reconcile the small accuracy spread with the practical claim that encoding choice is consequential for modeling decisions, and clarify when representation dependence is harmful versus merely different.
  4. [§3.3, Table 1, §4.3] §3.3 / Table 1: HINT uses model-derived Grad-CAM saliency as the alignment target (self-supervised adaptation). Large degradations (Computers −36%, SonyAIBORobotSurface1 −14.14%) are reported but under-analyzed. If HINT is offered as evidence that attention guidance can reduce encoding-sensitive behavior when diagnostics agree, the paper needs a clearer account of when the self-derived target is reliable versus when it reinforces the wrong cues, and whether those failures undermine the diagnostic suite that selected the “encoding-divergent” cases.
minor comments (6)
  1. [Footnote 1 / title] Footnote 1 expands VEIL as “Visual Encoding Illusion in Learning,” while the title uses “Hijacking.” Align the expansion with the title term or explain the distinction.
  2. [Fig. 4] Fig. 4 caption says “FordB and ECG5000” but the subcaptions are “UMAP- Yoga” and “UMAP- ECG5000”; fix the mismatch.
  3. [§3.2, Eq. (1)] Eq. (1) writes CKA(X,Y) with H_a, H_b introduced just above; the formula uses X,Y without stating X=H_a, Y=H_b. Minor notational cleanup.
  4. [§3.2 Sensitivity Indices] ESI_Probe = 1 − mean off-diagonal probe accuracy mixes a similarity-style index with an accuracy scale; state the accuracy range and whether chance-level baselines are subtracted so ESI values are comparable across multi-class datasets.
  5. [References] Several citations appear as CoRR/arXiv or “VTBench” concurrent work; ensure final versions and venue details are updated where available.
  6. [Abstract, §4.3] Abstract says attention-guided training “appears to mitigate this effect when encoding sensitivity is consistently identified”; Table 1 also shows null and large negative deltas. Soften the abstract sentence to match the “diagnostic intervention, not reliable mitigation” stance in §4.3.

Circularity Check

1 steps flagged

No significant circularity: diagnostics are applied to independently trained models; ESI is a post-hoc transform of measured matrices, not a quantity defined to equal the claim.

specific steps
  1. self definitional [Section 3.3 HINT Attention Guidance; L_total = L_cls + λ · L_HINT]
    "Before computing the alignment loss, we mask the most salient Grad-CAM regions with a white bounding box, occluding them to isolate the model’s reliance on those cues. The masked image is then passed through the model to produce a second Grad-CAM map, used as the alignment signal. ... Unlike prior work relying on human annotations, our approach derives supervision from model-driven saliency, enabling a self-supervised adaptation of HINT for chart-based TSC."

    The attention-alignment target is generated from the same model’s own Grad-CAM saliency (after occlusion), so the HINT loss is partially self-referential. This is mild and confined to an optional mitigation experiment whose failures are reported; it does not force the main hijacking claim, which rests on independent CKA/probe/attribution measurements.

full rationale

VEIL is an empirical diagnostic study, not a first-principles derivation. Models are trained independently per encoding under a fixed protocol; CKA, linear-probe transfer, Grad-CAM, PCA/UMAP, and perturbations are then measured on those models. ESI_CKA = 1 − mean off-diagonal CKA and ESI_Probe = 1 − mean off-diagonal probe accuracy are simple summary statistics of the measured matrices, not parameters fitted so that the hijacking claim holds by construction. The paper itself repeatedly concedes that the joint pattern cannot separate illegitimate shortcuts from legitimate encoding-specific evidence (Sections 3.2, 4.2, Discussion), so the central framing is interpretive rather than definitional. HINT uses model-derived saliency as a self-supervised alignment target, introducing mild self-reference, but this is an optional intervention whose mixed results are reported honestly and is not load-bearing for the main claim. No uniqueness theorem, ansatz smuggled via self-citation, or fitted input renamed as prediction appears. The derivation chain is therefore self-contained against external UCR benchmarks; circularity burden is negligible.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The work is empirical; it rests on standard ML assumptions (CKA as similarity, linear probes as transfer lower bound, Grad-CAM as attention proxy) plus the authors' operational definition of hijacking. No free physical constants or new particles; free parameters are ordinary training hyper-parameters held fixed. Invented entities are conceptual labels, not ontological posits.

free parameters (5)
  • Adam learning rate = 1e-3
    Fixed at 1e-3 for all runs; not tuned per dataset, but still a free choice that could interact with encoding.
  • weight decay = 1e-2
    Fixed at 1e-2; same comment as learning rate.
  • early-stopping patience = 10
    Fixed at 10 epochs; affects which checkpoint is evaluated.
  • HINT loss weight λ
    Balances classification and attention-alignment losses; value not reported in the provided text, yet required for the mitigation experiments.
  • image resolution = 128x128
    All charts rendered at 128 imes128; choice can alter edge and density cues.
axioms (5)
  • domain assumption Linear CKA is a valid measure of representation similarity across encoders trained on visually distinct inputs.
    Invoked throughout Section 3.2 and Results 4.1; standard in representation-learning literature but still an assumption about what 'alignment' means.
  • domain assumption Cross-encoding linear-probe accuracy lower-bounds shared task-relevant information.
    Section 3.2; treats logistic regression on frozen features as a pure transfer test.
  • domain assumption Grad-CAM heatmaps on the final convolutional layer faithfully indicate the visual evidence used for classification.
    Section 3.2 and Figure 2; known limitations of Grad-CAM are not quantified.
  • domain assumption Identical training protocol (same CNN, optimizer, early stopping, no per-dataset tuning) isolates encoding effects from optimization effects.
    Section 3.1; necessary for causal attribution of performance differences to chart type.
  • ad hoc to paper Joint pattern of low CKA + low transfer + attribution/perturbation sensitivity diagnoses 'hijacking' rather than legitimate encoding-specific evidence.
    Core definitional move of the paper (Introduction and Discussion); the authors themselves note that single metrics cannot establish task-irrelevance.
invented entities (2)
  • visual encoding hijacking no independent evidence
    purpose: Name the phenomenon of encoding-dependent representations that undermine cross-encoding alignment and transfer.
    Operationally defined by the joint diagnostic suite; no independent physical existence claimed outside the measured behavior.
  • Encoding Sensitivity Index (ESI_CKA, ESI_Probe) no independent evidence
    purpose: Scalar summary of off-diagonal CKA or probe matrices so datasets can be ranked by encoding dependence.
    Simple transforms of existing matrices; useful but definitional.

pith-pipeline@v1.1.0-grok45 · 14090 in / 3369 out tokens · 29550 ms · 2026-07-11T04:28:16.297974+00:00 · methodology

0 comments
read the original abstract

Rendering time series as chart images for CNN-based classification has become increasingly common in time-series classification (TSC). However, it remains unclear whether models learn underlying temporal patterns or rely on encoding-specific visual cues introduced by chart design. We present VEIL: a systematic study examining how chart encodings influence learned representations through complementary analyses of similarity, transferability, and attribution. Attention-guided training appears to mitigate this effect when encoding sensitivity is consistently identified across diagnostics, but provides limited or negative benefit when such signals are absent. These findings position VEIL within the broader question of how machines perceive visualizations -- extending graphical perception from human readers to vision models -- and show that visualization design choices shape learned representations in ways that warrant treating chart-based TSC as a representation and measurement problem rather than a simple modeling decision.

Figures

Figures reproduced from arXiv: 2607.05641 by Dongyu Liu, Madhumitha Venkatesan, Suranjana Sooraj, Xuyang Chen.

Figure 1
Figure 1. Figure 1: VEIL Pipeline. We convert time series into chart-based visual encodings (line, area, bar, scatter), learn features using CNNs, analyze [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Grad-CAM heatmap overlays on original chart images [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: (a)-(b) UMAP projections of learned embeddings colored [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Encoding Sensitivity Index (ESI) values for CKA and [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Analysis of impact of chart-feature perturbations. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    Alain and Y

    G. Alain and Y . Bengio. Understanding intermediate layers using lin- ear classifier probes.ICLR Workshop, 2017. 1, 2

    2017

  2. [2]

    Alghanemi et al

    S. Alghanemi et al. Enhancing multivariate time series forecasting through integration of CWT scalograms as CNN channels. InStudies in Systems, Decision and Control. Springer, 2024. doi: 10.1007/978-3 -031-71649-2 3 1

    doi:10.1007/978-3 2024

  3. [3]

    M. K. Chung. Introduction to logistic regression.arXiv preprint arXiv:2008.13567, 2020. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2008

  4. [4]

    H. A. Dau, A. Bagnall, K. Kamgar, C.-C. M. Yeh, Y . Zhu, S. Gharghabi, C. A. Ratanamahatana, and E. Keogh. The UCR time series classification archive.IEEE/CAA Journal of Automatica Sinica, 6(6):1293–1305, 2019. doi: 10.1109/JAS.2019.1911747 1

    doi:10.1109/jas.2019.1911747 2019

  5. [5]

    Eckmann, S

    J.-P. Eckmann, S. O. Kamphorst, and D. Ruelle. Recurrence plots of dynamical systems.Europhysics Letters, 4(9):973–977, 1987. doi: 10 .1209/0295-5075/4/9/004 3

    1987

  6. [6]

    N. M. Foumani, L. Miller, C. W. Tan, G. I. Webb, G. Forestier, and A. Dempster. Deep learning for time series classification and extrinsic regression: A current survey.ACM Computing Surveys, 56(9):1–45,

  7. [7]

    doi: 10.1145/3649448 1

    doi:10.1145/3649448

  8. [8]

    Geirhos, J.-H

    R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2:665–673, 2020. doi: 10. 1038/s42256-020-00257-z 1, 2

    2020

  9. [9]

    Geirhos, P

    R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet-trained CNNs are biased towards texture; in- creasing shape bias improves accuracy and robustness. InProceedings of the International Conference on Learning Representations (ICLR),

  10. [10]

    D. W. Griffin and J. S. Lim. Signal estimation from modified short- time Fourier transform.IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(2):236–243, 1984. doi: 10.1109/TASSP.1984. 1164317 3

    work page doi:10.1109/tassp.1984 1984

  11. [11]

    Hatami, Y

    N. Hatami, Y . Gavet, and J. Debayle. Classification of time-series images using deep convolutional neural networks. InProceedings of the 10th International Conference on Machine Vision (ICMV), p. 106960Y , 2018. doi: 10.1117/12.2309486 1

    doi:10.1117/12.2309486 2018

  12. [12]

    J. D. Hunter. Matplotlib: A 2d graphics environment.Computing in Science & Engineering, 9(3):90–95, 2007. doi: 10.1109/MCSE.2007 .55 2

    doi:10.1109/mcse.2007 2007

  13. [13]

    M. A. Islam, M. Kowal, P. Esser, S. Jia, B. Ommer, K. G. Derpanis, and N. D. B. Bruce. Shape or texture: Understanding discrimina- tive features in cnns. In9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 2

    2021

  14. [14]

    Ismail Fawaz, G

    H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller. Deep learning for time series classification: A review.Data Mining and Knowledge Discovery, 33(4):917–963, 2019. doi: 10. 1007/s10618-019-00619-1 1

    2019

  15. [15]

    I. T. Jolliffe.Principal Component Analysis. Springer, New York, 2nd ed., 2002. 1, 2

    2002

  16. [16]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InProceedings of the International Conference on Learning Repre- sentations (ICLR), 2015. 2

    2015

  17. [17]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. InProceedings of the International Conference on Machine Learning (ICML), pp. 3519–3529, 2019. 1, 2

    2019

  18. [18]

    S. Lee, M. Chang, S. Park, and J. Seo. Assessing graphical percep- tion of image embedding models using channel effectiveness. In2024 IEEE Visualization and Visual Analytics (VIS), pp. 226–230. IEEE,

  19. [19]

    Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou. Time series as images: Vision transformer for irregularly sampled time series. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. 1

    2023

  20. [20]

    S. Long, A. Chatzimparmpas, E. Alexander, M. Kay, and J. Hullman. Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization. InProceedings of the 2025 CHI Confer- ence on Human Factors in Computing Systems, pp. 1–20, 2025. 2

    2025

  21. [21]

    S. G. Mallat. A theory for multiresolution signal decomposition: The wavelet representation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):674–693, 1989. doi: 10.1109/34.192463 3

    doi:10.1109/34.192463 1989

  22. [22]

    M. C. Mariani, M. A. A. Bhuiyan, O. K. Tweneboah, and H. Gonzalez- Huizar. Transforming time series into texture images: A fusion of recurrence plots and gramian angular fields. InProceedings of the Hawaii University International Conferences (HUIC-STEM), 2024. 1

    2024

  23. [23]

    McInnes, J

    L. McInnes, J. Healy, N. Saul, and L. Großberger. UMAP: Uniform manifold approximation and projection for dimension reduction.Jour- nal of Open Source Software, 3(29):861, 2018. doi: 10.21105/joss. 00861 2

    doi:10.21105/joss 2018

  24. [24]

    Middlehurst, P

    M. Middlehurst, P. Sch ¨afer, and A. Bagnall. Bake off redux: A review and experimental evaluation of recent time series classification algo- rithms.Data Mining and Knowledge Discovery, 38(4):1958–2031,

    1958

  25. [25]

    doi: 10.1007/s10618-024-01022-1 2

    doi:10.1007/s10618-024-01022-1

  26. [26]

    J. Ni, Z. Zhao, C. Shen, H. Tong, D. Song, W. Cheng, D. Luo, and H. Chen. Harnessing vision models for time series analysis: A survey. arXiv preprint arXiv:2502.08869, 2025. 1

    Pith/arXiv arXiv 2025

  27. [27]

    Y . K. Oh, H. Kim, and S. Kim. TSSI: Time series as screenshot images for multivariate time series classification using convolutional neural networks.Computers & Industrial Engineering, 2025. doi: 10.1016/j .cie.2024.110820 1

    doi:10.1016/j 2025

  28. [28]

    R. A. Proma, G. J. Quadri, and P. Rosen. Evaluating line chart strate- gies for mitigating density of temporal data: The impact on trend, pre- diction, and decision-making. InInternational Symposium on Visual Computing, pp. 223–235. Springer, 2025. 2

    2025

  29. [29]

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 618–626, 2017. doi: 10. 1109/ICCV.2017.74 1, 2

    2017

  30. [30]

    R. R. Selvaraju, S. Lee, Y . Shen, H. Jin, S. Ghosh, L. Heck, D. Ba- tra, and D. Parikh. Taking a HINT: Leveraging explanations to make vision and language models more grounded. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2591–2600, 2019. 2

    2019

  31. [31]

    Shortcut Learning Susceptibility in Vision Classifiers

    P. Suhail, V . Goel, and A. Sethi. Shortcut learning susceptibility in vision classifiers.arXiv preprint arXiv:2502.09150, 2025. 2

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representations

    M. Venkatesan, X. Chen, and D. Liu. Vtbench: A multimodal frame- work for time-series classification with chart-based representations. CoRR, abs/2604.27259, 2026. 1, 2, 3, 4

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  33. [33]

    Wang and T

    Z. Wang and T. Oates. Imaging time-series to improve classification and imputation. InProceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI), pp. 3939–3945,

  34. [34]

    H. Wu, T. Hu, Y . Liu, H. Zhou, J. Wang, and M. Long. Timesnet: Tem- poral 2d-variation modeling for general time series analysis.arXiv preprint arXiv:2210.02186, 2022. 1

    Pith/arXiv arXiv 2022

  35. [35]

    B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu. Fault detection and identification method: 3D-CNN combined with continuous wavelet transform.Computers & Chemical Engineering, p. 108761, 2024. doi: 10.1016/j.compchemeng.2024.108761 1

    work page doi:10.1016/j.compchemeng.2024.108761 2024