Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data

Anik Ghosh

REVIEW 2 major objections 2 minor 22 references

Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The modality gap in zero-shot IMU activity recognition is a training-time issue fixed by contrastive training on discriminative text prototypes.

desk verdict The paper measures a clear lift in zero-shot IMU HAR alignment and accuracy on PAMAP2 from contrastive training plus richer prototypes, but does not fully rule out confounding differences in the experimental setup. read the letter →

arxiv 2606.10789 v1 pith:IRDFLMI2 submitted 2026-06-09 cs.LG

Anik Ghosh This is my paper · ORCID

classification cs.LG

keywords zero-shotlearninghumanactivityrecognitionIMUsensorscontrastivetrainingmodalitygapSentence-BERTmacroF1PAMAP2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes that poor alignment between IMU sensor embeddings and text class representations arises from the encoder's training objective rather than an inherent barrier. Training a temporal convolutional network with cross-entropy on simple label-name Sentence-BERT prototypes produces embeddings whose mean cosine similarity to the text prototypes is only 0.30, while switching to contrastive training on richer discriminative activity descriptions raises the similarity to 0.69. The improved alignment holds across inference methods, and the best combination with inverted softmax correction reaches 73.2 percent accuracy and 0.583 macro F1 on four unseen PAMAP2 classes versus 58.3 percent accuracy and 0.34 macro F1 for the label-name baseline. Richer descriptions compress inter-prototype distances in embedding space because of overlapping biomechanical terms, yet the alignment gain persists when the descriptions stay sufficiently distinctive. The work also shows that accuracy misleads on imbalanced test sets and recommends macro F1 as the standard metric.

What carries the argument

Contrastive training of a temporal convolutional network against Sentence-BERT prototypes built from discriminative activity descriptions, followed by inverted softmax correction at inference.

What would settle it

Running the identical TCN architecture and hyperparameters on the same PAMAP2 split but keeping only label-name prototypes throughout training and inference should produce no rise above 0.30 mean cosine similarity or 58.3 percent accuracy if the claim is correct.

Watch

Extended reading notes

Core claim

The modality gap between sensor embeddings and semantic class representations in zero-shot IMU HAR is a training-time phenomenon governed by the encoder objective. A TCN trained with cross-entropy over label-name Sentence-BERT prototypes yields sensor embeddings with mean cosine similarity of 0.30 to the corresponding text prototypes, while replacing those targets with discriminative activity descriptions raises the similarity to 0.69. This alignment improvement transfers across all three inference methods. The strongest configuration pairs contrastive training with inverted softmax correction and reaches 73.2 percent accuracy and 0.583 macro F1 on the unseen classes, against 58.3 percent ac

Load-bearing premise

The alignment gains and accuracy lift are caused by the switch in training objective and the richer prototypes rather than by differences in hyperparameters, data preprocessing, or the particular choice of four unseen classes.

Editorial extensions

If this is right

The alignment gain from richer prototypes and contrastive training transfers consistently across the three tested inference methods.
Richer text descriptions reduce inter-prototype separability in Sentence-BERT space because of shared biomechanical vocabulary.
Macro-averaged F1 is required as the primary metric for ZSL-HAR benchmarks because overall accuracy misleads under class imbalance.
Contrastive training plus inverted softmax correction produces the highest unseen-class accuracy and macro F1 among the seven evaluated configurations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same contrastive-plus-richer-prototype recipe could be tested on other sensor modalities such as audio or wearable ECG to check whether the training-time gap is modality-specific.
Evaluating the method on additional datasets with more than four unseen classes would test whether the reported gains depend on the particular PAMAP2 split.
Prototype construction could be extended with explicit separability regularization to offset the compression effect observed in Sentence-BERT space.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The main result is that contrastive training with richer activity descriptions raises mean cosine similarity from 0.30 to 0.69 and lifts accuracy to 73.2% and macro F1 to 0.583 on the four held-out classes, versus 58.3% and 0.34 for the label-name cross-entropy baseline. The improvement holds across the three inference methods tested.

The work does a useful job of running the seven training-plus-inference combinations on the same PAMAP2 subject hold-out split and reporting the alignment numbers directly. The secondary observation that richer text descriptions compress the prototype cloud in Sentence-BERT space is also worth noting for anyone building text-based prototypes.

The soft spot is the lack of explicit confirmation that every hyperparameter, augmentation, learning-rate schedule, and preprocessing step stayed identical between the two training pipelines. The four unseen classes are also fixed without a sensitivity check. If any of those details differed, part of the 15-point accuracy gain could come from those choices rather than the objective or prototype richness alone. The abstract reports no error bars or statistical tests, which makes the size of the effect harder to judge.

This paper is aimed at researchers working on practical zero-shot wearable sensing. A reader who needs concrete numbers on PAMAP2 with these exact setups will get value from the controlled comparisons and the macro-F1 recommendation. It is worth sending to peer review so the controls and statistical details can be checked and tightened if necessary.

Referee Report

2 major / 2 minor

Summary. The paper claims that the modality gap in zero-shot IMU-based human activity recognition is a training-time phenomenon that can be closed by switching from cross-entropy on label-name Sentence-BERT prototypes to contrastive training on richer discriminative activity descriptions. On PAMAP2 with a fixed 14-seen/4-unseen split and subjects 108/109 held out, this yields mean cosine similarity rising from 0.30 to 0.69 and, when combined with inverted-softmax inference, unseen-class accuracy of 73.2% and macro F1 of 0.583 versus 58.3% / 0.34 for the label-name baseline. Seven configurations are evaluated; richer prototypes are noted to reduce inter-prototype separability in text space.

Significance. If the reported gains are causally attributable to the change in encoder objective and prototype construction, the work supplies a practical recipe for improving zero-shot generalization on wearable sensor data. Strengths include direct measurement of embedding alignment on a standard public dataset with subject hold-out and explicit comparison of multiple inference methods; these elements make the empirical claims falsifiable and reproducible within the manuscript's scope.

major comments (2)

[experimental evaluation (description of the two training pipelines)] The central attribution of the accuracy lift (58.3% → 73.2%) and cosine-similarity gain (0.30 → 0.69) to the training-objective change assumes that every hyper-parameter, augmentation schedule, learning-rate schedule, batch size, and prototype-construction detail was held identical between the two pipelines. The manuscript provides no explicit statement confirming this isolation across the seven configurations; without it the observed improvements cannot be unambiguously credited to contrastive training and richer prototypes rather than uncontrolled differences.
[results and abstract] The 4 unseen classes are fixed without any sensitivity analysis or alternative splits; combined with the absence of error bars or statistical tests on the reported accuracies and F1 scores, this limits the strength of the claim that the method reliably closes the modality gap.

minor comments (2)

[results] The abstract states that overall accuracy is misleading under class imbalance and recommends macro F1, yet the main results table should explicitly foreground macro F1 for all seven configurations to make this recommendation operational.
[methods] Notation for the inverted-softmax correction and the precise construction of the richer activity descriptions should be formalized in a dedicated methods subsection rather than described narratively.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address the two major comments point-by-point below, proposing targeted revisions to improve clarity and acknowledge limitations without altering the core empirical findings.

read point-by-point responses

Referee: [experimental evaluation (description of the two training pipelines)] The central attribution of the accuracy lift (58.3% → 73.2%) and cosine-similarity gain (0.30 → 0.69) to the training-objective change assumes that every hyper-parameter, augmentation schedule, learning-rate schedule, batch size, and prototype-construction detail was held identical between the two pipelines. The manuscript provides no explicit statement confirming this isolation across the seven configurations; without it the observed improvements cannot be unambiguously credited to contrastive training and richer prototypes rather than uncontrolled differences.

Authors: We confirm that all hyperparameters (including learning rate, batch size, augmentation schedule, optimizer settings, and TCN architecture), data preprocessing, and subject hold-out were identical between the cross-entropy and contrastive pipelines. The sole controlled differences are the training objective and the text prototype content. We will insert an explicit paragraph in the experimental setup section stating this isolation to make the attribution unambiguous. revision: yes
Referee: [results and abstract] The 4 unseen classes are fixed without any sensitivity analysis or alternative splits; combined with the absence of error bars or statistical tests on the reported accuracies and F1 scores, this limits the strength of the claim that the method reliably closes the modality gap.

Authors: The 14/4 split with subjects 108/109 held out follows the exact protocol of prior PAMAP2 zero-shot HAR papers to ensure comparability. We performed no sensitivity analysis over alternative unseen-class combinations and report point estimates from the single held-out test set without error bars or significance tests. The observed gains are consistent across all seven configurations, but we acknowledge this limits generalizability claims. We will add a limitations subsection and revise the abstract and results to include appropriate caveats. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; all results are direct empirical measurements on external PAMAP2 data

full rationale

The paper evaluates seven configurations on the PAMAP2 dataset (14 seen / 4 unseen classes, subjects 108/109 held out) and reports measured accuracy, macro F1, and cosine similarities. No step defines a quantity in terms of a fitted parameter that is then re-used as a 'prediction.' No self-citation chains, uniqueness theorems, or ansatzes are invoked to justify core claims. The derivation chain consists entirely of experimental comparisons whose outputs are independent of the inputs by construction.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on the empirical comparison of training objectives and prototype text on one public dataset.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data." pith.science (2026). https://pith.science/paper/IRDFLMI2

@misc{pith2026260610789,
  author       = {Pith},
  title        = {Pith review of: Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRDFLMI2}},
  note         = {Machine review of arXiv:2606.10789}
}

read the original abstract

Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations. We systematically evaluate seven configurations combining three inference methods with two training pipelines on the PAMAP2 dataset, using 14 seen and 4 unseen activity classes with subjects 108 and 109 held out for testing. We find that the modality gap is a training-time phenomenon governed by the encoder objective. A temporal convolutional network (TCN) trained with cross-entropy over label-name Sentence- BERT prototypes yields sensor embeddings with a mean cosine similarity of 0.30 to the corresponding text prototypes, while replacing the label-name prototype targets with discriminative activity descriptions raises this to 0.69. This alignment improvement transfers consistently across all three inference methods. The strongest result combines contrastive training with inverted softmax correction, achieving 73.2% accuracy and 0.583 macro F1 on unseen classes, compared to 58.3% accuracy and 0.34 macro F1 for the label-name baseline. A secondary finding is that richer text descriptions reduce inter-prototype separability in Sentence-BERT space, because shared biomechanical vocabulary causes the language model to compress the prototype cloud. This effect does not negate the benefits of contrastive alignment provided prototype descriptions retain sufficient discriminative vocabulary. We also demonstrate that overall accuracy is a misleading primary metric when test-set class distributions are imbalanced, and recommend macro-averaged F1 as the standard reporting metric for ZSL-HAR benchmarks.

Figures

Figures reproduced from arXiv: 2606.10789 by the authors.

**Figure 2.** Modality gap per seen class. Contrastive training raised the mean text-sensor cosine from 0.3027 in the baseline to 0.6326 with rich biomechanical descriptions, and further to 0.6851 with discriminative descriptions. This confirms that the training objective directly controls the magnitude of the modality gap, and that the choice of prototype description influences the degree of alignment achieved. 6.2 Prototype Sep… view at source ↗

**Figure 3.** Prototype separability using class labels versus activity descriptions. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Comparison of all approaches under the baseline pipeline (A1 to A4b). [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: Comparison of all approaches under the modified pipeline (A1-M to A3-M). [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Prototype centroid similarity. In a well-aligned space the diagonal entry for each unseen-class row should be the highest in that row. Running and descending stairs satisfy this under the contrastive encoder, consistent with their stronger recall in A2-M. Vacuum cleani…

**Figure 7.** Figure 7: t-SNE prototype placements for class labels versus class descriptions. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 5 canonical work pages

[1]

Elkobaisi, and Kyandoghere Kyamakya

Fadi Al Machot, Mohammed R. Elkobaisi, and Kyandoghere Kyamakya. Zero-shot human activity recognition using non-visual sensors.Sensors, 20(3):825, 2020

2020
[2]

De Silva et al

D.Y. De Silva et al. SEZ-HARN: Self-explainable zero-shot human activity recognition network.arXiv preprint arXiv:2507.00050, 2025

work page arXiv 2025
[3]

Deelaka et al

P.N. Deelaka et al. TEZARNet: TEmporal zero-shot activity recognition network. InNeural Information Processing, ICONIP 2023, volume 1969 ofCommunications in Computer and Information Science, Singapore, 2024. Springer

2023
[4]

Limitations in employing natural language supervision for sensor-based human activity recognition— And ways to overcome them

Harish Haresamudram, Apoorva Beedu, Mashfiqui Rabbi, Sougata Saha, Irfan Essa, and Thomas Ploetz. Limitations in employing natural language supervision for sensor-based human activity recognition— And ways to overcome them. InProceedings of the AAAI Conference on Artificial Intelligence, 2025. arXiv:2408.12023

work page arXiv 2025
[5]

HARGPT: Are LLMs zero-shot human activity recognizers? In2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pages 38–43

Sijie Ji, Xinzhe Zheng, and Chenshu Wu. HARGPT: Are LLMs zero-shot human activity recognizers? In2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pages 38–43. IEEE, 2024

2024
[6]

Zechen Li, Baiyu Chen, Hao Xue, and Flora D. Salim. ZARA: Training-free motion time-series reasoning via evidence-grounded LLM agents.arXiv preprint arXiv:2508.04038, 2025

work page Pith review arXiv 2025
[7]

Sensorllm: Aligning large language models with motion sensors for human activity recognition,

Zechen Li et al. SensorLLM: Aligning large language models with motion sensors for human activity recognition.arXiv preprint arXiv:2410.10624, 2024. To appear at EMNLP 2025

work page arXiv 2024
[8]

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. InAdvances in Neural Information Processing Systems, volume 35, 2022

2022

Show all 22 references

[9]

Characterizing word embeddings for zero-shot sensor-based human activity recognition.Sensors, 19(22):5043, 2019

Moe Matsuki, Paula Lago, and Sozo Inoue. Characterizing word embeddings for zero-shot sensor-based human activity recognition.Sensors, 19(22):5043, 2019

2019
[10]

IMU2CLIP: Language-grounded motion sensor translation with multimodal contrastive learning

Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, and Babak Dama- vandi. IMU2CLIP: Language-grounded motion sensor translation with multimodal contrastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13246...

2023
[11]

Association for Computational Linguistics
[12]

Radford et al

A. Radford et al. Learning transferable visual models from natural language supervision. InProceedings of ICML, 2021

2021
[13]

Radovanovic et al

M. Radovanovic et al. Hubs in space: Popular nearest neighbors in high-dimensional data.Journal of Machine Learning Research, 11:2487–2531, 2010

2010
[14]

Reimers and I

N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP, 2019

2019
[15]

Reiss and D

A. Reiss and D. Stricker. Introducing a new benchmarked dataset for activity monitoring. In16th IEEE International Symposium on Wearable Computers (ISWC), pages 108–109, 2012

2012
[16]

Schonfeld et al

E. Schonfeld et al. Generalized zero- and few-shot learning via aligned variational autoencoders. In Proceedings of CVPR, 2019

2019
[17]

Smith, David H

Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In5th International Conference on Learning Representations (ICLR 2017), 2017

2017
[18]

Catherine Tong, Jinchen Ge, and Nicholas D. Lane. Zero-shot learning for IMU-based activity recognition using video embeddings.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(4):1–23, 2021. 16

2021
[19]

Lampert, Bernt Schiele, and Zeynep Akata

Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly.IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2251–2265, 2018

2018
[20]

Exploring the capabilities of LLMs for IMU-based fine- grained human activity understanding

Lilin Xu, Kaiyuan Hou, and Xiaofan Jiang. Exploring the capabilities of LLMs for IMU-based fine- grained human activity understanding. InProceedings of the 2nd International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys ’25), Irvine, CA, ...

2025
[21]

Large language model- guided semantic alignment for human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(4), 2025

Hua Yan, Heng Tan, Yi Ding, Pengfei Zhou, Vinod Namboodiri, and Yu Yang. Large language model- guided semantic alignment for human activity recognition.Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(4), 2025

2025
[22]

Gupta, and Jingbo Shang

Xiyuan Zhang, Diyan Teng, Ranak Roy Chowdhury, Shuheng Li, Dezhi Hong, Rajesh K. Gupta, and Jingbo Shang. UniMTS: Unified pre-training for motion time series. InAdvances in Neural Information Processing Systems, volume 37, pages 107469–107493, 2024. 17

2024

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.

[1] [1]

Elkobaisi, and Kyandoghere Kyamakya

Fadi Al Machot, Mohammed R. Elkobaisi, and Kyandoghere Kyamakya. Zero-shot human activity recognition using non-visual sensors.Sensors, 20(3):825, 2020

2020

[2] [2]

De Silva et al

D.Y. De Silva et al. SEZ-HARN: Self-explainable zero-shot human activity recognition network.arXiv preprint arXiv:2507.00050, 2025

work page arXiv 2025

[3] [3]

Deelaka et al

P.N. Deelaka et al. TEZARNet: TEmporal zero-shot activity recognition network. InNeural Information Processing, ICONIP 2023, volume 1969 ofCommunications in Computer and Information Science, Singapore, 2024. Springer

2023

[4] [4]

Limitations in employing natural language supervision for sensor-based human activity recognition— And ways to overcome them

Harish Haresamudram, Apoorva Beedu, Mashfiqui Rabbi, Sougata Saha, Irfan Essa, and Thomas Ploetz. Limitations in employing natural language supervision for sensor-based human activity recognition— And ways to overcome them. InProceedings of the AAAI Conference on Artificial Intelligence, 2025. arXiv:2408.12023

work page arXiv 2025

[5] [5]

HARGPT: Are LLMs zero-shot human activity recognizers? In2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pages 38–43

Sijie Ji, Xinzhe Zheng, and Chenshu Wu. HARGPT: Are LLMs zero-shot human activity recognizers? In2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys), pages 38–43. IEEE, 2024

2024

[6] [6]

Zechen Li, Baiyu Chen, Hao Xue, and Flora D. Salim. ZARA: Training-free motion time-series reasoning via evidence-grounded LLM agents.arXiv preprint arXiv:2508.04038, 2025

work page Pith review arXiv 2025

[7] [7]

Sensorllm: Aligning large language models with motion sensors for human activity recognition,

Zechen Li et al. SensorLLM: Aligning large language models with motion sensors for human activity recognition.arXiv preprint arXiv:2410.10624, 2024. To appear at EMNLP 2025

work page arXiv 2024

[8] [8]

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. InAdvances in Neural Information Processing Systems, volume 35, 2022

2022