Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities

Sourish Gunesh Dhekane; Thomas Ploetz

REVIEW 3 major objections 5 minor 18 references

Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Smart-home activity recognition can work without sending data to an LLM.

desk verdict A plausible zero-shot HAR alternative to LLM prompting, but the headline SOTA comparison is unquantified and the paper needs code/data before the claim holds. read the letter →

arxiv 2507.21964 v1 pith:DBI4JDRR submitted 2025-07-29 cs.AI cs.LG

Sourish Gunesh Dhekane , Thomas Ploetz This is my paper · ORCID

classification cs.AIcs.LG

keywords humanactivityrecognitionzero-shotclassificationsmarthomessentenceembeddingslanguagemodelingsensordatadescriptionsLLM-free

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that zero-shot human activity recognition (HAR) in smart homes can be done without large language models and without any training data, by converting sensor windows and activity labels into natural-language text and comparing their sentence embeddings. If true, this would remove the privacy and reliability problems of existing 'prompt-the-LLM' systems and make HAR deployable offline and locally in new homes. The authors demonstrate the approach on six public datasets, reporting accuracies above 60% on four of them and comparable results to two LLM-based baselines, while showing that both hand-crafted summary generation and activity descriptions are necessary.

What carries the argument

The central mechanism is embedding-space arithmetic between two hand-crafted text modules: a Summary Generation module that converts a sensor window into a fixed-skeleton sentence covering time, duration, top-k locations, and top-k sensors with contextual tags, and an Activity Descriptor module that produces one sentence per activity class from floorplan and metadata. A pre-trained sentence encoder, all-distilroberta-v1, maps both to vectors, and cosine similarity selects the class whose description sits closest to the datapoint summary embedding.

What would settle it

A concrete test would be to run the method on a new smart home dataset without modifying the summary skeleton or descriptions; if the accuracy drops to the level of the Cairo and Kyoto7 results for activities whose descriptions are not well defined semantically, the transferability assumption fails. A direct comparison would be to ask different human annotators to write activity descriptions for the same dataset and show that the performance varies widely with the wording of the descriptions.

Watch

Extended reading notes

Core claim

The paper argues that the essential ingredient for zero-shot HAR is generating the two textual descriptions at the right granularity: a datapoint summary that captures the time of occurrence, duration, top locations, and top fired sensors with their object context, and a one-sentence activity description that captures the duration, location, and sensor signatures expected for that activity. Once these texts are embedded with a pre-trained sentence encoder (all-distilroberta-v1), the correct activity label is simply the class description whose embedding has the highest cosine similarity to the datapoint summary embedding. The authors support this by an ablation that shows a drastic drop in accuracy if either component is removed, and by a comparison in which their zero-shot performance is comparable to that of LLM-based SOTA baselines.

Load-bearing premise

The approach assumes that the hand-written sentence templates and activity descriptions, derived from floorplans and dataset metadata, transfer to any new smart home and capture enough of an activity to separate it from every other class in the embedding space.

Editorial extensions

If this is right

If the approach holds, smart-home HAR can be deployed in new homes without collecting any labeled data and without any network connection to a language model service.
The same pipeline extends to few-shot recognition simply by adding the embeddings of a few labeled sensor windows to the set of candidate class descriptions, which the paper reports improves weighted F1 by about 13.5% on average across five of six datasets.
The comparison with two LLM-based baselines suggests that the commonsense knowledge needed for this task can be supplied by hand-written summaries and activity descriptions rather than by prompting an LLM.
The ablation results imply that the accuracy of such systems depends more on the descriptive fidelity of the generated text than on the choice of similarity metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper leaves implicit is automating the summary and description generation, e.g., from floorplans and sensor metadata, since the current hand-crafted templates are the main barrier to scaling the approach to arbitrary smart homes.
The paper's own results on Cairo and Kyoto7 suggest that the method's ceiling is set by how precisely an activity can be verbally distinguished from its semantic neighbors, which connects to the deeper question of whether linguistic descriptions alone can separate activities that differ mainly in fine-grained sensor signatures.
Because the method exposes a clean embedding geometry, a testable extension would be a fully automatic pipeline that uses the same sentence encoder to generate data summaries from raw event streams and to propose activity descriptions, removing the need for human-written templates.
The observed sensitivity to randomly chosen few-shot samples points toward combining this embedding approach with an active-learning selector that picks maximally informative sensor windows, which would turn the few-shot extension into a more reliable human-in-the-loop tool.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this paper has a real idea—replace LLM prompting with sentence-embedding similarity between hand-crafted data summaries and activity descriptions—and the six-dataset case study is useful. But the central claim that it matches LLM-based SOTA is not supported by the evidence shown. Two baselines appear, no numbers are reported, there are no error bars or significance tests, and Figure 2 is a qualitative visual comparison on one dataset per baseline. The abstract and introduction promise three baselines; only two appear.

What is actually new: Fritsch et al. explored embedding-space zero-shot without activity descriptions; this paper adds activity descriptions and runs a six-dataset study. That is a legitimate incremental contribution. The method is genuinely zero-shot in the sense that no sensor data is used for training, and it runs offline on a small encoder, which addresses real privacy and reliability concerns of LLM prompting. The ablation shows both components matter, and the few-shot extension is a sensible add-on. The discussion is honest about failures on semantically similar classes (Cairo, Kyoto7) and about the manual nature of summary generation.

Soft spots: the evaluation is thin exactly where the headline claim is strongest. Without baseline numbers, 'comparable performance' is unverifiable. The summaries and descriptions are hand-crafted per dataset, including special heuristic rules (e.g., MARBLE phone sensors). That makes the method less automatic than the zero-shot framing suggests, and this should be stated more prominently. No code or data is released, so the numbers in Table 1 are not reproducible. The few-shot results show variance and some degradation with more samples; the authors acknowledge this, which is fair.

Who this is for: researchers working on zero-shot HAR or privacy-preserving smart home analytics will get value from the case study and the clear articulation of the design space. It deserves a serious referee, but only with a request for major revision: quantitative SOTA comparison, code/data, and a clearer separation of automated vs. hand-engineered components.

I would not cite this version in my own work yet, but the idea is worth watching.

Referee Report

3 major / 5 minor

Summary. The paper proposes a zero-shot human activity recognition (HAR) method for smart homes that replaces LLM prompting with language modeling: sensor data are converted into a hand-crafted textual summary, activity labels are converted into hand-crafted textual descriptions, and a pre-trained sentence encoder (all-distilroberta-v1) is used to embed both; classification is performed by cosine similarity to the nearest class description. The method is evaluated in a zero-shot setting on six public datasets (Aruba, Milan, Cairo, Kyoto7, MARBLE, ARAS), reporting accuracies above 60% on four of them, a few-shot extension, and ablations over the summary module, the activity-description module, the sentence encoder, and the similarity metric. The central claim is that this embedding-based approach achieves performance comparable to LLM-based SOTA baselines while avoiding external LLM service dependence and privacy risks.

Significance. If the central claim is supported, the contribution is practically significant: a zero-shot HAR pipeline that runs offline with a small pre-trained sentence encoder and no labeled data would address real privacy and reliability concerns of LLM-prompting approaches. The paper's ablations demonstrate that both the summary-generation and activity-description components are necessary for the reported accuracy, which is a useful design finding, and the few-shot extension is a sensible human-in-the-loop direction. However, the evidence presented is not yet rigorous enough to support the headline claim of comparability to SOTA LLM baselines, because the baseline comparison is qualitative only, and no code or textual artifacts are released for reproduction. The method also performs poorly on the semantically difficult Cairo and Kyoto7 datasets, and the authors correctly acknowledge that limitation in Section 4.5.

major comments (3)

[Section 4.2, Figure 2, and Section 1] The central claim of 'comparable performance to three LLM-based SOTA baselines' is not supported by the evidence shown. Figure 2 compares only two baselines—LLM + Iterative Evolution on ARAS and ADL-LLM on MARBLE—and only visually, with no numerical accuracy or F1 values for those baselines, no error bars, and no significance tests. The text in Section 1 promises three baselines but only two appear in the evaluation. Because 'comparable' cannot be quantified from the figure, the reader cannot tell whether the difference is one point or twenty points. Please provide numerical results for all three claimed baselines on all datasets on which they are evaluated, with variance estimates, and correct the baseline count or explicitly list the three baselines.
[Sections 3.1, 3.2, and 4.2 / Table 1] The method is not reproducible as described because it relies on manually engineered text artifacts that are not shipped: the exact summary skeleton templates, the per-dataset special heuristic rules of Section 3.1.5, and the complete set of activity descriptions of Section 3.2 are not provided in full for all six datasets. Since these textual definitions are the entire predictive machinery, a reader cannot re-implement the method without reverse-engineering them from examples. Moreover, the contribution of the special rules is not isolated in the ablation: Table 1 removes either the whole summary module or the whole activity-description module, but never removes only the special rules. Please release the templates and descriptions (e.g., as supplementary material) and add an ablation that toggles the special rules.
[Section 4.2, Table 1, and Figure 3] The evaluation lacks a comparison against trivial baselines, which is necessary to interpret the reported accuracies and confusion matrices. On Cairo the proposed method obtains 0.46 accuracy and on Kyoto7 0.50, and Figure 3 shows confusion between semantically similar classes; without reporting the majority-class baseline or a random-guess baseline (adjusted for class imbalance), it is unclear whether the method is actually extracting signal on these datasets or merely exploiting label priors. Please report per-dataset class distributions, majority-class accuracy, and the macro-F1 values for such baselines, alongside the confusion matrices already shown.

minor comments (5)

[Table 1] The header 'w/osummary' appears to be a typo; it should read 'w/o summary' for consistency with the other ablation rows.
[Section 4.2, paragraph 1] The sentence beginning 'on the MARBLE dataset' should start with a capital 'On'.
[Section 3.3] The sentence encoder all-distilroberta-v1 is mentioned without a citation; please add a reference for the model so that the exact encoder version is identifiable.
[Figure 2 and Figure 3] The figures are likely difficult to read in print because the panels are small and no numeric labels are given on the bars; please use larger panels, add value labels, and describe in the captions what is being compared and on which dataset each comparison is performed.
[Section 3.1.5] The description of special rules is underspecified: please state how many rules were used per dataset and which activities they applied to, because these rules are a form of dataset-specific engineering that affects the claimed data-independent nature of the method.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proposed zero-shot embedding pipeline is self-contained and does not reduce to its inputs.

full rationale

The derivation chain is explicit and non-circular. Sensor windows are converted to textual summaries by fixed templates (time, duration, top-k locations, top-k sensors; Sec. 3.1), activity labels are converted to one-sentence descriptions using floorplans and dataset metadata (Sec. 3.2), and both are embedded with a fixed pre-trained encoder (all-distilroberta-v1) followed by argmax cosine similarity (Sec. 3.3). No parameter is fitted to the evaluation data, and no equation defines the summary in terms of the label; the hand-crafted descriptions are external class knowledge, not fitted quantities. The only self-citations are [8], a survey, and [14], the authors' prior TDOST work, which supplies the general idea of language modeling sensor triggers and the CASAS evaluation setup; neither is invoked as an unverified uniqueness theorem or as a substitute for the current experiments. The paper's ablations (Table 1) test the contribution of its own modules, which is the opposite of circularity. The headline claim of being 'comparable to three LLM-based SOTA baselines' is under-evidenced (only two baselines, no numerical results), but that is an experimental-rigor problem, not a circularity reduction. Overall, the evaluation is an empirical demonstration against six external datasets rather than an argument that derives its conclusion from its own assumptions.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No numerically fitted parameters appear; the main free choices are design decisions (k=2, sentence encoder, special rules) rather than data-fitted constants. The method rests on the assumption that sentence embeddings of hand-written summaries and descriptions are semantically aligned, which is a domain assumption that the paper's own results only partially support.

free parameters (2)

k in top-k locations/sensors = 2
Chosen by default (Section 3.1.3-3.1.4); affects summary content and therefore classification.
Special heuristic rules for activities
Ad hoc rules added for MARBLE phone activities in Section 3.1.5; dataset-specific design decisions that influence results.

assumptions (3)

domain assumption Pre-trained sentence embeddings place semantically similar texts close together in embedding space.
The whole pipeline relies on this, Section 3.3.
ad hoc to paper The fixed summary skeleton (time, duration, top-k locations, top-k sensors) captures the discriminative signature of activities across all six datasets.
Design based on observations of dataset floorplans and activities, Section 3.1.
domain assumption Activity descriptions derived from metadata and floorplans are precise enough to be closer to their correct summaries than to other classes in embedding space.
Central to zero-shot classification, Section 3.2; the paper admits this fails for vague classes like Wake or Personal Hygiene.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities." pith.science (2026). https://pith.science/paper/DBI4JDRR

@misc{pith2026250721964,
  author       = {Pith},
  title        = {Pith review of: Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DBI4JDRR}},
  note         = {Machine review of arXiv:2507.21964}
}

read the original abstract

Developing zero-shot human activity recognition (HAR) methods is a critical direction in smart home research -- considering its impact on making HAR systems work across smart homes having diverse sensing modalities, layouts, and activities of interest. The state-of-the-art solutions along this direction are based on generating natural language descriptions of the sensor data and feeding it via a carefully crafted prompt to the LLM to perform classification. Despite their performance guarantees, such ``prompt-the-LLM'' approaches carry several risks, including privacy invasion, reliance on an external service, and inconsistent predictions due to version changes, making a case for alternative zero-shot HAR methods that do not require prompting the LLMs. In this paper, we propose one such solution that models sensor data and activities using natural language, leveraging its embeddings to perform zero-shot classification and thereby bypassing the need to prompt the LLMs for activity predictions. The impact of our work lies in presenting a detailed case study on six datasets, highlighting how language modeling can bolster HAR systems in zero-shot recognition.

Figures

Figures reproduced from arXiv: 2507.21964 by the authors.

**Figure 1.** The proposed zero-shot solution models sensor data and activities of interest using natural language via the novel [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Comparison of our zero-shot approach with two SOTA methods: LLM + Iterative Evolution [10] and ADL-LLM [5]. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Zero-shot performance on challenging Cairo and Kyoto7 datasets with semantically similar activities of interest. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Few-shot performance in terms of the weighted F1 score for varying number of labeled datapoints per class. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

[1]

Hande Alemdar, Halil Ertan, Ozlem Durmaz Incel, and Cem Ersoy. 2013. ARAS human activity datasets in multiple homes with multiple residents. In 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. IEEE, 232–235

work page 2013
[2]

Luca Arrotta, Claudio Bettini, and Gabriele Civitarese. 2021. The marble dataset: Multi-inhabitant activities of daily living combining wearable and environmen- tal sensors data. In International Conference on Mobile and Ubiquitous Systems: Computing, Networking, and Services . Springer, 451–468

work page 2021
[3]

Gorka Azkune, Aitor Almeida, and Eneko Agirre. 2020. Cross-environment ac- tivity recognition using word embeddings for sensor and activity representation. Neurocomputing 418 (2020), 280–290

work page 2020
[4]

Hui Chen, Charles Gouin-Vallerand, Kévin Bouchard, Sébastien Gaboury, Mélanie Couture, Nathalie Bier, and Sylvain Giroux. 2024. Enhancing Human Activity Recognition in Smart Homes with Self-Supervised Learning and Self-Attention. Sensors 24, 3 (2024), 884. Thou Shalt Not Prompt: Zero-Shot HAR in Smart Homes via Language Modeling of Sensor Data & Activitie...

work page 2024
[5]

Gabriele Civitarese, Michele Fiori, Priyankar Choudhary, and Claudio Bettini

work page
[6]

Ian Cleland, Luke Nugent, Federico Cruciani, and Chris Nugent. 2024. Leveraging large language models for activity recognition in smart environments. In 2024 International Conference on Activity and Behavior Computing (ABC) . IEEE, 1–8

work page 2024
[7]

Diane J Cook, Aaron S Crandall, Brian L Thomas, and Narayanan C Krishnan
[8]

Sourish Gunesh Dhekane and Thomas Ploetz. 2025. Transfer Learning in Sensor- Based Human Activity Recognition: A Survey. Comput. Surveys (2025)

work page 2025

Show all 18 references

[9]

Stefan Gerd Fritsch, Federico Cruciani, Vitor Fortes Rey, Ian Cleland, Luke Nugent, Paul Lukowicz, and Chris Nugent. 2024. Hierarchical Zero-Shot Approach for Human Activity Recognition in Smart Homes. In International Conference on Ubiquitous Computing and Ambient Intelligenc...

2024
[10]

Jiayuan Gao, Yingwei Zhang, Yiqiang Chen, Tengxiang Zhang, Boshi Tang, and Xiaoyu Wang. 2024. Unsupervised Human Activity Recognition Via Large Lan- guage Models and Iterative Evolution. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processi...

2024
[11]

Hao Niu, Huy Quang Ung, and Shinya Wada. 2022. Source Domain Selection for Cross-House Human Activity Recognition with Ambient Sensors. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) . IEEE, 754–759

2022
[12]

Andrea Rosales Sanabria, Franco Zambonelli, and Juan Ye. 2021. Unsupervised domain adaptation in activity recognition: A GAN-based approach. IEEE Access 9 (2021), 19421–19438

2021
[13]

Naoto Takeda, Roberto Legaspi, Yasutaka Nishimura, Kazushi Ikeda, Thomas Plötz, and Sonia Chernova. 2024. A Synergistic Large Language Model and Supervised Learning Approach to Zero-Shot and Continual Activity Recognition in Smart Homes. In 2024 9th International Conference on...

2024
[14]

Megha Thukral, Sourish Gunesh Dhekane, Shruthi K Hiremath, Harish Haresamu- dram, and Thomas Ploetz. 2025. Layout-Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST). Pro- ceedings of the ACM on Interactive, Mobile, Wearab...

2025
[15]

Qingxin Xia, Takuya Maekawa, and Takahiro Hara. 2023. Unsupervised human activity recognition through two-stage prompting with chatgpt. arXiv preprint arXiv:2306.02140 (2023)

2023 arXiv
[16]

Yunqian Yu, Kun Tang, and Yaqing Liu. 2023. A fine-tuning based approach for daily activity recognition between smart homes. Applied Sciences 13, 9 (2023), 5706. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

2023
[2012]

Computer 46, 7 (2012), 62–69

CASAS: A smart home in a box. Computer 46, 7 (2012), 62–69

2012
[2024]

arXiv preprint arXiv:2407.01238 (2024)

Large language models are zero-shot recognizers for activities of daily living. arXiv preprint arXiv:2407.01238 (2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Hande Alemdar, Halil Ertan, Ozlem Durmaz Incel, and Cem Ersoy. 2013. ARAS human activity datasets in multiple homes with multiple residents. In 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. IEEE, 232–235

work page 2013

[2] [2]

Luca Arrotta, Claudio Bettini, and Gabriele Civitarese. 2021. The marble dataset: Multi-inhabitant activities of daily living combining wearable and environmen- tal sensors data. In International Conference on Mobile and Ubiquitous Systems: Computing, Networking, and Services . Springer, 451–468

work page 2021

[3] [3]

Gorka Azkune, Aitor Almeida, and Eneko Agirre. 2020. Cross-environment ac- tivity recognition using word embeddings for sensor and activity representation. Neurocomputing 418 (2020), 280–290

work page 2020

[4] [4]

Hui Chen, Charles Gouin-Vallerand, Kévin Bouchard, Sébastien Gaboury, Mélanie Couture, Nathalie Bier, and Sylvain Giroux. 2024. Enhancing Human Activity Recognition in Smart Homes with Self-Supervised Learning and Self-Attention. Sensors 24, 3 (2024), 884. Thou Shalt Not Prompt: Zero-Shot HAR in Smart Homes via Language Modeling of Sensor Data & Activitie...

work page 2024

[5] [5]

Gabriele Civitarese, Michele Fiori, Priyankar Choudhary, and Claudio Bettini

work page

[6] [6]

Ian Cleland, Luke Nugent, Federico Cruciani, and Chris Nugent. 2024. Leveraging large language models for activity recognition in smart environments. In 2024 International Conference on Activity and Behavior Computing (ABC) . IEEE, 1–8

work page 2024

[7] [7]

Diane J Cook, Aaron S Crandall, Brian L Thomas, and Narayanan C Krishnan

[8] [8]

Sourish Gunesh Dhekane and Thomas Ploetz. 2025. Transfer Learning in Sensor- Based Human Activity Recognition: A Survey. Comput. Surveys (2025)

work page 2025