pith. sign in

arxiv: 1907.00124 · v1 · pith:Z3JR7GWKnew · submitted 2019-06-29 · 💻 cs.CR

Helion: Enabling a Natural Perspective of Home Automation

Sunil Manandhar , Kevin Moran , Kaushal Kafle , Ruhao Tang , Denys Poshyvanyk , Adwait Nadkarni This is my paper

Pith reviewed 2026-05-25 13:23 UTC · model grok-4.3

classification 💻 cs.CR
keywords home automationsmart homesIoT securityuser routinesevent sequencesnaturalnesssecurity policies
0
0 comments X

The pith

Helion models patterns in user-created smart home routines to generate realistic event sequences for security research.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Helion as a framework that extracts regularities from user-driven home automation routines to produce natural sequences of events. It begins with the observation that these routines display repeatable semantic patterns, which the system models to create scenarios that reflect actual end-user behavior. Evaluation on a corpus of over 30,000 events shows that the generated sequences are viewed as reasonable by external evaluators. The scenarios then support the creation of home security and safety policies. This approach reduces the effort required compared with methods limited to analyzing IoT apps alone.

Core claim

Helion identifies the regularities present in user-driven home automation routines and models their inherent semantic patterns, or naturalness, to generate valid sequences of events that could realistically occur in an end-user's home. A corpus built from 273 routines collected from 40 users empirically supports the naturalness hypothesis. The resulting scenarios prove useful for designing 17 home security and safety policies while requiring significantly less effort than existing app-bounded approaches.

What carries the argument

Modeling of semantic patterns in user-driven home automation event sequences to generate natural scenarios.

Load-bearing premise

Smart home event sequences created by users contain consistent semantic patterns that can be modeled to produce new valid sequences.

What would settle it

A controlled study in which end-users rate a majority of Helion-generated sequences as unrealistic or impossible in their own homes.

Figures

Figures reproduced from arXiv: 1907.00124 by Adwait Nadkarni, Denys Poshyvanyk, Kaushal Kafle, Kevin Moran, Ruhao Tang, Sunil Manandhar.

Figure 1
Figure 1. Figure 1: shows an alternate approach that leverages scenarios, which may make Alice’s task significantly easier. Imagine (mis)use cases Inspect scenarios home automation scenarios Manually specified policies Semi-automatically specified policies [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An overview of the Hϵlion framework, which models home automation sequences to construct natural scenarios. Stakeholders use tools that analyze or execute scenarios to obtain actionable outcomes. trained model’s perplexity (or its log-transformed version, cross￾entropy) on unseen data. These are standard metrics used to test the viability of statistical language modeling for modeling any corpora. A trained… view at source ↗
Figure 3
Figure 3. Figure 3: Cross-entropy of the n-gram model over the HOME, Guten￾berg, and C# corpora, as well as the C# corpus without syntactic tokens. 6 Evaluating the Naturalness of HOME (RQ1) We test our naturalness hypothesis by comparing the cross-entropy of real user-driven home automation sequences with that of a popular natural language and software corpora. Recall that cross-entropy is a measure of how perplexed a model … view at source ↗
Figure 4
Figure 4. Figure 4: Device Selection Screen [25] SmartThings. Accessed December 2018. Yale Assure Lock with Bluetooth (Zig￾bee). https://www.smartthings.com/products/yale-assure-lock-with-bluetooth￾zigbee. [26] SmartThings. Accessed February 2019. SmartThings Classic App. https://play. google.com/store/apps/details?id=com.smartthings.android. [27] SmartThings. Accessed February 2019. SmartThings Web IDE. https://graph.api. sm… view at source ↗
Figure 6
Figure 6. Figure 6: Routine Creation [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Execution Indicator: Time of the week [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Importance of routines to the users [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sources of ideas for routines [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Setup preferred by the users to create routines C Policies generated using Hϵlion [PITH_FULL_IMAGE:figures/full_fig_p015_12.png] view at source ↗
Figure 16
Figure 16. Figure 16: Screen to provide additional feedback [PITH_FULL_IMAGE:figures/full_fig_p015_16.png] view at source ↗
read the original abstract

Security researchers have recently discovered significant security and safety issues related to home automation and developed approaches to address them. Such approaches often face design and evaluation challenges which arise from their restricted perspective of home automation that is bounded by the IoT apps they analyze. The challenges of past work can be overcome by relying on a deeper understanding of realistic home automation usage. More specifically, the availability of natural home automation scenarios, i.e., sequences of home automation events that may realistically occur in an end-user's home, could help security researchers design better security/safety systems. This paper presents Helion, a framework for building a natural perspective of home automation. Helion identifies the regularities in user-driven home automation, i.e., from user-driven routines that are increasingly being created by users through intuitive platform UIs. Our intuition for designing Helion is that smart home event sequences created by users exhibit an inherent set of semantic patterns, or naturalness that can be modeled and used to generate valid and useful scenarios. To evaluate our approach, we first empirically demonstrate that this naturalness hypothesis holds, with a corpus of 30,518 home automation events, constructed from 273 routines collected from 40 users. We then demonstrate that the scenarios generated by Helion are reasonable and valid from an end-user perspective, through an evaluation with 16 external evaluators. We further show the usefulness of Helion's scenarios by generating 17 home security/safety policies with significantly less effort than existing approaches. We conclude by discussing key takeaways and future research challenges enabled by Helion's natural perspective of home automation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper presents Helion, a framework for generating natural home automation scenarios by modeling semantic patterns in user-created routines. It claims to empirically support the naturalness hypothesis via a corpus of 30,518 events from 273 routines collected from 40 users, validate generated scenarios as reasonable via feedback from 16 external evaluators, and demonstrate usefulness by producing 17 home security/safety policies with significantly less effort than prior approaches.

Significance. If the modeling and validation hold, the work is significant for IoT security research by shifting from app-bounded analysis to realistic user-driven scenarios, potentially improving design and evaluation of security/safety mechanisms. Strengths include the scale of the user corpus and the external evaluator assessment; the policy generation example provides a concrete downstream application.

major comments (3)
  1. [§4] §4 (Corpus construction): The paper provides insufficient detail on routine collection from the 40 users, event extraction process, and any filtering steps applied to reach 30,518 events; without this, it is not possible to assess selection bias or confirm that the corpus rigorously tests the naturalness hypothesis.
  2. [§5.1] §5.1 (Evaluator study): The validity assessment lacks description of the survey instrument, exact rating criteria for 'reasonable and valid,' inter-rater reliability metrics, and how the 16 evaluators were selected; these omissions are load-bearing for the central claim that Helion scenarios are reasonable from an end-user perspective.
  3. [§6] §6 (Policy generation): The claim of 'significantly less effort' than existing approaches requires a quantitative baseline comparison (e.g., time or steps for manual policy creation); the current example alone does not substantiate the usefulness assertion.
minor comments (2)
  1. [Abstract and §4] The abstract states the corpus was 'constructed from 273 routines' but the methods section should explicitly state whether duplicate events across routines were deduplicated and how temporal ordering was preserved.
  2. [§3] Notation for event sequences and semantic patterns should be formalized earlier (e.g., in §3) to improve readability of the modeling description.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each major comment below and will revise the manuscript to provide the requested details and clarifications.

read point-by-point responses

  1. Referee: [§4] §4 (Corpus construction): The paper provides insufficient detail on routine collection from the 40 users, event extraction process, and any filtering steps applied to reach 30,518 events; without this, it is not possible to assess selection bias or confirm that the corpus rigorously tests the naturalness hypothesis.

    Authors: We agree that additional details are needed to allow assessment of selection bias. In the revised manuscript, we will expand §4 with descriptions of the routine collection method from the 40 users, the event extraction process from the 273 routines, the filtering steps applied to reach 30,518 events, and an explicit discussion of potential biases and mitigation steps. revision: yes

  2. Referee: [§5.1] §5.1 (Evaluator study): The validity assessment lacks description of the survey instrument, exact rating criteria for 'reasonable and valid,' inter-rater reliability metrics, and how the 16 evaluators were selected; these omissions are load-bearing for the central claim that Helion scenarios are reasonable from an end-user perspective.

    Authors: We acknowledge that more transparency is required for the evaluator study. The revised version will include a full description of the survey instrument, the exact rating criteria, inter-rater reliability metrics, and details on evaluator selection and recruitment to better support the validity claims. revision: yes

  3. Referee: [§6] §6 (Policy generation): The claim of 'significantly less effort' than existing approaches requires a quantitative baseline comparison (e.g., time or steps for manual policy creation); the current example alone does not substantiate the usefulness assertion.

    Authors: We agree that the usefulness claim would be strengthened by a quantitative baseline. In the revision, we will add to §6 a quantitative comparison, such as estimated time or steps required for manual policy creation versus using Helion, to substantiate the 'significantly less effort' assertion. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper grounds its naturalness hypothesis in an external corpus of 30,518 events from 273 routines collected from 40 users, then validates generated scenarios via 16 independent external evaluators and demonstrates policy generation separately. No derivation step reduces by construction to fitted parameters, self-definitions, or load-bearing self-citations; the modeling and evaluation remain independent of the target claims.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the domain assumption that user routines contain modelable semantic patterns; no free parameters or invented entities are explicitly described in the abstract.

axioms (1)
  • domain assumption Smart home event sequences created by users exhibit an inherent set of semantic patterns that can be modeled to generate valid scenarios.
    This is explicitly stated as the core intuition for designing Helion in the abstract.
invented entities (1)
  • Helion framework no independent evidence
    purpose: To identify regularities in user-driven home automation and generate natural scenarios.
    The proposed system itself; no independent evidence provided beyond the described evaluations.

pith-pipeline@v0.9.0 · 5835 in / 1221 out tokens · 30312 ms · 2026-05-25T13:23:09.304235+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 47 canonical work pages

  1. [1]

    We thoroughly describe this methodology in Sec

    Collecting routines from users: To collect routines from users, we use a survey methodology that is conceptually similar to prior work [35]. We thoroughly describe this methodology in Sec. 4.1, with additional survey artifacts provided in Appendix A. At the end of this survey process, the raw data collected from users consists of two components: (i) routi...

    work page

  2. [2]

    If the door lock is locked

    Representing smart home events as tokens : In the context of this paper, the tokens are home automation events parsed from struc- tured natural language descriptions of user-driven routines. A home automation event can denote a change in the state of a device (e.g., lock the door) or the home ( e.g., the user is away). To model the varying attributes of h...

    work page 2017

  3. [3]

    Introduction to Execution Indicators: We propose a novel ab- straction for users to stipulate the approximate order in which rou- tines may execute, i.e., routine-specific execution indicators. That is, we consider the possibility that end-users have some intuition re- garding when certain routines execute, based on when certain device or environmental ev...

    work page

  4. [4]

    abnormal

    Specifying Execution Indicators and Scheduling Routines: Execution indicators constitute the time and frequency of the po- tential execution of a routine. As users may not be able to specify precise values, we collect such indicators by allowing users to pick broad ranges of values organized into three types:(1) the time-range indicator (e.g., early morni...

    work page 2017

  5. [5]

    To enable this step, we provided the participants with a broad device list consisting of 70 unique types of devices available in the market

    Selecting devices: First, participants selected devices that they could envision (or already have) in their smart home. To enable this step, we provided the participants with a broad device list consisting of 70 unique types of devices available in the market. We constructed this list using resources such as websites and mobile apps of all the device part...

    work page

  6. [6]

    We asked the participants to provide triggers and actions in a plain English text to allow them to express any func- tionality desired

    Creating routines: After selecting devices, the participants were given a short tutorial on routines, and asked to create one or more routines using the devices that they had previously selected, along with general smart home variables such as the user being home/away, temperature, and time. We asked the participants to provide triggers and actions in a p...

    work page

  7. [7]

    not sure

    Specifying Execution Indicators: After creating routines, partic- ipants specified the time-range, day-range and frequency indicators for the routines they created, shown in Fig. 7, 8 and 9 respectively. Participants could select from predetermined ranges, as well as indi- cate “anytime” for routines that could occur at any time (i.e., with 7 Conference’1...

    work page 2017

  8. [8]

    That is, given a scenario, this module provides a snapshot for each event, which shows the holistic state of the home on the event’s execution

    Snapshot Module: This module tracks the evolution of states of individual devices and the home, as events are executed in the home (e.g., the “locked” state of the door lock, the home/away mode). That is, given a scenario, this module provides a snapshot for each event, which shows the holistic state of the home on the event’s execution

    work page

  9. [9]

    It’s very convenient. The advan- tage of having the sequences is that they set up a likely or unlikely scenarios without me having to invent it

    Execution Engine: To allow the dynamic execution of the scenar- ios predicted by Hϵlion, we built an execution engine on top of the SmartThings platform. This engine can execute scenarios on real and virtual devices. Our current setup has more than 15 real devices, and can provision an arbitrary number of configurable virtual devices. 8.1 Helping Research...

    work page 2017

  10. [10]

    Accessed December 2018

    Bo-June Hsu. Accessed December 2018. MIT Language Modeling Toolkit. https: //github.com/mitlm/mitlm

    work page 2018

  11. [11]

    Littman, and Blase Ur

    Will Brackenbury, Abhimanyu Deora, Jillian Ritchey, Jason Vallee, Weijia He, Guan Wang, Michael L. Littman, and Blase Ur. 2019. How Users Interpret Bugs in Trigger-Action Programming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, New York, NY , USA, Article 552, 12 pages. https://doi.org/10.1145/3290605.3300782

    work page doi:10.1145/3290605.3300782 2019

  12. [12]

    Z Berkay Celik, Leonardo Babun, Amit Kumar Sikder, Hidayet Aksu, Gang Tan, Patrick McDaniel, and A Selcuk Uluagac. 2018. Sensitive Information Tracking in Commodity IoT. In Proceedings of the 27th USENIX Security Symposium (USENIX)

    work page 2018

  13. [13]

    Berkay Celik, Patrick McDaniel, and Gang Tan

    Z. Berkay Celik, Patrick McDaniel, and Gang Tan. 2018. Soteria: Automated IoT Safety and Security Analysis. In 2018 USENIX Annual Technical Conference (USENIX ATC). 147–158

    work page 2018

  14. [14]

    Berkay Celik, Gang Tan, and Patrick McDaniel

    Z. Berkay Celik, Gang Tan, and Patrick McDaniel. 2019. IoTGuard: Dynamic Enforcement of Security and Safety Policy in Commodity IoT. In 2019 NDSS Symposium. To appear

    work page 2019

  15. [15]

    Stanley F Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language 13, 4 (1999)

    work page 1999

  16. [16]

    Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. 2019. Empowering End Users in Debugging Trigger-Action Rules. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, New York, NY , USA, Article 388, 13 pages. https://doi.org/10.1145/3290605.3300618

    work page doi:10.1145/3290605.3300618 2019

  17. [17]

    Wenbo Ding and Hongxin Hu. 2018. On the Safety of IoT Device Physical Interaction Control. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS). 832–846

    work page 2018

  18. [18]

    Earlence Fernandes, Jaeyeon Jung, and Atul Prakash. 2016. Security analysis of emerging smart home applications. In Security and Privacy (SP), 2016 IEEE Symposium on. 636–654

    work page 2016

  19. [19]

    Accessed May 2019

    Heather Pickstock. Accessed May 2019. Fire ripped through Bristol house because of cat’s electrical blanket. https://www.bristolpost.co.uk/news/bristol-news/fire- ripped-through-bristol-house-2336875

    work page 2019

  20. [20]

    Hindle, E.T

    A. Hindle, E.T. Barr, Z. Su, M. Gabel, and P. Devanbu. 2012. On the Naturalness of Software. In International Conference on Software Engineering (ICSE’12). 837–847

    work page 2012

  21. [21]

    Yunhan Jack Jia, Qi Alfred Chen, Shiqi Wang, Amir Rahmati, Earlence Fernandes, Z Morley Mao, Atul Prakash, and Shanghai JiaoTong Unviersity. 2017. ContexIoT: Towards providing contextual integrity to appified IoT platforms. InProceedings of the 2017 Network and Distributed System Security Symposium (NDSS)

    work page 2017

  22. [22]

    Kaushal Kafle, Kevin Moran, Sunil Manandhar, Adwait Nadkarni, and Denys Poshyvanyk. 2019. A Study of Data Store-based Home Automation. In Proceed- ings of the 9th ACM Conference on Data and Application Security and Privacy (CODASPY)

    work page 2019

  23. [23]

    Mario Linares-Vasquez, Martin White, Carlos Bernal-Cardenas, Kevin Moran, and Denys Poshyvanyk. 2015. Mining Android App Usages for Generating Actionable GUI-based Execution Scenarios. In Proceedings of the 12th Working Conference on Mining Software Repositories. 111–122

    work page 2015

  24. [24]

    Manning and Hinrich Schütze

    Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge, Massachusetts

    work page 1999

  25. [25]

    Accessed May 2019

    Mia Sims. Accessed May 2019. Electric blanket blamed for Las Vegas house fire. https://www.reviewjournal.com/local/local-las-vegas/electric-blanket- blamed-for-las-vegas-house-fire-1567476/

    work page 2019

  26. [26]

    Accessed May 2019

    Nathalie Sturgeon. Accessed May 2019. Fire that left Syrian family homeless started with blanket near baseboard heater. https://www.cbc.ca/news/canada/new- brunswick/fire-cause-syrian-family-1.5033662

    work page 2019

  27. [27]

    Accessed Feb 2019

    Nest Labs. Accessed Feb 2019. Meet the Nest app. https://nest.com/app/

    work page 2019

  28. [28]

    Accessed June 2018

    Nest Labs. Accessed June 2018. Nest Developers. https://developers.nest.com///

    work page 2018

  29. [29]

    Accessed June 2018

    Nest Labs. Accessed June 2018. Works with Nest. https://nest.com/works-with- nest//

    work page 2018

  30. [30]

    Krishnamurthy, Ed- ward J

    Dang Tu Nguyen, Chengyu Song, Zhiyun Qian, Srikanth V . Krishnamurthy, Ed- ward J. M. Colbert, and Patrick McDaniel. 2018. IotSan: Fortifying the Safety of IoT Systems. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (CoNEXT). 191–203

    work page 2018

  31. [31]

    Mitali Palekar, Earlence Fernandes, and Franziska Roesner. 2019. Analysis of the Susceptibility of Smart Home Programming Interfaces to End User Error. InIEEE Workshop on the Internet of Safe Things (SafeThings) (SafeThings’19). ACM, New York, NY , USA

    work page 2019

  32. [32]

    Musfiqur Rahman, Dharani Palani, and Peter Rigby. 2019. Natural Software Revisted. In Proceedings of the 41st International Conference on Software Engi- neering Companion (ICSE ’19). IEEE Press, Montreal, QC Canada, to appear

    work page 2019

  33. [33]

    Samsung. 2018. Samsung SmartThings SmartApp Public Repository. https: //github.com/SmartThingsCommunity/SmartThingsPublic. Figure 4: Device Selection Screen

    work page 2018

  34. [34]

    Accessed December 2018

    SmartThings. Accessed December 2018. Yale Assure Lock with Bluetooth (Zig- bee). https://www.smartthings.com/products/yale-assure-lock-with-bluetooth- zigbee

    work page 2018

  35. [35]

    Accessed February 2019

    SmartThings. Accessed February 2019. SmartThings Classic App. https://play. google.com/store/apps/details?id=com.smartthings.android

    work page 2019

  36. [36]

    Accessed February 2019

    SmartThings. Accessed February 2019. SmartThings Web IDE. https://graph.api. smartthings.com

    work page 2019

  37. [37]

    Accessed December 2018

    SmartThings Community. Accessed December 2018. Execution Times Increasing. https://community.smartthings.com/t/execution-times-increasing/19979

    work page 2018

  38. [38]

    Accessed December 2018

    SmartThings Community. Accessed December 2018. Help! Old SmartApp au- tomation still running. https://community.smartthings.com/t/help-old-smartapp- automation-still-running/70213

    work page 2018

  39. [39]

    Accessed June 2018

    Smartthings Developers. Accessed June 2018. Documentation. http://developer. smartthings.com/

    work page 2018

  40. [40]

    Accessed June 2018

    SmartThings Support. Accessed June 2018. Routines in the SmartThings Classic app. https://support.smartthings.com/hc/en-us/articles/205380034-Routines-in- the-SmartThings-Classic-app

    work page arXiv 2018

  41. [41]

    Accessed May 2019

    Statista. Accessed May 2019. Forecast market size of the global smart home market from 2016 to 2022 (in billion U.S. dollars). https://www.statista.com/ statistics/682204/global-smart-home-market-size/

    work page 2019

  42. [42]

    Milijana Surbatovich, Jassim Aljuraidan, Lujo Bauer, Anupam Das, and Limin Jia. 2017. Some Recipes Can Do More Than Spoil Your Appetite: Analyzing the Security and Privacy Risks of IFTTT Recipes. In Proceedings of the 26th International Conference on World Wide Web. 1501–1510

    work page 2017

  43. [43]

    Yuan Tian, Nan Zhang, Yueh-Hsun Lin, XiaoFeng Wang, Blase Ur, XianZheng Guo, and Patrick Tague. 2017. SmartAuth: User-Centered Authorization for the Internet of Things. In Proceedings of the 26th USENIX Security Symposium

    work page 2017

  44. [44]

    Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and Michael L Littman. 2014. Practical trigger-action programming in the smart home. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 803–812

    work page 2014

  45. [45]

    Qi Wang, Wajih Ul Hassan, Adam Bates, and Carl Gunter. 2018. Fear and Logging in the Internet of Things. In Network and Distributed Systems Symposium

    work page 2018

  46. [46]

    Accessed June 2018

    Yeti. Accessed June 2018. Yeti - Simplify the control of your smart home. https: //getyeti.co/

    work page 2018

  47. [47]

    Accessed June 2018

    Yonomi. Accessed June 2018. Yonomi app – Yonomi. https://www.yonomi.co. A Additional Survey Questions Aside from collecting routines and execution indicators, we asked users additional questions during the survey, illustrated in Figures 12, 11, 10, and 13. B Survey Instrument for the Routine Comparison and Sequence Generation Studies This section provides...

    work page 2018