Pith. sign in

REVIEW 3 major objections 4 minor 56 references

Passively watching everyday browsing can surface more useful web automations than people invent for themselves.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 11:00 UTC pith:NHKPP25E

load-bearing objection Working ambient-discovery system that surfaces far more automatable browser patterns than users invent, with honest small-N evidence and typical HCI scope limits. the 3 major comments →

arxiv 2607.10531 v1 pith:NHKPP25E submitted 2026-07-12 cs.HC cs.AIcs.SE

Motif: Discovering and Automating Personal Web Workflows

Shaokang Jiang , Daye Nam This is my paper
classification cs.HC cs.AIcs.SE
keywords ambient automation discoveryend-user programmingprogramming by demonstrationweb automationLLM code generationusage pattern miningTampermonkeyinitiation gap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most end-user automation tools still assume you already know which of your habits are worth programming. Motif drops that assumption: it quietly logs ordinary browser activity, uses an LLM to mine recurring multi-step patterns, designs simple program blueprints, and only generates a local Tampermonkey script after the user reviews and confirms. In a multi-day study with eight people, the system found far more automatable patterns than participants could name on their own; most of those patterns matched real routines, and a majority of the generated scripts worked and were kept. The paper argues that grounding code generation in observed behavior, rather than pure natural-language prompts, closes the initiation gap and yields personalized, cheap, deterministic programs instead of repeated LLM-agent calls.

Core claim

The paper establishes ambient automation discovery as a workable end-user programming paradigm: a passive pipeline that turns ordinary browser logs into ranked, reviewable, and deployable local programs discovers substantially more automatable personal workflows than users can self-identify, and most of those programs match routines and remain useful after deployment.

What carries the argument

Ambient automation discovery pipeline (Motif): continuous multi-modal browser logging, LLM action abstraction, pattern mining with occurrence threshold, UX-oriented program design (trigger / description / notification), ranking, user review, and generation of scoped Tampermonkey scripts.

Load-bearing premise

A few days of passive logging, a fixed frequency threshold, and one commercial LLM are enough to surface patterns that are both representative of longer-term habits and reliably programmable inside page-scoped browser scripts.

What would settle it

Repeat the multi-day protocol with a larger, more diverse cohort and longer logging windows; if users still cannot self-identify most Motif patterns yet the generated scripts fail to match routines or stop working outside the lab, the central usefulness claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • End-user programming tools can shift from “tell me what to build” to “here is what you already do that a program could own.”
  • Personalized, deterministic local scripts can replace repeated LLM-agent executions for recurring web chores, cutting cost and privacy exposure.
  • Observed behavioral traces supply richer specifications than unaided natural-language prompts, reducing the under-specification problem of vibe coding.
  • Users keep control: every automation is reviewed, editable in plain language, and installed only after confirmation.
  • Longer continuous observation should surface rarer but still high-value personal workflows that short self-reflection misses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same ambient-mining idea could be applied to desktop or mobile OS event streams once local multimodal models are strong enough, not only Chrome.
  • Combining Motif-style pattern discovery with hybrid program-plus-agent execution might cover complex DOM sites that pure Tampermonkey scripts cannot reach.
  • If users systematically prefer regenerating designs over editing them, future interfaces may need a still higher abstraction layer that translates intent into program blueprints without exposing technical descriptions.
  • Workplace deployments would need stronger audit and consent layers; the same logs that help individuals could become surveillance instruments if the control model is inverted.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. Motif proposes ambient automation discovery: a Chrome extension plus native host that passively logs UI events, screenshots, and network context, abstracts them with an LLM into actions, mines recurring programmable patterns, designs user-facing triggers/descriptions/notifications, ranks candidates, and (after user confirmation) emits Tampermonkey scripts. The central claim is that this closes the initiation gap of PBD and vibe coding. In a multi-day study (avg 5.5 days) with eight student participants, Motif surfaced ~22 patterns per person (threshold ≥5 occurrences); of 40 participant-selected patterns, 85% matched routines and 60% produced working programs, while self-reflection yielded only ~1.3 ideas per person and vibe coding succeeded for only two participants. Follow-up surveys indicated continued use of many scripts.

Significance. If the result holds, the paper offers a useful reframing of end-user programming around discovery rather than specification, with a concrete pipeline that produces local, deterministic, reusable scripts instead of per-run LLM agents. Strengths include the multi-day in-the-wild logging design, explicit comparison to self-reflection and vibe coding, candid threats-to-validity discussion (short window, self-selection of five patterns, residual privacy risks), and the practical choice of Tampermonkey as a deployable artifact. The work is a solid systems+HCI contribution that could influence personal automation tooling even if the absolute success rates are later revised.

major comments (3)
  1. Section 6.1 / Figure 3 and the strongest claim (avg 22 Motif patterns vs ~1.3 self-identified; 85% of 40 reviewed matched routines): the comparison is load-bearing for the initiation-gap claim, yet self-reflection was researcher-guided (Section 5.1) and Motif review was limited to five self-selected patterns of interest after only 4–8 days of logging (threshold ≥5). Section 6.2 further notes that Motif patterns are shorter/more scoped than users’ high-level ideas and that usefulness ranking was largely ignored. Without reporting match/usefulness rates over the full discovered set (or a random sample) and without an unguided self-reflection baseline, the claim that Motif discovers substantially more useful automations than users invent is only weakly supported for longer-term, non-student use.
  2. Section 4.9 and 6.3: program success (60% overall; 82% low-difficulty / 86% page-load vs 45% medium / 54% button) is measured only on the self-selected subset and is constrained by Tampermonkey’s page-level scope (explicit failures for browser pop-ups, complex/dynamic DOMs, multi-model Gemini workflows). The paper correctly lists these limits, but the abstract and RQ3 framing still present the 60% figure as evidence that Motif generates useful programs at scale. A clearer separation of in-scope success rate from out-of-scope failures, plus an analysis of how often high-ranked patterns fall outside Tampermonkey capability, is needed before the deployability claim can be taken as general.
  3. Section 5.3 and free parameters (occurrence threshold=5, 500-action batches, ≤60 s chunks, Gemini 3 Flash temperatures 0.1/0.2): the pipeline is entirely dependent on a single commercial LLM and ad-hoc thresholds with no ablation or sensitivity analysis. Given that pattern mining and program design are the core technical contributions, the manuscript should either (a) report how pattern count/quality and program success change under modest threshold or model variation, or (b) explicitly bound claims to this fixed configuration rather than presenting ambient discovery as robust.
minor comments (4)
  1. Table 1 and participant IDs: numbering jumps (P1, P3–P9); two dropouts are mentioned but not labeled, which makes cross-referencing awkward.
  2. Figure 1 / Figure 2 captions and UI labels contain typos ("Notification Stataus", "Refine", "α Controller") that should be cleaned for camera-ready.
  3. Section 4.8 privacy discussion is candid but still under-specifies residual risks (prompt injection into generated scripts, behavioral profiling); a short concrete mitigation roadmap would strengthen the paper without changing claims.
  4. Appendix prompts are valuable for reproducibility; consider also releasing the exact occurrence-threshold and ranking prompts as machine-readable artifacts.

Circularity Check

0 steps flagged

No circularity: empirical system + user-study paper whose claims rest on participant judgments and observed program execution, not on quantities defined by the same inputs used to generate them.

full rationale

Motif is an HCI systems paper whose load-bearing claims (Motif surfaces more automatable patterns than users self-identify; ~85% of reviewed patterns match routines; ~60% yield working Tampermonkey scripts that participants plan to keep) are evaluated against external participant self-reports, think-alouds, and post-deployment follow-up surveys after multi-day passive logging. The pipeline (action abstraction, LLM pattern mining with occurrence threshold ≥5, program design/ranking, generation) uses design choices and a commercial LLM, but these are not fitted parameters that are then re-presented as independent predictions of the same quantities. Self-citations ([19], [20]) appear only as background on vibe-coding costs and prompt engineering; they do not supply uniqueness theorems, ansätze, or uniqueness results that force the main empirical findings. There are no equations equating a derived quantity to a fitted input by construction, no renaming of known empirical patterns as novel unifications, and no self-citation chain that substitutes for the user-study evidence. The evaluation is therefore self-contained against its stated external benchmarks (participant agreement, working scripts, continued use).

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on a small set of design thresholds, an LLM capability assumption, and the premise that short personal logs contain programmable, user-valued patterns. No physical constants or mathematical free parameters; free parameters are engineering knobs chosen by the authors.

free parameters (4)
  • occurrence_threshold = 5
    Patterns must appear ≥5 times before program design is offered (Section 4.9); directly controls which candidates reach users.
  • action_chunk_size = 60s / 8 screenshots / 3k tokens
    Raw events grouped by ≤60 s continuous activity, 15 s inactivity, or 8 screenshots; abstraction context capped at ~3 k tokens.
  • pattern_mining_batch = 500
    LLM pattern identification triggered every 500 new actions.
  • LLM_temperatures = 0.1 / 0.2
    0.1 for abstraction/mining, 0.2 for program design; chosen for stability vs. creativity.
axioms (4)
  • domain assumption A commercial multimodal LLM (Gemini 3 Flash) can reliably abstract noisy UI+screenshot+network logs into high-level actions and identify programmable sequences without excessive hallucination.
    Core of Sections 4.2–4.3; no independent verification beyond the study outcomes.
  • domain assumption Page-scoped Tampermonkey userscripts are a sufficient and preferable target representation for the majority of personal web automations (vs. full browser agents or OS-level automation).
    Stated in DG4 and Section 4.7; limitations section acknowledges out-of-scope cases (browser chrome, canvas, complex SPAs).
  • domain assumption Users will review, refine, and adopt ambiently discovered programs when given natural-language designs and a simple UI.
    DG3 and evaluation design; partially supported by the 60% success and follow-up retention numbers.
  • domain assumption Standard sequence-mining and PBD initiation assumptions (user must first recognize the task) create a real initiation gap that ambient observation can close.
    Literature framing in Sections 2.2–2.3; used to motivate the system.
invented entities (2)
  • ambient automation discovery paradigm no independent evidence
    purpose: Frame the shift from user-initiated demonstration/specification to passive observation + recommendation + program generation.
    Introduced as the paper’s conceptual contribution (Section 3); evaluated via Motif.
  • Motif system (Chrome extension + Rust native host + LLM pipeline) no independent evidence
    purpose: Concrete realization that collects multi-modal browser logs, mines patterns, designs programs, and emits Tampermonkey scripts.
    The artifact under study; no external independent implementation exists yet.

pith-pipeline@v1.1.0-grok45 · 26093 in / 3048 out tokens · 36707 ms · 2026-07-14T11:00:50.475049+00:00 · methodology

0 comments
read the original abstract

Recent advances in LLMs and existing work on programming by demonstration have made it possible for end users to create automations by explicitly demonstrating their behavior to LLMs. However, these approaches rely on the assumption that users know what to automate and what is capable of being automated. Additionally, automation via LLM agents is often expensive compared with programs. We introduce Motif, a system that passively observes everyday browser activity to discover recurring interaction patterns that are programmable, makes recommendations to users whenever a pattern is discovered and generate a program to install after user confirmation. Users can review, and refine the program using natural language. We evaluated Motif in a multi-day study, comparing its ambient discoveries against automations users attempted to build via ``vibe coding.'' With eight participants, Motif discovered more automatable patterns than users recognized. Most of them matched participants' routines and were useful. Follow-up surveys showed most would continue using Motif-generated programs.

Figures

Figures reproduced from arXiv: 2607.10531 by Daye Nam, Shaokang Jiang.

Figure 1
Figure 1. Figure 1: Overview of Motif. Motif supports ambient automation discovery by streaming a pipeline of passive data collection, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: User interface. Right top (𝛼 − 𝛽): controls for managing system monitoring and data collection. Right bottom (1-4): menu for reviewing generated programs. Left (A-H): an interface showing discovered programs once the user clicks on a pattern. 4.3 Pattern Mining Motif then uses LLMs to identify all programmable sequences as patterns. The LLM is instructed to focus on automatable sequences, find and merge ne… view at source ↗
Figure 3
Figure 3. Figure 3: Motif identified more automatable patterns (avg [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Post-study survey results. Most participants re [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Followup survey results. extreme automation by vibe coding could pose a malicious program by essentially preventing the user from accessing the original page. 6.5 RQ5: Long-term usefulness We sent a follow-up survey 2–4 days after participants completed the study session, asking about their continued use of the worked programs generated by the Motif. Six participants (P1, P3, P4, P5, P8, P9) responded to t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 7 canonical work pages

  1. [1]

    Simone Agostinelli, Marco Lupia, Andrea Marrella, and Massimo Mecella. 2022. Reactive synthesis of software robots in RPA from user interface logs.Computers in Industry142 (2022), 103721

    2022

  2. [2]

    Siraaj Akhtar, Saad Khan, and Simon Parkinson. 2025. LLM-based event log analysis techniques: A survey.arXiv preprint arXiv:2502.00677(2025)

    Pith/arXiv arXiv 2025

  3. [3]

    Saleema Amershi, Jalal Mahmud, Jeffrey Nichols, Tessa Lau, and German Attana- sio Ruiz. 2013. LiveAction: Automating Web Task Model Generation.ACM Trans. Interact. Intell. Syst.3, 3, Article 14 (Oct. 2013), 23 pages. doi:10.1145/2533670. 2533672

    work page doi:10.1145/2533670 2013

  4. [4]

    Fraser Anderson, Tovi Grossman, and George Fitzmaurice. 2017. Trigger-Action- Circuits: Leveraging Generative Design to Enable Novices to Design and Build Circuitry. InProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology(Québec City, QC, Canada)(UIST ’17). Association Conference’17, July 2017, Washington, DC, USA Shaokang...

    doi:10.1145/3126594 2017

  5. [5]

    Deniz Arsan, Ali Zaidi, Aravind Sagar, and Ranjitha Kumar. 2021. App-based task shortcuts for virtual assistants. InThe 34th Annual ACM Symposium on User Interface Software and Technology. 1089–1099

    2021

  6. [6]

    Richard Atterer, Monika Wnuk, and Albrecht Schmidt. 2006. Knowing the user’s every move: user activity tracking for website usability evaluation and implicit interaction. InProceedings of the 15th International Conference on World Wide Web(Edinburgh, Scotland)(WWW ’06). Association for Computing Machinery, New York, NY, USA, 203–212. doi:10.1145/1135777.1135811

    work page doi:10.1145/1135777.1135811 2006

  7. [7]

    Aveni, Hila Mor, Armando Fox, and Björn Hartmann

    Timothy J. Aveni, Hila Mor, Armando Fox, and Björn Hartmann. 2025. Generative Trigger-Action Programming with Ply. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, Article 33, 17 pages. doi:10.1145/ 3746059.3747638

    arXiv 2025

  8. [8]

    Axiom AI Limited. 2025. Axiom.ai: Browser Automation for Humans and AI. https://axiom.ai/. Accessed: 2026-06-29

    2025

  9. [9]

    Alan F Blackwell. 2002. First steps in programming: A rationale for attention in- vestment models. InProceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments. IEEE, 2–10

    2002

  10. [10]

    Antonio Bosco, Adriano Augusto, Marlon Dumas, Marcello La Rosa, and Gian- carlo Fortino. 2019. Discovering automatable routines from user interaction logs. InInternational conference on business process management. Springer, 144–162

    2019

  11. [11]

    Chasins, Maria Mueller, and Rastislav Bodik

    Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scrap- ing Distributed Hierarchical Web Data. InProceedings of the 31st Annual ACM Symposium on User Interface Software and Technology(Berlin, Germany)(UIST ’18). Association for Computing Machinery, New York, NY, USA, 963–975. doi:10.1145/3242587.3242661

    doi:10.1145/3242587.3242661 2018

  12. [12]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176(2023)

    Pith/arXiv arXiv 2023

  13. [13]

    Daehyoun Choi, Hind R’bigui, and Chiwoon Cho. 2021. Candidate digital tasks selection methodology for automation with robotic process automation.Sustain- ability13, 16 (2021), 8980

    2021

  14. [14]

    Vincent J Hellendoorn, Sebastian Proksch, Harald C Gall, and Alberto Bacchelli

  15. [15]

    In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

    When code completion fails: A case study on real-world completions. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 960–970

    2019

  16. [16]

    Jaylin Herskovitz, Andi Xu, Rahaf Alharbi, and Anhong Guo. 2024. ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Program- ming. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–15

    2024

  17. [17]

    Marko Horvat. 2025. What is Vibe coding and when should you use it (or not)? Authorea Preprints(2025)

    2025

  18. [18]

    Forrest Huang, Gang Li, Tao Li, and Yang Li. 2024. Automatic macro mining from interaction traces at scale. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–16

    2024

  19. [19]

    Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, and Jeffrey P. Bigham. 2026. Mod- eling Distinct Human Interaction in Web Agents. arXiv:2602.17588 [cs.CL] https://arxiv.org/abs/2602.17588

    Pith/arXiv arXiv 2026

  20. [20]

    Shaokang Jiang and Michael Coblenz. 2024. An analysis of the costs and benefits of autocomplete in ides.Proceedings of the ACM on Software Engineering1, FSE (2024), 1284–1306

    2024

  21. [21]

    Shaokang Jiang and Daye Nam. 2025. Beyond the Prompt: An Empirical Study of Cursor Rules.arXiv preprint arXiv:2512.18925(2025)

    arXiv 2025

  22. [22]

    Martin Kleppmann, Adam Wiggins, Peter Van Hardenberg, and Mark Mc- Granaghan. 2019. Local-first software: you own your data, in spite of the cloud. InProceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software. 154–178

    2019

  23. [23]

    Amy J Ko, Brad A Myers, and Htet Htet Aung. 2004. Six learning barriers in end- user programming systems. In2004 IEEE Symposium on Visual Languages-Human Centric Computing. IEEE, 199–206

    2004

  24. [24]

    Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide.Data mining and knowledge discovery18, 1 (2009), 140–181

    2009

  25. [25]

    Tessa Lau. 2009. Why Programming by Demonstration Systems Fail: Lessons Learned for Usable AI.AI Magazine30, 4 (2009), 65–67

    2009

  26. [26]

    Van-Hoang Le and Hongyu Zhang. 2023. Log parsing with prompt-based few-shot learning. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2438–2449

    2023

  27. [27]

    Volodymyr Leno, Artem Polyvyanyy, Marlon Dumas, Marcello La Rosa, and Fabrizio Maria Maggi. 2021. Robotic Process Mining: Vision and Challenges: V. Leno et al.Business & Information Systems Engineering63, 3 (2021), 301–314

    2021

  28. [28]

    Haber, Tara Matthews, and Tessa Lau

    Gilly Leshed, Eben M. Haber, Tara Matthews, and Tessa Lau. 2008. CoScripter: automating & sharing how-to knowledge in the enterprise. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 1719–1728. doi:10.1145/1357054.1357323

    doi:10.1145/1357054.1357323 2008

  29. [29]

    Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A stage-based model of personal informatics systems. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Atlanta, Georgia, USA)(CHI ’10). Association for Computing Machinery, New York, NY, USA, 557–566. doi:10.1145/1753326.1753409

    doi:10.1145/1753326.1753409 2010

  30. [30]

    Jiawen Li, Zheng Ning, Yuan Tian, and Toby Jia-jun Li. 2025. ALLOY: Gen- erating Reusable Agent Workflows from User Demonstration.arXiv preprint arXiv:2510.10049(2025)

    arXiv 2025

  31. [31]

    Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. InProceedings of the 2017 CHI conference on human factors in computing systems. 6038–6049

    2017

  32. [32]

    Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019. Pumice: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations. InProceedings of the 32nd annual ACM symposium on user interface software and technology. 577–589

    2019

  33. [33]

    What It Wants Me To Say

    Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code- Generating Large Language Models. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). ACM, 1–31. do...

    doi:10.1145/3544548 2023

  34. [34]

    Yimeng Liu, Misha Sra, Jeevana Priya Inala, and Chenglong Wang. 2025. ReUseIt: Synthesizing Reusable AI Agent Workflows for Web Automation.arXiv preprint arXiv:2510.14308(2025)

    arXiv 2025

  35. [35]

    Wendy E Mackay. 1991. Triggers and barriers to customizing software. InProceed- ings of the SIGCHI conference on Human factors in computing systems. 153–160

    1991

  36. [36]

    Bamshad Mobasher, Robert Cooley, and Jaideep Srivastava. 2000. Automatic personalization based on Web usage mining.Commun. ACM43, 8 (Aug. 2000), 142–151. doi:10.1145/345124.345169

    work page doi:10.1145/345124.345169 2000

  37. [37]

    Surendran

    Nuria Oliver, Greg Smith, Chintan Thakkar, and Arun C. Surendran. 2006. SWISH: semantic analysis of window titles and switching history. InProceedings of the 11th International Conference on Intelligent User Interfaces(Sydney, Australia) (IUI ’06). Association for Computing Machinery, New York, NY, USA, 194–201. doi:10.1145/1111449.1111492

    work page doi:10.1145/1111449.1111492 2006

  38. [38]

    Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, and Jinwook Seo

  39. [39]

    In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25)

    Leveraging Multimodal LLM for Inspirational User Interface Search. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 579, 22 pages. doi:10.1145/3706598.3714213

    doi:10.1145/3706598.3714213 2025

  40. [40]

    Roy D Pea. 1987. User centered system design: new perspectives on human- computer interaction.Journal educational computing research3 (1987), 129–134

    1987

  41. [41]

    Pernice, K

    K. Pernice, K. Whitenton, J. Nielsen, and Nielsen Norman Group. 2014.How People Read on the Web: The Eyetracking Evidence. Nielsen Norman Group. https://books.google.com/books?id=nQ2-oQEACAAJ

    2014

  42. [42]

    Kevin Pu, Jim Yang, Angel Yuan, Minyi Ma, Rui Dong, Xinyu Wang, Yan Chen, and Tovi Grossman. 2023. DiLogics: Creating Web Automation Programs with Diverse Logics. InProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology(San Francisco, CA, USA)(UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 74, 15...

    arXiv 2023

  43. [43]

    Road to Ramen LLC. 2025. Browserflow: Web Scraping & Web Automation. https://browserflow.app/. Accessed: 2026-06-29

    2025

  44. [44]

    Dietterich

    Jianqiang Shen, Erin Fitzhenry, and Thomas G. Dietterich. 2009. Discovering frequent work procedures from resource connections. InProceedings of the 14th International Conference on Intelligent User Interfaces(Sanibel Island, Florida, USA) (IUI ’09). Association for Computing Machinery, New York, NY, USA, 277–286. doi:10.1145/1502650.1502690

    doi:10.1145/1502650.1502690 2009

  45. [45]

    Dietterich, and Jonathan L

    Jianqiang Shen, Lida Li, Thomas G. Dietterich, and Jonathan L. Herlocker. 2006. A hybrid learning system for recognizing user tasks from desktop activities and email messages. InProceedings of the 11th International Conference on Intelligent User Interfaces(Sydney, Australia)(IUI ’06). Association for Computing Machinery, New York, NY, USA, 86–92. doi:10....

    work page doi:10.1145/1111449.1111473 2006

  46. [46]

    Pradyumna Shome, Sashreek Krishnan, and Sauvik Das. 2025. Why Johnny Can’t Use Agents: Industry Aspirations vs. User Realities with AI Agent Software.arXiv preprint arXiv:2509.14528(2025)

    Pith/arXiv arXiv 2025

  47. [47]

    Rehan Syed, Suriadi Suriadi, Michael Adams, Wasana Bandara, Sander JJ Leemans, Chun Ouyang, Arthur HM Ter Hofstede, Inge Van De Weerd, Moe Thandar Wynn, and Hajo A Reijers. 2020. Robotic process automation: contemporary themes and challenges.Computers in industry115 (2020), 103162

    2020

  48. [48]

    A Aylin Tokuç and Tamer Dag. 2025. Predicting user purchases from clickstream data: A comparative analysis of clickstream data representations and machine learning models.IEEE Access13 (2025), 43796–43817

    2025

  49. [49]

    Kelsey Turbeville, Jennarong Muengtaweepongsa, Samuel Stevens, Jason Moss, Amy Pon, Kyra Lee, Charu Mehra, Jenny Gutierrez Villalobos, and Ranjitha Kumar. 2024. LLM-powered Multimodal Insight Summarization for UX Testing. InProceedings of the 26th International Conference on Multimodal Interaction. 4–11. Motif: Discovering and Automating Personal Web Work...

    2024

  50. [50]

    Blase Ur, Elyse McManus, Melwyn Pak Yong Ho, and Michael L. Littman. 2014. Practical trigger-action programming in the smart home. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Toronto, Ontario, Canada)(CHI ’14). Association for Computing Machinery, New York, NY, USA, 803–812. doi:10.1145/2556288.2557420

    work page doi:10.1145/2556288.2557420 2014

  51. [51]

    J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang

  52. [52]

    InProceedings of the 2023 CHI conference on human factors in computing systems

    Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. InProceedings of the 2023 CHI conference on human factors in computing systems. 1–21

    2023

  53. [53]

    action": Provide a concise, human-readable description of what the user did. -

    Lefan Zhang, Weijia He, Olivia Morkved, Valerie Zhao, Michael L. Littman, Shan Lu, and Blase Ur. 2020. Trace2TAP: Synthesizing Trigger-Action Programs from Traces of Behavior.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.4, 3, Article 104 (Sept. 2020), 26 pages. doi:10.1145/3411838 A Prompt used Abstracting actions You are an AI analyzing a user’s ...

    work page doi:10.1145/3411838 2020

  54. [54]

    Execution Toolkit: You decide the best technical ap- proach to achieve the program_description goal based on the provided traces. Choose the most reliable and ef- ficient strategy from the following viable options: - Net- work Requests: Consolidate manual clicks into background `fetch()òr `GM_xmlhttpRequest`calls. Safeguard: Only simu- late state-changing...

  55. [55]

    Planning First: BEFORE writing any code, you MUST output your thought process using two strict XML blocks: <plan> and <automation_steps>. - Inside <plan>: Analyze the traces and the scenario, evaluate the trade-offs of the strategies in your toolkit, and explicitly state which ap- proach is best for this specific task and why. - Inside <au- tomation_steps...

  56. [56]

    ${patternName}

    Format Requirements: You MUST output in exactly two parts. PART 1: The <plan> block, followed by the <au- tomation_steps> block. PART 2: Immediately following the </automation_steps> tag, output ONLY the raw JavaScript code starting with the standard // ==UserScript== header. Do not add any conversational text before or after the code. === BLUEPRINT (WHAT...