REVIEW 3 major objections 6 minor 137 references
IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis
T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that externalizing analytic intent into shared, editable rules and checking every AI prompt against those rules improves awareness, reflection, and early conflict detection in human-AI collaborative data analysis.
desk verdict Design contribution is real; the central awareness claim outruns the evidence—but this is a solid HCI systems paper worth reviewing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the shared rule: a structured template with a name, author, severity level (info/warn/error), a natural-language description, a trigger condition, and a grounding context field. The rule is designed as a boundary object interpretable by both humans and LLM agents, serving as a concrete artifact for negotiation—accepting, editing, or rejecting a rule becomes a coordination act that builds common ground. Three LLM-based agents carry the workflow: a rule-checking agent that matches prompt and notebook state against trigger conditions, a linting agent that proposes minimal prompt revisions, and a rule-proposal agent that derives more specific rules from triggered ones. Rules persist in the workspace across sessions and propagate in real time to all collaborators, allowing coordination norms to evolve.
What would settle it
Observe a real team using a notebook coordination layer for a full project cycle while a matched team uses a standard AI assistant. The central claim fails if the coordination team does not report higher awareness of collaborators' intent or earlier conflict detection, or if most proposed rules are rejected or deactivated within the first weeks as the rule board grows. A cheaper controlled check: with a fixed rule set, measure whether inter-coder agreement on which rules 'should fire' drops when the notebook context contains stale or contradictory cells.
Extended reading notes
Core claim
IntentLint's central discovery is that encoding analytic intent as shared, machine-readable rules creates a feedback loop between intent articulation and action. When a user submits a prompt, a rule-proposal agent infers the intent behind it and suggests a rule for review, while a rule-checking agent evaluates the prompt against all existing rules; if a conflict with a collaborator's prior goals or assumptions is found, a linting agent surfaces a message with concrete prompt revisions. The authors report that participants made substantially more prompt edits, rated their prompts as clearer and more precise, accepted or refined most proposed rules, and reported that potential problems were identified significantly earlier compared to a GitHub Copilot baseline. They also report strong agreement between the LLM-based rule-checker and human judgment, supporting the feasibility of natural-language trigger conditions as a coordination mechanism.
Load-bearing premise
The findings come from a simulation of collaboration—each participant worked alone on a notebook pre-populated with a teammate's supposed work—so the claim that the system improves real shared understanding rests on that handoff being a faithful stand-in for actual multi-user teamwork over time.
Editorial extensions
If this is right
- Analysts using a similar coordination layer will catch conflicts with collaborators' prior decisions before code is generated, reducing downstream rework and increasing confidence in AI-generated code.
- Prompt quality improves: study participants made more prompt edits and rated their prompts as clearer and more precise with IntentLint than with a standard AI assistant baseline.
- Teams can convert one-off analytical decisions into reusable coordination norms by accepting or refining proposed rules, growing the rule base over time without imposing documentation overhead.
- Structured rules complement rather than replace documentation, providing an actionable alternative to inline comments that may be incomplete, outdated, or AI-generated.
- The direction of effort shifts from editing code to articulating intent, which the paper interprets as supporting deeper analytical reasoning and cognitive offloading of routine validation tasks.
Reading between the lines
- One corollary the paper does not pursue: the same rule template could extend beyond notebooks to any channel where a team prompts an LLM, such as chat tools or agent configuration files, since rule-checking operates on prompt text plus a structured context snapshot rather than notebook-specific UI.
- The reported LLM–human rule-trigger agreement of 0.64–0.91 sets a ceiling: most false triggers and missed triggers likely trace to how well the model interprets natural-language trigger conditions against context, making model interpretation the likeliest lever for precision.
- Participants rejected rules they saw as one-time decisions or too strict for exploratory analysis, suggesting a testable prediction that coordination-layer value peaks at a moderate rule count and declines as the rule board accumulates narrow or redundant rules.
- If awareness gains persist outside the lab, a practical consequence is that teams could rely less on mandatory documentation and more on prompt-time checks, changing the effort required to onboard new members to inherited analyses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces IntentLint, a VSCode extension implementing a rule-based coordination layer for AI-assisted collaborative notebooks, with two mechanisms: analytic intent scaffolding (structured, editable rules capturing goals, assumptions, and rationales) and prompt-time linting (checking prompts against shared rules before code generation). It reports a formative study with five analysts, a multi-agent LLM implementation, and a within-subjects study with 16 analysts comparing IntentLint to a GitHub Copilot baseline on pre-populated notebooks. The paper's central claim is that IntentLint improves awareness of collaborators' intent and encourages reflection on analytic strategies.
Significance. If the result holds, this is a valuable design contribution: a concrete, implementable coordination layer for the increasingly common multi-human and multi-agent data analysis setting, with a useful rule taxonomy and design guidelines. The paper's strengths include a working proof-of-concept, a clear rule template, event-log collection, inter-coder reliability checking for rule firing, a thoughtful failure-case analysis, and a candid limitation section. The main obstacle is that the headline claim about collaborative awareness is supported only by self-report in a simulated handoff; this is a construct-validity gap rather than a statistical subtlety, and it needs to be addressed before the abstract's causal wording is justified.
major comments (3)
- [§5.1.4, §5.2.3 (Fig. 5, Q4); §6.4] The central claim that IntentLint "improves awareness of collaborators' intent" is not operationally tested. Each participant worked individually on a pre-populated notebook; the "collaborator" is a static artifact, and Q4 asks for a self-report ("I understood my collaborator's intent enough to safely build on their work") with no ground-truth measure of whether participants actually recovered the goals, assumptions, or constraints that the prior work's authors encoded. The authors' own §6.4 acknowledges that the design "cannot fully examine how persistent rules evolve, become stale, conflict, or support real collaboration over time," which is precisely the construct the abstract claims to improve. Please add an objective comprehension measure (e.g., asking participants to reconstruct the prior author's goals and assumptions and scoring against expert-authored ground truth) or revise the claim to "perceived awareness of represented intent" throughout the abstract and Section 5.2.3.
- [§5.2.3, Fig. 5] The quantitative support for the reflection and awareness claims rests on ten self-defined Likert items tested with Wilcoxon signed-rank tests without any correction for multiple comparisons, and only categorical p thresholds are reported. With n=16, uncorrected testing across ten items produces an inflated chance of at least one false positive. Please report exact p-values and effect sizes (e.g., matched rank-biserial correlation), apply a correction or clearly justify not doing so, and adjust the strength of the affected claims accordingly.
- [§5.2.2, Fig. 4; §5.2.5] The comparison of event counts (more prompt edits and fewer code edits under IntentLint) is used as behavioral evidence of a workflow shift, but the two conditions used different tasks and notebooks (Task 1 sale timing vs. Task 2 attrition), counterbalanced but not shown to be comparable in difficulty or prompt-edit propensity. No statistical test is reported for these counts. Please provide per-task event rates, test the difference, or explicitly present the counts as descriptive only and remove the causal "shift in effort" wording.
minor comments (6)
- [§5.2.2] The statement that participants accepted a majority of proposed rules (86 out of 157) "suggesting that the system's rule generation aligned well with collaboration concerns" treats acceptance of system proposals as evidence of value; acceptance could reflect acquiescence or perceived cost of rejecting. Please temper this inference or triangulate it with the interview data.
- [Fig. 5 caption] The caption contains a typo ("self-defned") and the questionnaire in Appendix A.2.3 lists 12 items while Figure 5 reports only Q1–Q10; please clarify why Q11 and Q12 are omitted.
- [Fig. 8 caption] The caption contains a typo ("Githug CoPilot" should be "GitHub Copilot").
- [§3.1.2] The subsection heading "Construction a Computational Notebook Technology Probe" should read "Constructing a Computational Notebook Technology Probe."
- [§5.2.2] The paper reports SUS scores "based on the UMUX-LITE" without explaining the conversion or why the UMUX-LITE items are labeled as SUS; please describe the scoring procedure.
- [Appendix A.4] Appendix A.4 appears to be empty in the provided manuscript; please populate it or remove the heading.
Circularity Check
No significant circularity: the awareness and reflection claims are evaluated by independent user measures, not derived from the system's own rule templates.
full rationale
This paper contains no mathematical derivation, so the usual circularity modes (fitted parameter renamed as prediction, uniqueness imported from authors, ansatz smuggled via citation) do not arise. The closest self-referential elements are the default rules in Table 2, which are constructed from the formative study and later assessed through participant acceptance of proposed rules (86 of 157, Section 5.2.2), and a few related-work citations to the authors' own prior papers ([20], [66], [74], [90]). Neither element is load-bearing: rule acceptance is an independent behavioral measure of user endorsement rather than a quantity computed from the rule-proposal mechanism itself, and the self-citations support background claims about notebooks and dashboard practice rather than the paper's central awareness and reflection conclusion. The central evidence is a within-subjects comparison of self-reported Likert responses and interaction logs between IntentLint and a GitHub Copilot baseline (Figures 4 and 5); these measures are not defined in terms of the system's rule templates or LLM agent outputs. The main weakness is construct validity rather than circularity: as the authors explicitly acknowledge in Section 6.4, participants worked individually on pre-populated notebooks, so the 'understood my collaborator's intent' item measures perceived understanding of a simulated handoff, not verified understanding of a real collaborator. This limitation is disclosed and affects generalization, not the logical independence of the evaluation. No circular step can be quoted from the paper, so the score is 0.
Assumptions & free parameters
free parameters (1)
- Default rule set (Table 2)
assumptions (5)
- domain assumption Shared rules, as boundary objects, externalize analytic intent effectively for both humans and AI agents.
- domain assumption The LLM-based rule-checking agent correctly interprets natural-language trigger conditions with sufficient reliability.
- domain assumption The 15 challenges synthesized from 25 prior papers are representative of collaborative data analysis breakdowns.
- domain assumption Self-reported Likert responses measure genuine changes in awareness and reflection rather than demand characteristics.
- domain assumption Working on a pre-populated notebook alone approximates asynchronous collaboration with absent teammates.
Cite this review
Pith. "Pith review of IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis." pith.science (2026). https://pith.science/paper/KDANTYNM
@misc{pith2026260804331,
author = {Pith},
title = {Pith review of: IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/KDANTYNM}},
note = {Machine review of arXiv:2608.04331}
}
read the original abstract
In human-AI collaborative data analysis, as analyses rapidly evolve, the artifacts meant to capture shared understanding often become incomplete or difficult to interpret, leading to undocumented assumptions, cross-user misaligned intent, context-poor prompts, and unwanted agent behaviors. To address these challenges, we introduce a rule-based coordination layer with two interaction mechanisms, intent scaffolding and prompt-time linting, that make analytic intent explicit and actionable during human-AI collaborative data analysis. We implement them in IntentLint, a proof-of-concept system that infers analytic intent from shared notebooks, represents it as structured, editable rules, and checks users' prompts against shared rules. IntentLint helps analysts externalize and refine their intent and proactively checks prompts for potential conflicts. A study with 16 data analysts shows that IntentLint improves awareness of collaborators' intent and encourages reflection on analytic strategies, and provides design implications for supporting more aligned and transparent human-AI collaborative data analysis.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
[n. d.]. Ames Housing Dataset — kaggle.com. https://www.kaggle.com/datasets/ shashanknecrothapa/ames-housing-dataset/data. [Accessed 25-03-2026]
2026
-
[2]
Sara Alspaugh, Nava Zokaei, Andrea Liu, Cindy Jin, and Marti A Hearst. 2018. Futzing and moseying: Interviews with professional data analysts on exploration practices.IEEE transactions on visualization and computer graphics25, 1 (2018), 22–31
2018
-
[3]
Leilani Battle and Jeffrey Heer. 2019. Characterizing exploratory visual analysis: A literature review and evaluation of analytic provenance in tableau. InComputer graphics forum, Vol. 38. Wiley Online Library, 145–159
2019
-
[4]
Mary Beth Kery and Brad A. Myers. 2017. Exploring exploratory programming. In2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 25–29. doi:10.1109/VLHCC.2017.8103446
arXiv 2017
-
[5]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psy- chology.Qualitative Research in Psychology3, 2 (2006), 77–101. doi:10.1191/ 1478088706qp063oa
2006
-
[6]
Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health11, 4 (2019), 589–597
2019
-
[7]
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted 11 UIST ’26, November 02–05, 2026, Detroit, MI, USA Feng et al. decision-making.Proceedings of the ACM on Human-computer Interaction5, CSCW1 (2021), 1–21
2021
-
[8]
Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjana- sith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E Hassan, et al. 2025. Agent READMEs: An Empirical Study of Context Files for Agentic Coding.arXiv preprint arXiv:2511.12884(2025)
arXiv 2025
Show all 137 references
-
[9]
Souti Chattopadhyay, Ishita Prasad, Austin Z Henley, Anita Sarma, and Titus Barik. 2020. What’s wrong with computational notebooks? Pain points, needs, and design opportunities. InProceedings of the 2020 CHI conference on human factors in computing systems. 1–12
2020
-
[10]
positive fric- tion
Zeya Chen and Ruth Schmidt. 2024. Exploring a behavioral model of “positive fric- tion” in human-AI interaction. InInternational Conference on Human-Computer Interaction. Springer, 3–22
2024
-
[11]
Davide Chicco, Luca Oneto, and Erica Tavazzi. 2022. Eleven quick tips for data cleaning and feature engineering.PLOS Computational Biology18, 12 (2022), e1010718
2022
-
[12]
Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z Henley. 2023. Conversational challenges in ai- powered data science: Obstacles, needs, and design opportunities.arXiv preprint arXiv:2310.16164(2023)
2023 arXiv
-
[13]
HH Clark. 1991. Grounding in communication.Perspectives on socially shared cognition/American Psychological Association(1991)
1991
-
[14]
Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process.Cognition22, 1 (1986), 1–39
1986
-
[15]
Anna L Cox, Sandy JJ Gould, Marta E Cecchinato, Ioanna Iacovides, and Ian Ren- free. 2016. Design frictions for mindful interactions: The case for microboundaries. InProceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. 1389–1397
2016
-
[16]
Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. InProceedings of the 1992 ACM conference on Computer-supported cooperative work. 107–114
1992
-
[17]
Ilker Etikan, Sulaiman Abubakar Musa, Rukayya Sunusi Alkassim, et al . 2016. Comparison of convenience sampling and purposive sampling.American journal of theoretical and applied statistics5, 1 (2016), 1–4
2016
-
[18]
KJ Feng, Tzu-Sheng Kuo, Quan Ze, Inyoung Cheong, Kenneth Holstein, Amy X Zhang, et al. 2025. PolicyPad: Collaborative Prototyping of LLM Policies.arXiv preprint arXiv:2509.19680(2025)
2025
-
[19]
KJ Feng, Kevin Pu, Matt Latzke, Tal August, Pao Siangliulue, Jonathan Bragg, Daniel S Weld, Amy X Zhang, and Joseph Chee Chang. 2025. Cocoa: Co-planning and co-execution with ai agents.arXiv preprint arXiv:2412.10999(2025)
2025
-
[20]
Li Feng, Ryan Yen, Yuzhe You, Mingming Fan, Jian Zhao, and Zhicong Lu. 2024. Coprompt: Supporting prompt sharing and referring in collaborative natural language programming. InProceedings of the 2024 CHI conference on human factors in computing systems. 1–21
2024
-
[21]
Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka-Wei Lee, and Simon Perrault
-
[22]
Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios
-
[23]
Ken Gu, Madeleine Grunde-McLaughlin, Andrew McNutt, Jeffrey Heer, and Tim Althoff. 2024. How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. InProc. CHI’24(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 1015, 22 pa...
2024
-
[24]
Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M. Drucker
-
[25]
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. 2024. Blade: Benchmarking language model agents for data-driven science.arXiv preprint arXiv:2408.09667 (2024)
2024
-
[26]
Ian Guelman, Arthur Gregório Leal, Laerte Xavier, and Marco Tulio Valente
-
[27]
Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware.Computer Supported Cooperative Work (CSCW)11, 3 (2002), 411–446
2002
-
[28]
Robert Haase. 2025. Towards transparency and knowledge exchange in AI- assisted data analysis code generation.Nature Computational Science5, 4 (2025), 271–272
2025
-
[29]
Edward W He, Vanessa Bellotti, Alexandra Scott, Jiaohao Xu, Ashley Suh, Jen Rogers, and Remco Chang. 2025. The Effects of Belief Elicitation in Visual Data Analysis: A Longitudinal Classroom Study.IEEE Transactions on Visualization and Computer Graphics(2025)
2025
-
[30]
On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation.arXiv preprint arXiv:2408.14007(2024)
2024
-
[31]
Jeffrey Heer and Maneesh Agrawala. 2007. Design considerations for collabo- rative visual analytics. In2007 IEEE symposium on visual analytics science and technology. IEEE, 171–178
2007
-
[32]
Oliver Huang, Muhammad Fatir, Steven Luo, Sangho Suh, Hariharan Subra- monyam, and Carolina Nobre. 2025. Narrative Scaffolding: A Narrative-First Framework for Data-Driven Sensemaking.arXiv preprint arXiv:2512.18920(2025)
2025
-
[33]
Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. 2024. DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yas...
2024 doi
-
[34]
Drucker, and Robert DeLine
Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine
-
[35]
Alex Kale, Ziyang Guo, Xiao Li Qiao, Jeffrey Heer, and Jessica Hullman. 2023. Evm: Incorporating model checking into exploratory visual analysis.IEEE Transactions on Visualization and Computer Graphics30, 1 (2023), 208–218
2023
-
[36]
Sean Kandel, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012. Enterprise data analysis and visualization: An interview study.IEEE transactions on visualization and computer graphics18, 12 (2012), 2917–2926
2012
-
[37]
Sean Kandel, Ravi Parikh, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012. Profiler: Integrated statistical analysis and visualization for data quality assessment. InProceedings of the International Working Conference on Advanced Visual Interfaces. 547–554
2012
-
[38]
Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Zachary Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. In Proceedings of the 37th Annual ACM Symposium...
2024
-
[39]
Shaokang Jiang and Daye Nam. 2025. An Empirical Study of Developer-Provided Context for AI Coding Assistants in Open-Source Projects.arXiv preprint arXiv:2512.18925(2025)
2025
-
[40]
Mary Beth Kery and Brad A Myers. 2018. Interactions for untangling messy history in a computational notebook. In2018 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 147–155
2018
-
[41]
John, and Brad A
Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018. The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CH...
2018
-
[42]
David Kirsh. 2010. Thinking with external representations.AI & society25, 4 (2010), 441–454
2010
-
[43]
Laura Koesten, Emilia Kacprzak, Jeni Tennison, and Elena Simperl. 2019. Collab- orative Practices with Structured Data: Do Tools Support What Users Need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)(CHI ’19). Associati...
2019
-
[44]
Mary Beth Kery, Bonnie E John, Patrick O’Flaherty, Amber Horvath, and Brad A Myers. 2019. Towards effective foraging by data scientists to find past analysis choices. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13
2019
-
[45]
Ratanond Koonchanok, Gauri Yatindra Tawde, Gokul Ragunandhan Narayanasamy, Shalmali Walimbe, and Khairi Reda. 2023. Visual belief elicitation reduces the incidence of false discovery. InProceedings of the 2023 CHI conference on human factors in computing systems. 1–17
2023
-
[46]
Michelle S Lam, Fred Hohman, Dominik Moritz, Jeffrey P Bigham, Kenneth Hol- stein, and Mary Beth Kery. 2025. Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–24
2025
-
[47]
Florian Lehmann, Krystsina Shauchenka, and Daniel Buschek. 2025. Collabo- rative Document Editing with Multiple Users and AI Agents.arXiv preprint arXiv:2509.11826(2025)
2025
-
[48]
Clayton Lewis. 1982. Using the’thinking-aloud’method in cognitive interface design.Research Report RC9265, IBM TJ Watson Research Center(1982)
1982
-
[49]
Ratanond Koonchanok, Parul Baser, Abhinav Sikharam, Nirmal Kumar Raveen- dranath, and Khairi Reda. 2021. Data prophecy: Exploring the effects of belief elicitation in visual analytics. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12
2021
-
[50]
Jianping Kelvin Li, Shenyu Xu, Yecong Ye, and Kwan-Liu Ma. 2020. Resolving conflicting insights in asynchronous collaborative visual analysis. InComputer Graphics Forum, Vol. 39. Wiley Online Library, 497–509. 12 IntentLint UIST ’26, November 02–05, 2026, Detroit, MI, USA
2020
-
[51]
Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023. In- sightPilot: An LLM-Empowered Automated Data Exploration System. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Yansong Feng and Els Lef...
2023 doi
-
[52]
Paula Maddigan and Teo Susnjak. 2023. Chat2vis: Generating data visualizations via natural language using chatgpt, codex and gpt-3 large language models.Ieee Access11 (2023), 45181–45193
2023
-
[53]
Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification.Proceedings of the ACM on Software Engineering1, FSE (2024), 2332–2354
2024
-
[54]
James R Lewis, Brian S Utesch, and Deborah E Maher. 2013. UMUX-LITE: when there’s no time for the SUS. InProceedings of the SIGCHI conference on human factors in computing systems. 2099–2102
2013
-
[55]
1988.The psychology of everyday things.Basic books
Donald A Norman. 1988.The psychology of everyday things.Basic books
1988
-
[56]
Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse.Human factors39, 2 (1997), 230–253
1997
-
[57]
Soya Park, April Yi Wang, Ban Kawas, Q Vera Liao, David Piorkowski, and Marina Danilevsky. 2021. Facilitating knowledge sharing from domain experts to data scientists for building nlp models. InProceedings of the 26th International Conference on Intelligent User Interfaces. 585–596
2021
-
[58]
Samir Passi and Steven J Jackson. 2018. Trust in data science: Collaboration, translation, and accountability in corporate data science projects.Proceedings of the ACM on human-computer interaction2, CSCW (2018), 1–28
2018
-
[59]
Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. InProceedings of the 2019 CHI conference on human factors in co...
2019
-
[60]
Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643
1999
-
[61]
Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2022. Eliciting best practices for collaboration with computational notebooks.Proceedings of the ACM on Human-Computer Interaction6, CSCW1 (2022), 1–41
2022
-
[62]
Eric D Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen. 2015. Characteriz- ing provenance in visualization and data analysis: an organizational framework of provenance types and purposes.IEEE transactions on visualization and computer graphics22, 1 (2015), 31–40
2015
-
[63]
Mizanur Rahman, Amran Bhuiyan, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Ridwan Mahbub, Ahmed Masry, Shafiq Joty, and Enamul Hoque. 2025. Llm-based data science agents: A survey of capabilities, challenges, and future directions.arXiv preprint arXiv:2510.04023(2025)
2025
-
[64]
João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire
-
[65]
Anthony C Robinson. 2008. Collaborative synthesis of visual analytic results.. In IEEE V AST. 67–74
2008
-
[66]
Jen Rogers and Anamaria Crisan. 2023. Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work. InProc. CHI’23. Article 837, 22 pages. doi:10.1145/3544548.3580819
2023
-
[67]
Adam Rule, Aurélien Tabard, and James D Hollan. 2018. Exploration and expla- nation in computational notebooks. InProceedings of the 2018 CHI conference on human factors in computing systems. 1–12
2018
-
[68]
Jakob Schoeffer, Maria De-Arteaga, and Niklas Kuehl. 2024. Explanations, fairness, and appropriate reliance in human-AI decision-making. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18
2024
-
[69]
Leixian Shen, Haotian Li, Yun Wang, and Huamin Qu. 2024. From Data to Story: Towards Automatic Animated Data Video Creation with LLM-Based Multi-Agent Systems. In2024 IEEE VIS Workshop on Data Storytelling in an Era of Generative AI (GEN4DS). 20–27. doi:10.1109/GEN4DS63889.2024.00008
2024
-
[70]
Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. 2023. Visualising data science workflows to support third-party note- book comprehension: an empirical study.Empirical Software Engineering28, 3 (2023), 58
2023
-
[71]
Gareth Terry, Nikki Hayfield, Victoria Clarke, Virginia Braun, et al. 2017. The- matic analysis.The SAGE handbook of qualitative research in psychology2, 17-37 (2017), 25
2017
-
[72]
Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, and Yingcai Wu. 2024. Chartgpt: Leveraging llms to generate charts from abstract natural language.IEEE Transactions on Visualization and Computer Graphics31, 3 (2024), 1731–1745
2024
-
[73]
Yuan Tian, Chuhan Zhang, Xiaotong Wang, Sitong Pan, Weiwei Cui, Haidong Zhang, Dazhen Deng, and Yingcai Wu. 2025. ReSpark: Leveraging Previous Data Reports as References to Generate New Reports with LLMs. InProceedings of the 38th Annual ACM Symposium on User Interface Softwar...
2025
-
[74]
Melanie Tory, Lyn Bartram, Brittany Fiore-Gartland, and Anamaria Crisan. 2023. Finding Their Data Voice: Practices and Challenges of Dashboard Users.IEEE Computer Graphics and Applications43, 1 (2023), 22–36
2023
-
[75]
Priyan Vaithilingam, Munyeong Kim, Frida-Cecilia Acosta-Parenteau, Daniel Lee, Amine Mhedhbi, Elena L Glassman, and Ian Arawjo. 2025. Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale. InProceedings of the 38th Annual ACM Symposium on User Inte...
2025
-
[76]
Susan Leigh Star and James R Griesemer. 1989. Institutional ecology,translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39.Social studies of science19, 3 (1989), 387–420
1989
-
[77]
April Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2024. Don’t step on my toes: resolving editing conflicts in real-time collaboration in computa- tional notebooks. InProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments. 47–52
2024
-
[78]
April Yi Wang, Anant Mittal, Christopher Brooks, and Steve Oney. 2019. How Data Scientists Use Computational Notebooks for Real-Time Collaboration.Proc. ACM Hum.-Comput. Interact.3, CSCW, Article 39 (Nov. 2019), 30 pages
2019
-
[79]
April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2022. Documentation matters: Human-centered ai system to assist data science code documentation in com- putational notebooks.ACM Transactions on Compute...
2022
-
[80]
April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020. Callisto: Capturing the" Why" by Connecting Conversations with Computational Narra- tives. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13
2020
-
[81]
Chenglong Wang, John Thompson, and Bongshin Lee. 2023. Data Formula- tor: AI-powered Concept-driven Visualization Authoring.IEEE Transactions on Visualization and Computer Graphics(2023)
2023
-
[82]
Eric A Vance, Jessica L Alzen, and Heather S Smith. 2022. Creating shared understanding in statistics and data science collaborations.Journal of Statistics and Data Science Education30, 1 (2022), 54–64
2022
-
[83]
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. InProceedings of the 2017 conference on empirical methods in natural language processing. 2253–2263
2017
-
[84]
Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian
-
[85]
Catherine Yeh, Donghao Ren, Yannick Assogba, Dominik Moritz, and Fred Hohman. 2025. Exploring empty spaces: Human-in-the-loop data augmentation. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–19
2025
-
[86]
Zamfirescu-Pereira, Richmond Y
J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang
-
[87]
Emanuel Zgraggen, Zheguang Zhao, Robert Zeleznik, and Tim Kraska. 2018. Investigating the effect of the multiple comparisons problem in visual analysis. InProceedings of the 2018 chi conference on human factors in computing systems. 1–12
2018
-
[88]
Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray
Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human- AI Collaboration in Data Science: Exploring Data Scientists’ Perceptions of Auto- mated AI.Proc.CSCW’193, Article 211 (Nov. 2...
2019 doi
-
[89]
Shuning Zhang, Hui Wang, and Xin Yi. 2025. Exploring collaboration patterns and strategies in human-ai co-creation through the lens of agency: A scoping review of the top-tier hci literature.Proceedings of the ACM on Human-Computer Interaction9, 7 (2025), 1–43
2025
-
[90]
Jian Zhao, Michael Glueck, Petra Isenberg, Fanny Chevalier, and Azam Khan
-
[91]
InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–14
-
[92]
Zheguang Zhao, Lorenzo De Stefani, Emanuel Zgraggen, Carsten Binnig, Eli Upfal, and Tim Kraska. 2017. Controlling false discoveries during interactive data exploration. InProceedings of the 2017 acm international conference on man- agement of data. 527–540
2017
-
[93]
Yunfan Zhou, Xiwen Cai, Qiming Shi, Yanwei Huang, Haotian Li, Huamin Qu, Di Weng, and Yingcai Wu. 2025. Xavier: Toward Better Coding Assistance in Authoring Tabular Data Wrangling Scripts. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–16
2025
-
[94]
InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23)
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). Article 437, 21 pages. doi:10. 1145/3544548.3581388
2023
-
[96]
Amy X Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? roles, workflows, and tools.Proceedings of the ACM on Human-Computer Interaction4, CSCW1 (2020), 1–23
2020
-
[100]
Yuheng Zhao, Junjie Wang, Linbing Xiang, Xiaowen Zhang, Zifei Guo, Cagatay Turkay, Yu Zhang, and Siming Chen. 2024. Lightva: Lightweight visual ana- lytics with llm agent-based task planning and execution.IEEE Transactions on Visualization and Computer Graphics31, 9 (2024), 6162–6177
2024
-
[103]
Jiajun Zhu, Xinyu Cheng, Zhongsu Luo, Yunfan Zhou, Xinhuan Shu, Di Weng, and Yingcai Wu. 2025. ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. ...
2025
-
[104]
This system’s capabilities meet my requirements
-
[105]
A.2.2 NASA-TLX
This system is easy to use. A.2.2 NASA-TLX
-
[106]
How mentally demanding was the task?
-
[107]
How physically demanding was the task?
-
[108]
How hurried or rushed was the pace of the task?
-
[109]
How successful were you in accomplishing what you were asked to do?
-
[110]
How hard did you have to work to accomplish your level of performance? Match rules to prompt, return triggered set Rule-checking agent Propose prompt revision Linting agent Suggest new rules Rule-proposal agent Dispatches agents Orchestrator Prompt Linter panelJupyter notebook...
-
[111]
How insecure, discouraged, irritated, stressed, and annoyed were you? A.2.3 Self-Defined Likert Scale Items
-
[112]
I thought carefully about my analytical goals and reasoning [42]
-
[113]
I clearly expressed my intent in the workspace (for my collabo- rators and AI to understand)
-
[114]
I felt confident about which parts of the notebook I could safely build on [14]
-
[115]
I understood my collaborator’s intent enough to safely build on their work [27]
-
[116]
I felt the analytical reasoning was preserved across sessions [62, 67]
-
[117]
The process of writing prompts, iterating on them to generate better code, and aligning with collaborators felt natural during my analysis [55]
-
[118]
The system helped me think of considerations (e.g., documenting my analysis and assumptions) I had overlooked [60]
-
[119]
The system helped me carefully consider my analytical approach before running code [60]
-
[120]
I felt confident that my prompts are clear and precise
-
[121]
The system accurately identified potential problems (e.g., con- flicts in collaboration)
-
[122]
The effort of filling in the rules was worth it for the collaboration [55]
-
[123]
I am satisfied with the overall suggestions from the system. A.3 User interface ofIntentLintand baseline 14 IntentLint UIST ’26, November 02–05, 2026, Detroit, MI, USA b c a Figure 7: Full user interface ofIntentLintwith Jupyter notebook in the VS Code editor Figure 8: Baselin...
2026
-
[124]
This may reference variables, dataframes, columns, or methods that exist (or should exist) in the notebook
User Prompt A natural language instruction the user intends to send to an AI coding assistant for code generation. This may reference variables, dataframes, columns, or methods that exist (or should exist) in the notebook
-
[125]
Notebook Intermediate Representation (IR) A structured summary of the current notebook state, including: - **cell_contents**: A list of cells, each with: - ‘cell_index‘: position in the notebook - ‘author‘: the user who wrote/last edited this cell - ‘content‘: the Python sourc...
-
[126]
R1", "R7
Active Rules A list of collaboration rules, each with: - ‘rule_id‘: identifier (e.g., "R1", "R7") - ‘name‘: human-readable rule name - ‘rule_type‘: "static" (structurally detectable) or "proposal" (requires semantic judgment) - ‘severity‘: "error" (blocks execution), "warn" (c...
2026
-
[127]
This is the text that was evaluated by the rule-checking agent and found to trigger one or more rules
Original Prompt The user’s natural language instruction as originally written. This is the text that was evaluated by the rule-checking agent and found to trigger one or more rules
-
[128]
R2", "R9
The current notebook state (cell contents, variables, execution context) 3. A set of triggered rules (each with a name, description, severity, and a relevance explanation describing why it was triggered) The rules that the rule-checking agent determined were triggered, each wi...
-
[129]
Revision Principles Apply these principles in priority order: Principle 1: Minimal Change Make the smallest edit that resolves each triggered rule
Prompt Author The identity of the user who wrote the original prompt, used to determine cross-user relationships with notebook cell authors. Revision Principles Apply these principles in priority order: Principle 1: Minimal Change Make the smallest edit that resolves each trig...
2026
-
[130]
This is the prompt that triggered one or more existing rules
User Prompt The natural language instruction that was evaluated by the rule-checking agent. This is the prompt that triggered one or more existing rules
-
[131]
Notebook Intermediate Representation (IR)
-
[132]
R2", "R9
Triggered Rules The list of rules that the rule-checking agent determined were triggered, each with: - ‘rule_id‘: the parent rule’s identifier (e.g., "R2", "R9") - ‘severity‘: the parent rule’s severity - ‘relevance‘: why the rule was triggered in this specific context - ‘cont...
-
[133]
It fires in a strict subset of cases where the parent rule fires
More Specific Than Its Parent The proposed rule narrows the parent rule’s scope to a concrete element in the current notebook: a specific variable name, column, data source, threshold value, preprocessing step, analysis method, or authorship pattern. It fires in a strict subse...
-
[134]
The trigger condition must be expressible in terms of: (a) keywords/references in the prompt text, (b) variables/cells/imports in the notebook IR, and (c) authorship metadata
Evaluable at Prompt Time The rule can be checked by reading a user’s natural language prompt against the notebook IR — it does NOT require executing code, inspecting runtime output, or accessing external resources. The trigger condition must be expressible in terms of: (a) key...
-
[135]
The proposed rule would trigger only when the specific, relevant condition holds
Reduces False Positives or False Negatives The proposed rule either: - **Reduces false positives**: The parent rule triggered but the concern was only partially relevant. The proposed rule would trigger only when the specific, relevant condition holds. - **Reduces false negati...
-
[136]
Captures a Team Convention or Decision The best proposed rules codify implicit knowledge that was revealed during the triggering event: a naming convention, a preferred method, a data dependency, a threshold rationale, or a workflow ordering that the team has adopted but not f...
-
[137]
warn if Cell 7 is modified
Durable Beyond This Single Notebook The rule should remain useful across multiple analysis sessions and notebooks, not just for the current cell edit. Rules that are too specific to a single transient state (e.g., "warn if Cell 7 is modified") are not useful. What Makes a BAD ...
-
[2015]
InProceedings of the 28th annual acm symposium on user interface software & technology
Datatone: Managing ambiguity in natural language interfaces for data visualization. InProceedings of the 28th annual acm symposium on user interface software & technology. 489–500
-
[2017]
Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs.IEEE transactions on visualization and computer graphics24, 1 (2017), 340–350
2017
-
[2019]
InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19)
Managing Messes in Computational Notebooks. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3290605.3300500
2019
-
[2021]
Understanding and improving the quality and reproducibility of Jupyter notebooks.Empirical Software Engineering26, 4 (2021), 65
2021
-
[2023]
CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis.ACM Transactions on Computer-Human Interaction31, 1 (2023), 1–38
2023
-
[2024]
How Do Analysts Understand and Verify AI-Assisted Data Analyses?. In Proc. CHI’21. Article 748, 22 pages. doi:10.1145/3613904.3642497
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.