Pith. sign in

REVIEW 3 major objections 6 minor 137 references

IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read The paper claims that externalizing analytic intent into shared, editable rules and checking every AI prompt against those rules improves awareness, reflection, and early conflict detection in human-AI collaborative data analysis.

desk verdict Design contribution is real; the central awareness claim outruns the evidence—but this is a solid HCI systems paper worth reviewing. read the letter →

arxiv 2608.04331 v1 pith:KDANTYNM submitted 2026-08-05 cs.HC

classification cs.HC
keywords human-AIcollaborationdataanalysisintentscaffoldingprompt-timelintingsharedunderstandingcomputationalnotebooksLLMagentscoordinationrules
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in human-AI collaborative data analysis, shared understanding breaks down because analytic intent stays implicit in notebooks and prompts. To fix this, it introduces a rule-based coordination layer with two mechanisms: intent scaffolding, which turns goals, assumptions, and rationales into structured, editable rules, and prompt-time linting, which checks each new prompt against those rules before code is generated. The central claim is that this combination makes collaborators' intent explicit and catches misalignment early, improving coordination in teams that mix people with LLM agents. A study with 16 data analysts reports that the system improved awareness of collaborators' intent, prompted deeper reflection on analytic strategy, and surfaced conflicts sooner than a standard AI assistant. If right, this shifts the burden of coordination from passive documentation that nobody reads to actionable checks at the moment of prompting.

What carries the argument

The central object is the shared rule: a structured template with a name, author, severity level (info/warn/error), a natural-language description, a trigger condition, and a grounding context field. The rule is designed as a boundary object interpretable by both humans and LLM agents, serving as a concrete artifact for negotiation—accepting, editing, or rejecting a rule becomes a coordination act that builds common ground. Three LLM-based agents carry the workflow: a rule-checking agent that matches prompt and notebook state against trigger conditions, a linting agent that proposes minimal prompt revisions, and a rule-proposal agent that derives more specific rules from triggered ones. Rules persist in the workspace across sessions and propagate in real time to all collaborators, allowing coordination norms to evolve.

What would settle it

Observe a real team using a notebook coordination layer for a full project cycle while a matched team uses a standard AI assistant. The central claim fails if the coordination team does not report higher awareness of collaborators' intent or earlier conflict detection, or if most proposed rules are rejected or deactivated within the first weeks as the rule board grows. A cheaper controlled check: with a fixed rule set, measure whether inter-coder agreement on which rules 'should fire' drops when the notebook context contains stale or contradictory cells.

Watch

Extended reading notes

Core claim

IntentLint's central discovery is that encoding analytic intent as shared, machine-readable rules creates a feedback loop between intent articulation and action. When a user submits a prompt, a rule-proposal agent infers the intent behind it and suggests a rule for review, while a rule-checking agent evaluates the prompt against all existing rules; if a conflict with a collaborator's prior goals or assumptions is found, a linting agent surfaces a message with concrete prompt revisions. The authors report that participants made substantially more prompt edits, rated their prompts as clearer and more precise, accepted or refined most proposed rules, and reported that potential problems were identified significantly earlier compared to a GitHub Copilot baseline. They also report strong agreement between the LLM-based rule-checker and human judgment, supporting the feasibility of natural-language trigger conditions as a coordination mechanism.

Load-bearing premise

The findings come from a simulation of collaboration—each participant worked alone on a notebook pre-populated with a teammate's supposed work—so the claim that the system improves real shared understanding rests on that handoff being a faithful stand-in for actual multi-user teamwork over time.

Editorial extensions

If this is right

  • Analysts using a similar coordination layer will catch conflicts with collaborators' prior decisions before code is generated, reducing downstream rework and increasing confidence in AI-generated code.
  • Prompt quality improves: study participants made more prompt edits and rated their prompts as clearer and more precise with IntentLint than with a standard AI assistant baseline.
  • Teams can convert one-off analytical decisions into reusable coordination norms by accepting or refining proposed rules, growing the rule base over time without imposing documentation overhead.
  • Structured rules complement rather than replace documentation, providing an actionable alternative to inline comments that may be incomplete, outdated, or AI-generated.
  • The direction of effort shifts from editing code to articulating intent, which the paper interprets as supporting deeper analytical reasoning and cognitive offloading of routine validation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One corollary the paper does not pursue: the same rule template could extend beyond notebooks to any channel where a team prompts an LLM, such as chat tools or agent configuration files, since rule-checking operates on prompt text plus a structured context snapshot rather than notebook-specific UI.
  • The reported LLM–human rule-trigger agreement of 0.64–0.91 sets a ceiling: most false triggers and missed triggers likely trace to how well the model interprets natural-language trigger conditions against context, making model interpretation the likeliest lever for precision.
  • Participants rejected rules they saw as one-time decisions or too strict for exploratory analysis, suggesting a testable prediction that coordination-layer value peaks at a moderate rule count and declines as the rule board accumulates narrow or redundant rules.
  • If awareness gains persist outside the lab, a practical consequence is that teams could rely less on mandatory documentation and more on prompt-time checks, changing the effort required to onboard new members to inherited analyses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces IntentLint, a VSCode extension implementing a rule-based coordination layer for AI-assisted collaborative notebooks, with two mechanisms: analytic intent scaffolding (structured, editable rules capturing goals, assumptions, and rationales) and prompt-time linting (checking prompts against shared rules before code generation). It reports a formative study with five analysts, a multi-agent LLM implementation, and a within-subjects study with 16 analysts comparing IntentLint to a GitHub Copilot baseline on pre-populated notebooks. The paper's central claim is that IntentLint improves awareness of collaborators' intent and encourages reflection on analytic strategies.

Significance. If the result holds, this is a valuable design contribution: a concrete, implementable coordination layer for the increasingly common multi-human and multi-agent data analysis setting, with a useful rule taxonomy and design guidelines. The paper's strengths include a working proof-of-concept, a clear rule template, event-log collection, inter-coder reliability checking for rule firing, a thoughtful failure-case analysis, and a candid limitation section. The main obstacle is that the headline claim about collaborative awareness is supported only by self-report in a simulated handoff; this is a construct-validity gap rather than a statistical subtlety, and it needs to be addressed before the abstract's causal wording is justified.

major comments (3)
  1. [§5.1.4, §5.2.3 (Fig. 5, Q4); §6.4] The central claim that IntentLint "improves awareness of collaborators' intent" is not operationally tested. Each participant worked individually on a pre-populated notebook; the "collaborator" is a static artifact, and Q4 asks for a self-report ("I understood my collaborator's intent enough to safely build on their work") with no ground-truth measure of whether participants actually recovered the goals, assumptions, or constraints that the prior work's authors encoded. The authors' own §6.4 acknowledges that the design "cannot fully examine how persistent rules evolve, become stale, conflict, or support real collaboration over time," which is precisely the construct the abstract claims to improve. Please add an objective comprehension measure (e.g., asking participants to reconstruct the prior author's goals and assumptions and scoring against expert-authored ground truth) or revise the claim to "perceived awareness of represented intent" throughout the abstract and Section 5.2.3.
  2. [§5.2.3, Fig. 5] The quantitative support for the reflection and awareness claims rests on ten self-defined Likert items tested with Wilcoxon signed-rank tests without any correction for multiple comparisons, and only categorical p thresholds are reported. With n=16, uncorrected testing across ten items produces an inflated chance of at least one false positive. Please report exact p-values and effect sizes (e.g., matched rank-biserial correlation), apply a correction or clearly justify not doing so, and adjust the strength of the affected claims accordingly.
  3. [§5.2.2, Fig. 4; §5.2.5] The comparison of event counts (more prompt edits and fewer code edits under IntentLint) is used as behavioral evidence of a workflow shift, but the two conditions used different tasks and notebooks (Task 1 sale timing vs. Task 2 attrition), counterbalanced but not shown to be comparable in difficulty or prompt-edit propensity. No statistical test is reported for these counts. Please provide per-task event rates, test the difference, or explicitly present the counts as descriptive only and remove the causal "shift in effort" wording.
minor comments (6)
  1. [§5.2.2] The statement that participants accepted a majority of proposed rules (86 out of 157) "suggesting that the system's rule generation aligned well with collaboration concerns" treats acceptance of system proposals as evidence of value; acceptance could reflect acquiescence or perceived cost of rejecting. Please temper this inference or triangulate it with the interview data.
  2. [Fig. 5 caption] The caption contains a typo ("self-defned") and the questionnaire in Appendix A.2.3 lists 12 items while Figure 5 reports only Q1–Q10; please clarify why Q11 and Q12 are omitted.
  3. [Fig. 8 caption] The caption contains a typo ("Githug CoPilot" should be "GitHub Copilot").
  4. [§3.1.2] The subsection heading "Construction a Computational Notebook Technology Probe" should read "Constructing a Computational Notebook Technology Probe."
  5. [§5.2.2] The paper reports SUS scores "based on the UMUX-LITE" without explaining the conversion or why the UMUX-LITE items are labeled as SUS; please describe the scoring procedure.
  6. [Appendix A.4] Appendix A.4 appears to be empty in the provided manuscript; please populate it or remove the heading.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the awareness and reflection claims are evaluated by independent user measures, not derived from the system's own rule templates.

full rationale

This paper contains no mathematical derivation, so the usual circularity modes (fitted parameter renamed as prediction, uniqueness imported from authors, ansatz smuggled via citation) do not arise. The closest self-referential elements are the default rules in Table 2, which are constructed from the formative study and later assessed through participant acceptance of proposed rules (86 of 157, Section 5.2.2), and a few related-work citations to the authors' own prior papers ([20], [66], [74], [90]). Neither element is load-bearing: rule acceptance is an independent behavioral measure of user endorsement rather than a quantity computed from the rule-proposal mechanism itself, and the self-citations support background claims about notebooks and dashboard practice rather than the paper's central awareness and reflection conclusion. The central evidence is a within-subjects comparison of self-reported Likert responses and interaction logs between IntentLint and a GitHub Copilot baseline (Figures 4 and 5); these measures are not defined in terms of the system's rule templates or LLM agent outputs. The main weakness is construct validity rather than circularity: as the authors explicitly acknowledge in Section 6.4, participants worked individually on pre-populated notebooks, so the 'understood my collaborator's intent' item measures perceived understanding of a simulated handoff, not verified understanding of a real collaborator. This limitation is disclosed and affects generalization, not the logical independence of the evaluation. No circular step can be quoted from the paper, so the score is 0.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on design assumptions rather than fitted parameters. No numeric parameters are fitted to data beyond the hand-authored default rule set. The main burden is that shared rules externalize intent effectively, that LLM-based rule checking is reliable, and that the evaluation setting approximates real collaboration.

free parameters (1)
  • Default rule set (Table 2)
    Hand-authored from a literature synthesis and formative study; not fitted to outcome data, but it determines which prompts are flagged and thus directly shapes the evaluation results.
assumptions (5)
  • domain assumption Shared rules, as boundary objects, externalize analytic intent effectively for both humans and AI agents.
    Section 4.2 introduces rules as the core abstraction; the entire benefit claim depends on this representation being usable by both parties.
  • domain assumption The LLM-based rule-checking agent correctly interprets natural-language trigger conditions with sufficient reliability.
    Section 5.2.1 evaluates agreement (M=0.86) but the system still depends on this performance in practice; false positives and negatives are documented but not fully resolved.
  • domain assumption The 15 challenges synthesized from 25 prior papers are representative of collaborative data analysis breakdowns.
    Section 3.1.1 states the synthesis is 'not intended to be exhaustive' but uses it to ground the formative study and design rules.
  • domain assumption Self-reported Likert responses measure genuine changes in awareness and reflection rather than demand characteristics.
    All key outcomes (Q1-Q10) are self-reports; no objective behavioral measure of understanding is used.
  • domain assumption Working on a pre-populated notebook alone approximates asynchronous collaboration with absent teammates.
    Section 6.4 acknowledges this limitation and notes it cannot capture emergent team dynamics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis." pith.science (2026). https://pith.science/paper/KDANTYNM

@misc{pith2026260804331,
  author       = {Pith},
  title        = {Pith review of: IntentLint: Supporting Intent Scaffolding and Prompt-time Linting in Human-AI Collaborative Data Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDANTYNM}},
  note         = {Machine review of arXiv:2608.04331}
}
read the original abstract

In human-AI collaborative data analysis, as analyses rapidly evolve, the artifacts meant to capture shared understanding often become incomplete or difficult to interpret, leading to undocumented assumptions, cross-user misaligned intent, context-poor prompts, and unwanted agent behaviors. To address these challenges, we introduce a rule-based coordination layer with two interaction mechanisms, intent scaffolding and prompt-time linting, that make analytic intent explicit and actionable during human-AI collaborative data analysis. We implement them in IntentLint, a proof-of-concept system that infers analytic intent from shared notebooks, represents it as structured, editable rules, and checks users' prompts against shared rules. IntentLint helps analysts externalize and refine their intent and proactively checks prompts for potential conflicts. A study with 16 data analysts shows that IntentLint improves awareness of collaborators' intent and encourages reflection on analytic strategies, and provides design implications for supporting more aligned and transparent human-AI collaborative data analysis.

Figures

Figures reproduced from arXiv: 2608.04331 by the authors.

Figure 1
Figure 1. Overview of how our mechanisms work. When Devin submits a prompt (1), the rule-proposal agent analyzes the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The IntentLint interface features three panels: Prompt Linter(a), Rule Proposals(b), and Rule Board(c). The prompt linter displays linting results for the current user prompt, along with explanations and suggested improve￾ments. The Rule Proposals panel allows users to review and edit proposed rules before adding them to the system. The Rule Board displays all active and inactive rules, which users can view and mana… view at source ↗
Figure 3
Figure 3. Rule-based Interactions: rule triggering and rule [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distribution of system log events for 16 participants [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: User perception of the utility of Baseline and IntentLint, measured on self-defned 7-point Likert scales (Appendix A.2.3). Dots represent the mean differences of our system compared to the Baseline. Bars indicate the 95% CI calculated using the studentized bootstrap me…
Figure 6
Figure 6. Figure 6: System Architecture 6. How insecure, discouraged, irritated, stressed, and annoyed were you? A.2.3 Self-Defined Likert Scale Items. 1. I thought carefully about my analytical goals and reasoning [42]. 2. I clearly expressed my intent in the workspace (for my collabo￾ra…
Figure 7
Figure 7. Figure 7: Full user interface of IntentLint with Jupyter notebook in the VS Code editor [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Baseline (Githug CoPilot) with Jupyter notebook in the VS Code editor [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

137 extracted references · 54 canonical work pages

  1. [1]

    [n. d.]. Ames Housing Dataset — kaggle.com. https://www.kaggle.com/datasets/ shashanknecrothapa/ames-housing-dataset/data. [Accessed 25-03-2026]

  2. [2]

    Sara Alspaugh, Nava Zokaei, Andrea Liu, Cindy Jin, and Marti A Hearst. 2018. Futzing and moseying: Interviews with professional data analysts on exploration practices.IEEE transactions on visualization and computer graphics25, 1 (2018), 22–31

  3. [3]

    Leilani Battle and Jeffrey Heer. 2019. Characterizing exploratory visual analysis: A literature review and evaluation of analytic provenance in tableau. InComputer graphics forum, Vol. 38. Wiley Online Library, 145–159

  4. [4]

    Mary Beth Kery and Brad A. Myers. 2017. Exploring exploratory programming. In2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 25–29. doi:10.1109/VLHCC.2017.8103446

  5. [5]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psy- chology.Qualitative Research in Psychology3, 2 (2006), 77–101. doi:10.1191/ 1478088706qp063oa

  6. [6]

    Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health11, 4 (2019), 589–597

  7. [7]

    Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted 11 UIST ’26, November 02–05, 2026, Detroit, MI, USA Feng et al. decision-making.Proceedings of the ACM on Human-computer Interaction5, CSCW1 (2021), 1–21

  8. [8]

    Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa, Brittany Reid, Kundjana- sith Thonglek, Pattara Leelaprute, Arnon Rungsawang, Bundit Manaskasemsak, Bram Adams, Ahmed E Hassan, et al. 2025. Agent READMEs: An Empirical Study of Context Files for Agentic Coding.arXiv preprint arXiv:2511.12884(2025)

Show all 137 references
  1. [9]

    Souti Chattopadhyay, Ishita Prasad, Austin Z Henley, Anita Sarma, and Titus Barik. 2020. What’s wrong with computational notebooks? Pain points, needs, and design opportunities. InProceedings of the 2020 CHI conference on human factors in computing systems. 1–12

  2. [10]

    positive fric- tion

    Zeya Chen and Ruth Schmidt. 2024. Exploring a behavioral model of “positive fric- tion” in human-AI interaction. InInternational Conference on Human-Computer Interaction. Springer, 3–22

  3. [11]

    Davide Chicco, Luca Oneto, and Erica Tavazzi. 2022. Eleven quick tips for data cleaning and feature engineering.PLOS Computational Biology18, 12 (2022), e1010718

  4. [12]

    Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z Henley. 2023. Conversational challenges in ai- powered data science: Obstacles, needs, and design opportunities.arXiv preprint arXiv:2310.16164(2023)

  5. [13]

    HH Clark. 1991. Grounding in communication.Perspectives on socially shared cognition/American Psychological Association(1991)

  6. [14]

    Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process.Cognition22, 1 (1986), 1–39

  7. [15]

    Anna L Cox, Sandy JJ Gould, Marta E Cecchinato, Ioanna Iacovides, and Ian Ren- free. 2016. Design frictions for mindful interactions: The case for microboundaries. InProceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. 1389–1397

  8. [16]

    Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. InProceedings of the 1992 ACM conference on Computer-supported cooperative work. 107–114

  9. [17]

    Ilker Etikan, Sulaiman Abubakar Musa, Rukayya Sunusi Alkassim, et al . 2016. Comparison of convenience sampling and purposive sampling.American journal of theoretical and applied statistics5, 1 (2016), 1–4

  10. [18]

    KJ Feng, Tzu-Sheng Kuo, Quan Ze, Inyoung Cheong, Kenneth Holstein, Amy X Zhang, et al. 2025. PolicyPad: Collaborative Prototyping of LLM Policies.arXiv preprint arXiv:2509.19680(2025)

  11. [19]

    KJ Feng, Kevin Pu, Matt Latzke, Tal August, Pao Siangliulue, Jonathan Bragg, Daniel S Weld, Amy X Zhang, and Joseph Chee Chang. 2025. Cocoa: Co-planning and co-execution with ai agents.arXiv preprint arXiv:2412.10999(2025)

  12. [20]

    Li Feng, Ryan Yen, Yuzhe You, Mingming Fan, Jian Zhao, and Zhicong Lu. 2024. Coprompt: Supporting prompt sharing and referring in collaborative natural language programming. InProceedings of the 2024 CHI conference on human factors in computing systems. 1–21

  13. [21]

    Jie Gao, Kenny Tsu Wei Choo, Junming Cao, Roy Ka-Wei Lee, and Simon Perrault

  14. [22]

    Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G Karahalios

  15. [23]

    Ken Gu, Madeleine Grunde-McLaughlin, Andrew McNutt, Jeffrey Heer, and Tim Althoff. 2024. How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. InProc. CHI’24(Honolulu, HI, USA)(CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 1015, 22 pa...

  16. [24]

    Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M. Drucker

  17. [25]

    Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, et al. 2024. Blade: Benchmarking language model agents for data-driven science.arXiv preprint arXiv:2408.09667 (2024)

  18. [26]

    Ian Guelman, Arthur Gregório Leal, Laerte Xavier, and Marco Tulio Valente

  19. [27]

    Carl Gutwin and Saul Greenberg. 2002. A descriptive framework of workspace awareness for real-time groupware.Computer Supported Cooperative Work (CSCW)11, 3 (2002), 411–446

  20. [28]

    Robert Haase. 2025. Towards transparency and knowledge exchange in AI- assisted data analysis code generation.Nature Computational Science5, 4 (2025), 271–272

  21. [29]

    Edward W He, Vanessa Bellotti, Alexandra Scott, Jiaohao Xu, Ashley Suh, Jen Rogers, and Remco Chang. 2025. The Effects of Belief Elicitation in Visual Data Analysis: A Longitudinal Classroom Study.IEEE Transactions on Visualization and Computer Graphics(2025)

  22. [30]

    On the Quality of AI-Generated Source Code Comments: A Comprehensive Evaluation.arXiv preprint arXiv:2408.14007(2024)

  23. [31]

    Jeffrey Heer and Maneesh Agrawala. 2007. Design considerations for collabo- rative visual analytics. In2007 IEEE symposium on visual analytics science and technology. IEEE, 171–178

  24. [32]

    Oliver Huang, Muhammad Fatir, Steven Luo, Sangho Suh, Hariharan Subra- monyam, and Carolina Nobre. 2025. Narrative Scaffolding: A Narrative-First Framework for Data-Driven Sensemaking.arXiv preprint arXiv:2512.18920(2025)

  25. [33]

    Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. 2024. DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yas...

  26. [34]

    Drucker, and Robert DeLine

    Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine

  27. [35]

    Alex Kale, Ziyang Guo, Xiao Li Qiao, Jeffrey Heer, and Jessica Hullman. 2023. Evm: Incorporating model checking into exploratory visual analysis.IEEE Transactions on Visualization and Computer Graphics30, 1 (2023), 208–218

  28. [36]

    Sean Kandel, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012. Enterprise data analysis and visualization: An interview study.IEEE transactions on visualization and computer graphics18, 12 (2012), 2917–2926

  29. [37]

    Sean Kandel, Ravi Parikh, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012. Profiler: Integrated statistical analysis and visualization for data quality assessment. InProceedings of the International Working Conference on Advanced Visual Interfaces. 547–554

  30. [38]

    Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Zachary Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. In Proceedings of the 37th Annual ACM Symposium...

  31. [39]

    Shaokang Jiang and Daye Nam. 2025. An Empirical Study of Developer-Provided Context for AI Coding Assistants in Open-Source Projects.arXiv preprint arXiv:2512.18925(2025)

  32. [40]

    Mary Beth Kery and Brad A Myers. 2018. Interactions for untangling messy history in a computational notebook. In2018 IEEE symposium on visual languages and human-centric computing (VL/HCC). IEEE, 147–155

  33. [41]

    John, and Brad A

    Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018. The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems(Montreal QC, Canada)(CH...

  34. [42]

    David Kirsh. 2010. Thinking with external representations.AI & society25, 4 (2010), 441–454

  35. [43]

    Laura Koesten, Emilia Kacprzak, Jeni Tennison, and Elena Simperl. 2019. Collab- orative Practices with Structured Data: Do Tools Support What Users Need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)(CHI ’19). Associati...

  36. [44]

    Mary Beth Kery, Bonnie E John, Patrick O’Flaherty, Amber Horvath, and Brad A Myers. 2019. Towards effective foraging by data scientists to find past analysis choices. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13

  37. [45]

    Ratanond Koonchanok, Gauri Yatindra Tawde, Gokul Ragunandhan Narayanasamy, Shalmali Walimbe, and Khairi Reda. 2023. Visual belief elicitation reduces the incidence of false discovery. InProceedings of the 2023 CHI conference on human factors in computing systems. 1–17

  38. [46]

    Michelle S Lam, Fred Hohman, Dominik Moritz, Jeffrey P Bigham, Kenneth Hol- stein, and Mary Beth Kery. 2025. Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–24

  39. [47]

    Florian Lehmann, Krystsina Shauchenka, and Daniel Buschek. 2025. Collabo- rative Document Editing with Multiple Users and AI Agents.arXiv preprint arXiv:2509.11826(2025)

  40. [48]

    Clayton Lewis. 1982. Using the’thinking-aloud’method in cognitive interface design.Research Report RC9265, IBM TJ Watson Research Center(1982)

  41. [49]

    Ratanond Koonchanok, Parul Baser, Abhinav Sikharam, Nirmal Kumar Raveen- dranath, and Khairi Reda. 2021. Data prophecy: Exploring the effects of belief elicitation in visual analytics. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12

  42. [50]

    Jianping Kelvin Li, Shenyu Xu, Yecong Ye, and Kwan-Liu Ma. 2020. Resolving conflicting insights in asynchronous collaborative visual analysis. InComputer Graphics Forum, Vol. 39. Wiley Online Library, 497–509. 12 IntentLint UIST ’26, November 02–05, 2026, Detroit, MI, USA

  43. [51]

    Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023. In- sightPilot: An LLM-Empowered Automated Data Exploration System. InProceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Yansong Feng and Els Lef...

  44. [52]

    Paula Maddigan and Teo Susnjak. 2023. Chat2vis: Generating data visualizations via natural language using chatgpt, codex and gpt-3 large language models.Ieee Access11 (2023), 45181–45193

  45. [53]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification.Proceedings of the ACM on Software Engineering1, FSE (2024), 2332–2354

  46. [54]

    James R Lewis, Brian S Utesch, and Deborah E Maher. 2013. UMUX-LITE: when there’s no time for the SUS. InProceedings of the SIGCHI conference on human factors in computing systems. 2099–2102

  47. [55]

    1988.The psychology of everyday things.Basic books

    Donald A Norman. 1988.The psychology of everyday things.Basic books

  48. [56]

    Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse.Human factors39, 2 (1997), 230–253

  49. [57]

    Soya Park, April Yi Wang, Ban Kawas, Q Vera Liao, David Piorkowski, and Marina Danilevsky. 2021. Facilitating knowledge sharing from domain experts to data scientists for building nlp models. InProceedings of the 26th International Conference on Intelligent User Interfaces. 585–596

  50. [58]

    Samir Passi and Steven J Jackson. 2018. Trust in data science: Collaboration, translation, and accountability in corporate data science projects.Proceedings of the ACM on human-computer interaction2, CSCW (2018), 1–28

  51. [59]

    Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. InProceedings of the 2019 CHI conference on human factors in co...

  52. [60]

    Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643

  53. [61]

    Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. 2022. Eliciting best practices for collaboration with computational notebooks.Proceedings of the ACM on Human-Computer Interaction6, CSCW1 (2022), 1–41

  54. [62]

    Eric D Ragan, Alex Endert, Jibonananda Sanyal, and Jian Chen. 2015. Characteriz- ing provenance in visualization and data analysis: an organizational framework of provenance types and purposes.IEEE transactions on visualization and computer graphics22, 1 (2015), 31–40

  55. [63]

    Mizanur Rahman, Amran Bhuiyan, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Ridwan Mahbub, Ahmed Masry, Shafiq Joty, and Enamul Hoque. 2025. Llm-based data science agents: A survey of capabilities, challenges, and future directions.arXiv preprint arXiv:2510.04023(2025)

  56. [64]

    João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire

  57. [65]

    Anthony C Robinson. 2008. Collaborative synthesis of visual analytic results.. In IEEE V AST. 67–74

  58. [66]

    Jen Rogers and Anamaria Crisan. 2023. Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work. InProc. CHI’23. Article 837, 22 pages. doi:10.1145/3544548.3580819

  59. [67]

    Adam Rule, Aurélien Tabard, and James D Hollan. 2018. Exploration and expla- nation in computational notebooks. InProceedings of the 2018 CHI conference on human factors in computing systems. 1–12

  60. [68]

    Jakob Schoeffer, Maria De-Arteaga, and Niklas Kuehl. 2024. Explanations, fairness, and appropriate reliance in human-AI decision-making. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18

  61. [69]

    Leixian Shen, Haotian Li, Yun Wang, and Huamin Qu. 2024. From Data to Story: Towards Automatic Animated Data Video Creation with LLM-Based Multi-Agent Systems. In2024 IEEE VIS Workshop on Data Storytelling in an Era of Generative AI (GEN4DS). 20–27. doi:10.1109/GEN4DS63889.2024.00008

  62. [70]

    Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. 2023. Visualising data science workflows to support third-party note- book comprehension: an empirical study.Empirical Software Engineering28, 3 (2023), 58

  63. [71]

    Gareth Terry, Nikki Hayfield, Victoria Clarke, Virginia Braun, et al. 2017. The- matic analysis.The SAGE handbook of qualitative research in psychology2, 17-37 (2017), 25

  64. [72]

    Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, and Yingcai Wu. 2024. Chartgpt: Leveraging llms to generate charts from abstract natural language.IEEE Transactions on Visualization and Computer Graphics31, 3 (2024), 1731–1745

  65. [73]

    Yuan Tian, Chuhan Zhang, Xiaotong Wang, Sitong Pan, Weiwei Cui, Haidong Zhang, Dazhen Deng, and Yingcai Wu. 2025. ReSpark: Leveraging Previous Data Reports as References to Generate New Reports with LLMs. InProceedings of the 38th Annual ACM Symposium on User Interface Softwar...

  66. [74]

    Melanie Tory, Lyn Bartram, Brittany Fiore-Gartland, and Anamaria Crisan. 2023. Finding Their Data Voice: Practices and Challenges of Dashboard Users.IEEE Computer Graphics and Applications43, 1 (2023), 22–36

  67. [75]

    Priyan Vaithilingam, Munyeong Kim, Frida-Cecilia Acosta-Parenteau, Daniel Lee, Amine Mhedhbi, Elena L Glassman, and Ian Arawjo. 2025. Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale. InProceedings of the 38th Annual ACM Symposium on User Inte...

  68. [76]

    Susan Leigh Star and James R Griesemer. 1989. Institutional ecology,translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907-39.Social studies of science19, 3 (1989), 387–420

  69. [77]

    April Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2024. Don’t step on my toes: resolving editing conflicts in real-time collaboration in computa- tional notebooks. InProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments. 47–52

  70. [78]

    April Yi Wang, Anant Mittal, Christopher Brooks, and Steve Oney. 2019. How Data Scientists Use Computational Notebooks for Real-Time Collaboration.Proc. ACM Hum.-Comput. Interact.3, CSCW, Article 39 (Nov. 2019), 30 pages

  71. [79]

    April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2022. Documentation matters: Human-centered ai system to assist data science code documentation in com- putational notebooks.ACM Transactions on Compute...

  72. [80]

    April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020. Callisto: Capturing the" Why" by Connecting Conversations with Computational Narra- tives. InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13

  73. [81]

    Chenglong Wang, John Thompson, and Bongshin Lee. 2023. Data Formula- tor: AI-powered Concept-driven Visualization Authoring.IEEE Transactions on Visualization and Computer Graphics(2023)

  74. [82]

    Eric A Vance, Jessica L Alzen, and Heather S Smith. 2022. Creating shared understanding in statistics and data science collaborations.Journal of Statistics and Data Science Education30, 1 (2022), 54–64

  75. [83]

    Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. InProceedings of the 2017 conference on empirical methods in natural language processing. 2253–2263

  76. [84]

    Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian

  77. [85]

    Catherine Yeh, Donghao Ren, Yannick Assogba, Dominik Moritz, and Fred Hohman. 2025. Exploring empty spaces: Human-in-the-loop data augmentation. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–19

  78. [86]

    Zamfirescu-Pereira, Richmond Y

    J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang

  79. [87]

    Emanuel Zgraggen, Zheguang Zhao, Robert Zeleznik, and Tim Kraska. 2018. Investigating the effect of the multiple comparisons problem in visual analysis. InProceedings of the 2018 chi conference on human factors in computing systems. 1–12

  80. [88]

    Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray

    Dakuo Wang, Justin D. Weisz, Michael Muller, Parikshit Ram, Werner Geyer, Casey Dugan, Yla Tausczik, Horst Samulowitz, and Alexander Gray. 2019. Human- AI Collaboration in Data Science: Exploring Data Scientists’ Perceptions of Auto- mated AI.Proc.CSCW’193, Article 211 (Nov. 2...

  81. [89]

    Shuning Zhang, Hui Wang, and Xin Yi. 2025. Exploring collaboration patterns and strategies in human-ai co-creation through the lens of agency: A scoping review of the top-tier hci literature.Proceedings of the ACM on Human-Computer Interaction9, 7 (2025), 1–43

  82. [90]

    Jian Zhao, Michael Glueck, Petra Isenberg, Fanny Chevalier, and Azam Khan

  83. [91]

    InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

    WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–14

  84. [92]

    Zheguang Zhao, Lorenzo De Stefani, Emanuel Zgraggen, Carsten Binnig, Eli Upfal, and Tim Kraska. 2017. Controlling false discoveries during interactive data exploration. InProceedings of the 2017 acm international conference on man- agement of data. 527–540

  85. [93]

    Yunfan Zhou, Xiwen Cai, Qiming Shi, Yanwei Huang, Haotian Li, Huamin Qu, Di Weng, and Yingcai Wu. 2025. Xavier: Toward Better Coding Assistance in Authoring Tabular Data Wrangling Scripts. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–16

  86. [94]

    InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23)

    Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). Article 437, 21 pages. doi:10. 1145/3544548.3581388

  87. [96]

    Amy X Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? roles, workflows, and tools.Proceedings of the ACM on Human-Computer Interaction4, CSCW1 (2020), 1–23

  88. [100]

    Yuheng Zhao, Junjie Wang, Linbing Xiang, Xiaowen Zhang, Zifei Guo, Cagatay Turkay, Yu Zhang, and Siming Chen. 2024. Lightva: Lightweight visual ana- lytics with llm agent-based task planning and execution.IEEE Transactions on Visualization and Computer Graphics31, 9 (2024), 6162–6177

  89. [103]

    Jiajun Zhu, Xinyu Cheng, Zhongsu Luo, Yunfan Zhou, Xinhuan Shu, Di Weng, and Yingcai Wu. 2025. ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. ...

  90. [104]

    This system’s capabilities meet my requirements

  91. [105]

    A.2.2 NASA-TLX

    This system is easy to use. A.2.2 NASA-TLX

  92. [106]

    How mentally demanding was the task?

  93. [107]

    How physically demanding was the task?

  94. [108]

    How hurried or rushed was the pace of the task?

  95. [109]

    How successful were you in accomplishing what you were asked to do?

  96. [110]

    How hard did you have to work to accomplish your level of performance? Match rules to prompt, return triggered set Rule-checking agent Propose prompt revision Linting agent Suggest new rules Rule-proposal agent Dispatches agents Orchestrator Prompt Linter panelJupyter notebook...

  97. [111]

    How insecure, discouraged, irritated, stressed, and annoyed were you? A.2.3 Self-Defined Likert Scale Items

  98. [112]

    I thought carefully about my analytical goals and reasoning [42]

  99. [113]

    I clearly expressed my intent in the workspace (for my collabo- rators and AI to understand)

  100. [114]

    I felt confident about which parts of the notebook I could safely build on [14]

  101. [115]

    I understood my collaborator’s intent enough to safely build on their work [27]

  102. [116]

    I felt the analytical reasoning was preserved across sessions [62, 67]

  103. [117]

    The process of writing prompts, iterating on them to generate better code, and aligning with collaborators felt natural during my analysis [55]

  104. [118]

    The system helped me think of considerations (e.g., documenting my analysis and assumptions) I had overlooked [60]

  105. [119]

    The system helped me carefully consider my analytical approach before running code [60]

  106. [120]

    I felt confident that my prompts are clear and precise

  107. [121]

    The system accurately identified potential problems (e.g., con- flicts in collaboration)

  108. [122]

    The effort of filling in the rules was worth it for the collaboration [55]

  109. [123]

    I am satisfied with the overall suggestions from the system. A.3 User interface ofIntentLintand baseline 14 IntentLint UIST ’26, November 02–05, 2026, Detroit, MI, USA b c a Figure 7: Full user interface ofIntentLintwith Jupyter notebook in the VS Code editor Figure 8: Baselin...

  110. [124]

    This may reference variables, dataframes, columns, or methods that exist (or should exist) in the notebook

    User Prompt A natural language instruction the user intends to send to an AI coding assistant for code generation. This may reference variables, dataframes, columns, or methods that exist (or should exist) in the notebook

  111. [125]

    Notebook Intermediate Representation (IR) A structured summary of the current notebook state, including: - **cell_contents**: A list of cells, each with: - ‘cell_index‘: position in the notebook - ‘author‘: the user who wrote/last edited this cell - ‘content‘: the Python sourc...

  112. [126]

    R1", "R7

    Active Rules A list of collaboration rules, each with: - ‘rule_id‘: identifier (e.g., "R1", "R7") - ‘name‘: human-readable rule name - ‘rule_type‘: "static" (structurally detectable) or "proposal" (requires semantic judgment) - ‘severity‘: "error" (blocks execution), "warn" (c...

  113. [127]

    This is the text that was evaluated by the rule-checking agent and found to trigger one or more rules

    Original Prompt The user’s natural language instruction as originally written. This is the text that was evaluated by the rule-checking agent and found to trigger one or more rules

  114. [128]

    R2", "R9

    The current notebook state (cell contents, variables, execution context) 3. A set of triggered rules (each with a name, description, severity, and a relevance explanation describing why it was triggered) The rules that the rule-checking agent determined were triggered, each wi...

  115. [129]

    Revision Principles Apply these principles in priority order: Principle 1: Minimal Change Make the smallest edit that resolves each triggered rule

    Prompt Author The identity of the user who wrote the original prompt, used to determine cross-user relationships with notebook cell authors. Revision Principles Apply these principles in priority order: Principle 1: Minimal Change Make the smallest edit that resolves each trig...

  116. [130]

    This is the prompt that triggered one or more existing rules

    User Prompt The natural language instruction that was evaluated by the rule-checking agent. This is the prompt that triggered one or more existing rules

  117. [131]

    Notebook Intermediate Representation (IR)

  118. [132]

    R2", "R9

    Triggered Rules The list of rules that the rule-checking agent determined were triggered, each with: - ‘rule_id‘: the parent rule’s identifier (e.g., "R2", "R9") - ‘severity‘: the parent rule’s severity - ‘relevance‘: why the rule was triggered in this specific context - ‘cont...

  119. [133]

    It fires in a strict subset of cases where the parent rule fires

    More Specific Than Its Parent The proposed rule narrows the parent rule’s scope to a concrete element in the current notebook: a specific variable name, column, data source, threshold value, preprocessing step, analysis method, or authorship pattern. It fires in a strict subse...

  120. [134]

    The trigger condition must be expressible in terms of: (a) keywords/references in the prompt text, (b) variables/cells/imports in the notebook IR, and (c) authorship metadata

    Evaluable at Prompt Time The rule can be checked by reading a user’s natural language prompt against the notebook IR — it does NOT require executing code, inspecting runtime output, or accessing external resources. The trigger condition must be expressible in terms of: (a) key...

  121. [135]

    The proposed rule would trigger only when the specific, relevant condition holds

    Reduces False Positives or False Negatives The proposed rule either: - **Reduces false positives**: The parent rule triggered but the concern was only partially relevant. The proposed rule would trigger only when the specific, relevant condition holds. - **Reduces false negati...

  122. [136]

    Captures a Team Convention or Decision The best proposed rules codify implicit knowledge that was revealed during the triggering event: a naming convention, a preferred method, a data dependency, a threshold rationale, or a workflow ordering that the team has adopted but not f...

  123. [137]

    warn if Cell 7 is modified

    Durable Beyond This Single Notebook The rule should remain useful across multiple analysis sessions and notebooks, not just for the current cell edit. Rules that are too specific to a single transient state (e.g., "warn if Cell 7 is modified") are not useful. What Makes a BAD ...

  124. [2015]

    InProceedings of the 28th annual acm symposium on user interface software & technology

    Datatone: Managing ambiguity in natural language interfaces for data visualization. InProceedings of the 28th annual acm symposium on user interface software & technology. 489–500

  125. [2017]

    Supporting handoff in asynchronous collaborative sensemaking using knowledge-transfer graphs.IEEE transactions on visualization and computer graphics24, 1 (2017), 340–350

  126. [2019]

    InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19)

    Managing Messes in Computational Notebooks. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems(Glasgow, Scotland Uk)(CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3290605.3300500

  127. [2021]

    Understanding and improving the quality and reproducibility of Jupyter notebooks.Empirical Software Engineering26, 4 (2021), 65

  128. [2023]

    CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis.ACM Transactions on Computer-Human Interaction31, 1 (2023), 1–38

  129. [2024]

    How Do Analysts Understand and Verify AI-Assisted Data Analyses?. In Proc. CHI’21. Article 748, 22 pages. doi:10.1145/3613904.3642497

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.