Pith. sign in

REVIEW 3 major objections 5 minor 111 references

Steering Semantic Data Processing With DocWrangler

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that an IDE with in-situ notes, LLM-assisted prompt refinement, and LLM-assisted operation decomposition bridges the gaps between analysts, their unstructured data, and their LLM-powered pipelines, and that users respond…

desk verdict A thoughtful design-study paper with genuinely observed user strategies; the main soft spot is that the LLM-as-judge behind the two AI-driven features is never validated, which undercuts the system-claims but not the qualitative findings. read the letter →

arxiv 2504.14764 v1 pith:BSC4JTEJ submitted 2025-04-20 cs.HC cs.DB

Shreya Shankar , Bhavya Chopra , Mawil Hasan , Stephen Lee , Björn Hartmann , Joseph M. Hellerstein , Aditya G. Parameswaran , Eugene Wu This is my paper · ORCID
classification cs.HCcs.DB
keywords semanticdataprocessinglargelanguagemodelshuman-AIinteractionmixed-initiativeinterfacewranglingpromptrefinementoperationdecompositionqualitativeuserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models turn familiar data operators like map, reduce, and filter into tools that work on unstructured text, but building such semantic data processing pipelines is hard for reasons beyond prompt quality: analysts must understand messy data before they can specify what to extract, and must specify pipelines before they have that understanding. DocWrangler is an IDE built to close those gaps, with notes attached directly to documents and outputs, an assistant that turns those notes into improved prompts, and a background judge that suggests breaking up operations that are too complex for one LLM call. The paper's central claim is that these features let users iterate between inspecting, specifying, and refining, and that users develop systematic strategies in that loop, such as reframing open-ended extraction as classification to make validation easier and deliberately writing vague prompts to learn about their data. This matters because it shifts the design target for LLM data tools from optimizing a fixed pipeline toward supporting the exploratory loop in which the goal itself takes shape.

What carries the argument

The machinery is a notebook-plus-spreadsheet IDE in which every stage of the pipeline remains visible, plus three named features. In-Situ User Notes are annotations stored as the operation, attribute, free-text, and a color tag, attached directly to documents and LLM outputs and persisting across runs. LLM-Assisted Prompt Refinement packages the current prompt, output schema, sample documents, and the relevant notes into a context-window-limited message and returns a revised prompt with visible diffs that the user can edit or branch. LLM-Assisted Operation Decomposition uses a background LLM-as-judge on a sample of outputs to detect failures, then invokes an accuracy optimizer to propose multi-step restructurings. Together they turn observations into specifications and surface generalization failures, which is how the paper bridges its three gulfs.

What would settle it

Track whether the observed strategies survive in users with no LLM experience and in multi-day projects: if beginners or long-horizon teams do not convert open-ended operations into classifiers, do not use vague prompts to learn about data, or do not benefit from notes feeding prompt refinement, the gulf-bridging claims would be limited to expert, short-session use. A cleaner test: in a randomized comparison, half the groups use an IDE without these three features; if the strategies and error-recovery patterns appear equally without the features, then the features are not what enables them.

Watch

Extended reading notes

Core claim

On its own terms, the paper discovers that semantic data processing is not primarily a problem of writing better prompts but a three-way interaction problem among user, data, and pipeline, with distinct gulfs of comprehension, specification, and generalization. DocWrangler's three features each address one gulf: in-situ notes give analysts a persistent place to record what they see, prompt refinement converts those notes into concrete specification changes, and operation decomposition flags when a single LLM call cannot handle the documents and restructures the operation. The behavioral finding, from the think-aloud study and the public deployment, is that users treat their pipelines as tools for learning rather than as final specifications: they add explanation fields and boolean flags just to check the model's reasoning, they convert open-ended tasks into classification problems with histograms for validation, and they sometimes prompt vaguely on purpose to surface what the data and the LLM contain.

Load-bearing premise

The study's general claims rest on the assumption that the ten think-aloud participants, all experienced LLM users recruited from the system's own community, together with the self-selected users of the public deployment, behave the way typical analysts would over the long, multi-day workflows of real document analysis, even though they were observed for one to two hours.

Editorial extensions

If this is right

  • Tools that build on this result should treat in-situ notes as first-class inputs to pipeline revision, because prompt refinement works by feeding those notes directly into the revision context.
  • The classifier-reframing strategy should be explicitly supported: when users convert open-ended extraction into categorical outputs, provide histograms and distribution views that make validation cheap.
  • Even when users reject automatic decomposition, the decomposed plan's examples of failures help them verify and fix their own operations, so presenting failure cases is valuable independent of applying the suggestion.
  • Since most deployed pipelines have two or fewer operations and evolve mainly through prompt and schema changes, fast iteration on single operations with cached intermediates matters more than large pipeline orchestration.
  • Prompt refinement is used proactively to turn vague instructions into concrete specifications, sometimes with no notes at all; this suggests the feature serves as a scaffolding aid for novices as much as a note-application tool.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the classifier-reframing strategy could itself be automated: a system that detects skewed output distributions and proposes a finer-grained or differently scaled output attribute would preempt the manual pivots seen in the study.
  • A testable extension is to measure learning, not just accuracy: if pipelines are epistemic artifacts, the success of a tool like DocWrangler should appear as goal reformulation and question discovery during a session, not only as correct final output.
  • The three-gulf account likely transfers to other generative-AI tasks where the 'data' is a single artifact, such as drafting or image generation; the paper gestures at this, but the transfer is my extrapolation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents DocWrangler, an IDE for semantic data processing in which users assemble LLM-powered operators (map, reduce, filter, resolve) over unstructured documents. Three novel features are introduced: in-situ user notes for annotating documents and outputs, LLM-assisted prompt refinement that turns notes into prompt revisions, and LLM-assisted operation decomposition that proactively suggests restructuring operations that appear too complex. The evaluation consists of a 10-participant think-aloud study and a public deployment with over 1,500 recorded sessions. The central empirical claims are that users develop systematic strategies such as converting open-ended extraction tasks into classifier-like tasks, using vague prompts as epistemic tools to learn about data, and opportunistically realigning pipeline goals in response to LLM behavior. The paper also presents design implications for human-AI collaboration, framed around the three gulfs of comprehension, specification, and generalization.

Significance. If the central claims hold, the paper makes a timely and useful contribution to both data-systems research and human-AI interaction: it articulates a design space for semantic data processing IDEs, contributes three concrete features, and documents user behaviors that are not obvious from prior work. The qualitative findings are richly reported with participant quotes, example workflows, and explicit counts, and four authors independently performed open and axial coding. The public deployment adds ecological validity and the paper is honest about its limitations (Section 8). The main weakness is that the two AI-driven features—prompt refinement and operation decomposition—are claimed to bridge the specification and generalization gulfs, yet the LLM-as-judge component that drives them is not validated at all; the evidence base for the features' effectiveness is therefore thinner than the abstract's language suggests. With modest additional validation or carefully qualified claims, the paper would be a solid design-probe contribution.

major comments (3)
  1. [§4.3, §6.2.3, §7] The central claim that Operation Decomposition bridges the generalization gulf is not supported by any validation of its trigger. Section 4.3 states that decomposition is triggered by an LLM-as-judge (gpt-4o-mini) that samples five documents per run and returns True/False plus failure reasons, and that the accuracy optimizer selects the plan judged best by the same judge. The paper reports no accuracy, precision, or human-agreement data for this judge, and the only supporting evidence in the user study is anecdotal confirmation by P4 and P9 that specific notifications about missed information were accurate. Given that the related work itself cites evidence of LLM-as-judge unreliability [72, 105], the system may mislead users rather than bridge the gulf. This is load-bearing for the paper's central claim about the feature's role. The revision should either validate the judge on a human-labeled sample, report acceptance and outcome data from the deployment, or explicitly reframe the feature as an untested design probe.
  2. [§7] The deployment analysis does not substantiate the quantitative claims about AI assistance or pipeline evolution. Section 7 reports 150 prompt-refinement uses and 95 chat uses, but there is no acceptance rate, no outcome measurement, and no qualitative trace showing whether the suggestions improved the pipelines; users may have invoked and then discarded the suggestions. Additionally, the telemetry analysis was itself performed with DocWrangler using gpt-4o and only 50 extractions were manually verified, with no inter-rater reliability or error analysis. The percentages such as "53% of pipelines grew more complex" and "47% of prompts became more specific" therefore rest on unvalidated LLM classifications. The paper should either provide a manual audit of a larger sample, report agreement for the telemetry coding, or present these numbers as illustrative rather than measured facts.
  3. [§5, §8] The generalizability of the qualitative findings is limited by the participant and session characteristics, and although Section 8 acknowledges this, the abstract and several findings use unqualified language such as "how users develop systematic strategies." All ten participants were recruited from the DocETL Discord, all had prior experience with LLMs, four had prior DocETL experience, and the sessions lasted only one to two hours. For the qualitative claims this is an acceptable design-probe limitation, but the wording should be qualified throughout (e.g., "our participants" and "self-selected deployment users") so that the claims are not read as population-level generalizations.
minor comments (5)
  1. [Abstract and §7] The number of deployment sessions is described inconsistently as "1,500+ recorded sessions" in the abstract and "over 1,500 pipeline executions" and "1,500+ uses" in Section 7; please use a single consistent measure.
  2. [§5 and Table 3] There is a typo "prosidential debates" in the study protocol text; it should be "presidential debates". The same typo appears in Table 3.
  3. [§4.2.4] The sentence "Clicking on the notification triggers the the Operation Decomposition feature" contains a duplicated "the".
  4. [§7] The definition of pipeline evolution uses "consecutive pipeline versions created within 5-minute intervals," but the choice of 5 minutes is not justified; different thresholds would likely change the reported percentages, and the sensitivity to this parameter should be discussed.
  5. [§4, artifact availability] The paper mentions that DocWrangler is containerized via Docker and available at a public playground, but it does not include a repository URL or an artifact availability statement. Adding a link to the open-source code would strengthen reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical systems/HCI study; its qualitative claims rest on user observation and telemetry, and no prediction is defined as its own input.

full rationale

DocWrangler is a design and evaluation paper, not a derivation: the central claims are that three IDE features bridge interaction gulfs and that a think-aloud study and public deployment reveal user strategies. Those claims are supported by observed behavior (Section 6), logged pipeline telemetry (Section 7), and acknowledged limitations (Section 8), not by an equation or reduction. The closest self-referential elements are (i) the Operation Decomposition feature uses gpt-4o-mini as an LLM-as-judge to flag 'too complex' operations and to select the highest-accuracy candidate plan (Section 4.3), and (ii) the deployment telemetry was itself analyzed with DocWrangler, with a manual verification of 50 samples (Section 7). Neither makes the paper's claims true by construction: the paper does not define correctness as the judge's output, and the judge's reliability is an internal-validity concern rather than a logical circle. DocETL [71] is a self-citation but is not load-bearing, because Section 3 explicitly states other semantic data processing systems could serve as the backend. References [72,105] are cited for LLM-as-judge methodology and its known limitations, not as an external uniqueness theorem that forces the paper's design choices. Section 8's limitation that participants were tech-savvy, sessions were short, and findings come from one specific system further bounds the empirical basis instead of concealing a circular step. No parameter is fitted to a subset of data and then renamed as a prediction, and no equation equates the target result with its input. Accordingly, no circular step meets the evidentiary bar of quote-plus-reduction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No numeric parameters are fitted; the paper makes qualitative claims about tool design and user behavior. The hand-chosen implementation constants (e.g., 5-document LLM-as-judge sample, 7-bin histograms) do not affect the central findings. The axioms above are the unproved premises the qualitative conclusions rest on.

assumptions (5)
  • domain assumption LLM outputs are reliable enough for the semantic operations studied (extraction, classification, summarization).
    DocWrangler's utility and the observed user strategies presuppose that gpt-4o-mini and similar models produce usable results on the study tasks; the paper does not benchmark accuracy against ground truth.
  • domain assumption Think-aloud and interview statements reveal actual cognitive processes and strategies.
    The findings in Section 6 rest on this standard HCI assumption; no independent behavioral measures are used.
  • domain assumption The 10 Discord-recruited participants and 1,500+ self-selected deployment sessions represent the broader population of semantic data processing users.
    Section 8 acknowledges tech-savvy, self-selecting users; the paper's general conclusions depend on this.
  • domain assumption LLM-as-judge outputs are accurate enough to drive operation decomposition and telemetry analysis.
    Section 4.3 uses gpt-4o-mini to flag inaccuracies, and Section 7 uses it to analyze telemetry, with only 50 manual verifications; no human agreement metrics are reported.
  • domain assumption The gulf framework from Figure 1 is a valid model of user challenges in semantic data processing.
    The design goals and evaluation are organized around the three gulfs; if the framing is invalid, the significance of the features changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Steering Semantic Data Processing With DocWrangler." pith.science (2026). https://pith.science/paper/BSC4JTEJ

@misc{pith2026250414764,
  author       = {Pith},
  title        = {Pith review of: Steering Semantic Data Processing With DocWrangler},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BSC4JTEJ}},
  note         = {Machine review of arXiv:2504.14764}
}
read the original abstract

Unstructured text has long been difficult to automatically analyze at scale. Large language models (LLMs) now offer a way forward by enabling {\em semantic data processing}, where familiar data processing operators (e.g., map, reduce, filter) are powered by LLMs instead of code. However, building effective semantic data processing pipelines presents a departure from traditional data pipelines: users need to understand their data to write effective pipelines, yet they need to construct pipelines to extract the data necessary for that understanding -- all while navigating LLM idiosyncrasies and inconsistencies. We present \docwrangler, a mixed-initiative integrated development environment (IDE) for semantic data processing with three novel features to address the gaps between the user, their data, and their pipeline: {\em (i) In-Situ User Notes} that allows users to inspect, annotate, and track observations across documents and LLM outputs, {\em (ii) LLM-Assisted Prompt Refinement} that transforms user notes into improved operations, and {\em (iii) LLM-Assisted Operation Decomposition} that identifies when operations or documents are too complex for the LLM to correctly process and suggests decompositions. Our evaluation combines a think-aloud study with 10 participants and a public-facing deployment (available at \href{https://docetl.org/playground}{docetl.org/playground}) with 1,500+ recorded sessions, revealing how users develop systematic strategies for their semantic data processing tasks; e.g., transforming open-ended operations into classifiers for easier validation and intentionally using vague prompts to learn more about their data or LLM capabilities.

Figures

Figures reproduced from arXiv: 2504.14764 by the authors.

Figure 1
Figure 1. Effective semantic data processing requires interaction between users, their pipeline of LLM calls, and their data (both [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Screenshot of the DocWrangler IDE showing our notebook-style pipeline constructor (A) and spreadsheet-like data inspector (B). User notes on documents and LLM-generated data, provided in-situ, and the raw document collection are displayed in the left and right sidebars, respectively. minimal support for the crucial human-centered “outer” loop of discovery and refinement. DocWrangler provides such an envi￾ronment for… view at source ↗
Figure 3
Figure 3. Workflow for analyzing patient discomfort from medical transcripts ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: In-situ user notes feature in DocWrangler (D2). (A) User selects attribute to inspect. (B) Document viewer dialog shows attribute statistics and enables in-situ notes. (C) User could inspect documents side-by-side with split-screen view, if they want. (D) User adds a n…
Figure 5
Figure 5. Figure 5: Prompt Refinement workflow (D3). (A) User initiates improvement via “Improve” button. (B) A dialog opens, showing the current operation and the user’s relevant in-situ notes. (C) AI suggests improved prompt addressing notes (focusing on behavioral discomfort). (D) Impr…
Figure 6
Figure 6. Figure 6: The Operation Decomposition workflow (D3). (A) A notification suggests decomposing a complex operation, simul￾taneous extraction of discomfort and symptoms for long documents. (B) A dialog explains why the operation exceeds LLM capabilities, with examples of inconsiste…
Figure 7
Figure 7. Figure 7: Participant actions by phase: Initialize (create/edit operations), Inspect (review outputs), and Improve (refine prompts or decompose operations). Engagement was bal￾anced across phases, with frequent transitions between them. • Users manipulate semantic operations to …
Figure 8
Figure 8. Figure 8: Participants converted open-ended prompts into [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Adding an unnest operation to analyze symptoms individually. (A) User adds a new operation. (B) “Run” button executes with previously-cached operation outputs, while (C) “Run Fresh” reprocesses all operations from scratch. (D) Output viewer shows operation selectivity …
Figure 10
Figure 10. Figure 10: Adding a reduce operation to summarize symptom data. (A) The user adds a new operation. (B) They specify which field to group by. (C) The help menu shows the AI assistant. (D) The AI assistant helps with Jinja syntax. (E) User copies the assistant’s suggestion to copy…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

111 extracted references · 39 canonical work pages

  1. [1]

    Shm Garanganao Almeda, JD Zamfirescu-Pereira, Kyu Won Kim, Pradeep Mani Rathnam, and Bjoern Hartmann. 2024. Prompting for Discovery: Flexible Sense-Making for AI Art-Making with Dreamsheets. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–17

    2024

  2. [2]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems . 1–13

    2019

  3. [3]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. [n. d.]. Guidelines for Human- AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow Scotland Uk, 2019-05-02). ...

    arXiv 2019

  4. [4]

    Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh

    Eric Anderson, Jonathan Fritz, Austin Lee, Bohou Li, Mark Lindblad, Henry Lindeman, Alex Meyer, Parth Parmar, Tanvi Ranade, Mehul A. Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh. [n. d.]. The Design of an LLM-powered Unstructured Analytics System. arXiv:2409.00847 [cs] http://arxiv.org/abs/2409.00847

    arXiv

  5. [5]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L Glassman. 2024. ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–18

    2024

  6. [6]

    Marcia J Bates. 1989. The design of browsing and berrypicking techniques for the online search interface. Online review 13, 5 (1989), 407–424

    1989

  7. [7]

    Leilani Battle, Remco Chang, and Michael Stonebraker. 2016. Dynamic prefetch- ing of data tiles for interactive visualization. In Proceedings of the 2016 Interna- tional Conference on Management of Data . 1363–1375

    2016

  8. [8]

    Dan Bennett, Oussama Metatla, Anne Roudaut, and Elisa D Mekler. 2023. How does HCI understand human agency and autonomy?. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–18

    2023

Show all 111 references
  1. [9]

    Alan Blackwell and Thomas Green. 2003. Notational systems–the cognitive dimensions of notations framework. HCI models, theories, and frameworks: toward an interdisciplinary science. Morgan Kaufmann 234 (2003)

    2003

  2. [10]

    Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Gross- man. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–14

    2023

  3. [11]

    Bazyli Brzoska. 2025. gpt-tokenizer: A Token Byte Pair Encoder/Decoder for OpenAI Models. https://www.npmjs.com/package/gpt-tokenizer Accessed: 2025-04-07

    2025

  4. [12]

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Talwalkar. 2024. Need Help? Designing Proactive AI Assistants for Programming. arXiv preprint arXiv:2410.04596 (2024)

    2024 arXiv

  5. [13]

    Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z Henley. 2023. Conversational challenges in ai- powered data science: Obstacles, needs, and design opportunities.arXiv preprint arXiv:2310.16164 (2023)

    2023 arXiv

  6. [14]

    Michael Coblenz, Chris Martens, and Luke Church. 2021. Programming Lan- guages + Human-Computer Interaction: Continuing the story at SPLASH 2020. SIGPLAN Blog. https://blog.sigplan.org/2021/07/06/programming-languages- human-computer-interaction-continuing-the-story-at-splash-...

    2021

  7. [15]

    Mary Czerwinski, Eric Horvitz, and Susan Wilhite. 2004. A diary study of task switching and interruptions. In Proceedings of the SIGCHI conference on Human factors in computing systems . 175–182

    2004

  8. [16]

    Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, and Daniel Buschek

  9. [17]

    Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data process- ing on large clusters. Commun. ACM 51, 1 (2008), 107–113

    2008

  10. [18]

    AnHai Doan, Jeffrey F Naughton, Raghu Ramakrishnan, Akanksha Baid, Xi- aoyong Chai, Fei Chen, Ting Chen, Eric Chu, Pedro DeRose, Byron Gao, et al

  11. [19]

    Dudley and Per Ola Kristensson

    John J. Dudley and Per Ola Kristensson. [n. d.]. A Review of User Interface Design for Interactive Machine Learning. 8, 2 ([n. d.]), 1–37. https://doi.org/10. 1145/3185517

  12. [20]

    KJ Feng, Kevin Pu, Matt Latzke, Tal August, Pao Siangliulue, Jonathan Bragg, Daniel S Weld, Amy X Zhang, and Joseph Chee Chang. 2024. Cocoa: Co- Planning and Co-Execution with AI Agents. arXiv preprint arXiv:2412.10999 Shreya Shankar1†, Bhavya Chopra 1†, Mawil Hasan 1, Stephen...

    2024

  13. [21]

    Raul Castro Fernandez, Aaron J Elmore, Michael J Franklin, Sanjay Krishnan, and Chenhao Tan. 2023. How large language models will disrupt data management. Proceedings of the VLDB Endowment 16, 11 (2023), 3302–3309

    2023

  14. [22]

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K Kummerfeld, and Elena L Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–21

    2024

  15. [23]

    Lam, Ranjay Krishna, Daniel S

    Madeleine Grunde-McLaughlin, Michelle S. Lam, Ranjay Krishna, Daniel S. Weld, and Jeffrey Heer. [n. d.]. Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows. arXiv:2312.11681 [cs] http://arxiv.org/abs/ 2312.11681

    arXiv

  16. [24]

    Ken Gu, Madeleine Grunde-McLaughlin, Andrew McNutt, Jeffrey Heer, and Tim Althoff. 2024. How do data analysts respond to ai assistance? a wizard-of-oz study. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–22

    2024

  17. [25]

    Sumit Gulwani. 2016. Programming by examples: Applications, algorithms, and ambiguity resolution. InAutomated Reasoning: 8th International Joint Conference, IJCAR 2016, Coimbra, Portugal, June 27–July 2, 2016, Proceedings 8 . Springer, 9–14

    2016

  18. [26]

    Hossein Hassani and Emmanuel Sirmal Silva. 2023. The role of ChatGPT in data science: how ai-assisted conversational interfaces are revolutionizing the field. Big data and cognitive computing 7, 2 (2023), 62

    2023

  19. [27]

    Jeffrey Heer. 2019. Agency plus automation: Designing artificial intelligence into interactive systems. Proceedings of the National Academy of Sciences 116, 6 (2019), 1844–1850

    2019

  20. [28]

    Jeffrey Heer, Joseph M Hellerstein, and Sean Kandel. 2015. Predictive Interaction for Data Transformation.. In CIDR. Citeseer

    2015

  21. [29]

    Joseph M Hellerstein, Michael Stonebraker, James Hamilton, et al. 2007. Archi- tecture of a database system. Foundations and Trends® in Databases 1, 2 (2007), 141–259

    2007

  22. [30]

    Harry Hochheiser and Ben Shneiderman. 2004. Dynamic query tools for time series data sets: timebox widgets for interactive exploration. Information Visu- alization 3, 1 (2004), 1–18

    2004

  23. [31]

    Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems . 159–166

    1999

  24. [32]

    Gary W Johnson and Richard Jennings. 2001. LabVIEW graphical programming. McGraw-Hill Professional

    2001

  25. [33]

    Adam Tauman Kalai and Santosh S Vempala. 2024. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing. 160–171

    2024

  26. [34]

    Sean Kandel, Jeffrey Heer, Catherine Plaisant, Jessie Kennedy, Frank Van Ham, Nathalie Henry Riche, Chris Weaver, Bongshin Lee, Dominique Brodbeck, and Paolo Buono. 2011. Research directions in data wrangling: Visualizations and transformations for usable and credible data. In...

    2011

  27. [35]

    Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wran- gler: Interactive visual specification of data transformation scripts. In Proceed- ings of the sigchi conference on human factors in computing systems . 3363–3372

    2011

  28. [36]

    Sean Kandel, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012. Enterprise data analysis and visualization: An interview study.IEEE transactions on visualization and computer graphics 18, 12 (2012), 2917–2926

    2012

  29. [37]

    Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Zachary Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. In Proceedings of the 37th Annual ACM Symposium...

    2024

  30. [38]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Ke- shav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. [n. d.]. DSPy: Compiling Declarative Language Model Calls into...

    arXiv

  31. [39]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–21

    2024

  32. [40]

    David Kirsh and Paul Maglio. 1994. On distinguishing epistemic from pragmatic action. Cognitive science 18, 4 (1994), 513–549

    1994

  33. [41]

    Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, et al. 2016. Jupyter Notebooks–a publishing format for reproducible computational workflows. In Positioning ...

    2016

  34. [42]

    Michelle S Lam, Janice Teoh, James A Landay, Jeffrey Heer, and Michael S Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High- Level Concepts Using LLooM. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–28

    2024

  35. [43]

    Doris Jung Lin Lee and Aditya G Parameswaran. 2018. The Case for a Visual Dis- covery Assistant: A Holistic Solution for Accelerating Visual Data Exploration. IEEE Data Eng. Bull. 41, 3 (2018), 3–14

    2018

  36. [44]

    Doris Jung-Lin Lee, Dixin Tang, Kunal Agarwal, Thyne Boonmark, Caitlyn Chen, Jake Kang, Ujjaini Mukhopadhyay, Jerry Song, Micah Yong, Marti A Hearst, et al. 2021. Lux: always-on visualization recommendations for exploratory dataframe workflows. arXiv preprint arXiv:2105.00121 (2021)

    2021 arXiv

  37. [45]

    Arthur Li. 2013. Handbook of SAS ® DATA Step Programming. CRC press

    2013

  38. [46]

    Jingyi Li, Eric Rawn, Jacob Ritchie, Jasper Tran O’Leary, and Sean Follmer. 2023. Beyond the artifact: power as a lens for creativity support tools. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–15

    2023

  39. [47]

    Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano

  40. [48]

    Chunwei Liu, Gerardo Vitagliano, Brandon Rose, Matt Prinz, David Andrew Samson, and Michael Cafarella. 2025. PalimpChat: Declarative and Interactive AI analytics. arXiv preprint arXiv:2502.03368 (2025)

    2025 arXiv

  41. [49]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  42. [50]

    Vivian Liu and Lydia B Chilton. 2022. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI conference on human factors in computing systems . 1–23

    2022

  43. [51]

    Qianou Ma, Weirui Peng, Hua Shen, Kenneth Koedinger, and Tongshuang Wu. 2024. What you say= what you want? Teaching humans to articulate requirements for LLMs. arXiv preprint arXiv:2409.08775 (2024)

    2024 arXiv

  44. [52]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  45. [53]

    Samuel Madden, Michael Cafarella, Michael Franklin, and Tim Kraska. 2024. Databases Unbound: Querying All of the World’s Bytes with AI. Proceedings of the VLDB Endowment 17, 12 (2024), 4546–4554

    2024

  46. [54]

    Wes McKinney et al . 2011. pandas: a foundational Python library for data analysis and statistics. Python for high performance and scientific computing 14, 9 (2011), 1–9

    2011

  47. [55]

    Andrew M McNutt, Chenglong Wang, Robert A Deline, and Steven M Drucker

  48. [56]

    Midjourney. 2025. Midjourney: AI-based Image Generation Service. https: //www.midjourney.com/. Accessed: 2025-04-09

    2025

  49. [57]

    Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How data science workers work with data: Discovery, capture, curation, design, creation. In Proceedings of the 2019 CHI conference on human factors in c...

    2019

  50. [58]

    Tamara Munzner. 2014. Visualization analysis and design . CRC press

    2014

  51. [59]

    Allen Newell. 1972. Human problem solving. Upper Saddle River/Prentive Hall (1972)

    1972

  52. [60]

    In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

    On the design of ai-powered code assistants for notebooks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–16

    2023

  53. [61]

    Tobias Nipkow. 2003. Jinja: Towards a comprehensive formal semantics for a Java-like language. In Proc. Marktobderdorf Summer School . IOS Press Amster- dam

    2003

  54. [62]

    D. A. Norman. 1987. Some observations on mental models . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 241–244

    1987

  55. [63]

    Aditya Parameswaran, Shreya Shankar, Parth Asawa, Naman Jain, and Yujie Wang. 2023. Revisiting Prompt Engineering via Declarative Crowdsourcing. (2023). https://par.nsf.gov/biblio/10531530

    2023

  56. [64]

    Liana Patel, Siddharth Jha, Carlos Guestrin, and Matei Zaharia. 2024. Lotus: Enabling semantic queries with llms over tables of unstructured and structured data. arXiv preprint arXiv:2407.11418 (2024)

    2024 arXiv

  57. [65]

    Jakob Nielsen. 1994. Usability engineering. Morgan Kaufmann

    1994

  58. [66]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10684–10695

    2022

  59. [67]

    Arvind Satyanarayan. 2024. Intelligence as Agency. In Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology . 1–3

    2024

  60. [68]

    Donald A Schön. 1979. The reflective practitioner. New York (1979)

    1979

  61. [69]

    Vidya Setlur and Melanie Tory. 2022. How do you converse with an analytical chatbot? revisiting gricean maxims for designing analytical conversational behavior. InProceedings of the 2022 CHI conference on human factors in computing systems. 1–17. Steering Semantic Data Process...

    2022

  62. [70]

    Vijayshankar Raman and Joseph M Hellerstein. 2001. Potter’s wheel: An inter- active data cleaning system. In VLDB, Vol. 1. 381–390

    2001

  63. [71]

    Parameswaran, and Eugene Wu

    Shreya Shankar, Aditya G. Parameswaran, and Eugene Wu. 2024. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. arXiv:2410.12189 [cs.DB] https://arxiv.org/abs/2410.12189

    2024 arXiv

  64. [72]

    Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software an...

    2024

  65. [73]

    Ben Shneiderman. 2003. The eyes have it: A task by data type taxonomy for information visualizations. In The craft of information visualization . Elsevier, 364–371

    2003

  66. [74]

    Ben Shneiderman. 2020. Human-centered artificial intelligence: Reliable, safe & trustworthy. International Journal of Human–Computer Interaction 36, 6 (2020), 495–504

    2020

  67. [75]

    Shreya Shankar, Haotian Li, Parth Asawa, Madelon Hulsebos, Yiming Lin, J Zamfirescu-Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G Parameswaran, and Eugene Wu. 2024. SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines. Proc. VLDB Endow (2024)

    2024

  68. [76]

    Tarique Siddiqui, Albert Kim, John Lee, Karrie Karahalios, and Aditya Parameswaran. 2016. Effortless Data Exploration with zenvisage: An Expressive and Interactive Visual Analytics System. Proceedings of the VLDB Endowment 10, 4 (2016)

    2016

  69. [77]

    Tarique Siddiqui, Paul Luh, Zesheng Wang, Karrie Karahalios, and Aditya Parameswaran. 2018. Shapesearch: flexible pattern-based querying of trend line visualizations. Proceedings of the VLDB Endowment 11, 12 (2018)

    2018

  70. [78]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test- time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024)

    2024 arXiv

  71. [79]

    Robert Soden, Laura Devendorf, Richmond Y Wong, Lydia B Chilton, Ann Light, and Yoko Akama. 2020. Embracing uncertainty in HCI. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems . 1–8

    2020

  72. [80]

    Nischal Shrestha, Titus Barik, and Chris Parnin. 2021. Unravel: A fluent code explorer for data wrangling. In The 34th Annual ACM Symposium on User Interface Software and Technology. 198–207

    2021

  73. [81]

    Chris Stolte, Diane Tang, and Pat Hanrahan. 2002. Polaris: A system for query, analysis, and visualization of multidimensional relational databases. IEEE Transactions on visualization and computer graphics 8, 1 (2002), 52–65

    2002

  74. [82]

    Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models. IEEE transactions on visualization and computer graphic...

    2022

  75. [83]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–19

    2024

  76. [84]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling multilevel exploration and sensemaking with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–18

    2023

  77. [85]

    Software Arts

    Inc. Software Arts. 1979. VisiCalc. Computer software. https://www.bricklin. com/visicalc.htm Originally developed by Dan Bricklin and Bob Frankston

    1979

  78. [86]

    Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. 2022. Task ambiguity in humans and language models. arXiv preprint arXiv:2212.10711 (2022)

    2022 arXiv

  79. [87]

    Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The metacognitive demands and opportunities of generative AI. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–24

    2024

  80. [88]

    Steven J Taylor, Robert Bogdan, and Marjorie L DeVault. 2015. Introduction to qualitative research methods: A guidebook and resource . John Wiley & Sons

    2015

  81. [89]

    Martin Tricaud and Michel Beaudouin-Lafon. 2023. Revisiting creative be- haviour as an epistemic process: lessons from 12 computational artists and designers. In Proceedings of the 35th Australian Computer-Human Interaction Conference. 175–190

    2023

  82. [90]

    Annalisa Szymanski, Noah Ziems, Heather A Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A Metoyer. 2025. Limitations of the LLM-as-a-Judge approach for evaluating LLM outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent...

    2025

  83. [91]

    Jiayi Wang and Guoliang Li. 2025. Aop: Automated and interactive llm pipeline orchestration for answering complex queries. CIDR

    2025

  84. [92]

    Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020. Gener- alizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur) 53, 3 (2020), 1–34

    2020

  85. [93]

    Colin Ware. 2019. Information visualization: perception for design . Morgan Kaufmann

    2019

  86. [94]

    Michael Williams and Tami Moser. 2019. The art of coding and thematic exploration in qualitative research. International management review 15, 1 (2019), 45–55

    2019

  87. [95]

    VERBI Software. 2022. MAXQDA 2022. Berlin, Germany. https://www.maxqda. com

    2022

  88. [96]

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transpar- ent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human factors in computing systems. 1–22

    2022

  89. [97]

    Yifan Wu, Joseph M Hellerstein, and Arvind Satyanarayan. 2020. B2: Bridging code and interactive visualization in computational notebooks. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology . 152–165

    2020

  90. [98]

    Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian

  91. [99]

    Kuat Yessenov, Shubham Tulsiani, Aditya Menon, Robert C Miller, Sumit Gul- wani, Butler Lampson, and Adam Kalai. 2013. A colorful approach to text processing by example. In Proceedings of the 26th annual ACM symposium on User interface software and technology . 495–504

    2013

  92. [100]

    Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2015. Voyager: Exploratory analysis via faceted browsing of visualization recommendations. IEEE transactions on visualization and computer graphics 22, 1 (2015), 649–658

    2015

  93. [101]

    JD Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI cats: Lessons from designing a chatbot by prompting GPT-3. In Proceedings of the 2023 ACM Designing Interactive Systems Conference . 2206–2220

    2023

  94. [102]

    Sepanta Zeighami, Yiming Lin, Shreya Shankar, and Aditya Parameswaran. [n. d.]. LLM-Powered Proactive Data Systems. Data Engineering ([n. d.]), 90

  95. [103]

    Jingyue Zhang and Ian Arawjo. [n. d.]. ChainBuddy: An AI Agent System for Generating LLM Pipelines. arXiv:2409.13588 [cs] http://arxiv.org/abs/2409. 13588

    arXiv

  96. [104]

    In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

    WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology . 1–14

  97. [105]

    Run” button executes with previously-cached operation outputs, while (C) “Run Fresh

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [n. d.]. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. arXiv:2306.05685 [cs] http://ar...

    arXiv

  98. [106]

    JD Zamfirescu-Pereira, Eunice Jun, Michael Terry, Qian Yang, and Björn Hart- mann. 2025. Beyond Code Generation: LLM-supported Exploration of the Program Design Space. arXiv preprint arXiv:2503.06911 (2025)

    2025 arXiv

  99. [110]

    Xinyang Zhao, Xuanhe Zhou, and Guoliang Li. 2024. Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs. Proceedings of the VLDB Endowment 17, 12 (2024), 4481–4484

    2024

  100. [2009]

    ACM SIGMOD Record 37, 4 (2009), 14–20

    Information extraction challenges in managing unstructured data. ACM SIGMOD Record 37, 4 (2009), 14–20

    2009

  101. [2022]

    arXiv preprint arXiv:2209.01390 (2022)

    How to prompt? Opportunities and challenges of zero-and few-shot learning for human-AI interaction in creative applications of generative models. arXiv preprint arXiv:2209.01390 (2022)

    2022 arXiv

  102. [2023]

    Advances in Neural Information Processing Systems 36 (2023), 46534–46594

    Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2023), 46534–46594

    2023

  103. [2024]

    arXiv preprint arXiv:2405.14696 (2024)

    A Declarative System for Optimizing AI Workloads. arXiv preprint arXiv:2405.14696 (2024)

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.