Pith. sign in

REVIEW 4 major objections 6 minor 76 references

Dynamic Prompt Middleware: Contextual Prompt Refinement Controls for Comprehension Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Users preferred dynamically generated prompt refinement controls over a fixed preset list, reporting more control, lower barriers to providing context, and more exploration of comprehension tasks, though the effects of controls on the…

desk verdict Worth reading for the system design and qualitative findings, but the central causal claim about dynamic generation is underdetermined because the Static PRC baseline is a specific, possibly unrepresentative option set. read the letter →

arxiv 2412.02357 v1 pith:U763277X submitted 2024-12-03 cs.HC cs.AI

Ian Drosos , Jack Williams , Advait Sarkar , Nicholas Wilson This is my paper · ORCID
classification cs.HCcs.AI
keywords promptmiddlewaredynamicUIgenerationusercontrolgenerativeAIcomprehensiontasksexplanationinterfaceshuman-AIinteractionrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that prompt middleware—interfaces that help users construct prompts—can be improved by generating the control elements themselves on the fly, tailored to the user's current prompt and task. The authors built a system whose UI options (radio buttons, checkboxes, text boxes) are produced by a language model based on the user's input, and compared it to a version offering a static, preset list of generally useful refinements. In a formative survey (n=38) and a controlled within-subjects study (n=16) with comprehension tasks such as explaining spreadsheet formulas, Python code, and text passages, participants preferred the dynamic version for affording more control and lowering barriers to providing context, while also finding it more mentally demanding and harder to reason about. The central claim is that dynamically generating prompt refinements can improve the user experience of generative AI workflows, and the paper derives design implications for future such systems.

What carries the argument

The central object is the 'Option Module'—an LLM agent that takes the user's prompt, conversation history, and current session options as input, and returns a set of prompt options as JSON conforming to a TypeScript schema. These options are rendered as radio buttons, checkboxes, and free-text fields with initial values; a serializer converts the selected options into textual prompt refinements that are appended to the prompt sent to a separate Chat Module, allowing deterministic additions, removals, or modifications of the user's original prompt. A two-tier structure separates inline options (regenerated for every user input) from session options (applied to all subsequent prompts), giving users both task-specific tailoring and persistent preferences.

What would settle it

Run the same within-subjects study with a Static PRC list that is carefully validated by domain experts as well-matched to the specific tasks (e.g., a preset list distilled from the Dynamic PRC system's own most common generations), and measure both preference and objective explanation quality; if the dynamic advantage disappears or reverses, the reported preference is an artifact of the particular static list rather than the generation mechanism.

Watch

Extended reading notes

Core claim

The paper's central claim is that dynamic prompt middleware—where an LLM analyzes the user's prompt and generates a set of refinement options rendered as GUI controls—gives users more effective control over AI responses than a static, pre-defined list of refinements, for comprehension and learning tasks. The evidence is a preference for the Dynamic PRC approach in a within-subjects study: participants reported that it afforded more control, lowered the barrier to providing context, and encouraged exploration and reflection, at the cost of greater perceived complexity and difficulty in predicting how individual options would affect the output. The paper presents this as a trade-off between standardized but predictable support (Static PRC) and adaptive but less predictable support (Dynamic PRC), with users favoring the adaptive form.

Load-bearing premise

The comparison assumes that the Static PRC preset list is a fair, non-degenerate baseline; if the fixed options were unusually ill-suited to the study tasks, the observed Dynamic PRC advantage would reflect baseline weakness rather than the value of dynamic generation.

Editorial extensions

If this is right

  • Dynamic prompt middleware can improve the user experience of generative AI workflows by increasing perceived control and reducing the effort needed to express context.
  • Users can generate new controls through natural language or by directly editing JSON, enabling them to create refinements not anticipated by the system.
  • The approach encourages exploration and reflection on comprehension tasks, suggesting potential as a 'tool for thought' or metacognitive scaffold.
  • Design implications include giving users control over how and where the AI applies options, leveraging user context and data to generate helpful controls, and supporting direct manipulation of generated options.
  • The remaining barrier—reasoning about the effects of generated controls on final output—points toward integrating explainability mechanisms such as response diffs or more transparent option application.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible extension is that dynamic generation could benefit not just comprehension tasks but any LLM-steering interface, including code generation, data analysis, and creative writing; the evaluated mechanism is general and modular.
  • The paper measures perceived control and preference, not objective explanation quality or learning outcomes; a testable extension would examine whether dynamic options actually improve comprehension test scores or task performance.
  • The static baseline's fairness is the main internal-validity concern: if the preset list happened to be poorly matched to the study tasks, the observed dynamic advantage would be an artifact; future work should validate static option sets against expert-derived alternatives.
  • The reported difficulty of predicting option effects suggests that dynamically generated controls could be paired with XAI-style transparency, such as showing exactly how each option changed the prompt or the response, which the paper's own design implications begin to outline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper addresses the challenge of prompting generative AI for comprehension tasks (explaining spreadsheet formulas, code, text) by proposing 'Dynamic Prompt Refinement Control' (Dynamic PRC), which uses an LLM-based Option Module to generate task-specific UI controls for refining prompts, compared with 'Static Prompt Refinement Control' (Static PRC), which offers a fixed preset list of controls. The paper reports a formative survey (n=38) that motivates two design goals (direct control and adaptive/dynamic control), a within-subjects user study (n=16) across six comprehension tasks, and qualitative interviews. The main results are a stated participant preference for Dynamic PRC, higher perceived effectiveness of Dynamic PRC for controlling AI output, and several qualitative themes: Dynamic PRC lowers barriers to providing context, improves perceived control, offers guidance, and encourages exploration/reflection, while making reasoning about the effects of controls more difficult. The paper concludes with design implications for future dynamic prompt middleware systems.

Significance. If the quantitative claims are robust, the paper makes a useful contribution to HCI research on generative AI interaction by articulating a new design space (dynamic prompt middleware), providing a concrete implementation with full system prompts in appendices, and offering qualitative findings that are well supported by participant quotes and coherent thematic analysis. The design implications in Section 7 (control over how options are applied, leveraging user context, direct manipulation) are actionable and plausible. The paper explicitly frames its design goals as aligning with prior work on controllable and adaptable explanations, which is a strength. However, the central quantitative claim of superiority for Dynamic PRC is currently undermined by an inappropriate statistical test for paired data and by a confounded baseline condition, so the paper's main evidence rests on qualitative insights and descriptive statistics.

major comments (4)
  1. [§5.3, §6.2.1] The manuscript reports Mann-Whitney U tests for the within-subjects comparison of Dynamic PRC and Static PRC. This is statistically inappropriate because each participant contributed data to both conditions, so the observations are paired, not independent. The correct test is the Wilcoxon signed-rank test (or a paired t-test if assumptions hold). The reported significant results for effectiveness (U=73.5, p=0.0245) and for needing more control (U=190.5, p=0.0174) may not remain significant under a paired analysis. Notably, the median effectiveness rating is identical (6.0 vs 6.0) in the two conditions, and the significance rests on distributional differences; a paired test is needed to establish whether the within-subject direction is consistent. Because these p-values are the only quantitative support for the claim that Dynamic PRC is 'significantly more effective,' this is a load-bearing issue.
  2. [§4.3, §6.2.2] The Static PRC baseline is confounded with the content of the controls. Section 4.3 states that the Static PRC preset list was derived from the formative survey but excludes options that 'required Dynamic PRC generation or interpretation' and keeps only 'generally applicable' controls; Appendix A.3 confirms that the list contains six generic controls (expertise, length, role, type, start, tone). Dynamic PRC, in contrast, generates 3-5 task-specific controls per prompt. Thus the experiment varies both the generation mechanism (static vs dynamic) and the content (generic vs task-adapted). The observed preference for Dynamic PRC may reflect the poor fit of the generic preset list rather than the value of dynamic generation. This is highlighted by the Task 4 result in Section 6.2.2, where Static PRC significantly outperformed Dynamic PRC (U=12.0, p=0.0355), suggesting that the static list can be adequate for some tasks. Without a Static PRC-Plus condition with a richer, task-specific preset list (or at least evidence that the six generic controls are applicable to the six study tasks), the central claim that Dynamic PRC is preferred because of dynamic generation is not fully supported.
  3. [§6.2.2] The one significant task-level result (Task 4, Dynamic PRC worse than Static PRC, U=12.0, p=0.0355) is reported without correction for multiple comparisons across six tasks and multiple questionnaire items. With this many tests, one significant result at p≈0.036 is expected by chance. The interpretation that 'the options provided in Static PRC are an effective set of refinements for code comprehension tasks' goes beyond what the data can support. The authors should either apply a multiple-comparison correction (e.g., Bonferroni or FDR) or explicitly label these as exploratory and refrain from task-specific claims.
  4. [§6.1.1] The key outcome of the study — participant preference for Dynamic PRC over Static PRC (median=2.0, mean=2.81 on a 1-7 scale where 1 is Dynamic) — is reported descriptively without any inferential test. Given that the abstract and conclusion state that 'Results show a preference for the Dynamic PRC approach,' this central claim should be supported by a statistical test, e.g., a Wilcoxon signed-rank test against the neutral value (4) or a sign test on the paired preference direction. As written, the preference claim rests on descriptive statistics and is not backed by significance testing.
minor comments (6)
  1. [§5.2, Task B1] The Python code snippet contains stray closing braces (e.g., 'import pandas as pd }' and similar after each line) that appear to be formatting artifacts; these should be corrected for clarity.
  2. [§6.2.1] The text states that Dynamic PRC was 'significantly more effective' than Static PRC, but the medians are identical (6.0 vs 6.0) and the difference appears only in the means and the (inappropriate) U test. Please clarify that the result is distributional and not a median shift.
  3. [§5.3] The protocol description says 'We performed and report Mann-Whitney U Tests'; for a within-subjects design this should be a paired test. If the authors intend to retain Mann-Whitney U, they need to justify the independence assumption, which is not met here.
  4. [§6.3] The terms 'mental load,' 'mental demand,' and 'cognitive effort' are used somewhat interchangeably; for consistency with the NASA-TLX instrument, the authors should align their terminology.
  5. [§6.4.2] The sentence in §6.1.1 that 'the affordances that Dynamic PRC brought ... led to participants responding that they preferred' uses causal language that may overstate what self-report data can establish; suggest tempering to 'participants reported preferring.'
  6. [§6.4] The thematic analysis mentions 'negotiated-agreement [37, 51]' but does not report inter-rater reliability statistics (e.g., Cohen's kappa); including a brief reliability measure would strengthen the qualitative claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims rest on a controlled empirical user study and independent design-goal elicitation, not on definitions, fitted parameters, or load-bearing self-citations.

full rationale

The paper's central claims are empirical: a formative survey (n=38) elicits design goals D.1 and D.2, and a within-subjects user study (n=16) measures preferences and perceived effectiveness. These results are self-reports and behavioral observations, not derivations from the system's definition. The Static PRC option list is specified independently in Appendix A.3, and the Dynamic PRC generation prompt is shown in Appendix A.1; neither is constructed to force the reported outcome. The design goals are grounded in the formative survey and are noted to align with an external paper [65], not with an unverified self-citation. Author self-citations such as [11] and [34] are used for background context about prompting challenges and do not carry the load of the main conclusion. There are no fitted parameters, no 'predictions' of data used to build the system, and no imported uniqueness theorem. The acknowledged limitation about not comparing against a baseline chat condition and the potential that Static PRC's preset list was not task-optimized are validity threats, not circularity. Thus no circular step can be exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its central claim rests on domain assumptions about task representativeness, self-report validity, the fairness of the static baseline, and the representativeness of the specific LLM implementation.

assumptions (4)
  • domain assumption Comprehension and learning tasks are a representative and ecologically valid class of GenAI use.
    The paper justifies via prevalence and diversity (Section 1), but the study tasks are limited to spreadsheet formulas, code, text summarization, and data-analysis concepts (Section 5.2).
  • domain assumption Self-reported Likert ratings and think-aloud comments are valid measures of user control, understanding, and mental load.
    The central claim relies on subjective scales in Sections 6.1 to 6.3; no objective performance measure of control is used.
  • domain assumption The two systems differ only in dynamic versus static option generation, with all other elements identical.
    The authors state every other element is identical (Section 5.3), but the static option list could interact with task choice, making the baseline comparison less clean.
  • domain assumption gpt4-turbo with the provided system prompts is a reliable and representative instantiation of dynamic prompt middleware.
    System prompts in Appendix A.1 were iterated until reliable; results may not generalize to other LLMs or prompt formulations, as the authors note in Section 7.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Prompt Middleware: Contextual Prompt Refinement Controls for Comprehension Tasks." pith.science (2026). https://pith.science/paper/U763277X

@misc{pith2026241202357,
  author       = {Pith},
  title        = {Pith review of: Dynamic Prompt Middleware: Contextual Prompt Refinement Controls for Comprehension Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U763277X}},
  note         = {Machine review of arXiv:2412.02357}
}
read the original abstract

Effective prompting of generative AI is challenging for many users, particularly in expressing context for comprehension tasks such as explaining spreadsheet formulas, Python code, and text passages. Prompt middleware aims to address this barrier by assisting in prompt construction, but barriers remain for users in expressing adequate control so that they can receive AI-responses that match their preferences. We conduct a formative survey (n=38) investigating user needs for control over AI-generated explanations in comprehension tasks, which uncovers a trade-off between standardized but predictable support for prompting, and adaptive but unpredictable support tailored to the user and task. To explore this trade-off, we implement two prompt middleware approaches: Dynamic Prompt Refinement Control (Dynamic PRC) and Static Prompt Refinement Control (Static PRC). The Dynamic PRC approach generates context-specific UI elements that provide prompt refinements based on the user's prompt and user needs from the AI, while the Static PRC approach offers a preset list of generally applicable refinements. We evaluate these two approaches with a controlled user study (n=16) to assess the impact of these approaches on user control of AI responses for crafting better explanations. Results show a preference for the Dynamic PRC approach as it afforded more control, lowered barriers to providing context, and encouraged exploration and reflection of the tasks, but that reasoning about the effects of different generated controls on the final output remains challenging. Drawing on participant feedback, we discuss design implications for future Dynamic PRC systems that enhance user control of AI responses. Our findings suggest that dynamic prompt middleware can improve the user experience of generative AI workflows by affording greater control and guide users to a better AI response.

Figures

Figures reproduced from arXiv: 2412.02357 by the authors.

Figure 1
Figure 1. Schematic overview of Dynamic Prompt Refinement Control interface for generating inline prompt refinement options to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. One frame of the control interface presented to users as part of the formative design probe to collect feedback for a final [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. User flow with the Dynamic PRC system. (1) User submits their prompt. (2) The Option Module generates a set of options to [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Prompt options model organizes the explanation into discrete steps which the user finds preferable to the previous response (6). Using the newly added session options, the Chat Module regenerates the response (7). 4.2 Two-tier Dynamic Prompt Refinement Control We devel…
Figure 5
Figure 5. Figure 5: Dynamic PRC overall system flow. Inline: (1) The Option Module takes the user’s prompt input and conversation history. (2) The Option Module returns a set of prompt options with initial values based on the prompt. (3) Prompt options are rendered inline using a renderin…
Figure 6
Figure 6. Figure 6: Static PRC model [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Static PRC options available during the study. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Task-sets A and B PRC against Static PRC to better understand the effectiveness of different approaches of control for users, and the trade-offs of each. Each user study took approximately 90 minutes. Participants were assigned Dynamic PRC and Static PRC conditions and…
Figure 9
Figure 9. Figure 9: Overall design of the study. options helped them get an explanation that helps them understand the task, and rate how confident they would be to answer questions about the task with the explanation in hand (reported in Section 6.2.2, Appendix B.1). After a task-set was…
Figure 10
Figure 10. Figure 10: Overall direct comparison of conditions for participant tool preference, mental demand, feelings of being rushed, successful, [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Participant reported preferences around control from Strongly Disagree to Strongly Agree. Sub-graph A. captures participant [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Participant rated effectiveness of each condition in the control of AI responses for the comprehension tasks they saw. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Measurement of how effective the options were in each condition at helping the participant control the response and assist [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Participant rated mental load for each condition, rated after the completion of a task set. [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: PRC exploration support can be improved by generating diffs of the various option selections for users. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: High fidelity prototype showing richer PRC modalities like interactive maps to choose EV range and visual representations of [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 44 canonical work pages

  1. [1]

    Dynamic Abstractions Workshop: Building the Next Generation of Cognitive Tools and Interfaces

    2024. Dynamic Abstractions Workshop: Building the Next Generation of Cognitive Tools and Interfaces. UIST’24 Workshop, Pittsburgh, USA, October 13-16, 2024. https://dynamic-uist24.vercel.app/ Organized by Sang Ho Suh, Hai Dang, Ryan Yen, Josh Pollock, Ian Arawjo, Rubaiat Habib, Nazmus Saquib, Hari Subramonyam, Jingyi, and Arvind Satyanarayan

    work page 2024

  2. [2]

    Bernstein, Joon Sung Park, Meredith Ringel Morris, Saleema Amershi, Lydia Chilton, and Mitchell L

    Michael S. Bernstein, Joon Sung Park, Meredith Ringel Morris, Saleema Amershi, Lydia Chilton, and Mitchell L. Gordon. 2023. Architecting Novel Interactions With Generative AI Models. In In the 35th Annual ACM Symposium on User Interface Software and Technology Workshop (UIST Workshop Manuscript submitted to ACM 28 Ian Drosos, Jack Williams, Advait Sarkar,...

    work page 2023

  3. [3]

    Michelle Brachman, Amina El-Ashry, Casey Dugan, and Werner Geyer. 2024. How Knowledge Workers Use and Want to Use LLMs in an Enterprise Context. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems . 1–8

    work page 2024

  4. [4]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (01 2006), 77–101. https://doi.org/10.1191/1478088706qp063oa

    doi:10.1191/1478088706qp063oa 2006

  5. [5]

    Daniel Buschek. 2024. Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools. In Proceedings of the 2024 ACM Designing Interactive Systems Conference . 2719–2737

    2024

  6. [6]

    Yiru Chen, Ryan Li, Austin Mac, Tianbao Xie, Tao Yu, and Eugene Wu. 2022. Nl2interface: Interactive visualization interface generation from natural language queries. arXiv preprint arXiv:2209.08834 (2022)

    arXiv 2022

  7. [7]

    Yiru Chen and Eugene Wu. 2022. Pi2: End-to-end interactive visualization interface generation from queries. In Proceedings of the 2022 International Conference on Management of Data . 1711–1725

    work page 2022

  8. [8]

    Ruijia Cheng, Titus Barik, Alan Leung, Fred Hohman, and Jeffrey Nichols. 2024. BISCUIT: Scaffolding LLM-Generated Code with Ephemeral UIs in Computational Notebooks. arXiv:2404.07387 [cs.HC] https://arxiv.org/abs/2404.07387

    arXiv 2024

Show all 76 references
  1. [9]

    Dhole, Ramraj Chandradevan, and Eugene Agichtein

    Kaustubh D. Dhole, Ramraj Chandradevan, and Eugene Agichtein. 2023. An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback. arXiv:2311.11226 [cs.AI] https://arxiv.org/abs/2311.11226

    2023 arXiv

  2. [10]

    Victor Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Danushka Bol...

    2023 doi

  3. [11]

    It’s like a rubber duck that talks back

    Ian Drosos, Advait Sarkar, Xiaotong Xu, Carina Negreanu, Sean Rintel, and Lev Tankelevitch. 2024. "It’s like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study. In Proceedings of the 3rd Annual M...

    2024

  4. [12]

    Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al. 2023. Explainable AI (XAI): Core ideas, techniques, and solutions. Comput. Surveys 55, 9 (2023), 1–33

    2023

  5. [13]

    Upol Ehsan, Q Vera Liao, Michael Muller, Mark O Riedl, and Justin D Weisz. 2021. Expanding explainability: Towards social transparency in ai systems. In Proceedings of the 2021 CHI conference on human factors in computing systems . 1–19

    2021

  6. [14]

    Upol Ehsan, Q Vera Liao, Samir Passi, Mark O Riedl, and Hal Daumé III. 2024. Seamful XAI: Operationalizing Seamful Design in Explainable AI. Proceedings of the ACM on Human-Computer Interaction 8, CSCW1 (2024), 1–29

    2024

  7. [15]

    Upol Ehsan, Samir Passi, Q Vera Liao, Larry Chan, I Lee, Michael Muller, Mark O Riedl, et al. 2021. The who in explainable ai: How ai background shapes perceptions of ai explanations. arXiv preprint arXiv:2107.13509 (2021)

    2021 arXiv

  8. [16]

    Upol Ehsan and Mark O Riedl. 2020. Human-centered explainable ai: Towards a reflective sociotechnical approach. In HCI International 2020-Late Breaking Papers: Multimodality and Intelligence: 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, ...

    2020

  9. [17]

    Upol Ehsan and Mark O Riedl. 2024. Explainability pitfalls: Beyond dark patterns in explainable AI. Patterns 5, 6 (2024)

    2024

  10. [18]

    Upol Ehsan, Koustuv Saha, Munmun De Choudhury, and Mark O Riedl. 2023. Charting the sociotechnical gap in explainable ai: A framework to address the gap in xai. Proceedings of the ACM on human-computer interaction 7, CSCW1 (2023), 1–32

    2023

  11. [19]

    Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Automated rationale generation: a technique for explainable AI and its effects on human perceptions. In Proceedings of the 24th international conference on intelligent user interfaces . 263–274

    2019

  12. [20]

    Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Martina Mara, Marc Streit, Sandra Wachter, Andreas Riener, and Mark O Riedl. 2021. Operationalizing human-centered perspectives in explainable AI. In Extended abstracts of the 2021 CHI conference on human factors in computing sys...

    2021

  13. [21]

    Upol Ehsan, Philipp Wintersberger, Q Vera Liao, Elizabeth Anne Watkins, Carina Manger, Hal Daumé III, Andreas Riener, and Mark O Riedl. 2022. Human-Centered Explainable AI (HCXAI): beyond opening the black-box of AI. In CHI conference on human factors in computing systems exte...

    2022

  14. [22]

    Facebook. [n. d.]. React - The library for web and native user interfaces. https://github.com/facebook/react. Accessed: 2024-09-27

    2024

  15. [23]

    Gordon, Carina Negreanu, Nadia Polikarpova, Advait Sarkar, and Benjamin Zorn

    Kasra Ferdowsi, Jack Williams, Ian Drosos, Andrew D. Gordon, Carina Negreanu, Nadia Polikarpova, Advait Sarkar, and Benjamin Zorn. 2023. COLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computi...

    2023

  16. [24]

    Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (Funchal, Madeira, Portugal) (IUI ’04). Association for Computing Machinery, New York, NY, USA, 93–100....

    2004

  17. [25]

    Krzysztof Z Gajos, Jacob O Wobbrock, and Daniel S Weld. 2007. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th annual ACM symposium on User interface software and technology . 231–240

    2007

  18. [26]

    Randy Goebel, Ajay Chander, Katharina Holzinger, Freddy Lecue, Zeynep Akata, Simone Stumpf, Peter Kieseberg, and Andreas Holzinger. 2018. Explainable AI: the new 42?. In International cross-domain conference for machine learning and knowledge extraction . Springer, 295–303

    2018

  19. [27]

    David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. 2019. XAI—Explainable artificial intelligence. Science robotics 4, 37 (2019), eaay7120. Manuscript submitted to ACM Dynamic Prompt Middleware: Contextual Prompt Refinement Controls fo...

    2019

  20. [28]

    Hart and Lowell E

    Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https: //do...

    1988 doi

  21. [29]

    Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. 2018. Metrics for explainable AI: Challenges and prospects. arXiv preprint arXiv:1812.04608 (2018)

    2018 arXiv

  22. [30]

    Andreas Holzinger, Anna Saranti, Christoph Molnar, Przemyslaw Biecek, and Wojciech Samek. 2022. Explainable AI methods-a brief overview. In International workshop on extending explainable AI beyond deep models and classifiers . Springer, 13–38

    2022

  23. [31]

    The IEEE and The Open Group. [n. d.]. diff. The Open Group Base Specifications Issue 8. https://pubs.opengroup.org/onlinepubs/9699919799/ utilities/diff.html. Accessed: 2024-09-27

    2024

  24. [32]

    Jovanovic

    B. Jovanovic. 2024. Generative UI: The future of dynamic user experiences . https://bootcamp.uxdesign.cc/generative-ui-the-future-of-dynamic-user- experiences [Accessed September-2024]

    2024

  25. [33]

    Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition. arXiv preprint arXiv:2407.02651 (2024)

    2024 arXiv

  26. [34]

    What It Wants Me To Say

    Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. “What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In Proceedings of the 20...

    2023

  27. [35]

    Xiao Ma, Swaroop Mishra, Ariel Liu, Sophie Ying Su, Jilin Chen, Chinmay Kulkarni, Heng-Tze Cheng, Quoc Le, and Ed Chi. 2024. Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responses. In Extended Abstracts of the 2024 CHI Conference on Human Factors ...

    2024 doi

  28. [36]

    Stephen MacNeil, Andrew Tran, Joanne Kim, Ziheng Huang, Seth Bernstein, and Dan Mogil. 2023. Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances. arXiv:2307.01142 [cs.HC] https://arxiv.org/abs/2307.01142

    2023 arXiv

  29. [37]

    Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 72 (nov 2019), 23 pages. https://doi.org/10.1145/3359174

    2019 doi

  30. [38]

    Microsoft. [n. d.]. TypeScript. https://github.com/microsoft/typescript. Accessed: 2024-09-27

    2024

  31. [39]

    Microsoft. 2024. Microsoft Copilot. https://copilot.microsoft.com/. Accessed: 2024-09-26

    2024

  32. [40]

    Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence 267 (2019), 1–38

    2019

  33. [41]

    Kate Moran and Sarah Gibbons. 2024. Generative UI and Outcome-Oriented Design. https://www.nngroup.com/articles/generative-ui/ Accessed: 2024-10-01

    2024

  34. [42]

    Moran and S

    K. Moran and S. Gibbons. 2024. Is UI dead? how conversational models are taking over . https://www.nngroup.com/articles/generative-ui/ [Accessed 25-September-2024]

    2024

  35. [43]

    Jeffrey Nichols, Brad A Myers, Michael Higgins, Joseph Hughes, Thomas K Harris, Roni Rosenfeld, and Kevin Litwack. 2003. Personal universal controllers: controlling complex appliances with GUIs and speech. In CHI’03 Extended Abstracts on Human Factors in Computing Systems . 624–625

    2003

  36. [44]

    Jeffrey Nichols, Brad A Myers, and Brandon Rothrock. 2006. UNIFORM: automatically generating consistent remote control user interfaces. In Proceedings of the SIGCHI conference on Human Factors in computing systems . 611–620

    2006

  37. [45]

    Jeffrey Nichols, Brandon Rothrock, Duen Horng Chau, and Brad A Myers. 2006. Huddle: automatically generating interfaces for systems of multiple connected appliances. In Proceedings of the 19th annual ACM symposium on User interface software and technology . 279–288

    2006

  38. [46]

    OpenAI. 2024. Structured Outputs. https://platform.openai.com/docs/guides/structured-outputs/supported-schemas. Accessed: 2024-09-26

    2024

  39. [47]

    Tomas Petricek. 2022. No-code, no thought? substrates for simple programming for all. https://tomasp.net/blog/2022/no-code-substrates/

    2022

  40. [48]

    Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis , Vol. 5. McLean, VA, USA, 2–4

    2005

  41. [49]

    Ivan Poupyrev. 2023. The Ultimate Interface. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23 Adjunct). Association for Computing Machinery, New York, NY, USA, Article 111, 2 pages. https://doi...

    2023

  42. [50]

    Microsoft Research. 2024. Tools for Thought. https://www.microsoft.com/en-us/research/project/t4t/. Accessed: 2024-09-26

    2024

  43. [51]

    J. Saldana. 2021. The Coding Manual for Qualitative Researchers . SAGE Publications. https://books.google.co.uk/books?id=RwcVEAAAQBAJ

    2021

  44. [52]

    Advait Sarkar. 2022. Is explainable AI a race against model complexity?. In Workshop on Transparency and Explanations in Smart Systems (TeXSS), in conjunction with ACM Intelligent User Interfaces (IUI 2022) (CEUR Workshop Proceedings, 3124) . 192–199. http://ceur-ws.org/Vol-31...

    2022

  45. [53]

    Advait Sarkar. 2023. Exploring Perspectives on the Impact of Artificial Intelligence on the Creativity of Knowledge Work: Beyond Mechanised Plagiarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (Olde...

    2023

  46. [54]

    Advait Sarkar. 2023. Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?. In Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Cascais, Portugal) (Onw...

    2023

  47. [55]

    Advait Sarkar. 2024. AI Should Challenge, Not Obey. Commun. ACM (Sept. 2024), 5 pages. https://doi.org/10.1145/3649404 Online First. Manuscript submitted to ACM 30 Ian Drosos, Jack Williams, Advait Sarkar, and Nicholas Wilson

    2024 doi

  48. [56]

    Advait Sarkar. 2024. Intention Is All You Need. In Proceedings of the 35th Annual Conference of the Psychology of Programming Interest Group (PPIG 2024)

    2024

  49. [57]

    Advait Sarkar. 2024. Large Language Models Cannot Explain Themselves. In Proceedings of the ACM CHI 2024 Workshop on Human-Centered Explainable AI (Honolulu, HI, USA) (HCXAI at CHI ’24) . https://doi.org/10.48550/arXiv.2405.04382

    2024 doi arXiv

  50. [58]

    Advait Sarkar, Xiaotong (Tone) Xu, Neil Toronto, Ian Drosos, and Christian Poelitz. 2024. When Copilot Becomes Autopilot: Generative AI’s Critical Risk to Knowledge Work and a Critical Solution. In Proceedings of the Annual Conference of the European Spreadsheet Risks Interest...

    2024

  51. [59]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling multilevel exploration and sensemaking with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1–18

    2023

  52. [60]

    Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The Metacognitive Demands and Opportunities of Generative AI. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, ...

    2024

  53. [61]

    Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas. 2013. Content analysis and thematic analysis: Implications for con- ducting a qualitative descriptive study. Nursing & Health Sciences 15, 3 (2013), 398–405. https://doi.org/10.1111/nhs.12048 arXiv:https://onlinelibrary.w...

    2013 doi

  54. [62]

    Glassman, Jeevana Priya Inala, and Chenglong Wang

    Priyan Vaithilingam, Elena L. Glassman, Jeevana Priya Inala, and Chenglong Wang. 2024. DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association f...

    2024

  55. [63]

    Priyan Vaithilingam and Philip J Guo. 2019. Bespoke: Interactively synthesizing custom GUIs from command-line applications by demonstration. In Proceedings of the 32nd annual ACM symposium on user interface software and technology . 563–576

    2019

  56. [64]

    Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. 2019. Explainable AI: A brief survey on history, research areas, approaches and challenges. In Natural language processing and Chinese computing: 8th cCF international conference, NLPCC 2019, dunhuang, ...

    2019

  57. [65]

    Litao Yan, Alyssa Hwang, Zhiyuan Wu, and Andrew Head. 2024. Ivie: Lightweight Anchored Explanations of Just-Generated Code. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New Y...

    2024

  58. [66]

    JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–21

    2023

  59. [67]

    m ult i − s e l e c t− checkbox

    Haoci Zhang, Viraj Raj, Thibault Sellam, and Eugene Wu. 2018. Precision interfaces for different modalities. In Proceedings of the 2018 International Conference on Management of Data . 1777–1780. A System Prompts A.1 Option Module c o n s t i n s t r u c t i o n s : s t r i n ...

    2018

  60. [68]

    Rate how much the options helped you in getting an explanation that helps you understand the task (1-7 Lowest to Highest)

  61. [69]

    How confident are you that you can answer questions about this task with this explanation? (i.e., how much does this explanation help you?) (1-7 Lowest to Highest)

  62. [70]

    B.2 Post-Condition Questionnaire

    What other options do you think would be useful? [Free-Text]. B.2 Post-Condition Questionnaire

  63. [71]

    (2) The controls were useful at helping me better understand the concepts involved in the tasks

    Answer each statement with your level of agreement (From Strongly Disagree to Strongly Agree): (1) The controls were effective at helping me control AI output and explanations. (2) The controls were useful at helping me better understand the concepts involved in the tasks. Man...

  64. [72]

    (2) The AI understood my intent and made the right edits

    Answer each statement with your level of agreement (From Strongly Disagree to Strongly Agree): (1) It was easy to complete the tasks using the tool provided. (2) The AI understood my intent and made the right edits

  65. [73]

    Rate how successful you were in the tasks you saw (From Perfect to Failure): (1) How successful would you rate yourself in accomplishing this task?

  66. [74]

    Rate your experience during the tasks (From Very Low to Very High): (1) How mentally demanding was this task with this tool? (2) How hurried or rushed were you during this task? (3) How hard did you have to work to accomplish your level of performance? (4) How insecure, discou...

  67. [75]

    Compare each tool (A and B) for the below statements and choose your preference (From 1 - A to 7 - B):. (1) Which tool would you prefer to use? (2) Which tool was more mentally demanding to communicate? (3) Which tool made you feel hurried or rushed during the task? (4) Which ...

  68. [76]

    (2) Learning concepts the AI has used in its responses is important to me

    Rate your agreement to the statements below (From Strongly Disagree to Strongly Agree): (1) Control over AI-generated explanations is important to me. (2) Learning concepts the AI has used in its responses is important to me. (3) My needs around AI explanations change based on...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.