Pith. sign in

REVIEW 4 major objections 4 minor 58 references

OnGoal shows that embedding LLM-run goal tracking into a linear chat interface lowers the effort users spend evaluating and reviewing their conversational goals, while encouraging new prompting strategies.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

OnGoal is an LLM chat interface that infers, merges, and evaluates user goals in real time and visualizes their progress, tested with 20 users on a writing task.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A well-built interface and thoughtful qualitative study undercut by an inverted significance test and an acknowledged anchoring confound; the headline 'less time and effort' claim does not survive their own data. the 4 major comments →

arxiv 2508.21061 v1 pith:DRE6DYEH submitted 2025-08-28 cs.HC cs.AIcs.LG

OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models

Adam Coscia , Shunan Guo , Eunyee Koh , Alex Endert This is my paper
classification cs.HC cs.AIcs.LG
keywords conversational goalsgoal trackingLLM chat interfacevisualizationmulti-turn dialoguesensemakinguser studywriting task
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the familiar linear chat interface can be extended to help users manage evolving conversational goals, and that doing so pays off. It presents OnGoal, a chat interface with an LLM-run pipeline that infers goals from user messages, merges or replaces them as the conversation evolves, and evaluates whether each LLM response confirms, contradicts, or ignores each goal, with explanations and quoted examples. In a 20-participant writing study against a plain chat baseline, OnGoal users spent less effort reading and reviewing, reported lower mental demand, and spent more time deliberately reviewing goal progress during validation. The result matters because long multi-turn LLM conversations are common and hard to navigate, and most existing tools either restructure the chat or analyze it after the fact rather than surfacing the user's own goals in real time.

Core claim

The paper's central claim is that tracking and visualizing conversational goals inside a linear chat interface improves users' ability to evaluate and review goal progress in multi-turn dialogue with LLMs. OnGoal runs a three-stage goal pipeline—infer, merge, evaluate—using a generative LLM to parse the user's questions, requests, offers, and suggestions, consolidate them over turns, and judge each LLM response as confirming, contradicting, or ignoring each goal. Those judgments are shown as colored goal glyphs under messages, as explanations with verbatim example phrases, and as timeline and event summaries in a side panel. In the study, users of OnGoal self-reported lower effort and mental

What carries the argument

The load-bearing mechanism is the LLM-assisted conversational goal pipeline: three prompted stages that (1) infer goals from user messages by classifying clauses as questions, requests, offers, or suggestions; (2) merge newly inferred goals with the existing list by keeping, combining, or replacing them; and (3) evaluate the latest assistant response against each merged goal as confirm, contradict, or ignore, with a one-sentence explanation and verbatim examples. Around this pipeline, OnGoal renders goal glyphs under each message, a progress panel with goals, timeline (Sankey-style), and events tabs, an individual-goal drill-down, and text highlighting for key phrases, similar sentences, and

Load-bearing premise

The study's key results are self-reported effort and confidence, and the paper itself notes that higher agreement with the system 'is likely due to the influence of seeing the evaluation on the screen'; if users were simply anchoring to visible judgments, the benefits would not be genuine. Also, the GPT-4o goal pipeline was never checked against expert-annotated goal states.

What would settle it

Run the same writing study with the system's evaluations hidden until after the participant answers, or compare the pipeline's evaluations against expert judgments on the same six-goal writing tasks; if effort and confidence gains disappear or pipeline accuracy is near chance, the central claim weakens.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • OnGoal users spent less self-reported effort reading and reviewing the chat, with lower mental demand, despite spending slightly more time actively reviewing during validation.
  • Users with goal feedback explored more prompting strategies to overcome miscommunication, rather than repeating the same prompt.
  • Users felt more confident evaluating goals and agreed more often with the system's goal evaluations; when they disagreed, confidence dropped.
  • Design implications: offer multiple ways to communicate goals, visualize where evaluations align with user focus, provide goal alerts and progress snapshots, and let users give feedback on goal evaluations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If part of the benefit is anchoring on visible evaluations, then a variant that hides or delays the system's verdict could separate genuine sensemaking gains from mere agreement; this is testable in a follow-up study.
  • The pipeline's goal ontology (questions, requests, offers, suggestions) is task-shaped; porting OnGoal to programming, data analysis, or learning would require re-inferring goal types, and the paper's design implications suggest local, fine-grained goals may be needed for complex artifacts.
  • Longitudinal or open-ended settings may favor the timeline and events views, which scored lowest in this fixed-goal writing task; the paper leaves open whether those views pay off when goals are self-created and evolve freely.
  • User feedback on evaluations—thumbs up/down or editable evaluations—could personalize the pipeline over time, but the paper only points to this direction without testing it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces OnGoal, an interface that augments a linear LLM chat with a three-stage goal pipeline (infer, merge, evaluate) driven by GPT-4o, plus inline goal glyphs, a progress panel with goals/timeline/events tabs, an individual goal view, and text-highlighting techniques. The design is motivated by three challenges around evaluating and reviewing conversational goals in multi-turn dialogue. The authors evaluate OnGoal against a baseline chat interface without goal tracking or visualizations in a between-subjects study (N=20) on a two-boss writing task. They report that OnGoal users spent less time reading but more time evaluating and reviewing, self-reported lower effort and mental demand, felt more confident and more often agreed with the system's evaluations, and adopted more varied prompting strategies. The paper synthesizes these findings into design implications for goal communication, cognitive load, interactivity, and feedback in LLM chat interfaces.

Significance. The system addresses a genuine and timely problem—helping users monitor evolving conversational goals in long, multi-turn LLM dialogues—and the design implications are potentially useful to the UIST community. The implementation is transparent, with prompts included in the appendix, and the study uses think-aloud protocols and bootstrapped confidence intervals rather than relying solely on statistical tests. However, the central empirical claims are compromised by an acknowledged anchoring confound, by internally inconsistent statistical reporting, and by an abstract claim ('less time') that directly contradicts the paper's own time measurements. As submitted, the quantitative evaluation does not support the causal headline; the paper's credible contributions are the system design and the qualitative themes, not the measured performance claims.

major comments (4)
  1. [Abstract and §5.2.1, Fig. 6] The abstract states that 'participants spent less time and effort to achieve their goals,' but the reported times show OnGoal users spent slightly more time evaluating (B:29.8[20.9,38.6] vs O:34.1[22.9,45.3]) and reviewing (B:19.7[15,24.5] vs O:24.4[17.6,31.2]), and notably more time validating reviewing (B:16.8[12,21.7] vs O:30[21.2,38.9]). Total time across reading/evaluating/reviewing is not lower with OnGoal. The only reduction is reading time. This mismatch between the abstract and the data is load-bearing for the paper's central claim.
  2. [§5.1 'Measures and analysis' vs §5.2.1] The paper defines significance as non-overlapping 95% confidence intervals, then states 'we observed several significant differences (i.e., overlapping 95% confidence intervals).' This inverts the stated criterion. The misuse is not merely cosmetic: 'strong evidence' is assigned to overlapping intervals (e.g., reviewing effort B[2.5,3.9] vs O[1.7,2.6]) and even to a case where the intervals overlap slightly (validating review time B[12,21.7] vs O[21.2,38.9]). Consequently, the 'strong/weak/no evidence' labels throughout §5.2.1 are unreliable and should be recomputed under a consistent rule.
  3. [§5.1 Tasks and §5.2.1 'Interactions and answers'] The evaluation task asks participants to rate whether the LLM satisfied a goal and whether that goal was consistently addressed; in the OnGoal condition, the system's confirm/contradict/ignore evaluations and explanations are visible on screen before participants answer the same questions. The paper itself states, 'This is likely due to the influence of seeing the evaluation on the screen.' Therefore the higher agreement with the system and higher confidence when agreeing cannot be interpreted as improved user evaluation or review; it may reflect anchoring/conformity to the displayed judgment. The same mechanism could also explain lower self-reported effort and mental demand if users offload the judgment to the UI. This confound affects the central outcome measures and is acknowledged by the authors in the results section.
  4. [§6.2 and §5.2.1 'OnGoal pipeline accuracy'] The accuracy of the goal pipeline is untested: no expert-annotated benchmark was run, and users rated the evaluate stage significantly lower than the infer stage (2.9 vs 4.1 out of 5). Because the displayed evaluations are the intervention, their correctness is central to interpreting both the agreement results and the effort/confidence results. Without independent validation of the pipeline, the observed agreement between users and the system is uninterpretable as evidence of improved goal management. The paper should either provide such validation or explicitly limit its claims to perceived usefulness and qualitative experience.
minor comments (4)
  1. [Introduction] Typo: 'design design challenges' appears in the contributions list.
  2. [Figure 6 and Figure 7] Figure text contains misspellings ('baselin' for 'baseline') and informal annotations ('uncovering deeper insights!') that are inconsistent with a journal-style presentation.
  3. [§5.2.2] Typo: 'the the goal glyphs' should be 'the goal glyphs.' Also, use consistent capitalization of 'OnGoal' throughout (e.g., 'onGoal' appears in §5.2.2).
  4. [§5.1] The paper uses the phrase 'statistically significant' in places but reports only bootstrapped CIs; the terminology should be aligned with the stated CI-based approach, and the small per-condition sample (N=10) should be explicitly acknowledged as a limitation in the quantitative claims.

Circularity Check

1 steps flagged

Headline goal-evaluation benefit rests on user agreement with evaluations displayed on screen; the paper itself concedes the anchoring confound.

specific steps
  1. fitted input called prediction [Section 5.2.1, 'Interactions and answers' (Fig. 8)]
    "While both interfaces evaluated goals after each turn, evaluations were only visible to participants using OnGoal. We found participants tended to agree more with the system when using OnGoal... This is likely due to the influence of seeing the evaluation on the screen."

    Participants in OnGoal see the system's goal evaluation on screen before answering the same evaluate/review questions. The paper then treats agreement with that displayed evaluation as evidence that OnGoal improves goal evaluation/review. Agreement is therefore not an independent outcome: the reference answer is the stimulus the user just saw. The paper explicitly concedes this ('likely due to the influence of seeing the evaluation on the screen'). Since §6.2 also states the pipeline's accuracy is untested on expert-annotated benchmarks, agreement with an unvalidated displayed label cannot validate user evaluation quality. The reported lower effort/mental demand may likewise reflect offloading to the visible evaluation rather than improved goal management.

full rationale

This is an empirical HCI paper, not a mathematical derivation, so the circularity is not equation-level. The central quantitative support for the claim that OnGoal helps users 'better evaluate and review' their conversational goals is the user-agreement result: users agreed more with the system when using OnGoal and were more confident when agreeing. But in the OnGoal condition the system's evaluations were displayed on screen before the user answered the same questions, making the agreement measure partially constructed by the interface itself. The paper acknowledges this anchoring confound. Additional evidence (qualitative themes, feature usefulness, reading-time reductions) is independent of this particular reduction, which is why the paper is not wholly circular. However, because the pipeline's evaluation accuracy is explicitly untested (§6.2), the agreement metric cannot serve as external validation either. No load-bearing self-citation or imported uniqueness theorem was found; the Adobe-affiliated related-work citations are not central to the claims. Score 6 reflects one central, partially construction-based circularity in the main outcome measure.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The system has no numeric fitted parameters, but the hand-crafted LLM prompts are de facto parameters. The axioms are the domain assumptions about goal definition, LLM reliability, measurement validity, and sample size.

free parameters (1)
  • Goal pipeline prompt templates = Hand-crafted GPT-4o prompts (Appendix A)
    The infer, merge, evaluate, and keyphrase stages are defined by these prompts. No quantitative tuning or benchmark validation is reported, so they act as hand-set parameters of the system.
axioms (4)
  • domain assumption Conversational goals can be defined as questions, requests, offers, or suggestions (Stivers et al. [42]).
    The entire goal pipeline is built on this definition (Section 4.1); other goal types are excluded.
  • domain assumption GPT-4o with the provided prompts accurately infers, merges, and evaluates goals.
    No expert-annotated benchmark is run; Section 6.2 admits pipeline accuracy is untested.
  • domain assumption Self-reported effort, mental demand, and agreement with the system are valid measures of goal-management performance.
    Section 5.2.1 uses these self-reports as outcome measures, but the visible system evaluations may anchor responses.
  • domain assumption Ten participants per condition in a between-subjects design is sufficient to support 'strong evidence' claims.
    Small sample with no pre-registration; the paper relies on bootstrap CIs but misapplies the non-overlap rule.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models." pith.science (2026). https://pith.science/paper/DRE6DYEH

@misc{pith2026250821061,
  author       = {Pith},
  title        = {Pith review of: OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRE6DYEH}},
  note         = {Machine review of arXiv:2508.21061}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As multi-turn dialogues with large language models (LLMs) grow longer and more complex, how can users better evaluate and review progress on their conversational goals? We present OnGoal, an LLM chat interface that helps users better manage goal progress. OnGoal provides real-time feedback on goal alignment through LLM-assisted evaluation, explanations for evaluation results with examples, and overviews of goal progression over time, enabling users to navigate complex dialogues more effectively. Through a study with 20 participants on a writing task, we evaluate OnGoal against a baseline chat interface without goal tracking. Using OnGoal, participants spent less time and effort to achieve their goals while exploring new prompting strategies to overcome miscommunication, suggesting tracking and visualizing goals can enhance engagement and resilience in LLM dialogues. Our findings inspired design implications for future LLM chat interfaces that improve goal communication, reduce cognitive load, enhance interactivity, and enable feedback to improve LLM performance.

Figures

Figures reproduced from arXiv: 2508.21061 by Adam Coscia, Alex Endert, Eunyee Koh, Shunan Guo.

Figure 1
Figure 1. Figure 1: OnGoal tracks and visualizes conversational goals such as requests and suggestions in multi-turn dialogue with LLMs, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example of the conversational goal pipeline in action. OnGoal uses a generative LLM (e.g., GPT-4o) and prompt [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The three tabs in the progress panel. In the goals tab (A), users can control goals by locking or completing them, create [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Clicking on a goal in the goals tab (Fig. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Usage scenario of using OnGoal to analyze a CSV dataset. After giving the LLM the dataset, the user iteratively forms [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Time spent reading messages, evaluating goals, and reviewing goals between interfaces and task phases. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Responses to the NASA Task Load Index (TLX) for [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Turns taken, goals confirmed / contradicted / ignored, goal variability, and final answers to the initial and validation [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Usefulness ratings for various OnGoal features. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 27 canonical work pages · 1 internal anchor

  1. [1]

    Glassman

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L. Glassman. 2024. ChainForge: A Visual Toolkit for Prompt Engineer- ing and LLM Hypothesis Testing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Asso- ciation for Computing Machinery, New York, NY, USA, Article 3...

    arXiv 2024

  2. [2]

    Trevor Ashby, Adithya Kulkarni, Jingyuan Qi, Minqian Liu, Eunah Cho, Vaibhav Kumar, and Lifu Huang. 2024. Towards Effective Long Conversation Genera- tion with Dynamic Topic Tracking and Recommendation. In Proceedings of the 17th International Natural Language Generation Conference , Saad Mahamood, Nguyen Le Minh, and Daphne Ippolito (Eds.). Association f...

    work page 2024

  3. [3]

    Jeff Baker, Donald Jones, and Jim Burkman. 2009. Using visual representations of data to enhance sensemaking in data exploration tasks. Journal of the Association for Information Systems 10, 7 (2009), 2

    work page 2009

  4. [4]

    Elizabeth L Bjork, Robert A Bjork, et al. 2011. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society 2, 59-68 (2011)

    2011

  5. [5]

    Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D 3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2301–2309. doi:10.1109/TVCG.2011.185

    doi:10.1109/tvcg.2011.185 2011

  6. [6]

    Boyatzis

    R.E. Boyatzis. 1998. Transforming Qualitative Information: Thematic Analysis and Code Development. SAGE Publications

    work page 1998

  7. [7]

    Bursztyn, Jennifer Healey, Eunyee Koh, Nedim Lipka, and Larry Birn- baum

    Victor S. Bursztyn, Jennifer Healey, Eunyee Koh, Nedim Lipka, and Larry Birn- baum. 2021. Developing a Conversational Recommendation Systemfor Nav- igating Limited Options. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI EA ’21) . Asso- ciation for Computing Machinery, New York, NY, USA, Articl...

    work page arXiv 2021

  8. [8]

    Zixin Chen, Jiachen Wang, Meng Xia, Kento Shigyo, Dingdong Liu, Rong Zhang, and Huamin Qu. 2025. StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions. IEEE Transactions on Visualization and Computer Graphics 31, 1 (2025), 908–918. doi:10.1109/TVCG.2024.3456363

    arXiv 2025

  9. [9]

    Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review 54, 1 (01 Jan 2021), 755–810. doi:10.1007/s10462-020-09866-x

    doi:10.1007/s10462-020-09866-x 2021

  10. [10]

    Pierre Dragicevic. 2016. Fair Statistical Communication in HCI . Springer Interna- tional Publishing, Cham, 291–330. doi:10.1007/978-3-319-26633-6_13

    doi:10.1007/978-3-319-26633-6_13 2016

  11. [11]

    Haodong Duan, Jueqi Wei, Chonghua Wang, Hongwei Liu, Yixiao Fang, Songyang Zhang, Dahua Lin, and Kai Chen. 2023. Botchat: Evaluating llms’ capabilities of having multi-turn dialogues. arXiv preprint arXiv:2310.13650 (2023)

    Pith/arXiv arXiv 2023

  12. [12]

    K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data. the MIT Press

    work page 1984

  13. [13]

    Siwei Fu, Jian Zhao, Hao Fei Cheng, Haiyi Zhu, and Jennifer Marlow. 2018. T-Cal: Understanding Team Conversational Data with Calendar-based Visualization. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal, QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. doi:10.1145/3173574.3174074

    arXiv 2018

  14. [14]

    Jie Gao, Simret Araya Gebreegziabher, Kenny Tsu Wei Choo, Toby Jia-Jun Li, Simon Tangi Perrault, and Thomas W Malone. 2024. A Taxonomy for Human- LLM Interaction Modes: An Initial Exploration. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’24). Association for Computing Machinery, New Yo...

    arXiv 2024

  15. [15]

    Kummerfeld, and Elena L

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 838, 21 pages. doi:1...

    doi:10.1145/3613904 2024

  16. [16]

    Akash Gupta, Ivaxi Sheth, Vyas Raina, Mark Gales, and Mario Fritz. 2024. LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conver- sational History. In ICML 2024 Workshop on Foundation Models in the Wild . https://openreview.net/forum?id=WnMcWR9n3P

    work page 2024

  17. [17]

    Havre, B

    S. Havre, B. Hetzler, and L. Nowell. 2000. ThemeRiver: visualizing theme changes over time. In IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings. 115–123. doi:10.1109/INFVIS.2000.885098

    arXiv 2000

  18. [18]

    Hernandez-Bocanegra and Jürgen Ziegler

    Diana C. Hernandez-Bocanegra and Jürgen Ziegler. 2023. Explaining Recom- mendations through Conversations: Dialog Model and the Effects of Interface Type and Degree of Interactivity. ACM Trans. Interact. Intell. Syst. 13, 2, Article 6 (April 2023), 47 pages. doi:10.1145/3579541

    doi:10.1145/3579541 2023

  19. [19]

    Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2021. Integrated taxonomy of errors in chat-oriented dialogue sys- tems. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue , Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina D...

    work page doi:10.18653/v1/2021.sigdia 2021

  20. [20]

    Matt-Heun Hong and Anamaria Crisan. 2023. Conversational AI Threads for Visualizing Multidimensional Datasets. arXiv:2311.05590 [cs.HC] https://arxiv. org/abs/2311.05590

    Pith/arXiv arXiv 2023

  21. [21]

    Shih-Hong Huang, Ya-Fang Lin, Zeyu He, Chieh-Yang Huang, and Ting-Hao Ken- neth Huang. 2024. How Does Conversation Length Impact User’s Satisfaction? A Case Study of Length-Controlled Conversations with LLM-Powered Chatbots. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA ’24). Association for Computing Machi...

    arXiv 2024

  22. [23]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406 (2022)

    Pith/arXiv arXiv 2022

  23. [24]

    Kim, Rafael A

    Joshua Y. Kim, Rafael A. Calvo, N. J. Enfield, and Kalina Yacef. 2021. A Systematic Review on Dyadic Conversation Visualizations. In Companion Publication of the 2021 International Conference on Multimodal Interaction (Montreal, QC, Canada) (ICMI ’21 Companion) . Association for Computing Machinery, New York, NY, USA, 137–147. doi:10.1145/3461615.3485396

    work page arXiv 2021

  24. [25]

    Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2024. Understanding Users’ Dissatisfaction with ChatGPT Responses: Types, Resolv- ing Tactics, and the Effect of Knowledge Level. In Proceedings of the 29th In- ternational Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI ’24). Association for Computing Machinery, New Yor...

    arXiv 2024

  25. [26]

    Atharva Kulkarni, Bo-Hsiang Tseng, Joel Moniz, Dhivya Piraviperumal, Hong Yu, and Shruti Bhargava. 2024. SynthDST: Synthetic Data is All You Need for Few-Shot Dialog State Tracking. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 1988–2001

    work page 2024

  26. [27]

    Philippe Laban, Jesse Vig, Marti Hearst, Caiming Xiong, and Chien-Sheng Wu

  27. [28]

    Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson. 2025. The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems . A...

    doi:10.1145/3706598 2025

  28. [29]

    Andrew Li, Zhenduo Wang, Ethan Mendes, Duong Minh Le, Wei Xu, and Alan Ritter. 2024. ChatHF: Collecting Rich Human Feedback from Real-time Conver- sations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . 270–279

    work page 2024

  29. [30]

    Jenny T Liang, Chenyang Yang, and Brad A Myers. 2023. A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges. arXiv preprint arXiv:2303.17125 (2023)

    Pith/arXiv arXiv 2023

  30. [31]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638

    doi:10.1162/tacl_a_00638 2024

  31. [32]

    Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien- Ming Huang. 2023. Llm-powered conversational voice assistants: Interac- tion patterns, opportunities, challenges, and design guidelines. arXiv preprint arXiv:2309.13879 (2023)

    Pith/arXiv arXiv 2023

  32. [33]

    Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. Direct- gpt: A direct manipulation interface to interact with large language models. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–16

    work page 2024

  33. [34]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishin- skaya, Maja Trebacz, and Jan Leike. 2024. LLM Critics Help Catch LLM Bugs. arXiv preprint arXiv:2407.00215 (2024)

    Pith/arXiv arXiv 2024

  34. [35]

    Aditi Mishra, Utkarsh Soni, Anjana Arunkumar, Jinbin Huang, Bum Chul Kwon, and Chris Bryan. 2023. Promptaid: Prompt exploration, perturbation, testing 16 OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models UIST ’25, September 28-October 1, 2025, Busan, Republic of Korea and iteration using visual analyti...

    Pith/arXiv arXiv 2023

  35. [36]

    Cheng Niu, Xingguang Wang, Xuxin Cheng, Juntong Song, and Tong Zhang

    work page

  36. [37]

    Andrea Papenmeier, Dagmar Kern, Gwenn Englebienne, and Christin Seifert

    work page

  37. [38]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Enhancing Dialogue State Tracking Models through LLM-backed User- Agents Simulation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 8724–8741

    work page

  38. [39]

    Chi, Nathanael Schärli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. 2023. Large Language Models Can Be Easily Distracted by Irrelevant Context. In Proceedings of the 40th Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Re- search, Vol. 202), Andreas Krause, Emma Brunskill, Kyung...

    work page 2023

  39. [40]

    Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, et al. 2024. Wildfeedback: Aligning llms with in-situ user interactions and feedback. arXiv preprint arXiv:2408.15549 (2024)

    Pith/arXiv arXiv 2024

  40. [42]

    Tanya Stivers, N J Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter de Ruiter, Kyung- Eun Yoon, and Stephen C Levinson. 2009. Universals and cultural variation in turn-taking in conversation. Proc. Natl. Acad. Sci. U. S. A. 106, 26 (June 2009), 10587–10592

    work page 2009

  41. [43]

    Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Arti...

    arXiv 2024

  42. [44]

    Brodrick Stigall, Ryan Rossi, Jane Hoffswell, Xiang Chen, Shunan Guo, Fan Du, Eunyee Koh, and Kelly Caine. 2023. On Chatbots for Visual Exploratory Data Analysis. In 2023 IEEE International Conference on Big Data (BigData) . 5924–5929. doi:10.1109/BigData59044.2023.10386335

    arXiv 2023

  43. [45]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling Multilevel Exploration and Sensemaking with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 1, 18 pages. doi...

    arXiv 2023

  44. [46]

    Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2024. The metacognitive demands and opportunities of generative AI. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–24

    work page 2024

  45. [47]

    Lovis Bero Suchmann, Nicole Krämer, and Jürgen Ziegler. 2023. Branching Preferences: Visualizing Non-linear Topic Progression in Conversational Rec- ommender Systems. In Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (Limassol, Cyprus) (UMAP ’23 Ad- junct). Association for Computing Machinery, New York, NY,...

    work page arXiv 2023

  46. [48]

    1983.The visual display of quantitative information

    Edward R Tufte and Peter R Graves-Morris. 1983.The visual display of quantitative information. Vol. 2. Graphics press Cheshire, CT

    work page 1983

  47. [49]

    Gina Danielle Venolia and Carman Neustaedter. 2003. Understanding sequence and reply relationships within email conversations: a mixed-model visualization. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI ’03). Association for Computing Machinery, New York, NY, USA, 361–368. doi:10.1145/6426...

    work page arXiv 2003

  48. [50]

    Inequalities faced by women in access to permanent positions in astronomy in France

    A. Tat and M.S.T. Carpendale. 2002. Visualising human dialog. In Proceedings Sixth International Conference on Information Visualisation . 16–21. doi:10.1109/IV. 2002.1028751

    work page internal anchor Pith review Pith/arXiv arXiv 2002

  49. [51]

    Tao Wang, Mandy Keck, and Zana Vosough. 2021. Discussion Flows: An Interac- tive Visualization for Analyzing Engagement in Multi-Party Meetings. In EuroVis 2021 - Short Papers , Marco Agus, Christoph Garth, and Andreas Kerren (Eds.). The Eurographics Association. doi:10.2312/evs.20211060

    work page doi:10.2312/evs.20211060 2021

  50. [52]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  51. [53]

    Ben Wang, Jiqun Liu, Jamshed Karimnazarov, and Nicolas Thompson. 2024. Task supportive and personalized human-large language model interaction: A user study. In Proceedings of the 2024 Conference on Human Information Interaction and Retrieval. 370–375

    work page 2024

  52. [54]

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New...

    work page arXiv 2022

  53. [55]

    J Diego Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang

  54. [56]

    Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023. ScatterShot: Interactive In-context Example Curation for Text Transformation. In Proceedings of the 28th International Conference on Intelligent User Interfaces (Sydney, NSW, Australia)(IUI ’23). Association for Computing Machinery, New York, NY, USA, 353–367. doi:10.1145/358...

    arXiv 2023

  55. [60]

    clauses

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S...

    work page 2023

  56. [2022]

    ACM Trans

    It’s Complicated: The Relationship between User Trust, Model Accuracy and Explanations in AI. ACM Trans. Comput.-Hum. Interact. 29, 4, Article 35 (March 2022), 33 pages. doi:10.1145/3495013

    doi:10.1145/3495013 2022

  57. [2023]

    In Proceedings of the 2023 CHI conference on human factors in computing systems

    Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems. 1–21

    work page 2023

  58. [2024]

    In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

    Beyond the chat: Executable and verifiable text-editing with llms. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–23

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.