Pith. sign in

REVIEW 4 major objections 6 minor 37 references

Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read The paper claims that a Conversation Progress Guide—a progress bar with subtask markers in a conversational AI interface—significantly improves users' self-efficacy compared to a conventional chat interface.

desk verdict Plausible progress-UI idea with a clean prototype, but the headline comparative claim rests on a between-group test that was never run. read the letter →

arxiv 2501.12001 v2 pith:CZFCXJIR submitted 2025-01-21 cs.HC

classification cs.HC
keywords self-efficacyprogressbarconversationalAIhuman-AIinteractionmasteryexperienceuserinterfacedesignstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Conversation Progress Guide (CPG), a visual interface for text-based conversational AI that shows users a progress bar with markers that light up as subtasks of their goal are completed. The central claim is that this interface raises users' self-efficacy—their belief in their ability to succeed at the task—by giving them frequent 'mastery experiences,' the most powerful source of self-efficacy in Bandura's theory. The authors argue that because conversational AI often produces failures and dissatisfaction, users' confidence can erode, and that providing visible partial successes can offset that erosion. A user study with 22 participants compared a CPG-enhanced chat to a conventional chat on a math/encryption task, and the paper reports significantly larger self-efficacy gains in the CPG group, with no significant differences in cognitive load, task time, or satisfaction. If the claim holds, it implies that a lightweight UI addition—not a change to the underlying AI—could make conversational assistants more empowering.

What carries the argument

The central object is the Conversation Progress Guide (CPG), a UI layer consisting of a progress bar and subtask markers, driven by a separate Progress Feedback Agent. The agent re-evaluates the conversation history after each exchange and decides, based on hand-crafted evaluation rules, whether a predefined subtask (e.g., 'Multiplication of Primes') has been completed; when it returns a positive judgment, the interface activates the corresponding marker in the predefined order. This design translates the user's actual conversation into an accumulating visual record of partial achievements, which is the concrete carrier of the claimed self-efficacy effect.

What would settle it

A direct between-group statistical test on pre-to-post self-efficacy change scores (e.g., independent-samples t-test or ANCOVA with baseline covariate) would settle the comparative claim; if the difference is not significant at the conventional threshold, the claim that the CPG significantly improves self-efficacy relative to a conventional interface is not supported by the data. Additionally, measuring the evaluator's judgments against human raters on the same conversation logs would test the assumption that the markers reflect real progress.

Watch

Extended reading notes

Core claim

The paper's central claim is that a visual progress guide for text-based conversational AI—a progress bar whose markers light up as the user completes subtasks—produces significantly greater gains in task-specific self-efficacy than a conventional chat interface, without adding cognitive load or reducing satisfaction. The mechanism is the reinforcement of 'mastery experiences': each time a subtask is judged complete, a marker appears, giving the user visible evidence of success. This contrasts with ordinary chat, where failures and ambiguous responses can accumulate with no visible record of partial progress. In the authors' user study, 22 participants performed an RSA encryption task with either a CPG-enhanced GPT-3.5-based chat or the same chat without the progress display; both groups improved on a six-item self-efficacy survey, and the authors report that the CPG group improved significantly more, with larger effect sizes on most items. They conclude that the interface succeeds in strengthening self-efficacy by leveraging partial successes, and that the effect is achieved without harming task performance, efficiency, or user experience.

Load-bearing premise

The entire claim rests on the automated evaluator that decides when a subtask is finished; if that evaluator frequently misjudges completion, the progress markers would deliver spurious mastery experiences and the self-efficacy effect would not be genuine.

Editorial extensions

If this is right

  • Layering the CPG onto an existing conversational AI requires no change to the underlying model, so any chat service with linearly decomposable tasks could adopt the same visual feedback.
  • Users who see their subtask completions accumulate may gain confidence in their ability to use conversational AI for multi-turn problem solving, which could increase their willingness to engage with such tools.
  • Because the interface adds no measurable cognitive load and does not alter task time or satisfaction, it offers a low-cost way to improve user experience in goal-oriented conversations.
  • The approach is deliberately limited to tasks with well-defined sub-steps; open-ended or purely social conversations would not receive meaningful progress markers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A plausible but untested extension is that the CPG's effect is largest for users who begin with low self-efficacy, since mastery experiences are thought to matter most when confidence is fragile; a stratified analysis by baseline score would reveal this.
  • The hand-crafted evaluation rules could be replaced by a learning-based subtask detector, which would let the interface generalize beyond the linear, pre-defined tasks used here without human rule authoring.
  • The present analysis compares within-group changes; a direct between-group test on change scores, such as an ANCOVA with baseline as covariate, would be a natural and more definitive statistical check of the comparative claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the Conversation Progress Guide (CPG), a UI that displays subtask-completion markers on a progress bar during text-based conversational AI interactions. The authors claim that CPG provides mastery experiences and thereby improves self-efficacy. They report a user study with 22 participants (control vs. experimental) performing an RSA encryption task, measuring self-efficacy before and after the task, cognitive load, satisfaction, task time, and interaction count. The paper concludes that CPG significantly improved self-efficacy compared to the default interface, without increasing cognitive load or harming satisfaction or task efficiency.

Significance. The topic is relevant and timely: self-efficacy is known to affect learning and persistence, and conversational AI failures may undermine it. The CPG concept is novel in this application domain, the implementation is functional, and the study includes a control group. The paper also candidly discusses several limitations, including issues with the progress evaluator and the self-efficacy scale. However, the central comparative claim is not supported by the reported statistical analysis, and the measurement design has a circularity problem. As presented, the evidence does not establish that CPG improves self-efficacy beyond the default interface. If properly validated and reanalyzed, the concept could be useful, but the current study is insufficient to support the headline claim.

major comments (4)
  1. [Abstract; Section 4.3; Section 5.1] The central claim that CPG led to 'significant improvements in self-efficacy measures compared to those using a conventional conversational AI' is not supported by the analysis. Section 4.3 reports (a) pre-task between-group t-tests showing no baseline differences, (b) within-group pre-post t-tests for each group, and (c) Cohen's d values in Table 4. No independent-samples test on change scores (post − pre) and no group × time interaction test is reported. Table 4 shows that Cohen's d is larger in the control group for Q1 and Q5 and similar for Q6, so the descriptive pattern is mixed. The conclusion of a significant between-group difference therefore rests on a comparison that was never performed.
  2. [Section 3.2; Section 4.2; Table 5] The Progress Feedback Agent, implemented with GPT-4, is the sole mechanism for lighting subtask markers, but its accuracy is never validated. No ground truth annotations, inter-rater agreement, or error rates are reported for the evaluation rules in Figure 4 and Figure 9. The paper itself documents failures: P7 saw the completion modal without understanding what triggered it, and P14 abandoned the experiment after the system failed to recognize task completion. If the agent frequently misjudges subtask completion, the purported mastery experiences are spurious. The reliability of the intervention needs to be established before its effect on self-efficacy can be interpreted.
  3. [Section 3.3; Section 4.1; Table 2] The self-efficacy survey items largely restate the subtasks displayed by the progress bar (e.g., Q2 asks about obtaining the RSA private key, which corresponds to the 'Private Key Generation' marker). Participants in the experimental group who see a marker light up may simply answer the matching survey item with higher confidence, so the observed gain may reflect direct feedback from the interface rather than a genuine change in self-efficacy. This item-wise overlap between the intervention display and the outcome measure is a measurement confound that threatens construct validity. The study needs a self-efficacy scale that does not directly mirror the displayed subtasks.
  4. [Section 4.3, Cognitive Load and Task Time] The paper interprets null results as evidence that the CPG has no effect ('confirming the system does not increase cognitive load'). With 22 participants total (approximately 11 per group), the statistical power to detect differences in cognitive load, satisfaction, task time, and interaction count is very low. The absence of significant p-values should not be equated with evidence of equivalence. Exact p-values and confidence intervals should be reported, and the wording should reflect that these are null findings rather than demonstrations of no impact.
minor comments (6)
  1. [Section 4.2] There are typos: 'an web application' should be 'a web application', and 'For the Progress Feedback Agent, employed the GPT-4 model' is missing a subject ('we employed').
  2. [Section 4.3] The within-group comparisons are t-tests, but the type (paired vs. independent) is not explicitly stated; please state that paired t-tests were used for pre-post comparisons.
  3. [Section 4.3] Exact p-values are not reported for the non-significant tests (cognitive load, satisfaction, task time, interaction count); only 'above 0.05' is given. Reporting the exact values would aid interpretation.
  4. [Keywords] The keyword 'Conservation Interface' appears to be a typo; it likely should read 'Conversation Interface'.
  5. [Section 4.3, Table 4] The version of Cohen's d used (e.g., pooled SD vs. SD of change scores) is not defined; please specify the formula.
  6. [Section 4.1] Please report the number of participants in each group; the text only gives the total (22).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical and no fitted parameter, self-citation chain, or definitional reduction carries the conclusion.

full rationale

This is an empirical HCI user study rather than a mathematical derivation, and none of the enumerated circularity patterns appears. The intervention (CPG subtask markers) is motivated by Bandura's mastery-experience theory, and the outcome is a task-specific self-efficacy survey. There is a content overlap between survey items (Table 2, Q2–Q5) and the subtask markers (Section 4.2), which is a genuine measurement/demand-characteristics concern, but it is not a case where the conclusion is true by construction: the pre-post self-efficacy changes are empirical observations, no parameter is fitted and then renamed as a prediction, and no self-citation is load-bearing. The paper's comparative claim in the abstract ("significant improvements ... compared to those using a conventional conversational AI") is not actually established by the Section 4.3 analysis, which reports only pre-task balance tests, within-group pre-post t-tests, and Cohen's d values, with no between-group test on change scores or group-by-time interaction. That is a statistical inference gap, not circularity. Under the hard rules, the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The central claim depends on the hand-crafted subtask structure, the unvalidated GPT-4 evaluator, the experimenter-designed survey, and the theoretical framework of self-efficacy. None of these are externally benchmarked; they are all introduced or assumed within the paper.

free parameters (4)
  • Number of subtasks for the RSA task = 6
    The authors state they empirically tested and found 3 to 7 subtasks to be manageable; for this task they chose 6. This is a hand-picked design parameter that affects the level of granularity of progress feedback.
  • Subtask evaluation rules for the Progress Feedback Agent = Hand-crafted rules per subtask (see Figure 4 and Appendix Figure 9)
    The rules that determine whether a subtask is complete are authored by the experimenters without validation against a ground truth. Different rules would change which markers light up and thus alter the intervention.
  • LLM choices = GPT-3.5-turbo for task agent, GPT-4 for evaluator
    The authors deliberately chose GPT-3.5-turbo to induce more errors, simulating realistic failure scenarios. This choice affects task difficulty and may influence self-efficacy outcomes.
  • Self-efficacy survey items = 6 Likert items (1-5)
    The survey was constructed by the authors for this experiment. It has no reported reliability or validation, and its items are directly tied to the subtask labels displayed by the CPG.
assumptions (3)
  • domain assumption Bandura's self-efficacy theory, including the four sources of self-efficacy, with mastery experiences as the most powerful source.
    The entire design rationale rests on this theory, cited in Sections 2 and 3.3. The paper does not test the theory itself.
  • domain assumption GPT-4 can accurately evaluate whether a conversation has completed a predefined subtask.
    This is required for the Progress Feedback Agent to function. No validation is provided, and the paper reports failure cases (P7, P14).
  • domain assumption Task-specific self-efficacy is measurable through six self-report Likert items.
    The paper cites Bandura's scale construction guidance, but does not validate the instrument empirically.
invented entities (2)
  • Progress Feedback Agent
    purpose: Monitors the conversation and decides whether each subtask has been completed, triggering visual markers.
    This is a new software component introduced by the paper. Its accuracy is not independently assessed, and no external handle is provided beyond the paper's own prototypes.
  • Subtask Marker
    purpose: A visual indicator that lights up on the progress bar when the Progress Feedback Agent detects a completed subtask.
    This is a new UI element. Its effect on self-efficacy is the central claim, but it has no existence outside this paper's prototype.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI." pith.science (2026). https://pith.science/paper/CZFCXJIR

@misc{pith2026250112001,
  author       = {Pith},
  title        = {Pith review of: Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CZFCXJIR}},
  note         = {Machine review of arXiv:2501.12001}
}
read the original abstract

In this study, we introduce the Conversation Progress Guide (CPG), a system designed for text-based conversational AI interactions that provides a visual interface to represent progress. Users often encounter failures when interacting with conversational AI, which can negatively affect their self-efficacy-an individual's belief in their capabilities, reducing their willingness to engage with these services. The CPG offers visual feedback on task progress, providing users with mastery experiences, a key source of self-efficacy. To evaluate the system's effectiveness, we conducted a user study assessing how the integration of the CPG influences user engagement and self-efficacy. Results demonstrate that users interacting with a conversational AI enhanced by the CPG showed significant improvements in self-efficacy measures compared to those using a conventional conversational AI.

Figures

Figures reproduced from arXiv: 2501.12001 by the authors.

Figure 1
Figure 1. The system architecture of the CPG, highlighting the interaction between the Task Agent and the Progress Feedback [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. User Interface of Conversation Progress Guide [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The task instruction for user study. Task. We designed the task as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Evaluation rules for the “Multiplication of Primes” subtask, as used by the Progress Feedback Agent to assess [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Average scores of Pre- and Post-Self-Efficacy Survey [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Average Cognitive Load scores for the experimental [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Average satisfaction scores for the experimental [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: Evaluation rules used by the Progress Feedback Agent to assess task completion and trigger progress markers. These [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 33 canonical work pages

  1. [1]

    Iman Al Hashmi, Hilal Alsabti, Omar Al Omari, Yusra Al Nasseri, and Atika Kha- laf. 2022. Development, feasibility and acceptability of a self-efficacy-enhancing smartphone application among pregnant women with gestational diabetes mel- litus: single-arm pilot clinical trial. BMC Pregnancy and Childbirth 22, 1 (2022), 358

  2. [2]

    Roger Azevedo and Jennifer G Cromley. 2004. Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of educational psychology 96, 3 (2004), 523

  3. [3]

    Albert Bandura. 1978. Self-efficacy: Toward a unifying theory of behavioral change. Advances in behaviour research and therapy 1, 4 (1978), 139–161

  4. [4]

    Albert Bandura et al. 1986. Social foundations of thought and action. Englewood Cliffs, NJ 1986, 23-28 (1986), 2

  5. [5]

    Albert Bandura et al. 2006. Guide for constructing self-efficacy scales.Self-efficacy beliefs of adolescents 5, 1 (2006), 307–337

  6. [6]

    Albert Bandura, Linda Reese, and Nancy E Adams. 1982. Microanalysis of action and fear arousal as a function of differential levels of perceived self-efficacy. Journal of personality and social psychology 43, 1 (1982), 5

  7. [7]

    Albert Bandura and Sebastian Wessels. 1997. Self-efficacy. Cambridge University Press Cambridge

  8. [8]

    Roy F Baumeister, Ellen Bratslavsky, Mark Muraven, and Dianne M Tice. 2018. Ego depletion: Is the active self a limited resource? In Self-regulation and self- control. Routledge, 16–44

Show all 37 references
  1. [9]

    John M Carroll. 1997. Human–computer interaction: Psychology as a science of design. International journal of human-computer studies 46, 4 (1997), 501–522

  2. [10]

    what" and

    Edward L Deci and Richard M Ryan. 2000. The" what" and" why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry 11, 4 (2000), 227–268. CHI ’25, April 26-May 1, 2025, Yokohama, Japan Jeong et al

  3. [11]

    Pawel Dybala, Michal Ptaszynski, Rafal Rzepka, and Kenji Araki. 2009. Activating Humans with Humor–A Dialogue System That Users Want to Interact with.IEICE TRANSACTIONS on Information and Systems 92, 12 (2009), 2394–2401

  4. [12]

    Morgan C Evans, Adela Kapuscinska, Maya Greenholt, Junchao Lin, Xuanyuan Liu, Tianyi Zhang, Jessica Hammer, and Geoff Kaufman. 2021. Designing a self-efficacy game for health literacy in marginalized communities. In Extended Abstracts of the 2021 CHI Conference on Human Factor...

  5. [13]

    David F Feldon, Rebecca Brockbank, and Kaylee Litson. 2023. Direct effects of cognitive load on self-efficacy during instruction. Journal of Educational Psychology (2023)

  6. [14]

    Jessica Gale, Meltem Alemdar, Christopher Cappelli, and David Morris. 2021. A mixed methods study of self-efficacy, the sources of self-efficacy, and teaching experience. In Frontiers in Education, Vol. 6. Frontiers Media SA, 750599

  7. [15]

    Sandra G Hart. 1986. NASA task load index (TLX). (1986)

  8. [16]

    Xiaoxia Huang, Richard E Mayer, and Ellen L Usher. 2020. Better together: Effects of four self-efficacy-building strategies on online statistical learning. Contemporary Educational Psychology 63 (2020), 101924

  9. [17]

    Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots classification and design techniques. In Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanc...

  10. [18]

    Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2024. Under- standing users’ dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In Proceedings of the 29th International Confer- ence on Intelligent User Interfac...

  11. [19]

    Claudia Leopold, Elke Sumfleth, and Detlev Leutner. 2013. Learning with sum- maries: Effects of representation mode and type of learning activity on compre- hension and transfer. Learning and Instruction 27 (2013), 40–49

  12. [20]

    Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang, Tara Safavi, Mengt- ing Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, et al. 2024. Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models. arXiv preprint arX...

  13. [21]

    Samiha Marwan, Preya Shabrina, Alex Milliken, Ian Menezes, Veronica Catete, Thomas W Price, and Tiffany Barnes. 2021. Promoting students’ progress- monitoring behavior during block-based programming. In Proceedings of the 21st Koli Calling International Conference on Computing...

  14. [22]

    Richard E Mayer. 2002. Multimedia learning. In Psychology of learning and motivation. Vol. 41. Elsevier, 85–139

  15. [23]

    Brad A Myers. 1983. Incense: A system for displaying data structures. ACM SIGGRAPH Computer Graphics 17, 3 (1983), 115–125

  16. [24]

    Brad A Myers. 1985. The importance of percent-done progress indicators for computer-human interfaces. ACM SIGCHI Bulletin 16, 4 (1985), 11–17

  17. [25]

    Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. Educational psychologist 38, 1 (2003), 1–4

  18. [26]

    Frank Pajares. 1996. Self-efficacy beliefs in academic settings. Review of educa- tional research 66, 4 (1996), 543–578

  19. [27]

    Frank Pajares and M David Miller. 1994. Role of self-efficacy and self-concept beliefs in mathematical problem solving: A path analysis. Journal of educational psychology 86, 2 (1994), 193

  20. [28]

    Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv pre...

  21. [29]

    Doeun Park, Myounglee Choo, Bohyun Jin, Un Sun Chung, Jinwoo Kim, Junghan Lee, and Yee-Jin Shin. 2023. Utilizing a Conversational Agent to Promote Self- efficacy in Children: A Pilot Study on Low Cognitive Ability Children with Attention Deficit Hyperactivity Disorder. In Exte...

  22. [30]

    André Rodrigues, André RB Santos, Kyle Montague, and Tiago Guerreiro. 2021. Promoting Self-Efficacy Through an Effective Human-Powered Nonvisual Smart- phone Task Assistant. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–19

  23. [31]

    Dale H Schunk and Maria K DiBenedetto. 2021. Self-efficacy and human motiva- tion. In Advances in motivation science . Vol. 8. Elsevier, 153–179

  24. [32]

    Ralf Schwarzer. 2014. Self-efficacy: Thought control of action . Taylor & Francis

  25. [33]

    Ralf Schwarzer, Matthias Jerusalem, J Weinman, Stephen Wright, M Johnston, et al. 1995. Measures in health psychology: A user’s portfolio. Causal and control beliefs. Causal and control beliefs 1, 011 (1995), 35–37

  26. [34]

    Abigail See and Christopher D Manning. 2021. Understanding and predicting user dissatisfaction in a neural generative chatbot. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue . 1–12

  27. [35]

    Guinan Su, Yanwu Yang, and Jie Guo. 2023. Prompt Your Mind: Refine Personal- ized Text Prompts within Your Mind. arXiv preprint arXiv:2311.05114 (2023)

  28. [36]

    John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257–285

  29. [37]

    Jeroen JG Van Merrienboer and John Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational psy- chology review 17 (2005), 147–177. Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.