REVIEW 4 major objections 6 minor 37 references
Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that a Conversation Progress Guide—a progress bar with subtask markers in a conversational AI interface—significantly improves users' self-efficacy compared to a conventional chat interface.
desk verdict Plausible progress-UI idea with a clean prototype, but the headline comparative claim rests on a between-group test that was never run. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Conversation Progress Guide (CPG), a UI layer consisting of a progress bar and subtask markers, driven by a separate Progress Feedback Agent. The agent re-evaluates the conversation history after each exchange and decides, based on hand-crafted evaluation rules, whether a predefined subtask (e.g., 'Multiplication of Primes') has been completed; when it returns a positive judgment, the interface activates the corresponding marker in the predefined order. This design translates the user's actual conversation into an accumulating visual record of partial achievements, which is the concrete carrier of the claimed self-efficacy effect.
What would settle it
A direct between-group statistical test on pre-to-post self-efficacy change scores (e.g., independent-samples t-test or ANCOVA with baseline covariate) would settle the comparative claim; if the difference is not significant at the conventional threshold, the claim that the CPG significantly improves self-efficacy relative to a conventional interface is not supported by the data. Additionally, measuring the evaluator's judgments against human raters on the same conversation logs would test the assumption that the markers reflect real progress.
Extended reading notes
Core claim
The paper's central claim is that a visual progress guide for text-based conversational AI—a progress bar whose markers light up as the user completes subtasks—produces significantly greater gains in task-specific self-efficacy than a conventional chat interface, without adding cognitive load or reducing satisfaction. The mechanism is the reinforcement of 'mastery experiences': each time a subtask is judged complete, a marker appears, giving the user visible evidence of success. This contrasts with ordinary chat, where failures and ambiguous responses can accumulate with no visible record of partial progress. In the authors' user study, 22 participants performed an RSA encryption task with either a CPG-enhanced GPT-3.5-based chat or the same chat without the progress display; both groups improved on a six-item self-efficacy survey, and the authors report that the CPG group improved significantly more, with larger effect sizes on most items. They conclude that the interface succeeds in strengthening self-efficacy by leveraging partial successes, and that the effect is achieved without harming task performance, efficiency, or user experience.
Load-bearing premise
The entire claim rests on the automated evaluator that decides when a subtask is finished; if that evaluator frequently misjudges completion, the progress markers would deliver spurious mastery experiences and the self-efficacy effect would not be genuine.
Editorial extensions
If this is right
- Layering the CPG onto an existing conversational AI requires no change to the underlying model, so any chat service with linearly decomposable tasks could adopt the same visual feedback.
- Users who see their subtask completions accumulate may gain confidence in their ability to use conversational AI for multi-turn problem solving, which could increase their willingness to engage with such tools.
- Because the interface adds no measurable cognitive load and does not alter task time or satisfaction, it offers a low-cost way to improve user experience in goal-oriented conversations.
- The approach is deliberately limited to tasks with well-defined sub-steps; open-ended or purely social conversations would not receive meaningful progress markers.
Reading between the lines
- A plausible but untested extension is that the CPG's effect is largest for users who begin with low self-efficacy, since mastery experiences are thought to matter most when confidence is fragile; a stratified analysis by baseline score would reveal this.
- The hand-crafted evaluation rules could be replaced by a learning-based subtask detector, which would let the interface generalize beyond the linear, pre-defined tasks used here without human rule authoring.
- The present analysis compares within-group changes; a direct between-group test on change scores, such as an ANCOVA with baseline as covariate, would be a natural and more definitive statistical check of the comparative claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Conversation Progress Guide (CPG), a UI that displays subtask-completion markers on a progress bar during text-based conversational AI interactions. The authors claim that CPG provides mastery experiences and thereby improves self-efficacy. They report a user study with 22 participants (control vs. experimental) performing an RSA encryption task, measuring self-efficacy before and after the task, cognitive load, satisfaction, task time, and interaction count. The paper concludes that CPG significantly improved self-efficacy compared to the default interface, without increasing cognitive load or harming satisfaction or task efficiency.
Significance. The topic is relevant and timely: self-efficacy is known to affect learning and persistence, and conversational AI failures may undermine it. The CPG concept is novel in this application domain, the implementation is functional, and the study includes a control group. The paper also candidly discusses several limitations, including issues with the progress evaluator and the self-efficacy scale. However, the central comparative claim is not supported by the reported statistical analysis, and the measurement design has a circularity problem. As presented, the evidence does not establish that CPG improves self-efficacy beyond the default interface. If properly validated and reanalyzed, the concept could be useful, but the current study is insufficient to support the headline claim.
major comments (4)
- [Abstract; Section 4.3; Section 5.1] The central claim that CPG led to 'significant improvements in self-efficacy measures compared to those using a conventional conversational AI' is not supported by the analysis. Section 4.3 reports (a) pre-task between-group t-tests showing no baseline differences, (b) within-group pre-post t-tests for each group, and (c) Cohen's d values in Table 4. No independent-samples test on change scores (post − pre) and no group × time interaction test is reported. Table 4 shows that Cohen's d is larger in the control group for Q1 and Q5 and similar for Q6, so the descriptive pattern is mixed. The conclusion of a significant between-group difference therefore rests on a comparison that was never performed.
- [Section 3.2; Section 4.2; Table 5] The Progress Feedback Agent, implemented with GPT-4, is the sole mechanism for lighting subtask markers, but its accuracy is never validated. No ground truth annotations, inter-rater agreement, or error rates are reported for the evaluation rules in Figure 4 and Figure 9. The paper itself documents failures: P7 saw the completion modal without understanding what triggered it, and P14 abandoned the experiment after the system failed to recognize task completion. If the agent frequently misjudges subtask completion, the purported mastery experiences are spurious. The reliability of the intervention needs to be established before its effect on self-efficacy can be interpreted.
- [Section 3.3; Section 4.1; Table 2] The self-efficacy survey items largely restate the subtasks displayed by the progress bar (e.g., Q2 asks about obtaining the RSA private key, which corresponds to the 'Private Key Generation' marker). Participants in the experimental group who see a marker light up may simply answer the matching survey item with higher confidence, so the observed gain may reflect direct feedback from the interface rather than a genuine change in self-efficacy. This item-wise overlap between the intervention display and the outcome measure is a measurement confound that threatens construct validity. The study needs a self-efficacy scale that does not directly mirror the displayed subtasks.
- [Section 4.3, Cognitive Load and Task Time] The paper interprets null results as evidence that the CPG has no effect ('confirming the system does not increase cognitive load'). With 22 participants total (approximately 11 per group), the statistical power to detect differences in cognitive load, satisfaction, task time, and interaction count is very low. The absence of significant p-values should not be equated with evidence of equivalence. Exact p-values and confidence intervals should be reported, and the wording should reflect that these are null findings rather than demonstrations of no impact.
minor comments (6)
- [Section 4.2] There are typos: 'an web application' should be 'a web application', and 'For the Progress Feedback Agent, employed the GPT-4 model' is missing a subject ('we employed').
- [Section 4.3] The within-group comparisons are t-tests, but the type (paired vs. independent) is not explicitly stated; please state that paired t-tests were used for pre-post comparisons.
- [Section 4.3] Exact p-values are not reported for the non-significant tests (cognitive load, satisfaction, task time, interaction count); only 'above 0.05' is given. Reporting the exact values would aid interpretation.
- [Keywords] The keyword 'Conservation Interface' appears to be a typo; it likely should read 'Conversation Interface'.
- [Section 4.3, Table 4] The version of Cohen's d used (e.g., pooled SD vs. SD of change scores) is not defined; please specify the formula.
- [Section 4.1] Please report the number of participants in each group; the text only gives the total (22).
Circularity Check
No significant circularity: the central claims are empirical and no fitted parameter, self-citation chain, or definitional reduction carries the conclusion.
full rationale
This is an empirical HCI user study rather than a mathematical derivation, and none of the enumerated circularity patterns appears. The intervention (CPG subtask markers) is motivated by Bandura's mastery-experience theory, and the outcome is a task-specific self-efficacy survey. There is a content overlap between survey items (Table 2, Q2–Q5) and the subtask markers (Section 4.2), which is a genuine measurement/demand-characteristics concern, but it is not a case where the conclusion is true by construction: the pre-post self-efficacy changes are empirical observations, no parameter is fitted and then renamed as a prediction, and no self-citation is load-bearing. The paper's comparative claim in the abstract ("significant improvements ... compared to those using a conventional conversational AI") is not actually established by the Section 4.3 analysis, which reports only pre-task balance tests, within-group pre-post t-tests, and Cohen's d values, with no between-group test on change scores or group-by-time interaction. That is a statistical inference gap, not circularity. Under the hard rules, the appropriate finding is no significant circularity with score 0.
Assumptions & free parameters
free parameters (4)
- Number of subtasks for the RSA task =
6
- Subtask evaluation rules for the Progress Feedback Agent =
Hand-crafted rules per subtask (see Figure 4 and Appendix Figure 9)
- LLM choices =
GPT-3.5-turbo for task agent, GPT-4 for evaluator
- Self-efficacy survey items =
6 Likert items (1-5)
assumptions (3)
- domain assumption Bandura's self-efficacy theory, including the four sources of self-efficacy, with mastery experiences as the most powerful source.
- domain assumption GPT-4 can accurately evaluate whether a conversation has completed a predefined subtask.
- domain assumption Task-specific self-efficacy is measurable through six self-report Likert items.
invented entities (2)
-
Progress Feedback Agent
-
Subtask Marker
Cite this review
Pith. "Pith review of Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI." pith.science (2026). https://pith.science/paper/CZFCXJIR
@misc{pith2026250112001,
author = {Pith},
title = {Pith review of: Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI},
year = {2026},
howpublished = {\url{https://pith.science/paper/CZFCXJIR}},
note = {Machine review of arXiv:2501.12001}
}
read the original abstract
In this study, we introduce the Conversation Progress Guide (CPG), a system designed for text-based conversational AI interactions that provides a visual interface to represent progress. Users often encounter failures when interacting with conversational AI, which can negatively affect their self-efficacy-an individual's belief in their capabilities, reducing their willingness to engage with these services. The CPG offers visual feedback on task progress, providing users with mastery experiences, a key source of self-efficacy. To evaluate the system's effectiveness, we conducted a user study assessing how the integration of the CPG influences user engagement and self-efficacy. Results demonstrate that users interacting with a conversational AI enhanced by the CPG showed significant improvements in self-efficacy measures compared to those using a conventional conversational AI.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Iman Al Hashmi, Hilal Alsabti, Omar Al Omari, Yusra Al Nasseri, and Atika Kha- laf. 2022. Development, feasibility and acceptability of a self-efficacy-enhancing smartphone application among pregnant women with gestational diabetes mel- litus: single-arm pilot clinical trial. BMC Pregnancy and Childbirth 22, 1 (2022), 358
work page 2022
-
[2]
Roger Azevedo and Jennifer G Cromley. 2004. Does training on self-regulated learning facilitate students’ learning with hypermedia? Journal of educational psychology 96, 3 (2004), 523
work page 2004
-
[3]
Albert Bandura. 1978. Self-efficacy: Toward a unifying theory of behavioral change. Advances in behaviour research and therapy 1, 4 (1978), 139–161
work page 1978
-
[4]
Albert Bandura et al. 1986. Social foundations of thought and action. Englewood Cliffs, NJ 1986, 23-28 (1986), 2
work page 1986
-
[5]
Albert Bandura et al. 2006. Guide for constructing self-efficacy scales.Self-efficacy beliefs of adolescents 5, 1 (2006), 307–337
work page 2006
-
[6]
Albert Bandura, Linda Reese, and Nancy E Adams. 1982. Microanalysis of action and fear arousal as a function of differential levels of perceived self-efficacy. Journal of personality and social psychology 43, 1 (1982), 5
work page 1982
-
[7]
Albert Bandura and Sebastian Wessels. 1997. Self-efficacy. Cambridge University Press Cambridge
work page 1997
-
[8]
Roy F Baumeister, Ellen Bratslavsky, Mark Muraven, and Dianne M Tice. 2018. Ego depletion: Is the active self a limited resource? In Self-regulation and self- control. Routledge, 16–44
work page 2018
Show all 37 references
-
[9]
John M Carroll. 1997. Human–computer interaction: Psychology as a science of design. International journal of human-computer studies 46, 4 (1997), 501–522
1997
-
[10]
what" and
Edward L Deci and Richard M Ryan. 2000. The" what" and" why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry 11, 4 (2000), 227–268. CHI ’25, April 26-May 1, 2025, Yokohama, Japan Jeong et al
2000
-
[11]
Pawel Dybala, Michal Ptaszynski, Rafal Rzepka, and Kenji Araki. 2009. Activating Humans with Humor–A Dialogue System That Users Want to Interact with.IEICE TRANSACTIONS on Information and Systems 92, 12 (2009), 2394–2401
2009
-
[12]
Morgan C Evans, Adela Kapuscinska, Maya Greenholt, Junchao Lin, Xuanyuan Liu, Tianyi Zhang, Jessica Hammer, and Geoff Kaufman. 2021. Designing a self-efficacy game for health literacy in marginalized communities. In Extended Abstracts of the 2021 CHI Conference on Human Factor...
2021
-
[13]
David F Feldon, Rebecca Brockbank, and Kaylee Litson. 2023. Direct effects of cognitive load on self-efficacy during instruction. Journal of Educational Psychology (2023)
2023
-
[14]
Jessica Gale, Meltem Alemdar, Christopher Cappelli, and David Morris. 2021. A mixed methods study of self-efficacy, the sources of self-efficacy, and teaching experience. In Frontiers in Education, Vol. 6. Frontiers Media SA, 750599
2021
-
[15]
Sandra G Hart. 1986. NASA task load index (TLX). (1986)
1986
-
[16]
Xiaoxia Huang, Richard E Mayer, and Ellen L Usher. 2020. Better together: Effects of four self-efficacy-building strategies on online statistical learning. Contemporary Educational Psychology 63 (2020), 101924
2020
-
[17]
Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. 2019. A survey on conversational agents/chatbots classification and design techniques. In Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanc...
2019
-
[18]
Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2024. Under- standing users’ dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In Proceedings of the 29th International Confer- ence on Intelligent User Interfac...
2024
-
[19]
Claudia Leopold, Elke Sumfleth, and Detlev Leutner. 2013. Learning with sum- maries: Effects of representation mode and type of learning activity on compre- hension and transfer. Learning and Instruction 27 (2013), 40–49
2013
-
[20]
Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang, Tara Safavi, Mengt- ing Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, et al. 2024. Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models. arXiv preprint arX...
2024 arXiv
-
[21]
Samiha Marwan, Preya Shabrina, Alex Milliken, Ian Menezes, Veronica Catete, Thomas W Price, and Tiffany Barnes. 2021. Promoting students’ progress- monitoring behavior during block-based programming. In Proceedings of the 21st Koli Calling International Conference on Computing...
2021
-
[22]
Richard E Mayer. 2002. Multimedia learning. In Psychology of learning and motivation. Vol. 41. Elsevier, 85–139
2002
-
[23]
Brad A Myers. 1983. Incense: A system for displaying data structures. ACM SIGGRAPH Computer Graphics 17, 3 (1983), 115–125
1983
-
[24]
Brad A Myers. 1985. The importance of percent-done progress indicators for computer-human interfaces. ACM SIGCHI Bulletin 16, 4 (1985), 11–17
1985
-
[25]
Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. Educational psychologist 38, 1 (2003), 1–4
2003
-
[26]
Frank Pajares. 1996. Self-efficacy beliefs in academic settings. Review of educa- tional research 66, 4 (1996), 543–578
1996
-
[27]
Frank Pajares and M David Miller. 1994. Role of self-efficacy and self-concept beliefs in mathematical problem solving: A path analysis. Journal of educational psychology 86, 2 (1994), 193
1994
-
[28]
Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv pre...
2020 arXiv
-
[29]
Doeun Park, Myounglee Choo, Bohyun Jin, Un Sun Chung, Jinwoo Kim, Junghan Lee, and Yee-Jin Shin. 2023. Utilizing a Conversational Agent to Promote Self- efficacy in Children: A Pilot Study on Low Cognitive Ability Children with Attention Deficit Hyperactivity Disorder. In Exte...
2023
-
[30]
André Rodrigues, André RB Santos, Kyle Montague, and Tiago Guerreiro. 2021. Promoting Self-Efficacy Through an Effective Human-Powered Nonvisual Smart- phone Task Assistant. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–19
2021
-
[31]
Dale H Schunk and Maria K DiBenedetto. 2021. Self-efficacy and human motiva- tion. In Advances in motivation science . Vol. 8. Elsevier, 153–179
2021
-
[32]
Ralf Schwarzer. 2014. Self-efficacy: Thought control of action . Taylor & Francis
2014
-
[33]
Ralf Schwarzer, Matthias Jerusalem, J Weinman, Stephen Wright, M Johnston, et al. 1995. Measures in health psychology: A user’s portfolio. Causal and control beliefs. Causal and control beliefs 1, 011 (1995), 35–37
1995
-
[34]
Abigail See and Christopher D Manning. 2021. Understanding and predicting user dissatisfaction in a neural generative chatbot. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue . 1–12
2021
-
[35]
Guinan Su, Yanwu Yang, and Jie Guo. 2023. Prompt Your Mind: Refine Personal- ized Text Prompts within Your Mind. arXiv preprint arXiv:2311.05114 (2023)
2023 arXiv
-
[36]
John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257–285
1988
-
[37]
Jeroen JG Van Merrienboer and John Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational psy- chology review 17 (2005), 147–177. Conversation Progress Guide : UI System for Enhancing Self-Efficacy in Conversational AI...
2005
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.