Exploring Student Feedback Needs and Design Opportunities in Data Storytelling Education
Pith reviewed 2026-05-21 10:47 UTC · model grok-4.3
The pith
In data storytelling education, participants viewed on-demand and process feedback as effective while automatic and outcome feedback appeared slightly more persuasive.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Through participant observations (N=8), interviews (N=6), and two design workshops (N=8/10), the study identified concrete feedback needs in a data storytelling course and evaluated four feedback modes in the Story Studio application. Participants perceived on-demand and process feedback modes as effective, but automatic and outcome feedback as slightly more persuasive.
What carries the argument
Feedback strategies varied along frequency (on-demand versus automatic), seamlessness (process versus outcome), and accountability inside an AI-assisted narrative storytelling application.
If this is right
- AI-augmented storytelling systems should offer both on-demand and automatic options so students can choose the level of guidance they need at different stages.
- Designers need to balance effectiveness for learning with persuasiveness when implementing process and outcome feedback.
- Instructors can extend their feedback capacity by letting AI tools handle routine checks while preserving human oversight for complex narrative decisions.
- Future tools could adapt feedback modes dynamically based on the current step in a student's storytelling workflow.
Where Pith is reading between the lines
- The same feedback-mode distinctions may prove useful in adjacent creative-technical courses such as data visualization or science communication.
- Classroom trials over a full semester could test whether the reported preferences produce measurable gains in final storytelling quality.
- Accountability features built into AI feedback might increase student reflection on how data choices shape the narrative.
Load-bearing premise
The small samples of students and educators studied hold preferences that represent data storytelling learners and instructors in general.
What would settle it
A larger-scale study with more diverse participants that finds most learners prefer automatic feedback over on-demand or rate outcome feedback as less persuasive than process feedback.
Figures
read the original abstract
Data storytelling workflows ask learners to integrate analytical, design, and narrative skills, but instructors rarely have the capacity to provide detailed feedback at each step. Computational and AI-assisted storytelling offers opportunities to support student learning, but how feedback should be structured effectively remains unclear. To address this gap, we conducted a two-phase participatory design study. Through participant observations (N=8) and interviews (N=6), the first phase explored learners and educators' feedback needs and challenges in a data storytelling course. The second phase conducted two design workshops (N=8/10) to design and evaluate feedback strategies (frequency, seamlessness, accountability) for Story Studio: an AI-assisted narrative storytelling application. Our findings show that participants perceived on-demand and process feedback modes as effective, but automatic and outcome feedback as slightly more persuasive. We discuss implications for designing AI-augmented storytelling systems that adapt their feedback modes to the diverse needs and expectations of students.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a two-phase participatory design study on feedback needs in data storytelling education. Phase 1 uses participant observations (N=8) and interviews (N=6) to identify challenges in providing detailed feedback across analytical, design, and narrative steps. Phase 2 employs two design workshops (N=8/10) to co-design and evaluate feedback strategies (frequency, seamlessness, accountability) for Story Studio, an AI-assisted narrative storytelling application. The central finding is that participants perceived on-demand and process feedback modes as effective, while automatic and outcome feedback modes were viewed as slightly more persuasive. The authors derive design implications for adaptive AI-augmented storytelling systems.
Significance. If the reported perceptions are stable, the work fills a practical gap in structuring feedback for complex, multi-skill data storytelling workflows where instructor capacity is limited. It contributes grounded, user-derived insights to HCI and AI-augmented education by directly involving learners and educators in identifying needs and prototyping strategies. Strengths include the multi-method participatory approach (observations, interviews, workshops) that ties findings directly to participant input without reliance on fitted parameters or self-referential derivations.
major comments (1)
- [Findings from Phase 2 workshops and Discussion of design implications] The central claim that automatic and outcome feedback modes are 'slightly more persuasive' (while on-demand/process are effective) is load-bearing for the design implications for Story Studio and similar tools. This distinction is drawn from small samples (N=8 observations, N=6 interviews, N=8/10 workshops) with no reported details on thematic saturation, participant diversity metrics, or how many individuals drove the 'slightly more persuasive' qualifier. Without these, the finding risks reflecting idiosyncratic responses rather than a stable pattern that can reliably inform adaptive feedback design.
minor comments (2)
- [Abstract and Methods] Clarify the exact structure of the two design workshops (e.g., whether N=8 and N=10 refer to separate workshops or combined participants) to improve reproducibility of the participatory process.
- [Methods] The manuscript would benefit from explicit discussion of how qualitative data were analyzed (e.g., thematic analysis steps, inter-coder agreement if used) to strengthen the transparency of how perceptions were derived from the raw observations and interviews.
Simulated Author's Rebuttal
We thank the referee for their constructive review and for recognizing the value of our multi-method participatory design approach in addressing feedback challenges in data storytelling education. We address the major comment on the robustness of the Phase 2 findings and their implications for design below.
read point-by-point responses
-
Referee: The central claim that automatic and outcome feedback modes are 'slightly more persuasive' (while on-demand/process are effective) is load-bearing for the design implications for Story Studio and similar tools. This distinction is drawn from small samples (N=8 observations, N=6 interviews, N=8/10 workshops) with no reported details on thematic saturation, participant diversity metrics, or how many individuals drove the 'slightly more persuasive' qualifier. Without these, the finding risks reflecting idiosyncratic responses rather than a stable pattern that can reliably inform adaptive feedback design.
Authors: We appreciate this observation on how our qualitative findings are presented. Participatory design studies of this nature intentionally use modest sample sizes to prioritize depth of engagement and co-creation over breadth or statistical generalizability, as reflected in our two workshops (N=8 and N=10). The distinction between on-demand/process modes (viewed as effective for supporting learning steps) and automatic/outcome modes (viewed as slightly more persuasive for motivation and accountability) arose iteratively from group discussions and prototype evaluations across both sessions. While we did not formally report thematic saturation metrics or participant diversity statistics beyond the shared course context, the themes showed consistency in participant input. We acknowledge that the original manuscript provides limited transparency on the precise number of individuals contributing to the 'slightly more persuasive' perception, given the collaborative workshop format. In revision, we will expand the Methods to detail our iterative thematic analysis process, add a summary of participant backgrounds and experience levels, and incorporate additional direct quotes from the workshops to illustrate the basis for the effectiveness versus persuasiveness distinction. We will also adjust the Discussion to frame the design implications as context-specific insights derived from learner and educator input, rather than broadly stable patterns, and note the need for further validation in larger studies. These changes will strengthen the manuscript without altering the core participatory findings. revision: yes
Circularity Check
No circularity: qualitative findings reported directly from participant data
full rationale
The paper is a two-phase participatory design study using observations (N=8), interviews (N=6), and workshops (N=8/10). Central claims consist of direct summaries of participant perceptions on feedback modes with no equations, fitted parameters, model predictions, or self-citation chains that reduce the result to its inputs by construction. Data collection and reporting form an empirical chain that does not match any enumerated circularity pattern.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Participatory design methods with small groups of learners and educators can reliably surface actionable feedback needs and design opportunities
Lean theorems connected to this paper
-
IndisputableMonolith/Foundation/AbsoluteFloorClosure.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
participants perceived on-demand and process feedback modes as effective, but automatic and outcome feedback as slightly more persuasive
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Peter Bacchetti, Steven G. Deeks, and Joseph M. McCune. 2011. Breaking Free of Sample Size Dogma to Perform Innovative Translational Research. Science Translational Medicine 3, 87 (2011), 87ps24. doi:10.1126/scitranslmed.3001623
-
[2]
Robert L. Bangert-Drowns, Chen-Lin C. Kulik, James A. Kulik, and Mary Morgan
-
[3]
Review of Educational Research 61, 2 (1991), 213–238
The Instructional Effect of Feedback in Test-Like Events. Review of Educational Research 61, 2 (1991), 213–238. doi:10.3102/00346543061002213
-
[4]
Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psy- chology. Qualitative Research in Psychology 3, 2 (2006), 77–101. doi:10.1191/ 1478088706qp063oa
work page 2006
-
[5]
Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose, California, USA, 981–992. doi:10.1145/2858036.2858498 Exploring Student Feedback Needs and Design Opportunities in Data Storytelling Education CHI ’26, April 13–17, 2026, Barcelona, Spain
-
[6]
S. Campbell, M. Greenwood, S. Prior, T. Shearer, K. Walkem, S. Young, D. Bywaters, and K. Walker. 2020. Purposive sampling: complex or simple? Research case examples. Journal of Research in Nursing 25, 8 (2020), 652–661. doi:10.1177/ 1744987120927206
work page 2020
-
[7]
Matthew Chalmers and Areti Galani. 2004. Seamful interweaving: heterogene- ity in the theory and design of interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. 243–252
work page 2004
-
[8]
S. Cheah and S. Li. 2020. The Effect of Structured Feedback on Performance: The Role of Attitude and Perceived Usefulness. Sustainability 12, 5 (2020), 2101. doi:10.3390/su12052101
-
[9]
Mihaly Csikszentmihalyi. 1996. Creativity: Flow and the Psychology of Discovery and Invention. Harper Perennial, New York
work page 1996
-
[10]
Y. Demchenko, C. G. J. José, S. Brewer, and T. Wiktorski. 2021. Edison data science framework (EDSF): Addressing demand for data science and analytics compe- tences for the data-driven digital economy. In 2021 IEEE Global Engineering Education Conference (EDUCON). IEEE, 1682–1687
work page 2021
-
[11]
B. Dykes. 2019. Effective data storytelling: how to drive change with data, narrative and visuals. John Wiley & Sons
work page 2019
-
[12]
Steve Graham, Karen Harris, and Michael Hebert. 2011. Informing Writing: The Benefits of Formative Assessment. A Report from Carnegie Corporation of New York. Carnegie Corporation of New York (2011)
work page 2011
-
[13]
Geneva D. Haertel, Terry P. Vendlinski, Daisy Rutstein, Angela DeBarger, Britte H. Cheng, Eric B. Snow, Cynthia D’Angelo, Christopher J. Harris, Louise Yarnall, and Liliana Ructtinger. 2016. General Introduction to Evidence-Centered Design. In Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications, James W. Pellegrino and Edys S...
work page 2016
-
[14]
doi:10.4324/9780203781302-6
-
[15]
Taha Hassan, Bob Edmison, Daron Williams, Larry Cox II, Matthew Louvet, Bart Knijnenburg, and D Scott McCrickard. 2024. Simplify, consolidate, intervene: facilitating institutional support with mental models of learning management system use. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1–23
work page 2024
-
[16]
John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research 77, 1 (2007), 81–112
work page 2007
-
[17]
Eric Horvitz, Johnson Apacible, and Mudumbai Subramani. 1999. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Proceedings of INTERACT ’99: Seventh IFIP Conference on Human-Computer Interaction. IOS Press, 263–269
work page 1999
-
[18]
Daniel R. Ilgen, Cynthia D. Fisher, and M. Susan Taylor. 1979. Consequences of In- dividual Feedback on Behavior in Organizations. Journal of Applied Psychology 64, 4 (1979), 349–371. doi:10.1037/0021-9010.64.4.349
-
[19]
Qasim Khalid, Naveed Arshad, Nasir Khan, Taha Hassan, Fahad Javed, and Ja- hangir Ikram. 2014. Home energy management system using NILM and low-cost HAN. Journal of electronic science and technology 12, 1 (2014), 20–25
work page 2014
-
[20]
Avraham N. Kluger and Angelo DeNisi. 1996. The effects of feedback interven- tions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 119, 2 (1996), 254–284
work page 1996
-
[21]
C. N. Knaflic. 2022. Storytelling with you: Plan, create, and deliver a stellar presentation. John Wiley & Sons
work page 2022
-
[22]
R. Kosara and J. Mackinlay. 2013. Storytelling: The next step for visualization. Computer 46, 5 (2013), 44–50
work page 2013
-
[23]
Raymond W. Kulhavy and Richard C. Anderson. 1972. Delay-retention effect with multiple-choice tests. Journal of Educational Psychology 63, 5 (1972), 505–512
work page 1972
-
[24]
B. Lee, N. H. Riche, P. Isenberg, and S. Carpendale. 2015. More than telling a story: Transforming data into visually shared stories. IEEE Computer Graphics and Applications 35, 5 (2015), 84–90
work page 2015
-
[25]
P. M. Leidig and L. Cassel. 2020. ACM taskforce efforts on computing competen- cies for undergraduate data science curricula. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 519– 520
work page 2020
-
[26]
Noble Lo, Alan Wong, and Sumie Chan. 2025. The impact of generative AI on essay revisions and student engagement. Computers and Education Open (2025), 100249
work page 2025
-
[27]
Charles A MacArthur. 2016. Instruction in evaluation and revision. (2016)
work page 2016
-
[28]
National Academies of Sciences, Engineering, and Medicine. 2018. Data science for undergraduates: Opportunities and options. National Academies Press
work page 2018
-
[29]
National Academies of Sciences, Engineering, and Medicine. 2020. Roundtable on data science postsecondary education: A compilation of meeting highlights
work page 2020
-
[30]
D. Nolan and S. Stoudt. 2021. Communicating with data: The art of writing for data science. Oxford University Press
work page 2021
-
[31]
Ernesto Panadero, Anders Jonsson, Leire Pinedo, and Belén Fernández-Castilla
-
[32]
Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analytic review.Educational Psychology Review 35, 4 (2023), 113
work page 2023
-
[33]
Robert J. Ploutz-Snyder, J. Fiedler, and Alan H. Feiveson. 2014. Justifying small-n research in scientifically amazing settings: Challenging the notion that only "big-n" studies are worthwhile. Journal of Applied Physiology 116, 9 (2014), 1251–1252. doi:10.1152/japplphysiol.01340.2013
-
[34]
Peng Ren, Yanhua Wang, and Fang Zhao. 2023. Re-understanding of Data Sto- rytelling Tools from a Narrative Perspective. Visual Intelligence 1, 11 (2023). doi:10.1007/s44267-023-00011-0
-
[35]
Maha Sajid, Syed Ibrahim Mustafa Shah Bukhari, Bo Ji, and Brendan David-John
-
[36]
Just stop doing everything for now!
" Just stop doing everything for now!": Understanding security attacks in remote collaborative mixed reality. In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 623–633
work page 2025
-
[37]
Donald A. Schön. 1983. The Reflective Practitioner: How Professionals Think in Action. Basic Books, New York
work page 1983
-
[38]
E. Segel and J. Heer. 2010. Narrative visualization: Telling stories with data. IEEE Transactions on Visualization and Computer Graphics 16, 6 (2010), 1139–1148
work page 2010
-
[39]
Danqing Shi, Xinyue Xu, Fuling Sun, Yang Shi, and Nan Cao. 2020. Calliope: Automatic visual data story generation from a spreadsheet. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 453–463
work page 2020
-
[40]
D. Showkat and E. P. Baumer. 2021. Where do stories come from? Examining the exploration process in investigative data journalism. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–31
work page 2021
-
[41]
Valerie J. Shute. 2008. Focus on formative feedback. Review of Educational Research 78, 1 (2008), 153–189
work page 2008
-
[42]
Timmers, Jet Braber-van den Broek, and Smits M
Caroline F. Timmers, Jet Braber-van den Broek, and Smits M. van den Berg
-
[43]
Learning and Individual Differences 21 (2015), 123–134
Motivational Beliefs, Self-Regulated Learning and Academic Performance in Higher Education: A Structural Equation Model. Learning and Individual Differences 21 (2015), 123–134. doi:10.1016/j.lindif.2010.07.007
-
[44]
Elizabeth J. Tisdell. 2025. Qualitative Research: A Guide to Design and Implementation (5 ed.). Jossey-Bass, Hoboken, New Jersey
work page 2025
-
[45]
Huichen Will Wang, Larry Birnbaum, and Vidya Setlur. 2025. Jupybara: Opera- tionalizing a design space for actionable data analysis and storytelling with llms. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–24
work page 2025
-
[46]
Qing Wang, Zhutian Li, Sai Fu, Weiwei Cui, and Huamin Qu. 2018. Narvis: Authoring Narrative Slideshows for Introducing Data Visualization Designs. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 779–
work page 2018
-
[47]
doi:10.1109/TVCG.2018.2865234
-
[48]
X. Wang, Y. Chen, A. G. Godley, and C. P. Rosé. 2018. Public Peer Review Motivates Higher Quality Feedback. In Proceedings of the International Conference of the Learning Sciences (ICLS). International Society of the Learning Sciences
work page 2018
-
[49]
G. Wu, S. Guo, J. Hoffswell, G. Y.-Y. Chan, R. A. Rossi, and E. Koh. 2023. Socrates: Data story generation via adaptive machine-guided elicitation of user feedback. IEEE Transactions on Visualization and Computer Graphics (2023)
work page 2023
-
[50]
Louise Yarnall, Hui Yang, Sophia Ouyang, and Lujie K Chen. 2025. Laying Foundations for Scalable Coaching for Data Storytelling: A Evidence-Centered Design Approach. InProceedings of the Twelfth ACM Conference on Learning@ Scale. 227–231
work page 2025
- [51]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.