pith. sign in

arxiv: 2605.16271 · v1 · pith:7REQ5AR5new · submitted 2026-04-02 · 💻 cs.HC

Exploring Student Feedback Needs and Design Opportunities in Data Storytelling Education

Pith reviewed 2026-05-21 10:47 UTC · model grok-4.3

classification 💻 cs.HC
keywords data storytellingfeedback designAI-assisted learningparticipatory designeducational technologynarrative visualizationstudent needs
0
0 comments X

The pith

In data storytelling education, participants viewed on-demand and process feedback as effective while automatic and outcome feedback appeared slightly more persuasive.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Data storytelling requires learners to combine analytical, design, and narrative skills, but instructors rarely have time to give detailed feedback at each step of the process. This paper reports a two-phase participatory design study that first observed students and interviewed educators to map feedback needs, then ran workshops to prototype and test feedback strategies inside an AI-assisted storytelling tool. Participants judged on-demand feedback that students request themselves and process feedback that guides intermediate steps as effective for learning. Automatic feedback delivered without prompting and outcome feedback centered on final results scored slightly higher on persuasiveness. These distinctions suggest that AI systems could better support creative data work by switching between feedback styles according to student expectations.

Core claim

Through participant observations (N=8), interviews (N=6), and two design workshops (N=8/10), the study identified concrete feedback needs in a data storytelling course and evaluated four feedback modes in the Story Studio application. Participants perceived on-demand and process feedback modes as effective, but automatic and outcome feedback as slightly more persuasive.

What carries the argument

Feedback strategies varied along frequency (on-demand versus automatic), seamlessness (process versus outcome), and accountability inside an AI-assisted narrative storytelling application.

If this is right

  • AI-augmented storytelling systems should offer both on-demand and automatic options so students can choose the level of guidance they need at different stages.
  • Designers need to balance effectiveness for learning with persuasiveness when implementing process and outcome feedback.
  • Instructors can extend their feedback capacity by letting AI tools handle routine checks while preserving human oversight for complex narrative decisions.
  • Future tools could adapt feedback modes dynamically based on the current step in a student's storytelling workflow.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same feedback-mode distinctions may prove useful in adjacent creative-technical courses such as data visualization or science communication.
  • Classroom trials over a full semester could test whether the reported preferences produce measurable gains in final storytelling quality.
  • Accountability features built into AI feedback might increase student reflection on how data choices shape the narrative.

Load-bearing premise

The small samples of students and educators studied hold preferences that represent data storytelling learners and instructors in general.

What would settle it

A larger-scale study with more diverse participants that finds most learners prefer automatic feedback over on-demand or rate outcome feedback as less persuasive than process feedback.

Figures

Figures reproduced from arXiv: 2605.16271 by Jennifer Posada, Jiaqi Gong, Louise Yarnall, Lujie Karen Chen, Taha Hassan.

Figure 1
Figure 1. Figure 1: Study timeline overview. (Top left). Study 1 activities (observations, interviews, concept development, and design [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Study methodology overview. (Top left) Participant observations, interviews. (Bottom-left) Concept development, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Storyboard activity overview. (Top) This shows [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Conceptual design overview clockwise from left. (Top-left) Section 1 shows Concept sketch 1 which is the first [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Story Studio website features. (Top-left) Story Studio website main page featuring workspace with data plots. (Bottom￾left) Narrative story suggestions. (Top-right) Address feedback feature showing feedback on data story and button "address to proceed". 5.4 Study Findings [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Survey results: (from left) all six feedback alternatives (on-demand, automatic, skip, address, process, outcome) rated [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Survey results: automatic vs. on-demand feedback [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Survey results: outcomes vs. process feedback ac [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Gantt Chart overview. A.6 Survey Analysis This shows Survey Analysis results of mean and standard deviation. feature dimension average std dv on demand feedback effective 4.57 0.53 automatic feedback effective 4.14 1.07 skip effective 4.14 0.90 address effective 4.29 1.25 process effective 4.86 0.38 outcomes effective 4.71 0.49 on demand feedback persuasive 3.71 1.11 automatic feedback persuasive 4.00 0.8… view at source ↗
read the original abstract

Data storytelling workflows ask learners to integrate analytical, design, and narrative skills, but instructors rarely have the capacity to provide detailed feedback at each step. Computational and AI-assisted storytelling offers opportunities to support student learning, but how feedback should be structured effectively remains unclear. To address this gap, we conducted a two-phase participatory design study. Through participant observations (N=8) and interviews (N=6), the first phase explored learners and educators' feedback needs and challenges in a data storytelling course. The second phase conducted two design workshops (N=8/10) to design and evaluate feedback strategies (frequency, seamlessness, accountability) for Story Studio: an AI-assisted narrative storytelling application. Our findings show that participants perceived on-demand and process feedback modes as effective, but automatic and outcome feedback as slightly more persuasive. We discuss implications for designing AI-augmented storytelling systems that adapt their feedback modes to the diverse needs and expectations of students.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript reports a two-phase participatory design study on feedback needs in data storytelling education. Phase 1 uses participant observations (N=8) and interviews (N=6) to identify challenges in providing detailed feedback across analytical, design, and narrative steps. Phase 2 employs two design workshops (N=8/10) to co-design and evaluate feedback strategies (frequency, seamlessness, accountability) for Story Studio, an AI-assisted narrative storytelling application. The central finding is that participants perceived on-demand and process feedback modes as effective, while automatic and outcome feedback modes were viewed as slightly more persuasive. The authors derive design implications for adaptive AI-augmented storytelling systems.

Significance. If the reported perceptions are stable, the work fills a practical gap in structuring feedback for complex, multi-skill data storytelling workflows where instructor capacity is limited. It contributes grounded, user-derived insights to HCI and AI-augmented education by directly involving learners and educators in identifying needs and prototyping strategies. Strengths include the multi-method participatory approach (observations, interviews, workshops) that ties findings directly to participant input without reliance on fitted parameters or self-referential derivations.

major comments (1)
  1. [Findings from Phase 2 workshops and Discussion of design implications] The central claim that automatic and outcome feedback modes are 'slightly more persuasive' (while on-demand/process are effective) is load-bearing for the design implications for Story Studio and similar tools. This distinction is drawn from small samples (N=8 observations, N=6 interviews, N=8/10 workshops) with no reported details on thematic saturation, participant diversity metrics, or how many individuals drove the 'slightly more persuasive' qualifier. Without these, the finding risks reflecting idiosyncratic responses rather than a stable pattern that can reliably inform adaptive feedback design.
minor comments (2)
  1. [Abstract and Methods] Clarify the exact structure of the two design workshops (e.g., whether N=8 and N=10 refer to separate workshops or combined participants) to improve reproducibility of the participatory process.
  2. [Methods] The manuscript would benefit from explicit discussion of how qualitative data were analyzed (e.g., thematic analysis steps, inter-coder agreement if used) to strengthen the transparency of how perceptions were derived from the raw observations and interviews.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive review and for recognizing the value of our multi-method participatory design approach in addressing feedback challenges in data storytelling education. We address the major comment on the robustness of the Phase 2 findings and their implications for design below.

read point-by-point responses
  1. Referee: The central claim that automatic and outcome feedback modes are 'slightly more persuasive' (while on-demand/process are effective) is load-bearing for the design implications for Story Studio and similar tools. This distinction is drawn from small samples (N=8 observations, N=6 interviews, N=8/10 workshops) with no reported details on thematic saturation, participant diversity metrics, or how many individuals drove the 'slightly more persuasive' qualifier. Without these, the finding risks reflecting idiosyncratic responses rather than a stable pattern that can reliably inform adaptive feedback design.

    Authors: We appreciate this observation on how our qualitative findings are presented. Participatory design studies of this nature intentionally use modest sample sizes to prioritize depth of engagement and co-creation over breadth or statistical generalizability, as reflected in our two workshops (N=8 and N=10). The distinction between on-demand/process modes (viewed as effective for supporting learning steps) and automatic/outcome modes (viewed as slightly more persuasive for motivation and accountability) arose iteratively from group discussions and prototype evaluations across both sessions. While we did not formally report thematic saturation metrics or participant diversity statistics beyond the shared course context, the themes showed consistency in participant input. We acknowledge that the original manuscript provides limited transparency on the precise number of individuals contributing to the 'slightly more persuasive' perception, given the collaborative workshop format. In revision, we will expand the Methods to detail our iterative thematic analysis process, add a summary of participant backgrounds and experience levels, and incorporate additional direct quotes from the workshops to illustrate the basis for the effectiveness versus persuasiveness distinction. We will also adjust the Discussion to frame the design implications as context-specific insights derived from learner and educator input, rather than broadly stable patterns, and note the need for further validation in larger studies. These changes will strengthen the manuscript without altering the core participatory findings. revision: yes

Circularity Check

0 steps flagged

No circularity: qualitative findings reported directly from participant data

full rationale

The paper is a two-phase participatory design study using observations (N=8), interviews (N=6), and workshops (N=8/10). Central claims consist of direct summaries of participant perceptions on feedback modes with no equations, fitted parameters, model predictions, or self-citation chains that reduce the result to its inputs by construction. Data collection and reporting form an empirical chain that does not match any enumerated circularity pattern.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The study rests on standard HCI assumptions about the validity of participatory design for surfacing user needs, with no free parameters, no new invented entities, and no ad-hoc axioms beyond domain norms.

axioms (1)
  • domain assumption Participatory design methods with small groups of learners and educators can reliably surface actionable feedback needs and design opportunities
    Invoked implicitly in the two-phase study design and interpretation of workshop outcomes.

pith-pipeline@v0.9.0 · 5699 in / 1118 out tokens · 42617 ms · 2026-05-21T10:47:25.648356+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

51 extracted references · 51 canonical work pages

  1. [1]

    Deeks, and Joseph M

    Peter Bacchetti, Steven G. Deeks, and Joseph M. McCune. 2011. Breaking Free of Sample Size Dogma to Perform Innovative Translational Research. Science Translational Medicine 3, 87 (2011), 87ps24. doi:10.1126/scitranslmed.3001623

  2. [2]

    Bangert-Drowns, Chen-Lin C

    Robert L. Bangert-Drowns, Chen-Lin C. Kulik, James A. Kulik, and Mary Morgan

  3. [3]

    Review of Educational Research 61, 2 (1991), 213–238

    The Instructional Effect of Feedback in Test-Like Events. Review of Educational Research 61, 2 (1991), 213–238. doi:10.3102/00346543061002213

  4. [4]

    Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psy- chology. Qualitative Research in Psychology 3, 2 (2006), 77–101. doi:10.1191/ 1478088706qp063oa

  5. [5]

    Kelly Caine. 2016. Local Standards for Sample Size at CHI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San Jose, California, USA, 981–992. doi:10.1145/2858036.2858498 Exploring Student Feedback Needs and Design Opportunities in Data Storytelling Education CHI ’26, April 13–17, 2026, Barcelona, Spain

  6. [6]

    Campbell, M

    S. Campbell, M. Greenwood, S. Prior, T. Shearer, K. Walkem, S. Young, D. Bywaters, and K. Walker. 2020. Purposive sampling: complex or simple? Research case examples. Journal of Research in Nursing 25, 8 (2020), 652–661. doi:10.1177/ 1744987120927206

  7. [7]

    Matthew Chalmers and Areti Galani. 2004. Seamful interweaving: heterogene- ity in the theory and design of interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. 243–252

  8. [8]

    Cheah and S

    S. Cheah and S. Li. 2020. The Effect of Structured Feedback on Performance: The Role of Attitude and Perceived Usefulness. Sustainability 12, 5 (2020), 2101. doi:10.3390/su12052101

  9. [9]

    Mihaly Csikszentmihalyi. 1996. Creativity: Flow and the Psychology of Discovery and Invention. Harper Perennial, New York

  10. [10]

    Demchenko, C

    Y. Demchenko, C. G. J. José, S. Brewer, and T. Wiktorski. 2021. Edison data science framework (EDSF): Addressing demand for data science and analytics compe- tences for the data-driven digital economy. In 2021 IEEE Global Engineering Education Conference (EDUCON). IEEE, 1682–1687

  11. [11]

    B. Dykes. 2019. Effective data storytelling: how to drive change with data, narrative and visuals. John Wiley & Sons

  12. [12]

    Steve Graham, Karen Harris, and Michael Hebert. 2011. Informing Writing: The Benefits of Formative Assessment. A Report from Carnegie Corporation of New York. Carnegie Corporation of New York (2011)

  13. [13]

    Haertel, Terry P

    Geneva D. Haertel, Terry P. Vendlinski, Daisy Rutstein, Angela DeBarger, Britte H. Cheng, Eric B. Snow, Cynthia D’Angelo, Christopher J. Harris, Louise Yarnall, and Liliana Ructtinger. 2016. General Introduction to Evidence-Centered Design. In Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications, James W. Pellegrino and Edys S...

  14. [14]

    doi:10.4324/9780203781302-6

  15. [15]

    Taha Hassan, Bob Edmison, Daron Williams, Larry Cox II, Matthew Louvet, Bart Knijnenburg, and D Scott McCrickard. 2024. Simplify, consolidate, intervene: facilitating institutional support with mental models of learning management system use. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1–23

  16. [16]

    John Hattie and Helen Timperley. 2007. The power of feedback. Review of Educational Research 77, 1 (2007), 81–112

  17. [17]

    Eric Horvitz, Johnson Apacible, and Mudumbai Subramani. 1999. Notification, disruption, and memory: Effects of messaging interruptions on memory and performance. In Proceedings of INTERACT ’99: Seventh IFIP Conference on Human-Computer Interaction. IOS Press, 263–269

  18. [18]

    Ilgen, Cynthia D

    Daniel R. Ilgen, Cynthia D. Fisher, and M. Susan Taylor. 1979. Consequences of In- dividual Feedback on Behavior in Organizations. Journal of Applied Psychology 64, 4 (1979), 349–371. doi:10.1037/0021-9010.64.4.349

  19. [19]

    Qasim Khalid, Naveed Arshad, Nasir Khan, Taha Hassan, Fahad Javed, and Ja- hangir Ikram. 2014. Home energy management system using NILM and low-cost HAN. Journal of electronic science and technology 12, 1 (2014), 20–25

  20. [20]

    Kluger and Angelo DeNisi

    Avraham N. Kluger and Angelo DeNisi. 1996. The effects of feedback interven- tions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin 119, 2 (1996), 254–284

  21. [21]

    C. N. Knaflic. 2022. Storytelling with you: Plan, create, and deliver a stellar presentation. John Wiley & Sons

  22. [22]

    Kosara and J

    R. Kosara and J. Mackinlay. 2013. Storytelling: The next step for visualization. Computer 46, 5 (2013), 44–50

  23. [23]

    Kulhavy and Richard C

    Raymond W. Kulhavy and Richard C. Anderson. 1972. Delay-retention effect with multiple-choice tests. Journal of Educational Psychology 63, 5 (1972), 505–512

  24. [24]

    B. Lee, N. H. Riche, P. Isenberg, and S. Carpendale. 2015. More than telling a story: Transforming data into visually shared stories. IEEE Computer Graphics and Applications 35, 5 (2015), 84–90

  25. [25]

    P. M. Leidig and L. Cassel. 2020. ACM taskforce efforts on computing competen- cies for undergraduate data science curricula. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 519– 520

  26. [26]

    Noble Lo, Alan Wong, and Sumie Chan. 2025. The impact of generative AI on essay revisions and student engagement. Computers and Education Open (2025), 100249

  27. [27]

    Charles A MacArthur. 2016. Instruction in evaluation and revision. (2016)

  28. [28]

    National Academies of Sciences, Engineering, and Medicine. 2018. Data science for undergraduates: Opportunities and options. National Academies Press

  29. [29]

    National Academies of Sciences, Engineering, and Medicine. 2020. Roundtable on data science postsecondary education: A compilation of meeting highlights

  30. [30]

    Nolan and S

    D. Nolan and S. Stoudt. 2021. Communicating with data: The art of writing for data science. Oxford University Press

  31. [31]

    Ernesto Panadero, Anders Jonsson, Leire Pinedo, and Belén Fernández-Castilla

  32. [32]

    Effects of rubrics on academic performance, self-regulated learning, and self-efficacy: A meta-analytic review.Educational Psychology Review 35, 4 (2023), 113

  33. [33]

    Ploutz-Snyder, J

    Robert J. Ploutz-Snyder, J. Fiedler, and Alan H. Feiveson. 2014. Justifying small-n research in scientifically amazing settings: Challenging the notion that only "big-n" studies are worthwhile. Journal of Applied Physiology 116, 9 (2014), 1251–1252. doi:10.1152/japplphysiol.01340.2013

  34. [34]

    Peng Ren, Yanhua Wang, and Fang Zhao. 2023. Re-understanding of Data Sto- rytelling Tools from a Narrative Perspective. Visual Intelligence 1, 11 (2023). doi:10.1007/s44267-023-00011-0

  35. [35]

    Maha Sajid, Syed Ibrahim Mustafa Shah Bukhari, Bo Ji, and Brendan David-John

  36. [36]

    Just stop doing everything for now!

    " Just stop doing everything for now!": Understanding security attacks in remote collaborative mixed reality. In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 623–633

  37. [37]

    Donald A. Schön. 1983. The Reflective Practitioner: How Professionals Think in Action. Basic Books, New York

  38. [38]

    Segel and J

    E. Segel and J. Heer. 2010. Narrative visualization: Telling stories with data. IEEE Transactions on Visualization and Computer Graphics 16, 6 (2010), 1139–1148

  39. [39]

    Danqing Shi, Xinyue Xu, Fuling Sun, Yang Shi, and Nan Cao. 2020. Calliope: Automatic visual data story generation from a spreadsheet. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2020), 453–463

  40. [40]

    Showkat and E

    D. Showkat and E. P. Baumer. 2021. Where do stories come from? Examining the exploration process in investigative data journalism. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–31

  41. [41]

    Valerie J. Shute. 2008. Focus on formative feedback. Review of Educational Research 78, 1 (2008), 153–189

  42. [42]

    Timmers, Jet Braber-van den Broek, and Smits M

    Caroline F. Timmers, Jet Braber-van den Broek, and Smits M. van den Berg

  43. [43]

    Learning and Individual Differences 21 (2015), 123–134

    Motivational Beliefs, Self-Regulated Learning and Academic Performance in Higher Education: A Structural Equation Model. Learning and Individual Differences 21 (2015), 123–134. doi:10.1016/j.lindif.2010.07.007

  44. [44]

    Elizabeth J. Tisdell. 2025. Qualitative Research: A Guide to Design and Implementation (5 ed.). Jossey-Bass, Hoboken, New Jersey

  45. [45]

    Huichen Will Wang, Larry Birnbaum, and Vidya Setlur. 2025. Jupybara: Opera- tionalizing a design space for actionable data analysis and storytelling with llms. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. 1–24

  46. [46]

    Qing Wang, Zhutian Li, Sai Fu, Weiwei Cui, and Huamin Qu. 2018. Narvis: Authoring Narrative Slideshows for Introducing Data Visualization Designs. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2018), 779–

  47. [47]

    doi:10.1109/TVCG.2018.2865234

  48. [48]

    X. Wang, Y. Chen, A. G. Godley, and C. P. Rosé. 2018. Public Peer Review Motivates Higher Quality Feedback. In Proceedings of the International Conference of the Learning Sciences (ICLS). International Society of the Learning Sciences

  49. [49]

    G. Wu, S. Guo, J. Hoffswell, G. Y.-Y. Chan, R. A. Rossi, and E. Koh. 2023. Socrates: Data story generation via adaptive machine-guided elicitation of user feedback. IEEE Transactions on Visualization and Computer Graphics (2023)

  50. [50]

    Louise Yarnall, Hui Yang, Sophia Ouyang, and Lujie K Chen. 2025. Laying Foundations for Scalable Coaching for Data Storytelling: A Evidence-Centered Design Approach. InProceedings of the Twelfth ACM Conference on Learning@ Scale. 227–231

  51. [51]

    J. Zhao, S. Xu, S. Chandrasegaran, C. Bryan, F. Du, A. Mishra, et al. 2021. Chart- Story: Automated Partitioning, Layout, and Captioning of Charts into Comic- Style Narratives. arXiv preprint (2021). arXiv:2103.03996