Pith. sign in

REVIEW 2 major objections 6 minor 62 references

REVA: Supporting LLM-Generated Programming Feedback Validation at Scale Through User Attention-based Adaptation

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read REVA helps instructors review AI-generated programming feedback with better misconception coverage and less context switching.

desk verdict A real systems contribution with a well-designed study, but the headline quality numbers rest on an annotation process that needs more transparency before the claim is taken at face value. read the letter →

arxiv 2507.11470 v1 pith:CMIKR6YV submitted 2025-07-15 cs.HC

Xiaohang Tang , Sam Wong , Zicheng He , Yalong Yang , Yan Chen This is my paper
classification cs.HC
keywords LLM-generatedfeedbackprogrammingeducationhuman-AIcollaborationvalidationattention-basedadaptationrevisionpropagationcognitivecontextswitchingwithin-subjectuserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REVA is a human-AI system for course instructors who must validate large volumes of AI-generated programming feedback before sending it to students. The paper's central claim is that by learning from what instructors highlight and edit, the system can order the review queue around similar misconceptions and propagate accepted revisions to semantically similar submissions, cutting cognitive context-switching and repetitive work. In a within-subject lab study with 12 instructors, feedback validated with REVA covered misconceptions with higher recall (0.86 vs 0.55) and precision (0.90 vs 0.71), and scored higher on overall quality (0.34 vs 0.04) than feedback validated with a baseline version without these two features. The authors also report faster initial reactions to each new submission and lower perceived mental demand. If the claim holds, the instructor-validation bottleneck that currently limits LLM-generated feedback in real classes becomes substantially easier to scale.

What carries the argument

The carrying mechanism is attention-based adaptation, defined as dynamically reshaping the review workflow from implicit signals of instructor focus. Concretely, REVA's two load-bearing components are user-initiated semantic filters, which convert highlighted code or feedback into an LLM-generated filter that reorders and highlights upcoming submissions, and revision propagation, which extracts a revision goal from an accepted edit, matches semantically similar code-feedback pairs, and proposes adapted revisions for human verification. The system's assumption is that semantic repetition in student submissions is high enough that these matches are useful; an off-the-shelf large language model does the interpretation and matching.

What would settle it

Have independent annotators who are blind to the study condition and to the design team code all 475 code-feedback pairs for misconception ground truth and feedback coverage, require a reported inter-rater agreement such as Cohen's kappa above 0.8, and recompute the recall, precision, and quality comparisons; the central claim collapses if the REVA-vs-baseline differences no longer reach significance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that user attention, expressed through ordinary highlighting and editing actions, is a reliable signal for adapting a validation workflow. REVA captures this signal in two features: semantic filters, which an instructor creates by highlighting a problematic code segment or feedback phrase and which reorder the remaining submissions so similar issues are reviewed together; and revision propagation, which takes an accepted edit, extracts its goal, matches code and feedback patterns in upcoming submissions, and proposes adapted versions that the instructor can accept or reject. In the study, these features produced significantly higher misconception-coverage precision and recall, better overall feedback quality scores, more revisions per session, and lower reaction time when starting to validate a new code-feedback pair (18.69 vs 29.77 seconds), with no significant increase in validation time per pair.

Load-bearing premise

The quality gains rest on the assumption that the two researchers' hand-coded misconception ground truth and quality scores are valid, but no inter-rater reliability statistic is reported, the coders are on the design team, and only 89 of 475 feedback pairs (about 18%) were scored.

Editorial extensions

If this is right

  • Instructors can validate roughly the same number of feedback items per session while cutting the time spent before first action on each item, making it feasible to review feedback for larger course sections.
  • Accepted edits propagate to similar submissions, shifting instructor effort from repeating the same fix to spotting misconceptions the system hasn't caught.
  • The higher recall and precision of misconception coverage mean students are less likely to receive feedback that misses their actual error or includes irrelevant points.
  • Lower perceived mental demand and sustained revision engagement across a session suggest the approach reduces the fatigue that makes long validation sessions impractical.
  • Quality gains do not come at the cost of speed, since per-item validation time did not significantly increase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes, the filter-plus-propagation pattern could be tested in other high-volume human review of LLM output, such as essay grading, legal document review, or clinical note auditing, where a reviewer validates many similar drafts; this is an extrapolation, not a study result.
  • Because the study measured instructor-side quality rather than student learning, a field deployment comparing student outcomes with and without REVA-validated feedback would be the natural next test.
  • The 20-minute lab sessions may understate the benefit in real grading marathons, where fatigue and context-switch costs accumulate; a longer field study is needed to see whether the engagement effect persists.
  • Using highlighting as an implicit priority signal could be adopted more broadly as a low-cost way to adapt AI workflows without explicit user configuration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents REVA, a human-AI system that supports instructors in reviewing and revising large volumes of LLM-generated programming feedback. REVA contributes two core mechanisms: attention-based semantic filtering and queue reordering, and automatic propagation of instructor revisions to semantically similar code-feedback pairs. The authors report a formative study (N=20) that motivates three design goals, and a within-subjects lab study (N=12) comparing REVA against a baseline without semantic filtering and propagation. The main quantitative claims are that REVA significantly increases the number of revisions, reduces reaction time to first action, and improves misconception-coverage precision (0.90 vs. 0.71), recall (0.86 vs. 0.55), and overall feedback quality (0.34 vs. 0.04), as summarized in Table 2. The paper concludes that REVA significantly improves both the efficiency and quality of feedback review and revision compared with the baseline.

Significance. If the main results are trustworthy, REVA addresses a real and timely bottleneck: enabling instructors to validate AI-generated programming feedback at scale without sacrificing quality. The study has notable strengths: it uses real student code submissions, a counterbalanced within-subjects design, a baseline that isolates the two novel features, and descriptive statistics for all NASA-TLX items. The reported effect sizes for precision and recall are large, and the system contribution is well motivated by the formative findings on context switching and revision reuse. The principal weakness is not the system design but the measurement of feedback quality: the ground-truth annotation is produced by two members of the research team with no reported inter-rater reliability or blinding, which places the central quality claim at risk. The paper does not contain mathematical derivations or fitted parameters, so the usual circularity concerns are absent; the concern is instead about the validity and independence of the outcome measure.

major comments (2)
  1. [§7, Table 2] The central quality claim rests on the precision/recall/overall-quality comparisons in Table 2, but the ground-truth annotation is not yet shown to be reliable or unbiased. The manuscript reports that two researchers annotated misconception types and coded feedback coverage, but it does not report an inter-rater reliability statistic (e.g., Cohen's kappa for misconception annotation, coverage coding, or the -1/0/1 quality scale), despite citing reliability norms [37]. It also does not state whether the coders were blind to condition or whether the misconception taxonomy was independently validated. This is load-bearing because REVA's propagation mechanism inserts instructor-revised misconception text into subsequent feedback; if coders were not blind, they could count system-inserted content as coverage and thereby inflate the recall and precision gaps. The limitations section (8.4) does not acknowledge this measurement threat. Please report IRR, describe coder training and independence, state whether coders were blind, and provide an external validation or third-coder check of the misconception ground truth.
  2. [§7, reaction-time analysis] The efficiency claim relies in part on the reaction-time result, but the metric as defined is confounded by the propagation feature. Reaction time is defined as the time before performing the first concrete action when validating a new feedback pair. In REVA, propagated revision suggestions are already displayed when the participant opens a propagated pair (Section 5.2.5), so the first concrete action can be accepting a system-generated suggestion rather than an instructor-initiated sensemaking action. The lower reaction time may therefore reflect system initiative rather than reduced context-switching cost. Please report the distribution of first-action types (e.g., accept propagation vs. edit vs. filter creation), or use a metric such as time to first independent edit, to separate these accounts.
minor comments (6)
  1. [§1, §5.2.1, §5.2.3, §8.2] Typos and wording: 'An novel system' should be 'A novel system'; 'maximum feedback's impact' should be 'maximize feedback's impact'; 'Based on the our observed' should be 'Based on our observed'; 'participatns' should be 'participants'.
  2. [Table 1] The TLX table reports medians and means but no inferential statistics; if no hypothesis tests were run for these items, say so explicitly, and avoid framing the qualitative observation that participants 'perceived low levels' as a comparative result.
  3. [§7] The overall quality score is a three-level ordinal scale (-1/0/1) but is analyzed with a t-test on means; consider a mixed-effects model or nonparametric test that accounts for repeated measures and the bounded, ordinal nature of the outcome, and report effect sizes or confidence intervals.
  4. [§7] The paper runs many outcome tests (TLX dimensions, revision counts, precision, recall, quality, reaction time) without any multiple-comparison correction; report adjusted p-values or justify the planned family of tests.
  5. [§7] The sentence 'we sampled 89 out of 475 (Around 18% coverage)' should be edited for grammar and capitalization, and the paper should clarify whether the 89 pairs are unique code-feedback pairs or pair-participant observations.
  6. [§3.3.1] The formative study also reports that 'two researchers independently annotated' feedback quality without an inter-rater reliability statistic; since this classification informs the design goals, please report agreement there as well.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: REVA's claims rest on an empirical within-subjects study with externally annotated ground truth, not on a self-referential derivation.

full rationale

REVA does not present a formal derivation or fitted model; its central claims are empirical comparisons of instructor behavior and output quality in a within-subjects study. The precision/recall/quality metrics in Table 2 are computed against a misconception ground truth that was annotated independently of the system's outputs (Section 7: "we annotated misconception types for each code, and used it as the ground truth for issue coverage"), with the sampled pairs randomly shuffled before feedback annotation. Although the two researchers who coded the data are part of the research team and no inter-rater reliability statistic is reported, that is a measurement-validation threat rather than circular reasoning: the ground truth labels are not defined in terms of REVA's outputs, the study was not used to fit any parameter, and there is no equation or definitional chain that makes the outcome equal to the input. The authors' self-citations in the related-work sections are contextual and not load-bearing for the evaluation. REVA's propagation mechanism can inflate revision counts or content coverage by reusing instructor edits, but that is the intervention being tested, and its value is judged by instructor acceptance and by the independently defined misconception taxonomy. The central claims therefore have independent empirical content; concerns about annotation reliability, blinding, and the 18% sample belong under correctness risk, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric parameters are fitted; the system relies on LLM matching. The key unvalidated assumptions are the cognitive-switching theory, LLM semantic matching accuracy, and annotation reliability. These are external premises, not derived in the paper.

assumptions (4)
  • domain assumption Task-switching costs generalize to programming feedback review.
    The design leans on Monsell [39] and Wylie and Allport [55] to assume that grouping similar feedback reduces mental transitions. The study does not directly measure switch costs separately from reaction time.
  • domain assumption LLM-generated feedback is unreliable enough to require instructor validation.
    Motivates the problem; supported by cited work on LLM inaccuracies, but treated as a premise for why REVA is needed.
  • domain assumption Semantic repetition in code-feedback pairs can be detected reliably by off-the-shelf LLMs and embeddings.
    REVA's filter and propagation depend on this; no precision evaluation of the matching is provided beyond stress-testing claims in Section 5.3.
  • domain assumption The two researchers' misconception annotations are an accurate ground truth.
    No inter-rater reliability is reported; this grounds precision, recall, and quality scores in Section 7.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REVA: Supporting LLM-Generated Programming Feedback Validation at Scale Through User Attention-based Adaptation." pith.science (2026). https://pith.science/paper/CMIKR6YV

@misc{pith2026250711470,
  author       = {Pith},
  title        = {Pith review of: REVA: Supporting LLM-Generated Programming Feedback Validation at Scale Through User Attention-based Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMIKR6YV}},
  note         = {Machine review of arXiv:2507.11470}
}
read the original abstract

This paper introduces REVA, a human-AI system that expedites instructor review of voluminous AI-generated programming feedback by sequencing submissions to minimize cognitive context shifts and propagating instructor-driven revisions across semantically similar instances. REVA introduces a novel approach to human-AI collaboration in educational feedback by adaptively learning from instructors' attention in the review and revision process to continuously improve the feedback validation process. REVA's usefulness and effectiveness in improving feedback quality and the overall feedback review process were evaluated through a within-subjects lab study with 12 participants.

Figures

Figures reproduced from arXiv: 2507.11470 by the authors.

Figure 1
Figure 1. The workflow of REVA. There are three steps to reviewing and providing AI-generated programming feedback at scale using REVA. (Step 1) Collect Student Code Submissions where instructors gather submissions from hundreds of students to a coding exercise. (Step 2) Instructor Review and Revise AI Feedback, the core innovation of REVA, where instructors efficiently validate feedback through two key features: defining fil… view at source ↗
Figure 2
Figure 2. User Interface of the probe system for creating and reviewing LLM-generated feedback. (1) Instructor can select different feedback types, which will select a subset of (2) feedback components. (3) Instructors can then generate feedback for preview. 3.3.3 Instructors prefer to review similar programming issues or feedback in the same batch to reduce context switching. When re￾viewing large amounts of feedback, instru… view at source ↗
Figure 3
Figure 3. User Interface for REVA: (1) A customized feedback template is created using (2) modular feedback components like issue and strategy, with (3) feedback panel showing the generated feedback. Users can apply (4) predefined filters or (5) create user-defined filters to target specific errors, which populate the (6) filtered feedback list with matching submissions. Once feedback is applied, (7) a propagated revision lis… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Revisions: REVA supports different forms of human-AI feedback revision. (1a) Highlighting text and (1b) highlighting code will allow user to edit feedback. (2) Changes are be made inline and (3) a description of changes are displayed decision ( [PITH_FULL_IMAGE:figure…
Figure 5
Figure 5. Figure 5: Abstraction Levels: Feedback content can be controlled by the level of abstraction of the content through a (1) Slider and (2) Grid that shows all feedback combinations. interactions in the review panel. Besides manually typing, to al￾low users to initiate the revision…
Figure 6
Figure 6. Figure 6: ) that capture users’ attention model at the semantic level, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 6
Figure 6. Figure 6: Filtering: Filtering allows instructor to focus on specific issues in student submissions. (1) Filters are initiated by directly highlighting related comment or code. (2) After clicking "Add Filter", a one line summary of the filter is created by LLM. Upcoming submissi…
Figure 7
Figure 7. Figure 7: Feedback propagation: Feedback propagation automatically propagate revision to similar feedback. After the user (1) accepts a revision suggestion, (2) REVA extracts the revision with code and feedback patterns, and (3) propagates the revision to applicable feedback. Wh…
Figure 8
Figure 8. Figure 8: Number of revisions (orange) completed by P8 through revision suggestion with the cost of time (blue) for each [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

62 extracted references · 31 canonical work pages

  1. [37]

    Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 72 (nov 2019), 23 pages. doi:10.1145/3359174

    doi:10.1145/3359174 2019

  2. [1]

    Ian Arawjo, Chelse Swoopes, Priyan Vaithilingam, Martin Wattenberg, and Elena L Glassman. 2024. Chainforge: A visual toolkit for prompt engineering and llm hypothesis testing. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems . 1–18

    work page 2024

  3. [2]

    Ebrahim Babaei, Tilman Dingler, Benjamin Tag, and Eduardo Velloso. 2025. Should we use the NASA-TLX in HCI? A review of theoretical and methodological issues around Mental Workload Measurement. International Journal of Human- Computer Studies (2025), 103515

    work page 2025

  4. [3]

    Brian P Bailey and Shamsi T Iqbal. 2008. Understanding changes in mental work- load during execution of goal-directed tasks and its application for interruption management. ACM Transactions on Computer-Human Interaction (TOCHI) 14, 4 (2008), 1–28

    work page 2008

  5. [4]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021). https://api.semanticscholar.org/CorpusID:262580630

    work page 2021

  6. [5]

    Bernstein, Greg Little, Robert C

    Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery...

    arXiv 2010

  7. [6]

    Jelmer P Borst, Niels A Taatgen, and Hedderik van Rijn. 2015. What makes interruptions disruptive? A process-model account of the effects of the problem state bottleneck on task interruption and resumption. In Proceedings of the 33rd annual ACM conference on human factors in computing systems . 2971–2980

    work page 2015

  8. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

Show all 62 references
  1. [8]

    Deborah L Butler and Philip H Winne. 1995. Feedback and self-regulated learning: A theoretical synthesis. Review of educational research 65, 3 (1995), 245–281

    1995

  2. [9]

    Çiğdem Suzan Çardak and Kıymet Selvi. 2016. Increasing teacher candidates’ ways of interaction and levels of learning through action research in a blended course. Computers in Human Behavior 61 (2016), 488–506

    2016

  3. [10]

    Yan Chen, Jaylin Herskovitz, Gabriel Matute, April Wang, Sang Won Lee, Walter S Lasecki, and Steve Oney. 2020. EdCode: Towards Personalized Support at Scale for Remote Assistance in CS Education. In2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . ...

    2020

  4. [11]

    Anna Comas-Quinn. 2011. Learning to teach online or learning to become an online teacher: An exploration of teachers’ experiences in a blended learning course. ReCALL 23, 3 (2011), 218–232

    2011

  5. [12]

    Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press

    1993

  6. [13]

    Iria Estévez-Ayres, Patricia Callejo, Miguel Ángel Hombrados-Herrera, Carlos Alario-Hoyos, and Carlos Delgado Kloos. 2024. Evaluation of LLM Tools for Feedback Generation in a Course on Concurrent Programming. International Journal of Artificial Intelligence in Education (2024), 1–17

    2024

  7. [14]

    Hagit Gabbay and Anat Cohen. 2024. Combining LLM-Generated and Test- Based Feedback in a MOOC for Programming. In Proceedings of the Eleventh ACM Conference on Learning@ Scale . 177–187

    2024

  8. [16]

    Kummerfeld, and Elena L

    Katy Ilonka Gero, Chelse Swoopes, Ziwei Gu, Jonathan K. Kummerfeld, and Elena L. Glassman. 2024. Supporting Sensemaking of Large Language Model Outputs at Scale. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Associati...

    2024 doi

  9. [17]

    Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller

  10. [18]

    Elena L Glassman, Tianyi Zhang, Björn Hartmann, and Miryung Kim. 2018. Visualizing API usage examples at scale. InProceedings of the 2018 CHI Conference on Human Factors in Computing Systems . 1–12

    2018

  11. [19]

    Kummerfeld, and Elena L

    Ziwei Gu, Ian Arawjo, Kenneth Li, Jonathan K. Kummerfeld, and Elena L. Glass- man. 2024. An AI-Resilient Text Rendering Technique for Reading and Skimming Documents. In Proceedings of the 2024 CHI Conference on Human Factors in Com- puting Systems (Honolulu, HI, USA) (CHI ’24)...

    2024

  12. [20]

    Sumit Gulwani. 2011. Automating string processing in spreadsheets using input- output examples. ACM Sigplan Notices 46, 1 (2011), 317–330

    2011

  13. [21]

    John Hattie. 2012. Visible learning for teachers: Maximizing impact on learning . Routledge

    2012

  14. [22]

    John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research 77, 1 (2007), 81–112

    2007

  15. [23]

    Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017. Writing reusable code feedback at scale with mixed-initiative program synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale (Cambridge...

    2017

  16. [24]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232 ...

    2023 arXiv

  17. [25]

    Lucas Jasper Jacobsen, Jonathan Rohlmann, and Kira Elena Weber. 2025. AI Feedback in Education: The Impact of Prompt Design and Human Expertise on LLM Performance. (2025)

    2025

  18. [26]

    Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue: Exploring large language model responses with interactive diagrams. In Proceed- ings of the 36th annual ACM symposium on user interface software and technology . 1–20

    2023

  19. [27]

    Hita Kambhamettu, Jamie Flores, and Andrew Head. 2024. Traceable Text: Deepening Reading of AI-Generated Summaries with Phrase-Level Provenance REV A: Supporting LLM-Generated Programming Feedback Validation at Scale Links. arXiv preprint arXiv:2409.13099 (2024)

    2024 arXiv

  20. [28]

    Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011. Wran- gler: interactive visual specification of data transformation scripts. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association fo...

    2011

  21. [29]

    Onur Karademir, Atezaz Ahmad, Jan Schneider, Daniele Di Mitri, Ioana Jivet, and Hendrik Drachsler. 2021. Designing the Learning Analytics Cockpit - A Dashboard that Enables Interventions. In Methodologies and Intelligent Systems for Technology Enhanced Learning, 11th Internati...

    2021

  22. [30]

    Nguyen, Alfie Abdul-Rahman, Benjamin Bach, Min Chen, Euan Freeman, and Cagatay Turkay

    Saiful Khan, Phong H. Nguyen, Alfie Abdul-Rahman, Benjamin Bach, Min Chen, Euan Freeman, and Cagatay Turkay. 2022. Propagating Visual Designs to Nu- merous Plots and Dashboards. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2022), 86–95. doi:10.1109/TVCG.2021.3114828

    2022

  23. [31]

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–21

    2024

  24. [32]

    Thomas Kosch, Jakob Karolus, Johannes Zagermann, Harald Reiterer, Albrecht Schmidt, and Paweł W. Woźniak. 2023. A Survey on Measuring Cognitive Workload in Human-Computer Interaction. ACM Comput. Surv. 55, 13s, Article 283 (July 2023), 39 pages. doi:10.1145/3582272

    2023 doi

  25. [33]

    Philippe Laban, Jesse Vig, Marti Hearst, Caiming Xiong, and Chien-Sheng Wu

  26. [34]

    Henry Lieberman. 2001. Your wish is my command: Programming by example . Elsevier

    2001

  27. [35]

    Alan Lundgard and Arvind Satyanarayan. 2022. Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2022), 1073–1083. doi:10.1109/TVCG.2021.3114770

    2022

  28. [36]

    Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the gpt-3 large language model. In Proceedings of the 2022 ACM conference on international computing education research-volume 2 . 37–39

    2022

  29. [38]

    Bryan Min and Haijun Xia. 2025. Feedforward in Generative AI: Opportunities for a Design Space. arXiv preprint arXiv:2502.14229 (2025)

    2025 arXiv

  30. [39]

    Stephen Monsell. 2003. Task switching. Trends in cognitive sciences 7, 3 (2003), 134–140

    2003

  31. [40]

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  32. [41]

    Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How Beginning Programmers and Code LLMs (Mis) read Each Other. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–26

    2024

  33. [42]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Franc...

    2023

  34. [43]

    Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. 2024. Au- tomating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for ...

    2024

  35. [44]

    Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review 106, 4 (1999), 643

    1999

  36. [45]

    Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard G Baraniuk. 2024. Pedagogical alignment of large language models. arXiv preprint arXiv:2402.05000 (2024)

    2024 arXiv

  37. [46]

    Pragnya Sridhar, Aidan Doyle, Arav Agarwal, Chris Bogart, Jaromir Savelka, and Majd F. Sakr. 2023. Harnessing LLMs in Curricular Design: Using GPT- 4 to Support Authoring of Learning Objectives. In LLM@AIED. https://api. semanticscholar.org/CorpusID:259308858

    2023

  38. [47]

    Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling multilevel exploration and sensemaking with large language models. In Proceedings of the 36th annual ACM symposium on user interface software and technology. 1–18

    2023

  39. [48]

    Wannapon Suraworachet, Jennifer Seon, and Mutlu Cukurova. 2024. Pre- dicting challenge moments from students’ discourse: A comparison of GPT- 4 to two traditional natural language processing approaches. In Proceedings of the 14th Learning Analytics and Knowledge Conference (Ky...

    2024

  40. [49]

    Xiaohang Tang, Sam Wong, Kevin Pu, Xi Chen, Yalong Yang, and Yan Chen. 2024. VizGroup: An AI-assisted Event-driven System for Collaborative Programming Learning Analytics. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA...

    2024

  41. [50]

    Janneke Van de Pol, Monique Volman, and Jos Beishuizen. 2010. Scaffolding in teacher–student interaction: A decade of research. Educational psychology review 22 (2010), 271–296

    2010

  42. [51]

    April Yi Wang, Yan Chen, John Joon Young Chung, Christopher Brooks, and Steve Oney. 2021. PuzzleMe: Leveraging Peer Assessment for In-Class Programming Exercises. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–24

    2021

  43. [52]

    Theodora Worledge, Tatsunori Hashimoto, and Carlos Guestrin. 2024. The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations. arXiv preprint arXiv:2411.17375 (2024)

    2024 arXiv

  44. [53]

    Shaffer, and Yan Chen

    Tong Wu, Xiaohang Tang, Sam Wong, Xi Chen, Clifford A. Shaffer, and Yan Chen

  45. [54]

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Transpar- ent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Comput- ing Systems (New Orleans, LA, USA) (CHI ’22). ...

    2022

  46. [55]

    switch costs

    Glenn Wylie and Alan Allport. 2000. Task switching and the measurement of “switch costs”. Psychological research 63 (2000), 212–233

    2000

  47. [56]

    Zexing Xu, Zhuang Luo, Yichuan Li, Kyumin Lee, and S Rasoul Etesami. 2024. From Critique to Clarity: A Pathway to Faithful and Personalized Code Explana- tions with Large Language Models. arXiv preprint arXiv:2501.14731 (2024)

    2024 arXiv

  48. [57]

    Gautam Yadav, Ying-Jui Tseng, and Xiaolin Ni. 2023. Contextualizing problems to student interests at scale in intelligent tutoring system using large language models. arXiv preprint arXiv:2306.00190 (2023)

    2023 arXiv

  49. [58]

    Kexin Bella Yang, Vanessa Echeverria, Zijing Lu, Hongyu Mao, Kenneth Hol- stein, Nikol Rummel, and Vincent Aleven. 2023. Pair-up: prototyping human-AI co-orchestration of dynamic transitions between individual and collaborative learning in the classroom. In Proceedings of the ...

    2023

  50. [59]

    Ashley Ge Zhang, Yan Chen, and Steve Oney. 2023. VizProg: Identifying Mis- understandings By Visualizing Students’ Coding Progress. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–16

    2023

  51. [60]

    Ashley Ge Zhang, Xiaohang Tang, Steve Oney, and Yan Chen. 2024. CFlow: Supporting Semantic Flow Analysis of Students’ Code in Programming Problems at Scale. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (Atlanta, GA, USA) (L@S ’24). Association for Computin...

    2024

  52. [2015]

    ACM Transactions on Computer-Human Interaction (TOCHI) 22, 2, Article 7 (March 2015), 35 pages

    OverCode: Visualizing variation in student solutions to programming problems at scale. ACM Transactions on Computer-Human Interaction (TOCHI) 22, 2, Article 7 (March 2015), 35 pages. doi:10.1145/2699751

    2015 doi

  53. [2024]

    In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24)

    Beyond the Chat: Executable and Verifiable Text-Editing with LLMs. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST ’24). Association for Computing Machin- ery, New York, NY, USA, Article 20, 23 pages. doi:10...

  54. [2025]

    (2025), 1260–1266

    The Impact of Group Discussion and Formation on Student Performance: An Experience Report in a Large CS1 Course. (2025), 1260–1266. doi:10.1145/ 3641554.3701973

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.