Pith. sign in

REVIEW 3 major objections 6 minor 82 references

AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AutoCue claims that making implicit mouse and keyboard inputs visible as standardized visual cues in screencast tutorials reduces learner rewinds, stuck events, and completion time, supported by a 24-participant between-subjects study in…

desk verdict A promising workflow paper with a solid user-study core, but the automatic-inference table is internally inconsistent and the abstract overstates the subjective results. read the letter →

arxiv 2608.04910 v1 pith:PP4FVR5K submitted 2026-08-05 cs.HC

Shengyang Luo , Shengyao Luo , Xiaolei Guo , Fengze Zhang , James Liang , Yingjie Victor Chen This is my paper · ORCID
classification cs.HC
keywords screencasttutorialssoftwarelearninginstructionalvisualcueshuman-in-the-loopAItutorialaugmentationmultimodalLLMcognitiveloadAutodeskMaya
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Following a screencast tutorial often fails because the instructor's mouse clicks, double-clicks, drags, and keyboard-modified gestures remain invisible or implicit in the video. The paper claims that externalizing these inputs as a consistent set of visual cues—mouse-button icons, click indicators, a fixed keyboard overlay, and integrated key-plus-mouse symbols placed near the cursor—reduces how often learners rewind and get stuck. AutoCue is a multimodal-LLM pipeline that finds candidate interaction moments from frame-to-frame differences, OCR-readable interface text, narration, and official software documentation, then labels each moment from a constrained interaction menu and renders aligned cue layers or editable artifacts. In a between-subjects study with 24 Maya users, the augmented tutorial gave a median completion time of 1067 seconds versus 1404 seconds and a median of 3 breakdowns versus 7, with significant group differences and large effect sizes. On UI-mediated events (menus, dialogs, window transitions), the automatic component reaches a micro-averaged recall of 95.10% and precision of 96.91%.

What carries the argument

AutoCue's central object is a three-module augmentation pipeline: difference-driven frame analysis (M1) that selects candidate timestamps, constrained multimodal LLM inference (M2) that maps an evidence bundle to one interaction label from an official documentation-derived menu, and cue rendering and export (M3) that overlays the tutorial or writes editable JSON artifacts. The instructional visual cue grammar is the load-bearing design artifact: icon-based mouse buttons, circular indicators separating single from double clicks, a consistent bottom-screen keyboard overlay, and integrated key-plus-mouse composite cues placed at the cursor to respect spatial contiguity and reduce split attention. What carries the argument is the combination of evidence grounding (frames, OCR, narration) with a constrained label space, so inferences stay renderable and reviewable.

What would settle it

Take several raw screencasts across different applications, run AutoCue's automatic component with no expert completion, and compare learners against the original videos on completion time and breakdown frequency. If the benefit in the Maya study disappears or flips when the 11 expert-completed cues are removed, the claim that the AutoCue workflow itself reduces breakdowns would be falsified; a simpler check compares auto-only cues against expert-completed cues on the same tutorial.

Watch

Extended reading notes

Core claim

The central claim is that much of the friction in following software screencasts comes from missing input information, and that this information can be recovered from the screencast itself and visualized with a coherent cue grammar. AutoCue treats input inference as a constrained labeling problem: a multimodal LLM receives local frame pairs, an ROI with OCR tokens, time-aligned narration, and a documentation-derived interaction menu, and must return exactly one interaction label per candidate timestamp. The empirical core is the user-study result that the AutoCue-workflow-augmented tutorial made learners faster and less prone to breakdowns than the original tutorial: completion time differed significantly ($U=20$, $p=0.003$) with Cliff's $\delta=-0.72$, and breakdown frequency differed significantly ($U=25.5$, $p=0.007$) with Cliff's $\delta=-0.65$. The technical validation adds that UI-mediated events are recovered with 95.10% micro-averaged recall and 96.91% precision, supporting the claim that the automatic part of the pipeline handles the interaction class with the clearest visual feedback.

Load-bearing premise

The load-bearing premise is that one Maya cup-modeling tutorial, with an expert manually adding 11 of the 24 cues, is representative enough of real screencast learning that the measured reduction in breakdowns and completion time transfers to other tutorials and to the fully automatic pipeline.

Editorial extensions

If this is right

  • Creators of raw screencasts without input metadata can produce a cue-augmented tutorial in a semi-automated workflow instead of hand-authoring every click, drag, and modifier.
  • Learners following an AutoCue-augmented tutorial are expected to rewind less, get stuck less often, and finish tasks faster, at least in feature-rich 3D applications comparable to Maya.
  • The high recall and precision on UI-mediated events implies that menus, tool dialogs, and window transitions can be augmented automatically with near-manual accuracy, while ambiguous geometry or state changes still need human review.
  • Because the software-specific knowledge sits in the M2 interaction menu, the pipeline transfers to other GUI applications with observable visual feedback by swapping documentation-derived labels and prompts.
  • A standardized cue vocabulary applied across tutorials would reduce the cost learners pay for re-learning cue conventions whenever they switch creators or platforms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to vary cue density deliberately: the study's diminishing-returns pattern suggests that augmenting only a handful of high-stakes cues, such as the double-click that opens the Revolve window, may account for most of the benefit, and that exhaustive cueing would add little.
  • The same evidence-bundle machinery could be pointed at other implicit states, such as selection sets, object modes, or view changes, by extending the interaction menu, effectively turning legacy screencasts into structured, searchable action transcripts.
  • One experienced participant's complaint about cue size hints at an expertise-reversal effect: an adaptive cue grammar that reduces cue persistence or opacity for advanced users would likely outperform the fixed overlay, which is a directly testable follow-up.
  • The editable JSON layer separates inference from rendering, so cue layers could be versioned or crowdsourced over old tutorials, converting a library of legacy screencasts into consistently styled learning materials.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents AutoCue, a multimodal LLM-assisted, human-in-the-loop pipeline that adds instructional visual cues to screencast tutorials lacking input metadata. The authors report a formative survey (n=87 valid) and contextual inquiry (n=8) identifying missing mouse, keyboard, and keyboard-modified input information as a key cause of rewinds and getting stuck, then design a cue grammar and instantiate the pipeline in Autodesk Maya. Technical validation on 30 clips from 5 Maya tutorials reports micro-averaged recall of 95.10% and precision of 96.91% for UI-mediated events. A between-subjects user study (N=24, 12 per condition) comparing the raw tutorial with an AutoCue-workflow-augmented tutorial finds significantly faster task completion (median 1067 s vs 1404 s, U=20, p=0.003) and fewer breakdowns (median 3 vs 7, U=25.5, p=0.007), with large effect sizes; subjective ratings were directionally favorable but not significant at p<0.05.

Significance. If the reported results hold, the paper makes a useful contribution: it provides empirical evidence that externalizing implicit input actions through a consistent, action-proximal cue grammar can improve follow-along learning in feature-rich software, and it demonstrates a plausible architecture for generating such cues from raw screencasts while constraining LLM output to documentation-derived interaction menus. The user-study analysis is methodologically sound for a small between-subjects design: appropriate nonparametric tests, effect sizes, and shift estimates are reported, and the paper is careful to frame the evaluation as testing the human-in-the-loop workflow rather than a fully autonomous system. The main caveat is that the automatic-validity claim currently rests on Table 1, whose raw counts are internally inconsistent, so that part of the contribution needs correction before the reported numbers can be accepted.

major comments (3)
  1. [Section 6.5.2, Table 1] Table 1 is internally inconsistent under the standard definitions stated in its caption. The recall column reports correct predictions over 102 validated events (28+15+22+15+17), while the precision column reports correct predictions over 97 predicted events (27+15+20+15+17); if the same true-positive set underlies both metrics, the two numerators must agree, and the reported micro-averaged recall (95.10%) and precision (96.91%) cannot both hold. Specifically, V1 reports 28/29 recall but 27/28 precision, and V3 reports 22/23 recall but 20/22 precision, which would imply that an event counted as correct for recall is not counted as correct for precision. Because Section 6.5.2's automatic-validity claim is supported solely by these counts, the authors must either correct the raw counts, clarify how true positives were defined separately for recall and precision, or recompute the micro-averages before the claim can be accepted.
  2. [Abstract and Section 7.3.2] The abstract's statement that the AutoCue-augmented tutorial 'showed improved learner-reported experience' overstates the reported results. In Section 7.3.2, none of the three Likert comparisons reached p<0.05 (Q1: p=0.196; Q2: p=0.083; Q3: p=0.102), and the text itself describes only 'directional trends.' The abstract should be revised to say that subjective measures were directionally more favorable, or the subjective comparisons should be omitted from the summary claims.
  3. [Section 7 and Section 8.5] The user study compared the raw tutorial against an augmented tutorial in which an expert instructor completed 11 of 24 cues by hand, so the study establishes the value of the human-in-the-loop AutoCue workflow, not of fully automatic augmentation. The paper mostly scopes its claims correctly, but the conclusion and abstract should make this boundary explicit to prevent readers from attributing the observed benefits to automatic inference alone; Section 8.5's limitation statement should be reflected in the contribution wording.
minor comments (6)
  1. [Section 6.2.1 and Section 6.4.2] Several pipeline parameters that likely affect the reported metrics are not given values or a sensitivity analysis, including the MAD and Canny thresholds, the Otsu/morphological processing settings, the 0.5-second frame sampling interval, the narration alignment window, and the 0.80 high-confidence inclusion threshold.
  2. [Section 6.5.1] The expert ground-truth annotations are described as produced by two expert Maya instructors, but no inter-annotator agreement is reported; since recall and precision are computed against this ground truth, reporting agreement or a reconciliation procedure would strengthen the validation.
  3. [Section 7.3.1] The Cliff's delta for breakdown frequency is reported only as a point estimate (-0.65), while the completion-time effect includes a 95% confidence interval; adding a confidence interval for the breakdown effect would make the two effect-size reports consistent.
  4. [Table 1] The caption should define how a prediction is counted as 'correct' (for example, exact label match with what timestamp tolerance and ROI overlap), since that definition is needed to adjudicate the recall/precision numerator discrepancy.
  5. [Section 6.5.2] The paper reports per-video standard deviations for recall and precision but does not describe how these relate to the micro-averaged values; clarifying whether these are standard deviations across the five V1-V5 clips or across the 30 segments would avoid ambiguity.
  6. [General] No data or artifact availability statement is included; releasing the interaction menu, the exact LLM prompt, the annotated validation data, and the augmented tutorial video would support reproducibility and help resolve the counting discrepancy in Table 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AutoCue's claims are empirically evaluated against external annotations and an A/B user study, with no fitted inputs renamed as predictions and no load-bearing self-citation chain.

full rationale

The paper's derivation chain is empirical rather than analytic, and I found no step where a claimed result reduces by construction to its own inputs. The formative survey and contextual inquiry motivate design goals, but the system itself is evaluated in two independent, external ways. First, the technical validation compares AutoCue's automatically generated cues for UI-mediated events against ground-truth annotations made by two expert Maya instructors, which is an external benchmark and not derived from the system's own evidence bundles. Second, the user study is a between-subjects A/B comparison of the original raw tutorial against the AutoCue-workflow-augmented tutorial, measuring task completion time, breakdowns, and learner-reported experience; no parameter is fitted to those outcome measures and then re-reported as a prediction. The cue grammar is grounded in externally published multimedia learning and cognitive load theory, not in a self-citation, and no uniqueness theorem from the authors' prior work is invoked to force a choice. The closest structural feature is the paper's voluntary scoping of automatic validation to UI-mediated events, which are defined as actions with explicit visual or textual interface feedback; this is precisely the evidential tier the M1/M2 pipeline is designed to exploit. That scoping narrows the claim rather than making it tautological, because correct recognition still requires the LLM to select the right interaction label among a constrained menu, and the reported recall/precision numbers come from testing against expert labels, not from the definitions. Similarly, the fact that the expert instructor completed 11 of 24 cues in the user study is an external-validity limitation about what the workflow includes, not a circularity, since the comparison is of the complete workflows, not of the automatic component alone. The internal inconsistency in Table 1's raw recall and precision counts (the true-positive numerators are 97 for recall but 94 for precision, which cannot both be correct under standard definitions) is a correctness or reproducibility concern that needs re-derivation, but it is not an instance of the paper's conclusion being equivalent to an input. Overall, the paper's claims are supported by independent measurements and do not exhibit the circular patterns this pass targets.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not depend on fitted numerical parameters; the free parameters are pipeline settings that affect coverage. The main epistemic cost is the domain assumption that interactions leave observable UI feedback and that the UI-mediated subset carries the learning benefit. No new physical or theoretical entities are introduced; the visual cues are rendered design elements, not postulated mechanisms.

free parameters (4)
  • High-confidence inclusion threshold = 0.80
    Section 6.4.2: predictions below 0.80 are kept in the editable artifact and not shown in the automatic video. The threshold is a design choice that determines which cues learners see.
  • Frame sampling interval = 0.5 s
    Section 6.5.3: the pipeline samples frames at 0.5-second intervals for multimodal inference, and the authors note this can miss brief UI events.
  • MAD and Canny edge thresholds = not reported
    Section 6.2.1: candidate moments trigger when either signal exceeds its threshold, but the paper does not report the threshold values.
  • Narration alignment window = not reported
    Section 6.2.3: narration segments are aligned within a fixed temporal window; the window size is not specified.
assumptions (4)
  • standard math L1 pixel distance, Otsu thresholding, and Canny edge detection are adequate to localize interaction moments in screencasts.
    Invoked in Section 6.2.1 as the difference-driven action filter; these are standard image-processing tools used without validation for this domain.
  • domain assumption Critical input actions can be inferred from screen visuals, OCR text, narration, and documentation-derived interaction menus.
    Core assumption of M2 (Section 6.3); the paper validates this only for UI-mediated events with explicit feedback.
  • domain assumption UI-mediated events are representative of the learner-relevant critical interactions that cause rewinds and stuck events.
    Section 6.5.1 says UI-mediated events account for about 40% of learner-relevant critical steps and restricts all reported metrics to this tier; whether this subset carries the user-study benefit is not directly measured.
  • ad hoc to paper An expert instructor's completion of 11 of 24 cues yields a tutorial that represents the AutoCue workflow.
    Section 7: the user study compares the original tutorial against the expert-completed augmented tutorial, so the measured effect includes human authoring effort, not only automatic inference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials." pith.science (2026). https://pith.science/paper/PP4FVR5K

@misc{pith2026260804910,
  author       = {Pith},
  title        = {Pith review of: AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PP4FVR5K}},
  note         = {Machine review of arXiv:2608.04910}
}
read the original abstract

Tutorial videos are widely used for learning feature-rich software, yet following screencast tutorials often breaks down in practice. Through a survey and contextual inquiry, we found that learners frequently rewind or get stuck because critical input information, especially mouse actions and keyboard-modified operations, is often implicit or missing in tutorials without input metadata. To address this problem, we present AutoCue, a multimodal LLM-assisted, human-in-the-loop tutorial augmentation pipeline for externalizing implicit inputs as instructional visual cues. AutoCue integrates frame-to-frame visual changes, narration signals, and operation guidance from official software manuals to infer likely mouse and key-modifier actions, then produces aligned cue layers and editable artifacts for human refinement. Grounded in multimedia learning and cognitive load theory, we further develop a visual cue grammar for representing mouse, keyboard, and combined inputs in software-learning tutorials. We instantiate and evaluate AutoCue in Autodesk Maya, focusing automatic inference on selected UI-mediated interactions with observable visual or textual feedback while supporting more ambiguous state changes through editable authoring artifacts. In a between-subjects study with 24 participants, the AutoCue-augmented tutorial reduced task completion time and interaction breakdowns and showed improved learner-reported experience.

Figures

Figures reproduced from arXiv: 2608.04910 by the authors.

Figure 1
Figure 1. Overview of the AutoCue system workflow. AutoCue takes an original screencast tutorial without input metadata [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Three categories of implicit input information iden [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Contextual inquiry setup. Participants watched [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Instructional visual cue vocabulary for external [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Before–after example of AutoCue augmentation. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: ROI extraction from adjacent frames. AutoCue com [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: UI events vs. complex state changes. UI events pro [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Task completion time and breakdown frequency for [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

82 extracted references · 41 canonical work pages

  1. [1]

    Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankan- halli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. InProceedings of the 2018 CHI conference on human factors in computing systems. 1–18. doi:10.1145/3173574.3174156

    arXiv 2018

  2. [2]

    Franck Amadieu, Claudette Mariné, and Carole Laimay. 2011. The attention- guiding effect and cognitive load in the comprehension of animations.Computers in human behavior27, 1 (2011), 36–40. doi:10.1016/j.chb.2010.05.009

    work page doi:10.1016/j.chb.2010.05.009 2011

  3. [3]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. InProceedings of the 2019 chi conference on human factors in computing systems. 1–13. doi:10.1145/3290605.3300233

    arXiv 2019

  4. [4]

    Paul Ayres and John Sweller. 2005. The split-attention principle in multimedia learning.The Cambridge handbook of multimedia learning2 (2005), 135–146

    2005

  5. [6]

    Samuelle Bourgault, Li-Yi Wei, Jennifer Jacobs, and Rubaiat Habib Kazi. 2025. Narrative Motion Blocks: Combining Direct Manipulation and Natural Language Interactions for Animation Creation. InProceedings of the 2025 ACM Designing Interactive Systems Conference. 1366–1386. doi:10.1145/3715336.3735766

    arXiv 2025

  6. [7]

    Jagvir Brar and Hans Van Der Meij. 2017. Complex software training: Harnessing and optimizing video instruction.Computers in human behavior70 (2017), 475–

    2017

  7. [8]

    H David Brecht and Suzanne M Ogilby. 2008. Enabling a comprehensive teaching strategy: Video lectures.Journal of Information Technology Education. Innovations in Practice7 (2008), 71. doi:10.28945/198

    work page doi:10.28945/198 2008

  8. [9]

    John Canny. 1986. A Computational Approach to Edge Detection.IEEE Transactions on Pattern Analysis and Machine Intelligence8, 6 (1986), 679–698. doi:10.1109/TPAMI.1986.4767851

    arXiv 1986

Show all 82 references
  1. [10]

    Valjean M Cashen and Kenneth L Leicht. 1970. Role of the isolation effect in a formal educational setting.Journal of Educational Psychology61, 6p1 (1970), 484. doi:10.1037/h0030286

    1970 doi

  2. [11]

    Paul Chandler and John Sweller. 1992. The split-attention effect as a factor in the design of instruction.British Journal of Educational Psychology62, 2 (1992), 233–246. doi:10.1111/j.2044-8279.1992.tb01017.x

    1992

  3. [12]

    Yuexi Chen, Vlad I Morariu, Anh Truong, and Zhicheng Liu. 2024. TutoAI: a cross- domain framework for AI-assisted mixed-media tutorial creation on physical tasks. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–17. doi:10.1145/3613904.3642443

    2024

  4. [13]

    Peggy Chi, Senpo Hu, Lei Shi, Tanya Kraljic, Justin Secor, Tao Dong, Irfan Essa, and Mike Cleron. 2025. WatchWithMe: LLM-Based Interactive Guided Watching of Review Videos. InProceedings of the 7th ACM Conference on Conversational User Interfaces. 1–15. doi:10.1145/3719160.3736624

    2025

  5. [14]

    Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. InProceedings of the 25th annual ACM symposium on User interface software and technology. 93–102. doi:10.1145/2380116.2380130

    2012

  6. [15]

    Pei-Yu Chi, Bongshin Lee, and Steven M Drucker. 2014. DemoWiz: re-performing software demonstrations for a live presentation. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1581–1590. doi:10.1145/ 2556288.2557254

    2014

  7. [16]

    Antonella De Angeli, Uday Athavankar, Anirudha Joshi, Lynne Coventry, and Graham I Johnson. 2004. Introducing ATMs in India: a contextual inquiry.Inter- acting with Computers16, 1 (2004), 29–44. doi:10.1016/j.intcom.2003.11.003

    2004 doi

  8. [17]

    Björn B De Koning, Huib K Tabbers, Remy MJP Rikers, and Fred Paas. 2007. Attention cueing as a means to enhance learning from an animation.Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition21, 6 (2007), 731–746. doi:1...

    2007 doi

  9. [18]

    Björn B De Koning, Huib K Tabbers, Remy MJP Rikers, and Fred Paas. 2009. Towards a framework for attention cueing in instructional animations: Guidelines for research and design.Educational psychology review21, 2 (2009), 113–140. doi:10.1007/s10648-009-9098-7

    2009 doi

  10. [19]

    I’m}{ Stuck!

    Bryan Dosono, Jordan Hayes, and Yang Wang. 2015.{“I’m}{ Stuck!”}: A Con- textual Inquiry of People with Visual Impairments in Authentication. InEleventh Symposium On Usable Privacy and Security (SOUPS 2015). 151–168

    2015

  11. [20]

    My toxic trait is thinking I’ll remember this

    Ian Drosos, Advait Sarkar, and Andrew D Gordon. 2024. " My toxic trait is thinking I’ll remember this": gaps in the learner experience of video tutorials for feature-rich software.arXiv preprint arXiv:2404.07114(2024). doi:10.48550/arXiv. 2404.07114

    2024 doi arXiv

  12. [21]

    Dutta, A

    A. Dutta, A. Gupta, and A. Zissermann. 2016. VGG Image Annotator (VIA). http://www.robots.ox.ac.uk/vgg/software/via/

    2016

  13. [23]

    Robert L Fowler and Anne S Barker. 1974. Effectiveness of highlighting for retention of text material.Journal of Applied Psychology59, 3 (1974), 358. doi:10. 1037/h0036750

    1974

  14. [24]

    C Ailie Fraser, Julia M Markel, N James Basa, Mira Dontcheva, and Scott Klemmer

  15. [26]

    Matthew Fyfield, Michael Henderson, and Michael Phillips. 2022. Improving in- structional video design: A systematic review.Australasian Journal of Educational Technology38, 3 (2022), 155–183. doi:10.14742/ajet.7296

    2022 doi

  16. [27]

    Nathan Garrett. 2021. Segmentation’s failure to improve software video tutorials. British Journal of Educational Technology52, 1 (2021), 318–336. doi:10.1111/bjet. 13000

    2021 doi

  17. [28]

    Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. InAdvances in psy- chology. Vol. 52. Elsevier, 139–183. doi:10.1016/S0166-4115(08)62386-9

    1988 doi

  18. [29]

    James Hartley and Mark Trueman. 1985. A research strategy for text desigbers: The role of headings.Instructional science14, 2 (1985), 99–155

    1985

  19. [30]

    JaidedAI. 2024. EasyOCR: Ready-to-use OCR with 80+ supported languages. GitHub repository. https://github.com/JaidedAI/EasyOCR Version 1.7.2

    2024

  20. [31]

    Qiao Jin, Yu Liu, Ye Yuan, Bo Han, Feng Qian, and Svetlana Yarosh. 2024. Virtual reality, real pedagogy: A contextual inquiry of instructor practices with VR video. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–21. doi:10.1145/3613904.3642510

    2024

  21. [32]

    Kimia Kiani, George Cui, Andrea Bunt, Joanna McGrenere, and Parmit K Chilana

  22. [33]

    Juho Kim, Philip J Guo, Carrie J Cai, Shang-Wen Li, Krzysztof Z Gajos, and Robert C Miller. 2014. Data-driven interaction techniques for improving naviga- tion of educational videos. InProceedings of the 27th annual ACM symposium on User interface software and technology. 563–...

    2014

  23. [34]

    Juho Kim, Philip J Guo, Daniel T Seaton, Piotr Mitros, Krzysztof Z Gajos, and Robert C Miller. 2014. Understanding in-video dropouts and interaction peaks inonline lecture videos. InProceedings of the first ACM conference on Learning@ scale conference. 31–40. doi:10.1145/25563...

    2014

  24. [35]

    Melina Klepsch, Florian Schmitz, and Tina Seufert. 2017. Development and validation of two instruments measuring intrinsic, extraneous, and germane cognitive load.Frontiers in psychology8 (2017), 1997. doi:10.3389/fpsyg.2017.01997

    2017

  25. [36]

    Sven Krome, Steffen P Walz, and Stefan Greuter. 2016. Contextual inquiry of future commuting in autonomous cars. InProceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 3122–3128. doi:10. 1145/2851581.2892336

    2016

  26. [37]

    Doug Kueker and Joi Moore. 2024. Learning from screencast software tutorials: A comparison of cognitive load in dual and single-monitor learning environments. Journal of Computer Assisted Learning40, 1 (2024), 118–135. doi:10.1111/jcal.12875

    2024 doi

  27. [38]

    Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013. Commu- nity enhanced tutorials: improving tutorials with multiple demonstrations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1779–1788. doi:10.1145/2470654.2466235

    2013

  28. [39]

    Franklin Mingzhe Li, Michael Xieyang Liu, Shaun K Kane, and Patrick Carrington

  29. [40]

    Lars Lischke, Sven Mayer, Andreas Preikschat, Markus Schweizer, Ba Vu, Paweł W Woźniak, and Niels Henze. 2018. Understanding large display environments: Contextual inquiry in a control room. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems....

    2018 doi

  30. [41]

    Steven A Lloyd and Chuck L Robertson. 2012. Screencast tutorials enhance student learning of statistics.Teaching of Psychology39, 1 (2012), 67–71. doi:10. 1177/0098628311430640

    2012

  31. [42]

    Robert F Lorch Jr and Elizabeth Pugzles Lorch. 1996. Effects of organizational signals on free recall of expository text.Journal of educational psychology88, 1 (1996), 38. doi:10.1037/0022-0663.88.1.38

    1996 doi

  32. [43]

    Patricia D Mautone and Richard E Mayer. 2001. Signaling as a cognitive guide in multimedia learning.Journal of educational Psychology93, 2 (2001), 377

    2001

  33. [44]

    Richard E Mayer. 1997. Multimedia learning: Are we asking the right questions? Educational psychologist32, 1 (1997), 1–19. doi:10.1207/s15326985ep3201_1

    1997 doi

  34. [45]

    Richard E Mayer. 2002. Multimedia learning. InPsychology of learning and motivation. Vol. 41. Elsevier, 85–139. doi:10.1016/S0079-7421(02)80005-6

    2002 doi

  35. [46]

    Richard E Mayer. 2005. Cognitive theory of multimedia learning.The Cam- bridge handbook of multimedia learning41, 1 (2005), 31–48. doi:10.1017/ CBO9780511816819.004

    2005

  36. [47]

    Martin Merkt, Anne Ballmann, Julia Felfeli, and Stephan Schwan. 2018. Pauses in educational videos: Testing the transience explanation against the structuring explanation.Computers in Human Behavior89 (2018), 399–410. doi:10.1016/j.chb. 2018.01.013

    2018 doi

  37. [48]

    Lori S Mestre. 2012. Student preference for tutorial design: A usability study. Reference Services Review40, 2 (2012), 258–276. doi:10.1108/00907321211228318

    2012 doi

  38. [49]

    Evelyn Navarrete, Andreas Nehring, Sascha Schanze, Ralph Ewerth, and Anett Hoppe. 2025. A closer look into recent video-based learning research: A com- prehensive review of video characteristics, tools, technologies, and learning effectiveness.International Journal of Artifici...

    2025 doi

  39. [50]

    Cuong Nguyen and Feng Liu. 2015. Making software tutorial video responsive. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1565–1568. doi:10.1145/2702123.2702209

    2015

  40. [51]

    OpenAI. 2026. Introducing GPT-5.4. https://openai.com/index/introducing-gpt- 5-4/. Accessed: 2026-05-10

    2026

  41. [52]

    Nobuyuki Otsu. 1979. A Threshold Selection Method from Gray-Level Histograms. IEEE Transactions on Systems, Man, and Cybernetics9, 1 (1979), 62–66. doi:10. 1109/TSMC.1979.4310076

    1979

  42. [53]

    Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F Cohen. 2011. Pause-and-play: automatically linking screencast video tutorials with applications. InProceedings of the 24th annual ACM symposium on User interface software and te...

    2011

  43. [54]

    Richard J Radke, Srinivas Andra, Omar Al-Kofahi, and Badrinath Roysam. 2005. Image change detection algorithms: a systematic survey.IEEE transactions on image processing14, 3 (2005), 294–307. doi:10.1109/TIP.2004.838698

    2005

  44. [55]

    Vasiliki Ragazou and Ilias Karasavvidis. 2023. Effects of Signaling and Practice Types in Video-Based Software Training.Education Sciences13, 6 (2023), 602. doi:10.3390/educsci13060602

    2023 doi

  45. [56]

    Mary Elizabeth Raven and Alicia Flanders. 1996. Using contextual inquiry to learn about your audiences.ACM SIGDOC Asterisk Journal of Computer Documentation 20, 1 (1996), 1–13. doi:10.1145/227614.227615

    1996

  46. [57]

    Samant Saurabh and Sanjana Gautam. 2019. Modelling and statistical analysis of YouTube’s educational videos: A channel Owner’s perspective.Computers & Education128 (2019), 145–158. doi:10.1016/j.compedu.2018.09.003

    2019 doi

  47. [58]

    Noah L Schroeder and Ada T Cenkci. 2018. Spatial contiguity and spatial split- attention effects in multimedia learning environments: A meta-analysis.Educa- tional Psychology Review30, 3 (2018), 679–701. doi:10.1007/s10648-018-9435-9

    2018 doi

  48. [59]

    Alessandra Semeraro and Laia Turmo Vidal. 2022. Visualizing instructions for physical training: Exploring visual cues to support movement learning from instructional videos. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–16. doi:10.1145/34911...

    2022

  49. [60]

    1983.Image analysis and mathematical morphology

    Jean Serra. 1983.Image analysis and mathematical morphology. Academic Press, Inc

    1983

  50. [61]

    John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science12, 2 (1988), 257–285. doi:10.1207/s15516709cog1202_4

    1988 doi

  51. [62]

    Richard Tang, Xing-Dong Yang, Scott Bateman, Joaquim Jorge, and Anthony Tang. 2015. Physio@ Home: Exploring visual guidance and feedback techniques for physiotherapy exercises. InProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 4123–4132. do...

    2015

  52. [63]

    Petra Ten Hove and Hans van der Meij. 2015. Like it or not. What characterizes YouTube’s more popular instructional videos?Technical communication62, 1 (2015), 48–62

    2015

  53. [64]

    Anh Truong, Peggy Chi, David Salesin, Irfan Essa, and Maneesh Agrawala

  54. [65]

    Hans Van der Meij. 2017. Reviews in instructional video.Computers & education 114 (2017), 164–174. doi:10.1016/j.compedu.2017.07.002

    2017 doi

  55. [66]

    Hans Van der Meij and Constanze Hopfner. 2022. Eleven guidelines for the design of instructional videos for software training.Technical Communication 69, 3 (2022), 5–23. doi:10.55177/tc786532

    2022 doi

  56. [67]

    Hans van der Meij and Jasmijn Maseland. 2021. Practice schedules in a video- based software training arrangement.Social Sciences & Humanities Open3, 1 (2021), 100133. doi:10.1016/j.ssaho.2021.100133

    2021

  57. [68]

    Hans Van Der Meij and Jan Van Der Meij. 2014. A comparison of paper-based and video tutorials for software learning.Computers & education78 (2014), 150–159. doi:10.1016/j.compedu.2014.06.003

    2014 doi

  58. [69]

    Hans van der Meij and Jan van der Meij. 2016. Demonstration-based training (DBT) in the design of a video tutorial for software training.Instructional science 44, 6 (2016), 527–542. doi:10.1007/s11251-016-9394-9

    2016 doi

  59. [70]

    Hans van der Meij and Jan van der Meij. 2016. The effects of reviews in video tutorials.Journal of computer assisted learning32, 4 (2016), 332–344. doi:10.1111/ jcal.12136

    2016

  60. [71]

    Hans Van Der Meij, Jan Van Der Meij, Tessa Voerman, and Evert Duipmans. 2018. Supporting motivation, task performance and retention in video tutorials for software training.Educational technology research and development66, 3 (2018), 597–614. doi:10.1007/s11423-017-9560-z

    2018 doi

  61. [72]

    Jan van der Meij and Hans van der Meij. 2015. A test of the design of a video tutorial for software training.Journal of Computer Assisted Learning31, 2 (2015), 116–132. doi:10.1111/jcal.12082

    2015 doi

  62. [73]

    Tamara Van Gog. 2021. The signaling (or cueing) principle in multimedia learning. InThe Cambridge handbook of multimedia learning. Cambridge University Press, 221–230. doi:10.1017/CBO9781139547369.014

    2021 doi

  63. [74]

    Maarten W Van Someren, Yvonne F Barnard, Jacobijn AC Sandberg, et al. 1994. The think aloud method: a practical approach to modelling cognitive processes. London: AcademicPress11, 6 (1994)

    1994

  64. [75]

    Xue Wang, Lin Lin, Meiqi Han, and J Michael Spector. 2020. Impacts of cues on learning: Using eye-tracking technologies to examine the functions and designs of added cues in short instructional videos.Computers in Human Behavior107 (2020), 106279. doi:10.1016/j.chb.2020.106279

    2020

  65. [76]

    Saelyne Yang, Anh Truong, Juho Kim, and Dingzeyu Li. 2025. VideoMix: Ag- gregating How-To Videos for Task-Oriented Learning. InProceedings of the 30th International Conference on Intelligent User Interfaces. 1564–1580. doi:10.1145/ 3708359.3712144

    2025

  66. [77]

    Saelyne Yang, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2024. AQuA: Automated question-answering in software tutorial videos with visual anchors. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–19. doi:10.1145/3613904.3642752

    2024

  67. [78]

    Saelyne Yang, Jisu Yim, Aitolkyn Baigutanova, Seoyoung Kim, Minsuk Chang, and Juho Kim. 2022. SoftVideo: Improving the Learning Experience of Software Tutorial Videos with Collective Interaction Data. InProceedings of the 27th Inter- national Conference on Intelligent User Int...

    2022 doi

  68. [79]

    Running Zhao, Zhihan Jiang, Xinchen Zhang, Chirui Chang, Handi Chen, Weipeng Deng, Luyao Jin, Xiaojuan Qi, Xun Qian, and Edith CH Ngai. 2025. NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understanding. InProceedings of the 38t...

    2025

  69. [80]

    Yihao Zhu and Qinyi Zhou. 2023. Docent: Digital Operation-Centric Elicitation of Novice-friendly Tutorials. InAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–3. doi:10.1145/3586182. 3625121

    2023 doi

  70. [485]

    doi:10.1016/j.chb.2017.01.014 GI 2026, June 09–12, 2026, Waterloo, ON, Canada Luo et al

    2017 doi

  71. [2019]

    One-Size-Fits-All

    Beyond" One-Size-Fits-All" Understanding the Diversity in How Software Newcomers Discover and Make Use of Help Resources. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14. doi:10.1145/ 3290605.3300570

    2019

  72. [2020]

    In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

    ReMap: Lowering the barrier to help-seeking with multimodal search. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 979–986. doi:10.1145/3379337.3415592

  73. [2021]

    InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems

    Automatic generation of two-level hierarchical tutorials from instruc- tional makeup videos. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–16. doi:10.1145/3411764.3445721

    2021

  74. [2024]

    In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    A Contextual Inquiry of People with Vision Impairments in Cooking. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–14. doi:10.1145/3613904.3642233

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.