{"id":"c89e197d-d4b2-400a-8843-4e3eb5ac312a","arxiv_id":"2608.04910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-assisted pipeline that infers implicit mouse and keyboard input from screencast tutorials and overlays visual cues reduced task completion time and breakdowns in a between-subjects Maya study.","lead":"AutoCue is a pipeline that watches screencast software tutorials, infers mouse clicks and keyboard-modified actions that the video does not show, and overlays visual cue icons to make those actions explicit. In a 24-person study with Autodesk Maya, learners using the augmented tutorial finished faster and got stuck less often than learners using the original video.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's recall and precision counts are internally inconsistent, so the strongest automatic-validity claim is not currently supported.","rationale":"The reader's verdict bundles two claims: the user-study workflow benefit and the automatic-component metric for UI-mediated events. The user-study result in Section 7.3 is statistically supported, though the treatment condition included expert-authored cues for 11 of 24 events, so it tests the full human-in-the-loop workflow rather than the automatic pipeline alone. The automatic-component claim, however, rests entirely on Table 1, and simple arithmetic on the numbers printed in the paper shows an internal inconsistency: the true-positive count implied by recall is 97 while the true-positive count implied by precision is 94. This is a concrete correctness risk, not a matter of external validity or consensus. It can be settled by re-deriving the metrics from the raw annotations, which the paper should provide. If the corrected values remain close to those reported, the automatic-validity claim survives; if not, the paper's scoped evidence for automatic inference weakens substantially. Because the user study may still support the workflow-level contribution, the appropriate verdict remains conditional rather than an outright rejection, but the condition must now include a corrected and internally consistent Table 1. This is why my agreement with the reader is partial: we both flag the automatic component as a weak point, but the specific arithmetic inconsistency is a new concern beyond the reader's single-tutorial generalizability worry.","tokens_in":22694,"tokens_out":6382,"duration_ms":58064,"concrete_test":"Request the per-event annotation table from Supplement 4.x and recompute recall and precision from raw true-positive, false-negative, and false-positive counts. Verify that the true-positive set is identical for both metrics. One concrete check: sum the recall numerators and the precision numerators across videos; if the sums differ, at least one set of counts is erroneous. Then report corrected micro-averaged values and state whether the Section 6.5.2 claim remains above the stated threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.5.2 reports micro-averaged recall 95.10% and precision 96.91% for UI-mediated events. In Table 1, the recall numerators are 28, 15, 22, 15, and 17 (sum 97; true events sum 102), while the precision numerators are 27, 15, 20, 15, and 17 (sum 94; predicted events sum 97). Under standard definitions, recall and precision share the same true-positive set, so a prediction counted as correct for recall must also be counted as correct for precision. The two sets of raw counts therefore cannot both be correct. If the true-positive count is 97, precision would be 97/97 = 100%, not 96.91%; if the true-positive count is 94, recall would be 94/102 = 92.16%, not 95.10%. No explanation for the discrepancy is provided in Section 6.5.3 or elsewhere. Because Table 1 is the only direct evidence for the paper's scoped automatic-inference claim, the reported metrics need re-derivation before that claim can be relied upon.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents AutoCue, a multimodal LLM-assisted, human-in-the-loop pipeline that adds instructional visual cues to screencast tutorials lacking input metadata. The authors report a formative survey (n=87 valid) and contextual inquiry (n=8) identifying missing mouse, keyboard, and keyboard-modified input information as a key cause of rewinds and getting stuck, then design a cue grammar and instantiate the pipeline in Autodesk Maya. Technical validation on 30 clips from 5 Maya tutorials reports micro-averaged recall of 95.10% and precision of 96.91% for UI-mediated events. A between-subjects user study (N=24, 12 per condition) comparing the raw tutorial with an AutoCue-workflow-augmented tutorial finds significantly faster task completion (median 1067 s vs 1404 s, U=20, p=0.003) and fewer breakdowns (median 3 vs 7, U=25.5, p=0.007), with large effect sizes; subjective ratings were directionally favorable but not significant at p<0.05.","tokens_in":22848,"tokens_out":7343,"duration_ms":59980,"significance":"If the reported results hold, the paper makes a useful contribution: it provides empirical evidence that externalizing implicit input actions through a consistent, action-proximal cue grammar can improve follow-along learning in feature-rich software, and it demonstrates a plausible architecture for generating such cues from raw screencasts while constraining LLM output to documentation-derived interaction menus. The user-study analysis is methodologically sound for a small between-subjects design: appropriate nonparametric tests, effect sizes, and shift estimates are reported, and the paper is careful to frame the evaluation as testing the human-in-the-loop workflow rather than a fully autonomous system. The main caveat is that the automatic-validity claim currently rests on Table 1, whose raw counts are internally inconsistent, so that part of the contribution needs correction before the reported numbers can be accepted.","major_comments":[{"comment":"Table 1 is internally inconsistent under the standard definitions stated in its caption. The recall column reports correct predictions over 102 validated events (28+15+22+15+17), while the precision column reports correct predictions over 97 predicted events (27+15+20+15+17); if the same true-positive set underlies both metrics, the two numerators must agree, and the reported micro-averaged recall (95.10%) and precision (96.91%) cannot both hold. Specifically, V1 reports 28/29 recall but 27/28 precision, and V3 reports 22/23 recall but 20/22 precision, which would imply that an event counted as correct for recall is not counted as correct for precision. Because Section 6.5.2's automatic-validity claim is supported solely by these counts, the authors must either correct the raw counts, clarify how true positives were defined separately for recall and precision, or recompute the micro-averages before the claim can be accepted.","section":"Section 6.5.2, Table 1"},{"comment":"The abstract's statement that the AutoCue-augmented tutorial 'showed improved learner-reported experience' overstates the reported results. In Section 7.3.2, none of the three Likert comparisons reached p<0.05 (Q1: p=0.196; Q2: p=0.083; Q3: p=0.102), and the text itself describes only 'directional trends.' The abstract should be revised to say that subjective measures were directionally more favorable, or the subjective comparisons should be omitted from the summary claims.","section":"Abstract and Section 7.3.2"},{"comment":"The user study compared the raw tutorial against an augmented tutorial in which an expert instructor completed 11 of 24 cues by hand, so the study establishes the value of the human-in-the-loop AutoCue workflow, not of fully automatic augmentation. The paper mostly scopes its claims correctly, but the conclusion and abstract should make this boundary explicit to prevent readers from attributing the observed benefits to automatic inference alone; Section 8.5's limitation statement should be reflected in the contribution wording.","section":"Section 7 and Section 8.5"}],"minor_comments":[{"comment":"Several pipeline parameters that likely affect the reported metrics are not given values or a sensitivity analysis, including the MAD and Canny thresholds, the Otsu/morphological processing settings, the 0.5-second frame sampling interval, the narration alignment window, and the 0.80 high-confidence inclusion threshold.","section":"Section 6.2.1 and Section 6.4.2"},{"comment":"The expert ground-truth annotations are described as produced by two expert Maya instructors, but no inter-annotator agreement is reported; since recall and precision are computed against this ground truth, reporting agreement or a reconciliation procedure would strengthen the validation.","section":"Section 6.5.1"},{"comment":"The Cliff's delta for breakdown frequency is reported only as a point estimate (-0.65), while the completion-time effect includes a 95% confidence interval; adding a confidence interval for the breakdown effect would make the two effect-size reports consistent.","section":"Section 7.3.1"},{"comment":"The caption should define how a prediction is counted as 'correct' (for example, exact label match with what timestamp tolerance and ROI overlap), since that definition is needed to adjudicate the recall/precision numerator discrepancy.","section":"Table 1"},{"comment":"The paper reports per-video standard deviations for recall and precision but does not describe how these relate to the micro-averaged values; clarifying whether these are standard deviations across the five V1-V5 clips or across the 30 segments would avoid ambiguity.","section":"Section 6.5.2"},{"comment":"No data or artifact availability statement is included; releasing the interaction menu, the exact LLM prompt, the annotated validation data, and the augmented tutorial video would support reproducibility and help resolve the counting discrepancy in Table 1.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is within scope for GI and the user-study result is potentially valuable, but the Table 1 inconsistency in the technical validation must be resolved before acceptance, and the abstract should be aligned with the nonsignificant subjective outcomes. Please ask the authors to supply raw per-event confusion matrices or the annotated dataset, since the current text does not allow the discrepancy to be adjudicated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper has a credible, theory-grounded user study showing that adding visual input cues to a Maya screencast helps learners, but the technical validation table for the automatic inference component has an internal inconsistency that needs fixing before the system claims can be trusted.\n\nWhat's actually new: AutoCue is a sensible new combination—constrained multimodal-LLM inference over frame differences, OCR, narration, and a documentation-derived interaction menu—paired with a visual cue grammar for mouse and keyboard-modified inputs. The formative work (survey plus contextual inquiry) is well executed and gives a real motivation for the system. The human-in-the-loop output (editable artifacts plus augmented video) is a practical design choice that matches the reality that some interactions are too ambiguous for full automation.\n\nWhat the paper does well: the user study (N=24, between-subjects) uses appropriate nonparametric tests and reports large effects (Cliff's delta -0.72 for completion time, -0.65 for breakdowns). The paper is transparent about many of its limitations, which I appreciate.\n\nThe soft spots, in proportion: the main issue is Table 1. The recall numerators and precision numerators are supposed to share the same true-positive set, but for V1 and V3 they differ (28 vs 27, and 22 vs 20). The micro-averaged recall (95.10%) and precision (96.91%) come from the raw columns, but those columns cannot both be correct. So the strongest automatic-validity claim is not currently supported. Also, the abstract overstates the subjective results: none of the three Likert items reached significance. The user study had an expert hand-complete 11 of 24 cues, so it tests the workflow, not the automatic pipeline alone. Breakdown recording was unblinded, a minor but real threat. No code or data is provided.\n\nThese are fixable. The user-study effect is large and the workflow contribution is plausible; the technical claim needs re-derivation and the abstract needs to be toned down. This paper will interest HCI researchers working on software learning, tutorial augmentation, and LLM-assisted authoring. It deserves a serious referee.\n\nRecommendation: send it to peer review, but ask the authors to correct or explain Table 1 and align the abstract with the non-significant subjective results.","headline":"A promising workflow paper with a solid user-study core, but the automatic-inference table is internally inconsistent and the abstract overstates the subjective results.","tokens_in":23448,"tokens_out":2977,"would_cite":false,"duration_ms":24355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoCue claims that making implicit mouse and keyboard inputs visible as standardized visual cues in screencast tutorials reduces learner rewinds, stuck events, and completion time, supported by a 24-participant between-subjects study in…","keywords":["screencast tutorials","software learning","instructional visual cues","human-in-the-loop AI","tutorial augmentation","multimodal LLM","cognitive load","Autodesk Maya"],"falsifier":"Take several raw screencasts across different applications, run AutoCue's automatic component with no expert completion, and compare learners against the original videos on completion time and breakdown frequency. If the benefit in the Maya study disappears or flips when the 11 expert-completed cues are removed, the claim that the AutoCue workflow itself reduces breakdowns would be falsified; a simpler check compares auto-only cues against expert-completed cues on the same tutorial.","tokens_in":22425,"feed_emoji":"🖱️","tokens_out":8997,"duration_ms":71009,"temperature":0.7,"pith_summary":"Following a screencast tutorial often fails because the instructor's mouse clicks, double-clicks, drags, and keyboard-modified gestures remain invisible or implicit in the video. The paper claims that externalizing these inputs as a consistent set of visual cues—mouse-button icons, click indicators, a fixed keyboard overlay, and integrated key-plus-mouse symbols placed near the cursor—reduces how often learners rewind and get stuck. AutoCue is a multimodal-LLM pipeline that finds candidate interaction moments from frame-to-frame differences, OCR-readable interface text, narration, and official software documentation, then labels each moment from a constrained interaction menu and renders aligned cue layers or editable artifacts. In a between-subjects study with 24 Maya users, the augmented tutorial gave a median completion time of 1067 seconds versus 1404 seconds and a median of 3 breakdowns versus 7, with significant group differences and large effect sizes. On UI-mediated events (menus, dialogs, window transitions), the automatic component reaches a micro-averaged recall of 95.10% and precision of 96.91%.","feed_headline":"Overlaid input cues cut screencast breakdowns by more than half","feed_subtitle":"A 24-person Maya study found that making implicit clicks and key modifiers visible cut task time and stuck moments.","key_machinery":"AutoCue's central object is a three-module augmentation pipeline: difference-driven frame analysis (M1) that selects candidate timestamps, constrained multimodal LLM inference (M2) that maps an evidence bundle to one interaction label from an official documentation-derived menu, and cue rendering and export (M3) that overlays the tutorial or writes editable JSON artifacts. The instructional visual cue grammar is the load-bearing design artifact: icon-based mouse buttons, circular indicators separating single from double clicks, a consistent bottom-screen keyboard overlay, and integrated key-plus-mouse composite cues placed at the cursor to respect spatial contiguity and reduce split attention. What carries the argument is the combination of evidence grounding (frames, OCR, narration) with a constrained label space, so inferences stay renderable and reviewable.","core_discovery":"The central claim is that much of the friction in following software screencasts comes from missing input information, and that this information can be recovered from the screencast itself and visualized with a coherent cue grammar. AutoCue treats input inference as a constrained labeling problem: a multimodal LLM receives local frame pairs, an ROI with OCR tokens, time-aligned narration, and a documentation-derived interaction menu, and must return exactly one interaction label per candidate timestamp. The empirical core is the user-study result that the AutoCue-workflow-augmented tutorial made learners faster and less prone to breakdowns than the original tutorial: completion time differed significantly ($U=20$, $p=0.003$) with Cliff's $\\delta=-0.72$, and breakdown frequency differed significantly ($U=25.5$, $p=0.007$) with Cliff's $\\delta=-0.65$. The technical validation adds that UI-mediated events are recovered with 95.10% micro-averaged recall and 96.91% precision, supporting the claim that the automatic part of the pipeline handles the interaction class with the clearest visual feedback.","pith_inferences":["A testable extension is to vary cue density deliberately: the study's diminishing-returns pattern suggests that augmenting only a handful of high-stakes cues, such as the double-click that opens the Revolve window, may account for most of the benefit, and that exhaustive cueing would add little.","The same evidence-bundle machinery could be pointed at other implicit states, such as selection sets, object modes, or view changes, by extending the interaction menu, effectively turning legacy screencasts into structured, searchable action transcripts.","One experienced participant's complaint about cue size hints at an expertise-reversal effect: an adaptive cue grammar that reduces cue persistence or opacity for advanced users would likely outperform the fixed overlay, which is a directly testable follow-up.","The editable JSON layer separates inference from rendering, so cue layers could be versioned or crowdsourced over old tutorials, converting a library of legacy screencasts into consistently styled learning materials."],"forward_implications":["Creators of raw screencasts without input metadata can produce a cue-augmented tutorial in a semi-automated workflow instead of hand-authoring every click, drag, and modifier.","Learners following an AutoCue-augmented tutorial are expected to rewind less, get stuck less often, and finish tasks faster, at least in feature-rich 3D applications comparable to Maya.","The high recall and precision on UI-mediated events implies that menus, tool dialogs, and window transitions can be augmented automatically with near-manual accuracy, while ambiguous geometry or state changes still need human review.","Because the software-specific knowledge sits in the M2 interaction menu, the pipeline transfers to other GUI applications with observable visual feedback by swapping documentation-derived labels and prompts.","A standardized cue vocabulary applied across tutorials would reduce the cost learners pay for re-learning cue conventions whenever they switch creators or platforms."],"supporting_citations":[{"why":"Supplies the multimedia learning theory that motivates cueing and spatial-contiguity design choices.","marker":"[45]"},{"why":"Provides the cognitive theory of multimedia learning that grounds the cue vocabulary in working-memory limits.","marker":"[46]"},{"why":"Articulates the split-attention principle used to justify placing keyboard-modified mouse cues at the cursor.","marker":"[4]"},{"why":"Provides the attention-cueing framework for dynamic materials that supports the cueing mechanism.","marker":"[18]"},{"why":"Supplies the signaling (cueing) principle that explains how explicit cues lower extraneous processing.","marker":"[73]"},{"why":"Provides prior evidence that added cues in instructional videos shape attention and learning.","marker":"[75]"},{"why":"Names the multimodal LLM used for constrained interaction inference in M2.","marker":"[51]"},{"why":"Surveys image change-detection algorithms that ground M1's difference-driven candidate selection.","marker":"[54]"}],"fun_headline_variants":["Visual cues for hidden mouse and key actions speed up tutorials","AutoCue reveals implicit inputs to cut tutorial friction","Making hidden mouse clicks visible cuts task time and breakdowns","LLM-infused cues make software tutorials faster to follow","AutoCue overlays input cues to slash tutorial rewind time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one Maya cup-modeling tutorial, with an expert manually adding 11 of the 24 cues, is representative enough of real screencast learning that the measured reduction in breakdowns and completion time transfers to other tutorials and to the fully automatic pipeline.","fun_headline_variants_meta":{"raw":{"variants":["Visual cues for hidden mouse and key actions speed up tutorials","AutoCue reveals implicit inputs to cut tutorial friction","Making hidden mouse clicks visible cuts task time and breakdowns","LLM-infused cues make software tutorials faster to follow","AutoCue overlays input cues to slash tutorial rewind time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2380,"prompt_tokens":1005,"completion_tokens":1375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1293}},"tokens_in":621,"tokens_out":1375,"duration_ms":9512,"temperature":1.0,"reasoning_tokens":1293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:37:07.929856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take several raw screencasts across different applications, run AutoCue's automatic component with no expert completion, and compare learners against the original videos on completion time and breakdown frequency. If the benefit in the Maya study disappears or flips when the 11 expert-completed cues are removed, the claim that the AutoCue workflow itself reduces breakdowns would be falsified; a simpler check compares auto-only cues against expert-completed cues on the same tutorial.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cognitive theory of multimedia learning that grounds the cue vocabulary in working-memory limits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the attention-cueing framework for dynamic materials that supports the cueing mechanism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the signaling (cueing) principle that explains how explicit cues lower extraneous processing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Names the multimodal LLM used for constrained interaction inference in M2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Surveys image change-detection algorithms that ground M1's difference-driven candidate selection."}],"review_version":1}