{"id":"0855a700-10bf-405d-aacf-206c06caa827","arxiv_id":"2411.08768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Directly feeding sampled desktop-recording frames to a vision-language model extracts click/select/scroll/drag/type action sequences more reliably than explicitly computing frame differences first.","lead":"This paper tests two ways to use vision-language models to turn screen recordings into lists of user actions like clicks and typing. The simpler method, feeding frames directly to a VLM, reaches roughly 70 to 80 percent accuracy on operation type and beats a method that first computes frame differences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic matching metric is contradicted by the paper's own replay data: one case has perfect semantic precision/recall yet fails replay, so the reported 70–80% accuracy rests on an unvalidated proxy.","rationale":"Both the reader and I identify the semantic metric as the weakest assumption. I go a step further by pointing to internal evidence: Table VII's 'click/text icon/tab' case has semantic P=R=1 but replay failure, which the paper does not mention in its summary. This is a concrete falsification of the claim that the semantic metric is consistent with replay success. Because the abstract claims 70–80% accuracy and replayability, and because the ACTREAL full-metric numbers are substantially lower (0.53 recall for GPT-4o DF), the true capability is likely overstated. The paper remains a plausible first exploration of VLM-based desktop action extraction, with new datasets and detailed prompts in the appendix; the ablations are informative and the negative result about explicit UI-change extraction is interesting. However, the central evaluation proxy is not trustworthy until the replay validation is extended and the threshold sensitivity is reported. This warrants the CONDITIONAL verdict the reader recommended; my concern does not change that verdict but provides a sharper justification for it.","tokens_in":26449,"tokens_out":4347,"duration_ms":124492,"concrete_test":"Re-run the replay experiment beyond the nine selected ACTONE cases: use the authors' modified UFO tool on all 40 ACTONE videos (or a random sample of at least 20), comparing per-case semantic P/R against replay success. Record every case with semantic P=1/R=1 that fails replay, and report the false-positive rate. Also run a threshold sensitivity analysis on the 0.7 BERT cosine cutoff (0.5, 0.6, 0.7, 0.8) and on the greedy matching order; if metric values shift by more than 5 points for any GPT-4o row in Tables II/III, the reported accuracy cannot be taken at face value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim (Section V-B: 'accuracy of 70%–80% in identifying operation types ... re-playable through RPA-like processes') depends entirely on the semantic comparison metric defined in Section IV-B. That metric treats an action as correct when the operation type matches exactly and BERT cosine similarity of detail/context exceeds the manually tuned threshold of 0.7, matched greedily in chronological order. The paper attempts to validate this metric with replay tests on nine ACTONE cases (Table VII). The paper states that 'two of the three unsuccessful cases exhibited lower Precision and Recall values, whereas the six successful cases all achieved one for both Precision and Recall.' But Table VII contains a direct counterexample: the case 'click/text icon/tab' has semantic Precision = 1 and Recall = 1, yet the replay result is 'no'. Perfect semantic scores are therefore not sufficient for actual executable correctness; the metric can certify a sequence as correct even when an RPA tool cannot replay it. This is not an external criticism: it is the paper's own data contradicting its claim that 'Precision and Recall metrics derived from semantic matching are consistent with actual success rates of RPA replay.' Since the 70–80% numbers are computed with this metric, and the only external check is nine simple ACTONE videos (none from ACTREAL), the headline accuracy may be materially inflated. This is load-bearing for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two vision-language-model (VLM) based methods for extracting user action sequences from desktop screen recordings. The Direct Frame-Based approach (DF) feeds sampled frames directly to a VLM and uses sliding-window processing with modules for action proposal, correction, and merging. The Differential Frame-Based approach (DiffF) first computes explicit frame differences via computer-vision techniques, then asks a VLM to interpret textual descriptions of localized UI changes. The authors evaluate both methods on two datasets: ACTONE, a self-curated set of 40 short videos each containing one or two simple actions of a single type, and ACTREAL, a set of 41 videos adapted from GUI-World with longer, multi-action sequences. Evaluation uses precision and recall computed both at the operation-type level and over the full (operation, detail, context) tuple, with semantic matching based on BERT cosine similarity and a manually tuned threshold of 0.7. The paper reports that GPT-4o with DF achieves operation-level recall of 0.83 and precision of 0.81 on ACTONE, and 0.82 recall and 0.70 precision on ACTREAL, while the full-tuple metrics are substantially lower. A replay-based validation on nine ACTONE cases using a modified version of the authors' own UFO tool succeeded in six cases. The central claim is that the DF approach achieves 70%–80% accuracy in identifying operation types and that the extracted sequences are re-playable through RPA-like processes.","tokens_in":26754,"tokens_out":4911,"duration_ms":43828,"significance":"If the claims are validated, this work would be an early demonstration of VLM-based extraction of user actions from desktop video, with potential downstream applications in Robotic Process Automation, tutorial generation, and user modeling. The two datasets, the modular method design, and the detailed prompt templates are potentially useful resources for the community. However, the current evaluation is not yet at the standard needed to support the headline claims: the datasets are small (40 and 41 videos), no variance estimates or significance tests are provided, the primary metric is a semantic proxy whose validity is partially contradicted by the paper's own replay data, and the abstract overstates the level of accuracy by omitting the 'operation type' qualifier. The strengths are the clear task formulation and the explicit, reproducible experimental setup, but these need to be matched by more rigorous evaluation and more carefully hedged claims.","major_comments":[{"comment":"The paper states that 'Precision and Recall metrics derived from semantic matching are consistent with actual success rates of RPA replay,' but Table VII contains a direct counterexample: the case 'click/text icon/tab' has semantic Precision = 1 and Recall = 1 yet the replay result is 'no'. This means that a perfect semantic score does not guarantee executable correctness, so the metric can certify a sequence as correct even when an RPA tool cannot replay it. Since the 70%–80% accuracy figures in Section V-B and the abstract are computed with this metric, the replayability claim is not adequately supported. Please either reconcile this discrepancy, provide a refined metric, or temper the claim that extracted sequences are re-playable through RPA-like processes.","section":"Section IV-B, Table VII"},{"comment":"The abstract says the DF approach achieves 'an accuracy of 70% to 80% in identifying user actions,' but the reported numbers in Table II are for operation-type identification only: GPT-4o with DF has operation recall 0.83 and precision 0.81, while the full operation-detail-context ('All') metrics are 0.71 recall and 0.68 precision. On ACTREAL (Table III), the 'All' metrics are even lower, with recall 0.53. The phrase 'identifying user actions' without the 'operation type' qualifier is therefore misleading; the claims in Sections I and V-B should be explicitly scoped to operation-type identification, and the full-tuple accuracy should be reported wherever the headline accuracy is cited.","section":"Section V-B and Abstract"},{"comment":"The evaluation is based on only 40 and 41 videos, and no standard errors, confidence intervals, or significance tests are reported. The conclusion that DF is more reliable than DiffF in real-world scenarios relies on differences such as GPT-4o 'All' recall of 0.53 versus 0.27 on ACTREAL, but without variance estimates these differences could be due to chance. Please provide per-video results or bootstrap confidence intervals, at least for the primary DF-versus-DiffF comparisons. Additionally, the replay validation covers only nine ACTONE cases and none from ACTREAL, so the claim that extracted sequences are generally re-playable through RPA is only weakly supported.","section":"Sections IV-A/IV-B and Tables II/III"},{"comment":"The semantic matching procedure relies on a manually tuned BERT cosine-similarity threshold of 0.7, but the paper does not provide a sensitivity analysis or justification for this value. Since every reported precision and recall number depends on this threshold, the choice is load-bearing for the central claim. Please include a threshold sweep showing how the metrics vary with the threshold, or provide an alternative justification for why 0.7 is the correct operating point.","section":"Section IV-B"}],"minor_comments":[{"comment":"There is a typo: 're-playable though Robotic Process Automation' should read 're-playable through Robotic Process Automation.'","section":"Abstract"},{"comment":"The footnote says the prompts 'are not included in current draft' and will be open-sourced, but the full prompts are actually provided in the Appendix (Tables VIII–XXI). Please remove or update this note.","section":"Section III-A, footnote"},{"comment":"The replay validation paragraph states that 'two of the three unsuccessful cases exhibited lower Precision and Recall values,' but does not mention that one unsuccessful case ('click/text icon/tab') achieved perfect scores on both metrics. This omission makes the validation seem more favorable than it is; please be explicit about the exception.","section":"Section IV-B"},{"comment":"The column headers are confusing: the table lists 'Semantic Matching' and 'Successful Replay?' but the columns are actually 'Precision (All)', 'Recall (All)', and a yes/no replay result. Please restructure the table or the headers for clarity.","section":"Table VII"},{"comment":"The related work section could better differentiate the paper from GUI agent works such as UFO and OmniParser, which also process screen content with VLMs; the claim of being the 'first' attempt should be verified and hedged accordingly.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"This is a preliminary arXiv preprint that introduces a useful task and two datasets, but the evaluation is not yet at the level needed to support the headline claims. The replay counterexample in Table VII is a load-bearing weakness that the authors must address; they should also temper the abstract's unqualified 'accuracy' claim. With careful revision and additional experiments or reanalysis, the paper could be appropriate for a vision or multimodal venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate first shot at a useful task—turning desktop recordings into structured action sequences via VLMs. The two benchmarks (ACTONE, ACTREAL) and the negative result that explicit UI-difference extraction hurts performance are worth knowing about. But the evaluation has a load-bearing inconsistency you should see before trusting the numbers.\n\nThe good parts first. Framing action-sequence extraction from desktop video as a VLM task is new relative to the GUI-agent and video-understanding literature. The pipeline (action proposer, corrector, merger) is sensible engineering, and the ablations show the corrector matters and that adding explicit UI-change bounding boxes doesn't help. The operation-type results—GPT-4o DF at 0.83/0.81 recall/precision on ACTONE and 0.82/0.70 on ACTREAL—are plausible and based on exact matching of five discrete types.\n\nNow the soft spots, in proportion. The stress-test note is right: Table VII contains a direct counterexample to the paper's claim that semantic precision/recall are consistent with replay success. The case 'click/text icon/tab' has semantic precision = 1 and recall = 1, yet replay fails. That is the paper's own data, and it means the 70–80% accuracy figure—which uses this semantic metric for the 'All' components—can certify a sequence as correct when an RPA tool cannot replay it. The operation-type numbers are less affected because they use exact matching, but the abstract's 'accuracy in identifying user actions' overstates by dropping the 'operation type' qualifier. Also: only 40 and 41 videos, no error bars or significance tests, replay on only nine simple ACTONE cases, and code/data promised but not released.\n\nNone of this makes the paper worthless. It makes the headline accuracy claims conditional. The fix is straightforward: resolve the Table VII contradiction, report operation-type and full-tuple metrics separately with confidence intervals, and release the benchmarks. Who should read it: anyone working on GUI agents, RPA from demo videos, or VLM video understanding. It deserves a serious referee; I'd send it to review with major revision, not desk-reject.","headline":"First VLM task formulation for desktop action extraction with two benchmarks, but the paper's own replay data contradicts its semantic metric, so the headline accuracy claims need caution.","tokens_in":27280,"tokens_out":3179,"would_cite":true,"duration_ms":26712,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper demonstrates that vision-language models can extract user action sequences from desktop screen recordings with 70–80% accuracy on operation type, and that feeding frames directly outperforms explicitly computing UI changes.","keywords":["desktop screen recordings","user action extraction","vision-language models","operation type classification","Robotic Process Automation","UI change detection","action sequence benchmark","semantic matching"],"falsifier":"Take a sample of predicted action sequences from ACTONE and ACTREAL, have independent human judges mark each predicted action as correct/incorrect against the ground truth, and compare those labels with the paper's semantic-matching verdicts; if the agreement is low, the 70–80% figure is an artifact of the metric. A second, direct check is to replay predicted sequences on a larger, unbiased set of cases and see whether replay success tracks the semantic precision/recall.","tokens_in":26246,"feed_emoji":"🖥️","tokens_out":10894,"duration_ms":83980,"temperature":0.7,"pith_summary":"The paper sets out to show that a vision-language model can turn a desktop screen recording into a structured sequence of user actions — click, select, scroll, drag, type — with operation-type accuracy around 70%–80% and with sequences that are executable by a robotic-process-automation tool. It compares two pipelines: one (DF) feeds sampled frames directly to the model, the other (DiffF) first computes explicit pixel-level UI changes and feeds those to the model. Across a curated dataset and a realistic benchmark, the direct pipeline performs best, and explicit UI-change extraction tends to degrade accuracy instead of helping it. If correct, this makes video a practical source of behavioral data for automation, tutorials, and personalization, and it argues that current models are better left to infer actions from raw frames.","feed_headline":"Desktop video yields action sequences with 70–80% accuracy","feed_subtitle":"Directly feeding frames to a VLM beats explicit UI-change detection, and the actions replay through RPA.","key_machinery":"The machinery is the action tuple $(O,D,C)$ — operation type (one of five), the detail (UI element), and the context (application) — combined with a three-module VLM pipeline: an Action Proposer that outputs candidate actions with reasoning, an Action Corrector that deletes redundant and invalid actions and fills missing information, and an Action Merger that stitches overlapping sliding windows. In DiffF, a Frame Difference Localizer computes pixel-difference masks using computer-vision filters and thresholds, and a Frame Difference Descriptor narrates those changes before the same proposer/corrector modules run. The evaluation device is a semantic matching procedure that accepts a prediction when the operation type matches exactly and the BERT-cosine similarity of the detail and context descriptions lies above a manually tuned threshold of 0.7, followed by greedy chronological alignment.","core_discovery":"The central discovery is that the straightforward DF pipeline works: given a desktop recording, uniformly sampled frames, and a large vision-language model prompted to propose, correct, and merge actions, the model identifies the operation type with recall and precision around 0.8 (GPT-4o: 0.83/0.81 on ACTONE, 0.82/0.70 on ACTREAL). When the full action tuple — operation, UI element, and application context — is evaluated, performance drops to 0.71/0.68 and 0.53/0.45 respectively, but the paper still reports the operation-level numbers as the headline accuracy because RPA replay succeeded in 6 of 9 tested cases. A second discovery is that DiffF, which adds an explicit computer-vision stage to localize and describe UI changes before action inference, does not improve over DF and often hurts precision, which the paper attributes to the model over-focusing on localized boxes and losing global context.","pith_inferences":["The 70–80% figure applies to operation-type recognition only; the full-tuple numbers on ACTREAL (0.53 recall) show that recovering the exact UI element and context is still the hard part of the task.","The 0.7 BERT-cosine threshold is tuned by hand on these datasets; a principled alignment method, or a learned threshold, could change the reported accuracies even if the relative ranking of DF and DiffF holds.","DiffF's poor result may be an artifact of giving the VLM precomputed pixel-diff boxes; a model that computes temporal differences internally, rather than relying on an external CV stage, might capture the same signal without the context loss.","Because replay validation covers only nine click/type cases from the curated dataset, a direct next experiment is to replay a larger, more diverse sample (including drag, scroll, select) to see whether the semantic metric remains predictive."],"forward_implications":["Operation-type predictions are accurate enough to drive RPA replay: 6 of 9 tested cases succeeded, and the successful cases all had perfect semantic precision/recall.","The DF pipeline is the better default: the two ablation studies show that adding frame differences to DF's input or adding all frames to DiffF's proposer yields only marginal or negative gains.","Smaller VLMs (4o-mini, Flash) lose 0.2–0.5 in precision, showing that model capacity is a major factor in this task.","The error breakdown points to visual hallucination/blindness as the dominant failure mode for DF and reasoning problems for DiffF, which tells future work where to invest."],"supporting_citations":[{"why":"Supplies the base videos and categories for the real-world benchmark ACTREAL.","marker":"[3]"},{"why":"Used to record the self-curated ACTONE dataset's screen videos.","marker":"[19]"},{"why":"Provides the GPT-4o/4o-mini models that are the main evaluated large VLMs.","marker":"[20]"},{"why":"Provides the Gemini1.5-Pro/Flash models used as the comparison VLM series.","marker":"[8]"},{"why":"Provides the BERT embeddings used in the semantic matching evaluation metric.","marker":"[23]"},{"why":"The VLM-based UI automation tool used in the RPA replay validation.","marker":"[33]"},{"why":"Provides the computer-vision primitives for the Frame Difference Localizer in DiffF.","marker":"[2]"}],"fun_headline_variants":["Simple VLM beats UI-change detection for desktop action extraction","VLM extracts user actions from desktop recordings at 70-80% accuracy","Direct frame VLM outperforms explicit UI-change for action extraction","First VLM method extracts replayable action sequences from desktop video","UI-change cues reduce VLM accuracy for desktop action extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy numbers assume that a BERT-cosine similarity score above a hand-tuned 0.7 threshold, combined with the paper's greedy chronological matching, correctly identifies when a predicted action matches the ground-truth action; if that match rule misclassifies actions, the 70–80% accuracy collapses.","fun_headline_variants_meta":{"raw":{"variants":["Simple VLM beats UI-change detection for desktop action extraction","VLM extracts user actions from desktop recordings at 70-80% accuracy","Direct frame VLM outperforms explicit UI-change for action extraction","First VLM method extracts replayable action sequences from desktop video","UI-change cues reduce VLM accuracy for desktop action extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2790,"prompt_tokens":942,"completion_tokens":1848,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1761}},"tokens_in":558,"tokens_out":1848,"duration_ms":12082,"temperature":1.0,"reasoning_tokens":1761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:20:01.435486+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of predicted action sequences from ACTONE and ACTREAL, have independent human judges mark each predicted action as correct/incorrect against the ground truth, and compare those labels with the paper's semantic-matching verdicts; if the agreement is low, the 70–80% figure is an artifact of the metric. A second, direct check is to replay predicted sequences on a larger, unbiased set of cases and see whether replay success tracks the semantic precision/recall.","supporting_citations":[{"cited_title":"bert-embedding","cited_arxiv_id":null,"evidence_quote":"Provides the BERT embeddings used in the semantic matching evaluation metric."},{"cited_title":"Ufo: A ui-focused agent for windows os interaction, 2024","cited_arxiv_id":null,"evidence_quote":"The VLM-based UI automation tool used in the RPA replay validation."},{"cited_title":"Gui-world: A dataset for gui-oriented multimodal llm-based agents, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the base videos and categories for the real-world benchmark ACTREAL."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to record the self-curated ACTONE dataset's screen videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GPT-4o/4o-mini models that are the main evaluated large VLMs."},{"cited_title":"Gemini models","cited_arxiv_id":null,"evidence_quote":"Provides the Gemini1.5-Pro/Flash models used as the comparison VLM series."}],"review_version":1}