{"id":"646a2d9e-0af7-4f4a-a649-f68e80e140d9","arxiv_id":"2501.09355","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A simple SSIM and object-count-change algorithm can detect proactive intervention moments in AR tasks with high recall but low precision compared to heavy baselines.","lead":"YETI is a lightweight algorithm that uses frame similarity and changes in counted objects to decide when an AI assistant should proactively step in during an AR-guided task. It reports higher recall and F-measure than heavier multimodal baselines on the HoloAssist benchmark, while using far less data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ±5 s tolerance window plus YETI's high firing rate makes chance-level recall likely; without a rate-matched null baseline, the claimed superiority over HoloAssist is not established.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: with a generous 5-second tolerance window and no random or rate-matched baseline, high recall can be produced by a detector that fires often, including one with no real understanding of intervention moments. My quantitative reading of the paper's own numbers strengthens this concern. From Table 4 and supplementary Table 6, Global YETI's precision and recall imply about 2,120 detections on a test set with 1,005 intervention labels over 24,537 frames; a uniform random detector with the same firing rate would achieve roughly 61% recall and F-measure near 0.39 under the stated rule. This means the most impressive-sounding number, 88.31 recall, is heavily inflated by the evaluation protocol, and the comparison to HoloAssist is not controlled for operating point. The symmetric window also conflates reactive trailing detections with proactive ones, which matters for a paper about proactive intervention. I do not claim the method is certainly useless: the feature-efficiency argument is plausible, the algorithm is transparent, and the paper makes falsifiable claims. But the published evaluation does not rule out a chance-level explanation for most of the observed gap. The fix is straightforward: report a random/fixed-rate null baseline, a one-sided temporal test, and preferably a validation split for hyperparameters. Because the reader's CONDITIONAL verdict already reflects the need for such checks, I leave the verdict unchanged rather than escalating to REJECT; the concern is serious but the evidence is not yet sufficient to prove the central claim false outright.","tokens_in":17456,"tokens_out":12837,"duration_ms":121389,"concrete_test":"Reproduce the evaluation on the same HoloAssist intervention split with a null baseline that emits exactly the same number of candidate frames as Global YETI (about 2,120, estimated from TP/P), drawn uniformly at random over the 24,537 test frames, and score it with the paper's ±5 s rule; repeat over at least 100 seeds. If the mean null recall is at least 60% or the null F-measure approaches the HoloAssist R+H+E value of 42.28, the headline claim is not supported. As a secondary check, re-score YETI using only the 5 s before each label to verify that detections precede the ground-truth intervention timestamps rather than trailing them.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 4 sells Global YETI's recall (88.31 vs 37.59) and F-measure (56.17 vs 42.28) as evidence of a better intervention detector, but the evaluation rule declares any YETI frame within ±5 s of a labelled intervention a true positive. The paper's supplementary Table 6 reports 1,005 intervention labels over 24,537 test frames. Global YETI's precision (41.86) and recall (88.31) imply roughly 2,120 detections on the test split. A null detector that fires at 2,120 uniformly random frames would be expected to hit about 61% of the labels under the identical ±5 s rule (1 − (1 − 11/24537)^2120), with an F-measure around 0.39. That value is already close to the best HoloAssist baseline (F = 42.28), so YETI's headline recall is only about 27 points above chance rather than 51 points above the baseline. Because no random, periodic, or rate-matched baseline is reported, the observed advantage is confounded with detection frequency: YETI's low precision shows it is firing far more often than the HoloAssist detectors. The symmetric window also credits detections that occur after the human intervention, so the result does not demonstrate proactive timing. The central claim therefore rests on an evaluation protocol that has not been shown to separate signal from chance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces YETI, a deterministic algorithm for detecting moments when an AI agent should proactively intervene in AR-mediated procedural tasks. YETI computes two lightweight signals from egocentric video: per-frame object counts from a VLM (PaliGemma) and SSIM between consecutive frames; the algorithm triggers an intervention when the object-count change reaches a local extremum, subject to filtering and temporal spacing constraints. The authors evaluate YETI on the HoloAssist benchmark and report substantially higher recall and F-measure than the HoloAssist baselines for proactive intervention detection, with far smaller feature footprints.","tokens_in":17700,"tokens_out":4670,"duration_ms":41402,"significance":"The proposed approach is refreshingly simple, interpretable, and potentially deployable on resource-constrained devices; if the evaluation were sound, the paper would make a useful contribution to proactive agent design. The algorithmic specification is clear and deterministic, and the authors provide extensive ablation tables. However, the current evaluation protocol does not establish the claimed advantage, so the contribution's significance is presently conditional.","major_comments":[{"comment":"The ±5 s symmetric tolerance window, combined with YETI's low precision (41.86) and high recall (88.31), means the detector fires about 2,120 times on the 24,537 test frames. Under the same evaluation rule, a null detector choosing that many frames uniformly at random would be expected to hit roughly 61% of the 1,005 intervention labels (1 − (1 − 11/24537)^2120) and attain an F-measure near 0.39, already close to the best HoloAssist baseline (F = 42.28). Because no random, periodic, or rate-matched baseline is reported, the headline recall of 88.31 vs. the baseline 37.59 is not established as signal; a large part of the apparent improvement could be an artifact of YETI's much higher detection rate. In addition, the symmetric window credits detections that occur after the labeled intervention, so the results do not demonstrate proactive timing.","section":"Section 4.2 and Table 4; Supplementary Table 6"},{"comment":"The four hyperparameters (SSIM threshold τ, conversation interval m, extrema range r, episode length k) are selected on the test set, and the ablation tables report test-set metrics. This is not a valid comparison: the HoloAssist baselines are presumably evaluated with fixed settings, and YETI's numbers reflect test-set optimization. Furthermore, Global YETI uses the first local extrema of each test video to set the per-video threshold, implicitly using test information within each video. The authors should perform model selection on a validation split or via cross-validation and report the held-out test performance.","section":"Section 4.1 and Table 3; Supplementary Tables 10-15"},{"comment":"The procedure for aggregating HoloAssist per-class results into the 'overall' row is described only as 'averaging.' This is ambiguous and likely inappropriate: equal-weight averaging of per-class precision/recall/F-measure ignores the large class imbalance (e.g., 474 Confirm Action vs. 142 Correct Mistake test frames in Table 6). The resulting overall numbers for the baselines are not comparable to YETI's overall numbers, which appear to be computed by pooling all intervention frames. The authors should either report pooled (micro-averaged) metrics for both YETI and HoloAssist, or clearly state the aggregation rule.","section":"Section 4.1, Table 4"},{"comment":"The alignment signal is the raw change in object count between consecutive frames. Because PaliGemma's counts are noisy and the signal is not calibrated to task-specific ground truth, the extrema detected by YETI may correspond to counting noise rather than meaningful action state changes. The paper provides no analysis of PaliGemma counting accuracy on this dataset, nor any comparison against a simple motion/activity baseline. At minimum, the authors should report the distribution of the VLM counting errors and show that the extrema coincide with annotated interventions more often than would be expected from the counting noise alone.","section":"Section 3.3 and Figure 2"}],"minor_comments":[{"comment":"The '6500 times less memory' comparison mixes feature types and units (per-frame embeddings vs. scalar signals); the comparison should state the per-frame dimensionality and total storage per video.","section":"Table 1 and Section 1"},{"comment":"The pseudocode uses undefined variables ('ci' in line 8, 'fcurrent' in line 23) and the control flow for episode and conversation intervals is hard to follow; please rewrite with clear definitions and a worked example.","section":"Algorithm 1"},{"comment":"Section 4.1 states that the evaluation includes 482 videos 'out of the total HoloAssist dataset' but the dataset description in Section 3.1 says the dataset has 482 interaction sequences; clarify the relationship between these numbers.","section":"Section 4.1 vs. Section 3.1"},{"comment":"The supplementary text contains speculative claims such as 'YETI likely employs more streamlined feature representations'; these should be removed or supported by evidence.","section":"Supplementary Section 6"},{"comment":"The authors state that the 5-second window 'is consistent with the method used to evaluate the HoloAssist baseline model [21]'; please provide a precise citation to the HoloAssist protocol and state whether the original baselines used the same window or a different one.","section":"Section 4.2"},{"comment":"The phrase '60,000 times more memory-efficient' in Section 5 is inconsistent with the '6,500 times' value in Table 1 and the Introduction; please reconcile these numbers.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is not yet established because the evaluation does not separate signal from chance. The authors should be asked to add a rate-matched null baseline and to redo the hyperparameter selection on a validation set before reconsideration. I do not see grounds for rejection, as the method is simple and the issues are addressable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper proposes a simple, interpretable detector for proactive intervention in AR egocentric video — SSIM frame filtering plus changes in VLM-counted objects — and claims it beats HoloAssist baselines on recall and F-measure while using far smaller features. The idea has merit, but the evaluation does not support the headline claim. Under the ±5 s tolerance window, a null detector firing at YETI's rate would already hit about 61% of the 1,005 labels, for an F-measure near 0.39 — close to the best HoloAssist baseline (42.28). Without any random, periodic, or rate-matched baseline, the 88.31 recall is not established as signal.\n\nWhat is genuinely new: combining SSIM with a VLM object-count delta as a trigger for proactive intervention is, as far as I know, not in prior literature. The alignment signal is computed from frames alone, independent of the intervention labels, so the circularity concern is minimal. The feature-efficiency claim (SSIM plus counts are orders of magnitude smaller than pose, IMU, or depth streams) is credible and practically relevant for on-device AR.\n\nThe soft spots are real and load-bearing. First, the ±5 s window is too generous given the detector fires roughly 2,120 times over the 24,537 test frames. A random detector would get F≈0.39, so the comparison is confounded with firing rate. Second, the HoloAssist baselines are aggregated by averaging per-class metrics, which is not the correct way to compute overall precision/recall/F1. Third, the four hyperparameters in Table 3 are selected on the test set, and the ablations in the supplement are all test-set results. Also, the symmetric window gives credit to detections after the human intervention, so the \"proactive\" timing claim is not demonstrated. No code or data are released, so the numbers cannot be independently reproduced. The text also inconsistently claims 6,500× and 60,000× feature-size reductions; minor.\n\nWho is this for? People working on proactive assistance, egocentric video, or temporal detection evaluation. The paper is a useful cautionary example about tolerance-window evaluations and a plausible starting point for a lightweight intervention detector — but only if the evaluation is redone.\n\nMy recommendation: this deserves a serious referee, not a desk reject, because the idea is clear, the flaws are identifiable and fixable, and a proper null baseline could rescue the central claim. But as it stands, the quantitative comparison to HoloAssist should be treated as unverified.","headline":"A plausible lightweight heuristic for proactive intervention detection, but the headline results rest on an evaluation protocol that does not separate signal from chance.","tokens_in":18285,"tokens_out":3901,"would_cite":false,"duration_ms":36450,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"YETI claims an AR agent can decide when to proactively help using only two lightweight video signals, reaching 88.31 recall on HoloAssist interventions.","keywords":["proactive AI","augmented reality","intervention detection","egocentric video","structural similarity","object counting","vision-language model","HoloAssist"],"falsifier":"Re-run the same evaluation with a baseline that fires intervention frames at fixed intervals or at random times; if that baseline's recall under the five-second window approaches YETI's 88 percent, then the recall figure is an artifact of the tolerance window rather than a sign that the signals align with true intervention moments.","tokens_in":17240,"feed_emoji":"🤖","tokens_out":7762,"duration_ms":65034,"temperature":0.7,"pith_summary":"The paper argues that an augmented-reality assistant does not need heavy multimodal sensor streams to know when to proactively step in and help a user. It proposes YETI, an algorithm that runs on two tiny per-frame signals: structural similarity (SSIM) between consecutive frames and a change-in-object-count signal produced by a lightweight vision-language model. On the HoloAssist benchmark, YETI detects proactive intervention moments with substantially higher recall than the HoloAssist baselines while using features orders of magnitude smaller. The practical stake is that proactive help in AR—catching mistakes, confirming actions, giving follow-up instructions—could run in real time on resource-constrained head-worn devices.","feed_headline":"Proactive AI help spotted at 88% recall from two tiny signals","feed_subtitle":"Just frame similarity and object-count changes tell an AR assistant when to speak up.","key_machinery":"The load-bearing objects are two per-frame signals computed at 1 frame per second. The first is SSIM, a standard image-similarity measure between consecutive frames, used as a filter: frames whose SSIM exceeds a threshold $\\tau$ are considered too static to warrant intervention. The second is the alignment signal, the per-second change in the number of objects counted by a lightweight vision-language model prompted to report object count; a large upward or downward change in this count is taken as evidence that the user is doing something worth commenting on. The YETI algorithm then selects intervention frames as local extrema of the alignment signal, subject to a conversation interval $m$ (minimum gap between interventions) and an episode interval $k$ (at most one intervention per episode). The two variants differ in whether the extrema threshold is fixed from the first episode (Global) or updated from recent history (Local).","core_discovery":"On its own terms, the paper's central claim is that the moments when an expert should proactively intervene in a guided procedural task are detectable from two low-cost signals: the structural similarity of consecutive egocentric frames and the change in the number of objects visible in the scene. Combining these signals, YETI flags an intervention frame when the object-count change hits a local extremum after redundant frames are filtered by an SSIM threshold. In the paper's evaluation against the HoloAssist benchmark, the Global variant of YETI achieves 88.31 recall and 56.17 F-measure on proactive interventions, compared with 37.59 recall and 42.28 F-measure for the strongest HoloAssist baseline (RGB+hand+eye), and its two features together take roughly 41 MB versus hundreds of MB for the baseline feature sets. The paper presents this as evidence that proactive intervention detection can be both more sensitive and far cheaper, making real-time AR assistance practical.","pith_inferences":["If the matching window were tightened from 5 seconds to 1-2 seconds, YETI's recall would likely drop, so whether the 88 percent figure survives stricter temporal alignment is a testable question the paper leaves open.","The same two signals could transfer to other egocentric procedural benchmarks for mistake detection or step anticipation, because neither signal depends on HoloAssist-specific annotations.","Precision could be raised by gating YETI's triggers on a second condition, such as the presence of a hand or a gaze shift, without losing the on-the-fly, low-memory advantage.","Because YETI needs no depth or IMU data, it is a candidate for deployment on camera-only AR glasses, a scenario the paper frames but does not test on actual hardware."],"forward_implications":["An AR assistant using YETI-style signals could run proactive intervention detection on-device in real time, because the two features are about 41 MB in total rather than hundreds of MB.","High recall across all three intervention types (confirm action, correct mistake, follow up) means the assistant would rarely miss a moment to speak up, though the lower precision implies it would also interrupt users more often.","The Global YETI variant, which fixes the extrema threshold from the first episode, gives the best recall and F-measure balance in the paper's experiments.","The approach is task-agnostic across the 482 HoloAssist sequences: it does not require hand pose, gaze, depth, or IMU sensors, so it applies wherever egocentric video is available."],"supporting_citations":[{"why":"HoloAssist supplies the egocentric AR task videos, the proactive intervention labels, and the baseline results that YETI is compared against.","marker":"[21]"},{"why":"Supplies the lightweight vision-language model used to produce the per-frame object count for the alignment signal.","marker":"[3]"},{"why":"Defines the Structural Similarity Index Measure that YETI uses as the frame-filtering signal.","marker":"[22]"},{"why":"TallyQA is cited as the counting dataset the vision-language model was trained on, supporting the reliability of the object-count estimates.","marker":"[2]"}],"fun_headline_variants":["Two signals, 88% recall: AR assistant knows when to speak up","Tiny AI agent predicts when to help in AR tasks","AI agent catches mistakes early with just two cues","Proactive AR aid: 88% recall from two cheap signals","YETI agent: proactive intervention at 88% recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand only if the HoloAssist intervention labels are the correct ground truth for when an agent should intervene, and if counting any YETI detection within a five-second window of such a label as a true positive is a fair way to score performance.","fun_headline_variants_meta":{"raw":{"variants":["Two signals, 88% recall: AR assistant knows when to speak up","Tiny AI agent predicts when to help in AR tasks","AI agent catches mistakes early with just two cues","Proactive AR aid: 88% recall from two cheap signals","YETI agent: proactive intervention at 88% recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2839,"prompt_tokens":1076,"completion_tokens":1763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":1677}},"tokens_in":692,"tokens_out":1763,"duration_ms":11699,"temperature":1.0,"reasoning_tokens":1677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:05:42.421785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same evaluation with a baseline that fires intervention frames at fixed intervals or at random times; if that baseline's recall under the five-second window approaches YETI's 88 percent, then the recall figure is an artifact of the tolerance window rather than a sign that the signals align with true intervention moments.","supporting_citations":[{"cited_title":"Paligemma: A versatile 3b vlm for trans- fer, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the lightweight vision-language model used to produce the per-frame object count for the alignment signal."},{"cited_title":"Tal- lyqa: Answering complex counting questions","cited_arxiv_id":null,"evidence_quote":"TallyQA is cited as the counting dataset the vision-language model was trained on, supporting the reliability of the object-count estimates."}],"review_version":1}