{"id":"6419d406-8daa-4737-9e0a-1f4a2c027654","arxiv_id":"2507.23226","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper reports that VLM-based systems ViDDAR and VIM-Sense detect obstructing or misleading AR content with claimed accuracies of 92.15% and 88.94% on custom datasets, with all experimental details deferred to prior papers.","lead":"This paper summarizes two augmented-reality safety systems that use vision-language models to detect virtual overlays which block or distort real-world information. It is a short doctoral-consortium overview, reporting accuracies of 92.15% and 88.94% on the authors' own datasets while deferring methods and evidence to earlier papers.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels for VIM-Sense rest on subjective user ratings with no inter-rater reliability reported, so the 88.94% accuracy claim may not measure objective semantic manipulation.","rationale":"The reader's weakest_assumption is that the custom datasets' ground truth labels define task-detrimental content and that accuracy claims rest on this definition. I agree with this identification. The reader's verdict of UNVERDICTED is correct because the manuscript itself is a short position/doctoral-consortium summary: it references system papers without including their experimental protocols, and the numbers cannot be verified from the text. My critique sharpens the concern: the user-study annotation process is not merely under-documented; the reported 'average agreement level score of 4.55/5' is not a proper inter-rater reliability measure, and no such measure is reported for the VIM-Sense ratings either. A useful sanity check is whether the risk is primarily 'not verifiable in this manuscript' versus 'likely wrong.' The internal logic of the systems is plausible and consistent with the cited prior work; the obstruction-detection pipeline (VLM + detection/segmentation masks + threshold) is a reasonable design, so I do not claim the numbers are false. The central claim would be true if the annotation labels are objective and reproducible, but this condition is the weakest link. No internal inconsistency is evident, and the future-work sections are clearly framed as plans. Given the manuscript is explicitly a Doctoral Consortium summary, an unchanged UNVERDICTED verdict is the honest outcome; the concern I raise is the concrete reason why verification of the cited papers is necessary before ACCEPT.","tokens_in":3645,"tokens_out":1496,"duration_ms":13961,"concrete_test":"Re-analyze the AR-VIM and ViDDAR datasets to compute inter-rater reliability (Fleiss' kappa or Krippendorff's alpha) and per-rater agreement; if kappa is below 0.6, the label noise alone could account for the reported accuracy gap. Independently re-derive or reproduce the 88.94% and 92.15% accuracy numbers from the code, data, and protocols in references [5], [6], and [7].","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety claim is that VIM-Sense detects task-detrimental content with 88.94% accuracy on AR-VIM. In Section 2, AR-VIM is described as 452 raw-AR video pairs 'annotated through an IRB-approved user study involving 18 participants, who rated whether the AR version misrepresents the real-world information.' No inter-rater agreement statistic, rating scale, or label-generation procedure is reported, and the downstream accuracy is measured against these ratings. If the ratings are subjective or inconsistent, the accuracy number reflects agreement with a noisy human heuristic rather than objective manipulation detection. The same concern applies to ViDDAR's 92.15% accuracy: its 306 raw-AR pairs are 'manually annotated' with obstruction labels, with only a 4.55/5 'average agreement level score' reported, which is not a standard inter-rater reliability measure (e.g., Cohen's kappa or Fleiss' kappa). Because the manuscript provides no dataset statistics, no error bars, and no external benchmark, the reader's UNVERDICTED verdict is appropriate; the load-bearing condition that ground truth is objective and reproducible is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a framework and two detection systems for task-detrimental augmented reality content. ViDDAR detects obstruction attacks by combining VLM-based key-object identification with object detection and segmentation, and VIM-Sense detects semantic information-manipulation attacks through OCR and multimodal VLM reasoning. The paper reports high detection accuracies and latencies, and proposes three future research directions: perceptually aligned quality assessment, multimodal attack detection, and lightweight VLM deployment. The manuscript is written as a short consortium-style position paper, with the core experimental evidence largely attributed to the authors' earlier or submitted work rather than to experiments described in the text.","tokens_in":3806,"tokens_out":3116,"duration_ms":36727,"significance":"If the reported performance is reproducible, the work addresses a genuine gap: existing AR safety methods rely on geometric modeling or low-level image metrics and do not capture semantic manipulation. The modular AR-Edge-Cloud design and the deployment on the Meta Quest 3 are concrete strengths, and the problem framing around task-detrimental content is useful for future work. However, the manuscript as submitted is not self-contained: the central accuracy figures are asserted without an experimental protocol, and the ground truth rests on subjective annotations with no inter-rater reliability evidence. The contribution is promising but not yet established by the evidence contained in this paper.","major_comments":[{"comment":"The claim that ViDDAR achieved up to 92.15% detection accuracy with 533 ms latency is not supported by an experimental protocol in this paper: there is no description of the dataset composition beyond pair counts, no train/test split, no parameter settings for the obstruction-ratio threshold, no baseline comparisons, and no error bars. The reported 'average agreement level score of 4.55/5' is not an inter-rater reliability statistic; Cohen's or Fleiss' kappa or a similar measure is needed to establish that the ground-truth obstruction labels are reproducible.","section":"Section 2 (ViDDAR)"},{"comment":"The 88.94% accuracy on AR-VIM is computed against ratings from 18 participants who judged whether the AR version misrepresents the real-world information, but the manuscript reports no inter-rater agreement statistic, no rating scale details, and no label-aggregation procedure. If the participant ratings are inconsistent, the reported accuracy reflects agreement with a noisy subjective heuristic rather than objective detection of semantic manipulation. This is load-bearing for the central safety claim and must be addressed with standard reliability metrics and a clear label-generation protocol.","section":"Section 2 (VIM-Sense)"},{"comment":"The core experimental results are attributed to the authors' prior work, references [5] and [7], and reference [5] is described as 'Submitted to IEEE ISMAR, 2025', meaning it is not available for verification. A journal manuscript must provide sufficient experimental detail itself; citing an unpublished manuscript for the central accuracy and latency claims prevents reviewers and readers from checking that the results are real and reproducible.","section":"Section 2 and References"},{"comment":"The text states that VIM-Sense is based on a 'formal taxonomy' defined along 'attack format and attack purpose', but the axes and their categories are never defined in the manuscript. Without a concrete specification of the taxonomy, the claim that the system generalizes 'across different attack formats' cannot be evaluated, and the reader cannot tell which attack types the 452 AR-VIM pairs actually cover.","section":"Section 2 (VIM taxonomy)"},{"comment":"The manuscript says VIM-Sense 'consistently outperforming baseline methods, including OCR-only and VLM-only approaches', but provides no numerical results, no baseline descriptions, and no evaluation metric definition for those comparisons. This makes the superiority claim unverifiable even for the reported dataset.","section":"Section 2 (baselines)"}],"minor_comments":[{"comment":"The AR-Edge-Cloud architecture diagram is not explained in the text; the functions of 'Data Receiving Module', 'Response Post-Processing Module', and 'Light-Weight Processing Modules' are unclear to the reader.","section":"Figure 1"},{"comment":"The sentence 'allowing the system to generalize across different attack formats purposes' is missing a conjunction; it should read 'across different attack formats and purposes'.","section":"Section 2 (VIM-Sense)"},{"comment":"Reference [5] is listed as 'Submitted to IEEE ISMAR, 2025'; if the manuscript is submitted, the citation should be marked as unpublished and the text should not rely on it for the central evaluation.","section":"References"},{"comment":"The phrase 'seeks feedback on perceptual modeling, multimodal AR content implementation, and lightweight model adaptation' is appropriate for a doctoral consortium abstract but unusual for a journal paper; the authors should reframe the contribution for a journal audience.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads like a Doctoral Consortium or workshop position statement rather than a full research paper. It is very short and the core evidence is delegated to two prior papers by the same group, one of which is still under submission. For a journal venue, the authors should either expand the manuscript into a self-contained paper with detailed experiments, dataset statistics, inter-rater reliability measures, baselines, and error bars, or reframe it explicitly as a research vision paper. The self-citations themselves are legitimate, but relying on an unpublished manuscript for the main numbers is a problem for the review process. The topic is relevant and timely, so a major revision with added experimental detail is appropriate if the authors can supply it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a doctoral consortium summary, not a research paper. The two systems, ViDDAR and VIM-Sense, are already described in the authors' prior publications, and the text explicitly says so. What's new here is the framing of task-detrimental AR content, the two-axis taxonomy for visual information manipulation attacks, and three future directions. That is a thin amount of new material, but the paper does it well: the problem is clearly motivated, the architecture is described at an appropriate level, and the future directions are concrete and tied to open problems in the field.\n\nThe main weakness is that the accuracy claims (92.15% and 88.94%) appear in the text without any experimental detail. No protocols, no baselines, no error bars, no dataset statistics beyond pair counts. The ground-truth labels are human-generated, and the only agreement metric reported is an average Likert score of 4.55/5, which is not a standard inter-rater reliability measure. The stress-test note is right: if the human ratings are noisy, the accuracy numbers reflect agreement with a human heuristic rather than an objective ground truth. That said, this is a summary, and the authors are transparent about the fact that the evidence lives in their prior papers. The soft spot is not that the summary is short; it's that the safety claims are stated as settled results when a reader of this text cannot verify them.\n\nWho is this for? Someone who wants a high-level overview of the authors' AR safety work and a sense of where they're heading. It could be useful for a reading group discussing research directions in AR safety or VLM-based scene understanding. It is not for someone trying to evaluate whether the systems actually work.\n\nFor peer review: I would not send this to a research journal as a full paper. It would be a desk reject with an invitation to cite the underlying papers. But as a doctoral consortium submission or a workshop position statement, it's appropriate and should be reviewed as such.","headline":"A clear, honest doctoral-consortium summary of two published AR safety systems; fine for a workshop, but not a self-contained research paper.","tokens_in":4346,"tokens_out":3297,"would_cite":false,"duration_ms":36049,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that two vision-language-model-based systems, ViDDAR and VIM-Sense, can detect task-detrimental augmented reality content—both obstruction and semantic manipulation—with reported accuracy of 92.15% and 88.94%, respectively.","keywords":["augmented reality","vision-language model","obstruction attack","visual information manipulation attack","AR content safety","multimodal reasoning","scene understanding","latency-sensitive deployment"],"falsifier":"Take the 306 raw-AR pairs and 452 AR-VIM clips, have independent annotators relabel them under objective criteria (such as blinded expert agreement on clear obstruction or semantic change), and measure inter-annotator agreement; low agreement would mean the reported accuracies are not measuring a stable ground truth.","tokens_in":3398,"feed_emoji":"🥽","tokens_out":4166,"duration_ms":41318,"temperature":0.7,"pith_summary":"This paper argues that the safety risks of augmented reality—virtual content that blocks critical real-world information or subtly changes its meaning—can be detected automatically by combining vision-language models with segmentation, OCR, and multimodal reasoning. It reports two systems: ViDDAR, which flags obstruction attacks with up to 92.15% detection accuracy in about half a second, and VIM-Sense, which flags semantic manipulations of real-world information with 88.94% accuracy. The intended payoff is a scalable, human-aligned safety layer for AR devices that works across varied scenarios. The paper also outlines three next steps: perceptually aligned quality assessment, detection of audio-visual attacks, and lightweight VLM deployment.","feed_headline":"Two AI systems catch AR overlays that hide or twist reality","feed_subtitle":"ViDDAR flags blocking overlays at 92% accuracy; VIM-Sense catches subtle scene edits at 89%.","key_machinery":"The load-bearing mechanism is the contrast between the raw real-world image and the AR-composited image, interpreted through a vision-language model. For obstruction, the VLM picks the objects that matter, segmentation models produce precise masks, and an obstruction ratio computed from mask overlap triggers a flag. For manipulation, OCR extracts text from both views, a prompt states the differences, and the VLM decides whether those differences alter the scene's meaning. This design replaces geometric scene reconstruction and low-level image metrics with semantic, language-grounded comparison.","core_discovery":"The central claim is that task-detrimental AR content, defined as virtual elements that obstruct critical information or manipulate user perception, can be reliably detected by systems that let a VLM interpret scene semantics and then verify those semantics against rendered AR output. ViDDAR asks a VLM to identify semantically important objects, localizes them with detection and segmentation models, and compares their masks against the AR overlay to compute an obstruction ratio; it reached 92.15% accuracy with 533 ms average latency on a custom dataset of 306 raw-AR pairs. VIM-Sense extracts text via OCR from raw and AR frames, builds natural-language prompts describing differences, and has the VLM judge whether meaning was manipulated; it reached 88.94% accuracy on the 452-pair AR-VIM dataset. The paper further claims that the approach works on commercial hardware (Meta Quest 3 for ViDDAR, mobile platforms for VIM-Sense), albeit at latencies that are usable for general-purpose but not time-critical AR.","pith_inferences":["The two systems cover complementary attack types, so a combined pipeline could offer end-to-end protection; the paper does not report such an integration.","The large latency gap between ViDDAR (533 ms) and VIM-Sense (7.17 s) suggests that semantic manipulation detection will need hierarchical inference (edge pre-screening, cloud verification) before it can serve time-critical applications.","The accuracy numbers rest on manually annotated, custom datasets with no external benchmark; independent replication on a standardized set of AR attacks would determine whether the approach generalizes.","The taxonomy of VIM attacks along format and purpose could be a useful starting point for a broader shared vocabulary for AR content safety, including audio-visual manipulations."],"forward_implications":["If ViDDAR's results hold, AR devices can automatically detect when virtual overlays hide critical objects such as signs or hazards, and can respond by making the overlay transparent, within about half a second.","If VIM-Sense's results hold, even subtle edits—a changed character on a sign, a misleading arrow—can be caught before the user acts on false information.","The modular pipeline (VLM for semantics, off-the-shelf segmentation and OCR for grounding) means detection can be tuned per application by changing prompts and thresholds without rebuilding the system.","The reported latencies suggest the architecture is deployable on commercial headsets and phones for most non-time-critical AR uses.","The proposed extension to audio-aware multimodal detection anticipates AR attacks that lie outside the visual channel."],"supporting_citations":[{"why":"Describes ViDDAR, the system whose 92.15% accuracy and 533 ms latency underpin the obstruction detection claim.","marker":"[7]"},{"why":"Describes VIM-Sense, the multimodal reasoning system whose 88.94% accuracy on AR-VIM underpins the manipulation detection claim.","marker":"[5]"},{"why":"Reports the Meta Quest 3 implementation that demonstrates ViDDAR on commercial AR hardware.","marker":"[6]"},{"why":"Earlier AR output security work using object detection and OCR that this paper positions as a limited baseline.","marker":"[3]"},{"why":"Survey of geometric occlusion modeling in AR that motivates the shift to semantic detection.","marker":"[4]"}],"fun_headline_variants":["Two AI systems catch AR overlays that hide or twist reality","AR safety: VLMs flag bad overlays at 92% and 89%","New detectors stop AR from hiding or distorting truth","ViDDAR and VIM-Sense tackle deceptive AR content","AI checks AR scenes for obstruction and manipulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that the custom datasets' manual labels and user ratings correctly define what content is harmful or misleading; if those labels are subjective or unrepresentative of real AR attacks, the accuracy numbers do not establish safety.","fun_headline_variants_meta":{"raw":{"variants":["Two AI systems catch AR overlays that hide or twist reality","AR safety: VLMs flag bad overlays at 92% and 89%","New detectors stop AR from hiding or distorting truth","ViDDAR and VIM-Sense tackle deceptive AR content","AI checks AR scenes for obstruction and manipulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000135,"raw_usage":{"total_tokens":1113,"prompt_tokens":883,"completion_tokens":230,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":145}},"tokens_in":499,"tokens_out":230,"duration_ms":3249,"temperature":1.0,"reasoning_tokens":145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:55:48.125346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 306 raw-AR pairs and 452 AR-VIM clips, have independent annotators relabel them under objective criteria (such as blinded expert agreement on clear obstruction or semantic change), and measure inter-annotator agreement; low agreement would mean the reported accuracies are not measuring a stable ground truth.","supporting_citations":[{"cited_title":"Lebeck, K","cited_arxiv_id":null,"evidence_quote":"Describes ViDDAR, the system whose 92.15% accuracy and 533 ms latency underpin the obstruction detection claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes VIM-Sense, the multimodal reasoning system whose 88.94% accuracy on AR-VIM underpins the manipulation detection claim."},{"cited_title":"Latif and X","cited_arxiv_id":null,"evidence_quote":"Reports the Meta Quest 3 implementation that demonstrates ViDDAR on commercial AR hardware."}],"review_version":1}