{"id":"f218129e-db54-4787-b900-941736adf7c6","arxiv_id":"2507.16704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.","lead":"Screen2AX turns a single macOS screenshot into a structured accessibility tree using object detection, image captioning, and group detection, and the authors release three new macOS UI datasets plus a task benchmark. The reported result is that these generated trees help a GPT-4 agent click the correct UI element more often than built-in macOS accessibility data or OmniParser V2.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 77% F1 is an edge-based score evaluated against the same manually-corrected developer accessibility trees used as training targets; Sections 5.3 and 6.1.1 concede these labels are inconsistent, so the metric may certify fidelity to flawed metadata rather than true accessibility quality.","rationale":"The reader's weakest assumption is that the manually corrected accessibility labels used as ground truth are accurate and consistent enough to train and score the system. I agree this is the most load-bearing concern: Section 4.1 derives the labels from built-in accessibility metadata, Section 5.3 concedes possible inaccuracies, and Section 6.1.1 gives concrete examples of inconsistent developer-provided labels. Because the same corrected trees are both the training targets for hierarchy generation and the evaluation targets for the 77% F1 claim, the metric can reward a model for reproducing the exact flaws the system is supposed to fix. The paper does include substantial independent support in the form of open datasets, code, and a reproducible benchmark, and the pipeline is clearly described; but none of that resolves the label-quality question. The abstract's additional 2.2x claim is also not directly supported by Table 8, where 33.7% versus 16.9% is only a 2.0x improvement, though I treat the ground-truth circularity as the primary issue. A clean expert re-annotation of a held-out subset is the direct, feasible test: if the hierarchy metrics and agent success rates survive that check, the central claim is substantially strengthened; if they do not, the conditional acceptance should become a rejection or a request for re-evaluation. Since the reader already issued a conditional verdict and requested exactly this kind of clarification, my analysis does not change the verdict.","tokens_in":21510,"tokens_out":3791,"duration_ms":36155,"concrete_test":"Select 100 screenshots from Screen2AX-Tree that were not used for any model training or validation. Have two macOS accessibility experts independently re-annotate the full hierarchy (elements, groups, parent-child edges, bounding boxes) directly from screenshots, blind to both the built-in accessibility metadata and the published Screen2AX labels; measure inter-annotator agreement. Recompute Table 6 edge F1, leaves F1, CM, and GED for Screen2AX against this clean ground truth, and rerun the Screen2AX-Task agent evaluation on the corresponding subset. If edge F1 drops by more than 5 points, if the Screen2AX-vs-built-in success gap in Table 8 reverses or shrinks below statistical significance, or if inter-annotator agreement is low (edge-set agreement below 0.7), the 77% and 2.2x claims are not robust to ground-truth noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Screen2AX 'accurately infers hierarchy trees, achieving a 77% F1 score in reconstructing a complete accessibility tree' rests on ground truth whose quality is asserted but never independently verified. Section 4.1 states the Screen2AX-Tree and Screen2AX-Group labels were produced by taking built-in macOS accessibility metadata and 'manually correct[ing]' it with Roboflow. Section 5.3 then admits the dataset 'may contain inaccuracies or inconsistencies,' and Section 6.1.1 confirms that the underlying developer-provided metadata is 'inconsistently labeled and positioned across different applications' (e.g., semantically similar buttons with different or missing roles, inconsistent bounding boxes). Since both the YOLOv11 hierarchy model and the 77% evaluation use these same corrected system trees, any systematic error—invisible elements, shifted boxes, inconsistent grouping—is simultaneously learned and rewarded. The paper reports no inter-annotator agreement, no held-out clean re-annotation, and no comparison of the corrected labels against an independent expert standard. The 77% figure is also only the edge-based F1 from Table 6; leaves F1 is 31% and group CM is 55%, so calling it a 'complete accessibility tree' score overstates what was measured. The same concern applies to the agent claim: Screen2AX-Task success is measured against human-annotated boxes and captions, but the hierarchy representation used by the agent is only as good as the label source it was trained on. If the ground-truth corrections encode the annotators' or developers' grouping conventions, the reported gains may reflect fitting those conventions rather than generating genuinely better accessibility metadata.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Screen2AX, a screenshot-only pipeline for generating hierarchical macOS accessibility metadata, combining YOLOv11-based element and group detection, OCR, BLIP-based icon captioning, and a heuristic or learned grouping step. It introduces three datasets (Screen2AX-Tree, Screen2AX-Element, Screen2AX-Group) and the Screen2AX-Task agent benchmark, and evaluates the pipeline on element detection, captioning, hierarchy reconstruction, and agent grounding. The central claims are that Screen2AX reconstructs complete accessibility trees at 77% F1 and yields a 2.2x improvement over native accessibility representations for agent task execution while surpassing OmniParser V2 on ScreenSpot.","tokens_in":21854,"tokens_out":4899,"duration_ms":45532,"significance":"The work addresses a real and under-served problem, and the released datasets and open-source code are valuable community assets. If the headline results survived closer scrutiny, this would be a meaningful step toward vision-only accessibility generation for desktop UIs. However, the evidence as presented does not yet support the complete-tree and 2.2x claims; the strengths are the resource contribution and a plausible pipeline, not the current quantitative overstatements.","major_comments":[{"comment":"The '77% F1 score in reconstructing a complete accessibility tree' is an edge-based F1 (Table 6, 'F1 Score, % 77'), not a tree-level or complete-tree metric. The same table reports Leaves F1 of 31% and a Container Match (CM) of 55%; the abstract should either specify 'edge-based F1' or aggregate the full set of metrics, since 77% as stated materially overstates the measured reconstruction quality.","section":"Abstract and Table 6"},{"comment":"The ground-truth trees and group labels were produced by manually correcting the same built-in macOS accessibility metadata that the paper documents as inconsistent, and Section 5.3 concedes that the dataset 'may contain inaccuracies or inconsistencies.' Because this label source supplies both the YOLOv11 hierarchy training targets and the evaluation targets, the reported 77% F1 can certify agreement with a noisy label convention rather than with true accessibility structure. The manuscript should report inter-annotator agreement on a re-annotated held-out subset and compare corrected labels against an independent expert standard.","section":"Sections 4.1, 5.3, and 6.1.1"},{"comment":"The abstract's '2.2x performance improvement over native accessibility representations' is not recoverable from Table 8: 33.7/16.9 = 1.99, not 2.2. The text's 16.8 percentage-point absolute gain is not a multiplicative improvement. Please correct the claim or provide the computation that yields 2.2x.","section":"Section 5.4.2 and Table 8"},{"comment":"The component metrics are low enough to undercut the end-to-end story: element detection F1 is 43.8% (Table 3), leaves F1 is 31% (Table 6), and Screen2AX-Task success is 33.7% (Table 8). The paper should explain how an edge-based F1 of 77% is compatible with these numbers, and should not describe the detected result as a 'complete' accessibility tree.","section":"Tables 3 and 6, and Section 5.4.2"},{"comment":"The claimed advantages over OmniParser V2 (36.6 vs. 31.9 on ScreenSpot; 33.7 vs. 28.0 on Screen2AX-Task) and the 3.0-3.7 percentage-point gains from hierarchy are reported without confidence intervals, repeated trials, or significance tests, despite GPT-4 stochasticity and modest sample sizes (172 ScreenSpot instructions; 166 Screen2AX-Task images). Add variance estimates or significance testing before asserting superiority.","section":"Tables 7 and 8"}],"minor_comments":[{"comment":"The text contains the typo 'OmniPars r v2' in the sentence 'improvement of 5.7% compared to OmniPars r v2.'","section":"Section 5.4.2"},{"comment":"The sentence 'we initially use the ocrmac tool ... to extract text from bu tons' contains a typo; 'bu tons' should be 'buttons.'","section":"Section 6.1.2"},{"comment":"The opening sentence 'we qualitatively evaluate each core component of our stem' should read 'system' instead of 'stem.'","section":"Section 6"},{"comment":"Both references list the SeeClick paper with overlapping arXiv identifiers; consolidate them to avoid duplicate citations.","section":"References [9] and [10]"},{"comment":"The phrase 'we comprised 435 images' should be 'we compiled 435 images' or 'the dataset comprises 435 images' to match the intended meaning.","section":"Section 4.1.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a vision/accessibility venue, but the authors should be asked to reconcile the abstract with Tables 6 and 8 before publication. The open datasets are a genuine contribution and should not be delayed by the claim revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on Screen2AX. The genuinely new thing here is empirical: three public macOS UI datasets (112 apps, 1127 screenshots) with element, group, and hierarchy annotations, plus a small task benchmark. If you work on desktop GUI agents or accessibility metadata, those datasets are the real contribution, and the code is out. The ScreenSpot number (36.6% vs OmniParser V2's 31.9% on macOS tasks) is an external, public-benchmark comparison that supports the claim that a vision-only tree helps agent grounding. I credit that: it's a comparison they didn't control.\n\nThe soft spots are in the framing, not the pipeline. The abstract says \"77% F1 in reconstructing a complete accessibility tree.\" What Table 6 actually reports is edge-based F1 of 77%; leaves F1 is 31%, and container match is 55%. Those are very different statements. Same for the \"2.2x improvement over native accessibility\": Table 8 gives 33.7% vs 16.9%, which is a 1.99x ratio, not 2.2x. The number doesn't come from the table. The text correctly describes it as an absolute improvement of 16.8 percentage points, so the abstract is just sloppy—but sloppy in the headline claim is the worst place.\n\nThe deeper concern is ground truth. The hierarchy labels are \"manually corrected\" built-in macOS accessibility trees, corrected in Roboflow. Section 5.3 and Section 6.1.1 both concede the underlying developer metadata is inconsistent (missing roles, shifted boxes, invisible elements). So the YOLOv11 hierarchy model is trained and scored against the same corrected trees, with no inter-annotator agreement, no held-out clean re-annotation, and no independent expert standard. The model may be learning the annotators' grouping conventions rather than producing objectively better metadata. The 77% edge F1 could partly certify fidelity to flawed labels. That's not fatal—the ScreenSpot and Screen2AX-Task results use human-annotated boxes, so they're less exposed—but the hierarchy metrics are weaker than the abstract pretends.\n\nAlso worth noting: no error bars, single runs, small held-out sets (172 ScreenSpot tasks, 166 Task images). And they call it \"real-time\" in the abstract while the limitations section says latency needs work.\n\nWho's it for: people building desktop GUI agents and accessibility tooling; the datasets are the thing to take away. It deserves a serious referee. My recommendation: send it to review, but require the authors to (1) rephrase the abstract to match Tables 6 and 8, (2) add uncertainty estimates, and (3) either validate the corrected labels against an independent standard or discuss the circularity risk explicitly.","headline":"A genuinely useful macOS accessibility dataset and pipeline, but the abstract's 77% 'complete tree' F1 and 2.2x agent gain are not what Tables 6 and 8 actually show.","tokens_in":22418,"tokens_out":3391,"would_cite":true,"duration_ms":29763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Screen2AX claims that a single macOS screenshot contains enough visual information to reconstruct the application's full accessibility tree, and reports a 77% F1 score for tree reconstruction.","keywords":["accessibility metadata","macOS","UI element detection","hierarchy generation","AI agents","screen parsing","object detection","dataset"],"falsifier":"Re-annotate a held-out set of the released screenshots with independent annotators who build trees from scratch without seeing built-in metadata; if inter-annotator agreement is low, or Screen2AX matches the original labels but not the independent trees, the 77% F1 cannot stand as evidence of accurate reconstruction.","tokens_in":21328,"feed_emoji":"♿","tokens_out":4799,"duration_ms":48333,"temperature":0.7,"pith_summary":"This paper tries to establish that desktop accessibility metadata can be generated automatically from pixels rather than sourced from developers. It reports that only about a third of macOS applications expose full accessibility metadata, and that a vision pipeline can recover a usable accessibility tree from one screenshot. If true, screen readers and AI agents would no longer depend on developers having annotated custom UI elements. The paper backs this with three released datasets, a 77% F1 tree-reconstruction score, and agent benchmarks in which the generated hierarchy outperforms built-in metadata and the OmniParser V2 baseline.","feed_headline":"Screen2AX builds macOS accessibility trees from one screenshot","feed_subtitle":"It beats built-in metadata and OmniParser for AI agent tasks, with a 77% tree-reconstruction F1.","key_machinery":"The central object is the accessibility tree itself: a window-rooted hierarchy whose internal nodes are semantic groups such as toolbars and panels, and whose leaves are UI elements carrying role, name, value, and bounding box. Screen2AX's mechanism is a three-stage vision pipeline: a fine-tuned YOLOv11 detector localizes and classifies elements; OCR plus a fine-tuned BLIP model supplies text and icon captions; and a second YOLOv11 model trained on a single AXGroup class predicts group boxes that are assembled into the final tree. The tree format is what lets the output plug into the same JSON grounding format used by agents and accessibility consumers.","core_discovery":"Screen2AX's central claim is that the full macOS accessibility hierarchy—element bounding boxes, roles, names, descriptions, and parent-child grouping—can be recovered from a single screenshot. The pipeline detects and classifies UI elements with a fine-tuned YOLOv11 model, extracts text with OCR, captions icon-only buttons with a fine-tuned BLIP model, and uses a second YOLOv11 model to predict semantic groups that are assembled into a tree mirroring the macOS AXWindow/AXGroup structure. On its own dataset, the method reaches 65.4% element-detection accuracy at IoU 0.5, a 77% F1 for edge-based tree reconstruction, and a 55% group-match score. When the resulting tree is fed to a GPT-4-based agent, task success rises to 33.7% versus 16.9% with built-in accessibility metadata and 28.0% with OmniParser V2, and ScreenSpot grounding success reaches 36.6% versus 31.9%.","pith_inferences":["Editorial inference: the same pixel-to-tree pipeline could plausibly transfer to Windows, Linux, or web interfaces if similar labeled hierarchies existed, since the method itself is not macOS-specific.","Editorial inference: the reported one-third full-support statistic implies a large deployment surface, but real-world usefulness would need validation with screen-reader users, not only agent benchmarks.","Editorial inference: because a single screenshot captures one static state, popovers, hover states, and live-updating content would likely require temporal or multi-frame input to be handled reliably.","Editorial inference: the mobile-icon domain gap visible in the BLIP captioning failures suggests that a desktop-specific icon captioning dataset would directly improve both caption quality and agent task success."],"forward_implications":["Applications with missing or incomplete developer-authored accessibility metadata could receive a usable accessibility tree without any developer changes, simply from a screenshot.","AI agents grounded in Screen2AX-generated trees would perform roughly twice as well on simple UI tasks as agents grounded in native macOS accessibility metadata.","The hierarchical structure itself contributes to agent performance, with the full tree improving task success by about 3% over a flat list of the same detected elements.","The released datasets give future work a common benchmark for macOS UI element detection, grouping, and accessibility-tree reconstruction.","Real-time generation at roughly 0.2 seconds per element-detection pass makes the approach a plausible component of live accessibility tooling."],"supporting_citations":[{"why":"Introduced the vision-only mobile accessibility generation pipeline whose grouping and metadata steps Screen2AX adapts to desktop.","marker":"[44]"},{"why":"Screen Parsing supplies the semantic-group framing and the edge/leaf/group metrics used to score hierarchy reconstruction.","marker":"[41]"},{"why":"OmniParser V2 is the state-of-the-art screen parser that Screen2AX compares against and outperforms on both agent benchmarks.","marker":"[23]"},{"why":"ScreenSpot provides the macOS grounding benchmark on which Screen2AX reports 36.6% success versus OmniParser's 31.9%.","marker":"[10]"},{"why":"BLIP is the vision-language model fine-tuned for icon captioning, the source of Screen2AX's element descriptions.","marker":"[20]"},{"why":"GPT-4 is the language model used as the agent for both benchmark evaluations and for judging caption semantic equivalence.","marker":"[27]"},{"why":"Prior YOLO UI detection results motivate the choice of YOLOv11 for both element and group detection.","marker":"[11]"}],"fun_headline_variants":["Screen2AX: vision-only macOS accessibility tree generation","Screen2AX reconstructs macOS accessibility trees with 77% F1","One screenshot, full macOS accessibility tree: Screen2AX","Screen2AX infers macOS accessibility hierarchy from one image","Screen2AX beats native and OmniParser for macOS accessibility trees"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human-corrected accessibility labels used as ground truth are accurate and consistent enough that a model trained on them is truly learning to reconstruct accessibility structure, not learning the labeling quirks of the annotators.","fun_headline_variants_meta":{"raw":{"variants":["Screen2AX: vision-only macOS accessibility tree generation","Screen2AX reconstructs macOS accessibility trees with 77% F1","One screenshot, full macOS accessibility tree: Screen2AX","Screen2AX infers macOS accessibility hierarchy from one image","Screen2AX beats native and OmniParser for macOS accessibility trees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000424,"raw_usage":{"total_tokens":2220,"prompt_tokens":1035,"completion_tokens":1185,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":1100}},"tokens_in":651,"tokens_out":1185,"duration_ms":9764,"temperature":1.0,"reasoning_tokens":1100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:04:13.632849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a held-out set of the released screenshots with independent annotators who build trees from scratch without seeing built-in metadata; if inter-annotator agreement is low, or Screen2AX matches the original labels but not the independent trees, the 77% F1 cannot stand as evidence of accurate reconstruction.","supporting_citations":[{"cited_title":"Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels","cited_arxiv_id":"2101.04893","evidence_quote":"Introduced the vision-only mobile accessibility generation pipeline whose grouping and metadata steps Screen2AX adapts to desktop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4 is the language model used as the agent for both benchmark evaluations and for judging caption semantic equivalence."}],"review_version":1}