{"id":"33e7fb91-3abe-4bbe-ae8f-1e6879f0260d","arxiv_id":"2607.03100","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Runtime interaction traces plus criterion-specific evidence packets let an LLM judge dynamic WCAG criteria far more accurately than unconstrained browser-agent audits.","lead":"Flow-A11y runs natural-language user scenarios in a real browser, records runtime traces, and judges 45 interaction-dependent WCAG criteria with evidence-gated LLM analysis. It reaches 45.6% oracle agreement versus 3.3% for a generic browser agent, showing a workable path to automate dynamic accessibility checks that static scanners miss.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The comparative claim rests on a deliberately weak baseline and author-built oracle; the 10× gain may not show that runtime evidence packets specifically enable dynamic WCAG assessment.","rationale":"The reader correctly flags the author-built oracle as a weak assumption and lands on CONDITIONAL with high confidence; that is the right overall posture. The more load-bearing soft spot for the strongest claim, however, is the causal attribution in RQ2: the 10× accuracy gap is real against the published baseline, but that baseline is intentionally stripped of scenario structure as well as of evidence packets, so it cannot isolate the paper’s distinctive mechanism. The historical pre/post layer comparison is acknowledged by the authors as non-causal. A single controlled ablation on frozen traces would settle whether the evidence-packet design, rather than merely “run a scenario and ask an LLM,” is what enables the reported gains. Until that (or an equivalent independent oracle + stronger baseline) exists, CONDITIONAL remains appropriate; the concern does not justify REJECT because the engineering result and the relative improvement over unconstrained agents are still useful and honestly reported.","tokens_in":15922,"tokens_out":740,"duration_ms":8770,"concrete_test":"Re-run the 17 completed scenarios once, freeze the resulting runtime traces, then score three judges on the identical 765 rows: (1) full Flow-A11y, (2) the same LLM with only the raw ordered trace and WCAG text (no family packets, no pre-LLM gate, no evidence-ref verifier), (3) the published naive audit. If exact accuracy and fail precision of (2) remain within ~5 points of (1), the packet/gate layer is not the enabling factor the claim asserts; if (1) still substantially beats (2) and both beat (3), the claim holds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that criterion-specific runtime evidence packets enable automated assessment of interaction-dependent WCAG barriers, evidenced by 45.6% exact accuracy and 41.4% fail precision vs 3.3% and zero fails for a generic browser-use audit (RQ2, Tables III–V). That comparison is load-bearing for the “enable” claim, yet the baseline is defined as a naive agent given only a URL, step budget, and criterion IDs, with no scenario, probes, evidence packets, or gating (Section V-A). It therefore confounds the contribution of runtime evidence packets with the contribution of scenario-driven execution itself. The paper also reports a historical pre-layer vs final-pipeline comparison (Table VIII) that improves fail precision from 23.5% to 41.4% and eliminates invalid refs, but the runs use different denominators (630 vs 765) and multiple simultaneous changes, so the authors correctly call it descriptive only. Without a controlled ablation that holds the interaction trace fixed and toggles only the evidence-packet / gate / verifier stack, the central causal attribution to “criterion-specific runtime evidence packets” remains under-supported relative to the wording of the claim. The author-constructed 19×45 oracle (Section V-B) is a secondary but related risk: labels come from the same authors reviewing the same sites without independent AT execution, so agreement figures can be inflated by shared observation of the same DOM/focus artifacts the system later cites.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"Flow-A11y is a flow-aware accessibility testing system that treats multi-step interaction scenarios, rather than static page snapshots, as the unit of WCAG assessment. Given a URL and a natural-language task, it executes the flow in a real browser, records an ordered multimodal runtime trace, projects criterion-specific evidence packets via nine probe families, gates unsupported judgments before LLM invocation, verifies that FAIL/WARNING outputs cite resolvable evidence fields, and exports auditable criterion-level artifacts. On a new author-built benchmark of 19 public-web scenarios covering 45 dynamic WCAG criteria (765 scored rows after two task failures), the system reports 45.6% exact five-way status accuracy and 41.4% fail precision (12/31 oracle failures), versus 3.3% accuracy and zero true fails for a naive browser-use global audit. A descriptive historical comparison attributes a precision gain (23.5%→41.4%) and elimination of invalid evidence references to the evidence-calibration layer. The paper argues that runtime traces supply actionable evidence for interaction-dependent WCAG criteria that page-level scanners cannot assess.","tokens_in":16288,"tokens_out":1070,"duration_ms":15363,"significance":"The problem framing is timely and well motivated: many WCAG 2.2 criteria (keyboard traps, focus order/visibility, status messages, hover/focus content, etc.) are inherently interaction-dependent, and mainstream scanners (axe-core, Lighthouse, WAVE) are snapshot-bound by design. A unified, evidence-gated pipeline spanning 45 dynamic criteria, together with a public-web scenario benchmark and auditable artifact packages, is a genuine contribution relative to prior work that targets isolated barrier classes (keyboard, dialogs, reflow). Strengths include explicit evidence gating, mechanical evidence-reference verification (0 invalid refs in the final run), separation of execution from judgment, and unusually candid reporting of incomplete rates, zero-accuracy families, and the non-causal status of the pre-layer comparison. If the causal attribution to criterion-specific evidence packets can be tightened and the oracle threat better bounded, the work would offer a practical path for automating a class of accessibility checks that currently require manual evaluation.","major_comments":[{"comment":"RQ2 / §V-A and Tables III–V: The load-bearing claim that “criterion-specific, runtime-evidence-grounded dynamic WCAG analysis” enables the reported gains is not isolated by the baseline. The naive browser-use audit receives only a URL, step budget, and criterion IDs—no scenario, no probes, no evidence packets, no gate. The 45.6% vs 3.3% comparison therefore confounds scenario-driven execution with the evidence-packet/gate/verifier stack. A controlled ablation that holds the same interaction traces fixed and toggles only the evidence layer (or at least a scenario-aware agent without packets) is needed before the abstract and RQ2 wording can attribute the 10× gain specifically to criterion-specific evidence packets.","section":null},{"comment":"Abstract and Table VIII: The abstract states that “its evidence-calibration layer improves fail precision from 23.5% to 41.4% and eliminates invalid evidence references.” §VII-B correctly labels this comparison descriptive (different scored denominators 630 vs 765; multiple simultaneous pipeline changes). Presenting that precision gain as a primary result in the abstract overstates what the data support. Either run a controlled same-denominator ablation of the packet/gate/validator stack, or rephrase the abstract to report only the fully controlled Flow-A11y vs naive-audit comparison.","section":null},{"comment":"§V-B (oracle construction) and §VIII (construct validity): The 19×45 oracle is author-labeled from live-site review without independent assistive-technology execution (screen readers, etc.). Shared observation of the same DOM/focus/mutation artifacts that the system later cites can inflate agreement and fail precision. With only 31 oracle-fail rows, even modest labeling bias moves the central precision/recall figures. At minimum, report inter-rater agreement on a double-labeled subset, or validate a sample of FAIL/PASS labels with AT-based review; otherwise the absolute accuracy claims should be framed more cautiously as agreement with author judgment rather than ground-truth WCAG conformance.","section":null},{"comment":"§VI-C / Table VI: Three probe families (animation/flashing, timing/interruption, consistency/navigation) score 0% exact accuracy, and media’s 100% is entirely NOT APPLICABLE. The paper correctly notes that the evidence gate abstains rather than fabricates, but the claim of a “unified framework … currently covering 45 dynamic success criteria spanning 9 probe families” is then largely aspirational for half the families. Either expand the scenario set so those families produce applicable runtime evidence, or narrow the claimed coverage to families with non-trivial applicable rows and treat the rest as future work in the abstract and contributions.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing: this is a clean systems paper that actually runs multi-step public-web flows, builds criterion-specific runtime evidence, and gates LLM judgments so they cannot invent citations. On the same 765-row oracle it beats a generic browser-use audit 45.6% vs 3.3% exact accuracy and finds 12 of 31 oracle fails where the baseline finds none. That relative result is real and useful.\n\nWhat is new is the packaging, not the idea of driving a browser for a11y. Chiou et al. already hit keyboard traps, dialogs, and reflow; GenA11y and others already put LLMs on WCAG. Flow-A11y unifies 45 interaction-dependent criteria under one probe/evidence-packet/pre-LLM-gate/reference-verifier stack and ships a 19-scenario public-web oracle. The engineering is careful: incomplete when evidence is missing, fail-confidence downgrades, auditable artifacts, zero invalid evidence refs after calibration. They are honest that media “perfect accuracy” is mostly not-applicable gating, and that animation/timing/consistency score zero because the scenarios never produced the right states.\n\nSoft spots, in proportion. The load-bearing RQ2 baseline is a naive agent with URL + criterion IDs and no scenario, so it confounds “run the flow” with “build evidence packets.” The paper itself calls the pre-layer vs final comparison descriptive (different denominators, multiple changes at once). So the abstract’s “runtime traces / evidence packets enable…” claim is a bit stronger than the controlled evidence. The 19×45 oracle is author-built without AT execution; fail n=31 is small; two scenarios never completed. None of that kills the comparative result, but it caps how hard you can lean on absolute precision/recall.\n\nMath and citations look fine for a systems paper—no free parameters, related work is in the right neighborhood, threats section is candid. This is for people building accessibility tooling or continuous a11y in CI, not for pure WCAG theory. I would send it to peer review; referees should demand a fixed-trace ablation of the packet/gate/verifier stack and independent oracle checks, not a desk reject. Worth engaging if you care about dynamic WCAG automation.","headline":"Solid systems paper with a real relative win and a useful benchmark; the 10× claim is real but the causal story about evidence packets is oversold relative to a deliberately weak baseline.","tokens_in":16884,"tokens_out":577,"would_cite":true,"duration_ms":11798,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Runtime interaction traces let automated tools judge dynamic WCAG accessibility barriers that static page scanners cannot see.","keywords":["web accessibility","WCAG","dynamic accessibility testing","flow-aware testing","browser interaction testing","runtime evidence","accessibility automation","software testing"],"falsifier":"Have independent accessibility experts, using assistive technology, re-label the 765 scenario-criterion rows without seeing Flow-A11y’s outputs, then recompute exact five-way accuracy and fail precision; a large drop would falsify the claim that the system’s findings match true interaction-dependent WCAG outcomes.","tokens_in":16822,"feed_emoji":"♿","tokens_out":941,"duration_ms":19898,"temperature":0.7,"pith_summary":"Modern web apps often hide accessibility failures until a user actually acts: keyboard traps, lost focus, unannounced status updates, and modal problems appear only after concrete steps. Static checkers and free-form browser agents largely miss these interaction-dependent WCAG criteria, so teams still rely on manual review. Flow-A11y treats a natural-language user scenario as the unit of analysis. It drives a real browser, records an ordered multimodal runtime trace, builds criterion-specific evidence packets, gates unsupported judgments, and emits only findings that cite resolvable runtime evidence. On 19 real public-web scenarios covering 45 dynamic criteria, it reaches 45.6 percent exact oracle agreement and 41.4 percent fail precision—more than ten times a generic browser-agent audit that finds zero failures—arguing that grounded runtime evidence is a practical path to automating checks that page-level tools leave to people.","feed_headline":"Runtime traces automate WCAG checks page scanners miss","feed_subtitle":"Flow-A11y beats a generic browser agent by more than ten times on 45 interaction-dependent criteria.","key_machinery":"The criterion-specific evidence packet: a compact, family-scoped bundle of focus sequences, keyboard events, DOM mutations, live-region signals, media state, and related runtime facts drawn from the ordered trace, which both gates LLM judgment and forces every fail or warning to cite resolvable fields.","core_discovery":"The paper establishes that interaction-dependent WCAG criteria become machine-assessable when an ordered runtime trace is projected into family-specific evidence packets, gated for sufficiency and applicability, and judged only with resolvable evidence references. Under that discipline Flow-A11y matches a manual oracle on 45.6 percent of 765 scenario-criterion rows and recovers 12 of 31 true failures at 41.4 percent precision, while a generic browser agent scores 3.3 percent accuracy and zero true failures. Criterion-specific runtime evidence, not mere browser exploration, is what makes dynamic accessibility automation feasible.","pith_inferences":["Continuous-integration pipelines could run scenario-driven traces on interactive UI changes to catch regressions in focus order or live-region announcements before release.","The same evidence-packet pattern may transfer to mobile accessibility, where dynamic content transitions already motivate specialized detectors.","If sites published machine-readable scenario descriptions, third-party auditors could re-run the same flows without reverse-engineering tasks.","A precision-over-recall calibration implies product teams may prefer fewer high-confidence tickets over noisy full-coverage reports."],"forward_implications":["Page-level scanners can be paired with flow-aware tools that cover keyboard, focus, status-message, and related dynamic criteria they currently leave to manual review.","Accessibility reports can return auditable, artifact-backed findings instead of unsupported free-form model assertions.","Curated natural-language scenario libraries become reusable assets for dynamic WCAG testing, analogous to functional end-to-end suites.","Evidence gating and fail-confidence validation become a reusable pattern for any LLM-based quality judgment that must stay grounded in runtime artifacts.","Zero-accuracy families such as animation, timing, and consistency become concrete targets for criterion-aware scenario design."],"fun_headline_variants":["Runtime traces turn dynamic WCAG into machine-checkable evidence","Flow-A11y: ordered traces make interaction WCAG oracle-matchable","Evidence packets from flows beat page scanners on 45 WCAG criteria","Gated runtime evidence lifts fail precision to 41.4% on dynamic WCAG","Interaction WCAG becomes automatable when traces carry resolvable proof"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The accuracy and precision numbers rest on an author-built oracle of PASS and FAIL labels for the same public sites; if those manual labels systematically miss or invent barriers, the reported agreement collapses.","fun_headline_variants_meta":{"raw":{"variants":["Runtime traces turn dynamic WCAG into machine-checkable evidence","Flow-A11y: ordered traces make interaction WCAG oracle-matchable","Evidence packets from flows beat page scanners on 45 WCAG criteria","Gated runtime evidence lifts fail precision to 41.4% on dynamic WCAG","Interaction WCAG becomes automatable when traces carry resolvable proof"]},"model":"grok-4.5","effort":"low","cost_usd":0.00218,"raw_usage":{"total_tokens":950,"prompt_tokens":832,"num_sources_used":0,"completion_tokens":98,"cost_in_usd_ticks":21800000,"prompt_tokens_details":{"text_tokens":832,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":832,"tokens_out":98,"duration_ms":1557,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T04:53:53.284065+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Have independent accessibility experts, using assistive technology, re-label the 765 scenario-criterion rows without seeing Flow-A11y’s outputs, then recompute exact five-way accuracy and fail precision; a large drop would falsify the claim that the system’s findings match true interaction-dependent WCAG outcomes.","supporting_citations":[],"review_version":1}