{"id":"dce14cf8-8e50-450c-97f0-66c61f16bcf8","arxiv_id":"2507.14293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"WebGuard introduces an action-level risk dataset for web agents and shows that a fine-tuned 7B model improves risk-prediction accuracy from about 38% to 80% and high-risk recall from 20% to 76%, still below deployment standards.","lead":"WebGuard is a new dataset of 4,939 human-labeled web actions (safe, low risk, high risk) from 193 real websites, with benchmarks showing frontier LLMs often miss high-risk actions. Fine-tuning a 7B vision-language model on the data roughly doubles accuracy and sharply increases high-risk recall, though performance still falls short of deployment-grade reliability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth label validity is the load-bearing concern: no inter-annotator agreement is reported and unannotated actions default to SAFE, so the measured accuracy and recall gains cannot be interpreted until labels are independently revalidated.","rationale":"The paper's contribution is a dataset and an empirical claim that fine-tuning on it yields a usable guardrail. Both contributions depend on the 4,939 action-level risk labels being correct and stable. The experiments never generate their own ground truth; every zero-shot and fine-tuned number is scored against the annotation product. If the labels are noisy, systematically biased, or defined by omission, the reported frontier-model deficiencies and the SFT gains are not measuring action risk. The reader's weakest-assumption analysis identifies exactly this point, and I agree with it. I would add one concrete mechanism: the default-SAFE rule turns missed annotations into negative examples, which is a stronger failure mode than ordinary label noise because it is invisible in any review that only re-checks existing labels. The class-count discrepancy (4,956 vs 4,939) reinforces that the dataset statistics have not been fully reconciled. This concern is addressable: an independent reannotation study with inter-annotator agreement and default-SAFE probes would either validate the labels or quantify the bias. Because the concern is real but not yet demonstrated, the reader's CONDITIONAL verdict remains appropriate. Secondary issues such as missing hyperparameters, the omission of the promised F1 metric from Table 2, and the ambiguous duplicate rows for text-only versus multimodal models do not change this assessment. I therefore recommend leaving the reader's verdict unchanged.","tokens_in":16791,"tokens_out":7452,"duration_ms":75541,"concrete_test":"Run an independent reannotation study on a stratified random sample of 300 released test actions (100 per class) plus 100 elements that WebGuard left unannotated on the same saved pages. Have two fresh annotators, uninvolved in the original project, label the 300 actions using only the public risk schema, and ask them to judge whether each of the 100 default-SAFE elements is state-changing. Compute Cohen's kappa or Krippendorff's alpha for the 300 labeled actions and the fraction of default-SAFE elements reclassified as LOW or HIGH. Then recompute WebGuard-VL-7B accuracy and HIGH-risk recall restricted to the 300 actions whose original label is confirmed by both fresh annotators.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that WebGuard makes action-level risk prediction feasible and that fine-tuning lifts accuracy to ~80% and HIGH-risk recall to ~76%—inherits every ground-truth label. Section 2.2 describes a protocol in which trained annotators label state-changing actions and then states: 'All remaining unannotated actions are then labeled as SAFE by default.' This design converts annotation omissions into negative labels: if an annotator misses a consequential action, the result is not a labeling error but a SAFE example. No inter-annotator agreement is reported, and the Annotation Reviewing step uses professionals trained on the same schema and guidelines, so it can catch inconsistencies with the authors' rubric but cannot detect a systematic blind spot about what counts as state-changing. Because the model evaluations are measured against these potentially biased labels, both the frontier-model deficiency result and the fine-tuning improvement could reflect learning annotator behavior rather than true action risk. A concrete warning sign is that the stated class totals (1,108 + 2,284 + 1,564 = 4,956) do not match the stated dataset size (4,939), so the dataset statistics are not fully reconciled. The load-bearing assumption is therefore not that the schema is reasonable, but that the labels are a stable, independently verifiable ground truth; that assumption is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WebGuard, an action-level dataset of 4,939 human-annotated web actions collected from 193 real websites across 22 domains, labeled with a three-tier SAFE/LOW/HIGH risk schema. The authors benchmark frontier LLMs and fine-tuned Qwen2.5/Qwen2.5-VL models on four held-out generalization splits (Long-Tail, Cross-Domain, Cross-Website, Cross-Action). They report that fine-tuning Qwen2.5-VL-7B lifts accuracy from roughly 37% to 80% and HIGH-risk recall from roughly 20% to 76%, while frontier models remain below 60% accuracy and below 60% HIGH-risk recall in zero-shot settings. The dataset, annotation tools, and trained models are released publicly.","tokens_in":17183,"tokens_out":7464,"duration_ms":78682,"significance":"If the label quality is confirmed, WebGuard would be a valuable community resource: it is the first large-scale action-level real-world dataset for web-agent guardrails, and the finding that frontier models have weak zero-shot risk prediction while modest fine-tuned models improve substantially is concrete and falsifiable. The paper uses held-out evaluation splits, includes challenging long-tail websites, and is unusually open about its limitations, including the explicit statement in Appendix G that performance remains below the reliability needed for deployment. The circularity concern raised in the stress-test does not land: the fine-tuned models are scored on actions not used in training, so the main accuracy and recall numbers are not artifacts of memorizing the training labels. The label-validity concern, however, does land: the ground-truth protocol has no reported inter-annotator agreement and a default-SAFE rule that can convert annotation omissions into negative labels, so the evaluation inherits a risk that must be addressed before the headline numbers can be interpreted.","major_comments":[{"comment":"Section 2.2 states that 'All remaining unannotated actions are then labeled as SAFE by default' and reports no inter-annotator agreement. This protocol converts annotation omissions into negative labels: if an annotator overlooks a state-changing element, the resulting item is recorded as a SAFE example rather than as a labeling error. Because every accuracy and recall figure in Table 2 is scored against labels produced this way, the measured fine-tuning gains and frontier-model deficits could reflect annotators' blind spots rather than true action risk. The review step checks the recorded labels and snapshots, but it cannot detect elements that were never annotated. Please report a sampled inter-annotator agreement study, estimate the fraction of SAFE labels that came from the default rule rather than explicit annotation, and audit a random set of unannotated elements to estimate missed state-changing actions.","section":"Section 2.2 (Data Curation Process)"},{"comment":"Section 2.2 reports 1,108 HIGH, 2,284 LOW, and 1,564 SAFE actions, which sum to 4,956, while the dataset is stated to contain 4,939 actions. The split sizes in Section 4.1 (143 + 1,669 + 650 + 495 + 1,982 = 4,939) are consistent, so the class-total mismatch is not merely a typo in the total. The per-class counts must be reconciled and the per-split class breakdown reported, since the recall denominators in Table 2 cannot be verified otherwise.","section":"Section 2.2 and Section 4.1"},{"comment":"Section 4.3 claims that fine-tuned Qwen2.5-VL-7B 'improves HIGH-risk recall by over 60 points across all splits.' According to Table 2, the Cross-Domain improvement is 66.8 - 22.3 = 44.5 points, so the claim is not supported by the table. Please correct the sentence or provide the averaged or weighted numbers that justify it; this matters because the abstract and introduction use these recall gains as headline evidence.","section":"Section 4.3 and Table 2"},{"comment":"Section 4.1 reports that the Long-Tail split contains only 143 actions, and Table 2 reports no confidence intervals or per-class counts for any split. On 143 examples, a small number of prediction changes moves accuracy by roughly a point, and the reported differences between WebGuard-VL-7B and other models in that row are within plausible sampling variation. Please report bootstrap confidence intervals and per-class denominators, especially for the Long-Tail split, before drawing conclusions about long-tail generalization difficulty.","section":"Section 4.1 (Test splits) and Table 2"}],"minor_comments":[{"comment":"The headline numbers are internally inconsistent: the abstract reports accuracy from 37% to 80% and HIGH recall from 20% to 76%; the introduction reports 38% to 80% and recall from 20% to 'over 80%'; Section 4.3 says the average accuracy is 80.4%, while the four WebGuard-VL-7B rows in Table 2 average 83.6%. Please reconcile the exact reported values.","section":"Abstract, Section 1, Section 4.3, Table 2"},{"comment":"Section 4.2 says 'we report the average F1 score,' but Table 2 contains no F1 column; either report the F1 values or delete this sentence.","section":"Section 4.2 (Evaluation Metrics)"},{"comment":"Each model appears twice in Table 2, and the only differentiation is the caption note that text-only models are in gray. Add an explicit row descriptor, such as '(A11y)' for the text-only variant, so the table is interpretable without the caption.","section":"Table 2"},{"comment":"The label 'TestLong-Tail' should be 'Test Long-Tail', and the operational definition of the long-tail traffic distribution should be given in the main text, including the source of traffic data and the cutoff used, since the claim that these websites are underrepresented is central to that split.","section":"Section 4.1 (Test splits)"},{"comment":"Appendix G lists deployment-related limitations but does not mention the label-validity risk described in my first major comment; if the authors add the label audit, they should also state the residual label risk in the limitations section.","section":"Appendix G (Limitations)"}],"recommendation":"major_revision","confidential_remarks":"The held-out evaluation design and the public release of data and models are genuine strengths, and I am not requesting rejection. My recommendation is driven by the label-validity issue, which is standard to require for a safety benchmark, and by several internal inconsistencies in the reported numbers. If the authors provide a label audit (inter-annotator agreement and an omission-rate estimate), reconcile the class totals and headline numbers, and add confidence intervals for the small Long-Tail split, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on WebGuard. The core contribution is real: the first action-level, real-website safety dataset for web agent guardrails, 4,939 actions across 193 sites and 22 domains, with a three-tier SAFE/LOW/HIGH schema. That fills an actual gap—prior work is task-level, simulated, or mobile-only. The four generalization splits (long-tail, cross-domain, cross-website, cross-action) are thoughtful, and the finding that zero-shot frontier models land under 60% accuracy and then a fine-tuned Qwen2.5-VL-7B jumps to ~80% accuracy and ~76-86% HIGH recall is a concrete, useful result. The error analysis figures (intermediate steps in a passport application, checkout button that doesn't commit) are instructive. I'd cite this dataset if I were working on web agent safety.\n\nThe soft spots are real but addressable. The biggest is the ground-truth label validity. Section 2.2 says all remaining unannotated actions are labeled SAFE by default. That means an annotator's miss becomes a negative label, not a labeling error. Combined with no inter-annotator agreement reported, the accuracy and recall numbers are only as good as the annotators' exhaustive coverage. The three-person review catches schema inconsistencies but cannot catch a systematic blind spot about what counts as state-changing. The stress-test note is right to flag this as the load-bearing assumption. I'd want an independent re-annotation of a random sample, or at least agreement statistics on a subset, before trusting the 80% number.\n\nThere's also a small internal inconsistency: the stated class totals (1,108 + 2,284 + 1,564 = 4,956) don't match the stated dataset size (4,939). That's minor—likely a typo or a counting change—but it should be fixed. And the long-tail test split is only 143 actions, so those numbers carry wide error bars; no confidence intervals anywhere.\n\nThe evaluation itself is not circular: the held-out splits mean the fine-tuning gains are measured on actions not used for training, so the main result stands once labels are validated. The central argument holds up conditional on that.\n\nBottom line: this deserves a serious referee. The dataset is a valuable resource, the paper is clearly written, and the flaws are fixable rather than fatal. I'd send it to review with a request for label validation. Bring to reading group? yes—the risk schema and the default-SAFE issue are worth discussing.","headline":"WebGuard ships a genuinely useful action-level risk dataset for web agents, but the headline numbers rest on unvalidated human labels—worth serious refereeing, not blind acceptance.","tokens_in":17574,"tokens_out":1751,"would_cite":true,"duration_ms":501885,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebGuard claims that a few thousand human-labeled actions can train a generalizable guardrail that predicts web-agent action risk far better than zero-shot frontier models.","keywords":["web agent safety","action-level risk assessment","guardrail","three-tier risk schema","generalization benchmark","supervised fine-tuning","multimodal web agents","long-tail websites"],"falsifier":"Take a random sample of the actions that were labeled SAFE by default because no annotator recorded them, have independent annotators re-examine each page and assign a risk label using the same schema, and measure inter-annotator agreement. If a substantial fraction of default-SAFE actions are judged LOW or HIGH by independent annotators, or if Cohen's kappa on a re-annotated subset of all actions is low, the benchmark's ground truth is not stable enough to support the reported accuracy and recall claims.","tokens_in":16604,"feed_emoji":"🛡️","tokens_out":5395,"duration_ms":58079,"temperature":0.7,"pith_summary":"WebGuard is a dataset and benchmark built to answer one question before a web agent acts: if the agent clicks this element or submits this form on a live website, how much harm could follow? The paper collects 4,939 human-annotated actions from 193 real websites across 22 domains, labels each action SAFE, LOW, or HIGH under a three-tier risk schema, and splits the data to test generalization to new actions, new websites, new domains, and long-tail sites. The paper's central finding is that zero-shot frontier models are not adequate guardrails: they score under 60% accuracy and under 60% recall on HIGH-risk actions, whereas a Qwen2.5-VL-7B model fine-tuned on WebGuard reaches about 80% accuracy and 76-80% HIGH-risk recall. The claim that matters is that action-level risk prediction on real websites is learnable from a few thousand examples, while also being still too unreliable for high-stakes deployment without further progress.","feed_headline":"Fine-tuning lifts web-agent risk detection from 37% to 80%","feed_subtitle":"A 4,939-action human-labeled benchmark shows frontier models miss risky web actions; small tuned models catch far more.","key_machinery":"The load-bearing object is the three-tier risk schema (SAFE, LOW, HIGH), which turns an open-ended safety question into a concrete classification target: SAFE covers trivial, reversible, non-state-changing actions; LOW covers minor consequences that only affect the user; HIGH covers significant, irreversible, or legally and financially consequential actions that persist beyond the session. Around this schema, the paper builds an annotation pipeline that records each action with page snapshots, bounding boxes, element metadata, and URLs, and defines the guardrail task as $f(S, A, R) \\to y$: predict a risk label from webpage state, proposed action, and schema. The same schema is used to prompt frontier models and to supervise fine-tuning, making the label set and its definitions the unit that carries both the data and the model evaluation.","core_discovery":"The paper's claim is that the safety of web agents can and should be enforced at the level of individual actions, not whole tasks, and that a generalizable guardrail can be trained for that purpose. WebGuard provides the first large-scale action-level resource for this: 4,939 real-world actions annotated with a three-tier risk schema, collected from 193 live websites in 22 domains including 15 long-tail sites. The authors show that prompting frontier LLMs to reason through state understanding, outcome reasoning, and risk classification yields less than 60% accuracy and less than 60% recall on high-risk actions, and that supervised fine-tuning of Qwen2.5-VL-7B on WebGuard lifts average accuracy from roughly 37-38% to 80.4% and HIGH-risk recall from about 20% to over 76%, with a 3B variant also surpassing much larger zero-shot models. The authors present this as evidence that dedicated, fine-tuned guardrails are the right direction, and explicitly caution that even the best model remains below the near-perfect reliability that real deployment would demand.","pith_inferences":["The default-SAFE convention for unannotated actions means the dataset's true coverage of risky actions depends on how exhaustively annotators explored each page; an independent re-audit of default-SAFE actions would reveal whether risk recall is overstated.","Because the error analysis shows models rely on surface cues like button labels, the dataset could be extended with post-execution screenshots or before-and-after state diffs to train outcome prediction rather than label prediction.","The same three-tier schema and fine-tuning recipe could transfer to mobile GUIs or desktop applications, where action-level risk labeling is currently sparse.","A fourth category or a continuous risk score would let users set a safety threshold instead of choosing between LOW and HIGH, which the current three-class formulation cannot express."],"forward_implications":["Fine-tuning on WebGuard turns a 7B open-weight model into a guardrail that outperforms much larger zero-shot frontier models on accuracy and high-risk recall, so task-specific supervision is a viable path to agent safety.","Text-only accessibility-tree inputs beat screenshots in zero-shot prompting, but the pattern reverses after fine-tuning, meaning modality choice should be made jointly with training data.","The Cross-Domain split produces the smallest gains after fine-tuning, so generalizing to entirely unseen domains is the main open obstacle for deployable guardrails.","Frontier reasoning models still underestimate HIGH-risk actions under zero-shot prompting, so strong general reasoning alone does not remove the need for a dedicated risk classifier.","Even the best tuned guardrail remains short of near-perfect accuracy and recall, so high-stakes deployment should keep a human in the loop."],"supporting_citations":[{"why":"Provides the WebOlympus annotation tool used to record actions with screenshots, bounding boxes, and element metadata.","marker":"[58]"},{"why":"Supplies the Qwen2.5-VL base models that are fine-tuned into WebGuard guardrails.","marker":"[3]"},{"why":"GPT-4o is a frontier baseline whose zero-shot performance motivates the need for dedicated guardrails.","marker":"[20]"},{"why":"GuardAgent represents the prompting-plus-retrieval guardrail approach that WebGuard benchmarks against.","marker":"[47]"},{"why":"SafeArena is a simulation-based task-level safety benchmark that WebGuard contrasts with real-world action-level labels.","marker":"[43]"},{"why":"BrowserART is another simulation-based safety benchmark used in the comparison of dataset granularity and environment type.","marker":"[24]"},{"why":"MobileSafetyBench grounds the comparison of task-level labels in simulated mobile environments.","marker":"[26]"},{"why":"Provides an action-level mobile UI impact dataset, the closest prior granularity to WebGuard.","marker":"[55]"},{"why":"Set-of-Mark prompting grounds the bounding-box visual representation used in the multimodal guardrail setting.","marker":"[50]"}],"fun_headline_variants":["WebGuard: action-level guardrail lifts agent risk detection from 37% to 80%","37% to 80%: fine-tuned web agent safety net catches risky actions","Benchmark shows frontier web agents fail at risk detection; tuning fixes it","New dataset trains guardrails that spot risky web actions 80% of the time","WebGuard: fine-tuned 7B model beats frontier LLMs on action risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human labels are correct: the risk schema is defined by the authors, annotators were asked to exhaustively label state-changing actions, any action not annotated is labeled SAFE by default, and no inter-annotator agreement is reported. If annotators missed or disagreed on risky actions, the reported accuracy and recall numbers do not measure what they appear to measure.","fun_headline_variants_meta":{"raw":{"variants":["WebGuard: action-level guardrail lifts agent risk detection from 37% to 80%","37% to 80%: fine-tuned web agent safety net catches risky actions","Benchmark shows frontier web agents fail at risk detection; tuning fixes it","New dataset trains guardrails that spot risky web actions 80% of the time","WebGuard: fine-tuned 7B model beats frontier LLMs on action risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4217,"prompt_tokens":1061,"completion_tokens":3156,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":3061}},"tokens_in":677,"tokens_out":3156,"duration_ms":25093,"temperature":1.0,"reasoning_tokens":3061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:59:33.889624+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the actions that were labeled SAFE by default because no annotator recorded them, have independent annotators re-examine each page and assign a risk label using the same schema, and measure inter-annotator agreement. If a substantial fraction of default-SAFE actions are judged LOW or HIGH by independent annotators, or if Cohen's kappa on a re-annotated subset of all actions is low, the benchmark's ground truth is not stable enough to support the reported accuracy and recall claims.","supporting_citations":[{"cited_title":"WebOlympus: An open platform for web agents on live websites","cited_arxiv_id":null,"evidence_quote":"Provides the WebOlympus annotation tool used to record actions with screenshots, bounding boxes, and element metadata."},{"cited_title":"Hendryx, Summer Yue, and Zifan Wang","cited_arxiv_id":null,"evidence_quote":"BrowserART is another simulation-based safety benchmark used in the comparison of dataset granularity and environment type."},{"cited_title":"From interaction to impact: Towards safer ai agent through understanding and evaluating mobile ui operation impacts","cited_arxiv_id":null,"evidence_quote":"Provides an action-level mobile UI impact dataset, the closest prior granularity to WebGuard."}],"review_version":1}