{"id":"03bb2fb7-524f-4214-909c-4ce2ec79a874","arxiv_id":"2607.05910","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PolicyShiftGuard, trained with randomized policy SFT plus matched pass/block boundary pairs, reaches 76.9 Avg. F1 and 72.1 Avg. PSS on a new policy-shift image-guardrail benchmark where existing VLMs stay brittle.","lead":"This paper shows that image safety models fail when the same picture must be allowed under one product rule and blocked under another, and introduces a benchmark plus a compact training recipe that forces models to flip decisions with the policy. Product teams and safety researchers should care because real deployments change rules by age group, region, and brand, not by a single fixed unsafe taxonomy.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"SOTA claim is load-bearing on a small, rule-defined image set whose labels may not match real policies.","rationale":"The paper’s internal story is coherent: fixed-taxonomy models can get F1 without policy flips; matched pass/block pairs (Table 6) and concise no-think supervision drive large PSS gains; latency Pareto is real. The load-bearing risk is not an internal contradiction but external validity of the evaluation target. With only 265 images and deterministic labels from author rules, the SOTA numbers can overstate real policy-adaptive competence. The reader already flagged this as the weakest assumption; I agree it is the single most important concern and does not justify moving off CONDITIONAL. A human re-label check on flip pairs would settle whether the concern lands without requiring a full re-benchmark.","tokens_in":44827,"tokens_out":525,"duration_ms":6006,"concrete_test":"Independently re-label a stratified 100-instance subset of Adaptive+Shift (covering all 7 categories and both pass/block sides of flip pairs) with 2–3 human moderators given only the natural-language policy text and image—no attribute schema or rule engine. Recompute F1 and PSS for PolicyShiftGuard-7B and the top two baselines under these human labels. If Avg. PSS drops >10 points or the ranking vs. Gemini/GPT reverses, the SOTA claim is not externally valid.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Table 2: PolicyShiftGuard-7B 76.9 Avg. F1 / 72.1 Avg. PSS SOTA) is measured almost entirely on PolicyShiftBench: 2,000 instances over only 265 unique images, with labels produced by three-VLM majority-voted attributes plus the authors’ hand-written executable rules (Sec. 2.2–2.3; App. B–C). PSS further groups by same image and category, so a few boundary images can dominate the headline metric. Human audits give 88%/90% blind accuracy and 95% qualified rates, but that only checks consistency with the same rule catalog, not whether attributes or rules match real product policies. If the rule engine systematically mis-specifies exemptions (medical, news, cultural, OCR, etc.), both the benchmark and BP-Adapt optimize the wrong target; transfer gains on UnSafeBench/SafeEditBench are weaker and do not fully rescue the policy-shift SOTA claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that image safety is policy-dependent rather than an intrinsic image property, and that current VLMs and specialized guardrails fail under policy shifts. It introduces PolicyShiftBench (2,000 policy-discriminative instances over 265 images, 28 policy variants, Adaptive vs held-out Shift splits) and Policy Shift Score (PSS), which scores same-image pass/block flips. It then proposes PolicyShiftGuard (3B/7B), trained with Randomized Policy SFT followed by Boundary-Pair Policy Adaptation that combines CE with a pairwise margin separating matched pass/block policies. On PolicyShiftBench the 7B model reports 76.9 Avg. F1 and 72.1 Avg. PSS (Table 2), with transfer gains on UnSafeBench and SafeEditBench (Table 3), a favorable latency–performance trade-off (Figure 3), and ablations isolating randomization, no-think supervision, and the pair loss (Tables 4–6; Figure 5).","tokens_in":45126,"tokens_out":883,"duration_ms":16808,"significance":"Policy-adaptive visual guardrailing is a genuine deployment gap: product, age, region, and institutional policies routinely re-label the same image. The paper’s main contributions are concrete and useful: (i) a compositional, scenario-grounded benchmark with executable rules over atomic attributes; (ii) PSS, which cleanly separates ordinary unsafe recognition from policy-flip sensitivity; (iii) a compact two-stage recipe whose pair-loss ablation is large and interpretable (Table 6: +17.7 Avg. F1 for 7B when L_pair is added). Strengths include image-disjoint train/eval construction, Adaptive vs held-out Shift policies, multi-model baselines including closed-source systems, transfer evaluation, latency reporting, and rerun variation in Appendix E.2. If the benchmark and labels are accepted as a reasonable proxy for runtime policy bundles, the result is a clear advance over fixed-taxonomy image guardrails and a practical training recipe for compact policy-conditioned models.","major_comments":[{"comment":"Table 2 / §2.1: The headline SOTA (76.9 Avg. F1, 72.1 Avg. PSS) is measured on only 265 unique images (2,000 instances). PSS further macro-averages over same-image, same-category flip groups (§E.1), so a modest number of boundary images can dominate the metric. The manuscript should report image-level or group-level variance (e.g., bootstrap over images/groups) and discuss how sensitive Avg. PSS is to removing high-leverage images; without this, the SOTA claim is harder to interpret than ordinary F1 on large fixed-taxonomy sets.","section":null},{"comment":"§2.2–2.3 and Appendix B–C: Ground-truth labels are produced by three-VLM majority-voted attributes plus author-written executable rules. Human audits (88%/90% blind accuracy; 95% qualified rate) check consistency with that same rule catalog, not external product/legal policies. This is load-bearing for both the benchmark and BP-Adapt: if attributes or exemptions (medical, news, cultural, OCR, etc.) systematically mis-specify real boundaries, models optimize the authors’ rule engine. The paper should more explicitly scope claims as ‘policy-rule following under this catalog,’ add failure analysis of attribute errors that flip labels, and avoid overstating fidelity to real deployment policies without external policy-expert validation.","section":null},{"comment":"Table 2, Adaptive vs Shift: PolicyShiftGuard-7B remains strong on Shift PSS (70.4) but drops sharply in Shift F1 (86.8 → 67.0), while Gemini-3-Flash-Preview leads Shift F1 (74.2). The average SOTA therefore mixes in-distribution policy following with held-out generalization. The main claim should more carefully separate ‘best average policy-adaptive score on this bench’ from ‘best held-out policy generalization,’ and discuss why F1 degrades more than PSS on Shift.","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: most image guardrails still act like fixed unsafe detectors, and this paper shows that with a same-image flip metric (PSS) and matched pass/block training. That framing is right for product moderation, and the empirical gap is real—plenty of models post okay F1 with near-zero PSS.\n\nWhat is actually new is the package, not the slogan. PolicyShiftBench pairs 265 images with multi-policy prompts (2k eval instances), compositional bundles over seven risk categories, Adaptive vs held-out Shift policies, and labels from voted attributes plus executable rules. PolicyShiftGuard then does RP-SFT plus BP-Adapt with an explicit pair margin. Ablations land: pair loss is load-bearing (+10–18 Avg F1), concise no-think beats think/RL for the deployed decision, and the 7B model also improves latency. Transfer to UnSafeBench/SafeEditBench is positive though not as dramatic as the home-bench SOTA. Code/data release links and human audit numbers (88/90 blind, 95% qualified) are present; citation pattern is fair to LlavaGuard, SafeEdit/SafeGuard-VL, and related work.\n\nSoft spots, in proportion. The stress note is directionally right but not a collapse: 265 unique images is thin for a headline SOTA, and PSS is group-macro over same-image flips, so boundary images matter a lot. Labels are auditable and not circular with the trained model, but they still encode the authors’ attribute schema and rule catalog; human audit checks consistency with that catalog more than external product truth. Free hyperparameters (loss weights, margin, LRs) exist, though the weight sweep looks stable. Concurrent policy-conditioned work already sits nearby, so novelty is the flip metric + boundary-pair recipe, not inventing policy text conditioning.\n\nWho this is for: multimodal safety and moderation people who care about product-specific rules, not pure vision theory. Math is ordinary CE + margin losses; data pipeline is the real artifact. I would send it to peer review. Engage if you work on guardrails; skim if you only need the PSS idea.","headline":"Solid systems paper: same-image policy flips are the real contribution; SOTA is real on their bench but rests on a small, rule-labeled image set.","tokens_in":45748,"tokens_out":529,"would_cite":true,"duration_ms":8428,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Image safety is not a property of the image: guardrails must flip decisions when the runtime policy changes.","keywords":["image guardrails","policy-adaptive safety","vision-language models","PolicyShiftBench","boundary-pair training","content moderation","policy shift score"],"falsifier":"Hold out a new set of real product policies never seen in training or the benchmark catalog; if PolicyShiftGuard’s Policy Shift Score on same-image pass/block flips falls back toward the near-zero scores of ordinary guardrails, the claimed adaptation fails.","tokens_in":45709,"feed_emoji":"🛡️","tokens_out":686,"duration_ms":8270,"temperature":0.7,"pith_summary":"Most image guardrails treat safety as if it were baked into the picture. In real products the same photo can be fine for one audience, blocked for another, and newly disallowed when a rule is rewritten. This paper argues that a useful guardrail must read the currently supplied policy, bind it to fine-grained visual evidence, and revise its verdict when the boundary moves. To measure that skill it builds PolicyShiftBench: 2,000 policy-discriminative cases over 265 images, each image judged under multiple policies so the same visual evidence must sometimes pass and sometimes block. It then trains PolicyShiftGuard with a two-stage recipe: first randomized policy supervised fine-tuning so the model actually follows full policy bundles, then Boundary-Pair Policy Adaptation that pairs the identical image under a blocking policy and a passing policy and forces the unsafe score to separate. The resulting 7B model reaches 76.9 average F1 and 72.1 average Policy Shift Score, far above ordinary vision-language models and specialized guardrails that often detect risky cues yet almost never flip with the policy, while also running faster via a five-token decision format.","feed_headline":"Same image, new policy: guardrails that actually flip","feed_subtitle":"A 7B model hits 72 Policy Shift Score by training matched pass/block pairs, not fixed unsafe tags","key_machinery":"Boundary-Pair Policy Adaptation (BP-Adapt): for a fixed image and risk category, train one policy that must block and one that must pass, combining ordinary label losses with a pairwise margin that makes the blocking policy receive a higher unsafe score than the passing policy under identical visual evidence.","core_discovery":"Existing vision-language models and specialized image guardrails largely fail at policy adaptation: they can score nontrivial F1 on unsafe content yet collapse on Policy Shift Score, showing they rely on image-level priors rather than the active rule. A compact policy-conditioned model trained with randomized policy SFT plus matched pass/block boundary-pair adaptation closes most of that gap, reaching state-of-the-art 76.9 Avg. F1 and 72.1 Avg. PSS on PolicyShiftBench and transferring to external safety benchmarks.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Same image, shifting rules: policy-conditioned guardrails that adapt","Matched pass/block pairs lift Policy Shift Score to 72.1","Image guardrails fail policy flips; RP-SFT + BP-Adapt closes the gap","PolicyShiftGuard: 7B model reads the active rule, not fixed unsafe tags","Fixed priors collapse on PolicyShiftBench; boundary pairs restore adaptation"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"Ground-truth labels are treated as correct once three vision models majority-vote atomic image attributes and the authors’ hand-written policy rules are executed on those attributes.","fun_headline_variants_meta":{"raw":{"variants":["Same image, shifting rules: policy-conditioned guardrails that adapt","Matched pass/block pairs lift Policy Shift Score to 72.1","Image guardrails fail policy flips; RP-SFT + BP-Adapt closes the gap","PolicyShiftGuard: 7B model reads the active rule, not fixed unsafe tags","Fixed priors collapse on PolicyShiftBench; boundary pairs restore adaptation"]},"model":"grok-4.5","effort":"low","cost_usd":0.004662,"raw_usage":{"total_tokens":1395,"prompt_tokens":879,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":46620000,"prompt_tokens_details":{"text_tokens":879,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":433,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":879,"tokens_out":83,"duration_ms":5121,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T01:48:09.309600+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold out a new set of real product policies never seen in training or the benchmark catalog; if PolicyShiftGuard’s Policy Shift Score on same-image pass/block flips falls back toward the near-zero scores of ordinary guardrails, the claimed adaptation fails.","supporting_citations":[],"review_version":2}