{"id":"4b87843a-027a-4339-bf5d-0562ee825ef5","arxiv_id":"2507.20503","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Conditioning a multimodal guardrail on retrieved 'precedent' reasoning traces, rather than static policy definitions, improves few-shot and novel-policy content-moderation F1 scores on UnsafeBench.","lead":"This paper trains a vision-language model to moderate images by referencing past examples (precedents) of similar content and the reasoning behind their safety labels, rather than relying on fixed written policies. The approach improves scores on an 11-policy safety benchmark, especially when only a few examples per policy are available, which matters for companies enforcing evolving content rules.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval policy-match rate is never measured; the reported gains may stem from accidentally retrieving the right policy rather than from precedent-based reasoning.","rationale":"The reader's weakest assumption identifies exactly the condition I find most load-bearing: retrieval must return a precedent whose policy and rationale are relevant to the test image, or the model is conditioned on the wrong criteria. The paper's own acknowledgment that irrelevant context degrades performance (citing Shi et al., 2023) makes this more than a stylistic concern; it is a potential failure mode of the central mechanism. The existing ablation in Table 4 shows that thresholding helps, but it does not establish that the surviving retrieved precedents are policy-correct, nor does it separate the contribution of correct-policy identification from the contribution of the precedent's rationale. Without a policy-match-rate measurement, the headline few-shot and leave-one-out gains could be driven by retrieval acting as a policy classifier rather than by the precedent-based reasoning the paper advocates. This does not invalidate the method; the qualitative examples and consistent gains across settings are real evidence that precedents can help. But the mechanism is underspecified at its most critical juncture, and one targeted measurement would settle it. Because the reader already assigned a CONDITIONAL verdict and this concern is the same one they flagged, no verdict change is needed; the paper should be asked to supply this evidence before full acceptance.","tokens_in":17678,"tokens_out":5124,"duration_ms":59461,"concrete_test":"Re-run the Sec. 3.2 retrieval on the UnsafeBench validation split and report the top-1 policy-match rate: the fraction of test images whose retrieved precedent (CLIP ViT-L/14 image embedding, threshold 0.8) has the same policy as the test image. Then compare three inference conditions on the same test set: (a) the paper's actual retrieval, (b) oracle retrieval (precedent guaranteed to be from the same policy), and (c) the correct policy definition alone, without any precedent. If (b) substantially exceeds (a), retrieval quality is the bottleneck; if (c) matches (a), the precedent's rationale is not adding signal beyond policy identification. Either outcome would determine whether the headline F1 gains are attributable to precedent-based reasoning or to the retrieval step accidentally supplying the correct policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that conditioning a guardrail on retrieved precedents, rather than on fixed policy definitions, is what drives the reported F1 improvements. The load-bearing step is the retrieval mechanism in Sec. 3.2: under the paper's no-policy evaluation protocol, the retrieved precedent's policy and rationale are the only channel by which the model learns which policy applies to the test image. Retrieval is based on CLIP image similarity, not on policy relevance, and the paper never measures how often the top-1 precedent (after the 0.8 threshold in Table 4) actually belongs to the same policy as the test image. If retrieval frequently returns an off-policy precedent, the distraction finding the paper itself cites (Shi et al., 2023) implies the model is actively pushed toward the wrong decision. The F1 advantage could then reflect a better-than-chance policy match rate rather than a robust precedent-based mechanism. The leave-one-out result is especially vulnerable: with a held-out policy, the most visually similar images are likely the few-shot examples of that same held-out policy, so the reported 16.7% gain may reduce to near-duplicate retrieval from the adaptation set, not generalization to a genuinely novel policy. Table 4 is not sufficient to resolve this because it reports end-task F1 only, which conflates retrieval errors with prompt-format and threshold effects, and it does not report how many retrieved precedents are actually policy-relevant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new paradigm for customizable multimodal RAI guardrails: instead of conditioning a vision-language model on fixed policy definitions, it conditions predictions on 'precedents,' which are structured tuples (image, caption, policy-violation label, rationale, policy) collected from the model itself via a critique-revise cycle. At training time, the model is LoRA fine-tuned on revised captions and labels ('reflective fine-tuning'); at inference time, a CLIP-based retriever selects the most relevant precedent and its policy/rationale is inserted into the prompt. Experiments on UnsafeBench with 11 policies compare the method against CLIP-based classifiers, proprietary models with ICL, and fine-tuned LLaVA baselines under few-shot (16 images/policy), larger-data (160/640 images/policy), and leave-one-out novel-policy protocols. The authors report consistent F1 gains—for example, 0.688 vs. 0.622 for fine-tuned LLaVA in the few-shot setting, and a 16.7% F1 improvement in the leave-one-out setting—and include ablations of retrieval configuration, critique-revise coverage, and random-vs-relevant precedents.","tokens_in":17953,"tokens_out":4778,"duration_ms":49535,"significance":"If the empirical claims hold, the paper makes a useful contribution to customizable multimodal content moderation: the precedent formulation is simple and interpretable, the critique-revise mechanism avoids human annotation, and the evaluation spans several protocols and model families. The code release, the random-precedent control (Table 7), and the component ablations (Table 4) are clear strengths. However, the central mechanism—retrieval of policy-relevant precedents—is never directly measured, and the headline gains are point estimates without variance or significance testing. These gaps currently prevent the paper from fully establishing its generalization claims. The approach is not circular: precedents are constructed from labeled training data and performance is measured on held-out test splits.","major_comments":[{"comment":"The central claim is that retrieved precedents, rather than fixed policies, drive the improvements. Under the paper's no-policy evaluation protocol, the retrieved precedent's policy and rationale are the only channel by which the model learns which policy applies to the test image. Yet the paper never measures how often the retrieved precedent is policy-relevant. Table 4 reports end-task F1 only, which conflates retrieval quality with prompt-format and threshold effects; for example, rows 7 and 8 differ in both retrieval subject and threshold. Please report the fraction of retrieved precedents whose policy matches the ground-truth policy of the test image, per category and as a function of the similarity threshold, and show that the F1 gain is not driven solely by better-than-chance policy matching. This is load-bearing because the paper itself cites Shi et al. (2023) to argue that irrelevant context can degrade performance.","section":"Section 3.2 / Table 4"},{"comment":"The leave-one-out (LOO) protocol may overstate generalization to novel policies if the precedent database includes the 16 few-shot adaptation images of the held-out policy. Since retrieval is based on CLIP image similarity, test images from the novel category are likely to retrieve near-duplicates from that same adaptation set, which would make the reported 16.7% F1 gain reflect instance-level memorization rather than generalization to a genuinely new policy. Please clarify whether the precedent database and the retrieval index exclude all training/adaptation images of the held-out category during each LOO fold; if they do not, quantify the overlap (e.g., duplicate or near-duplicate rate and retrieval policy-match rate) and, if possible, evaluate a variant where the held-out policy's few-shot images are withheld from the database.","section":"Section 4.4 / Table 2"},{"comment":"All quantitative results are point estimates: no standard deviations, confidence intervals, or significance tests are reported. This is especially important for the headline LOO improvement in Table 2, which is the average of 11 single runs, and for the few-shot comparisons, where results can be highly sensitive to which 16 images are sampled. Please report results over multiple random seeds or bootstrap resamples, and indicate statistical significance for the main claims (few-shot F1, the 6.6% and 6.8% improvements in Section 1, and the 16.7% LOO improvement).","section":"Tables 1-3 and Figure 4"}],"minor_comments":[{"comment":"The introduction reports 6.6% and 6.8% F1 improvements for few-shot and full-dataset settings, but Table 1 shows only the few-shot setting; the full-dataset comparison appears only as a curve in Figure 4 without numeric overall F1 values. Please report the full-dataset numbers explicitly.","section":"Section 1 / Table 1 / Figure 4"},{"comment":"The notation 'N-shot data points' should be defined precisely; the reader must infer from Section 4.1 that N=16 in the main protocol and N=160/640 in the scaling experiments.","section":"Section 3.2"},{"comment":"The column headers 'Subject Th Policy Rationale' are hard to parse; define 'Th' as the similarity threshold and clarify what the checkmarks indicate.","section":"Table 4"},{"comment":"The text says LLaVA improves 'by8%' and 'by 20%' on Public and Personal Health and Spam, but Table 1 shows improvements of roughly 8 and 20 percentage points; use 'points' or 'percentage points' rather than percent.","section":"Section 4.3"},{"comment":"The figure caption labels '(a) Precedent Construction' and '(b) Precedent Utilization' but the panels in the figure are labeled 'Precedent Construction' and 'Precedent Utilization'; align the caption with the figure, and note that 'LLaV A' is inconsistently spaced throughout.","section":"Figure 2 / Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The empirical claims are credible and the paper has clear strengths (code release, multiple evaluation protocols, and a thoughtful ablation of the random-precedent control), but the retrieval-policy-match-rate gap is load-bearing for the central 'precedents instead of policies' claim. The fix—measuring retrieval quality and clarifying the LOO protocol—is within the scope of a revision, so I would not reject. The reviewer's stress-test note aligns with my reading; I would ask the authors to address that concern head-on before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is clean: instead of conditioning a multimodal guardrail on a fixed policy string, retrieve a \"precedent\"—a similar image's policy, caption, and rationale—and make the prediction conditional on that. On UnsafeBench, this consistently beats ICL with all policies and fine-tuning on fixed policies, including a 16.7-point F1 gain in the leave-one-out novel-policy setting. The formulation itself, plus the critique-revise collection mechanism that recovers initially misclassified examples, is a genuine step forward for the guardrail-customization problem.\n\nWhat the paper does well: the evaluation adopts a realistic policy-agnostic protocol, which is a real improvement over UnsafeBench's original setup. The ablations in Table 4 show that both the policy definition and the rationale in the precedent matter, and that image retrieval beats text retrieval. Table 7 shows that randomly sampled few-shot examples actually hurt, so retrieval quality is doing real work. Code is released, and the limitations section is honest about the dependence on the base VLM's capabilities.\n\nThe soft spots are mostly about evidence quality. There are no error bars or significance tests anywhere; every conclusion rests on point estimates. The LOO protocol is run once per category, so the headline 16.7% is an average of 11 single runs with no variance. The stress-test concern about retrieval is valid in spirit: the paper never measures how often the retrieved precedent actually belongs to the same policy as the test image. If it doesn't, the prompt can actively mislead the model, as the paper itself notes via the Shi et al. citation. That said, the indirect evidence is fairly strong: the threshold-based image retrieval in Table 4 and the random-example comparison in Table 7 both suggest that relevant retrieval is necessary. The authors just haven't pinned down whether the gain comes from correct policy selection, reasoning quality, or both. I would ask them to report retrieval policy-match rate as a function of threshold, ideally per category.\n\nOne smaller issue: the \"full-dataset\" claim in the abstract (6.8% improvement) is backed only by a figure, not by a table with the actual numbers in the main text. And the \"best template\" reporting for baselines is a bit loose, though not disqualifying.\n\nThe central claim holds up. This is a useful method paper for people building customizable multimodal guardrails, not a paradigm shift. It deserves a serious referee, and I would accept it after the authors add error bars or significance tests and directly measure retrieval policy-match rate.","headline":"A solid method paper for customizable multimodal guardrails: conditioning on retrieved precedents beats fixed policy definitions, but the retrieval mechanism's policy-match rate is never directly measured.","tokens_in":18474,"tokens_out":2387,"would_cite":true,"duration_ms":27448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-modal safety guardrail that judges images by retrieved past rulings beats fixed-policy guardrails, especially on novel policies.","keywords":["multi-modal guardrails","responsible AI","precedent-based prediction","critique-revise mechanism","retrieval-augmented generation","few-shot policy customization","vision-language models","content moderation"],"falsifier":"Corrupt the retrieval channel by replacing the nearest-neighbour precedent with a randomly chosen precedent from a different policy while keeping everything else fixed, and measure the F1 drop on UnsafeBench; the paper's mechanism predicts a large drop, whereas a small drop would show that most of the gain comes from the critique-revise fine-tuning rather than from the retrieved precedent itself.","tokens_in":17499,"feed_emoji":"🛡️","tokens_out":8908,"duration_ms":80153,"temperature":0.7,"pith_summary":"Custom RAI guardrails have to filter images according to user-defined policies that are often new, culturally specific, and short on labeled examples. This paper argues that instead of conditioning the guardrail on fixed policy definitions, the model should judge each image by analogy to retrieved precedents: structured records of how a similar past image was judged, including its caption, policy-violation label, rationale, and the policy it was assessed under. The authors build such precedents with a fully automatic critique-revise loop and show that retrieving the nearest precedent improves both fine-tuned and fine-tuning-free vision-language models. On UnsafeBench's 11 policies, the approach reports 6.6-point F1 gains in the few-shot setting, 6.8 points at full data scale, and 16.7 points when a policy is held out and treated as novel. If the claim holds, guardrail customization becomes a matter of supplying a few examples and retrieving well, rather than retraining or rewriting rules for every new policy.","feed_headline":"Past-image precedents lift novel-policy guardrail F1 by 16.7 points","feed_subtitle":"Similar past rulings let image guardrails adapt to new policies from just 16 examples per policy, beating fixed rules.","key_machinery":"The central object is the precedent database. Each precedent is a structured reasoning trace—image, revised caption, binary policy-violation label, rationale, and the policy under which the judgment was made—collected by a critique-revise mechanism: the VLM first captions and judges an image, critiques its own caption for missed or misread details, revises it, and re-judges; only cases whose revised prediction matches the ground-truth label enter the database. At training time the database feeds reflective fine-tuning, where the model is supervised to reproduce the revised caption and label. At inference time the nearest precedent is retrieved with CLIP image embeddings (ViT-L/14) and its policy and rationale are inserted into the prompt, so the model is asked to decide by analogy to that similar case.","core_discovery":"The central claim is that a vision-language guardrail's judgment should be conditioned on precedents rather than on the policy text itself. A precedent is a structured tuple of image, revised caption, binary violation label, rationale, and policy, and the paper's discovery is that such traces can be produced automatically through a critique-revise mechanism that recovers initially misclassified examples. With only 16 labeled images per policy, the method reaches an overall F1 of 0.688, against 0.622 for the fixed-policy LoRA baseline LlavaGuard and 0.584 for GPT-4o with in-context learning; for a held-out novel policy the method reaches 0.659 F1 versus 0.492 for LlavaGuard. The paper also reports that precedent-based conditioning degrades far less than policy-based conditioning when the number of policies grows from 1 to 11.","pith_inferences":["The paper's design implicitly turns policy customization into a retrieval-quality problem: if users build the database from a few examples, then the system's ceiling is set by how well the similarity metric finds the right precedent, not by how well the model knows the policy text.","A testable extension is to make retrieval policy-aware or rationale-aware—for example, weighting image similarity by the precedent's policy or by the rationale's relevance—since the paper only ablates image versus caption embeddings and a similarity threshold.","Because the retrieved precedent's policy and rationale are injected verbatim into the prompt, the precedent database becomes an attack surface: a poisoned or mislabeled precedent could steer a borderline image toward either a false alarm or a miss. The paper does not test adversarial manipulations of the database.","The same precedent structure could be carried to other modalities, such as video or audio, by swapping the retrieval encoder while keeping the critique-revise collection loop; the authors list cross-modality extension as future work, but the mechanism itself is modality-agnostic."],"forward_implications":["With only 16 labeled images per policy, adding retrieved precedents to the prompt lifts fine-tuning-free GPT-4o from 0.584 to 0.726 F1 and Claude-3 Sonnet from 0.569 to 0.691 F1.","Precedent-based reflective fine-tuning plus retrieval (Re-FT + RAG) reaches 0.688 F1 in the few-shot setting, surpassing fixed-policy LoRA fine-tuning (LlavaGuard, 0.622) and the CLIP-based classifiers MultiHeaded (0.507) and Q16 (0.533).","In the leave-one-out simulation of novel policies, the precedent method averages 0.659 F1 versus 0.492 for LlavaGuard, a 16.7-point gain, indicating it can adapt to unseen policies from a few examples.","Scaling from 1 to 11 policies costs the precedent method only about 0.041–0.045 F1, while ICL and SFT baselines lose 0.086–0.128, so precedent-based conditioning scales better as policy counts grow.","At the full data scale (640 images per policy), the local LLaVA-based precedent model outperforms the larger proprietary GPT-4 and Claude-3 models, suggesting competitive guardrails can be built without proprietary APIs."],"supporting_citations":[{"why":"It supplies UnsafeBench, the 11-policy dataset and labels used for all training, evaluation, and policy definitions.","marker":"Qu et al., 2024"},{"why":"It provides LlavaGuard, the fixed-policy LoRA fine-tuning baseline that the precedent methods are compared against and surpass.","marker":"Helff et al., 2024"},{"why":"It shows that irrelevant in-context context degrades model performance, motivating retrieval of only the most relevant precedent.","marker":"Shi et al., 2023"},{"why":"It gives CLIP, whose ViT-L/14 image encoder is the retrieval mechanism for selecting the most relevant precedent.","marker":"Radford et al., 2021"},{"why":"It provides LLaVA-1.5 13B, the vision-language backbone used for reflective fine-tuning, ICL, and RAG inference.","marker":"Liu et al., 2024"},{"why":"It provides MultiHeaded, a CLIP linear-probing baseline used in few-shot and full-data comparisons.","marker":"Qu et al., 2023"},{"why":"It provides Q16, a CLIP prompt-tuning baseline whose results the precedent method must exceed.","marker":"Schramowski et al., 2022"},{"why":"It provides Contriever, the text retriever used in the ablation comparing image-based versus text-based precedent retrieval.","marker":"Izacard et al., 2021"},{"why":"It documents GPT-4o, the proprietary model used to show that precedent-based RAG improves fine-tuning-free guardrails.","marker":"Achiam et al., 2023"},{"why":"It documents Claude-3 Sonnet, the proprietary model evaluated with ICL and with precedent-based RAG.","marker":"Anthropic, 2023"}],"fun_headline_variants":["Precedent-based guardrails beat fixed policies by 16.7 F1","16 examples per policy: precedents lift novel-policy F1 by 16.7","Novel-policy guardrail F1 up 16.7 with precedent conditioning","Guardrails adapt via precedents, not policy text: 16.7 F1 gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nearest retrieved image is the most informative precedent for judging the current image under the correct policy; if the nearest neighbour is irrelevant, the prompt conditions the model on the wrong policy and rationale, and the paper does not measure how often that happens.","fun_headline_variants_meta":{"raw":{"variants":["Precedent-based guardrails beat fixed policies by 16.7 F1","16 examples per policy: precedents lift novel-policy F1 by 16.7","Novel-policy guardrail F1 up 16.7 with precedent conditioning","Guardrails adapt via precedents, not policy text: 16.7 F1 gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001177,"raw_usage":{"total_tokens":4872,"prompt_tokens":957,"completion_tokens":3915,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":3825}},"tokens_in":573,"tokens_out":3915,"duration_ms":28617,"temperature":1.0,"reasoning_tokens":3825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:42:07.526797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Corrupt the retrieval channel by replacing the nearest-neighbour precedent with a randomly chosen precedent from a different policy while keeping everything else fixed, and measure the F1 drop on UnsafeBench; the paper's mechanism predicts a large drop, whereas a small drop would show that most of the gain comes from the critique-revise fine-tuning rather than from the retrieved precedent itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides Q16, a CLIP prompt-tuning baseline whose results the precedent method must exceed."}],"review_version":1}