{"id":"4e3b8e81-309b-4a2f-9797-925f516f5db6","arxiv_id":"2501.00192","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A zero-shot MLLM pipeline that objectifies safety rules, decomposes them into preconditions, and uses debiased token probabilities plus cascaded reasoning achieves 94.8% accuracy on a synthetic image safety benchmark.","lead":"A new pipeline called CLUE lets multimodal AI models judge whether images are safe by reading written safety rules, without needing human-labeled training data. It breaks rules into simpler checks, filters irrelevant ones, and uses token probabilities and step-by-step reasoning to decide, reporting large accuracy gains on a synthetic benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OS Bench is generated from the same LLM-objectified rules CLUE is designed to apply, so the headline 95.9% recall may reflect benchmark-specific artifacts rather than generalizable zero-shot safety judgment; the paper provides no external or human-validated evaluation.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation is self-referential because OS Bench is generated from the same objectified rules that CLUE is designed to apply. My reading confirms this is the central threat to the paper's claim that CLUE is 'highly effective for zero-shot image safety judgment tasks.' The method's internal ablations are informative and the components are individually plausible, but the headline numbers are only as valid as the benchmark. The circularity is concrete: the rules in Table 1 were revised by an LLM to maximize an LLM-assessed 'objectiveness' score, and the unsafe/safe images in OS Bench were generated from prompts derived from those same rules (Appendix B, Table 8). This means the test set rewards exactly the kind of surface-level correspondence that the relevance scanning and token-probability stages are built to exploit. The paper's own motivating scenario is a user who 'may need to frequently update safety rules,' which makes generalization to a new constitution the key claim; the paper never tests that. A held-out-rule and external-image test, or even a human-labeled subset, would settle whether the method generalizes or merely fits the benchmark distribution. Because the reader already conditioned acceptance on external validation, my verdict is UNCHANGED rather than a move to REJECT: the method is not shown to be invalid, only insufficiently validated.","tokens_in":19542,"tokens_out":4238,"duration_ms":47230,"concrete_test":"Run CLUE unchanged on a held-out evaluation built from a different source: (a) hold out 4 of the 14 rules from ALL stages including objectification threshold selection, precondition extraction, and hyperparameter tuning, then generate unsafe/borderline-safe images for those held-out rules with a different text-to-image model and no manual filtering; and (b) on a 200-image random subset of OS Bench, obtain human labels for safety under the Table 1 constitution. If the F1 score drops by more than 5 points on the held-out rules or human agreement is substantially below the reported accuracy, the central claim is benchmark-specific rather than a general zero-shot safety judgment capability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 constructs OS Bench by generating unsafe images from prompts derived from the objectified constitution (Table 1) that CLUE is specifically engineered to apply, and safe images from near-miss prompts. The objectified constitution itself was produced by an LLM whose 'objectiveness score' is self-assessed via the same LLM-judge prompt (Figure 9). Hence the evaluation is circular at two levels: (i) the rule set is LLM-generated and self-validated, and (ii) the test images are generated from that same rule set. The reported 95.9% recall / 94.8% accuracy therefore measures how well CLUE recognizes text-to-image renderings of its own rule vocabulary (e.g., 'throat slit', 'legs spread >90 degrees') rather than whether it can judge safety under a new constitution — which is the paper's motivating scenario. The paper provides no human-verified labels, no external benchmark, and no held-out constitution. Even the score distributions in Figure 6 and the objectification comparison in Table 5 are computed on OS Bench, so they cannot rule out that the method exploits generation artifacts; for instance, the CLIP relevance filter may already separate unsafe from safe by image-rule similarity. The absence of released code and data compounds this, as the benchmark cannot be independently inspected or stress-tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLUE, a multi-stage zero-shot pipeline for image safety judgment under a textual safety constitution. CLUE objectifies subjective rules via LLM-based rewriting, filters irrelevant rules using CLIP similarity, decomposes rules into precondition chains, makes fast judgments from debiased token probabilities (contrasting image-bearing vs. image-free queries, and whole vs. centric-region-removed images), and falls back to cascaded chain-of-thought reasoning when confidence is low. The authors construct a new benchmark, OS Bench, by generating text-to-image diffusion samples from seed prompts derived from the same objectified rules CLUE is designed to apply, plus near-miss safe prompts. On OS Bench, CLUE with InternVL2-76B achieves 95.9% recall, 94.8% accuracy, and 0.949 F1, substantially outperforming naive zero-shot baselines and off-the-shelf fine-tuned safety classifiers. The paper includes ablations of each module, an efficiency analysis, and rule-level result breakdowns.","tokens_in":19913,"tokens_out":5046,"duration_ms":50639,"significance":"If the reported performance were independent of the benchmark construction, CLUE would be a meaningful step toward label-free, constitution-driven image safety judgment: it directly addresses known failure modes of naive MLLM prompting (subjective rules, long rule lists, token-probability biases) and its modular design is sensible. The paper also gives credit for exposing concrete bias phenomena and proposing practical debiasing strategies. However, the current evidence is not yet convincing. The evaluation is closed-loop: the test images are generated from prompts that instantiate the exact objectified rules the method applies, the ground-truth labels are assigned by construction rather than by independent human annotation, and the objectification quality is self-assessed by an LLM judge of the same kind the method replaces. No external benchmark, no held-out constitution, and no human-verified labels are provided, and code/data are not released. The headline numbers may therefore reflect pattern recognition of generated artifacts rather than generalizable zero-shot safety judgment.","major_comments":[{"comment":"The evaluation on OS Bench is circular with respect to the method's motivating scenario. Unsafe images are generated from seed prompts derived from the objectified constitution in Table 1, which is exactly the set of rules CLUE is engineered to apply, and safe images are generated from prompts that are often only loosely related to the rule (e.g., for 'Legs spread', the safe seed is 'A person standing casually in swim trunks...' rather than a true near-miss). Labels are assigned by construction, not by human annotation or an independent external benchmark. Consequently, the reported 95.9% recall / 94.8% accuracy (Table 2) may reflect how well CLUE recognizes text-to-image renderings of its own rule vocabulary rather than its ability to judge arbitrary images under a new constitution. Please provide an external evaluation: for example, run CLUE on existing datasets such as UnsafeBench with an appropriately mapped constitution, or have human annotators label a held-out set of real-world images using the objectified rules, and report performance on that set.","section":"Section 4.1, Table 8"},{"comment":"The rule-objectification process is self-validated and its output is reused as the test rules. The objectiveness score of each rule is computed by an LLM judge using the prompt in Figure 9, which is the same style of 'LLM-as-a-judge' evaluation that the paper is trying to replace, and no human verification is reported for whether the objectified rules preserve the semantics of the original constitution. Since the objectified rules in Table 1 are then used both to generate OS Bench and as the test constitution, the contribution of objectification is entangled with the method's ability to follow the rewritten text. Please report human agreement on the objectiveness scores and on the semantic equivalence of original vs. objectified rules, or evaluate the complete pipeline on an independent, human-authored constitution that was not used to construct the benchmark.","section":"Section 3.1 and Figure 9"},{"comment":"The dataset-construction filtering step is underspecified and could bias the results. The paper states that low-quality and poorly aligned images are filtered out 'to ensure accurate labeling', but it does not specify what alignment metric is used, how the threshold is set, or how many images are discarded per rule. If the filtering relies on a model similar to the CLIP-based relevance scanner used in Section 3.2, or if it preferentially retains easy-to-classify images, the reported accuracy and recall on OS Bench would be inflated. Please specify the exact filtering procedure, report per-rule retention rates, and verify that the filtering does not advantage CLUE over the baselines.","section":"Section 4.1"},{"comment":"The objectification ablation is too limited to support the claim that objectification is crucial. It compares a single rule ('Legs spread') with 50 positive and 50 negative generated images, and the safe vs. unsafe prompts for that rule (Table 8) are not genuinely borderline. Moreover, the objectiveness scores in Table 5 are LLM-generated, so the comparison between score 6 and score 9 is not independently validated. Please expand the ablation to multiple rules, use human-verified near-miss negatives, and consider reporting the method's performance with the original (non-objectified) constitution as an additional baseline.","section":"Section 4.3, Table 5"},{"comment":"The comparison to fine-tuned baselines (Table 3) is not a fair head-to-head for the zero-shot setting. Q16, SD Filter, NSFW Detector, and LLaVA Guard are used as pre-trained on their own datasets and are not given the constitution, so the large margin in Table 3 conflates the advantage of receiving the test-specific rules with the advantage of CLUE's architecture. The more controlled zero-shot baselines in Table 2 are appropriate, but they are still evaluated only on the closed-loop OS Bench. Please either fine-tune or prompt the baselines with the same objectified constitution in a comparable zero-shot protocol, or clearly separate the 'generalization beyond training rules' claim from the 'effective zero-shot judge' claim.","section":"Section 4.2 and Appendix C"}],"minor_comments":[{"comment":"The sentence 'we propose a MLLM-based method includes objectifying safety rules' is grammatically incomplete; 'includes' should be 'that includes' or 'consisting of'.","section":"Abstract"},{"comment":"The column header 'Architecutre' is misspelled; it should be 'Architecture'.","section":"Table 2, Table 3, Table 9"},{"comment":"The text says 'we present the distribution of these differences on OS Bench in Figure 4', but the referenced figure is Figure 8. Please correct the cross-reference.","section":"Section 4.3, Figure 8"},{"comment":"Line 9 is labeled 'Precondition Extraction (offline)' but appears inside the per-rule loop; clarify whether the extraction is cached per rule and only performed once across images.","section":"Algorithm 1"},{"comment":"Several 'safe' seed prompts are not borderline with respect to the corresponding rule (e.g., 'A fully clothed buttocks of a man' vs. 'A man with most of the buttocks not covered'). Consider reporting human-judged closeness to the rule boundary to substantiate the 'borderline safe' terminology.","section":"Appendix B, Table 8"},{"comment":"The conclusion states that the method 'pav[es] the way for future improvements' but does not discuss the key limitation that the current evaluation is on a synthetic, self-constructed benchmark. A brief limitation paragraph would help readers calibrate the claims.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the OS Bench construction is the central weakness. I do not believe this is a fatal flaw in the method itself; the multi-stage design is reasonable and the paper identifies real MLLM failure modes. However, the current evaluation cannot support the strong quantitative claims without external validation. I would be willing to reconsider after the authors add (a) an evaluation on an independent, human-annotated or external benchmark, (b) a held-out constitution experiment, and (c) transparency on the filtering procedure and objectification validation. I also note that the paper's comparison to fine-tuned baselines is framed in a way that overstates the advantage; this should be repositioned as a motivation rather than a direct comparison. The paper does not release code or data, which makes independent verification difficult; I encourage the authors to release at least the OS Bench construction details and evaluation scripts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the pipeline is real, the ablations are honest, and the headline numbers are probably reproducible on their own benchmark—but that benchmark is a closed loop. OS Bench is generated from the same objectified rules CLUE is designed to apply, and the labels come from whether the generation satisfied those rules. So the 95.9% recall tells you the method recognizes text-to-image renderings of its own rule vocabulary, not that it generalizes to a new constitution.\n\nWhat's genuinely new: the combination of rule objectification, precondition chains, debiased token probabilities (both the no-image-token baseline and the centric-region-removal trick), and cascaded reasoning is a coherent pipeline. I especially liked the idea of decomposing rules into preconditions and the demonstration that MLLMs fail on the full rule but succeed on the precondition (Figures 12-14). The ablations are consistent: removing precondition extraction or debiased token probability drops accuracy by several points, which supports the design.\n\nThe soft spots are mostly around evidence. The benchmark construction (Section 4.1 and Appendix B) starts with seed prompts, uses LLM rewriting, and then filters out low-quality or misaligned images—that filter removes exactly the hard cases. The objectiveness scores are self-assessed by the LLM judge. No code, no data, no human-verified labels, no external benchmark. The relevance scanning might already separate unsafe/safe by CLIP similarity, so the later stages could be less tested than the tables suggest. This is a significant limitation, but not fatal: the method is coherent and the problem is real. The paper would be much stronger with a human-validated subset or a held-out constitution.\n\nThe efficiency numbers (22-102 seconds per image) are honest and include the cost. The claim that this can auto-label data for smaller models is plausible but untested.\n\nBottom line: this is a serious engineering contribution and a good example of how to decompose a complex safety rule for MLLM judging. The current evaluation doesn't support the generalization claim. I'd send it to peer review, but ask for external validation or a clear limitation statement. I wouldn't cite it as evidence of zero-shot safety judgment; I'd cite it as a method paper if I needed the pipeline details.","headline":"A well-engineered zero-shot pipeline whose headline numbers rest on a closed-loop synthetic benchmark, so treat them as evidence of self-consistency rather than real-world generalization.","tokens_in":20390,"tokens_out":2492,"would_cite":false,"duration_ms":25770,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-shot MLLM pipeline flags unsafe images at 95.9% recall without any human-labeled data.","keywords":["image safety judgment","zero-shot MLLM","constitutional AI","precondition chains","token probability debiasing","relevance filtering","content moderation","safety constitution"],"falsifier":"Evaluate CLUE on a held-out set of real-world images labeled by human moderators, drawn from a platform's actual flagged content rather than generated from seed prompts tied to the objectified rules, and compare its recall and F1 to the reported 95.9% and 0.949. If the scores fall to near the naive zero-shot baselines (55-65% recall), the claim that CLUE generalizes zero-shot is falsified.","tokens_in":19326,"feed_emoji":"🛡️","tokens_out":9497,"duration_ms":77414,"temperature":0.7,"pith_summary":"The paper claims that a carefully engineered zero-shot prompting pipeline can outperform both naive zero-shot prompts and existing fine-tuned safety classifiers on the task of deciding whether an image violates a written safety constitution, without any human-labeled training data. The proposed method, CLUE, works by rewriting vague safety rules into objective ones, filtering each image against every rule with a CLIP similarity check, decomposing each rule into a chain of simple preconditions, and then judging each precondition with debiased token probabilities, falling back to chain-of-thought reasoning only on uncertain cases. On the authors' new synthetic benchmark, OS Bench, CLUE reaches a 95.9% recall, 94.8% accuracy, and 0.949 F1 score with InternVL2-76B, far above the zero-shot baselines and the fine-tuned classifiers evaluated off-the-shelf. The significance, if true, is that safety rules could be updated and applied without human annotation, since the judgment rests on the MLLM's pretrained knowledge plus a structured decomposition of the policy text.","feed_headline":"Unsafe-image judge reaches 95.9% recall with zero training","feed_subtitle":"CLUE splits rules into preconditions, strips token bias, and reasons only when needed.","key_machinery":"The central mechanism is the precondition chain together with the debiased token-probability score. A precondition chain rewrites a long safety rule into a Boolean formula of simple, verifiable statements, so the MLLM never has to reason about the whole policy at once. Each precondition is scored as $P(\\text{Yes})/(P(\\text{Yes})+P(\\text{No}))$, then the score is debiased twice: once by comparing with the score of the same prompt without image tokens (removing language-prior bias), and once by comparing the whole image's score with the score of the image after its central object is cropped out by OWLv2 (removing spatial-correlation bias). A rule is flagged as violated only when every precondition in its chain is satisfied; that combination is what converts a subjective policy text into a sequence of binary visual checks a pretrained MLLM can perform reliably.","core_discovery":"CLUE establishes that zero-shot MLLMs can serve as reliable constitutional image judges provided three obstacles are removed: subjective rule wording, long complex rule text, and token-level biases from language priors and non-centric image regions. The core discovery is that each obstacle has a concrete workaround: rules are objectified to a 9/10 objectivity score, each rule is converted into a logically complete precondition chain such as [person or animal visible] AND [visible, bloody injuries] AND [injuries indicate imminent death], and each precondition is judged by a Yes/No token-probability score that is debiased twice, once by subtracting the score of an image-less query and once by comparing the whole image with a version whose central object has been removed by OWLv2. When the debiased score is unambiguous, no further computation is needed; otherwise a cascaded chain-of-thought reasoning step supplies a rating and rationale. The result is a large margin over naive zero-shot prompts on OS Bench and a substantial margin over fine-tuned models such as Q16, Stable Diffusion Safety Checker, NSFW Detector, and LLaVA Guard when these are evaluated without retraining on the new rules.","pith_inferences":["OS Bench's unsafe images are generated from the same objectified rules CLUE applies, so the reported F1 likely overstates real-world performance; a human-labeled benchmark sampled from actual platform content would test the generalization claim.","The same decomposition-and-debiasing recipe should transfer to other zero-shot multimodal judging tasks, such as hate-meme detection, misinformation, or advertising-compliance checks, whenever a policy can be objectified into checkable preconditions.","The rule-objectification floor of 9/10 is a heuristic; for high-stakes or adversarial deployments, an auditor might require a 10/10 score or human sign-off on every rewritten rule.","The reported running times (22-102 seconds per image on an A100) suggest CLUE in its present form is best used as an offline auto-labeler rather than as an inline moderation filter, though distillation into a smaller model could change that."],"forward_implications":["Safety guidelines can be edited or replaced at runtime without retraining, because CLUE only needs to re-run rule objectification and precondition extraction after a policy change.","Because CLUE labels images on its own, its output can serve as pseudo-labels to distill a smaller one-step MLLM, which the paper identifies as a way to cut inference cost while keeping accuracy.","Relevance scanning filters out roughly two-thirds of the rules for a typical image while retaining 96.6% of the ground-truth violated rules, making per-image inspection tractable.","The per-rule breakdown shows CLUE resolves fine-grained borderline cases, such as legs spread beyond 90 degrees versus a casual stance, at 95-99% F1 on OS Bench."],"supporting_citations":[{"why":"Supplies the CLIP model used to compute image-rule similarity in relevance scanning.","marker":"Radford et al., 2021"},{"why":"Provides InternVL2-8B and InternVL2-76B, the primary MLLM backbones on which CLUE is evaluated.","marker":"Chen et al., 2023"},{"why":"Provides Qwen2-VL-7B-Instruct, another MLLM backbone showing the method transfers across models.","marker":"Wang et al., 2024"},{"why":"Provides the LLM-as-Optimizer procedure used to objectify the safety constitution.","marker":"Yang et al., 2024"},{"why":"Provides the impartial-judge prompt template used to score rule objectivity during objectification.","marker":"Zheng et al., 2024"},{"why":"Provides OWLv2, the open-vocabulary detector used to find and crop the central region for spatial debiasing.","marker":"Minderer et al., 2024"},{"why":"Demonstrates language-prior bias in vision-language token probabilities, motivating the image-less debiasing strategy.","marker":"Lin et al., 2024"},{"why":"Documents token-probability bias in MLLMs, referenced as additional motivation for the debiased judgment step.","marker":"Zhang et al., 2024"},{"why":"Provides LLaVA Guard, a fine-tuned safety judge baseline that CLUE outperforms on OS Bench without human labeling.","marker":"Helff et al., 2024"},{"why":"Provides Q16, a fine-tuned baseline that CLUE outperforms, showing rule-specific training does not generalize.","marker":"Schramowski et al., 2022"}],"fun_headline_variants":["Zero-shot image safety judge beats fine-tuned models","CLUE: zero-shot image safety judge hits 95.9% recall","Zero-training image safety judge uses only rules","MLLM image safety judge: debias, preconditions, cascade","95.9% recall for zero-shot image safety judge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that OS Bench, a synthetic dataset whose unsafe images are generated with exactly the same objectified rules that CLUE is built to apply, is a valid proxy for real-world image safety judgment; the paper does not supply independent human-verified labels or an external benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot image safety judge beats fine-tuned models","CLUE: zero-shot image safety judge hits 95.9% recall","Zero-training image safety judge uses only rules","MLLM image safety judge: debias, preconditions, cascade","95.9% recall for zero-shot image safety judge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000715,"raw_usage":{"total_tokens":3282,"prompt_tokens":1082,"completion_tokens":2200,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":698,"completion_tokens_details":{"reasoning_tokens":2116}},"tokens_in":698,"tokens_out":2200,"duration_ms":20851,"temperature":1.0,"reasoning_tokens":2116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:56:17.866881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate CLUE on a held-out set of real-world images labeled by human moderators, drawn from a platform's actual flagged content rather than generated from seed prompts tied to the objectified rules, and compare its recall and F1 to the reported 95.9% and 0.949. If the scores fall to near the naive zero-shot baselines (55-65% recall), the claim that CLUE generalizes zero-shot is falsified.","supporting_citations":[],"review_version":1}