Pith. sign in

REVIEW 3 major objections 5 minor 48 references

Simple image edits like color inversion and blur bypass commercial AI moderation APIs while remaining human-recognizable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 15:13 UTC pith:OEVSLFQW

load-bearing objection Solid black-box audit: cheap one-shot and intensity-limited transforms flip three commercial image-moderation APIs, so they are not standalone security boundaries. the 3 major comments →

arxiv 2607.28187 v1 pith:OEVSLFQW submitted 2026-07-30 cs.AI cs.CRcs.SE

Old Tricks, New Models: How Simple Image Transformations Break Modern AI-based Content Moderation

Marco Alecci , Francesco Marchiori , Iyiola Emmanuel Olatunji , Tegawend\'e F. Bissyand\'e , Jacques Klein This is my paper
classification cs.AI cs.CRcs.SE
keywords content moderationimage transformationsblack-box robustnessfoundation modelscommercial APIsunsafe-to-safe evasionmultimodal safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Commercial image-moderation APIs built on large foundation models are sold as broader, smarter safety filters than older task-specific classifiers. This paper asks whether that upgrade also makes them harder to evade. Through roughly 600,000 black-box queries against OpenAI omni-moderation, Amazon Rekognition, and Google Cloud SafeSearch, the authors show that seven cheap, model-agnostic transformations—blur, noise, salt-and-pepper, channel swaps, grayscale, and color inversion—can flip unsafe decisions to safe without gradients, surrogate models, or any knowledge of the target. Even fixed one-shot edits succeed while images stay recognizable to people, and success rates remain high under moderate perceptual-similarity constraints. Robustness varies sharply by content type: multimodal memes and self-harm are especially fragile. The practical claim is that swapping in a foundation-model API does not by itself create a reliable security boundary; these systems must be treated as one layer in a larger pipeline and stress-tested under everyday visual transformations.

Core claim

All three evaluated commercial image-moderation services can be bypassed by inexpensive, model-agnostic image transformations that require no gradients, surrogate models, or target knowledge. Fixed edits such as color inversion and grayscale already induce unsafe-to-safe flips while preserving human-recognizable content, and intensity-dependent attacks succeed under non-trivial MS-SSIM similarity thresholds; robustness further varies by dataset and harm category, with multimodal and self-harm content especially vulnerable. Therefore foundation-model moderation APIs are not standalone security boundaries.

What carries the argument

Attack Success Rate (ASR) under system-specific baselines: for each API, clean images first establish which items are flagged unsafe; a transformation succeeds only when it flips that same API’s decision to safe, measured both for one-shot edits and for intensity-dependent edits constrained by multi-scale structural similarity (MS-SSIM).

Load-bearing premise

That turning each provider’s own labels and scores into a simple safe/unsafe bit, then only attacking images that provider already calls unsafe, fairly measures real-world bypass risk rather than taxonomy or threshold mismatch.

What would settle it

Re-run the same seven transformations on a fresh sample of images that each provider currently flags unsafe; if unsafe-to-safe flips drop near zero under moderate MS-SSIM thresholds (for example τ ≥ 0.6), the claimed vulnerability is gone.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Platforms that treat a single commercial moderation API as their primary filter can be systematically evaded by automated, low-cost preprocessing before upload.
  • Moderation pipelines need explicit transformation-aware checks or secondary classifiers, not only foundation-model scores.
  • Self-harm and multimodal meme content require stricter layered review than homogeneous pornographic imagery under current APIs.
  • Robustness evaluations of new moderation models should include fixed and intensity-scaled common transforms under perceptual-similarity budgets, not only optimized adversarial noise.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same cheap transforms are likely to transfer to other closed multimodal safety APIs that share similar vision backbones or training distributions.
  • Defenders may need lightweight inverse or canonicalizing preprocessors (re-colorization, de-noise, channel normalization) before the moderation call, at the cost of extra latency.
  • Because free-tier APIs are attractive to small developers, the measured vulnerability may disproportionately affect services that lack resources for multi-layer review.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a large-scale black-box robustness audit of three commercial image-moderation APIs (OpenAI omni-moderation, Amazon Rekognition, Google Cloud SafeSearch) under seven simple, model-agnostic image transformations (box/Gaussian blur, Gaussian and salt-and-pepper noise, split-merge RGB, grayscale, color inversion). Using system-specific clean baselines, ASR defined only on initially unsafe images, and MS-SSIM-constrained intensities, the authors show non-zero one-shot bypass rates (notably color inversion up to ~44% on Amazon) and substantial intensity-dependent ASRs under moderate similarity thresholds, with further variation across LSPD, UnsafeBench, and Hateful Memes and across sexual/violence/self-harm categories on OpenAI. They conclude that foundation-model moderation APIs are not, by themselves, a reliable security boundary and should be used inside layered pipelines. The evaluation comprises on the order of 600,000 API queries.

Significance. If the empirical pattern holds, the result is practically important: it shows that widely deployed, including free, commercial moderation APIs can be evaded by inexpensive, gradient-free transforms that require no surrogate or target knowledge, undercutting the assumption that moving from task-specific classifiers to foundation-model APIs automatically yields a robust safety boundary. Strengths include the careful black-box design (system-specific baselines rather than dataset ground truth; ASR restricted to initially unsafe images; shared MS-SSIM constraints; category filtering to image-supported harms), the multi-provider and multi-dataset scope, and the explicit release of experimental artifacts. The work is an audit rather than a new attack method, but the scale and deployment relevance make it a useful contribution to AI safety and content-moderation evaluation.

major comments (3)
  1. [Section 4.1] Section 4.1: Cross-provider ASR depends on ad-hoc binarization rules that are not equivalent to OpenAI’s flagged decision—Amazon is treated as unsafe if any moderation label is returned, Google if any considered category is LIKELY or VERY_LIKELY. These thresholds can inflate or deflate ASR relative to real multi-threshold deployments and make the three ASRs only loosely comparable. The paper should either (i) report sensitivity over confidence/likelihood cutoffs (e.g., Amazon ≥50/70/90; Google including POSSIBLE) or (ii) clearly frame RQ1 as a directional existence result rather than a calibrated cross-provider ranking.
  2. [Abstract; Sections 3.2, 4.1] Abstract finding (2) and Sections 3.2/4.1: The claim that transforms preserve content “recognizable to humans” is supported only by MS-SSIM thresholds and author assertion. MS-SSIM is a useful proxy but is not a human study, and fixed transforms such as full color inversion can remain structurally similar while changing appearance substantially. Either add a small human recognition/label-consistency check on a subsample of successful bypasses, or qualify the claim to “perceptually similar under MS-SSIM” and remove stronger human-recognizability language from the abstract.
  3. [Sections 4.1–4.2; Limitations; Abstract] Sections 4.1–4.2: The claim that “all three commercial services” are vulnerable is established only on the LSPD pornographic sample; broader dataset and category analyses (UnsafeBench, Hateful Memes, self-harm) are OpenAI-only for cost reasons (Limitations). That design choice is understandable, but the abstract and conclusion currently over-generalize provider-level vulnerability across harm types. Restrict the multi-harm claims to OpenAI, or add at least a small multi-category spot-check on Amazon/Google.
minor comments (5)
  1. [Table 1; Figure 2; Appendix A.1] Table 1 / Figure 2: Report the denominator (number of initially unsafe images) per provider alongside ASR so readers can assess sampling uncertainty; Appendix Table 3 has baselines but they are easy to miss when reading RQ1.
  2. [Section 3.2; RQ3] Section 3.2: Specify the exact ε grids and max intensities tested for each intensity-dependent transform; RQ3 normalizes by max tested intensity, so those ranges are needed for reproducibility.
  3. [Figure 1; Appendix A.3] Figure 1 and Appendix figures: The paper already warns about sensitive content; ensure camera-ready redactions cannot be reverse-engineered and that figure captions state that evaluations used uncensored inputs.
  4. [Section 6] Related Work / Discussion: Briefly note non-stationarity of live commercial APIs (model updates can change ASR) as a threat to longitudinal validity; the Limitations section is otherwise appropriately scoped.
  5. [Throughout] Minor typos and spacing artifacts appear throughout (e.g., “surrogatemodels”, “orknowledgeofthetargetsystem”, “evaluatedunderrealistictransformations”). A full proofread pass is needed.

Circularity Check

0 steps flagged

No significant circularity: empirical black-box audit whose success metric is observed third-party API decision flips, not a fitted or self-defined prediction.

full rationale

The paper’s load-bearing claims are observational: after applying fixed or intensity-parameterized image transforms, commercial moderation APIs (OpenAI omni-moderation, Amazon Rekognition, Google SafeSearch) change initially-unsafe decisions to safe. ASR is defined as the fraction of system-specific clean-unsafe images that flip; baselines are the APIs’ own clean outputs, not dataset ground truth; intensity-dependent results are constrained by MS-SSIM thresholds. None of these steps rearrange a fitted parameter into a ‘prediction,’ import uniqueness from the authors, or define the outcome in terms of itself. Prior self-citations (Alecci et al. 2023; Marchiori et al. 2025) only motivate the transformation suite; the measured bypass rates are independent black-box outcomes on external services (~600k queries). Binarization rules and the initial-unsafe filter are measurement choices, not circular reductions. Score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on experimental definitions and domain choices rather than new physical entities or fitted laws. Load-bearing ingredients are the binarization of heterogeneous API outputs, the unsafe-only evaluation filter, the MS-SSIM preservation proxy for ‘still recognizable,’ and the assumption that the chosen datasets/transforms represent realistic evasion pressure.

free parameters (4)
  • MS-SSIM acceptance thresholds τ = evaluated across τ in {0.0,...,0.9}
    Intensity-dependent ASR is reported after selecting the strongest tested ε with MS-SSIM ≥ τ; the discrete τ grid shapes which bypasses count as valid.
  • Transformation intensity grids ε
    Minimum successful intensity and constrained ASR depend on the authors’ chosen tested ranges/scales per transform family.
  • Google SafeSearch unsafe threshold = LIKELY or VERY_LIKELY
    Binary unsafe label requires at least one considered category at LIKELY or VERY_LIKELY; alternate cutoffs would change ASR.
  • LSPD sample size = 1000
    Cross-provider comparison uses a 1000-image random sample justified by a power/margin calculation rounded up from ~665.
axioms (4)
  • domain assumption A moderation bypass is an unsafe→safe decision change under the provider’s own clean baseline, independent of dataset ground-truth labels.
    Stated in Section 3.3/4.1 to separate robustness from annotation/policy mismatch; this defines the measured quantity.
  • domain assumption MS-SSIM above a threshold is an adequate proxy that transformed content remains substantially similar / practically relevant rather than destroyed.
    Used throughout RQ1–RQ2 to avoid uninformative total degradation; human perceptual validation is not separately measured.
  • ad hoc to paper Amazon images with any returned moderation label, and Google images with any considered category ≥ LIKELY, are comparable ‘unsafe’ decisions to OpenAI flagged outputs.
    Section 4.1 mapping enables cross-provider ASR; it is a reasonable but non-unique operationalization.
  • domain assumption The selected transforms (blur, noise, grayscale, invert, split-merge) are representative inexpensive model-agnostic attacks available to real adversaries.
    Motivated by prior ‘non-mathematical’ attack work cited in Section 3.2.

pith-pipeline@v1.2.0-daily-grok45 · 18258 in / 3291 out tokens · 64050 ms · 2026-07-31T15:13:40.401661+00:00 · methodology

0 comments
read the original abstract

While automated content-moderation systems have become essential for screening harmful content at scale, conventional task-specific classifiers often provide limited policy cov- erage and contextual understanding. Recently, commercial multimodal moderation APIs built on large foundation models have been introduced with the promise of providing broader and more capable safety filters. In this work, we analyze whether this shift also yields more robust image moderation. We conduct a large-scale black-box evaluation on three established commercial image-moderation services and compare their robustness. By evaluating seven simple, model-agnostic image transformations across multiple providers, datasets, harm categories, perceptual-similarity constraints, and transformation intensities, we find that: (1) all three commercial services can be bypassed using inexpensive image transformations that require no gradients, surrogate models, or knowledge of the target system; (2) even fixed transformations such as color inversion and grayscale conversion induce unsafe-to-safe decision changes while preserving content that remains recognizable to humans; (3) their robustness varies substantially across datasets and harm categories, with multimodal content and self-harm exhibiting pronounced vulnerabilities. This yields the conclusion that replacing conventional moderation classifiers with foundation-model-based APIs does not, by itself, provide a reliable security boundary. Such systems must be evaluated under realistic transformations and deployed as one component of a layered moderation pipeline rather than as standalone safety filters.

Figures

Figures reproduced from arXiv: 2607.28187 by Francesco Marchiori, Iyiola Emmanuel Olatunji, Jacques Klein, Marco Alecci, Tegawend\'e F. Bissyand\'e.

Figure 1
Figure 1. Figure 1: Example of a Salt and Pepper transformation. Both [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ASR of intensity-dependent transformations across different minimum acceptable MS-SSIM thresholds on LSPD. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ASR of intensity-dependent transformations across the three datasets and different minimum acceptable MS-SSIM [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average category-wise ASR on the UnsafeBench [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of the normalized minimum successful [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Successful unsafe-to-safe transition using Split-Merge RGB against Google Cloud SafeSearch. (a) Original image (b) Transformed image [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Successful unsafe-to-safe transition using Salt￾and-Pepper Noise against OpenAI omni-moderation. (a) Original image (b) Transformed image [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Successful unsafe-to-safe transition using Gaussian Noise against OpenAI omni-moderation. (a) Original image (b) Transformed image [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 7 linked inside Pith

  1. [1]

    Proceedings of the 26th international symposium on research in attacks, intrusions and defenses , pages=

    Your attack is too dumb: Formalizing attacker scenarios for adversarial transferability , author=. Proceedings of the 26th international symposium on research in attacks, intrusions and defenses , pages=

  2. [2]

    2024 , howpublished =

    Upgrading the Moderation. 2024 , howpublished =

    2024

  3. [3]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    A holistic approach to undesired content detection in the real world , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  4. [4]

    arXiv preprint arXiv:1312.6199 , year=

    Intriguing properties of neural networks , author=. arXiv preprint arXiv:1312.6199 , year=

    Pith/arXiv arXiv

  5. [5]

    International Conference on Learning Representations , year =

    Explaining and Harnessing Adversarial Examples , author =. International Conference on Learning Representations , year =

  6. [6]

    International Conference on Learning Representations , year =

    Towards Deep Learning Models Resistant to Adversarial Attacks , author =. International Conference on Learning Representations , year =

  7. [7]

    Proceedings of the 2017 ACM on Asia conference on computer and communications security , pages=

    Practical black-box attacks against machine learning , author=. Proceedings of the 2017 ACM on Asia conference on computer and communications security , pages=

    2017

  8. [8]

    2025 , organization=

    Marchiori, Francesco and Alecci, Marco and Pajola, Luca and Conti, Mauro , booktitle=. 2025 , organization=

    2025

  9. [9]

    International Conference on Learning Representations , year =

    Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , author =. International Conference on Learning Representations , year =

  10. [10]

    Proceedings of the 40th international conference on software engineering , pages=

    Deeptest: Automated testing of deep-neural-network-driven autonomous cars , author=. Proceedings of the 40th international conference on software engineering , pages=

  11. [11]

    International conference on learning representations , volume=

    Augmix: A simple method to improve robustness and uncertainty under data shift , author=. International conference on learning representations , volume=

  12. [12]

    International Conference on Machine Learning , pages=

    Adversarial robustness against the union of multiple perturbation models , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  13. [13]

    International Conference on Learning Representations , year =

    Perceptual Adversarial Robustness: Defense against Unseen Threat Models , author =. International Conference on Learning Representations , year =

  14. [14]

    Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , pages=

    Lost in moderation: How commercial content moderation apis over-and under-moderate group-targeted hate speech and linguistic variations , author=. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , pages=

    2025

  15. [15]

    Qu, Yiting and Shen, Xinyue and Wu, Yixin and Backes, Michael and Zannettou, Savvas and Zhang, Yang , booktitle=

  16. [16]

    arXiv preprint arXiv:2504.01081 , year=

    ShieldGemma 2: Robust and tractable image content moderation , author=. arXiv preprint arXiv:2504.01081 , year=

    Pith/arXiv arXiv

  17. [17]

    IEEE Transactions on Information Forensics and Security , year=

    Promptguard: Soft prompt-guided unsafe content moderation for text-to-image models , author=. IEEE Transactions on Information Forensics and Security , year=

  18. [18]

    arXiv preprint arXiv:2606.09700 , year=

    What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks , author=. arXiv preprint arXiv:2606.09700 , year=

    Pith/arXiv arXiv

  19. [19]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    Making mllms blind: Adversarial smuggling attacks in mllm content moderation , author=. Findings of the Association for Computational Linguistics: ACL 2026 , pages=

    2026

  20. [20]

    Proceedings of the 11th ACM workshop on artificial intelligence and security , pages=

    All you need is "love" evading hate speech detection , author=. Proceedings of the 11th ACM workshop on artificial intelligence and security , pages=

  21. [21]

    2018 , publisher=

    Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media , author=. 2018 , publisher=

    2018

  22. [22]

    2019 , publisher=

    Behind the Screen: Content Moderation in the Shadows of Social Media , author=. 2019 , publisher=

    2019

  23. [23]

    Perspective. n.d. , howpublished =

  24. [24]

    International Journal of Data Science and Analytics , volume=

    Deep learning for hate speech detection: a comparative study , author=. International Journal of Data Science and Analytics , volume=. 2025 , publisher=

    2025

  25. [25]

    Cloud Natural Language Documentation , year =

  26. [26]

    Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021) , pages=

    HateBERT: Retraining BERT for abusive language detection in English , author=. Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021) , pages=

    2021

  27. [27]

    Findings of the association for computational linguistics: EMNLP 2021 , pages=

    fBERT: A neural transformer for identifying offensive content , author=. Findings of the association for computational linguistics: EMNLP 2021 , pages=

    2021

  28. [28]

    2021 International Joint Conference on Neural Networks (IJCNN) , pages=

    On-device content moderation , author=. 2021 International Joint Conference on Neural Networks (IJCNN) , pages=. 2021 , organization=

    2021

  29. [29]

    2022 , howpublished =

    Schuhmann, Christoph and. 2022 , howpublished =

    2022

  30. [30]

    2016 , howpublished =

    Open. 2016 , howpublished =

    2016

  31. [31]

    2026 , howpublished =

    What Is Azure. 2026 , howpublished =

    2026

  32. [32]

    Advances in neural information processing systems , volume=

    The Hateful Memes Challenge: Detecting hate speech in multimodal memes , author=. Advances in neural information processing systems , volume=

  33. [33]

    arXiv preprint arXiv:2012.14891 , year=

    Detecting hate speech in multi-modal memes , author=. arXiv preprint arXiv:2012.14891 , year=

    Pith/arXiv arXiv 2012

  34. [34]

    Moderation and Guardrailing , year =

  35. [35]

    arXiv preprint arXiv:2312.06674 , year=

    Llama guard: Llm-based input-output safeguard for human-ai conversations , author=. arXiv preprint arXiv:2312.06674 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2407.21772 , year=

    Shieldgemma: Generative ai content moderation based on gemma , author=. arXiv preprint arXiv:2407.21772 , year=

    Pith/arXiv arXiv

  37. [37]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    ToViLaG: Your visual-language generative model is also an evildoer , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  38. [38]

    International Journal of Intelligent Engineering and Systems , volume=

    LSPD: A large-scale pornographic dataset for detection and classification , author=. International Journal of Intelligent Engineering and Systems , volume=

  39. [39]

    IEEE transactions on image processing , volume=

    Image quality assessment: from error visibility to structural similarity , author=. IEEE transactions on image processing , volume=. 2004 , publisher=

    2004

  40. [40]

    2024 , howpublished =

    Horwitz, Jeff , title =. 2024 , howpublished =

    2024

  41. [41]

    2021 , howpublished =

    Adams, Richard , title =. 2021 , howpublished =

    2021

  42. [42]

    2022 , howpublished =

    Das, Shanti , title =. 2022 , howpublished =

    2022

  43. [43]

    2023 , howpublished =

    Lomas, Natasha , title =. 2023 , howpublished =

    2023

  44. [44]

    The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers , volume =

    Multiscale Structural Similarity for Image Quality Assessment , author =. The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers , volume =. 2003 , organization =

    2003

  45. [45]

    Moderations , year =

  46. [46]

    2026 , howpublished =

    Using the Image and Video Moderation. 2026 , howpublished =

    2026

  47. [47]

    2026 , howpublished =

    Detect Explicit Content (. 2026 , howpublished =

    2026

  48. [48]

    arXiv preprint arXiv:2406.05113 , year=

    LlavaGuard: An open vlm-based framework for safeguarding vision datasets and models , author=. arXiv preprint arXiv:2406.05113 , year=

    Pith/arXiv arXiv