Pith. sign in

REVIEW 3 major objections 5 minor 85 references

VModA: An Effective Framework for Adaptive NSFW Image Moderation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VMODA claims a training-free pipeline of ROI zoom-in, chain-of-thought description, and aggregated sampling lets stock vision-language models beat commercial NSFW detectors by up to 54.3 accuracy points across five content categories.

desk verdict Useful zero-shot moderation framework and a genuine label-noise contribution, but the headline 54.3% gain is recall on all-positive datasets, not accuracy. read the letter →

arxiv 2505.23386 v1 pith:MXWLX2OM submitted 2025-05-29 cs.CV cs.AI

Han Bao , Qinying Wang , Zhi Chen , Qingming Li , Xuhong Zhang , Changjiang Li , Zonghui Wang , Shouling Ji
show 1 more author
Wenzhi Chen
This is my paper
classification cs.CVcs.AI
keywords NSFWimagemoderationvision-languagemodelschain-of-thoughtpromptingadaptiveregion-of-interestzero-shotadaptationhatefulmemedetectionoutputaggregationlabelnoise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that NSFW-image moderation can be made adaptive and detail-aware without retraining, by wrapping a stock vision-language model (VLM) in a pipeline that stages the review from easy to hard. The framework, VMODA, first improves and dissects the image (super-resolution, text extraction, and a zoom-in on semantically selected regions), then runs a standard violation check under a prompt that names the image type and the applicable rule, and finally, if no violation is found, escalates to a chain-of-thought semantic pass that describes objects, states, and rhetorical devices before a large language model makes the final call. The paper claims this design beats commercial detectors and trained meme classifiers across six benchmarks, by up to 54.3 percentage points of accuracy on a school-bullying set, and that it adapts to different rule sets, scenarios, and VLM backbones with little variance. A sympathetic reader would care because the method is training-free, so it can track shifting platform regulations without data collection or fine-tuning, and because the authors show the framework doubles as a benchmark audit tool, flagging hundreds of mislabeled samples in public datasets.

What carries the argument

The load-bearing object is the VMODA pipeline itself, with two mechanisms carrying most of the weight. The first is the adaptive region-of-interest (ROI) zoom-in: an open-set tagging model labels image regions, a grounding detector frames the relevant objects, and the VLM is shown the full image together with a marked, enlarged crop, so its reading of the detail is conditioned on the whole scene (formalized as $\hat{R}_{p_i} = \arg\max( P(R_{p_i} \mid T^V, I_{p_i}) + P(R_{p_i} \mid T^V, P(I \mid I_{p_i})) )$). The second is the chain-of-thought (CoT) hierarchical description, which forces three escalating levels of description (objects, states, rhetorical devices), each consuming the previous level and each separately screened for violations. Around both sits an iterative output-aggregation strategy: at temperature 1 the VLM is sampled ten times per question, and an LLM merges the answers while discarding refusals, so that low-probability but informative responses survive. The whole review is progressive (text, whole image, zoomed regions, object reading, state reading, rhetorical reading, final LLM review), and any stage that flags a violation ends the process.

What would settle it

Take the school-bullying test set and add an equal number of benign classroom and school-corridor images, then run VMODA and the best commercial baseline on the balanced set and compute accuracy, precision, and recall. If a 'flag everything' rule also matches VMODA's recall while VMODA's precision on the benign half falls well below its reported 0.950 accuracy, the headline 54.3-point gain is an artifact of the one-class metric rather than evidence of superior moderation.

Watch

Extended reading notes

Core claim

The central claim is that a training-free orchestration of a VLM and a large language model can outperform both commercial NSFW APIs and task-specific trained classifiers. VMODA attributes the gain to three mechanisms: conditional prompting that injects the moderation rule and the image type as prior knowledge; an adaptive region-of-interest step that anchors zoomed-in segments to the full image with bounding overlays, so the VLM sees detail without losing context; and a chain-of-thought hierarchy that forces the model to move from object identification to state description to rhetorical-device analysis, with an LLM aggregating ten sampled outputs at each stage to suppress refusals and sanitized answers. On the reported numbers, VMODA reaches 0.985 accuracy on a pornographic-versus-neutral binary set, 0.950 on school-bullying frames versus 0.407 for the best commercial tool, and detection ratios above 0.8 on high-severity protest violence where commercial tools stay near 0.5. The paper further claims the framework exposes label noise in all five public benchmarks, and that re-annotation by human raters confirmed most of the disputed labels.

Load-bearing premise

The load-bearing premise is that the reported 'accuracy' on the blood and school-bullying datasets measures real moderation skill; those datasets contain only NSFW images, so the score is a recall rate, and a moderation system that flagged every image as a violation would score 100 percent on them.

Editorial extensions

If this is right

  • Platforms could update moderation rules by editing a prompt rather than collecting and re-labeling training data, since VMODA is training-free and rule-adaptive.
  • Fine-grained rules such as the school-bullying definition buy precision on their target scene but sacrifice recall on broader violence; the paper's cross-prompt experiment quantifies that trade-off, so deployment can choose between high-precision and high-coverage prompts.
  • The framework works across four VLM backbones with small accuracy variance (standard deviation at or below 0.051 on five of six datasets), so the reported gains are not tied to one model's prompting quirks.
  • Benchmark label noise is large enough to matter: the paper reports 62 of 82 disputed porn test images were normal, 30 of 43 disputed blood images were not bloody scenes, and hundreds of meme labels were inconsistent, with corrections submitted to maintainers and two datasets confirming updates.
  • Every stage of the progressive pipeline recovers content the earlier stages miss; the paper reports that advanced semantic analysis alone contributes between 27.8% and 88.8% of meme detections, so each design component earns its place.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ROI zoom-in with full-image anchoring is a general remedy for the wide-receptive-field blindness of VLM encoders, and could transfer to fine-grained visual question answering, document layout understanding, or medical image reading where small regions carry the decision.
  • The iterative aggregation trick (sampling at high temperature and having an LLM merge answers while discarding refusals) is a jailbreak-adjacent technique: the same mechanism that recovers useful moderation signals could lower the cost of extracting disallowed content from safety-aligned models, a dual-use risk the paper does not address.
  • If the label-noise findings replicate on other benchmarks, the practical takeaway is that nontrivial fractions of public NSFW datasets inherit noise from keyword-based collection or shifting standards, which strengthens the case for training-free moderation over retraining on those sets.
  • The demonstrated link between rule granularity and the precision-recall trade-off suggests a testable deployment recipe the paper leaves implicit: run several prompts at different granularities in parallel and pick by the platform's own policy threshold.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VMODA, a zero-shot framework for NSFW image moderation that combines a base VLM with an LLM aggregator. The pipeline consists of preprocessing (super-resolution, adaptive ROI zoom-in, text extraction), standard image moderation with conditional prompting and iterative output aggregation, and advanced semantic analysis using a CoT-based hierarchical description strategy. The authors evaluate VMODA on six datasets covering pornography, blood/gore, protest violence, school bullying, and hateful/harmful memes, comparing against commercial APIs (Sightengine, AWS Rekognition, Azure, Imagga) and academic baselines (Pro-Cap, ExplainHM). They report up to a 54.3% accuracy improvement over baselines, and also contribute a label-noise discovery and correction effort on public benchmark datasets, plus a real-world meme detection case study.

Significance. If its evaluation claims were fully supported, VMODA would be a valuable training-free alternative to category-specific NSFW detectors, and the paper's design choices (e.g., iterative output aggregation to handle VLM refusals, CoT-based semantics extraction) are generally reasonable and potentially transferable. The label-correction effort submitted to dataset maintainers is a concrete community contribution. However, the central headline improvement rests on a one-class evaluation that conflates accuracy with recall. The two-class experiments (porn and meme datasets) are more convincing and show consistent gains, so the framework's core idea has merit, but the evidence as presented does not support the abstract's broad claim of an accuracy improvement across all NSFW types.

major comments (3)
  1. [V-A4, Table I, Abstract] The blood and school-bully datasets contain no negative samples: Section V-A1 states that the blood set uses all 1,305 images depicting bleeding scenes and the school-bully set uses 300 frames that 'are all considered violations.' Consequently, the 'accuracy' reported in Table I for these columns is identical to recall (TP / total positives). The headline '54.3% accuracy improvement' is computed from the school-bully column (Sightengine 0.407 vs. Ours GPT4o 0.950), so it is a recall gain, not an accuracy gain. A trivial classifier that labels every image as NSFW would score 100% under this metric. The abstract's claim of 'up to a 54.3% accuracy improvement across NSFW types' is therefore unsupported for these categories. The manuscript should either rename this metric to recall/sensitivity with an explicit caveat, or add a negative-image set for these categories so that true accuracy and false-positive rates can be measured.
  2. [V-C, Table IV] The cross-prompt experiment also evaluates on positive-only sets: the protest subset with violence score ≥ 0.6 (all violent) and the entire school-bully set (all violations). The 'Acc' columns in Table IV are thus recall values, not accuracy. The conclusion in the text that this experiment 'quantifies its specificity and generality' is not justified because specificity (true-negative rate) cannot be estimated without negative examples. This undermines the adaptability claim for scenario-based rules. Please re-frame the reported numbers as detection rates on positive-only samples, or include a balanced set to support claims about specificity and accuracy.
  3. [V-A4, Section V-B] The sentence 'For datasets that lack positive samples, specifically the blood and school bully datasets, we report only accuracy' appears to contain a typo: the datasets lack negative samples, not positive samples. This misstatement is not merely cosmetic; it reflects the metric issue: when no negatives are present, 'accuracy' is a misnomer. The paper's broader comparative claims in Section V-B ('VMODA consistently achieves strong performance in all NSFW categories') should be revised to separate one-class sensitivity results from two-class accuracy results, and the 54.3% figure should be attributed to sensitivity improvement only.
minor comments (5)
  1. [V-A3] Typo: 'experiement setting' should be 'experiment setting'.
  2. [IV-B1, prompt PS] The system prompt begins 'Your are a professional image reviewer' and should read 'You are a professional image reviewer.'
  3. [Figure 4 caption] The caption states that the bars illustrate 'the proportion detected by each module when identifying negative samples.' Based on the text in Section VI-B, this should read 'positive samples' (i.e., NSFW images), since the figure reports the contribution of each step to detecting violations.
  4. [V-D] The robustness analysis reports standard deviations of accuracy across the four VLMs but does not provide any confidence intervals or significance tests; given the stochastic sampling (n=10) and aggregation, a short discussion of variance or a table of per-backbone repeats would improve reliability.
  5. [References] Reference [31] is incomplete in the bibliography: the author list is truncated to 'Y. M. X. S. Y. Qu, N. Y. M. Backes, and S. Z. Y. Zhang' instead of listing the full author names.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VMODA is a zero-shot prompting framework whose predictions are not fitted to the test labels; the metric concern on all-positive datasets is a validity issue, not circularity.

full rationale

The paper proposes VMODA, a zero-shot moderation framework composed of prompting and image-preprocessing modules. No parameters are trained or fitted to the evaluation datasets; the rule strings are simply inserted into prompts, and the base VLMs/LLMs are used as frozen models. The headline 'up to 54.3% accuracy improvement' derives from Table I comparisons against external commercial tools (Sightengine, AWS, Azure) and trained academic models (Pro-Cap, ExplainHM). Those baselines are independent of VMODA's own outputs, so the comparison is not self-referential. The main caveat identified by the reviewer is that the blood and school-bully datasets contain only positive (NSFW) samples, making 'accuracy' equal to recall; the 54.3% gap is therefore a sensitivity difference, not a true accuracy difference. This is a measurement-validity critique, not a circularity: the paper never defines its predictions in terms of the ground-truth labels, nor does it fit any parameter to the test data. There are no self-citations, no imported uniqueness theorems, and no ansatz that is justified solely by citing the authors' own prior work. The framework's use of dataset-derived rules as prompt inputs is the intended zero-shot mechanism; it does not reduce the prediction to the label by construction because the VLM/LLM must still perform visual understanding to map an image to the rule. All design choices (ROI zoom-in, CoT description, aggregation) are evaluated by ablation/contribution analysis against the same independent baselines. Thus the derivation chain is self-contained and externally benchmarked.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new entities. It relies on pre-trained models and standard prompting techniques. The only hand-chosen numbers are the temperature, sample count, and super-resolution size, none of which are fitted to the test sets. However, the per-dataset choice of which modules to apply (e.g., disabling ROI zoom-in for memes) is a post-hoc configuration that affects results.

free parameters (3)
  • Sampling temperature for VLM = 1
    Set to 1 to balance output diversity and quality; not fitted to test data.
  • Number of sampling iterations n = 10
    Chosen for efficiency-diversity tradeoff; not fitted.
  • Super-resolution target size = 2048 pixels (longer side)
    Chosen to enhance details; downscaled if VLM cannot handle.
assumptions (4)
  • domain assumption Base VLMs can produce reliable image descriptions and judgments when prompted appropriately.
    The entire framework depends on the VLM's understanding; the paper shows vanilla VLMs struggle, but the framework assumes the prompting fixes this.
  • domain assumption The LLM used for aggregation can correctly filter refusals, combine multiple VLM responses, and avoid hallucinations.
    The iterative aggregation strategy relies on the LLM's ability to extract useful information from noisy samples.
  • domain assumption The rule text provided in the system prompt is sufficient for the VLM/LLM to understand and apply the moderation policy.
    The paper constructs rules from dataset definitions and assumes the model interprets them correctly.
  • domain assumption The public NSFW datasets and their labels, despite identified noise, are reliable enough for evaluating relative performance.
    The paper reports accuracy against original labels while also showing many labels are wrong; this creates uncertainty in the reported numbers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VModA: An Effective Framework for Adaptive NSFW Image Moderation." pith.science (2026). https://pith.science/paper/MXWLX2OM

@misc{pith2026250523386,
  author       = {Pith},
  title        = {Pith review of: VModA: An Effective Framework for Adaptive NSFW Image Moderation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXWLX2OM}},
  note         = {Machine review of arXiv:2505.23386}
}
read the original abstract

Not Safe/Suitable for Work (NSFW) content is rampant on social networks and poses serious harm to citizens, especially minors. Current detection methods mainly rely on deep learning-based image recognition and classification. However, NSFW images are now presented in increasingly sophisticated ways, often using image details and complex semantics to obscure their true nature or attract more views. Although still understandable to humans, these images often evade existing detection methods, posing a significant threat. Further complicating the issue, varying regulations across platforms and regions create additional challenges for effective moderation, leading to detection bias and reduced accuracy. To address this, we propose VModA, a general and effective framework that adapts to diverse moderation rules and handles complex, semantically rich NSFW content across categories. Experimental results show that VModA significantly outperforms existing methods, achieving up to a 54.3% accuracy improvement across NSFW types, including those with complex semantics. Further experiments demonstrate that our method exhibits strong adaptability across categories, scenarios, and base VLMs. We also identified inconsistent and controversial label samples in public NSFW benchmark datasets, re-annotated them, and submitted corrections to the original maintainers. Two datasets have confirmed the updates so far. Additionally, we evaluate VModA in real-world scenarios to demonstrate its practical effectiveness.

Figures

Figures reproduced from arXiv: 2505.23386 by the authors.

Figure 1
Figure 1. Three examples highlighting challenges VLMs face in pornographic moderation. The first image and response [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The overview of VMODA. The top of the diagram represents the VLM and LLM based on different system prompt strategies. The colored steps in the flowchart indicate the use of the corresponding prompt strategies and models during execution. content to harass, spread hate, and target vulnerable indi￾viduals, particularly minors, for blackmail, financial gain, or inciting violence. These actions contribute to a hostile o… view at source ↗
Figure 3
Figure 3. An example of the region of interest zoom-in. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Contributions of each module to the moderation of NSFW datasets [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Samples of the incorrect original label in porn dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 8
Figure 8. Figure 8: No blood image in blood dataset. F. Annotation Results in Hateful/Harmful Datasets The results of manual annotation in Hateful/Harmful datasets are shown in Table VI. The second row (Inconsistent) of the table shows the number of samples where the original label was fo…
Figure 7
Figure 7. Figure 7: NSFW content in details of the images. The NSFW [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 9
Figure 9. Figure 9: Samples of the incorrect original label in two meme [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Harmful samples in the real world. K. Examples of NSFW memes in Real World [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 58 canonical work pages

  1. [1]

    Not safe for work,

    Wikipedia, “Not safe for work,” https://en.wikipedia.org/wiki/No t safe for work, 2024, accessed: 2024-07-25. [Online]. Available: https://en.wikipedia.org/wiki/Not safe for work

    2024

  2. [2]

    Momo challenge hoax,

    “Momo challenge hoax,” https://en.wikipedia.org/wiki/Momo Challen ge hoax, April 2025, wikipedia, the free encyclopedia

    2025

  3. [3]

    Multi-level detector for pornographic content using cnn models,

    Q.-H. Nguyen, H.-L. Tran, T.-T. Nguyen, D.-D. Phan, D.-L. Vuet al., “Multi-level detector for pornographic content using cnn models,” in 2020 RIVF international conference on computing and communication technologies (RIVF). IEEE, 2020, pp. 1–5

    2020

  4. [4]

    Pornographic content classification using deep-learning,

    A. Tabone, K. Camilleri, A. Bonnici, S. Cristina, R. Farrugia, and M. Borg, “Pornographic content classification using deep-learning,” in Proceedings of the 21st ACM symposium on document engineering, 2021, pp. 1–10

    2021

  5. [5]

    A motion-based approach for real-time detection of pornographic content in videos,

    J. Geremias, E. K. Viegas, A. S. Britto Jr, and A. O. Santin, “A motion-based approach for real-time detection of pornographic content in videos,” inProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, 2022, pp. 1066–1073

    2022

  6. [6]

    Moderating illicit online image promotion for unsafe user generated content games using large vision language models,

    K. Guo, A. Utkarsh, W. Ding, I. Ondracek, Z. Zhao, G. Freeman, N. Vishwamitra, and H. Hu, “Moderating illicit online image promotion for unsafe user generated content games using large vision language models,” in33st USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association, 2024

    work page 2024

  7. [7]

    Audiovisual dependency attention for violence detection in videos,

    W. Pang, W. Xie, Q. He, Y . Li, and J. Yang, “Audiovisual dependency attention for violence detection in videos,”IEEE Transactions on Mul- timedia, vol. 25, pp. 4922–4932, 2022

    work page 2022

  8. [8]

    Ai-assisted edge vision for violence detection in iot-based industrial surveillance networks,

    F. U. M. Ullah, K. Muhammad, I. U. Haq, N. Khan, A. A. Heidari, S. W. Baik, and V . H. C. de Albuquerque, “Ai-assisted edge vision for violence detection in iot-based industrial surveillance networks,”IEEE Transactions on Industrial Informatics, vol. 18, no. 8, pp. 5359–5370, 2021

    work page 2021

Show all 85 references
  1. [9]

    Decouple and resolve: transformer-based models for online anomaly detection from weakly la- beled videos,

    T. Liu, C. Zhang, K.-M. Lam, and J. Kong, “Decouple and resolve: transformer-based models for online anomaly detection from weakly la- beled videos,”IEEE Transactions on Information Forensics and Security, vol. 18, pp. 15–28, 2022

    2022

  2. [10]

    The hateful memes challenge: Detecting hate speech in multimodal memes,

    D. Kiela, H. Firooz, A. Mohan, V . Goswami, A. Singh, P. Ringshia, and D. Testuggine, “The hateful memes challenge: Detecting hate speech in multimodal memes,”Advances in neural information processing systems, vol. 33, pp. 2611–2624, 2020

    2020

  3. [11]

    You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content,

    X. He, S. Zannettou, Y . Shen, and Y . Zhang, “You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content,”arXiv preprint arXiv:2308.05596, 2023

    2023 arXiv

  4. [12]

    Pro-cap: Leveraging a frozen vision-language model for hateful meme detection,

    R. Cao, M. S. Hee, A. Kuek, W.-H. Chong, R. K.-W. Lee, and J. Jiang, “Pro-cap: Leveraging a frozen vision-language model for hateful meme detection,” inProceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 5244–5252

    2023

  5. [13]

    Momenta: A multimodal framework for detecting harmful memes and their targets,

    S. Pramanick, S. Sharma, D. Dimitrov, M. S. Akhtar, P. Nakov, and T. Chakraborty, “Momenta: A multimodal framework for detecting harmful memes and their targets,” inFindings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 4439–4455

    2021

  6. [14]

    Image moderation,

    Sightengine, “Image moderation,” https://sightengine.com/image-mod eration, 2024, accessed: 2024-08-17

    2024

  7. [15]

    Content moderation,

    Clarifai, “Content moderation,” https://www.clarifai.com/solutions/con tent-moderation, 2024, accessed: 2024-08-17

    2024

  8. [16]

    Amazon rekognition,

    Amazon Web Services, “Amazon rekognition,” https://aws.amazon.com /cn/rekognition/, accessed: 2024-08-17

    2024

  9. [17]

    Ai content safety,

    Microsoft Azure, “Ai content safety,” https://azure.microsoft.com/en-u s/products/ai-services/ai-content-safety/, 2024, accessed: 2024-08-17

    2024

  10. [18]

    Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos,

    N. AlDahoul, M. J. T. Tan, H. R. Kasireddy, and Y . Zaki, “Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos,”arXiv preprint arXiv:2411.17123, 2024

    2024 arXiv

  11. [19]

    Teenagers exposed to ’horrific’ content online - and this survey reveals the scale of the problem,

    T. Cohen, “Teenagers exposed to ’horrific’ content online - and this survey reveals the scale of the problem,” https://news.sky.com/story/teenagers-exposed-to-horrific-content-o nline-and-this-survey-reveals-the-scale-of-the-problem-13331556, 2025, sky News

    2025

  12. [20]

    Exposed or erased: Algorithmic censorship of nudity in art,

    P. Riccio, T. Hofmann, and N. Oliver, “Exposed or erased: Algorithmic censorship of nudity in art,” inProceedings of the CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–17

    2024

  13. [21]

    Detection and categorization of suggestive thumb- nails: A step towards a safer internet,

    M. Oliveira Franca, “Detection and categorization of suggestive thumb- nails: A step towards a safer internet,” 2021

    2021

  14. [22]

    Auditing image-based nsfw classifiers for content filtering,

    W. Leu, Y . Nakashima, and N. Garcia, “Auditing image-based nsfw classifiers for content filtering,” inThe 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024, pp. 1163–1173

    2024

  15. [23]

    Nipples, memes, and algorithmic failure: Nsfw critique of tumblr censorship,

    E. Pilipets and S. Paasonen, “Nipples, memes, and algorithmic failure: Nsfw critique of tumblr censorship,”New Media & Society, vol. 24, no. 6, pp. 1459–1480, 2022

    2022

  16. [24]

    Lever- aging large-scale multimedia datasets to refine content moderation models,

    I. Sarridis, C. Koutlis, O. Papadopoulou, and S. Papadopoulos, “Lever- aging large-scale multimedia datasets to refine content moderation models,” in2022 IEEE Eighth International Conference on Multimedia Big Data (BigMM). IEEE, 2022, pp. 125–132

    2022

  17. [25]

    Who’s in and who’s out? a case study of multimodal clip-filtering in datacomp,

    R. Hong, W. Agnew, T. Kohno, and J. Morgenstern, “Who’s in and who’s out? a case study of multimodal clip-filtering in datacomp,”arXiv preprint arXiv:2405.08209, 2024

    2024 arXiv

  18. [26]

    Unsafebench: Benchmarking image safety classifiers on real-world and ai-generated images,

    Y . Qu, X. Shen, Y . Wu, M. Backes, S. Zannettou, and Y . Zhang, “Unsafebench: Benchmarking image safety classifiers on real-world and ai-generated images,”arXiv preprint arXiv:2405.03486, 2024

    2024 arXiv

  19. [27]

    Evaluation of content moderation software for nudity and pornography detection in various scenarios,

    N. Aldahoul, H. Abdul Karim, M. A. Momo, M. Sy, and M. J. Tan, “Evaluation of content moderation software for nudity and pornography detection in various scenarios,” 07 2023

    2023

  20. [28]

    Stealthy porn: Understanding real-world adversarial images for illicit online promotion,

    K. Yuan, D. Tang, X. Liao, X. Wang, X. Feng, Y . Chen, M. Sun, H. Lu, and K. Zhang, “Stealthy porn: Understanding real-world adversarial images for illicit online promotion,” in2019 IEEE Symposium on Security and Privacy (SP). IEEE, 2019, pp. 952–966

    2019

  21. [29]

    Automatic content mod- eration on social media,

    D. Karabulut, C. Ozcinar, and G. Anbarjafari, “Automatic content mod- eration on social media,”Multimedia Tools and Applications, vol. 82, no. 3, pp. 4439–4463, 2023

    2023

  22. [30]

    Adult nudity and sexual activity,

    I. Meta Platforms, “Adult nudity and sexual activity,” https://transparency.meta.com/en-us/policies/community-standards/a dult-nudity-sexual-activity/, 2024, accessed: 2024-12-26

    2024

  23. [31]

    From meme to threat: On the hateful meme understanding and induced hateful content generation in open-source vision language models

    Y . M. X. S. Y . Qu, N. Y . M. Backes, and S. Z. Y . Zhang, “From meme to threat: On the hateful meme understanding and induced hateful content generation in open-source vision language models.”

  24. [32]

    Tovilag: Your visual-language generative model is also an evildoer,

    X. Wang, X. Yi, H. Jiang, S. Zhou, Z. Wei, and X. Xie, “Tovilag: Your visual-language generative model is also an evildoer,” inThe 2023 Conference on Empirical Methods in Natural Language Processing, 2024

    2023

  25. [33]

    Seeing without looking: Analysis pipeline for child sexual abuse datasets,

    C. Laranjeira da Silva, J. Macedo, S. Avila, and J. dos Santos, “Seeing without looking: Analysis pipeline for child sexual abuse datasets,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 2189–2205

    2022

  26. [34]

    Bert: Pre-training of deep bidirectional transformers for language understanding. arxiv,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding. arxiv,” arXiv preprint arXiv:1810.04805, 2019

    2019 arXiv

  27. [35]

    Towards explainable harmful meme detection through multimodal debate between large language models,

    H. Lin, Z. Luo, W. Gao, J. Ma, B. Wang, and R. Yang, “Towards explainable harmful meme detection through multimodal debate between large language models,” inProceedings of the ACM on Web Conference 2024, 2024, pp. 2359–2370

    2024

  28. [36]

    Investigating moderation challenges to combating hate and harassment: The case of mod-admin power dynamics and feature misuse on reddit,

    M. Tabassum, A. Mackey, A. Schuett, and A. Lerner, “Investigating moderation challenges to combating hate and harassment: The case of mod-admin power dynamics and feature misuse on reddit,” in30th USENIX Security Symposium (USENIX Security 24). USENIX Associa- tion, 2024

    2024

  29. [37]

    Understand- ing the security and privacy implications of online toxic content on refugees,

    A. Arunasalam, H. Farrukh, E. Tekcan, and Z. B. Celik, “Understand- ing the security and privacy implications of online toxic content on refugees,” inUSENIX Security Symposium, 2024

    2024

  30. [38]

    Behind the tube: Exploitative monetization of content on{YouTube},

    A. Chu, A. Arunasalam, M. O. Ozmen, and Z. B. Celik, “Behind the tube: Exploitative monetization of content on{YouTube},” in31st USENIX Security Symposium (USENIX Security 22), 2022, pp. 2171– 2188

    2022

  31. [39]

    No easy way out: the effectiveness of deplatforming an extremist forum to suppress hate and harassment,

    A. V . Vu, A. Hutchings, and R. Anderson, “No easy way out: the effectiveness of deplatforming an extremist forum to suppress hate and harassment,”arXiv preprint arXiv:2304.07037, 2023

    2023 arXiv

  32. [40]

    Adult content moderation,

    Imagga, “Adult content moderation,” https://imagga.com/solutions/adu lt-content-moderation, 2024, accessed: 2024-08-17

    2024

  33. [41]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PMLR, 2021, pp. 8748–8763

    2021

  34. [42]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational conference on machine learning. PMLR, 2022, pp. 12 888–12 900

    2022

  35. [43]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  36. [44]

    Deepseek- vl: Towards real-world vision-language understanding,

    H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, Y . Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan, “Deepseek- vl: Towards real-world vision-language understanding,” 2024

    2024

  37. [45]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,”arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  38. [46]

    Gpt-4o system card,

    OpenAI, “Gpt-4o system card,” 2024, accessed: 2024-08-20. [Online]. Available: https://openai.com/index/gpt-4o-system-card/

    2024

  39. [47]

    Lion: Empowering multimodal large language model with dual-level visual knowledge,

    G. Chen, L. Shen, R. Shao, X. Deng, and L. Nie, “Lion: Empowering multimodal large language model with dual-level visual knowledge,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 540–26 550

    2024

  40. [48]

    Osprey: Pixel understanding with visual instruction tuning,

    Y . Yuan, W. Li, J. Liu, D. Tang, X. Luo, C. Qin, L. Zhang, and J. Zhu, “Osprey: Pixel understanding with visual instruction tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28 202–28 211

    2024

  41. [49]

    V*: Guided visual search as a core mechanism in multimodal llms,

    P. Wu and S. Xie, “V*: Guided visual search as a core mechanism in multimodal llms,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 084–13 094

    2024

  42. [50]

    Gpt4roi: Instruction tuning large language model on region- of-interest,

    S. Zhang, P. Sun, S. Chen, M. Xiao, W. Shao, W. Zhang, Y . Liu, K. Chen, and P. Luo, “Gpt4roi: Instruction tuning large language model on region- of-interest,”arXiv preprint arXiv:2307.03601, 2023

    2023 arXiv

  43. [51]

    Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning,

    L. Zhao, E. Yu, Z. Ge, J. Yang, H. Wei, H. Zhou, J. Sun, Y . Peng, R. Dong, C. Hanet al., “Chatspot: Bootstrapping multimodal llms via precise referring instruction tuning,”arXiv preprint arXiv:2307.09474, 2023

    2023 arXiv

  44. [52]

    Lyrics: Boosting fine-grained language-vision alignment and comprehension via semantic-aware visual objects,

    J. Lu, R. Gan, D. Zhang, X. Wu, Z. Wu, R. Sun, J. Zhang, P. Zhang, and Y . Song, “Lyrics: Boosting fine-grained language-vision alignment and comprehension via semantic-aware visual objects,”arXiv preprint arXiv:2312.05278, 2023

    2023 arXiv

  45. [53]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhouet al., “Chain-of-thought prompting elicits reasoning in large language models,”Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  46. [54]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  47. [55]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  48. [56]

    Facebook community standards,

    I. Meta Platforms, “Facebook community standards,” https://transpar ency.meta.com/policies/community-standards, 2024, accessed: 2024-12- 26

    2024

  49. [57]

    Community guidelines,

    YouTube, “Community guidelines,” 2024, accessed: 2024-12-26. [Online]. Available: https://www.youtube.com/howyoutubeworks/polici es/community-guidelines/

    2024

  50. [58]

    X rules,

    X. Corporation, “X rules,” https://help.x.com/en/rules-and-policies/x-r ules, 2024, accessed: 2024-12-26

    2024

  51. [59]

    Visual robustness benchmark for visual question answering (vqa),

    M. Farhan Ishmam, I. Tashdeed, T. Asir Saadat, M. Hamjajul Ashmafee, R. M. Kamal, D. Abu, and M. A. Hossain, “Visual robustness benchmark for visual question answering (vqa),”arXiv e-prints, pp. arXiv–2407, 2024

    2024

  52. [60]

    Real-esrgan: Training real- world blind super-resolution with pure synthetic data,

    X. Wang, L. Xie, C. Dong, and Y . Shan, “Real-esrgan: Training real- world blind super-resolution with pure synthetic data,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1905–1914

    2021

  53. [61]

    Open-set image tagging with multi-grained text supervision,

    X. Huang, Y .-J. Huang, Y . Zhang, W. Tian, R. Feng, Y . Zhang, Y . Xie, Y . Li, and L. Zhang, “Open-set image tagging with multi-grained text supervision,”arXiv e-prints, pp. arXiv–2310, 2023

    2023

  54. [62]

    Grounding dino: Marrying dino with grounded pre- training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhuet al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,”arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  55. [63]

    On large language models’ resilience to coercive interrogation,

    Z. Zhang, G. Shen, G. Tao, S. Cheng, and X. Zhang, “On large language models’ resilience to coercive interrogation,” in2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2024, pp. 252– 252

    2024

  56. [64]

    The effect of sampling temperature on prob- lem solving in large language models,

    M. Renze and E. Guven, “The effect of sampling temperature on prob- lem solving in large language models,”arXiv preprint arXiv:2402.05201, 2024

    2024 arXiv

  57. [65]

    Detecting harmful memes and their targets,

    S. Pramanick, D. Dimitrov, R. Mukherjee, S. Sharma, M. S. Akhtar, P. Nakov, and T. Chakraborty, “Detecting harmful memes and their targets,” inFindings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 2783–2796

    2021

  58. [66]

    Protest activity detec- tion and perceived violence estimation from social media images,

    D. Won, Z. C. Steinert-Threlkeld, and J. Joo, “Protest activity detec- tion and perceived violence estimation from social media images,” in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 786–794

    2017

  59. [67]

    old violence dataset,

    H. C. M. university of Technology, “old violence dataset,” https://universe.roboflow.com/ho-chi-minh-university-of-tec hnology-eqlzy/old-violence-0tsy1, nov 2023, visited on 2024-08-20. [Online]. Available: https://universe.roboflow.com/ho-chi-minh-unive rsity-of-technology-eq...

    2023

  60. [68]

    Violent or graphic content,

    YouTube, “Violent or graphic content,” https://support.google.com/you tube/answer/2802008, 2024, accessed: 2024-12-26

    2024

  61. [69]

    What is bullying?

    U. D. of Health and H. Services, “What is bullying?” https://www.stop bullying.gov/bullying/what-is-bullying, 2024, accessed: 2024-12-26

    2024

  62. [70]

    Yahoo open nsfw,

    Y . Inc., “Yahoo open nsfw,” https://github.com/yahoo/open nsfw, 2016, accessed: 2024-08-25

    2016

  63. [71]

    Llama 3: Model cards and prompt formats,

    Meta, “Llama 3: Model cards and prompt formats,” 2024, accessed: 2024-08-26. [Online]. Available: https://llama.meta.com/docs/model-c ards-and-prompt-formats/meta-llama-3

    2024

  64. [72]

    Multitask prompt tuning enables parameter-efficient transfer learning,

    Z. Wang, R. Panda, L. Karlinsky, R. Feris, H. Sun, and Y . Kim, “Multitask prompt tuning enables parameter-efficient transfer learning,” arXiv preprint arXiv:2303.02861, 2023

    2023 arXiv

  65. [73]

    G. Cloud. (2024) Data labeling. Accessed: 2024-12-26. [Online]. Available: https://cloud.google.com/use-cases/data-labeling

    2024

  66. [74]

    Memedroid rules,

    Memedroid, “Memedroid rules,” https://www.memedroid.com/rules, 2024, accessed: 2024-08-25

    2024

  67. [75]

    Enabling contextual soft moderation on social media through contrastive textual deviation,

    P. Paudel, M. H. Saeed, R. Auger, C. Wells, and G. Stringhini, “Enabling contextual soft moderation on social media through contrastive textual deviation,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 4409–4426

    2024

  68. [76]

    The menlo report: Ethical principles guiding information and communication technology research,

    E. Kenneally and D. Dittrich, “The menlo report: Ethical principles guiding information and communication technology research,”Available at SSRN 2445102, 2012

    2012

  69. [77]

    The velocity of censorship:{High-Fidelity}detection of microblog post deletions,

    T. Zhu, D. Phipps, A. Pridgen, J. R. Crandall, and D. S. Wallach, “The velocity of censorship:{High-Fidelity}detection of microblog post deletions,” in22nd USENIX Security Symposium (USENIX Security 13), 2013, pp. 227–240

    2013

  70. [78]

    {CensMon}: A web censorship monitor,

    E. Athanasopoulos, S. Ioannidis, and A. Sfakianakis, “{CensMon}: A web censorship monitor,” inUSENIX Workshop on Free and Open Communications on the Internet (FOCI 11), 2011

    2011

  71. [79]

    Automated detection and fingerprinting of censorship block pages,

    B. Jones, T.-W. Lee, N. Feamster, and P. Gill, “Automated detection and fingerprinting of censorship block pages,” inProceedings of the 2014 Conference on Internet Measurement Conference, 2014, pp. 299–304

    2014

  72. [80]

    Covfefe,

    Wikipedia, “Covfefe,” https://en.wikipedia.org/wiki/Covfefe, 2024, accessed: 2024-08-29. [Online]. Available: https://en.wikipedia.org/w iki/Covfefe APPENDIX A. Annotation Steps The annotation mainly includes the following steps: 1) Define labeling guidelines: our annotation e...

    2024

  73. [81]

    Focus primarily on objective descriptions of objects and characters

  74. [82]

    Include specific names of individuals, events, or back- grounds mentioned in any descriptions

  75. [83]

    Exclude any descriptions of atmosphere, sense, or sug- gestions related to objects, characters, or the overall image

  76. [84]

    If any description indicates a policy violation, state the violated policies directly instead of providing a summary description

  77. [85]

    Format your response as follows: Summary Description: [your description or policy violation] ”Descriptions:{descriptions} Please provide ten descriptions from VLM in parentheses. C. Comprehensive Moderation Prompt The comprehensive prompt designT C for the final compre- hensiv...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.