{"id":"fa2010a6-5359-4c0b-ba41-606b0c7c59b8","arxiv_id":"2412.16446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Vision transformers, particularly LITv2, outperform ResNet baselines on a newly curated 3-class pornography classification dataset, but the evaluation is limited by dataset overlap and tuning issues.","lead":"This paper compares several vision transformer models against older ResNet models for detecting pornographic and 'porn-indicative' images, using newly assembled datasets from Reddit, ImageNet, and existing collections. The authors report that transformer models, especially a variant called LITv2, achieve higher classification accuracy than the ResNet baselines, though the comparisons with commercial and prior tools are on different data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most load-bearing concern: P2 clean test images are sampled from ImageNet (Section IV-A), the same source as the ImageNet-1K checkpoints used for fine-tuning (Table VI); without deduplication, reported test accuracies and the ViT superiority claim may be inflated by pretraining memorization.","rationale":"I read the paper's central claim as the empirical superiority of ViT-family models over ResNet baselines on pornography/clean classification, supported mainly by Table X. The strongest evidence is the >95% test accuracy of fine-tuned ViT/DeiT/Swin on P2 2-class versus 89.5% for ResNet18. This evidence is valid only if the test set is disjoint from pretraining. The paper's own dataset construction makes that disjointness unlikely for the clean class, and the conclusion acknowledges overlap concerns only for ACI. The reader identified the same assumption; I agree. A concrete deduplication check would settle the matter. I am not raising an internal inconsistency: the tables are internally consistent, but the external validity of the test set is the weak point. If deduplication shows no overlap, the concern is resolved and the conditional verdict can stand. If overlap is found, the main quantitative claim needs to be recomputed on a clean subset. Hence no change to the reader's CONDITIONAL verdict.","tokens_in":11904,"tokens_out":5231,"duration_ms":43924,"concrete_test":"Request the authors to release the exact ImageNet image identifiers (or perceptual hashes) used for the clean class of the P2 test set and check them against the ImageNet-1K training split. Then re-run the fine-tuning protocol of Table X on the subset of P2 from which all overlapping or near-duplicate clean test images are removed, and compare the accuracy of ViT-base, DeiT-S, Swinv2-S, and ResNet18. If the transformer advantage narrows or reverses on the deduplicated subset, the Section IV-C claim that ViTs 'surpassed the benchmark pre-trained models' is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A states that neutral/clean images in the P2 dataset are selected from ImageNet, while Table VI lists ViT-base, DeiT-S, and Swinv2-S as pre-trained on ImageNet-1K. Since ImageNet-1K is the standard subset of the full ImageNet and the clean class is human-related (e.g., person classes that are included in ImageNet-1K), there is a high probability that the same or near-duplicate images appear in both the fine-tuning pretraining set and the P2 test set. This violates the assumption that test images are unseen. The Conclusion only acknowledges a possible overlap between ImageNet-1K and ACI, not the more direct P2/ImageNet-1K overlap. The consequence is that the 95%+ test accuracies in Table X may reflect memorization of the clean class rather than generalization. Because the transformer models are larger and have higher capacity, the gap over ResNet18 in the 2-class P2 comparison could be partly an artifact of pretraining leakage, rather than evidence of superior architecture. The separate comparison to AttM-CNN's 92.72% is also not on the same test set, but the P2 leakage is the more load-bearing issue for the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies vision transformer (ViT) classifiers for sensitive-image detection, motivated by CSAM moderation. The authors construct a three-class dataset (P2) with clean, porn-indicative, and pornographic images, and also use the ACI benchmark. They compare fine-tuned ViT-base, DeiT-S, and Swinv2-S models against a fine-tuned ResNet18, as well as against published results of AttM-CNN and the Bumble private-detector model. On the two-class P2 test set, the fine-tuned ViT models report accuracies above 95%, exceeding ResNet18 (89.50%) and the 92.72% figure quoted for AttM-CNN. On the three-class P2 dataset, LITv2 models trained from scratch are reported to outperform Swin and DeiT. The paper concludes that ViT networks surpass pre-trained ResNet models and established detection methods for this task.","tokens_in":12158,"tokens_out":3666,"duration_ms":33331,"significance":"If the central comparison were sound, the paper would provide a useful practical evaluation of transformer backbones for a socially important content-moderation task, and the proposed three-class P2 dataset could be a resource for future work. The internal comparison among transformer variants (LITv2 vs Swin vs DeiT, and the effect of high/low-frequency attention ratios) is informative and appears internally consistent. However, the headline claims against established methods rest on three load-bearing weaknesses: possible train/test overlap with ImageNet-1K pretraining, a Bumble threshold tuned on the test set, and an AttM-CNN accuracy taken from a different dataset. These issues currently leave the main superiority claim unsupported, although they are in principle addressable with a revised experimental protocol.","major_comments":[{"comment":"The clean images in the P2 dataset are sampled from ImageNet (Section IV-A), while the fine-tuned ViT, DeiT, and Swinv2 models use checkpoints pre-trained on ImageNet-1K (Table VI). The paper provides no deduplication or exclusion of P2 test images that may overlap with the pretraining distribution. As the authors partially acknowledge in the Conclusion only for ACI, this overlap can inflate test accuracy through memorization. Because the reported superiority over ResNet18 in Table X is the central claim, the authors must either demonstrate that the P2 clean test images are disjoint from ImageNet-1K, remove overlapping images, or retrain the comparison without pretraining data that contains the test distribution.","section":"Section IV-A, Table VI, Table X"},{"comment":"The Bumble comparison is compromised by test-set threshold selection. The paper states that 'we selected 20% as the boundary threshold between clean and pornography categories after a series of threshold comparisons,' and then reports accuracies of 89.27% and 89.87% on P2 and ACI. Tuning the threshold on the test set and then reporting accuracy on the same test set makes the comparison with Bumble partly circular and overstates the authors' model's advantage. The threshold should be fixed on a validation set or the comparison should be reported as a precision-recall curve over thresholds.","section":"Section IV-C"},{"comment":"The comparison with AttM-CNN is not apples-to-apples. The paper claims that ViT models achieved accuracy greater than 95% on the P2 two-class dataset, 'surpassing the performance of AttM-CNN [6], which achieved an accuracy of 92.72%.' However, the 92.72% figure is reported by Gangwar et al. on their own dataset, not on the P2 or ACI test sets used in this paper. A meaningful comparison requires evaluating AttM-CNN (or a reimplementation) on the same test sets as the proposed models.","section":"Section IV-C"},{"comment":"All reported accuracies are single numbers without error bars, repeated runs, or statistical significance tests. The differences among the fine-tuned transformer models are small (for example, ViT-base 0.9596, DeiT-S 0.9522, Swinv2-S 0.9520 in Table X), and without variance estimates it is unclear whether these differences or even the gap over ResNet18 are reliable. Given the potential ImageNet overlap identified above, the authors should report multiple seeds with means and standard deviations, or otherwise establish that the observed gaps are not noise.","section":"Tables VII-X"}],"minor_comments":[{"comment":"The heading says 'RE-TRAINING FROM CHECKPOINTS PRE-TRAINED ON IMAGENET-1K,' but the described procedure is fine-tuning from checkpoints, not retraining from scratch; please use 'fine-tuning' for clarity.","section":"Table X"},{"comment":"The LITv2 attention-head ratio is described only via the HiLo figure and the statement that α is the ratio of low-frequency heads. Please state explicitly whether the reported values 0.1, 0.4, and 0.9 correspond to α or to the high-frequency fraction in Tables VII and VIII.","section":"Section III-C and Table VII"},{"comment":"The porn-indicative class has 512 training images but 640 validation images; this reversed size ratio is unusual and should be explained or corrected, as it may affect the reported validation accuracy for the three-class task.","section":"Table III"},{"comment":"The phrase 'despite the occurrence of overfitting' is asserted without supporting training/validation curves or regularization details; please either provide evidence of overfitting or rephrase.","section":"Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is an empirical comparison and a new three-class dataset. The novelty is modest, but the practical relevance is clear. The suggested revisions (deduplication against ImageNet-1K, fixing the Bumble threshold on a validation set, and evaluating AttM-CNN on the same test sets) are feasible in principle and would determine whether the central claim survives. I would not recommend rejection at this stage, but the current version's headline claims are not supported by the reported experimental design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThis is a reasonable empirical paper about applying vision transformers to pornography detection, but its central quantitative claims are undermined by a test-set leakage problem that the authors themselves half-notice. The genuinely new thing is a small three-class dataset ('porn-indicative' as a middle category) and a comparison across ViT, DeiT, Swin, LITv2 versus ResNet18. That is useful, if modest. The internal transformer comparisons are plausible and the paper does not oversell the from-scratch results.\n\nThe soft spots are real and some are load-bearing. First, the clean test images in P2 are sampled from ImageNet (Section IV-A), and the fine-tuned checkpoints are pre-trained on ImageNet-1K (Table VI). The conclusion only worries about ImageNet-1K vs ACI overlap, but the more direct problem is P2-clean vs ImageNet pretraining. Without deduplication or a leakage analysis, the >95% test accuracies on 2-class P2 may partly reflect memorization, and the ViT-vs-ResNet gap could be an artifact of larger capacity memorizing better. This is the main issue. Second, the Bumble comparison tunes the 20% threshold on the test data and then reports that accuracy; that is circular. Third, AttM-CNN's 92.72% comes from a different dataset, so the claim to surpass established methods is not supported by a same-protocol comparison. No error bars or repeated runs, either.\n\nThat said, the internal comparison among transformer variants (LITv2 vs Swin vs DeiT on the 3-class task) seems honest, and the limited-data findings (porn-indicative accuracy below 50% for most models) are worth knowing. The authors also clearly flag some limitations. The paper is not a wreck; it just overclaims in the abstract and comparisons.\n\nWho is this for? Anyone working on content moderation benchmarks or CSAM-adjacent classification who wants a starting point on dataset construction and model selection. It deserves a serious referee, but acceptance should be conditional on addressing leakage (dedup or separate clean set), same-protocol baselines, and error bars.","headline":"Useful empirical comparison of ViTs for pornography classification, but the reported gains over prior methods are not trustworthy because the clean test images come from ImageNet, the same pool as the fine-tuning checkpoints.","tokens_in":12671,"tokens_out":2927,"would_cite":false,"duration_ms":24365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision transformers surpass CNNs on sensitive-image classification benchmarks.","keywords":["vision transformer","pornography classification","CSAM detection","self-attention","HiLo attention","image classification","content moderation","benchmark dataset"],"falsifier":"Run a near-duplicate search between the P2 clean test images (and the ACI test set) and the ImageNet-1K training set used for pre-training. If a substantial fraction of test images have near-duplicates in the pre-training set, the fine-tuned transformer accuracy is inflated and the comparison against ResNet18 and AttM-CNN is not fair; without such overlap, the claim stands.","tokens_in":11715,"feed_emoji":"🛡️","tokens_out":4629,"duration_ms":34993,"temperature":0.7,"pith_summary":"The paper asks whether vision transformer backbones can classify pornographic and porn-indicative images better than the convolutional networks that have dominated this task. It builds two new benchmark datasets, a two-class and a three-class version, with the third class covering images that are suggestive but not explicit, and evaluates ViT, DeiT, Swin, and LITv2 models against pre-trained ResNet18 and published CNN methods. The central claim is that fine-tuned transformer models exceed 95% test accuracy on the two-class dataset, surpassing the 92.72% reported for the AttM-CNN method, and that LITv2 models, which separate high- and low-frequency attention, generalize best on the harder three-class task. If true, this gives content-moderation and law-enforcement systems a concrete accuracy gain by swapping ResNet backbones for transformer encoders.","feed_headline":"Vision transformers beat CNNs on sensitive-image tests","feed_subtitle":"Fine-tuned ViT, DeiT, Swin, and LITv2 top 95% accuracy on a two-class pornography benchmark, surpassing prior CNN methods.","key_machinery":"The load-bearing mechanism is the multi-head self-attention of vision transformers, which lets the model relate image patches across the whole image instead of through local receptive fields. LITv2's HiLo attention sharpens this by splitting attention heads into high-frequency heads for local fine detail and low-frequency heads for global structure; the paper argues this balance is why LITv2 models perform best on the three-class data. The pre-trained checkpoints from ImageNet-1K supply the starting features, and the fine-tuning protocol on each dataset is what produces the reported gains.","core_discovery":"On the paper's own terms, the discovery is that self-attention-based vision transformers are better suited to pornography and porn-indicative image classification than the pre-trained ResNet18 baseline and established CNN-based detectors. Fine-tuned ViT-base, DeiT-S, and Swinv2-S each reach above 95% accuracy on the two-class P2 test set, outperforming AttM-CNN's 92.72%; on the three-class P2 set, which adds an ambiguous 'porn-indicative' category, the same models maintain roughly 89-90% accuracy while ResNet18 drops to 78.75%. The paper also finds that LITv2 models trained from scratch approach the accuracy of ImageNet-pretrained transformers on the two main classes, and that a LITv2-medium with a 0.9 attention-head ratio reaches 84.86% on the three-class validation set, the best among the from-scratch models.","pith_inferences":["A direct test of the overlap concern would be to compare model accuracy on clean P2 test images that are confirmed unique versus those with near-duplicates in ImageNet-1K; the gap would quantify how much of the transformer advantage is memorization.","The same three-class formulation could be extended to video frames or to finer CSAM severity gradations, where the porn-indicative category is even more loaded.","The attention-map visualizations the authors plan could reveal whether transformer errors on porn-indicative images stem from missing local cues or from global context, which would guide whether high-frequency head ratios should be tuned per dataset.","Because the P2 datasets are small and imbalanced in the indicative class, the reported three-class numbers likely understate real-world performance on rare categories; balancing or synthetic augmentation would be a natural next probe."],"forward_implications":["Content-moderation systems can expect a practical accuracy gain by replacing ResNet18 backbones with fine-tuned ViT, DeiT, Swin, or LITv2 encoders.","The three-class formulation with a porn-indicative category is workable for transformer models, which keep roughly 89-90% accuracy on the two main classes when the ambiguous class is added.","LITv2-style attention that mixes local high-frequency and global low-frequency information gives better three-class performance than pure ViT or Swin-style attention.","Larger transformer variants with more parameters improve accuracy when compute is available.","Fine-tuning from ImageNet-1K checkpoints is substantially better than training from scratch for this task."],"supporting_citations":[{"why":"Supplies the ACI benchmark dataset used for comparison.","marker":"[4]"},{"why":"Supplies the public P2 dataset that the paper reorganizes into 2-class and 3-class P2.","marker":"[5]"},{"why":"Defines the AttM-CNN baseline whose 92.72% accuracy the transformers surpass.","marker":"[6]"},{"why":"Provides the pre-trained ResNet18 approach used as the main convolutional baseline.","marker":"[13]"},{"why":"Introduces the ViT architecture the paper fine-tunes.","marker":"[30]"},{"why":"Introduces the Swin transformer with shifted windows.","marker":"[31]"},{"why":"Introduces LITv2 HiLo attention, the mechanism behind the best from-scratch results.","marker":"[32]"},{"why":"Provides the state-of-the-art nudity-classification comparison the paper relates its accuracy to.","marker":"[33]"},{"why":"Supplies the Bumble commercial moderation model used as a practical comparison.","marker":"[34]"}],"fun_headline_variants":["Vision transformers beat CNNs on sensitive-image tests","ViT and DeiT surpass CNN detectors on porn benchmarks","Self-attention models top 95% on two-class pornography test","From-scratch LITv2 matches pretrained transformers on sensitive data","Sensitive image classification: Transformers top CNN baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the clean test images taken from ImageNet do not overlap with the ImageNet-1K images used to pre-train the transformer models, so the reported accuracy reflects genuine generalization rather than memory.","fun_headline_variants_meta":{"raw":{"variants":["Vision transformers beat CNNs on sensitive-image tests","ViT and DeiT surpass CNN detectors on porn benchmarks","Self-attention models top 95% on two-class pornography test","From-scratch LITv2 matches pretrained transformers on sensitive data","Sensitive image classification: Transformers top CNN baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":4028,"prompt_tokens":945,"completion_tokens":3083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":2998}},"tokens_in":561,"tokens_out":3083,"duration_ms":19484,"temperature":1.0,"reasoning_tokens":2998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:33:54.027394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a near-duplicate search between the P2 clean test images (and the ACI test set) and the ImageNet-1K training set used for pre-training. If a substantial fraction of test images have near-duplicates in the pre-training set, the fine-tuned transformer accuracy is inflated and the comparison against ResNet18 and AttM-CNN is not fair; without such overlap, the claim stands.","supporting_citations":[{"cited_title":"Smart content recognition from images using a mixture of convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the ACI benchmark dataset used for comparison."},{"cited_title":"20k nudity dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the public P2 dataset that the paper reorganizes into 2-class and 3-class P2."},{"cited_title":"AttM- CNN: Attention and metric learning based CNN for pornography, age and child sexual abuse (CSA) detection in images,","cited_arxiv_id":null,"evidence_quote":"Defines the AttM-CNN baseline whose 92.72% accuracy the transformers surpass."},{"cited_title":"Laying foundations for effective machine learning in law enforce- ment. Majura – A labelling schema for child exploitation materials,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained ResNet18 approach used as the main convolutional baseline."},{"cited_title":"Fast vision transformers with HiLo at- tention,","cited_arxiv_id":null,"evidence_quote":"Introduces LITv2 HiLo attention, the mechanism behind the best from-scratch results."},{"cited_title":"State-of-the-art in nudity classification: A comparative analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art nudity-classification comparison the paper relates its accuracy to."},{"cited_title":"The Bumble’s private detector model","cited_arxiv_id":null,"evidence_quote":"Supplies the Bumble commercial moderation model used as a practical comparison."}],"review_version":1}