Pith. sign in

REVIEW 3 major objections 6 minor 105 references

SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Camera settings and lighting, not just scene content, drive what vision models see: all 52 tested models lose accuracy on under- and over-exposed images, and nearly identical photos taken with slightly different settings flip their answers.

desk verdict Solid controlled benchmark and exposure-level findings, but the 'sensitivity to minute settings' claim is undercut by a metric that reduces to below-chance accuracy. read the letter →

arxiv 2505.15628 v1 pith:O35ITQVQ submitted 2025-05-21 cs.CV

classification cs.CV
keywords captureconditionscameraparametersexposurevaluedatasetbiasimageclassificationobjectdetectionvisualquestionansweringhumanbaseline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Camera settings and lighting, not just scene content, are a first-order driver of what deep-learning vision models see. The paper supports this with a new benchmark, SNAP (Shutter speed, ISO seNsitivity, and APerture): 100 scenes of everyday objects photographed with densely sampled camera parameters under two controlled lighting levels, re-indexed by exposure offset from the camera's own well-exposed setting. Across 23 image classifiers, 16 object detectors, and 13 vision-language models, accuracy peaks on well-exposed images and falls off asymmetrically on under- and over-exposed ones. All models also flip their answers on photos that look essentially identical but were taken with slightly different settings, and no VQA model equals peak human accuracy on well-exposed images. An analysis of 13 common training datasets finds their images concentrated in a narrow band of auto-exposure settings, which the authors call a significant capture bias.

What carries the argument

SNAP is the load-bearing object: a dataset of 37,558 images of 10 everyday object categories photographed with a Canon DSLR at one-stop intervals across shutter speeds 1/4000 to 30 seconds, ISO 100 to 6400, and F-numbers 5.6 to 22, under 1000 lux and 10 lux illumination, with lighting, white balance, and focal length fixed. Two derived quantities carry the analysis. The EV offset re-indexes each image by exposure relative to the camera's auto setting, using $EV = \log_2(F^2/t) - \log_2(\mathrm{ISO}/100)$ per illumination condition, so that 0 means well-exposed and each step means doubled or halved light; this turns exposure equivalence (many parameter triplets producing the same light) into a single axis. The parameter sensitivity metric then measures, within sets of same-scene, same-EV-offset images, the percentage of sets whose coefficient of variation exceeds 1, quantifying instability from nearly imperceptible settings changes.

What would settle it

Recover capture metadata for images currently missing Exif tags in one analyzed dataset, for example by re-fetching a random sample of OpenImages' Flickr sources and extracting F-number, ISO, and shutter speed, and compare the EV-offset distribution with the tagged 0.1% subset. If the untagged images turn out to be dominated by edited, synthetic, or camera-unknown images with different exposure statistics, the claim that computer vision datasets are significantly biased by capture conditions would hold only for camera-origin photos. A second test: digitally re-expose well-exposed SNAP images to the same EV offsets; if the accuracy curves do not reproduce, the effect belongs to the physical capture pipeline (sensor noise, tone mapping, JPEG compression) rather than to pixel brightness alone.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that capture conditions constitute a measurable, systematic source of error that current benchmarks ignore. Models are susceptible to image perturbations caused by slight variations of camera parameters: even within a fixed exposure level, sets of near-identical images yield inconsistent predictions, with parameter sensitivity exceeding 20% for some classifiers. Exposure level itself acts as a global performance curve: peak top-1 accuracy lies between EV offset $-2$ and $+2$, with under-exposure hurting classifiers more than over-exposure, while object detection fails hardest on over-exposed images and is dominated by misclassification errors inherited from ImageNet-pretrained backbones. The paper further claims that common vision datasets are significantly biased: across 13 datasets with over a billion images combined, F-number, ISO, and shutter-speed distributions are long-tailed with peaks at a few one-stop values, most photos were taken on auto, and capture dates cluster in the past, and this bias propagates into downstream tasks. Finally, on the VQA task, humans and machines separate in a specific way: four VLMs match or beat average human accuracy, yet no model reaches the 89% human peak on well-exposed images, while humans collapse on heavily under-exposed images that models still partially read.

Load-bearing premise

The whole dataset-bias argument rests on the small fraction of images that carry Exif tags for shutter speed, F-number, and ISO, only 3.3% of the 1.3 billion images overall and as little as 0.1% for OpenImages, being representative of what those datasets actually contain.

Editorial extensions

If this is right

  • Model accuracy reported on standard benchmarks is partly a statement about the camera settings in the training data, so capture-condition coverage should be reported alongside accuracy.
  • Exposure-robustness curves become a reproducible diagnostic: every model's SNAP curve is unimodal around EV offset 0 with asymmetric tails, so new models can be compared on that axis without re-running the full benchmark.
  • Object-detection failures under extreme exposure are mostly misclassifications inherited from ImageNet-pretrained backbones, meaning fixes to the backbone, not the detection head, are the lever.
  • Scaling data and parameters helps but does not cure the problem: the largest CLIP-pretrained models still waver on near-identical images and none matches the human peak on well-exposed VQA images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The natural experiment the paper does not run is a treatment group: fine-tune models on SNAP-style capture-balanced data and remeasure parameter sensitivity; the paper's own analysis of LAION's exposure diversity predicts sensitivity would shrink, which would turn the diagnostic into a training recipe.
  • If digitally re-exposing well-exposed SNAP images to the same EV offsets reproduces the accuracy curves, exposure robustness can be built cheaply as a data-augmentation layer; if it does not, camera-in-the-loop collection like SNAP is the only route.
  • The human-machine asymmetry on over-exposed images suggests models' high-level features are more sensitive to clipped highlights than human perception is, which points toward explicit tone-mapping or exposure-normalization modules as the targeted fix.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SNAP, a new image dataset collected with a DSLR under controlled lighting, densely sampling shutter speed, ISO, and F-number at 1-stop intervals across 100 scenes from 10 object categories. The authors evaluate 23 image classifiers, 16 object detectors, and 13 vision-language models, and run a 43-subject human VQA study with brief masked presentation. They also analyze Exif metadata from 13 large vision datasets and report systematic capture bias. The main reported findings are that model accuracy drops on under- and over-exposed images relative to well-exposed images, that models are sensitive to "minute variations" of camera parameters at matched exposure, and that current training datasets under-represent capture diversity.

Significance. If the exposure-level findings hold, SNAP is a valuable benchmark: it provides real-camera parameter coverage at a density not available in existing robustness datasets, releases code and data, and evaluates a broad model zoo across three tasks with a human baseline. The consistent peak at well-exposed EV offsets across tasks is a useful, falsifiable observation, and the manual collection effort is substantial. However, the matched-exposure sensitivity claim rests on a statistically invalid metric, and the dataset-bias conclusion is extrapolated from very small Exif subsets; both need re-analysis before the paper's central claims can be accepted.

major comments (3)
  1. [3.3 (Parameter sensitivity)] For the binary per-image metrics used in image classification and VQA hard accuracy, the coefficient of variation is a deterministic function of the mean: for a set with mean accuracy p, CV = sqrt((1-p)/p), so the condition CV > 1 is exactly p < 0.5. Therefore PS is the percentage of (scene, EV-offset) sets in which the model's accuracy is below 50%, not a measure of how much predictions fluctuate across camera settings. This directly affects the matched-exposure claims in Section 4.1 ("All models are susceptible to image perturbations caused by slight variations of camera parameters"), Section 4.3 ("All VLMs are sensitive to camera parameter variations"), and the abstract's "minute variations" statement. A set in which the model is wrong three times out of five contributes to PS, while a set whose per-image outcomes alternate substantially around a mean of 0.55 does not. Please replace PS with a metric that actually compares per-image outcomes across different parameter settings within a scene (for example, the variance of per-image accuracy across settings, or a permutation test against within-setting variability), or remove the matched-exposure conclusion.
  2. [3.1 / Table 1] The capture-bias analysis is performed on Exif-tagged subsets whose coverage varies from 61.6% (YFCC15M) down to 0.1% (OpenImages, Wukong) and 1.4% (LAION400M). The paper states in Section 3.1 that "Overall, we found significant biases across all datasets" and uses the resulting distributions in Figs. 1 and A.4 to support the abstract's claim that "computer vision datasets are significantly biased." No evidence is provided that images with all three Exif tags are representative of the full dataset; for Common Crawl-derived corpora, images without metadata may be screenshots, stripped web images, or non-camera files. Please either supply a representativeness check (for example, comparing a labeled sample of images without Exif on content or source statistics) or explicitly restrict the dataset-bias conclusions to the Exif-tagged subpopulations.
  3. [3.4 / 4.3 (Human experiment)] The human baseline used 200 ms image presentation followed by a mask, while the VLMs were given the full image with no time limit and could generate lengthy answers. The paper uses this baseline to state that "4 models reach or surpass the average accuracy of human subjects" (Section 4.3) and that humans peak at 89% on well-exposed images. The time-limited protocol may substantially lower human accuracy, especially on under-exposed images, so the model-vs-human comparisons are not apples-to-apples. Since the VLMs are not feedforward and are not subject to the same presentation constraint, the claim that models "reach or surpass" humans should be rephrased as "surpass a brief-presentation human baseline," or additional human data without the time limit should be collected. The conclusion that models do not reach human accuracy on well-exposed images is conservative under this mismatch, but the "surpass human" statements are not.
minor comments (6)
  1. [3.2.1] The text says "We captured 10 scenes with 2–5 objects from each object category," but Appendix B states there are 100 unique scenes (10 per category); please clarify the wording.
  2. [4.3 / Appendix F.4] Question numbering is inconsistent: Section 3.2.2 defines Q3 as multiple-choice categorization and Q4 as multiple-choice counting, but Section 4.3 refers to "MC categorization (Q5)" and Fig. F.4(d) is labeled "Q4: Categorization, multiple-choice." Please renumber or re-label consistently.
  3. [4.2] The text says "SP is highest for the oLRP FN component" and should read "PS is highest."
  4. [4.1 / Fig. 4] The sentence "This is evident from high parameter sensitivity (PS) w.r.t. top-1 accuracy that reaches over 20% some models (Fig. 4)" appears to cite the wrong figure; Fig. 4 shows mean top-1 accuracy versus EV offset, while the PS values are the blue circles in Fig. 3.
  5. [References] Reference [6] for OpenImages v7 points to a semantic segmentation paper (Benenson and Ferrari); please cite the actual OpenImages dataset.
  6. [3 (Methodology)] The opening line "This session discusses" should be "This section discusses."

Circularity Check

1 steps flagged · score 6.0 of 10

The 'parameter sensitivity' metric (PS) reduces, for binary metrics, to the fraction of sets with below-50% accuracy, so the central claim of sensitivity to minute camera variations is an artifact of the metric definition; the paper's other empirical findings remain self-contained.

  1. self definitional [Section 3.3, 'Parameter sensitivity (PS)' definition; applied in Section 4.1 to claim 'All models are susceptible...']
    "Parameter sensitivity (PS) measures sensitivity of the metrics to camera parameters... to measure fluctuations in models’ results within each set, we compute coefficient of variation (CV), defined as the ratio of the std to mean (CV = σ µ) of the metric; ... we compute PS as the percentage of sets with CV > 1. ... All models are susceptible to image perturbations caused by slight variations of camera parameters. This is evident from high parameter sensitivity (PS)... Even the top-5 performing models inconsistently classify nearly 10% of the scenes that look essentially the same."

    For top-1 classification accuracy and VQA hard accuracy, each image contributes a binary 0/1. For a set with mean p, σ = sqrt(p(1−p)), so CV = sqrt((1−p)/p). Therefore CV > 1 if and only if p < 0.5. By construction, PS is the percentage of same-scene/same-EV-offset sets whose mean accuracy is below 50%, not the percentage of sets whose results fluctuate across camera parameter combinations. A set with p=0.1 (mostly wrong) counts as 'sensitive', while a set with p=0.8 and genuinely mixed success does not. Hence the conclusion that models are susceptible to 'minute variations of camera settings' and 'inconsistently classify' scenes reduces, through the metric definition, to below-50% mean accuracy; the threshold of 1 has no statistical justification.

full rationale

The paper is an empirical benchmark rather than a derivation. The capture-bias analysis (Section 3.1) is a direct measurement of Exif metadata using the standard EV formula; the SNAP dataset is collected and annotated independently; the model evaluations and the human study are external measurements. No fitted parameter is renamed as a prediction, and the few self-citations (e.g., [2,86,93]) are background related work, not load-bearing. The one significant definitional circularity is the PS metric: because top-1 and hard accuracy are binary, CV collapses to a deterministic function of the mean, so the CV>1 threshold selects exactly the sets with accuracy below 50%. The paper's headline claim about sensitivity to 'minute variations of camera settings' rests on this PS value and therefore reduces by construction to a statement about low mean accuracy, not about variability across camera parameters. The major-exposure results (accuracy vs EV offset in Figs. 4, 6, 9) are independent empirical evidence and are not affected by this issue, which is why the overall circularity is partial rather than total.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's central claims rest on three assumptions: representativeness of Exif subsets for the bias analysis, correctness of the standard EV formula as a grouping device, and the interpretation that same-EV images are visually equivalent enough that performance fluctuations indicate sensitivity to camera settings. No free parameters are fitted to model outputs; the PS threshold is a hand-chosen metric constant.

free parameters (1)
  • Parameter sensitivity CV threshold = 1.0
    PS is defined as the percentage of EV-offset bins with coefficient of variation > 1. The threshold 1.0 is chosen without justification and the resulting 'sensitivity to minute variations' claim depends on it.
assumptions (3)
  • domain assumption The subset of images with complete Exif metadata is representative of each analyzed dataset's capture-condition distribution.
    Used in Section 3.1 to draw conclusions about capture bias in 13 datasets; metadata coverage ranges from 0.1% to 61.6%, and images with Exif may systematically differ from those without.
  • standard math The standard exposure value formula EV = log2(F^2/t) - log2(ISO/100) plus the auto-mode reference correctly identifies well-exposed images independent of illumination.
    Invoked in Section 3.2.1 and Appendix B to bin images into EV offsets and to define 'best exposure' via the camera's auto settings for each light level.
  • domain assumption Images with the same EV offset and lighting 'look essentially the same' to models, so accuracy variation within such bins is attributable to minute camera-setting differences rather than visible scene differences.
    Underlies the parameter-sensitivity analysis in Section 3.3 and the conclusion in Section 4.1 that all models are susceptible to slight camera parameter variations; aperture and ISO changes also alter depth of field and noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks." pith.science (2026). https://pith.science/paper/O35ITQVQ

@misc{pith2026250515628,
  author       = {Pith},
  title        = {Pith review of: SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O35ITQVQ}},
  note         = {Machine review of arXiv:2505.15628}
}
abstract

Generalization of deep-learning-based (DL) computer vision algorithms to various image perturbations is hard to establish and remains an active area of research. The majority of past analyses focused on the images already captured, whereas effects of the image formation pipeline and environment are less studied. In this paper, we address this issue by analyzing the impact of capture conditions, such as camera parameters and lighting, on DL model performance on 3 vision tasks -- image classification, object detection, and visual question answering (VQA). To this end, we assess capture bias in common vision datasets and create a new benchmark, SNAP (for $\textbf{S}$hutter speed, ISO se$\textbf{N}$sitivity, and $\textbf{AP}$erture), consisting of images of objects taken under controlled lighting conditions and with densely sampled camera settings. We then evaluate a large number of DL vision models and show the effects of capture conditions on each selected vision task. Lastly, we conduct an experiment to establish a human baseline for the VQA task. Our results show that computer vision datasets are significantly biased, the models trained on this data do not reach human accuracy even on the well-exposed images, and are susceptible to both major exposure changes and minute variations of camera settings. Code and data can be found at https://github.com/ykotseruba/SNAP

Figures

Figures reproduced from arXiv: 2505.15628 by the authors.

Figure 1
Figure 1. Normalized distribution of camera settings [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Data collection setup consists of a Canon EOS Rebel T7 camera on the tripod, a table covered with non-reflective cloth back￾drop, and LED lights on either side. Object categories. Dense sampling of camera param￾eters is time-consuming to collect. Thus to make the data suitable for all three vision tasks, we chose object categories present in both ImageNet and COCO—the most common datasets for training image classifi… view at source ↗
Figure 3
Figure 3. Box plots show range and mean top-1 accuracy values [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Image classification top-1 (%) across exposure [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Box plots show range and mean oLRP values for [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Object detection at different exposure levels in SNAP. Each point is a mean oLRP of the model predictions on a given EV offset. Model labels are sorted by oLRP from worst to best) The majority of existing object detection models are pre-trained on ImageNet and COCO, th…
Figure 7
Figure 7. Figure 7: Mean hard accuracy (green) and soft accuracy (blue) on all questions for models and human subject accuracy (red line). VLMs are comparable to humans in average accuracy across all questions. We first assess the average performance of the tested VLMs against the human s…
Figure 8
Figure 8. Figure 8: For example, PaliGemma, VILA, and DeepSeekVL-7B almost never deviate from the answer [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Mean hard accuracy on all questions and exposures in SNAP for VLMs and humans. The low average performance of the human sub￾jects is mainly due to sharp drop-off on under￾exposed images (EV offset < -3). These im￾ages appear mostly black so the performance of the human…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

105 extracted references · 50 canonical work pages

  1. [1]

    http://www.gphoto.com/

    gPhoto2. http://www.gphoto.com/

  2. [2]

    On sensor bias in experimental methods for comparing interest-point, saliency, and recognition algorithms

    Alexander Andreopoulos and John K Tsotsos. On sensor bias in experimental methods for comparing interest-point, saliency, and recognition algorithms. TPAMI, 34(1):110–126, 2011

  3. [3]

    Why do deep convolutional networks generalize so poorly to small image transformations? arXiv:1805.12177, 2018

    Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv:1805.12177, 2018

  4. [4]

    Unexplored faces of robustness and out-of-distribution: Covariate shifts in environment and sensor domains

    Eunsu Baek, Keondo Park, Jiyoon Kim, and Hyung-Sin Kim. Unexplored faces of robustness and out-of-distribution: Covariate shifts in environment and sensor domains. In CVPR, 2024

  5. [5]

    ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

    Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019

  6. [6]

    From colouring-in to pointillism: revisiting semantic segmentation supervision

    Rodrigo Benenson and Vittorio Ferrari. From colouring-in to pointillism: revisiting semantic segmentation supervision. arXiv:2210.14142, 2022

  7. [7]

    Are we done with ImageNet? arXiv:2006.07159, 2020

    Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? arXiv:2006.07159, 2020

  8. [8]

    PaliGemma: A versatile 3B VLM for transfer

    Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaliGemma: A versatile 3B VLM for transfer. arXiv:2407.07726, 2024

Show all 105 references
  1. [9]

    A digital image processing pipeline for modelling of realistic noise in synthetic images

    Oleksandra Bielova, Ronny Hansch, Andreas Ley, and Olaf Hellwich. A digital image processing pipeline for modelling of realistic noise in synthetic images. In CVPRW, 2019

  2. [10]

    YOLOv4: Optimal speed and accuracy of object detection

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection. arXiv:2004.10934, 2020

  3. [11]

    COYO-700M: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. COYO-700M: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset, 2022

  4. [12]

    End-to-end object detection with Transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In ECCV, 2020

  5. [13]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021

  6. [14]

    On the generalization mystery in deep learning

    Satrajit Chatterjee and Piotr Zielinski. On the generalization mystery in deep learning. arXiv:2203.10036, 2022

  7. [15]

    GazeGAN: A generative adversarial saliency model based on invariance analysis of human gaze during scene free viewing

    Zhaohui Che, Ali Borji, Guangtao Zhai, Xiongkuo Min, Guodong Guo, and Patrick Le Callet. GazeGAN: A generative adversarial saliency model based on invariance analysis of human gaze during scene free viewing. arXiv:1905.06803, 2019

  8. [16]

    Benchmarking robustness of adaptation methods on pre-trained vision-language models

    Shuo Chen, Jindong Gu, Zhen Han, Yunpu Ma, Philip Torr, and V olker Tresp. Benchmarking robustness of adaptation methods on pre-trained vision-language models. In NeurIPS, 2023

  9. [17]

    PaLI: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. PaLI: A jointly-scaled multilingual language-image model. In ICLR, 2023

  10. [18]

    Reproducible scaling laws for contrastive language-image learning

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023

  11. [19]

    ImageNet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009

  12. [20]

    Understanding how image quality affects deep neural networks

    Samuel Dodge and Lina Karam. Understanding how image quality affects deep neural networks. In QoMEX, 2016

  13. [21]

    A study and comparison of human and deep learning recognition performance under visual distortions

    Samuel Dodge and Lina Karam. A study and comparison of human and deep learning recognition performance under visual distortions. In ICCCN, 2017. 10

  14. [22]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021

  15. [23]

    In search of robust measures of generalization

    Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M Roy. In search of robust measures of generalization. In NeurIPS, 2020

  16. [24]

    The Pascal Visual Object Classes (VOC) Challenge

    Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes (VOC) Challenge. IJCV, 88:303–338, 2010

  17. [25]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR, 2024

  18. [26]

    Yolov3: An incremental improvement

    Ali Farhadi and Joseph Redmon. Yolov3: An incremental improvement. arXiv:1804.02767, 2018

  19. [27]

    Shortcut learning in deep neural networks

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020

  20. [28]

    Generalisation in humans and deep neural networks

    Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In NeurIPS, 2018

  21. [29]

    Fast R-CNN

    Ross Girshick. Fast R-CNN. In ICCV, 2015

  22. [30]

    Truth or backpropaganda? an empirical investigation of deep learning theory

    Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth or backpropaganda? an empirical investigation of deep learning theory. In ICLR, 2020

  23. [31]

    Strengths and weaknesses of deep learning models for face recognition against image degradations

    Klemen Grm, Vitomir Štruc, Anais Artiges, Matthieu Caron, and Hazım K Ekenel. Strengths and weaknesses of deep learning models for face recognition against image degradations. IET Biometrics, 7(1):81–89, 2017

  24. [32]

    Wukong: A 100 million large-scale Chinese cross-modal pre-training benchmark

    Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale Chinese cross-modal pre-training benchmark. In NeurIPS, 2022

  25. [33]

    ExifTool

    Phil Harvey. ExifTool. https://exiftool.org/, 2016

  26. [34]

    Recent advances in deep learning theory

    Fengxiang He and Dacheng Tao. Recent advances in deep learning theory. arXiv:2012.10931, 2020

  27. [35]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022

  28. [36]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016

  29. [37]

    Benchmarking neural network robustness to common corruptions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019

  30. [38]

    Natural adversarial examples

    Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021

  31. [39]

    Scene recognition with CNNs: Objects, scales and dataset bias

    Luis Herranz, Shuqiang Jiang, and Xiangyang Li. Scene recognition with CNNs: Objects, scales and dataset bias. In CVPR, 2016

  32. [40]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Info...

  33. [41]

    Visual Genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32–73, 2017

  34. [42]

    Robustness and repeatability of saliency models subjected to visual degradations

    Olivier Le Meur. Robustness and repeatability of saliency models subjected to visual degradations. In ICIP, 2011

  35. [43]

    Can multiple-choice questions really be useful in detecting the abilities of llms? In LREC-COLING, 2024

    Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful in detecting the abilities of llms? In LREC-COLING, 2024. 11

  36. [44]

    Exploring plain vision transformer backbones for object detection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022

  37. [45]

    VILA: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On pre-training for visual language models. In CVPR, 2024

  38. [46]

    Microsoft COCO: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014

  39. [47]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024

  40. [48]

    Evaluation of LBP and deep texture descriptors with a new robustness benchmark

    Li Liu, Paul Fieguth, Xiaogang Wang, Matti Pietikäinen, and Dewen Hu. Evaluation of LBP and deep texture descriptors with a new robustness benchmark. In ECCV, 2016

  41. [49]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In ECCV, 2024

  42. [50]

    SSD: Single shot multibox detector

    Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. SSD: Single shot multibox detector. In ECCV, 2016

  43. [51]

    Swin Transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021

  44. [52]

    A decade’s battle on dataset bias: Are we there yet? In ICLR, 2025

    Zhuang Liu and Kaiming He. A decade’s battle on dataset bias: Are we there yet? In ICLR, 2025

  45. [53]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022

  46. [54]

    DeepSeek-VL: Towards real-world vision-language understand- ing

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. DeepSeek-VL: Towards real-world vision-language understand- ing. arXiv:2403.05525, 2024

  47. [55]

    Benchmarking robustness in object detection: Autonomous driving when winter is coming

    Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv:1907.07484, 2019

  48. [56]

    Simple open-vocabulary object detection

    Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In ECCV, 2022

  49. [57]

    MNIST-C: A robustness benchmark for computer vision

    Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. arXiv:1906.02337, 2019

  50. [58]

    Uniform convergence may be unable to explain generalization in deep learning

    Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In NeurIPS, 2019

  51. [59]

    Evaluation of visual saliency analysis algorithms in noisy images

    Yuzhen Niu, Lingling Ke, and Wenzhong Guo. Evaluation of visual saliency analysis algorithms in noisy images. Machine Vision and Applications, 27(6):915–927, 2016

  52. [60]

    Localization recall precision (LRP): A new performance metric for object detection

    Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Localization recall precision (LRP): A new performance metric for object detection. In ECCV, 2018

  53. [61]

    One metric to measure them all: Localisation recall precision (LRP) for evaluating visual detection tasks

    Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. One metric to measure them all: Localisation recall precision (LRP) for evaluating visual detection tasks. TPAMI, 44(12):9446–9463, 2021

  54. [62]

    Im2Text: Describing images using 1 million captioned photographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using 1 million captioned photographs. In NeurIPS, 2011

  55. [63]

    NMS strikes back

    Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, and Philipp Krähenbühl. NMS strikes back. arXiv:2212.06137, 2022

  56. [64]

    PsychoPy2: Experiments in behavior made easy

    Jonathan Peirce, Jeremy R Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51:195–203, 2019. 12

  57. [65]

    Dataset issues in object recognition

    Jean Ponce, Tamara L Berg, Mark Everingham, David A Forsyth, Martial Hebert, Svetlana Lazebnik, Marcin Marszalek, Cordelia Schmid, Bryan C Russell, Antonio Torralba, C Williams, and A Zisserman. Dataset issues in object recognition. In Toward Category-level Object Recognition,...

  58. [66]

    Basics Photography 07: Exposure

    David Präkel. Basics Photography 07: Exposure . A V A Publishing, 2009

  59. [67]

    Dataset growth

    Ziheng Qin, Zhaopan Xu, Yukun Zhou, Zangwei Zheng, Zebang Cheng, Hao Tang, Lei Shang, Baigui Sun, Xiaojiang Peng, Radu Timofte, et al. Dataset growth. In ECCV, 2024

  60. [68]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021

  61. [69]

    Camera exposure determination

    Sidney F Ray et al. Camera exposure determination. In Ralph Jacobson, Sidney Ray, Geoffrey G. Attridge, and Norman Axford, editors, The Manual of Photography, pages 310–318. Routledge, 2000

  62. [70]

    ImageNet-21K pretraining for the masses

    Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-21K pretraining for the masses. arXiv:2104.10972, 2021

  63. [71]

    Fine-grained recognition in the noisy wild: Sensitivity analysis of convolutional neural networks approaches

    Erik Rodner, Marcel Simon, Robert B Fisher, and Joachim Denzler. Fine-grained recognition in the noisy wild: Sensitivity analysis of convolutional neural networks approaches. arXiv:1610.06756, 2016

  64. [72]

    Effects of degradations on deep neural network architectures

    Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv:1807.10108, 2018

  65. [73]

    LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs

    Christoph Schuhmann, Robert Kaczmarczyk, Aran Komatsuzaki, Aarush Katta, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs. In NeurIPSW, 2021

  66. [74]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In CVPR, 2019

  67. [75]

    Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018

  68. [76]

    Assessing visually-continuous corruption robustness of neural networks relative to human performance

    Huakun Shen, Boyue Caroline Hu, Krzysztof Czarnecki, Lina Marsso, and Marsha Chechik. Assessing visually-continuous corruption robustness of neural networks relative to human performance. In WACV, 2025

  69. [77]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015

  70. [78]

    WIT: Wikipedia- based image text dataset for multimodal multilingual machine learning

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. WIT: Wikipedia- based image text dataset for multimodal multilingual machine learning. In SIGIR, 2021

  71. [79]

    Convnets and ImageNet beyond accuracy: Understanding mistakes and uncovering biases

    Pierre Stock and Moustapha Cisse. Convnets and ImageNet beyond accuracy: Understanding mistakes and uncovering biases. In ECCV, 2018

  72. [80]

    A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models

    Namjoon Suh and Guang Cheng. A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models. Annual Review of Statistics and Its Application , 12, 2024

  73. [81]

    YFCC100M: The new data in multimedia research

    Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016

  74. [82]

    A deeper look at dataset bias

    Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. In Domain Adaptation in Computer Vision Applications , pages 37–55. 2017

  75. [83]

    Unbiased look at dataset bias

    Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011

  76. [84]

    A robustness analysis of deep q networks

    Adam W Tow, Sareh Shirazi, Jürgen Leitner, Niko Sünderhauf, Michael Milford, and Ben Upcroft. A robustness analysis of deep q networks. Australasian Conference on Robotics and Automation, 2016

  77. [85]

    From ImageNet to image classification: Contextualizing progress on benchmarks

    Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From ImageNet to image classification: Contextualizing progress on benchmarks. In ICML, 2020

  78. [86]

    Why does data-driven beat theory-driven computer vision? In ICCVW, 2019

    John Tsotsos, Iuliia Kotseruba, Alexander Andreopoulos, and Yulong Wu. Why does data-driven beat theory-driven computer vision? In ICCVW, 2019. 13

  79. [87]

    Probing the effect of selection bias on generalization: A thought experiment

    John K Tsotsos and Jun Luo. Probing the effect of selection bias on generalization: A thought experiment. arXiv:2105.09934, 2021

  80. [88]

    CSPNet: A new backbone that can enhance learning capability of CNN

    Chien-Yao Wang, Hong-Yuan Mark Liao, Yueh-Hua Wu, Ping-Yang Chen, Jun-Wei Hsieh, and I-Hau Yeh. CSPNet: A new backbone that can enhance learning capability of CNN. In CVPRW, 2020

  81. [89]

    QwenV2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. QwenV2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv:2409.12191, 2024

  82. [90]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv:2411.10442, 2024

  83. [91]

    InternImage: Exploring large-scale vision foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. InternImage: Exploring large-scale vision foundation models with deformable convolutions. In CVPR, 2023

  84. [92]

    Physical adversarial attack meets computer vision: A decade survey

    Hui Wei, Hao Tang, Xuemei Jia, Zhixiang Wang, Hanxun Yu, Zhubo Li, Shin’ichi Satoh, Luc Van Gool, and Zheng Wang. Physical adversarial attack meets computer vision: A decade survey. TPAMI, 46(12):9797–9817, 2024

  85. [93]

    Active control of camera parameters for object detection algorithms

    Yulong Wu and John Tsotsos. Active control of camera parameters for object detection algorithms. arXiv:1705.05685, 2017

  86. [94]

    Statistic analysis of millions of digital photos

    Dietmar Wueller and Reiner Fageth. Statistic analysis of millions of digital photos. InDigital Photography IV, volume 6817, page 68170L. International Society for Optics and Photonics, 2008

  87. [95]

    Statistic analysis of millions of digital photos 2017

    Dietmar Wueller and Reiner Fageth. Statistic analysis of millions of digital photos 2017. Electronic Imaging, 2018(5):1–4, 2018

  88. [96]

    Does robustness on ImageNet transfer to downstream tasks? In CVPR, 2022

    Yutaro Yamada and Mayu Otani. Does robustness on ImageNet transfer to downstream tasks? In CVPR, 2022

  89. [97]

    Yocto-Light-V3 product page

    Yoctopuce. Yocto-Light-V3 product page. https://www.yoctopuce.com/EN/products/ usb-environmental-sensors/yocto-light-v3 . Accessed April 5, 2024

  90. [98]

    Understanding bias in large-scale visual datasets

    Boya Zeng, Yida Yin, and Zhuang Liu. Understanding bias in large-scale visual datasets. In NeurIPS, 2024

  91. [99]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023

  92. [100]

    Understanding deep learning (still) requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021

  93. [101]

    DINO: DETR with improved denoising anchor boxes for end-to-end object detection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023

  94. [102]

    DETRs beat YOLOs on real-time object detection

    Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. DETRs beat YOLOs on real-time object detection. In CVPR, 2024

  95. [103]

    Exif as language: Learning cross-modal associations between images and camera metadata

    Chenhao Zheng, Ayush Shrivastava, and Andrew Owens. Exif as language: Learning cross-modal associations between images and camera metadata. In CVPR, 2023

  96. [104]

    Improving the robustness of deep neural networks via stability training

    Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In CVPR, 2016

  97. [105]

    Manual” and ‘Manual exposure

    Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 14 A Computer vision dataset properties VOC2007ImageNet SBUCOCO OpenImages v7 YFCC15M CC3M ImageNet21K CC12M WIT LAION400M Wukong CO...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.