Pith. sign in

REVIEW 3 major objections 4 minor 74 references

Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pre-trained classifier's biases can be detected with no labeled data, using only a textual description of its task.

desk verdict Genuinely new unsupervised setting, honest first pipeline, but retrieval quality and small samples keep the central claim from being proven; worth a serious referee. read the letter →

arxiv 2504.20902 v1 pith:QR6UP5CN submitted 2025-04-29 cs.CV cs.AIcs.LG

Quentin Guimard , Moreno D'Incà , Massimiliano Mancini , Elisa Ricci This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords unsupervisedbiasdetectionvisualclassifierauditinglargelanguagemodelstext-to-imageretrievalscorepseudo-labeleddatasetsopen-setdiscoverymodelfairness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

A person who downloads a pre-trained visual classifier usually has no labeled validation set and no ready-made list of failure modes. This paper proposes C2B, a framework that claims to recover the classifier's biases from nothing more than a plain-language description of the task it was built for. An LLM proposes candidate bias attributes and classes for each target class, converts them into captions like 'a photo of a rectangular birdhouse', and a retrieval engine pulls images for those captions from an unlabeled pool; accuracy gaps on those images become bias scores. On the two tested benchmarks the paper reports that C2B finds more ground-truth-annotated biases than a supervised open-set baseline, with lower miss rates, and that roughly two-thirds to four-fifths of the biases it detects are not present in the benchmark annotations. The framework is training-free, so the same pipeline can be re-run with stronger language models or retrievers as they become available.

What carries the argument

The object doing the work is the signed bias score of Eq. (1), which turns a retrieved image set into a number between $-1$ and $1$: for a fixed target class and bias attribute, it is the model's accuracy on images showing one bias class minus its average accuracy on the other bias classes of the same attribute. This score is what lets an unlabeled retrieval step yield a ranked list of biases, with positive values meaning the model does better on that bias class and negative values meaning it does worse. Two swappable components carry the rest: an LLM that proposes the bias vocabulary and generates captions, and a text-to-image retriever that supplies the pseudo-labeled images; in the reported experiments these are a lightweight instruction-tuned LLM, a contrastive language-image model, a 12-million-image corpus, and a web search engine. Because both components are task-agnostic and training-free, the same pipeline can be aimed at any classification task described in natural language.

What would settle it

Take a classification task with ground-truth labels for both target classes and bias factors, run C2B on the unlabeled pool, and then recompute the bias scores using only the ground-truth-labeled images that satisfy each caption. If the ranking of biases changes substantially when retrieval noise is removed, the pipeline's outputs are driven by retrieval artifacts rather than by the classifier's actual per-group accuracy differences.

Watch

Extended reading notes

Core claim

The central claim is that bias discovery for visual classifiers can be reformulated as a language-and-retrieval problem, removing the need for task-specific labels. C2B starts with a task description and an output class set; a large language model produces, for each class, a list of bias attributes and bias classes. A second prompting stage turns each target-bias pair into a caption, and a contrastive vision-language model retrieves the top-$k$ images for that caption from a large unlabeled database. Treating the captions as pseudo-labels, the paper computes the bias score $\phi_{y,i,j}=A_y(f,b_{i,j})-\frac{1}{n_i-1}\sum_{k\neq j}A_y(f,b_{i,k})$, the per-class accuracy gap between one bias class and its siblings. The paper reports that this pipeline outperforms a supervised open-set bias detector at recalling ground-truth biases on CelebA and ImageNet-X, while also flagging many biases absent from those annotated sets.

Load-bearing premise

The load-bearing premise is that the retrieved images really show the combination described in each caption: if 'a photo of a rectangular birdhouse' returns images that are not rectangular birdhouses, the accuracy gaps C2B reports measure retrieval noise rather than model bias, and the paper's own VQA check finds fewer than 50% of retrieved images match both the intended target and bias class.

Editorial extensions

If this is right

  • A non-expert can audit any downloaded classifier with a one-sentence task description, removing the annotation bottleneck that restricts bias detection to datasets someone has already labeled.
  • Bias detection is no longer limited to a predefined list: C2B proposes class-specific attributes, so it can name biases like 'camera angle: three-quarter' that benchmark annotations never record.
  • In the paper's experiments, C2B misses fewer known ground-truth biases than the supervised baseline (76.9–85.1% vs. 92.3–95.9% on the tested models and tasks), while detecting biases the annotations do not contain.
  • Because C2B is training-free and modular, improving the LLM or the retriever should translate directly into better bias discovery without retraining the pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own VQA check shows fewer than half of retrieved images contain both the intended target and bias class; if a retriever with stronger compositional understanding replaces the current one, hit rates should rise and false-hit rates fall. This can be tested by re-running C2B's two evaluation protocols with a different retriever.
  • The top-down LLM proposal step means biases that are not linguistically expressible, or outside the LLM's knowledge, will likely stay invisible; a hybrid that clusters the classifier's errors on retrieved images first and then uses the LLM to name each cluster could combine this approach with bottom-up discovery.
  • Because the framework needs only a text description, it could in principle audit classifiers in specialized domains such as medical imaging, where the user has no labels and biases are tied to acquisition devices or hospital settings; the paper lists this as a direction but does not demonstrate it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Classifier-to-Bias (C2B), a pipeline for unsupervised bias detection in pretrained visual classifiers. Given only a textual task description, C2B uses an LLM to propose bias attributes and bias classes for each target class, generates captions for each target-bias pair, retrieves top-k images from CC12M or Bing via CLIP-style retrieval, and then computes per-class accuracy differences on this pseudo-labeled image set to score each proposed bias. The authors claim this is the first framework that detects biases without task-specific labeled data and without a predefined bias list. They evaluate on CelebA and ImageNet-X, comparing against the supervised B2T baseline using two evaluation protocols: matching detected biases to ground-truth annotations via SBERT similarity, and measuring agreement with VQA-generated bias pseudo-labels. The paper also includes ablations on retrieval accuracy, domain shift, threshold sensitivity, LLM choice, and retrieved-image diversity.

Significance. If the central claim held, C2B would fill a real gap: a user downloading a model could audit it with no labeled validation set and no predefined bias ontology. The paper has clear strengths in presentation and reproducibility: the code repository is public, the exact prompts are reported in Appendix I, and the supplementary material contains unusually extensive ablations, including controlled retrieval from labeled subsets (Appendix B), VQA-based retrieval checks (Appendix F), and threshold sensitivity analysis (Appendix C). However, the significance is currently limited by a load-bearing weakness: the pseudo-labeled retrieved images are only weakly validated, and the paper's own VQA analysis shows that fewer than half of retrieved images contain both the intended target and bias classes. Because the bias scores in Eq. (1) are computed over these images, the reported 'detected biases' may reflect retrieval artifacts rather than classifier behavior. The core idea is promising and modular, but the evidence as presented does not yet establish that C2B measures what it claims to measure.

major comments (3)
  1. [Sec. 3.4, Eq. (5); Appx. F, Tables 17-18] The load-bearing premise of the method is that the top-k retrieved images in Eq. (5) form valid pseudo-labeled datasets D(b,y), so that the accuracy values in Eq. (1) measure classifier performance on images of class y exhibiting bias class b. The paper's own VQA evaluation contradicts this premise: the 'both' accuracy is 46.5% (Bing, CelebA), 42.4% (CC12M, CelebA), 42.8% (Bing, ImageNet-X), and 38.1% (CC12M, ImageNet-X). Thus, for a majority of target-bias cells, the retrieved images do not contain the intended combination, and the per-class accuracy differences in Eq. (1) can reflect retrieval difficulty or off-target content (e.g., 'spherical birdhouse' images being rarer and harder to retrieve) rather than classifier bias. The central claim therefore requires either a demonstration that bias scores are stable when computed on VQA-confirmed subsets only, or a correction/calibration procedure that accounts for retrieval noise.
  2. [Sec. 4.2.1, Tables 1-2; Appx. B, Tables 5-6] The quantitative grounding for the main claim is weak in several connected ways. Ground-truth matching recovers only 10.8-12.3% of CelebA biases and 7.5-11.2% of ImageNet-X biases, with Detected->GT precision of 12.8-14.2% and 2.5-3.0% respectively. The controlled-retrieval experiments in Appendix B show that retrieving from the labeled ImageNet subset (IMAGENET-GT) lowers the ground-truth hit rate while raising VQA agreement, which is consistent with external-setting scores being dominated by retrieval noise. Moreover, all reported scores use k=20 images per cell, and no confidence intervals or significance tests are provided, so the accuracy differences behind phi are high-variance. The authors should report bootstrap or repeated-retrieval confidence intervals and show that the ranking of detected biases is stable under resampling, especially before claiming superiority over B2T.
  3. [Sec. 3.3-3.5; Appx. E, Tables 15-16] The discovery claim is constrained by the proposal mechanism in a way that is not fully accounted for. The pipeline proposes biases with an LLM, generates captions from those proposals, retrieves images for them, and then scores the same proposals; Appendix E shows the LLM misses roughly 67-71% of ground-truth biases, so the detected set is limited to what the LLM can verbalize. The 'new bias' percentages (68-83% on ImageNet-X) count biases absent from the annotation set, but the paper does not independently verify that these are real classifier biases rather than retrieval artifacts or semantically similar but off-target proposals. The authors should provide a validation step for novel detected biases, for example human evaluation on a sample of VQA-confirmed retrieved images, before interpreting the open-set results as evidence of discovering biases beyond the annotated sets.
minor comments (4)
  1. [Sec. 4.2.1, paragraph after Tables 1-2] The text says 'On ImageNet-X (Tab. 1),' but the ImageNet-X results are in Table 2; the cross-reference should be corrected.
  2. [Appx. B, Table 6 caption] The caption of Table 6 states 'Agreement between detected biases and VQA on CelebA,' but the table reports ImageNet-X results; this appears to be a copy-paste error.
  3. [Sec. 3.1, Eq. (1)] The definition of Ay(f,b_i,j) is not fully specified: the authors should clarify whether this is the accuracy on the subset of retrieved images for that target-bias pair, and how the per-class accuracy is computed when the target class is an attribute (e.g., 'smiling' vs. 'not smiling') rather than a single category.
  4. [Sec. 4.1, Implementation details] The paper reports per-class LLM generation times in Appendix J but does not report the total wall-clock cost of running the full pipeline over all 1000 ImageNet classes and all retrieved images; a brief cost estimate would help readers judge practical usability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: C2B measures the target classifier on caption-pseudo-labeled retrieved images; acknowledged retrieval noise is a validity caveat, not a by-construction reduction.

full rationale

C2B's chain is not circular. Eqs. (2)-(4) generate bias proposals and captions from the task text, Eq. (5) retrieves images by CLIP similarity, and Eq. (1) then computes per-class accuracy differences of the audited classifier on those images. The bias scores are therefore a genuine (if noisy) measurement of the classifier's outputs on pseudo-labeled samples, not a restatement of the LLM proposals or captions. Imperfect retrieval, which the paper explicitly concedes in Appendix A ('less than 50% of retrieved images correspond to both the intended target and bias classes according to VQA evaluation') and quantifies in Tables 17-18, can make individual bias scores unreliable, but that is an estimation-validity problem, not a circular one: the output does not reduce to the input by construction. Evaluation is anchored externally to CelebA and ImageNet-X ground-truth annotations and to VQA pseudo-labels, and the comparison to B2T is a fair benchmark. The only self-citation, OpenBias [17], is cited as inspiration in Related Work and is not load-bearing for C2B's claims. No circular step is exhibited, so the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of unverified components: LLM proposals, caption generation, retrieval pseudo-labels, and bias scoring on noisy images. The main numeric choices are evaluation thresholds; the method itself is training-free, so no learned parameters are fitted. No new physical or conceptual entities are introduced.

free parameters (3)
  • top-k retrieved images per caption = 20
    Eq. (5) fixes k=20; no stability analysis is reported, and per-class accuracy computed on 20 images is noisy.
  • bias detection threshold tau = 0.05
    Used to declare a ground-truth or detected bias present (Sec. 4.2.1); sensitivity is shown only in the appendix and the value is chosen without a principled criterion.
  • SBERT matching threshold = 0.9
    Semantic match between proposed and ground-truth biases; described as 'empirically set to avoid too many false positives' (Sec. 4.2.1, Appx K). Results change substantially at 0.8 and 0.95 (Appx C).
assumptions (5)
  • domain assumption The natural-language task description accurately and sufficiently specifies the task, input domain, and class list.
    Invoked in Sec. 3.2 and in the hand-written task descriptions of Figs. 17-18; the quality of all downstream proposals depends on this text.
  • domain assumption Text-to-image retrieval returns images that actually contain the target class and bias class described in each caption.
    Used in Eq. (5) and Sec. 3.4; Appendix F shows less than 50% of retrieved images satisfy both conditions according to VQA.
  • domain assumption Accuracy differences on pseudo-labeled retrieved images, as defined in Eq. (1), measure classifier bias rather than retrieval artifacts or domain shift.
    This is the core interpretation of the bias scores in Sec. 3.5; Appendix B shows retrieval noise and domain shift materially affect results.
  • domain assumption LLM-generated bias proposals are visually identifiable in images and cover a useful fraction of real biases.
    Sec. 3.3 relies on this; Appendix E reports LLM proposals cover only about 29% of CelebA and 33% of ImageNet-X ground-truth biases.
  • domain assumption VQA pseudo-labels are a valid oracle for validating biases not present in ground-truth annotations.
    Used in Sec. 4.2.2; Appendix G reports low informedness on ImageNet-X factors (BM=0.154 for LLaVA-1.5), weakening this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers." pith.science (2026). https://pith.science/paper/QR6UP5CN

@misc{pith2026250420902,
  author       = {Pith},
  title        = {Pith review of: Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QR6UP5CN}},
  note         = {Machine review of arXiv:2504.20902}
}
read the original abstract

A person downloading a pre-trained model from the web should be aware of its biases. Existing approaches for bias identification rely on datasets containing labels for the task of interest, something that a non-expert may not have access to, or may not have the necessary resources to collect: this greatly limits the number of tasks where model biases can be identified. In this work, we present Classifier-to-Bias (C2B), the first bias discovery framework that works without access to any labeled data: it only relies on a textual description of the classification task to identify biases in the target classification model. This description is fed to a large language model to generate bias proposals and corresponding captions depicting biases together with task-specific target labels. A retrieval model collects images for those captions, which are then used to assess the accuracy of the model w.r.t. the given biases. C2B is training-free, does not require any annotations, has no constraints on the list of biases, and can be applied to any pre-trained model on any classification task. Experiments on two publicly available datasets show that C2B discovers biases beyond those of the original datasets and outperforms a recent state-of-the-art bias detection baseline that relies on task-specific annotations, being a promising first step toward addressing task-agnostic unsupervised bias detection.

Figures

Figures reproduced from arXiv: 2504.20902 by the authors.

Figure 1
Figure 1. We explore the novel task of bias discovery when we are [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of C2B. Given a specific task with an associated description and a pre-trained classification model , our approach leverages an LLM to identify potential candidate bias attributes and corresponding classes. These candidate biases are then used to prompt the LLM to generate captions, which are subsequently utilized by a retrieval module to collect a dataset of images for bias testing. This dataset enables ev… view at source ↗
Figure 3
Figure 3. Examples of biases found by C2B for the high cheekbones target attribute on CelebA. angle of view - angled angle of view - frontal background - rural background - urban color - blue color - green color - red image quality - high image quality - low image resolution - high image resolution - low lighting - daytime lighting - nighttime object size - large object size - small orientation - landscape orientation - portr… view at source ↗
Figures from the paper (21 more)
Figure 6
Figure 6. Figure 6: Accuracy of VLM-based retrieval. retrieve images with combinations of attributes, with a re￾call @ 0.01 · K just above 40%. For ImageNet-X (right), CLIP is fairly accurate, especially for low values of A, with more than 80% and 70% recall at 0.05·K to retrieve images w…
Figure 5
Figure 5. Figure 5: Examples of biases found by C2B for the birdhouse class on ImageNet-X (ViT B 16 SWAG). X, using handcrafted captions as input: combinations of at￾tributes on CelebA, and classes and factors on ImageNet-X. Metric. We use recall@K, where K varies for each caption to matc…
Figure 7
Figure 7. Figure 7: Strongest detected biases of FaceXFormer over all target attributes on face attribute classification. 1.0 0.5 0.0 0.5 1.0 Bias score (European polecat, background complexity: simple background) (Norwich Terrier, focus: sharp focus) (shield, object presence: shield abse…
Figure 9
Figure 9. Figure 9: Strongest detected biases of ResNet101 V2 over all target classes on image classification. 1.0 0.5 0.0 0.5 1.0 Bias score (Saluki, Composition: Off-center Saluki) (lighter, background complexity: complex) (Norwich Terrier, focus: sharp focus) (hand plane, color: green)…
Figure 10
Figure 10. Figure 10: Strongest detected biases of ResNet152 V2 over all tar￾get classes on image classification. 1.0 0.5 0.0 0.5 1.0 Bias score (acorn, texture: scaly) (shield, object presence: shield absent) (ocarina, angle: oblique) (Norwich Terrier, focus: sharp focus) (projector, obje…
Figure 12
Figure 12. Figure 12: Most bias-affected target attributes for [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Most bias-affected target classes for ResNet50 V2. 0.0 0.5 1.0 1.5 2.0 Bias magnitude orange Boxer strainer cock slip-on shoe slide rule titi printer chiton shield Target class [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 15
Figure 15. Figure 15: Most bias-affected target classes for [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 17
Figure 17. Figure 17: Task description for face attribute classification. [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 19
Figure 19. Figure 19: System prompt for bias generation. Think about what characteristic or feature of the image could impact the performance of the model. Think about potential spurious correlations and potential failure types [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 21
Figure 21. Figure 21: System prompt for caption template generation. [PITH_FULL_IMAGE:figures/full_fig_p025_21.png]
Figure 23
Figure 23. Figure 23: System prompt for caption generation. Do not introduce any new bias in the captions. Do not add new attributes. The only attributes included in the caption should be the given target class and the given bias class [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 25
Figure 25. Figure 25: Example of biases proposed by Gemma, for the [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]
Figure 26
Figure 26. Figure 26: Example of biases proposed by Llama, for the [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: Example of biases proposed by Phi, for the [PITH_FULL_IMAGE:figures/full_fig_p027_27.png]
Figure 28
Figure 28. Figure 28: Example of SBERT similarity scores between ground-truth attributes (used as ground-truth biases) and biases proposed by C2B [PITH_FULL_IMAGE:figures/full_fig_p028_28.png]
Figure 29
Figure 29. Figure 29: Ground-truth bias matrix of FaceXFormer on CelebA. [PITH_FULL_IMAGE:figures/full_fig_p030_29.png]
Figure 30
Figure 30. Figure 30: Ground-truth bias matrix of ResNet50 V2 for the 50 classes with the strongest biases on ImageNet-X. upright piano Japanese Chin agaric red panda red king crab buckle dhole American bullfrog sloth bear drumstick southern black widow grey fox guinea pig baguette dingo b…
Figure 31
Figure 31. Figure 31: Ground-truth bias matrix of ResNet101 V2 for the 50 classes with the strongest biases on ImageNet-X. 20 [PITH_FULL_IMAGE:figures/full_fig_p031_31.png]
Figure 32
Figure 32. Figure 32: Ground-truth bias matrix of ResNet152 V2 for the 50 classes with the strongest biases on ImageNet-X. agaric upright piano dhole red panda American bullfrog impala Gila monster sloth bear drumstick beaver Airedale Terrier moving van buckle breakwater jaguar speaker sou…
Figure 33
Figure 33. Figure 33: Ground-truth bias matrix of ViT B 16 SWAG E2E V1 for the 50 classes with the strongest biases on ImageNet-X. 21 [PITH_FULL_IMAGE:figures/full_fig_p032_33.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 57 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 15

    arXiv 2024

  2. [2]

    FairML: ToolBox for diagnosing bias in predictive modeling

    Julius A Adebayo et al. FairML: ToolBox for diagnosing bias in predictive modeling. PhD thesis, Massachusetts Institute of Technology, 2016. 1

    work page 2016

  3. [3]

    Estimat- ing example difficulty using variance of gradients

    Chirag Agarwal, Daniel D’souza, and Sara Hooker. Estimat- ing example difficulty using variance of gradients. In CVPR,

    work page

  4. [4]

    Diffusion visual counterfactual explana- tions

    Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explana- tions. NeurIPS, 2022. 2

    work page 2022

  5. [5]

    Kudchadkar, and James Fackler

    Razvan Azamfirei, Sapna R. Kudchadkar, and James Fackler. Large language models and the perils of their hallucinations. Critical Care, 27(1):120, 2023. 2

    work page 2023

  6. [6]

    Learning to split for au- tomatic bias detection

    Yujia Bao and Regina Barzilay. Learning to split for au- tomatic bias detection. arXiv preprint arXiv:2204.13749 ,

    arXiv

  7. [7]

    Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them

    Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https : / / github . com / rom1504 / clip - retrieval, 2022. 5

    work page 2022

  8. [8]

    R. K. E. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mo- jsilovi´c, S. Nagar, K. Natesan Ramamurthy, J. Richards, D. Saha, P. Sattigeri, M. Singh, K. R. Varshney, and Y . Zhang. Ai fairness 360: An extensible toolkit for detecting and mit- igating algorithmic bias. IBM Journal of Research and De- velopm...

    work page 2019

Show all 74 references
  1. [9]

    Man is to computer program- mer as woman is to homemaker? debiasing word embed- dings

    Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer program- mer as woman is to homemaker? debiasing word embed- dings. In NeurIPS, 2016. 1

    2016

  2. [10]

    Gender shades: Inter- sectional accuracy disparities in commercial gender classi- fication

    Joy Buolamwini and Timnit Gebru. Gender shades: Inter- sectional accuracy disparities in commercial gender classi- fication. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, 2018. 1

    2018

  3. [11]

    Fairvis: Visual analytics for discovering intersectional bias in machine learning

    ´Angel Alexander Cabrera, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau. Fairvis: Visual analytics for discovering intersectional bias in machine learning. In 2019 IEEE Conference on Visual An- alytics Science and Technology (VAST), pages 46–...

    2019

  4. [12]

    Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR,

  5. [13]

    Davide Chicco, Niklas T ¨otsch, and Giuseppe Jurman. The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and marked- ness in two-class confusion matrix evaluation.BioData Min- ing, 14(1):13, 2021. 11

    2021

  6. [14]

    Say my name: a model’s bias dis- covery framework

    Massimiliano Ciranni, Luca Molinaro, Carlo Alberto Bar- bano, Attilio Fiandrotti, Vittorio Murino, Vito Paolo Pastore, and Enzo Tartaglione. Say my name: a model’s bias dis- covery framework. arXiv preprint arXiv:2408.09570, 2024. 2

    2024

  7. [15]

    What could go wrong? discovering and describing failure modes in computer vision

    Gabriela Csurka, Tyler L Hayes, Diane Larlus, and Riccardo V olpi. What could go wrong? discovering and describing failure modes in computer vision. In ECCV-WS, 2024. 3

    2024

  8. [16]

    The spotlight: A general method for discov- ering systematic errors in deep learning models

    Greg d’Eon, Jason d’Eon, James R Wright, and Kevin Leyton-Brown. The spotlight: A general method for discov- ering systematic errors in deep learning models. In Proceed- ings of the 2022 ACM Conference on Fairness, Accountabil- ity, and Transparency, pages 1962–1981, 2022. 1, 2

    2022

  9. [17]

    Openbias: Open-set bias detection in text-to-image generative models

    Moreno D’Inc `a, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection in text-to-image generative models. In CVPR, 2024. 1, 2

    2024

  10. [18]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 ,

    arXiv

  11. [19]

    Domino: Discovering sys- tematic errors with cross-modal embeddings

    Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean- Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Re. Domino: Discovering sys- tematic errors with cross-modal embeddings. In ICLR, 2022. 1, 2

    2022

  12. [20]

    Datacomp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Or- gad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ra- manujan, Yonatan Bitton, Kalyani Marathe, Stephen Mus...

    2023

  13. [21]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50 (3):1097–1179, 2024. 2

    2024

  14. [22]

    Adaptive testing of computer vision models

    Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive testing of computer vision models. In ICCV, 2023. 1, 3

    2023

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  16. [24]

    Women also snowboard: Over- coming bias in captioning models

    Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Over- coming bias in captioning models. In ECCV, 2018. 1

    2018

  17. [25]

    Generating counterfactual explanations with natural language

    Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Generating counterfactual explanations with natural language. In ICML-WS, 2018. 2

    2018

  18. [26]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. NeurIPS,

  19. [27]

    https://huggingface.co/. 1

  20. [28]

    Imagenet-x: Understanding model mistakes with factor of variation annotations

    Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vin- cent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet-x: Understanding model mistakes with factor of variation annotations. In ICLR, 2023. 1, 2, 6

    2023

  21. [29]

    Miss- ingness bias in model debugging

    Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Aleksander Madry. Miss- ingness bias in model debugging. In ICLR, 2022. 2

    2022

  22. [30]

    Distilling model failures as directions in latent space

    Saachi Jain, Hannah Lawrence, Ankur Moitra, and Alek- sander Madry. Distilling model failures as directions in latent space. In ICLR, 2023. 1, 2

    2023

  23. [31]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12), 2023. 2

    2023

  24. [32]

    Lewis, and Guha Balakrishnan

    Krish Kabra, Kathleen M. Lewis, and Guha Balakrishnan. Gelda: A generative language annotation framework to re- veal visual biases in image generators. In CVPRW, 2024. 2

    2024

  25. [33]

    Discovering and miti- gating visual biases through keyword explanation

    Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jinwoo Shin. Discovering and miti- gating visual biases through keyword explanation. In CVPR,

  26. [34]

    Udis: Unsupervised discovery of bias in deep visual recognition models

    Arvindkumar Krishnakumar, Viraj Prabhu, Sruthi Sudhakar, and Judy Hoffman. Udis: Unsupervised discovery of bias in deep visual recognition models. In BMVC, 2021. 1, 2

    2021

  27. [35]

    Discover the unknown biased attribute of an image classifier

    Zhiheng Li and Chenliang Xu. Discover the unknown biased attribute of an image classifier. In ICCV, 2021. 2

    2021

  28. [36]

    Discover and mitigate unknown biases with debiasing alternate net- works

    Zhiheng Li, Anthony Hoogs, and Chenliang Xu. Discover and mitigate unknown biases with debiasing alternate net- works. In ECCV, 2022. 2

    2022

  29. [37]

    Just train twice: Improving group robustness without training group information

    Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghu- nathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In ICML, 2021. 2

    2021

  30. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 7, 11

    2023

  31. [39]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR,

  32. [40]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 11

    2024

  33. [41]

    Deep learning face attributes in the wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015. 2, 5

    2015

  34. [42]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 5

    2021

  35. [43]

    Torchvision: Py- torch’s computer vision library.https://github.com/ pytorch/vision, 2016

    TorchVision maintainers and contributors. Torchvision: Py- torch’s computer vision library.https://github.com/ pytorch/vision, 2016. 5

    2016

  36. [44]

    A survey on bias and fairness in machine learning

    Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) , 54 (6):1–35, 2021. 3

    2021

  37. [45]

    Gemma: Open mod- els based on gemini research and technology

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open mod- els based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. 15

    2024 arXiv

  38. [46]

    Chat markup language chatml (preview), 2024

    Microsoft. Chat markup language chatml (preview), 2024. https://learn.microsoft.com/en-us/azure/ ai- services/openai/how- to/chat- markup- language, Last accessed on 2024-11-21. 13

    2024

  39. [47]

    Model cards for model reporting

    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency , pages 220–229,

  40. [48]

    Facexformer: A unified transformer for fa- cial analysis

    Kartik Narayan, Vibashan VS, Rama Chellappa, and Vishal M Patel. Facexformer: A unified transformer for fa- cial analysis. arXiv preprint arXiv:2403.12960, 2024. 5

    2024 arXiv

  41. [49]

    Dissecting racial bias in an algorithm used to manage the health of populations

    Ziad Obermeyer, Brian Powers, Christine V ogeli, and Send- hil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464): 447–453, 2019. 1

    2019

  42. [50]

    Omiye, Jenna C

    Jesutofunmi A. Omiye, Jenna C. Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou. Large lan- guage models propagate race-based medicine. npj Digital Medicine, 6(1):195, 2023. 2

    2023

  43. [51]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 4

    2021

  44. [52]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In ENMLP, 2019. 6

    2019

  45. [53]

    PRIME: Prioritizing interpretability in failure mode extraction

    Keivan Rezaei, Mehrdad Saberi, Mazda Moayeri, and Soheil Feizi. PRIME: Prioritizing interpretability in failure mode extraction. In ICLR, 2024. 1, 3

    2024

  46. [54]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge. IJCV, 115(3):211–252, 2015. 5

    2015

  47. [55]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Lai...

    2022

  48. [56]

    A multimodal automated interpretability agent

    Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and An- tonio Torralba. A multimodal automated interpretability agent. In ICML, 2024. 3

    2024

  49. [57]

    Fairrag: Fair human genera- tion via fair retrieval augmentation

    Robik Shrestha, Yang Zou, Qiuyu Chen, Zhiheng Li, Yusheng Xie, and Siqi Deng. Fairrag: Fair human genera- tion via fair retrieval augmentation. In CVPR, 2024. 2

    2024

  50. [58]

    Revisiting weakly supervised pre-training of visual perception models

    Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Doll ´ar, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In CVPR, 2022. 5

    2022

  51. [59]

    Salient imagenet: How to discover spurious features in deep learning? In ICLR, 2022

    Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In ICLR, 2022. 2

    2022

  52. [60]

    Understanding failures of deep networks via robust feature extraction

    Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In CVPR, 2021. 1

    2021

  53. [61]

    No subclass left behind: Fine- grained robustness in coarse-grained classification problems

    Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher R ´e. No subclass left behind: Fine- grained robustness in coarse-grained classification problems. NeurIPS, 2020. 2

    2020

  54. [62]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In CVPR, 2022. 2

    2022

  55. [63]

    When does dough become a bagel? analyzing the remaining mistakes on imagenet

    Vijay Vasudevan, Benjamin Caine, Raphael Gontijo Lopes, Sara Fridovich-Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. In NeurIPS, 2022. 1

    2022

  56. [64]

    Revise: A tool for measuring and mitigating bias in visual datasets

    Angelina Wang, Alexander Liu, Ryan Zhang, Anat Kleiman, Leslie Kim, Dora Zhao, Iroha Shirai, Arvind Narayanan, and Olga Russakovsky. Revise: A tool for measuring and mitigating bias in visual datasets. IJCV, 130(7):1790–1810,

  57. [65]

    Dis- covering bugs in vision models using off-the-shelf image generation and captioning

    Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Dis- covering bugs in vision models using off-the-shelf image generation and captioning. In NeurIPS-WS, 2022. 2

    2022

  58. [66]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023. 2

    2023

  59. [67]

    Siren’s song in the ai ocean: a survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yu- long Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. 2

    2023 arXiv

  60. [68]

    Discover and mitigate multiple biased subgroups in im- age classifiers

    Zeliang Zhang, Mingqian Feng, Zhiheng Li, and Chenliang Xu. Discover and mitigate multiple biased subgroups in im- age classifiers. In CVPR, 2024. 1, 2

    2024

  61. [69]

    Under- standing and evaluating racial biases in image captioning

    Dora Zhao, Angelina Wang, and Olga Russakovsky. Under- standing and evaluating racial biases in image captioning. In ICCV, 2021. 1

    2021

  62. [70]

    Men also like shopping: Reducing gender bias amplification using corpus-level constraints

    Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, 2017. 1

    2017

  63. [71]

    camera angle: three-quarter

    Zaiying Zhao, Soichiro Kumano, and Toshihiko Yamasaki. Language-guided detection and mitigation of unknown dataset bias. arXiv preprint arXiv:2406.02889, 2024. 2 CLASSIFIER -TO-BIAS: Toward Unsupervised Automatic Bias Detection for Visual Classifiers Supplementary Material Tab...

    2024 arXiv

  64. [72]

    Generate a list of visually-identifiable bias attributes that could influence the performance of a pre-trained classifier for a given task

  65. [73]

    For each bias attribute, provide a list of bias classes that represent all the potential values of this attribute

  66. [74]

    bias attribute

    Output the list of bias attributes and their classes in JSON format. Figure 19. System prompt for bias generation. Think about what characteristic or feature of the image could impact the performance of the model. Think about potential spurious correlations and potential failu...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.