A pipeline that uses LLM-generated counterfactual captions and control-conditioned diffusion to produce label-preserving synthetic test images that expose vision model weaknesses and improve retraining.
Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generation
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Distribution shift is a major source of failure for machine learning models. However, evaluating model reliability under distribution shift can be challenging, especially since it may be difficult to acquire counterfactual examples that exhibit a specified shift. In this work, we introduce the notion of a dataset interface: a framework that, given an input dataset and a user-specified shift, returns instances from that input distribution that exhibit the desired shift. We study a number of natural implementations for such an interface, and find that they often introduce confounding shifts that complicate model evaluation. Motivated by this, we propose a dataset interface implementation that leverages Textual Inversion to tailor generation to the input distribution. We then demonstrate how applying this dataset interface to the ImageNet dataset enables studying model behavior across a diverse array of distribution shifts, including variations in background, lighting, and attributes of the objects. Code available at https://github.com/MadryLab/dataset-interfaces.
citation-role summary
background 1
citation-polarity summary
fields
cs.CV 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
DILLEMA: Diffusion and Large Language Models for Multi-Modal Augmentation
A pipeline that uses LLM-generated counterfactual captions and control-conditioned diffusion to produce label-preserving synthetic test images that expose vision model weaknesses and improve retraining.