REVIEW 4 cited by
BREEDS: Benchmarks for Subpopulation Shift
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We develop a methodology for assessing the robustness of models to subpopulation shift---specifically, their ability to generalize to novel data subpopulations that were not observed during training. Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity. We then validate that the corresponding shifts are tractable by obtaining human baselines for them. Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of off-the-shelf train-time robustness interventions. Code and data available at https://github.com/MadryLab/BREEDS-Benchmarks .
Forward citations
Cited by 4 Pith papers
-
Enhancing Conformal Prediction via Class Similarity
Adding a class-similarity penalty to conformal scores can shrink prediction sets and reduce the number of semantic groups they span.
-
Moment Alignment: Unifying Gradient and Hessian Matching for Domain Generalization
A unified moment-alignment theory bounds target-domain error by cross-domain differences in loss derivatives, and the new CMA algorithm implements exact gradient and Hessian matching in closed form.
-
Targeted Forgetting of Image Subgroups in CLIP Models
A three-stage forgetting, reminding, and restoring pipeline lets CLIP forget a targeted image subgroup without pre-training data while keeping zero-shot performance.
-
Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies
The paper proposes a one-to-one mapping between six causes of distribution shift and several AI safety issues, arguing for mutual method transfer through aligned definitions.
Discussion (0). Sign in to comment.