REVIEW 6 cited by
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.
Forward citations
Cited by 6 Pith papers
-
Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins
Small hyperbolic models (146M–3B) report 100% creative-seed preference, 90.7% compliance-gap detection, and a selective-gating skeleton–wallpaper memory pilot as a companion-AI stack.
-
An autonomous agent for auditing and improving the reliability of clinical AI models
ModelAuditor is an LLM agent that audits clinical imaging models, selects metrics and shifts, proposes targeted augmentations, and recovers part of the performance lost under real-world distribution shifts.
-
QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions
QuEst gives point estimates and asymptotic confidence intervals for quantile-based distributional measures by optimally combining scarce observed data with abundant model-imputed data.
-
$\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
ΔEnergy, an energy-change OOD score for CLIP, and its EBM fine-tuning loss simultaneously improve OOD detection and covariate-shift generalization.
-
CaTE Data Curation for Trustworthy AI
A synthesis of data curation practices for trustworthy AI, framed around an actionable definition of trustworthiness and a decision tree.
-
The Science of Evaluating Foundation Models
A survey-and-checklist proposal that organizes LLM evaluation into an ABCD framework (Algorithm, Big Data, Computation, Domain Expertise) for context-aware, documented assessment.
Discussion (0). Continue with ORCID to comment.