REVIEW 8 cited by
Diagnosing Model Performance Under Distribution Shift
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Prediction models can perform poorly when deployed to target distributions different from the training distribution. To understand these operational failure modes, we develop a method, called DIstribution Shift DEcomposition (DISDE), to attribute a drop in performance to different types of distribution shifts. Our approach decomposes the performance drop into terms for 1) an increase in harder but frequently seen examples from training, 2) changes in the relationship between features and outcomes, and 3) poor performance on examples infrequent or unseen during training. These terms are defined by fixing a distribution on $X$ while varying the conditional distribution of $Y \mid X$ between training and target, or by fixing the conditional distribution of $Y \mid X$ while varying the distribution on $X$. In order to do this, we define a hypothetical distribution on $X$ consisting of values common in both training and target, over which it is easy to compare $Y \mid X$ and thus predictive performance. We estimate performance on this hypothetical distribution via reweighting methods. Empirically, we show how our method can 1) inform potential modeling improvements across distribution shifts for employment prediction on tabular census data, and 2) help to explain why certain domain adaptation methods fail to improve model performance for satellite image classification.
Forward citations
Cited by 8 Pith papers
-
Uncovering Bias Mechanisms in Observational Studies
Covariances between the size of causal bias and conditional variances of treatment, selection, and outcome form a fingerprint that distinguishes transportability, confounding, and selection bias mechanisms.
-
"Who experiences large model decay and why?" A Hierarchical Framework for Diagnosing Heterogeneous Performance Drift
SHIFT is a hierarchical hypothesis-testing method that detects subgroups with large model performance decay under distribution shift and explains the decay via variable-subset-specific covariate or outcome shifts.
-
General and Estimable Learning Bound Unifying Covariate and Concept Shifts
The authors define a total pair concept shift on the optimal transport plan between source and target covariates, yielding a Lipschitz-based target error bound that handles stochastic labels, general losses, and misma...
-
Explaining Concept Shift with Interpretable Feature Attribution
SGShift attributes concept shift to a sparse set of features by fitting a penalized generalized additive update term on top of the source model.
-
When the Past Misleads: Rethinking Training Data Expansion Under Temporal Distribution Shifts
Expanding the historical training window can degrade model performance and fairness under concept shift, with the harm appearing mainly when training data are large.
-
Realistic Evaluation of TabPFN v2 in Open Environments
TabPFN v2 underperforms tree-based models on most open-environment tabular tasks and is only preferable on small, covariate-shifted, class-balanced data.
-
Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings
Training on LLM text embeddings can cause tabular classifiers to collapse to single-class predictions, which spuriously inflates Accuracy-on-the-Line correlations.
-
Data Heterogeneity Modeling for Trustworthy Machine Learning
A survey that frames heterogeneity-aware machine learning as a paradigm spanning data collection, training, evaluation, and deployment, drawing mostly on the authors' prior results.
Discussion (0). Sign in to comment.