Pith. sign in

REVIEW 5 cited by

Leveraging Unlabeled Data to Predict Out-of-Distribution Performance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2201.04234 v3 pith:LAM3G2AV submitted 2022-01-11 cs.LG stat.ML

Saurabh Garg , Sivaraman Balakrishnan , Zachary C. Lipton , Behnam Neyshabur , Hanie Sedghi This is my paper
classification cs.LGstat.ML
keywords targetaccuracyconfidencedatamethodmethodsmodelperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. ATC outperforms previous methods across several model architectures, types of distribution shifts (e.g., due to synthetic corruptions, dataset reproduction, or novel subpopulations), and datasets (Wilds, ImageNet, Breeds, CIFAR, and MNIST). In our experiments, ATC estimates target performance $2$-$4\times$ more accurately than prior methods. We also explore the theoretical foundations of the problem, proving that, in general, identifying the accuracy is just as hard as identifying the optimal predictor and thus, the efficacy of any method rests upon (perhaps unstated) assumptions on the nature of the shift. Finally, analyzing our method on some toy distributions, we provide insights concerning when it works. Code is available at https://github.com/saurabhgarg1996/ATC_code/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. One task to rule them all: A closer look at traffic classification generalizability

    cs.NI 2025-07 conditional novelty 7.0 of 10

    Traffic classifiers that seem near-perfect on their own datasets fall to 30-40% accuracy on another network's same-task data, and a 1-Nearest Neighbor baseline is competitive.

  2. Transductive Model Selection under Prior Probability Shift

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Under prior probability shift, selecting hyperparameters via accuracy estimates on the unlabelled test batch (transductive model selection) outperforms cross-validation-based selection.

  3. LanePerf: a Performance Estimation Framework for Lane Detection

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LanePerf estimates lane-detection F1 on unlabeled target domains by fusing CLIP image features with lane features, achieving MAE 0.117 and Spearman's rho 0.727 on OpenLane.

  4. From Tea Leaves to System Maps: A Survey and Framework on Context-aware Machine Learning Monitoring

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A systematic review of 94 studies proposes C-SAR, a three-dimensional framework (System, Aspect, Representation) describing how contextual information is used in ML monitoring, with 20 recurring patterns.

  5. Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A statistical non-inferiority test on estimated per-sample correctness probabilities flags when a classifier's accuracy on unlabeled user data drops by more than a chosen margin relative to its test set.

Pith tools