Pith. sign in

REVIEW 4 major objections 4 minor 78 references

Consensus-Driven Active Model Selection

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CODA performs active model selection by combining classifier consensus with Bayesian inference, reducing annotation effort by over 70% compared to prior state of the art.

desk verdict A strong unsupervised consensus prior plus a valuable benchmark, but the active component is nearly inert and the 70% claim is unsupported. read the letter →

arxiv 2507.23771 v1 pith:TT4V7OO7 submitted 2025-07-31 cs.LG cs.AIcs.CV

Justin Kay , Grant Van Horn , Subhransu Maji , Daniel Sheldon , Sara Beery This is my paper
classification cs.LGcs.AIcs.CV
keywords activemodelselectionconsensuspriorsDawid-SkeneBayesianinferenceexpectedinformationgainlabelefficiencyzoobenchmarksuite
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the best model among a pool of existing classifiers can be found with far fewer human labels than current active model selection methods require. It proposes CODA, which treats each candidate model as having a per-class confusion matrix and uses the consensus of all models' predictions to initialize Bayesian beliefs about which model is best. A human label is then requested for the data point whose hypothetical label would most reduce uncertainty about the identity of the best model. Across 26 curated model selection tasks spanning vision and language, CODA outperforms all prior methods on 18 tasks, often cutting cumulative regret in half or more, and it can sometimes identify a near-optimal model with fewer than 25 labels. The practical payoff is that users of large model repositories could choose a deployment model without building and annotating a large validation set.

What carries the argument

The machinery is a Bayesian Dawid-Skene model of classifier predictions, where each model hk has a C×C confusion matrix whose rows follow Dirichlet priors. The priors are initialized from consensus pseudo-labels: predictions from all models are summed, the argmax class becomes a pseudo-label, and each model's confusion matrix is estimated against those pseudo-labels with a diagonal-weighted regularization. At each step, PBest(hk) is computed by the integral ∫01 fk(x) ∏l≠k Fl(x) dx, where fk and Fl are class-marginal-weighted mixtures of Beta densities and cumulative distributions for each model's diagonal accuracy. Data points are acquired by expected information gain, a virtual update that computes the entropy of PBest under each possible label and weights it by the estimated class probability of the point. True labels enter as a partial Dirichlet-categorical update with a small learning rate η=0.01, which keeps the consensus prior dominant.

What would settle it

Construct a pool of candidate models that all share the same systematic bias toward a majority class, then run CODA and random sampling across varying levels of class imbalance; if CODA's cumulative regret at step 100 does not degrade relative to random sampling as the shared bias strengthens, the claim that the consensus prior is the mechanism behind its success is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that per-model confusion matrices with consensus-initialized Dirichlet priors give a probabilistic distribution PBest over which candidate is best, and that actively labeling points by expected information gain with respect to this distribution is a highly label-efficient model selection strategy. CODA computes PBest by pairing each model's estimated class-conditional accuracy (a mixture of Beta distributions derived from the Dirichlet rows of its confusion matrix) and integrating the probability that one model's draw exceeds all others. Each candidate data point is scored by how much its hypothetical label would reduce the Shannon entropy of PBest, and real labels update the Dirichlet parameters through a small partial-count update. The paper reports that this recipe reduces the annotation effort required to discover the best model by upwards of 70% compared to the previous state of the art, and that the same consensus-initialized prior alone matches or exceeds unsupervised model selection methods on 20 of 26 benchmarks.

Load-bearing premise

The load-bearing premise is that consensus pseudo-labels give a reliable Dirichlet prior over each model's confusion matrix, and that this prior is strong enough that a few real labels cannot overturn it when the consensus is systematically wrong.

Editorial extensions

If this is right

  • If CODA's central claim is correct, model selection for pre-trained model zoos becomes practical in the low-label regime, with near-optimal models found using dozens rather than hundreds or thousands of human annotations.
  • Unsupervised domain adaptation pipelines, which currently struggle to validate adapted models without target labels, could use CODA's consensus priors as a strong unsupervised model selection signal and then refine with a handful of labels.
  • The released 26-task benchmark suite provides a shared evaluation ground for future active model selection research, covering domain shifts, WILDS-style distribution shifts, model zoos, and GLUE language tasks.
  • CODA's framework is explicitly designed for accuracy-based multi-class losses, so extending it to other metrics or to non-classification tasks would require a new derivation of PBest but the consensus prior and acquisition structure generalize.
  • Because the consensus prior alone already matches prior unsupervised selection methods on most tasks, CODA effectively unifies unsupervised and active model selection in one framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to replace the fixed class-marginal estimate with an explicit imbalance correction: the paper's CoLA and CivilComments failures suggest that when the true class prior is more skewed than the consensus estimate, the PBest computation can confidently select a degenerate biased model.
  • The partial-update design implies that CODA's success depends on the consensus prior being roughly correct; a pool where all models share the same systematic error should defeat the method, and constructing synthetic pools with controlled shared bias would directly probe this boundary.
  • CODA's acquisition objective optimizes entropy over the model identity PBest, not over the eventual regret of the chosen model; an alternative that directly minimizes expected regret under the current posterior might further improve label efficiency in the small-sample regime.
  • The method could be combined with active testing, since labels acquired for model selection also provide unbiased estimates of individual model accuracy; the paper mentions this as future work, and a joint allocation strategy is a concrete follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CODA, a consensus-driven probabilistic method for pool-based active model selection. CODA models each candidate classifier with a Dawid-Skene-style confusion matrix, initializes Dirichlet priors from the consensus of model predictions on unlabeled data, computes a distribution PBest over which model is the best via Beta-mixture integration, selects points to label by expected information gain in PBest, and updates the Dirichlet parameters with a partial-update rule. The authors curate a benchmark of 26 model selection tasks from DomainNet126, WILDS, MSV, and GLUE, and compare CODA against random sampling, uncertainty sampling, active testing, VMA, and ModelSelector. The main results are reported as cumulative regret at step 100, with CODA claimed to be state-of-the-art on 18 of 26 tasks and to reduce annotation effort by upwards of 70% relative to prior work.

Significance. The benchmark suite and code release are valuable contributions to the active model selection literature, and the idea of using consensus information to build an informative prior over confusion matrices is well motivated. If the active label-acquisition component were shown to be responsible for the reported gains, CODA would be an important step forward. The paper is also careful to include an ablation of the prior design, an ablation of the acquisition function, and a failure analysis on the two tasks where CODA underperforms random sampling. However, as submitted, the central claim that CODA is an active method that reduces annotation effort is not fully supported: the arithmetic of the default hyperparameters makes the active updates negligible relative to the consensus prior, and the paper does not provide a direct comparison between the active method and its own unsupervised prior. These issues are load-bearing for the main contribution.

major comments (4)
  1. [Sec. 4.2 Eq. (6) and Sec. 4.5 Eq. (18)] The active update is arithmetically negligible relative to the consensus prior, and the paper does not demonstrate that labeled examples materially change PBest or the model ranking. With the default values alpha=0.1 and T=0.5, Eq. (6) gives a per-row Dirichlet concentration of roughly (sum_c' beta_c,c' + 0.1 R_c)/0.5, where R_c is the number of consensus pseudo-observations in row c; for a binary task with R_c ~ 500 this is about 104. In contrast, Eq. (18) adds only eta=0.01 per label, so after 100 labels the total increment for a model is about 1, a sub-1% shift in the posterior concentration. Thus PBest at step 100 is essentially the step-0 consensus prior, and the reported cumulative regret improvements may be attributable to the unsupervised prior rather than to active selection. This is consistent with the supplementary result in Sec. 9.2, where the consensus prior alone already matches or exceeds previous unsupervised model selection on 20 of 26 tasks. The failure cases on CoLA and CivilComments (Sec. 7, Fig. 4) are also consistent with a prior that is confidently wrong and cannot be corrected within 100 labels. To support the central claim, the paper should report an ablation over eta (e.g., eta=0, 0.01, 1), show how frequently the model ranking changes as labels are acquired, and quantify the fraction of the cumulative-regret reduction that comes from the active updates rather than from the step-0 prior.
  2. [Abstract and Sec. 7, Tab. 1] The abstract's claim of 'reducing the annotation effort required to discover the best model by upwards of 70% compared to the previous state-of-the-art' is not derived from any reported experiment. Tab. 1 reports cumulative regret at step 100, and Sec. 7 reports percentage reductions in cumulative regret relative to the next-best method, but neither of these directly measures annotation effort or the number of labels needed to discover the best model. Fig. 1 shows the fraction of benchmarks converged to under 1% regret as a function of labels, which could support a label-efficiency statement, but the 70% figure is not computed or defined. The authors should either remove the unquantified claim or state explicitly which metric and which comparison produce the 70% reduction.
  3. [Sec. 4.4, Eqs. (16)-(17)] The acquisition function is underspecified because it relies on a per-point class probability pi(c | x_i) that is never defined. The text says 'as in Eq. (10), without marginalizing,' but Eq. (10) defines only the marginal class prevalence pi-hat(c) as an average over all data points. A per-point conditional distribution is a different object, and its estimator is not given. In particular, it is unclear whether pi(c | x_i) is computed using the posterior means of the confusion matrices or some other point estimate, and whether the summation over models in Eq. (10) is intended to be retained. This makes the EIG computation irreproducible and should be corrected by providing an explicit formula for pi(c | x_i).
  4. [Sec. 9.2 and Tab. 5 vs. Sec. 7 and Tab. 1] The paper reports that the consensus prior alone matches or exceeds the state of the art in unsupervised model selection on 20 of 26 tasks (Sec. 9.2, Tab. 5), and the active method is state of the art on 18 of 26 tasks (Tab. 1). Because the active method's posterior is dominated by the same consensus prior, the reader cannot distinguish the contribution of the active label acquisition from the contribution of the unsupervised initialization. A direct comparison between the active curves in Tab. 1/Fig. 1 and the step-0 prior-only selection is necessary to establish that labels add value. Without this comparison, the phrase 'active model selection' overstates what is demonstrated.
minor comments (4)
  1. [Sec. 6, ModelSelector baseline] The update rule for ModelSelector is written as 'PBest,t+1(hk) = 1-epsilon/epsilon x PBest,t+1(hk)' with the same variable on both sides; this appears to be a typo, likely for PBest,t(hk) on the right-hand side.
  2. [Eq. (10)] The notation M_{k,c',c} in Eq. (10) is used without specifying whether this is a random variable, a posterior mean, or the prior parameter; clarifying this would make the PBest computation less ambiguous.
  3. [Fig. 1 caption and Sec. 7] Fig. 1 says 'within 1% accuracy' while Sec. 7 says 'within 1% regret'; these should be made consistent.
  4. [Supplemental Sec. 9.2] There are minor typos in the supplement: 'minmium nuclear norm' should be 'minimum nuclear norm', 'Bath nuclear norm' should be 'Batch nuclear norm', and the GLUE task 'mrcp' in Tab. 5 should be 'mrpc'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the consensus prior is a disclosed heuristic and the evaluation uses external ground truth.

full rationale

CODA's derivation chain is not circular. The consensus prior (Eqs. 5-7) is a disclosed heuristic computed from model predictions, but the target h* is defined independently by ground-truth loss (Eq. 1), and all active updates (Eq. 18) and acquisition scores (Eqs. 16-17) use oracle labels obtained during the benchmark. The paper evaluates against ground truth on 26 external tasks and compares to five baselines, so the central claim is self-contained and falsifiable. The self-referential flavor of consensus-based priors is a known bias, not a definitional equivalence: the paper's own failure analysis (CoLA, CivilComments) shows the prior can be confidently wrong and the ground-truth evaluation penalizes it. There are no load-bearing self-citations or imported uniqueness theorems. The unsupervised step-0 prior coincides with EnsV on one-hot tasks, but the paper discloses this comparison (Sec. 9.2) and does not present it as a new contribution, so it is not circular renaming. The weakness that the eta=0.01 active update may be negligible relative to the prior is a correctness/ablation concern, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. Its free parameters are global hyperparameters tuned on initial experiments. The core assumptions are the Dawid-Skene generative model, independence of model accuracies, and the reliability of consensus pseudo-labels.

free parameters (3)
  • alpha = 0.1
    Blend weight between consensus empirical estimate and static prior in Eq. (6); selected based on 'a limited set of initial experiments' rather than per-task tuning.
  • T = 0.5
    Temperature controlling the number of initial pseudo-counts in Eq. (6); same selection process as alpha.
  • eta = 0.01
    Learning rate for partial Dirichlet updates in Eq. (18); chosen for stability, but this small value makes the posterior dominated by the prior.
assumptions (4)
  • domain assumption Each classifier's predictions follow a Dawid-Skene categorical model with a per-class confusion matrix, independent across data points.
    Invoked in Sec. 4.1; the entire probabilistic framework depends on this generative model. It ignores dependencies that real classifiers exhibit.
  • domain assumption Model accuracies are independent random draws from Beta mixture distributions.
    Sec. 4.3 Eq. (13)-(15); used to compute PBest. Correlated models, especially those sharing training data, violate this independence and make the integration an approximation.
  • domain assumption Consensus pseudo-labels are an acceptable proxy for ground truth for initializing priors.
    Sec. 4.2 Eq. (5); when all models share systematic errors (e.g., class imbalance), the consensus is biased, leading to the failures documented in Sec. 7.
  • ad hoc to paper Per-point class probability π(c|x_i) can be derived from current beliefs as 'in Eq. (10), without marginalizing'.
    Sec. 4.4 Eq. (16); used for EIG acquisition, but no explicit formula is provided for this quantity, leaving the acquisition function partially unspecified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consensus-Driven Active Model Selection." pith.science (2026). https://pith.science/paper/TT4V7OO7

@misc{pith2026250723771,
  author       = {Pith},
  title        = {Pith review of: Consensus-Driven Active Model Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TT4V7OO7}},
  note         = {Machine review of arXiv:2507.23771}
}
read the original abstract

The widespread availability of off-the-shelf machine learning models poses a challenge: which model, of the many available candidates, should be chosen for a given data analysis task? This question of model selection is traditionally answered by collecting and annotating a validation dataset -- a costly and time-intensive process. We propose a method for active model selection, using predictions from candidate models to prioritize the labeling of test data points that efficiently differentiate the best candidate. Our method, CODA, performs consensus-driven active model selection by modeling relationships between classifiers, categories, and data points within a probabilistic framework. The framework uses the consensus and disagreement between models in the candidate pool to guide the label acquisition process, and Bayesian inference to update beliefs about which model is best as more information is collected. We validate our approach by curating a collection of 26 benchmark tasks capturing a range of model selection scenarios. CODA outperforms existing methods for active model selection significantly, reducing the annotation effort required to discover the best model by upwards of 70% compared to the previous state-of-the-art. Code and data are available at https://github.com/justinkay/coda.

Figures

Figures reproduced from arXiv: 2507.23771 by the authors.

Figure 1
Figure 1. We introduce CODA, a consensus-driven method for active model selection. This figure shows the number of labels needed to converge to the optimal or near-optimal (within 1% ac￾curacy) model in a benchmark suite of 26 model selection tasks. CODA is significantly more label-efficient than prior work, iden￾tifying a near-optimal model with fewer than 25 labeled examples over 50% of the time, and with fewer than 100 lab… view at source ↗
Figure 2
Figure 2. CODA for active model selection. Simplified example with two models. At time step 0 we instantiate a Bayesian framework for tracking model performance over time (Sec. 4.1), using the consensus of all model predictions to instantiate per-model priors (Sec. 4.2). At each time step we perform three actions: (1) We estimate PBest, the current probability that each model is best, by integrating over our current beliefs (… view at source ↗
Figure 3
Figure 3. Active model selection average results. We visual￾ize regret (top row) and cumulative regret (bottom row) from time steps 1 to 100, median value across all tasks within benchmarks. Lower is better. CODA is consistently the best performer over time for all settings except the binary classification tasks in WILDS. Full per-task results in supplemental. gret and cumulative regret over time in more detail in [PITH_FULL… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Failure analysis on CivilComments and CoLA. CODA may underestimate the performance of very biased classi￾fiers in early steps (CivilComments, left), but overestimate them in later steps (CoLA, right) when there is also data imbalance present (blue bars). More details i…
Figure 6
Figure 6. Figure 6: Ablation of CODA acquisition function. We use the CODA probabilistic framework and compare different data point acquisition functions: random sampling, uncertainty-based sampling, and expected information gain (EIG, the default). We see that EIG typically improves upon…
Figure 7
Figure 7. Figure 7: “Success rate” of each model in selecting the absolute best model at each time step. Mean over 5 random seeds. In all datasets, several methods have not yet selected the absolute best model by time step 100. Task Random Uncertainty Active VMA Model CODA Sampling Testin…
Figure 8
Figure 8. Figure 8: Results on all benchmarks. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 10
Figure 10. Figure 10: A visual depiction of the integration technique used [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 68 canonical work pages

  1. [1]

    Invariant risk minimization

    Martin Arjovsky, L ´eon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. 6

    arXiv 1907

  2. [2]

    From detection of individual metastases to classification of lymph node status at the pa- tient level: the camelyon17 challenge

    Peter Bandi, Oscar Geessink, Quirine Manson, Mar- cory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the pa- tient level: the camelyon17 challenge. IEEE transactions on medical imaging, 38(2):550–5...

    work page 2018

  3. [3]

    Recognition in terra incognita

    Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European confer- ence on computer vision (ECCV), pages 456–473, 2018. 2

    work page 2018

  4. [4]

    The iwildcam 2018 challenge dataset.arXiv preprint arXiv:1904.05986, 2019

    Sara Beery, Grant Van Horn, Oisin Mac Aodha, and Pietro Perona. The iwildcam 2018 challenge dataset.arXiv preprint arXiv:1904.05986, 2019. 6

    arXiv 2018

  5. [5]

    Nuanced metrics for measur- ing unintended bias with real data for text classification

    Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measur- ing unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web confer- ence, pages 491–500, 2019. 6

    work page 2019

  6. [6]

    Lean crowdsourcing: Combining humans and machines in an on- line system

    Steve Branson, Grant Van Horn, and Pietro Perona. Lean crowdsourcing: Combining humans and machines in an on- line system. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition , pages 7474–7483,

    work page

  7. [7]

    A general model for aggregating annota- tions across simple, complex, and multi-object annotation tasks

    Alexander Braylan, Madalyn Marabella, Omar Alonso, and Matthew Lease. A general model for aggregating annota- tions across simple, complex, and multi-object annotation tasks. Journal of Artificial Intelligence Research , 78:901– 973, 2023. 3

    work page 2023

  8. [8]

    Monai: An open-source framework for deep learning in healthcare

    M Jorge Cardoso, Wenqi Li, Richard Brown, Nic Ma, Eric Kerfoot, Yiheng Wang, Benjamin Murrey, Andriy Myro- nenko, Can Zhao, Dong Yang, et al. Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701, 2022. 1

    arXiv 2022

Show all 78 references
  1. [9]

    Transferability vs

    Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In Interna- tional conference on machine learning , pages 1081–1090. PMLR, 2019. 6

    2019

  2. [10]

    Functional map of the world

    Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6172–6180, 2018. 6

    2018

  3. [11]

    Towards discriminability and diversity: Batch nuclear-norm maximization under label in- sufficient situations

    Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qing- ming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label in- sufficient situations. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition ...

    2020

  4. [12]

    Gradually vanishing bridge for adver- sarial domain adaptation

    Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adver- sarial domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12455–12464, 2020. 6

    2020

  5. [13]

    Committee-based sampling for training probabilistic classifiers

    Ido Dagan and Sean P Engelson. Committee-based sampling for training probabilistic classifiers. In Machine learning proceedings 1995, pages 150–157. Elsevier, 1995. 6, 7

    1995

  6. [14]

    Maximum like- lihood estimation of observer error-rates using the em algo- rithm

    Alexander Philip Dawid and Allan M Skene. Maximum like- lihood estimation of observer error-rates using the em algo- rithm. Journal of the Royal Statistical Society: Series C (Ap- plied Statistics), 28(1):20–28, 1979. 2, 3, 4, 15

    1979

  7. [15]

    Better prac- tices for domain adaptation

    Linus Ericsson, Da Li, and Timothy Hospedales. Better prac- tices for domain adaptation. In International Conference on Automated Machine Learning, pages 4–1. PMLR, 2023. 1, 2

    2023

  8. [16]

    On sta- tistical bias in active learning: How and when to fix it

    Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On sta- tistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665, 2021. 6

    2021 arXiv

  9. [17]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research , 17(59):1–35, 2016. 6

    2016

  10. [18]

    Pytorch-wildlife: A collaborative deep learning framework for conservation

    Andres Hernandez, Zhongqi Miao, Luisa Vargas, Rahul Dodhia, Pablo Arbelaez, and Juan M Lavista Ferres. Pytorch-wildlife: A collaborative deep learning framework for conservation. arXiv preprint arXiv:2405.12930, 2024. 1

    2024 arXiv

  11. [19]

    Learning whom to trust with mace

    Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. Learning whom to trust with mace. In Pro- ceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, 2013. 3

    2013

  12. [20]

    Towards reliable model selection for unsupervised domain adaptation: An empirical study and a certified baseline

    Dapeng Hu, Mi Luo, Jian Liang, and Chuan-Sheng Foo. Towards reliable model selection for unsupervised domain adaptation: An empirical study and a certified baseline. In The Thirty-eight Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track, 2025. 2, 13

    2025

  13. [21]

    Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037

    Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learn- ing, pages 2029–2037. PMLR, 2018. 6

    2018

  14. [22]

    HuggingFace Models, 2025

    HuggingFace. HuggingFace Models, 2025. 1

    2025

  15. [23]

    Minimum class confusion for versatile domain adaptation

    Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 464–480. Springer, 2020. 6

    2020

  16. [24]

    Uda-bench: Revisiting common assumptions in un- 9 supervised domain adaptation using a standardized frame- work

    Tarun Kalluri, Sreyas Ravichandran, and Manmohan Chan- draker. Uda-bench: Revisiting common assumptions in un- 9 supervised domain adaptation using a standardized frame- work. In European Conference on Computer Vision, pages 199–220. Springer, 2024. 1, 2

    2024

  17. [25]

    Online active model selection for pre-trained classifiers

    Mohammad Reza Karimi, Nezihe Merve G ¨urel, Bojan Karlaˇs, Johannes Rausch, Ce Zhang, and Andreas Krause. Online active model selection for pre-trained classifiers. In International Conference on Artificial Intelligence and Statistics, pages 307–315. PMLR, 2021. 2, 3

    2021

  18. [26]

    Anytime model selection in linear ban- dits, 2023

    Parnian Kassraie, Nicolas Emmenegger, Andreas Krause, and Aldo Pacchiano. Anytime model selection in linear ban- dits, 2023. 3

    2023

  19. [27]

    The caltech fish counting dataset: A benchmark for multiple- object tracking and counting

    Justin Kay, Peter Kulits, Suzanne Stathatos, Siqi Deng, Erik Young, Sara Beery, Grant Van Horn, and Pietro Perona. The caltech fish counting dataset: A benchmark for multiple- object tracking and counting. In European Conference on Computer Vision, pages 290–311. Springer, 2022. 2

    2022

  20. [28]

    Unsuper- vised domain adaptation in the real world: A case study in sonar video

    Justin Kay, Suzanne Stathatos, Siqi Deng, Erik Young, Pietro Perona, Sara Beery, and Grant Van Horn. Unsuper- vised domain adaptation in the real world: A case study in sonar video. In NeurIPS 2023 Computational Sustainability: Promises and Pitfalls from Theory to Deployment, 2023. 1

    2023

  21. [29]

    Align and distill: Unifying and improving domain adaptive object detection

    Justin Kay, Timm Haucke, Suzanne Stathatos, Siqi Deng, Erik Young, Pietro Perona, Sara Beery, and Grant Van Horn. Align and distill: Unifying and improving domain adaptive object detection. arXiv preprint arXiv:2403.12029, 2024. 1

    2024 arXiv

  22. [30]

    Wilds: A benchmark of in-the- wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning...

    2021

  23. [31]

    Active testing: Sample-efficient model evalu- ation

    Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evalu- ation. In International Conference on Machine Learning , pages 5753–5763. PMLR, 2021. 2, 3, 6

    2021

  24. [32]

    Active surrogate estimators: An active learning approach to label-efficient model evaluation

    Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Thomas Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. Advances in Neural Information Processing Systems , 35:24557–24570,

  25. [33]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  26. [34]

    Deeper, broader and artier domain generaliza- tion

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generaliza- tion. In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017. 6

    2017

  27. [35]

    A survey on deep active learning: Recent advances and new frontiers

    Dongyuan Li, Zhen Wang, Yankai Chen, Renhe Jiang, Weip- ing Ding, and Manabu Okumura. A survey on deep active learning: Recent advances and new frontiers. IEEE Trans- actions on Neural Networks and Learning Systems, 2024. 1, 2

    2024

  28. [36]

    Domain adaptation with auxiliary target domain-oriented classifier

    Jian Liang, Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domain-oriented classifier. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16632–16642, 2021. 6

    2021

  29. [37]

    Contextual active online model selection with expert advice

    Xuefeng Liu, Fangfang Xia, Rick L Stevens, and Yuxin Chen. Contextual active online model selection with expert advice. In ICML2022 Workshop on Adaptive Experimental Design and Active Learning in the Real World. ICML, 2022. 3

    2022

  30. [38]

    Learning transferable features with deep adaptation net- works

    Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jor- dan. Learning transferable features with deep adaptation net- works. In International conference on machine learning , pages 97–105. PMLR, 2015. 6

    2015

  31. [39]

    Conditional adversarial domain adapta- tion

    Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adapta- tion. Advances in neural information processing systems , 31, 2018. 6

    2018

  32. [40]

    Active model selection: A variance minimization approach

    Mitsuru Matsuura and Satoshi Hara. Active model selection: A variance minimization approach. In NeurIPS 2023 Work- shop on Adaptive Experimental Design and Active Learning in the Real World. 2, 3, 6

    2023

  33. [41]

    Accuracy on the line: on the strong correlation between out-of-distribution and in- distribution generalization

    John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in- distribution generalization. In International confere...

    2021

  34. [42]

    Unsu- pervised domain adaptation: A reality check

    Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Unsu- pervised domain adaptation: A reality check. arXiv preprint arXiv:2111.15672, 2021. 1, 2, 6

    2021 arXiv

  35. [43]

    Three new validators and a large-scale benchmark rank- ing for unsupervised domain adaptation

    Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. Three new validators and a large-scale benchmark rank- ing for unsupervised domain adaptation. arXiv preprint arXiv:2208.07360, 2022. 1, 2, 6, 13, 15

    2022 arXiv

  36. [44]

    Active testing: An efficient and robust framework for estimating ac- curacy

    Phuc Nguyen, Deva Ramanan, and Charless Fowlkes. Active testing: An efficient and robust framework for estimating ac- curacy. In International Conference on Machine Learning , pages 3759–3768. PMLR, 2018. 2

    2018

  37. [45]

    All mod- els are wrong, some are useful: Model selection with limited labels

    Patrik Okanovic, Andreas Kirsch, Jannes Kasper, Torsten Hoefler, Andreas Krause, and Nezihe Merve G¨urel. All mod- els are wrong, some are useful: Model selection with limited labels. arXiv preprint arXiv:2410.13609, 2024. 2, 3, 6, 7, 13, 16

    2024 arXiv

  38. [46]

    Bioimage model zoo: a community-driven resource for accessible deep learning in bioimage analysis

    Wei Ouyang, Fynn Beuttenmueller, Estibaliz G ´omez-de Mariscal, Constantin Pape, Tom Burke, Carlos Garcia- L´opez-de Haro, Craig Russell, Luc ´ıa Moya-Sans, Cristina De-La-Torre-Guti´errez, Deborah Schmidt, et al. Bioimage model zoo: a community-driven resource for accessible ...

    2022

  39. [47]

    The benefits of a model of annotation

    Rebecca J Passonneau and Bob Carpenter. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326, 2014. 2, 3, 4

    2014

  40. [48]

    Comparing bayesian models of annotation

    Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585, 2018. 3

    2018

  41. [49]

    Moment matching for multi-source domain adaptation

    Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1406–1415,

  42. [50]

    Snorkel: Rapid train- ing data creation with weak supervision

    Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R ´e. Snorkel: Rapid train- ing data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases, page 269, 2017. 3

    2017

  43. [51]

    Do cifar-10 classifiers generalize to cifar- 10? arXiv preprint arXiv:1806.00451, 2018

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar- 10? arXiv preprint arXiv:1806.00451, 2018. 2

    2018 arXiv

  44. [52]

    Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400

    Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? In International conference on machine learning , pages 5389–5400. PMLR, 2019. 2

    2019

  45. [53]

    Ad- versarial branch architecture search for unsupervised domain adaptation

    Luca Robbiano, Muhammad Rameez Ur Rahman, Fabio Galasso, Barbara Caputo, and Fabio Maria Carlucci. Ad- versarial branch architecture search for unsupervised domain adaptation. In Proceedings of the IEEE/CVF winter confer- ence on applications of computer vision , pages 2918–2928,

  46. [54]

    Maximum classifier discrepancy for unsuper- vised domain adaptation

    Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tat- suya Harada. Maximum classifier discrepancy for unsuper- vised domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 3723–3732, 2018. 6

    2018

  47. [55]

    Semi-supervised domain adaptation via minimax entropy

    Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Dar- rell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. InProceedings of the IEEE/CVF inter- national conference on computer vision , pages 8050–8058,

  48. [56]

    Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density

    Kuniaki Saito, Donghyun Kim, Piotr Teterwak, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF international conference on computer vision , pages 91...

    2021

  49. [57]

    Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generali- sation

    Amartya Sanyal, Yaxi Hu, Yaodong Yu, Yian Ma, Yixin Wang, and Bernhard Sch¨olkopf. Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generali- sation. arXiv preprint arXiv:2406.19049, 2024. 2

    2024 arXiv

  50. [58]

    Active comparison of prediction models

    Christoph Sawade, Niels Landwehr, and Tobias Scheffer. Active comparison of prediction models. Advances in neural information processing systems, 25, 2012. 2, 3

    2012

  51. [59]

    Active learning literature survey

    Burr Settles. Active learning literature survey. 2009. 1, 2

    2009

  52. [60]

    Evaluat- ing multiple models using labeled and unlabeled data

    Divya Shanmugam, Shuvom Sadhuka, Manish Raghavan, John Guttag, Bonnie Berger, and Emma Pierson. Evaluat- ing multiple models using labeled and unlabeled data. arXiv preprint arXiv:2501.11866, 2025. 2

    2025

  53. [61]

    Information-theoretical learning of discriminative clusters for unsupervised domain adaptation

    Yuan Shi and Fei Sha. Information-theoretical learning of discriminative clusters for unsupervised domain adaptation. arXiv preprint arXiv:1206.6438, 2012. 6

    2012 arXiv

  54. [62]

    Deep coral: Correlation alignment for deep domain adaptation

    Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer vision– ECCV 2016 workshops: Amsterdam, the Netherlands, Oc- tober 8-10 and 15-16, 2016, proceedings, part III 14 , pages 443–450. Springer, 2016. 6

    2016

  55. [63]

    Influence of ai’s uncertainty in the dawid- skene aggregation for human-ai crowdsourcing

    Takumi Tamura, Hiroyoshi Ito, Satoshi Oyama, and At- suyuki Morishima. Influence of ai’s uncertainty in the dawid- skene aggregation for human-ai crowdsourcing. In Interna- tional Conference on Information, pages 232–247. Springer,

  56. [64]

    Id and ood performance are sometimes inversely correlated on real-world datasets

    Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Ab- basnejad. Id and ood performance are sometimes inversely correlated on real-world datasets. Advances in Neural Infor- mation Processing Systems, 36:71703–71722, 2023. 2

    2023

  57. [65]

    Lean multiclass crowdsourcing

    Grant Van Horn, Steve Branson, Scott Loarie, Serge Be- longie, and Pietro Perona. Lean multiclass crowdsourcing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2714–2723, 2018. 3

    2018

  58. [66]

    An overview of statistical learning the- ory

    Vladimir N Vapnik. An overview of statistical learning the- ory. IEEE transactions on neural networks, 10(5):988–999,

  59. [67]

    Glue: A multi-task benchmark and analysis platform for natural language un- derstanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language un- derstanding. arXiv preprint arXiv:1804.07461, 2018. 6

    2018 arXiv

  60. [68]

    The multidimensional wisdom of crowds

    Peter Welinder, Steve Branson, Pietro Perona, and Serge Be- longie. The multidimensional wisdom of crowds. Advances in neural information processing systems, 23, 2010. 3

    2010

  61. [69]

    Can we evaluate domain adaptation models with- out target-domain labels? arXiv preprint arXiv:2305.18712,

    Jianfei Yang, Hanjie Qian, Yuecong Xu, Kai Wang, and Li- hua Xie. Can we evaluate domain adaptation models with- out target-domain labels? arXiv preprint arXiv:2305.18712,

    arXiv

  62. [70]

    Towards accurate model selection in deep unsuper- vised domain adaptation

    Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selection in deep unsuper- vised domain adaptation. In International Conference on Machine Learning, pages 7124–7133. PMLR, 2019. 1, 2, 13 11 Supplemental Material

    2019

  63. [71]

    Success rate

    Additional results 9.1. Alternate metrics Here we report tabular results for several alternate metrics, providing additional points of comparison to supplement our main results of cumulative regret at step 100 (Tab. 1). 9.1.1. Variance between seeds Task RandomUncertaintyActiv...

  64. [72]

    Dawid-Skene data generating process cyi ci c 1.true class drawn from item categorical prior 2

    Implementation details 10.1. Dawid-Skene data generating process cyi ci c 1.true class drawn from item categorical prior 2. confusion rows drawn from dirichlets3. class draw indexes confusion matrix

  65. [73]

    A visual depiction of the Dawid-Skene [14] data gener- ating process that we adapt to active model selection

    prediction drawn from row categorical ~ ~ ~ ~ ~ x₃0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0 x₂ x₃ x₁ 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 1.0 x₂ x₃ x₁ 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.00.0 0.2 0.4 0.6 0.8 ...

  66. [74]

    Each data point’s true class label yi is drawn randomly from per-data-point prior distributions over which class that data point could be, yi ∼ Cat(π(xi))

  67. [75]

    To accommodate Bayesian updates, we initialize each θk,c to be a Dirich- let prior

    Each row of the classifier’s confusion matrix is drawn randomly from per-row distributions, Mk, c,· ∼ θk,c, where θk,c is the prior distribution over what the row of the confusion matrix could be. To accommodate Bayesian updates, we initialize each θk,c to be a Dirich- let prior

  68. [76]

    The sampled true class indexes into the corresponding row of the classifier’s confusion matrix, Mk, yi

  69. [77]

    The classifier’s prediction for that data point is sam- pled from the distribution over that row’s cells, ˆck,i ∼ Cat(Mk, yi ). 10.2. Computing PBest We illustrate visually the computation ofPBest from Sec. 4.3 in Fig. 10 in the simplified case of two models. To compute the pr...

  70. [78]

    source-only

    Data and model details We provide more details about the datasets and models in our benchmarking suite in Tab. 6, Tab. 7, and Tab. 8. Model 1 Probability Density Model 2 Cumulative Density Accuracy P(Model 1 best) fmodel1(p) Fmodel2(p) integrate p Figure 10. A visual depiction...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.