Transductive Model Selection under Prior Probability Shift

Alejandro Moreo; Fabrizio Sebastiani; Lorenzo Volpi

REVIEW 3 major objections 5 minor 20 references

Transductive Model Selection under Prior Probability Shift

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that under prior probability shift, model selection by estimating accuracy on the unlabelled batch outperforms cross-validation on training data, and demonstrates this advantage across 25 datasets.

desk verdict Solid, carefully controlled application of CAP to transductive model selection under prior probability shift; the main caveat is that the ranking quality of the CAP estimates is never directly validated, though the oracle gap provides indirect support. read the letter →

arxiv 2507.22647 v1 pith:O4PL34CO submitted 2025-07-30 cs.LG

Lorenzo Volpi , Alejandro Moreo , Fabrizio Sebastiani This is my paper

classification cs.LG

keywords modelselectionhyperparameteroptimisationclassifieraccuracypredictionpriorprobabilityshiftlabeltransductivelearningdatasetquantification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When data arrive in batches whose class proportions differ from the training set (prior probability shift), the standard way of choosing a classifier—cross-validation on labelled training data—systematically picks suboptimal models. The paper proposes transductive model selection (TMS): for each unlabelled batch, estimate each candidate model's accuracy directly on that batch using a classifier accuracy prediction method tailored to prior probability shift, then deploy the model with the highest estimated accuracy. Experiments across 25 datasets with many shifted test bags per dataset show that TMS-based selection outperforms cross-validation-based selection, and the advantage grows with the amount of shift. The paper also proves that training-set accuracy estimates cannot be trusted under prior probability shift unless the classifier has equal true positive and true negative rates, which is not true in general.

What carries the argument

The load-bearing component is the CAP method O-LEAP KDEy, from the LEAP family: it estimates the cells of the classifier's contingency table on the unlabelled batch by solving a system of linear equations encoding the prior-probability-shift assumptions, using KDEy-ML as an internal quantifier for class proportions; any accuracy measure then follows from the table. The argument that this is needed rests on the observation that class-conditional error rates are invariant under prior probability shift but class priors are not, so accuracy on training data is a biased estimate of accuracy on the batch. In algorithmic terms, TMS chooses the model maximizing the estimated accuracy, expressed as the CAP prediction evaluated directly on the unlabelled batch.

What would settle it

On a collection of test bags with known labels, compute the true accuracy of every candidate model and the O-LEAP KDEy estimate for each bag; if on any dataset the model ranked first by the estimator has lower true accuracy than the model ranked first by cross-validation, the claimed TMS advantage fails for that dataset.

Watch

Extended reading notes

Core claim

Under prior probability shift, model selection should be transductive: rather than choosing hyperparameters by cross-validation on the training distribution, choose them by estimating accuracy on the specific unlabelled batch to be classified. The paper formalizes why cross-validation fails: for any classifier, the accuracy on the training distribution equals the accuracy on the shifted batch only if true positive rate equals true negative rate, so the IID accuracy estimate is biased. TMS replaces the validation-accuracy step with a classifier accuracy prediction (CAP) estimate, in particular O-LEAP KDEy, which solves a linear system under the prior-probability-shift assumptions to estimate the contingency table on the unlabelled batch. In the experiments, TMS-All—selecting the best model across classifier types and hyperparameter configurations by CAP-estimated accuracy—achieves higher accuracy than IMS-All and per-type IMS baselines, and its accuracy approaches the oracle that always picks the true best model as the amount of shift grows.

Load-bearing premise

The CAP method O-LEAP KDEy must estimate accuracy on the unlabelled batch accurately enough that the argmax over candidate models identifies a genuinely better model than cross-validation would pick.

Editorial extensions

If this is right

TMS-All outperforms IMS-All and single-classifier IMS baselines under prior probability shift on the 25 datasets tested.
The advantage over IMS grows with the L1 distance between training and batch class proportions; for near-IID batches, TMS and IMS perform similarly.
TMS accuracy approaches the oracle that always picks the best model as shift increases, whereas IMS accuracy degrades.
TMS is not restricted to prior probability shift: swapping the CAP method allows handling other shifts such as covariate shift.
TMS is suited to strictly transductive applications like technology-assisted review, e-discovery, content moderation, and systematic review production.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's formal argument implies that any model selection criterion based on labelled training data is biased under prior probability shift whenever the classifier's true positive rate differs from its true negative rate, which extends the critique beyond the tested hyperparameter grid to Bayesian optimization and architecture search run on training data.
Because the advantage grows with shift amount, TMS appears most valuable in volatile environments; an online variant that re-estimates accuracy per arriving batch and switches models accordingly is a natural next step not explored in the paper.
The paper does not directly validate the CAP estimates themselves; a practical safeguard would be to compare TMS's chosen model against the IMS pick per batch and flag large disagreements, since those are the cases where the CAP estimator is most likely to err.
The method's practical benefit is bounded by the accuracy of the underlying KDEy-ML quantifier; improvements in quantification accuracy would directly translate into better model selection, while poor quantification on high-dimensional or sparse batches would erode the advantage.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is new and clean: instead of doing cross-validation on labeled training data, choose hyperparameters by estimating accuracy on the unlabeled batch via CAP methods tailored to PPS. That framing is not present in the cited CAP papers, and the paper shows empirically that it helps. The experimental setup is careful—PPS is simulated via APP so the assumptions hold exactly, 1000 bags per dataset, 25 datasets, and code is on GitHub. The formal argument that CV accuracy is biased under PPS is correct for the binary case and properly rests on the external lemma from [11]; the multiclass extension is asserted without proof, which is a minor gap.

The soft spots are real but not fatal. The stress-test note is right that the paper never directly checks whether the dAcc estimates from O-LEAP KDEy rank the candidate models correctly. That is the mechanism behind the whole method. Figure 1's oracle gap is indirect evidence, and the paper would be much stronger with a scatter of estimated vs. true accuracy, or a report of how often TMS-All picks the oracle's choice. A no-skill baseline (random selection among candidates) would also help; without it, some of the gain could just come from having a large pool of models with at least a few good ones. The claim of a "substantial performance advantage" overstates what is a modest ~2.6 point average gain, and there is no global paired significance test, only per-dataset Wilcoxon superscripts. Still, the advantage is consistent: TMS-All wins or ties on most datasets and the oracle gap narrows as shift increases.

The self-citation of O-LEAP KDEy is appropriate—it is the natural component for this task, and the paper does not hide that the method is built from prior work. The writing is clear and the scope is honestly stated.

Who is this for? Researchers working on dataset shift, quantification, or model selection in anti-causal problems. It deserves a serious referee. My recommendation: send it to peer review, but ask the authors to add a direct analysis of the CAP estimates' ranking quality and a random-selection baseline. That would close the load-bearing gap.

I would not cite it in my own work in the next year, but that is a scope choice, not a quality judgment.

Referee Report

3 major / 5 minor

Summary. The paper proposes transductive model selection (TMS), a method for hyperparameter optimization and model selection when the unlabeled test batch is affected by prior probability shift (PPS). The authors argue that standard inductive model selection (IMS) via cross-validation is biased under PPS, prove this for binary problems in Section 2.1, and then present a method that uses classifier accuracy prediction (CAP) on the unlabeled batch to select the best candidate model. The CAP component is O-LEAP KDEy from the authors' prior work. Experiments on 25 UCI datasets use the Artificial Prevalence Protocol to simulate PPS, compare TMS-All against IMS-All and other baselines, and report that TMS-All achieves higher average accuracy (0.771 vs 0.745) and follows the oracle more closely under strong shift.

Significance. If the result holds, this is a useful contribution: it gives a principled reason to distrust cross-validation under PPS and provides a concrete, reproducible method for selecting models on the unlabeled batch. The formal binary-case argument in Section 2.1 is correct and relies on the standard lemma that P(X|Y)=Q(X|Y) implies P(f(X)|Y)=Q(f(X)|Y). The experimental protocol is strong: PPS is simulated exactly through APP, 1000 bags are drawn per dataset, and code is publicly available. The comparison against IMS and TSVM baselines is sensible. However, the central mechanism of the method, namely that the CAP estimates ψ_hθ(Ui) are reliable enough to rank models on the unlabeled batch, is not directly validated; this is the main weakness. The multiclass extension of the theoretical argument is also asserted without proof.

major comments (3)

[§3, Algorithm 2, Table 1] The central mechanism of TMS is that the CAP estimates dAcc = ψ_hθ(Ui) correctly rank the candidate models, since Algorithm 2 selects h*θ = argmax_θ dAcc. However, the paper reports only the final accuracy of the selected model and never validates the quality of these estimates (e.g., correlation or rank correlation between dAcc and true accuracy, calibration, or the fraction of bags in which TMS-All selects the model chosen by the oracle). The oracle gap in Figure 1 is indirect evidence: it does not show how often TMS-All identifies the oracle's choice. Moreover, there is no no-skill baseline (e.g., random selection among candidates) to confirm that the advantage over IMS-All is due to CAP informativeness rather than to some other property of the selection procedure. This is load-bearing because the proposed explanation of the experimental results is exactly that CAP estimates are reliable enough to rank models; without this validation, the 'substantial performance advantage' is an end-to-end result without a demonstrated mechanism. Please add direct ranking-quality metrics and/or a no-skill ablation.
[§2.1, Eq. (3)] The formal argument is developed only for binary problems; the sentence 'A similar reasoning holds for the multiclass case' asserts without proof that CV accuracy is also biased under PPS for multiclass classifiers. Since the experiments include multiclass datasets with up to 26 classes, the theoretical motivation for the multiclass setting is unproven. Either provide the multiclass proof or explicitly restrict the theoretical claim to the binary case.
[§3, Table 1] The headline claim of a substantial advantage is supported only by a difference in average accuracy (0.771 vs 0.745) across the 25 datasets. No aggregate paired significance test is reported. The per-dataset Wilcoxon tests compare methods against the best method for that dataset, not TMS-All against IMS-All directly. On several datasets TMS-All is numerically worse than IMS-All (e.g., poker-hand, chess, letter, nursery, hand-digits, digits), so a paired test across datasets or across the 1000 bags is needed to establish that the advantage is not driven by a few datasets. This is load-bearing for the central empirical claim.

minor comments (5)

[Header] The running title in the header reads 'Tranductive Model Selection under Prior Probability Shift' (typo for 'Transductive').
[Table 1] The column header is rendered ambiguously (e.g., '∅ IMS TMS∅-LR'); the header should clearly separate the three groups and individual columns.
[§3, Class weights] The description of the binary class-weight grid G = (0.2, 0.4, 0.6, 0.8) could state more explicitly that the assignments are (g, 1−g) for each g in G; the current phrase 'all combinations' is slightly confusing.
[References] References [6] and [7] appear to be the same work (ICLR and arXiv versions); consider citing only one.
[Figure 1] The term 'oracle' is not formally defined in the text; a one-sentence definition would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TMS reuses a published CAP estimator as an off-the-shelf component, and the headline result is evaluated on held-out bag accuracy rather than on the CAP estimates themselves.

full rationale

The derivation chain is not circular. Section 2.1's proof that cross-validation accuracy is unreliable under prior probability shift relies on Equations (1)-(3) and on an external lemma from Lipton et al. [11]; it does not assume the TMS conclusion. Algorithm 2 defines TMS as selecting the classifier that maximizes the CAP estimate dAcc = ψ_hθ(Ui), which is exactly the proposed method rather than a hidden reduction of the output to the input; the paper does not claim to derive the CAP estimator from first principles. O-LEAP KDEy is adopted from the authors' prior work [18] and disclosed as such, but the paper's central experimental claim is measured as the true accuracy of the selected model on held-out bags under the Artificial Prevalence Protocol, a quantity defined independently of the CAP estimates. No fitted parameter is renamed as a prediction: the comparison is between model selection strategies, and the reported accuracy is not the CAP estimate itself. The oracle curve in Figure 1 is an external benchmark, and the absence of a no-skill CAP ablation is a robustness or correctness concern, not a circularity. The only self-citation is [18] for the CAP component; that citation is not used to prove the empirical advantage, which is supported by the experiments reported in Table 1. Therefore no circular step can be exhibited under the hard rules requiring a quote and a specific reduction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

Free parameters are confined to experimental protocol choices (class weight grids) explicitly marked arbitrary by the authors; the TMS algorithm itself introduces no fitted parameters. Axioms are the PPS assumptions, the external lemma used in the formal proof, the unproven reliability of the CAP estimator for model ranking, and the asserted multiclass extension of the bias argument. No invented entities.

free parameters (2)

binary class weight grid = G = (0.2, 0.4, 0.6, 0.8)
Hand-chosen grid for binary class weights (Section 3, Hyperparameters); defines the candidate pool of class-weight configurations but is not a parameter of the TMS method itself.
multiclass high class weight = 2/n (e.g., 0.66 for n=3)
Arbitrary high weight in the multiclass class-weight pool, explicitly acknowledged as arbitrary by the authors in footnote 4; affects the candidate pool and thus the comparison, but not the TMS methodology.

assumptions (4)

domain assumption PPS assumptions hold for the application domain
Section 2 defines PPS as P(Y) not equal to Q(Y) and P(X|Y) = Q(X|Y); the method and experiments assume these conditions are met.
standard math Lemma 1 from Lipton et al. [11]
Section 2.1 uses the result that P(X|Y)=Q(X|Y) implies P(f(X)|Y)=Q(f(X)|Y) for any deterministic measurable f, cited to [11].
domain assumption O-LEAP KDEy provides accurate accuracy estimates on unlabelled batches under PPS
TMS's argmax selection presumes CAP estimates rank candidate models correctly; this is not proven in the paper, but adopted from the authors' prior work [18].
ad hoc to paper The multiclass extension of the CV-bias argument holds
Section 2.1 states 'a similar reasoning holds for the multiclass case' without proof; the motivation for TMS in multiclass settings depends on this extension.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Transductive Model Selection under Prior Probability Shift." pith.science (2026). https://pith.science/paper/O4PL34CO

@misc{pith2026250722647,
  author       = {Pith},
  title        = {Pith review of: Transductive Model Selection under Prior Probability Shift},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O4PL34CO}},
  note         = {Machine review of arXiv:2507.22647}
}

read the original abstract

Transductive learning is a supervised machine learning task in which, unlike in traditional inductive learning, the unlabelled data that require labelling are a finite set and are available at training time. Similarly to inductive learning contexts, transductive learning contexts may be affected by dataset shift, i.e., may be such that the IID assumption does not hold. We here propose a method, tailored to transductive classification contexts, for performing model selection (i.e., hyperparameter optimisation) when the data exhibit prior probability shift, an important type of dataset shift typical of anti-causal learning problems. In our proposed method the hyperparameters can be optimised directly on the unlabelled data to which the trained classifier must be applied; this is unlike traditional model selection methods, that are based on performing cross-validation on the labelled training data. We provide experimental results that show the benefits brought about by our method.

Figures

Figures reproduced from arXiv: 2507.22647 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

[11]

C., Wang, Y ., and Smola, A

Lipton, Z. C., Wang, Y ., and Smola, A. J. (2018). Detecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), pages 3128– 3136, Stockholm, SE

work page 2018
[1]

Esuli, A., Fabris, A., Moreo, A., and Sebastiani, F. (2023). Learning to quantify. Springer Nature, Cham, CH

work page 2023
[2]

and Flach, P

Fawcett, T. and Flach, P. (2005). A response to Webb and Ting’s ‘On the application of ROC analysis to predict classification performance under varying class distributions’. Machine Learning, 58(1):33–38

work page 2005
[3]

L., Tummers, L., Teijema, J

Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., and van de Schoot, R. (2023). Performance of active learning models for screening prioritization in systematic reviews: A simulation study into the average time to discover relevant records. Systematic Reviews, 10(100)

work page 2023
[4]

Forman, G. (2008). Quantifying counts and costs via classification. Data Mining and Knowledge Discovery , 17(2):164–206

work page 2008
[5]

G., and Vapnik, V

Gammerman, A., V ovk, V . G., and Vapnik, V . (1998). Learning by transduction. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI 1998), pages 148–155, Madison, US

work page 1998
[6]

C., Neyshabur, B., and Sedghi, H

Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. (2022a). Leveraging unlabeled data to predict out-of-distribution performance. In Proceedings of the 10th International Conference on Learning Repre- sentations (ICLR 2022), Virtual Event

work page 2022
[7]

C., Neyshabur, B., and Sedghi, H

Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. (2022b). Leveraging unlabeled data to predict out-of-distribution performance. arXiv:2201.04234 [cs.LG]

arXiv 2022

Show all 20 references

[8]

D., Pickens, J., and Yang, E

Gray, L., Lewis, D. D., Pickens, J., and Yang, E. (2024). High-recall retrieval via technology-assisted review. In Proceedings of the 47th ACM Conference on Research and Development in Information Retrieval (SIGIR 2024) , pages 2987–2988, Washington, US

2024
[9]

Guillory, D., Shankar, V ., Ebrahimi, S., Darrell, T., and Schmidt, L. (2021). Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2021), pages 1134—-1144, Montreal, CA

2021
[10]

Joachims, T. (1999). Transductive inference for text classification using support vector machines. InProceedings of the 16th International Conference on Machine Learning (ICML 1999), pages 200–209, Bled, SL

1999
[12]

Moreo, A., Gonz ´alez, P., and del Coz, J. J. (2025). Kernel density estimation for multiclass quantification. Machine Learning, 114(4)

2025
[13]

Oard, D. W. and Webber, W. (2013). Information retrieval for e-discovery.Foundations and Trends in Information Retrieval, 7(2/3):99–237

2013
[14]

Sch ¨olkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., and Mooij, J. M. (2012). On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012), Edinburgh, UK

2012
[15]

Smith, N. A. and Tromble, R. W. (2004). Sampling uniformly from the unit simplex. Technical report, Johns Hopkins University. https://www.cs.cmu.edu/~nasmith/papers/smith+tromble.tr04.pdf

2004
[16]

Storkey, A. (2009). When training and test sets are different: Characterizing learning transfer. In Qui ˜nonero- Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D., editors, Dataset shift in machine learning , pages 3–28. The MIT Press, Cambridge, US

2009
[17]

and M ¨uller, K

Sugiyama, M. and M ¨uller, K. (2005). Model selection under covariate shift. In Proceedings of the 15th Interna- tional Conference on Artificial Neural Networks (ICANN 2005), pages 235–240, Warsaw, PL

2005
[18]

V olpi, L., Moreo, A., and Sebastiani, F. (2024). A simple method for classifier accuracy prediction under prior probability shift. In Proceedings of the 27th International Conference on Discovery Science (DS 2024) , pages 267–283, Pisa, IT. 8 Tranductive Model Selection under...

2024
[19]

D., and Frieder, O

Yang, E., Lewis, D. D., and Frieder, O. (2021). TAR on social media: A framework for online content moderation. In Proceedings of the 2nd International Conference on Design of Experimental Search & Information REtrieval Systems (DESIRES 2021), pages 147–155, Padova, IT

2021
[20]

and Czy ˙z, P

Ziegler, A. and Czy ˙z, P. (2024). Bayesian quantification with black-box estimators. Transactions on Machine Learning Research, 2024. 9

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [11]

C., Wang, Y ., and Smola, A

Lipton, Z. C., Wang, Y ., and Smola, A. J. (2018). Detecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), pages 3128– 3136, Stockholm, SE

work page 2018

[2] [1]

Esuli, A., Fabris, A., Moreo, A., and Sebastiani, F. (2023). Learning to quantify. Springer Nature, Cham, CH

work page 2023

[3] [2]

and Flach, P

Fawcett, T. and Flach, P. (2005). A response to Webb and Ting’s ‘On the application of ROC analysis to predict classification performance under varying class distributions’. Machine Learning, 58(1):33–38

work page 2005

[4] [3]

L., Tummers, L., Teijema, J

Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., and van de Schoot, R. (2023). Performance of active learning models for screening prioritization in systematic reviews: A simulation study into the average time to discover relevant records. Systematic Reviews, 10(100)

work page 2023

[5] [4]

Forman, G. (2008). Quantifying counts and costs via classification. Data Mining and Knowledge Discovery , 17(2):164–206

work page 2008

[6] [5]

G., and Vapnik, V

Gammerman, A., V ovk, V . G., and Vapnik, V . (1998). Learning by transduction. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence (UAI 1998), pages 148–155, Madison, US

work page 1998

[7] [6]

C., Neyshabur, B., and Sedghi, H

Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. (2022a). Leveraging unlabeled data to predict out-of-distribution performance. In Proceedings of the 10th International Conference on Learning Repre- sentations (ICLR 2022), Virtual Event

work page 2022

[8] [7]

C., Neyshabur, B., and Sedghi, H

Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi, H. (2022b). Leveraging unlabeled data to predict out-of-distribution performance. arXiv:2201.04234 [cs.LG]

arXiv 2022