{"id":"c5e0e8bc-fbef-4e6e-81de-0bc4483af932","arxiv_id":"2507.22647","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Under prior probability shift, selecting hyperparameters via accuracy estimates on the unlabelled test batch (transductive model selection) outperforms cross-validation-based selection.","lead":"Machine learning models are usually tuned by checking accuracy on labeled data, but this fails when the test data has shifted class proportions. This paper proposes tuning classifiers directly on the unlabelled test batch by estimating accuracy under prior probability shift, and shows modest accuracy gains across 25 datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core mechanism—CAP estimates reliably ranking models on the shifted batch—is asserted but never directly validated, leaving the headline advantage unsupported.","rationale":"The reader's weakest assumption already pinpointed the CAP estimator's reliability, and my analysis agrees: the method's entire value proposition rests on the accuracy of the dAcc ranking. I found no additional flaw in the formal bias argument (Section 2.1) nor in the APP experimental setup; the PPS assumptions are simulated exactly, so internal inconsistency is not the issue. The missing piece is a direct validation of the selection mechanism. The paper's reported results are consistent with the claim, but they are also consistent with a weaker alternative: that TMS-All's advantage comes from the transductive per-batch selection protocol or from uncontrolled dataset effects, with CAP estimates merely correlated. The concrete test distinguishes these. Because the reader already assigned CONDITIONAL and flagged this exact assumption, my stress pass does not change the verdict; it strengthens the condition under which acceptance is appropriate. I recommend keeping CONDITIONAL (i.e., UNCHANGED relative to the reader's verdict) until the CAP ranking quality is directly reported.","tokens_in":11062,"tokens_out":5684,"duration_ms":72217,"concrete_test":"On a representative subset of the 25 datasets, rerun the TMS-All protocol while logging, for every bag Ui and every candidate θ, both the CAP estimate dAcc and the true accuracy on Ui (available since the simulation knows the labels). Compute per-bag Spearman rank correlation between dAcc and true accuracy across candidates, and the per-bag agreement rate between argmax_θ dAcc and argmax_θ (true accuracy). Also run a control where dAcc is replaced by random noise (or by the trivial predicted-class-proportion baseline) and compare TMS-All's final accuracy against IMS-All. If median rank correlation is below ~0.5, or if the random-CAP control matches or exceeds the real TMS-All, the central claim that CAP quality drives the advantage is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that TMS-All beats IMS-All because the O-LEAP KDEy accuracy estimates produced on the unlabelled batch correctly rank the candidate models. This mechanism is the whole method: Algorithm 2 sets h*θ = argmax_θ ψ_hθ(Ui). Yet the evaluation only reports the final accuracy of the chosen model; it never reports the accuracy, calibration, or ranking quality of the dAcc estimates themselves. With many candidate hyperparameter configurations per dataset, even unbiased but noisy CAP estimates can make the argmax select a model far from the true best, and the observed average gain of ~2.6 points could be driven by a few datasets rather than by reliable ranking. Figure 1's oracle gap is indirect evidence at best: it does not show how often TMS-All actually identifies the oracle's choice. A no-skill CAP baseline (e.g., random dAcc) is not included, so there is no ablation establishing that the advantage requires CAP quality. This is load-bearing: if the CAP estimates are not sufficiently reliable to rank models, the paper's proposed mechanism does not explain the experimental results, and the conditional acceptance hinges on an unverified component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes transductive model selection (TMS), a method for hyperparameter optimization and model selection when the unlabeled test batch is affected by prior probability shift (PPS). The authors argue that standard inductive model selection (IMS) via cross-validation is biased under PPS, prove this for binary problems in Section 2.1, and then present a method that uses classifier accuracy prediction (CAP) on the unlabeled batch to select the best candidate model. The CAP component is O-LEAP KDEy from the authors' prior work. Experiments on 25 UCI datasets use the Artificial Prevalence Protocol to simulate PPS, compare TMS-All against IMS-All and other baselines, and report that TMS-All achieves higher average accuracy (0.771 vs 0.745) and follows the oracle more closely under strong shift.","tokens_in":11167,"tokens_out":7756,"duration_ms":87432,"significance":"If the result holds, this is a useful contribution: it gives a principled reason to distrust cross-validation under PPS and provides a concrete, reproducible method for selecting models on the unlabeled batch. The formal binary-case argument in Section 2.1 is correct and relies on the standard lemma that P(X|Y)=Q(X|Y) implies P(f(X)|Y)=Q(f(X)|Y). The experimental protocol is strong: PPS is simulated exactly through APP, 1000 bags are drawn per dataset, and code is publicly available. The comparison against IMS and TSVM baselines is sensible. However, the central mechanism of the method, namely that the CAP estimates ψ_hθ(Ui) are reliable enough to rank models on the unlabeled batch, is not directly validated; this is the main weakness. The multiclass extension of the theoretical argument is also asserted without proof.","major_comments":[{"comment":"The central mechanism of TMS is that the CAP estimates dAcc = ψ_hθ(Ui) correctly rank the candidate models, since Algorithm 2 selects h*θ = argmax_θ dAcc. However, the paper reports only the final accuracy of the selected model and never validates the quality of these estimates (e.g., correlation or rank correlation between dAcc and true accuracy, calibration, or the fraction of bags in which TMS-All selects the model chosen by the oracle). The oracle gap in Figure 1 is indirect evidence: it does not show how often TMS-All identifies the oracle's choice. Moreover, there is no no-skill baseline (e.g., random selection among candidates) to confirm that the advantage over IMS-All is due to CAP informativeness rather than to some other property of the selection procedure. This is load-bearing because the proposed explanation of the experimental results is exactly that CAP estimates are reliable enough to rank models; without this validation, the 'substantial performance advantage' is an end-to-end result without a demonstrated mechanism. Please add direct ranking-quality metrics and/or a no-skill ablation.","section":"§3, Algorithm 2, Table 1"},{"comment":"The formal argument is developed only for binary problems; the sentence 'A similar reasoning holds for the multiclass case' asserts without proof that CV accuracy is also biased under PPS for multiclass classifiers. Since the experiments include multiclass datasets with up to 26 classes, the theoretical motivation for the multiclass setting is unproven. Either provide the multiclass proof or explicitly restrict the theoretical claim to the binary case.","section":"§2.1, Eq. (3)"},{"comment":"The headline claim of a substantial advantage is supported only by a difference in average accuracy (0.771 vs 0.745) across the 25 datasets. No aggregate paired significance test is reported. The per-dataset Wilcoxon tests compare methods against the best method for that dataset, not TMS-All against IMS-All directly. On several datasets TMS-All is numerically worse than IMS-All (e.g., poker-hand, chess, letter, nursery, hand-digits, digits), so a paired test across datasets or across the 1000 bags is needed to establish that the advantage is not driven by a few datasets. This is load-bearing for the central empirical claim.","section":"§3, Table 1"}],"minor_comments":[{"comment":"The running title in the header reads 'Tranductive Model Selection under Prior Probability Shift' (typo for 'Transductive').","section":"Header"},{"comment":"The column header is rendered ambiguously (e.g., '∅ IMS TMS∅-LR'); the header should clearly separate the three groups and individual columns.","section":"Table 1"},{"comment":"The description of the binary class-weight grid G = (0.2, 0.4, 0.6, 0.8) could state more explicitly that the assignments are (g, 1−g) for each g in G; the current phrase 'all combinations' is slightly confusing.","section":"§3, Class weights"},{"comment":"References [6] and [7] appear to be the same work (ICLR and arXiv versions); consider citing only one.","section":"References"},{"comment":"The term 'oracle' is not formally defined in the text; a one-sentence definition would improve clarity.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope. The main risk is that the empirical advantage is attributed to CAP reliability without direct evidence; the authors' prior CAP paper [18] should be checked for whether O-LEAP KDEy is known to rank models reliably under PPS. The requested additions (ranking-quality metrics, a no-skill baseline, and an aggregate significance test) are all feasible within the current experimental setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is new and clean: instead of doing cross-validation on labeled training data, choose hyperparameters by estimating accuracy on the unlabeled batch via CAP methods tailored to PPS. That framing is not present in the cited CAP papers, and the paper shows empirically that it helps. The experimental setup is careful—PPS is simulated via APP so the assumptions hold exactly, 1000 bags per dataset, 25 datasets, and code is on GitHub. The formal argument that CV accuracy is biased under PPS is correct for the binary case and properly rests on the external lemma from [11]; the multiclass extension is asserted without proof, which is a minor gap.\n\nThe soft spots are real but not fatal. The stress-test note is right that the paper never directly checks whether the dAcc estimates from O-LEAP KDEy rank the candidate models correctly. That is the mechanism behind the whole method. Figure 1's oracle gap is indirect evidence, and the paper would be much stronger with a scatter of estimated vs. true accuracy, or a report of how often TMS-All picks the oracle's choice. A no-skill baseline (random selection among candidates) would also help; without it, some of the gain could just come from having a large pool of models with at least a few good ones. The claim of a \"substantial performance advantage\" overstates what is a modest ~2.6 point average gain, and there is no global paired significance test, only per-dataset Wilcoxon superscripts. Still, the advantage is consistent: TMS-All wins or ties on most datasets and the oracle gap narrows as shift increases.\n\nThe self-citation of O-LEAP KDEy is appropriate—it is the natural component for this task, and the paper does not hide that the method is built from prior work. The writing is clear and the scope is honestly stated.\n\nWho is this for? Researchers working on dataset shift, quantification, or model selection in anti-causal problems. It deserves a serious referee. My recommendation: send it to peer review, but ask the authors to add a direct analysis of the CAP estimates' ranking quality and a random-selection baseline. That would close the load-bearing gap.\n\nI would not cite it in my own work in the next year, but that is a scope choice, not a quality judgment.","headline":"Solid, carefully controlled application of CAP to transductive model selection under prior probability shift; the main caveat is that the ranking quality of the CAP estimates is never directly validated, though the oracle gap provides indirect support.","tokens_in":11802,"tokens_out":1724,"would_cite":false,"duration_ms":23224,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that under prior probability shift, model selection by estimating accuracy on the unlabelled batch outperforms cross-validation on training data, and demonstrates this advantage across 25 datasets.","keywords":["model selection","hyperparameter optimisation","classifier accuracy prediction","prior probability shift","label shift","transductive learning","dataset shift","quantification"],"falsifier":"On a collection of test bags with known labels, compute the true accuracy of every candidate model and the O-LEAP KDEy estimate for each bag; if on any dataset the model ranked first by the estimator has lower true accuracy than the model ranked first by cross-validation, the claimed TMS advantage fails for that dataset.","tokens_in":10739,"feed_emoji":"🎯","tokens_out":5058,"duration_ms":51932,"temperature":0.7,"pith_summary":"When data arrive in batches whose class proportions differ from the training set (prior probability shift), the standard way of choosing a classifier—cross-validation on labelled training data—systematically picks suboptimal models. The paper proposes transductive model selection (TMS): for each unlabelled batch, estimate each candidate model's accuracy directly on that batch using a classifier accuracy prediction method tailored to prior probability shift, then deploy the model with the highest estimated accuracy. Experiments across 25 datasets with many shifted test bags per dataset show that TMS-based selection outperforms cross-validation-based selection, and the advantage grows with the amount of shift. The paper also proves that training-set accuracy estimates cannot be trusted under prior probability shift unless the classifier has equal true positive and true negative rates, which is not true in general.","feed_headline":"Tune models on unlabelled data, not training folds","feed_subtitle":"When class proportions shift, estimating accuracy on the target batch beats cross-validation, and grows stronger with shift.","key_machinery":"The load-bearing component is the CAP method O-LEAP KDEy, from the LEAP family: it estimates the cells of the classifier's contingency table on the unlabelled batch by solving a system of linear equations encoding the prior-probability-shift assumptions, using KDEy-ML as an internal quantifier for class proportions; any accuracy measure then follows from the table. The argument that this is needed rests on the observation that class-conditional error rates are invariant under prior probability shift but class priors are not, so accuracy on training data is a biased estimate of accuracy on the batch. In algorithmic terms, TMS chooses the model maximizing the estimated accuracy, expressed as the CAP prediction evaluated directly on the unlabelled batch.","core_discovery":"Under prior probability shift, model selection should be transductive: rather than choosing hyperparameters by cross-validation on the training distribution, choose them by estimating accuracy on the specific unlabelled batch to be classified. The paper formalizes why cross-validation fails: for any classifier, the accuracy on the training distribution equals the accuracy on the shifted batch only if true positive rate equals true negative rate, so the IID accuracy estimate is biased. TMS replaces the validation-accuracy step with a classifier accuracy prediction (CAP) estimate, in particular O-LEAP KDEy, which solves a linear system under the prior-probability-shift assumptions to estimate the contingency table on the unlabelled batch. In the experiments, TMS-All—selecting the best model across classifier types and hyperparameter configurations by CAP-estimated accuracy—achieves higher accuracy than IMS-All and per-type IMS baselines, and its accuracy approaches the oracle that always picks the true best model as the amount of shift grows.","pith_inferences":["The paper's formal argument implies that any model selection criterion based on labelled training data is biased under prior probability shift whenever the classifier's true positive rate differs from its true negative rate, which extends the critique beyond the tested hyperparameter grid to Bayesian optimization and architecture search run on training data.","Because the advantage grows with shift amount, TMS appears most valuable in volatile environments; an online variant that re-estimates accuracy per arriving batch and switches models accordingly is a natural next step not explored in the paper.","The paper does not directly validate the CAP estimates themselves; a practical safeguard would be to compare TMS's chosen model against the IMS pick per batch and flag large disagreements, since those are the cases where the CAP estimator is most likely to err.","The method's practical benefit is bounded by the accuracy of the underlying KDEy-ML quantifier; improvements in quantification accuracy would directly translate into better model selection, while poor quantification on high-dimensional or sparse batches would erode the advantage."],"forward_implications":["TMS-All outperforms IMS-All and single-classifier IMS baselines under prior probability shift on the 25 datasets tested.","The advantage over IMS grows with the L1 distance between training and batch class proportions; for near-IID batches, TMS and IMS perform similarly.","TMS accuracy approaches the oracle that always picks the best model as shift increases, whereas IMS accuracy degrades.","TMS is not restricted to prior probability shift: swapping the CAP method allows handling other shifts such as covariate shift.","TMS is suited to strictly transductive applications like technology-assisted review, e-discovery, content moderation, and systematic review production."],"supporting_citations":[{"why":"Supplies the O-LEAP KDEy CAP method that TMS uses to estimate accuracy on the unlabelled batch.","marker":"[18]"},{"why":"Provides KDEy-ML, the quantification method internally used by LEAP to estimate class proportions.","marker":"[12]"},{"why":"Provides Lemma 1 guaranteeing P(f(X)|Y)=Q(f(X)|Y) for any measurable f, the step that makes the bias proof work.","marker":"[11]"},{"why":"Earlier evidence that cross-validation is biased under dataset shift, motivating the need for TMS.","marker":"[17]"},{"why":"Defines the Artificial Prevalence Protocol used to simulate prior probability shift in the experiments.","marker":"[4]"},{"why":"Provides background on quantification and the prevalence sampling procedure used to build test bags.","marker":"[1]"},{"why":"Supplies the algorithm used to sample prevalence vectors uniformly from the unit simplex.","marker":"[15]"},{"why":"Defines transductive inference and serves as the TSVM baseline that TMS is compared against.","marker":"[5]"}],"fun_headline_variants":["Tune hyperparameters on the unlabelled target batch","When class priors shift, validate on the test data instead","Transductive model selection: tune on the batch you'll classify","Beat cross-validation by tuning on shifted unlabelled data","Choose models via target-batch accuracy, not training folds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CAP method O-LEAP KDEy must estimate accuracy on the unlabelled batch accurately enough that the argmax over candidate models identifies a genuinely better model than cross-validation would pick.","fun_headline_variants_meta":{"raw":{"variants":["Tune hyperparameters on the unlabelled target batch","When class priors shift, validate on the test data instead","Transductive model selection: tune on the batch you'll classify","Beat cross-validation by tuning on shifted unlabelled data","Choose models via target-batch accuracy, not training folds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1274,"prompt_tokens":887,"completion_tokens":387,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":503,"tokens_out":387,"duration_ms":4226,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:26:46.776985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a collection of test bags with known labels, compute the true accuracy of every candidate model and the O-LEAP KDEy estimate for each bag; if on any dataset the model ranked first by the estimator has lower true accuracy than the model ranked first by cross-validation, the claimed TMS advantage fails for that dataset.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the O-LEAP KDEy CAP method that TMS uses to estimate accuracy on the unlabelled batch."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides KDEy-ML, the quantification method internally used by LEAP to estimate class proportions."},{"cited_title":"C., Wang, Y ., and Smola, A","cited_arxiv_id":null,"evidence_quote":"Provides Lemma 1 guaranteeing P(f(X)|Y)=Q(f(X)|Y) for any measurable f, the step that makes the bias proof work."},{"cited_title":"and M ¨uller, K","cited_arxiv_id":null,"evidence_quote":"Earlier evidence that cross-validation is biased under dataset shift, motivating the need for TMS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Artificial Prevalence Protocol used to simulate prior probability shift in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides background on quantification and the prevalence sampling procedure used to build test bags."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the algorithm used to sample prevalence vectors uniformly from the unit simplex."},{"cited_title":"G., and Vapnik, V","cited_arxiv_id":null,"evidence_quote":"Defines transductive inference and serves as the TSVM baseline that TMS is compared against."}],"review_version":1}