{"id":"6c139b83-1243-4b20-8064-330a360b34d3","arxiv_id":"2505.05868","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A three-stage estimator recovers target label proportions and the known-data fraction under open set label shift, and corrects a frozen classifier to the target without retraining.","lead":"This paper estimates how the mix of known image classes and an extra unknown class changes between training and test data, using a frozen classifier and an out-of-distribution detector with a reference set of unknown samples. It then adapts the original classifier to the new mix without retraining, which matters for deploying pre-trained models in changing environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-OOD reference and ad hoc T rescaling fall outside the proved guarantees, so the estimates used in experiments can be biased by an arbitrary calibration factor.","rationale":"The paper's theoretical skeleton is clean: the transformation of OSLS to a K+1-class closed-set likelihood in Lemma 4.2 is sound given Assumptions 3.2 and 3.3, and the EM derivation in Appendix C follows the MLLS/MAPLS template. Theorems 4.1 and 4.4 are coherent as stated, assuming a genuine OOD reference dataset. The load-bearing weakness is exactly the gap between the assumed OOD reference and the deployed pseudo-OOD construction. The reader's weakest_assumption identifies the same issue, and the concrete test above would determine whether the gap is empirically benign or whether the reported results depend on the arbitrary choice of T and on matching the wrong OOD distribution. This does not move the verdict: the manuscript is a solid conditional accept that needs either a proof of bias bounds for the pseudo-OOD estimator or a sensitivity analysis over γ and T, plus a benchmark against Garg et al. (2022).","tokens_in":61416,"tokens_out":9769,"duration_ms":100512,"concrete_test":"On CIFAR100 with the OpenMax OOD classifier and the LT10-forward r=1 setting of Table 18, run the full pipeline with three different OOD references while keeping everything else fixed: (a) the paper's pseudo-OOD set Doγ with γ=0.1, T=2; (b) a held-out subset of the actual Near OOD distribution used at test time, not overlapping the target set Dt; (c) pseudo-OOD sets with T swept over {1, 1.5, 2, 3} at fixed γ. Compare the resulting estimates of ρs, ρt, and π to ground truth, and check whether the differences exceed the concentration bounds of Theorems 4.1 and 4.4 at δ=0.05. If the pseudo-reference results differ substantially from the true-OOD-reference results, or vary materially with T, then the pseudo-OOD relaxation is not validated by the paper's theory.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5 replaces the ground-truth OOD reference dataset used in Theorems 4.1 and 4.4 with pseudo-OOD samples Doγ = {(1-γ)xi + γε} generated from source ID samples plus Gaussian noise, then rescales the OOD score by a hand-set factor T in Eq. (16). Theorems 4.1 and 4.4 provide concentration bounds only when µ0 = E_{X_s|B_s=0}[h(X)] is estimated from samples drawn from p(x|y=K+1), the true source/target OOD conditional distribution. For Doγ, E_{Doγ}[h] is not generally equal to µ0, and the paper's claim that T rescaling makes the expectations match is not supported by any theorem, lemma, or appendix analysis. Because ρs enters the construction of ˜c in Eq. (7), and because the EM stage of Lemma 4.2 and the corrected ρt in Eqs. (13)-(14) all depend on µ0, a biased or incorrectly rescaled ˆµ0 propagates through every stage of the pipeline. The stated probability guarantees in Theorems 4.1 and 4.4 therefore do not apply to the actual algorithm evaluated in Section 5. The Remark in Section 4.5 acknowledges this relaxation, but no error bound or consistency result is given for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Open Set Label Shift (OSLS), where the target label distribution gains an extra out-of-distribution (OOD) class. The authors propose a three-stage estimator: (1) estimate the source ID data ratio ρ_s using an OOD classifier h and an OOD reference dataset; (2) reformulate the target likelihood as a K+1-class closed-set label shift problem and apply an EM algorithm to estimate target ID class proportions π and the target ID ratio ρ_t; and (3) correct ρ_t for imperfect OOD classifiers under a relaxed assumption. They provide concentration inequalities for the source-ratio estimator and the corrected target-ratio estimator, prove EM convergence for the MLE, and construct an adapted classifier in Eq. (17). Experiments on CIFAR-10/100 and ImageNet-200 with five OOD detectors show consistent improvements over closed-set label-shift baselines across many shift configurations. The paper releases code and reports detailed ablations, including sensitivity to the pseudo-OOD mixing coefficient γ.","tokens_in":61721,"tokens_out":11683,"duration_ms":111979,"significance":"The likelihood factorization in Lemma 4.2 is elegant: it reduces OSLS to a K+1-class closed-set label shift problem, making existing CSLS machinery applicable. The EM derivation is sound under Assumptions 3.2 and 3.3, and Theorems 4.1 and 4.4 provide genuine finite-sample concentration bounds for the setting where the OOD reference dataset contains true OOD samples. The paper also demonstrates good empirical performance across a wide range of datasets, shift types, and OOD detectors, and it releases publicly available code. However, the actual experimental pipeline replaces the ground-truth OOD reference with pseudo-OOD samples and an ad hoc rescaling factor T, which is outside the scope of the proved theorems; this gap materially weakens the theoretical support for the reported experimental results.","major_comments":[{"comment":"The pseudo-OOD reference dataset D_o^γ = {(1-γ)x_i + γ ε} and the rescaling factor T are not covered by The theoretical guarantees. Theorems 4.1 and 4.4 require the reference dataset to consist of i.i.d. samples from p(x|y=K+1), the true source/target OOD conditional distribution. For D_o^γ, E_{D_o^γ}[h] is not generally equal to µ0 = E_{X_s|B_s=0}[h(X)], and the statement in §4.5 that the T rescaling makes the expectations match is not supported by any theorem or lemma. Because ˆµ0 enters the source ratio estimator (2) and the target ratio correction (13)–(14), a biased ˆµ0 propagates through ρ_s, π, and ρ_t, so the concentration bounds in Theorems 4.1 and 4.4 do not apply to the algorithm actually evaluated in Section 5. The Remark in §4.1 acknowledges the relaxation, but no bias bound, consistency result, or even a formal condition on T is provided. This gap affects the central claim that sampling errors of the estimators are quantified. I recommend either adding a rigorous analysis of the pseudo-OOD bias (e.g., bounding |E_{D_o^γ}[h]/T - µ0| in terms of γ and properties of h) or repositioning the pseudo-OOD variant as a heuristic and validating the theoretical claims on experiments that use a true OOD reference dataset.","section":"§4.5, Eqs. (15)–(16); §5 experiments"},{"comment":"The claim that the MAP-OLS EM algorithm converges to the global MAP estimate πMAP is not established. Proposition C.2 proves that the posterior is strictly convex in π for fixed ρ_t and strictly convex in ρ_t for fixed π, but the objective is not shown to be jointly convex in (π, ρ_t). For a non-jointly-convex objective, standard EM theory only guarantees convergence to a stationary point or a local maximum, not the global maximum. Since the experimental results use the MAP variant (as stated in §5.2), the convergence claim in Theorem 4.3 and Proposition C.2 should be weakened to local convergence, or a joint convexity argument (or an additional condition) should be supplied.","section":"§4.3 and Appendix C.5, Proposition C.2"},{"comment":"The comparison omits Garg et al. 14, which is the only existing method specifically designed for Open Set Label Shift. The authors justify this by noting that Garg et al. requires retraining and does not report the estimation error metric (w - ŵ)^2/K. While this is a reasonable practical concern, the absence of any comparison against the existing OSLS method means that the claim of 'superior performance' is not demonstrated against the state of the art in OSLS, only against closed-set methods and a uniform baseline. At the very least, the paper should report results on a setting where a runnable implementation of Garg et al. can be compared, or clearly state that no OSLS baseline currently reports this metric and discuss the expected trade-off.","section":"§5.2, Table 4"}],"minor_comments":[{"comment":"Definition 3.1 lists the given inputs as D_s, D_t, f, and h, but does not include the OOD reference dataset D_o, which Section 4.1 introduces as required. This inconsistency should be resolved, either by adding D_o to the problem definition or by explicitly stating that D_o is an auxiliary input that may be replaced by pseudo-OOD data.","section":"Definition 3.1 and §4.1"},{"comment":"There are minor typos: 'Tource domain distribution' should be 'Target domain distribution', and 'subsset' should be 'subset'.","section":"Appendix A"},{"comment":"The word 'pipline' should be 'pipeline'.","section":"Appendix B.3"},{"comment":"The hyperparameters γ and T appear in Algorithm 2 without any selection procedure or sensitivity analysis. Appendix E.2 ablates γ but not T; please add a sensitivity study for T or provide a principled way to choose it.","section":"§4.5 and Appendix D.3"},{"comment":"Some entries for the Ash OOD detector on CIFAR-10 show very large standard deviations (e.g., Table 15, Ash Near r=1.0: 0.280±0.086 and later values above 1.0), which suggests instability. This should be mentioned in the discussion of the results.","section":"Appendix F, Tables 15–17"},{"comment":"The chain of equalities deriving ps(y=K+1|x) = 1-h(x) is hard to follow and contains an apparent typo in the denominator (ps(b=0) appears twice). A cleaner derivation would improve readability.","section":"Appendix C.3, Eq. (36)"}],"recommendation":"major_revision","confidential_remarks":"The core probabilistic derivation is sound when a true OOD reference dataset is available, and the experimental breadth is good. The main barrier is the gap between the theoretical guarantees and the pseudo-OOD implementation used in the experiments. If the authors can close that gap (or clearly separate theory and heuristic) and strengthen the comparison against the existing OSLS method, the paper could become a solid contribution. The MAP convergence claim also needs correction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core reduction here is real: the authors rewrite the open set label shift likelihood as a K+1 closed-set label shift problem by appending 1-h(x) as a synthetic class and absorbing the ID/OOD ratios into the class priors (Eqs. 5-7). That transformation is not in Garg et al. 14, and it is the right way to reuse the closed-set EM machinery. The EM updates are standard but correct under Assumptions 3.2 and 3.3, and Theorems 4.1 and 4.4 give honest Hoeffding-style bounds when the reference OOD dataset is drawn from the true OOD distribution. The paper deserves credit for shipping code, running a wide benchmark suite, and being upfront that the pseudo-OOD reference is a relaxation.\n\nThe soft spot is exactly where the stress-test note points: Section 4.5 replaces the true OOD reference with Do^gamma = {(1-gamma)x + gamma eps} and then rescales h by a hand-set factor T. All the concentration guarantees in Theorems 4.1 and 4.4 are for mu0 = E[h(X)|B=0]; nothing shows that the rescaled expectation over Do^gamma equals mu0. The remark acknowledges the relaxation but gives no bound or consistency argument. Since the experiments all use the pseudo-OOD reference, the theory in the paper does not cover the evaluated algorithm. That is a real gap, though not a fatal one--the gamma ablation shows the estimates are fairly stable in gamma, and the empirical wins over the closed-set baselines are consistent. T is fixed at 2 with no sensitivity study, which is a minor omission you could fix in revision.\n\nTwo smaller issues. Equation (14) has a typo: it subtracts mu0/(mu1-mu0) from rho_t instead of (rho_t - mu0)/(mu1-mu0). And the paper skips comparing with Garg et al. 14; the stated reason (they retrain per experiment) is understandable, but a small-scale comparison on CIFAR-10 would make the novelty claim more convincing.\n\nWho should read this: people working on label shift or OOD detection who want a no-retraining correction procedure. It is a useful method paper, not a deep theory paper. I would send it to a serious referee: the main reduction is sound, the code is available, and the pseudo-OOD gap is fixable with additional analysis or at least a T-sensitivity experiment.\n\nRecommendation: accept for peer review with requests to fix the typo, address the pseudo-OOD gap, and consider a comparison with Garg et al.\n\nBest.","headline":"Genuinely useful reduction of open-set label shift to K+1 closed-set label shift with EM and concentration bounds, but the pseudo-OOD reference used in all experiments sits outside the proved guarantees.","tokens_in":62264,"tokens_out":2449,"would_cite":true,"duration_ms":25057,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that open set label shift can be estimated and corrected using only a source ID classifier, an ID/OOD score, and a reference OOD set, without retraining the source network.","keywords":["open set label shift","label shift estimation","out-of-distribution detection","EM algorithm","maximum likelihood estimation","test-time adaptation","classifier correction","pseudo-OOD reference"],"falsifier":"Take a fixed source dataset and a held-out true OOD set, compute the mean OOD score on the true OOD set and on the pseudo-OOD set of Eq. (15) with the paper's chosen \\gamma and T, and check whether the difference exceeds the Hoeffding-scale bound. Alternatively, run the full estimator twice on the same unlabeled target set, once with a genuine OOD reference and once with the pseudo-OOD reference; if the estimated \\pi and \\rho_t differ by more than the stated concentration bounds, the pseudo-OOD relaxation is invalid.","tokens_in":61205,"feed_emoji":"🎯","tokens_out":5267,"duration_ms":54871,"temperature":0.7,"pith_summary":"The paper tries to show that when a target test set contains an extra out-of-distribution class, the target class proportions and the fraction of target data that is in-distribution can be recovered from unlabeled target data, a source classifier, an ID/OOD score, and a reference OOD set. The method rewrites the open-set likelihood as a closed-set label-shift likelihood with K+1 classes, so that standard EM machinery applies to the reparameterized target distribution. A three-stage procedure first estimates the source ID ratio, then runs an EM algorithm for the target ID class probabilities and ID ratio, then corrects the target ID ratio when the OOD score is imperfect. If the claims hold, the source classifier can be reweighted to the target domain without retraining or fine-tuning.","feed_headline":"Correct open-set label shift without retraining the classifier","feed_subtitle":"A three-stage EM scheme recovers target class proportions and the ID/OOD ratio, then reweights the source model.","key_machinery":"The carrying object is the augmented K+1-class classifier \\tilde f(x)_i = h(x)f(x)_i for each ID class and \\tilde f(x)_{K+1} = 1-h(x) for the OOD class, paired with the reparameterized source and target label vectors \\tilde c = [\\rho_s c_1,\\dots,\\rho_s c_K,1-\\rho_s] and \\tilde\\pi = [\\rho_t\\pi_1,\\dots,\\rho_t\\pi_K,1-\\rho_t]. Writing the target negative log likelihood with these objects turns the open-set likelihood into exactly the closed-set label shift likelihood, so the EM updates of the closed-set problem apply unchanged; the ID data ratio appears as the sum of the first K entries of \\tilde\\pi. A second mechanism, the ratio correction in Eq. (14), subtracts the offset \\hat\\mu_0/(\\hat\\mu_1-\\hat\\mu_0) to debias the estimated target ID ratio when the OOD score is not a true posterior but still separates ID from OOD on average and behaves uniformly across ID classes.","core_discovery":"The central claim is that open set label shift reduces to an augmented closed-set label shift problem. By combining a calibrated ID classifier f and an ID/OOD classifier h into the augmented classifier \\tilde f with entries h(x)f(x)_i for ID classes and 1-h(x) for the OOD class, the target negative log likelihood becomes identical in form to the K+1-class closed-set label shift likelihood. Maximum likelihood in the reparameterized variable \\tilde\\pi = [\\rho_t\\pi_1,\\dots,\\rho_t\\pi_K,1-\\rho_t] therefore yields both the target ID label distribution and the target ID data ratio, and the corrected classifier in Eq. (17) approximates p_t(y|x). Theorems 4.1 and 4.4 provide concentration bounds for the source ID ratio estimator and for a corrected target ID ratio estimator under a relaxed assumption on the OOD score.","pith_inferences":["The pseudo-OOD construction suggests the method may not need a physically realistic OOD reference at all: any set whose average score under h matches the true OOD average would work, making the actual requirement a calibration condition rather than a distributional match.","If the OOD score's expected response is not exactly equal across ID classes, the correction in Theorem 4.4 inherits a bias proportional to those class-average differences; measuring those differences on a source validation set would predict when the method degrades.","Because the objective is convex in the reparameterized target vector, adding Dirichlet and Beta priors gives a route to uncertainty estimates for the target proportions and ID ratio without new optimization machinery.","The estimator chain could be applied sequentially to arriving target batches, repeatedly correcting the ID ratio as new unlabeled data appear, which would amount to an online open-set label shift estimator."],"forward_implications":["Target label proportions and the ID/OOD mix can be estimated from unlabeled target data plus a reference OOD set, removing the need for target labels.","The corrected classifier requires no retraining or fine-tuning, so adapting a frozen or expensive source model to a shifting target becomes cheap.","Even an imperfect OOD score can be used, as long as it responds differently to ID versus OOD data and uniformly across ID classes, after applying the linear correction of Theorem 4.4.","The Hoeffding-based concentration bounds make sample-size requirements explicit: estimation error decays at the usual rate \\sqrt{\\log(1/\\delta)/n} in the smallest of the involved dataset sizes.","Because the open-set objective is the closed-set objective in disguise, existing closed-set label-shift estimation and correction software can be reused for the open-set case."],"supporting_citations":[{"why":"Provides the EM algorithm for closed-set label shift whose updates the open-set EM generalizes.","marker":"[47]"},{"why":"Establishes convexity and MLE convergence of the closed-set EM objective, used to justify convergence of the reparameterized open-set EM.","marker":"[1]"},{"why":"Provides consistency guarantees for the EM estimator under classifier calibration, which supports Assumption 3.3A.","marker":"[12]"},{"why":"Supplies the linear-system view and estimation error bounds that motivate identifying the ID ratio from score expectations.","marker":"[32]"},{"why":"Supplies the Hoeffding concentration inequality behind the sampling-error bounds in Theorems 4.1 and 4.4.","marker":"[55]"},{"why":"Provides the MAP/prior EM framework and the correction-style classifier construction that Eq. (17) extends.","marker":"[62]"},{"why":"Defines the open set label shift problem and the prior domain-adaptation approach that this work avoids by not retraining the classifier.","marker":"[14]"},{"why":"Justifies the invariance of maximum likelihood under reparameterization, used to map the estimated \\tilde\\pi back to \\pi and \\rho_t.","marker":"[41]"}],"fun_headline_variants":["Fix label shift with an OOD reference, no retraining","Three-stage EM corrects target labels and OOD ratio","Open-set label shift solved by augmenting closed-set","Estimate target class mix and OOD proportion via EM","Reweight source model to target without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole estimator chain stands on the assumption that the pseudo-OOD reference set, made by adding Gaussian noise to source images and rescaling the OOD score by T, has the same expected score under the OOD classifier as the true out-of-distribution distribution; if that single expectation is off, both the source ID ratio and the corrected target ID ratio become biased and the theorem bounds no longer apply.","fun_headline_variants_meta":{"raw":{"variants":["Fix label shift with an OOD reference, no retraining","Three-stage EM corrects target labels and OOD ratio","Open-set label shift solved by augmenting closed-set","Estimate target class mix and OOD proportion via EM","Reweight source model to target without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1497,"prompt_tokens":931,"completion_tokens":566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":547,"tokens_out":566,"duration_ms":5632,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:54:31.572119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed source dataset and a held-out true OOD set, compute the mean OOD score on the true OOD set and on the pseudo-OOD set of Eq. (15) with the paper's chosen \\gamma and T, and check whether the difference exceeds the Hoeffding-scale bound. Alternatively, run the full estimator twice on the same unlabeled target set, once with a genuine OOD reference and once with the pseudo-OOD reference; if the estimated \\pi and \\rho_t differ by more than the stated concentration bounds, the pseudo-OOD relaxation is invalid.","supporting_citations":[{"cited_title":"Adjusting the outputs of a classifier to new a priori probabil- ities: a simple procedure","cited_arxiv_id":null,"evidence_quote":"Provides the EM algorithm for closed-set label shift whose updates the open-set EM generalizes."},{"cited_title":"Maximum likelihood with bias-corrected calibration is hard- to-beat at label shift adaptation","cited_arxiv_id":null,"evidence_quote":"Establishes convexity and MLE convergence of the closed-set EM objective, used to justify convergence of the reparameterized open-set EM."},{"cited_title":"A unified view of label shift estimation.Ad- vances in Neural Information Processing Systems, 33:3290– 3300, 2020","cited_arxiv_id":null,"evidence_quote":"Provides consistency guarantees for the EM estimator under classifier calibration, which supports Assumption 3.3A."},{"cited_title":"De- tecting and correcting for label shift with black box predic- tors","cited_arxiv_id":null,"evidence_quote":"Supplies the linear-system view and estimation error bounds that motivate identifying the ID ratio from score expectations."},{"cited_title":"High-dimensional probability: An intro- duction with applications in data science","cited_arxiv_id":null,"evidence_quote":"Supplies the Hoeffding concentration inequality behind the sampling-error bounds in Theorems 4.1 and 4.4."},{"cited_title":"Label shift estimation for class-imbalance problem: A bayesian approach","cited_arxiv_id":null,"evidence_quote":"Provides the MAP/prior EM framework and the correction-style classifier construction that Eq. (17) extends."},{"cited_title":"Domain adaptation under open set label shift","cited_arxiv_id":null,"evidence_quote":"Defines the open set label shift problem and the prior domain-adaptation approach that this work avoids by not retraining the classifier."},{"cited_title":"Machine learning: a probabilistic perspec- tive","cited_arxiv_id":null,"evidence_quote":"Justifies the invariance of maximum likelihood under reparameterization, used to map the estimated \\tilde\\pi back to \\pi and \\rho_t."}],"review_version":1}