{"id":"b4de8de7-bc22-4144-b6c2-77972fccfcd1","arxiv_id":"2505.05180","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OpenworldAUC jointly measures base/new detection and domain-specific classification via pairwise ranking, and GMoP optimizes it with gated domain-specific prompts.","lead":"This paper introduces OpenworldAUC, a unified scoring metric for open-world prompt tuning that rewards both correct detection of known versus unknown images and correct classification of each image. It also presents a prompt-tuning method, GMoP, that optimizes this score and reports gains over ten existing methods across 15 benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5.1's equivalence claim is false: the arithmetic sum objective differs from 1−OpenworldAUC by E[(1−I_g)(1−I_h)], so GMoP is not provably maximizing OpenworldAUC.","rationale":"The reader's conditional verdict focuses on Thm. 5.2's unestimated pseudo-to-real partition shift. That concern is valid, but the more load-bearing weakness is Prop. 5.1's false equivalence, because it breaks the formal claim that GMoP maximizes OpenworldAUC. If the optimization objective is only a heuristic surrogate, the theory in Thm. 5.2 bounds a risk that is not the metric, and the 'unified evaluation and optimization' story loses its theoretical closure. The metric itself and the empirical benchmarking (15 datasets, released code, multiple ablations) remain credible, so the paper should be conditionally accepted with a required correction or qualification of Prop. 5.1 and a careful restatement of Thm. 5.2. I disagree with the reader's specific weakest-assumption choice but share the overall CONDITIONAL assessment.","tokens_in":47938,"tokens_out":17307,"duration_ms":174420,"concrete_test":"Algebraically check Prop. 5.1: compute S = E[1−I_g + I_g(1−I_r)I_h + 1−I_h] and C = 1−E[I_g I_r I_h]; S−C = E[(1−I_g)(1−I_h)]. Then instantiate a counterexample with two expected-indicator configurations, e.g., (a) P(I_g=1)=P(I_h=1)=0.5 with perfect correlation and I_r=1, giving OpenworldAUC=0.5, S=1.0; (b) P(I_g=1)=P(I_h=1)=0.8 independent with I_r=0.5, giving OpenworldAUC=0.32, S=0.72. The arithmetic objective ranks (b) better while OpenworldAUC ranks (a) better, disproving equivalence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Prop. 5.1 states that maximizing OpenworldAUC is equivalent to minimizing E[1[y_b≠g(x_b)] + 1[y_n≠h(x_n)] + 1[y_b=g(x_b)]·1[r(x_b)≤r(x_n)]·1[y_n=h(x_n)]]. With arithmetic '+', this is false. For a pair where both classifiers err, the sum is 2 whereas 1−I_g I_r I_h is 1. Algebraically, with I_g=1[y_b=g], I_h=1[y_n=h], I_r=1[r(x_b)>r(x_n)], the objective equals 2−I_g−I_h+I_g I_h−I_g I_r I_h = (1−I_g I_r I_h)+(1−I_g)(1−I_h). Thus minimizing it is equivalent to maximizing OpenworldAUC only when E[(1−I_g)(1−I_h)] is constant; in general it is not. The proof's Tab.5 fills every row with 0/1 values, silently treating '+' as logical OR rather than indicator arithmetic. Since OP0 and OPfin, the losses actually optimized, are arithmetic sums (CE plus squared-ranking), the claimed equivalence is the only formal link between GMoP and OpenworldAUC. This is a correctness risk in the central optimization claim, independent of the unestimated partition-shift term in Thm. 5.2 (which the reader correctly flagged). The empirical results and released code still support the method as a heuristic, so this does not warrant rejection, but the equivalence should be corrected or explicitly downgraded to a surrogate claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OpenworldAUC, a metric for open-world prompt tuning that jointly evaluates base-to-new detection and classification in both domains through a pairwise ranking formulation, and argues it is insensitive to the base/new sample ratio. To optimize the metric, it introduces GMoP, a gated mixture-of-prompts approach with a base classifier prompt, multiple detector prompts trained on pseudo base/new partitions, and a fixed zero-shot new-domain classifier. A generalization bound is stated for the empirical risk. Experiments on recognition, domain generalization, and cross-dataset benchmarks compare against ten methods and report improvements when evaluating with OpenworldAUC, with code released.","tokens_in":48278,"tokens_out":6870,"duration_ms":67913,"significance":"If the optimization-theory issues were repaired, the work would be a solid contribution: the metric definition (Prop. 4.2) is sound and addresses a real evaluation gap in open-world prompt tuning; the empirical study is broad (many datasets and competitors, released code); and the pseudo-partition idea for detector training is practically promising. However, the formal link between the objective actually optimized by GMoP and the OpenworldAUC metric is currently broken by an incorrect equivalence claim in Prop. 5.1, and the proof of the generalization bound contains an unsupported step that treats discrete partition distributions as continuous simplex densities. These issues are load-bearing for the central optimization and theoretical claims; once repaired, the metric and empirical method would constitute a valuable contribution. The released code supports reproducibility of the empirical results.","major_comments":[{"comment":"The equivalence in Prop. 5.1 is incorrect. With indicators I_g=1[y_b=g(x_b)], I_r=1[r(x_b)>r(x_n)], I_h=1[y_n=h(x_n)], the arithmetic objective in Prop. 5.1 has expectation E[(1-I_g)+(1-I_h)+I_g(1-I_r)I_h] = 1 - OpenworldAUC + E[(1-I_g)(1-I_h)]. The extra term E[(1-I_g)(1-I_h)] does not vanish in general, so minimizing the stated objective is not equivalent to maximizing OpenworldAUC. The truth table in Tab. 5 is valid only if '+' is interpreted as logical OR, not arithmetic addition; the empirical risks in (OP0) and (OPfin) are arithmetic sums. Since this equivalence is the only formal link between GMoP and OpenworldAUC, the paper should either prove a surrogate-gap bound (e.g., control E[(1-I_g)(1-I_h)] under training conditions) or explicitly present the objective as a heuristic surrogate for OpenworldAUC rather than an equivalent reformulation.","section":"Section 5.1, Prop. 5.1, App. A.5 and Tab. 5"},{"comment":"Term (a) of the proof of Thm. 5.2 bounds |E_Y[R_D] - E_hatY[R_hatD]| by an integral over the simplex S^{C-1} of |E(P)-E'(P)| dP, and then uses Lemma B.9 (simplex volume 1/C!) to obtain the factor ||E(P)-E'(P)||_infinity/C!. However, E and E' are, by definition, distributions over discrete class partitions, not probability densities on the continuous simplex; the integral representation and the resulting C! factor are not justified. Moreover, the term ||E(P)-E'(P)||_infinity/(C_b+C_n)! depends on the unknown true partition distribution and is never estimated or controlled by the algorithm, so the stated guarantee is conditional on a quantity the method does not handle. Please repair the derivation or restate the theorem explicitly as a conditional bound dependent on an unestimated partition-shift term.","section":"Section 5.2, Thm. 5.2, App. B.4, term (a)"},{"comment":"The claimed lower bound on HitRaten is not proportional to OpenworldAUC u. From the derived inequality, HitRaten >= 1 - (1-u)/a where a = MissRateb; this is an affine function of u, and for a < 1-u the bound is negative and vacuous. Consequently, the subsequent claim that optimizing OpenworldAUC 'can guarantee the model performance on the new domain under arbitrary base/new ratio' (Sec. 4, discussion after Prop. 4.3) is not supported as stated. Please qualify the statement with the condition u > 1 - a and describe the bound as affine rather than proportional, or remove the overclaim.","section":"Section 4.2, Prop. 4.3, App. A.4"}],"minor_comments":[{"comment":"The sentence 'based on the open-source library open-source library XCurve' contains a duplicated phrase; please remove one instance.","section":"Section 6.1"},{"comment":"The caption labels the third and fourth panels both with 'c)'; the fourth panel should be labeled 'd)'.","section":"Figure 1 caption"},{"comment":"The word 'dasdsa' appears before the table of contents in the appendix; this appears to be an accidental insertion and should be deleted.","section":"Appendix table of contents"},{"comment":"The notation for the partition-shift term is inconsistent: the theorem statement in the main text uses ||E-E'||_infinity/(C_b+C_n)!, while the proof in App. B.4 uses ||E(P)-E'(P)||_infinity/C!. Please unify the notation.","section":"Section 5.2 / App. B.4"},{"comment":"The abstract states '15 benchmarks', while the experimental section reports 11 recognition datasets, 4 ImageNet variants, 7 cross-dataset targets, and 10 imbalance variants; please clarify the exact counting of benchmarks to avoid apparent inconsistency.","section":"Abstract / Section 6.1"},{"comment":"The phrase 'joint possibility' should be 'joint probability' in the statement of Prop. 4.2.","section":"Section 4.2, Prop. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The metric contribution is valuable and the empirical study is extensive, but the theoretical section needs careful repair before publication. The authors' prior work on AUC optimization (OpenAUC, XCurve) is used heavily in the surrogate losses; this is not circular, but a clearer statement of what is inherited versus novel would be helpful for reviewers. The appendix contains several typos (including 'dasdsa' and a duplicated phrase) that suggest the final version was assembled quickly; these are minor but should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is OpenworldAUC: a single pairwise metric that requires the detector to rank a base sample above a new sample while both domain classifiers make correct predictions. It fills a real gap in open-world prompt tuning, where HM ignores detection, OverallAcc is sensitive to the base/new ratio, and AUROC ignores classification. The experiments are broad (15 benchmarks, imbalance resampling, domain and cross-dataset shifts), the gains over ten competitors are consistent if modest, and the code is public. Prop. 4.2, which rewrites the metric as a joint probability, checks out.\n\nThe soft spots are all in the theory, and they are real. The stress-test note is right: Proposition 5.1's equivalence is false under arithmetic '+'. The objective differs from 1−OpenworldAUC by E[(1−I_g)(1−I_h)], and the proof's truth table handles the '+' as logical OR rather than indicator arithmetic. So the formal link between GMoP and the metric is not established; the method should be presented as optimizing a surrogate, or the proposition corrected. This is load-bearing for the optimization claim but not fatal to the empirical results.\n\nTheorem 5.2 also has a separate issue. The term for partition-distribution shift treats discrete distributions over partitions as densities on a simplex and bounds the difference by a sup norm times 1/C!, which comes from a volume integral that does not obviously apply to discrete allocations. The bound also leaves the pseudo-to-real partition shift and the zero-shot new-class error unestimated, so it is weaker than the abstract's 'theoretical guarantees' suggests. Both of these are fixable or qualifiable.\n\nThe related work is honestly positioned against OpenAUC and DeCoOp, but a direct empirical comparison with the authors' earlier OpenAUC is missing. That is a small omission, and it would sharpen the claim of incremental value.\n\nOverall, this is a useful paper for people in VLM prompt tuning, open-set/OOD evaluation, and AUC optimization. I would not desk reject. I would send it to a careful referee and tell the authors to correct or downgrade Prop. 5.1, repair the partition term in Thm. 5.2, and add the OpenAUC comparison. I would bring it to our next reading group.","headline":"Solid metric and experiments for open-world prompt tuning, but the key equivalence in Prop. 5.1 is wrong and Thm. 5.2 needs repair.","tokens_in":48823,"tokens_out":6293,"would_cite":true,"duration_ms":56913,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenworldAUC unifies base-to-new detection and classification into one distribution-insensitive metric for open-world prompt tuning.","keywords":["OpenworldAUC","open-world prompt tuning","base-to-new detection","gated mixture-of-prompts","pairwise ranking metric","generalization bound","CLIP prompt tuning","distribution-insensitive evaluation"],"falsifier":"Run any competitor on an open-world split where the true new classes lie in a different distributional family from a random half of the training classes (e.g., base classes are photos and new classes are sketches or satellite images) and compare OpenworldAUC: if GMoP's OpenworldAUC collapses relative to a baseline whose detector was trained on genuine OOD signals, the pseudo-partition proxy is the bottleneck.","tokens_in":47690,"feed_emoji":"🎯","tokens_out":5094,"duration_ms":37657,"temperature":0.7,"pith_summary":"This paper argues that open-world prompt tuning needs one evaluation metric that rewards a model only when it both detects which domain an image belongs to and classifies the image correctly inside that domain. It shows existing metrics fall short individually and proposes OpenworldAUC, defined as the joint probability over a base/new pair that the detector ranks the base sample above the new one while both domain classifiers predict correctly. To optimize it, the paper introduces Gated Mixture-of-Prompts, a divide-and-conquer scheme with separate prompts for detection and base classification plus a fixed zero-shot new-domain classifier, and provides a generalization bound. If the proposal holds, model selection for open-world tasks becomes a single-number comparison instead of juggling HM, OverallAcc, and AUROC against each other.","feed_headline":"One metric unifies open-world detection and classification for prompt tuning","feed_subtitle":"OpenworldAUC is a single number that stays stable under any base/new ratio and is optimized by gated prompts.","key_machinery":"The central object is the OpenworldAUC pairwise metric together with its equivalent surrogate objective. The metric counts a pair $(x_b,x_n)$ only if the base sample is correctly classified, the new sample is correctly classified, and the detector ranks $x_b$ above $x_n$. Its replacement loss is $\\mathbf{1}[y_b\\neq g(x_b)]+\\mathbf{1}[y_n\\neq h(x_n)]+\\mathbf{1}[y_b=g(x_b)]\\mathbf{1}[r(x_b)\\le r(x_n)]\\mathbf{1}[y_n=h(x_n)]$, approximated by a squared ranking loss weighted by sigmoid gates derived from classifier confidence. The pseudo base-to-new partition, averaged over $K$ random splits of the known classes, simulates the unseen domain during training and produces the detector ensembles whose maximum score is used as $r$.","core_discovery":"OpenworldAUC, defined as $\\mathbb{E}_{(x_b,y_b)\\sim D_b,(x_n,y_n)\\sim D_n}[\\mathbf{1}[y_b=g(x_b)]\\cdot \\mathbf{1}[r(x_b)>r(x_n)]\\cdot \\mathbf{1}[y_n=h(x_n)]]$, is a unified metric for open-world prompt tuning that jointly evaluates base-to-new detection (P1), base classification (P2), and new classification (P2), while remaining insensitive to the base/new sample ratio because it uses pairwise ranking rather than class-marginal accuracy. The paper establishes that HM ignores detection, OverallAcc is dominated by the more numerous domain, and AUROC ignores classification, so none alone evaluates an open-world model faithfully. GMoP optimizes this metric by assigning a dedicated prompt to each of the detector, base classifier, and (zero-shot) new classifier, and the gating mechanism weights the pairwise ranking loss toward correctly classified samples. Theorem 5.2 bounds the generalization gap through the empirical loss, the zero-shot error of the new classifier, and three stochastic error terms involving the number of pseudo partitions, the base sample size, and the discrepancy between the pseudo and true base/new partitions. Across fifteen benchmarks, the method reports higher OpenworldAUC and better stage-wise trade-offs than ten competitors while using fewer trainable parameters than some of them.","pith_inferences":["A direct corollary: any existing prompt-tuning method could be converted into an OpenworldAUC learner solely by reweighting its training objective with the gated pairwise term, without changing the prompt architecture, so the metric itself, not GMoP, is the primary contribution.","The metric could be imported into other staged recognition pipelines, such as open-set recognition or test-time-agnostic long-tail learning, wherever a detector fronts a classifier and ground truth for a target split is unavailable.","A testable extension: swap the fixed zero-shot new classifier for a lightweight adversarially trained new-domain classifier and measure whether the irreducible zero-shot error term in Theorem 5.2 shrinks enough to improve OpenworldAUC, especially on fine-grained new classes.","If pseudo partitions are class-level while the true new domain is distribution-level, the bound's partition-shift term predicts a measurable degradation; comparing OpenworldAUC under class-split versus dataset-split new domains would quantify that gap and tell practitioners where the metric can be trusted."],"forward_implications":["Open-world model selection becomes a single-number decision guided by OpenworldAUC, replacing separate stage-wise metrics where a model strong in one stage can appear better than one that balances both.","Prompt tuning methods can be re-ranked by a metric that is stable under changed base/new test ratios, so deployment decisions no longer depend on assumptions about the unobserved domain distribution.","The gated division of labor among prompts provides a recipe for optimizing ranking-based objectives when the full label space is unavailable: pseudo partitions plus confidence gates.","The reported results suggest that a modest set of prompt parameters, around 26.6k, suffices to improve over methods with larger prompt budgets, which matters for low-resource adaptation of vision-language models.","The generalization bound implies that more pseudo partitions and more base shots monotonically reduce the excess risk, a prediction testable on any downstream benchmark."],"supporting_citations":[{"why":"introduces the open-world prompt tuning task and the pseudo base-to-new partition of the training classes that OpenworldAUC and GMoP build upon","marker":"(Zhou et al., 2024)"},{"why":"provides the frozen vision-language backbone whose text prompts are tuned","marker":"(Radford et al., 2021)"},{"why":"defines the AUC/ROC machinery that the pairwise ranking formulation and its computation adapt","marker":"(Fawcett, 2006)"},{"why":"supplies the OOD-detection baseline using maximum softmax probability, which the final detector score resembles","marker":"(Hendrycks & Gimpel, 2017)"},{"why":"establishes the base-to-new prompt tuning setting and the CoOp baseline that GMoP extends","marker":"(Zhou et al., 2022b)"},{"why":"provides the multiclass AUC optimization theory and surrogate loss framework used for the ranking objective","marker":"(Yang et al., 2022)"},{"why":"motivates the cross-entropy regularization at $\\lambda\\in[1/2,1]$ used to stabilize the AUC-style objective","marker":"(Yang et al., 2023)"}],"fun_headline_variants":["OpenworldAUC: one metric to detect and classify in open worlds","Only one metric needed: OpenworldAUC for open-world tuning","Single metric for joint detection and classification in prompt tuning","OpenworldAUC: ratio-agnostic unified metric for open-world tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument assumes the pseudo base/new split of the training classes behaves like the true base/new split of the data; if that proxy is badly unrepresentative of the real new domain, the guarantee includes an unmeasured shift term and the method's new-class promise rests on the zero-shot classifier alone.","fun_headline_variants_meta":{"raw":{"variants":["OpenworldAUC: one metric to detect and classify in open worlds","Only one metric needed: OpenworldAUC for open-world tuning","Single metric for joint detection and classification in prompt tuning","OpenworldAUC: ratio-agnostic unified metric for open-world tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3974,"prompt_tokens":1101,"completion_tokens":2873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":717,"completion_tokens_details":{"reasoning_tokens":2798}},"tokens_in":717,"tokens_out":2873,"duration_ms":21129,"temperature":1.0,"reasoning_tokens":2798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:10:45.663393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run any competitor on an open-world split where the true new classes lie in a different distributional family from a random half of the training classes (e.g., base classes are photos and new classes are sketches or satellite images) and compare OpenworldAUC: if GMoP's OpenworldAUC collapses relative to a baseline whose detector was trained on genuine OOD signals, the pseudo-partition proxy is the bottleneck.","supporting_citations":[{"cited_title":"Decoop: Robust prompt tuning with out-of-distribution detection","cited_arxiv_id":null,"evidence_quote":"introduces the open-world prompt tuning task and the pseudo base-to-new partition of the training classes that OpenworldAUC and GMoP build upon"},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"provides the frozen vision-language backbone whose text prompts are tuned"},{"cited_title":"An introduction to ROC analysis","cited_arxiv_id":null,"evidence_quote":"defines the AUC/ROC machinery that the pairwise ranking formulation and its computation adapt"},{"cited_title":"and Gimpel, K","cited_arxiv_id":null,"evidence_quote":"supplies the OOD-detection baseline using maximum softmax probability, which the final detector score resembles"},{"cited_title":"Learning with multiclass AUC: theory and algorithms, 2022","cited_arxiv_id":null,"evidence_quote":"provides the multiclass AUC optimization theory and surrogate loss framework used for the ranking objective"}],"review_version":1}