{"id":"70ec5032-a13c-44b9-b15a-e89e98abab84","arxiv_id":"2608.12035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 11 medical imaging scenarios, a good adapted model usually exists, but label-free validators miss it by 6.1 points on average, and no tested validator is consistently reliable.","lead":"Unsupervised domain adaptation (UDA) promises to make medical AI work on new hospitals and scanners without new labels. This large benchmark tests the full deployment pipeline, adaptation plus model selection, and finds the real bottleneck is choosing the right model without target labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-best checkpoints are selected and evaluated on the same small target validation set, so the 6.1-point average gap is inflated by winner's curse; the 'large and structural' claim should be re-tested with a nested or bias-corrected oracle.","rationale":"The reader's weakest assumption concerned the representativeness of the 13 validators and the range restriction from top-10 pooling in Section 4.3. Those are legitimate concerns about the 'structural' interpretation. I identify a different, more fundamental issue that affects the headline gap itself: the Oracle is selected and evaluated on the same finite target validation set, so its reported accuracy is biased upward by selection over many noisy estimates. This inflates the 6.1-point average gap and the 10.5-point maximum gap that anchor the paper's central claim. Validator-selected models are not label-selected, so the comparison is asymmetric. The within- and across-algorithm Spearman analyses in Sections 4.3 provide support for validator unreliability, but the magnitude and 'structural' wording of the gap should not rest on an in-sample Oracle maximum. The proposed nested-split check would directly quantify the bias. If the unbiased gap remains around 5-6 points, the concern is minor; if it drops substantially, the abstract and conclusion overstate the distance to deployment. Because the paper's overall empirical contribution remains valuable and the issue is addressable with a re-analysis, the verdict should stay conditional rather than being rejected outright.","tokens_in":49663,"tokens_out":6509,"duration_ms":67440,"concrete_test":"On the ADNI-1 to ADNI-2 scenario and the largest CXR scenarios, split the target validation set into two halves (A and B). Use half A to select the Oracle-best checkpoint from the full pool, evaluate it on half B, then swap and average; repeat over multiple partitions to reduce variance. Recompute the mean and maximum selection gap across all 11 scenarios using this unbiased Oracle estimate. If the average gap falls substantially below 6.1 points (e.g., below 3 points) or the maximum gap falls below 10.5 points, the headline gap is partly an artifact of evaluating the Oracle on its own selection set and the 'structural' claim needs to be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, a 6.1-point average selection gap between validator-selected and Oracle-selected models (Section 4.2, Figure 3), is computed by selecting the Oracle checkpoint as the argmax target validation accuracy over the full checkpoint pool and then reporting that same validation accuracy as the Oracle's performance. Since target validation sets are small (around 100 samples for brain MRI, per Section 4.4) and the pool contains roughly 1,500 checkpoints per scenario, the Oracle accuracy is the maximum of many noisy estimates and is positively biased by winner's curse. Validator-selected models are not chosen using target labels, so their reported accuracies are unbiased estimates; the gap therefore mixes genuine validator deficiency with the cost of estimating the best model from a finite sample. This inflation directly affects the headline gap magnitude and the strength of the 'structural' conclusion. A label-based selector deployed with the same finite validation set would also suffer this bias, so the Oracle is not a clean upper bound on what a good selector could achieve. The paper's within-algorithm and across-algorithm Spearman analyses may still show validators are unreliable, but the 6.1-point 'large and structural' gap as stated is not a sound estimate of the shortfall attributable to label-free selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the complete unsupervised domain adaptation (UDA) pipeline for medical imaging classification, treating both adaptation and label-free model selection as parts of one deployment problem. Across eleven cross-domain scenarios (brain MRI, chest X-ray, retina), ten UDA algorithms, thirteen label-free validators, and over 80,000 trained checkpoints, it reports that a capable adapted model usually exists under oracle selection, but the models selected by label-free validators fall substantially short of the best available model, with an average accuracy gap of 6.1 points and a maximum of 10.5 points. The authors attribute this gap to a structural limitation of current validators, supported by Spearman correlations showing no validator is consistently reliable within or across algorithms. They further show that ensembling validator-selected checkpoints or spending a small target-labeling budget narrows but does not close the gap.","tokens_in":49864,"tokens_out":3644,"duration_ms":36266,"significance":"The paper addresses a practically important and previously understudied problem: model selection in the UDA deployment pipeline. It provides a large-scale, multi-dataset benchmark that includes medical-specific UDA algorithms and a broad set of validators, and it is one of the first studies to evaluate selection across algorithms rather than only within a single algorithm. The paper ships open-source code and reports both mean±std and median with 95% confidence intervals, which strengthens the reliability of its descriptive statistics. If the central claim survives the methodological concerns below, the paper would be a valuable reference for the medical imaging and UDA communities, highlighting that adaptation accuracy does not translate into deployable performance unless reliable label-free selection is solved.","major_comments":[{"comment":"The headline selection gap of 6.1 points is computed by selecting the Oracle checkpoint as the argmax over roughly 1,500 checkpoints on the same small target validation set (around 100 samples for brain MRI, per §4.4) and then reporting that same validation accuracy as the Oracle's performance. Because validator-selected models are not chosen using target labels, their reported accuracies are unbiased estimates, while the Oracle accuracy is the maximum of many noisy estimates and is therefore positively biased by winner's curse. The gap thus mixes genuine validator deficiency with the cost of estimating the best model from a finite sample, so the 'large and structural' claim in §4.2 and the abstract is not a sound estimate of the shortfall attributable to label-free selection. I recommend re-estimating the gap with a nested oracle (e.g., select on one half of the target validation set and evaluate on the other half) or a bias-corrected oracle, and reporting both the raw and corrected gaps.","section":"§4.2, Figure 3"},{"comment":"The across-algorithm Spearman correlations are computed after restricting each algorithm's checkpoints to the validator's top-10 scores before pooling. This range restriction attenuates correlation coefficients, so the observed collapse to near zero may be a procedural artifact rather than evidence that validators are structurally unreliable for across-algorithm selection. The paper's claim that 'the gap therefore appears largely structural' relies on these correlations. I request a re-analysis using the full checkpoint pool (or a random subsample), or a formal justification for why top-10 pooling is the correct protocol for the deployment setting. Without this, the 'structural' conclusion is not established for the across-algorithm case.","section":"§4.3, Figure 6(b) and Figure 7"},{"comment":"The conclusion that no evaluated validator is consistently reliable is based on only 13 validators, all from prior general-vision or shallow-adaptation literature. The paper states this gap is 'largely structural,' which is a strong claim about the impossibility of reliable label-free selection given the current validator paradigm, but a sample of 13 validators is limited evidence for such structurality. I recommend softening the language to 'no validator among those evaluated' or adding an explicit representative-coverage argument for the validator set.","section":"§4.3, Figures 5 and 7"}],"minor_comments":[{"comment":"The p-value is printed as 'p = 0.0.678' in the figure caption; this should be 'p = 0.678'.","section":"§4.3, Figure 6(b)"},{"comment":"The supplementary tables are not referenced individually in the main text; adding a sentence in §6.1 pointing to the corresponding main-text figures or claims would improve navigability.","section":"Tables 5–37"},{"comment":"The heatmap colorbar in Figure 5 ranges only from 0 to 1.0, but several cells contain negative correlations (e.g., -0.42). The colorbar should include the full range of the displayed values, or the text should clarify how negative values are mapped.","section":"Figure 5"},{"comment":"For the small labeling budget, the paper does not specify explicitly whether the checkpoint selected on the labeled subset is evaluated on the full target validation set or on the same labeled subset. If the latter, the reported gains are inflated; please clarify.","section":"§4.4, Table 4"},{"comment":"The adaptation strength is varied over λ∈{0.1,0.5,1.0}, but there is no discussion of whether this hyperparameter range is sufficient for the algorithms to show their typical behavior; a sentence justifying the range would be helpful.","section":"§3, Implementation Details"}],"recommendation":"major_revision","confidential_remarks":"The winner's-curse concern is real and affects the paper's central quantitative claim. However, it is fixable by re-running the oracle evaluation with a nested split or a bias-correction procedure, and the qualitative message about validator unreliability is likely to survive in attenuated form. The across-algorithm Spearman protocol also needs to be revisited. I therefore recommend major revision rather than rejection, with the request that the authors provide the corrected numbers and re-state the strength of the 'structural' claim accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know: this is a genuinely useful benchmark for UDA model selection in medical imaging, the first large one that evaluates adaptation and label-free selection together over eleven clinical scenarios. The finding that a capable adapted model usually exists while no evaluated validator reliably finds it is important, and the across-algorithm pooling analysis is new. But the headline gap—6.1 points average between validator-selected and 'Oracle' models—is inflated by winner's curse and should be re-estimated before you trust the 'structural' language.\n\nThe paper deserves credit. Ten algorithms, 13 validators, 80k checkpoints, nine datasets, balanced accuracy, mean±std plus medians with CIs, and a stated code release. The authors are careful to note that Best Validator and Best Pair are identified with target labels, so the realized deployment gap is only larger. The backbone robustness check is a nice touch.\n\nThe soft spots are real but not fatal. The Oracle selects the argmax target validation accuracy over roughly 1,500 checkpoints and then reports that same validation accuracy as its performance. With target validation sets around 100 samples, that maximum is a noisy maximum and positively biased. A label-based selector on the same finite set would suffer the same bias, so Oracle is not a clean upper bound for what a good selector could achieve. The gap magnitude is therefore overestimated. The 'structural' conclusion would be more convincing with a nested oracle that selects on one split and evaluates on another. Also, the across-algorithm correlation analysis uses each algorithm's top-10 checkpoints by validator score, which restricts range and can suppress correlations—not a fatal flaw, but it cuts against the strength of the claim. The ensembling strategy gains little over the best validator on average, and the abstract oversells it a bit.\n\nBottom line: the empirical scope is real, the selection problem is real, and this benchmark will be useful to anyone working on UDA deployment or medical imaging transfer. It deserves a serious referee. I'd ask for a bias-corrected oracle or an explicit caveat before publication, but I would not desk reject it.\n\nBest,","headline":"A solid, large medical-imaging benchmark for UDA model selection, but the 6.1-point 'structural' gap is inflated by winner's curse and needs a bias-corrected oracle.","tokens_in":50435,"tokens_out":2449,"would_cite":true,"duration_ms":23628,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the bottleneck for clinically deployable unsupervised domain adaptation is not adaptation itself but label-free model selection: a capable checkpoint usually exists, yet 13 evaluated validators leave a large…","keywords":["unsupervised domain adaptation","label-free model selection","medical imaging","domain shift","clinical deployment","checkpoint selection","benchmark evaluation"],"falsifier":"On the same checkpoint pools, compute across-algorithm Spearman correlations pooling all checkpoints rather than the top-10 per algorithm, or test a new label-free score on all 11 scenarios: if any validator shows consistently positive correlation (say $\\rho>0.5$) across most scenarios, or selects a checkpoint within 2 percentage points of the oracle across most scenarios, the structural-gap claim is refuted.","tokens_in":49423,"feed_emoji":"🩺","tokens_out":10367,"duration_ms":88269,"temperature":0.7,"pith_summary":"Unsupervised domain adaptation (UDA) trains models on a labeled source domain and an unlabeled target domain, but a clinical deployment also has to choose which of the many trained checkpoints to ship, without target labels to test them on. This paper evaluates that complete pipeline together across eleven clinically relevant medical imaging transfer scenarios, ten UDA algorithms, thirteen label-free selection methods, and over 80,000 checkpoints. The central finding is that a capable adapted model usually exists in the pool, but every evaluated label-free selector leaves a large gap to the best available model: on average 6.1 percentage points, up to 10.5. The paper concludes that the remaining distance to clinical deployment lies mostly in the selection step, not in adaptation, and that ensembling or a small target-labeling budget narrows but does not close the gap.","feed_headline":"No label-free selector reliably finds the best adapted model","feed_subtitle":"A good adapted model usually exists; picking it without target labels is the unsolved step.","key_machinery":"The central object is the 'validator,' a label-free scoring function $V(\\theta) \\mapsto s \\in \\mathbb{R}$ that assigns a scalar score to each checkpoint $\\theta$ without using target labels; the deployed model is $\\theta^\\star = \\arg\\max_{\\theta\\in\\Theta} V(\\theta)$, with $\\Theta$ the pool of checkpoints across algorithms, hyperparameters, and training iterations. Comparing a validator's choice with the oracle choice defines the selection gap. To test whether the gap is a bad choice among validators or a structural limitation, the paper uses the Spearman rank correlation between validation scores and true target accuracy, measured both within single algorithms and on an across-algorithm pool formed by taking each algorithm's top-10 checkpoints by validator score. The combination of the selection gap and the Spearman diagnostic is what turns the observation that 'validators sometimes fail' into a claim about structural unreliability.","core_discovery":"Under oracle selection (choosing with target labels), the across-algorithm checkpoint pool beats source-only in every scenario and approaches target-supervised training, so adaptation itself usually works. The failure is label-free model selection: under the best of the 13 validators, selected models average 6.1 percentage points below the oracle-best model (maximum 10.5), and the same gap appears across four different backbone architectures. The gap is structural: for within-algorithm selection a validator such as BNM (V) can be highly reliable on one algorithm ($\\rho=0.88$) and reversed on another ($\\rho=-0.42$), while for across-algorithm selection most validator correlations are weak or negative. Ensembling validator-selected checkpoints, and spending a small labeled target budget (about 20 samples for brain MRI, 4–5% for other modalities), both reduce the gap, but both remain below the oracle. The conclusions are drawn for binary classification measured by balanced accuracy.","pith_inferences":["Extension: the across-algorithm collapse of validator correlations suggests that validators should be designed for cross-algorithm comparability rather than tuned within one algorithm; absolute score calibration across heterogeneous checkpoints may be the missing ingredient.","Extension: the budget results imply a hybrid deployment recipe the paper does not push: use a label-free validator to shortlist checkpoints, then spend a small number of target labels to choose among the shortlist, which combines the two strategies and could reach the oracle with fewer than 25 labels.","Extension: if the structural gap persists for segmentation or detection, where prediction structure is richer, label-free selection difficulty could grow; the paper itself lists this as future work, and the likely direction is that the gap widens.","Extension: a testable consequence is that validator disagreements could be used as an uncertainty signal: checkpoints that many validators rank highly are more likely to be near the oracle, which would motivate studying consensus across validator scores rather than any single score."],"forward_implications":["Deployable medical UDA is gated by model selection: even when a strong adapted checkpoint exists, a practitioner using any of the 13 evaluated validators will, on average, leave 6.1 points of balanced accuracy unrealized.","No single validator can be pre-chosen as a default for clinical use; reliability flips with algorithm and scenario, so the same validator that ranks one algorithm's checkpoints well can rank another's in reverse.","Across-algorithm selection is harder than within-algorithm selection: pooling checkpoints from all algorithms makes most validator correlations weak or negative, so choosing which UDA algorithm to deploy is an unresolved sub-problem.","Ensembling the checkpoints selected by all validators across algorithms recovers much of the gap in several scenarios and can even match the best hindsight validator, but it is computationally costly and can hurt when poor checkpoints are naively averaged.","A small target-labeling budget (about 20 samples for brain MRI, 4–5% for other scenarios) selects models more accurately than the best validator or ensembling, but still leaves a gap to the oracle."],"supporting_citations":[{"why":"Supplies the Source-Risk validator, a source-guided selection baseline.","marker":"[12]"},{"why":"Supplies the MixVal validator, a target-based label-free scoring method.","marker":"[17]"},{"why":"Supplies the MCC validator, an information-maximization target-based criterion.","marker":"[20]"},{"why":"Provides the 'reality check' benchmark of UDA model selection that this study extends to the complete pipeline.","marker":"[34]"},{"why":"Supplies the DEV-N, ClassAMI, and BNM(V) validators and a large-scale ranking benchmark.","marker":"[35]"},{"why":"Supplies the SND validator, a target-based soft neighborhood density score.","marker":"[38]"},{"why":"Supplies the IWCV validator, an importance-weighted cross-validation source-guided score.","marker":"[43]"},{"why":"Supplies the TransScore validator, a target-based transferability score.","marker":"[52]"},{"why":"Supplies the DEV validator, a source-guided deep embedded validation score.","marker":"[53]"}],"fun_headline_variants":["UDA's real bottleneck: picking the best model without labels","Adaptation works, label-free selection doesn't","No validator reliably picks the best adapted model","Label-free selection gaps medical UDA from clinical use","Good adapted models exist; validators can't find them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The structural-gap claim assumes the 13 label-free selection methods are representative of such methods and that the top-10-per-algorithm pooling protocol is a fair test of ranking; if an untested selector is consistently reliable, or the pooling rule artificially lowers correlations, the gap would not be structural, and the finding is also scoped to binary classification with balanced accuracy.","fun_headline_variants_meta":{"raw":{"variants":["UDA's real bottleneck: picking the best model without labels","Adaptation works, label-free selection doesn't","No validator reliably picks the best adapted model","Label-free selection gaps medical UDA from clinical use","Good adapted models exist; validators can't find them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":3010,"prompt_tokens":941,"completion_tokens":2069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":557,"tokens_out":2069,"duration_ms":13195,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:18:43.177205+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same checkpoint pools, compute across-algorithm Spearman correlations pooling all checkpoints rather than the top-10 per algorithm, or test a new label-free score on all 11 scenarios: if any validator shows consistently positive correlation (say $\\rho>0.5$) across most scenarios, or selects a checkpoint within 2 percentage points of the oracle across most scenarios, the structural-gap claim is refuted.","supporting_citations":[{"cited_title":"Mixed samples as probes for un- supervised model selection in domain adaptation.Advances in Neural Information Processing Systems, 36:37923–37941,","cited_arxiv_id":null,"evidence_quote":"Supplies the MixVal validator, a target-based label-free scoring method."},{"cited_title":"Minimum class confusion for versatile domain adaptation","cited_arxiv_id":null,"evidence_quote":"Supplies the MCC validator, an information-maximization target-based criterion."},{"cited_title":"Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density","cited_arxiv_id":null,"evidence_quote":"Supplies the SND validator, a target-based soft neighborhood density score."},{"cited_title":"Unsupervised domain adaptation by backpropagation","cited_arxiv_id":null,"evidence_quote":"Supplies the Source-Risk validator, a source-guided selection baseline."},{"cited_title":"Covariate shift adaptation by importance weighted cross validation.JMLR, 8(5), 2007","cited_arxiv_id":null,"evidence_quote":"Supplies the IWCV validator, an importance-weighted cross-validation source-guided score."},{"cited_title":"Can we evaluate domain adaptation models without target-domain labels? InInternational Conference on Learn- ing Representations, pages 35061–35081, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the TransScore validator, a target-based transferability score."},{"cited_title":"Towards accurate model selection in deep unsupervised domain adap- tation","cited_arxiv_id":null,"evidence_quote":"Supplies the DEV validator, a source-guided deep embedded validation score."}],"review_version":1}