Pith. sign in

REVIEW 3 major objections 6 minor 34 references

A two-level agreement vote among UDA models can pick algorithm and hyperparameters without any target labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 17:04 UTC pith:SLHZXBO5

load-bearing objection Solid practical selection rule for medical UDA; real gains over validators, but selection and reported accuracy share the same target val split. the 3 major comments →

arxiv 2607.28125 v1 pith:SLHZXBO5 submitted 2026-07-30 cs.CV cs.AI

Towards Practical Algorithm Selection for Unsupervised Domain Adaptation in Medical Imaging

Yiheng Xiong , Luisa Gall\'ee , Daniel Santak Wolf , Heiko Hillenhagen , Michael G\"otz This is my paper
classification cs.CV cs.AI
keywords unsupervised domain adaptationalgorithm selectionmedical imaginglabel-free model selectiondomain shiftbrain MRIchest X-ray
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

In clinical imaging, unsupervised domain adaptation (UDA) can transfer a model from a labeled hospital to an unlabeled one, but practitioners cannot score candidates on the target domain because labels are missing. This paper claims that joint algorithm-and-hyperparameter selection is still possible: build a label-free reference by letting many existing validators each nominate a checkpoint inside every algorithm, then majority-vote those nominations across algorithms on every unlabeled target image. Score every candidate by how closely its predictions match that reference, and deploy the single highest-scoring checkpoint. On seven brain-MRI and chest X-ray transfers the method reaches higher average target accuracy than any individual validator and roughly halves the remaining gap to the best checkpoint one could pick with labels. The same ranking stays stable when the algorithm pool shrinks or when fewer checkpoints are kept per run, so the procedure is meant to be usable under realistic compute budgets.

Core claim

Given a heterogeneous pool of UDA checkpoints spanning algorithms, hyperparameters, and training iterations, a two-level agreement reference—validators nominating within each algorithm, then majority vote across algorithms—produces a label-free target against which candidates can be ranked, selecting a single deployable model whose target accuracy substantially beats individual validators and closes much of the gap to the labeled oracle.

What carries the argument

Two-level agreement reference: Level 1 lets each validator pick its best checkpoint inside every algorithm; Level 2 majority-votes those nominations per target sample to form a pseudo-label reference; each candidate is then scored by mean per-class agreement with that reference and the top scorer is deployed.

Load-bearing premise

The majority vote among validator-picked checkpoints from different algorithms is assumed not to share the same systematic mistakes, so ranking by agreement with that vote still tracks true target performance.

What would settle it

On a held-out medical UDA transfer, if the checkpoint with highest agreement to the two-level reference is systematically far below both the best individual validator and the labeled oracle—especially when algorithms share failure modes—the selection claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A clinic can train several UDA algorithms with a few hyperparameter settings, keep periodic checkpoints, and pick one model for deployment without labeling target scans.
  • Adding algorithms from different paradigms improves the upper bound the selector can reach, while the same criterion still works on smaller pools when compute is limited.
  • Selection quality does not require dense checkpointing; retaining a modest fraction of snapshots per run remains effective.
  • The same label-free ranking can be recomputed on any new unlabeled target batch, so the chosen model can be refreshed when data drift.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same two-level vote could be tried as a stopping rule during a single long UDA run, not only as a post-hoc selector across finished runs.
  • If shared failure modes among adversarial or pseudo-label methods dominate the vote, weighting validators by diversity of their nominated algorithms may be a natural next control.
  • Extending the agreement score from balanced accuracy on binary tasks to sensitivity-weighted or multi-class metrics would test whether the reference still ranks clinically preferred operating points.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a label-free criterion for jointly selecting a UDA algorithm and its hyperparameters for medical imaging classification. From a pool of checkpoints spanning multiple algorithms and hyperparameter settings, it builds a two-level agreement reference without target labels: (Level 1) each of several existing validators nominates the best checkpoint within each algorithm; (Level 2) those nominations are aggregated by majority vote over unlabeled target samples to form a reference prediction. Every candidate is then scored by macro per-class agreement with this reference (Eqs. 2–5), and the single highest-scoring checkpoint is deployed. On seven brain-MRI and CXR transfer scenarios the method reports 86.3% average target balanced accuracy versus 81.0% for the best individual validator (Class-AMI), cutting the gap to a labeled Oracle from 10.4% to 5.1% (Table 2), with ablations on aggregation structure, algorithm-pool size/composition, and checkpoint retention (Table 3, Fig. 2).

Significance. Joint algorithm-and-hyperparameter selection under unlabeled targets is a genuine clinical bottleneck that most UDA validator work leaves open. The two-level construction is a clear, implementable idea; the empirical scope (two modalities, seven transfers, different CXR backbones, five-fold means±std, many established validators, Oracle upper bound, and pool/density ablations) is stronger than typical medical UDA selection papers. Open-sourced code further supports reproducibility. If the selection quality holds under a stricter evaluation protocol, the work is a useful step toward practical UDA deployment rather than another adaptation loss.

major comments (3)
  1. [§3 Experimental Setup; Eqs. 3–5; Table 2] §3 Experimental Setup explicitly states that no separate target test set is held out and that target performance is measured on the same target validation split used to form the Level-2 reference (Eq. 3) and agreement scores s(θ) (Eq. 4). Selection and reported accuracy therefore share the identical unlabeled samples. A checkpoint can maximize agreement with consensus errors or finite-sample idiosyncrasies of that split and still inflate Table 2 numbers without being near-best under the broader target distribution. This is load-bearing for the central claim (86.3% vs 81.0%, Oracle gap 5.1% vs 10.4%). Please re-evaluate with a protocol that separates the unlabeled data used to build ˜y and s(θ) from the labeled target split used only for reporting accuracy (e.g., select on target-train unlabeled features, report on target-val), or provide an equivalent held-out analysis. Fig. 2 ablations
  2. [§2.2–2.3; Table 2; Limitations] The method’s premise (§2.2–2.3) is that majority vote among validator-nominated checkpoints across heterogeneous algorithms yields a reference correlated enough with true labels to rank a near-best single checkpoint. Shared failure modes among algorithms/validators would make both s(θ) and in-sample accuracy look good while ˆθ is suboptimal. Beyond overall averages, the paper should quantify reference quality (e.g., agreement of ˜y with true target labels per scenario, or cases where the selected checkpoint matches a systematically wrong consensus) and report how often the selected ˆθ is within a small margin of the Oracle versus merely better than Class-AMI. The remaining 5.1% Oracle gap and the larger shortfall on RSNA→Child CXR (Table 2) are natural places for this analysis; without it, the weakest assumption stays largely untested.
  3. [Table 3; §4 Discussion] Table 3 shows that the flat “All checkpoints” baseline already reaches 82.4% overall (vs 81.0% best validator and 86.3% two-level). The discussion offers a plausible diversity argument, but there is no statistical test or fold-level comparison establishing that the two-level gain over All checkpoints / Per-validator is reliable rather than scenario-noise. Given that the paper’s novelty is precisely the two-level structure, please report paired fold-wise differences (or confidence intervals) for Table 3 and clarify whether Level-1 validator nomination is necessary when a simple all-checkpoint vote is already competitive.
minor comments (6)
  1. [Abstract] Abstract and elsewhere: “algori-thms” line-break hyphenation appears in the compiled text; clean up for camera-ready.
  2. [§2.3 Eq. (4)] Eq. (4): the macro per-class match to ˜y is reasonable, but for the imbalanced CXR sets (Table 1) it would help to state why macro agreement was preferred over micro agreement or balanced accuracy against the reference, and whether ranking is sensitive to that choice.
  3. [§2.1; §3] The validator pool V is listed in §2.1 but which subset is used in the main experiments is not stated in one place; a short explicit list (and optionally a leave-one-validator-out check) would aid reproducibility.
  4. [Fig. 1] Fig. 1 is helpful but the “Config.” / algorithm / validator layout is dense; ensuring the Level-1 vs Level-2 boundary is visually sharper would help non-specialist readers.
  5. [§1; §5] Claim of being “the first” joint algorithm-and-hyperparameter selection method (§1, §5) should briefly contrast the sequential algorithm-then-hyperparameter protocol of Yang et al. [32] so the novelty boundary is precise.
  6. [Abstract / footnote] Code link is given as “Complete UDA Pipeline” without a stable URL/DOI in the text; add a persistent link in the camera-ready version.

Circularity Check

0 steps flagged

No significant circularity: ensemble-consensus selection is a proxy procedure evaluated against external labeled target accuracy, not a derivation that reduces to its inputs.

full rationale

This is an empirical UDA model-selection paper, not a first-principles derivation. The two-level agreement reference (Eqs. 2–3) and agreement score s(θ) (Eq. 4) are deliberately built from candidate predictions and validators; that self-reference is the method, not a hidden reduction of a claimed prediction to a fitted input. Reported results in Table 2 and Fig. 2 are balanced accuracy computed with held-out target labels that are never used in selection—an external evaluation metric. Validators and UDA algorithms are taken from the external literature (DEV, Entropy, Class-AMI, DANN, CDAN, MCD, etc.); there is no load-bearing self-citation, uniqueness theorem imported from the authors, or ansatz smuggled via prior own work. Sharing the same unlabeled target split for building the reference and for reporting labeled accuracy is a methodological limitation (possible in-sample ranking bias), but it does not make the accuracy numbers equal to s(θ) by construction. No step in the chain reduces a claimed result to its definitional inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

This is an empirical systems/methods paper. The central claim rests on standard UDA training assumptions, the usefulness of existing label-free validators as weak signals, and the design choice that cross-algorithm majority among those signals is a good unlabeled proxy. No physical constants or deep new ontology; free choices are training grids, pool composition, and the agreement scoring formula.

free parameters (4)
  • adaptation weight λ grid = {0.1, 0.5, 1.0}
    Each algorithm is run at λ ∈ {0.1, 0.5, 1.0}; this discrete hand-chosen grid defines the hyperparameter axis of the candidate pool the selector chooses from.
  • checkpoints per run / retention = 50 per run (default)
    50 post-warmup checkpoints per run (150 per algorithm) are saved on a uniform schedule; ablations subsample retention ratios 0.1–0.9. Density is a design choice affecting the pool.
  • validator set V composition = fixed catalog (DEV, Entropy, Class-AMI, etc.)
    Which label-free validators participate in Level-1 nomination changes the reference; the paper uses a fixed catalog of source-guided and target-specific validators without a learned weighting.
  • agreement score aggregation (macro per-class match) = macro-averaged per-class indicator match
    Eq. 4 averages per-class agreement fractions; alternative agreements (micro accuracy, KL on probabilities) are not selected by data-driven tuning in the paper but the macro form is a modeling choice that defines the ranking.
axioms (5)
  • domain assumption Standard UDA setup: labeled source and unlabeled target; model trained with L = L_cls + λ L_adapt generalizes better on target than source-only when adaptation works.
    Stated in §2.1 as the problem setting; all candidates are produced under this assumption.
  • domain assumption Existing label-free validators are noisy but positively informative proxies for target performance within an algorithm’s checkpoint trajectory.
    Level 1 (Eq. 2) treats argmax_θ v(θ) as a meaningful nomination; without this, the reference collapses.
  • ad hoc to paper Majority vote across algorithms and validators on unlabeled target samples yields a reference correlated with true labels enough to rank a strong single checkpoint.
    Core design hypothesis of §2.2–2.3 (Eqs. 3–5); supported empirically here but not derived from a proven bound.
  • ad hoc to paper Target validation balanced accuracy on the same unlabeled split used to form agreement scores is an adequate report of selection quality (no separate target test set).
    Explicit experimental-setup choice in §3; labels unused for selection but evaluation and selection features share the split.
  • domain assumption Binary medical classification transfers considered (AD vs CN; pneumonia vs non-pneumonia) are representative enough to support the practical-selection claim.
    Dataset construction in §3; limitations section notes multi-class/segmentation remain future work.
invented entities (2)
  • two-level agreement reference (validator-nominated within-algorithm checkpoints aggregated by across-algorithm majority vote) no independent evidence
    purpose: Provide a label-free pseudo-target against which every checkpoint is scored for joint algorithm and hyperparameter selection.
    Defined in §2.2 Eqs. 2–3; not a physical entity but a new procedural object. Independent evidence is only the empirical correlation with Oracle-selected performance in this study.
  • agreement score s(θ) as macro per-class match to the agreement reference no independent evidence
    purpose: Scalar ranking criterion to pick a single deployable checkpoint from the full multi-algorithm pool.
    Eq. 4–5; standard agreement idea specialized to this reference. No external falsifiable handle beyond the paper’s selection tables.

pith-pipeline@v1.2.0-daily-grok45 · 14158 in / 3765 out tokens · 87355 ms · 2026-07-31T17:04:14.632447+00:00 · methodology

0 comments
read the original abstract

Numerous unsupervised domain adaptation (UDA) algori-thms exist, but for clinical practice, selecting the best-suited one along with proper hyperparameters often remains unclear, as the unlabeled deployment (target) domain prevents direct evaluation. We propose a label-free criterion that jointly selects the algorithm and hyperparameters for UDA. Given a pool of candidate models from multiple algorithms trained with different hyperparameters, our approach scores each candidate against an agreement reference, and selects the one with the highest score. The agreement reference is constructed in two levels without using target labels. First, we leverage multiple label-free selection signals, using each to nominate a model within every algorithm. Second, the nominated models are aggregated across algorithms to form a reference prediction for each unlabeled target sample. The candidate whose predictions agree most with this reference is then selected for deployment. Experimental results on four brain MRI and four chest X-ray datasets across seven clinically relevant transfer scenarios show that our method achieves better selection performance than other methods and remains effective across different algorithm pools. Our approach takes a step towards practical, label-free algorithm selection for clinical deployment of UDA.

Figures

Figures reproduced from arXiv: 2607.28125 by Daniel Santak Wolf, Heiko Hillenhagen, Luisa Gall\'ee, Michael G\"otz, Yiheng Xiong.

Figure 1
Figure 1. Figure 1: Two-level agreement reference construction. Each candidate combines an [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ablation on the algorithm pool (left) and the fraction of checkpoints [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 4 linked inside Pith

  1. [1]

    https://www.kaggle.com/datasets/ tawsifurrahman/covid19-radiography-database

    Covid-19 radiography database. https://www.kaggle.com/datasets/ tawsifurrahman/covid19-radiography-database

  2. [2]

    https://www.kaggle.com/datasets/ omkarmanohardalvi/lungs-disease-dataset-4-types

    Lungs disease dataset (4 types). https://www.kaggle.com/datasets/ omkarmanohardalvi/lungs-disease-dataset-4-types

  3. [3]

    In: CVPR

    Chen, L., Chen, H., Wei, Z., et al.: Reusing the task-specific classifier as a discrimi- nator: Discriminator-free adversarial domain adaptation. In: CVPR. pp. 7181–7190 (2022)

    2022

  4. [4]

    In: CVPR

    Cui, S., Wang, S., Zhuo, J., et al.: Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In: CVPR. pp. 3941–3950 (2020)

    2020

  5. [5]

    In: 2009 IEEE conference on computer vision and pattern recognition

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    2009

  6. [6]

    International psychogeriatrics21(4), 672–687 (2009) 10 Y

    Ellis, K.A., Bush, A.I., Darby, D., et al.: The australian imaging, biomarkers and lifestyle (aibl) study of aging: methodology and baseline characteristics of 1112 individuals recruited for a longitudinal study of alzheimer’s disease. International psychogeriatrics21(4), 672–687 (2009) 10 Y. Xiong et al

    2009

  7. [7]

    Medical Image Analysis83, 102664 (2023)

    Feng, Y., Wang, Z., Xu, X., Wang, Y., Fu, H., Li, S., Zhen, L., Lei, X., Cui, Y., Ting, J.S.Z., et al.: Contrastive domain adaptation with consistency match for automated pneumonia diagnosis. Medical Image Analysis83, 102664 (2023)

    2023

  8. [8]

    In: ICML

    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: ICML. pp. 1180–1189. PMLR (2015)

    2015

  9. [9]

    Journal of machine learning research17(59), 1–35 (2016)

    Ganin, Y., Ustinova, E., Ajakan, H., et al.: Domain-adversarial training of neural networks. Journal of machine learning research17(59), 1–35 (2016)

    2016

  10. [10]

    Medical image analysis71, 102076 (2021)

    Guan, H., Liu, Y., Yang, E., et al.: Multi-site mri harmonization via attention- guided deep domain adaptation for brain disorder identification. Medical image analysis71, 102076 (2021)

    2021

  11. [11]

    In: CVPR

    He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)

    2016

  12. [12]

    Advances in Neural Infor- mation Processing Systems36, 37923–37941 (2023)

    Hu, D., Liang, J., Liew, J.H., Xue, C., Bai, S., Wang, X.: Mixed samples as probes for unsupervised model selection in domain adaptation. Advances in Neural Infor- mation Processing Systems36, 37923–37941 (2023)

    2023

  13. [13]

    NeurIPS37, 135883–135903 (2024)

    Hu, D., Luo, R., Liang, J., et al.: Towards reliable model selection for unsuper- vised domain adaptation: An empirical study and a certified baseline. NeurIPS37, 135883–135903 (2024)

    2024

  14. [14]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)

    2017

  15. [15]

    Journal of Magnetic Resonance Imag- ing: An Official Journal of the International Society for Magnetic Resonance in Medicine27(4), 685–691 (2008)

    Jack Jr, C.R., Bernstein, M.A., Fox, N.C., et al.: The alzheimer’s disease neu- roimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imag- ing: An Official Journal of the International Society for Magnetic Resonance in Medicine27(4), 685–691 (2008)

    2008

  16. [16]

    In: ECCV

    Jin, Y., Wang, X., Long, M., et al.: Minimum class confusion for versatile domain adaptation. In: ECCV. pp. 464–480. Springer (2020)

    2020

  17. [17]

    cell172(5), 1122–1131 (2018)

    Kermany, D.S., Goldbaum, M., Cai, W., et al.: Identifying medical diagnoses and treatable diseases by image-based deep learning. cell172(5), 1122–1131 (2018)

    2018

  18. [18]

    Computers in Biology and Medicine170, 107912 (2024)

    Kumari, S., Singh, P.: Deep learning for unsupervised domain adaptation in med- ical imaging: Recent advancements and future perspectives. Computers in Biology and Medicine170, 107912 (2024)

    2024

  19. [19]

    In: CVPR

    Liang, J., Hu, D., Feng, J.: Domain adaptation with auxiliary target domain- oriented classifier. In: CVPR. pp. 16632–16642 (2021)

    2021

  20. [20]

    IEEE Journal of Biomedical and Health Informatics27(12), 5848–5859 (2023)

    Liu, W., Ni, Z., Chen, Q., Ni, L.: Attention-guided partial domain adaptation for automated pneumonia diagnosis from chest x-ray images. IEEE Journal of Biomedical and Health Informatics27(12), 5848–5859 (2023)

    2023

  21. [21]

    In: ICML

    Long, M., Cao, Y., Wang, J., et al.: Learning transferable features with deep adap- tation networks. In: ICML. pp. 97–105. PMLR (2015)

    2015

  22. [22]

    NeurIPS31(2018)

    Long, M., Cao, Z., Wang, J., et al.: Conditional adversarial domain adaptation. NeurIPS31(2018)

    2018

  23. [23]

    arXiv preprint arXiv:1711.10288 (2017)

    Morerio, P., Cavazza, J., Murino, V.: Minimal-entropy correlation alignment for unsupervised deep domain adaptation. arXiv preprint arXiv:1711.10288 (2017)

    Pith/arXiv arXiv 2017

  24. [24]

    arXiv preprint arXiv:2111.15672 (2021)

    Musgrave, K., Belongie, S., Lim, S.N.: Unsupervised domain adaptation: A reality check. arXiv preprint arXiv:2111.15672 (2021)

    Pith/arXiv arXiv 2021

  25. [25]

    arXiv preprint arXiv:2208.07360 (2022)

    Musgrave, K., Belongie, S., Lim, S.N.: Three new validators and a large- scale benchmark ranking for unsupervised domain adaptation. arXiv preprint arXiv:2208.07360 (2022)

    Pith/arXiv arXiv 2022

  26. [26]

    arXiv preprint arXiv:1710.06924 (2017) Towards Practical Algorithm Selection for UDA in Medical Imaging 11

    Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., Saenko, K.: Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 (2017) Towards Practical Algorithm Selection for UDA in Medical Imaging 11

    Pith/arXiv arXiv 2017

  27. [27]

    In: ICCV

    Saito, K., Kim, D., Teterwak, P., et al.: Tune it the right way: Unsupervised valida- tion of domain adaptation via soft neighborhood density. In: ICCV. pp. 9184–9193 (2021)

    2021

  28. [28]

    In: CVPR

    Saito, K., Watanabe, K., Ushiku, Y., et al.: Maximum classifier discrepancy for unsupervised domain adaptation. In: CVPR. pp. 3723–3732 (2018)

    2018

  29. [29]

    JMLR8(5) (2007)

    Sugiyama, M., Krauledat, M., Müller, K.R.: Covariate shift adaptation by impor- tance weighted cross validation. JMLR8(5) (2007)

    2007

  30. [30]

    Tu, W., Deng, W., Gedeon, T., et al.: Assessing model out-of-distribution gener- alization with softmax prediction probability baselines and a correlation method

  31. [31]

    In: CVPR

    Wang, X., Peng, Y., Lu, L., et al.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: CVPR. pp. 2097–2106 (2017)

    2097

  32. [32]

    Yang, J., Qian, H., Xu, Y., Wang, K., Xie, L.: Can we evaluate domain adaptation models without target-domain labels? In: International Conference on Learning Representations. vol. 2024, pp. 35061–35081 (2024)

    2024

  33. [33]

    In: ICML

    You, K., Wang, X., Long, M., et al.: Towards accurate model selection in deep unsupervised domain adaptation. In: ICML. pp. 7124–7133. PMLR (2019)

    2019

  34. [34]

    IEEE TIP29, 7834–7844 (2020)

    Zhang, Y., Wei, Y., Wu, Q., et al.: Collaborative unsupervised domain adaptation for medical image diagnosis. IEEE TIP29, 7834–7844 (2020)

    2020