REVIEW 4 major objections 5 minor 18 references
Multi-Source COVID-19 Detection via Variance Risk Extrapolation
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that VREx domain-aligned pretraining followed by Mixup fine-tuning reaches a 0.96 average macro F1 for COVID-19 detection on chest CT scans from four hospitals.
desk verdict Thin challenge report: the 0.96 F1 is not evidence for VREx/Mixup without an ERM baseline or ablations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the VREx training objective, written by the paper as $\min_\theta \left( \frac{1}{n}\sum_{i=1}^n \mathcal{L}_i(\theta) + \lambda \operatorname{Var}[\mathcal{L}_i(\theta)] \right)$, where $\mathcal{L}_i$ is the classification loss on domain $i$ and $\lambda$ balances average risk against its variance; this term is what enforces cross-domain consistency during pretraining. The second half of the mechanism is Mixup, which generates virtual examples $\tilde{x} = \lambda x_i + (1-\lambda)x_j$ and $\tilde{y} = \lambda y_i + (1-\lambda)y_j$ from pairs of training samples, potentially across domains, to regularize fine-tuning. The encoder on which both stages run is not specified, so the contribution is carried by the two training losses rather than by a new architecture.
What would settle it
Train the same encoder on the same data with ordinary loss minimization in place of the VREx penalty and with Mixup disabled; if the average macro F1 stays at 0.96, or if none of the four per-domain scores changes, the proposed stages are not responsible for the reported generalization.
Extended reading notes
Core claim
The central claim is that the paper's two-stage framework—VREx domain-aligned pretraining followed by Mixup domain-mixed fine-tuning—classifies COVID-19 versus non-COVID chest CT scans with an average macro F1 of 0.96 when evaluated across four distinct hospital domains. In Stage 1, the model minimizes a weighted combination of the mean per-domain empirical risk and the variance of those risks, so no single institution's loss pattern is allowed to dominate. In Stage 2, Mixup interpolates inputs and labels, and the paper argues this smooths decision boundaries and reduces overfitting to domain-specific features. The 0.96 macro F1 on the held-out validation set is presented as evidence that the model has captured domain-invariant representations rather than memorizing center-specific cues.
Load-bearing premise
The load-bearing assumption is that the reported 0.96 macro F1 comes from the VREx and Mixup stages rather than from the unspecified pretrained encoder or the particular validation set, and the paper gives no baseline or ablation that would separate those causes.
Editorial extensions
If this is right
- A hospital with labeled local data can run the same two-stage recipe and expect a detector that carries over to other centers without retraining on their labels, since neither stage needs target-domain data.
- The variance penalty gives a concrete way to stop models latching onto scanner- and protocol-specific cues whenever source domains are known during training.
- Cross-domain Mixup during fine-tuning provides a route to smoother decision boundaries on small per-center datasets, as the 1,124 training scans here require.
- The 0.96 aggregate macro F1 becomes a benchmark number that future multi-source COVID-19 detectors can be compared against on the same validation set.
Reading between the lines
- Per-domain F1 scores would show whether the 0.96 average is uniform across the four hospitals or carried by one; the paper reports only the aggregate, but macro averaging makes this check straightforward.
- Applying the VREx variance penalty during fine-tuning as well as pretraining, or weighting the penalty per domain, are natural variants the paper leaves untested.
- Since the method needs only known source-domain labels, the same recipe could transfer to other multi-source medical imaging tasks, such as pneumonia or lung-nodule classification, whenever hospital of origin is recorded.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript reports a challenge solution for the Multi-Source COVID-19 Detection Challenge, which requires binary classification of chest CT scans into COVID and Non-COVID categories across four hospital sources. The proposed method is a two-stage framework: Stage 1 uses Variance Risk Extrapolation (VREx) as a domain-aligned pretraining objective that minimizes the variance of empirical risks across source domains, and Stage 2 fine-tunes the model with Mixup data augmentation. The paper reports an average macro F1 score of 0.96 on the validation set and claims that this demonstrates strong generalization. The manuscript is very short, contains no comparison to empirical risk minimization (ERM), no ablations, no per-domain results, no repeated-run statistics, and no hyperparameter details.
Significance. If the reported result is reproducible and the attribution to VREx and Mixup is correct, the manuscript would demonstrate a practically useful recipe for cross-institutional COVID-19 CT detection, which is a relevant clinical application. However, the paper currently does not establish that the two proposed components cause the improvement over the unspecified encoder or over standard training, and both VREx and Mixup are well-known techniques. The main value of the paper is therefore as a challenge-solution report rather than as a methodological advance. The manuscript is concise and the problem is clearly stated, but the lack of rigorous evaluation limits its significance.
major comments (4)
- [Section 4] The central claim that VREx pretraining and Mixup fine-tuning yield an average macro F1 of 0.96 is unsupported because Section 4 reports only a single scalar with no ERM baseline, no lambda=0 control, no VREx-only or Mixup-only ablation, no per-domain macro F1, and no repeated-run statistics. Add these experiments, including seed variance and a per-domain breakdown, or the attribution of the score to the proposed components does not follow.
- [Section 2, Stage 1] The encoder is never specified: the text says the framework builds on the CMC network [4-7], but those references are COVID-19 CT classification methods, and the architecture, pretraining data, and initialization are omitted. Without an ERM baseline using the same encoder, or an ablation that removes the VREx term from Eq. (1), a reader cannot determine whether the 0.96 F1 comes from VREx and Mixup or from a pretrained representation that already separates COVID from Non-COVID.
- [Section 3] The validation set composition in Table 1 is given only in aggregate: 180 Non-COVID vs 128 COVID, with no per-domain counts. Because the reported macro F1 averages four sources and some sources may have very few positive cases, the aggregate number is unstable and under-specified; report the per-domain distribution and interval estimates such as bootstrap confidence intervals or per-seed results.
- [Section 2] The experimental configuration is not reproducible: the value of lambda in Eq. (1), the Mixup parameter in Eq. (2), optimizer, learning rate, batch size, number of epochs, image preprocessing, and evaluation protocol are all missing. Provide a complete hyperparameter table, or state that the challenge protocol fixes these values with a citation.
minor comments (5)
- [Eqs. (1) and (2)] The symbol lambda is used both for the VREx penalty coefficient in Eq. (1) and for the Mixup interpolation weight in Eq. (2); using different symbols (e.g., beta for Mixup) would avoid ambiguity.
- [Figure 1] Figure 1 appears as a low-resolution diagram with garbled or unreadable labels in the provided manuscript; redraw it with clear text for the loss terms and stage division.
- [Section 2] The phrase 'PHAROS-AFE-AIMI' is not defined; state the full name of the challenge and provide a reference or URL.
- [Abstract] The abstract says the result 'demonstrating strong generalization,' but only a validation set is used; no test set is discussed, so generalization beyond the validation split is not actually demonstrated.
- [References] Reference [18] is cited as an arXiv preprint; the published ICLR 2018 version of the Mixup paper should be cited if available.
Circularity Check
No significant circularity: the paper reports a directly measured validation F1; neither VREx/Mixup attribution nor the CMC self-citation forces the result by construction.
full rationale
The paper contains no derivation chain whose output is equivalent to its input. Section 4 reports an average macro F1 of 0.96 as a direct measurement on the validation set, not as a quantity fitted from, or defined in terms of, the training objective. Equation (1) is the standard VREx objective and Eq. (2) is standard Mixup; neither equation implicitly contains the reported F1, and no parameter is tuned so that the F1 follows by construction. The reference to the authors' earlier CMC network [4-7] is used as an architectural foundation, but the paper's central claim is an empirical score, not a theorem derived from that citation. The absence of an ERM baseline, ablations, per-domain statistics, or repeated-run variance weakens the causal attribution of the score to VREx and Mixup, but that is a lack of experimental support rather than circularity. Under the hard rules, concerns about attribution without controlled comparison do not constitute a circular step because no specific equation or fitted quantity reduces to its own input. The honest finding is therefore no significant circularity.
Assumptions & free parameters
free parameters (3)
- VREx variance penalty coefficient lambda
- Mixup interpolation weight or distribution parameter
- Training hyperparameters (learning rate, batch size, epochs, optimizer)
assumptions (5)
- domain assumption The VREx objective in Eq. (1) encourages domain-invariant representations that generalize to unseen domains.
- domain assumption Mixup in Eq. (2) improves robustness and generalization.
- domain assumption The four training domains adequately represent the relevant sources of domain shift.
- domain assumption The validation set is an unbiased sample from the target distribution and the labels are correct.
- domain assumption The unspecified encoder has sufficient capacity and initialization to support the classification task.
Cite this review
Pith. "Pith review of Multi-Source COVID-19 Detection via Variance Risk Extrapolation." pith.science (2026). https://pith.science/paper/DSNMFRWH
@misc{pith2026250623208,
author = {Pith},
title = {Pith review of: Multi-Source COVID-19 Detection via Variance Risk Extrapolation},
year = {2026},
howpublished = {\url{https://pith.science/paper/DSNMFRWH}},
note = {Machine review of arXiv:2506.23208}
}
read the original abstract
We present our solution for the Multi-Source COVID-19 Detection Challenge, which aims to classify chest CT scans into COVID and Non-COVID categories across data collected from four distinct hospitals and medical centers. A major challenge in this task lies in the domain shift caused by variations in imaging protocols, scanners, and patient populations across institutions. To enhance the cross-domain generalization of our model, we incorporate Variance Risk Extrapolation (VREx) into the training process. VREx encourages the model to maintain consistent performance across multiple source domains by explicitly minimizing the variance of empirical risks across environments. This regularization strategy reduces overfitting to center-specific features and promotes learning of domain-invariant representations. We further apply Mixup data augmentation to improve generalization and robustness. Mixup interpolates both the inputs and labels of randomly selected pairs of training samples, encouraging the model to behave linearly between examples and enhancing its resilience to noise and limited data. Our method achieves an average macro F1 score of 0.96 across the four sources on the validation set, demonstrating strong generalization.
Figures
Reference graph
Works this paper leans on
-
[1]
A large imaging database and novel deep neural ar- chitecture for covid-19 diagnosis
Anastasios Arsenos, Dimitrios Kollias, and Stefanos Kol- lias. A large imaging database and novel deep neural ar- chitecture for covid-19 diagnosis. In 2022 IEEE 14th Im- age, Video, and Multidimensional Signal Processing Work- shop (IVMSP), page 1–5. IEEE, 2022. 1
2022
-
[2]
Data-driven covid-19 detection through medical imaging
Anastasios Arsenos, Andjoli Davidhi, Dimitrios Kollias, Panos Prassopoulos, and Stefanos Kollias. Data-driven covid-19 detection through medical imaging. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), page 1–5. IEEE, 2023
2023
-
[3]
Covid- 19 computer-aided diagnosis through ai-assisted ct imaging analysis: Deploying a medical ai system
Demetris Gerogiannis, Anastasios Arsenos, Dimitrios Kol- lias, Dimitris Nikitopoulos, and Stefanos Kollias. Covid- 19 computer-aided diagnosis through ai-assisted ct imaging analysis: Deploying a medical ai system. In 2024 IEEE In- ternational Symposium on Biomedical Imaging (ISBI), pages 1–4. IEEE, 2024. 1
2024
-
[4]
Cmc-cov19d: Contrastive mixup classification for covid-19 diagnosis
Junlin Hou, Jilan Xu, Rui Feng, Yuejie Zhang, Fei Shan, and Weiya Shi. Cmc-cov19d: Contrastive mixup classification for covid-19 diagnosis. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 454–461,
-
[5]
Periphery-aware covid-19 diagnosis with contrastive repre- sentation enhancement
Junlin Hou, Jilan Xu, Longquan Jiang, Shanshan Du, Rui Feng, Yuejie Zhang, Fei Shan, and Xiangyang Xue. Periphery-aware covid-19 diagnosis with contrastive repre- sentation enhancement. Pattern Recognition, 118:108005, 2021
2021
-
[6]
Cmc v2: Towards more accu- rate covid-19 detection with discriminative video priors
Junlin Hou, Jilan Xu, Nan Zhang, Yi Wang, Yuejie Zhang, Xiaobo Zhang, and Rui Feng. Cmc v2: Towards more accu- rate covid-19 detection with discriminative video priors. In European Conference on Computer Vision, pages 485–499. Springer, 2022
work page 2022
-
[7]
Boosting covid-19 severity detec- tion with infection-aware contrastive mixup classification
Junlin Hou, Jilan Xu, Nan Zhang, Yuejie Zhang, Xiaobo Zhang, and Rui Feng. Boosting covid-19 severity detec- tion with infection-aware contrastive mixup classification. In European Conference on Computer Vision, pages 537–551. Springer, 2022. 1
work page 2022
-
[8]
Deep neural archi- tectures for prediction in healthcare
Dimitrios Kollias, Athanasios Tagaris, Andreas Stafylopatis, Stefanos Kollias, and Georgios Tagaris. Deep neural archi- tectures for prediction in healthcare. Complex & Intelligent Systems, 4(2):119–131, 2018. 1
2018
Show all 18 references
-
[9]
Deep transparent prediction through latent represen- tation analysis
Dimitrios Kollias, N Bouas, Y Vlaxos, V Brillakis, M Se- feris, Ilianna Kollia, Levon Sukissian, James Wingate, and S Kollias. Deep transparent prediction through latent represen- tation analysis. arXiv preprint arXiv:2009.07044, 2020
2009 arXiv
-
[10]
Transpar- ent adaptation in deep medical image diagnosis
Dimitris Kollias, Y Vlaxos, M Seferis, Ilianna Kollia, Levon Sukissian, James Wingate, and Stefanos D Kollias. Transpar- ent adaptation in deep medical image diagnosis. In TAILOR, page 251–267, 2020
2020
-
[11]
Mia-cov19d: Covid-19 detection through 3-d chest ct image analysis
Dimitrios Kollias, Anastasios Arsenos, Levon Soukissian, and Stefanos Kollias. Mia-cov19d: Covid-19 detection through 3-d chest ct image analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , page 537–544, 2021
2021
-
[12]
Ai-mia: Covid-19 detection and severity analysis through medical imaging
Dimitrios Kollias, Anastasios Arsenos, and Stefanos Kollias. Ai-mia: Covid-19 detection and severity analysis through medical imaging. In European Conference on Computer Vi- sion, page 677–690. Springer, 2022
2022
-
[13]
Ai-enabled analysis of 3-d ct scans for diagnosis of covid-19 & its severity
Dimitrios Kollias, Anastasios Arsenos, and Stefanos Kollias. Ai-enabled analysis of 3-d ct scans for diagnosis of covid-19 & its severity. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICAS- SPW), page 1–5. IEEE, 2023
2023
-
[14]
A deep neural architecture for harmonizing 3-d input data analysis and decision making in medical imaging
Dimitrios Kollias, Anastasios Arsenos, and Stefanos Kollias. A deep neural architecture for harmonizing 3-d input data analysis and decision making in medical imaging. Neuro- computing, 542:126244, 2023
2023
-
[15]
Domain adaptation explainability & fairness in ai for medical image analysis: Diagnosis of covid-19 based on 3-d chest ct-scans
Dimitrios Kollias, Anastasios Arsenos, and Stefanos Kol- lias. Domain adaptation explainability & fairness in ai for medical image analysis: Diagnosis of covid-19 based on 3-d chest ct-scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...
2024
-
[16]
Sam2clip2sam: Vision language model for segmentation of 3d ct scans for covid-19 detection
Dimitrios Kollias, Anastasios Arsenos, James Wingate, and Stefanos Kollias. Sam2clip2sam: Vision language model for segmentation of 3d ct scans for covid-19 detection. arXiv preprint arXiv:2407.15728, 2024. 1
2024 arXiv
-
[17]
Out-of-distribution general- ization via risk extrapolation (rex)
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution general- ization via risk extrapolation (rex). In International confer- ence on machine learning, pages 5815–5826. PMLR, 2021. 1
2021
-
[18]
mixup: Beyond empirical risk minimiza- tion
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimiza- tion. arXiv preprint arXiv:1710.09412, 2017. 2 3
2017 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.