REVIEW 4 major objections 6 minor 35 references
EigenRank by Committee: A Data Subset Selection and Failure Prediction paradigm for Robust Deep Learning based Medical Image Segmentation
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The largest eigenvalue of a Dice-similarity matrix between committee models ranks which medical scans to label and which will fail to segment.
desk verdict A clear and genuinely novel scan-level committee-disagreement method with sound linear algebra, but the empirical support is thinner than the claims; worth serious peer review with requests for baselines, error bars, and code. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the per-scan Dice matrix $D_j$, a $t \times t$ matrix of pairwise Dice coefficients between segmentations produced by $t$ committee models on scan $j$. The largest eigenvalue $\lambda_{\max}^j$ measures how strongly the committee disagrees on that scan. The paper argues $D_j$ is positive semidefinite by writing it as a Hadamard product of an inner-product matrix and a Cauchy matrix, so its eigenvalues are legitimate Shannon-type information; the largest eigenvalue is used because for near-consensus committees it dominates the full Von Neumann entropy.
What would settle it
Give Eigenrank a larger, multi-site spine MRI dataset and run many random baseline draws. If Eigenrank-selected training subsets do not beat random subsets in mean Dice on a common held-out set, or if the scans it ranks hardest do not show lower Dice for a newly trained U-Net, the central claim is false.
Extended reading notes
Core claim
Eigenrank builds an ensemble of segmentation models, each trained on a small random subset of scans. For every remaining scan, it forms a matrix whose entries are Dice scores comparing each pair of models' segmentations of that scan, then takes the largest eigenvalue as a measure of inter-model disagreement. Scans with the strongest disagreement are added to the training set; the same ranking flags scans where a separately trained model is most likely to fail.
Load-bearing premise
Everything rests on the assumption that the amount of disagreement among committee models on a scan is a reliable proxy for both how much that scan would improve a U-Net and whether a different U-Net will fail on it.
Editorial extensions
If this is right
- A lab can choose a 21-scan training subset from 100 unlabeled spine MRIs and get a segmentation model with lower variance in Dice scores across a validation cohort than a same-sized random subset.
- In the failure-prediction setting, removing scans ranked hardest by Eigenrank raises the mean and lowers the standard deviation of Dice for the remaining validation cases, so the ranking can be used to route scans to human review.
- Because selection happens at the whole-scan level and does not require ground truth on the candidate scans, it can be run directly on a hospital image archive before any annotation effort is spent.
- The same committee-disagreement construction transfers to other segmentation metrics such as the Jaccard index or surface distance, since the Dice matrix is just one possible inter-model comparison.
- Models trained on Eigenrank-selected subsets are claimed to be both more accurate and more robust than randomly trained models, making the approach a candidate for clinical workflow integration.
Reading between the lines
- A natural test not in the paper is to compare Eigenrank against pixel-level uncertainty active learning, to see whether scan-level disagreement captures information that local methods miss.
- If the paper's conjecture that the largest eigenvalue dominates the Von Neumann entropy as the number of committee models grows is proven, the heuristic rank-ordering by this single number would be asymptotically equivalent to the full information measure.
- One could close the loop in practice: scans flagged by Eigenrank as likely failures for the current model could be routed to a human, segmented, and added to the next training run, making the framework a self-improving clinical workflow.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EigenRank, an iterative committee-based algorithm for two related problems in medical image segmentation: (a) selecting a subset of scans for manual annotation so that a U-Net trained on that subset is more accurate and robust than a U-Net trained on a random subset of the same size, and (b) identifying scans on which a trained segmentation model is likely to fail. The selection score is the largest eigenvalue of a pairwise Dice matrix computed from segmentations produced by an ensemble of models; the algorithm iteratively adds the k scans with the smallest such scores. The authors prove that the Dice matrix is positive semidefinite via a Hadamard product of an inner-product matrix and a Cauchy matrix, state a conjecture (Eq. 11) connecting the largest eigenvalue with Von Neumann entropy, and report experiments on spinal canal segmentation (100 axial MRI scans) and intervertebral disk segmentation (103 sagittal MRI scans), comparing EigenRank-selected training sets with randomly selected sets and evaluating failure prediction by iteratively removing scans scored as difficult.
Significance. If the headline claims were fully supported, the method would be a practically valuable contribution to annotation-efficient medical image segmentation and to clinical silent-failure detection. The manuscript has clear strengths: the algorithm is precisely specified; the positive-semidefiniteness argument for the Dice matrix is explicit and sound; the Von Neumann connection is honestly labeled as a conjecture rather than overclaimed; and validation is performed on two real clinical MRI tasks. The central idea of using committee disagreement at the whole-scan level is coherent and worth pursuing. However, the empirical support as presented is not yet strong enough for the paper's two central claims: the random-selection comparison lacks statistical characterization, the failure-prediction experiment lacks a nontrivial control, and the theoretical framing contains an internal sign inconsistency. These are fixable with additional experiments and careful rewriting, but they are load-bearing at present.
major comments (4)
- [Section III.A.1, Tables I-II and IV-V] The claim that Eigenrank yields more robust models than random selection rests on a single random draw. Tables I and II (spinal canals) and Tables IV and V (disks) report only one random-selection run, with no confidence intervals, repeated draws, or significance tests. With sample sizes of 100 and 103 scans and differences in standard deviation as small as 0.035 vs. 0.041, the observed advantage could plausibly reflect the particular random subset rather than a property of Eigenrank. The authors should repeat random selection many times (or otherwise characterize its distribution) and report confidence intervals or paired tests over repetitions. In addition, Table I is explicitly biased against random selection because the fixed validation set excludes Eigenrank-selected (complex) cases while retaining them for the random model; the authors acknowledge this, but the unbiased comparison in Table II is the one that needs statistical support.
- [Section III.A.2, Tables III and VI] The failure-prediction experiment is not controlled in a way that supports the claim that Eigenrank 'predicts' failure. The experiment removes scans that Eigenrank scores as difficult and shows that their average Dice is low; however, any procedure that removes low-Dice cases would produce exactly this pattern. Without a random-elimination control, or a comparison against standard uncertainty baselines (e.g., softmax entropy, test-time dropout, or ensemble variance), the results in Tables III and VI do not demonstrate that the committee-disagreement score is informative beyond chance. The paper should add such controls. It must also state whether the committee models used to compute the disagreement score are the same as the model Dm whose failure is being predicted, are trained on the same 15 scans, or are independent; as written, the relationship between committee disagreement on a scan and Dm's Dice on that scan is an unstated assumption.
- [Section II.E and Section IV.A] There is an internal contradiction between the stated selection criterion and the claimed connection to Von Neumann entropy. In the iterative step, the algorithm collects L = {lambda_max(D_j)} and 'use[s] images corresponding to the k-smallest values in L' to construct the next subset. But for a Dice matrix, complete agreement gives D_j = J with lambda_max = t, while complete disagreement gives D_j = I with lambda_max = 1; hence lambda_max is a measure of agreement, not disagreement. Selecting the smallest lambda_max is consistent with selecting hard/disagreeing cases, but the text repeatedly calls lambda_max a 'measure of disagreement' and later says Eigenrank 'effectively looks for cases with the highest Von Neumann entropy.' If Eq. (8)-(11) are taken literally, minimizing lambda_max minimizes (approximately) lambda_1 log(lambda_1), i.e., minimizes the entropy measure, not maximizes it. This needs to be corrected: either the description of lambda_max as disagreement and the entropy interpretation must be revised, or the algorithm's selection direction must be reconciled with the stated goal. As written, the central theoretical motivation does not match the implemented procedure.
- [Section II.E, algorithm display] The pseudocode uses the phrase 'images corresponding to the minimum k principal eigenvalues of the Dice matrix.' Since L stores the scalar lambda_max(j) for each scan, the intended rule is evidently the 'k smallest values of lambda_max,' not 'minimum k principal eigenvalues' of a single matrix. Please rephrase to avoid ambiguity, and ensure the direction of selection (smallest lambda_max) is explicitly reconciled with the text that describes lambda_max as a disagreement measure.
minor comments (6)
- [Section IV.A, Eq. (5)] There is a typo in the definition of the Cauchy matrix: C^{pq}_j is written as 1/(|s_p|+|s_p|), but it should be 1/(|s_p|+|s_q|).
- [Section III.A.1, table cross-references] The text refers to 'table III' when describing the data subset selection results, but the corresponding caption is Table I; the table numbering should be checked throughout the manuscript.
- [Section II.D] In the definition of the Dice coefficient, the phrase 'anndSj' appears to be a typo for 'and S_j.'
- [Section IV.B] The word 'manenr' appears to be a typo for 'manner.'
- [Section III.A.1 and Figure 3] The qualitative claim that Eigenrank selects cases that are 'much more complex' is supported by only three example scans; please either quantify the image-level properties (e.g., presence of hardware, scoliosis, contrast) or explicitly present these examples as illustrative rather than as evidence.
- [Section IV.A, Figure 6] The simulation supporting Conjecture (11) is described only briefly; please specify the number of simulated matrices, the range of epsilon, and how matrices are generated so that the result is reproducible.
Circularity Check
No circularity found: the Eigenrank score is computed from committee segmentations without ground truth and is validated empirically; limited experimental controls are a statistical concern, not a circular one.
full rationale
The central derivation chain is self-contained. Eigenrank's ranking quantity λmax(Dj) is defined directly from pairwise Dice scores between segmentations produced by an ensemble of models on each scan (Section II.E), with no ground-truth Dice and no fitted parameter entering the score. The DSS claim is an empirical comparison: subsets chosen by low λmax are used to train a new residual U-Net, whose mean and standard-deviation Dice on held-out validation scans are compared with a randomly chosen subset; the ranking is not constructed from those validation outcomes. The failure-prediction claim is likewise empirical: Dm is trained on 15 scans, Eigenrank (committee disagreement) removes validation scans, and the removed scans' ground-truth Dice are then measured and found low. Since the selection criterion does not use those Dice values, the result is not forced by construction. The Von Neumann information discussion is explicitly labeled a conjecture (Eq. 11) and is not needed to define the algorithm; the positive-semidefinite proof for Dj rests on external results (Nader; Bhatia), not on the paper's own claims. The only author-overlapping citations ([12], [13]) support non-central choices such as the 20x augmentation factor and prior task descriptions, so they are not load-bearing. The lack of multiple random baselines and of a random-elimination control weakens the empirical support, but that is a validity/correctness concern, not circularity.
Assumptions & free parameters
free parameters (3)
- Subset size k per iteration =
3
- Number of Eigenrank iterations =
7 (21 training scans)
- Augmentation multiplicity for selection models =
20 patches per input patch
assumptions (3)
- standard math Schur product theorem and positive semidefiniteness of Cauchy matrices
- domain assumption Scan-level inter-model disagreement measured by the largest eigenvalue of the Dice matrix is a valid proxy for training informativeness and for model failure
- ad hoc to paper Conjecture that λ1 log λ1 dominates the Von Neumann entropy of the Dice matrix as t grows
Cite this review
Pith. "Pith review of EigenRank by Committee: A Data Subset Selection and Failure Prediction paradigm for Robust Deep Learning based Medical Image Segmentation." pith.science (2026). https://pith.science/paper/JRNPVZYD
@misc{pith2026190806337,
author = {Pith},
title = {Pith review of: EigenRank by Committee: A Data Subset Selection and Failure Prediction paradigm for Robust Deep Learning based Medical Image Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/JRNPVZYD}},
note = {Machine review of arXiv:1908.06337}
}
read the original abstract
Translation of fully automated deep learning based medical image segmentation technologies to clinical workflows face two main algorithmic challenges. The first, is the collection and archival of large quantities of manually annotated ground truth data for both training and validation. The second is the relative inability of the majority of deep learning based segmentation techniques to alert physicians to a likely segmentation failure. Here we propose a novel algorithm, named `Eigenrank' which addresses both of these challenges. Eigenrank can select for manual labeling, a subset of medical images from a large database, such that a U-Net trained on this subset is superior to one trained on a randomly selected subset of the same size. Eigenrank can also be used to pick out, cases in a large database, where deep learning segmentation will fail. We present our algorithm, followed by results and a discussion of how Eigenrank exploits the Von Neumann information to perform both data subset selection and failure prediction for medical image segmentation using deep learning.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Tensorflow: a system for large-scale machine learning
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al. , “Tensorflow: a system for large-scale machine learning.” in OSDI, vol. 16, 2016, pp. 265–283
2016
-
[2]
Y . Bhalgat, M. Shah, and S. Awate, “Annotation-cost minimization for medical image segmentation using suggestive mixed supervision fully convolutional networks,” arXiv preprint arXiv:1812.11302 , 2018
work page Pith review arXiv 2018
-
[3]
Bhatia, Positive definite matrices
R. Bhatia, Positive definite matrices. Princeton university press, 2009, vol. 24
work page 2009
-
[4]
S. Buus, S. Lauemøller, P. Worning, C. Kesmir, T. Frimurer, S. Cor- bet, A. Fomsgaard, J. Hilden, A. Holm, and S. Brunak, “Sensitive quantitative predictions of peptide-mhc binding by a ‘query by com- mittee’artificial neural network approach,” Tissue antigens , vol. 62, no. 5, pp. 378–384, 2003
work page 2003
-
[5]
Chollet et al., “Keras,” 2015
F. Chollet et al., “Keras,” 2015
2015
-
[6]
Incorpo- rating expert feedback into active anomaly discovery,
S. Das, W.-K. Wong, T. Dietterich, A. Fern, and A. Emmott, “Incorpo- rating expert feedback into active anomaly discovery,” in 2016 IEEE 16th International Conference on Data Mining (ICDM) . IEEE, 2016, pp. 853–858
work page 2016
-
[7]
A Strategy of MR Brain Tissue Images' Suggestive Annotation Based on Modified U-Net
Y . Deng, Y . Sun, Y . Zhu, M. Zhu, W. Han, and K. Yuan, “A strategy of mr brain tissue images’ suggestive annotation based on modified u-net,” arXiv preprint arXiv:1807.07510 , 2018
work page Pith review arXiv 2018
-
[8]
Leveraging uncertainty estimates for predicting segmentation quality,
T. DeVries and G. W. Taylor, “Leveraging uncertainty estimates for predicting segmentation quality,” arXiv preprint arXiv:1807.00502 , 2018
arXiv 2018
Show all 35 references
-
[9]
Deep active learning for axon-myelin segmentation on histology data,
M. L. di Scandalea, C. S. Perone, M. Boudreau, and J. Cohen-Adad, “Deep active learning for axon-myelin segmentation on histology data,” arXiv preprint arXiv:1907.05143 , 2019
1907 arXiv
-
[10]
Deep cnn ensembles and suggestive annotations for infant brain mri segmentation,
J. Dolz, C. Desrosiers, L. Wang, J. Yuan, D. Shen, and I. B. Ayed, “Deep cnn ensembles and suggestive annotations for infant brain mri segmentation,” arXiv preprint arXiv:1712.05319 , 2017
2017 arXiv
-
[11]
Selective sampling using the query by committee algorithm,
Y . Freund, H. S. Seung, E. Shamir, and N. Tishby, “Selective sampling using the query by committee algorithm,” Machine learning, vol. 28, no. 2-3, pp. 133–168, 1997
1997
-
[12]
Extreme aug- mentation: Can deep learning based medical image segmentation be trained using a single manually delineated scan?
B. Gaonkar, A. Bui, M. Brown, and L. Macyszyn, “Extreme aug- mentation: Can deep learning based medical image segmentation be trained using a single manually delineated scan?” arXiv preprint arXiv:1810.01621, 2018
2018 arXiv
-
[13]
Deep learning in the small sample size setting: cascaded feed forward neural net- works for medical image segmentation,
B. Gaonkar, D. Hovda, N. Martin, and L. Macyszyn, “Deep learning in the small sample size setting: cascaded feed forward neural net- works for medical image segmentation,” in Medical Imaging 2016: Computer-Aided Diagnosis , vol. 9785. International Society for Optics and Photo...
2016
-
[14]
Query by committee made real,
R. Gilad-Bachrach, A. Navot, and N. Tishby, “Query by committee made real,” in Advances in neural information processing systems , 2006, pp. 443–450
2006
-
[15]
Explaining and harness- ing adversarial examples,
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harness- ing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
2014 arXiv
-
[16]
Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique,
H. Greenspan, B. Van Ginneken, and R. M. Summers, “Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique,” IEEE Transactions on Medical Imaging , vol. 35, no. 5, pp. 1153–1159, 2016
2016
-
[17]
Assessing reliability and challenges of un- certainty estimations for medical image segmentation,
A. Jungo and M. Reyes, “Assessing reliability and challenges of un- certainty estimations for medical image segmentation,” arXiv preprint arXiv:1907.03338, 2019
1907 arXiv
-
[18]
Medical image file formats,
M. Larobina and L. Murino, “Medical image file formats,” pp. 200– 206, 2014
2014
-
[19]
The first step for neuroimaging data analysis: DICOM to NIfTI conversion,
X. Li, P. S. Morgan, J. Ashburner, J. Smith, and C. Rorden, “The first step for neuroimaging data analysis: DICOM to NIfTI conversion,” Journal of Neuroscience Methods , vol. 264, pp. 47–56, may 2016
2016
-
[20]
The design of simpleitk,
B. C. Lowekamp, D. T. Chen, L. Ib ´a˜nez, and D. Blezek, “The design of simpleitk,” Frontiers in neuroinformatics, vol. 7, p. 45, 2013
2013
-
[21]
On the positive semi-definite property of similarity matrices,
R. Nader, A. Bretto, B. Mourad, and H. Abbas, “On the positive semi-definite property of similarity matrices,” Theoretical Computer Science, vol. 755, pp. 13–28, 2019
2019
-
[22]
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,
A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 427–436
2015
-
[23]
U-Net: Convolutional Networks for Biomedical Image Segmentation,
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Medical Image Computing and Computer-Assisted Intervention {\textendash} MIC- CAI 2015. Cham: Springer, Cham, oct 2015, pp. 234–241
2015
-
[24]
Active learn- ing in recommender systems,
N. Rubens, M. Elahi, M. Sugiyama, and D. Kaplan, “Active learn- ing in recommender systems,” in Recommender systems handbook . Springer, 2015, pp. 809–846
2015
-
[25]
apricot: Submodu- lar selection for data summarization in python,
J. Schreiber, J. Bilmes, and W. S. Noble, “apricot: Submodu- lar selection for data summarization in python,” arXiv preprint arXiv:1906.03543, 2019
1906 arXiv
-
[26]
Active learning literature survey,
B. Settles, “Active learning literature survey,” University of Wisconsin- Madison Department of Computer Sciences, Tech. Rep., 2009
2009
-
[27]
Query by committee,
H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” in Proceedings of the fifth annual workshop on Computational learning theory. ACM, 1992, pp. 287–294
1992
-
[28]
Active deep learning with fisher information for patch-wise semantic segmentation,
J. Sourati, A. Gholipour, J. G. Dy, S. Kurugol, and S. K. Warfield, “Active deep learning with fisher information for patch-wise semantic segmentation,” in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support . Springer, 2018, pp. 83–91
2018
-
[29]
Current procedural terminology (CPT)
C. P. Terminology, “Current procedural terminology (CPT).” JAMA : the journal of the American Medical Association , vol. 212, pp. 873– 874, 1970
1970
-
[30]
Submodularity in data subset selection and active learning,
K. Wei, R. Iyer, and J. Bilmes, “Submodularity in data subset selection and active learning,” inInternational Conference on Machine Learning, 2015, pp. 1954–1963
2015
-
[31]
Suggestive annotation: A deep active learning framework for biomedical image segmentation,
L. Yang, Y . Zhang, J. Chen, S. Zhang, and D. Z. Chen, “Suggestive annotation: A deep active learning framework for biomedical image segmentation,” in International conference on medical image comput- ing and computer-assisted intervention. Springer, 2017, pp. 399–407
2017
-
[32]
User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliabil- ity,
P. A. Yushkevich, J. Piven, H. C. Hazlett, R. G. Smith, S. Ho, J. C. Gee, and G. Gerig, “User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliabil- ity,” Neuroimage, vol. 31, no. 3, pp. 1116–1128, jul 2006
2006
-
[33]
Road extraction by deep residual u-net,
Z. Zhang, Q. Liu, and Y . Wang, “Road extraction by deep residual u-net,” IEEE Geoscience and Remote Sensing Letters , vol. 15, no. 5, pp. 749–753, 2018
2018
-
[34]
Deep learning based instance segmentation in 3d biomedical images using weak annotation,
Z. Zhao, L. Yang, H. Zheng, I. H. Guldner, S. Zhang, and D. Z. Chen, “Deep learning based instance segmentation in 3d biomedical images using weak annotation,” inInternational Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2018, pp. 352–360
2018
-
[35]
A brief introduction to weakly supervised learning,
Z.-H. Zhou, “A brief introduction to weakly supervised learning,” National Science Review , vol. 5, no. 1, pp. 44–53, 2017
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.