{"id":"32720aa9-5afc-460d-85fb-b349d41c6eb4","arxiv_id":"1908.06337","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Eigenrank ranks scans by the largest eigenvalue of a Dice-similarity matrix between committee models, and this ranking improves training-data selection and predicts segmentation failure on spine MRI.","lead":"This paper introduces Eigenrank, a method that selects which medical scans get expert labeling by measuring how much a committee of segmentation models disagrees on each scan. The selected scans train more consistent U-Net models and also flag scans where the model is likely to fail.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The committee-disagreement proxy is plausible but under-tested: a single random baseline for subset selection and no random-elimination control for failure prediction leave both main claims unverified.","rationale":"The reader's conditional verdict is appropriate. The algorithm is clearly described and the positive-semidefiniteness argument for the Dice matrix is sound; the selection direction (smallest λmax corresponds to least pairwise agreement) is coherent, and the information-theoretic framing, while not fully normalized, is not the load-bearing element. What is load-bearing is the empirical claim that this disagreement score predicts both training value and failure. That claim currently rests on two small datasets, one random baseline in the DSS experiment, and no control in the failure-prediction experiment. The paper's own sections acknowledge that the data are limited and that the conjecture connecting λmax to Von Neumann information requires future work, but the core issue is experimental, not mathematical. The authors should provide repeated random-selection runs, confidence intervals, a random-elimination control, and at least one standard uncertainty baseline for failure prediction. Until then, the central claim is plausible but not firmly established; the reader's conditional verdict should stand unchanged.","tokens_in":12766,"tokens_out":7778,"duration_ms":88327,"concrete_test":"A decisive check would be a controlled rerun on both spine datasets: (1) repeat the DSS comparison with at least 50 random selections and random committee initializations, reporting mean and 95% CI of Dice mean and standard deviation on a fixed held-out test set spanning the full difficulty range; (2) in the failure-prediction experiment, add a random-elimination control at each iteration and compare Eigenrank's ranking against softmax-entropy and test-time-dropout uncertainty using area under the ROC curve for detecting scans below a clinically defined Dice threshold; (3) specify and, if necessary, retrain the committee so that its disagreement is measured for models of the same architecture and training distribution as Dm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the committee disagreement score λmax(D_j), the largest eigenvalue of the pairwise-Dice matrix, is a reliable proxy for both training informativeness and failure of a U-Net. The paper's support for this proxy is not yet adequate. For data subset selection (Sections II.E, III.A.1, III.B), the comparison is against one randomly chosen subset of the same size; with a single run there are no confidence intervals, so the consistently lower standard deviations reported in Tables I-II and IV-V could reflect the particular random draw rather than a property of Eigenrank. For failure prediction (Section III.A.2, Tables III and VI), the experiment removes cases that Eigenrank scores as difficult and reports that their average Dice is low. But any method that removes low-Dice cases will produce exactly this pattern; without a random-elimination control, or a comparison to uncertainty-based baselines such as softmax entropy, test-time dropout, or ensemble variance, one cannot tell whether Eigenrank selects failures better than chance. The text also does not state whether the committee models used to score the validation scans are the same as Dm, the model whose failure is being predicted, or are independent of it; if the committee is trained on different data, the relation between committee disagreement and Dm failure is an unstated assumption rather than a demonstrated result. The underlying idea is coherent, but the load-bearing proxy is not yet pinned down.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EigenRank, an iterative committee-based algorithm for two related problems in medical image segmentation: (a) selecting a subset of scans for manual annotation so that a U-Net trained on that subset is more accurate and robust than a U-Net trained on a random subset of the same size, and (b) identifying scans on which a trained segmentation model is likely to fail. The selection score is the largest eigenvalue of a pairwise Dice matrix computed from segmentations produced by an ensemble of models; the algorithm iteratively adds the k scans with the smallest such scores. The authors prove that the Dice matrix is positive semidefinite via a Hadamard product of an inner-product matrix and a Cauchy matrix, state a conjecture (Eq. 11) connecting the largest eigenvalue with Von Neumann entropy, and report experiments on spinal canal segmentation (100 axial MRI scans) and intervertebral disk segmentation (103 sagittal MRI scans), comparing EigenRank-selected training sets with randomly selected sets and evaluating failure prediction by iteratively removing scans scored as difficult.","tokens_in":13041,"tokens_out":7023,"duration_ms":75349,"significance":"If the headline claims were fully supported, the method would be a practically valuable contribution to annotation-efficient medical image segmentation and to clinical silent-failure detection. The manuscript has clear strengths: the algorithm is precisely specified; the positive-semidefiniteness argument for the Dice matrix is explicit and sound; the Von Neumann connection is honestly labeled as a conjecture rather than overclaimed; and validation is performed on two real clinical MRI tasks. The central idea of using committee disagreement at the whole-scan level is coherent and worth pursuing. However, the empirical support as presented is not yet strong enough for the paper's two central claims: the random-selection comparison lacks statistical characterization, the failure-prediction experiment lacks a nontrivial control, and the theoretical framing contains an internal sign inconsistency. These are fixable with additional experiments and careful rewriting, but they are load-bearing at present.","major_comments":[{"comment":"The claim that Eigenrank yields more robust models than random selection rests on a single random draw. Tables I and II (spinal canals) and Tables IV and V (disks) report only one random-selection run, with no confidence intervals, repeated draws, or significance tests. With sample sizes of 100 and 103 scans and differences in standard deviation as small as 0.035 vs. 0.041, the observed advantage could plausibly reflect the particular random subset rather than a property of Eigenrank. The authors should repeat random selection many times (or otherwise characterize its distribution) and report confidence intervals or paired tests over repetitions. In addition, Table I is explicitly biased against random selection because the fixed validation set excludes Eigenrank-selected (complex) cases while retaining them for the random model; the authors acknowledge this, but the unbiased comparison in Table II is the one that needs statistical support.","section":"Section III.A.1, Tables I-II and IV-V"},{"comment":"The failure-prediction experiment is not controlled in a way that supports the claim that Eigenrank 'predicts' failure. The experiment removes scans that Eigenrank scores as difficult and shows that their average Dice is low; however, any procedure that removes low-Dice cases would produce exactly this pattern. Without a random-elimination control, or a comparison against standard uncertainty baselines (e.g., softmax entropy, test-time dropout, or ensemble variance), the results in Tables III and VI do not demonstrate that the committee-disagreement score is informative beyond chance. The paper should add such controls. It must also state whether the committee models used to compute the disagreement score are the same as the model Dm whose failure is being predicted, are trained on the same 15 scans, or are independent; as written, the relationship between committee disagreement on a scan and Dm's Dice on that scan is an unstated assumption.","section":"Section III.A.2, Tables III and VI"},{"comment":"There is an internal contradiction between the stated selection criterion and the claimed connection to Von Neumann entropy. In the iterative step, the algorithm collects L = {lambda_max(D_j)} and 'use[s] images corresponding to the k-smallest values in L' to construct the next subset. But for a Dice matrix, complete agreement gives D_j = J with lambda_max = t, while complete disagreement gives D_j = I with lambda_max = 1; hence lambda_max is a measure of agreement, not disagreement. Selecting the smallest lambda_max is consistent with selecting hard/disagreeing cases, but the text repeatedly calls lambda_max a 'measure of disagreement' and later says Eigenrank 'effectively looks for cases with the highest Von Neumann entropy.' If Eq. (8)-(11) are taken literally, minimizing lambda_max minimizes (approximately) lambda_1 log(lambda_1), i.e., minimizes the entropy measure, not maximizes it. This needs to be corrected: either the description of lambda_max as disagreement and the entropy interpretation must be revised, or the algorithm's selection direction must be reconciled with the stated goal. As written, the central theoretical motivation does not match the implemented procedure.","section":"Section II.E and Section IV.A"},{"comment":"The pseudocode uses the phrase 'images corresponding to the minimum k principal eigenvalues of the Dice matrix.' Since L stores the scalar lambda_max(j) for each scan, the intended rule is evidently the 'k smallest values of lambda_max,' not 'minimum k principal eigenvalues' of a single matrix. Please rephrase to avoid ambiguity, and ensure the direction of selection (smallest lambda_max) is explicitly reconciled with the text that describes lambda_max as a disagreement measure.","section":"Section II.E, algorithm display"}],"minor_comments":[{"comment":"There is a typo in the definition of the Cauchy matrix: C^{pq}_j is written as 1/(|s_p|+|s_p|), but it should be 1/(|s_p|+|s_q|).","section":"Section IV.A, Eq. (5)"},{"comment":"The text refers to 'table III' when describing the data subset selection results, but the corresponding caption is Table I; the table numbering should be checked throughout the manuscript.","section":"Section III.A.1, table cross-references"},{"comment":"In the definition of the Dice coefficient, the phrase 'anndSj' appears to be a typo for 'and S_j.'","section":"Section II.D"},{"comment":"The word 'manenr' appears to be a typo for 'manner.'","section":"Section IV.B"},{"comment":"The qualitative claim that Eigenrank selects cases that are 'much more complex' is supported by only three example scans; please either quantify the image-level properties (e.g., presence of hardware, scoliosis, contrast) or explicitly present these examples as illustrative rather than as evidence.","section":"Section III.A.1 and Figure 3"},{"comment":"The simulation supporting Conjecture (11) is described only briefly; please specify the number of simulated matrices, the range of epsilon, and how matrices are generated so that the result is reproducible.","section":"Section IV.A, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is on arXiv from 2019 and revised in 2021; the experiments are small and the empirical claims are stated more strongly in the abstract than the current evidence supports. No concerns about attribution or novelty disclosure came up; the authors cite their own prior work appropriately as related work. The main path to publication is to add statistical controls and to fix the sign inconsistency in the Von Neumann entropy narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the Eigenrank paper. The core idea is genuinely new as far as I can tell: instead of pixel-level uncertainty, they compare whole scans by building a t-by-t matrix of Dice scores between committee segmentations and ranking scans by its largest eigenvalue. The PSD proof for the Dice matrix, via the Hadamard product of a Gram matrix and a Cauchy matrix, is correct, and the algorithm is stated precisely enough to reimplement. They also deserve credit for presenting the Von Neumann connection as an explicit unproven conjecture (Eq. 11) rather than overclaiming it. What the experiments show is honest but thin. On 100 spinal-canal and 103 disk MRI scans, Eigenrank-selected training subsets give mean Dice about equal to random selection and standard deviations lower by 0.01 to 0.09. That is a plausible robustness gain, but the numbers come from a single random baseline per iteration with no confidence intervals or significance tests. The first validation table uses a shared clean set and shows smaller gains; the second uses method-specific left-out sets, which is a more realistic clinical scenario but also stacks the deck because Eigenrank's selected hard cases are removed from its own validation. For failure prediction, eliminating low-agreement cases and showing the remaining average Dice rises is close to a tautology: any score that correlates with difficulty would do the same. Without a random-elimination control or a comparison to softmax entropy, test-time dropout, or ensemble variance, we cannot tell whether Eigenrank finds failures better than chance. There is also an unstated assumption in Section II.F: the committee models used to score scans are not shown to be independent of the model Dm whose failure is being predicted. If the committee overlaps with Dm, the failure-prediction claim is weaker. The paper does not ship code or data, so the results cannot be checked without contacting the authors. This is not fatal for an idea paper, but it limits confidence. The self-citations are to related prior work and are not used as evidence for the core claim, so I do not see a citation-pattern problem. Bottom line: the contribution is clearly described, the math is solid, and the authors are honest about the conjecture. The empirical support is not yet firm. I would send this to peer review, with a strong request for multi-seed runs, error bars, a random-elimination control, uncertainty baselines, and code or data. I would not cite it as established until those are in, but I would bring it to a reading group to discuss the method.","headline":"A clear and genuinely novel scan-level committee-disagreement method with sound linear algebra, but the empirical support is thinner than the claims; worth serious peer review with requests for baselines, error bars, and code.","tokens_in":698,"tokens_out":1394,"would_cite":false,"duration_ms":44780,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The largest eigenvalue of a Dice-similarity matrix between committee models ranks which medical scans to label and which will fail to segment.","keywords":["Eigenrank","data subset selection","failure prediction","medical image segmentation","query by committee","Dice coefficient","Von Neumann information","U-Net"],"falsifier":"Give Eigenrank a larger, multi-site spine MRI dataset and run many random baseline draws. If Eigenrank-selected training subsets do not beat random subsets in mean Dice on a common held-out set, or if the scans it ranks hardest do not show lower Dice for a newly trained U-Net, the central claim is false.","tokens_in":12561,"feed_emoji":"","tokens_out":4817,"duration_ms":47906,"temperature":0.7,"pith_summary":"The paper proposes Eigenrank, a committee-based method that picks which scans from an unlabeled database should be manually annotated so that a U-Net trained on them is more robust than a U-Net trained on the same number of randomly chosen scans. It also uses the same disagreement score to predict which scans a trained segmentation model will fail on, something most medical-image segmentation systems do not do. If correct, the method would let hospitals spend limited annotation effort on the most informative scans and would alert clinicians when the automated segmentation is likely to be wrong.","feed_headline":"One eigenvalue ranks MRI scans for labeling and failure risk","feed_subtitle":"Committee disagreement picks the scans that make U-Net segmentation more robust than random selection.","key_machinery":"The central object is the per-scan Dice matrix $D_j$, a $t \\times t$ matrix of pairwise Dice coefficients between segmentations produced by $t$ committee models on scan $j$. The largest eigenvalue $\\lambda_{\\max}^j$ measures how strongly the committee disagrees on that scan. The paper argues $D_j$ is positive semidefinite by writing it as a Hadamard product of an inner-product matrix and a Cauchy matrix, so its eigenvalues are legitimate Shannon-type information; the largest eigenvalue is used because for near-consensus committees it dominates the full Von Neumann entropy.","core_discovery":"Eigenrank builds an ensemble of segmentation models, each trained on a small random subset of scans. For every remaining scan, it forms a matrix whose entries are Dice scores comparing each pair of models' segmentations of that scan, then takes the largest eigenvalue as a measure of inter-model disagreement. Scans with the strongest disagreement are added to the training set; the same ranking flags scans where a separately trained model is most likely to fail.","pith_inferences":["A natural test not in the paper is to compare Eigenrank against pixel-level uncertainty active learning, to see whether scan-level disagreement captures information that local methods miss.","If the paper's conjecture that the largest eigenvalue dominates the Von Neumann entropy as the number of committee models grows is proven, the heuristic rank-ordering by this single number would be asymptotically equivalent to the full information measure.","One could close the loop in practice: scans flagged by Eigenrank as likely failures for the current model could be routed to a human, segmented, and added to the next training run, making the framework a self-improving clinical workflow."],"forward_implications":["A lab can choose a 21-scan training subset from 100 unlabeled spine MRIs and get a segmentation model with lower variance in Dice scores across a validation cohort than a same-sized random subset.","In the failure-prediction setting, removing scans ranked hardest by Eigenrank raises the mean and lowers the standard deviation of Dice for the remaining validation cases, so the ranking can be used to route scans to human review.","Because selection happens at the whole-scan level and does not require ground truth on the candidate scans, it can be run directly on a hospital image archive before any annotation effort is spent.","The same committee-disagreement construction transfers to other segmentation metrics such as the Jaccard index or surface distance, since the Dice matrix is just one possible inter-model comparison.","Models trained on Eigenrank-selected subsets are claimed to be both more accurate and more robust than randomly trained models, making the approach a candidate for clinical workflow integration."],"supporting_citations":[{"why":"Provides the query-by-committee active-learning paradigm that Eigenrank extends to scan-level segmentation.","marker":"[27]"},{"why":"Introduces the U-Net architecture used as the segmentation model throughout the experiments.","marker":"[23]"},{"why":"Supplies the residual U-Net variant that is the concrete network trained and validated in the paper.","marker":"[33]"},{"why":"Establishes the positive semidefinite property of similarity matrices, which underpins the eigenvalue-based disagreement measure.","marker":"[21]"},{"why":"Provides the Cauchy-matrix and Schur product results used to prove that the Dice matrix is positive semidefinite.","marker":"[3]"},{"why":"Motivates the extreme augmentation used to train committee models from the small selected subsets.","marker":"[12]"}],"fun_headline_variants":["EigenRank: committee picks MRI scans that boost U-Net","Eigenvalue of model disagreement finds scans needing labels","Committee ensemble ranks scans for labeling and failure risk","EigenRank: eigenvalues select scans and predict segmentation failure"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the amount of disagreement among committee models on a scan is a reliable proxy for both how much that scan would improve a U-Net and whether a different U-Net will fail on it.","fun_headline_variants_meta":{"raw":{"variants":["EigenRank: committee picks MRI scans that boost U-Net","Eigenvalue of model disagreement finds scans needing labels","Committee ensemble ranks scans for labeling and failure risk","EigenRank: eigenvalues select scans and predict segmentation failure"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000606,"raw_usage":{"total_tokens":2749,"prompt_tokens":792,"completion_tokens":1957,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":408,"tokens_out":1957,"duration_ms":15944,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:48:41.916296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give Eigenrank a larger, multi-site spine MRI dataset and run many random baseline draws. If Eigenrank-selected training subsets do not beat random subsets in mean Dice on a common held-out set, or if the scans it ranks hardest do not show lower Dice for a newly trained U-Net, the central claim is false.","supporting_citations":[{"cited_title":"Query by committee,","cited_arxiv_id":null,"evidence_quote":"Provides the query-by-committee active-learning paradigm that Eigenrank extends to scan-level segmentation."},{"cited_title":"U-Net: Convolutional Networks for Biomedical Image Segmentation,","cited_arxiv_id":null,"evidence_quote":"Introduces the U-Net architecture used as the segmentation model throughout the experiments."},{"cited_title":"Road extraction by deep residual u-net,","cited_arxiv_id":null,"evidence_quote":"Supplies the residual U-Net variant that is the concrete network trained and validated in the paper."},{"cited_title":"On the positive semi-definite property of similarity matrices,","cited_arxiv_id":null,"evidence_quote":"Establishes the positive semidefinite property of similarity matrices, which underpins the eigenvalue-based disagreement measure."},{"cited_title":"Bhatia, Positive definite matrices","cited_arxiv_id":null,"evidence_quote":"Provides the Cauchy-matrix and Schur product results used to prove that the Dice matrix is positive semidefinite."},{"cited_title":"Extreme Augmentation : Can deep learning based medical image segmentation be trained using a single manually delineated scan?","cited_arxiv_id":"1810.01621","evidence_quote":"Motivates the extreme augmentation used to train committee models from the small selected subsets."}],"review_version":1}