{"id":"97a8f928-1d91-4696-834c-0e11e0d58875","arxiv_id":"2505.00308","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A Bayesian ordinal classifier with calibrated uncertainty thresholds is proposed for radiotherapy auto-contour QA, but the reported high accuracy is undermined by a missing majority-class baseline in a 93.1% Class 2 test set.","lead":"An AI quality-check model for radiotherapy contours uses Bayesian ordinal classification with uncertainty thresholds to label auto-generated contours as acceptable, needing major revision, or rejectable. The authors report over 90% accuracy with only 30 manually labeled training cases, but the test set is 93% acceptable contours and no trivial-baseline comparison is made.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No majority-class baseline: on a 93.1% Class-2 test set, the reported 92.9-93.6% accuracies are at or below the trivial always-Class-2 predictor, so the central >90% accuracy claim is not established.","rationale":"I read the reader's verdict as REJECT with high confidence. My independent pass converges on the same rejection, but via a slightly different route. The reader's formal weakest_assumption is the calibration-to-test distribution match (Section II.A.3a). That assumption is indeed untested, but it is not the most load-bearing issue: even if calibration and test distributions matched perfectly, the reported numerical results would still fail to establish the central claim because they are essentially equal to the trivial majority-class accuracy on a test set that is 93.1% Class 2. The paper actually contains the information needed to see this: Table A1 in the Appendix states the class distribution, and Table 6 reports 92.9% overall accuracy and 93.6% confident accuracy for the headline transfer-learning model. Without a majority baseline, per-class metrics, or a confusion matrix, these numbers are compatible with a model that almost always says Class 2 and rarely identifies the clinically important Class 0 and Class 1 outliers. This is a validation-gap concern rather than an internal inconsistency; the method and code are described in enough detail that the missing baseline can be computed. I therefore agree with the reader's verdict (REJECT) and would not change it. I also credit the paper for a fairly detailed methods section and for including the class distribution in the Appendix, which is what makes the concern checkable.","tokens_in":19126,"tokens_out":4393,"duration_ms":40982,"concrete_test":"Re-run the transfer-learning evaluation (Table 6 row '30(34)') on the 47-subject test set and report: (1) the full 3x3 confusion matrix of confident predictions; (2) per-class recall for Class 0 and Class 1; (3) the accuracy of an always-Class-2 predictor on the same confident subset; (4) a bootstrap 95% confidence interval for the difference between the model's confident accuracy and the always-Class-2 accuracy. If the interval includes zero, or if the baseline accuracy on the confident subset is greater than or equal to the reported 93.6%, the claim that the method achieves >90% accuracy beyond majority-class prediction is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section III.B, Table 6) is that fine-tuning the BOC model with 30 manual labels and calibrating with 34 subjects yields >90% test accuracy, with 93.6% accuracy on 98.2% of cases under the calibrated threshold. The Appendix (Table A1) reports that 93.1% of the 1,922 test slices are Class 2 by clinician majority vote. Therefore an 'always predict Class 2' rule already achieves 93.1% accuracy on the full test set and essentially the same accuracy on any large confident subset that is dominated by Class 2. The paper never reports this majority-class baseline, nor per-class recall for Class 0 and Class 1, nor a confusion matrix for the transfer-learning model, so it cannot be determined whether the uncertainty threshold is selecting genuinely reliable predictions or merely exploiting the class prior. In fact, the uncalibrated transfer-learning overall accuracy (92.9%) is below the majority baseline (93.1%). Because the calibration threshold is fit on calibration data to hit a 90% accuracy line, selecting cases with low predicted uncertainty can trivially meet that line if those cases are mostly Class 2. As a result, the central quantitative claim does not demonstrate predictive skill beyond a trivial classifier; the claimed clinical workload reduction is not supported by the reported metric.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a Bayesian Ordinal Classification (BOC) model with Monte Carlo dropout uncertainty quantification to assess auto-segmented rectum contour quality in radiotherapy. The method is trained under three label-availability scenarios: geometric surrogate labels, transfer learning with limited manual labels, and fully supervised training. A calibration step selects an uncertainty threshold to meet a target accuracy (e.g., 90%), and only low-uncertainty predictions are accepted. The central reported result is that fine-tuning with 30 manually labeled subjects and calibrating on 34 subjects yields 92.9% overall accuracy and 93.6% accuracy on 98.2% of test cases with uncertainty below the calibrated threshold.","tokens_in":19418,"tokens_out":4260,"duration_ms":43636,"significance":"If the central claim were established, the paper would make a useful practical contribution: an uncertainty-gated QA model that avoids ground-truth contours and reduces manual review burden, with a formal variance decomposition for ordinal classification. The mathematical derivation in Section II.A.2 is sound, and the use of a separate calibration set and a held-out test set is methodologically appropriate. However, the reported test accuracies are not compared with the trivial majority-class baseline on a heavily imbalanced test set, and the central accuracy claim is therefore not supported by the evidence as presented.","major_comments":[{"comment":"The test set contains 93.1% Class 2 slices by clinician majority vote, so an always-Class-2 classifier achieves 93.1% accuracy. The transfer-learning model with 30 subjects achieves 92.9% overall accuracy, below that baseline, and the calibrated confident-subset accuracy of 93.6% on 98.2% of cases is only 0.5 percentage points above it. The manuscript reports neither a majority-class baseline nor a confusion matrix or per-class recall for the transfer-learning model, so the claimed >90% accuracy and the workload-reduction claim are not established. The reader cannot tell whether the uncertainty threshold selects genuinely reliable predictions or merely exploits the class prior.","section":"Section III.B, Table 6 and Appendix Table A1"},{"comment":"The calibration procedure selects a threshold to achieve a target accuracy on calibration data and then reports accuracy on the test subset below that threshold. With 98.2% of test cases below the threshold, the confident subset is nearly the full test set, and its 93.6% accuracy is again indistinguishable from the 93.1% majority baseline. A valid demonstration requires reporting the confusion matrix on the confident subset, per-class recall and precision, and the accuracy of an always-Class-2 predictor restricted to the same subset. The stated assumption that the calibration and test distributions match is not verified, and the calibration set's class distribution is not reported.","section":"Section II.A.3a and Table 6"},{"comment":"The claim of a strong correlation between model uncertainty and manual-label uncertainty rests on a Spearman correlation of 1 computed from only three grouped points (full agreement, partial agreement, complete disagreement). With three groups, this only shows that the group means are monotonically ordered; it does not demonstrate a statistically meaningful correlation across individual cases. The authors should report the per-group distributions or a case-level correlation, and state the sample size in each agreement group.","section":"Section II.A.2b and Fig. 4"}],"minor_comments":[{"comment":"The phrase 'achieved over an accuracy of over 90%' is redundant and grammatically awkward; please revise.","section":"Abstract"},{"comment":"Table 5 is difficult to parse: the 'Uncertainty Level' column entries are not self-explanatory, and the rows for the 90% threshold do not clearly indicate which threshold applies. Add explicit column headers and complete cell entries.","section":"Table 5"},{"comment":"The table defines overall accuracy and confident-subset accuracy, but it would be much more informative to include per-class recall or a confusion matrix for the transfer-learning and from-scratch models, along with the calibration set's class distribution.","section":"Table 6"},{"comment":"Reference [13] is incomplete, as it lacks venue and year information, and the caption of Fig. A1 panel e says 'Class 0' while the text says clinicians assessed that case as 'Class 1'; please reconcile these details.","section":"References and Fig. A1"}],"recommendation":"reject","confidential_remarks":"The paper's formal UQ contribution is credible, but the empirical evaluation as written cannot support the central claim because the reported accuracies are at or below the majority-class baseline. The missing baseline and confusion matrices are not cosmetic; they change the conclusion. If the authors were to reframe the contribution around per-class sensitivity or around a decision-theoretic comparison against clinician-only review, a resubmission might be viable, but the current manuscript does not establish clinical benefit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the BOC-with-MC-dropout package is a sensible integration and the variance derivation for ordinal labels looks correct. But the paper's central claim—that calibrated uncertainty gives >90% accuracy and reduces manual reviews—is not established, because the test set is 93.1% Class 2 and the model's overall accuracy (92.9%) is below the trivial always-Class-2 baseline. The paper reports this imbalance itself in Table A1 but never runs the baseline.\n\nWhat's genuinely new: first application, as far as I know, of Bayesian ordinal classification (CORN-style) with MC-dropout uncertainty to auto-contour QA. The derivation of predictive variance for the ordinal sum is a useful piece, though it's a straightforward law-of-total-variance application. The three data scenarios (surrogate labels, transfer learning with 30 labels, full supervision) are well designed, and the finding that 20–30 manual subjects suffice is interesting if it holds.\n\nSoft spots, in order of severity:\n1. No majority-class baseline. On a test set with 93.1% Class 2, an always-Class-2 predictor gets 93.1%. The transfer model's 92.9% overall is below that; the calibrated 93.6% on 98.2% of cases is only marginally above the full-set baseline and not compared to the baseline restricted to those cases. No confusion matrix or per-class recall is given, so we cannot tell whether the model is detecting any Class 0/1 at all. This alone undermines the abstract's 'over 90% accuracy' framing.\n2. The calibration distribution-shift assumption is explicit but untested. The threshold is set on calibration data; if the deployment distribution shifts even a little, the 90% guarantee disappears. A simple robustness check on a held-out fold would help.\n3. The Spearman correlation of 1 between manual-label entropy and model uncertainty is computed over three groups. With three points, that's not evidence of a strong monotonic relationship; it's nearly inevitable if the means are ordered.\n\nMinor: single organ, single institution; no comparison to prior CNN-based QA or a no-UQ baseline. The cited literature looks appropriate.\n\nWho this is for: researchers working on auto-contour QA and adaptive radiotherapy workflows. It deserves a serious referee—the method is plausible and the problem is real—but the empirical section needs major revision before any claim of clinical workload reduction can be taken seriously. I'd send it to review and require the baselines and confusion matrix as a condition of acceptance.","headline":"Clever integration of ordinal classification and MC-dropout uncertainty for contour QA, but the headline accuracy claims are not distinguished from the trivial majority-class rule on a badly imbalanced test set.","tokens_in":19950,"tokens_out":2707,"would_cite":false,"duration_ms":26770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A calibrated Bayesian ordinal classifier screens radiotherapy auto-contours with over 90% accuracy using only 30 manual labels, needing no ground-truth contours at run time.","keywords":["auto-contour quality assessment","Bayesian ordinal classification","uncertainty quantification","online adaptive radiotherapy","deep learning","Monte Carlo dropout","surrogate labels","radiotherapy QA"],"falsifier":"Apply the exact calibrated threshold (e.g., $\\tau=0.33$ for the transfer-learning model) to an independent cohort in which acceptable contours are the minority rather than 93.1% of slices; if accuracy on accepted low-uncertainty cases drops below 90%, the distribution-transfer assumption is disproved. A reader can also compute the always-Class-2 accuracy on the current test set to see how much of the reported accuracy is explained by class imbalance alone.","tokens_in":18951,"feed_emoji":"🩻","tokens_out":9092,"duration_ms":94890,"temperature":0.7,"pith_summary":"This paper claims that auto-contour quality in radiotherapy can be assessed without ground-truth contours at run time, using a Bayesian ordinal classifier that outputs both a quality grade and an uncertainty for each 2D slice. The method is built to work when manual labels are absent, scarce, or plentiful: geometric surrogate labels cover the first case, transfer learning with as few as 30 labeled subjects covers the second, and direct supervision covers the third. A calibration step converts the uncertainty estimate into a threshold that meets a prespecified clinical accuracy target, and only predictions below that threshold are presented as confident. If the claim holds, clinicians in online adaptive radiotherapy could trust an automated screen for most auto-contours and spend manual review only on uncertain, rejected, or warning-triggering slices. The reported result is over 90% accuracy against manual labels after fine-tuning with 30 labeled subjects, with more than 98% of test slices accepted as confident and 93.6% of those confident calls correct.","feed_headline":"30 manual labels push contour QA past 90% accuracy","feed_subtitle":"Bayesian uncertainty threshold accepts only confident contour calls, so clinicians review fewer slices in adaptive radiotherapy.","key_machinery":"The load-bearing object is the Bayesian ordinal classifier, a ResNet-34 with dropout after each activation layer trained with the CORN conditional-probability loss; dropout turns it into a variational Bayesian network whose predictive distribution is approximated by 20 Monte Carlo forward passes. Ordinal coding is $y^*=y_1^*+y_2^*$, where the two binary decisions are 'at least Class 1' and 'at least Class 2', so the model respects that a rejected contour is worse than one needing major revision. Uncertainty is not the softmax entropy of a single pass but the total variance $\\mathrm{Var}(y^*)$ decomposed through the law of total variance into within-weight and across-weight components, estimated from the Monte Carlo draws. The calibration step converts that variance into a threshold $\\tau$ by sorting calibration slices by uncertainty and reading off the uncertainty level at which cumulative accuracy crosses the target; only slices with uncertainty below $\\tau$ are reported as confident. Geometric surrogate labels, set by DSC, SDSC, and HD95 with the final class taken as the worst of the three metric ratings, provide the training signal when manual labels do not exist.","core_discovery":"On the paper's own terms, the discovery is that an ordinal three-class Bayesian classifier, trained on CT slices overlaid with auto-contours, can be calibrated so that its confident predictions meet a prespecified clinical accuracy. Quality is coded as $y^*=y_1^*+y_2^*$, with $y_1^*=1$ meaning at least 'major revision' and $y_2^*=1$ meaning at least 'no/minor revision'; each binary event is modeled conditionally, and Monte Carlo dropout approximates the posterior over network weights. The variance of $y^*$ is used as predictive uncertainty, and a calibration curve of accuracy versus uncertainty yields a threshold for accepting predictions. In the limited-label scenario, pretraining on geometric surrogate labels (DSC, SDSC, HD95) and fine-tuning with 30 manually labeled subjects gives 92.9% overall accuracy, and applying the threshold to meet the 90% target gives 93.6% accuracy over 98.2% of the test slices. The same uncertainty measure tracks clinician disagreement, so low-uncertainty predictions are exactly the slices where the automated assessment can replace manual review.","pith_inferences":["Not developed in the paper: the same surrogate-label thresholds would have to be re-derived per organ, so the portability claim is about the BOC-plus-calibration machinery, not about the rectum-specific numbers.","An untested consequence: on a test set that is 93.1% Class 2, a trivial always-'acceptable' predictor already achieves 93.1% overall accuracy; the clinically decisive quantities are the recall and precision on Class 0 and Class 1 slices among confident predictions, which the paper does not isolate.","A likely operational extension: because uncertainty comes from 20 forward passes on 2D slices, the QA screen can run on the same machine as the auto-segmentation model and add only seconds per patient, making it usable inside an online adaptive session."],"forward_implications":["With zero manual labels, geometric surrogate labels plus data augmentation give a usable QA screen: 92.8% manual-label accuracy on 60.5% of test slices, enough to pre-screen contours where annotation is impossible.","With 20 to 30 manually labeled subjects used as fine-tuning, the model becomes confident on 96.5% to 98.7% of test slices and holds above 90% accuracy on those accepted predictions.","The same uncertainty threshold can be set to meet 80%, 85%, or 90% clinical accuracy targets, trading how many cases are auto-screened against how reliable the accepted calls are.","Model uncertainty tracks clinician disagreement: slices on which the three clinicians disagreed receive higher model uncertainty, so low-uncertainty predictions are the cases where a clinician can trust the assessment most.","Triggering a warning only when the model confidently disagrees with the clinician's 'no revision' judgment gives a decision-support loop that does not require ground-truth contours in deployment."],"supporting_citations":[{"why":"Supplies the proof that a network trained with dropout is a Bayesian neural network, justifying MC sampling of the weights.","marker":"[22]"},{"why":"Provides the rank-consistent conditional-probability ordinal loss (CORN) used to train the two binary classifiers.","marker":"[23]"},{"why":"Defines MC dropout inference, which yields the 20 sampled weight sets used for predictions and uncertainty estimates.","marker":"[31]"},{"why":"Supports the choice of T=20 Monte Carlo passes as a practical balance between cost and uncertainty precision.","marker":"[15]"},{"why":"Motivates threshold-based surrogate labeling of auto-contour quality and notes size-dependence issues with DSC.","marker":"[8]"},{"why":"Gives evidence that geometric metrics correlate with clinician quality rankings and that multiple metrics outperform a single metric.","marker":"[26]"},{"why":"Introduces SDSC, the surface metric used alongside DSC and HD95 in the surrogate labeling rule.","marker":"[29]"},{"why":"Provides the prior three-class CNN contour-QA baseline that this study extends with ordinal structure and uncertainty quantification.","marker":"[9]"}],"fun_headline_variants":["Bayesian QA: 30 labels, 93% accuracy","Calibrated uncertainty: 30 labels hit 93% contour QA","Contour QA: 30 manual labels, 93% accuracy, 98% confident","Uncertainty-aware AI: 30 labels, 93% contour QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration cohort and the future test cohort are assumed to produce the same distribution of contour-quality uncertainties, so a threshold tuned on 34 subjects keeps its 90% accuracy promise when applied to new patients.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian QA: 30 labels, 93% accuracy","Calibrated uncertainty: 30 labels hit 93% contour QA","Contour QA: 30 manual labels, 93% accuracy, 98% confident","Uncertainty-aware AI: 30 labels, 93% contour QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":3121,"prompt_tokens":1025,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2014}},"tokens_in":641,"tokens_out":2096,"duration_ms":15098,"temperature":1.0,"reasoning_tokens":2014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:45:16.056622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the exact calibrated threshold (e.g., $\\tau=0.33$ for the transfer-learning model) to an independent cohort in which acceptable contours are the minority rather than 93.1% of slices; if accuracy on accepted low-uncertainty cases drops below 90%, the distribution-transfer assumption is disproved. A reader can also compute the always-Class-2 accuracy on the current test set to see how much of the reported accuracy is explained by class imbalance alone.","supporting_citations":[{"cited_title":"Uncertainty Assessment for Deep Learning Radiotherapy Applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the proof that a network trained with dropout is a Bayesian neural network, justifying MC sampling of the weights."},{"cited_title":"Towards reliable head and neck cancers locoregional recurrence prediction using delta-radiomics and learning with rejection option,","cited_arxiv_id":null,"evidence_quote":"Provides the rank-consistent conditional-probability ordinal loss (CORN) used to train the two binary classifiers."},{"cited_title":"Uncertainty in Multitask Learning: Joint Representations for Probabilistic MR -only Radiotherapy Planning,","cited_arxiv_id":null,"evidence_quote":"Defines MC dropout inference, which yields the 20 sampled weight sets used for predictions and uncertainty estimates."},{"cited_title":"The model was trained using the Adam optimizer with default hyperparameters ( β1 = 0.9, and β2 = 0.999) over 1 × 105 iterations, leveraging the dice loss function","cited_arxiv_id":null,"evidence_quote":"Motivates threshold-based surrogate labeling of auto-contour quality and notes size-dependence issues with DSC."},{"cited_title":"Using Spatial Probability Maps to Hi ghlight Potential Inaccuracies in Deep Learning -Based Contours: Facilitating Online Adaptive Radiation Therapy - ClinicalKey","cited_arxiv_id":null,"evidence_quote":"Introduces SDSC, the surface metric used alongside DSC and HD95 in the surrogate labeling rule."},{"cited_title":"As shown in Table 2, the ROC AUC values increased from 0.821 to 0.932 and from 0.778 to 0.808 for predicting Class 2 and Class 1 , respectively","cited_arxiv_id":null,"evidence_quote":"Provides the prior three-class CNN contour-QA baseline that this study extends with ordinal structure and uncertainty quantification."}],"review_version":1}