{"id":"9a015fad-c652-4b79-a2c7-a0e60e6bb05a","arxiv_id":"2501.08963","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training-aware conformal risk control method triages IMRT plans for measurement QA, achieving 100% sensitivity and 76% specificity on pooled hospital data while reducing measurements by 75%.","lead":"This paper proposes a machine-learning method that predicts which cancer radiation therapy plans are safe enough to skip a time-consuming quality assurance measurement. The method uses conformal prediction with a risk-control penalty during training, and on the authors' data it flags no unsafe plan while cutting the measurement workload by about 75%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deployed interval averages per-minibatch risk-control widths and therefore does not inherit the Appendix G guarantee; the safety claim rests on an unproven heuristic.","rationale":"The reader's weakest assumption identifies exactly the gap between the per-minibatch conformal risk control guarantee and the final averaged interval. My independent reading of Sections 3.4, 3.5, and Appendix G confirms that the proof in Appendix G covers the λ chosen by Eq. (7) on a single calibration set, not an average of interval widths selected on different minibatches during training. The paper's own language ('a heuristic to average the interval', 'we may lose the coverage guarantee in theory') supports this reading. The feature-selection ambiguity (whether the t-test in Appendix C uses test labels) is also concerning, but the interval-guarantee gap is more directly tied to the central claim of risk-controlled safety and is acknowledged in the text. The empirical results are promising, and the paper is transparent about the heuristic nature of the final interval, so the appropriate verdict remains CONDITIONAL rather than rejection; the authors should either prove a guarantee for the averaged interval or explicitly relabel the method as an empirical heuristic, and ideally validate the feature-selection protocol. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":14317,"tokens_out":2364,"duration_ms":26489,"concrete_test":"Recompute Table 1's 'Ours' row with a guaranteed final calibration: after training with the proposed objective, hold out a fresh calibration split, compute nonconformity scores, select a single λ via Eq. (7), and form I = λ · err on that calibration set, then evaluate on the test split. Compare sensitivity, specificity, reduction in measurement, and interval width to the reported averaged-I results. If the empirically guaranteed version maintains the reported performance, the concern is mitigated; if specificity or reduction drops materially, the reported gains stem from the unguaranteed averaging heuristic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that training-aware conformal risk control provides a robust, risk-controlled triage of IMRT plans. In Section 3.5, the method sets the final interval to [ŷ_test − I, ŷ_test + I], where I is the average of one-sided interval widths generated during training by per-minibatch conformal risk control. The risk control guarantee in Appendix G (Theorem 1) applies to the λ selected by Eq. (7) on a fixed calibration set, with a loss that is non-increasing in λ and right-continuous. Averaging over minibatch-specific λ values yields an I that is not the output of any single calibration step; the monotonicity and exchangeability arguments in the proof do not transfer to this averaged quantity. The paper itself acknowledges this: Section 3.4 calls the averaging 'a heuristic,' and Section 3.5 states 'we may lose the coverage guarantee in theory.' Consequently, the headline property of bounded risk (e.g., sensitivity ≥ 1−α) is not formally established for the deployed interval. The empirical sensitivity of 1.0 in Table 1 is reassuring but is based on only 19 unsafe plans across the pooled datasets, so the safety claim is not statistically strong. This is the load-bearing weakness in the paper's argument: the method's name and abstract imply a conformal risk control guarantee that the final interval does not actually possess.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-aware conformal risk control method for triaging IMRT plans in quality assurance. The method combines conformal training (Stutz et al., 2021) with conformal risk control (Angelopoulos et al., 2022) by incorporating the clinical 95% gamma passing rate threshold and the associated sensitivity risk into the training objective. During training, a two-layer MLP is optimized with a penalty based on the lower conformal bound, and per-minibatch conformal risk control intervals are computed; the final deployed interval uses the average of these per-minibatch one-sided widths. The method is evaluated on two real-world IMRT datasets from Johns Hopkins Hospital, both pooled and under distribution shift, and compared with split conformal prediction, conformal quantile regression, conformal risk control, and conformal training. Reported metrics include sensitivity, specificity, reduction in measurement, coverage, and interval width.","tokens_in":14557,"tokens_out":5651,"duration_ms":53992,"significance":"If the risk-control claim were rigorously established, the paper would make a useful contribution to clinical machine learning: it addresses a real safety-critical workflow, incorporates a clinically meaningful threshold into the conformal framework, and reports an apparent large improvement in specificity and measurement reduction over baselines while maintaining sensitivity. The paper provides a public code link and evaluates on real clinical data, which are strengths. However, the central theoretical guarantee is not established for the final deployed interval, and the empirical safety evidence is based on a very small number of unsafe plans. The application is timely, but the methodological advance beyond combining existing conformal training and risk control is incremental.","major_comments":[{"comment":"The final deployed interval width I is the average of per-minibatch one-sided interval widths obtained during conformal training, not the output of the calibration step in Eq. (7). The risk control guarantee in Appendix G applies only to the λ selected by Eq. (7) on a fixed calibration set with a fixed model, and the proof relies on exchangeability of the calibration and test points. In the training-aware method, the model changes after each minibatch while the same validation set is reused for calibration, so the exchangeability condition is violated even for the per-minibatch intervals, and no argument transfers the guarantee to the averaged width. The paper acknowledges this (\"we may lose the coverage guarantee in theory\"), but the abstract and title still describe the method as risk control. The empirical sensitivity of 1.0 in Table 1 is based on only 19 unsafe plans in the pooled data and is not a substitute for a formal guarantee. Please either provide a valid risk bound for the averaged-width procedure or explicitly reframe the contribution as a heuristic with only empirical validation, and qualify the \"risk control\" terminology throughout.","section":"Section 3.5, Eq. (8), and Appendix G, Theorem 1"},{"comment":"The two-sample t-test used for feature selection is described without stating whether it is applied to the training split only or to the full pooled dataset. The text says \"in our data exploration\" and \"we find 12 features have a statistical difference between their distributions between 'safe' and 'unsafe',\" which suggests the test may use labels from the entire dataset, including the held-out test set. If test labels are used to select features, the out-of-sample sensitivity and specificity in Tables 1 and 2 are optimistically biased. Please clarify the exact data used for the t-test and, if necessary, move feature selection inside the training split and rerun the experiments.","section":"Section 4.4 (Feature Selection)"},{"comment":"The claim \"we are able to guarantee 100% sensitivity in the pooled data\" is an empirical observation, not a guarantee. The pooled dataset contains only 19 plans below 95% GPR (4 in Dataset 1 and 15 in Dataset 2), and after splitting into train, validation, and test, the test set contains a small and variable number of positive examples. A sensitivity of 1 out of a handful of positives has a wide confidence interval. Moreover, under distribution shift (Table 3, Dataset 1 as test), the proposed method's sensitivity drops to 0.86 in the prospective-threshold evaluation, so the safety property is not robust to shift. Please report confidence intervals for sensitivity (e.g., Clopper-Pearson) and state the small-sample limitation in the abstract and conclusion.","section":"Tables 1 and 3, Section 5"},{"comment":"The description of how the ensemble confidence interval is formed contains an apparent typo: \"the lower bound of the confidence interval would be the smallest value of the upper bound estimates made by each model in the ensemble.\" This should presumably read \"smallest value of the lower bound estimates.\" As written, the lower bound is an undefined quantity, and it is unclear how the ensemble intervals were actually computed, which affects the reproducibility of the reported interval widths.","section":"Section 4.2 (Experimental Setup)"}],"minor_comments":[{"comment":"The risk function uses a hard threshold at 95% GPR. In clinical practice, GPR measurements have noise, and plans near the threshold may be misclassified. Please discuss the sensitivity of the method to the exact threshold choice and to measurement noise.","section":"Section 3.3, Eq. (5)"},{"comment":"The penalty term is described as MSE + max(0, (lower conformal bound - actual value)), while the risk function penalizes only the case where the lower bound exceeds 95 and the true GPR is below 95. The relationship between the two is not explicitly stated. Clarify how the penalty is computed per minibatch and how it aligns with the risk function.","section":"Section 3.5"},{"comment":"The base model row reports no interval width or coverage. Consider reporting the standard deviation of the base model's predictions or the width of a naive interval to aid comparison with the conformal methods.","section":"Table 1"},{"comment":"The feature selection comparison in Table 6 uses a threshold of \"99.99\" that is not defined in the text. Please specify what this threshold refers to.","section":"Appendix C"},{"comment":"There are several typos and inconsistencies: \"exchangability\" should be \"exchangeability\" (Section 4.5), \"one-side\" should be \"one-sided\" (Sections 3.4 and 3.5), and \"descrptions\" should be \"descriptions\" (Appendix B). Please proofread.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is suitable in scope for ML4H, but the central risk-control guarantee is not valid for the deployed interval, and the authors themselves acknowledge this. The empirical results are promising but rest on very few positive examples. I recommend major revision, asking the authors to either prove a valid guarantee for the averaged-width procedure or clearly present the method as a heuristic with appropriate qualifications. I also recommend asking for clarification on the feature-selection data split, as this could affect the validity of all reported test metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful application paper: it combines conformal risk control with conformal training to triage IMRT QA plans, and on two single-site datasets it reports sensitivity 1.0, specificity 0.76, and a 75% reduction in measurement with an average interval width of 5.81 on pooled data. That is a meaningful practical result for a safety-critical, workload-heavy clinical process. Second, the headline 'risk control' guarantee does not actually apply to the deployed interval. The final interval uses I = average of per-minibatch one-sided widths from training-time conformal risk control, and the Appendix G theorem only covers the per-minibatch λ-selected intervals. The paper itself calls the averaging a heuristic and says 'we may lose the coverage guarantee in theory.' So the name overstates what is proven.\n\nWhat's new: the combination of CRC with conformal training, plus the risk-aligned penalty based on the 95% GPR threshold, is not in the cited literature. The authors also compare against standard CP, CQR, CRC, and conformal training, which is good diligence. The code is available, and the authors are honest about the small data and the heuristic averaging.\n\nSoft spots: First, the missing guarantee is the load-bearing issue. The empirical sensitivity of 1.0 is on only 19 unsafe plans in the pooled data (4 in dataset 1, 15 in dataset 2), so the safety claim is not statistically strong; a single missed unsafe plan would drop sensitivity to 0.95. Second, the feature selection protocol is ambiguous: the t-test for selecting features appears to be run before the train/validation/test split, using labels from the full pooled dataset. If so, that selection is not exchangeable and can leak test information into the model choice, though the effect is likely modest since only 12 features are selected and the model is then trained on train labels only. The paper should clarify the timing. Third, the comparison under distribution shift shows sensitivity drops to 0.86 (prospective threshold, D1 test), which is not reassuring, though that is clearly labeled.\n\nOverall: the paper is worth engaging. A serious referee should ask the authors to either provide a formal guarantee for the averaged interval (perhaps via a post-hoc calibration on a held-out set) or re-label the method as an empirical heuristic. The clinical idea is sound and the empirical results are promising, but the safety claim depends on either a proof or a larger unsafe-plan sample.\n\nRecommendation: send to peer review.","headline":"Useful clinical triage paper with a genuine novelty, but the 'risk control' label overstates the guarantee for the deployed interval; still worth refereeing.","tokens_in":15103,"tokens_out":2062,"would_cite":false,"duration_ms":19630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-aware conformal risk control achieves 100% sensitivity and 76% specificity on IMRT QA triage, cutting measurement by 75%.","keywords":["conformal prediction","conformal risk control","conformal training","IMRT quality assurance","gamma passing rate","uncertainty quantification","treatment plan triage","radiation therapy"],"falsifier":"Count, on a fresh exchangeable calibration set, the fraction of plans whose true gamma passing rate is below 95% but whose prediction interval lower bound is above 95%; if that empirical risk exceeds the chosen $\\alpha = 0.1$ by more than sampling error, the central safety claim fails. A stronger check compares this risk with the risk of an interval built by applying Eq. (7) directly to the same calibration set.","tokens_in":1841,"feed_emoji":"🎯","tokens_out":2400,"duration_ms":86834,"temperature":0.7,"pith_summary":"The paper asks whether conformal prediction, a distribution-free framework for producing statistically valid prediction intervals, can make measurement-based quality assurance (QA) for intensity modulated radiation therapy (IMRT) faster without weakening safety. Its answer is a training-aware conformal risk control method: the model is trained with a penalty that mirrors an asymmetric clinical risk function, so the resulting intervals are tight enough to mark many plans safe without measurement while keeping sensitivity at 100% on the pooled data. In experiments on two real IMRT plan datasets, the method triaged about 75% of plans as safe, at sensitivity 1.0 and specificity 0.76, with a smaller average interval width than the baselines it is compared with. The importance, if the claim holds, is that a safety-critical clinical workflow can be made more efficient with rigorous uncertainty quantification rather than with point predictions alone.","feed_headline":"IMRT QA can skip 75% of measurements while catching every unsafe plan","feed_subtitle":"Training-aware conformal risk control keeps sensitivity at 100% with intervals tight enough to triage most plans.","key_machinery":"The central object is the training-aware conformal risk control objective: a conformal training procedure whose penalty term is the clinical risk function of conformal risk control. Conformal risk control selects a width-scaling parameter $\\lambda$ to keep the expected loss below a user-set level $\\alpha$; conformal training backpropagates through the interval-construction step so the model learns to produce tighter intervals. The paper's mechanism combines these: in each minibatch it builds intervals $[\\hat{y} - \\lambda\\cdot \\mathrm{err}, \\hat{y} + \\lambda\\cdot \\mathrm{err}]$ using the CRC $\\lambda$-selection rule, with its one-sided risk function $\\ell(C_\\lambda(X_i),Y_i)=1$ if the lower bound is above 95 while $Y_i<95$ and 0 otherwise, and uses the lower-bound hinge penalty $\\mathrm{MSE} + \\max(0,\\text{lower bound} - \\text{true value})$ as the training loss. This is what lets the model both shrink intervals and keep unsafe plans from being cleared.","core_discovery":"The central discovery claimed is that conformal risk control can be folded into model training, not just applied post hoc, and that doing so yields prediction intervals that are both clinically safe and practically useful for IMRT QA triage. The paper defines a one-sided risk function that charges a loss exactly when a prediction interval's lower bound lies above the 95% gamma passing rate threshold while the true GPR is below it, and uses the average risk on validation data to select the interval-scaling parameter $\\lambda$ per training minibatch. During training the model minimizes mean squared error plus a penalty that pushes the lower bound below the threshold for unsafe plans; at deployment the interval is $[\\hat{y} - I, \\hat{y} + I]$ with $I$ the average one-sided calibration width from training, and a plan is triaged as safe only if its interval's lower bound exceeds the threshold. The paper reports sensitivity 1.0, specificity 0.76, and a 75% reduction in measurement with an average interval width of 5.81 on pooled data, and shows this beats the conformal prediction, conformal quantile regression, conformal risk control, and conformal training baselines on specificity and measurement reduction while matching their sensitivity.","pith_inferences":["The principle of training with the clinical risk function in the loop is not specific to gamma passing rates: any safety-critical regression task with an asymmetric cost structure, such as predicting a lab value where a false 'normal' is dangerous, could reuse the same training-aware risk control recipe.","Because the averaged interval width used at deployment is not produced by a fresh conformal calibration step, the formal risk bound proven for the per-minibatch $\\lambda$-selected intervals does not automatically transfer to the deployed interval; a hospital adopting the method would want to recompute the empirical risk on its own calibration data before trusting the 75% triage rate.","The pooled-data result comes from two temporally separated cohorts on one machine at one institution; testing on multi-institution, multi-machine data would clarify how much of the specificity gain survives genuine device and workflow variation.","A natural extension is to make the triage decision depend on disease site or treatment region, since the paper notes that the same gamma passing rate can carry different clinical meaning for different anatomical sites."],"forward_implications":["On pooled data the method achieves sensitivity 1.0, specificity 0.76, and a 75% reduction in measurement, with average interval width 5.81, beating all four conformal baselines on specificity and reduction.","Because sensitivity is 100% in the pooled experiments, no plan whose measured gamma passing rate is below the 95% threshold is triaged as safe, so the reported workload reduction does not come at the cost of passing an unsafe plan.","Incorporating the clinical decision threshold and risk function into training produces meaningfully tighter intervals than post hoc split conformal prediction (5.81 vs 12.29 on pooled data), showing the training objective matters for interval usefulness.","Under distribution shift between the two time-separated datasets, all conformal methods lose some performance, and the proposed method's prospective-threshold sensitivity drops to 0.86 in one direction, underscoring that the benefits rely on calibration and test data being exchangeable."],"supporting_citations":[{"why":"Supplies conformal risk control and its expected-risk guarantee, the framework the proposed method extends into the training process.","marker":"Angelopoulos et al. (2022)"},{"why":"Supplies conformal training, the training-with-interval procedure that the proposed method modifies with a risk-based penalty.","marker":"Stutz et al. (2021)"},{"why":"Establishes the AAPM TG-218 tolerance limits, the clinical basis for the 95% gamma passing rate threshold used as the triage decision boundary.","marker":"Miften et al. (2018)"},{"why":"Supplies the plan complexity features and the random-forest/XGBoost baselines whose low sensitivity motivates the conformal approach.","marker":"Lam et al. (2019)"},{"why":"Defines the split-conformal prediction baseline and the coverage-guarantee framework used for comparison.","marker":"Angelopoulos and Bates (2021)"},{"why":"Defines conformal quantile regression, a baseline the proposed method must outperform on specificity and interval width.","marker":"Romano et al. (2019)"},{"why":"Provides the CNN gamma-passing-rate predictor whose sensitivity and specificity the paper cites as motivation for needing uncertainty-aware triage.","marker":"Interian et al. (2018)"}],"fun_headline_variants":["Training-aware risk control triages 75% of IMRT QA with perfect safety","Conformal training skips 75% of IMRT checks, keeps 100% sensitivity","IMRT QA workload cut by 75% via training-aware conformal risk","New conformal method reduces IMRT QA measurements by 75% safely","One-sided risk control in training trims IMRT QA by 75%"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The safety claim rests on the assumption that the average interval width used at deployment inherits the risk control property proven for the per-minibatch calibrated intervals; the paper does not prove that averaging preserves the guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Training-aware risk control triages 75% of IMRT QA with perfect safety","Conformal training skips 75% of IMRT checks, keeps 100% sensitivity","IMRT QA workload cut by 75% via training-aware conformal risk","New conformal method reduces IMRT QA measurements by 75% safely","One-sided risk control in training trims IMRT QA by 75%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1348,"prompt_tokens":967,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":583,"tokens_out":381,"duration_ms":4086,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:13:43.642439+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count, on a fresh exchangeable calibration set, the fraction of plans whose true gamma passing rate is below 95% but whose prediction interval lower bound is above 95%; if that empirical risk exceeds the chosen $\\alpha = 0.1$ by more than sampling error, the central safety claim fails. A stronger check compares this risk with the risk of an interval built by applying Eq. (7) directly to the same calibration set.","supporting_citations":[],"review_version":1}