Pith. sign in

REVIEW 4 major objections 5 minor 25 references

Training-Aware Risk Control for Intensity Modulated Radiation Therapies Quality Assurance with Conformal Prediction

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training-aware conformal risk control achieves 100% sensitivity and 76% specificity on IMRT QA triage, cutting measurement by 75%.

desk verdict Useful clinical triage paper with a genuine novelty, but the 'risk control' label overstates the guarantee for the deployed interval; still worth refereeing. read the letter →

arxiv 2501.08963 v1 pith:YPA7JZK5 submitted 2025-01-15 cs.LG

Kevin He , David Adam , Sarah Han-Oh , Anqi Liu This is my paper · ORCID
classification cs.LG
keywords conformalpredictionriskcontroltrainingIMRTqualityassurancegammapassingrateuncertaintyquantificationtreatmentplantriageradiationtherapy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether conformal prediction, a distribution-free framework for producing statistically valid prediction intervals, can make measurement-based quality assurance (QA) for intensity modulated radiation therapy (IMRT) faster without weakening safety. Its answer is a training-aware conformal risk control method: the model is trained with a penalty that mirrors an asymmetric clinical risk function, so the resulting intervals are tight enough to mark many plans safe without measurement while keeping sensitivity at 100% on the pooled data. In experiments on two real IMRT plan datasets, the method triaged about 75% of plans as safe, at sensitivity 1.0 and specificity 0.76, with a smaller average interval width than the baselines it is compared with. The importance, if the claim holds, is that a safety-critical clinical workflow can be made more efficient with rigorous uncertainty quantification rather than with point predictions alone.

What carries the argument

The central object is the training-aware conformal risk control objective: a conformal training procedure whose penalty term is the clinical risk function of conformal risk control. Conformal risk control selects a width-scaling parameter $\lambda$ to keep the expected loss below a user-set level $\alpha$; conformal training backpropagates through the interval-construction step so the model learns to produce tighter intervals. The paper's mechanism combines these: in each minibatch it builds intervals $[\hat{y} - \lambda\cdot \mathrm{err}, \hat{y} + \lambda\cdot \mathrm{err}]$ using the CRC $\lambda$-selection rule, with its one-sided risk function $\ell(C_\lambda(X_i),Y_i)=1$ if the lower bound is above 95 while $Y_i<95$ and 0 otherwise, and uses the lower-bound hinge penalty $\mathrm{MSE} + \max(0,\text{lower bound} - \text{true value})$ as the training loss. This is what lets the model both shrink intervals and keep unsafe plans from being cleared.

What would settle it

Count, on a fresh exchangeable calibration set, the fraction of plans whose true gamma passing rate is below 95% but whose prediction interval lower bound is above 95%; if that empirical risk exceeds the chosen $\alpha = 0.1$ by more than sampling error, the central safety claim fails. A stronger check compares this risk with the risk of an interval built by applying Eq. (7) directly to the same calibration set.

Watch

Extended reading notes

Core claim

The central discovery claimed is that conformal risk control can be folded into model training, not just applied post hoc, and that doing so yields prediction intervals that are both clinically safe and practically useful for IMRT QA triage. The paper defines a one-sided risk function that charges a loss exactly when a prediction interval's lower bound lies above the 95% gamma passing rate threshold while the true GPR is below it, and uses the average risk on validation data to select the interval-scaling parameter $\lambda$ per training minibatch. During training the model minimizes mean squared error plus a penalty that pushes the lower bound below the threshold for unsafe plans; at deployment the interval is $[\hat{y} - I, \hat{y} + I]$ with $I$ the average one-sided calibration width from training, and a plan is triaged as safe only if its interval's lower bound exceeds the threshold. The paper reports sensitivity 1.0, specificity 0.76, and a 75% reduction in measurement with an average interval width of 5.81 on pooled data, and shows this beats the conformal prediction, conformal quantile regression, conformal risk control, and conformal training baselines on specificity and measurement reduction while matching their sensitivity.

Load-bearing premise

The safety claim rests on the assumption that the average interval width used at deployment inherits the risk control property proven for the per-minibatch calibrated intervals; the paper does not prove that averaging preserves the guarantee.

Editorial extensions

If this is right

  • On pooled data the method achieves sensitivity 1.0, specificity 0.76, and a 75% reduction in measurement, with average interval width 5.81, beating all four conformal baselines on specificity and reduction.
  • Because sensitivity is 100% in the pooled experiments, no plan whose measured gamma passing rate is below the 95% threshold is triaged as safe, so the reported workload reduction does not come at the cost of passing an unsafe plan.
  • Incorporating the clinical decision threshold and risk function into training produces meaningfully tighter intervals than post hoc split conformal prediction (5.81 vs 12.29 on pooled data), showing the training objective matters for interval usefulness.
  • Under distribution shift between the two time-separated datasets, all conformal methods lose some performance, and the proposed method's prospective-threshold sensitivity drops to 0.86 in one direction, underscoring that the benefits rely on calibration and test data being exchangeable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The principle of training with the clinical risk function in the loop is not specific to gamma passing rates: any safety-critical regression task with an asymmetric cost structure, such as predicting a lab value where a false 'normal' is dangerous, could reuse the same training-aware risk control recipe.
  • Because the averaged interval width used at deployment is not produced by a fresh conformal calibration step, the formal risk bound proven for the per-minibatch $\lambda$-selected intervals does not automatically transfer to the deployed interval; a hospital adopting the method would want to recompute the empirical risk on its own calibration data before trusting the 75% triage rate.
  • The pooled-data result comes from two temporally separated cohorts on one machine at one institution; testing on multi-institution, multi-machine data would clarify how much of the specificity gain survives genuine device and workflow variation.
  • A natural extension is to make the triage decision depend on disease site or treatment region, since the paper notes that the same gamma passing rate can carry different clinical meaning for different anatomical sites.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a training-aware conformal risk control method for triaging IMRT plans in quality assurance. The method combines conformal training (Stutz et al., 2021) with conformal risk control (Angelopoulos et al., 2022) by incorporating the clinical 95% gamma passing rate threshold and the associated sensitivity risk into the training objective. During training, a two-layer MLP is optimized with a penalty based on the lower conformal bound, and per-minibatch conformal risk control intervals are computed; the final deployed interval uses the average of these per-minibatch one-sided widths. The method is evaluated on two real-world IMRT datasets from Johns Hopkins Hospital, both pooled and under distribution shift, and compared with split conformal prediction, conformal quantile regression, conformal risk control, and conformal training. Reported metrics include sensitivity, specificity, reduction in measurement, coverage, and interval width.

Significance. If the risk-control claim were rigorously established, the paper would make a useful contribution to clinical machine learning: it addresses a real safety-critical workflow, incorporates a clinically meaningful threshold into the conformal framework, and reports an apparent large improvement in specificity and measurement reduction over baselines while maintaining sensitivity. The paper provides a public code link and evaluates on real clinical data, which are strengths. However, the central theoretical guarantee is not established for the final deployed interval, and the empirical safety evidence is based on a very small number of unsafe plans. The application is timely, but the methodological advance beyond combining existing conformal training and risk control is incremental.

major comments (4)
  1. [Section 3.5, Eq. (8), and Appendix G, Theorem 1] The final deployed interval width I is the average of per-minibatch one-sided interval widths obtained during conformal training, not the output of the calibration step in Eq. (7). The risk control guarantee in Appendix G applies only to the λ selected by Eq. (7) on a fixed calibration set with a fixed model, and the proof relies on exchangeability of the calibration and test points. In the training-aware method, the model changes after each minibatch while the same validation set is reused for calibration, so the exchangeability condition is violated even for the per-minibatch intervals, and no argument transfers the guarantee to the averaged width. The paper acknowledges this ("we may lose the coverage guarantee in theory"), but the abstract and title still describe the method as risk control. The empirical sensitivity of 1.0 in Table 1 is based on only 19 unsafe plans in the pooled data and is not a substitute for a formal guarantee. Please either provide a valid risk bound for the averaged-width procedure or explicitly reframe the contribution as a heuristic with only empirical validation, and qualify the "risk control" terminology throughout.
  2. [Section 4.4 (Feature Selection)] The two-sample t-test used for feature selection is described without stating whether it is applied to the training split only or to the full pooled dataset. The text says "in our data exploration" and "we find 12 features have a statistical difference between their distributions between 'safe' and 'unsafe'," which suggests the test may use labels from the entire dataset, including the held-out test set. If test labels are used to select features, the out-of-sample sensitivity and specificity in Tables 1 and 2 are optimistically biased. Please clarify the exact data used for the t-test and, if necessary, move feature selection inside the training split and rerun the experiments.
  3. [Tables 1 and 3, Section 5] The claim "we are able to guarantee 100% sensitivity in the pooled data" is an empirical observation, not a guarantee. The pooled dataset contains only 19 plans below 95% GPR (4 in Dataset 1 and 15 in Dataset 2), and after splitting into train, validation, and test, the test set contains a small and variable number of positive examples. A sensitivity of 1 out of a handful of positives has a wide confidence interval. Moreover, under distribution shift (Table 3, Dataset 1 as test), the proposed method's sensitivity drops to 0.86 in the prospective-threshold evaluation, so the safety property is not robust to shift. Please report confidence intervals for sensitivity (e.g., Clopper-Pearson) and state the small-sample limitation in the abstract and conclusion.
  4. [Section 4.2 (Experimental Setup)] The description of how the ensemble confidence interval is formed contains an apparent typo: "the lower bound of the confidence interval would be the smallest value of the upper bound estimates made by each model in the ensemble." This should presumably read "smallest value of the lower bound estimates." As written, the lower bound is an undefined quantity, and it is unclear how the ensemble intervals were actually computed, which affects the reproducibility of the reported interval widths.
minor comments (5)
  1. [Section 3.3, Eq. (5)] The risk function uses a hard threshold at 95% GPR. In clinical practice, GPR measurements have noise, and plans near the threshold may be misclassified. Please discuss the sensitivity of the method to the exact threshold choice and to measurement noise.
  2. [Section 3.5] The penalty term is described as MSE + max(0, (lower conformal bound - actual value)), while the risk function penalizes only the case where the lower bound exceeds 95 and the true GPR is below 95. The relationship between the two is not explicitly stated. Clarify how the penalty is computed per minibatch and how it aligns with the risk function.
  3. [Table 1] The base model row reports no interval width or coverage. Consider reporting the standard deviation of the base model's predictions or the width of a naive interval to aid comparison with the conformal methods.
  4. [Appendix C] The feature selection comparison in Table 6 uses a threshold of "99.99" that is not defined in the text. Please specify what this threshold refers to.
  5. [Throughout] There are several typos and inconsistencies: "exchangability" should be "exchangeability" (Section 4.5), "one-side" should be "one-sided" (Sections 3.4 and 3.5), and "descrptions" should be "descriptions" (Appendix B). Please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core derivation is self-contained and the held-out evaluation does not feed back into the model.

full rationale

The paper's central derivation chain is not circular. The conformal risk control selection rule in Eq. (7) is taken from Angelopoulos et al. (2022) and re-derived in Appendix G, so the risk-control result is an externally sourced mathematical fact rather than an assumption defined in terms of the paper's own conclusion. The proposed training-aware method uses conformal risk control only to form per-minibatch penalty terms during training, and the final interval width I is an average of training-time one-sided widths; the paper explicitly labels this averaging a 'heuristic' and states that 'we may lose the coverage guarantee in theory.' That is an acknowledged safety/rigor gap, not a circular step. All sensitivity, specificity, coverage, and interval-width numbers in Tables 1-4 are produced on held-out test data after training and calibration, so the reported predictions are genuine out-of-sample results rather than fitted values renamed as predictions. The only self-citations (Prinster et al., 2022, 2023) appear in a future-work sentence and are not load-bearing. Therefore no prediction or first-principles result reduces to its inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the exchangeability assumption, the clinical validity of the 95% GPR threshold, the pooling assumption for the two datasets, and the unstated assumption that feature selection does not leak test information. The final interval width is determined by the training procedure rather than by an independent calibration, so the risk guarantee is not formally inherited. No new entities are introduced.

free parameters (5)
  • Feature selection p-value threshold = 0.05
    Features with a two-sample t-test p-value below 0.05 between safe and unsafe plans are selected (Section 4.4). The threshold and the resulting 12-feature set are chosen from the data and directly affect model input.
  • Base model hyperparameters = hidden nodes 100, sigmoid, 1500 epochs, learning rate 0.01
    Selected by grid search to minimize validation MSE (Section 4.5, Appendix D). These choices affect the model's predictive performance and the resulting intervals.
  • Risk level alpha = 0.1
    Set by the authors for all conformal methods (Appendix D). It controls the desired risk/miscoverage level and directly influences interval width and the final triage behavior.
  • Training class balancing ratio = balanced (counts matched)
    The training set is balanced between plans below and above 95% GPR to handle class imbalance (Section 4.5). This alters the training distribution and affects the learned model.
  • Number of ensemble models = 5
    The base model is an ensemble of 5 MLPs with different initializations (Section 4.2), affecting the stability and width of the conformal intervals.
assumptions (4)
  • domain assumption Exchangeability of calibration and test samples
    Invoked in Section 3.1 and 3.3 for the conformal prediction and risk control guarantees. It holds for the pooled random split approximation but is violated in the cross-dataset experiments, as the authors acknowledge.
  • domain assumption Pooling of Dataset 1 and Dataset 2 is valid
    Section 4.1 asserts the two datasets come from dosimetrically matched machines and 'the data generating mechanism is totally the same', justifying pooling and treating the two periods as exchangeable.
  • domain assumption The 95% GPR threshold with 3%/3mm criteria is the correct clinical safety threshold
    The entire risk function and evaluation hinge on classifying plans below/above 95% GPR as unsafe/safe (Sections 3.3, 4.3).
  • ad hoc to paper Feature selection does not use test labels
    The paper does not state whether the t-test feature selection in Section 4.4 is confined to training folds. The presented procedure, if applied to the full pooled data, would leak test label information into model inputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training-Aware Risk Control for Intensity Modulated Radiation Therapies Quality Assurance with Conformal Prediction." pith.science (2026). https://pith.science/paper/YPA7JZK5

@misc{pith2026250108963,
  author       = {Pith},
  title        = {Pith review of: Training-Aware Risk Control for Intensity Modulated Radiation Therapies Quality Assurance with Conformal Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPA7JZK5}},
  note         = {Machine review of arXiv:2501.08963}
}
read the original abstract

Measurement quality assurance (QA) practices play a key role in the safe use of Intensity Modulated Radiation Therapies (IMRT) for cancer treatment. These practices have reduced measurement-based IMRT QA failure below 1%. However, these practices are time and labor intensive which can lead to delays in patient care. In this study, we examine how conformal prediction methodologies can be used to robustly triage plans. We propose a new training-aware conformal risk control method by combining the benefit of conformal risk control and conformal training. We incorporate the decision making thresholds based on the gamma passing rate, along with the risk functions used in clinical evaluation, into the design of the risk control framework. Our method achieves high sensitivity and specificity and significantly reduces the number of plans needing measurement without generating a huge confidence interval. Our results demonstrate the validity and applicability of conformal prediction methods for improving efficiency and reducing the workload of the IMRT QA process.

Figures

Figures reproduced from arXiv: 2501.08963 by the authors.

Figure 1
Figure 1. The workflow of a radiation oncology department for delivering IMRT treatment. The IMRT process involves multiple different disciplines and professionals in the workflow to design and implement a personalized treatment plan based on a patient’s unique disease condition. The IMRT QA process is the evaluation process after a plan is designed and before a plan is deployed in the treatment. It is a safety-critical task … view at source ↗
Figure 2
Figure 2. Depiction of coronal absorbed dose distribution in the A) patient and correspondingly in the B) quality assurance phantom detector plane. Appendix B. Feature Descriptions and More Details on Feature Extraction As mentioned, we follow the feature definition and calculation in the previous work (Lam et al., 2019). We demonstrate their feature names and descrptions in [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 19 canonical work pages

  1. [1]

    Angelopoulos and S

    A. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021

    arXiv 2021

  2. [2]

    Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster

    A. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814, 2022

    arXiv 2022

  3. [3]

    Argaw, E

    P. Argaw, E. Healey, and I. Kohane. Identifying heterogeneous treatment effects in multiple outcomes using joint confidence intervals. Proceedings of Machine Learning Research, 2022

    work page 2022

  4. [4]

    M. Chan, A. Witztum, and G. Valdes. Integration of ai and machine learning in radiotherapy qa. Frontiers of Artificial Intelligence, 2020

    work page 2020

  5. [5]

    Gospodarowicz

    D.Jaffray and M. Gospodarowicz. Cancer: Disease Control Priorities. World Bank, 2015

    work page 2015

  6. [6]

    Eaton, JP

    DJ. Eaton, JP. Byrne, and SJ. Thomas VP. Cosgrove. Unintended doses in radiotherapy-over, under, and outside. British Journal of Radiology, 2018

    work page 2018

  7. [7]

    Ezzell, J

    G. Ezzell, J. Burmeister, T. LoSasso N. Dogan, J. Mechalakos, D. Mihailidis, A. Molineu, J. Palta, C.Ramsey, B. Salter, J. Shi, P. Xia, N. Yue, and Y. Xiao. Imrt commissioning: multi-ple institution planning and dosimetry comparisons, a report from aapm task group 119. Medical Physics, 36: 0 5359--5373, 2009

    work page 2009

  8. [8]

    Adaptive conformal inference under distribution shift

    Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34: 0 1660--1672, 2021

    work page 2021

Show all 25 references
  1. [9]

    Interian, V

    Y. Interian, V. Rideout, V. Kearney, E. Gennatas, O. Morin, J. Cheung, T. Solberg, and G. Valdes. Deep nets vs expert designed features in medical physics: An imrt qa case study. Medical Physics, 45: 0 2672--2680, 2018

    2018

  2. [10]

    Robert L

    C. Robert L. Simon and Philippe Meyer. Artificial intelligence for quality assurance in radiotherapy. Cancer/Radiothérapie, 25: 0 623--626, 2021

    2021

  3. [11]

    D. Lam, H. Li X. Zhang, Y. Deshan, B. Schott, T. Zhao, W. Zhang, S. Mutic, and B. Sun. Predicting gamma passing rates for portal dosimetry-based imrt qa using machine learning. Medical Physics, 46: 0 4666--4675, 2019

    2019

  4. [12]

    Miften, A

    M. Miften, A. Olch, D. Mihailidis, J. Moran, T. Pawlicki, A. Molineu, H. Li, K. Wijesooriya, J. Shi, P. Xia, N. Papanikolaou, and D. Low. Tolerance limits and methodologies for imrt measurement-based verification qa: Recommendations of aapm task group no. 218. Medical Physics,...

    2018

  5. [13]

    Palta, C

    J. Palta, C. Liu, and J. Li. Quality assurance of intensity modulated radiation therapy. International Journal of Radiation Oncology, 2008

    2008

  6. [14]

    Papadopoulos, A

    H. Papadopoulos, A. Gammerman, and V. Vovk. Reliable diagnosis of acute abdominal pain with conformal prediction. Engineering Intelligent Systems, 2009

    2009

  7. [15]

    Pereira, S

    T. Pereira, S. Cardoso, M. Guerreiro, A. Mendonça de, and S. Madeira. Targeting the uncertainty of predictions at patient-level using an ensemble of classifiers coupled with calibration methods, venn-abers, and conformal predictors: A case study in ad. Journal of Biomedical In...

    2019

  8. [16]

    Distribution-free uncertainty quantification for classification under label shift

    Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence, pages 844--853. PMLR, 2021

    2021

  9. [17]

    Jaws: Auditing predictive uncertainty under covariate shift

    Drew Prinster, Anqi Liu, and Suchi Saria. Jaws: Auditing predictive uncertainty under covariate shift. Advances in Neural Information Processing Systems, 35: 0 35907--35920, 2022

    2022

  10. [18]

    Jaws-x: Addressing efficiency bottlenecks of conformal prediction under standard and feedback covariate shift

    Drew Prinster, Suchi Saria, and Anqi Liu. Jaws-x: Addressing efficiency bottlenecks of conformal prediction under standard and feedback covariate shift. In International Conference on Machine Learning, pages 28167--28190. PMLR, 2023

    2023

  11. [19]

    Romano, E

    Y. Romano, E. Patterson, and E. Candès. Conformalized quantile regression. arXiv preprint arXiv:1905.03222, 2019

    1905 arXiv

  12. [20]

    Smilowitz, I

    J. Smilowitz, I. Das, and and S. Kry and I. Marshall D. Mihailidis and Z. Ouhib and T. Ritter and M. Snyder and L. Fairobent V. Feygelman, and B. Fraass. Aapm medical physics practice guideline 5.a.: Commissioning and qa of treatment planning dose calculations — megavoltage ph...

    2015

  13. [21]

    Stutz, Krishnamurthy (Dj) Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet

    D. Stutz, Krishnamurthy (Dj) Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. arXiv preprint arXiv:2110.09192, 2021

    2021 arXiv

  14. [22]

    Conformal prediction under covariate shift

    Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    G. Valdes. A mathematical framework for virtual imrt qa using machine learning. Medical Physics, 43: 0 4323--4334, 2016

    2016

  16. [24]

    Vazquez and J

    J. Vazquez and J. Facelli. Conformal prediction in clinical medical sciences. Journal of Healthcare Informatics Research, 2022

    2022

  17. [25]

    V. Vovk, A. Gammerman, and G. Shafer. Algorithmic Learning in a Random World. Springer, 2005

    2005

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.