pith. sign in

arxiv: 2510.21934 · v3 · submitted 2025-10-24 · 💻 cs.LG · stat.ML

Joint Score-Threshold Optimization for Interpretable Risk Assessment

Fardin Ganjkhanloo , Emmett Springer , Erik H. Hoyer , Daniel L. Young , Kimia Ghobadi This is my paper

Pith reviewed 2026-05-18 04:09 UTC · model grok-4.3

classification 💻 cs.LG stat.ML
keywords risk assessmentmixed-integer programminginterpretable scoringordinal classificationthreshold optimizationclinical decision supportasymmetric losshealthcare optimization
0
0 comments X

The pith

Mixed-integer programming jointly optimizes both point scores and risk category thresholds to handle incomplete labels and asymmetric error costs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to build interpretable point-based risk scores by solving one optimization program that chooses both the weights on patient features and the cutoffs separating low, medium, and high risk groups. Standard supervised learning fails here because outcome labels are usually missing for middle risk levels and because the cost of underestimating risk grows with how far the mistake is. The proposed framework adds constraints that stop categories from vanishing when data is scarce for them and uses an objective that penalizes larger ordinal errors more than small ones. It also lets users add rules such as keeping most scores positive or limiting changes to an existing tool, and it supplies a simpler continuous version whose solutions speed up finding the final integer solution.

Core claim

A mixed-integer programming model simultaneously determines the point values assigned to each risk factor and the numeric thresholds that map total scores to ordinal risk categories. Threshold constraints prevent any category from collapsing when labels exist only for the extreme groups, while an asymmetric objective weights misclassifications by their ordinal distance. The same model accepts sign restrictions, sparsity requirements, and bounds on deviation from an incumbent scoring system. A continuous relaxation of the integer program supplies warm-start solutions that reduce solve time without changing the quality of the final solution.

What carries the argument

The mixed-integer program that jointly selects feature weights and ordinal thresholds subject to label-scarcity constraints and an asymmetric distance-aware objective.

If this is right

  • Governance rules such as sign restrictions on weights, sparsity limits, and bounds on changes to an existing tool can be enforced inside the same optimization.
  • The continuous relaxation produces warm-start points that shorten the time required to reach an optimal integer solution.
  • The resulting scoring system remains a simple sum of points that clinicians can compute by hand or embed in existing workflows.
  • The same structure applies to any ordinal risk tool where middle-category outcomes are censored by early intervention.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same joint-optimization idea could be tested on other clinical scales such as pressure-ulcer or delirium risk tools where label patterns are similar.
  • One could measure whether the data-driven thresholds produced by the model differ systematically from the round-number cutoffs clinicians currently use.
  • If the continuous relaxation consistently yields the same final integer solution, it may be possible to replace the full MIP with the relaxed version in time-sensitive settings.

Load-bearing premise

The added threshold constraints and distance-weighted penalties must match how clinicians actually experience and prioritize different kinds of mistakes.

What would settle it

On a dataset that supplies labels for every risk category, compare the out-of-sample performance of the jointly optimized thresholds against thresholds chosen by clinicians or by a two-stage method that fixes thresholds first.

Figures

Figures reproduced from arXiv: 2510.21934 by Daniel L. Young, Emmett Springer, Erik H. Hoyer, Fardin Ganjkhanloo, Kimia Ghobadi.

Figure 1
Figure 1. Figure 1: Stratification of inpatient admissions based on fall events and targeted interventions. [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Score distributions for all labeled data (test and train data together) [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Score differentials (MIP score - JHFRAT score) for all patients in dataset [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Receiver Operating Characteristic (ROC) and Precision-Recall performance curves across [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗
read the original abstract

Risk assessment tools in healthcare commonly employ point-based scoring systems that map patients to ordinal risk categories via thresholds. While electronic health record (EHR) data presents opportunities for data-driven optimization of these tools, two fundamental challenges impede standard supervised learning: (1) labels are often available only for extreme risk categories due to intervention-censored outcomes, and (2) misclassification cost is asymmetric and increases with ordinal distance. We propose a mixed-integer programming (MIP) framework that jointly optimizes scoring weights and category thresholds in the face of these challenges. Our approach prevents label-scarce category collapse via threshold constraints, and utilizes an asymmetric, distance-aware objective. The MIP framework supports governance constraints, including sign restrictions, sparsity, and minimal modifications to incumbent tools, ensuring practical deployability in clinical workflows. We further develop a continuous relaxation of the MIP problem to provide warm-start solutions for more efficient MIP optimization. We apply the proposed score optimization framework to a case study of inpatient falls risk assessment using the Johns Hopkins Fall Risk Assessment Tool.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes a mixed-integer programming (MIP) framework for jointly optimizing scoring weights and ordinal category thresholds in point-based healthcare risk assessment tools. It addresses label scarcity for intermediate categories via explicit threshold constraints, employs an asymmetric distance-aware objective to reflect increasing misclassification costs, and incorporates governance constraints (sign restrictions, sparsity, minimal changes to incumbent tools). A continuous relaxation is introduced to generate warm-start solutions for the MIP solver. The framework is illustrated on a case study adapting the Johns Hopkins Fall Risk Assessment Tool to inpatient falls data.

Significance. If the empirical claims hold, the work provides a principled, optimization-based route to data-driven yet interpretable risk scores that respects clinical constraints and censored outcomes. The joint treatment of weights and thresholds, combined with the asymmetric objective and governance features, offers a clear advance over sequential or unconstrained approaches. The continuous relaxation for warm-starting is a practical engineering contribution that could aid real-world deployment.

major comments (2)
  1. [Continuous relaxation (methodology section)] The description of the continuous relaxation states that it supplies warm-start solutions that accelerate MIP solves while leaving the final integer solution unchanged. However, no wall-clock timings, objective-value comparisons, or parameter-equivalence checks between cold-start and warm-start runs are reported on the Johns Hopkins inpatient falls data. Because branch-and-bound performance is known to be initialization-sensitive, this missing verification directly undermines the deployability argument that rests on the relaxation as a key enabler.
  2. [Case study section] The case-study application to the Johns Hopkins Fall Risk Assessment Tool is presented without quantitative performance metrics, ablation results, or solver statistics. The central claim that the MIP framework improves upon existing tools therefore rests on descriptive formulation rather than demonstrated gains in risk stratification or computational efficiency.
minor comments (1)
  1. [Objective function] Clarify the precise definition of the asymmetric distance-aware objective function and how the ordinal distance penalty is scaled; a small illustrative example would aid readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments on our manuscript. We address each major comment point by point below and have made revisions to strengthen the empirical support for our claims.

read point-by-point responses

  1. Referee: [Continuous relaxation (methodology section)] The description of the continuous relaxation states that it supplies warm-start solutions that accelerate MIP solves while leaving the final integer solution unchanged. However, no wall-clock timings, objective-value comparisons, or parameter-equivalence checks between cold-start and warm-start runs are reported on the Johns Hopkins inpatient falls data. Because branch-and-bound performance is known to be initialization-sensitive, this missing verification directly undermines the deployability argument that rests on the relaxation as a key enabler.

    Authors: We agree that the absence of empirical solver performance data limits the strength of the deployability argument. Although the continuous relaxation is constructed to yield feasible warm-starts that preserve the optimal integer solution, we have added a dedicated subsection with wall-clock timings, objective-value comparisons, and branch-and-bound statistics (nodes explored, optimality gaps) for cold-start versus warm-start MIP runs on the Johns Hopkins inpatient falls dataset. These results are reported in a new table and accompanying text. revision: yes

  2. Referee: [Case study section] The case-study application to the Johns Hopkins Fall Risk Assessment Tool is presented without quantitative performance metrics, ablation results, or solver statistics. The central claim that the MIP framework improves upon existing tools therefore rests on descriptive formulation rather than demonstrated gains in risk stratification or computational efficiency.

    Authors: The case study was designed to demonstrate the practical incorporation of clinical governance constraints and censored-label handling rather than to serve as a comprehensive benchmark. We acknowledge, however, that quantitative evidence is necessary to substantiate improvement claims. We have expanded the case-study section to include risk-stratification metrics (AUC, Brier score, and ordinal misclassification rates), ablation experiments (removing the asymmetric objective or individual governance constraints), and solver statistics (solve times and solution quality with and without the relaxation). revision: yes

Circularity Check

0 steps flagged

No circularity: direct MIP formulation with external case-study grounding

full rationale

The manuscript presents a constructive mixed-integer programming formulation that jointly optimizes scoring weights and ordinal thresholds under explicit constraints for label scarcity and asymmetric costs. This is an optimization model rather than a derivation chain that reduces to prior fitted parameters or self-citations. The continuous relaxation is introduced solely as a computational aid for warm-starting the MIP solver; its correctness does not depend on the target solution being presupposed. The Johns Hopkins inpatient falls case study supplies an external, real-world benchmark independent of the model's internal parameters. No load-bearing step equates a claimed prediction or uniqueness result to its own inputs by construction.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim rests on domain assumptions about ordinal costs and label censoring plus modeling choices for constraints; no new physical entities are postulated.

free parameters (1)
  • threshold constraint bounds
    Bounds chosen to prevent category collapse; their specific values are modeling decisions that affect feasible solutions.
axioms (2)
  • domain assumption Misclassification cost increases with ordinal distance between categories.
    Invoked to define the asymmetric objective in the MIP formulation.
  • domain assumption Labels are available only for extreme risk categories due to intervention censoring.
    Stated as a fundamental challenge that the threshold constraints address.

pith-pipeline@v0.9.0 · 5721 in / 1258 out tokens · 34587 ms · 2026-05-18T04:09:16.191084+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages · 1 internal anchor

  1. [1]

    The braden scale for predicting pressure sore risk.Nursing research, 36(4):205–210, 1987

    Nancy Bergstrom, Barbara J Braden, Antoinette Laguzza, and Victoria Holman. The braden scale for predicting pressure sore risk.Nursing research, 36(4):205–210, 1987

    work page 1987

  2. [2]

    Rank consistent ordinal regression for neural networks with application to age estimation.Pattern Recognition Letters, 140:325–331, 2020

    Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation.Pattern Recognition Letters, 140:325–331, 2020

    work page 2020

  3. [3]

    Clinical assessment of venous thromboembolic risk in surgical patients

    JA Caprini, JI Arcelus, JH Hasty, AC Tamhane, and F Fabrega. Clinical assessment of venous thromboembolic risk in surgical patients. InSeminars in thrombosis and hemostasis, volume 17, pages 304–312, 1991

    work page 1991

  4. [4]

    Support vector ordinal regression.Neural computation, 19(3):792–815, 2007

    Wei Chu and S Sathiya Keerthi. Support vector ordinal regression.Neural computation, 19(3):792–815, 2007

    work page 2007

  5. [5]

    Learning from partial labels.The Journal of Machine Learning Research, 12:1501–1536, 2011

    Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels.The Journal of Machine Learning Research, 12:1501–1536, 2011

    work page 2011

  6. [6]

    Fall prevention in acute care hospitals: a randomized trial.Jama, 304(17):1912–1918, 2010

    Patricia C Dykes, Diane L Carroll, Ann Hurley, Stuart Lipsitz, Angela Benoit, Frank Chang, Seth Meltzer, Ruslana Tsurikova, Lyubov Zuyov, and Blackford Middleton. Fall prevention in acute care hospitals: a randomized trial.Jama, 304(17):1912–1918, 2010

    work page 1912

  7. [7]

    The foundations of cost-sensitive learning

    Charles Elkan. The foundations of cost-sensitive learning. InInternational joint conference on artificial intelligence, volume 17, pages 973–978. Lawrence Erlbaum Associates Ltd, 2001

    work page 2001

  8. [8]

    Provably consistent partial-label learning.Advances in neural information processing systems, 33:10948–10960, 2020

    Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning.Advances in neural information processing systems, 33:10948–10960, 2020

    work page 2020

  9. [9]

    Benjamin A Goldstein, Ann Marie Navar, Michael J Pencina, and John PA Ioannidis. Op- portunities and challenges in developing risk prediction models with electronic health records data: a systematic review.Journal of the American Medical Informatics Association: JAMIA, 24(1):198, 2016

    work page 2016

  10. [10]

    What do we need to build explainable AI systems for the medical domain?

    Andreas Holzinger, Chris Biemann, Constantinos S Pattichis, and Douglas B Kell. What do we need to build explainable ai systems for the medical domain?arXiv preprint arXiv:1712.09923, 2017. 21

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  11. [11]

    6-clicks

    Diane U Jette, Mary Stilphen, Vinoth K Ranganathan, Sandra D Passek, Frederick S Frost, and Alan M Jette. Am-pac “6-clicks” functional assessment scores predict acute care hospital discharge destination.Physical therapy, 94(9):1252–1261, 2014

    work page 2014

  12. [12]

    Tailored bayes: a risk modeling framework under unequal misclassification costs

    Solon Karapanagiotis, Umberto Benedetto, Sach Mukherjee, Paul DW Kirk, and Paul J Newcombe. Tailored bayes: a risk modeling framework under unequal misclassification costs. Biostatistics, 24(1):85–107, 2023

    work page 2023

  13. [13]

    Progressive identification of true labels for partial-label learning

    Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. Ininternational conference on machine learning, pages 6500–6510. PMLR, 2020

    work page 2020

  14. [14]

    Regression models for ordinal data.Journal of the Royal Statistical Society: Series B (Methodological), 42(2):109–127, 1980

    Peter McCullagh. Regression models for ordinal data.Journal of the Royal Statistical Society: Series B (Methodological), 42(2):109–127, 1980

    work page 1980

  15. [15]

    Ordinal regression with multiple output cnn for age estimation

    Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 4920–4928, 2016

    work page 2016

  16. [16]

    David Oliver, M Britton, P Seed, FC Martin, and AH Hopper. Development and evaluation of evidence based risk assessment tool (stratify) to predict which elderly inpatients will fall: case-control and cohort studies.Bmj, 315(7115):1049–1053, 1997

    work page 1997

  17. [17]

    Preventing falls and fall-related injuries in hospitals.Clinics in geriatric medicine, 26(4):645–692, 2010

    David Oliver, Frances Healey, and Terry P Haines. Preventing falls and fall-related injuries in hospitals.Clinics in geriatric medicine, 26(4):645–692, 2010

    work page 2010

  18. [18]

    Partial proportional odds models for ordinal response variables.Journal of the Royal Statistical Society: Series C (Applied Statistics), 39(2):205–217, 1990

    Bercedis Peterson and Frank E Harrell Jr. Partial proportional odds models for ordinal response variables.Journal of the Royal Statistical Society: Series C (Applied Statistics), 39(2):205–217, 1990

    work page 1990

  19. [19]

    The johns hopkins fall risk assessment tool: postimplementation evaluation.Journal of nursing care quality, 22(4):293–298, 2007

    Stephanie S Poe, Maria Cvach, Patricia B Dawson, Harriet Straus, and Elizabeth E Hill. The johns hopkins fall risk assessment tool: postimplementation evaluation.Journal of nursing care quality, 22(4):293–298, 2007

    work page 2007

  20. [20]

    Machine learning in medicine.New England Journal of Medicine, 380(14):1347–1358, 2019

    Alvin Rajkomar, Jeffrey Dean, and Isaac Kohane. Machine learning in medicine.New England Journal of Medicine, 380(14):1347–1358, 2019

    work page 2019

  21. [21]

    Smooth hinge classification.Proceeding of Massachusetts Institute of Technology, 2005

    Jason DM Rennie. Smooth hinge classification.Proceeding of Massachusetts Institute of Technology, 2005

    work page 2005

  22. [22]

    Loss functions for preference levels: Regression with discrete ordered labels

    Jason DM Rennie and Nathan Srebro. Loss functions for preference levels: Regression with discrete ordered labels. InProceedings of the IJCAI multidisciplinary workshop on advances in preference handling, volume 1, pages 1–6. AAAI Press, Menlo Park, CA, 2005

    work page 2005

  23. [23]

    Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature machine intelligence, 1(5):206–215, 2019

    Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.Nature machine intelligence, 1(5):206–215, 2019

    work page 2019

  24. [24]

    Ranking with large margin principle: Two approaches

    Amnon Shashua and Anat Levin. Ranking with large margin principle: Two approaches. Advances in neural information processing systems, 15, 2002

    work page 2002

  25. [25]

    Learning optimized risk scores.Journal of Machine Learning Research, 20(150):1–75, 2019

    Berk Ustun and Cynthia Rudin. Learning optimized risk scores.Journal of Machine Learning Research, 20(150):1–75, 2019. 22

    work page 2019

  26. [26]

    Adaptive graph guided disambiguation for partial label learning

    Deng-Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 83–91, 2019

    work page 2019

  27. [27]

    Partial label learning with unlabeled data

    Qian-Wei Wang, Yufeng Li, Zhi-Hua Zhou, et al. Partial label learning with unlabeled data. In IJCAI, pages 3755–3761, 2019

    work page 2019

  28. [28]

    Solving the partial label learning problem: An instance-based approach

    Min-Ling Zhang and Fei Yu. Solving the partial label learning problem: An instance-based approach. InIJCAI, pages 4048–4054, 2015. 23

    work page 2015