Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Learning optimal treatment strategies for intraoperative hypotension using deep reinforcement learning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A deep reinforcement learning agent trained on 50,021 surgeries recommends intraoperative fluid and vasopressor doses that outscore physicians' actual care in off-policy evaluation.

desk verdict Large-cohort RL for intraoperative hypotension with a promising descriptive analysis, but the central off-policy claim is built on an invalid importance-sampling estimator. read the letter →

arxiv 2505.21596 v1 pith:DJT4O3PT submitted 2025-05-27 q-bio.QM cs.AIcs.LG

Esra Adiyeke , Tianqi Liu , Venkata Sai Dheeraj Naganaboina , Han Li , Tyler J. Loftus , Yuanfang Ren , Benjamin Shickel , Matthew M. Ruppert
show 5 more authors
Karandeep Singh Ruogu Fang Parisa Rashidi Azra Bihorac Tezcan Ozrazgat-Baslanti
This is my paper
classification q-bio.QMcs.AIcs.LG
keywords reinforcementlearningintraoperativehypotensionacutekidneyinjurydeepQ-networksoff-policyevaluationvasopressorsperioperativecareclinicaldecisionsupport
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a deep reinforcement learning agent can recommend intravenous fluid and vasopressor doses during surgery that would reduce intraoperative hypotension and postoperative acute kidney injury (AKI). Using 50,021 surgeries from a single quaternary care center, it trains a Dueling Double Deep Q-Network on 15-minute patient states and evaluates the learned policy against physician decisions, random dosing, and zero-drug dosing. The central result is that the RL policy has a higher estimated policy value than all three alternatives, and that AKI prevalence was lowest among patients whose actual doses matched the model's recommendations. A sympathetic reader is meant to take this as evidence that data-driven, real-time dosing suggestions could improve perioperative outcomes.

What carries the argument

The machinery is a Dueling Double Deep Q-Network (D3QN) trained on 200 cluster-discretized patient states, with a 25-action space (5 IV fluid levels by 5 vasopressor levels) and a reward combining a +15/−15 long-term AKI term with a −1.75 per-step penalty for hypotension (MAP < 65 mmHg or 20% below baseline). A KL-divergence penalty keeps the agent's action distribution near physician practice, and a modified weighted importance sampling estimator, replacing the cumulative importance ratio $\rho_{1:t}$ with $\rho''_{1:t} = \rho_{1:t}/(1+\rho_{1:t})$, is used to estimate policy value off-policy from the recorded physician trajectories.

What would settle it

The central claim would be settled by recomputing the policy-value comparison with the standard, unmodified weighted importance sampling formula on the same test data; if the agent's advantage over physician, random, and zero-drug policies disappears or its confidence intervals overlap, the modified estimator is the reason for the result. A synthetic Markov decision process with known optimal values would also show whether the $\rho/(1+\rho)$ transform recovers the true policy value.

Watch

Extended reading notes

Core claim

The central claim is that a D3QN-based reinforcement learning policy trained on retrospective electronic health record data can learn a better intraoperative dosing strategy for IV fluids and vasopressors than physicians currently deliver. The model reproduces 69% of physician vasopressor decisions and, in the remaining cases, tends to recommend less vasopressor and more IV fluid. Under the paper's weighted importance sampling evaluation, the agent policy yields a higher estimated cumulative reward than the physician policy, a random policy, and a zero-drug policy; the paper also reports that surgeries where clinicians gave doses close to the model's suggestions had the lowest AKI prevalence in the first three postoperative days. The intended upshot is that following the policy could reduce AKI and related complications driven by intraoperative hypotension.

Load-bearing premise

The evaluation's headline comparison relies on a modified reweighting formula for estimating what a policy would have earned, and the paper gives no proof that this modification still yields an unbiased or consistent estimate; if the estimate is biased, the reported advantage over physician care collapses.

Editorial extensions

If this is right

  • If the policy value comparison is right, an RL-based recommender could be deployed as a real-time clinical decision-support tool in the operating room, updating dosing suggestions every 15 minutes.
  • Physician adherence to model recommendations would be expected to lower postoperative AKI rates, since lower AKI prevalence was observed when administered doses aligned with the model's choices.
  • The model's tendency to recommend less vasopressor and more IV fluid than physicians suggests a testable hypothesis: some current intraoperative vasopressor use is excessive relative to fluid resuscitation.
  • Because the model was trained and tested on a single center's data, its policy would need retraining or fine-tuning on other populations before generalizing.
  • The 15-minute action cadence, shorter than most prior RL clinical work, implies the method is aimed at fast-moving intraoperative physiology rather than slower ICU trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: The modified importance-sampling estimator $\rho'' = \rho/(1+\rho)$ has no stated unbiasedness or consistency proof in the paper; a small simulation study comparing it with the standard WIS estimator on synthetic MDPs with known policy values would settle whether the reported policy-value ordering is an artifact of the estimator.
  • Extension: The AKI-alignment result is observational and confounded by indication; a prospective or instrumental-variable analysis could test whether the association survives adjustment for patient severity.
  • Extension: The action space's percentile-based discretization ties the meaning of "optimal" to the historical dose distribution of this hospital; a transfer study to another hospital would reveal whether the policy learns physiology or local practice patterns.
  • Extension: Because the reward puts a fixed +15 penalty on AKI and −1.75 per hypotensive epoch, the policy implicitly encodes a cost trade-off between AKI and transient hypotension; clinical stakeholders could elicit their own weights to produce patient-specific policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript develops a Deep Q-Network (D3QN) reinforcement learning agent to recommend intravenous fluid and vasopressor doses every 15 minutes during major surgery. The reward function combines a postoperative acute kidney injury (AKI) term (+15 for no AKI within 3 days, -15 for AKI) with an intraoperative hypotension penalty. Using a single-center retrospective cohort of 50,021 surgeries, the authors report that the RL policy replicates a substantial fraction of physician vasopressor and fluid decisions, yields a higher estimated policy value than physician, random, and zero-drug policies under a modified weighted importance sampling (WIS) estimator, and that patients whose actual doses align with model recommendations have the lowest AKI prevalence. The stated conclusion is that implementing the model's policy could reduce postoperative AKI.

Significance. If the central claims were supported, this would be a clinically important contribution to perioperative decision support, drawing on a large cohort and addressing a relevant outcome. The paper also provides useful clinical context, action-space design, and feature importance analysis. However, the main quantitative result rests on a nonstandard importance-sampling estimator for which no unbiasedness or consistency argument is supplied, and the AKI-alignment analysis uses the same outcome that defines the training reward without adjustment for confounding. The current evidence does not establish that the RL policy is superior to physician care or that following its recommendations would reduce AKI.

major comments (3)
  1. [Supplemental Methods, Eqs. 11-15] The modified WIS estimator is not a valid off-policy estimator. Equation 12 defines rho'' = rho/(1 + rho), which is not proportional to the cumulative importance ratio rho. The estimator in Eqs. 13-15 therefore converges to E_b[rho'' G] / E_b[rho''] rather than to V(pi_e), unless rho'' is proportional to rho, which it is not. No unbiasedness or consistency proof is provided, and the stability motivation does not justify changing the estimand. Because the learned policy is deterministic (Eq. 8), rho is zero for any trajectory containing a single mismatched action and extremely large for fully matched trajectories; the logistic transform compresses high-ratio trajectories differentially, so the relative ranking of policies in Fig. 5A can change. This is load-bearing: Fig. 5A is the basis for the abstract claim of higher estimated policy value. The authors should replace this estimator with standard WIS or per-decision importance sampling, with confidence intervals, or provide a rigorous proof that the modified estimator is consistent and does not alter the ranking.
  2. [Results, Model Evaluation and Fig. 4] The AKI alignment analysis is not a validation of the policy. The same AKI outcome is the dominant component of the reward function (Eq. 2), so the agent is directly optimized to select actions correlated with no-AKI episodes in the training distribution. In addition, the comparison in Fig. 4 and the abstract statement that AKI prevalence was lowest in patients whose doses aligned with the model do not adjust for patient risk, surgery type, disease severity, or other confounders; patients whose physicians happened to follow the agent's recommendations may differ systematically. The conclusion that implementation 'has the potential to reduce postoperative AKI' is therefore overstated. The authors should reframe this as an association and provide adjusted analyses or a proper causal evaluation framework.
  3. [Performance Evaluation and Fig. 5A] The policy value comparison lacks statistical support. The figure shows distributions over 50 random initializations, but the paper does not report numerical summaries (median, IQR), overlap between distributions, or a statistical test comparing the RL policy with the physician, random, and zero-drug policies. Even if the estimator were valid, the claim that the RL policy is 'higher' than the alternatives would need uncertainty quantification.
minor comments (5)
  1. [Table 1] The entry for 'AKI during hospitalization' in the test cohort reads '27,72 (18)'; this should be '2,772 (18)'.
  2. [Supplemental Table 2] The test cohort has substantial missingness for end-tidal CO2 (45.81%), respiratory rate (4.94%), peak inspiratory pressure (9.90%), and core temperature (7.79%). The manuscript does not describe how missing values are handled or whether this distribution shift could affect model evaluation.
  3. [Methods, Model Details] The main text states that the number of clusters was determined using 'silhouette analysis,' while the supplement refers to the 'elbow method for silhouette analysis'; these descriptions should be reconciled.
  4. [Data and code availability] No data or code availability statement is provided; given the nonstandard estimator and the complexity of the pipeline, releasing code would greatly improve reproducibility.
  5. [Figure 5A] The figure would benefit from explicit median and interquartile range values for each policy, as well as a paired comparison across the 50 model initializations.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'higher estimated policy value' claim is the RL training objective restated: WIS evaluates the same reward function the D3QN was trained to maximize, so part of the central result reduces by construction.

  1. self definitional [Methods: Action Space and Reward Function (Eqs 1-3); Supplemental Methods Eqs 5, 8, 14; Results Fig. 5A]
    "The model resulted in a higher estimated policy value compared to the physicians' actual treatments, as well as random and zero-drug policies. ... We developed a reward function as a combination of long- and short-term rewards with two major parts to consider. ... V_WIS^(i) = (rho''_{1:t_i}^{(i)} / w_D) (sum_{t=1}^{t_i} gamma^{t-1} r_t)."

    The policy value estimated by WIS (Supplemental Eq. 14) is the expected discounted sum of r_t, where r_t = r_aki + r_hypo (Eqs 1-3). The D3QN loss (Eq. 5) trains Q toward r + gamma max Q, i.e., the same cumulative reward. The agent policy is the argmax over Q (Eq. 8). Therefore reporting that this argmax policy has higher estimated value than physician, random, or zero policies on the same reward is a restatement of the optimization objective, not an independent test of clinical superiority. The magnitude is determined by how well the Q-network fit the training reward, not by any external outcome measure. This is a 'prediction' (higher policy value) equivalent to its input (the reward used for training) by construction.

full rationale

The only defensible circular step is the self-definitional evaluation metric: the 'estimated policy value' is exactly the cumulative reward the D3QN was trained to maximize, so the Fig. 5A comparison restates the optimization objective. The AKI-prevalence alignment analysis uses an external clinical outcome, though it is confounded and shares the AKI label used in the reward, so it provides partial independent content. The modified WIS estimator (Supplemental Eqs 11-15) is not shown to be unbiased or consistent; that is a correctness threat rather than circularity and does not, by itself, raise the circularity score. The data-pipeline self-citations (refs 32, 35) are support for cohort assembly, not load-bearing for the RL claim. Overall score 6: one central quantitative claim reduces by construction, but the paper also contains a non-circular empirical association.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central evaluation rests on several hand-chosen quantities (reward weights, cluster count, KL weight, action bin thresholds) and on the validity of a new off-policy estimator. The paper does not derive these from data or external benchmarks, and the AKI-alignment evidence is a confounded observational association using the same outcome that defines the reward.

free parameters (6)
  • AKI reward amplitude = +15 (no AKI), -15 (AKI)
    Hand-chosen constants in Eq. 2 that set the primary optimization target; no clinical or economic justification is given for the magnitude or balance with the hypotension penalty.
  • Hypotension penalty coefficient = -1.75 per hypotensive epoch
    Hand-chosen in Eq. 3; scales the immediate disincentive for MAP below 65 mmHg (or 20% below baseline) and affects how the agent trades off short-term hemodynamics against the terminal AKI reward.
  • Number of K-means++ clusters = 200
    Selected by silhouette analysis (Supplemental Methods C); determines the discrete state space and the transition matrix used in the MDP, and thereby the estimated physician behavior policy used for importance weighting.
  • Discount factor gamma = 0.99
    Chosen in Supplemental Methods C; with up to hundreds of 15-minute epochs per surgery, this determines the effective horizon of the value estimates.
  • KL divergence penalty weight alpha = not reported
    Introduced in Eq. 6 to penalize divergence between the agent's action distribution and physicians'; its value is never stated, yet it directly constrains the learned policy and the degree to which the agent can deviate from observed practice.
  • Action discretization thresholds = percentile-based bins (Supplemental Tables 4 and 5)
    Cutoffs for the 5x5 fluid/pressor action space are derived from the empirical distribution of historical doses; these thresholds define the action space and the headline agreement rates.
assumptions (6)
  • domain assumption The 16 clinical variables resampled at 15-minute intervals form a sufficient state for a Markov decision process (Markov property holds).
    Supplemental Methods C defines the state as these 16 variables; there is no verification that hidden factors, such as surgical stage, blood loss, or anesthesia depth, are not needed for optimal dosing.
  • domain assumption The reward function (Eqs 1-3) correctly encodes the clinical utility of treatment decisions.
    The relative weights of +15/-15 for AKI and -1.75 for hypotension are asserted without derivation from outcomes, costs, or patient preferences.
  • ad hoc to paper The modified WIS estimator (Eqs 11-15) is a statistically valid off-policy estimator.
    The paper replaces the importance ratio with rho/(1+rho) to control variance but provides no proof of unbiasedness or consistency; the estimator is not a recognized importance-sampling method.
  • domain assumption The physician behavior policy pi_b can be reliably estimated from the cohort and has positive support wherever the target policies have positive support.
    Supplemental Methods D assumes importance ratios pi_e/pi_b are well-defined; the paper does not describe how pi_b is estimated from sparse state-action counts, and zero counts in small clusters would make ratios infinite.
  • domain assumption The association between alignment with the model's recommendations and lower AKI is not driven by confounding.
    The Results section reports lower AKI prevalence in patients whose received doses aligned with the policy, with no adjustment for patient severity, case complexity, or other confounders.
  • domain assumption The test cohort is representative of the intended population despite covariate shift.
    Supplemental Table 2 shows much higher missingness for EtCO2, respiratory rate, and peak inspiratory pressure in the test period, and the handling of these missing values is not described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning optimal treatment strategies for intraoperative hypotension using deep reinforcement learning." pith.science (2026). https://pith.science/paper/DJT4O3PT

@misc{pith2026250521596,
  author       = {Pith},
  title        = {Pith review of: Learning optimal treatment strategies for intraoperative hypotension using deep reinforcement learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJT4O3PT}},
  note         = {Machine review of arXiv:2505.21596}
}
read the original abstract

Traditional methods of surgical decision making heavily rely on human experience and prompt actions, which are variable. A data-driven system generating treatment recommendations based on patient states can be a substantial asset in perioperative decision-making, as in cases of intraoperative hypotension, for which suboptimal management is associated with acute kidney injury (AKI), a common and morbid postoperative complication. We developed a Reinforcement Learning (RL) model to recommend optimum dose of intravenous (IV) fluid and vasopressors during surgery to avoid intraoperative hypotension and postoperative AKI. We retrospectively analyzed 50,021 surgeries from 42,547 adult patients who underwent major surgery at a quaternary care hospital between June 2014 and September 2020. Of these, 34,186 surgeries were used for model training and 15,835 surgeries were reserved for testing. We developed a Deep Q-Networks based RL model using 16 variables including intraoperative physiologic time series, total dose of IV fluid and vasopressors extracted for every 15-minute epoch. The model replicated 69% of physician's decisions for the dosage of vasopressors and proposed higher or lower dosage of vasopressors than received in 10% and 21% of the treatments, respectively. In terms of IV fluids, the model's recommendations were within 0.05 ml/kg/15 min of the actual dose in 41% of the cases, with higher or lower doses recommended for 27% and 32% of the treatments, respectively. The model resulted in a higher estimated policy value compared to the physicians' actual treatments, as well as random and zero-drug policies. AKI prevalence was the lowest in patients receiving medication dosages that aligned with model's decisions. Our findings suggest that implementation of the model's policy has the potential to reduce postoperative AKI and improve other outcomes driven by intraoperative hypotension.

Figures

Figures reproduced from arXiv: 2505.21596 by the authors.

Figure 2
Figure 2. Comparison of actions proposed by the trained agent (A) and taken by [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 5
Figure 5. Distribution of estimated values for physicians, agent, zero and random policies [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 52 canonical work pages

  1. [1]

    Long-term risk of mortality and acute kidney injury during hospitalization after major surgery

    Bihorac A, Yavas S, Subbiah S, et al. Long-term risk of mortality and acute kidney injury during hospitalization after major surgery. Annals of surgery. 2009;249(5):851-858

    work page 2009

  2. [2]

    Acute kidney injury in major abdominal surgery: incidence, risk factors, pathogenesis and outcomes

    Gameiro J, Fonseca JA, Neves M, Jorge S, Lopes JA. Acute kidney injury in major abdominal surgery: incidence, risk factors, pathogenesis and outcomes. Annals of intensive care. 2018;8:1-10

    work page 2018

  3. [3]

    Cardiac and vascular surgery–associated acute kidney injury: the 20th international consensus conference of the ADQI (acute disease quality initiative) group

    Nadim MK, Forni LG, Bihorac A, et al. Cardiac and vascular surgery–associated acute kidney injury: the 20th international consensus conference of the ADQI (acute disease quality initiative) group. Journal of the American Heart Association. 2018;7(11):e008834

    work page 2018

  4. [4]

    Postoperative acute kidney injury

    Park JT. Postoperative acute kidney injury. Korean journal of anesthesiology. 2017;70(3):258-266

    work page 2017

  5. [5]

    Postoperative acute kidney injury in adult non-cardiac surgery: joint consensus report of the Acute Disease Quality Initiative and PeriOperative Quality Initiative

    Prowle JR, Forni LG, Bell M, et al. Postoperative acute kidney injury in adult non-cardiac surgery: joint consensus report of the Acute Disease Quality Initiative and PeriOperative Quality Initiative. Nature Reviews Nephrology. 2021;17(9):605-618

    work page 2021

  6. [6]

    Perioperative acute kidney injury: prevention, early recognition, and supportive measures

    Romagnoli S, Ricci Z, Ronco C. Perioperative acute kidney injury: prevention, early recognition, and supportive measures. Nephron. 2018;140(2):105-110

    work page 2018

  7. [7]

    Epidemiology of surgery associated acute kidney injury (EPIS-AKI): a prospective international observational multi-center clinical study

    Zarbock A, Weiss R, Albert F, et al. Epidemiology of surgery associated acute kidney injury (EPIS-AKI): a prospective international observational multi-center clinical study. Intensive Care Medicine. 2023;49(12):1441-1455

    work page 2023

  8. [8]

    Biomarker-guided intervention to prevent acute kidney injury after major surgery: the prospective randomized BigpAK study

    Göcze I, Jauch D, Götz M, et al. Biomarker-guided intervention to prevent acute kidney injury after major surgery: the prospective randomized BigpAK study. In: LWW; 2018

    work page 2018

Show all 53 references
  1. [9]

    Prevention of cardiac surgery-associated AKI by implementing the KDIGO guidelines in high risk patients identified by biomarkers: the PrevAKI randomized controlled trial

    Meersch M, Schmidt C, Hoffmeier A, et al. Prevention of cardiac surgery-associated AKI by implementing the KDIGO guidelines in high risk patients identified by biomarkers: the PrevAKI randomized controlled trial. Intensive care medicine. 2017;43:1551-1561

    2017

  2. [10]

    Intraoperative hypotension and postoperative acute kidney injury: A systematic review

    Penev Y, Ruppert MM, Bilgili A, et al. Intraoperative hypotension and postoperative acute kidney injury: A systematic review. The American Journal of Surgery. 2024;232:45-53

    2024

  3. [11]

    Saugel B, Sander M, Katzer C, et al. Association of intraoperative hypotension and cumulative norepinephrine dose with postoperative acute kidney injury in patients having noncardiac surgery: a retrospective cohort analysis. British Journal of Anaesthesia. 2025;134(1):54-62

    2025

  4. [12]

    Intraoperative hypotension: Pathophysiology, clinical relevance, and therapeutic approaches

    Kouz K, Hoppe P, Briesenick L, Saugel B. Intraoperative hypotension: Pathophysiology, clinical relevance, and therapeutic approaches. Indian journal of anaesthesia. 2020;64(2):90-96

    2020

  5. [13]

    Effect of goal-directed haemodynamic therapy on postoperative complications in low–moderate risk surgical patients: a multicentre randomised controlled trial (FEDORA trial)

    Calvo-Vecino JM, Ripollés-Melchor J, Mythen M, et al. Effect of goal-directed haemodynamic therapy on postoperative complications in low–moderate risk surgical patients: a multicentre randomised controlled trial (FEDORA trial). British journal of anaesthesia. 2018;120(4):734-744

    2018

  6. [14]

    Salmasi V, Maheshwari K, Yang D, et al. Relationship between intraoperative hypotension, defined by either reduction from baseline or absolute thresholds, and acute kidney and myocardial injury after noncardiac surgery: a retrospective cohort analysis. Anesthesiology. 2017;126...

    2017

  7. [15]

    Wijnberge M, Geerts BF, Hol L, et al. Effect of a machine learning–derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the HYPE randomized clinical trial. Jama. 202...

    2020

  8. [16]

    Reinforcement learning in healthcare: A survey

    Yu C, Liu J, Nemati S, Yin G. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR). 2021;55(1):1-36

    2021

  9. [17]

    Reinforcement learning in surgery

    Datta S, Li Y, Ruppert MM, et al. Reinforcement learning in surgery. Surgery. 2021;170(1):329-332. 23

    2021

  10. [18]

    Reinforcement Learning for Clinical Applications

    Khezeli K, Siegel S, Shickel B, Ozrazgat-Baslanti T, Bihorac A, Rashidi P. Reinforcement Learning for Clinical Applications. Clinical Journal of the American Society of Nephrology. 2023;18(4):521-523

    2023

  11. [19]

    Reinforcement learning for clinical decision support in critical care: comprehensive review

    Liu S, See KC, Ngiam KY, Celi LA, Sun X, Feng M. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet research. 2020;22(7):e18477

    2020

  12. [20]

    The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care

    Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine. 2018;24(11):1716-1720

    2018

  13. [21]

    Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment

    Nanayakkara T, Clermont G, Langmead CJ, Swigon D. Unifying cardiovascular modelling with deep reinforcement learning for uncertainty aware control of sepsis treatment. PLOS Digital Health. 2022;1(2):e0000012

    2022

  14. [22]

    Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning

    Peng X, Ding Y, Wihl D, et al. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. Paper presented at: AMIA Annual Symposium Proceedings2018

  15. [23]

    Optimal vasopressin initiation in septic shock: the OVISS reinforcement learning study

    Kalimouttou A, Kennedy JN, Feng J, et al. Optimal vasopressin initiation in septic shock: the OVISS reinforcement learning study. JAMA. 2025

    2025

  16. [24]

    Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units

    Yu C, Ren G, Dong Y. Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC medical informatics and decision making. 2020;20:1-8

    2020

  17. [25]

    Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep Q networks

    Lopez-Martinez D, Eschenfeldt P, Ostvar S, Ingram M, Hur C, Picard R. Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep Q networks. Paper presented at: 2019 41st annual international conference of the IEEE engineering...

    2019

  18. [26]

    A dual mode adaptive basal-bolus advisor based on reinforcement learning

    Sun Q, Jankovic MV, Budzinski J, et al. A dual mode adaptive basal-bolus advisor based on reinforcement learning. IEEE journal of biomedical and health informatics. 2018;23(6):2633-2641

    2018

  19. [27]

    Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients

    Lee HY, Chung S, Hyeon D, et al. Reinforcement learning model for optimizing dexmedetomidine dosing to prevent delirium in critically ill patients. npj Digital Medicine. 2024;7(1):325

    2024

  20. [28]

    An interpretable RL framework for pre-deployment modeling in ICU hypotension management

    Zhang K, Wang H, Du J, et al. An interpretable RL framework for pre-deployment modeling in ICU hypotension management. npj Digital Medicine. 2022;5(1):173

    2022

  21. [29]

    Learning to treat hypotensive episodes in sepsis patients using a counterfactual reasoning framework

    Jeter R, Lehman L-W, Josef C, Shashikumar S, Nemati S. Learning to treat hypotensive episodes in sepsis patients using a counterfactual reasoning framework. medRxiv. 2021:2021.2003. 2003.21252863

    2021

  22. [30]

    Clinician-in-the-loop decision making: Reinforcement learning with near-optimal set-valued policies

    Tang S, Modi A, Sjoding M, Wiens J. Clinician-in-the-loop decision making: Reinforcement learning with near-optimal set-valued policies. Paper presented at: International Conference on Machine Learning2020

  23. [31]

    Identifying distinct, effective treatments for acute hypotension with SODA-RL: safely optimized diverse accurate reinforcement learning

    Futoma J, Masood MA, Doshi-Velez F. Identifying distinct, effective treatments for acute hypotension with SODA-RL: safely optimized diverse accurate reinforcement learning. AMIA summits on translational science proceedings. 2020;2020:181

    2020

  24. [32]

    MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery

    Bihorac A, Ozrazgat-Baslanti T, Ebadi A, et al. MySurgeryRisk: development and validation of a machine-learning risk algorithm for major complications and death after surgery. Annals of surgery. 2019;269(4):652-662

    2019

  25. [33]

    Kidney disease: Improving global outcomes (KDIGO) acute kidney injury work group

    Kellum JA, Lameire N, Aspelin P, et al. Kidney disease: Improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury. Kidney International Supplements. 2012;2(1):1-138

    2012

  26. [34]

    New creatinine-and cystatin C–based equations to estimate GFR without race

    Inker LA, Eneanya ND, Coresh J, et al. New creatinine-and cystatin C–based equations to estimate GFR without race. New England Journal of Medicine. 2021;385(19):1737- 1749. 24

    2021

  27. [35]

    Development and validation of a race- agnostic computable phenotype for kidney health in adult hospitalized patients

    Ozrazgat-Baslanti T, Ren Y, Adiyeke E, et al. Development and validation of a race- agnostic computable phenotype for kidney health in adult hospitalized patients. Plos one. 2024;19(4):e0299332

    2024

  28. [36]

    A new method of classifying prognostic comorbidity in longitudinal studies: development and validation

    Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of chronic diseases. 1987;40(5):373-383

    1987

  29. [37]

    A unified approach to interpreting model predictions

    Lundberg SM, Lee S-I. A unified approach to interpreting model predictions. Advances in neural information processing systems. 2017;30

    2017

  30. [38]

    Learning important features through propagating activation differences

    Shrikumar A, Greenside P, Kundaje A. Learning important features through propagating activation differences. Paper presented at: International conference on machine learning2017

  31. [39]

    Bootstrapping with models: Confidence intervals for off- policy evaluation

    Hanna J, Stone P, Niekum S. Bootstrapping with models: Confidence intervals for off- policy evaluation. Paper presented at: Proceedings of the AAAI Conference on Artificial Intelligence2017

  32. [40]

    Data-efficient off-policy policy evaluation for reinforcement learning

    Thomas P, Brunskill E. Data-efficient off-policy policy evaluation for reinforcement learning. Paper presented at: International conference on machine learning2016

  33. [41]

    Vasopressor dose equivalence: A scoping review and suggested formula

    Goradia S, Sardaneh AA, Narayan SW, Penm J, Patanwala AE. Vasopressor dose equivalence: A scoping review and suggested formula. Journal of Critical Care. 2021;61:233-240

    2021

  34. [42]

    Incidence of intraoperative hypotension during non-cardiac surgery in community anesthesia practice: a retrospective observational analysis

    Saasouh W, Christensen AL, Xing F, et al. Incidence of intraoperative hypotension during non-cardiac surgery in community anesthesia practice: a retrospective observational analysis. Perioperative Medicine. 2023;12(1):29

    2023

  35. [43]

    Association of intraoperative hypotension with acute kidney injury after elective noncardiac surgery

    Sun LY, Wijeysundera DN, Tait GA, Beattie WS. Association of intraoperative hypotension with acute kidney injury after elective noncardiac surgery. Anesthesiology. 2015;123(3):515-523

    2015

  36. [44]

    Hospital costs associated with intraoperative hypotension among non-cardiac surgical patients in the US: a simulation model

    Keuffel EL, Rizzo J, Stevens M, Gunnarsson C, Maheshwari K. Hospital costs associated with intraoperative hypotension among non-cardiac surgical patients in the US: a simulation model. Journal of Medical Economics. 2019;22(7):645-651

    2019

  37. [45]

    Artificial intelligence and surgical decision- making

    Loftus TJ, Tighe PJ, Filiberto AC, et al. Artificial intelligence and surgical decision- making. JAMA surgery. 2020;155(2):148-158

    2020

  38. [46]

    Identifying decision points for safe and interpretable reinforcement learning in hypotension treatment

    Zhang K, Wang Y, Du J, et al. Identifying decision points for safe and interpretable reinforcement learning in hypotension treatment. arXiv preprint arXiv:210103309. 2021

    2021

  39. [47]

    Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care

    Peine A, Hallawa A, Bickenbach J, et al. Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care. NPJ Digit Med. 2021;4(1):32

    2021

  40. [48]

    A reinforcement learning approach to weaning of mechanical ventilation in intensive care units

    Prasad N, Cheng L-F, Chivers C, Draugelis M, Engelhardt BE. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. arXiv preprint arXiv:170406300. 2017. 25 Supplemental Materials Learning optimal treatment strategies for intraoperative ...

    2017

  41. [49]

    (Supplemental Figure 1)

    the surgery was < 60 minutes, 6) 3-day or 7-day acute kidney injury (AKI) status was missing due to insufficient serum creatinine data available, 7) cardiac surgeries. (Supplemental Figure 1). B. Dosage pre-processing and action space Intravenous fluids (IV) included boluses a...

  42. [50]

    S is the state of a patient (in our model, it contains 16 clinical variables)

  43. [51]

    A is the finite set of actions for state S (in our model, the doses intravenous fluids and vasopressors are discretized into 25 actions) 28

  44. [52]

    R(s′) is the immediate reward received for transitioning to next state S′

  45. [53]

    Our model operated on 15-minute time intervals, within which multiple measurements were recorded

    Γ is the discounting factor, that indicates the decay of influences for future rewards than an immediate reward. Our model operated on 15-minute time intervals, within which multiple measurements were recorded. We considered resampling by averaging all measurements within each...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.