REVIEW 3 major objections 4 minor 17 references
Balancing utility and cost in dynamic treatment regimes
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Backward induction over four contrast functions yields a treatment regime that maximizes expected outcome minus the cost of treatments and diagnostic tests, and a linear-model version learns it with regret that shrinks as the sample grows.
desk verdict A genuine extension of Q-learning to covariate-assessment decisions, with honest regret bounds that separate estimation from misspecification—but the practical claim is only tested under perfect linearity. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is a four-stage backward induction over contrast functions $\Delta_{2t}$, $\Delta_{2c}$, $\Delta_{1t}$, and $\Delta_{1c}$, which measure the profit advantage of the second treatment, the second covariate set, the first treatment, and the first covariate set over their respective baselines. The algorithm estimates the unrestricted versions of these contrasts with cross-fitted residual-on-residual (R-learner) regressions and then projects them onto the actually assessed covariate subsets with nested regressions; the sign or argmax of each estimated contrast is the learned decision rule at that stage.
What would settle it
Generate data from a two-stage model whose true contrast functions are strongly nonlinear, such as a threshold effect in the second-stage outcome, run Balanced Q-learning with linear working models at increasing sample sizes, and compare the achieved profit with the oracle profit; if the regret does not shrink as $n$ grows, the vanishing-misspecification condition in Assumption 6 is the binding assumption.
Extended reading notes
Core claim
The central discovery is that the usual Q-learning dynamic programming step can be duplicated for measurement decisions: at each of two stages the decision-maker chooses both which additional covariates to collect and which treatment to give, and the optimal joint policy is obtained by backward induction over Q-functions for the second treatment, the second covariate set, the first treatment, and the first covariate set. Theorem 3.1 shows that the regime selecting the argmax at each of these four steps dominates every other two-stage regime in expected profit. Theorem 4.2 shows that when the contrast functions are estimated through linear working models, the profit gap between the estimated and the optimal regime is $O_p\{(1/\sqrt{n})^{r+1}+\epsilon_n^{r+1}\}$, where $r$ is a margin exponent and $\epsilon_n$ is the misspecification error of the linear models; hence under correct specification the estimated regime converges to the global profit optimum.
Load-bearing premise
The practical claim rests on Assumption 6: the linear working models for the four contrast functions must be accurate enough that the misspecification error $\epsilon_n$ shrinks to zero; if treatment effects are strongly nonlinear, the learned balanced rule has no guarantee of approaching the profit-maximizing regime.
Editorial extensions
If this is right
- A clinic using the learned regime will adaptively order fewer or cheaper diagnostic tests for patients whose baseline profile makes the extra information unlikely to change treatment.
- When the contrast functions are linear, the regret bound gives a fast convergence rate, so the method attains the profit-optimal regime in large samples.
- The asymptotic normality result supports plug-in confidence intervals for the contrast parameters, allowing a clinician to test whether a specific covariate or test contributes enough to utility to justify its cost.
- As the cost weight or the prices of tests increase, the learned regime shifts from large covariate sets to smaller, cheaper ones, preserving most of the outcome benefit while cutting expenditure.
Reading between the lines
- The same backward-induction accounting could be extended to more than two stages; each new stage adds a pair of contrast functions, but the misspecification error and the number of candidate subsets grow, so sparse or greedy search would be needed in practice.
- If covariate measurement itself can change patient behavior or outcome (for example via anxiety or labeling effects), the paper's profit formula would need an additional direct-effect term; this is empirically testable in a randomized comparison of identical information collected under different disclosure protocols.
- The method's design suggests a natural decision-support use: list candidate test bundles with prices on a hospital menu, fit the contrasts from historical records, and the learned rule becomes a personalized test-ordering guideline.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Balanced Q-learning (BQL) for two-stage, binary-treatment dynamic treatment regimes in which both treatment assignment and covariate assessment incur costs. The authors define stage-wise Q-functions that alternately optimize the choice of which covariates to assess and which treatment to assign, prove in Theorem 3.1 under standard identification conditions that backward induction on these Q-functions maximizes expected profit, and propose a cross-fitted algorithm that estimates linear contrast functions via R-learning and nested regressions. The asymptotic results give root-n normality for the linear contrast parameters (Theorem 4.1) and a regret bound (Theorem 4.2) of order Op{(1/√n)^{r+1} + ε_n^{r+1}}, where ε_n is a uniform approximation error between the true contrast functions and the linear working models. The method is illustrated by simulations and by a MIMIC-III sepsis application comparing BQL with high-dimensional Q-learning (HDQ) and robust Q-learning (RQL).
Significance. If the results hold, the paper makes a useful contribution by extending Q-learning to settings in which covariate collection is costly and the choice of which covariates to assess is itself part of the treatment rule. The paper has clear strengths: Theorem 3.1 is a clean dynamic-programming argument under standard causal identification; Theorem 4.2 explicitly separates estimation error from misspecification error; and the target linear parameters are defined as population-level best linear approximations, so the theory is internally coherent and does not assume the working models are correctly specified ex ante. The detailed proofs and explicit variance expressions in Theorem S4.1 are valuable. The main weakness is that the practical claim of recovering the optimal utility-cost trade-off rests on Assumption 6, and the simulation evidence never exercises that assumption.
major comments (3)
- [Assumption 6; Theorem 4.2; Section 5.1] The regret bound in Theorem 4.2 is Op{(1/√n)^{r+1} + ε_n^{r+1}}, so the learned regime converges to the global profit-maximizing regime only when the uniform misspecification error ε_n tends to zero. This is not a mild smoothness condition. Because the target coefficients α*, β*, γ*, and δ* are least-squares projections, under misspecification the sign of the estimated linear score is not guaranteed to agree with the sign of the true contrast function even at the population level, and Theorem 4.2 leaves a constant regret floor. The simulations in Section 5.1 (Models 1–3 and Models 4–7 in the supplement) all generate outcomes with linear main effects and interactions, so ε_n = 0 by construction and Assumption 6 is never actually tested. To support the paper's practical claim, the authors should either add simulations with demonstrably nonlinear contrast functions (and report regret or policy value as a function of misspecification) or explicitly reframe the contribution as optimizing within a linear class, with Theorem 4.2 quantifying the approximation penalty.
- [Algorithm 1, Step 7] Algorithm 1 Step 7 says 'Repeat Steps 3–7' within Step 7 itself, which is self-referential and makes the procedure non-terminating as written. The intended instruction is almost certainly 'Repeat Steps 3–6' before constructing the cross-fitted pseudo-outcome. Also, Step 8 defines r-hat_{f1,i}^{-k} using bY^{1t}_{j1i} from Step 6 rather than the cross-fitted version bY^{1t,-k}_{j1i}; this needs clarification so the estimator is reproducible.
- [Section 5.2, Tables 2–4] The reported profit advantages of BQL over RQL and HDQ are averaged over 100 random splits, but no standard errors, confidence intervals, or interquartile ranges are given. Since the differences are often modest (for example, at τ=500 the profit values are 0.8253 for BQL versus 0.7441 for RQL and 0.7237 for HDQ) and the utility differences are even smaller, the reader cannot assess whether the advantage is systematic or within sampling variability; please report variability across the splits.
minor comments (4)
- [Assumption 6] In the left-hand side of Assumption 6, the second term writes Δ2c(S̄_{l2i}, j1, a1, j1) but the linear term is S̄_{l2i}^⊤ β*_{j1 a1 j2}; the first index should presumably be j2.
- [Section 2] The phrase 'potential value of of the second-stage covariate' contains a duplicated 'of'.
- [Section 3 and Figure 1] The notation S̄_{l2}, S̄_{l f 2}, S̄_{j2}, and related subsets is easy to confuse; a compact table defining all covariate subsets and their index sets would improve readability.
- [Section 5.1 and Figure 2] The selection-frequency and profit plots would be easier to interpret with Monte Carlo standard errors or shaded confidence bands, especially for the profit panels where differences between methods are small in some λ ranges.
Circularity Check
No circular derivation: the optimality theorem is standard backward induction, and the regret bound explicitly separates estimation error from misspecification error rather than renaming a fit as a prediction.
full rationale
The paper's derivation chain is self-contained and non-circular. Theorem 3.1 defines Q-functions recursively in a backward-induction fashion and proves that pointwise maximization of these Q-functions yields a regime whose profit dominates any other regime; this is a standard dynamic-programming argument under Assumption 1, and the profit decomposition in the proof tracks the nonnegative gap terms G2t, G2c, G1t explicitly. The estimation section then targets population-level best linear approximations (alpha*, beta*, gamma*, delta*) of the contrast functions, and Theorem 4.2 states regret = Op((1/sqrt n)^(r+1) + eps_n^(r+1)), where eps_n is the misspecification error defined in Assumption 6. This is not a fitted quantity presented as a prediction: the paper explicitly says the estimated linear regime reaches the global optimum only when eps_n -> 0, and it does not claim otherwise. The simulations use linear outcome models, so eps_n = 0 by construction, but that is a limitation of the numerical evidence, not a circularity in the theoretical claim. There are no load-bearing self-citations: the relevant references (R-learner, margin conditions, cross-fitting, Q-learning) are independent external results. One literal self-reference appears in Algorithm 1 Step 7, which says 'Repeat Steps 3-7' inside Step 7 itself; this is almost certainly a typo for Steps 3-6 and is a reproducibility issue rather than a circular derivation. No step of the paper reduces a claimed output to its own input by definition.
Assumptions & free parameters
assumptions (7)
- domain assumption Consistency, sequential ignorability, and positivity (Assumption 1).
- domain assumption Covariate assessments have no direct effect on the outcome; they affect Y only through treatment assignments.
- domain assumption Distributional invariance between observed and test populations for PS1, PS2|S1,A1, and PY|S1,A1,S2,A2, plus full covariate availability in the observed data.
- standard math Boundedness and positive variance conditions (Assumptions 2 and 4).
- domain assumption Nuisance estimators satisfy the convergence rate conditions in Assumption 3.
- domain assumption Margin conditions and bounded linear-model misspecification (Assumptions 5 and 6).
- domain assumption Cost functions and candidate assessment sets are known to the analyst.
Cite this review
Pith. "Pith review of Balancing utility and cost in dynamic treatment regimes." pith.science (2026). https://pith.science/paper/YX4HTIFY
@misc{pith2026250717360,
author = {Pith},
title = {Pith review of: Balancing utility and cost in dynamic treatment regimes},
year = {2026},
howpublished = {\url{https://pith.science/paper/YX4HTIFY}},
note = {Machine review of arXiv:2507.17360}
}
read the original abstract
Dynamic treatment regimes (DTRs) are personalized, adaptive strategies designed to guide the sequential allocation of treatments based on individual characteristics over time. Before each treatment assignment, covariate information is collected to refine treatment decisions and enhance their effectiveness. The more information we gather, the more precise our decisions can be. However, this also leads to higher costs during the data collection phase. In this work, we propose a balanced Q-learning method that strikes a balance between the utility of the DTR and the costs associated with both treatment assignment and covariate assessment. The performance of the proposed method is demonstrated through extensive numerical studies, including simulations and a real-data application to the MIMIC-III database.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
= E(Y 1c j1 − Y 1c jf 1 | Sl1). We specify the working model for ∆ 1c(sl1, j1) 43 as: ∆1c∗(sl1, j1) = s⊤ l1δ∗ j1, δ∗ j1 = arg min δ E{(Y 1c∗ j1 − Y 1c∗ jf 1 − S⊤ l1δ)2}. (S3.6) S4 Proof of the main results Proof of Theorem 3.1. Recall that J1 = π1c(Sl1), A1 = π1t(S ¯J1, J1), J2 = π2c(S ¯L2, J1, π1c(Sl1)), and A2 = π2t(S ¯J2, J1, π1t(S ¯J1, J1), J2), for s...
work page 2021
-
[2]
> 0)} −∆C,2c j2 . To approximate the restricted contrast function ¯∆2c(s¯l2, j1, a1, j2), we adopt a linear working model: ¯∆2c∗(s¯l f 2 , j1, a1, j2) = (s¯l f 2 , a1)⊤ ¯β∗ j1j2, ¯β∗ j1j2 = arg min β E{(Y 2c j1j2 − (S¯l f 2 , A1)⊤β)2}. Consider a linear working model for the unrestricted contrast ∆ 2c(s¯l f 2 , j1, a1, j2): ∆2c∗(s¯l2, j1, a1, j2) = s⊤ ¯l2...
-
[3]
| S¯l f 2 , A1] + X j2∈J2 I{π2c∗(S¯l2, j1, A1) = j2} ¯∆2c∗(S¯l f 2 , j1, A1, j2) − C 1t(A1). Moreover, note that ¯Q2t∗{¯S2, A1, π2t∗(S¯jf 2 , j1, A1, jf 2)} = ¯Q2t∗(¯S2, A1, 0) + ¯∆2t∗(¯S2, A1)I{∆2t∗(S¯jf 2 , j1, A1, jf
-
[4]
> 0} = f2(¯S2, A1) − C 2t(0) − g2(¯S2, A1)∆C,2t + ¯∆2t∗(¯S2, A1)[I{∆2t∗(S¯jf 2 , j1, A1, jf
-
[5]
> 0} −g2(¯S2, A1)]. This motivates the definition of Y 1t∗ j1 = Y − C 2t(A2) + (¯S2, A1)⊤ ¯α∗{I(S⊤ ¯jf 2 α∗ j1A1jf 2 > 0) − A2} − C 2c(jf
-
[6]
42 It follows that E(Y 1t∗ j1 − eY 1t∗ j1 | S1, A1) = 0
+ X j2∈J2 I(j2 = arg maxj′ 2 S⊤ ¯l2β∗ j1A1j′ 2 )(S¯l f 2 , A1i)⊤ ¯β∗ j1j2 − C 1t(A1). 42 It follows that E(Y 1t∗ j1 − eY 1t∗ j1 | S1, A1) = 0. The contrast can be estimated via the following model equivalently: Y 1t∗ j1 = ¯Q1t∗(S1, j1, 0) + A1{Q1t∗(S1, j1, 1) − ¯Q1t∗(S1, j1, 0)} + ε1t∗, E (ε1t∗ | S1, A1) = 0. Let g1(s1) = E(A1 | S1 = s1) denote the propen...
-
[7]
such that j2 ̸= j′ 2 and any δ ∈ (0, 1), by the convergence of bβj1a1j2 and bβj1a1j′ 2, there exist a constant C3δ depends on δ such that P (Cδ j2j′ 2 ) > 1 − δ, where Cδ j2j′ 2 = ( 1X a1=0 (∥ bβj1a1j2 − β∗ j1a1j2∥2 + ∥ bβj1a1j′ 2 − β∗ j1a1j′ 2 ∥2) ≤ C3δn−1/2 ) . Conditional on Cδ j2j′ 2 , there exists a constant C ′′′ such that |Ik|−1 X i∈Ik I( bBj1j2i ∩...
-
[8]
Consequently, P j2 S⊤ ¯L2 β∗ J1 ˆπ1tj2 I(FJ1j2) = max j2 S⊤ ¯L2 β∗ J1 ˆπ1tj2 ≥ P J2 S⊤ ¯L2 β∗ J1 ˆπ1tj2 I(DJ1j2), which implies that S⊤ ¯L2 β∗ J1 ˆπ1tj2 {I(DJ1j2) − I(FJ1j2)} ≤0. Moreover, since both ∪j2∈J2DJ1j2 and ∪j2∈J2FJ1j2 are full sets, as well as DJ1j2 ∩ DJ1j′ 2 and FJ1j2 ∩ FJ1j′ 2 are empty sets for j2 ̸= j′ 2, T4 ≤ E X j2∈J2 {∆2c(S ¯L2, J1, ˆπ1t,...
Show all 17 references
-
[9]
− S⊤ ¯L2β∗ J1 ˆπ1tj′ 2 }I(FJ1j′ 2) I(DJ1j2) = E X j2∈J2 {∆2c(S ¯L2, J1, ˆπ1t, j2) − S⊤ ¯L2β∗ J1 ˆπ1tj2} − X j′ 2∈J2 {∆2c(S ¯L2, J1, ˆπ1t, j′
-
[10]
− S⊤ ¯L2β∗ J1 ˆπ1tj′ 2 }I(FJ1j′ 2) I(DJ1j2) = E X j2∈J2 X j′ 2̸=J2 {∆2c(S ¯L2, J1, ˆπ1t, j2) − S⊤ ¯L2β∗ J1 ˆπ1tj2 − ∆2c(S ¯L2, J1, ˆπ1t, j′
-
[11]
+ S⊤ ¯L2β∗ J1 ˆπ1tj′ 2 }{I(DJ1j2 ∩ FJ1j′ 2)} ≤ E X j2∈J2 X j′ 2̸=j2 2ϵn · I(DJ1j2 ∩ FJ1j′ 2) . For each pairs j2 ̸= j′ 2, E{I(DJ1j2 ∩ FJ1j′ 2)} ≤P {∆2c(S ¯L2, J1, ˆπ1t, j2) ≥ ∆2c(S ¯L2, J1, ˆπ1t, j′ 2), S⊤ ¯L2β∗ J1 ˆπ1tj′ 2 ≥ S⊤ ¯L2β∗ J1 ˆπ1tj2} ≤ P {|S⊤ ¯L2(β∗ J1 ˆπ1tj2 − β∗ ...
-
[12]
− S⊤ ¯L2(β∗ J1 ˆπ1tj2 − bβJ1 ˆπ1tj2)|} ≤ E |S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − bβJ1 ˆπ1tj′
-
[13]
− S⊤ ¯L2(β∗ J1 ˆπ1tj2 − bβJ1 ˆπ1tj2)| × I{|S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − β∗ J1 ˆπ1tj2)| ≤ |S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − bβJ1 ˆπ1tj′
-
[14]
− S⊤ ¯L2(β∗ J1 ˆπ1tj2 − bβJ1 ˆπ1tj2)|} = O (∥ bβJ1 ˆπ1tj′ 2 − β∗ J1 ˆπ1tj′ 2 ∥2 + ∥ bβJ1 ˆπ1tj2 − β∗ J1 ˆπ1tj2∥2) × P {|S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − β∗ J1 ˆπ1tj2)| ≤ |S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − bβJ1 ˆπ1tj′
-
[15]
Hence T5 = Op{(1/√n)r+1}
− S⊤ ¯L2(β∗ J1 ˆπ1tj2 − bβJ1 ˆπ1tj2)|} = Op{(1/√n)r+1}, where the fourth equality is by ∥ bβj1a1j2 − β∗ j2a1j2∥2 = Op(n−1/2) for any j1, j2, and a1, as well 74 as Assumptions 2, and 5 with r ≥ 1. Hence T5 = Op{(1/√n)r+1}. Similarily, T6 = E X j2∈J2 X j′ 2̸=J2 {∆2c(S ¯L2, J1, ˆ...
-
[16]
+ S⊤ ¯L2β∗ J1 ˆπ1tj′ 2 }I(FJ1j2 ∩ EJ1j′ 2) = O ϵn · X j2∈J2 X j′ 2̸=j2 P {|S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − β∗ J1 ˆπ1tj2)| ≤ |S⊤ ¯L2(β∗ J1 ˆπ1tj′ 2 − bβJ1 ˆπ1tj′
-
[17]
Consequently, ˆG2c = Op{(1/√n)r+1 + ϵn(1/√n)r + ϵr+1 n }
− S⊤ ¯L2(β∗ J1 ˆπ1tj2 − bβJ1 ˆπ1tj2)|} = O{ϵn(1/√n)r}. Consequently, ˆG2c = Op{(1/√n)r+1 + ϵn(1/√n)r + ϵr+1 n }. Repeating the argument used to control the term ˆG2t, we have ˆG1t ≤ E ∆1t(S ¯J1, J1){I(∆1t(S ¯J1, J1) > 0) − I(S⊤ ¯J1 bγJ1 > 0)} = Op{(1/√n)r+1 + ϵn(1/√n)r + ϵr+1 ...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.