{"id":"dda887d3-9eca-4483-b072-4b7ee55959be","arxiv_id":"2507.17360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A two-stage Q-learning algorithm jointly chooses treatments and which covariates to collect, maximizing expected outcome minus assessment and treatment costs.","lead":"Balanced Q-learning decides not just what treatment to give but which diagnostic tests are worth paying for at each stage. The method maximizes expected health benefit minus treatment and testing costs, and is tested on ICU sepsis data from MIMIC-III.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.2's guarantee of global optimality requires ε_n→0 uniformly (Assumption 6); all simulations use exactly linear contrasts (ε_n=0), so the practical claim that BQL balances utility and cost is untested under misspecification.","rationale":"I read Theorem 3.1 and the proof in S4; the backward induction is standard and no algebraic error surfaced. Theorem 4.2 is stated honestly as conditional on Assumption 6, so the mathematics is not false. The concern is that the paper's simulations are constructed so that ε_n=0 (all outcome models are linear), and the MIMIC analysis cannot verify ε_n, leaving the method's advertised optimality unsupported outside the exactly-linear case. This is exactly the reader's weakest assumption, and it explains the CONDITIONAL verdict. I would not reject: the theoretical contribution is coherent, and the conditional regret bound is a real result. I would require either (a) a misspecification simulation, or (b) a statement that the practical claim requires ε_n small, before accepting the real-data conclusions. I also flag Algorithm 1 Step 7's self-reference as a reproducibility issue, but it is fixable and not the main statistical vulnerability.","tokens_in":57126,"tokens_out":10018,"duration_ms":118672,"concrete_test":"Simulate a two-stage DGP like Model 3 but with a nonlinear second-stage contrast, e.g., include S2,2^2 and an A1×S2,3 interaction in the outcome while BQL still fits linear working models. For n ∈ {500, 2000, 8000}, estimate ε_n by the oracle L∞ difference between the true contrast and its best linear approximation, and estimate regret against the oracle optimal regime using a large test set. If regret does not decrease with n and track ε_n^(r+1) (about ε_n^2 for r=1), then Assumption 6 is doing the work and the paper's practical claim fails outside exactly linear settings. As a secondary check, replace Algorithm 1 Step 7 with 'Repeat Steps 3–6 on D^{-k}' and confirm that the cross-fitted pseudo-outcomes used in Theorem 4.1's proof are what the algorithm computes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is Assumption 6: the restricted and unrestricted contrast functions Δ2t, Δ2c, Δ1t, Δ1c must be approximated uniformly, almost surely, by the linear working models with error ε_n. Theorem 4.2 gives regret Op{(1/√n)^(r+1)+ε_n^(r+1)}, so the learned regime converges to the globally optimal regime only if ε_n→0. This is not a mild smoothness condition: because the target coefficients α*, β*, γ*, δ* are least-squares projections, under misspecification the sign of the projected linear score need not maximize profit even among linear rules; the regret floor is a constant depending on ε_n. The paper's simulations (Models 1–3 and S1) all generate outcomes with linear main-effect and interaction terms, so ε_n=0 by construction; Assumption 6 is never exercised. In the MIMIC application there is no ground truth and no error bars (Tables 2–4), so the reported profit advantage of BQL over RQL/HDQ could be an artifact of the linearity assumption. The theory is internally consistent, but the central practical claim—that BQL actually recovers the utility-cost optimal regime in real data—stands or falls on an unverified uniform approximation. A secondary implementation issue is Algorithm 1 Step 7 ('Repeat Steps 3–7'), which is self-referential; this should be corrected (likely 'Steps 3–6') before the estimator is reproducible, but it does not change the statistical concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Balanced Q-learning (BQL) for two-stage, binary-treatment dynamic treatment regimes in which both treatment assignment and covariate assessment incur costs. The authors define stage-wise Q-functions that alternately optimize the choice of which covariates to assess and which treatment to assign, prove in Theorem 3.1 under standard identification conditions that backward induction on these Q-functions maximizes expected profit, and propose a cross-fitted algorithm that estimates linear contrast functions via R-learning and nested regressions. The asymptotic results give root-n normality for the linear contrast parameters (Theorem 4.1) and a regret bound (Theorem 4.2) of order Op{(1/√n)^{r+1} + ε_n^{r+1}}, where ε_n is a uniform approximation error between the true contrast functions and the linear working models. The method is illustrated by simulations and by a MIMIC-III sepsis application comparing BQL with high-dimensional Q-learning (HDQ) and robust Q-learning (RQL).","tokens_in":57390,"tokens_out":5690,"duration_ms":58936,"significance":"If the results hold, the paper makes a useful contribution by extending Q-learning to settings in which covariate collection is costly and the choice of which covariates to assess is itself part of the treatment rule. The paper has clear strengths: Theorem 3.1 is a clean dynamic-programming argument under standard causal identification; Theorem 4.2 explicitly separates estimation error from misspecification error; and the target linear parameters are defined as population-level best linear approximations, so the theory is internally coherent and does not assume the working models are correctly specified ex ante. The detailed proofs and explicit variance expressions in Theorem S4.1 are valuable. The main weakness is that the practical claim of recovering the optimal utility-cost trade-off rests on Assumption 6, and the simulation evidence never exercises that assumption.","major_comments":[{"comment":"The regret bound in Theorem 4.2 is Op{(1/√n)^{r+1} + ε_n^{r+1}}, so the learned regime converges to the global profit-maximizing regime only when the uniform misspecification error ε_n tends to zero. This is not a mild smoothness condition. Because the target coefficients α*, β*, γ*, and δ* are least-squares projections, under misspecification the sign of the estimated linear score is not guaranteed to agree with the sign of the true contrast function even at the population level, and Theorem 4.2 leaves a constant regret floor. The simulations in Section 5.1 (Models 1–3 and Models 4–7 in the supplement) all generate outcomes with linear main effects and interactions, so ε_n = 0 by construction and Assumption 6 is never actually tested. To support the paper's practical claim, the authors should either add simulations with demonstrably nonlinear contrast functions (and report regret or policy value as a function of misspecification) or explicitly reframe the contribution as optimizing within a linear class, with Theorem 4.2 quantifying the approximation penalty.","section":"Assumption 6; Theorem 4.2; Section 5.1"},{"comment":"Algorithm 1 Step 7 says 'Repeat Steps 3–7' within Step 7 itself, which is self-referential and makes the procedure non-terminating as written. The intended instruction is almost certainly 'Repeat Steps 3–6' before constructing the cross-fitted pseudo-outcome. Also, Step 8 defines r-hat_{f1,i}^{-k} using bY^{1t}_{j1i} from Step 6 rather than the cross-fitted version bY^{1t,-k}_{j1i}; this needs clarification so the estimator is reproducible.","section":"Algorithm 1, Step 7"},{"comment":"The reported profit advantages of BQL over RQL and HDQ are averaged over 100 random splits, but no standard errors, confidence intervals, or interquartile ranges are given. Since the differences are often modest (for example, at τ=500 the profit values are 0.8253 for BQL versus 0.7441 for RQL and 0.7237 for HDQ) and the utility differences are even smaller, the reader cannot assess whether the advantage is systematic or within sampling variability; please report variability across the splits.","section":"Section 5.2, Tables 2–4"}],"minor_comments":[{"comment":"In the left-hand side of Assumption 6, the second term writes Δ2c(S̄_{l2i}, j1, a1, j1) but the linear term is S̄_{l2i}^⊤ β*_{j1 a1 j2}; the first index should presumably be j2.","section":"Assumption 6"},{"comment":"The phrase 'potential value of of the second-stage covariate' contains a duplicated 'of'.","section":"Section 2"},{"comment":"The notation S̄_{l2}, S̄_{l f 2}, S̄_{j2}, and related subsets is easy to confuse; a compact table defining all covariate subsets and their index sets would improve readability.","section":"Section 3 and Figure 1"},{"comment":"The selection-frequency and profit plots would be easier to interpret with Monte Carlo standard errors or shaded confidence bands, especially for the profit panels where differences between methods are small in some λ ranges.","section":"Section 5.1 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for a statistical methodology journal, and the core theory appears sound. My main concerns are the strength of the empirical support for the practical optimality claim under misspecification and the self-referential step in Algorithm 1. The revision should address these before publication; I do not see evidence of circularity or other integrity problems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Net: this is a real extension of Q-learning to decisions about which covariates to collect, not just which treatment to prescribe. Theorem 3.1 is a clean backward-induction argument under standard identification, and Theorem 4.2 is honest about what it needs: estimation error and misspecification error enter separately, and the regret floor vanishes only when epsilon_n does. The paper does not oversell this—the discussion after Theorem 4.2 says so explicitly. That is the right way to handle the limitation.\n\nThe nested Q-function construction is new relative to the cited cost-sensitive DTR literature, which mostly treats treatment costs or fixed monitoring schedules. The authors also define target parameters as best linear approximations rather than pretending the truth is linear, which is a legitimate and defensible choice.\n\nThe soft spot is exactly where the stress-test note points. Assumption 6 requires uniform, almost-sure approximation of the true contrast functions by linear working models. That is a strong condition, and when it fails, the learned rule has no guarantee of approaching even the best linear rule, because the sign of a projection need not match the sign of the true contrast. Every simulation uses outcome models that are linear in the relevant contrasts, so epsilon_n = 0 by construction and the misspecification term is never exercised. The MIMIC application is suggestive but has no error bars, and there is no code or data artifact. Algorithm 1 Step 7 says 'Repeat Steps 3–7,' which is self-referential and presumably a typo for Steps 3–6; it should be fixed for reproducibility.\n\nThese are genuine weaknesses, but they are not fatal. The central claim is a conditional one, and the paper states its own condition. A serious referee can push for a nonlinear simulation, error bars, and code, and the core method would still stand. This is a solid method paper for people working on cost-sensitive DTRs or personalized diagnostic decisions. I would send it to peer review.","headline":"A genuine extension of Q-learning to covariate-assessment decisions, with honest regret bounds that separate estimation from misspecification—but the practical claim is only tested under perfect linearity.","tokens_in":57925,"tokens_out":1743,"would_cite":true,"duration_ms":21455,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Backward induction over four contrast functions yields a treatment regime that maximizes expected outcome minus the cost of treatments and diagnostic tests, and a linear-model version learns it with regret that shrinks as the sample grows.","keywords":["dynamic treatment regimes","utility-cost balance","Q-learning","covariate assessment costs","personalized medicine","R-learner","regret bound","two-stage clinical trials"],"falsifier":"Generate data from a two-stage model whose true contrast functions are strongly nonlinear, such as a threshold effect in the second-stage outcome, run Balanced Q-learning with linear working models at increasing sample sizes, and compare the achieved profit with the oracle profit; if the regret does not shrink as $n$ grows, the vanishing-misspecification condition in Assumption 6 is the binding assumption.","tokens_in":56891,"feed_emoji":"⚖️","tokens_out":9141,"duration_ms":93453,"temperature":0.7,"pith_summary":"This paper establishes that a dynamic treatment regime can be optimized for net benefit, not just for health outcome: the objective is expected outcome minus the cost of the treatments given and minus the cost of the diagnostic covariates chosen before each treatment. The proposed Balanced Q-learning method adds a covariate-assessment decision before each treatment decision and proves that the backward-induction rule maximizing four associated contrast functions is profit-optimal over all two-stage regimes. The practical estimator uses cross-fitted residual-on-residual regressions with linear working models, and the regret of the learned rule relative to the optimal rule is $O_p\\{(1/\\sqrt{n})^{r+1}+\\epsilon_n^{r+1}\\}$, so when the working models are correct or nearly so, the learned regime attains the global optimum as the sample grows. This matters because it turns the question of which diagnostic tests are worth running into a data-driven, patient-specific policy rather than a fixed menu.","feed_headline":"Cost-aware algorithm balances treatment benefit against test costs","feed_subtitle":"A new learning rule maximizes expected outcome minus spending on treatments and on the tests chosen for each patient.","key_machinery":"The carrying object is a four-stage backward induction over contrast functions $\\Delta_{2t}$, $\\Delta_{2c}$, $\\Delta_{1t}$, and $\\Delta_{1c}$, which measure the profit advantage of the second treatment, the second covariate set, the first treatment, and the first covariate set over their respective baselines. The algorithm estimates the unrestricted versions of these contrasts with cross-fitted residual-on-residual (R-learner) regressions and then projects them onto the actually assessed covariate subsets with nested regressions; the sign or argmax of each estimated contrast is the learned decision rule at that stage.","core_discovery":"The central discovery is that the usual Q-learning dynamic programming step can be duplicated for measurement decisions: at each of two stages the decision-maker chooses both which additional covariates to collect and which treatment to give, and the optimal joint policy is obtained by backward induction over Q-functions for the second treatment, the second covariate set, the first treatment, and the first covariate set. Theorem 3.1 shows that the regime selecting the argmax at each of these four steps dominates every other two-stage regime in expected profit. Theorem 4.2 shows that when the contrast functions are estimated through linear working models, the profit gap between the estimated and the optimal regime is $O_p\\{(1/\\sqrt{n})^{r+1}+\\epsilon_n^{r+1}\\}$, where $r$ is a margin exponent and $\\epsilon_n$ is the misspecification error of the linear models; hence under correct specification the estimated regime converges to the global profit optimum.","pith_inferences":["The same backward-induction accounting could be extended to more than two stages; each new stage adds a pair of contrast functions, but the misspecification error and the number of candidate subsets grow, so sparse or greedy search would be needed in practice.","If covariate measurement itself can change patient behavior or outcome (for example via anxiety or labeling effects), the paper's profit formula would need an additional direct-effect term; this is empirically testable in a randomized comparison of identical information collected under different disclosure protocols.","The method's design suggests a natural decision-support use: list candidate test bundles with prices on a hospital menu, fit the contrasts from historical records, and the learned rule becomes a personalized test-ordering guideline."],"forward_implications":["A clinic using the learned regime will adaptively order fewer or cheaper diagnostic tests for patients whose baseline profile makes the extra information unlikely to change treatment.","When the contrast functions are linear, the regret bound gives a fast convergence rate, so the method attains the profit-optimal regime in large samples.","The asymptotic normality result supports plug-in confidence intervals for the contrast parameters, allowing a clinician to test whether a specific covariate or test contributes enough to utility to justify its cost.","As the cost weight or the prices of tests increase, the learned regime shifts from large covariate sets to smaller, cheaper ones, preserving most of the outcome benefit while cutting expenditure."],"supporting_citations":[{"why":"Establishes the potential-outcome identification conditions (consistency, sequential ignorability, positivity) and the dynamic-programming view of optimal DTRs that Theorem 3.1 relies on.","marker":"Murphy, 2003"},{"why":"Provides the Q- and A-learning formulations of value functions that the four-decision Q-function construction extends.","marker":"Schulte et al., 2015"},{"why":"Supplies the Q-learning theory used to justify estimating the optimal regime through backward induction on Q-functions.","marker":"Clifton and Laber, 2020"},{"why":"Supplies the residual-on-residual regression construction adapted for the contrast functions and the comparison baseline used in the numerical studies.","marker":"Ertefaie et al., 2021"},{"why":"Shows how to fold treatment-assignment costs into decision boundaries as constant thresholds, which the paper adapts for the treatment-cost components.","marker":"Athey and Wager, 2021"},{"why":"Provides the high-dimensional Q-learning baseline that regularizes covariate selection and is used for comparison in the simulations and real data application.","marker":"Zhu et al., 2019"},{"why":"Supplies the R-learner framework and the nuisance-estimation error conditions used in Assumption 3.","marker":"Nie and Wager, 2021"},{"why":"Supplies the margin condition used in Assumption 5 to control the probability of being near decision boundaries.","marker":"Mammen and Tsybakov, 1999"},{"why":"Gives the $O_p(1/n)$ regret benchmark for linear fitted Q-iteration that the correctly specified case matches.","marker":"Hu et al., 2025"}],"fun_headline_variants":["Balanced Q-learning maximizes profit from treatments and tests","Cost-aware learning rule optimizes expected patient profit","Two-stage DTRs with test costs: profit-optimal via Q-learning","Dynamic treatment regimes that balance utility and test costs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical claim rests on Assumption 6: the linear working models for the four contrast functions must be accurate enough that the misspecification error $\\epsilon_n$ shrinks to zero; if treatment effects are strongly nonlinear, the learned balanced rule has no guarantee of approaching the profit-maximizing regime.","fun_headline_variants_meta":{"raw":{"variants":["Balanced Q-learning maximizes profit from treatments and tests","Cost-aware learning rule optimizes expected patient profit","Two-stage DTRs with test costs: profit-optimal via Q-learning","Dynamic treatment regimes that balance utility and test costs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":2981,"prompt_tokens":834,"completion_tokens":2147,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":2081}},"tokens_in":450,"tokens_out":2147,"duration_ms":16347,"temperature":1.0,"reasoning_tokens":2081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:50:51.042094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate data from a two-stage model whose true contrast functions are strongly nonlinear, such as a threshold effect in the second-stage outcome, run Balanced Q-learning with linear working models at increasing sample sizes, and compare the achieved profit with the oracle profit; if the regret does not shrink as $n$ grows, the vanishing-misspecification condition in Assumption 6 is the binding assumption.","supporting_citations":[],"review_version":1}