{"id":"0e218c9f-eb75-443b-a149-015531a0e664","arxiv_id":"2507.00312","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Optimal targeting in Markovian systems reduces to CADE thresholding with state-specific shadow-cost thresholds, estimable via state-level value iteration.","lead":"This paper shows that in dynamic systems with shared capacity, the optimal treatment rule compares each unit's direct treatment benefit to a state-specific threshold that captures congestion costs. It provides an algorithm for estimating these thresholds from observational data and reports gains over standard CATE targeting and offline reinforcement learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3's reward-rate extension defines Δ_i as the inter-event time before unit i, so Δ_i is realized before W_i and is not a function of the current state/action; the transformed outcome Ỹ_i = R_i − θ*Δ_i therefore violates the Markov reward structure of Assumption 1, and the claimed threshold…","rationale":"The reader's weakest_assumption (Assumption 1, exogenous covariates) is a valid scope limitation: the threshold characterization indeed requires P_S to be independent of X and P_X to be independent of S, and many real queueing systems violate this. However, it is an explicit modeling assumption, and the Section 2 theorem is internally coherent given it. The more load-bearing concern is the Section 3 reward-rate extension, which contains an internal temporal misalignment: Δ_i is defined as the backward inter-event time, so the transformed outcome is not a function of the current state and action, breaking the Markov reward assumption needed for the Bellman equation. This is not a disagreement with external consensus but a correctness risk in the paper's own framework. The issue is fixable by reindexing to the forward inter-event time Δ_{i+1}, and the reported simulations may already implement the corrected version, but the derivation and algorithm description as written do not. Because the core Section 2 threshold theorem appears sound and the flaw is localized to an extension, the reader's CONDITIONAL verdict remains appropriate; no change to the verdict is needed, though the revision conditions should include correcting Section 3's indexing and re-verifying the customer-support results.","tokens_in":35556,"tokens_out":25790,"duration_ms":317373,"concrete_test":"Re-derive the Bellman operator for the event-level M_n/M/1 queue using the forward inter-event time Δ_{i+1} = T_{i+1} − T_i instead of Δ_i, and compare with equation (29). Compute τ_{Δ^+}(x, s) = E[Δ_{i+1} | X=x, S_i=s, W=1] − E[Δ_{i+1} | X=x, S_i=s, W=0]; if this is nonzero while the paper's backward τ_Δ is zero, the operator (29) provably omits the congestion effect. Then run the customer-support experiment with SACT implemented using backward Δ_i: if it fails to approach the oracle reward rate (θ* = 8.720 in Figure 5) while the forward-Δ implementation succeeds, the Section 3 derivation is the cause of the discrepancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the paper defines Δ_i := T_i − T_{i−1} (the time before event i) and then constructs the transformed outcome Ỹ_i = R_i − θ*Δ_i, claiming the state-level Bellman characterization of Section 2 applies with state S_i = (K_i, A_i). This is internally inconsistent. Δ_i is realized before W_i is chosen: its conditional distribution given (X_i, S_i, W_i) is not determined by the current state/action because it depends on the previous state S_{i−1}. Consequently, τ_Δ(x, s) = E[Δ_i | X=x, S_i=s, W=1] − E[Δ_i | X=x, S_i=s, W=0] is identically zero, so the Bellman operator (29) reduces to the direct-reward CADE alone and cannot encode the very congestion-driven future arrival reduction the section is meant to capture. The correct Dinkelbach inner problem maximizes E[R_i − θ* Δ_{i+1}] (or a one-step reindexed reward), where Δ_{i+1} is the forward inter-event time whose conditional law depends on S_i and W_i; its CADE is nonzero and must appear in the threshold. Thus the threshold characterization for congestion-sensitive arrivals is unsupported as written, even though the Section 2 theorem for the indexed process appears sound.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies optimal treatment targeting in a Markovian system in which units arrive sequentially, each with covariates X_i, and the decision-maker assigns treatment W_i based on X_i and a system state S_i. The outcome Y_i may depend on the current state and action, and the action also affects the future state. Under a time-homogeneous MDP with exogenous covariates (Assumption 1), the paper proves (Theorem 2) that an optimal average-reward policy exists and takes the form π*(x,s)=I(τ(x,s)>c_s), where τ is the CADE and c_s = −Σ_{s'}(P_{S,1}(s'|s)−P_{S,0}(s'|s))V*(s') is a state-dependent shadow cost equal to the negative conditional average indirect effect. It proposes Algorithm 1 (SACT), which estimates τ via causal ML and V* via state-level relative value iteration, with sample-splitting by regenerative blocks. Regret bounds (Theorem 4) and a tabular comparison with collapsed-state RL (Proposition 5) are given, plus simulations in emergency-department routing and online customer support with congestion-sensitive arrivals, using a Dinkelbach reward-rate extension in Section 3.","tokens_in":35831,"tokens_out":19155,"duration_ms":214834,"significance":"The Section 2 characterization is an elegant and potentially practically important bridge between CATE-based targeting and dynamic programming: it reduces a high-dimensional MDP with covariates to a low-dimensional state-level Bellman recursion, with a decision rule that preserves the familiar CADE ranking within each state. The regret analysis and the finite-state comparison provide a credible argument for a statistical advantage over collapsed-state RL, and the numerical studies show large gains over direct targeting and generic offline RL. The main weakness is the reward-rate extension in Section 3, which is conceptually incorrect as written; because this extension underlies the customer-support experiment, the paper's claims about congestion-sensitive arrivals require revision. The core Section 2 theorem, however, appears sound.","major_comments":[{"comment":"The reward-rate extension is built on the transformed outcome Ỹ_i = R_i − θ*Δ_i, where Δ_i = T_i − T_{i−1} is the inter-event time preceding event i. Because Δ_i is realized before the action W_i is chosen, its conditional distribution given the current state and action is independent of W_i, so τ_Δ(x,s) = 0 identically. Consequently, the Bellman operator (29) does not charge the decision for the effect of treatment on future arrival times: the sojourn time associated with the decision is Δ_{i+1}, not Δ_i. The correct Dinkelbach inner problem for the semi-Markov reward rate θ(π) = E_π[R_i]/E_π[Δ_i] uses the forward inter-event time (e.g., Ỹ_i = R_i − θ*Δ_{i+1} or an equivalent reindexing), whose CADE is generically nonzero and belongs in the threshold. As written, the threshold rule (30) and the update (31) do not follow from the Section 2 theorem; the r_Δ,0(s) term in (31) is estimated under the behavior policy and is not the conditional mean backward recurrence time under the candidate policy π^(b), so the Dinkelbach iteration is also internally inconsistent. This undermines the theoretical claim of Section 3 and the reward-rate interpretation of the customer-support experiment in Section 4.2.","section":"§3, Eqs. (26)-(29), Section 4.2"}],"minor_comments":[{"comment":"The regenerative-block sample-splitting procedure is described only informally; a precise construction of the blocks (choice of regeneration state, block lengths, how cross-fitting assigns whole blocks to folds) would make the algorithm and the statements in Footnote 4 reproducible.","section":"§2.2 and Appendix C.2.1"},{"comment":"The uniform mixing assumption is imposed on the empirical kernel P̂_{S,π} as well as the true kernel, which is a strong condition not implied by Assumptions 1-2; the paper should clarify whether Assumption 3 is a high-level assumption the user must verify, and how the anchored-kernel suggestion of Zurek and Chen [2024] is incorporated into Algorithm 1.","section":"Assumption 3"},{"comment":"The caption refers to an 'effective state-specific threshold' but the figure displays a single horizontal line for the direct rule and the optimal rule is shown through the treated/untreated coloring; please make the display of the state-dependent thresholds under the optimal rule explicit.","section":"Figure 4 caption"},{"comment":"The symbol k is used both for the capacity bound and as a generic queue-length state, which is confusing (e.g., in Eq. (S3) the condition k < k appears); please introduce separate notation for the capacity limit and the running state.","section":"Section A.1 and B.1"},{"comment":"The expression for the Bellman operator (10) uses E_X[(τ(X,s)+((P_{S,1}−P_{S,0})v)(s))^+], which is correct, but the subsequent presentation in Eq. (13) uses a sample average over the evaluation set without restating that this approximates the covariate expectation; making this explicit in the text would help readers.","section":"Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The Section 3 problem is the main obstacle to publication: the reward-rate extension rests on a temporal mismatch between the decision and the sojourn time used in the transformed outcome. The fix is conceptually straightforward (use the forward inter-event time or an equivalent reindexing), but it changes the Bellman operator, the threshold formula, and the Dinkelbach update, and the customer-support experiment would need to be re-examined. The Section 2 theorem and the SACT algorithm for the per-arrival average-reward objective appear sound and are the paper's main contribution. The comparisons against RL baselines are honestly described, including the oracle tuning of discount factors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core Section 2 result is sound: under the exogenous-covariate MDP, the optimal policy really does reduce to a CADE threshold with state-dependent cutoff equal to the negative indirect effect, and the paper connects this cleanly to difference-of-Q and policy-gradient ideas. The SACT algorithm is a reasonable, clearly specified way to estimate those thresholds, and the state-level Bellman operator (10) is a genuinely useful reduction that avoids the full (X,S) state space. Second, the Section 3 reward-rate extension has a real indexing bug that I don't think is in the Section 2 core. The stress-test note is right: they define Δ_i as the inter-event time before unit i and build Ỹ_i = R_i − θ*Δ_i, then claim the Section 2 characterization applies with state S_i = (K_i,A_i). But Δ_i is realized before W_i is chosen and its conditional law does not depend on the current action; τ_Δ(x,s) is zero, so Bellman operator (29) cannot encode the congestion-driven arrival slowdown. The fix is to use Δ_{i+1}, the forward inter-event time, or reindex so the transformed reward depends on the current action. As written, the claimed threshold policy for congestion-sensitive arrivals (30) is unsupported. That said, the Section 2 theorem for the indexed process appears solid; the proofs in the appendix are detailed and the performance-difference argument is standard. The regret bounds are honest about relying on high-level uniform mixing and the regenerative-block splitting is presented informally, so those parts need tightening but are not fatal. Numerics are illustrative rather than exhaustive, and code/data are not released, which is worth asking for. The related-work discussion is genuinely careful, including the caveats about the collapsed-state comparison and the oracle tuning favors the RL baselines. Overall: the main idea is valuable, and the Section 3 issue is fixable. I would send a serious referee with instructions to focus on the reward-rate derivation, and I would not rely on (30) until the reindexing is corrected. The paper deserves referee time; my own verdict would be conditional on a corrected Section 3 and a real code release.","headline":"Theorem 2's threshold form is a clean Bellman-equation consequence and the SACT algorithm is a sensible practical bridge between CATE targeting and dynamic capacity, but Section 3's reward-rate extension has a real indexing flaw in the current draft.","tokens_in":36340,"tokens_out":577,"would_cite":true,"duration_ms":10154,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C40","62G05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Under mild assumptions, the optimal treatment policy in a dynamic system is a state-specific threshold on the direct treatment effect, learnable by augmenting CATE estimation with state-level value iteration.","keywords":["optimal targeting","conditional average treatment effect","conditional average direct effect","Markov decision process","state-level value iteration","congestion and capacity constraints","offline policy learning","average reward"],"falsifier":"Simulate the emergency-department parallel-queue system but generate patient covariates from a distribution that depends on the current fast-track queue length (or on past routing decisions), violating Assumption 1. If an exhaustive-search optimal policy strictly beats the CADE-threshold rule with the true state-specific thresholds computed from the true $V^*$, the characterization fails exactly under its stated boundary; equivalently, checking whether the estimated covariate distribution $P_X$ varies with $S$ in real queuing logs would test the premise.","tokens_in":35322,"feed_emoji":"🎯","tokens_out":7916,"duration_ms":82714,"temperature":0.7,"pith_summary":"This paper tries to show that optimal treatment targeting in a Markovian system with shared capacity is not as hard as general reinforcement learning. Under the assumption that units' covariates are drawn exogenously, the optimal policy takes the form of thresholding the Conditional Average Direct Effect (CADE) at a state-specific cutoff: treat a unit with covariate $x$ when the system is in state $s$ iff $\\tau(x,s)>c_s$. The cutoff $c_s$ equals the negative conditional average indirect effect of treatment at the optimum, i.e., the expected downstream cost of using capacity in that state. The paper then gives an algorithm, state-aware CADE thresholding (SACT), that combines standard CATE/CADE estimation with relative value iteration on the state space, and proves that its regret is governed by the CADE estimation rate. A practitioner would care because direct CATE targeting over-uses scarce resources under congestion, while generic policy learning over the full covariate-state space is needlessly statistically expensive.","feed_headline":"Optimal dynamic treatment rules are state thresholds","feed_subtitle":"CATE ranking stays, but the cutoff shifts with capacity cost; SACT learns it from one dataset.","key_machinery":"The key machinery is the state-level Bellman operator obtained by integrating out the exogenous covariates. Because $X_i$ is drawn from a fixed distribution $P_X$ independent of state and treatment, the continuation value collapses to a relative value function $V^*$ on the finite state space $\\mathcal{S}$, and the covariate enters only through the CADE $\\tau(x,s)$. This yields the operator $(T_{\\mathcal{S}}v)(s)=r_0(s)+(P_{S,0}v)(s)+\\mathbb{E}_X[(\\tau(X,s)+((P_{S,1}-P_{S,0})v)(s))_+]$, where $r_0(s)=\\mathbb{E}_X[\\eta_0(X,s)]$ is the average baseline reward and $P_{S,w}(s'|s)$ is the state transition kernel under treatment $w$. Algorithm 1 estimates $\\tau$, $r_0$, and the transition kernels on one sample split and applies relative value iteration to an empirical version of $T_{\\mathcal{S}}$; the fixed point $V^*$ supplies the thresholds $c_s$. The same operator is adapted to reward-rate objectives by replacing the outcome with $R_i-\\theta^*\\Delta_i$ and iterating via Dinkelbach's method.","core_discovery":"The central claim is Theorem 2: under time-homogeneous dynamics with exogenous covariates (Assumption 1) and a finite irreducible, aperiodic state process (Assumption 2), there exists an optimal deterministic policy of the form $\\pi^*(x,s)=\\mathbb{I}(\\tau(x,s)>c_s)$, where $c_s = -\\sum_{s'\\in\\mathcal{S}}(P_{S,1}(s'|s)-P_{S,0}(s'|s))V^*(s')$ and $V^*$ is the relative value function of the state-level average-reward Bellman equation. Thus the optimal dynamic targeting rule keeps the familiar CADE ranking within each state but changes the treatment cutoff, and the threshold is the dynamic shadow cost of assigning treatment in that state at the optimum. The direct targeting rule $\\mathbb{I}(\\tau(x,s)>0)$ is optimal only in the degenerate case where treatment does not affect state transitions. The paper supports this with regret bounds showing that the learned threshold policy has regret $O_P(n^{-(\\beta\\wedge 1/2)})$ when the CADE estimator achieves $L^2$ rate $n^{-\\beta}$.","pith_inferences":["One can read $c_s$ as a state-indexed shadow price; this suggests that related resource-allocation problems with convex congestion costs might admit analogous threshold rules, a direction the paper does not pursue.","A practical diagnostic suggested by the theory: estimate CADE and CAIE separately; a near-zero CAIE means direct targeting is near optimal, while a large CAIE indicates capacity effects dominate.","The regret analysis leaves margin conditions untouched; adding the usual margin assumption near the boundaries $\\tau(x,s)=c_s$ could plausibly sharpen the $n^{-(\\beta\\wedge 1/2)}$ bound to faster plug-in rates, though the paper does not claim this.","The finite-state assumption is used for the mixing and value bounds; extending SACT to continuous states would require function approximation for $V^*$ and a new analysis of the state-level operator."],"forward_implications":["Existing CATE/CADE estimators (e.g., causal forests) can be reused as-is; the dynamic adjustment is confined to a low-dimensional state-level value iteration.","Under congestion, direct targeting is suboptimal: it admits marginal patients who crowd out later high-benefit arrivals, and the gap grows with the strength of indirect effects.","The threshold $c_s$ equals the negative conditional average indirect effect at the optimal policy, so the optimal rule is a direct/indirect effect decomposition: treat iff CADE exceeds the shadow cost.","For systems where arrivals respond to congestion, the same threshold characterization holds for the long-run reward rate, using transformed outcome $R_i - \\theta^* \\Delta_i$ and Dinkelbach updates.","In tabular settings, exploiting the exogenous-covariate structure yields regret smaller than the collapsed-state Markov-decision-process baseline by a factor of order $\\min\\{(t_0+t_{\\mathrm{mix}})\\sqrt{|\\mathcal{X}||\\mathcal{S}|}, |\\mathcal{X}|\\}$, making the method attractive with moderately large covariate spaces."],"supporting_citations":[{"why":"Supplies the average-reward Bellman equation and relative value iteration that underpin Lemma 1 and Algorithm 1.","marker":"[Puterman, 1994]"},{"why":"Supplies the causal-forest CADE estimator used in the direct targeting rule and in SACT.","marker":"[Athey et al., 2019]"},{"why":"Introduces the CADE/CAIE decomposition whose terminology and interpretation anchor Theorem 2.","marker":"[Munro et al., 2025]"},{"why":"Provides the policy-gradient theorem used to identify the CADE/CAIE decomposition and the shadow-cost interpretation.","marker":"[Marbach and Tsitsiklis, 2001]"},{"why":"Supplies the fractional-programming update used for the reward-rate variant with state-dependent arrivals.","marker":"[Dinkelbach, 1967]"},{"why":"Provides the tabular MDP benchmark and simulation-lemma proof style used in Proposition 5.","marker":"[Kearns and Singh, 2002]"},{"why":"Cited as the simulation-lemma reference through which Bellman-operator error propagates into regret.","marker":"[Agarwal et al., 2026]"},{"why":"Fitted Q-iteration baseline in the numerical comparison, representing the collapsed-state approach.","marker":"[Ernst et al., 2005]"}],"fun_headline_variants":["State-dependent thresholds define optimal dynamic treatment","Dynamic treatment: cutoffs shift with system state","Optimal targeting uses state-specific treatment costs","CADE ranking plus state cost gives optimal policy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Assumption 1: each arriving unit's covariates are drawn from the same fixed distribution, independent of the system state and of past treatments; if the mix of arriving units shifts with congestion or with the policy, the state-level Bellman equation and the threshold characterization do not hold.","fun_headline_variants_meta":{"raw":{"variants":["State-dependent thresholds define optimal dynamic treatment","Dynamic treatment: cutoffs shift with system state","Optimal targeting uses state-specific treatment costs","CADE ranking plus state cost gives optimal policy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1353,"prompt_tokens":934,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":550,"tokens_out":419,"duration_ms":5641,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:18:53.537587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the emergency-department parallel-queue system but generate patient covariates from a distribution that depends on the current fast-track queue length (or on past routing decisions), violating Assumption 1. If an exhaustive-search optimal policy strictly beats the CADE-threshold rule with the true state-specific thresholds computed from the true $V^*$, the characterization fails exactly under its stated boundary; equivalently, checking whether the estimated covariate distribution $P_X$ varies with $S$ in real queuing logs would test the premise.","supporting_citations":[{"cited_title":"Treatment effects in market equilibrium","cited_arxiv_id":null,"evidence_quote":"Introduces the CADE/CAIE decomposition whose terminology and interpretation anchor Theorem 2."}],"review_version":1}