{"id":"5a493472-1fab-4c5b-93fd-ab2434446791","arxiv_id":"2412.05783","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-way deconfounder algorithm that models unmeasured confounders as per-trajectory and per-timestep latent factors and uses a neural tensor network for off-policy evaluation.","lead":"This paper proposes a new way to evaluate a decision policy from past data even when some hidden factors influence both decisions and outcomes. It splits hidden factors into two groups, one for each person and one for each time point, learns them with a neural network, and then simulates the target policy to estimate its value.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central consistency claim depends on Assumption 4 (embedding error ε_U,W→0), but no convergence or identifiability result is established for the neural tensor network estimator; the cited Proposition 3 only proves prediction-error decay in a linear model, not recovery of U_i, W_t.","rationale":"I read the paper as claiming a model-based OPE estimator that is consistent under TWUC, with Theorem 1 as the formal basis. The theorem is honest in making its bound depend on ε_P and ε_U,W, but the abstract-level consistency claim requires those errors to vanish at appropriate rates. The paper provides standard neural-network approximation arguments for ε_P, but for ε_U,W it offers only Assumption 4 and a pointer to Proposition 3. That pointer is insufficient: Proposition 3 is about prediction MSE in a linear model, not about recovering the latent factors, and the linear two-way model has a well-known additive-shift non-identifiability. The NTN estimator is nonlinear, and no identifiability or consistency theorem for its embeddings is given. The proof's treatment of I2 also hand-waves the Lipschitz/TV step, which is precisely where Assumption 4 enters. This is not an accusation of dishonesty; the empirical work is a real positive, including the sensitivity analysis showing degradation when TWUC is violated, and the code is provided. But the central theoretical claim is not closed. A CONDITIONAL verdict asking for a proof or substantial relaxation of Assumption 4, or for explicit identifiability conditions, is appropriate. I find no reason to move the reader's verdict.","tokens_in":29625,"tokens_out":8964,"duration_ms":98785,"concrete_test":"Run the Section D.1 linear two-way LSE with increasing N and T (e.g., 50, 100, 200, 400) and compute the factor-recovery error after fixing the additive-shift ambiguity, either by imposing the natural constraints Σ_i Û_i = Σ_i U_i and Σ_t Ŵ_t = Σ_t W_t or by minimizing over the shift. If the average squared embedding error does not decay to zero while the prediction MSE does, then Proposition 3 cannot be used to justify Assumption 4. Independently re-derive the I2 bound in Appendix B.5 with an explicit uniform Lipschitz constant for the NTN transition network; if the constant depends on the true embeddings or diverges, the T ε_U,W term in Theorem 1 is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1 gives E|η̂π − ηπ| ≤ CT Rmax ε_P + cT Rmax ε_U,W + cRmax N^{-1/2} + cRmax T^{α-1}; the advertised consistency therefore requires ε_U,W→0, in fact T ε_U,W→0. Assumption 4 is the only place where this enters, and the paper never proves it for the proposed NTN estimator. The justification 'According to Proposition 3' does not carry the weight: Proposition 3's proof bounds the prediction error σ²(N+T+2)/(NT) in a linear additive two-way model, not the parameter error (1/N)Σ_i E||Û_i−U_i||² or (1/T)Σ_t E||Ŵ_t−W_t||². Moreover, in that linear model the factors are identified only up to an additive constant: adding c to all U_i and subtracting c from all W_t leaves Xβ unchanged, so without an explicit normalization the Euclidean embedding error need not vanish even as predictions improve. For the nonlinear neural tensor network, no identifiability condition or consistency theorem for the learned embeddings is supplied. The proof of term I2 in Appendix B.5 also assumes that TV(P̂(·|a,o,Û,Ŵ), P̂(·|a,o,U,W)) is bounded by a uniform Lipschitz constant times the embedding distance; this is asserted rather than shown, and it is exactly the step that converts Assumption 4 into the T ε_U,W term. Thus the abstract's 'consistent policy value estimation' rests on an unsupported premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses off-policy evaluation (OPE) in reinforcement learning when unmeasured confounders affect both actions and transitions/rewards. It introduces a 'two-way unmeasured confounding' (TWUC) model in which each latent confounder decomposes into a trajectory-specific component U_i and a time-specific component W_t, and proposes a two-way deconfounder that learns these embeddings together with a transition model and a behavior-policy model using a neural tensor network, followed by Monte Carlo policy evaluation. The theoretical section states a finite-sample error bound (Theorem 1) that is linear in the transition error ε_P and the embedding error ε_{U,W}. Experiments on two synthetic environments and MIMIC-III compare the method against model-based, importance-sampling, doubly robust, and POMDP baselines.","tokens_in":29991,"tokens_out":7373,"duration_ms":74101,"significance":"The TWUC assumption is a genuinely useful middle ground between unconstrained and one-way confounding, and the joint NTN architecture plus the model-based plug-in estimator is a sensible way to instantiate it. The paper ships code, provides sample-splitting proof ideas, and includes a broad set of experiments, including a real-data study; these are real strengths. However, the paper's advertised consistency result is conditional on an unproved embedding-recovery assumption. The theoretical contribution therefore currently establishes a conditional guarantee only, and the empirical evaluation cannot substitute for the missing identifiability/convergence analysis.","major_comments":[{"comment":"The conclusion that E|η̂π−ηπ| vanishes as N,T grow requires ε_U,W→0 (in fact T ε_U,W→0), but Assumption 4 is never verified for the proposed neural tensor network estimator. The sentence 'According to Proposition 3, these errors are negligible under simple models' does not support this. Proposition 3 (Appendix B.4) bounds the prediction MSE of a linear additive two-way fixed-effects model; it does not bound (1/N)Σ_i E‖Û_i−U_i‖² or (1/T)Σ_t E‖Ŵ_t−W_t‖². Moreover, in that linear model the factors are identified only up to an additive constant, so Euclidean embedding error need not vanish even as predictions improve. For the nonlinear NTN estimator no identifiability condition or consistency theorem for the embeddings is supplied, so the central consistency claim of the abstract rests on an unsupported premise.","section":"Section 4, Assumption 4 and Theorem 1"},{"comment":"The proof states that 'as neural networks are Lipschitz continuous functions of their parameters, the above total variation norm is proportional to ∥(Ûᵀ_i,Ŵᵀ_t)ᵀ − (Uᵀ_i,Wᵀ_t)ᵀ∥₂' and then applies Assumption 4. This is not a consequence of Lipschitz continuity; it is an additional structural assumption that the learned transition distribution changes in total variation at a uniform rate in the embedding error, with a constant independent of (a,o) and of the estimated network parameters. No such bound or constant is stated. Since this is exactly the step that produces the term cT Rmax ε_U,W, the second term of Theorem 1 is not justified as written.","section":"Appendix B.5, Step 1 (bound on I2)"},{"comment":"Theorem 1 is stated only in a tabular setting with discrete observation, action, and latent spaces, while the proposed estimator in Section 3 is defined for continuous observations and uses a conditional Gaussian NTN; the experiments also use continuous observations. The paper does not provide a discretization or approximation argument showing that the tabular bound applies to the implemented estimator. Thus the theorem, even if Assumptions 1-5 held, would not cover the algorithm as used in the experiments.","section":"Section 4 versus Section 3"}],"minor_comments":[{"comment":"The displayed formula MSE(Ŷ)=σ²(N+T+2)/(NT) appears to be off by one: the projection matrix X(XᵀX)⁻Xᵀ has rank N+T+1 because the unit and time dummy columns sum to the same all-ones vector. The rate is unaffected.","section":"Appendix B.4"},{"comment":"In the tailored policy for normalized SOFA < 0.7, the text assigns p(A_v=1)=0.2 and then p(A_v=2)=p(A_v=1)=p(A_v=0)=0, which is internally inconsistent; please correct the intended probabilities.","section":"Appendix D.4"},{"comment":"The learning-rate search range is written as [0.005, 0.001], which is in decreasing order and is presumably meant to be [0.001, 0.005] or similar.","section":"Appendix C.2"},{"comment":"The text and table refer to one-way deconfounder variants as OWD-NI and OWD-NT, while Appendix C.3 labels them TWD-NI and TWD-NT; please standardize the naming.","section":"Section 5.4 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the abstract overclaims consistency: the theorem is a conditional bound, and the condition on embedding recovery is not established. If the authors can supply a nontrivial identifiability/convergence result for the NTN embeddings, the paper would be substantially stronger; otherwise the contribution should be repositioned as a practical method with conditional theory. I do not see a circularity problem: the loss does not fit the policy value, so the estimator is not constructed to match the target."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick version: the paper is a genuine contribution to OPE under unmeasured confounding, and the empirical package is solid. But the headline claim—consistent policy-value estimation under the two-way unmeasured confounding assumption—is not established by the theory. The finite-sample bound in Theorem 1 is conditional on Assumption 4, which says the learned embeddings are close in squared error to the true latent factors. That assumption is doing all the work, and the paper never proves it for the proposed neural tensor network estimator.\n\nWhat's new: the two-way latent structure is a natural middle ground between unconstrained and one-way confounding, and the authors correctly note that Bian et al. require additive latents and linear function approximation. The NTN-based joint estimation of transition and behavior-policy models is sensible. The experiments cover two simulations and MIMIC-III, with a sensitivity analysis that honestly shows performance degrading when the assumption is violated, plus an ablation supporting the joint loss and NTN. Code is provided. The discussion of the Wang-Blei deconfounder's limitations is fair.\n\nThe soft spot is in Section 4. The bound E|η̂π − ηπ| ≤ CT Rmax ε_P + cT Rmax ε_U,W + cRmax N^{-1/2} + cRmax T^{α-1} contains T Rmax ε_U,W, so consistency requires ε_U,W → 0, in fact T ε_U,W → 0. Assumption 4 is exactly that. The paper says, \"According to Proposition 3, these errors are negligible under simple models.\" Proposition 3 is a linear additive two-way model, and its proof bounds prediction error, not parameter error of the embeddings. In that linear model the factors are identified only up to an additive constant: adding c to all U_i and subtracting c from all W_t leaves the prediction unchanged, so the Euclidean error in Assumption 4 need not vanish even with perfect predictions. For the nonlinear NTN, no identification or consistency theorem is supplied. The proof of the I2 term in Appendix B.5 also simply asserts that the TV distance between transition estimates at different embeddings is Lipschitz in the embedding difference; that needs an argument. Minor point: the theorem is proved for a sample-splitting variant, while the experiments use the joint estimator; probably fixable, but it should be stated.\n\nThe citation pattern looks appropriate, and the limitations section is candid. This is for anyone working on OPE in confounded RL, especially deconfounding approaches. It deserves a serious referee. My main ask: prove that the training loss yields embeddings with vanishing error under TWUC (plus an identifiability condition), or state the theorem with the embedding error as an unavoidable assumption and temper the abstract.","headline":"A genuinely useful two-way deconfounder for OPE, but the consistency claim runs ahead of the theory: Assumption 4 is doing unproved work.","tokens_in":30511,"tokens_out":4260,"would_cite":true,"duration_ms":41279,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting unmeasured confounders into trajectory- and time-specific factors yields a consistent off-policy value estimator with a finite-sample error bound.","keywords":["off-policy evaluation","unmeasured confounding","two-way fixed effects","deconfounding","neural tensor network","model-based reinforcement learning","causal reinforcement learning","latent confounders"],"falsifier":"Simulate the tumor-growth data-generating process with known ground-truth $U_i$ and $W_t$, train the two-way deconfounder, and measure the mean-squared error of the estimated embeddings against the truth; if this error does not shrink as $N$ and $T$ grow, or if the policy-value error stays bounded away from zero while the other conditions of Theorem 1 hold, the consistency claim falls.","tokens_in":29421,"feed_emoji":"🧩","tokens_out":5944,"duration_ms":55727,"temperature":0.7,"pith_summary":"The paper asks how to evaluate a new policy from logged data when hidden confounders influence both actions and outcomes. It proposes a middle-ground assumption, two-way unmeasured confounding: every hidden confounder is either constant within a trajectory or constant across trajectories, so the latent count is N+T instead of N×T. Under this assumption a neural tensor network learns the confounders together with the environment dynamics, and a plug-in model-based estimator evaluates the target policy. The central theoretical claim is a finite-sample bound on the estimator's mean absolute error that vanishes when the transition error, confounder estimation error, and an autocorrelation exponent all shrink appropriately.","feed_headline":"Two-way latent split yields consistent off-policy value estimates","feed_subtitle":"Grouping hidden factors by trajectory and by time shrinks the error bound to zero as data grows.","key_machinery":"The central object is the two-way unmeasured confounding assumption, $Z_{i,t} = (U_i^\\top, W_t^\\top)^\\top$, which reduces the number of latent variables from $N \\times T$ to $N + T$ and makes consistent estimation feasible. The carrying mechanism is the neural tensor network (NTN), whose bilinear tensor products $u_i^\\top W^{[k]} w_t$ let the trajectory- and time-specific confounders interact nonlinearly; the same NTN output feeds both the transition network (conditional Gaussian mean and variance) and the actor network (behavior policy), and the joint loss is optimized to yield the estimated embeddings and dynamics. The estimator then plugs the learned embeddings into Monte Carlo rollouts of the target policy, relying on the latents being policy-agnostic.","core_discovery":"The paper's core claim is that consistent off-policy evaluation is achievable without no-unmeasured-confounding, Markov, or proxy assumptions, provided the unmeasured confounders have a two-way structure: each confounder is either a trajectory-specific latent factor $U_i$ or a time-specific latent factor $W_t$. The proposed estimator learns $U_i$ and $W_t$ jointly with a Gaussian transition model and the behavior policy through a neural tensor network, then runs Monte Carlo rollouts of the target policy using the learned model. Theorem 1 bounds the mean absolute error by $C T R_{\\max} \\varepsilon_P + c T R_{\\max} \\varepsilon_{U,W} + c R_{\\max} N^{-1/2} + c R_{\\max} T^{\\alpha-1}$; the first two terms come from transition and confounder estimation, the last two from sampling variability across trajectories and time. When those estimation errors vanish and $\\alpha < 1$, the bound tends to zero, so the estimator is consistent.","pith_inferences":["Editorial: the proof's hardest condition is Assumption 4, that the learned embeddings track the true latents in mean squared error; the paper only proves an analogue for a linear additive fixed-effects model, not for the neural tensor network, so verifying or repairing this step is the natural next test.","Editorial: the sensitivity analysis suggests a simple extension: add a residual term that is both trajectory- and time-specific and use its magnitude as a sensitivity parameter, allowing the method to report how much violation of two-way confounding the conclusions tolerate.","Editorial: combining the estimated transition model with a double-robust or marginalized-importance correction could remove the linear $T$ dependence under ergodicity, mirroring standard results in off-policy evaluation.","Editorial: in the MIMIC-III analysis the policy-ranking check is qualitative; a sharper validation would withhold final outcomes and compare TWD's recommended ranking against a clinical benchmark or an oracle informed by full patient records."],"forward_implications":["Under two-way unmeasured confounding, off-policy evaluation becomes feasible without external proxies or Markov assumptions, and the number of latent variables grows only as $N + T$.","The finite-sample bound shows the estimation error grows linearly in the horizon $T$, which the paper notes is unavoidable in general, while the standard-deviation terms decay with $N$ and $T^{1-\\alpha}$.","In the simulated dynamic process, tumor growth example, and MIMIC-III data, the estimator achieves lower logarithmic mean squared error and bias than no-unmeasured-confounding, POMDP-type, and one-way deconfounding baselines.","Because the two-way confounders are policy-agnostic, the plug-in estimator avoids estimating the distribution of the latent factors, simplifying the Monte Carlo construction.","A sensitivity analysis shows the method degrades gracefully as the two-way assumption is mildly violated, but collapses when the violation is complete."],"supporting_citations":[{"why":"Introduces the deconfounder idea of estimating latent confounders as plug-in substitutes; the paper positions itself as a deconfounding-type method and must avoid its identifiability pitfalls.","marker":"[Wang and Blei, 2019]"},{"why":"Supplies the neural tensor network architecture whose bilinear products model interactions between entity pairs, here the trajectory and time embeddings.","marker":"[Socher et al., 2013]"},{"why":"Extends the two-way fixed-effects idea to reinforcement learning; the paper relaxes its additive and linear restrictions and moves to nonlinear neural dynamics.","marker":"[Bian et al., 2023]"},{"why":"Applies two-way fixed-effects to counterfactual prediction in a contextual bandit setting, a direct precursor of the two-way confounding model.","marker":"[Dwivedi et al., 2022]"},{"why":"Provides the cross-fitting and sample-splitting device that the theorem's proof uses to make the estimated transition function independent of the estimated latent factors.","marker":"[Chernozhukov et al., 2018]"},{"why":"Supplies the lower bound showing that linear dependence on the horizon is unavoidable for model-based reinforcement learning, justifying the form of the error bound.","marker":"[Jiang, 2024]"},{"why":"Provides neural network approximation rates that support the assumption that the transition estimation error $\\varepsilon_P$ is small.","marker":"[Schmidt-Hieber, 2020]"},{"why":"Supplies the MIMIC-III intensive-care dataset used to demonstrate the method on real trajectories.","marker":"[Johnson et al., 2016]"}],"fun_headline_variants":["Two-way deconfounder yields consistent policy values","Confounder-free consistency for off-policy evaluation","Two-way latent model ensures reliable policy estimates","Consistent OPE without Markov or proxy assumptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the estimated neural embeddings converge to the true trajectory- and time-specific confounders in mean squared error; the paper assumes this rather than proving it for its own model.","fun_headline_variants_meta":{"raw":{"variants":["Two-way deconfounder yields consistent policy values","Confounder-free consistency for off-policy evaluation","Two-way latent model ensures reliable policy estimates","Consistent OPE without Markov or proxy assumptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":1106,"prompt_tokens":839,"completion_tokens":267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":209}},"tokens_in":455,"tokens_out":267,"duration_ms":3187,"temperature":1.0,"reasoning_tokens":209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:21:35.854061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the tumor-growth data-generating process with known ground-truth $U_i$ and $W_t$, train the two-way deconfounder, and measure the mean-squared error of the estimated embeddings against the truth; if this error does not shrink as $N$ and $T$ grow, or if the policy-value error stays bounded away from zero while the other conditions of Theorem 1 hold, the consistency claim falls.","supporting_citations":[{"cited_title":"Off-policy Evaluation in Doubly Inhomogeneous Environments","cited_arxiv_id":"2306.08719","evidence_quote":"Extends the two-way fixed-effects idea to reinforcement learning; the paper relaxes its additive and linear restrictions and moves to nonlinear neural dynamics."}],"review_version":1}