{"id":"15d8ac12-7881-42f9-8039-eb6e5bfdfbc6","arxiv_id":"2607.05375","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Fitted occupancy-ratio evaluation (FORE) contracts in KL divergence under only occupancy-ratio realizability, enabling offline policy evaluation without Bellman completeness.","lead":"This paper introduces FORE, an iterative algorithm that estimates the occupancy ratio between a target policy and offline data by repeatedly projecting an adjoint Bellman image onto a log-ratio class in KL divergence. It proves that ratio realizability alone — not Bellman completeness — suffices for off-policy evaluation, replacing a standard restrictive assumption.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Advertised neural/GBM implementations are outside the proven guarantees: the KL-projection contraction relies on convex H (A2), and nonconvex classes can fail even when the ratio is realizable.","rationale":"I read the paper in good faith: the population theory (Lemma 3.1, Theorem 4.1) is carefully developed, the finite-sample analysis in Appendix C follows a standard ERM template, and the claimed contrast with Bellman completeness is meaningful for convex exponential-family ratio classes. The most serious soft spot is the mismatch between the theory's convexity requirement (A2) and the algorithm's advertised nonconvex implementation (neural networks, gradient-boosted trees). This is a load-bearing gap because the Pythagorean projection inequality that eliminates completeness is exactly where convexity enters; without it, even realizable targets can fail to contract. The reader identified this as one of the weakest assumptions, and I agree. The other limitation—positivity/lower-tail margin (A3/A6) and the coverage-stopped estimand change—is real but is explicitly and honestly scoped in the paper's Limitations paragraph; the paper does not claim to estimate the full target value when support fails. That is a boundary of the result, not a defect. Because the reader's CONDITIONAL verdict already accounts for the convexity gap and the lack of reproducible code, my concern does not move the verdict; it sharpens the reason. The concrete two-point test would settle whether the nonconvex gap is actually breachable: if the projected recursion can increase KL error in a realizable, nonconvex W, then the advertised implementation is not merely unproven but can fail qualitatively.","tokens_in":74179,"tokens_out":13023,"duration_ms":149184,"concrete_test":"Construct a two-atom example: let ν be uniform on {0,1}, W={v0,v1} with v0=(0.9,0.1), v1=(0.1,0.9), so H is the two-point nonconvex log-ratio class. Choose a small tabular MDP whose adjoint Bellman operator Bπ_γ has fixed point ωπ,γ=v0 and, for some ω∈W, sends ω to u=(0.4,0.6). Computationally or analytically evaluate the exact population iterates of (6): Dν(ω^(k+1)∥v0) = Dν(Π_W Bπ_γ ω^(k)∥v0). If for any iterate this error increases rather than contracting by factor γ, then Lemma B.2's convexity requirement is essential and Theorem 4.1 does not extend to nonconvex H. If it still contracts, derive the missing inequality and state under which weaker condition it holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central no-completeness claim rests on the KL-projection contraction in Theorem 4.1, whose proof uses the Pythagorean-type inequality Lemma B.2. That lemma requires H to be convex, closed, and bounded (A2, A4), because the projection is taken onto a normalized exponential family over a convex natural-parameter space. Without convexity, the identifying score condition E_{\\bar u}(h-h*)-E_u(h-h*)\\ge0 need not hold, and the inequality D(\\Pi_W u\\| w) \\le D(u\\| w) for w\\in W can fail even when the true ratio is realizable. The paper's Section 3.3 explicitly suggests neural networks and gradient-boosted trees as implementation classes, and Section 3.3 even gives batched stochastic gradients for nonlinear classes. These are nonconvex function classes, so the advertised implementations do not satisfy A2. The gap is not merely cosmetic: for a two-point nonconvex class W={v0,v1}, one can choose an adjoint Bellman image u that is closer in reverse KL to v1 than to v0 while the true ratio is v0; then the KL projection sends u to v1 and D(\\Pi_W u\\| v0) can exceed D(u\\| v0), breaking the one-step contraction in Theorem 4.1. Thus, as stated, the theorem does not cover the method's recommended nonlinear learners. For convex log-ratio classes the argument appears sound, so this is a scoping gap rather than an internal inconsistency of Theorem 4.2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FORE, a fitted iterative method for estimating the discounted occupancy ratio ωπ,γ in offline policy evaluation. At each iteration, FORE replaces the exact adjoint Bellman image Bπγω by its KL projection onto a normalized exponential-family ratio class W = {ωh : h ∈ H}, using a single-level loss that depends only on initial-state moments and one-step target-policy transitions. The main theoretical contribution is a population contraction result (Theorem 4.1) showing that, when H is convex, closed, and bounded (A2, A4), the projected recursion contracts in KL divergence up to an approximation error εKL = inf_{v∈W} Dν(v∥ωπ,γ). This removes the need for Bellman completeness, adjoint Bellman completeness, or critic dual-completeness: realizability of the ratio itself suffices. The finite-sample result (Theorem 4.2) provides a high-probability bound on the generalized KL error of the empirical recursion, decomposing into an exponentially decaying iteration term, an εKL term, and a local-Rademacher statistical term. The paper also gives a doubly robust value estimator (Theorem 5.2), an occupancy-weighted FQE procedure (Theorem 5.3), and a coverage-stopped variant that targets the subprobability occupancy accumulated before the first uncovered state–action pair (Section 6). Numerical experiments on a Baird-style MRP and a linear-Gaussian MDP illustrate that FORE is stable where linear FQE diverges.","tokens_in":74538,"tokens_out":4128,"duration_ms":51348,"significance":"If the central claim holds, the paper makes an important contribution to offline policy evaluation: it identifies discounted occupancy-ratio realizability as a sufficient condition for fitted ratio evaluation without any completeness-type closure assumptions. The key insight — that the adjoint Bellman operator is a KL contraction and that KL projection aligns with this contraction — is elegant and, as far as the core derivations go, correctly worked out. The finite-sample analysis appears technically sound, and the doubly robust and coverage-stopped extensions are valuable in themselves. I verified several central pieces: Lemma 3.1 (joint convexity of KL plus data processing), the moment identity (3), the realizable-case Pythagorean argument behind Theorem 4.1, and the doubly robust identity in Theorem 5.2. The paper is also careful to state its assumptions explicitly and to acknowledge the positivity/coverage limitations. However, as detailed below, the advertised implementation scope (neural networks, gradient-boosted trees) is not covered by the proven guarantees, which require convexity of the log-ratio class. This is a substantial scoping gap that the authors should address, e","major_comments":[{"comment":"The paper advertises neural networks and gradient-boosted trees as implementation classes (Intro, §3.3, and §6.3), and §3.3 explicitly provides batched stochastic gradients for nonlinear classes. However, the central guarantees — Theorem 4.1, Theorem 4.2, and Theorem 6.3 — require Assumption A2 that the log-ratio class H be convex, closed, and totally bounded. This convexity is not incidental: Lemma B.2 (the Pythagorean inequality for KL projections onto the normalized exponential family) and the star-shapedness of H∆ and G× used in Lemma C.2 both rely on convexity of H. For a nonconvex two-point class W={v0,v1}, one can choose an adjoint Bellman image u that is closer in reverse KL to v1 than to v0 while v0 is the true ratio; then the KL projection maps u to v1 and Dν(Π_W u∥v0) can exceed Dν(u∥v0), breaking the one-step contraction that underlies Theorem 4.1. Thus, as stated, the theore","section":"§3.3, §4.1, Assumption A2, Lemma B.2"},{"comment":"Theorem 4.2 analyzes the exact-ERM version of Algorithm 1, requiring ĥ_{k+1} ∈ arg min_{h∈H} L̂(h;ω̂(k)). For the nonlinear classes advertised in §3.3, exact ERM is computationally intractable, and the batched stochastic gradient variant described in §3.3 is not analyzed: the theorem's uniform statistical bounds are for the exact minimizer, not for one or a few gradient steps. The numerical experiments use log-linear classes, so they are consistent with the theorem. But the practical algorithm proposed for nonlinear classes lacks the advertised finite-sample guarantee. This reinforces the scoping gap in the first major comment.","section":"Theorem 4.2 / Algorithm 1"}],"minor_comments":[{"comment":"The abstract states “each iteration can be implemented with standard supervised learners, such as gradient-boosted trees or neural networks.” Given Assumption A2, this sentence is misleading. Please qualify it to “convex supervised learners” or move the nonlinear potential to a heuristic remark.","section":"Abstract / Introduction"},{"comment":"The proof sketch for the realizable case is clear. However, the non-realizable case uses Lemma B.3, whose proof relies on Lemma B.2 (convex projection inequality). The dependence of the constant C_app = e^{4R} on A4 is fine, but the convexity dependence should be stated explicitly in the theorem statement, not only in A2.","section":"§4.1, Theorem 4.1"},{"comment":"The sentence “In practice, one may use flexible classifiers, such as neural networks or gradient-boosted trees” in §6.3 is again outside the theorem scope: the ratio class H must still be convex (A2) for Theorem 6.3. The classifier class C need not be convex, but the ratio class does.","section":"§6.3, Algorithm 2"},{"comment":"The reference list is unusually heavy on the authors' own work (van der Laan, Kallus, et al.) and includes several items dated 2025/2026 with IDs that may not be final. This makes it hard for a reader to separate established results from new claims; consider trimming self-citations and using stable versions where possible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is appropriate. The central theoretical insight — KL-projected adjoint Bellman iteration with realizability-only approximation — is sound for convex log-ratio classes, and the finite-sample analysis is careful. My recommendation for major_revision is driven by the paper's own advertising: the abstract and §3.3 recommend neural networks and gradient-boosted trees, but the proven guarantees require convexity of H. This is not a correctness error in the theorems, but it is a significant mismatch between the claim and the proof. A revision that sharply delineates the convex-class theorem and demotes nonlinear learners to heuristics (or supplies additional analysis) would make the contribution publishable. I would not reject, as the linear/convex case is already a meaningful advance over existing completeness-dependent analyses."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: FORE is a real result. The KL-aligned contraction idea — the adjoint Bellman operator contracts in relative entropy and the KL projection preserves that geometry — is genuinely new and lands. Under ratio realizability, the projected population recursion contracts without any Bellman completeness, and Theorem 4.2 gives a finite-sample bound that follows the standard empirical-process template. I hand-checked the key steps: Lemma 3.1, the moment identity, Theorem 4.1 in the realizable case, and the doubly robust identity. They hold. The paper is also honest about its limitations: full-ratio recovery needs positivity and a lower-tail margin, and the coverage-stopped extension changes the estimand.\n\nThe main soft spot is the one the stress-test flags: the theory requires the log-ratio class H to be convex (A2), because the one-step contraction relies on the Pythagorean inequality for KL projections onto an exponential family. But Section 3.3 recommends gradient-boosted trees and neural networks, and gives batched stochastic gradients for nonlinear classes. Those are not convex classes, so the advertised implementations are outside the proven guarantees. The stress-test's two-point counterexample shows the projection inequality can fail for nonconvex classes even when the ratio is realizable. So this is a real scoping gap. It is not a flaw in the main theorem for convex classes, but the paper should either add a caveat that the finite-sample guarantees are for convex H, or discuss what extra structure would extend them to neural nets. As written, a reader could reasonably think Algorithm 1 with a neural net is covered.\n\nThe other caveats are minor: no code or data for the experiments, and the self-citation density is high, but the cited prior work is legitimate and the novelty stands out from it.\n\nWho is this for? Researchers working on off-policy evaluation, DICE methods, and the realizability-vs-completeness question. It deserves a serious referee; the right response is major revision with a clear statement of the convexity scope, not rejection.","headline":"FORE's core claim — ratio realizability suffices, no Bellman completeness — is correct for convex log-ratio classes; the neural/GBM implementations advertised in Section 3.3 sit outside the proven guarantees, a real scoping gap but not a fatal one.","tokens_in":75071,"tokens_out":3156,"would_cite":true,"duration_ms":33214,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that offline policy evaluation can be based on fitting the discounted occupancy ratio alone, with no Bellman-completeness condition when the ratio is realizable.","keywords":["off-policy evaluation","discounted occupancy ratio","adjoint Bellman equation","KL projection","Bellman completeness","fitted iteration","offline reinforcement learning","coverage diagnosis"],"falsifier":"In a finite MDP with a one-dimensional feature where the true discounted occupancy ratio is provably outside the fitted log-linear class, run the population projected recursion; the theorem predicts KL error plateauing at a positive multiple of the KL approximation error rather than decaying to zero. If the error instead continues to zero, the claimed control by approximation error alone is false. Alternatively, a target policy that assigns zero probability to an offline-supported action should make the full-ratio bound fail and the method should only recover the coverage-stopped value.","tokens_in":73999,"feed_emoji":"🎯","tokens_out":4844,"duration_ms":58501,"temperature":0.7,"pith_summary":"The paper introduces FORE, a fitted iteration that estimates the discounted occupancy ratio by repeatedly applying a KL projection to adjoint Bellman updates. Its central claim is that this projected recursion contracts in relative entropy toward the true ratio at the discount factor, so the only substantive approximation requirement is that the true discounted occupancy ratio be realizable (or well approximated) by the chosen log-ratio class. No Bellman completeness of a value class, adjoint Bellman completeness of a ratio class, or dual completeness of a critic class is needed. The finite-sample bound separates a geometrically decaying initialization term, a KL approximation error, and a statistical error controlled by the complexity of the ratio class. A coverage-stopped variant targets the occupancy accumulated before the first uncovered state-action pair, giving a conservative value bound when full coverage fails.","feed_headline":"Fitted ratios free offline policy evaluation from Bellman completeness","feed_subtitle":"A KL-projected adjoint Bellman recursion contracts to the true ratio up to approximation error, dropping the completeness requirement.","key_machinery":"The central object is the adjoint Bellman operator combined with KL projection onto a normalized log-ratio class. The argument's load-bearing identity is the contraction of the adjoint Bellman map in relative entropy, which follows from joint convexity of KL divergence and the data-processing inequality for Markov kernels. A key lemma converts the KL-projected update into a single-level density-ratio objective evaluated on initial-state moments and one-step target-policy transitions. This aligns the geometry of the Bellman update with the geometry of the projection, which is what lets ratio realizability replace Bellman completeness.","core_discovery":"The discounted occupancy ratio is characterized as the unique fixed point of the adjoint Bellman operator. FORE replaces each exact adjoint Bellman image with its KL projection onto a normalized exponential-family log-ratio class. Because the unprojected adjoint map contracts relative entropy by the discount factor and the projection is taken in the same KL geometry, the population projected recursion satisfies a contraction up to a KL approximation error of the true ratio. The projection can be computed from one-step transition data through a single-level convex loss. This removes the need for any completeness assumption: the error is governed by how well the class approximates the fixed po","pith_inferences":["The theorem's convexity assumption is a real restriction: the proven guarantees cover closed convex log-ratio classes, so the advertised neural-network and gradient-boosted implementations are an extrapolation until nonconvex analogues of the projection inequality are established.","The method's reverse-KL projection requires the true ratio to be strictly positive on the offline support; near-deterministic target policies that put zero occupancy on some offline region fall outside the full-ratio guarantees, and only the coverage-stopped estimand is recovered.","A natural stress test is to find an MDP where ratio realizability holds but every standard critic-based estimator fails; if FORE remains stable there, it isolates ratio realizability as the essential ingredient rather than a side effect of minimax balance.","The same KL-contraction mechanism may extend to stationary-distribution correction and undiscounted settings under a one-step strong data-processing inequality, as the appendix sketches; verifying this in nonconvex classes would broaden the result."],"forward_implications":["Whenever the discounted occupancy ratio is realizable by the log-ratio class, FORE converges to the true ratio at a geometric rate without value-, adjoint-, or critic-completeness assumptions.","Each FORE iteration is a single-level convex (or stochastically optimized) density-ratio objective; no separate critic class has to be tuned.","The fitted ratio plugs into reward reweighting, doubly robust estimation, and occupancy-weighted FQE, with value-error bounds that factor the ratio error against a Bellman residual or value-approximation error rather than a completeness condition.","In datasets with partial coverage, coverage-stopped FORE estimates the occupancy accumulated before the first uncovered state-action pair, yielding a conservative lower bound on target value for nonnegative rewards and a diagnostic occupancy mass.","The approximation term in the value bound scales like the square root of the KL approximation error divided by one minus the discount, while the statistical term carries the usual horizon factor, so long-horizon behavior remains controlled."],"fun_headline_variants":["FORE drops Bellman completeness for fitted occupancy ratios","KL-projected ratio recursion escapes completeness assumption","Fitted occupancy ratio: no completeness, just realizability","Off-policy eval without completeness via adjoint Bellman fit","Coverage-stopped FORE gives conservative value lower bounds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The guarantee collapses if the target occupancy ratio is zero on a positive-mass offline region or the one-step target transitions are not covered by the data; then FORE no longer estimates the full target-policy value, only the coverage-stopped one.","fun_headline_variants_meta":{"raw":{"variants":["FORE drops Bellman completeness for fitted occupancy ratios","KL-projected ratio recursion escapes completeness assumption","Fitted occupancy ratio: no completeness, just realizability","Off-policy eval without completeness via adjoint Bellman fit","Coverage-stopped FORE gives conservative value lower bounds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1094,"prompt_tokens":811,"completion_tokens":283,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":219}},"tokens_in":555,"tokens_out":283,"duration_ms":3832,"temperature":1.0,"reasoning_tokens":219,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:27:48.457872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a finite MDP with a one-dimensional feature where the true discounted occupancy ratio is provably outside the fitted log-linear class, run the population projected recursion; the theorem predicts KL error plateauing at a positive multiple of the KL approximation error rather than decaying to zero. If the error instead continues to zero, the claimed control by approximation error alone is false. Alternatively, a target policy that assigns zero probability to an offline-supported action should make the full-ratio bound fail and the method should only recover the coverage-stopped value.","supporting_citations":[],"review_version":2}