{"id":"4b966207-9daf-4079-ba15-6f229de729d6","arxiv_id":"2505.12211","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ILQ clips model-imagined values of out-of-distribution actions at the maximum behavior value to limit optimism, reporting strong D4RL results, but its contraction proof is incorrect.","lead":"A new offline reinforcement learning method, ILQ, estimates values for actions missing from the training data by imagining them with a learned dynamics model and capping them at the best value seen in the data. The paper claims improved D4RL benchmark scores, but the convergence proof contains a mathematical error that invalidates the central theoretical claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ILB operator is not a gamma-contraction: the same-state max term y_lmt is only 1-Lipschitz, so Theorem 1 and the contraction-based Theorem 4 do not follow.","rationale":"The reader's weakest_assumption correctly identifies the out-of-support branch of the ILB operator as the failing point, and my independent check confirms it. The max-behavior-value term is a same-state max of Q, so it is 1-Lipschitz; the min with the gamma-discounted imagined value cannot recover a gamma contraction. The explicit counterexample with constant functions shows ||T_ILB Q1 - T_ILB Q2||_infinity = 0.75 > gamma*||Q1-Q2||_infinity for gamma=0.5. Since Theorem 4's proof relies on the contraction property at equations (40) and (42) to move gamma |QILB-Qbeta*| to the left-hand side, the error bound O(rmax/(1-gamma)^2) is not established. This is an internal inconsistency in the central theoretical argument, not merely a disagreement with prior assumptions. The paper itself states in A.7 that the theory is tabular, but the counterexample is tabular, so that caveat does not help. I therefore agree with the reader's REJECT verdict and see no reason to change it.","tokens_in":39963,"tokens_out":8512,"duration_ms":76389,"concrete_test":"Recompute Theorem 1 on the tabular MDP S={s}, A={a_in,a_ood}, beta(a_in|s)=1, r=0, gamma=0.5, bP(s|s,a)=1, delta=0, with Q1(s,.)=0.5 and Q2(s,.)=-0.5. Evaluate T_ILB under Eq. (2)-(4): for a_ood, T_ILB Q1 = min{0.25, 0.5} = 0.25 and T_ILB Q2 = min{-0.25, -0.5} = -0.5. The L_infinity distance is 0.75, exceeding gamma*||Q1-Q2||_infinity = 0.5, disproving the contraction. Then trace Theorem 4's proof: without the contraction step at Eqs. (40) and (42), the transposition yielding 1/(1-gamma) is invalid; verify whether any alternative argument in the appendix establishes the bound without contraction. If none does, the central theoretical claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central guarantee is invalid. In Definition 1 (Eq. 2-4), the out-of-support branch includes y_lmt = max_{a_hat in Supp(beta(s))} Q(s,a_hat), which compares Q-values at the same state s. Unlike y_img, this term has no discount factor and no expectation over next states. The proof of Theorem 1 claims the |y_lmt(Q1)-y_lmt(Q2)| case is 'analogous to (26)' and gamma-contractive, but |max_a Q1(s,a) - max_a Q2(s,a)| <= ||Q1-Q2||_infinity with constant 1, not gamma. The cross-case bound in (28)-(29) takes the max of this 1-Lipschitz term and the gamma-contractive y_img term, so the overall operator is only nonexpansive. A concrete tabular counterexample: S={s}, A={a_in,a_ood}, beta(a_in|s)=1, r=0, gamma=0.5, bP(s|s,a)=1, delta=0. Set Q1(s,.)=0.5 and Q2(s,.)=-0.5. Then T_ILB Q1(s,a_ood)=0.25 but T_ILB Q2(s,a_ood)=-0.5, giving distance 0.75 > gamma*||Q1-Q2||_infinity = 0.5. Therefore Theorem 1's gamma-contraction claim is false. Theorem 4's proof explicitly invokes the contraction inequality at Eqs. (40) and (42) (|T_ILB QILB - T_ILB Qbeta*| <= gamma |QILB-Qbeta*|) to obtain the 1/(1-gamma) factors; without contraction those rearrangements collapse, so the claimed same-magnitude OOD error bound is unproved. The empirical results and the tabular-MDP caveat in A.7 do not repair this internal proof error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Imagination-Limited Q-learning (ILQ), an offline RL algorithm that evaluates out-of-distribution (OOD) actions by computing an 'imagined' value with a learned dynamics model and then clipping it by the maximum in-support behavior value. For in-support actions it uses the standard Bellman backup. The authors claim that the resulting ILB operator is a gamma-contraction (Theorem 1), which guarantees convergence, and that the fixed-point value gap between ILQ and the support-constrained Bellman optimality operator is O(rmax/(1-gamma)^2) for both in- and out-of-distribution actions (Theorem 4). The paper reports state-of-the-art results on D4RL MuJoCo, Maze2D, and Adroit tasks, together with sensitivity analyses and ablations.","tokens_in":40378,"tokens_out":9390,"duration_ms":85306,"significance":"The algorithmic idea is simple and appealing, and the empirical evaluation is substantial: three D4RL domains, multiple task types, per-task hyperparameters, sensitivity studies for delta and eta, ablations of the two components of the OOD target, and a computation-time comparison. The authors also honestly state the tabular-MDP scope of their theory (Section A.7). If the contraction and error-bound theorems were correct, ILQ would offer a valuable alternative to pessimistic regularization in offline RL. However, the primary theoretical contribution is invalid: Theorem 1 is false and Theorem 4 is unsupported. The paper's contribution therefore reduces to an empirical heuristic, which may still be useful but is not the contribution the authors claim. No code is provided in the submission, so the reproducibility of the empirical results rests on the hyperparameter tables and the usual detail level.","major_comments":[{"comment":"The case |y^Q1_lmt - y^Q2_lmt| is not 'analogous to (26)'. From (24), y^Q_lmt = max_{â in Supp(beta(·|s))} Q(s,â) is computed at the same state s and has no discount or expectation structure; the best uniform bound is |max_a Q1(s,a) - max_a Q2(s,a)| <= ||Q1 - Q2||_inf, with constant 1. The cross-case bound in (28)-(29) takes the maximum of this 1-Lipschitz term and the gamma-contractive y_img term, so T_ILB is at best nonexpansive. A concrete violation of the claimed contraction is: S={s}, A={a_in, a_ood}, beta(a_in|s)=1, r=0, gamma=0.5, Phat(s|s,a)=1, delta=0; set Q1(s,·)=0.5 and Q2(s,·)=-0.5. Then T_ILB Q1(s,a_ood)=0.25 while T_ILB Q2(s,a_ood)=-0.5, giving distance 0.75 > 0.5 = gamma * ||Q1 - Q2||_inf. Theorem 1 is therefore false.","section":"Appendix A.1, proof of Theorem 1"},{"comment":"Equations (40) and (42) both use |T_ILB Q_ILB - T_ILB Q_beta*| <= gamma |Q_ILB - Q_beta*|, an inequality that follows only from the gamma-contraction of T_ILB. Since the operator is not a gamma-contraction (previous comment), the rearrangement in (41) and (46) that produces the 1/(1-gamma)^2 factors is invalid. Thus the central claim that OOD action-values have the same error magnitude as in-distribution ones is unproved.","section":"Appendix A.1, proof of Theorem 4"},{"comment":"The claimed convergence of value iteration also rests on the gamma-contraction property. With T_ILB only nonexpansive, the Banach fixed-point theorem does not apply, and the paper provides no alternative argument that the iterates converge to a unique fixed point. The tabular setting does not rescue this, since the counterexample in the first major comment is a tabular MDP.","section":"Section A.1, Theorem 1 and convergence"}],"minor_comments":[{"comment":"The equality |min(a,b)+delta - c| = max(|a+delta-c|, |b+delta-c|) is not true in general; for example, a=1, b=2, c=0, delta=0 gives LHS=1 and RHS=2. The proof should use an inequality (which is sufficient for the subsequent argument), not an equality.","section":"Appendix A.1, Eq. (43)"},{"comment":"The theorem statement refers to 'epsilon_r' but only epsilon_pi and epsilon_P are defined in Theorem 2; the notation should be corrected or the undefined quantity removed.","section":"Theorem 4 statement"},{"comment":"The support condition beta(a|s)>0 and the notation Supp(beta(·|s)) should be formally defined, since the behavior policy beta is used in a mathematical definition without being introduced as a distribution.","section":"Definition 1"},{"comment":"The abstract contains a typo: 'overly constraints' should be 'overly restrictive constraints'.","section":"Abstract"},{"comment":"The maze2d rows in Table 2 lack column separators, making the scores hard to read; this should be fixed in the camera-ready version.","section":"Section A.2, Table 2"},{"comment":"The offset delta is described as 'a hyperparameter with a small absolute value', but Table 4 lists values as large as 2 and -2; the text should clarify the practical range and the role of delta in compensating for estimation error.","section":"Section 4.2 and Table 4"}],"recommendation":"reject","confidential_remarks":"The core issue is not a matter of presentation: the ILB operator in Definition 1 does not enjoy the claimed contraction property, and the counterexample in my report is a valid tabular MDP. Even if the empirical results are strong, the main advertised contribution — a same-order error bound for OOD values — is unsupported. I do not see a way to fix this within the manuscript's current scope, since changing the operator (e.g., discounting y_lmt) would alter the method. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me get straight to it. The headline claim does not hold up: the ILB operator is not a gamma-contraction. The proof of Theorem 1 treats the |y_lmt(Q1) - y_lmt(Q2)| case as analogous to the Bellman case, but it is not. y_lmt is a max over Q-values at the same state, and the max is 1-Lipschitz. There is no discount factor in front of it. The stress-test counterexample is correct: with one state, two actions, beta supported only on a_in, r = 0, gamma = 0.5, Q1 = 0.5 and Q2 = -0.5 gives |T_ILB Q1 - T_ILB Q2| = 0.75 > 0.5 = gamma ||Q1 - Q2||_inf. So Theorem 1 is false as stated, and Theorem 4's rearrangement, which explicitly invokes contraction at Eqs. (40)-(42), collapses with it.\n\nThat is a load-bearing flaw. The paper's stated theoretical contribution, same-magnitude OOD error bounds, rests on that contraction, so I cannot accept the theory as written. It may be repairable: the operator is still nonexpansive, and nonexpansiveness plus additional structure can sometimes still yield convergence, but the analysis would need to be redone and the gamma-contraction claim dropped.\n\nWhat is genuinely new: the clipping idea, imagining OOD values with a learned dynamics model and capping them at the max behavior value, is a real combination that I have not seen in MCQ or ROMI-BCQ. The ablations suggest both components matter. The empirical coverage is broad and the results are decent, though the state-of-the-art wording is softened by per-task eta/delta tuning and baselines borrowed from several papers with different protocols. There is also an ambiguity: the appendix says they use MOPO reward penalties on some tasks, but the defined ILB operator has no such penalty. That needs clarification.\n\nWho gets value from this? Someone working on value regularization in offline RL who wants the empirical recipe and can treat the theory with suspicion. I would not cite the theorem. I would reject in current form, but it deserves a serious referee: the empirical method is plausible, the flaw is specific and fixable, and the paper engages honestly with the prior literature. My recommendation: reject now; invite a revision that either proves a corrected contraction or weakens the claims to match a nonexpansive analysis.","headline":"The empirical recipe is a plausible new combination, but the central contraction proof is wrong: the ILB operator is not gamma-contractive, so the paper's headline theoretical claims collapse.","tokens_in":40924,"tokens_out":3211,"would_cite":false,"duration_ms":32397,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces an Imagination-Limited Bellman operator that provably converges and bounds out-of-distribution value error in offline RL.","keywords":["offline reinforcement learning","out-of-distribution actions","Bellman operator","Q-learning","dynamics model","diffusion behavior policy","value regularization","D4RL benchmark"],"falsifier":"On a one-state MDP, take one in-support action $a_1$ and one out-of-support action $a_2$, with reward $2$ for $(s,a_2)$ and no future state. Let $Q_1(a_1)=Q_1(a_2)=1$ and $Q_2(a_1)=Q_2(a_2)=0$. Then $y^{\\mathrm{img}}=2$ for both, while $y^{\\mathrm{lmt}}$ is $1$ for $Q_1$ and $0$ for $Q_2$, so the ILB operator maps the OOD pair to $1$ and $0$; the sup-norm difference is $1$, not $\\gamma$, contradicting the claimed $\\gamma$-contraction.","tokens_in":39746,"feed_emoji":"🤖","tokens_out":7361,"duration_ms":67637,"temperature":0.7,"pith_summary":"Offline reinforcement learning fails when learned policies pick actions absent from the logged data, because value estimates for such out-of-distribution actions run away optimistically. The paper's ILQ method keeps the optimism that those actions deserve but caps it: it imagines what an OOD action would be worth under a learned dynamics model, then clamps that imagined value to the maximum value seen for in-distribution actions. On tabular MDPs, the paper proves the resulting Imagination-Limited Bellman operator converges to a unique fixed point and that its value error for OOD actions has the same order of magnitude as the error for in-distribution actions. On the D4RL benchmark the method reports higher average scores than the compared policy-constraint and value-regularization baselines across MuJoCo, Maze2D, and Adroit tasks.","feed_headline":"Capped imagined Q-values tame offline RL over-optimism","feed_subtitle":"New Bellman operator converges and keeps out-of-distribution value error at the same order as in-distribution error.","key_machinery":"The central object is the Imagination-Limited Bellman (ILB) operator, which splits the backup into two branches: for state-action pairs in the support of the behavior policy it applies the standard Bellman target $r+\\gamma\\,\\mathbb{E}\\max_{\\tilde a'} Q(s',\\tilde a')$; for out-of-support actions it takes $\\min\\{y^{\\mathrm{img}}, y^{\\mathrm{lmt}}\\}+\\delta$, where $y^{\\mathrm{img}}$ is a one-step bootstrapped value from an empirical dynamics model and $y^{\\mathrm{lmt}}$ is the maximum Q-value over actions sampled from a learned diffusion model of the behavior policy. The min operation is the load-bearing mechanism: it preserves the imagined value whenever it is below the behavior maximum, and only then clips it, which the paper argues retains reasonable optimism while steering the actor toward in-distribution actions.","core_discovery":"The paper's central claim is that a Bellman backup can be modified so that out-of-distribution actions are neither trusted blindly nor suppressed outright: for in-support $(s,a)$ it keeps the standard backup, and for out-of-support $(s,a)$ it uses the minimum of a model-imagined value and the maximum in-distribution value at that state, plus a small offset. The authors prove (Theorem 1) that this Imagination-Limited Bellman operator is a $\\gamma$-contraction in the sup norm, so repeated application converges to a unique fixed point. They then bound the gap between that fixed point and the support-constrained Bellman optimality value, obtaining $O(r_{\\max}/(1-\\gamma)^2)$ for both in- and out-of-distribution actions (Theorem 4), and argue this shows the bias introduced by ILQ is comparable to, not worse than, the bias on actions already in the data. The intended consequence is that offline learning can exploit genuinely good OOD actions while avoiding collapse from phantom high values, without the blanket pessimism of conservative value regularization.","pith_inferences":["The same min-cap idea could be applied to actor objectives or to model-based rollout policies, not just Q-targets; the paper only applies it inside the Bellman target.","A testable extension would replace the hard cap with a quantile of behavior values, trading off how much OOD optimism is retained; the paper's ablation suggests the cap is essential for stability in most tasks.","The paper's own appendix concedes that the Lipschitz reward assumption fails for sparse-reward Maze tasks, so the formal error-bound guarantee does not cover those empirical successes; closing that gap would require a weaker smoothness condition.","Monitoring $y^{\\mathrm{img}}-y^{\\mathrm{lmt}}$ during training could serve as a diagnostic for dynamics-model error: the paper's ablation shows the difference can grow explosively when the cap is removed."],"forward_implications":["Q-iteration under the ILB operator converges to a unique fixed point in tabular MDPs, so policy evaluation with capped OOD targets is stable.","The value gap between the ILB fixed point and the support-constrained optimal value is $O(r_{\\max}/(1-\\gamma)^2)$ for OOD actions, the same order as in-distribution actions, so the added bias from the cap does not grow relative to the in-data bias.","Because OOD values are capped by the maximum behavior value, the learned policy has a built-in incentive to favor actions that appear in the dataset when imagined values are unreliable.","The method outperforms the compared baselines on average across the D4RL MuJoCo, Maze2D, and Adroit suites in the paper's experiments."],"supporting_citations":[{"why":"Supplies the conservative value-regularization baseline whose bias ILQ aims to avoid, and the error-bound comparison ILQ aligns with within the support region.","marker":"[Kumar et al., 2020]"},{"why":"Provides the prior use of maximum behavior value as an OOD target and is the closest regularizer ILQ extends.","marker":"[Lyu et al., 2022]"},{"why":"Supplies the reward Lipschitz assumption and the behavior-value regularization baseline used for comparison.","marker":"[Huang et al., 2024]"},{"why":"Defines the batch-constrained policy-constraint baseline and the problem setting ILQ addresses.","marker":"[Fujimoto et al., 2019]"},{"why":"Provides the diffusion model machinery used to fit the behavior policy for sampling the limitation value.","marker":"[Ho et al., 2020]"},{"why":"Supplies the dynamics-model implementation used to generate imagined next states and rewards.","marker":"[Yu et al., 2020]"},{"why":"Provides the D4RL benchmark datasets on which all empirical comparisons are run.","marker":"[Fu et al., 2020]"},{"why":"Provides the implicit Q-learning baseline and is the source of several comparison numbers used in the tables.","marker":"[Kostrikov et al., 2022]"}],"fun_headline_variants":["Offline RL with bounded imagination beats over-optimism","Imagination-limited Q-learning: exploit good OOD, cap the rest","Capped model-imagined values keep offline RL honest","Offline RL: dream big but cap the value estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire convergence and error-bound argument rests on the assumption that the Imagination-Limited Bellman operator shrinks differences between Q-functions by the discount factor; if the cap by the maximum in-distribution value fails to shrink differences that fast, the convergence and bias conclusions do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Offline RL with bounded imagination beats over-optimism","Imagination-limited Q-learning: exploit good OOD, cap the rest","Capped model-imagined values keep offline RL honest","Offline RL: dream big but cap the value estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1366,"prompt_tokens":974,"completion_tokens":392,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":322}},"tokens_in":590,"tokens_out":392,"duration_ms":4140,"temperature":1.0,"reasoning_tokens":322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:38:30.840218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a one-state MDP, take one in-support action $a_1$ and one out-of-support action $a_2$, with reward $2$ for $(s,a_2)$ and no future state. Let $Q_1(a_1)=Q_1(a_2)=1$ and $Q_2(a_1)=Q_2(a_2)=0$. Then $y^{\\mathrm{img}}=2$ for both, while $y^{\\mathrm{lmt}}$ is $1$ for $Q_1$ and $0$ for $Q_2$, so the ILB operator maps the OOD pair to $1$ and $0$; the sup-norm difference is $1$, not $\\gamma$, contradicting the claimed $\\gamma$-contraction.","supporting_citations":[{"cited_title":"Mildly conservative q-learning for offline reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Provides the prior use of maximum behavior value as an OOD target and is the closest regularizer ILQ extends."},{"cited_title":"Offline reinforcement learning with behavior value regularization","cited_arxiv_id":null,"evidence_quote":"Supplies the reward Lipschitz assumption and the behavior-value regularization baseline used for comparison."},{"cited_title":"Off-policy deep reinforcement learning without exploration","cited_arxiv_id":null,"evidence_quote":"Defines the batch-constrained policy-constraint baseline and the problem setting ILQ addresses."},{"cited_title":"Offline reinforcement learning with implicit q-learning","cited_arxiv_id":null,"evidence_quote":"Provides the implicit Q-learning baseline and is the source of several comparison numbers used in the tables."}],"review_version":1}