{"id":"6de1dd79-2e44-4e74-b3cc-7a4421aa483f","arxiv_id":"2608.02034","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Expectile n-step Q-learning (ENQ) applies an upper-expectile loss to the action-value TD error, reducing the pessimistic bias of multi-step returns in off-policy RL; the authors prove contraction and bias bounds and report competitive results across 27 tasks.","lead":"A new reinforcement learning method, ENQ, replaces the symmetric error in n-step Q-learning with an asymmetric expectile error, so the learned value emphasizes good logged continuations instead of being dragged down by poor ones. The authors prove the method converges and show it matches or beats a recent state-of-the-art baseline on 27 robotic tasks while running faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim that ENQ beats LQL is not yet established: baseline table cells are copied from a prior paper with different hyperparameters, so the aggregate comparisons are confounded.","rationale":"I focus on the empirical comparison because it is the load-bearing support for the central claim that ENQ is competitive with LQL. The theoretical analysis is internally consistent: Theorem 1's contraction proof couples trajectories and applies Lemma 1(v); Theorem 2's decomposition is a direct telescoping identity; Theorem 3's bounds follow from the displacement identity. Assumption 2, the reader's identified weak point, is indeed idealizing, but it is used only for the tau=1 endpoint result (Theorem 2c); the practical tau=0.8 recommendation still benefits from the monotonicity of expectiles in tau even without full optimal coverage, so it is not the most load-bearing assumption. The empirical concern is more direct: the paper itself flags in Appendix G that all baselines are imported from a prior paper and that ENQ uses different hyperparameters. With no uncertainty for baseline cells, a 10-point aggregate gap is not interpretable. This does not invalidate the method; it makes the central empirical claim conditional on a matched re-run. The reader's CONDITIONAL verdict is therefore appropriate, and my read does not change it.","tokens_in":20730,"tokens_out":7759,"duration_ms":70749,"concrete_test":"Rerun LQL and TD-n in the same JAX codebase, using ENQ's exact hyperparameters (action chunk h=4, same gamma, same alpha, same ensemble aggregation, same batch size and transition budget) on all 27 task instances, and compare per-seed distributions rather than point means. If LQL's aggregate mean remains below ENQ's under matched settings with overlapping or non-overlapping CIs, the comparative claim holds; if the gap shrinks or reverses, the reported advantage is an artifact of unmatched hyperparameters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that with tau=0.8 and n=4, ENQ attains the higher aggregate mean in four of five comparisons with LQL. This claim rests on tables whose TD, TD-n, and LQL entries were not rerun; Appendix G states they are 'taken directly from the results reported by Abraham, Shi, and Finn (2026)' and retain the original implementation's action-chunk size h=5, domain-specific policy coefficients, and ensemble aggregation. ENQ runs use different settings in the same comparison: h=4, gamma=0.995 on humanoidmaze-medium vs 0.99 for the baselines, alpha=3 on antmaze-giant vs 5 for published LQL, and ensemble-mean vs ensemble-minimum aggregation on RoboMimic. Because the baseline cells have no confidence intervals or seed-level data, the aggregate margins (e.g., OGBench FQL total 69.0 vs 59.0) cannot be attributed to the ENQ loss rather than to these confounds. The theoretical contraction result (Theorem 1) is independent and appears sound, but the headline empirical contribution is underdetermined by the reported comparisons.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Expectile n-step Q-learning (ENQ), a modification of off-policy n-step TD in which the squared loss on the action-value error is replaced by an asymmetric expectile loss. The authors prove that the resulting population operator is a gamma^n-contraction for every tau in (0,1] and any dynamics and behavior policy (Theorem 1), decompose the pessimistic bias of n-step TD under deterministic dynamics (Theorem 2), show that at tau=1 the operator is unbiased at Q* under an optimal-coverage assumption, and derive two-sided bias bounds plus a fixed-point error bound under stochastic dynamics (Theorem 3). Corollary 1 connects the tau=1 fixed point to LQL's lower-bound inequality at separations n, 2n, and so on. Empirically, using tau=0.8 and n=4 across 27 OGBench and RoboMimic instances, ENQ is reported to attain a higher aggregate mean than LQL in four of five comparisons, with additional ablations on expectile level, horizon, ensemble size, expectile placement, and stochastic transitions.","tokens_in":20954,"tokens_out":4530,"duration_ms":44300,"significance":"The theoretical core is a genuine contribution: the contraction theorem is clean, the bias decomposition in Equation (8) is illuminating, and the connection to LQL's lower-bound inequality is novel. The appendix proofs are detailed and internally consistent, and the experimental protocol is careful in several respects, notably matching the number of sampled transitions per update across ENQ and LQL and reporting bootstrap confidence intervals for all newly run ENQ results. However, the headline empirical claim—that ENQ is competitive with or better than LQL on aggregate—is currently underdetermined because the baseline cells in the main comparison tables were copied from a prior paper with different hyperparameters rather than rerun under matched settings. The theoretical results stand on their own, but the paper's central empirical contribution needs either matched baseline runs or a substantially narrowed claim.","major_comments":[{"comment":"The main comparative tables compare newly run ENQ results against TD, TD-n, LQL, ReBRAC, and IQL cells that are 'taken directly from the results reported by Abraham, Shi, and Finn (2026)' and were not rerun. The ENQ runs in the same tables use different action-chunk sizes (h=4 versus h=5), different discount factors on humanoidmaze-medium (gamma=0.995 versus 0.99), different policy coefficients on antmaze-giant (alpha=3 versus 5), and different target-ensemble aggregation on RoboMimic (ensemble-mean versus ensemble-minimum). These settings can materially affect success rates, so aggregate margins such as the OGBench FQL total (69.0 versus 59.0 in Table 1) and the humanoidmaze-giant total (80.9 versus 75.7 in Table 2) cannot be attributed to the expectile loss. This undermines the central empirical claim in 'Comparative Evaluation' that ENQ attains the higher aggregate mean in four of five comparisons.","section":"Appendix G, 'Policy extraction and baseline provenance'; Tables 1–3"},{"comment":"Baseline cells have no confidence intervals or seed-level data; only ENQ cells carry 95% bootstrap intervals. Consequently, even the direction of the aggregate comparisons is not statistically assessable: a margin of, for example, 10 points on the OGBench FQL total could be within seed-level noise, and the absence of baseline intervals makes the reported ranking unverifiable. The authors should either rerun the baselines with the same seeds and report their own intervals, or clearly restrict the comparative claims to settings where all entries were produced by the same pipeline.","section":"Tables 1–3"},{"comment":"The theoretical mechanism that upper expectiles correct n-step pessimism is established only under Assumption 2 (optimal coverage), which requires that for every in-support (s,a) the behavior policy places positive probability on continuations whose cumulative suboptimality gap is arbitrarily small. This is a strong condition that may fail in real offline datasets. The paper is transparent about this, but the consequence is that the theoretical support for the method's headline configuration (tau=0.8, n=4) is indirect: Theorem 2(c) concerns the tau=1 endpoint, and the stochastic-dynamics bounds in Theorem 3 are stated in terms of unobservable drift and variance quantities. The authors should state more explicitly that the empirical gains at tau=0.8 are not directly predicted by the endpoint result, and ideally provide a diagnostic of approximate optimal coverage on the benchmark datasets.","section":"Theorem 2(c) and Assumption 2; Appendix F"}],"minor_comments":[{"comment":"There is an inconsistency about the TD-n baseline horizon: Table 5 lists the published TD-n horizon as 'n=8 (default)', while Table 2 labels the baseline 'TD-4' and the experimental section states that for humanoidmaze-giant the published TD-n baseline uses n=4. Please reconcile these statements.","section":"Table 5 and Table 2"},{"comment":"The abstract and introduction say tau is 'the only method-specific hyperparameter added beyond n-step TD', but the ENQ results also depend on the conservative coefficient rho and on the action-chunk size h, which vary across comparisons. Since rho and h are inherited from the ensemble aggregation and policy-extraction recipes, the phrasing should clarify that tau is the only new hyperparameter of the loss itself, not of the full method.","section":"Abstract and Section 'Expectile n-step Q-learning'"},{"comment":"The throughput profiling uses a single seed (seed 10001) for each ensemble size. This is acceptable for a profiling study, but the 1.27x to 1.77x speedup ratios in Table 7 should be described as single-seed measurements without implying tight error bars.","section":"Figure 3, left panel"}],"recommendation":"major_revision","confidential_remarks":"The theoretical sections are solid and the contraction result (Theorem 1) is a clean contribution. The main risk is the empirical comparison: because the LQL and TD baseline cells are copied from a prior paper with different hyperparameters and no uncertainty intervals, the paper's central comparative claims are not yet supported. This is fixable within the manuscript's scope if the authors rerun the baselines under matched settings or substantially reframe the empirical contribution as an ablation study of ENQ's own behavior. If the baselines cannot be rerun, the paper should be repositioned as a theoretical-plus-ablations paper rather than a comparative empirical paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely clean: apply an upper-expectile loss directly to the n-step TD residual on the action-value error, with tau the only added hyperparameter. That placement is new relative to IQL (state-value expectile), XQL, and LEQ (lower expectile on model rollouts). Second, the theory is the strong part and it holds up: Theorem 1's gamma^n contraction is a direct expectile-Lipschitz argument, Theorem 2's bias decomposition under deterministic dynamics is correct and the tau=1 endpoint connection to LQL's lower-bound inequality is a nice observation. The appendix proofs check out, and the paper is honest about its idealizations (exact operator, greedy bootstrap) in Appendix F.\n\nThe soft spot is exactly where the stress-test note lands. The main empirical tables compare ENQ against TD, TD-n, and LQL numbers copied from Abraham et al. (2026), and those cells carry different action-chunk sizes (h=5 vs h=4), different gamma on humanoidmaze-medium (0.995 vs 0.99), different alpha on antmaze-giant (3 vs 5), and different ensemble aggregation on RoboMimic. So the aggregate margins (e.g., OGBench FQL total 69.0 vs 59.0) cannot be attributed to the ENQ loss. The paper does disclose this in Appendix G, which is good, but disclosure doesn't remove the confound. To establish the empirical claim they need to rerun the baselines in their own codebase with matched hyperparameters. The ablations are more convincing because they are self-contained, but they don't fix the headline comparison.\n\nIs the central argument sound? The theoretical mechanism is plausible and rigorously derived; the empirical mechanism is plausible but underdetermined. Assumption 2 (optimal coverage) is strong but explicitly stated, and the paper doesn't hide that the tau=1 endpoint needs it. The practical tau=0.8 choice is motivated by ablations, but those ablations were run on the benchmark itself, so there is some selection risk on the fixed hyperparameters.\n\nWho is this for? Researchers working on off-policy value learning with expressive policies, multi-step returns, and offline-to-online RL. The operator theory and the expectile-on-action-value idea will be useful even if the empirical claims need another pass. I'd cite the theoretical part. The paper deserves a serious referee: the math is solid, the idea is simple and worth testing, and the empirical issues are fixable by rerunning baselines under matched settings. I'd send it to review with a clear request for that rerun, not desk-reject it.","headline":"Solid theory, honest paper, but the headline empirical comparison is confounded because the baselines were imported with different hyperparameters; worth refereeing for the operator analysis alone.","tokens_in":21493,"tokens_out":1175,"would_cite":true,"duration_ms":12206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that an upper-expectile loss on n-step action-value residuals yields a gamma^n-contraction with a unique fixed point, and shows that a fixed tau=0.8, n=4 setting is competitive with the long-horizon Q-learning baseline…","keywords":["expectile Q-learning","multi-step returns","off-policy reinforcement learning","n-step TD","long-horizon value learning","contraction operator","offline-to-online RL","pessimism bias"],"falsifier":"Take a deterministic MDP where the behavior policy, at some supported (s,a), always follows a strictly suboptimal action within the next n-1 steps, so the cumulative discounted gap has a positive lower bound on every logged continuation. Then, by the definition of the operator, $T_n^{1}$ Q*(s,a) will be strictly below Q*(s,a) on that pair, contradicting Theorem 2(c); observing exactly that inequality in a finite experiment would also flag missing near-optimal coverage in the dataset.","tokens_in":20527,"feed_emoji":"🎯","tokens_out":11000,"duration_ms":85436,"temperature":0.7,"pith_summary":"Multi-step returns spread reward information quickly in off-policy reinforcement learning, but they drag along the logged behavior policy's suboptimal choices, creating a pessimistic bias that grows with horizon. The paper introduces Expectile n-step Q-learning (ENQ), which replaces the symmetric squared n-step TD loss with an asymmetric loss that up-weights positive action-value residuals through an expectile level tau and down-weights negative ones. It proves that the resulting n-step Bellman operator is a gamma^n-contraction for every tau in (0,1] and horizon n, so it always has a unique fixed point; under deterministic dynamics with an optimal-coverage condition, the tau=1 endpoint removes the pessimistic bias exactly at Q* on supported pairs, and the fixed point satisfies the lower-bound optimality inequality used by long-horizon Q-learning at separations n, 2n, .... With a single setting (tau=0.8, n=4) held fixed across 27 manipulation and navigation task instances, ENQ is competitive with the long-horizon baseline and shows higher training throughput in a matched profile. If these guarantees carry over to practical neural critics, ENQ would be a drop-in, hyperparameter-light replacement for n-step TD that needs no importance ratios, auxiliary value networks, or trajectory-level constraints.","feed_headline":"Upper expectiles fix n-step Q-learning's off-policy pessimism","feed_subtitle":"One fixed tau=0.8 and n=4 beats or matches the long-horizon baseline on 27 robot tasks.","key_machinery":"The load-bearing mechanism is the asymmetric squared expectile loss on the scalar n-step TD residual. With expectile level $\\tau > 1/2$, positive residuals (target above current estimate) receive weight $\\tau$ and negative residuals weight $1-\\tau$, so the critic is pulled toward an upper expectile of the distribution of n-step targets induced by logged actions and transition noise; this is what down-weights suboptimal continuations without any likelihood ratio. The contraction proof couples two value functions on the same trajectory, showing $|Z_n^{Q_1} - Z_n^{Q_2}| \\le \\gamma^n \\|Q_1 - Q_2\\|_\\infty$ almost surely, then uses the 1-Lipschitz property of expectiles (Lemma 1(v)) to transfer the bound; the Banach fixed-point theorem yields the unique fixed point. The bias analysis rests on the telescoping identity $W_n = \\sum_{k=0}^{n-1} \\gamma^k \\epsilon_k - \\sum_{k=1}^{n-1} \\gamma^k \\Delta^*(s_k,a_k)$, which separates the mean-zero Bellman noise $\\epsilon_k$ from the nonnegative suboptimality gaps $\\Delta^*$ of the logged actions; the monotonicity of expectiles in $\\tau$ converts this decomposition into a one-sided bias that can be tuned by $\\tau$.","core_discovery":"The paper's central claim is that placing an upper-expectile loss directly on the n-step action-value residual corrects the off-policy pessimism of multi-step returns while keeping their fast reward propagation. Formally, define $Z_n^Q(s,a)$ as the n-step target whose intermediate actions are drawn from the behavior policy and whose bootstrap is greedy; the expectile n-step Bellman operator is $(T_n^{\\tau} Q)(s,a) = e_{\\tau}[Z_n^Q(s,a)]$. Theorem 1 states that for every $\\tau \\in (0,1]$ and $n \\ge 1$, with arbitrary stochastic dynamics and behavior policies, $\\|T_n^{\\tau} Q_1 - T_n^{\\tau} Q_2\\|_\\infty \\le \\gamma^n \\|Q_1 - Q_2\\|_\\infty$, so $T_n^{\\tau}$ is a $\\gamma^n$-contraction with a unique fixed point $Q_n^{\\tau}$. Under deterministic dynamics, the pathwise gap to $Q^*$ decomposes as $W_n = \\sum_{k=0}^{n-1} \\gamma^k \\epsilon_k - \\sum_{k=1}^{n-1} \\gamma^k \\Delta^*(s_k,a_k)$, a mean-zero noise term minus the discounted behavioral drift; therefore the $\\tau=1/2$ operator is exactly as pessimistic as uncorrected n-step TD, the fixed point $Q_n^{\\tau}$ is nondecreasing in $\\tau$ and bounded above by $Q^*$, and under optimal coverage the $\\tau=1$ operator fixes $Q^*$ on covered in-support pairs, its fixed point obeying the separation-n lower-bound inequality of LQL. Under stochastic dynamics, the one-application bias satisfies $-\\mu_n(s,a) \\le B_{\\tau}(s,a) \\le -\\mu_n(s,a) + \\kappa_{\\tau} \\sigma_n(s,a)$ with $\\kappa_{\\tau} = \\frac{2\\tau-1}{2(1-\\tau)}$ and horizon-independent variance constants, yielding a fixed-point error bound with the same constants as the LQL stochasticity analysis. Empirically, with $\\tau=0.8$ and $n=4$ fixed, ENQ attains the higher aggregate mean in four of five comparisons against the LQL baseline across 27 task instances, with a 1.27-1.77x training-throughput advantage in a matched profiling study.","pith_inferences":["The theorems analyze an exact population operator with greedy bootstrap, so they do not quantify function-approximation or finite-sample error; the natural next step, which the paper leaves open, is a finite-sample bound for expectile n-step regression under approximate operators.","The monotone dependence of the fixed point on tau suggests tau can serve as a practical bias dial; the ablation's finding that the preferred tau grows with n on one domain but not another implies tau and n should be selected jointly, an implication the paper states but does not resolve.","ENQ's throughput advantage widens with ensemble size, so as a backbone ENQ may scale better with critic-ensemble compute than trajectory-constraint methods; this is an extrapolation beyond the paper's data rather than a claim it makes.","The direct placement of the expectile on the action-value residual outperformed an n-step IQL-style baseline that routes the same asymmetry through a state-value function; a testable extension would vary the baseline's value-network capacity to see whether the placement effect persists."],"forward_implications":["ENQ is a drop-in replacement for n-step TD: it adds only the expectile level tau, and needs no action likelihoods, importance ratios, auxiliary value networks, or trajectory-level pairwise penalties.","The gamma^n-contraction property gives a well-posed learning target for every tau and horizon: a unique fixed point exists and iteration converges geometrically with modulus gamma^n.","At the tau=1 endpoint, the fixed point satisfies the separation-jn lower-bound inequality of LQL for j=1,2,..., so the trajectory-level optimality constraint is met by construction rather than by hinge penalties.","The fixed tau=0.8, n=4 configuration reaches the higher aggregate mean in four of five comparisons against LQL across 27 manipulation and navigation task instances, and achieves 1.27-1.77x higher measured training-step throughput.","The stochastic-dynamics bound produces a safe-expectile threshold tau_safe(s,a) = (sigma_n + 2 mu_n)/(2 sigma_n + 2 mu_n): larger behavioral drift permits larger tau, while with noisy upper tails a conservative ensemble penalty is needed to counter over-optimism."],"supporting_citations":[{"why":"Defines the expectile as the asymmetric least-squares minimizer and supplies Lemma 1's monotonicity and Lipschitz properties used throughout the operator analysis.","marker":"Newey and Powell 1987"},{"why":"Introduces n-step returns and temporal-difference prediction; the n-step target in Eq. (7) is the object ENQ modifies.","marker":"Sutton 1988"},{"why":"Defines the LQL baseline and its lower/upper-bound optimality inequalities, the comparison target and the inequality Corollary 1 recovers at separations n, 2n, ...; also supplies the constants M and V_infinity used in Theorem 3.","marker":"Abraham, Shi, and Finn 2026"},{"why":"Introduces expectile regression on state values for offline RL (IQL), the antecedent that Remark 2 contrasts with ENQ's action-value placement and that A3 compares empirically.","marker":"Kostrikov, Nair, and Levine 2021"},{"why":"Provides the Flow Q-Learning policy-extraction mechanism used in the main experiments.","marker":"Park, Li, and Levine 2025"},{"why":"Supplies the two manipulation domain datasets used in the RoboMimic comparisons.","marker":"Mandlekar et al. 2021"},{"why":"Supplies the 25 goal-conditioned task instances and their datasets used in the OGBench comparisons.","marker":"Park et al. 2025a"},{"why":"Provides the fixed-point theorem invoked in Theorem 1 to conclude existence and uniqueness of the fixed point under the gamma^n-contraction.","marker":"Banach 1922"},{"why":"Motivates the conservative ensemble aggregation (mean minus rho times standard deviation) in Eq. (4) and its counterweight role in ablation A4.","marker":"Ghasemipour, Gu, and Nachum 2022"},{"why":"Provides the action-chunking variant of the critic and actor, plus published baseline numbers for the chunked comparisons.","marker":"Li, Zhou, and Levine 2026"}],"fun_headline_variants":["Upper expectiles fix n-step Q-learning's pessimism","Expectile n-step Q-learning corrects multi-step bias","ENQ: Asymmetric loss for n-step off-policy Q-learning","One expectile level for unbiased multi-step Q-learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Assumption 2 requires that for every supported state-action pair, the logged data place positive probability on n-step continuations whose total discounted suboptimality gap is arbitrarily close to zero; if such near-optimal continuations are absent, Theorem 2(c)'s tau=1 endpoint correction no longer holds and the fixed point can fall strictly below Q*.","fun_headline_variants_meta":{"raw":{"variants":["Upper expectiles fix n-step Q-learning's pessimism","Expectile n-step Q-learning corrects multi-step bias","ENQ: Asymmetric loss for n-step off-policy Q-learning","One expectile level for unbiased multi-step Q-learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":3084,"prompt_tokens":1297,"completion_tokens":1787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":913,"completion_tokens_details":{"reasoning_tokens":1718}},"tokens_in":913,"tokens_out":1787,"duration_ms":14871,"temperature":1.0,"reasoning_tokens":1718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:02:15.475937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a deterministic MDP where the behavior policy, at some supported (s,a), always follows a strictly suboptimal action within the next n-1 steps, so the cumulative discounted gap has a positive lower bound on every logged continuation. Then, by the definition of the operator, $T_n^{1}$ Q*(s,a) will be strictly below Q*(s,a) on that pair, contradicting Theorem 2(c); observing exactly that inequality in a finite experiment would also flag missing near-optimal coverage in the dataset.","supporting_citations":[],"review_version":2}