{"id":"b4500b02-5c56-446f-9454-fcf941c46757","arxiv_id":"2502.00288","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ARSQ factorizes the soft Q-function into normalized per-dimension advantages and generates actions auto-regressively, improving continuous control from suboptimal demonstration data.","lead":"The paper introduces ARSQ, a value-based reinforcement learning algorithm that predicts continuous actions one dimension at a time using normalized per-dimension soft advantages on a coarse-to-fine action grid. It reports faster and better learning from suboptimal demonstration data on the D4RL and RLBench benchmarks than the CQN baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix B.2 switches ARSQ to a different BC loss (Eq. 22) on the six D4RL medium/medium-expert datasets, while Sec. 5.1 says both methods use Eq. (15); the paper never states whether the CQN baseline was updated, so the 1.62x headline gain may not isolate the auto-regressive mechanism.","rationale":"Theorem 4.3 itself checks out: Eq. (16) enforces the normalization condition Eq. (12) exactly, and the Appendix-A proof then correctly yields Q(s,a) = V(s) + sum_d A_d(s,a_-d,a_d); the factorization is elementary but valid, and the algorithm is consistent with it. The load-bearing weakness is empirical attribution of the headline gain. The reader's weakest_assumption (the unreported BC-loss variant) is exactly the most load-bearing concern I find, and Appendix B.2 is the passage that flags it: it documents a change in the experimental protocol that Sec. 5.1 describes differently. Because six of the nine D4RL datasets comprising the 1.62x average use Eq. (22) for ARSQ, and the paper is silent on what the CQN baseline used, the comparison cannot be verified as isolating the auto-regressive mechanism without a matched rerun. Secondary flags support, but do not replace, this concern: the fully-offline hopper-medium score of 99.2 +/- 0.5 (Table 1) exceeds every offline-RL baseline by a large margin and also exceeds ARSQ's own online+offline score on the same task, with no mechanism offered for why offline-only value learning would extrapolate that well; and the main D4RL results report three seeds without visible error bars. If the BC-loss confound lands, the central claim reduces to 'a stronger BC loss helps,' so the CONDITIONAL verdict remains appropriate and unchanged.","tokens_in":21335,"tokens_out":17896,"duration_ms":162648,"concrete_test":"Obtain the code (or re-implement ARSQ and the MLP-encoder CQN baseline described in App. C.3) and rerun the nine D4RL tasks under matched BC objectives with the same hyperparameters and three seeds: (a) both methods use Eq. (15); (b) both use Eq. (22); (c) the paper's apparent configuration, ARSQ with Eq. (22) and CQN with Eq. (15) on medium/medium-expert. If the ARSQ-vs-CQN gap in (a) or (b) is much smaller than in (c), or if the recomputed 1.62x average drops substantially, the reported gain is driven by the BC-loss mismatch rather than the auto-regressive decomposition. Report per-dataset normalized scores and the averaged ratio for each configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 1.62x average gain over CQN on D4RL (Fig. 4), attributed to the auto-regressive advantage decomposition. That attribution is valid only if ARSQ and CQN differ in the auto-regressive mechanism and in nothing else that matters. The paper itself documents a deviation from that condition. Sec. 5.1 states: 'both ARSQ and CQN incorporate the BC objective (Eq. (15)) towards offline dataset.' Appendix B.2 then introduces a different objective, Eq. (22), a log-sum-exp margin loss, and states 'we adopt this variant objective when working with such datasets', namely 'the medium and medium-expert series of datasets in D4RL.' That is six of the nine datasets contributing to the 1.62x average. The paper never states whether the CQN baseline was run with Eq. (15) or Eq. (22) on those six datasets. The BC loss directly controls how strongly the learned policy is pulled toward dataset actions, and on D4RL a stronger BC term alone can move normalized scores by large margins; the Appendix itself says Eq. (22) 'achieves better performance' in these regimes. If CQN was left on Eq. (15), part or all of the reported gap on medium/medium-expert datasets is attributable to the BC-loss change rather than to auto-regression, and the headline mechanism claim is not established. This is an internal-inconsistency flag, not a disagreement with consensus: the main-text and appendix descriptions of the experiment cannot both be true unless the baseline was also updated and the text merely omits that fact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Auto-Regressive Soft Q-Network (ARSQ), a value-based RL method for continuous control that discretizes actions in a coarse-to-fine hierarchy and predicts normalized 'dimensional soft advantages' auto-regressively over action dimensions. A theoretical result (Theorem 4.3) shows that, under a per-dimension normalization condition, the soft advantage A(s,a) equals the sum of dimensional advantages. The method combines this representation with a behavior-cloning objective and is evaluated on D4RL (with suboptimal offline data) and RLBench (with expert demonstrations plus online data), reporting an average 1.62x improvement over the CQN baseline on D4RL.","tokens_in":21684,"tokens_out":6316,"duration_ms":59072,"significance":"If the empirical claims are substantiated, ARSQ offers a useful value-based alternative for continuous control with suboptimal data, and the factored advantage representation is a valid soft-Q parameterization. The theoretical derivation is correct but is essentially a mathematical identity that follows from the normalization the authors enforce in Eq. (16); the paper's contribution therefore rests primarily on the experimental comparison. The paper is clearly written, provides code-level implementation details, and includes ablations that isolate the coarse-to-fine and auto-regressive components. The main risk is that the headline comparison with CQN currently does not cleanly isolate the auto-regressive mechanism because of an unstated change in the behavior-cloning loss for part of the D4RL evaluation.","major_comments":[{"comment":"Section 5.1 states that 'both ARSQ and CQN incorporate the BC objective (Eq. (15)) towards offline dataset.' Appendix B.2 introduces a different objective, Eq. (22), and says 'we adopt this variant objective when working with such datasets,' referring to the medium and medium-expert series of D4RL datasets. These are six of the nine datasets used in the main D4RL comparison. The paper never states whether the CQN baseline was also switched to Eq. (22) on those datasets. Because the BC loss directly controls how strongly the learned policy is pulled toward dataset actions, and because the appendix reports that Eq. (22) 'achieves better performance' in these regimes, the reported average 1.62x improvement over CQN may be partly attributable to the BC-loss change rather than to the auto-regressive advantage decomposition. Please clarify, for each D4RL dataset, which objective each method used, and if CQN was not updated, rerun the baseline with the same variant and re-report the results.","section":"Sec. 5.1 and Appendix B.2"},{"comment":"The abstract's headline claim of an 'average 1.62x performance improvement' over the SOTA value-based baseline is not backed by a per-dataset numeric table. Figure 4 shows learning curves without numerical converged values, and the additional curves in Appendix D (Fig. 15) also lack a table. Please provide a table reporting the mean and standard deviation over seeds for each of the nine D4RL datasets for ARSQ and CQN, together with the per-dataset ratio, so the 1.62x figure can be verified and the contribution of each dataset assessed.","section":"Abstract and Fig. 4; Appendix D"},{"comment":"The main empirical claims rest on comparisons that are reported only as learning curves averaged over three random seeds, with no error bars or shaded regions in Figs. 4-8. Given that the central message is a quantitative advantage over CQN, the absence of variance information makes it impossible to assess whether the reported gaps are statistically meaningful. Please add error bars or confidence bands, and state the number of evaluation episodes used for each point.","section":"Sec. 5.1 and Figs. 4-8"}],"minor_comments":[{"comment":"The proof of Theorem 4.3 uses both Z(s,a_-d) and Z_d(s,a_-d) with slightly inconsistent notation; please unify the notation to avoid confusion.","section":"Appendix A, Eq. (19)"},{"comment":"Algorithm 2 takes the minimum over two advantage networks, A_d(a_d) = min_i A^{θ_i}_d, and then renormalizes. Because each individual network satisfies the normalization in Eq. (16), the min of two normalized exponentials does not in general satisfy it, and the renormalization is a heuristic not covered by Theorem 4.3. Please add a sentence clarifying that the double-Q min is an algorithmic choice outside the theoretical identity.","section":"Algorithm 2"},{"comment":"There is a grammatical error in the first sentence: 'we incorporate an behavior cloning objective' should be 'we incorporate a behavior cloning objective.'","section":"Appendix B.2"},{"comment":"The 'Plain' ablation removes both the coarse-to-fine structure and dimensional conditioning, but it is not clear whether this variant corresponds exactly to an independent per-dimension Q-network with a single discretization level; please clarify the architecture of this baseline.","section":"Sec. 5.4 and Fig. 7"},{"comment":"Table 1 states that most baseline numbers are sourced from their respective papers while only DWBC is re-evaluated (marked with '*'); please specify the evaluation protocol (number of trajectories, number of seeds) used for the cited baselines so that the comparison with ARSQ, evaluated with 10 trajectories over 3 seeds, is meaningful.","section":"Sec. 5.3, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The load-bearing issue is the inconsistency between the main text (both methods use Eq. (15)) and Appendix B.2 (a different BC variant is adopted on six of nine D4RL datasets). This is fixable but requires either a clear statement that CQN was also updated to Eq. (22) or a rerun of the baseline. The theoretical result is correct yet modest, so the paper's acceptance depends on the empirical comparison; the lack of a per-dataset numeric table and the absence of error bars further weaken the current support. I would encourage the editor to request the clarifications and the numeric table before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: ARSQ is a genuinely clean engineering contribution—normalized per-dimension soft advantages with auto-regressive generation on top of CQN's coarse-to-fine discretization—and the Theorem 4.3 identity is correct. But the paper's own appendix introduces a BC-loss variant for six of the nine D4RL datasets without saying whether the CQN baseline got the same update, so the headline 1.62x gain over CQN cannot currently be attributed to the auto-regressive mechanism.\n\nWhat's new and good: The per-dimension normalization constraint (Eq. 16) turns a soft-Q policy into a product of conditional distributions with a clean identity; the proof is elementary but correct. The ablations (Swap, w/o CF, w/o Dim Cond, etc.) are informative and mostly match the design story. The RLBench 20-task evaluation in the appendix is broader than the main text suggests, and the method does beat CQN on most curves. The fully offline table is a useful extra, even if it mixes numbers from other papers.\n\nSoft spots, in order of seriousness. First, the BC-loss ambiguity is real and load-bearing. Section 5.1 says both ARSQ and CQN use Eq. (15); Appendix B.2 says that for medium and medium-expert datasets they adopt the log-sum-exp margin loss of Eq. (22) because it 'achieves better performance' in concentrated modes. That's six of the nine datasets feeding the 1.62x average, and the paper never states whether CQN was run with Eq. (15) or Eq. (22). A stronger BC term alone can move D4RL scores by large margins, so the mechanism claim is not established as written. This is fixable by reporting the protocol, but it has to be fixed.\n\nSecond, the statistics are thin: three seeds, no error bars in Figures 4 and 6, and the 1.62x number has no per-dataset breakdown. Third, the fully-offline hopper-medium score of 99.2 versus ~60 for DT and filtered BC looks like an outlier; the paper doesn't discuss it. Fourth, novelty is modest—Q-transformer and Seo & Abbeel already do autoregressive Q/action prediction, and the coarse-to-fine part is CQN's—but the normalization identity plus the D4RL/RLBench results is enough for a solid subfield paper. No code is released, which makes the empirical side harder to trust.\n\nVerdict: send it to review. A serious referee should focus on the BC-loss protocol and the statistical completeness, not on the theory, which holds. If the authors can confirm CQN received the same BC treatment, the paper is likely a genuine advance over CQN; if not, the headline claim shrinks to 'we also changed the BC loss,' which is a much smaller result.","headline":"Clean theory and a plausible algorithm, but the paper's own appendix muddies the headline 1.62x claim by switching BC losses on six datasets without saying whether the baseline was switched too.","tokens_in":22236,"tokens_out":4114,"would_cite":true,"duration_ms":35129,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Auto-regressive soft advantages let value-based RL learn better from suboptimal demonstrations.","keywords":["auto-regressive soft Q-learning","continuous control","value-based reinforcement learning","suboptimal demonstrations","dimensional soft advantage","coarse-to-fine discretization","D4RL","RLBench"],"falsifier":"Run the D4RL medium and medium-expert comparisons with CQN given the same behavior-cloning loss variant (Eq. 22) that ARSQ uses; if CQN then closes the gap, the reported gain is not caused by the auto-regressive advantage.","tokens_in":21119,"feed_emoji":"🤖","tokens_out":7207,"duration_ms":63831,"temperature":0.7,"pith_summary":"The paper claims that value-based reinforcement learning for continuous control fails on suboptimal data because it scores each action dimension independently, ignoring correlations, so that frequent suboptimal behaviors dominate the Q estimate. It proposes Auto-Regressive Soft Q-learning (ARSQ), which discretizes the action space coarse-to-fine and predicts a normalized dimensional soft advantage for each action dimension conditioned on the previously chosen ones. The key identity is that when each dimensional advantage is normalized to a probability distribution, the ordinary soft advantage equals the sum of dimensional advantages, so the Q-function remains a valid soft Q while the policy is generated one dimension at a time. On D4RL, ARSQ reports an average 1.62x performance improvement over the state-of-the-art value-based baseline; on RLBench it outperforms actor-critic and imitation baselines. If right, value-based RL can extract correct behavior from datasets where optimal and suboptimal demonstrations are mixed.","feed_headline":"1.62x gain on suboptimal control data via auto-regressive soft Q","feed_subtitle":"Predicting per-dimension advantages in sequence beats independent Q-functions on mixed-quality demonstrations.","key_machinery":"The central object is the dimensional soft advantage $A_d(s,a_{-d},a_d)$: the per-dimension advantage of choosing bin $a_d$ after earlier dimensions are fixed, together with the normalization constraint that turns $\\exp(A_d/\\alpha)$ into a conditional distribution. The log-sum-exp subtraction in Eq. (16) enforces that constraint in the network, so the sum of dimensional advantages equals the full soft advantage (Theorem 4.3). Around this core, ARSQ wraps a coarse-to-fine discretization of the continuous action space and a margin-based behavior-cloning loss (Eq. 15), with a log-softmax variant (Eq. 22) used on some datasets, to keep expert actions preferred during online fine-tuning.","core_discovery":"The central claim is that the soft Q-function of a multi-dimensional continuous action can be written auto-regressively: with dimensional soft advantages $A_d(s,a_{-d},a_d)$ satisfying $\\sum_{a_d} \\exp(A_d/\\alpha)=1$, the identity $\\sum_d A_d(s,a_{-d},a_d)=A(s,a)$ holds, so the sum of the dimensional advantages is the full soft advantage and the product of the conditional policies equals the global soft policy. ARSQ realizes this with a shared network with separate heads, hard-normalizing each head by log-sum-exp subtraction, and by conditioning both on previously sampled action dimensions and on coarser discretization levels. The authors argue that this removes the bias toward frequent suboptimal modes that independent per-dimension Q estimation inherits, and support it with a one-step toy example, a fully offline evaluation, and online-with-demonstration benchmarks.","pith_inferences":["One implication the paper leaves open is that the normalized-conditional-advantage construction is a recipe, not just an algorithm: any value network whose heads are normalized this way yields a factorized stochastic policy, so the identity could be grafted onto other offline RL methods.","An untested prediction of the paper's logic is that the gain over per-dimension baselines grows with the strength of cross-action correlation; on near-independent action dimensions the two approaches should converge.","Grouping action dimensions into blocks before the auto-regressive pass is a natural latency fix; the identity holds at group level (normalize over each block's bins) and would directly address the higher inference time the authors report.","The paper's error analysis suggests a cheap deployment diagnostic: compare the auto-regressive Q value with a plain value head on the same state-action pairs; large deviations would indicate the factorization is not faithfully representing the soft Q-function."],"forward_implications":["If the identity holds, any soft Q-function for continuous control can be represented by per-dimension conditional advantages, so value-based methods can handle correlations between action dimensions without enumerating all joint action bins.","Learning from datasets with many suboptimal demonstrations should improve automatically, because the policy samples from normalized conditional distributions instead of a blurred per-dimension Q average.","The coarse-to-fine hierarchy keeps the number of output heads small, so fine control precision does not require an exponential growth in discretized actions.","The method also works fully offline, matching or exceeding dedicated offline RL and offline imitation baselines on the D4RL aggregate score.","When demonstration quality degrades, the paper reports the advantage over the value-based baseline grows, reaching about 2x on the bottom 30% of trajectories."],"supporting_citations":[],"fun_headline_variants":["1.62x gain with auto-regressive soft Q on mixed demonstrations","Auto-regressive soft Q beats independent Q on suboptimal control","Learning control from suboptimal data via auto-regressive Q","Auto-regressive soft Q: 1.62x gain from suboptimal demonstrations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes ARSQ and the baseline faced the identical behavior-cloning loss; the appendix reveals a different loss variant for some datasets, so part of the gain could reflect that change rather than the auto-regressive advantage.","fun_headline_variants_meta":{"raw":{"variants":["1.62x gain with auto-regressive soft Q on mixed demonstrations","Auto-regressive soft Q beats independent Q on suboptimal control","Learning control from suboptimal data via auto-regressive Q","Auto-regressive soft Q: 1.62x gain from suboptimal demonstrations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000387,"raw_usage":{"total_tokens":2073,"prompt_tokens":1006,"completion_tokens":1067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1003}},"tokens_in":622,"tokens_out":1067,"duration_ms":7777,"temperature":1.0,"reasoning_tokens":1003,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:33:19.677526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the D4RL medium and medium-expert comparisons with CQN given the same behavior-cloning loss variant (Eq. 22) that ARSQ uses; if CQN then closes the gap, the reported gain is not caused by the auto-regressive advantage.","supporting_citations":[],"review_version":1}