{"id":"9a953515-a77d-4915-8c5d-efb364a86ad1","arxiv_id":"2607.21302","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An online RL method that learns a generative behavior prior from the replay buffer via a Q-guided CVAE and uses adaptive gradient correction to combine Q-guidance with expert-action supervision.","lead":"This paper proposes EBP, an online reinforcement learning algorithm that trains a Q-guided variational autoencoder on the agent's own replay buffer to generate 'expert' action priors, removing the need for offline expert demonstrations. The authors claim higher sample efficiency and more stable convergence than TD3, DDPG, SAC, and several prior-guided baselines across Gym, PyBullet, and DMControl continuous-control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EBP vs TD3 comparison is confounded: EBP uses actor updates every step (d=1) while TD3 is evaluated only with d=2; the paper's own PyBullet results show d=1 alone helps TD3, so the claimed advantage may be partly an update-frequency effect.","rationale":"The reader's weakest assumption about the Q-guided CVAE is reasonable and worth testing, but it is a mechanism-internal concern: even if the generated actions are not genuinely high-value, EBP could still outperform TD3 through other effects (e.g., regularization or exploration), so it does not directly threaten the headline comparison. The actor-update-frequency confound is more directly load-bearing because the central claim names TD3 as the baseline, and the paper's own PyBullet results demonstrate that d=1 alone improves TD3. Without a TD3(d=1) control in the Gym and DMControl main experiments, the reported advantage cannot be attributed to the proposed modules. The convergence analysis in Section IV-D is also mathematically problematic, but it is ancillary to the empirical claim; fixing it would not rescue an uncontrolled comparison. The appropriate verdict remains CONDITIONAL: the paper should add the missing control and temper the claim if the advantage over TD3(d=1) is not significant.","tokens_in":24022,"tokens_out":11593,"duration_ms":121651,"concrete_test":"Run TD3 with d=1 (synchronized updates) on all eight Gym tasks and all eight DMControl tasks, using the same seeds, evaluation protocol, and number of runs as EBP. Compute paired probability of improvement (RLiable) and a paired significance test (e.g., Wilcoxon signed-rank or paired bootstrap) between EBP and TD3(d=1) across tasks. If EBP's win rate over TD3(d=1) is significantly lower than over TD3(d=2), or is not significant on a majority of tasks, the paper should either add TD3(d=1) to all main figures/tables or qualify the 'significantly outperforms TD3' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline empirical claim (Section VII) is that EBP, with a fixed set of hyperparameters, significantly outperforms TD3. However, the comparison is not fixed-hyperparameter: Algorithm 1 and Table IV set EBP's actor update frequency to d=1 (every step), while TD3 is run with its default d=2 (delayed updates). Section V-C explicitly investigates this design choice and reports that TD3(d=1) outperforms TD3(d=2) on three of the four PyBullet tasks (HalfCheetahBullet, AntBullet, Walker2DBullet). This makes d a confound that acts in EBP's favor: the synchronized-update schedule alone can improve TD3. In the Gym main results (Table I, Fig. 3) and DMControl results (Fig. 5), no TD3(d=1) control is reported, so the reported advantage of EBP over TD3 in those benchmarks may be substantially due to update frequency rather than to the Q-guided CVAE, EPG, or PGC modules. The central claim therefore lacks a necessary control.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Expert Behavior Prior (EBP), an online RL algorithm built on TD3 (and a DDPG variant) that trains a Q-guided conditional variational autoencoder (Q-CVAE) on the current replay buffer, generates H candidate actions per state, selects the action with highest minimum target-Q as an 'expert' anchor, and adds a supervised imitation loss to the actor update. A Policy Gradient Correction (PGC) module reweights the supervised loss using the cosine similarity between Q-gradient and supervision-gradient. The authors compare EBP against TD3, DDPG, SAC, BAC, ALH, and NNPG on eight Gym, four PyBullet, and eight DMControl tasks, report multi-seed results, ablations, reward-noise robustness, RLiable meta-analysis, and runtime/memory overhead. The central claim is that, with fixed hyperparameters, EBP significantly outperforms TD3 and other behavior-prior methods.","tokens_in":24328,"tokens_out":5531,"duration_ms":57376,"significance":"If the empirical claims hold, EBP would be a practically useful and simple plug-in for improving TD3/DDPG in continuous control without offline data. The paper has notable strengths: a broad benchmark suite (20 tasks), ten seeds per task, ablations for the main components (α, H, µ, decay rate), an RLiable aggregate analysis, and explicit runtime/memory measurements. However, the headline comparison against TD3 is confounded by an un-matched policy update frequency: EBP uses d=1 while TD3 is run with its default d=2, and the paper's own PyBullet experiments show that TD3(d=1) alone can outperform TD3(d=2). The theoretical convergence section is also substantially overclaimed. These issues affect the central claim and require correction before the results can be accepted as stated.","major_comments":[{"comment":"The EBP-vs-TD3 comparison is confounded by policy update frequency. Algorithm 1 updates the actor every environment step (d=1), and Table IV assigns d=2 to TD3 while d=1 to EBP. Section V-C and Fig. 4 explicitly report that TD3(d=1) outperforms TD3(d=2) on three of the four PyBullet tasks (HalfCheetahBullet, AntBullet, Walker2DBullet). Thus the advantage of EBP over TD3 in the Gym (Table I, Fig. 3) and DMControl (Fig. 5) results may be substantially attributable to the synchronized update schedule rather than to the Q-CVAE, EPG, or PGC modules. Since the central claim in Section VII is that EBP 'with a fixed set of hyperparameters' outperforms TD3, this is not a fixed-hyperparameter comparison. Please add TD3(d=1) controls for Gym and DMControl, or rerun with matched update frequencies, and qualify the claim accordingly.","section":"Section IV-D, Eqs. (15)–(22)"},{"comment":"The 'convergence analysis' is not a convergence analysis. Equation (21) only bounds the norm of a single gradient step via triangle inequality and Cauchy–Schwarz; no policy improvement, descent, fixed-point, or convergence argument is provided. Moreover, Eq. (15) defines g as a function of the gradients; when differentiating µg·J_Sup in Eq. (18), the derivative of g with respect to ϕ is ignored. The derivation also drops the H×W average defined in Eq. (14), and the key assumption λ2 ≪ λ1 (Eq. 20) is asserted without evidence. Please either provide a genuine convergence or policy-improvement result, or relabel this section as a norm-bounding remark and remove the word 'convergence'.","section":"Section V-E, Fig. 12"},{"comment":"The claim that the Q-CVAE generates 'high-value' actions is validated only against the same Q-function used to train the model (Eq. 7) and to select anchors (Eq. 10). This is circular with respect to the mechanism: early in training, when Q estimates are inaccurate, the 'expert' anchor may exploit Q-estimation error rather than yield high true returns. The ablation in Fig. 12 therefore does not establish that the generated actions are actually high-value in the environment. Please provide rollout-based validation of the prior's actions, or compare against a behavior-cloned prior evaluated by true environment returns.","section":"Table I, Section VII, Fig. 13"},{"comment":"The claim of 'significantly outperforms' is not uniform across the reported results. At 200K timesteps, EBP is below TD3 on Ant (1070 vs 1308) and BipedalWalker (93 vs 118); at 2M timesteps, EBP is below NNPG on Hopper (3279 vs 3467). The RLiable aggregate (Fig. 13) is a useful summary, but it does not by itself support the universal wording in the abstract and Section VII. Please either narrow the claim to aggregated or final-performance improvement, or report per-task statistical significance with appropriate multiple-comparison control.","section":"Section V-E, Fig. 10; Section VI"},{"comment":"The paper's Limitation section (Section VI) identifies only the reward-noise phenomenon as an open question. Given the update-frequency confound and the circular validation of the Q-CVAE's 'expert' quality, the limitations should also acknowledge these issues, as they directly affect the interpretation of the empirical results.","section":"Eq. (18), Fig. 5 caption, Section V-E"}],"minor_comments":[{"comment":"The caption states 'four DMControl continuous control tasks' but eight DMControl environments are listed and eight panels are shown. Correct to 'eight'.","section":"Section V-E (Fig. 6 discussion)"},{"comment":"The notation 'EBP(w/ CV AE)' is confusing because 'w/' conventionally means 'with'. Since this variant is the one without the Q-guided loss, use 'EBP(w/o Q-CVAE)' or 'EBP(w/ CVAE-only)'.","section":"Section V-E (noise robustness)"},{"comment":"The sentence 'We attribute this phenomenon to the smoothing effect introduced in TD3's Q-value estimation process under noisy rewards...' appears twice verbatim in the noise-robustness discussion. Remove the duplicate.","section":"Eq. (18)"},{"comment":"The derivation in Eq. (18) is notational inconsistent: after substituting for µg, the expectation E_s[2(π−ã)·∇π] already equals ∇J_Sup, but the line then multiplies by ∇J_Sup again. Also, the H×W average from Eq. (14) is silently dropped. Please clean up the algebra and align the notation with Eq. (14).","section":"Section V-E, Eq. (14)"},{"comment":"The parameter m is described as 'Gradient similarity margin' in Table IV and set to 0.05·action_dim. Please clarify whether m is task-dependent and how the value was chosen; the fixed-hyperparameter claim is weakened if m scales with the action dimension while other hyperparameters are fixed.","section":"Section V-E, Eq. (14)"}],"recommendation":"major_revision","confidential_remarks":"The paper has substantial empirical breadth and an honest reporting style, but the central comparison against TD3 is confounded by the actor update frequency (d=1 vs d=2). Since the authors themselves show that TD3(d=1) improves over TD3(d=2) in PyBullet, the added value of the EBP modules over the update schedule is not yet established. This is fixable within the paper's scope by adding matched controls and tempering the claims; I would not reject outright, but the current version overclaims both empirically and theoretically."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a serious look. The core idea is real: train a CVAE on the online replay buffer with an auxiliary Q-maximization loss, sample a support set of actions, pick the highest-Q one, and use it as a supervised anchor for the actor, with an adaptive weight based on gradient cosine similarity. I don't recall that exact combination in the literature, and it's a reasonable way to get a policy prior without offline data. The empirical base is genuinely substantial: eight Gym, four PyBullet, eight DMControl tasks, ten seeds, an RLiable meta-analysis, and ablations on the main hyperparameters. That is more than most papers in this area ship.\n\nBut the headline claim is not as clean as stated. The convergence analysis in Section IV-D is mathematically flawed. Equation (18) drops the derivative of the adaptive weight g with respect to phi, and it writes the gradient of the supervised term twice. The lower bound lambda1 is not implied by the assumptions; adding a nonzero vector can reduce the norm of a sum below either component's norm. This section should be rewritten or removed.\n\nSecond, the comparison with TD3 is partly confounded. EBP updates the actor every step (d=1), while TD3 is run with its default d=2. The paper's own PyBullet experiments show TD3(d=1) helps on three of four tasks, and they do include TD3(d=1) as a control there — EBP beats it on all four. But in the Gym and DMControl suites, no TD3(d=1) control is reported, so part of the reported gap could be update frequency. That doesn't kill the contribution, but it means the 'fixed hyperparameters' and 'significantly outperforms TD3' claims are overstated.\n\nThird, the Q-guided CVAE is trained to maximize the same Q-networks that select the expert action. Early in training, when Q is inaccurate, the 'expert' may be exploiting Q-error rather than true returns. Figure 12 plots Q-values, not true returns, so it does not validate this assumption. A simple check against environment returns on a couple of tasks would help.\n\nAlso, no code is released. For a method with as many moving parts as this one — CVAE, Q-guidance, support-set selection, adaptive gradient correction — that limits reproducibility and makes it hard to separate genuine gains from implementation details.\n\nOverall: the central mechanism is plausible and worth testing, and the experiments are broad enough to deserve referee time. But the paper needs major revision before it can be accepted as a definitive contribution.","headline":"Genuinely new mechanism and a big empirical sweep, but the theory is broken, the headline comparison is partly confounded, and the code is missing.","tokens_in":24824,"tokens_out":3072,"would_cite":false,"duration_ms":29938,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Online RL can generate expert-like policy priors on the fly, without any offline demonstration data.","keywords":["expert behavior prior","behavior prior reinforcement learning","conditional variational autoencoder","Q-guided generative model","sample efficiency","continuous control","TD3","policy gradient correction"],"falsifier":"Intermittently execute the Q-CVAE's selected 'expert' actions directly in the environment (without using them for updates) and compare their true returns with the actor's actions throughout training. If the selected expert actions frequently yield lower true returns than the actor's actions, especially in the early phase, the mechanism's premise fails.","tokens_in":23878,"feed_emoji":"🤖","tokens_out":4429,"duration_ms":47180,"temperature":0.7,"pith_summary":"The paper tries to establish that reinforcement learning agents can obtain expert-quality behavior priors without any offline demonstration data. Its proposed EBP algorithm trains a conditional variational autoencoder on the agent's own replay buffer, using the Q-function as an additional loss to push generated actions toward high value. The generated actions are scored by the critics and the best one is used as a supervised anchor for the actor, with an adaptive weight that rebalances Q-guidance and expert supervision. Across 20 continuous-control tasks in Gym, PyBullet, and DMControl, with a single fixed hyperparameter set, EBP reports higher sample efficiency and more stable convergence than TD3 and several prior-guided baselines.","feed_headline":"Replay-buffer-trained generator beats TD3 in control tasks","feed_subtitle":"No offline demonstrations needed: a Q-guided autoencoder supplies expert action priors, boosting sample efficiency and stability.","key_machinery":"The Q-CVAE (Q-guided conditional variational autoencoder) is a CVAE trained on the replay buffer with a reconstruction term and a Q-maximization term, which generates diverse high-value candidate actions. The expert policy guidance (EPG) mechanism selects the highest min-Q action from a support set of H sampled actions to serve as the supervised anchor for the actor. The policy gradient correction (PGC) module computes an adaptive weight from the cosine similarity gap between Q-guidance and expert-supervision gradients, with a margin m, so the expert term corrects Q-gradients when they diverge and is softly clipped when they align.","core_discovery":"The central claim is that a generative policy prior can be produced online — from the replay buffer itself — rather than from static offline expert trajectories. The Q-CVAE learns to reconstruct observed actions while also maximizing Q-values through the double critic; the expert policy guidance step then draws H candidate actions, picks the one with the highest min-Q-value, and uses it as the target in a supervised actor loss. The policy gradient correction module weights this supervised loss by a ReLU function of the cosine similarity gap between the Q-gradient and the supervised gradient, so that supervision is strengthened when the two gradients disagree and weakened when they align. EBP","pith_inferences":["If the Q-function is overoptimistic early in training, the Q-CVAE's 'expert' anchors could be systematically biased toward actions that exploit Q error; the paper's ablation measures Q-estimates of generated actions, not their true returns, so it cannot rule this out.","The support-set mechanism that samples H candidates and picks the max-Q one is a form of optimism on a learned proposal distribution; viewing EBP this way suggests connections to other 'optimistic exploration' tricks.","A natural testable extension is to apply EBP to stochastic actor-critic algorithms or vision-based control, where the replay buffer is more diverse and the Q-function noisier.","The paper's own limitation note that moderate reward noise unexpectedly helped HalfCheetah suggests Q-uncertainty may have a dual role; a controlled study of injected stochasticity could reveal when noise aids exploration."],"forward_implications":["Online RL can obtain expert-like priors without offline data, removing the data-quality bottleneck of behavior prior RL.","A single fixed hyperparameter set transfers across Gym, PyBullet, and DMControl, suggesting the method does not rely on environment-specific tuning.","The PGC weighting mechanism provides a way to combine Q-guidance and supervised anchors, potentially reducing policy oscillation.","The approach improves both TD3 and DDPG bases, and gains are largest in the early stage (200K timesteps), indicating sample-efficiency gains.","EBP's performance drop under reward noise is smaller than TD3's on the Ant task, suggesting robustness to reward perturbation."],"fun_headline_variants":["No offline demos needed: expert priors from replay buffer","Online policy priors beat TD3 without expert trajectories","Q-guided autoencoder yields expert actions on the fly","Replay buffer alone powers expert-guided RL, no demos","From buffer to expert: sample-efficient RL sans offline data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the Q-guided CVAE, trained with the current Q-function on the online replay buffer, generates actions that are genuinely high-value in the environment; if Q-estimation is inaccurate early in training, the 'expert' anchors can be worse than the actor's own actions, and the supervision would then reinforce suboptimal behavior.","fun_headline_variants_meta":{"raw":{"variants":["No offline demos needed: expert priors from replay buffer","Online policy priors beat TD3 without expert trajectories","Q-guided autoencoder yields expert actions on the fly","Replay buffer alone powers expert-guided RL, no demos","From buffer to expert: sample-efficient RL sans offline data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1224,"prompt_tokens":754,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":498,"tokens_out":470,"duration_ms":4962,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T07:50:03.594229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Intermittently execute the Q-CVAE's selected 'expert' actions directly in the environment (without using them for updates) and compare their true returns with the actor's actions throughout training. If the selected expert actions frequently yield lower true returns than the actor's actions, especially in the early phase, the mechanism's premise fails.","supporting_citations":[],"review_version":1}