{"id":"c4cccea3-9049-4e96-b670-3f1244bd9d4a","arxiv_id":"2412.13662","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"State-to-Visual DAgger outperforms visual RL on hard manipulation tasks and is more stable and faster in wall-clock time, but offers little sample-efficiency benefit on easy tasks.","lead":"An empirical study compares State-to-Visual DAgger, which trains a state-based teacher then imitates it from pixels, against visual reinforcement learning across 16 control tasks. It finds the two-stage method is not universally better, but it wins on hard tasks, is more stable, and often trains faster in wall-clock time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual RL baseline representativeness is unvalidated on the hard tasks driving the headline; the DrQ-v2 comparison covers mostly easy DMControl tasks with downloaded curves and no hard ManiSkill/Adroit check.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the empirical basis for generalizing the single visual RL baseline to 'visual RL' as a whole is thin. My reading confirms that the validation against DrQ-v2 is limited to four DMControl tasks, mostly easy by the paper's own classification, uses externally downloaded learning curves rather than a shared protocol, and does not cover any of the hard ManiSkill/Adroit tasks where the central advantage is claimed. This is the most load-bearing concern because the paper's title and recommendations explicitly address 'visual RL' as a paradigm, and the hard-task advantage is the main positive result. If a stronger visual RL method performs better on those tasks, the practical guidance changes. The paper is otherwise transparent about limitations and provides code, but this particular gap is not merely a caveat; it directly supports the headline claim. The reader's CONDITIONAL verdict already reflects this, so I recommend no change to the verdict. A concrete re-run of a modern visual RL baseline on the hard tasks would settle whether the concern is real or whether AAC is indeed representative.","tokens_in":18855,"tokens_out":9068,"duration_ms":77861,"concrete_test":"Run DrQ-v2 (or a more recent visual RL algorithm such as DreamerV3 or TD-MPC2) on the paper's hard ManiSkill/Adroit tasks (PegInsertion, MoveBucket, OpenDrawer, Relocate, Pen, Hammer) and on Humanoid-Walk, using the same simulator, same observation format, 3 seeds, same total environment-step budget, and the paper's windowed asymptotic evaluation. Compare the normalized scores to AAC and to S2V Stage 2 from Table 7. If the new baseline's normalized score on these hard tasks exceeds AAC's by more than 15 points (or approaches S2V's), the paper's central claim that S2V markedly surpasses visual RL on hard tasks is not robust to the choice of visual RL baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation—prefer State-to-Visual DAgger for hard tasks—rests on the claim that the chosen visual RL baseline (Asymmetric Actor Critic with SAC) is representative of visual RL as a paradigm. The only external validation (Appendix D, Fig. 6) compares AAC against DrQ-v2 on four DMControl tasks using learning curves downloaded from DrQ-v2's official repository rather than re-run under the paper's own protocol. Those four tasks are predominantly 'easy' by the paper's own threshold; the single 'hard' task in the set (Humanoid-Walk) appears to be the one where the text says AAC did not surpass DrQ-v2 (it surpassed in three of four), though the numbering is ambiguous. No validation is provided on the ManiSkill/Adroit hard tasks (PegInsertion, MoveBucket, PickClutterYCB, OpenDrawer, Relocate, Pen, Hammer) where the headline advantage of S2V is largest (Fig. 3: hard tasks 66 vs 21 normalized). If a stronger visual RL method (DrQ-v2, DreamerV3, TD-MPC2) performs substantially better than AAC on these hard tasks—via data augmentation, better encoders, or model-based imagination—the 'marked superiority' of S2V over 'visual RL' could shrink or reverse. Additionally, the paper cites Laskin et al. 2020 (RAD) as 'DrQ-v2,' which is a different algorithm, further obscuring what exactly was validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks when a two-stage State-to-Visual DAgger (S2V) pipeline, in which a state-based SAC teacher is trained and then a visual policy is learned by online imitation, should be preferred over end-to-end visual RL. The authors compare their implementation of S2V against an Asymmetric Actor-Critic SAC baseline on 16 tasks from ManiSkill, DMControl, and Adroit, measuring asymptotic performance, sample efficiency, and wall-clock time. The main reported findings are that S2V markedly outperforms visual RL on tasks classified as hard, performs comparably or slightly worse on easy tasks, is usually more stable at convergence, does not offer a clear sample-efficiency advantage when both methods solve the task, and often saves wall-clock time. They also document implementation choices for S2V and provide practical recommendations.","tokens_in":19222,"tokens_out":6337,"duration_ms":53269,"significance":"If the comparison were fully convincing, the paper would provide useful, actionable guidance for practitioners choosing between two-stage privileged-imitation pipelines and direct visual RL: use S2V for hard tasks and wall-clock-sensitive settings, and visual RL for easy tasks or when state is unavailable. The scope is substantial (16 tasks, three benchmarks), the implementation details and hyperparameter tuning are transparent, the code is released, and the authors state their limitations explicitly. The main reason the significance is conditional is that the entire visual RL side of the comparison is represented by one baseline whose external validation is thin and partly mis-cited, and the easy/hard split that drives the headline is acknowledged to be ad hoc. With additional baseline validation on the hard tasks and robustness checks on the difficulty split, the paper would be a valuable reference.","major_comments":[{"comment":"The selection of Asymmetric Actor Critic (AAC) as the representative of visual RL is not validated on the tasks that drive the main conclusion. Appendix D compares AAC against DrQ-v2 on only four DMControl tasks using learning curves downloaded from DrQ-v2's official repository rather than curves re-run under the paper's protocol; three of these tasks are classified as easy under the paper's own 4M-step threshold, and the remaining hard task (Humanoid-Walk) appears to be the one where AAC does not surpass DrQ-v2. Moreover, the caption cites DrQ-v2 to Laskin et al. 2020, which is the RAD paper, not the DrQ-v2 paper. Since the headline advantage of S2V is concentrated in hard ManiSkill and Adroit tasks (Fig. 3: normalized score 66 vs 21), a stronger visual RL method such as DrQ-v2, DreamerV3, or TD-MPC2 could plausibly reduce or reverse the reported advantage on those tasks. Please either validate AAC against a strong visual RL baseline on the hard tasks that carry the central claim, or restrict the conclusion to the tested AAC-SAC implementation.","section":"Sec. 3, Appendix D, Fig. 6"},{"comment":"The easy/hard task split is defined by whether state-based RL converges within 4M environment steps, and the authors themselves concede that this classification is not rigorous. Because this split is then used to formulate the paper's central recommendation that S2V markedly surpasses visual RL in hard tasks, the threshold is load-bearing rather than cosmetic. The paper should define convergence operationally, justify the 4M cutoff, and show that the hard-task advantage is robust across alternative thresholds or a continuous difficulty measure. Without such robustness analysis, the headline result is partly a function of the chosen threshold, especially since the same split is used to define the comparison groups in Fig. 3 and Fig. 4.","section":"Sec. 5, Performance Comparison; Sec. 7"},{"comment":"The asymptotic performance metric averages the last 3% of total environment steps, but the paper does not verify that all methods have reached a plateau within the allotted budgets. For example, in Fig. 4 the Adroit Relocate S2V curve appears to still be increasing at the end of training, and several visual RL curves in ManiSkill tasks never leave near-zero performance, so the final-window average may conflate slow progress with asymptotic level. Please report plateau diagnostics or use a fixed-steps-after-convergence criterion; otherwise the asymptotic-performance claims in Sec. 5 are only statements about final-window performance.","section":"Sec. 4, Evaluation Metrics; Sec. 5, Performance Comparison"}],"minor_comments":[{"comment":"The reference for DrQ-v2 is misattributed to Laskin et al. 2020; DrQ-v2 is by Yarats et al. and is distinct from RAD. Please correct the citation and verify that the downloaded curves are indeed from DrQ-v2.","section":"Appendix D, Fig. 6"},{"comment":"Task names are inconsistent: OpenCabinetDrawer and PegInsertionSide in Appendix B and C appear as OpenDrawer and PegInsertion in Table 7 and Fig. 4. Please unify the naming.","section":"Sec. 4, Appendix B, Table 7"},{"comment":"The bullet claim that the first stage handles learning how to observe and the second stage handles learning how to act reverses the roles described in Sec. 3: Stage 1 learns an action policy from state observations, and Stage 2 learns the visual observation mapping.","section":"Appendix A"},{"comment":"The normalized scores combine success rates and returns divided by 1000 across benchmarks, which implicitly assumes these quantities are comparable. Consider plotting benchmarks separately or using a common standardization.","section":"Fig. 3"},{"comment":"The asymptotic window of 3% of total environment steps leads to different absolute window lengths for S2V and visual RL if their total step budgets differ; please clarify whether the window is computed per method or per task.","section":"Sec. 4, Evaluation Metrics"},{"comment":"The DMControl Humanoid-Walk visual RL 95% confidence interval includes a negative lower bound for a return that is nonnegative by construction; please report raw seed values or use a bootstrap interval.","section":"Table 7"},{"comment":"Appendix F describes real-world applicability through sim-to-real transfer, but this is prospective and not supported by experiments in the paper; consider presenting it explicitly as future outlook rather than as a validated property.","section":"Sec. 7, Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practical and timely question, and the released code plus transparent hyperparameter documentation are clear strengths. My main concern is that the central recommendation to prefer S2V for hard tasks rests on a single visual RL baseline whose external validation is thin, mis-cited, and mostly confined to easy DMControl tasks. This is fixable by additional experiments or by carefully qualifying the claims. The easy/hard threshold issue is acknowledged by the authors but still needs robustness analysis. I would support publication after a major revision that addresses these two load-bearing points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth a look. This is the first systematic comparison I know of between the two-stage State-to-Visual DAgger (state RL teacher, then online visual imitation) and a single-stage visual RL method, run across 16 tasks from ManiSkill, DMControl, and Adroit. The core finding—S2V markedly outperforms visual RL on hard tasks, matches or slightly trails on easy ones, and is more stable at convergence—shows up in the per-task learning curves and the aggregate histograms. The paper also does a real service by documenting implementation choices (off-policy DAgger with an early-stopping BC loss threshold) and by reporting wall-clock time including both stages, where the two-stage method wins consistently.\n\nThe soft spots are real but not disqualifying. The easy/hard split uses a hand-picked 4M-step threshold and the authors concede it is not rigorous. That split drives the central recommendation, so it deserves scrutiny; the per-task curves are consistent with the pattern, which helps, but the threshold is still post-hoc. More important, the visual RL baseline is a single method (Asymmetric Actor Critic with SAC), and the external validation against DrQ-v2 covers four DMControl tasks using learning curves downloaded from DrQ-v2's repository rather than re-run locally. Those four tasks are mostly easy by the paper's own definition, and the one hard one (Humanoid-Walk) appears to be where AAC did not beat DrQ-v2. There is no validation on the ManiSkill/Adroit hard tasks where the headline advantage is largest. So the broad claim \"S2V over visual RL\" would be safer as \"S2V over this asymmetric visual RL baseline.\" The citation of DrQ-v2 in the text points to Laskin et al. 2020 (RAD), which is a different algorithm—a minor error, but it muddies what exactly was validated.\n\nThree seeds is on the low side, but the confidence intervals and per-task curves give a reasonable picture, and the appendix documents tuning over 500+ experiments, which is more transparency than most empirical RL papers offer.\n\nBottom line: the paper is honest, well-scoped, and useful. It deserves a serious referee. I would send it to review and ask for either one stronger visual RL baseline on a few hard tasks or a softened title/claim that names the baseline explicitly.","headline":"A solid, honest empirical comparison of two-stage imitation vs visual RL; the hard/easy split is real but rests on a post-hoc threshold and a single, thinly validated baseline.","tokens_in":19720,"tokens_out":4066,"would_cite":true,"duration_ms":31326,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-stage state-policy-then-imitation pipeline should be preferred over visual reinforcement learning on hard control tasks, while visual RL remains competitive on easy tasks.","keywords":["visual reinforcement learning","imitation learning","DAgger","two-stage policy learning","asymmetric actor critic","state-to-visual transfer","manipulation benchmarks","wall-clock efficiency"],"falsifier":"Run a modern visual RL method (e.g., DrQ-v2 or an augmented-data SAC) on the paper's hard tasks—PegInsertion, MoveBucket, PickClutterYCB, Adroit Relocate—under identical environment-step and wall-clock budgets, and observe whether it reaches or exceeds the two-stage pipeline's final success rates; if it does on most of these tasks, the paper's central claim that State-to-Visual DAgger is preferred for hard tasks is falsified.","tokens_in":18697,"feed_emoji":"🤖","tokens_out":11210,"duration_ms":83365,"temperature":0.7,"pith_summary":"The paper asks when a two-stage 'State-to-Visual DAgger' pipeline—first train a low-dimensional state policy by reinforcement learning, then train a pixel-based policy by online imitation of that teacher—should be preferred over training a policy directly from pixels with visual reinforcement learning. Based on 16 tasks across ManiSkill, DMControl, and Adroit, it claims the two-stage method is markedly better on hard tasks, comparable or slightly worse on easy tasks, and usually more consistent and stable once converged. It also reports that the method does not reliably improve sample efficiency when both approaches can solve a task, but it often cuts wall-clock training time because the expensive image encoder is trained only in the second stage. The paper's practical payoff is a concrete decision rule: use the two-stage pipeline for challenging tasks, for wall-clock savings, or when a state policy already exists; use visual RL when no low-dimensional state is available or when minimal training intervention is wanted.","feed_headline":"Two-stage imitation wins on hard visual tasks","feed_subtitle":"Wins on hard tasks, ties on easy ones, cuts wall-clock time.","key_machinery":"The load-bearing mechanism is the two-stage State-to-Visual DAgger procedure itself. Stage 1 trains a state-policy teacher $\\pi_S$ with Soft Actor-Critic on low-dimensional observations that may include privileged simulator information; Stage 2 trains the visual policy $\\pi_V$ by DAgger—an online imitation algorithm that collects trajectories under the current visual policy, labels them with teacher actions $\\pi_S(o^S_t)$, and minimizes the mean squared error $\\|\\pi_V(o^V_t) - \\pi_S(o^S_t)\\|^2$—using an off-policy replay buffer and early stopping at an imitation-loss threshold. The comparison baseline is Asymmetric Actor Critic, which lets the critic see the state while the actor sees pixels, so both paradigms use privileged information and the difference is isolated to how the visual policy is learned. This design lets the paper attribute performance differences to the staged decomposition of perception and control rather than to access to privileged data.","core_discovery":"The central empirical claim is that State-to-Visual DAgger does not uniformly dominate visual RL, and the pattern of when it wins is governed by task difficulty as measured by state-based RL convergence. On the paper's 'hard' tasks—those where a state-based SAC teacher needs more than four million environment steps—the DAgger-trained visual policy finishes far ahead (for example, PegInsertion and Adroit Relocate move from near-zero success for visual RL to roughly 71% and 49%), while on 'easy' tasks the two methods are similar or visual RL is slightly ahead, as in Walker-Run and Swimmer-6. The paper further claims the two-stage pipeline gives tighter confidence intervals and smoother learning curves, meaning more predictable converged behavior and easier deployment checkpoint selection. It explains the hard-task gap by noting that state-based RL learns the control problem well while visual RL is held back by noisy gradients in visual exploration, and that DAgger's online data collection mitigates covariate shift in the second stage. In terms of efficiency, the paper concludes the method's apparent sample-efficiency edge on hard tasks mainly follows from its higher asymptotic performance, not from needing fewer environment steps, whereas its wall-clock advantage holds broadly because Stage 1 runs on cheap state observations.","pith_inferences":["Implication not tested here: replacing the paper's hard/easy threshold (state RL converging within four million steps) with a visual-RL-specific measure would likely preserve the ordering, because the state teacher is the stronger learner on those tasks.","Implication not tested here: the DMControl results suggest that when the underlying dynamics are easy, visual RL can outperform the state teacher itself, indicating a limit case where the two-stage pipeline should not be used.","Testable extension: the wall-clock advantage of the two-stage pipeline should grow as image resolution, camera count, or encoder size increase, since those costs are paid only in Stage 2.","Testable extension: ablating Stage 2 to offline behavioral cloning would isolate whether DAgger's online correction or the state teacher's control quality causes the hard-task gap."],"forward_implications":["On hard visual control tasks, practitioners should expect State-to-Visual DAgger to reach substantially higher final performance than visual RL, with the largest gaps on tasks visual RL essentially fails to solve.","On easy tasks, visual RL is sufficient and sometimes slightly better, so the extra stage of State-to-Visual DAgger buys little beyond faster wall-clock training.","Wall-clock time is where the two-stage method helps most broadly: even when sample efficiency is comparable, avoiding pixel-based RL in Stage 1 reduces training time.","The more consistent converged performance of State-to-Visual DAgger simplifies checkpoint selection and makes reported results less sensitive to random seeds.","If low-dimensional state observations are unavailable or cannot be simulated, visual RL is the only viable option among the two."],"supporting_citations":[{"why":"Supplies the DAgger online imitation algorithm used in Stage 2 to mitigate covariate shift.","marker":"Ross, Gordon, and Bagnell 2011"},{"why":"Supplies the Asymmetric Actor Critic method chosen as the visual RL baseline because it also uses privileged state information.","marker":"Pinto et al. 2017"},{"why":"Supplies Soft Actor-Critic, the RL backbone for both the state teacher in Stage 1 and the visual RL baseline.","marker":"Haarnoja et al. 2018"},{"why":"Supplies the ManiSkill2 benchmark and its eight manipulation tasks with dense rewards and observation interfaces.","marker":"Gu et al. 2023"},{"why":"Supplies the DMControl suite and the five locomotion and classical control tasks used in the comparison.","marker":"Tassa et al. 2018"},{"why":"Supplies the Adroit benchmark and its three dexterous hand manipulation tasks.","marker":"Rajeswaran et al. 2017"},{"why":"DrQ-v2 learning curves used in Appendix D to validate that Asymmetric Actor Critic is a representative visual RL baseline.","marker":"Laskin et al. 2020"}],"fun_headline_variants":["DAgger wins hard visual tasks, ties on easy ones","State-to-visual DAgger: only hard tasks beat visual RL","Hard tasks: DAgger surpasses visual RL, easy tasks tie","DAgger cuts wall-clock time, matches visual RL on easy","Visual RL loses to DAgger on hard tasks, ties easy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's recommendations assume that Asymmetric Actor Critic with SAC represents visual RL as a whole; the paper checks this against DrQ-v2 on only four DMControl tasks using that method's official learning curves rather than re-running it under the same protocol, so a stronger visual RL method on hard tasks could overturn the main conclusion.","fun_headline_variants_meta":{"raw":{"variants":["DAgger wins hard visual tasks, ties on easy ones","State-to-visual DAgger: only hard tasks beat visual RL","Hard tasks: DAgger surpasses visual RL, easy tasks tie","DAgger cuts wall-clock time, matches visual RL on easy","Visual RL loses to DAgger on hard tasks, ties easy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1390,"prompt_tokens":963,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":579,"tokens_out":427,"duration_ms":4246,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:54:21.397299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a modern visual RL method (e.g., DrQ-v2 or an augmented-data SAC) on the paper's hard tasks—PegInsertion, MoveBucket, PickClutterYCB, Adroit Relocate—under identical environment-step and wall-clock budgets, and observe whether it reaches or exceeds the two-stage pipeline's final success rates; if it does on most of these tasks, the paper's central claim that State-to-Visual DAgger is preferred for hard tasks is falsified.","supporting_citations":[],"review_version":1}