{"id":"6ef83841-5427-4f45-ba90-fba3e56141bb","arxiv_id":"2411.16532","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Task-Agnostic Policy Distillation pretrains a continual RL agent with curiosity-driven exploration and policy distillation, yielding faster learning and higher scores on five Atari games.","lead":"The paper adds a task-agnostic 'curiosity' pretraining phase to a continual reinforcement learning framework, where an agent explores Atari games without rewards and periodically copies its exploration policy into a knowledge base. The authors report that this pretraining speeds up learning and improves final scores on five Atari games compared to three continual learning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates distillation from the extra intrinsic-reward pretraining, so the central claim that distilled task-agnostic knowledge drives the gains is not yet supported.","rationale":"The reader's weakest assumption already flags the missing ablation separating intrinsic-reward exploration from distillation, and I agree that this is the most load-bearing gap. The paper's title, abstract, and conclusion all attribute downstream gains to task-agnostic policy distillation, but the experimental design never isolates that mechanism. TAPD receives substantially more environment interaction than the baselines before the progress phase, and part of that interaction occurs in two of the five evaluation games. A fair test of the distillation claim requires an ablation that matches the extra pretraining compute and environment access while removing only the distillation/compress operation. Without such a test, the reported improvements could plausibly come from additional curiosity-driven experience or from familiarity with SpaceInvaders and BeamRider, rather than from the distilled knowledge base. The internal inconsistency between Section 5.2 and Appendix B.3 about which games are in the task-agnostic Meta-Environment further complicates interpretation, though it is secondary to the missing ablation. I do not think this concern overturns the conditional verdict; the method is plausible and the code is promised, so conditional acceptance with a demand for the ablation is appropriate. The reader's verdict is already CONDITIONAL, so I recommend no change to the verdict.","tokens_in":22433,"tokens_out":2905,"duration_ms":36761,"concrete_test":"Run the following ablation with identical seeds, hyperparameters (Appendix B.3), and downstream protocol (Section 5.3): 'TAPD-no-distill', where the task-agnostic phase keeps the same intrinsic-reward training (25 samples from SpaceInvaders and BeamRider, 300k steps each) but disables the periodic compress/distillation so the knowledge base and lateral connections are never updated during the agnostic phase. If TAPD-no-distill matches TAPD's normalized scores and learning curves on all five games within seed variance, the distillation step is not load-bearing and the central claim fails as stated. Also verify the reproduced Progress & Compress baseline against the original Schwarz et al. (2018) results on at least one shared game to rule out a weak baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that task-agnostic distilled knowledge improves downstream continual RL. In the experiments (TAPD, Section 5.3), the task-agnostic phase adds two things simultaneously: (i) 7.5M timesteps of curiosity-driven exploration in SpaceInvaders and BeamRider, and (ii) periodic distillation of the exploratory policy into the knowledge base via KL loss with EWC (Section 4.2, Eq. 4; Appendix A). No experiment removes only the distillation step. Therefore, the observed gains over Progress & Compress could be due to the extra exploratory pretraining alone, especially on SpaceInvaders and BeamRider where the agent sees the same games during pretraining, rather than to the distilled knowledge. The paper's narrative in Section 5.3 attributes Pong improvements to the knowledge base, but a baseline that pretrains the active column with the same intrinsic reward for the same number of steps without compressing into the KB is missing. Without that ablation, the load-bearing component of the proposed method is unidentified. A secondary reproducibility issue: Section 5.2 says the agnostic Meta-Environment contains BeamRider and SpaceInvaders, while Appendix B.3 text says it includes Pong, SpaceInvaders, and BeamRider, making the exact pretraining distribution ambiguous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Task-Agnostic Policy Distillation (TAPD), which augments the Progress & Compress continual RL framework of Schwarz et al. with a task-agnostic phase. In this phase, an A2C agent maximizes an intrinsic curiosity reward derived from forward-model prediction error, periodically distills its exploratory policy into a knowledge base using KL divergence and online EWC, and samples games uniformly from a 'Meta-Environment' without task boundaries. The authors claim that the resulting distilled task-agnostic knowledge improves sample efficiency and positive forward transfer on five Atari games, and they report comparisons against Progress & Compress, Online EWC, and Progressive Nets. The paper includes code and extensive implementation details in the appendices.","tokens_in":22738,"tokens_out":4097,"duration_ms":35265,"significance":"If the central claim is supported, the paper makes a useful contribution to continual deep RL by showing that self-supervised task-agnostic exploration can be distilled and reused for faster downstream learning. The proposed framework directly targets learning without task boundaries, which is a recognized gap in the Progress & Compress line of work. The release of code and the inclusion of three baselines are also strengths. However, the current experimental evidence does not yet isolate the mechanism responsible for the reported gains, and several experimental inconsistencies undermine the strength of the empirical conclusions.","major_comments":[{"comment":"No ablation isolates the effect of distillation from the effect of the extra intrinsic-reward pretraining. In TAPD (Section 4.2, Equation 4), the task-agnostic phase introduces two changes relative to Progress & Compress: (i) 7.5M timesteps of curiosity-driven exploration on selected games, and (ii) periodic distillation of the exploratory policy into the knowledge base via KL plus EWC (Appendix A, Eq. A1). The reported gains in Fig. 4 and Table 2 could therefore stem from the additional pretraining alone, particularly on SpaceInvaders and BeamRider, which appear in the task-agnostic phase. A control that runs the same intrinsic-reward pretraining for the same number of steps without compressing into the knowledge base is missing. This ablation is necessary to support the paper's central claim that distilled task-agnostic knowledge drives the improvement.","section":"§4.2, §5.3, Fig. 4, Table 2"},{"comment":"The evaluation lacks error bars and significance tests. Fig. 4 reports averages over 4 seeds without variance; Table 2 reports averages over 8 seeds but no standard deviations; the two seed counts are inconsistent. On several tasks (e.g., DemonAttack and AirRaid in Table 2) the differences between TAPD and Progress & Compress are small relative to the between-visit fluctuations, so without confidence intervals or a paired test the claim that TAPD 'surpasses all three continual learning baselines' is not statistically supported.","section":"§5.3, §5.4, Fig. 4, Table 2"},{"comment":"Hyperparameters are tuned on the same downstream tasks used for evaluation. Section 5.1 states that Bayesian Hyperparameter Optimization was run 'to maximize normalized scores across tasks,' and Appendix B.3 lists multiple TAPD-specific hyperparameters (num-env-steps-agnostic, num-samples-drawn-in-task-agnostic-phase, ewc-lambda, etc.). Because the same five tasks are used for both tuning and reporting, the comparison to baselines is not a purely prospective test; the authors should either use a held-out validation task for tuning or report the sensitivity of the ranking to the tuned hyperparameters.","section":"§5.1, Appendix B.3"},{"comment":"There is a direct contradiction about which games appear in the Meta-Environment during the task-agnostic phase. Section 5.2 states 'the Meta-Environment, which includes BeamRider and SpaceInvaders,' while Appendix B.3 states 'The Meta-Environment includes Pong (P), SpaceInvaders (S), and BeamRider (B).' Table B3 lists only S and B. If Pong was in fact part of the task-agnostic phase, then the claim in Section 5.3 that TAPD achieved faster Pong learning through transfer to an unseen game is unsupported, because the agent would have had direct pretraining on Pong.","section":"§5.2, Appendix B.3"},{"comment":"The reproduced Progress & Compress baseline is not validated against the original results of Schwarz et al. (2018). Fig. 4 and Table 2 compare against a 'reproduced' baseline, but the paper provides no evidence that this reproduction matches the original method's behavior or scores on the same games. If the baseline underperforms its published implementation, the relative improvement of TAPD is overstated. The authors should report the original published Progress & Compress numbers alongside their reproduction, or explain any discrepancies.","section":"§5.3, Fig. 4"}],"minor_comments":[{"comment":"There is a typo: 'In this paper, we focuses on five different Atari games' should read 'we focus'.","section":"§5.1"},{"comment":"The symbol gamma is overloaded: it denotes both the reward discount factor and the forgetting constant in Eq. A1. Please use distinct notation for these two quantities.","section":"Appendix A"},{"comment":"The training-phase order for TAPD is listed as 'DemonAttack, Pong, BeamRider, SpaceInvaders, AirRaid,' but the experiments report the order Pong, SpaceInvaders, BeamRider, DemonAttack, AirRaid. This appears to be a transcription error and should be corrected.","section":"Table 3"},{"comment":"Progressive Nets are cited as [48] (Xu et al., 'Adaptive progressive continual learning'), but the method described is the original Progressive Networks of Rusu et al. [12]. Please align the citation with the actual method being used.","section":"References"},{"comment":"The number of seeds is reported inconsistently: Fig. 3 says 8 seeds, Fig. 4 says 4 seeds, and Table 2 says 8 seeds. Please state the exact seed count used for each result and explain any discrepancy.","section":"Fig. 3, Fig. 4, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central issue is not derivation-level circularity but missing experimental controls. The contradiction about whether Pong is in the task-agnostic pretraining distribution is especially important, because it directly affects the interpretation of the headline Pong result. The absence of an ablation of distillation and the lack of significance testing also need to be addressed before the empirical claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a plausible, clearly written empirical paper, but the main claim is not yet supported because the key ablation is missing. The task-agnostic phase adds two things at once—7.5M timesteps of curiosity-driven pretraining and periodic distillation into the knowledge base—and no experiment separates them. The gains over Progress & Compress could therefore come from the extra exploration alone, not from distilled knowledge.\n\nWhat is genuinely new: the specific integration—alternating intrinsic-reward exploration with KL distillation into a knowledge base, then reusing the knowledge base via lateral connections in the Progress & Compress framework—is not in the cited literature. The components are all prior work, but the task-agnostic phase as an explicit pretraining step is a real combination, and the paper is the first to evaluate it in continual RL on Atari. The writing is clear, the related work is accurate, and the code is released.\n\nThe soft spots are real and proportionate. The missing ablation is load-bearing; without a run that does intrinsic-reward pretraining without distillation, the paper cannot identify what makes TAPD work, and the Pong narrative is speculation. The hyperparameters were tuned by Bayesian optimization on the same downstream tasks used for evaluation, which biases the comparison in TAPD's favor. There are no error bars or significance tests; the learning curves in Fig. 4 are averaged over 4 seeds, and Table 2 over 8, but no variance is shown in the score curves. The Meta-Environment description is inconsistent—Section 5.2 says BeamRider and SpaceInvaders, Appendix B.3 says Pong, SpaceInvaders, and BeamRider. Minor but should be fixed. Also, the Progress & Compress baseline is called 'reproduced' but is not validated against the original results, which matters for a fair comparison.\n\nThe citation pattern looks fine. The self-citations are to relevant work and not inflated.\n\nWho is this for: people working in continual RL, especially on task-agnostic pretraining. It is not a field-reorganizing paper, but it is a reasonable incremental contribution. It deserves a serious referee: the question is well-posed and the result, if it holds up after the ablation, would be useful. My recommendation is to send it to peer review and require the missing ablation, variance reporting, and a corrected Meta-Environment description before acceptance.","headline":"Useful incremental idea, but the central claim needs a distillation ablation before it is supported.","tokens_in":23234,"tokens_out":2198,"would_cite":true,"duration_ms":20516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Task-Agnostic Policy Distillation (TAPD), a framework that adds a curiosity-driven, task-agnostic pre-training phase to Progress & Compress so that a continual reinforcement learning agent transfers exploratory…","keywords":["continual learning","reinforcement learning","task-agnostic learning","policy distillation","intrinsic motivation","curiosity-driven exploration","Atari 2600","self-supervised prediction"],"falsifier":"Run TAPD's task-agnostic phase on SpaceInvaders and BeamRider, then measure downstream learning curves on Pong, DemonAttack, and AirRaid against a control that receives the same number of environment steps of uniformly random exploration (or starts directly in the progress phase with the same total budget). If the control matches or beats TAPD's scores, the claim that distilled curiosity knowledge transfers to unseen games is falsified. A second check removes distillation: train the same intrinsic-reward policy for the same steps but never distill into the knowledge base; if performance is unchanged, periodic policy distillation is not the operative mechanism.","tokens_in":22221,"feed_emoji":"🕹️","tokens_out":7108,"duration_ms":61387,"temperature":0.7,"pith_summary":"The paper argues that a continual reinforcement learning agent can learn a sequence of tasks much more efficiently if it first spends a phase exploring without any task-specific goal, driven only by curiosity, and periodically distills what it discovers into a knowledge base. It introduces Task-Agnostic Policy Distillation (TAPD), which inserts this task-agnostic phase into the Progress & Compress framework: during the phase, an A2C policy maximizes an intrinsic reward equal to the log prediction error of a forward dynamics model, and the exploratory policy is repeatedly distilled into the knowledge base with online elastic weight consolidation. The agent then enters the standard progress and compress phases with a knowledge base that already contains general exploration skills, so downstream tasks are learned faster and with better sample efficiency. The authors evaluate TAPD on five Atari 2600 games—Pong, SpaceInvaders, BeamRider, DemonAttack, and AirRaid—and report that it outperforms Online EWC, Progressive Nets, and the Progress & Compress baseline across visits. A sympathetic reader would care because this addresses four obstacles to universal agents: catastrophic forgetting, positive forward transfer, scalability, and learning without task labels or clear task boundaries.","feed_headline":"Curiosity pretraining speeds up continual reinforcement learning","feed_subtitle":"Adding a task-agnostic exploration phase lets an agent learn new Atari games faster and with fewer steps.","key_machinery":"The load-bearing mechanism is the task-agnostic phase, an alternating loop of curiosity-driven exploration and knowledge-base distillation that runs before any task-specific learning. The intrinsic reward is the log forward-model prediction error in feature space, so the agent systematically seeks states it cannot yet predict. Distillation transfers the exploratory action distribution of the active column into the knowledge base using KL divergence, and online EWC prevents the knowledge base from forgetting earlier distilled exploration policies; lateral connections from the knowledge base then feed the active column in both the task-agnostic and progress phases. This loop is what converts raw curiosity into reusable, task-independent knowledge.","core_discovery":"The central discovery is that exploration itself can be treated as a distillable asset. TAPD adds an alternating task-agnostic phase to Progress & Compress. In each cycle the active column is trained for x steps with A2C to maximize the intrinsic return $\\mathbb{E}_\\pi[\\sum_n \\gamma^n r^i_{t+n}]$, where $r^i_t = \\log(L_{\\mathrm{forward}} + \\epsilon)$ and $L_{\\mathrm{forward}} = \\|\\phi(s_{t+1}) - \\hat\\phi(s_{t+1})\\|_2^2$ is the prediction error of a forward model $F(\\phi(s_t), a_t)$; the inverse dynamics model of the Intrinsic Curiosity Module is deliberately omitted. The exploratory policy is then distilled into the knowledge base by minimizing a KL divergence between the active and knowledge-base policies plus an online-EWC penalty that protects older distilled knowledge. After this phase, downstream tasks are learned through the usual progress phase (A2C with lateral connections from the knowledge base) and compress phase (distillation into the knowledge base with online EWC). The authors claim this makes the agent solve downstream tasks more efficiently and report higher scores than all three baselines on each visit to all five Atari games, with lower policy entropy indicating more directed exploration.","pith_inferences":["Because the task-agnostic phase uses only two games (SpaceInvaders and BeamRider), the paper leaves open whether the transfer is general curiosity or familiarity with shooter-like dynamics; a direct test would run the phase on a disjoint set of games and compare downstream transfer.","One testable extension suggested by the mechanism is to mix intrinsic and extrinsic rewards during the progress phase, which could further help long-horizon tasks where rewards are sparse; the paper mentions this as future work rather than testing it.","The framework predicts that the knowledge base's forward-model coverage of pretraining environments determines downstream speedup; measuring prediction-error reduction on unseen games would give a quantitative proxy for transfer.","The reported gains could partly reflect the extra environment interaction of the task-agnostic phase rather than distillation itself, so an ablation that matches total timesteps across all methods would isolate the operative contribution."],"forward_implications":["With a curiosity-pretrained knowledge base, TAPD reaches high scores on Pong by roughly 0.6 million timesteps, where Online EWC, Progressive Nets, and Progress & Compress show little improvement after 1.5 million; the paper interprets this as improved sample efficiency.","Because the task-agnostic phase samples tasks uniformly and never uses extrinsic rewards or task IDs, the framework works when task boundaries are unknown, so a single policy can move between games without being told which game it is playing.","Distilling the exploratory policy into the knowledge base and protecting it with online EWC accumulates exploration behavior across games, which the authors link to lower policy entropy and a better exploration-exploitation balance during later visits.","TAPD keeps only two networks and does not grow a new column per task, so it avoids the quadratic parameter growth of Progressive Nets and remains applicable in memory-limited settings.","Across three visits to five tasks, TAPD reports the highest average normalized performance and lower variance across tasks than Progressive Nets and Progress & Compress, indicating positive forward transfer."],"supporting_citations":[{"why":"Supplies the Progress & Compress framework that TAPD extends, including progress, compress, lateral connections, and online EWC, and serves as the main baseline.","marker":"[13]"},{"why":"Supplies the Intrinsic Curiosity Module whose forward-model prediction error defines the intrinsic reward used in the task-agnostic phase.","marker":"[7]"},{"why":"Supplies elastic weight consolidation, used as online EWC to protect the knowledge base during distillation.","marker":"[11]"},{"why":"Supplies progressive neural networks and lateral connections, the mechanism for reusing features from previously learned tasks; also a baseline.","marker":"[12]"},{"why":"Supplies the policy distillation formulation used to transfer exploratory behavior from the active column to the knowledge base.","marker":"[17]"},{"why":"Supplies the distillation principle that soft targets carry more information per sample, motivating the distillation of exploratory distributions.","marker":"[30]"},{"why":"Supplies the task-agnostic continual learning setting, where task IDs and task boundaries are absent, which the meta-environment simulation instantiates.","marker":"[14]"},{"why":"Supplies the Arcade Learning Environment and Atari 2600 games used for all experiments.","marker":"[16]"}],"fun_headline_variants":["Task-agnostic exploration distillation boosts continual RL","Distilling curiosity makes RL agents learn continually and fast","Exploration phase as reusable knowledge for continual RL","Self-supervised exploration distillation accelerates lifelong RL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the assumption that an exploration policy trained on two Atari games transfers useful exploratory knowledge to three unseen games, so that the measured performance gains come from that transferred knowledge rather than from extra training time or hyperparameter choices.","fun_headline_variants_meta":{"raw":{"variants":["Task-agnostic exploration distillation boosts continual RL","Distilling curiosity makes RL agents learn continually and fast","Exploration phase as reusable knowledge for continual RL","Self-supervised exploration distillation accelerates lifelong RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3461,"prompt_tokens":1023,"completion_tokens":2438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2379}},"tokens_in":639,"tokens_out":2438,"duration_ms":17257,"temperature":1.0,"reasoning_tokens":2379,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:00:20.404261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TAPD's task-agnostic phase on SpaceInvaders and BeamRider, then measure downstream learning curves on Pong, DemonAttack, and AirRaid against a control that receives the same number of environment steps of uniformly random exploration (or starts directly in the progress phase with the same total budget). If the control matches or beats TAPD's scores, the claim that distilled curiosity knowledge transfers to unseen games is falsified. A second check removes distillation: train the same intrinsic-reward policy for the same steps but never distill into the knowledge base; if performance is unchanged, periodic policy distillation is not the operative mechanism.","supporting_citations":[{"cited_title":"In: International Conference on Machine Learning, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the Progress & Compress framework that TAPD extends, including progress, compress, lateral connections, and online EWC, and serves as the main baseline."},{"cited_title":"In: ICLR (2016)","cited_arxiv_id":null,"evidence_quote":"Supplies the policy distillation formulation used to transfer exploratory behavior from the active column to the knowledge base."}],"review_version":1}