{"id":"40fa7bcb-a454-40a5-b46d-2439cf77b2d1","arxiv_id":"2506.14420","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SD3 separates skills by maximizing each skill's state density deviation from other skills and adds a VAE-based latent-space exploration reward, achieving modest benchmark gains.","lead":"This paper introduces SD3, a method that trains reinforcement learning agents to discover skills by pushing each skill's explored states away from the states explored by other skills, using an autoencoder to estimate state densities. It reports improved downstream task performance on standard benchmarks, although its gains over the strongest existing methods are often within statistical noise.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 2's proof is invalid: Eq. (18) conflates the per-input KL reward with an entropy difference, so the count-based exploration claim is an unsupported upper bound, not the stated equivalence.","rationale":"The reader's conditional verdict is appropriate, and the strongest parts of the paper are the extensive URLB evaluation and the clean Theorem 1 sandwich bound, which checks out. The weaker load-bearing point is the theoretical justification of the exploration bonus. The proof of Theorem 2 claims that the implemented KL reward approximately equals count-based exploration, but the derivation's first substantive equality, Eq. (18), is not a valid identity for the per-sample KL divergence used in Eq. (6). The subsequent inequalities therefore bound a different information-theoretic quantity, and an upper bound cannot yield the claimed two-sided approximation in Eq. (9). This is not an attack on the empirical method; the experimental results may still be valid. However, the paper's advertised 'provably efficient count-based' exploration is unsupported as written. Because this is a theorem-level flaw rather than a missing experiment, it strengthens the reader's condition: the authors should either repair the proof (e.g., by analyzing the UCB term they actually derive, or by proving a tight two-sided bound for the implemented reward) or remove and qualify the count-based claim. The density-estimation concern raised by the reader remains a secondary empirical gap; the analytic error in Appendix C is the sharper reason to keep the verdict conditional.","tokens_in":22529,"tokens_out":16029,"duration_ms":168429,"concrete_test":"Re-derive Appendix C with a concrete Gaussian instance: set r(h)=N(0,I) and Q_phi(h|s,z)=N(mu_{s,z}, Sigma_{s,z}) with non-degenerate Sigma (e.g., Sigma=2I, mu != 0). Evaluate Eq. (18): the left side is KL(N(mu,Sigma) || N(0,I)); the right side is H(Q_margin) - H(N(mu,Sigma)). Since Q_margin is an aggregate distribution over the dataset while the left side is a single-input KL, the equality fails for generic mu and Sigma. If this single algebraic check shows inequality, the proof chain from the implemented KL reward to count-based exploration is broken, and Theorem 2's conclusion does not follow.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is the proof of Theorem 2, which is supposed to justify the intra-skill exploration reward as count-based exploration. In Appendix C, Eq. (18) asserts E_Phi DKL(Q_phi(h|s,z) || r(h)) = H(Q_margin) - H(Q_phi(h|s,z)). For a fixed state-skill input, the left side is a per-input KL divergence to the prior r(h), while the right side is an entropy difference involving the aggregate encoding distribution Q_margin. These are different objects, and the equality is not generally true. Replacing the expectation over parameter posterior Phi with a point estimate phi does not repair the mismatch, because the implemented reward in Eq. (6) uses the fixed learned network. The subsequent inequalities (Eqs. 19-21) therefore bound an information-gain quantity that is not the actual KL reward, and the UCB/count expression at best upper-bounds that different quantity. Consequently, Theorem 2's claimed approximation r_exp_z(s) approx |S| / (2(N(s,z)+kappa)) is not derived even in tabular MDPs; the proof yields only an upper bound on a different quantity. Since the abstract and contributions advertise a count-based, provably efficient exploration bonus, this invalidates a central theoretical support. The empirical results are not refuted, but the paper's proposed explanation for why the exploration bonus works is unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SD3, an unsupervised skill-discovery method that rewards each skill for visiting states whose estimated state density deviates from the densities of other skills, and adds a latent-space KL-divergence reward for intra-skill exploration. A skill-conditional CVAE with soft modularization estimates the state densities and provides the latent representation. The paper claims two theoretical results: Theorem 1 bounds the SD3 objective by the mutual information I(S;Z) plus log λ, and Theorem 2 claims the intrinsic reward is approximately equivalent to count-based exploration in tabular MDPs. Experiments cover maze visualization, state-based URLB, pixel-based URLB, a robustness study, and ablations; SD3 achieves the best or near-best aggregate scores on the benchmarks.","tokens_in":22794,"tokens_out":7051,"duration_ms":77286,"significance":"If the claims hold, SD3 would be a competitive and scalable skill-discovery algorithm that unifies inter-skill diversity and exploration. The empirical evaluation is extensive: 12 state-based and 8 pixel-based downstream tasks, multiple seeds, aggregate statistics (IQM/mean/OG), ablations over λ, α, architecture, and skill selection, and a noise-robustness comparison. The paper does not release code or externally verified proofs, and Theorem 2 is not established as stated. With a corrected or appropriately weakened theoretical claim, the empirical contribution is substantial, but the current version overstates the theory and should not be accepted in its present form.","major_comments":[{"comment":"The proof of Theorem 2 does not establish the stated equivalence. Eq. (18) equates the fixed-input KL reward DKL[Qϕ(h|s,z)||r(h)] to an entropy difference H(Qmargin)-H(Qϕ(h|s,z)); these are different objects, and the equality is not generally valid because Qmargin is an aggregate encoding distribution while r(h) is a fixed prior. Replacing the expectation over the posterior Φ by a point estimate φ does not fix the mismatch, because Eq. (6) uses the fixed learned network. Consequently, Eqs. (19)-(21) bound an information-gain quantity that is not the implemented reward, and Eq. (36) gives only an upper bound on that different quantity. The final approximation '≈' in Eq. (37) and the phrase 'equivalent to count-based exploration' are therefore unsupported; no lower bound or tightness proof is supplied, and the tightness remarks in the appendix are qualitative. This is load-bearing because the abstract, the contributions list, and Section V-D use this theorem to justify the exploration bonus as provably efficient and UCB-like.","section":"Section III-C, Theorem 2 and Appendix C"},{"comment":"The entire inter-skill reward rsd3_z(s) in Eq. (7) depends on treating the CVAE ELBO as an estimate of log dπ_z(s). The paper never validates this estimate directly. The soft-modularization ablation in Section V-E.1 shows downstream-task gains but does not measure whether the estimated densities are accurate; the issue is particularly acute for image observations, where no density-estimation diagnostics are provided. Because the density model is trained on the policy's own samples, the 'deviation' signal is partly self-referential, and an inaccurate model could produce the reported behavior through a different mechanism. Please provide a validation of the density estimates (for example, on a tabular or low-dimensional control task where true occupancy can be approximated) or explicitly reframe the reward as 'deviation of the learned density model' rather than 'deviation of true state density.'","section":"Section III-A, Eq. (3) and Eq. (7)"},{"comment":"The proof also suffers from notation and index problems that prevent verification: m is used both as the number of data points and as the index of the last block in the covariance matrix, the block-diagonal form in Eq. (30) assumes an ordering of state-skill pairs that is not stated, and the final c/2 in Eq. (36) is silently identified with |S|/2 even though c was introduced as the output dimension of the linear decoder. These issues compound the gap in the first major comment, and they should be corrected if the theorem is retained in any form.","section":"Appendix C, Eqs. (30)-(36)"}],"minor_comments":[{"comment":"The text says 'The detailed scores are attached in Table III' for the pixel-based URLB results, but Table III is the robustness experiment; the pixel scores are in Table II. The reference to 'Fig. 6(a)' in the same section also appears to be a typo for Fig. 5.","section":"Section V-C"},{"comment":"Figure 8 reports the λ ablation only on Quadruped; please state whether λ-insensitivity was checked on other domains, since the conclusion is phrased generally.","section":"Section V-E.2"},{"comment":"The statement that the maximum of ISD3 is H(Z) is asserted without the measure-theoretic caveat that densities may vanish or that the maximum is approached only in a limiting sense; a short proof or reference would be helpful.","section":"Section III-A"},{"comment":"The state-based and pixel-based experiments use different backbones (DDPG vs. Dreamer), and the CVAE architecture and hyperparameters (number of modules, latent dimension, routing network size, reward scaling α per domain) are not fully specified; this is important for reproducibility, especially since no code release is mentioned.","section":"Section V-B and V-C"},{"comment":"The robustness claim is phrased as a general advantage of UCB-style bonuses over entropy-based exploration, but the experiment compares only SD3 against CIC; the conclusion should be restricted to this comparison unless additional baselines are added.","section":"Section V-D"}],"recommendation":"major_revision","confidential_remarks":"The paper has strong empirical results, but the central theoretical claim of Theorem 2 is not supported by the proof as written. I would recommend the editor require either a correct proof of the claimed count-based equivalence or a clear downgrade of that claim to a heuristic motivation. I would also encourage asking the authors for code release and for a direct validation of the density estimates, since the method's name and framing rest on those estimates. The overlap of the λ=1 objective with standard mutual information should be addressed explicitly in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the SD3 paper. Bottom line: it is a competent empirical skill-discovery paper with a plausible new mechanism, but the headline theoretical claim about count-based exploration is not proved; the proof in Appendix C is wrong at the first substantive step.\n\nWhat is new: the SD3 objective maximizes the log-ratio of a skill's state density to the density mixture of all skills, with λ softening the separation; at λ=1 this reduces to the standard I(S;Z), and Theorem 1 is a straightforward bound that checks out. The soft-modularized conditional VAE for density estimation and the KL-to-prior latent reward are genuine design choices not present in earlier work, though the density-separation idea itself is prefigured by ReST. The experimental campaign is above the norm: state and pixel URLB, maze visualizations, robustness to noise, and ablations over λ, α, and the modularization. The results are credible: SD3 is competitive with CIC and BeCL on IQM/mean, and it wins most pixel-based tasks, even if many per-task differences are within error bars.\n\nThe soft spots. First, Theorem 2 as stated is not derived. Eq. (18) sets the per-input KL reward equal to an entropy difference H(Q_margin) - H(Q_φ(h|s,z)); those are different quantities, and replacing the parameter posterior Φ with a point estimate does not fix it. The subsequent information-gain bound applies to a reward the algorithm does not actually compute. The appendix ends with an upper bound, not the claimed approximation r_exp ≈ |S|/(2(N+κ)), so the statement that the two are \"equivalent\" is unsupported. That matters because the abstract and contributions advertise the reward as provably count-based exploration; the empirical results stand on their own, but the theoretical justification should be corrected to a conjecture or a properly stated inequality. Second, the CVAE density estimates are never validated, and since the same samples train both the density model and the policy, the novelty signal is self-referential—common in this literature, but worth stating explicitly. Third, code and data are not provided, which makes the fine-grained numbers hard to check. These are real weaknesses, but the empirical core is not fraudulent; it is a plausible method that gains a handful of points over strong baselines.\n\nWho this is for: someone working on unsupervised skill discovery will find the method and benchmark numbers useful. The paper deserves a serious referee; I would not desk-reject it, but I would ask for a corrected Theorem 2, a validation plot of the density estimates, and ideally code. If the authors can fix or drop the count-based claim and release code, this becomes a solid contribution.","headline":"Competent empirical skill-discovery paper with a real proof error in Theorem 2; the mechanism and results are plausible but the count-based exploration claim is unsupported as written.","tokens_in":23351,"tokens_out":5848,"would_cite":true,"duration_ms":57985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unsupervised RL skills can be learned by pushing each skill into states that other skills avoid, with a latent-space bonus driving exploration; the paper reports state-of-the-art adaptation performance on the state- and pixel-based URLB…","keywords":["unsupervised reinforcement learning","skill discovery","state density deviation","conditional variational autoencoder","soft modularization","count-based exploration","intrinsic reward","URLB benchmark"],"falsifier":"Run SD3 in a small tabular MDP where true visit counts $N(s,z)$ and true state densities are computable exactly, and check two predictions: the exploration reward should decrease monotonically with $N(s,z)$ and track $|S|/(2(N(s,z)+\\kappa))$, and the density-deviation reward should be higher for states that other skills genuinely under-visit. If either correlation fails, the link between the practical network and the theorems is broken. In an image-based environment, the counterpart test is to measure the overlap between skill state distributions with an independently trained, reward-free encoder and ask whether skills that score high on $I_{SD3}$ actually visit disjoint state regions.","tokens_in":22310,"feed_emoji":"🎯","tokens_out":12337,"duration_ms":108398,"temperature":0.7,"pith_summary":"This paper proposes SD3, an unsupervised reinforcement learning method whose central idea is that skills should be learned by maximizing the deviation of each skill's state density from the state densities of all other skills: a skill is rewarded for occupying states that other skills rarely visit. Because entropy-based exploration and mutual-information skill objectives both scale poorly to large and image-based state spaces, SD3 estimates the densities with a conditional variational autoencoder that uses soft modularization to keep estimates stable when skills occupy very different regions, and it derives a second intrinsic reward from the autoencoder's latent space to push each skill to explore its own region. The paper proves that the density-deviation objective $I_{SD3}$ is sandwiched between the mutual information $I(S;Z)$ and $I(S;Z)+\\log\\lambda$ (Theorem 1), so the classic MI objective is a boundary case, and that the latent-space exploration reward approximates count-based exploration in tabular MDPs (Theorem 2). If the paper is right, the same two-term reward should learn distinguishable, well-exploring skills that adapt to downstream tasks faster than prior MI-based, entropy-based, and coverage-based methods, and it reports top aggregate scores on the state-based and pixel-based URLB benchmarks along with a robustness advantage under observation noise.","feed_headline":"Tops URLB benchmarks by pushing skills into separate state regions","feed_subtitle":"A latent-space bonus keeps each skill exploring while density deviation keeps skills distinct.","key_machinery":"The object that carries the argument is the pair of rewards derived from one conditional variational autoencoder. The autoencoder's ELBO, $L^{\\mathrm{elbo}}_z(s) = \\mathbb{E}_{Q(h|s,z)}[\\log P(s|h,z)] - D_{\\mathrm{KL}}[Q(h|s,z)\\|P(h|z)]$, is used as the estimate of each skill's log state density $\\log d^\\pi_z(s)$; soft modularization makes the encoder and decoder a skill-routed weighted combination of shared modules so that density estimates for very different skills do not interfere. The same network yields the intra-skill exploration reward $r^{\\mathrm{exp}}_z(s) = D_{\\mathrm{KL}}[Q_\\phi(h|s,z)\\|r(h)]$, which is small for states a skill has visited often and large for novel ones. The two theorems are the theoretical connectors: Theorem 1 proves the density-deviation objective sits strictly above the mutual information $I(S;Z)$ for $\\lambda\\geq1$ (recovering it exactly at $\\lambda=1$), and Theorem 2 proves the exploration reward is an upper bound on a UCB-style information gain that, under a linear decoder and tabular state-skill pairs, becomes $|S|/(2(N(s,z)+\\kappa))$ — the count-based exploration bonus.","core_discovery":"The paper claims that maximizing the density-deviation objective $I_{SD3} = \\mathbb{E}_{z\\sim p(z), s\\sim d^\\pi_z(s)}[\\log \\frac{\\lambda d^\\pi_z(s)}{\\lambda d^\\pi_z(s)p(z) + \\sum_{z'\\neq z} d^\\pi_{z'}(s)p(z')}]$ — with $\\lambda \\geq 1$ a weight that weakens the gradient pushing other skills' densities down — learns exactly the kind of skills that empowerment-based methods aim for but often miss: skills whose state-occupancy regions are clearly separated from one another while still covering enough of the state space to be useful. Theorem 1 states $I(S;Z) \\leq I_{SD3} \\leq I(S;Z) + \\log\\lambda$, with equality to $I(S;Z)$ at $\\lambda=1$, so the new objective is an upper bound on the classic mutual-information skill objective rather than a competing heuristic. Theorem 2 states that the exploration reward $r^{\\mathrm{exp}}_z(s) = D_{\\mathrm{KL}}[Q_\\phi(h|s,z) \\| r(h)]$ is approximately $|S|/(2(N(s,z)+\\kappa))$ in tabular MDPs, identifying the KL-to-prior term with well-studied count-based exploration. The paper's supporting evidence is empirical: SD3 achieves the highest IQM (77.37%), mean (76.19%), and optimality-gap (23.91%) scores among thirteen methods on the state-based URLB, improves on the strongest baselines on pixel-based Walker and Quadruped tasks using a Dreamer backbone, and retains more performance than CIC when observation noise is added during pre-training.","pith_inferences":["The $\\log\\lambda$ gap in Theorem 1 is not quantified in practice; an extension would be to characterize how $\\lambda$ trades off inter-skill separation against per-skill state coverage, so practitioners could set it to a target coverage level rather than tune it per environment.","Because the same network supplies both the density estimates and the exploration bonus, the two rewards are coupled: a density error inflates the exploration bonus and vice versa; freezing or periodically refreshing the autoencoder would be a direct test of whether decoupling the two stabilizes skill learning.","The paper's own conclusion notes that soft modularization limits SD3 to discrete skill spaces and that Theorem 2's analysis assumes tabular MDPs; extending density deviation to continuous skill spaces would require Monte Carlo estimates of the density sum in Eq. (1), whose variance is not analyzed.","The robustness result suggests that swapping other methods' entropy-based exploration terms for this KL-based count-like bonus, for instance in APS or BeCL, might improve their noise tolerance; the paper does not test that transfer directly."],"forward_implications":["Maximizing the SD3 objective with $\\lambda \\geq 1$ also increases the mutual information $I(S;Z)$, so the new objective strictly generalizes the MI-based skill discovery family that methods like DIAYN and DADS instantiate.","The latent-space exploration reward inherits count-based exploration's guarantees in tabular MDPs, giving practitioners a principled intra-skill exploration term that should prevent skill collapse as the diversity reward is maximized.","Because density estimation happens in the latent space of the autoencoder rather than over raw observations, the method is claimed to scale to pixel-based environments, and the reported pixel-based URLB results support that claim.","The robustness experiments show that the count-like bonus retains 86.33% of clean-task performance under observation noise, versus 71.68% for the entropy-based CIC, indicating that density-based and count-based exploration rewards are more stable pre-training objectives than entropy estimates."],"supporting_citations":[{"why":"URLB supplies the evaluation protocol: 2M pre-training steps, 100K fine-tuning steps, and the 12 state-based downstream tasks across Walker, Quadruped, and Jaco.","marker":"[19]"},{"why":"DIAYN is the canonical MI-based skill discovery method whose objective I(S;Z) Theorem 1 shows to be the lambda=1 special case of ISD3, and a baseline in the maze and URLB experiments.","marker":"[23]"},{"why":"BeCL is a contrastive competence-based baseline that SD3 is compared against on state-based URLB, where the two methods score closely (75.38% vs 77.37% IQM).","marker":"[24]"},{"why":"Metra is the metric-aware abstraction baseline that defines the standard for scalable, image-based skill discovery, and SD3 is compared against it on both state- and pixel-based URLB.","marker":"[25]"},{"why":"CIC is the contrastive intrinsic control baseline with the strongest entropy-style exploration; it is SD3's main competitor on state- and pixel-based tasks and the comparison target in the robustness experiments.","marker":"[28]"},{"why":"Pixel-based URLB defines the image-observation benchmark (Walker and Quadruped with 8 tasks) and the convention of using a model-based Dreamer backbone that the scalability experiments follow.","marker":"[29]"},{"why":"Soft modularization from multi-task RL is the technique the paper extends to the CVAE, enabling stable density estimates when different skills occupy very different state regions.","marker":"[32]"},{"why":"Count-based exploration theory supplies the provably efficient exploration guarantee that Theorem 2 connects the KL-based reward to in tabular MDPs.","marker":"[35]"},{"why":"Neural density models for count-based exploration bridge the tabular bonus to learned representations, the gap Theorem 2's linear-model argument is meant to close.","marker":"[36]"},{"why":"rliable provides the aggregate statistics (median, IQM, mean, optimality gap) used to compare algorithms robustly across seeds and tasks.","marker":"[54]"}],"fun_headline_variants":["SD3 wins URLB by separating skill state regions","New skill discovery objective beats URLB benchmarks","Density-deviation objective redefines skill discovery","Separating skill regions yields top URLB scores and better tasks","Maximize inter-skill state-density deviation for distinct behaviors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything depends on the autoencoder's reconstruction-based density estimate being a faithful stand-in for how often each skill truly visits each state in high-dimensional and image-based environments, and on the KL-to-prior exploration reward still rewarding rare states like visit counts do once the decoder is a deep network rather than the linear model that the proof of Theorem 2 assumes.","fun_headline_variants_meta":{"raw":{"variants":["SD3 wins URLB by separating skill state regions","New skill discovery objective beats URLB benchmarks","Density-deviation objective redefines skill discovery","Separating skill regions yields top URLB scores and better tasks","Maximize inter-skill state-density deviation for distinct behaviors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001221,"raw_usage":{"total_tokens":5084,"prompt_tokens":1067,"completion_tokens":4017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":3940}},"tokens_in":683,"tokens_out":4017,"duration_ms":24086,"temperature":1.0,"reasoning_tokens":3940,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:18:06.117041+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SD3 in a small tabular MDP where true visit counts $N(s,z)$ and true state densities are computable exactly, and check two predictions: the exploration reward should decrease monotonically with $N(s,z)$ and track $|S|/(2(N(s,z)+\\kappa))$, and the density-deviation reward should be higher for states that other skills genuinely under-visit. If either correlation fails, the link between the practical network and the theorems is broken. In an image-based environment, the counterpart test is to measure the overlap between skill state distributions with an independently trained, reward-free encoder and ask whether skills that score high on $I_{SD3}$ actually visit disjoint state regions.","supporting_citations":[{"cited_title":"URLB: Unsupervised reinforcement learning benchmark,","cited_arxiv_id":null,"evidence_quote":"URLB supplies the evaluation protocol: 2M pre-training steps, 100K fine-tuning steps, and the 12 state-based downstream tasks across Walker, Quadruped, and Jaco."},{"cited_title":"Diversity is all you need: Learning skills without a reward function,","cited_arxiv_id":null,"evidence_quote":"DIAYN is the canonical MI-based skill discovery method whose objective I(S;Z) Theorem 1 shows to be the lambda=1 special case of ISD3, and a baseline in the maze and URLB experiments."},{"cited_title":"Behavior contrastive learning for unsupervised skill discovery,","cited_arxiv_id":null,"evidence_quote":"BeCL is a contrastive competence-based baseline that SD3 is compared against on state-based URLB, where the two methods score closely (75.38% vs 77.37% IQM)."},{"cited_title":"METRA: Scalable unsupervised RL with metric-aware abstraction,","cited_arxiv_id":null,"evidence_quote":"Metra is the metric-aware abstraction baseline that defines the standard for scalable, image-based skill discovery, and SD3 is compared against it on both state- and pixel-based URLB."},{"cited_title":"Unsupervised reinforcement learning with contrastive intrinsic control,","cited_arxiv_id":null,"evidence_quote":"CIC is the contrastive intrinsic control baseline with the strongest entropy-style exploration; it is SD3's main competitor on state- and pixel-based tasks and the comparison target in the robustness experiments."},{"cited_title":"Mastering the unsupervised reinforce- ment learning benchmark from pixels,","cited_arxiv_id":null,"evidence_quote":"Pixel-based URLB defines the image-observation benchmark (Walker and Quadruped with 8 tasks) and the convention of using a model-based Dreamer backbone that the scalability experiments follow."},{"cited_title":"Multi-task reinforcement learning with soft modularization,","cited_arxiv_id":null,"evidence_quote":"Soft modularization from multi-task RL is the technique the paper extends to the CVAE, enabling stable density estimates when different skills occupy very different state regions."},{"cited_title":"Unifying count-based exploration and intrinsic motivation,","cited_arxiv_id":null,"evidence_quote":"Count-based exploration theory supplies the provably efficient exploration guarantee that Theorem 2 connects the KL-based reward to in tabular MDPs."},{"cited_title":"Count-based exploration with neural density models,","cited_arxiv_id":null,"evidence_quote":"Neural density models for count-based exploration bridge the tabular bonus to learned representations, the gap Theorem 2's linear-model argument is meant to close."},{"cited_title":"Deep reinforcement learning at the edge of the statistical precipice,","cited_arxiv_id":null,"evidence_quote":"rliable provides the aggregate statistics (median, IQM, mean, optimality gap) used to compare algorithms robustly across seeds and tasks."}],"review_version":1}