{"id":"f4d5d7c7-3ed5-4e56-90b7-55d45607c5d7","arxiv_id":"2412.03800","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"ELEMENT combines an average episodic state entropy reward with a kNN-graph lifelong entropy reward for reward-free RL exploration.","lead":"A new intrinsic reward method for reinforcement learning combines two scales of state entropy maximization: an episodic term that rewards diversity within each episode and a lifelong term that rewards visiting states not seen before. The paper reports that this combination explores MuJoCo and Mario environments better than four prior intrinsic-reward methods, and that the collected experience transfers to offline RL and downstream pre-training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Episodic module's additive decomposition (Eq. 9) is not established: Prop. 1 optimizes an upper bound under constant Tτ, and Fig. 5 validates on task-reward, not exploration, trajectories.","rationale":"The reader already identified the additive-decomposition assumption as the weakest point and issued CONDITIONAL. I agree that this is the most load-bearing issue: the episodic reward is the paper's main novelty, and both Proposition 1 and the empirical validation (Fig. 5) fall short of establishing Eq. 9 in the actual exploration regime. My concern extends the reader's in two concrete ways: the Fig. 5 check uses task-reward SAC trajectories rather than ELEMENT exploration trajectories, and the varying episode lengths from MuJoCo health termination make the constant-Tτ upper-bound argument inapplicable. I did not select Proposition 2's Rényi proof as the main concern because the lifelong reward in the implemented algorithm uses the kNN estimator (Eq. 15), so an invalid Rényi proportionality does not directly affect the tested method. Missing APT/Go-Explore baselines and post hoc estimator selection are real but secondary; they affect the breadth of the empirical claim, not the internal validity of the central mechanism. A direct decomposition-error measurement would settle whether the episodic module does what the paper claims. If it fails, the paper would need substantial revision, but the empirical results could still hold as a heuristic, so CONDITIONAL remains appropriate.","tokens_in":18019,"tokens_out":11389,"duration_ms":119983,"concrete_test":"Run ELEMENT episodic-only (β=0) on Ant and Hopper, and for each collected episode τ compute the relative decomposition error |H_{s∈τ}(s) − Σ_t rep(s_t)| / H_{s∈τ}(s), using rep(s) computed exactly as Eq. (11) from the replay buffer at collection time. Report the mean and 90th percentile, split by whether the episode ended by health violation or by the 1000-step truncation. If the median error exceeds roughly 20% or grows with Tτ variability, the episodic reward does not realize the claimed entropy objective; also compare against a control that assigns the true per-episode entropy H(τ) as a terminal reward to isolate the proxy's cost.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ELEMENT maximizes episodic state entropy depends on Eq. 9, Rep(τ) ≈ Σ_t rep(s_t). Proposition 1 does not prove this decomposition: it minimizes the upper bound L̃ = L + E[Tτ²·V_{s∈τ}(rep(s))] and assumes all episodes share the same length Tτ. A small upper bound can coexist with a large decomposition error when the variance term dominates, so the derived rep(s) in Eq. 11 is not certified to make Σ_t rep(s_t) close to H_{s∈τ}(s). The only empirical support, Fig. 5, is trained on roughly 2,000 trajectories from a task-reward SAC agent in Ant, not on ELEMENT's exploration policies. Moreover, MuJoCo episodes terminate stochastically on health violations (Section IV-A), so Tτ is not constant; the appendix's varying-length formula differs from Eq. 11. If the additive proxy is inaccurate under ELEMENT's own exploration distribution, the episodic reward does not actually maximize episodic state entropy, and Fig. 6's wins would reflect an unvalidated heuristic rather than the stated objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ELEMENT, an intrinsic-reward method for reward-free reinforcement learning that combines an episodic reward, defined as the average state entropy of episodes containing a given state (Eq. 11), with a lifelong reward based on kNN distances in a fast approximate kNN graph (Eq. 15). The authors claim that the episodic reward is the optimal solution to an upper bound of a trajectory-reward decomposition loss (Proposition 1) and that common entropy estimators are proportional to kNN-distance sums (Proposition 2). They evaluate ELEMENT against RND, NGU, RE3, and RISE in MuJoCo and Mario environments on episodic state entropy, lifelong state coverage, offline RL data collection, and task-agnostic pre-training.","tokens_in":18258,"tokens_out":8404,"duration_ms":90069,"significance":"If the claims hold, ELEMENT would be a practically useful, multiscale exploration module: the episodic/lifelong combination addresses the known vanishing-reward problem of purely lifelong entropy rewards, and the kNN graph reduces the per-step cost of lifelong entropy estimation from O(N^2) to a much smaller search-based cost. The paper contains a concrete algorithmic pipeline (Algorithms 1-3), qualitative behavioral visualizations, a broad set of environments, and an explicit discussion of limitations (fixed encoder, episode-length trade-off) in Section V. However, the theoretical support is currently not sound: Proposition 1's stated solution does not follow from the minimization problem, Proposition 2 contains sign and approximation errors, and the empirical validation of the key decomposition assumption is performed on task-reward trajectories rather than on ELEMENT's exploration distribution. These issues are load-bearing for the central claim that ELEMENT maximizes episodic and lifelong state entropy, so despite the promising empirical results, the manuscript requires substantial revision.","major_comments":[{"comment":"The stated optimal solution in Proposition 1 is not the minimizer of the stated loss. For a per-state function rep(s), minimizing E_{τ,t}[(H(τ)/Tτ - rep(s_t))^2] pointwise yields rep(s) = E[H(τ)/Tτ | s∈τ] (with the denominator P(s∈τ)), not the unnormalized expression E[H(τ)/Tτ · I(s∈τ)] in Eq. (13). Because the missing normalization depends on s through P(s∈τ), Eq. (13) is not a constant rescale of the true conditional expectation. Eq. (11) drops the 1/Tτ factor, so its relationship to the derived solution is also only up to an unstated scale. Moreover, the proof assumes constant Tτ, but Section IV-A and Section V(b) acknowledge that MuJoCo episodes terminate stochastically; the appendix's variable-length formula Σ Tτ H I / Σ Tτ^2 I is different from Eq. (11) and is not used. These points need to be corrected or the theoretical claim in the abstract, that the episodic reward provides an optimal solution for an upper bound, must be substantially downgraded.","section":"Section III-B, Eqs. (12)-(13) and Appendix A"},{"comment":"The proof of Proposition 2 contains a sign error that makes the expression undefined: Eq. (14c) and the derivation below Eq. (23) state H2(s) ∝ log Σ_i [ - Σ_{j∈kNN} κ(s_i,s_j) ], where the argument of the outer logarithm is negative. The derivation also omits the trace normalization in Eq. (2) and incorrectly replaces log Σ_i Σ_j κ(s_i,s_j)^2 with log Σ_i (Σ_j κ(s_i,s_j)). For the kNN estimator, Eq. (14b) equates log Σ_{j∈kNN} ||s_i-s_j||^2 with log ||s_i - s_i^{kNN}||^2, but the Kozachenko-Leonenko estimator in Eq. (4) applies the logarithm to the d-th power of a single distance, not to a sum of squared distances over k neighbors. Consequently, Proposition 2 does not establish that the lifelong reward r_l in Eq. (15) is proportional to any of the stated entropy estimators. The authors should either provide a correct derivation or reframe the kNN graph reward as a heuristic acceleration of kNN-distance-based intrinsic rewards.","section":"Section III-C, Proposition 2 and Appendix A.1.b"},{"comment":"The empirical validation of the additive decomposition in Eq. (9) is not performed on the distribution to which the claim applies. The text states that the authors 'train a SAC agent in the Ant environment using default task-driven rewards and record all historical states,' yielding about 2,000 trajectories, and Fig. 5 reports the resulting regression loss. This validates the decomposition for task-reward SAC trajectories, not for ELEMENT's exploration policies, which are trained without extrinsic reward. The central claim that ELEMENT maximizes episodic state entropy depends on the accuracy of the additive proxy under ELEMENT's own exploration distribution. The authors should re-run the decomposition validation on rollout data collected from ELEMENT checkpoints and report the approximation error on that distribution, or justify why the task-reward distribution suffices.","section":"Section IV-B, Fig. 5"},{"comment":"The claim that ELEMENT 'significantly outperforms state-of-the-art intrinsic rewards' is not supported by the reported statistics. In Table I, RISE has a higher mean than ELEMENT on Hopper (0.534 vs. 0.516), and the RE3 entry on Ant carries a very large error bar (±0.347); no significance tests or seed-level tables are provided for any of the comparisons. Figs. 6-8 report means with variance but do not include pairwise statistical comparisons. Since this claim appears in the abstract and is a central selling point, the authors should add appropriate statistical tests, report per-seed results, and either substantiate or temper the 'significantly outperforms' phrasing.","section":"Abstract and Section IV, Table I and Figs. 6-8"}],"minor_comments":[{"comment":"There are numerous typos and inconsistencies, including 'dimentionality' in Eq. (4), 'ElEMENT' in Section IV-B, 'unsuperivisedly' in Section VI, 'fatience' in Table II, and 'wihh' in the author affiliation; the paper also alternates between 'Mujoco' and 'MuJoCo'.","section":"Throughout and Section II-A"},{"comment":"The hyperparameter tables (Tables III-VI) cover only the MuJoCo environments; the Mario experiments do not report the ELEMENT-specific values (β, entropy estimator, k, graph search steps, graph update interval) that are needed to reproduce the Mario results in Figs. 6-8 and Table I.","section":"Section IV and Appendix: experimental settings"},{"comment":"The pseudocode writes 'Estimate rep = Hs∈Dτ(s)', but Eq. (11) requires assigning a per-state value based on the entropy of episodes containing that state; the algorithm should clarify whether the same episodic entropy value is assigned to every state in the trajectory or whether some count-based normalization is applied.","section":"Algorithm 3, line 14"},{"comment":"The episodic evaluation metric is the matrix-based entropy functional with α=1.001, while the episodic reward in Ant and Humanoid uses the same family of estimators (matrix-based Rényi with α=3); this objective-metric alignment is not circular, but it limits the independence of the episodic entropy evaluation and should be acknowledged in the experimental section.","section":"Section IV-B and Fig. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a promising empirical method and a sizable experimental study, but the theoretical justifications for both reward modules need correction before the central claims can be accepted. I would ask the authors to fix or remove the flawed Propositions 1 and 2, validate the decomposition under ELEMENT's own exploration distribution, and provide seed-level statistical support for the 'significantly outperforms' claim. The absence of a code release and the missing Mario hyperparameters also complicate reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper proposes ELEMENT, a combination of episodic and lifelong entropy-based intrinsic rewards. The lifelong reward is the standard kNN log-distance from APT/RE3, and the episodic reward — the average entropy of the episodes containing a state — is the genuinely new piece. The kNN graph speedup for the lifelong reward is a reasonable engineering contribution.\n\nWhat the paper does well: the idea of using episodic entropy to counteract the vanishing novelty of lifelong rewards is clearly motivated, and the qualitative results (fireworks trajectories) show the two terms behave as intended. The experiments cover five environments and include offline RL and pre-training evaluations, which is broader than many exploration papers.\n\nThe soft spots are real but not all equally damaging. The proof of Proposition 2 for the matrix-based Rényi estimator in the appendix contains a sign error — it takes the log of a negative quantity — so that part of the theory is invalid. That said, the algorithm uses the kNN estimator for the lifelong reward, so the flawed Rényi proof is not load-bearing; it should still be fixed. More important, Proposition 1 does not actually prove that the episodic reward in Eq. (11) makes Σ rep(s_t) close to H(τ). It shows that Eq. (11) minimizes an upper bound on a regression loss, under the assumption that all episodes have constant length Tτ. MuJoCo episodes terminate stochastically, so that assumption is violated, and the appendix's varying-Tτ formula differs from the main text. The empirical validation in Fig. 5 is trained on task-reward trajectories, not on ELEMENT's own exploration policy, so it does not close the gap. The additive decomposition remains a heuristic.\n\nOther issues: the entropy estimator is chosen per environment after seeing results, which is post hoc; APT, the closest predecessor for the lifelong reward, is not included as a baseline; and several of the reported wins overlap within error bars, despite the 'significantly outperforms' claim. No code is released.\n\nOverall, I think the core method is plausible and the episodic-plus-lifelong combination is a reasonable contribution, but the paper overclaims both theoretically and empirically. It deserves a serious peer review, not a desk reject. The referee should ask for a correct proof of Prop. 2, a proper treatment of varying Tτ, APT and Go-Explore baselines, significance tests, and code. If the authors can fix those, this becomes a useful paper for people working on reward-free exploration.","headline":"A plausible multiscale exploration method whose episodic reward is theoretically under-supported and whose lifelong reward is essentially APT/RE3; worth a careful revision, not a desk reject.","tokens_in":18813,"tokens_out":3966,"would_cite":false,"duration_ms":38362,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ELEMENT maximizes state entropy at two time scales—within each episode and across the agent's whole history—and reports that this dual-scale intrinsic reward outperforms four state-of-the-art exploration methods in episodic entropy…","keywords":["reinforcement learning","intrinsic motivation","maximum entropy exploration","episodic exploration","lifelong exploration","k-nearest neighbors entropy estimation","task-agnostic pre-training","offline reinforcement learning"],"falsifier":"Take ELEMENT's episodic reward and, on Ant-v4 or Walker2D-v4, compare it against an agent trained with the true trajectory-level episodic entropy delivered only at episode end and redistributed by a learned credit-assignment method; if ELEMENT does not match or beat that oracle, the additive decomposition is the failure point. A simpler diagnostic: record episodes during training and compute the per-trajectory error $|H_{s\\in\\tau}(s)-\\sum_t r_{\\mathrm{ep}}(s_t)|$; if the error grows as episodes get longer or terminate stochastically, the proxy reward is not actually maximizing episodic state entropy.","tokens_in":17738,"feed_emoji":"🧭","tokens_out":10760,"duration_ms":90624,"temperature":0.7,"pith_summary":"ELEMENT is an intrinsically motivated reinforcement learning framework that explores environments with no extrinsic reward by maximizing state entropy at two time scales at once: the entropy of states visited within a single episode, and the entropy of all states visited across the agent's lifetime. The paper's central claim is that these two objectives are complementary and that combining them outperforms four state-of-the-art intrinsic rewards—RE3, RISE, NGU, and RND—on episodic state entropy, lifelong state coverage, offline reinforcement learning data collection, and unsupervised pre-training for downstream tasks. It introduces two mechanism-level contributions: a non-parametric 'average episodic state entropy' reward that turns the trajectory-level episodic entropy objective into a per-state Markovian signal, and a k-nearest-neighbors graph that makes lifelong entropy estimation practical as the number of visited states grows. If the claims hold, ELEMENT offers a reusable intrinsic reward module for reward-free exploration and for gathering better datasets for offline reinforcement learning.","feed_headline":"Two-scale entropy reward beats single-scale exploration in RL","feed_subtitle":"Combining within-episode and lifetime state diversity beats four baselines and boosts offline RL data.","key_machinery":"The two load-bearing mechanisms are (1) the average episodic state entropy reward, $r_{\\mathrm{ep}}(s)=\\mathbb{E}_{\\tau \\ni s}[H_{s\\in\\tau}(s)]$, a Markovian per-state bonus that approximates the non-Markovian trajectory reward $H_{s\\in\\tau}(s)$; Proposition 1 identifies it as the optimal solution to an upper bound of a trajectory-reward decomposition loss, and Proposition 2 justifies the choice of estimator by showing that KDE, kNN, and matrix-based Rényi entropy are all proportional to sums of kNN distances. (2) A k-nearest-neighbors graph with fast approximate search and online updates provides the lifelong reward $r_l(s)=\\log(\\|s-\\mathrm{GNNS}(G,s)\\|_2+1)$ and reduces the lifelong entropy computation from $O(N^2)$ to roughly $O(Nk)$ with fixed search hyperparameters, making lifelong exploration scalable to millions of states. The reward used in training is the sum $r_i(s)=r_{\\mathrm{ep}}(s)+\\beta r_l(s)$, with the graph-update hyperparameters $U$ and $T_u$ controlling how often the lifelong memory refreshes.","core_discovery":"The paper claims that maximizing Shannon state entropy at two complementary scales—episodic and lifelong—yields better reward-free exploration than either scale alone or than predictive-uncertainty novelty. The episodic reward is defined as $r_{\\mathrm{ep}}(s)=\\mathbb{E}_{\\tau \\ni s}[H_{s\\in\\tau}(s)]$, the average entropy of the episodes that contain $s$, and Proposition 1 shows this function is the optimal solution to an upper-bounded reward-redistribution loss rather than to the exact decomposition of episodic entropy into per-state rewards. The lifelong reward is $r_l(s)=\\log(\\|s-\\mathrm{GNNS}(G,s)\\|_2+1)$, where GNNS returns $k$ nearest neighbors from a fast approximate kNN graph; Proposition 2 shows that kernel density estimation, kNN, and matrix-based Rényi entropy estimators are each proportional to sums of kNN distances, which justifies using the graph to estimate lifelong entropy. In experiments with SAC and A3C backbones, ELEMENT is reported to beat RE3, RISE, NGU, and RND across Hopper, Walker2D, Ant, Humanoid, and Mario in episodic entropy, unique visited states, offline CQL scores, and downstream task returns.","pith_inferences":["An extension the paper leaves implicit: the episodic reward is a credit-assignment scheme for non-Markovian exploration feedback, so the same average-entropy construction could be applied to other trajectory-level objectives such as coverage or skill-diversity scores, not just entropy.","A likely bottleneck not tested here is ELEMENT's fixed encoder; if the encoder were updated online, representation drift could inflate novelty and break the entropy estimates, so a natural test is to combine ELEMENT with an adaptively learned representation and compare state-coverage curves.","The Proposition 1 upper bound depends on within-episode reward variance, which suggests a testable correction: weighting or clipping $r_{\\mathrm{ep}}$ by its per-episode variance should tighten the bound and may improve performance on environments with stochastic termination.","The maze reward visualization shows that states near the start keep high episodic reward because they lead to high-entropy episodes, suggesting ELEMENT implicitly discovers bottleneck states; one could test whether the episodic reward correlates with state visitation betweenness in graph-theoretic terms."],"forward_implications":["If ELEMENT is right, an agent needs no task reward to build a good exploration policy: the dual-scale entropy signal alone produces diverse, reusable behaviors in continuous-control and platform environments.","The same reward can be used at pre-training time, so downstream tasks start from policies that already cover multiple motion skills, giving faster and sometimes zero-shot transfer.","Datasets collected by ELEMENT exploration policies improve offline reinforcement learning: normalized CQL scores in the paper increase on four of five environments compared with baseline exploration data.","The kNN graph makes lifelong entropy maximization computationally feasible, so the method can escape the first-in-first-out queues or downsampled buffers used by earlier kNN-based intrinsic rewards.","Because Proposition 2 links KDE, kNN, and Rényi estimators, ELEMENT can in principle be instantiated with any of these estimators depending on the environment."],"supporting_citations":[{"why":"Supplies the active pre-training baseline that first uses kNN particle distances as intrinsic rewards.","marker":"[4]"},{"why":"Provides RE3 and the empirical basis for using a fixed random encoder to estimate state entropy.","marker":"[6]"},{"why":"Provides RISE, the Rényi entropy baseline and estimator that ELEMENT compares against.","marker":"[7]"},{"why":"Gives the k-nearest-neighbors entropy estimator used for both episodic and lifelong rewards.","marker":"[13]"},{"why":"Supplies NGU, the episodic-intrinsic-reward baseline; the paper argues ELEMENT addresses NGU's novelty-vanishing issue with a simpler non-parametric reward.","marker":"[17]"},{"why":"Provides the matrix-based Rényi entropy functional used to estimate episodic state entropy in evaluations.","marker":"[18]"},{"why":"Supplies the fast approximate graph nearest-neighbor search that makes lifelong kNN reward computation tractable.","marker":"[19]"},{"why":"Supplies the online kNN graph update mechanism that keeps the lifelong graph current as new states arrive.","marker":"[20]"},{"why":"Gives the trajectory-wise reward redistribution loss whose upper bound Proposition 1 optimizes with the average episodic state entropy reward.","marker":"[32]"},{"why":"Provides Conservative Q-Learning, the offline RL algorithm used to evaluate the data collected by ELEMENT and baselines.","marker":"[39]"}],"fun_headline_variants":["Two-scale entropy reward beats single-scale exploration","Episodic plus lifelong entropy outperforms standard novelty","Multiscale entropy intrinsic reward for better RL exploration","Entropy on two timescales improves reward-free RL agents","ELEMENT: entropy exploration that aids offline RL and pre-training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All results depend on the assumption that an episode's entropy score can be approximated by adding up separate per-state rewards, even though the paper's proof establishes optimality only for an upper-bound loss and assumes constant episode length, while MuJoCo episodes can terminate early on health violations.","fun_headline_variants_meta":{"raw":{"variants":["Two-scale entropy reward beats single-scale exploration","Episodic plus lifelong entropy outperforms standard novelty","Multiscale entropy intrinsic reward for better RL exploration","Entropy on two timescales improves reward-free RL agents","ELEMENT: entropy exploration that aids offline RL and pre-training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1717,"prompt_tokens":1022,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":617}},"tokens_in":638,"tokens_out":695,"duration_ms":7105,"temperature":1.0,"reasoning_tokens":617,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:05:11.430902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take ELEMENT's episodic reward and, on Ant-v4 or Walker2D-v4, compare it against an agent trained with the true trajectory-level episodic entropy delivered only at episode end and redistributed by a learned credit-assignment method; if ELEMENT does not match or beat that oracle, the additive decomposition is the failure point. A simpler diagnostic: record episodes during training and compute the per-trajectory error $|H_{s\\in\\tau}(s)-\\sum_t r_{\\mathrm{ep}}(s_t)|$; if the error grows as episodes get longer or terminate stochastically, the proxy reward is not actually maximizing episodic state entropy.","supporting_citations":[{"cited_title":"Behavior from the void: Unsu- pervised active pre-training,","cited_arxiv_id":null,"evidence_quote":"Supplies the active pre-training baseline that first uses kNN particle distances as intrinsic rewards."},{"cited_title":"State entropy maximization with random encoders for efficient exploration,","cited_arxiv_id":null,"evidence_quote":"Provides RE3 and the empirical basis for using a fixed random encoder to estimate state entropy."},{"cited_title":"Rényi state entropy maximization for exploration acceleration in reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"Provides RISE, the Rényi entropy baseline and estimator that ELEMENT compares against."},{"cited_title":"Nearest neighbor estimates of entropy,","cited_arxiv_id":null,"evidence_quote":"Gives the k-nearest-neighbors entropy estimator used for both episodic and lifelong rewards."},{"cited_title":"Measures of entropy from data using infinitely divisible kernels,","cited_arxiv_id":null,"evidence_quote":"Provides the matrix-based Rényi entropy functional used to estimate episodic state entropy in evaluations."},{"cited_title":"Fast approximate nearest-neighbor search with k-nearest neighbor graph,","cited_arxiv_id":null,"evidence_quote":"Supplies the fast approximate graph nearest-neighbor search that makes lifelong kNN reward computation tractable."},{"cited_title":"Fast Online k-nn Graph Building","cited_arxiv_id":"1602.06819","evidence_quote":"Supplies the online kNN graph update mechanism that keeps the lifelong graph current as new states arrive."},{"cited_title":"Con- servative q-learning for offline reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides Conservative Q-Learning, the offline RL algorithm used to evaluate the data collected by ELEMENT and baselines."}],"review_version":1}