{"id":"f63aa1ed-e917-4444-93ae-3ca83d3de3cf","arxiv_id":"2501.09611","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EVaDE inserts three Gaussian-dropout convolutional layers into SimPLe reward models, raising mean human-normalized Atari 100K score from 0.525 to 0.682 in the paper's runs.","lead":"This paper adds three dropout-based convolutional layers, called EVaDE, to the reward model of SimPLe, a model-based reinforcement learning agent, so the agent explores different object interactions in Atari games. On the 100K Atari benchmark, the modified agent reports a higher mean human-normalized score than the original SimPLe baseline and several model-free methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Head-to-head EVaDE vs SimPLe(30) may be confounded by the 30-iteration schedule; the central comparison is internal, and the headline external gains are unverified because baseline numbers are point estimates.","rationale":"The reader identified the central concern as the unverified premise that intermediate channels capture object positions or events. That is a plausible concern about the interpretation of the mechanism, but it is not the most load-bearing for the paper's actual empirical claim. The paper's central claim is that EVaDE layers improve performance; the mechanism is a hypothesis. The most load-bearing issue for that claim is the confound between the architectural change (adding EVaDE layers) and the schedule change (15 to 30 iterations). The paper reports SimPLe(30) at 0.525 whereas the original SimPLe at 15 iterations is 0.443, so just changing the schedule appears to give a meaningful gain (0.443 to 0.525). The paper does not isolate the effect of the schedule from the effect of the layers. The ablation study does show each layer individually beating SimPLe(30), so the layers do have some effect, but the magnitude of the headline 30% improvement is confounded. A concrete test that runs vanilla SimPLe under the paper's 30-iteration schedule would settle this. The paper's internal comparison is otherwise well supported: five runs, per-game scores, IQM, a paired t-test. The external baseline comparisons are weaker because they use point estimates without error bars, but they are not the central claim. The PSRL/Thompson-sampling framing is an interpretation, not a derived guarantee, but this does not undermine the empirical result. I therefore recommend CONDITIONAL, as the reader did, but for a slightly different reason: the conditional acceptance should hinge on verifying that the improvement is not primarily due to the schedule change. The rationales for the reader's concern about the object/event premise are valid for the paper's explanatory narrative, but the central empirical claim is robust to that premise being false, since the visualizations in Figures 5-8 provide some supporting evidence for the mechanism, and the gains are measured behaviorally. The schedule confound is the most concrete, checkable threat to the headline numbers.","tokens_in":24206,"tokens_out":2336,"duration_ms":19569,"concrete_test":"Run vanilla SimPLe (unmodified, no EVaDE layers) under the paper's exact codebase and 30-iteration schedule for the 26-game suite, with the same number of runs and seeds, and compute the mean HNS and IQM. Then compare against the reported EVaDE-SimPLe results. If the vanilla 30-iteration SimPLe mean HNS is substantially above 0.525 (the value reported in the paper), the marginal benefit of the EVaDE layers is smaller than claimed, and the comparison should be recomputed. Alternatively, run EVaDE-SimPLe with the original 15-iteration schedule; if it does not show a comparable gain over vanilla 15-iteration SimPLe, the interaction effect is large.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is the improvement of EVaDE-SimPLe over SimPLe(30) (mean HNS 0.682 vs 0.525), and the abstract also claims a 79% improvement over CURL and 30% over vanilla SimPLe. The internal comparison is the load-bearing one: it is controlled, uses the same codebase, the same 30-iteration schedule, and reports five runs with per-game scores and a paired t-test (p = 3e-3). This comparison is fairly solid. The weakest load-bearing assumption is that the 30-iteration training schedule itself is not a major source of the improvement. The paper changes SimPLe from 15 to 30 iterations (Section 4.2) and reports vanilla SimPLe(30) at 0.525 HNS, but the original SimPLe is reported at 0.443 HNS. The paper's own Table 1 shows SimPLe(30) at 0.525, and EVaDE-SimPLe at 0.682. If the schedule change alone (with the same architecture) accounts for a large part of the gain, the marginal benefit of the EVaDE layers could be much smaller than the headline 30% suggests. The paper does not report a run of the original 15-iteration SimPLe under its own codebase, so we cannot separate the effect of the schedule from the effect of the layers. The ablation in Table 2 compares EVaDE layers against SimPLe(30), all under the 30-iteration schedule, so it partially addresses this, but it does not rule out a strong interaction between the schedule and the added noise layers, e.g., more iterations providing more opportunities for the exploration noise to pay off. A second, somewhat less central concern is that the external baseline comparisons (CURL, SimPLe, OTRainbow, Eff Rainbow) use published point estimates without error bars, but this does not threaten the internal EVaDE vs SimPLe(30) comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Event-based Variational Distributions for Exploration (EVaDE), a set of three noisy convolutional layers (event interaction, event weighting, event translation) that can be inserted into the reward model of a model-based reinforcement learning agent. The authors instantiate EVaDE inside SimPLe, forming EVaDE-SimPLe, and evaluate it on the 26-game Atari 100K benchmark. They report a mean human-normalized score (HNS) of 0.682 for EVaDE-SimPLe, compared to 0.525 for a 30-iteration SimPLe baseline (SimPLe(30)) run in the same codebase, and also compare against CURL and other published baselines. An ablation on 12 games shows that each layer individually improves over SimPLe(30), and all three together perform best. The paper also presents a representability theorem, visualizations of the learned layer behavior, and a statistical analysis (IQM and paired t-test) supporting the internal comparison.","tokens_in":24541,"tokens_out":7087,"duration_ms":70803,"significance":"If the internal results hold, the paper offers a practical and inexpensive plug-in module for improving the data efficiency of model-based RL in object-based domains, with a reproducible and statistically careful internal evaluation (five runs, IQM, paired t-test). The three-layer design is interpretable, and the visualizations give some evidence that the layers perform the intended operations. The main limitations are conceptual: the claimed connection to posterior sampling for RL (PSRL) is not derived, and the ablation does not isolate the event-based structure from generic dropout or added noise. The paper's secondary external comparisons rest on literature point estimates without variance, so the headline gains over CURL should be treated as indicative rather than established.","major_comments":[{"comment":"The paper repeatedly claims that EVaDE-SimPLe 'approximates PSRL' by maintaining an approximate posterior over reward functions, but no derivation or formal argument connects the procedure in Algorithm 1 to posterior sampling. In PSRL, a model is sampled from the posterior at the start of an episode, whereas Algorithm 1 samples one perturbed reward model per SimPLe iteration and then trains a policy against that single sample for 28.8M simulated interactions. This is closer to randomized reward functions or parameter-space noise than to posterior sampling. Because the title and motivation rest on the PSRL framing, the authors should either provide a formal connection (e.g., showing the sampled reward model corresponds to a sample from a well-defined posterior over reward parameters) or substantially weaken the claim to 'inspired by PSRL.'","section":"Section 3.5 and Algorithm 1 (Appendix A)"},{"comment":"The ablation shows that each EVaDE layer improves over SimPLe(30) and that their combination is best, but it does not include a control with generic multiplicative Gaussian dropout applied to the same reward network, nor a control with an unstructured additional layer of comparable parameter count. The authors' parameter-count argument (translation layer adds 4K parameters to a ~10M-parameter model) rules out a pure capacity effect, but it does not rule out the possibility that any form of multiplicative noise on the reward model helps exploration. Without such a control, the paper's central explanatory claim that the specifically structured event-based inductive bias is responsible for the gains is not empirically isolated.","section":"Section 4.4 (Ablation Studies)"},{"comment":"The abstract's headline '79% higher than CURL' is based on a single point estimate from the CURL paper, with no error bars or statistical test comparing the authors' five runs to that baseline. The internal EVaDE-SimPLe vs SimPLe(30) comparison is well supported by the reported five runs, paired t-test (p = 3e-3), and IQM, but the external comparisons to CURL, OTRainbow, and Eff. Rainbow should be presented as literature point estimates and should not be part of the paper's central claims without a proper uncertainty quantification.","section":"Section 4.3, Table 1, and abstract"}],"minor_comments":[{"comment":"The abstract says the 0.525 mean HNS is achieved by 'vanilla SimPLe agents'; since Section 4.2 later defines SimPLe(30) separately from the original 15-iteration SimPLe from [45] (which has mean HNS 0.443), the abstract should say 'SimPLe(30)' to avoid ambiguity.","section":"Abstract"},{"comment":"There is a typo in the sentence 'This affirms that the improvements obtained due to the addition of the EVaDE layers are robust...': the text reads 'Simple(30)' instead of 'SimPLe(30)'.","section":"Section 4.3"},{"comment":"Reference [39] is cited as the source of the SimPLe implementation used in the experiments, but [39] is 'Tensor2Tensor for Neural Machine Translation' by Vaswani et al. This citation appears to be incorrect; the authors should cite the actual repository or the original SimPLe paper's code.","section":"Appendix E.1"},{"comment":"In the proof of Claim 2, the sentence 'As stated in Equation 3, the event interaction layer computes every output yk_i,j' should read 'the event weighting layer' instead of 'event interaction layer'.","section":"Appendix B.3, Claim 2"},{"comment":"The caption says 'Learning curves of EVaDE-SimPLe agents, SimPLe(30) agents and agents which only add one of the EVaDE layers' but the figure itself shows curves for SimPLe(30), the three single-layer variants, and EVaDE-SimPLe; the caption should list all five curves explicitly, as the main text does.","section":"Figure 4 caption"},{"comment":"The random selection of the 12-game subset uses numpy without reporting a seed; since the selected games are not listed in the main text (only in Table 2), the seed or an explicit game list should be given for full reproducibility.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible empirical contribution, but the framing as PSRL is currently unsupported and the external comparisons in the abstract overstate the evidence. The citation error in Appendix E.1 (reference [39] is not a SimPLe implementation) should be checked during revision. The internal evaluation and code release are strengths that make the result credible, but the conceptual gaps prevent acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result holds up: EVaDE-SimPLe beats SimPLe(30) under the same 30-iteration schedule, with five seeds, per-game tables, IQM, a paired t-test, and ablations showing each layer helps. The stress-test worry about the schedule change is a red herring—the paper does not claim to beat the original 15-iteration SimPLe as its headline; the 30% improvement is over their own SimPLe(30). So the internal comparison is clean, and the empirical core is genuinely solid for a workshop-level contribution.\n\nWhat's new: the three EVaDE layers as a set, and their placement in the reward model of SimPLe. Each layer is simple—Gaussian multiplicative dropout with structured sparsity—but the combination is original, and the ablations confirm the whole is better than any single layer. The reporting is unusually careful: 26 games, 5 runs, IQM, per-game scores in the appendix. The visualizations are a nice plus.\n\nSoft spots, in proportion. The Thompson-sampling/PSRL framing is not established: Equation 1 is just a perturbed weight, and the paper offers no argument that it approximates a posterior. That said, the authors openly say the variational distribution need not be posterior-faithful to be useful for exploration, so this is a motivational looseness rather than a broken claim. The external comparisons to CURL and others use published point estimates without error bars, so the 79% figure should be quoted with caution. Theorem 1 is a representability statement and is fine but shallow; it does not support the PSRL interpretation. Minor: the phrase \"vanilla SimPLe\" in the abstract is ambiguous, and one citation for the codebase (Tensor2Tensor) looks wrong.\n\nWho this is for: people working on exploration in model-based RL, especially in object-based image domains. It is a useful plug-in idea with honest, reproducible-looking experiments. I would send it to peer review; a referee should ask for the PSRL framing to be softened or justified, and for a quick run of original 15-iteration SimPLe if they want to compare against the published 0.443, but the main claim is probably correct.","headline":"Solid empirical paper: three new dropout-based exploration layers give a controlled 30% gain over a same-schedule SimPLe(30) baseline on Atari 100K, though the PSRL framing is loose.","tokens_in":25124,"tokens_out":2310,"would_cite":false,"duration_ms":24701,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding three event-designed variational layers to SimPLe's reward model improves Atari 100K exploration, lifting mean human-normalized score by 30 percent over vanilla SimPLe and 79 percent over CURL.","keywords":["event-based variational distributions","Thompson sampling","model-based reinforcement learning","Gaussian dropout","exploration","SimPLe","Atari 100K","object-based domains"],"falsifier":"Replace the three EVaDE layers with generic Gaussian multiplicative dropout applied to ordinary fully populated 1x1 and 3x3 convolutions at the same positions in the network, keeping the parameter count matched, and compare mean human-normalized scores on the 26-game Atari 100K suite; if the control matches or exceeds EVaDE-SimPLe's 0.682 HNS, the event-based filter structure is not the causal factor.","tokens_in":23959,"feed_emoji":"🎮","tokens_out":4702,"duration_ms":47988,"temperature":0.7,"pith_summary":"The paper introduces Event-based Variational Distributions for Exploration (EVaDE), a set of three Gaussian-dropout convolutional layers inserted into the reward network of the model-based agent SimPLe. These layers make the reward function a sample from an approximate posterior, implementing a practical form of Thompson sampling for exploration. On the 100K Atari benchmark, the resulting agent, EVaDE-SimPLe, achieves a mean human-normalized score of 0.682, 79% higher than CURL's 0.381 and 30% higher than vanilla SimPLe(30)'s 0.525. The paper argues that perturbing object interactions, event importances, and object translations in object-based domains generates exploratory trajectories that help the agent escape local optima.","feed_headline":"Event-based dropout layers lift Atari 100K scores by 30 percent","feed_subtitle":"Adding three variational noise layers to SimPLe's reward model beats CURL by 79 percent.","key_machinery":"The three EVaDE layers are convolutional layers whose weights are perturbed multiplicatively by Gaussian noise, following $\\tilde{\\theta}^i_{env} \\leftarrow \\theta^i_{env}(1 + \\sigma^i_{env}\\epsilon^i)$ with $\\epsilon^i \\sim \\mathcal{N}(0,1)$, with variances trained jointly with the model. The noisy event interaction layer combines local patches across channels to perturb object interactions; the noisy event weighting layer scales each channel independently to up- or down-weight events; the noisy event translation layer computes noisy weighted sums over neighboring pixels to translate detected objects. Inserted between the deconvolutional layers of SimPLe's reward network (with weights shared with the transition network), they turn the reward function into a sample from an approximate posterior. A theorem shows each layer can represent the identity, so the augmented network retains the original's representable functions.","core_discovery":"The central claim is that in object-based domains, exploration in model-based reinforcement learning can be improved by designing the variational distribution to induce event-level perturbations: altering interactions between objects, up- or down-weighting the reward associated with detected events, and translating detected objects or events. The paper operationalizes this with three noisy convolutional layers—noisy event interaction, noisy event weighting, and noisy event translation—that apply Gaussian multiplicative dropout to their parameters and are inserted into the reward network of SimPLe. At each iteration, the policy is optimized against a reward function sampled from this event-aware variational family. EVaDE-SimPLe reports a mean HNS of 0.682 on Atari 100K, outperforming CURL (0.381) and SimPLe(30) (0.525), with a paired t-test p-value of 3e-3 and an IQM of 0.339 versus 0.202. The paper also proves that adding these layers does not reduce the representational capacity of the original network, since each EVaDE layer can represent the identity transformation.","pith_inferences":["A direct control experiment using generic Gaussian multiplicative dropout layers with the same parameter count and placement would test whether the structured event-based perturbations, rather than added capacity or simple regularization, cause the measured improvement.","The paper's premise that intermediate convolutional channels capture object positions or events could be verified on simulated manipulation or navigation domains with known object states; such verification would also inform where the layers should be inserted.","The design principle—choosing a variational distribution for the purpose of exploration rather than posterior fidelity—may extend to other model-based planners, including search-based agents that can sample reward models during simulated rollouts."],"forward_implications":["Any model-based RL agent with a convolutional reward model can insert the EVaDE layers to obtain an exploration mechanism without changing the training algorithm.","The three perturbation types provide distinct exploration behaviors, and their combination is more effective than any single layer in the reported ablation.","Adding the layers does not remove the network's ability to represent the true reward function, so observed gains are not explained by a loss of representational capacity.","The reported improvement over SimPLe(30) is statistically significant under a per-game paired t-test and holds under the outlier-robust inter-quartile mean metric."],"supporting_citations":[{"why":"Supplies the SimPLe algorithm and the deterministic world-model architecture into which the EVaDE layers are inserted.","marker":"[45]"},{"why":"Establishes dropout as a Bayesian approximation, grounding the claim that Gaussian dropout induces variational distributions over model parameters.","marker":"[12]"},{"why":"Provides the local reparameterization trick used to sample the Gaussian perturbations in Equation 1.","marker":"[20]"},{"why":"Defines posterior sampling for reinforcement learning, the framework that EVaDE-SimPLe approximates.","marker":"[26]"},{"why":"CURL is the main low-data-regime baseline whose mean human-normalized score is compared against EVaDE-SimPLe.","marker":"[21]"},{"why":"Advocates the inter-quartile mean metric used to show the improvement is robust to outlier games and runs.","marker":"[1]"}],"fun_headline_variants":["EVaDE beats CURL on Atari 100K with event-aware dropout","Event-based dropout layers lift SimPLe to 68.2% HNS on Atari","EVaDE: event-based variational Thompson sampling tops Atari 100K","Three event-aware dropout layers make SimPLe beat CURL on Atari","EVaDE-armed SimPLe achieves 68.2% HNS on Atari vs CURL's 38.1%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design rests on the premise that intermediate convolutional channels in this Atari reward model actually capture object positions or events, so that channel-wise or pixel-wise Gaussian perturbation corresponds to perturbing object interactions, event importance, and translation; if that premise is false, the measured gains could come from added capacity or generic dropout regularization rather than event-directed exploration.","fun_headline_variants_meta":{"raw":{"variants":["EVaDE beats CURL on Atari 100K with event-aware dropout","Event-based dropout layers lift SimPLe to 68.2% HNS on Atari","EVaDE: event-based variational Thompson sampling tops Atari 100K","Three event-aware dropout layers make SimPLe beat CURL on Atari","EVaDE-armed SimPLe achieves 68.2% HNS on Atari vs CURL's 38.1%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00129,"raw_usage":{"total_tokens":5266,"prompt_tokens":943,"completion_tokens":4323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":4204}},"tokens_in":559,"tokens_out":4323,"duration_ms":31705,"temperature":1.0,"reasoning_tokens":4204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:51:28.556456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the three EVaDE layers with generic Gaussian multiplicative dropout applied to ordinary fully populated 1x1 and 3x3 convolutions at the same positions in the network, keeping the parameter count matched, and compare mean human-normalized scores on the 26-game Atari 100K suite; if the control matches or exceeds EVaDE-SimPLe's 0.682 HNS, the event-based filter structure is not the causal factor.","supporting_citations":[{"cited_title":"Model Based Reinforcement Learning for Atari","cited_arxiv_id":null,"evidence_quote":"Supplies the SimPLe algorithm and the deterministic world-model architecture into which the EVaDE layers are inserted."},{"cited_title":"Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning","cited_arxiv_id":null,"evidence_quote":"Establishes dropout as a Bayesian approximation, grounding the claim that Gaussian dropout induces variational distributions over model parameters."},{"cited_title":"V ariational Dropout and the Local Reparameterization Trick","cited_arxiv_id":null,"evidence_quote":"Provides the local reparameterization trick used to sample the Gaussian perturbations in Equation 1."},{"cited_title":"Why is Posterior Sampling Better than Optimism for Reinforcement Learning? InInternational Conference on Machine Learning , pages 2701–2710, 2017","cited_arxiv_id":null,"evidence_quote":"Defines posterior sampling for reinforcement learning, the framework that EVaDE-SimPLe approximates."},{"cited_title":"CURL: Contrastive Unsupervised Representations for Reinforcement Learn- ing","cited_arxiv_id":null,"evidence_quote":"CURL is the main low-data-regime baseline whose mean human-normalized score is compared against EVaDE-SimPLe."},{"cited_title":"Deep Reinforcement Learning at the Edge of the Statistical Precipice","cited_arxiv_id":null,"evidence_quote":"Advocates the inter-quartile mean metric used to show the improvement is robust to outlier games and runs."}],"review_version":1}