{"id":"15c7f5ed-6d5e-4e63-9d2c-6909c292b703","arxiv_id":"2501.17842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Sparse-to-dense reward transitions, inspired by toddler learning, improve goal-oriented RL performance and generalization over fixed dense or sparse rewards.","lead":"This paper tests a training schedule where reinforcement learning agents start with sparse rewards and later switch to dense, distance-based rewards. Across robot arm, navigation, and maze tasks, the schedule improves success rates and sample efficiency, and the authors argue it smooths the learning landscape.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The S2D benefit may stem from stale sparse-reward transitions lingering in the replay buffer after the stage shift: Algorithm 1 neither clears nor relabels old transitions, so S2D is partly 'sparse+dense mixed replay', not a pure sparse-to-dense curriculum.","rationale":"The paper's central claim is that the Sparse-to-Dense reward transition smooths the policy loss landscape, yields wider minima, and improves generalization. The performance comparisons in Table 2 and the loss-landscape and sharpness analyses all rely on Algorithm 1, whose off-policy loop preserves pre-transition transitions without relabeling. This is the weakest link in the causal chain: the implementation conflates a temporal curriculum with mixed-reward replay. If old sparse-reward transitions are removed or relabeled at the switch, the S2D benefit may vanish in the SAC environments, undermining the central claim. The reader's identified weakest assumption matches this point exactly, and I find no more load-bearing concern. The sharpness metric's high variance and the absence of significance tests weaken the strength of the language but do not invalidate the mechanism itself; the loss-landscape visualizations are qualitative but secondary. The proposed experiment is straightforward and would settle whether the observed gains are due to the curriculum or to the continued presence of sparse-reward transitions. Since the paper already warrants conditional acceptance pending verification, my read does not change the reader's verdict.","tokens_in":27370,"tokens_out":4404,"duration_ms":46442,"concrete_test":"Re-run the SAC experiments in LunarLander, CartPole-Reacher, and UR5-Reacher with Algorithm 1 modified so that at the stage transition (T >= Tk) the replay buffer is either (a) emptied or (b) relabeled: for every stored transition, set r_tilde = r_t + F_k(s_t, a_t) using the new stage's potential function. Compare S2D against Only Dense under both modifications. If S2D no longer outperforms Only Dense, the reported advantage is attributable to stale sparse-reward replay rather than to the sparse-to-dense transition itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 stores each transition with the reward computed for the current stage (lines 13-16 of Algorithm 1) and never resets or relabels the replay buffer when the curriculum advances (lines 19-21). Therefore, after the sparse-to-dense switch, the agent continues to sample and train on transitions whose rewards were generated under the sparse MDP. The Only Dense baseline contains only dense-reward transitions throughout. Consequently, the headline S2D-versus-Only-Dense comparison in Table 2 does not isolate the effect of a temporal curriculum; it also includes continued off-policy training on exploratory sparse-reward data. The observed gains, e.g., CartPole-Reacher S2D(C2) 14.61 +/- 10.96 vs Only Dense 3.88 +/- 4.63, could be driven by this mixed-reward replay rather than by the transition itself. The wide-minima and loss-landscape conclusion inherits the same confound because the post-transition policy is trained on a mixture of reward functions. A clean test is to clear the buffer or relabel all stored rewards with the current-stage potential-based reward at the transition point; this directly distinguishes whether the temporal curriculum or the lingering sparse-reward data causes the reported advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a Sparse-to-Dense (S2D) reward curriculum for goal-oriented reinforcement learning, inspired by toddler development. The curriculum starts with sparse rewards and transitions to potential-based dense rewards while preserving optimal policies via potential-based reward shaping. The authors evaluate S2D against Only Sparse, Only Dense, Dense-to-Sparse, and intrinsic motivation baselines across seven main environments, including robotic arm control (UR5-Reacher), CartPole-Reacher, LunarLander, ViZDoom navigation, and Minecraft mazes. They report improved sample efficiency and success rates, and use a 'Cross-Density Visualizer' and a sharpness metric to argue that S2D smooths the policy loss landscape and leads to wider minima. They also reinterpret Tolman's maze experiments as evidence for the benefit of early free exploration.","tokens_in":27625,"tokens_out":6210,"duration_ms":55531,"significance":"The empirical scope is a clear strength: the paper evaluates across state-based and visual, discrete and continuous domains, and includes generalization tests in ViZDoom-Unseen and Cross Maze. The idea of a temporal reward-density curriculum with PBRS is well-motivated and practically relevant. If the reported effects are real, the loss-landscape analysis would be a useful tool for understanding reward shaping. However, the central comparison is threatened by a replay-buffer confound in Algorithm 1, and the quantitative support is weakened by large variances and absence of significance tests. The visual loss-landscape evidence is mostly qualitative. The manuscript is a reasonable candidate for publication after substantial revision.","major_comments":[{"comment":"The replay buffer is neither cleared nor relabeled when the curriculum advances, so after the sparse-to-dense transition the agent continues training on stale sparse-reward transitions. The Only Dense baseline never sees sparse rewards. The S2D-versus-Only-Dense comparisons in Table 2 therefore conflate a temporal curriculum with mixed-reward off-policy replay. Please run a control that clears or relabels the buffer at the transition point, and report whether the S2D advantage persists.","section":"§4.1, Algorithm 1 (lines 13–21)"},{"comment":"The performance claims are not supported by statistical evidence. Several of the reported means have very large standard deviations (e.g., LunarLander S2D(C2) 63.40±160.55 vs. Only Dense 139.68±14.90; CartPole S2D(C2) 14.61±10.96 vs. Only Dense 3.88±4.63), and no significance tests are reported. Please provide per-seed results, confidence intervals, and significance tests (or effect sizes) for the main pairwise comparisons.","section":"§6.1, Table 2"},{"comment":"The loss-landscape smoothing claim is based on visual inspection of 2D slices, and the sharpness metric is only reported at the end of training. Sharpness can be artificially low when the policy barely moves (as acknowledged for Only Sparse in Section 7.3). Please provide quantitative landscape measures (e.g., local Hessian norms or neighborhood loss variance) and report sharpness trajectories across training, not only the endpoint.","section":"§6.2–6.3, Figure 5 and Eq. (5)"},{"comment":"The optimal transition timing is described as 'within the first third of training' in the caption and 'first quarter' in Section 7.1, but for LunarLander the best S2D result uses C3 (400k of 1M steps, i.e., 40% of training). Please reconcile these statements and specify how N is chosen.","section":"§5.1.2 and Table 2 caption"}],"minor_comments":[{"comment":"The entries in the 'Dense Reward' column (e.g., 'γ Φ(s_{t+1}) − Φ(s_t) < 0.3') are ambiguous; clarify whether the dense reward is the potential-based difference itself or a threshold condition for granting a bonus.","section":"Table 1"},{"comment":"The citations for OpenAI Gym and MuJoCo are both [55]; the Gym reference should be [7].","section":"Table 3"},{"comment":"The number of trials is stated as 'at least six' in Table 2 and 'at least five' in Section 11.4; make the numbers consistent.","section":"Table 2 and §11.4"},{"comment":"The statement that 'the relative position of one landscape over another is not significant' is confusing, since the smoothing comparison relies on their relative relief; please clarify the normalization.","section":"§4.2"},{"comment":"The reward transition is reported at 3M steps, but the main text does not specify that the RNN analysis uses C3; add a pointer to the transition-timing table.","section":"Figure 9(a)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript builds heavily on the authors' AAAI 2024 paper [47], and the incremental contribution is mostly empirical. The replay-buffer confound, if not addressed, could lead to a misleading conclusion; the proposed control experiment is straightforward and should be decisive. The paper is within the scope of the journal, but the evidence should meet the standard of the field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper's most defensible new contribution is the empirical demonstration that a sparse-to-dense reward schedule often beats only-dense and only-sparse in a wide range of goal-oriented RL tasks. The loss-landscape and wide-minima story is the headline, but it is not yet established. The central comparison is confounded by the replay buffer: Algorithm 1 stores transitions with the reward of the stage in which they were collected and never clears or relabels them when the curriculum switches. So after the switch, the S2D agent trains on a mixture of sparse-reward and dense-reward transitions, while the Only Dense baseline trains on dense transitions throughout. That means the observed advantage could come from continued off-policy training on exploratory sparse data rather than from the transition itself. The stress-test note is correct, and a clean relabeling (or buffer reset) experiment is the obvious fix.\n\nWhat the paper does well: the breadth of environments is real. Seven main tasks, including custom ViZDoom and Minecraft mazes, plus appendix experiments with three RL algorithms. The PBRS grounding is standard and correctly cited, and the sharpness metric follows Foret et al. The loss landscape visualizer is a legitimate tool, and the observation that S2D landscapes look smoother is worth reporting. The paper also builds explicitly on the authors' earlier AAAI'24 work, so the novelty is incremental; this is an extension, not a new mechanism.\n\nThe soft spots are proportionate. No statistical significance tests anywhere, and several reported numbers have huge standard deviations (CartPole S2D C2: 14.61±10.96). The sharpness results do not uniformly support the wide-minima claim: in UR5, Only Dense has identical sharpness to S2D, and Only Sparse produces low sharpness with zero performance, which the authors acknowledge but don't resolve. The loss landscape smoothing is judged by eye. The Tolman maze reinterpretation is evocative but not evidence. The lack of released code and the custom environments make reproducibility harder.\n\nWho is this for? Practitioners looking for a low-cost reward schedule might take the empirical recipe. Researchers interested in how reward shaping affects optimization landscapes will find the hypothesis interesting but need the confound cleaned up first. The paper deserves a serious referee; the question is important and the experimental effort is substantial. I would send it to review, with the expectation of major revision: relabel or clear the replay buffer at transition, add significance tests, and ideally release code. As it stands, I would not cite it for the mechanism claim.","headline":"A broad empirical study of sparse-to-dense reward shaping whose loss-landscape headline is undercut by a mixed-replay confound and missing significance tests.","tokens_in":28172,"tokens_out":3288,"would_cite":false,"duration_ms":32628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse-to-dense reward schedules make RL agents learn faster, generalize better, and settle in wider minima.","keywords":["reinforcement learning","reward shaping","sparse rewards","curriculum learning","potential-based reward shaping","loss landscape","wide minima","generalization"],"falsifier":"Train the same S2D schedule but relabel every replay-buffer transition with the current stage's reward before each update, or clear the buffer at the transition point. If the S2D advantage over Only Dense shrinks or disappears under relabeling, the mechanism is the stale mixed rewards, not the sparse exploration phase. A second check: if S2D works by forming a cognitive map, its advantage should be measurable as better transfer when both the goal and spawn are randomized at evaluation, and should shrink when goal positions are fixed and visible.","tokens_in":27154,"feed_emoji":"🎯","tokens_out":3247,"duration_ms":26080,"temperature":0.7,"pith_summary":"This paper argues that reinforcement learning agents, like toddlers, learn best when they start with sparse rewards that force free exploration and then switch to dense, goal-shaped rewards. Across robotic arm, navigation, and maze tasks, the authors show that this Sparse-to-Dense (S2D) transition beats using only sparse, only dense, or reverse-order rewards on success rate and sample efficiency. They also provide evidence that the transition smooths the policy loss landscape, guiding the agent into wider, more generalizable minima. If correct, reward scheduling itself becomes a cheap way to improve RL robustness without changing the algorithm or the environment.","feed_headline":"Sparse-then-dense rewards speed up RL and widen its minima","feed_subtitle":"Training agents to explore before shaping goals beats fixed reward density and improves generalization.","key_machinery":"The load-bearing mechanism is the S2D curriculum built on potential-based reward shaping: the agent trains first under a sparse reward, then switches to a denser reward $R_i + F_i$ where $F_i(s,a) = \\gamma \\Phi_i(s') - \\Phi_i(s)$ with a distance-to-goal potential $\\Phi(s) = \\operatorname{diam}_p(S) - \\|s - g\\|_p$. PBRS guarantees the dense reward preserves the optimal policy, so the density change can only affect the optimization trajectory, not the task definition. This is what lets the authors attribute the gains to the transition itself. The analysis machinery is the Cross-Density Visualizer, a 3D plot of policy loss over randomly perturbed parameters, and the sharpness metric $\\max_{\\|\\epsilon\\|_p \\le \\rho} [L_\\pi(\\theta+\\epsilon) - L_\\pi(\\theta)]$, used to measure the width of the final minima.","core_discovery":"The central claim is that transitioning a goal-oriented RL agent from a sparse reward to a potential-based dense reward over the course of training—rather than fixing one reward density—improves learning performance, sample efficiency, and generalization. The authors define an S2D curriculum as a sequence of MDPs whose reward supports grow monotonically while the set of optimal policies shrinks monotonically, and they use potential-based reward shaping to preserve optimality. Empirically, across LunarLander, CartPole-Reacher, UR5-Reacher, ViZDoom, and Minecraft maze tasks, S2D outperforms Only Sparse, Only Dense, and Dense-to-Sparse baselines. Using a Cross-Density Visualizer of the policy loss landscape and a sharpness metric, they report that S2D smooths the loss landscape and converges to wider minima than other reward strategies. They reinterpret Tolman's latent-learning maze experiments, arguing that early sparse-reward free exploration builds robust initial parameters and faster-converging internal representations that later dense-reward learning exploits.","pith_inferences":["The loss-landscape smoothing is presented as a consequence of the sparse phase, but it could equally be caused by the off-policy replay buffer mixing two reward functions; a controlled comparison against relabeled or fresh-buffer variants would isolate the mechanism.","The toddler and Tolman analogies suggest that S2D may be most beneficial when the agent must form a spatial or object-centric model; in tasks where the goal is trivially locatable, the advantage should shrink or vanish—this is testable.","Because the supported-region inclusion condition compares only support sets, the paper's definition of 'denser' ignores reward magnitudes; two curricula with the same supports but different shaping strengths could in principle behave very differently.","If the sparse phase truly builds robust initial parameters, S2D should transfer as a pretraining schedule: an agent that explored under sparse rewards should fine-tune faster on a new goal distribution than one trained only on dense rewards from the start."],"forward_implications":["S2D is a broadly applicable, algorithm-agnostic reward schedule: it improved results across SAC, PPO, DQN, and A3C in the paper's experiments.","The optimal transition point is early—roughly the first quarter to third of training—aligning with the toddler critical-period analogy and giving practitioners a simple default rule.","When generalization to unseen layouts or textures is the bottleneck, starting with a sparse phase before dense shaping may make the learned policy less brittle.","S2D does not require changing the environment, the architecture, or the RL algorithm, so it can be layered on existing goal-conditioned pipelines.","The sharper RNN feature-convergence and more stable action frequencies after the transition are consistent with the idea that the sparse phase builds a reusable internal map, not just a better reward signal."],"supporting_citations":[{"why":"Potential-based reward shaping (Ng et al.): supplies the theoretical guarantee that adding the potential-based reward preserves the optimal policy, which justifies defining the S2D curriculum as optimality-preserving.","marker":"[43]"},{"why":"Loss landscape visualization method (Li et al.): the perturbation-based plotting approach the Cross-Density Visualizer is built on.","marker":"[35]"},{"why":"Sharpness-aware minimization (Foret et al.): supplies the sharpness metric used to quantify wide versus narrow minima.","marker":"[13]"},{"why":"Keskar et al.'s wide-minima generalization argument: the basis for linking lower sharpness to better generalization.","marker":"[28]"},{"why":"Tolman's maze experiments: the latent-learning evidence being reinterpreted as support for early sparse-reward free exploration.","marker":"[56]"},{"why":"SAC: the main RL algorithm used in the state-based experiments.","marker":"[19]"},{"why":"A3C: the algorithm used for the ViZDoom egocentric navigation experiments.","marker":"[40]"},{"why":"The authors' earlier work that this study builds on, establishing the toddler-inspired reward transition idea.","marker":"[47]"}],"fun_headline_variants":["Toddler-inspired reward shift boosts RL sample efficiency","Sparse-to-dense rewards smooth loss landscape and widen minima","Early free exploration then dense shaping speeds RL learning","S2D reward curriculum enhances RL generalization without fixed densities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That reheating the replay buffer with rewards collected under two different reward functions—mostly-zero sparse rewards early, dense shaped rewards later—still gives the optimizer a consistent learning signal, so the S2D advantage comes from the curriculum rather than from accidentally mixing reward scales.","fun_headline_variants_meta":{"raw":{"variants":["Toddler-inspired reward shift boosts RL sample efficiency","Sparse-to-dense rewards smooth loss landscape and widen minima","Early free exploration then dense shaping speeds RL learning","S2D reward curriculum enhances RL generalization without fixed densities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1894,"prompt_tokens":927,"completion_tokens":967,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":902}},"tokens_in":543,"tokens_out":967,"duration_ms":9047,"temperature":1.0,"reasoning_tokens":902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:32:51.701548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same S2D schedule but relabel every replay-buffer transition with the current stage's reward before each update, or clear the buffer at the transition point. If the S2D advantage over Only Dense shrinks or disappears under relabeling, the mechanism is the stale mixed rewards, not the sparse exploration phase. A second check: if S2D works by forming a cognitive map, its advantage should be measurable as better transfer when both the goal and spawn are randomized at evaluation, and should shrink when goal positions are fixed and visible.","supporting_citations":[{"cited_title":"Policy invariance under reward transformations: Theory and application to reward shaping","cited_arxiv_id":null,"evidence_quote":"Potential-based reward shaping (Ng et al.): supplies the theoretical guarantee that adding the potential-based reward preserves the optimal policy, which justifies defining the S2D curriculum as optimality-preserving."},{"cited_title":"Sharpness-aware minimization for efficiently improving generalization","cited_arxiv_id":null,"evidence_quote":"Sharpness-aware minimization (Foret et al.): supplies the sharpness metric used to quantify wide versus narrow minima."},{"cited_title":"On large-batch training for deep learning: Generalization gap and sharp minima","cited_arxiv_id":null,"evidence_quote":"Keskar et al.'s wide-minima generalization argument: the basis for linking lower sharpness to better generalization."},{"cited_title":"Asynchronous methods for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"A3C: the algorithm used for the ViZDoom egocentric navigation experiments."},{"cited_title":"Unveiling the significance of toddler-inspired reward transition in goal-oriented reinforcement learning","cited_arxiv_id":null,"evidence_quote":"The authors' earlier work that this study builds on, establishing the toddler-inspired reward transition idea."}],"review_version":1}