{"id":"ca9d7787-eb87-4b56-856c-33f2b3989533","arxiv_id":"2507.15287","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"MoE-GUIDE guides RL exploration by rewarding states that a mixture of autoencoders, trained on sparse state-only expert demonstrations, considers similar to expert data.","lead":"This paper introduces MoE-GUIDE, a reinforcement learning method that uses a mixture of autoencoders to turn incomplete, unlabeled expert demonstrations into a shaped exploration reward. Tests on five MuJoCo locomotion tasks suggest it can help agents explore in sparse-reward settings, though its advantage over plain reward learning is uneven in dense environments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness claim depends on oracle-tuned thresholds: ablation shows an order-of-magnitude change in Lmin flips performance from strong to catastrophic, and no selection rule is provided.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the intrinsic reward's utility depends on hand-picked thresholds and decay parameters. My stress-test agrees with this assessment. The ablation in Section 4.2.4 is the strongest evidence: a small absolute change in Lmin (0.006 to 0.03) moves Ant from near-SAC performance to strongly negative extrinsic reward, indicating a narrow good region. The lack of any automatic or validated selection procedure means the central 'robust' claim is not supported by the experiments as reported. I also note the Appendix A.1 proof that state-only intrinsic rewards do not alter optimal policies is incorrect (V_int depends on the state-visitation distribution, so it is not policy-order-preserving), but this is a secondary formal claim; the empirical claim of robustness is what the headline asserts. I agree with the reader's CONDITIONAL verdict: the idea is plausible and the sparse-reward results are encouraging, but the verification of the method's practical robustness requires held-out hyperparameter selection and a principled thresholding rule. I do not change the reader's verdict.","tokens_in":15699,"tokens_out":5166,"duration_ms":62420,"concrete_test":"Re-run the five MuJoCo experiments (or at least Ant and HalfCheetah) with a fixed, environment-agnostic threshold rule: set Lmin to the 5th percentile and Lmax to the 95th percentile of the reconstruction losses computed on the provided demonstration states, keep the same exponential mapping and beta decay schedule across environments, and compare final mean reward against ER-only. If MoE-GUIDE does not beat ER-only in Ant and sparse Walker2d while staying within one standard deviation in HalfCheetah, the robustness claim fails. Alternatively, report the fraction of a log-spaced Lmin x Lmax grid that beats ER-only; a small good region would confirm the tuning sensitivity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that MoE-GUIDE enables robust exploration and strong performance across sparse and dense reward environments with incomplete expert data. The load-bearing condition is that the reconstruction loss of the mixture of autoencoders, after applying thresholds Lmin and Lmax, yields an intrinsic reward landscape that usefully guides exploration. This condition is not established as robust because the thresholds are hand-picked per environment: reported Lmin values range over orders of magnitude (Ant 4e-5, Swimmer 0.01, HalfCheetah 0.1), and Section 4.2.4 shows that changing Lmin from 0.006 to 0.03 in Ant drops final mean extrinsic reward from 4139 to -1812.56, i.e. from competitive to catastrophic. The paper's suggested diagnostic (\"if random trajectories are well represented, Lmin is too large\") is not a quantitative selection rule and still requires a human to inspect reconstruction behavior. Similarly, Section 4.2.3 states that decay rates were chosen using prior knowledge that the HalfCheetah expert was weak, a signal unavailable in the intended setting. Without a principled way to set these parameters, the experiments demonstrate that MoE-GUIDE can work when thresholds are tuned by an oracle, not that it robustly works otherwise. This directly undermines the headline claim of robustness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MoE-GUIDE, a method for providing exploration guidance in reinforcement learning by training a mixture of autoencoder experts on state-only, possibly incomplete or imperfect demonstrations. The reconstruction loss of this mixture is mapped through a thresholded exponential function into an intrinsic reward, which is added to the extrinsic reward of a Soft Actor-Critic agent with an optional decay schedule. Experiments on MuJoCo benchmarks (Swimmer, Hopper, Walker2d, HalfCheetah, Ant) and a gridworld visualization study compare MoE-GUIDE against extrinsic-reward-only training, pretraining, and intrinsic-reward baselines, and include ablations on the number of experts, demonstration sparsity, decay rates, and mapping thresholds. The central claims are that the method enables robust and strong performance in both sparse and dense reward settings and that a state-only intrinsic reward does not alter the set of optimal policies, the latter supported by a purported proof in Appendix A.1.","tokens_in":16002,"tokens_out":4742,"duration_ms":53627,"significance":"If the central claims held, the method would be a practically useful and conceptually clean way to exploit unlabeled, state-only demonstrations for exploration in continuous control, an important and realistic setting. The paper has several strengths: it evaluates across five environments, includes sparse-reward and imperfect-expert variants, reports ablations on key components, and provides detailed hyperparameter tables in the appendix. The experimental gains on Ant, Walker2d, Swimmer, and the sparse-reward variants are notable, and the use of a mixture of autoencoders to handle modality diversity in demonstration data is a sensible and reasonably novel contribution. However, the load-bearing formal claim in Appendix A.1 is incorrect, and the robustness claim in the abstract is undermined by the demonstrated sensitivity of the method to hand-tuned thresholds and decay rates. The manuscript needs revision to correct the theoretical statement, add a principled threshold-selection procedure or substantially weaken the robustness claim, and more carefully separate environments where the method helps from those where it does not.","major_comments":[{"comment":"","section":"Appendix A.1 and Section 3"},{"comment":"","section":"Section 4.2.4, Table 15, and Table 4"},{"comment":"","section":"Section 4.2.3 and Tables 12, 13, 17"}],"minor_comments":[{"comment":"","section":"Equation (6)"},{"comment":"","section":"Equation (6)"},{"comment":"","section":"Table 4"},{"comment":"","section":"Appendix A.2"},{"comment":"","section":"Section 4.1"},{"comment":"","section":"Appendix C"},{"comment":"","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical core: the gains on Ant, Walker2d, Swimmer, and several sparse-reward settings are real and the state-only demonstration setup is practically motivated. However, the formal claim in Appendix A.1 is not just incomplete but false as stated, and the advertised robustness is undercut by the paper's own ablations. A major revision that removes or corrects the invariance claim, adds an explicit threshold-selection procedure or clearly restricts the claims to settings where thresholds can be tuned, and rebalances the abstract/conclusion to match the actual results would make the paper publishable. If the authors cannot provide a principled threshold-selection rule, the robustness claims should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, MoE-GUIDE is a plausible combination: mixture-of-autoencoder reconstruction loss turned into a thresholded intrinsic reward for guiding SAC with incomplete, state-only demonstrations. The experiments are reasonably thorough, and the sparse-reward results in Ant and Walker2d look genuinely useful. Second, the paper's two biggest claims do not survive contact with the paper's own content. The Appendix A proof that state-only intrinsic rewards preserve optimal policies is invalid; it assumes V_int is policy-independent, which it isn't. Their own illustrative example in A.2 shows an agent getting stuck on intrinsic rewards. And the abstract's 'robust exploration and strong performance in both sparse and dense reward environments' is contradicted by their dense HalfCheetah result (9867 vs 11217 for ER-only).\n\nWhat the paper does well: the ablations are honest. They report threshold sensitivity (0.006 works, 0.03 collapses), decay-rate sensitivity, and they admit in the discussion that dense HalfCheetah leaves little room for intrinsic shaping. Hyperparameter tables are complete. The gridworld visualizations are a nice way to see how the loss landscape changes with the number of experts.\n\nThe soft spots, in order: (1) The invariance proof is a load-bearing claim. Without it, the method is just shaping with a state-dependent reward, which can change the optimal policy. They need to either prove a special case or retract the claim. (2) The robustness headline depends on hand-picked thresholds and decay rates. Lmin spans 4e-5 to 0.1 across environments, and the ablation shows an order-of-magnitude change flips performance from strong to catastrophic. The proposed diagnostic (check if random states are reconstructed well) is qualitative, not a selection rule. And Section 4.2.3 admits decay rates were chosen because they knew the HalfCheetah expert was weak. That knowledge isn't available in the intended use case. (3) No code or data released, which makes the threshold tuning hard to reproduce.\n\nWho should read this: anyone working on demonstration-guided exploration in continuous control, especially with partial state-only data. It deserves a serious referee round. The empirical core is worth engaging, but the theory needs correcting and the claims need to be scaled back to what the evidence supports.","headline":"Solid empirical exploration method with two load-bearing caveats: the invariance proof is wrong and the robustness claim rests on per-environment hand-tuned thresholds.","tokens_in":16500,"tokens_out":3276,"would_cite":false,"duration_ms":34337,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A gated mixture of autoencoders trained on state-only, incomplete expert demonstrations produces a shaped intrinsic reward that guides reinforcement learning exploration without altering the optimal policy.","keywords":["reinforcement learning","intrinsic motivation","exploration","mixture of experts","autoencoder","imitation from observation","continuous control","Soft Actor-Critic"],"falsifier":"Run MoE-GUIDE on a new continuous-control task with demonstrations sampled every five steps, sweep $L_{\\min}$ across the values used in the Ant ablation, and compare final extrinsic reward against the extrinsic-only baseline: if no threshold beats the baseline, or if the best threshold is only discoverable by knowing the expert's strength in advance, the claim of robust guidance from incomplete data would fail.","tokens_in":15492,"feed_emoji":"🤖","tokens_out":11517,"duration_ms":115166,"temperature":0.7,"pith_summary":"MoE-GUIDE is a method for turning expert demonstrations that contain no actions, no next states, and possibly large gaps into an exploration signal for reinforcement learning. The authors train a gated mixture of bottleneck autoencoders on the states an expert visited, then map the mixture's reconstruction loss through a thresholded, exponentially decaying function to a bounded intrinsic reward. This reward is added to Soft Actor-Critic's extrinsic reward, steering the agent toward expert-like regions while the environment reward remains available. The authors argue that because the intrinsic reward depends only on the state, it can be decayed to zero without changing the optimal policy for the original task, and their experiments on five MuJoCo locomotion benchmarks show gains over extrinsic-reward-only and pretraining baselines, especially when rewards are sparse or demonstrations are imperfect.","feed_headline":"Autoencoder experts turn sparse demos into an exploration reward","feed_subtitle":"On MuJoCo tasks, a state-only similarity reward beats extrinsic-only baselines with gappy, imperfect demos.","key_machinery":"The carrying mechanism is the mixture of autoencoder experts together with the loss-to-reward mapping. A gating network assigns a weight to each autoencoder expert for an input state, the reconstruction is the weighted sum of expert outputs, and the reconstruction loss $L$ is converted to an intrinsic reward by $g(L)=\\kappa\\,\\mathrm{clip}(f((L-L_{\\min})/(L_{\\max}-L_{\\min})),0,1)$ with $f(x)=e^{-sx}$. States with loss below $L_{\\min}$ receive maximal reward, states above $L_{\\max}$ receive none, and the exponential steepness $s$ controls the falloff. The resulting state-only bonus is added to the Soft Actor-Critic Q-update with a strength $\\beta$ that can be decayed, and since it stays constant during training it can be computed once and stored in the replay buffer.","core_discovery":"The paper's central claim is that a reconstruction-loss landscape built from a mixture of autoencoder experts is a usable source of exploration guidance when demonstrations are unlabeled and incomplete. Each expert is a bottleneck autoencoder that specializes in part of the expert's behavior, and a gating network weights their reconstructions per state; the resulting loss at a state is converted by a mapping into a reward of 1 for expert-like states, 0 for far-from-expert states, and a smooth exponential falloff in between. The agent is guided toward states the expert visited without ever being shown explicit expert paths, actions, or transitions. In the paper's experiments this guidance improves on or matches baselines in most of the five MuJoCo domains, and it provides the largest gains in sparse partially observable variants where the position coordinate is hidden from both demonstrations and observations.","pith_inferences":["Applying the same thresholded loss-to-reward map to density estimators, ICM, or RND — models the paper names as future alternatives — would isolate whether the mixture of autoencoders or the shaping function is the main source of the benefit.","An automatic threshold-selection rule based on the reconstruction-loss distributions of held-out expert states versus random states could remove the main human-tuning step; the paper's sensitivity analysis suggests performance hinges on these thresholds.","Combining the state-similarity bonus with an episodic novelty bonus for revisited states, as the paper's gridworld already does with one-time rewards, could prevent the agent from lingering in expert-like regions and is a natural next test.","The policy-invariance claim is asymptotic in practice: during training the intrinsic reward deliberately biases exploration, so the practical contribution is faster discovery of good states, not a guarantee about the policy during training."],"forward_implications":["An agent can learn from demonstration data that contains only states with gaps, so data collection no longer requires recording actions, next states, or dense trajectories.","The mapping function gives practitioners explicit dials — $\\kappa$, $s$, $L_{\\min}$, $L_{\\max}$, and the decay $\\beta$ — to control how strongly and how long the agent is pulled toward expert-like states.","Guidance is most valuable where the extrinsic reward is sparse or partial; in dense-reward domains such as HalfCheetah, the paper finds the extrinsic reward alone leaves little room for improvement.","Because the intrinsic reward is computed once per state and stored in the replay buffer, the method adds little computational overhead to Soft Actor-Critic training.","If the intrinsic reward is decayed to zero over training, the paper argues the final policy can still be optimal for the original environment reward."],"supporting_citations":[{"why":"Supplies the reward-shaping invariance result used to argue the state-only intrinsic reward does not change the optimal policy.","marker":"Ng et al. (1999)"},{"why":"Defines the curiosity-driven intrinsic reward baseline that MoE-GUIDE builds on and compares against.","marker":"Pathak et al. (2017)"},{"why":"Defines the RND novelty baseline and the broader intrinsic-reward paradigm.","marker":"Burda et al. (2018)"},{"why":"Earlier demonstration that autoencoder reconstruction loss can serve as an intrinsic reward, the seed of the similarity model.","marker":"Klissarov et al. (2019)"},{"why":"The Soft Actor-Critic algorithm that MoE-GUIDE wraps with the intrinsic bonus.","marker":"Haarnoja et al. (2018a;b)"},{"why":"Representative demonstration-based RL that assumes complete trajectories, which MoE-GUIDE relaxes to state-only, gappy data.","marker":"Rajeswaran et al. (2017)"},{"why":"Source of the CILO expert demonstration datasets used for the perfect and imperfect expert experiments.","marker":"Gavenski et al. (2024)"}],"fun_headline_variants":["Mixture of autoencoders shapes rewards from incomplete demos","Autoencoder mixture turns partial demos into exploration signals","Sparse, imperfect demos become exploration rewards via autoencoder experts","Incomplete demos guide RL via autoencoder reconstruction loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the autoencoders' reconstruction loss reliably separates expert-like states from all other states, and only if a human picks the lower and upper loss thresholds well; the paper's own ablation shows a badly chosen threshold can collapse extrinsic reward.","fun_headline_variants_meta":{"raw":{"variants":["Mixture of autoencoders shapes rewards from incomplete demos","Autoencoder mixture turns partial demos into exploration signals","Sparse, imperfect demos become exploration rewards via autoencoder experts","Incomplete demos guide RL via autoencoder reconstruction loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3830,"prompt_tokens":958,"completion_tokens":2872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2803}},"tokens_in":574,"tokens_out":2872,"duration_ms":23635,"temperature":1.0,"reasoning_tokens":2803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:35:50.675157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MoE-GUIDE on a new continuous-control task with demonstrations sampled every five steps, sweep $L_{\\min}$ across the values used in the Ant ablation, and compare final extrinsic reward against the extrinsic-only baseline: if no threshold beats the baseline, or if the best threshold is only discoverable by knowing the expert's strength in advance, the claim of robust guidance from incomplete data would fail.","supporting_citations":[{"cited_title":"Explorative imitation learning: A path signature approach for continuous environments","cited_arxiv_id":null,"evidence_quote":"Source of the CILO expert demonstration datasets used for the perfect and imperfect expert experiments."}],"review_version":1}