{"id":"53ab4900-0d1b-4c28-a748-24954d95ceaf","arxiv_id":"2505.01396","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Injecting annealed noise into the observation-encoder latent of a diffusion policy during inference increases rollout diversity and, combined with success- and value-based data selection, improves imitation-learned robot policies through self-collected data.","lead":"SIME adds noise to the internal reasoning variables of a robot's diffusion policy during rollout, which makes the robot try different ways to solve manipulation tasks and produces more varied practice data. The paper reports that selecting the most valuable trials and segments from this data gives roughly 16% average improvement on RoboMimic tasks and a large gain in a real cup-stacking test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that latent perturbation creates distinct on-task modes is unmeasured; the reported gains could stem from stochastic failures plus selection, and the main comparison does not control selected-data volume.","rationale":"I read the paper as claiming a complete self-improvement loop: latent-space perturbation creates multimodal interaction data, selection extracts valuable subsets, and fine-tuning on those subsets improves a diffusion policy beyond its initial demonstrations. The simulation results are internally consistent, with four-seed means and standard deviations, and the selection ablations in Table III support the value of filtering. The paper also provides code and a qualitative real-world demonstration, which are real evidence. The weakest point is the link between the latent perturbation and actual behavioral multimodality. The success-rate histogram is evidence of increased trial-to-trial stochasticity, and the scenario-level success improvement (32% to 21% zero-success states) shows that exploration sometimes turns failures into successes. But stochastic action noise could also produce such a histogram; the paper's own ablation shows diffusion randomness (eta=1.0) yields little improvement, which argues against trivial noise, yet the amplitude and interaction of the latent noise are not quantified, and no mode-level metric is reported. The main comparison also does not control how many selected trajectories each collector produces, so part of the 16.1% average improvement could be a data-scale effect; Section V-A acknowledges data scales differ and only training iterations are fixed. This matches the reader's weakest assumption, so I do not move the verdict. The paper should remain CONDITIONAL: the proposed check would determine whether the modal-level-explanation is supported or whether the gains reduce to stochasticity plus selection.","tokens_in":10607,"tokens_out":7481,"duration_ms":86550,"concrete_test":"On Can-20, collect 1,000-state x10-attempt rollouts from the base policy with and without modal-level exploration, plus action-level-noise and eta=1.0 controls. For successful rollouts originating from the same initial state, compute a behavioral descriptor (e.g., end-effector or object pose sequence), cluster across trials, and report the effective number of modes or pairwise behavioral distance, restricted to successes. Then subsample the modal-exploration selected dataset to match the baseline selected-demo count and rerun the Table I comparison. If modal-exploration success clusters into a single dominant mode and the equal-data performance gap disappears, the central claim shifts from modal-level exploration to stochasticity plus selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's mechanism is that annealed latent noise in Eqs. 5-8 moves the policy between behavioral modes during inference, producing diverse on-task trajectories that inter/intra-demo selection can exploit. The quantitative support for this is Fig. 3, which plots the distribution of per-state success rates over 10 attempts on Can. A wider success-rate distribution (e.g., some states with 3/10 successes) is also exactly what a unimodal policy with added stochasticity would produce: it does not establish that the successful attempts use different strategies, nor that the failures are meaningful exploration rather than random errors. The trajectory visualizations in Fig. 4 and Fig. 6 are qualitative, show only one or two examples, and do not count modes. If the perturbation is just stronger action-level randomization, the improvements in Tables I-II could be driven by inter-demo selection (Table III shows selection alone contributes +10.8 percentage points) harvesting lucky successes, with no modal-level mechanism. The paper states that different data-collection methods yield different data scales and fixes only training iterations, so the main SIME-vs-baseline comparison also leaves open a selected-data-volume confound. This is the load-bearing weak point: the causal story relies on an unmeasured distinction between 'more random failures' and 'new behavioral modes'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SIME, a plug-and-play inference-time method for diffusion policies that injects annealed Gaussian noise into the observation encoder output (Eqs. 5-8) in order to induce what the authors call modal-level exploration. The resulting self-collected trajectories are filtered by inter-demo selection (keep successful trials from states with low success rate) and intra-demo selection (value-weighted regression, Eq. 9), then combined with the original demonstrations to fine-tune the policy. The method is evaluated on five RoboMimic tasks in both state and image space, with four seeds per experiment, and on a real-world cup-stacking task. The main empirical claim is that one round of SIME self-improvement yields on average about a 16.1% absolute success-rate gain, roughly twice the gain of the baseline self-improvement procedure, and that further rounds continue to improve performance.","tokens_in":11022,"tokens_out":6564,"duration_ms":69208,"significance":"If the central mechanism holds, SIME is a simple and attractive recipe for improving imitation-learning policies without additional human data: it does not modify the training objective, it is applicable to any diffusion policy with an encoder, and the accompanying code is released. The paper's strengths include multi-seed simulation results with reported standard deviations, ablation and selection tables that separately support the contribution of data selection, and a multi-round experiment showing continued gains. The main weakness is that the hypothesized modal-level exploration mechanism is only supported indirectly; the evidence does not yet distinguish genuinely new behavioral modes from a unimodal stochastic policy, and the headline comparison between SIME and the baseline does not control the amount of selected training data.","major_comments":[{"comment":"The diversity analysis does not establish that latent perturbation creates distinct behavioral modes. A wider distribution of per-state success rates over repeated trials is exactly what a unimodal policy with added stochasticity would produce; the statement that 56% of scenarios show partial success 'due to the multi-modal attempts of the policy' is an unsupported causal attribution. The trajectory visualizations in Fig. 4 and Fig. 6 are qualitative and show only one or two examples. Please provide a quantitative mode-level measure, for example by clustering successful trajectories by observed behavior and reporting the number of distinct clusters with and without SIME, or explicitly soften the claim to 'stochastic exploration' if such a measure is not available.","section":"Sec. V-B, Fig. 3"},{"comment":"The comparison between the baseline and SIME does not control the number of selected demonstrations. The statement in Sec. V-A2 that 'we fix the number of training iterations to ensure a fair comparison' controls compute, not data volume; because SIME collects more diverse data and both methods apply inter-demo selection, the selected dataset sizes very likely differ, yet no demo counts are reported in Tables I-II. Since Table III shows that at a fixed size of 40 demos selection alone contributes +10.8 percentage points (Random vs. SR<0.5), the differences in Tables I-II (e.g., +8.6% in vision space) may in part reflect a data-volume confound. Please report the number of selected demonstrations for both methods in the main comparison, or subsample to equal counts as in Table III.","section":"Sec. V-C, Tables I-II and Sec. V-A2"},{"comment":"The central exploration mechanism is under-specified: the noise scale sigma in Eq. (5) and the annealing thresholds kappa1 and kappa2 in Eq. (8) are never given numerical values or a sensitivity analysis, and the inter-demo threshold theta is fixed at 0.5 with no justification. Because the magnitude of the latent perturbation determines how strongly the policy deviates from its learned behavior, the method as described is not reproducible without these values. Please report the hyperparameters used in all experiments and include a sensitivity study (at least for sigma and theta).","section":"Sec. IV-B, Eqs. (5)-(8)"},{"comment":"The real-world component of the evaluation is a single run with no variance: 10 initial states and 5 attempts each result in 50 trials, and the reported 117.6% improvement is presented without uncertainty. In addition, the automated intra-demo selection via IQL is only sketched; Eq. (9) uses V_t and beta, but the paper does not define how V_t is computed, on which data the IQL value function is trained, or what beta value is used. Please either report multiple real-world runs with variance or clearly label this as a preliminary demonstration, and provide the missing details for the intra-demo selection procedure.","section":"Sec. IV-C, Eq. (9) and Sec. V-F, Table V"}],"minor_comments":[{"comment":"There are several typos and rendering artifacts, including 'S elf' and 'M odal-level' in the title, '10 trails' in Sec. V-B, and missing symbol labels in the legend of Fig. 3 ('DP( =0.0)' instead of 'DP(η=0.0)').","section":"Throughout"},{"comment":"The phrase 'perceptive perturbation' is likely intended to be 'perceptual perturbation'; please rephrase for clarity.","section":"Sec. IV-B"},{"comment":"The table reports Demo Nums for different selection strategies, but it is not stated whether the four seeds are aggregated or whether the numbers correspond to one seed; please clarify.","section":"Sec. V-D, Table III"},{"comment":"The related-work discussion would benefit from a clearer statement of how SIME differs from the state-novelty filtering of [27]: both methods use selection, but SIME adds an exploration mechanism; the current text says this but could be more explicit about the novelty claim.","section":"Sec. II-D"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about modal-level exploration lands: the current quantitative evidence does not distinguish new behavioral modes from stochastic failure modes, and the main comparison has a data-volume confound that Table III itself shows to be material. These issues are fixable with additional analysis and reporting, so I recommend major revision rather than rejection. The paper is well within the scope of the journal and the empirical gains are interesting even under the weaker interpretation of stochastic exploration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading and probably worth refereeing. What is actually new: it takes condition-annealed latent perturbation from CADS, applies it to robot diffusion policies at inference, and couples it with two selection stages (keep successful trials in low-success states, weight segments by value increments). The simulation results are carefully reported for four seeds, the ablation table supports that exploration plus selection beats fine-tuning on self-collected data alone, and the multi-round curve is a nice touch. The real-world demonstration, though limited, is honest about being a single task with a single run.\n\nWhere I agree with the stress-test note: the central mechanism is not directly measured. Fig. 3 shows a wider per-state success-rate distribution, but that is exactly what you'd get from action-level stochasticity. Fig. 4 and Fig. 6 are qualitative. There is no count of distinct behavioral modes, no metric for 'mode diversity' as opposed to variance. So the paper's headline claim—that modal-level exploration creates genuinely new on-task strategies—rests on an inference rather than a measurement. That is the load-bearing soft spot. I would not call it fatal: the selection results are strong enough that the pipeline works even if the mechanism turns out to be 'stochastic exploration plus filtering.' But the title and abstract oversell the modal-level story.\n\nSecond soft spot, also real: the main SIME-vs-baseline comparison fixes training iterations but not selected-data volume. Table III shows selection alone contributes most of the gain (10.8 of 11.6 percentage points), and random selection at matched volume nearly kills the improvement. So the comparison in Tables I-II may partly reflect the exploration method simply generating more usable successful data, not better modes. That confound is not addressed.\n\nMinor issues: real-world table has no variance, hyperparameters (sigma, kappa1, kappa2, beta, IQL details) are underspecified, and there is no empirical comparison to [27]'s novelty filtering, which is the closest prior method. All fixable.\n\nI disagree with the reader on one point: the circularity burden is low, and the paper does not define a target quantity by its own equations, so I don't see a circularity problem. The selection threshold theta=0.5 is a heuristic, but it is evaluated in Table III, not hidden.\n\nWho is this for: anyone working on autonomous data collection, self-improvement for imitation policies, or practical robot learning with limited demonstrations. It is a solid empirical contribution with a clear, reproducible setup (code and scripts promised), but the theoretical novelty is modest. A serious referee should engage with it, mainly to push the authors to either measure modes directly or soften the claim. I would accept it for peer review, and I'd ask for the missing measurements and ablations before publication.","headline":"Solid, well-run self-improvement pipeline for imitation policies; the 'modal-level' mechanism is plausibly real but the paper's own evidence does not yet distinguish new modes from added stochasticity plus selection.","tokens_in":11420,"tokens_out":703,"would_cite":true,"duration_ms":9296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The blocker to robot self-improvement is data diversity, not volume; annealed latent noise creates that diversity, and one retraining round on the policy's own interactions raises success by about 16.1 percent with no new demonstrations.","keywords":["policy self-improvement","modal-level exploration","diffusion policy","imitation learning","data selection","robot manipulation","latent-space noise","annealed exploration"],"falsifier":"Run the same pipeline with noise injected into a fixed random projection of the observation encoding, matched in variance and annealing; if the success-rate histogram and the self-improvement gain are indistinguishable from SIME's, the specific encoder-level locus is not the active ingredient. A complementary check is to count distinct successful trajectory modes per initial state under modal-level versus action-level noise; if the number of modes is unchanged and only failure variance grows, the diversity claim is refuted.","tokens_in":10399,"feed_emoji":"🤖","tokens_out":11242,"duration_ms":94441,"temperature":0.7,"pith_summary":"Robot policies trained from a handful of human demonstrations often repeat one behavior when left to practice alone, so the data they collect is redundant and fine-tuning on it does not help. The paper claims that the right place to explore is the policy's reasoning space, meaning the latent output of the observation encoder, rather than the action space. Adding annealed Gaussian noise there makes a diffusion policy try distinct solution modes at inference, and filtering the resulting interactions keeps successful portions of hard states and weights segments by predicted value gains. On five simulation tasks with 10-40 demonstrations, one round of this self-improvement raises average success by roughly 16.1 percent, and a real-world cup-stacking experiment shows a larger gain. If this is right, imitation-learning robots can get better from their own practice without additional human data or reward engineering.","feed_headline":"One practice round with latent noise lifts robot success 16%","feed_subtitle":"Injecting annealed noise into the policy's encoder output, not its actions, makes self-collected data worth learning from.","key_machinery":"The central object is a modal modulation factor $\\gamma(k)$ applied to the observation encoder's output during inference: the encoder output $\\psi(o_t)$ is replaced by $\\psi(o_t) + \\gamma(k)n$, where $n$ is Gaussian noise and $\\gamma(k)$ follows a linear annealing schedule, full noise early in denoising and zero at the end. The paper calls this modal-level exploration because it steers the conditional denoising process into different solution basins while preserving precision in the final steps. Two selection mechanisms then decide what the noise-generated data is worth learning from. Inter-demo selection keeps only the successful segments of trials in scenarios whose success rate is below a threshold, discarding easy scenarios and failed trials. Intra-demo selection weights each segment by $\\exp((V_{t+H}-V_t)/\\beta)$ using value estimates from Implicit Q-Learning, so the corrective portions of long trajectories are learned more strongly. Together they convert raw practice data into a focused training set that, combined with the original demonstrations, improves the policy.","core_discovery":"On the paper's own terms, the discovery is that a robot's ability to improve from its own experience is limited less by the size of the interaction dataset than by its diversity, and diversity is best created at the level of the policy's internal representation. Concretely, a diffusion policy's conditional latent $z_t = \\psi(o_t)$ is perturbed as $z_t = \\psi(o_t) + \\gamma(k)n$ with $n \\sim \\mathcal{N}(0,\\sigma^2 I)$ and an annealed gain $\\gamma(k)$ that fades as denoising proceeds. This produces distinct behavioral modes, such as grasping the same object from different sides, whereas action-level noise or maximum diffusion randomness mostly fails to change the behavior. When the resulting trajectories are filtered to keep successful portions of low-success-rate scenarios and reweighted by predicted value gains, the policy trained on the union of original demonstrations and selected self-collected data beats its initial version on every task reported. The headline number is an average success-rate increase of roughly 16.1 percent after one self-improvement round across state- and image-based variants.","pith_inferences":["A natural test of the mechanism is to measure the number of distinct successful trajectory modes per initial state, for example by clustering trajectories on waypoint geometry; such a count would distinguish genuine multi-modality from high-variance execution, which the paper's success-rate histogram does not do.","If modal-level exploration is the active ingredient, the same latent perturbation should transfer to other conditional generative policies, such as flow-matching or autoregressive action models, since it operates on a generic observation encoder rather than on diffusion-specific components.","The annealed noise profile could be made task-aware: tasks requiring precise contact at the end of the trajectory should keep the noise switched off longer, while tasks needing broad search could anneal earlier; this is a concrete hyperparameter extension the paper does not test.","One could also combine the selected self-collected data with a reward signal and run a value-based or policy-gradient update on top of it, which might extend the one-round gains beyond what weighted imitation alone achieves."],"forward_implications":["After one round of self-improvement, policies trained on only 10 to 40 demonstrations beat their initial versions on every reported task, with an average relative success gain of about 16.1 percent.","Continuing the process for several rounds keeps improving the policy, while the same pipeline without modal-level exploration shows diminishing gains in later rounds.","Action-level noise and maximum diffusion randomness do not produce useful practice data; the locus of exploration, in the encoder latent rather than the action, is what makes the difference.","Keeping only successful segments from low-success-rate scenarios is crucial: replacing inter-demo selection with random selection at the same sample count nearly removes the gain, and value-weighted intra-demo selection adds a further increment.","Because the method changes only inference-time noise and data filtering, it can be applied as a plug-in to an already trained imitation policy without retraining the architecture."],"supporting_citations":[{"why":"Defines the diffusion policy, the base policy class whose inference-time behavior SIME modifies.","marker":"[6]"},{"why":"Supplies the five simulated manipulation tasks and the behavior-cloning baseline that self-improvement must beat.","marker":"[26]"},{"why":"Establishes that naively fine-tuning on self-collected data is ineffective and motivates the filtering strategies.","marker":"[27]"},{"why":"Provides the condition-annealed sampling idea behind the linear annealing schedule of Eq. 8.","marker":"[30]"},{"why":"Implicit Q-Learning provides the value estimates used to automate intra-demo segment weighting.","marker":"[18]"},{"why":"Supplies the advantage-weighted regression objective that Eq. 9's exponentiated value increment adapts.","marker":"[29]"},{"why":"A fast deterministic diffusion sampler is used for inference during data collection, and its randomness parameter is tested as a baseline.","marker":"[34]"},{"why":"Cited evidence that diffusion policy stochasticity alone still fails to produce multi-modal exploration, motivating the modal-level noise.","marker":"[2]"}],"fun_headline_variants":["Latent-space noise beats action noise for robot self-improvement","Perturb policy's perception, not actions, to lift success 16%","Modal-level noise on robot perceptions improves self-training 16%","Adding noise to latent space gives robots 16% better self-improvement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the injected noise makes the robot try genuinely new ways of completing the task, not merely fail more randomly; if most of the added variety is random failure, the gains would come from data selection rather than from modal-level exploration.","fun_headline_variants_meta":{"raw":{"variants":["Latent-space noise beats action noise for robot self-improvement","Perturb policy's perception, not actions, to lift success 16%","Modal-level noise on robot perceptions improves self-training 16%","Adding noise to latent space gives robots 16% better self-improvement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3944,"prompt_tokens":942,"completion_tokens":3002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":2925}},"tokens_in":558,"tokens_out":3002,"duration_ms":20424,"temperature":1.0,"reasoning_tokens":2925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:19:03.878167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline with noise injected into a fixed random projection of the observation encoding, matched in variance and annealing; if the success-rate histogram and the self-improvement gain are indistinguishable from SIME's, the specific encoder-level locus is not the active ingredient. A complementary check is to count distinct successful trajectory modes per initial state under modal-level versus action-level noise; if the number of modes is unchanged and only failure variance grows, the diversity claim is refuted.","supporting_citations":[{"cited_title":"So You Think You Can Scale Up Autonomous Robot Data Collection?","cited_arxiv_id":null,"evidence_quote":"Establishes that naively fine-tuning on self-collected data is ineffective and motivates the filtering strategies."},{"cited_title":"From imitation to refinement–residual rl for precise visual assembly","cited_arxiv_id":null,"evidence_quote":"Cited evidence that diffusion policy stochasticity alone still fails to produce multi-modal exploration, motivating the modal-level noise."}],"review_version":1}