{"id":"80afaccd-c4ef-426f-bd17-2fad8454b490","arxiv_id":"2506.16565","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Removing unfamiliar visual distractors from the input observation before world-model rollout, then compositing them back into predicted frames, makes action verification robust and raises real-robot pick-and-place success from 20% to 70%.","lead":"This paper introduces ReOI, a test-time method that removes novel visual distractors from a robot's camera image before asking a world model to imagine future outcomes, then reinserts the distractors into the predicted frames. It reports that this intervention raises task success in a real robot pick-and-place verification setup from 20% to 70% when unfamiliar objects appear.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth-aware compositing of reinserted distractors renders collisions as occlusion, so the safety claim in §IV-B needs a direct rejection-rate test.","rationale":"The reader's weakest assumption prioritized detection (distractors degrade by frame 5), with post-hoc compositing listed secondarily. I find the compositing safety premise more load-bearing because it is a geometric consequence of the method itself: removing the obstacle from the dynamics means the trajectory cannot respond to it, and reinserting it with depth ordering necessarily converts intersection into occlusion. The paper's response to this concern—that unsafe trajectories are automatically rejected—is an assertion rather than an analysis, and it is exactly the kind of self-identified limitation that should be weighed heavily. A direct rejection-rate test would settle the question without needing more trials first. Because the claim could be supported by one experiment, I keep the reader's CONDITIONAL verdict rather than escalating; the condition should be this experiment. My agreement is partial because the reader did not identify the occlusion mechanism or propose this test.","tokens_in":12429,"tokens_out":9231,"duration_ms":97276,"concrete_test":"Run a controlled rejection experiment on the same real robot. Sample N=20 action plans whose waypoints are verified (by execution or motion-planning ground truth) to intersect the 3D bounding volume of a novel distractor, plus 20 plans that safely pass in front of or behind it. For each plan, run ReOI's reimagination and depth-aware compositing and feed the composited four-frame grids to the same GPT-4o verifier with the task prompt. Measure the rejection rate on true-collision plans and the false-rejection rate on safe plans. If the true-collision rejection rate is not near 1, or is not significantly higher than the safe-plan rejection rate, the §IV-B automatic-rejection claim fails and the reported 0.10 collision rate is not explained by the proposed mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ReOI's safety argument rests on post-hoc reinsertion of removed distractors into world-model rollouts (§IV-B). Because the distractor is absent during rollout, a plan that actually collides yields a robot trajectory that passes through the obstacle's volume. The paper claims such violations are automatically rejected since the robot must avoid contact, but the compositing procedure renders layers in back-to-front depth order: when the gripper trajectory intersects the distractor volume, the output shows one layer occluding the other, not interpenetration. That appearance is identical to a safe pass in front of or behind the obstacle. The VLM verifier therefore sees an occluded gripper or an occluded distractor, not a collision. The assertion in §IV-B that these violations are automatically rejected is not supported by the rendering pipeline and is in tension with it. This directly threatens the central claim that ReOI maintains a collision rate of 0.10 while rejecting unsafe plans, because the composited frames may not contain the evidence the verifier needs. Detection accuracy and trial count are secondary; this is a potential internal inconsistency in the safety mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reimagination with Observation Intervention (ReOI), a test-time, plug-in method for making visual world-model predictions robust to novel visual distractors in robot action-plan verification. Given a current observation, ReOI rolls out a world model on an in-distribution action plan, uses a GPT-4o prompt to identify objects that distort or disappear across the rollout, segments and inpaints those distractors, re-runs the world model on the modified observation, and then reinserts the distractors into the predicted frames via depth-aware compositing. The reimagined rollouts are scored by a VLM verifier that selects or rejects action plans. The paper evaluates ReOI with DINO-WM on a real Fanuc robot pick-and-place task with novel distractors, reporting improved SSIM/LPIPS prediction quality and a task success rate of 0.70 versus 0.20 for unmodified DINO-WM and 0.00 for a TrustRegion baseline, with collision rate 0.10.","tokens_in":12616,"tokens_out":6176,"duration_ms":60063,"significance":"The problem addressed is timely and important: visual world models are brittle under novel distractors, and existing mitigation strategies are training-time and task-specific. The proposed test-time intervention is modular, does not require retraining the dynamics model, and the qualitative results (Figures 1, 4, 5) are compelling. The paper also provides a detailed experimental setup, including hyperparameters and VLM prompts, which supports reproducibility. If the central claims hold, ReOI would be a useful contribution to world-model-based policy verification. However, the safety mechanism contains an internal tension, the quantitative evidence is statistically weak, and the evaluation protocol includes decoder fine-tuning on the test environment, so the current manuscript does not yet establish the claimed robustness.","major_comments":[{"comment":"The safety argument is internally inconsistent with the depth-aware compositing procedure. The paper states that trajectories where the gripper appears to pass through a reinserted distractor are 'automatically rejected during the verification process,' but the described back-to-front compositing renders interpenetration as occlusion: a gripper that actually intersects the distractor volume will appear either in front of or behind the distractor, visually indistinguishable from a safe pass. The VLM verifier therefore has no visual evidence of collision in the composited frame. This directly threatens the central claim that ReOI maintains a low collision rate while rejecting unsafe plans. Please supply a direct experiment measuring the verifier's rejection rate on rollouts known to collide with the reinserted distractor, or revise the verification procedure so that collision evidence is preserved in the rendered frames.","section":"IV-B"},{"comment":"The evaluation protocol appears to include a form of test-time adaptation that undermines the claim of a purely plug-in, training-free intervention. Appendix B states that the DINOv2 feature decoder is fine-tuned on 80 trajectories collected in the testing environment that 'intentionally includes various visual distractors not present during the initial training phase.' This gives the decoder access to the test-environment distractor distribution. Please clarify whether the DINO-WM and TrustRegion baselines use the same fine-tuned decoder; if they do not, the comparison is unfair, and if they do, the paper should qualify the claim that ReOI requires no knowledge of distractors at training time. In either case, report results without this decoder fine-tuning to isolate the contribution of ReOI.","section":"V and Appendix B"},{"comment":"The central quantitative claim rests on 10 trials with no error bars, confidence intervals, or statistical tests. With 7/10 versus 2/10 successes, the difference between ReOI and DINO-WM is not significant at the 5% level (two-sided Fisher's exact test p ≈ 0.07). Please report per-trial results, confidence intervals, and a significance test, or increase the number of trials. Table I also reports SSIM and LPIPS without sample counts or variance; please provide these so the reader can assess the stability of the reported improvements.","section":"Table II and Table I"},{"comment":"The detection step is load-bearing but not adequately evaluated. Section IV-A and Appendix A assume that novel distractors degrade within the first five frames of the world-model rollout, but the paper does not report a measured distractor-detection accuracy or failure rate, despite the main text promising such an evaluation in Appendix B. If a distractor is missed or degrades only after frame 5, no inpainting occurs and ReOI collapses to the baseline. Additionally, because GPT-4o is used both to detect distractors and to verify plans, correlated VLM errors could inflate the reported success. Please report detection accuracy, failure cases, sensitivity to the frame-5 choice, and plan-verification accuracy conditioned on detection success versus failure.","section":"Appendix A and Appendix B"}],"minor_comments":[{"comment":"The inpainting model is cited as [2], but reference [2] is Rovi-Aug, which is a data-augmentation method, not an image inpainting model. Please cite the actual inpainting method used.","section":"IV-A"},{"comment":"The word 'reimaginated' appears to be a typo; it should be 'reimagined.'","section":"Figure 10 caption"},{"comment":"The prompt text contains the typo 'Wach out' and the phrase 'looks safely' should likely be 'looks safe.'","section":"Figure 12 prompt"},{"comment":"In Equation (1), the notation o_t := o_{t:t+T} and a_t versus a^t is inconsistent; please define the action-plan indexing and horizon T explicitly.","section":"III"},{"comment":"There are spacing issues in the text, such as 'We use120multi-mode teleoperated demonstrations' and 'To this end, we propose Reimagination with Observation Intervention,ReOI,atest-timeandplug-instrategy.' Please proofread for formatting.","section":"V"},{"comment":"The TrustRegion baseline description does not specify the input space in which the r-balls are defined or the distance metric used for observation-action pairs; please add this detail so the baseline is reproducible.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper does not mention code release, and the reliance on proprietary GPT-4o for both detection and verification makes independent reproduction difficult. The editor may wish to ask the authors to release the prompts, the fine-tuned decoder details, and the trial-level data. The decoder fine-tuning on the test environment is a point that should be carefully scrutinized during revision, as it may substantially affect the interpretation of the reported gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe gist of ReOI: when a world-model rollout shows novel objects degrading implausibly, inpaint them out, re-roll the future from the cleaned observation, then paint the objects back in with depth ordering. Applied to DINO-WM on a real pick-and-place, it lifts verification success from 20% to 70% with 10% collisions. That's a real effect on a real bottleneck, and the modular, training-free framing is the right kind of step for the field.\n\nWhat's genuinely new: prior observation-intervention work targeted imitation or VLA policies; this applies the trick to the world model's imagined futures, with a nice detection signal (rollout artifacts) and a plausible reinsertion mechanism. The paper is also reasonably honest about scope: static distractors only, one task, and the decoder fine-tuning in Appendix B is disclosed, though its implications are understated.\n\nThe soft spots are uneven. The biggest one is the safety story around depth-aware compositing. Section IV-B asserts that trajectories where the gripper passes through a reinserted distractor are 'automatically rejected' because the robot must avoid contact. That doesn't follow. Compositing in depth order means an interpenetrating gripper and distractor render as clean occlusion—gripper in front or behind—so the VLM verifier may see exactly what a safe pass looks like. The paper gives no evidence that such violations are caught, and the collision rate in 10 trials doesn't settle it. This needs a direct test or a reworked argument before I'd trust the collision claim.\n\nThe empirical core is otherwise thin: 10 trials per method, no error bars or significance tests; SSIM/LPIPS in Table I have no variance; the decoder is fine-tuned on 80 trajectories from the test environment that include distractors, which muddies the 'test-time' novelty; and detection plus verification both use GPT-4o, so correlated VLM errors are a real possibility. None of these is fatal alone, but together they make the central numbers under-supported as stated.\n\nIf I were handling this, I'd send it to peer review—the idea is worth serious scrutiny—but I'd require the authors to (1) test the compositing rejection issue directly, (2) run more trials or at least report distributions, and (3) clarify what the decoder fine-tuning actually contributes. The paper is for people working on world-model robustness and MPC verification; they'll get value from the method even if the evaluation needs tightening.\n\nRecommendation: engage with it, but don't accept the safety claim on faith.","headline":"ReOI is a sensible modular idea with a real effect, but the depth-compositing safety story has a hole and the evaluation is too thin to take the 0.70 success rate at face value.","tokens_in":13186,"tokens_out":3715,"would_cite":true,"duration_ms":38042,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A test-time intervention detects novel objects that make world-model rollouts degrade, inpaints them, reimagines the future, and pastes them back, raising pick-and-place verification success from 0.20 to 0.70 in the presence of novel…","keywords":["test-time observation intervention","world models","visual model predictive control","visual distractors","action verification","inpainting","depth-aware compositing","robot manipulation"],"falsifier":"Run the same real-robot pick-and-place task with a novel distractor that DINO-WM predicts stably across all rollout frames, such as a visually simple colored block placed in the robot's path, and check whether the vision-language-model verifier using ReOI rejects the colliding plan; if it accepts the plan or shows the gripper passing through the reinserted block without flagging it, the detection premise fails.","tokens_in":12217,"feed_emoji":"🤖","tokens_out":7670,"duration_ms":70054,"temperature":0.7,"pith_summary":"World models let robots preview future frames for a planned action, but an object the model never saw in training can make those previews hallucinate: obstacles vanish, target objects get erased, and the downstream verifier picks a colliding plan. ReOI claims this failure can be fixed entirely at test time, without retraining the world model or the policy. The pipeline lets the world model roll out one in-distribution action, asks a vision-language model to flag objects that distort or disappear within the first five predicted frames, inpaints those objects out of the current observation, reimagines action outcomes from the cleaned input, and then pastes the distractors back in depth order so occlusion is correct. In a real robot pick-and-place task with novel distractors, this lifts the verifier's task success rate to 0.70, from 0.20 for unmodified DINO-WM and 0.00 for a trust-region rejection baseline, while holding collisions at 0.10. If correct, ReOI offers a plug-in robustness layer for model-based visual planning in open-world scenes.","feed_headline":"Clean-input trick lifts robot task success from 20% to 70%","feed_subtitle":"By erasing novel objects before world-model rollout and pasting them back after, robots avoid hallucinated futures.","key_machinery":"The central mechanism is Reimagination with Observation Intervention (ReOI), a three-stage test-time pipeline: detection, where the world model rolls out an in-distribution safety-check action and a vision-language model is shown the first and fifth frames plus a segmentation-mask overlay to name objects that have degraded or disappeared; intervention, where those regions are segmented and inpainted out of the current observation; and reimagination, where the original world model predicts each candidate plan from the modified observation, and the removed distractor layers are composited back in back-to-front depth order using per-layer depth estimates. The load-bearing identity is that physically implausible degradation across a short rollout is a reliable signature of out-of-distribution content; the pipeline never retrains or alters the dynamics model.","core_discovery":"ReOI establishes that world-model-based action verification can be made robust to unseen static distractors without modifying the world model, the policy, or the reward. It detects distractors via the model's own failure mode: novel objects degrade in physically implausible ways over an autoregressive rollout, so comparing the initial frame with the fifth rollout frame and querying a vision-language model identifies which segmented regions correspond to novel distractors. Those regions are inpainted out of the current observation, the modified observation is rerolled through the same world model to produce clean future outcomes, and the removed distractors are reinserted into every predicted frame using depth-aware layer compositing so the verifier sees correct occlusion. The paper reports that this reimagination improves SSIM and LPIPS of predicted outcomes and, in ten real-robot trials with novel distractors, lifts task success from 0.20 (DINO-WM) and 0.00 (TrustRegion) to 0.70 with a collision rate of 0.10.","pith_inferences":["Beyond the paper, the frame-5 detection rule is a design choice rather than a necessity: if slower-degrading distractors appear in longer rollouts, the same pipeline should work with a later reference frame, so the method can adapt to models with different autoregressive horizons.","Beyond the paper, because ReOI produces cleaner imagined rollouts generally, it should transfer beyond verification to model-predictive control loops and data augmentation for policy learning, although the paper only tests verification.","Beyond the paper, the post-hoc compositing assumes distractors are static and non-interactive: if a novel object can be pushed or grasped, reinserting it after the rollout would misrepresent physical interaction, so an extension would need to couple distractor dynamics into the rollout.","Beyond the paper, a vision-language-model-free detector, such as a pixel-level temporal-inconsistency measure between two rollout frames, would make ReOI cheaper and less dependent on a specific model; the paper's prompting approach could be replaced by a purely geometric consistency check."],"forward_implications":["A robot can verify action plans against novel static objects it has never seen, without retraining the world model or the policy, by cleaning the input observation at deployment time.","Removing novel distractors before rollout prevents the hallucination cascade that corrupts in-distribution components: the robot and target objects are predicted more faithfully (SSIM increases from 0.63 to 0.79, LPIPS drops from 0.14 to 0.09 on distractor-free components).","Because distractors are reinserted with correct depth ordering, the verifier can still see and reject collision-prone plans, keeping the collision rate at 0.10 while improving success over the conservative TrustRegion baseline, which succeeds 0.00.","The intervention is plug-in: it wraps any world model and any vision-language-model verifier, so the gains do not require changing pretrained weights or collecting new demonstrations."],"supporting_citations":[{"why":"Supplies DINO-WM, the base world model whose latent-space action-conditioned predictions ReOI wraps at test time.","marker":"[27]"},{"why":"Supplies the Diffusion Policy action sampler that proposes candidate action plans for verification.","marker":"[3]"},{"why":"Provides Grounded-SAM2 segmentation used both to localize distractor patches for inpainting and to split predicted frames into layers for depth-aware compositing.","marker":"[15]"},{"why":"Provides the inpainting model that removes identified distractors from the current observation.","marker":"[2]"},{"why":"Defines the TrustRegion baseline that rejects out-of-distribution observations and is the comparison for system-level success and collision rates.","marker":"[11]"},{"why":"Provides the prior observation-intervention idea for vision-language-action policies that ReOI adapts from model-free robustness to world-model verification.","marker":"[8]"},{"why":"Supplies the DINOv2 pretrained features that the DINO-WM encoder operates in and that the fine-tuned decoder maps back to images.","marker":"[13]"}],"fun_headline_variants":["Erase distractors, reimagine clean futures, lift robot success to 70%","World models predict better when you blur out new objects and paste them back","Clean-input reimagination lifts robot success from 20% to 70%","Trick: remove novel visual distractors from world model inputs, reinsert after"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on the assumption that every safety-relevant novel distractor will visibly warp, distort, or disappear within the first five frames of the world-model rollout, so a vision-language model comparing the first and fifth frames can flag it; if an obstacle stays visually plausible in prediction but still blocks the robot, ReOI will not intervene and degenerates to the brittle baseline.","fun_headline_variants_meta":{"raw":{"variants":["Erase distractors, reimagine clean futures, lift robot success to 70%","World models predict better when you blur out new objects and paste them back","Clean-input reimagination lifts robot success from 20% to 70%","Trick: remove novel visual distractors from world model inputs, reinsert after"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000924,"raw_usage":{"total_tokens":4012,"prompt_tokens":1047,"completion_tokens":2965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2878}},"tokens_in":663,"tokens_out":2965,"duration_ms":21980,"temperature":1.0,"reasoning_tokens":2878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:23:49.623765+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same real-robot pick-and-place task with a novel distractor that DINO-WM predicts stably across all rollout frames, such as a visually simple colored block placed in the robot's path, and check whether the vision-language-model verifier using ReOI rejects the colliding plan; if it accepts the plan or shows the gripper passing through the reinserted block without flagging it, the detection premise fails.","supporting_citations":[],"review_version":2}