{"id":"c5aa0ef7-2b34-4592-b8d0-65d8fe2e06b0","arxiv_id":"2411.14322","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A rearrangement agent that uses 3D Gaussian Splatting for the goal-state world model and DINOv2 patch features to detect and correct shuffled objects.","lead":"SplatR builds a 3D Gaussian Splat of a room in its goal configuration, then compares live camera views against rendered goal views to spot and fix shuffled objects. It reports improved rearrangement scores over prior methods on the AI2-THOR benchmark, though its success rate remains low.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an untested assumption that 3DGS goal renders are artifact-free at every queried viewpoint; a no-shuffle control is needed to rule out rendering-induced false positives.","rationale":"The reader's weakest assumption (pose drift) is not the most dangerous one because AI2-THOR supplies exact agent poses; the real load-bearing assumption is that the 3DGS render is a faithful, artifact-free image of the goal scene from every viewpoint the agent reaches. The entire downstream pipeline — object node creation, Hungarian matching, and pickup decisions — is driven by patchwise cosine similarity in Eq. (6), which cannot distinguish a rendering artifact from a real change. A no-shuffle control is the cleanest way to falsify or support this assumption: with zero moved objects, any detected change is necessarily a false positive from rendering or alignment. The paper's low Misplaced and the closeness of % Fixed and % Fixed Strict are consistent with few false positives, but those aggregate metrics are not a direct measurement, and Tab. 3 lacks per-episode variability. Because no code or episode list is released, I cannot verify whether the 26-episode sample or the baseline numbers are fully comparable; this reinforces a CONDITIONAL rather than an ACCEPT verdict. I do not think the internal logic of the method is broken, but the central empirical claim needs the control experiment and a sensitivity analysis before it can be taken as established.","tokens_in":11945,"tokens_out":8716,"duration_ms":91133,"concrete_test":"Run a no-shuffle control: for each of the same 26 episodes, execute the full SplatR pipeline but leave the scene in the goal configuration (zero objects shuffled). Count the number of object nodes created by the change-detection module per episode and any manipulation attempts. A correct change detector should produce zero nodes in every episode; any nonzero count quantifies the false-positive rate from rendering and alignment. If the control produces frequent spurious nodes, the reported % Fixed Strict and % Misplaced gains cannot be attributed to genuine change detection, and the SOTA claim would need to be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on patchwise DINOv2 comparison between a live RGB frame and a virtually rendered goal frame from a 3D Gaussian Splat (Secs. 3.4-3.5). For this comparison to be valid, the render must be both geometrically aligned and artifact-free at every viewpoint the agent visits. In AI2-THOR the agent pose is ground truth, so the real risk is not pose drift but rendering quality: the splat is trained only from the walkthrough trajectory, so at novel query viewpoints it can produce blur, floaters, missing geometry, or other artifacts. Any such artifact is indistinguishable, at the patch-feature level, from a displaced object and will be grouped into a spurious object node (Sec. 3.6). The paper's only artifact filter is a CLIP check against 'wall' and 'mirror' (Sec. 3.5), which cannot cover arbitrary render errors. The reported closeness of % Fixed and % Fixed Strict (Tab. 3) is suggestive but does not measure false positives directly. If artifact-induced false positives are non-negligible, both % Fixed Strict and % Misplaced would be corrupted, so the claimed SOTA improvement could be an artifact of render quality rather than of the method's change detection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SplatR, an embodied agent for the experience-goal visual rearrangement task. The agent first explores the goal-state scene, trains a 3D Gaussian Splatting (3DGS) model as a world representation, and then, during an unshuffle phase, renders the goal-state view from a virtual camera aligned with its current pose. Change detection is performed by patchwise dense feature matching with DINOv2 features between the live RGB frame and the rendered goal frame; detected changes are grouped into object nodes, refined with SAM, merged using visual and spatial similarity, and matched across shuffled and goal sets via the Hungarian algorithm or greedy assignment. The agent is evaluated on 26 randomly sampled episodes from the AI2-THOR RoomR 2-Phase challenge. The central claim is that SplatR improves over current state of the art on % Fixed Strict, % Misplaced, and % Energy Remaining, while the reported Success rate is lower than two baselines (Table 1). The authors also provide a comparison of Hungarian versus greedy matching and an ablation of fixed versus fixed-strict metrics.","tokens_in":12173,"tokens_out":4689,"duration_ms":45864,"significance":"If the central claim is valid, SplatR would demonstrate that a 3D Gaussian Splatting world model, combined with dense DINOv2 feature matching, is a viable representation for visual goal rearrangement, offering the advantage of continuous novel-view rendering and category-agnostic object matching. The method is zero-shot with respect to pretrained detection and segmentation models, which is a positive feature for generalization. However, the current evidence is insufficient to establish the claimed state-of-the-art improvement because the evaluation is based on a small, randomly sampled subset with no reported statistical uncertainty, and the comparison appears to be against published numbers that were likely obtained on the full dataset rather than on the same 26 episodes. Furthermore, the pipeline deliberately ignores the openness component of object state, which directly explains the low Success rate and limits the claim of improving over state of the art on the full task. The paper also lacks a control experiment for false positives caused by 3DGS rendering artifacts, which is a key risk for its change-detection mechanism.","major_comments":[{"comment":"The comparison underlying the state-of-the-art claim is not valid as reported. The authors evaluate SplatR on a randomly sampled subset of 26 episodes from the RoomR dataset, while the baseline numbers in Table 1 (TIDEE, CA VR, MaSS) appear to be the published results on the full dataset or a different evaluation setting. If the baselines were not re-run on the same 26 episodes, the comparison is confounded by the choice of subset, and the claimed improvements in % Fixed Strict, % Misplaced, and % Energy Remaining could be an artifact of the sampled episodes being easier for SplatR. The authors should either re-evaluate all baselines on their exact 26 episodes and report per-episode results, or clearly state that the comparison is against published aggregate numbers and temper the SOTA claim accordingly. In addition, a 26-episode sample with no seeds, no confidence intervals, and no error bars cannot support a strong comparative claim; at minimum, multiple random seeds and bootstrap intervals are needed.","section":"§4.1, Table 1"},{"comment":"The low Success rate is not a minor caveat but a direct limitation of the proposed method with respect to the benchmark's task definition. The authors state that the pipeline 'focuses exclusively on repositioning objects' and does not handle the degree-of-openness component of object state, which means episodes that require opening or closing objects will always fail. Since the AI2-THOR rearrangement challenge defines success based on restoring all object states, including openness, SplatR does not address the complete task. The claim of 'improvement over the current SOTA' is therefore selective: it holds for three metrics but not for Success, where SplatR is substantially worse than TIDEE+open-everything (3.85 vs 11.70). The paper should either extend the pipeline to handle openness (e.g., by planning a different action type for matched nodes with state changes), or explicitly reframe the contribution as a partial-solution method and not claim overall state-of-the-art performance on the benchmark.","section":"§5.2, Table 1"},{"comment":"The change-detection mechanism is not validated against rendering-induced false positives. SplatR compares a live RGB frame with a virtually rendered image from a 3DGS model trained only on the walkthrough trajectory. At novel viewpoints visited during unshuffle, the render may contain blur, floaters, missing geometry, or other artifacts. Because the pipeline groups any patchwise dissimilarity into object nodes, such artifacts are indistinguishable from genuinely displaced objects. The only filtering step is a CLIP check against the words 'wall' and 'mirror' (§3.5), which cannot cover arbitrary render errors. The paper does not report a no-shuffle control (i.e., running the full pipeline in a scene where no object has been shuffled and measuring the number of detections), nor does it provide any quantitative measure of false-positive detections. The close values of % Fixed and % Fixed Strict in Table 3 are suggestive but do not directly measure false positives, since % Fixed Strict is set to zero only if an already-correct object is moved during rearrangement, not if a spurious object node is generated but not acted upon. A no-shuffle control experiment is necessary to support the claim that the reported improvements reflect actual change detection rather than rendering artifacts.","section":"§3.4–§3.6"},{"comment":"Several parameters and implementation details that are load-bearing for the reported results are not specified. The node-merging rule in Eq. (7) depends on a weighting factor δ and a threshold τ_sim, but neither value is given in the paper or in an appendix. The pointcloud downsampling 'voxel grid size' in §3.3 is also unspecified. Since these parameters affect the quality of object grouping and node merging, and thus directly influence the reported Fixed Strict and Misplaced metrics, the results cannot be reproduced or assessed for robustness without their values. The authors should report all hyperparameters, and ideally include a sensitivity analysis showing that the results are stable to reasonable variations in δ, τ_sim, and the voxel size.","section":"§3.3, §3.7"}],"minor_comments":[{"comment":"The column labels in Table 3 are ambiguous: the header reads '% Fixed Strict % Fixed', but the text refers to 'Fixed Strict' and 'Fixed' inconsistently, and the value 36.34 for SplatR (HM) differs from the 36.35 reported in Table 1 for the same metric. The authors should clarify which column is which, reconcile the numerical inconsistency, and include the same number of decimal places throughout.","section":"Table 3"},{"comment":"The update rule for the fused CLIP feature vector after node merging is ambiguous as printed: the expression 'nj_o · gj + g / nj_o + 1' should be written as (n_j^o * g_j + g) / (n_j^o + 1) to make the numerator and denominator explicit. This is a minor readability issue but important for reproducibility.","section":"Eq. (10)"},{"comment":"The manuscript contains numerous typographical and grammatical errors, including 'pointcloudbased representation', 'is fast differentiable rendering capabilities', 'consist of corresponding image', and inconsistent use of 'SplatR' versus 'the agent'. A thorough language edit is needed before publication.","section":"Throughout"},{"comment":"The paper states that it 'leverages the 2D map module from [33]' but does not describe any of its properties, such as resolution, update mechanism, or how traversable regions are determined. Since the exploration strategy is a component of the pipeline, a brief description of this external module would help readers understand the exploration coverage and its potential effect on the final results.","section":"§3.3"},{"comment":"The limitation section mentions that the fixed DINOv2 patch size of 14×14 may be too coarse for small objects, but it does not quantify how many of the 26 episodes contain such objects or whether the reported metrics are affected. The authors could provide a per-object-category breakdown to give readers a sense of the practical impact.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's main issue is the fairness of the empirical comparison: SplatR is evaluated on a 26-episode subset while baseline numbers appear to be taken from published full-dataset results. This undermines the state-of-the-art claim even before considering the low Success rate and the lack of a false-positive control. I would recommend requiring the authors to re-run baselines on the same subset (or clearly prohibit the SOTA claim), report error bars over multiple seeds, and add a no-shuffle control experiment. The conceptual idea is interesting and publishable if these concerns are addressed, but the current evidence is not strong enough for acceptance. If the authors cannot reproduce the baselines on their subset, the paper should be reframed as a feasibility study rather than a SOTA comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper gives a genuinely new application—3D Gaussian Splatting as the goal-state world model for experience-goal rearrangement, with DINOv2 patchwise comparison in image space. The idea is clean and the authors are honest about limitations. But the empirical support is too weak to back the \"state of the art\" claim: 26 episodes, no error bars, no code or data, and Success lower than two baselines. The rendering-artifact concern from the stress test is real, though in AI2-THOR pose is ground truth, so misalignment is not the issue; artifact-induced false positives are. A no-shuffle control would help.\n\nWhat's new: as far as I know this is the first use of 3DGS for experience-goal rearrangement, and the integration with DINOv2 patchwise feature matching is a sensible way to compare a live view against a rendered goal view. The pipeline is coherent: exploration, splat training, patchwise change detection, node merging, bipartite matching, then rearrangement. They also state limitations clearly—small patch size for thin objects, memory costs, suboptimal exploration—and report zero-shot performance, which is credible.\n\nSoft spots: the evaluation is the main weakness. Twenty-six randomly sampled episodes is small, and without seeds or error bars we cannot tell if the reported numbers are meaningful. The SOTA claim is metric-selective: on Success they are at 3.85% versus TIDEE+open-everything's 11.7%, and CA VR also beats them there. They explain the openness limitation, but that means the abstract's \"improvements over the current state of the art\" is too strong. Table 3's column labels are ambiguous and need clarification. The stress-test point about 3DGS rendering artifacts deserves a direct answer: floaters or blur at novel viewpoints can be flagged as object changes by the patchwise comparison, and the CLIP wall/mirror filter covers only a narrow class of artifacts. A no-shuffle control—running the full pipeline when the scene is already at the goal—would quantify the false-positive floor. Also, the node-merging parameters delta and tau_sim appear without sensitivity analysis.\n\nVerdict: it is a reasonable systems contribution that deserves a serious referee, but the empirical claims need substantially stronger support. The authors should release code and data, run multiple seeds with error bars, clarify Table 3, and add a no-shuffle control. Once those are in place, the method's value will be much easier to assess.\n\nRecommendation: send to peer review with the expectation of major revisions; the core direction is promising, but the current evidence is not strong enough to establish the headline improvement.","headline":"A clean new application of 3DGS to rearrangement, but the empirical support is too thin to back the headline SOTA claim.","tokens_in":12709,"tokens_out":2552,"would_cite":true,"duration_ms":25428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3D Gaussian Splat of the goal scene, compared with DINOv2 patch features, lets an embodied agent detect and fix rearranged objects.","keywords":["3D Gaussian splatting","visual rearrangement","embodied AI","scene representation","dense feature matching","DINOv2","room rearrangement","world model"],"falsifier":"Take a fixed goal scene, render it through the splat with no object changes, then shift the virtual camera pose by small offsets or corrupt the splat slightly; if DINOv2 patchwise similarity flags many patches as changed, the reported detections are measuring rendering or alignment error, not object displacement.","tokens_in":11712,"feed_emoji":"🤖","tokens_out":8003,"duration_ms":64363,"temperature":0.7,"pith_summary":"SplatR tries to establish that an embodied agent can solve experience-goal visual rearrangement by storing the goal scene as a 3D Gaussian Splat. In the walkthrough phase the agent trains this splat; after the scene is shuffled, it places a virtual camera in the splat at its current pose and renders a matching goal view. The agent compares live and rendered views patch by patch with DINOv2 features, groups changed patches into objects, and rearranges them. The paper reports that this pipeline beats prior methods on the benchmark's key metrics (Fixed Strict, Misplaced, Energy Remaining), and argues the advantage comes from having consistent views of goal and current states in image space. If correct, this shows a continuous volumetric reconstruction plus foundation-model features can act as a practical goal-state representation for embodied rearrangement.","feed_headline":"Gaussian splat goal maps outperform prior rearrangement pipelines","feed_subtitle":"An agent stores the goal room as a splat, renders matching views, and uses patch features to spot changed objects.","key_machinery":"The load-bearing mechanism is the 3D Gaussian Splat: a set of 3D Gaussian primitives, each with mean, covariance, opacity, and color, trained by differentiable rasterization to reproduce the goal scene and render novel views in real time. A virtual camera placed at the agent's current pose turns this splat into a 'goal image' aligned with the live camera. The second mechanism is patchwise dense feature matching with DINOv2, in which each 14x14 image patch is represented by a visual feature vector and the cosine similarity of corresponding patches flags changed regions; adjacent changed patches are grouped into object detections. Object nodes carry CLIP embeddings and point clouds, and the Hungarian algorithm solves the category-agnostic bipartite matching between shuffled and goal nodes. Together these convert a static reconstruction into a rearrangement policy that ignores object-class detectors.","core_discovery":"The central claim is that a 3D Gaussian Splat of the goal configuration works as the world model for experience-goal rearrangement. Once the splat is trained from walkthrough observations, the agent renders the goal view from its current pose, producing an image pair (current, goal) that is consistent except for the object changes themselves. Change detection runs by cosine similarity between DINOv2 patch tokens of the two images; similar-dissimilar patch groups become coarse object masks, SAM refines them, and each object is stored as a node with a CLIP embedding, a point cloud, and a world-space center. At the end of exploration, Hungarian matching on CLIP similarities pairs shuffled objects with goal objects, and the agent picks and places accordingly. On the benchmark's Room Rearrangement subset, the paper reports 36.35% Fixed Strict, 0.62 Misplaced, and 0.63 Energy Remaining, improvements over the compared methods; the 3.85% Success rate is low because the pipeline only repositions objects and does not adjust open/close degrees.","pith_inferences":["A multi-scale or splat-embedded feature field would likely recover the small objects the fixed 14x14 patch misses; this can be tested by measuring detection rate as object size shrinks.","Adding pose noise to the virtual camera during evaluation would quantify how much of the change signal is alignment error, a cheap extension of the reported benchmark.","The same splat-plus-feature-match recipe could apply to any goal specification that can be rendered as a view, such as CAD models, sketches, or language-conditioned feature fields.","Because the paper evaluates zero-shot with respect to the target rooms, the framework's robustness to splatting artifacts is a key controlling factor for deployment beyond simulation."],"forward_implications":["Consistent image pairs can be generated from any viewpoint, so the agent no longer needs to revisit the exact goal viewpoint to detect changes.","Patch-level foundation-model features tolerate splatting artifacts better than pixel-level comparison, reducing false change detections on flat and reflective surfaces.","Combining change masks with SAM-refined object masks and CLIP embeddings gives category-agnostic matching, so no per-object detector trained on the target rooms is needed.","The reported benchmark numbers imply the method fixes more shuffled objects (36.35% Fixed Strict) and disturbs fewer unchanged objects (0.62 Misplaced) than the compared methods.","Because the pipeline only repositions objects and does not adjust openness, episodes that require opening or closing objects count as failures, keeping success rate low even when all displacements are corrected."],"supporting_citations":[{"why":"Supplies the differentiable 3D Gaussian splatting renderer used to build the goal-scene model and synthesize novel views.","marker":"[14]"},{"why":"Supplies the patch-level visual features used for dense feature matching between current and rendered images.","marker":"[29]"},{"why":"Defines the benchmark, the dataset, and the evaluation metrics the method is compared on.","marker":"[40]"},{"why":"Provides the 2D obstacle-map exploration module reused for data collection and one of the baselines to beat.","marker":"[33]"},{"why":"Provides a category-agnostic point-cloud baseline and motivates the category-agnostic matching design.","marker":"[22]"},{"why":"Supplies SAM, used to refine coarse patchwise masks into accurate object masks for manipulation.","marker":"[15]"},{"why":"Supplies CLIP embeddings used as semantic features in object nodes for matching shuffled and goal objects.","marker":"[32]"},{"why":"Supplies the Hungarian algorithm used for optimal bipartite matching between shuffled and goal object nodes.","marker":"[18]"}],"fun_headline_variants":["SplatR: 3D Gaussian splat as goal map beats priors","Gaussian splat goal world model boosts rearrangement","SplatR: dense feature matching on splat views improves","3D Gaussian splatting for experience goal rearrangement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes the goal-scene splat, viewed from the agent's current pose, lines up with the live camera closely enough that patch-feature differences mean real object changes rather than rendering artifacts or pose error.","fun_headline_variants_meta":{"raw":{"variants":["SplatR: 3D Gaussian splat as goal map beats priors","Gaussian splat goal world model boosts rearrangement","SplatR: dense feature matching on splat views improves","3D Gaussian splatting for experience goal rearrangement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1788,"prompt_tokens":975,"completion_tokens":813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":743}},"tokens_in":591,"tokens_out":813,"duration_ms":7395,"temperature":1.0,"reasoning_tokens":743,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:18:22.559260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed goal scene, render it through the splat with no object changes, then shift the virtual camera pose by small offsets or corrupt the splat slightly; if DINOv2 patchwise similarity flags many patches as changed, the reported detections are measuring rendering or alignment error, not object displacement.","supporting_citations":[{"cited_title":"Dinov2: Learning robust visual features with- out supervision, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the patch-level visual features used for dense feature matching between current and rendered images."},{"cited_title":"Visual room rearrangement, 2021","cited_arxiv_id":null,"evidence_quote":"Defines the benchmark, the dataset, and the evaluation metrics the method is compared on."},{"cited_title":"Harley, Paul Schydlo, Michael J","cited_arxiv_id":null,"evidence_quote":"Provides the 2D obstacle-map exploration module reused for data collection and one of the baselines to beat."},{"cited_title":"A category agnostic model for visual re- arrangment","cited_arxiv_id":null,"evidence_quote":"Provides a category-agnostic point-cloud baseline and motivates the category-agnostic matching design."}],"review_version":1}