{"id":"eb6f646c-a5c6-4428-b269-3eedd391a503","arxiv_id":"2411.14775","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CSE is a new synthetic multi-robot SLAM dataset in three indoor service environments with static/dynamic scenes, stereo depth, IMU, and ground truth poses.","lead":"Researchers built a simulated benchmark dataset for multiple service robots doing collaborative SLAM in hospitals, offices, and warehouses. It includes synchronized stereo and inertial data plus dynamic people, to test how well multi-robot mapping algorithms cope with real service environments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Static-vs-dynamic performance claims are confounded by non-identical robot trajectories and single-run evaluations; the dataset's central evidence that dynamic objects measurably affect SLAM is not yet established.","rationale":"Good-faith reading: the paper is a dataset contribution with a clear pipeline, detailed environment descriptions, and a public release. The dataset itself may indeed fill a gap among C-SLAM benchmarks. However, the paper's central claim that its challenges 'measurably affect' SLAM performance is supported only by static-vs-dynamic comparisons in Tables III and IV. Those comparisons are confounded because static and dynamic runs use different paths (Section III-D) and a single run per condition. This is more directly load-bearing than the sim-to-real gap because it affects the internal validity of the headline result even before any transfer question. The reader identified the sim-to-real gap as the weakest assumption; I partially agree, but the trajectory confound is the weakest link in the evidence as presented. The proposed check—replaying identical trajectories in static and dynamic scenes—would settle whether dynamic objects are the cause. If the authors can do this, or provide repeated trials with error bars, the central claim would be credible and the ACCEPT verdict would stand. Without it, the verdict should be CONDITIONAL on additional experiments.","tokens_in":12608,"tokens_out":4065,"duration_ms":42933,"concrete_test":"Re-run the baselines with matched trajectories: record the exact robot trajectories from each dynamic environment and replay them in the corresponding static scene (same waypoints, timestamps, and velocities), so the only change is the presence of dynamic objects; then recompute ATE for Tables III and IV. If the static/dynamic ATE gap disappears or reverses, the claimed effect of dynamic objects is unsupported. As a minimal complement, run each sequence at least 5 times under different ROS Navigation Stack seeds and report mean±std; if the static/dynamic gap is within run-to-run variance, the confound is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that CSE's dynamic-object challenges measurably affect SLAM performance—rests on a static/dynamic comparison that is internally confounded. Section III-C builds each environment in static and dynamic versions, and Section IV-C interprets performance differences as evidence that dynamic objects and occlusions degrade SLAM. But Section III-D states that static and dynamic scenarios share goal points only: \"due to variability introduced by the ROS Navigation Stack, the paths between goal points in static and dynamic scenarios are not perfectly identical but are almost similar.\" Different trajectories change viewpoint sequences, loop-closure structure, and encounter timing, all of which affect SLAM ATE independently of dynamic objects. Tables III and IV report one ATE per condition, with no repeated runs, error bars, or significance testing. Several dynamic results are not worse than static (e.g., Table III Hospital R3 ORB-SLAM3 RGB-D: 0.030 dynamic vs 0.040 static; Office R1 VINS-Fusion MonoInertial: 0.256 dynamic vs 2.546 static), so the direction of the effect is not consistent. The sim-to-real gap is a valid limitation, but even granting simulator realism, the paper has not established that its dynamic-object challenge, rather than path variation, drives the reported degradations. This matters because the benchmark's raison d'être is to isolate such challenges.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CSE, a synthetic multi-modal collaborative SLAM (C-SLAM) dataset generated in NVIDIA Isaac Sim for three indoor service environments (Hospital, Office, Warehouse). Each environment is provided in static and dynamic versions, with three simulated service robots (NVIDIA Carter) equipped with stereo RGB/depth, IMU, and ground-truth poses; pseudo-LiDAR conversion is also provided. The dataset is designed to include challenges such as homogeneous scenes, repeated objects, dynamic humans, severe occlusions, and large rotations, and the scenarios are built to contain intra- and inter-robot loop closures. The authors evaluate both single-robot (ORB-SLAM3, VINS-Fusion) and multi-robot (COVINS, Swarm-SLAM) methods, including variants with dynamic-feature removal, and report ATE for each sequence.","tokens_in":12868,"tokens_out":3354,"duration_ms":35789,"significance":"If the dataset proves usable, it fills a genuine gap: existing C-SLAM datasets are mostly urban-outdoor or small-laboratory indoor collections, while service robots operate in diverse indoor spaces with dynamic humans and visually challenging structures. The paper's strengths include open release, a tutorial for dataset generation, accurate time-synchronized ground truth from the simulator, and explicit scenario design for intra/inter-robot loop closures. The static/dynamic split is a useful idea for studying dynamic-object robustness, and the evaluation across multiple SLAM systems and sensor modalities is a valuable baseline. However, the central interpretive claim that static/dynamic differences are caused by dynamic objects is currently weakened by a trajectory confound, as detailed below.","major_comments":[{"comment":"The static-vs-dynamic comparison is structurally confounded. Section III-D states that static and dynamic scenarios share only the same goal points and that 'the paths between goal points in static and dynamic scenarios are not perfectly identical but are almost similar.' Because the ROS Navigation Stack produces different trajectories, the viewpoint sequence, loop-closure opportunities, and encounter timing differ between conditions. Tables III and IV then attribute ATE differences to dynamic objects, but the observed differences could equally arise from path variation. The inconsistency in direction (e.g., Table III Hospital R3 ORB-SLAM3 RGB-D is 0.040 static vs 0.030 dynamic; Office R1 VINS-Fusion MonoInertial is 2.546 static vs 0.256 dynamic) further undermines the causal reading. To support the claimed 'evaluate the efficiencies of SLAM algorithms dealing with dynamic objects,' the authors should either control the trajectories exactly (e.g., replay the same recorded path with dynamic objects toggled, which a simulator allows) or explicitly reframe the static/dynamic comparison as descriptive and acknowledge the confound when interpreting results.","section":"Section IV-C"},{"comment":"The quantitative evaluation is based on a single recorded dataset per condition, with no repeated runs or statistical analysis. While SLAM evaluation on a fixed sequence is deterministic, the data-collection process itself (robot trajectories) is stochastic due to the navigation stack; hence the reported ATE values are single samples from a distribution, and performance differences between conditions are not statistically characterized. For a benchmark paper, this is acceptable as a demonstration, but the narrative claims of 'notable reduction in performance' and 'significant impact' should be tempered, or the authors should provide multiple data-collection repetitions with variance estimates.","section":"Section IV-B"}],"minor_comments":[{"comment":"There is a stray curly quote in the Table III caption (\"The metric represents RMS ATE in meters.'\").","section":"Table III"},{"comment":"The sentence 'The all evaluation is conducted with EVO [26].' has a grammatical error; it should read 'All evaluations are conducted with EVO [26].'","section":"Section IV-B"},{"comment":"In the text, 'the ROS Navigation Stack' appears with inconsistent spacing (e.g., 'R OBOT' in Section III-D). Please fix the spacing in the robot names throughout.","section":"Section III-D"},{"comment":"The statement 'For IMU, we provide empirically tuned IMU parameters' would benefit from specifying the noise model (e.g., white noise and bias random walk values) or citing the simulator defaults, so users can judge sensor realism.","section":"Section III-B"},{"comment":"The sim-to-real limitation is candidly acknowledged, but the paper could briefly state which physical phenomena are expected to transfer (e.g., geometric and dynamic challenges) and which are not (e.g., photometric realism), to guide users.","section":"Section V-A"}],"recommendation":"major_revision","confidential_remarks":"The reader's report recommended acceptance, and I agree the dataset has promise, but I believe the static/dynamic confound is load-bearing for one of the paper's key claims—that CSE can isolate the effect of dynamic objects on SLAM performance. Because the authors use a simulator, they can readily fix this by replaying identical trajectories with/without dynamic objects. The single-run issue is secondary but worth addressing. I would not reject: the dataset itself is the main contribution and is likely valuable even if the static/dynamic results are reinterpreted as qualitative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nCSE is a genuinely useful synthetic dataset for multi-robot SLAM in indoor service environments. Three robots, three environments (hospital, office, warehouse), synchronized stereo RGB/depth, IMU, and ground truth from Isaac Sim, with static and dynamic versions and ROS Navigation Stack for realistic paths. That fills a real gap: existing C-SLAM datasets are mostly outdoor or small indoor labs, and this one is built around service-robot scenarios with intra/inter-robot loop closures. The tutorial and code are a plus.\n\nThe paper's main analytical claim is that dynamic objects measurably hurt SLAM performance. That doesn't hold up as presented. Section III-D says static and dynamic scenarios share only goal points, not trajectories: 'the paths between goal points in static and dynamic scenarios are not perfectly identical but are almost similar.' So performance differences are confounded with path variation. The tables show single runs, no error bars, no repeats. And several dynamic results are better than the static ones (e.g., Table III, Hospital R3 ORB-SLAM3 RGB-D 0.030 vs 0.040; Office R1 VINS-Fusion mono-inertial 2.546 vs 2.042). The paper's Section IV-C overstates the causal role of dynamic objects. To make that claim, the authors would need controlled trajectories across conditions and repeated runs.\n\nThe sim-to-real gap is acknowledged in Section V-A, but not validated. That is a limitation, not a fatal one—synthetic benchmarks are fine if the goal is stress-testing algorithms, but it means the conclusions are about the simulator, not real hospitals.\n\nMinor issues: 'empirically tuned IMU parameters' need a proper description; the dataset itself wasn't inspected, so the GitHub contents are a promise. These are fixable.\n\nOverall: the gap and the artifact are real and worth a serious look. The dataset should go to peer review, but the analysis section needs significant revision—either more rigorous controls or a more modest interpretation.","headline":"Useful synthetic C-SLAM benchmark, but the central claim that dynamic objects drive SLAM degradation is confounded by path variation and single-run evaluation.","tokens_in":13403,"tokens_out":3539,"would_cite":true,"duration_ms":31527,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new synthetic benchmark dataset, CSE, brings collaborative SLAM evaluation to indoor service environments and shows that dynamic objects and homogeneous scenes degrade current multi-robot SLAM.","keywords":["collaborative SLAM","multi-robot SLAM","service robots","synthetic dataset","indoor service environments","dynamic objects","benchmark","loop closure"],"falsifier":"Run the same ORB-SLAM3, VINS-Fusion, COVINS, and Swarm-SLAM configurations on a real indoor service environment with comparable geometry, synchronized multi-robot ground truth, and humans walking through the scene, then compare ATE and failure rates. If real-world dynamic degradation is much smaller, or if the simulator's homogeneous-scene ambiguities do not reproduce, the transferability claim is falsified.","tokens_in":12439,"feed_emoji":"🤖","tokens_out":5636,"duration_ms":50476,"temperature":0.7,"pith_summary":"The paper introduces CSE, a synthetic benchmark dataset for collaborative SLAM with multiple service robots in indoor service environments. It argues that existing C-SLAM datasets are collected in outdoor urban scenes or small indoor labs, and therefore miss the homogeneous corridors, visually redundant objects, occluding humans, and tight robot-to-robot encounters that service robots face. The dataset provides three robots navigating Hospital, Office, and Warehouse layouts, each in static and dynamic versions with precisely synchronized stereo RGB/depth, IMU, and ground-truth poses from simulation. Baseline experiments show that these challenges degrade state-of-the-art single-robot and multi-robot SLAM algorithms, demonstrating that the dataset captures conditions that matter for service robotics.","feed_headline":"New dataset stress-tests multi-robot SLAM in hospitals and offices","feed_subtitle":"Three robots, paired static/dynamic scenes, and simulator ground truth reveal where today's SLAM fails.","key_machinery":"The load-bearing mechanism is the paired static/dynamic environment construction. By keeping geometry and goals identical and toggling only the dynamic objects, the dataset isolates the effect of dynamic scenes on SLAM performance; the simulator provides exact time synchronization and ground truth that would be impractical to collect with multiple real robots. A second mechanism is the scenario design built from intra-robot, robot-to-robot, and robot-to-human interactions, which generates the loop-closure conditions C-SLAM must handle.","core_discovery":"The central contribution is a controlled dataset design: each of the three environments is rebuilt as a static and a dynamic variant with the same navigation goal points, so the only systematic difference is the presence of humans and moving objects. With three robots driven by autonomous navigation rather than manual teleoperation, the dataset includes intra- and inter-robot loop closures, follow/cross/revisit encounters, close-range occlusion, and recovery behaviors. The paper's evaluations show measurable ATE increases and algorithmic failures under dynamic conditions, including cases where place recognition succeeds but dynamic objects corrupt feature matching, and cases where centralized multi-robot SLAM propagates a single robot's failure to the whole team.","pith_inferences":["Because the dataset is fully synthetic, a natural next step is to validate the same baseline runs against real indoor service environments; the failure modes the authors observe should be tested for transfer before treating the benchmark as a proxy for physical deployment.","The observed invalid-matching failures suggest a concrete research direction: dynamic-keypoint-aware bundle adjustment and object tracking, not just dynamic-feature removal.","The dataset's structure could be extended to lifelong SLAM by adding the scene-change samples the authors have already prototyped, which would let researchers study long-term map maintenance in the same controlled environments.","The inclusion of robot recovery behavior and close-range robot-to-robot encounters gives a testbed for SLAM under abrupt stops and replanning, a condition rarely covered by manual-teleoperation datasets."],"forward_implications":["Dynamic objects and visually redundant structures cause significant ATE increases and algorithm halts in both single- and multi-robot SLAM, so service-environment C-SLAM must handle dynamic scenes explicitly.","Naive dynamic-feature removal helps in some dynamic sequences but hurts under extreme occlusion, indicating that simply deleting moving features is not sufficient.","Centralized C-SLAM fails more often than decentralized C-SLAM in this dataset because one robot's failure propagates through the shared backend.","The same-goal static/dynamic pairs make it possible to compare algorithms in a controlled way, since the only changed factor is the presence of dynamic objects.","Place recognition can succeed while dynamic objects still cause invalid feature matching, so C-SLAM evaluation needs to attend to feature-level quality, not just loop-closure detection."],"supporting_citations":[{"why":"Supplies the simulator used to generate photo-realistic, time-synchronized sensor data and ground-truth poses.","marker":"[14]"},{"why":"UTIAS, the first multi-robot SLAM dataset, establishes the early static indoor benchmark that CSE extends.","marker":"[8]"},{"why":"S3E provides the comparison table's baseline for multimodal multi-robot datasets and time-synchronization references.","marker":"[10]"},{"why":"AirMuseum represents the static indoor heterogeneous-agent C-SLAM dataset with SfM ground truth.","marker":"[9]"},{"why":"GRACO grounds the comparison against outdoor urban ground-aerial cooperative SLAM datasets.","marker":"[12]"},{"why":"Tian et al. supplies the closest prior multi-robot dataset with dynamic objects in indoor/outdoor scenes.","marker":"[13]"},{"why":"ORB-SLAM3 is the single-robot visual-inertial baseline whose failures on redundant scenes demonstrate the dataset's challenges.","marker":"[21]"},{"why":"Swarm-SLAM is the decentralized multi-robot baseline whose dynamic degradation is a main experimental result.","marker":"[24]"},{"why":"DS-SLAM's moving-consistency check is the basis for the dynamic-feature-removal variants used in the experiments.","marker":"[25]"},{"why":"OpenLORIS-Scene establishes the single-robot service-environment SLAM dataset with dynamic objects and textureless scenes that motivates the multi-robot gap.","marker":"[3]"}],"fun_headline_variants":["Simulated hospitals and offices expose multi-robot SLAM blind spots","New C-SLAM dataset: same routes, with and without humans to catch failures","Three-robot SLAM benchmark: dynamic scenes break current methods","Service-robot SLAM dataset pairs static and dynamic scenes for fair testing","CSE dataset: multi-robot SLAM fails when humans enter the scene"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's usefulness rests on the assumption that the simulator's rendered images, sensor noise, and animated humans behave enough like real hospitals, offices, and warehouses that the SLAM failures and performance rankings measured here carry over to physical service robots; the paper does not validate against real sensor data.","fun_headline_variants_meta":{"raw":{"variants":["Simulated hospitals and offices expose multi-robot SLAM blind spots","New C-SLAM dataset: same routes, with and without humans to catch failures","Three-robot SLAM benchmark: dynamic scenes break current methods","Service-robot SLAM dataset pairs static and dynamic scenes for fair testing","CSE dataset: multi-robot SLAM fails when humans enter the scene"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1257,"prompt_tokens":951,"completion_tokens":306,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":209}},"tokens_in":567,"tokens_out":306,"duration_ms":3470,"temperature":1.0,"reasoning_tokens":209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:54:13.575956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ORB-SLAM3, VINS-Fusion, COVINS, and Swarm-SLAM configurations on a real indoor service environment with comparable geometry, synchronized multi-robot ground truth, and humans walking through the scene, then compare ATE and failure rates. If real-world dynamic degradation is much smaller, or if the simulator's homogeneous-scene ambiguities do not reproduce, the transferability claim is falsified.","supporting_citations":[{"cited_title":"Nvidia isaac sim, https://developer.nvidia.com/isaac-sim","cited_arxiv_id":null,"evidence_quote":"Supplies the simulator used to generate photo-realistic, time-synchronized sensor data and ground-truth poses."},{"cited_title":"The utias multi- robot cooperative localization and mapping dataset,","cited_arxiv_id":null,"evidence_quote":"UTIAS, the first multi-robot SLAM dataset, establishes the early static indoor benchmark that CSE extends."},{"cited_title":"S3e: A large-scale multimodal dataset for collaborative slam,","cited_arxiv_id":null,"evidence_quote":"S3E provides the comparison table's baseline for multimodal multi-robot datasets and time-synchronization references."},{"cited_title":"Airmuseum: a heterogeneous multi-robot dataset for stereo-visual and inertial simultaneous localization and mapping,","cited_arxiv_id":null,"evidence_quote":"AirMuseum represents the static indoor heterogeneous-agent C-SLAM dataset with SfM ground truth."},{"cited_title":"Graco: A multimodal dataset for ground and aerial cooperative localization and mapping,","cited_arxiv_id":null,"evidence_quote":"GRACO grounds the comparison against outdoor urban ground-aerial cooperative SLAM datasets."},{"cited_title":"Resilient and distributed multi-robot visual slam: Datasets, experiments, and lessons learned,","cited_arxiv_id":null,"evidence_quote":"Tian et al. supplies the closest prior multi-robot dataset with dynamic objects in indoor/outdoor scenes."},{"cited_title":"Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,","cited_arxiv_id":null,"evidence_quote":"ORB-SLAM3 is the single-robot visual-inertial baseline whose failures on redundant scenes demonstrate the dataset's challenges."},{"cited_title":"Swarm-slam: Sparse decentralized collaborative simultaneous localization and mapping framework for multi-robot systems,","cited_arxiv_id":null,"evidence_quote":"Swarm-SLAM is the decentralized multi-robot baseline whose dynamic degradation is a main experimental result."},{"cited_title":"Ds-slam: A semantic visual slam towards dynamic environments,","cited_arxiv_id":null,"evidence_quote":"DS-SLAM's moving-consistency check is the basis for the dynamic-feature-removal variants used in the experiments."},{"cited_title":"Are we ready for service robots? the openloris- scene datasets for lifelong slam,","cited_arxiv_id":null,"evidence_quote":"OpenLORIS-Scene establishes the single-robot service-environment SLAM dataset with dynamic objects and textureless scenes that motivates the multi-robot gap."}],"review_version":1}