{"id":"25bda24a-6e74-4f65-9898-a81abd70be11","arxiv_id":"2508.16731","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A new suite of 24 realistic collaborative SLAM optimization benchmarks derived from real LiDAR data with simulated inter-robot communication.","lead":"COSMO-Bench provides 24 benchmark datasets for collaborative SLAM back-end optimization, built from real LiDAR data and a communication model. The authors synchronize single-robot trials into multi-robot sequences and release them with reference solutions, noise models, and outlier labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Temporal synchronization of single-robot trials is unvalidated; inter-robot measurement structure may be an artifact of arbitrary start-time offsets.","rationale":"The reader identified the temporal synchronization of single-robot trials as the weakest assumption, and my analysis converges on the same point. This is genuinely load-bearing: the benchmark's raison d'être is to provide realistic multi-robot C-SLAM data, and the inter-robot loop closures—the key feature distinguishing it from single-robot benchmarks—are generated by stitching independent trials. The paper is transparent about the method (Remark 1) and about communication-model limitations (Remark 5), but transparency does not substitute for validation. The concern is not that the authors are wrong, but that the core premise is asserted rather than tested. The proposed test is directly feasible: the authors have released the front-end and communication code, and they already provide the Nebula datasets in the same JRL format, so a statistical comparison of inter-robot measurement properties is a natural and low-cost check. I do not see a more damaging internal inconsistency: the front-end is composed of standard components, the noise models are empirically derived, and the reference solutions come from trusted sources. The missing validation of the synchronization premise justifies keeping the reader's CONDITIONAL verdict: the paper is a solid contribution if the premise holds, but that premise needs empirical support before the benchmark's realism claim can be fully accepted.","tokens_in":11958,"tokens_out":4927,"duration_ms":65663,"concrete_test":"Take one COSMO-Bench source sequence (e.g., kth_r3_00) and re-run the full pipeline 20 times, resampling the start-time offsets Δi and all communication-model randomness while keeping the front-end fixed. For each run, compute the number of inter-robot loop closures, the distribution of inter-robot relative translations/rotations, and the temporal gaps between inter-robot matches. If the coefficient of variation across runs for these statistics is comparable to or larger than the variation across the 12 distinct environments, the inter-robot structure is dominated by the arbitrary synchronization/communication seed, not by the environment. Additionally, run the same front-end on a real concurrently-collected multi-robot dataset (e.g., Nebula or the Lajoie et al. data used in Sec. IV-C) and compare these distributions; if COSMO-Bench's inter-robot statistics fall far outside the real-data","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that COSMO-Bench provides realistic benchmark datasets for distributed C-SLAM back-ends. The load-bearing step is Sec. IV-A: multiple single-robot trials, collected at different times, are temporally synchronized into a simulated multi-robot sequence by sampling relative start offsets Δi ~ N(0s, 40s). The paper asserts this 'effectively pretend[s] that the trials were gathered at the same time by multi-robots,' but provides no evidence that the resulting inter-robot measurement distribution matches a concurrently operating team. In a real multi-robot deployment, trajectories are coupled through shared tasks, collision avoidance, and simultaneous environmental dynamics. Here, independent trials are pasted together with random time shifts; there is no physical consistency between robots. Consequently, the spatial and temporal pattern of inter-robot loop closures—how often, where, and with what baseline robots observe the same scene—is determined more by the arbitrary Δi and by the stochastic communication model than by any realistic team behavior. This matters because the benchmark's value for evaluating C-SLAM back-ends depends on inter-robot measurements being representative of real deployments. The paper even criticizes partitioned single-robot benchmarks for not representing 'topological structure, viewpoint variance, or measurement distribution' of a multi-robot team; the synchronization approach may inherit similar issues, though it is certainly an improvement. Remark 1 acknowledges some side-effects but does not quantify their impact. The absence of any validation against real multi-robot data (e.g., the Nebula datasets the paper itself releases) leaves this premise untested. If the inter-robot measurement distribution is dominated by the synchronization seed, the benchmark's realism claim is compromised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COSMO-Bench, a suite of 24 benchmark datasets for distributed collaborative SLAM (C-SLAM) back-end evaluation. The datasets are generated by temporally synchronizing multiple single-robot LiDAR trials from the Multi-Campus Dataset and CU-Multi, then passing the synchronized data through a baseline C-SLAM front-end (LOAM odometry, distance-based keyframing, ScanContext loop detection, and KISS-Matcher registration). Inter-robot measurements are produced under one of two communication models: a Wi-Fi model fit to real three-robot communication data and a scaled \"Pro-Radio\" variant. Each dataset includes a survey-grade reference solution, empirical noise models, temporal information, and reference outlier classifications. The authors also convert the Nebula multi-robot datasets into the same JRL format. The central claim is that COSMO-Bench provides a realistic, reproducible, standardized testbed for distributed C-SLAM back-ends.","tokens_in":12386,"tokens_out":4550,"duration_ms":58339,"significance":"If the data-generation methodology is accepted, COSMO-Bench is a significant community resource: it addresses a genuine lack of standard benchmarks for distributed C-SLAM back-ends, provides open access with a DOI, uses real LiDAR data and a documented front-end, and includes reference solutions and temporal information. The authors are commendably explicit about several limitations (Remarks 1, 5, 8), and the conversion of the Nebula datasets into a common format is a useful extra contribution. The main risk is that the central realism claim rests on an unvalidated synchronization procedure and a manually parameterized communication model; both directly shape the inter-robot measurement structure that the benchmark is designed to provide.","major_comments":[{"comment":"The temporal synchronization procedure is the load-bearing step of the benchmark, but it is not validated. The method samples relative start offsets Δi ~ N(0s, 40s) and 'effectively pretend[s]' that independent trials were collected simultaneously by a multi-robot team. No evidence is provided that the resulting inter-robot measurement distribution, spatial topology, or temporal overlap matches a concurrently operating team. Remark 1 even allows a robot whose local data is finished to remain active and stationary and continue generating inter-robot measurements; this can create loop closures anchored at a fixed pose, a pattern one would not expect in a typical moving team. The paper itself criticizes partitioned single-robot benchmarks for not representing 'topological structure, viewpoint variance, or measurement distribution' of a real multi-robot team; the synchronization approach may","section":"Sec. IV-A"},{"comment":"The communication model's connectivity function ϕ(d) is 'manually fitted' to data from a single three-robot experiment and then scaled heuristically for Pro-Radio. The model's parameters (Pmax, α, β, rmax, dinit, dintf, B) are free parameters that directly control which inter-robot loop closures appear in all 24 datasets. The authors acknowledge that physical interference is not explicitly modeled (Remark 5), but the more basic concern is that a single manual fit from one environment may not generalize to the campus environments used in COSMO-Bench. Please report the fit residuals, parameter uncertainty, and ideally a sensitivity analysis showing how benchmark statistics (e.g., IRLC count, outlier rate, topology) vary with these parameters. This would let users assess how strongly conclusions drawn from the benchmark depend on the communication model choice.","section":"Sec. IV-C.3"},{"comment":"The reference outlier classifications are defined using the same empirical noise models that are computed from thresholded 'good' measurements in Sec. IV-D.3. A measurement is labeled an outlier if its residual exceeds the 95% χ² critical value of the fitted Gaussian noise model. This is self-referential: the noise model is estimated only from measurements already judged to be inliers by a user-supplied threshold, so the 5% outlier rate is partly baked into the Gaussian assumption rather than discovered from the data. The labels may not reflect true data association failures. Please report the sensitivity of the outlier labels to the inlier thresholds (e.g., 0.5 m, 0.05 rad) and, if possible, validate a subset against manual inspection or known failure cases.","section":"Sec. IV-D.4"}],"minor_comments":[{"comment":"The sentence 'with our benchmarks supporting observing the effects from different distributed C-SLAM back-ends' is awkward and could be rephrased for clarity.","section":"Sec. II"},{"comment":"The throughput plot uses channel labels C:1-2, C:1-3, C:2-3, but the legend is not self-contained; please include the channel labels directly in the plot or in the caption for readability.","section":"Fig. 3"},{"comment":"Since CU-Multi was 'designed intentionally for multi-robot applications,' the authors should briefly clarify whether its trials are truly independent or whether some were collected simultaneously, and why simultaneous trials were not used directly if available.","section":"Sec. IV-A"},{"comment":"The xz compression factor distribution is estimated from 20 scans from a single trial. This small sample size should be noted in the text, as the resulting μxz and σxz are used for all datasets.","section":"Sec. IV-C.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real community need and the dataset release is valuable, but the central realism claim depends on the unvalidated synchronization approach and the manually fitted communication model. I would recommend asking the authors to provide the validation or ablations described in major comments 1 and 2 before acceptance. It would also strengthen the paper to include a simple demonstration of a back-end evaluation on the benchmark, showing that the datasets are usable and that the reference labels behave as intended."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid contribution to the distributed C-SLAM benchmarking gap. The paper gives the community 24 datasets in a clean format, built from real LiDAR scans with a transparent pipeline, reference solutions, empirical noise models, and outlier labels. The temporal synchronization of single-robot trials into multi-robot sequences is a new trick, and the communication model derived from actual Wi-Fi data is a step up from the ad-hoc partitioning people have been using. The authors are also honest about limitations in Remarks 1, 5, and 8.\n\nThe main soft spot is exactly what the stress-test flags: there is no validation that stitching trials from different days with random start offsets (Δi ~ N(0s, 40s)) produces inter-robot measurement structure representative of a real concurrent team. The paper argues LiDAR is invariant to visual changes, which handles appearance, but it doesn't handle the fact that real teams are dynamically coupled through tasks and collision avoidance. The result could be that the spatial/temporal pattern of inter-robot loop closures is driven more by the random offsets and the communication model than by anything you'd see in an actual deployment. That's a legitimate concern, though not a fatal one—the authors could validate against the CU-Multi data, which includes actual multi-robot runs, or at least acknowledge that the benchmark targets applications where teams operate quasi-independently.\n\nA couple of smaller things: the outlier labels are generated using the same empirical noise models the datasets are built with, so there's a mild circularity, but the reference solutions are independent, so it's fine. And the paper doesn't include a sample evaluation showing the benchmark can discriminate between back-ends. That's common for dataset papers, but a simple experiment would have strengthened it.\n\nOverall, I think this deserves a serious referee. The resource itself is valuable, the methodology is transparent, and the flaws are addressable. I'd recommend publishing it, likely after minor revision.","headline":"A transparent, useful C-SLAM benchmark suite whose only real weakness is that the temporal synchronization premise is sensible but unvalidated.","tokens_in":12846,"tokens_out":2900,"would_cite":true,"duration_ms":29180,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COSMO-Bench provides 24 benchmark datasets for collaborative SLAM back-ends, built from real LiDAR data and realistic communication models.","keywords":["collaborative SLAM","multi-robot SLAM","benchmark","pose graph optimization","LiDAR","loop closure","outlier classification","communication model"],"falsifier":"Record a small team of robots driving concurrently through the same campus environments, run the same front-end pipeline, and compare inter-robot loop-closure rates, outlier fractions, and inter-robot distance distributions against COSMO-Bench's synchronized trials; a substantial mismatch would undermine the benchmark's representativeness.","tokens_in":11922,"feed_emoji":"🤖","tokens_out":3078,"duration_ms":32130,"temperature":0.7,"pith_summary":"The paper introduces COSMO-Bench, a suite of 24 benchmark datasets for evaluating distributed collaborative simultaneous localization and mapping (C-SLAM) back-end algorithms. The goal is to fill the gap left by the lack of standard multi-robot benchmarks, which have long existed for single-robot SLAM. The datasets are derived from real-world LiDAR data by temporally synchronizing multiple single-robot trials into plausible multi-robot sequences, then running a baseline front-end and a communication model based on actual multi-robot Wi-Fi data. Each dataset includes reference solutions, empirical noise models, and reference outlier classifications, so back-end algorithms can be compared fairly and reproducibly.","feed_headline":"24 real-world datasets for collaborative SLAM optimization","feed_subtitle":"New benchmark turns single-robot LiDAR runs into multi-robot testbeds with ground truth and outliers.","key_machinery":"The central mechanism is temporal synchronization: multiple single-robot trials collected in the same environment are shifted in time relative to an anchor trial, effectively simulating concurrent multi-robot operation. This is paired with a two-instance communication model (Wi-Fi and Pro-Radio) that samples connectivity based on inter-robot distance and models bandwidth sharing, deciding which inter-robot measurements occur. The baseline front-end pipeline, using LOAM odometry, keyframe sampling, ScanContext detection, and KISS-Matcher alignment, generates the actual measurements, and empirical noise models derived from reference solutions allow outlier classification via a chi-squared thre","core_discovery":"On its own terms, the paper establishes that a realistic distributed C-SLAM benchmark can be built entirely from existing single-robot LiDAR data by temporal synchronization, without needing new concurrent multi-robot collection. The resulting 24 datasets contain both intra-robot and inter-robot loop closures with realistic outlier rates, plus high-quality reference ground truth and reference outlier labels. The paper also converts the existing Nebula multi-robot datasets into the same format, making them directly usable with the new benchmark. The key claim is that these datasets meet the requirements of representative measurements, plentiful loop closures, long traversals, accurate referen","pith_inferences":["The synchronization of single-robot trials likely underrepresents correlated multi-robot perception, such as multiple robots observing the same dynamic object from different viewpoints, because the trials are physically disconnected.","The communication model's connectivity relies only on distance, ignoring physical obstacles like walls and vegetation; real cluttered environments may yield fewer inter-robot loop closures than the benchmark suggests.","Sequences formed from same-environment trials may produce denser inter-robot loop-closure opportunities than a team exploring disjoint areas, so back-end performance measured here could overestimate performance in exploration-oriented missions.","The methodology could be cheaply extended by mixing other open-source single-robot LiDAR trials to generate more benchmark sequences, but validating against true concurrent multi-robot data would be needed to confirm representativeness."],"forward_implications":["Researchers can compare distributed C-SLAM back-end algorithms on common data with known reference solutions and outlier labels, enabling reproducible results.","The realistic outlier rates and inter-robot measurement distributions support robustness testing of back-end algorithms under non-ideal conditions.","The JSON Robot Log (JRL) format makes the datasets human-readable and platform-agnostic, lowering the barrier to adoption.","Providing both Wi-Fi and Pro-Radio communication models lets researchers test sensitivity to network bandwidth and range characteristics.","Converting the Nebula datasets into the same format allows direct comparison with existing real-world multi-robot data from underground deployments."],"supporting_citations":[{"why":"Supplies the Multi-Campus Dataset LiDAR trials used to generate 10 of the benchmark sequences.","marker":"[33]"},{"why":"Supplies the CU-Multi Dataset LiDAR trials used to generate the remaining 2 benchmark sequences.","marker":"[30]"},{"why":"Provides real multi-robot communication data that the Wi-Fi connectivity model is fitted to.","marker":"[38]"},{"why":"LOAM is the odometry module in the baseline C-SLAM front-end that produces local measurements.","marker":"[35]"},{"why":"ScanContext is used for both intra-robot and inter-robot loop-closure detection.","marker":"[36]"},{"why":"KISS-Matcher computes the loop-closure measurements, including realistic outliers.","marker":"[37]"},{"why":"Provides the Nebula multi-robot datasets that are converted to JRL format and included for comparison.","marker":"[32]"}],"fun_headline_variants":["Single-robot LiDAR runs now yield multi-robot SLAM testbeds","Benchmark turns solo robot data into 24 collaborative SLAM sets","No new robots needed: COSMO-Bench repurposes LiDAR data for C-SLAM","24 datasets for collaborative SLAM built from existing LiDAR runs","From single to multi-robot: COSMO-Bench offers 24 testbed datasets"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Multiple single-robot trials recorded at different times, when shifted onto a common clock, produce inter-robot measurement patterns representative of a team actually driving together.","fun_headline_variants_meta":{"raw":{"variants":["Single-robot LiDAR runs now yield multi-robot SLAM testbeds","Benchmark turns solo robot data into 24 collaborative SLAM sets","No new robots needed: COSMO-Bench repurposes LiDAR data for C-SLAM","24 datasets for collaborative SLAM built from existing LiDAR runs","From single to multi-robot: COSMO-Bench offers 24 testbed datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2459,"prompt_tokens":628,"completion_tokens":1831,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":372,"completion_tokens_details":{"reasoning_tokens":1725}},"tokens_in":372,"tokens_out":1831,"duration_ms":12447,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:11:23.057593+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a small team of robots driving concurrently through the same campus environments, run the same front-end pipeline, and compare inter-robot loop-closure rates, outlier fractions, and inter-robot distance distributions against COSMO-Bench's synchronized trials; a substantial mismatch would undermine the benchmark's representativeness.","supporting_citations":[{"cited_title":"LOAM: Lidar odometry and mapping in real- time.,","cited_arxiv_id":null,"evidence_quote":"LOAM is the odometry module in the baseline C-SLAM front-end that produces local measurements."},{"cited_title":"Scan context: Egocentric spatial descriptor for place recognition within 3d point cloud map,","cited_arxiv_id":null,"evidence_quote":"ScanContext is used for both intra-robot and inter-robot loop-closure detection."},{"cited_title":"KISS-Matcher: Fast and robust point cloud registration revisited,","cited_arxiv_id":null,"evidence_quote":"KISS-Matcher computes the loop-closure measurements, including realistic outliers."},{"cited_title":"LAMP 2.0: A robust multi-robot SLAM system for operation in challenging large-scale underground environments,","cited_arxiv_id":null,"evidence_quote":"Provides the Nebula multi-robot datasets that are converted to JRL format and included for comparison."},{"cited_title":"MCD: Diverse large-scale multi-campus dataset for robot perception,","cited_arxiv_id":null,"evidence_quote":"Supplies the Multi-Campus Dataset LiDAR trials used to generate 10 of the benchmark sequences."},{"cited_title":"Multi-robot decentralized collabo- rative SLAM in planetary analogue environments: Dataset, challenges, and lessons learned,","cited_arxiv_id":null,"evidence_quote":"Provides real multi-robot communication data that the Wi-Fi connectivity model is fitted to."}],"review_version":1}