{"id":"cc40b1f3-cb6f-4506-9b2d-db5148469d04","arxiv_id":"2505.05076","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A simulation-based LiDAR dataset and a symmetric structural-change metric that together benchmark place recognition under construction and demolition at urban scale.","lead":"Construction and demolition can completely redraw a city skyline, wrecking the visual landmarks robots use to navigate. A new CARLA-based dataset, the City that Never Settles, simulates these extreme changes and shows that current LiDAR place recognition methods degrade sharply as structures change.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II reports different TCR_sym values for opposite traversal directions, contradicting Eq. (1)'s claimed symmetry; the quantitative evidence that CNS surpasses real-world benchmarks is therefore not yet established.","rationale":"I read the paper as a benchmark contribution: a synthetic LiDAR dataset with controlled construction and demolition, a new symmetric change metric, and baseline PR evaluations. The dataset release and the multi-method evaluation are real assets, and the CARLA-based generation is reproducible in principle. My stress-test focuses on the metric because the headline comparison to KAIST and the 'as TCR_sym increases' analysis both depend on a single scalar quantity that is claimed to be ordering-invariant. The direction-dependent entries in Table II are the most concrete point at which that claim fails. I am not alleging anything about the authors' intent; the discrepancy could be an undocumented sampling choice or a labeling error. But it is load-bearing: if the metric is not order-invariant under the protocol actually used, Table I's single values and Fig. 5's x-coordinates are ambiguous. The reader's weakest_assumption concerned sim-to-real transfer, which is a legitimate generalization concern, but I find the internal inconsistency more immediate and more directly testable from the released data. The reader did flag this inconsistency in the rationale, so my agreement is partial. I would keep the verdict conditional: the paper is valuable and the issue is addressable by recomputation and clarification, but the central quantitative comparison should not be accepted as-is.","tokens_in":5702,"tokens_out":8007,"duration_ms":82825,"concrete_test":"Using the released CNS point clouds and code, compute TCR_sym for Riverside 01↔02 and Suburbia 01↔02 under two protocols: (a) identical sampling for both sequences, and (b) the Table II protocol with database at 5 m and query at 10 m. Under (a), Eq. (1) requires equal values for both directions; if they differ, the metric implementation or the published tables are wrong. Under (b), report the values and state explicitly that the metric is order-dependent under the PR sampling, and state which direction Table I's single entry corresponds to.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that CNS exceeds real-world structural change as measured by the new symmetric metric rests on TCR_sym. Eq. (1) defines TCR_sym with unions over both source-target orderings, so for fixed point sets X(S) and X(T) the value must be invariant under swapping S and T. Table II, however, reports direction-dependent TCR_sym values for the same sequence pairs: Riverside 01→02 = 0.0986 but 02→01 = 0.1268, and Suburbia 01→02 = 0.1840 but 02→01 = 0.4699. Table I lists a single value for each pair (Riverside 0.0986, Suburbia 0.1840) that matches the first direction only. This means the published protocol does not realize the symmetry that motivates the metric. The likely cause is the PR preprocessing in Sec. V: database sequences are sampled at 5 m intervals and query sequences at 10 m intervals, so reversing the direction changes the input point cloud densities. In that case the 'symmetric' scalar is role-dependent, and Table I's single values are an arbitrary selection or an average that is not reported. Since TCR_sym is also the x-axis variable for the performance trends in Fig. 5 and Table II, the inconsistency propagates into the main benchmark analysis. The paper's comparative claim against KAIST therefore cannot be considered established until the metric is recomputed under a clearly symmetric protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the City that Never Settles (CNS) dataset, a CARLA-based simulated LiDAR dataset designed to study long-term place recognition under large-scale structural changes such as building construction and demolition. It also proposes TCR_sym, a supposedly symmetric version of the Temporal Change Ratio metric, and uses it to argue that CNS exhibits structural changes at least as large as those found in the real-world KAIST sequence pair. The authors benchmark four LiDAR place-recognition methods (Scan Context++, SOLiD, RING++, and BTC) on CNS and report substantial performance degradation as TCR_sym increases, supporting the need for more robust long-term place-recognition algorithms.","tokens_in":6004,"tokens_out":3464,"duration_ms":35348,"significance":"If the dataset and metric are sound, CNS would fill a real gap: existing benchmarks mostly capture limited or indoor structural changes, while CNS offers a reproducible, publicly available simulation benchmark at urban and suburban scale. The paper's strengths include the release of the dataset, the use of a standard simulator, and the comparison of several established place-recognition methods. However, the central quantitative claim—that CNS surpasses real-world structural change—depends entirely on TCR_sym, and the paper's own tables show that the computed metric is not symmetric as defined. This undermines the headline comparison, although the dataset itself may still be useful for benchmarking once the metric and preprocessing are fixed.","major_comments":[{"comment":"The function defined in Eq. (1) is symmetric by construction, since it takes the union over both source–target orderings. However, Table II reports different TCR_sym values for opposite traversal directions of the same pair: Riverside 01→02 = 0.0986 but 02→01 = 0.1268, and Suburbia 01→02 = 0.1840 but 02→01 = 0.4699. Table I lists only one value per pair (0.0986 for Riverside and 0.1840 for Suburbia), matching the first direction. The likely cause is the asymmetric preprocessing described in Section V, where database sequences are sampled at 5 m intervals and query sequences at 10 m intervals, so swapping the roles changes the input point sets entering Eq. (1). Because the claimed symmetry of TCR_sym is the paper's stated motivation and the Table I comparison against KAIST uses these non-symmetric values, the central claim that CNS encompasses more extensive transformations than real-world benchmarks is not established. Please recompute TCR_sym under a symmetric protocol (e.g., identical sampling density for both sets) or, if the asymmetry is intentional, define and justify an aggregate statistic (such as the maximum or mean over both directions) and update Table I and Fig. 5 accordingly.","section":"Section IV, Eq. (1) and Table II"},{"comment":"The TCR_sym values depend on hand-picked parameters: the 5 m voxel resolution and the nearest-neighbor threshold τ = 4.5 m. No sensitivity analysis is provided. Since TCR_sym is the x-axis in Fig. 5, the quantitative performance trends and the comparison with KAIST in Table I could change substantially under plausible variations of these parameters. The authors should report how TCR_sym varies over a range of voxel sizes and τ values, and confirm that the ranking of CNS against real-world datasets is stable.","section":"Section IV, threshold choices"},{"comment":"The only real-world comparison is the KAIST 01-to-06 pair, which requires aligning MulRan's KAIST 01 with HeLiPR's KAIST 06 using the LT-SLAM module. No alignment error, validation, or sensitivity of the resulting TCR_sym to the alignment quality is reported. The conclusion that CNS 'encompasses more extensive transformations than current real-world benchmarks' therefore rests on a single, potentially noisy cross-dataset measurement. Adding further real-world pairs (e.g., Oxford, Boreas, or other MulRan sequences) with clearly described alignment procedures would materially strengthen the claim. Absent that, the paper should temper the claim and explicitly discuss the uncertainty in the KAIST TCR_sym value.","section":"Section IV, Table I and real-world comparison"}],"minor_comments":[{"comment":"The first contribution bullet contains a grammatical error: 'We provides simulation-based dataset' should be 'We provide a simulation-based dataset.'","section":"Section I, contributions"},{"comment":"The notation 'TCR sym' appears with a line break in the abstract and is sometimes written as 'TCRsym' in the text; please use a consistent notation (e.g., 'TCR_sym').","section":"Abstract and throughout"},{"comment":"The text states that bold and underlined values indicate the best and second-best performance, but the rendered table does not visibly distinguish these formatting marks; please ensure the final version makes this clear.","section":"Section V, Table II"},{"comment":"The statement that database sequences are sampled at 5 m and query sequences at 10 m is directly related to the symmetry issue in TCR_sym; this should be explained and justified in the metric definition section, not only in the benchmark section.","section":"Section V, preprocessing"},{"comment":"The rightmost column of Fig. 4 is labeled with a dash ('-'), which is ambiguous; labeling it explicitly as the union of the two H sets would better illustrate the proposed symmetric construction.","section":"Figure 4"},{"comment":"The caption says 'The upper displays aerial views,' which is grammatically incomplete; it should read 'The upper part displays aerial views' or similar.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The dataset has clear potential value for the long-term place-recognition community, and the evaluation protocol is reproducible. However, the symmetry failure of TCR_sym is not a cosmetic issue: the paper's main claim that CNS exceeds real-world structural change is based on one-directional TCR_sym values. I recommend requiring the authors to recompute the metric under a well-defined symmetric protocol and to strengthen the real-world comparison before the paper is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the CNS dataset is a real resource: four CARLA maps, twelve sequences, simulated construction and demolition, released on GitHub, and evaluated with four standard LiDAR place-recognition methods. Second, the paper's central quantitative claim—that CNS exceeds real-world structural change—is undercut by an inconsistency in the metric protocol. The authors introduce TCR_sym as a symmetric variant of TCR, but in the experiments they sample database sequences at 5 m and query sequences at 10 m. Reversing source and target therefore changes the input point clouds, and Table II reports direction-dependent TCR_sym values for the same pair: Riverside 01→02 gives 0.0986 while 02→01 gives 0.1268; Suburbia 01→02 gives 0.1840 while 02→01 gives 0.4699. Table I lists a single value that matches the first direction only. The paper's Eq. (1) is symmetric for fixed point sets, so the metric itself is fine; the protocol is what breaks the symmetry. As a result, the comparison against the KAIST real-world sequence is not as clean as claimed, and the x-axis of the performance trends in Fig. 5 inherits the same direction dependence.\n\nThe dataset itself is useful and the evaluation is honest in its coverage. The authors do not overclaim about the methods: they report degradation and note that BTC and RING++ are more robust. That gives the paper value as a benchmark.\n\nThe softer spots are secondary. The thresholds (voxel 5 m, τ = 4.5 m, true-positive 7.5 m) are hand-picked with no sensitivity analysis; a benchmark should show that the main trends are robust to those choices. And the claim that simulated changes transfer to real construction is assumed rather than demonstrated—the only real-world anchor is one KAIST pair aligned with LT-SLAM, from the authors' own prior work. That is a limitation worth stating explicitly.\n\nMy recommendation: send it to peer review. The dataset deserves a serious referee. The symmetry issue is addressable—recompute TCR_sym under a common sampling scheme, or report both directions—and the paper would be a solid contribution once that is fixed.","headline":"A genuinely useful benchmark dataset whose central comparative claim currently rests on a metric protocol that breaks the symmetry the authors claim for it.","tokens_in":6473,"tokens_out":3464,"would_cite":true,"duration_ms":35579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simulated-LiDAR city dataset induces structural changes matching a four-year real benchmark, and current place-recognition methods degrade sharply on it.","keywords":["place recognition","LiDAR dataset","CARLA simulation","structural change","Temporal Change Ratio","benchmark","construction and demolition","long-term autonomy"],"falsifier":"Record a real urban sequence pair whose TCR_sym matches Metropolis 01→04 (0.2920) and run the same four methods on it; if their AUC and R@1 drops are much milder than those reported on CNS, the simulation's difficulty does not transfer to reality.","tokens_in":5522,"feed_emoji":"🏗️","tokens_out":9518,"duration_ms":89144,"temperature":0.7,"pith_summary":"The paper introduces the City that Never Settles (CNS), a CARLA-simulated LiDAR dataset in which buildings, trees, and landmarks are deliberately removed across four maps and twelve sequences to emulate large-scale urban construction and demolition. Its central claim is that CNS captures structural change at least as extreme as what a real city undergoes over four years, and that state-of-the-art place-recognition methods lose substantial accuracy on such change. To make this comparison order-independent, the paper defines TCR_sym, a symmetric version of an existing Temporal Change Ratio metric that measures what fraction of a voxelized point cloud has no near neighbor in the other sequence. On CNS, Scan Context++, SOLiD, RING++, and BTC all show accuracy falling as TCR_sym rises, with RING++ most resilient and BTC stable but lower in absolute performance. If correct, the dataset supplies a reproducible stress test for long-term place recognition under structural change, without waiting years for real construction.","feed_headline":"Simulated city beats four-year real-world change in LiDAR test","feed_subtitle":"A new metric and dataset show LiDAR place-recognition methods degrade as city structures change.","key_machinery":"The load-bearing object is TCR_sym, defined for source point cloud $X^{(S)}$ and target $X^{(T)}$ as $$\\mathrm{TCR}_{\\mathrm{sym}} = 1 - \\frac{|O($X^{{(S)}}$,$X^{{(T)}}$;\\tau) \\cup O($X^{{(T)}}$,$X^{{(S)}}$;\\tau)|}{|H($X^{{(S)}}$,$X^{{(T)}}$) \\cup H($X^{{(T)}}$,$X^{{(S)}}$)|},$$ where $O$ is the set of points with a nearest neighbor in the other cloud within $\\tau=4.5$ m after 5 m voxelization, and $H$ is the set of points whose addition does not enlarge the target's convex hull. Taking the union over both orderings removes source-target asymmetry, so demolition and construction are measured on the same scale. The data-generation machinery is the CARLA simulator: a 32-channel LiDAR with a $\\pm 22.5^\\circ$ vertical field of view and 120 m maximum range, an ideal zero-bias IMU, scripted routes, and manual removal of structures in the editor to create sequence pairs. TCR_sym carries the paper's quantitative argument because it is the same ruler applied to both the simulated and the real-world sequence pair.","core_discovery":"The paper's central discovery is that a simulated environment can generate structural change on the same scale as—or beyond—what a real city experiences over years, and that current LiDAR place-recognition methods are not robust to it. For the most heavily modified pair, Metropolis 01→04, TCR_sym is 0.2920, while the real-world comparison pair captured more than four years apart measures 0.1819. On that same pair, Scan Context++ achieves an AUC of 0.4442, SOLiD 0.2582, RING++ 0.7228, and BTC 0.5164, and all four show lower accuracy on the more changed sequences than on the less changed ones. RING++ degrades least, which the paper attributes to its Bird's Eye View encoding of road contours and distances; BTC remains comparatively flat but at lower accuracy. The paper concludes that existing descriptors rely on unchanged structures and need redesign for large-scale construction and demolition.","pith_inferences":["If CNS's difficulty transfers to reality, synthetic construction and demolition data could be used to train or fine-tune place-recognition descriptors before deployment in changing cities; the paper does not test this transfer.","TCR_sym could be decomposed per region or per object class to distinguish how much changed from where changed, which would help diagnose why certain descriptors fail.","A natural extension is to vary viewpoint and route overlap independently of structural change, something real datasets cannot isolate, to separate the effects of pose difference from the effects of building removal.","One could also measure TCR_sym between a simulated and a real scan of the same scene to validate directly whether the edited simulator geometry is distributionally close to real construction sites."],"forward_implications":["CNS provides a reproducible benchmark for construction- and demolition-scale change, letting researchers evaluate place recognition without waiting years for real redevelopment.","The reported degradation implies descriptors that rely on static geometry, such as Scan Context++ and SOLiD, will need mechanisms to ignore or re-weight changed structures.","RING++'s relative resilience suggests road-contour and BEV context is a promising direction for change-tolerant descriptors.","TCR_sym gives the community an order-independent ruler for comparing structural change across datasets, useful for reporting how extreme a benchmark actually is.","The sequence structure of CNS (01→04 construction, 04→01 demolition) supports separate studies of construction versus demolition difficulty."],"supporting_citations":[{"why":"Defines the original Temporal Change Ratio (TCR) and the construction-site benchmark from which TCR_sym is derived; removing it breaks the metric's lineage and comparison.","marker":"[1]"},{"why":"Supplies the CARLA urban driving simulator used to generate all CNS point clouds and sequences.","marker":"[2]"},{"why":"Provides one half of the real-world four-year-apart urban sequence pair used as the real-world change baseline.","marker":"[6]"},{"why":"Provides the other half of that real-world sequence pair, recorded with heterogeneous LiDAR at the same site.","marker":"[7]"},{"why":"Aligns the two real-world sequences into a common frame so their change ratio can be computed against CNS.","marker":"[8]"},{"why":"One of the four state-of-the-art place-recognition baselines evaluated on CNS; supplies the Scan Context++ descriptor results.","marker":"[9]"},{"why":"A second baseline (SOLiD) whose AUC, R@1, and F1 results on CNS document performance degradation.","marker":"[10]"},{"why":"A third baseline (RING++) whose BEV-based descriptor proves most resilient, providing the paper's main comparative evidence.","marker":"[11]"},{"why":"A fourth baseline (BTC) whose submap-based descriptor stays stable but low, informing the discussion of verification logic.","marker":"[12]"}],"fun_headline_variants":["Simulated city out-changes four years of real LiDAR data","New LiDAR dataset simulates major city rebuilds for place recognition","LiDAR place-recognition fails on simulated construction and demolition","TCR_sym metric reveals LiDAR place-recognition chokes on structural change"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that structural change measured by TCR_sym on CARLA point clouds—produced by editing a simulator world—predicts how real construction and demolition will degrade place-recognition algorithms.","fun_headline_variants_meta":{"raw":{"variants":["Simulated city out-changes four years of real LiDAR data","New LiDAR dataset simulates major city rebuilds for place recognition","LiDAR place-recognition fails on simulated construction and demolition","TCR_sym metric reveals LiDAR place-recognition chokes on structural change"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1273,"prompt_tokens":902,"completion_tokens":371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":518,"tokens_out":371,"duration_ms":3851,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:13:12.468994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a real urban sequence pair whose TCR_sym matches Metropolis 01→04 (0.2920) and run the same four methods on it; if their AUC and R@1 drops are much milder than those reported on CNS, the simulation's difficulty does not transfer to reality.","supporting_citations":[{"cited_title":"Helipr: Heterogeneous lidar dataset for inter-lidar place recognition under spatiotemporal variations,","cited_arxiv_id":null,"evidence_quote":"Provides the other half of that real-world sequence pair, recorded with heterogeneous LiDAR at the same site."},{"cited_title":"Lt-mapper: A modular framework for lidar-based lifelong mapping,","cited_arxiv_id":null,"evidence_quote":"Aligns the two real-world sequences into a common frame so their change ratio can be computed against CNS."},{"cited_title":"Scan context++: Structural place recognition robust to rotation and lateral variations in urban environments,","cited_arxiv_id":null,"evidence_quote":"One of the four state-of-the-art place-recognition baselines evaluated on CNS; supplies the Scan Context++ descriptor results."},{"cited_title":"Ring++: Roto-translation invariant gram for global localization on a sparse scan map,","cited_arxiv_id":null,"evidence_quote":"A third baseline (RING++) whose BEV-based descriptor proves most resilient, providing the paper's main comparative evidence."},{"cited_title":"Btc: A binary and triangle combined descriptor for 3-d place recognition,","cited_arxiv_id":null,"evidence_quote":"A fourth baseline (BTC) whose submap-based descriptor stays stable but low, informing the discussion of verification logic."}],"review_version":1}