Pith. sign in

REVIEW 3 major objections 6 minor 12 references

The City that Never Settles: Simulation-based LiDAR Dataset for Long-Term Place Recognition Under Extreme Structural Changes

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A simulated-LiDAR city dataset induces structural changes matching a four-year real benchmark, and current place-recognition methods degrade sharply on it.

desk verdict A genuinely useful benchmark dataset whose central comparative claim currently rests on a metric protocol that breaks the symmetry the authors claim for it. read the letter →

arxiv 2505.05076 v1 pith:VL3NLVQW submitted 2025-05-08 cs.RO cs.CV

Hyunho Song , Dongjae Lee , Seunghun Oh , Minwoo Jung , Ayoung Kim This is my paper · ORCID
classification cs.ROcs.CV
keywords placerecognitionLiDARdatasetCARLAsimulationstructuralchangeTemporalRatiobenchmarkconstructionanddemolitionlong-termautonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the City that Never Settles (CNS), a CARLA-simulated LiDAR dataset in which buildings, trees, and landmarks are deliberately removed across four maps and twelve sequences to emulate large-scale urban construction and demolition. Its central claim is that CNS captures structural change at least as extreme as what a real city undergoes over four years, and that state-of-the-art place-recognition methods lose substantial accuracy on such change. To make this comparison order-independent, the paper defines TCR_sym, a symmetric version of an existing Temporal Change Ratio metric that measures what fraction of a voxelized point cloud has no near neighbor in the other sequence. On CNS, Scan Context++, SOLiD, RING++, and BTC all show accuracy falling as TCR_sym rises, with RING++ most resilient and BTC stable but lower in absolute performance. If correct, the dataset supplies a reproducible stress test for long-term place recognition under structural change, without waiting years for real construction.

What carries the argument

The load-bearing object is TCR_sym, defined for source point cloud $X^{(S)}$ and target $X^{(T)}$ as $$\mathrm{TCR}_{\mathrm{sym}} = 1 - \frac{|O($X^{{(S)}}$,$X^{{(T)}}$;\tau) \cup O($X^{{(T)}}$,$X^{{(S)}}$;\tau)|}{|H($X^{{(S)}}$,$X^{{(T)}}$) \cup H($X^{{(T)}}$,$X^{{(S)}}$)|},$$ where $O$ is the set of points with a nearest neighbor in the other cloud within $\tau=4.5$ m after 5 m voxelization, and $H$ is the set of points whose addition does not enlarge the target's convex hull. Taking the union over both orderings removes source-target asymmetry, so demolition and construction are measured on the same scale. The data-generation machinery is the CARLA simulator: a 32-channel LiDAR with a $\pm 22.5^\circ$ vertical field of view and 120 m maximum range, an ideal zero-bias IMU, scripted routes, and manual removal of structures in the editor to create sequence pairs. TCR_sym carries the paper's quantitative argument because it is the same ruler applied to both the simulated and the real-world sequence pair.

What would settle it

Record a real urban sequence pair whose TCR_sym matches Metropolis 01→04 (0.2920) and run the same four methods on it; if their AUC and R@1 drops are much milder than those reported on CNS, the simulation's difficulty does not transfer to reality.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a simulated environment can generate structural change on the same scale as—or beyond—what a real city experiences over years, and that current LiDAR place-recognition methods are not robust to it. For the most heavily modified pair, Metropolis 01→04, TCR_sym is 0.2920, while the real-world comparison pair captured more than four years apart measures 0.1819. On that same pair, Scan Context++ achieves an AUC of 0.4442, SOLiD 0.2582, RING++ 0.7228, and BTC 0.5164, and all four show lower accuracy on the more changed sequences than on the less changed ones. RING++ degrades least, which the paper attributes to its Bird's Eye View encoding of road contours and distances; BTC remains comparatively flat but at lower accuracy. The paper concludes that existing descriptors rely on unchanged structures and need redesign for large-scale construction and demolition.

Load-bearing premise

The assumption that structural change measured by TCR_sym on CARLA point clouds—produced by editing a simulator world—predicts how real construction and demolition will degrade place-recognition algorithms.

Editorial extensions

If this is right

  • CNS provides a reproducible benchmark for construction- and demolition-scale change, letting researchers evaluate place recognition without waiting years for real redevelopment.
  • The reported degradation implies descriptors that rely on static geometry, such as Scan Context++ and SOLiD, will need mechanisms to ignore or re-weight changed structures.
  • RING++'s relative resilience suggests road-contour and BEV context is a promising direction for change-tolerant descriptors.
  • TCR_sym gives the community an order-independent ruler for comparing structural change across datasets, useful for reporting how extreme a benchmark actually is.
  • The sequence structure of CNS (01→04 construction, 04→01 demolition) supports separate studies of construction versus demolition difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If CNS's difficulty transfers to reality, synthetic construction and demolition data could be used to train or fine-tune place-recognition descriptors before deployment in changing cities; the paper does not test this transfer.
  • TCR_sym could be decomposed per region or per object class to distinguish how much changed from where changed, which would help diagnose why certain descriptors fail.
  • A natural extension is to vary viewpoint and route overlap independently of structural change, something real datasets cannot isolate, to separate the effects of pose difference from the effects of building removal.
  • One could also measure TCR_sym between a simulated and a real scan of the same scene to validate directly whether the edited simulator geometry is distributionally close to real construction sites.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the City that Never Settles (CNS) dataset, a CARLA-based simulated LiDAR dataset designed to study long-term place recognition under large-scale structural changes such as building construction and demolition. It also proposes TCR_sym, a supposedly symmetric version of the Temporal Change Ratio metric, and uses it to argue that CNS exhibits structural changes at least as large as those found in the real-world KAIST sequence pair. The authors benchmark four LiDAR place-recognition methods (Scan Context++, SOLiD, RING++, and BTC) on CNS and report substantial performance degradation as TCR_sym increases, supporting the need for more robust long-term place-recognition algorithms.

Significance. If the dataset and metric are sound, CNS would fill a real gap: existing benchmarks mostly capture limited or indoor structural changes, while CNS offers a reproducible, publicly available simulation benchmark at urban and suburban scale. The paper's strengths include the release of the dataset, the use of a standard simulator, and the comparison of several established place-recognition methods. However, the central quantitative claim—that CNS surpasses real-world structural change—depends entirely on TCR_sym, and the paper's own tables show that the computed metric is not symmetric as defined. This undermines the headline comparison, although the dataset itself may still be useful for benchmarking once the metric and preprocessing are fixed.

major comments (3)
  1. [Section IV, Eq. (1) and Table II] The function defined in Eq. (1) is symmetric by construction, since it takes the union over both source–target orderings. However, Table II reports different TCR_sym values for opposite traversal directions of the same pair: Riverside 01→02 = 0.0986 but 02→01 = 0.1268, and Suburbia 01→02 = 0.1840 but 02→01 = 0.4699. Table I lists only one value per pair (0.0986 for Riverside and 0.1840 for Suburbia), matching the first direction. The likely cause is the asymmetric preprocessing described in Section V, where database sequences are sampled at 5 m intervals and query sequences at 10 m intervals, so swapping the roles changes the input point sets entering Eq. (1). Because the claimed symmetry of TCR_sym is the paper's stated motivation and the Table I comparison against KAIST uses these non-symmetric values, the central claim that CNS encompasses more extensive transformations than real-world benchmarks is not established. Please recompute TCR_sym under a symmetric protocol (e.g., identical sampling density for both sets) or, if the asymmetry is intentional, define and justify an aggregate statistic (such as the maximum or mean over both directions) and update Table I and Fig. 5 accordingly.
  2. [Section IV, threshold choices] The TCR_sym values depend on hand-picked parameters: the 5 m voxel resolution and the nearest-neighbor threshold τ = 4.5 m. No sensitivity analysis is provided. Since TCR_sym is the x-axis in Fig. 5, the quantitative performance trends and the comparison with KAIST in Table I could change substantially under plausible variations of these parameters. The authors should report how TCR_sym varies over a range of voxel sizes and τ values, and confirm that the ranking of CNS against real-world datasets is stable.
  3. [Section IV, Table I and real-world comparison] The only real-world comparison is the KAIST 01-to-06 pair, which requires aligning MulRan's KAIST 01 with HeLiPR's KAIST 06 using the LT-SLAM module. No alignment error, validation, or sensitivity of the resulting TCR_sym to the alignment quality is reported. The conclusion that CNS 'encompasses more extensive transformations than current real-world benchmarks' therefore rests on a single, potentially noisy cross-dataset measurement. Adding further real-world pairs (e.g., Oxford, Boreas, or other MulRan sequences) with clearly described alignment procedures would materially strengthen the claim. Absent that, the paper should temper the claim and explicitly discuss the uncertainty in the KAIST TCR_sym value.
minor comments (6)
  1. [Section I, contributions] The first contribution bullet contains a grammatical error: 'We provides simulation-based dataset' should be 'We provide a simulation-based dataset.'
  2. [Abstract and throughout] The notation 'TCR sym' appears with a line break in the abstract and is sometimes written as 'TCRsym' in the text; please use a consistent notation (e.g., 'TCR_sym').
  3. [Section V, Table II] The text states that bold and underlined values indicate the best and second-best performance, but the rendered table does not visibly distinguish these formatting marks; please ensure the final version makes this clear.
  4. [Section V, preprocessing] The statement that database sequences are sampled at 5 m and query sequences at 10 m is directly related to the symmetry issue in TCR_sym; this should be explained and justified in the metric definition section, not only in the benchmark section.
  5. [Figure 4] The rightmost column of Fig. 4 is labeled with a dash ('-'), which is ambiguous; labeling it explicitly as the union of the two H sets would better illustrate the proposed symmetric construction.
  6. [Figure 3 caption] The caption says 'The upper displays aerial views,' which is grammatically incomplete; it should read 'The upper part displays aerial views' or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the CNS benchmark and PR evaluations rest on external methods and independent metrics; the TCR_sym symmetry mismatch is a correctness/consistency issue, not a reduction to inputs.

full rationale

The paper's main empirical result—that LiDAR PR methods degrade on the CNS dataset—is self-contained: it reports AUC/R@N/F1 from external algorithms (Scan Context++, SOLiD, RING++, BTC) on simulator-generated point clouds, with no fitted parameter or self-citation forcing the outcome. The dataset's structural changes are produced by explicit CARLA/Unreal edits, and the PR evaluation is independent of the TCR_sym metric. The quantitative claim that CNS exceeds real-world change uses the authors' proposed TCR_sym and the authors' prior LT-mapper/KAIST comparison; this introduces self-referential flavor but not a logical circle, because the KAIST numbers are measured on public real-world data. One genuine issue is that Eq. (1) defines TCR_sym as order-invariant via unions over both orderings, yet Table II reports different values for opposite directions (Riverside 01->02 = 0.0986 vs 02->01 = 0.1268; Suburbia 01->02 = 0.1840 vs 02->01 = 0.4699). This likely stems from the 10 m vs 5 m query/database sampling in Sec. V, meaning the tabulated quantity is not the symmetric function in Eq. (1). That inconsistency undermines the reliability of the Table I comparison, but it is an implementation/validation flaw rather than a circular derivation: the conclusion is not equivalent to its inputs by construction. Under the stated circularity criteria, no load-bearing step reduces to a fitted parameter, a self-citation chain, or a definitional identity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's contributions are a dataset and a metric, and the analysis rests on several assumptions about simulator realism and metric validity.

free parameters (3)
  • Voxel resolution = 5 m
    Point clouds are voxelized at 5 m to focus on structural changes; no sensitivity analysis provided.
  • Nearest-neighbor threshold tau = 4.5 m
    Set to 4.5 m in TCR_sym to classify unchanged points; choice is not justified or varied.
  • True-positive distance threshold = 7.5 m
    For PR evaluation, a predicted match is a true positive if within 7.5 m of the query location; threshold is arbitrary.
assumptions (4)
  • domain assumption CARLA sensor model produces LiDAR scans representative enough of real urban LiDAR for PR benchmarking.
    The whole dataset is built on the premise that simulator data transfers to real-world performance gaps.
  • domain assumption Deleting buildings and trees in Unreal Editor approximates real construction and demolition.
    The structural changes are manual deletions, not gradual construction phases with scaffolding and debris.
  • domain assumption Convex-hull-based O and H sets are a valid way to measure structural change.
    TCR_sym relies on convex hulls of entire scans, but no validation against human-labeled change or other metrics is given.
  • domain assumption LT-SLAM alignment of MulRan KAIST and HeLiPR KAIST is accurate enough for TCR_sym comparison.
    The only real-world baseline is aligned via this method; alignment errors would change the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The City that Never Settles: Simulation-based LiDAR Dataset for Long-Term Place Recognition Under Extreme Structural Changes." pith.science (2026). https://pith.science/paper/VL3NLVQW

@misc{pith2026250505076,
  author       = {Pith},
  title        = {Pith review of: The City that Never Settles: Simulation-based LiDAR Dataset for Long-Term Place Recognition Under Extreme Structural Changes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VL3NLVQW}},
  note         = {Machine review of arXiv:2505.05076}
}
read the original abstract

Large-scale construction and demolition significantly challenge long-term place recognition (PR) by drastically reshaping urban and suburban environments. Existing datasets predominantly reflect limited or indoor-focused changes, failing to adequately represent extensive outdoor transformations. To bridge this gap, we introduce the City that Never Settles (CNS) dataset, a simulation-based dataset created using the CARLA simulator, capturing major structural changes-such as building construction and demolition-across diverse maps and sequences. Additionally, we propose TCR_sym, a symmetric version of the original TCR metric, enabling consistent measurement of structural changes irrespective of source-target ordering. Quantitative comparisons demonstrate that CNS encompasses more extensive transformations than current real-world benchmarks. Evaluations of state-of-the-art LiDAR-based PR methods on CNS reveal substantial performance degradation, underscoring the need for robust algorithms capable of handling significant environmental changes. Our dataset is available at https://github.com/Hyunho111/CNS_dataset.

Figures

Figures reproduced from arXiv: 2505.05076 by the authors.

Figure 1
Figure 1. Structural changes over time in Metropolis. The upper part of the image shows how the city evolves over time through point cloud maps, where gray points represent unchanged areas and red points indicate structural changes. The lower part of the image illustrates two scenes captured at the same location, demonstrating how buildings are constructed over time, resulting in different observations. Abstract— Large-scale … view at source ↗
Figure 2
Figure 2. System Configuration Riverside Suburbia Metropolis Downtown 400m 200m 200m 200m [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The upper displays aerial views with trajectories over￾laid for Riverside 02, Suburbia 02, Downtown 04, and Metropolis 04. The below shows all the routes on each map, with red being the starting point and blue being the ending point. LiDAR. We employ a 32-channel LiDAR sensor, modeled after the Ouster OS1 design, mounted on the roof of a simulated vehicle. The vertical field of view is set to cover ±22.5 ◦ , with a … view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: AUC for each baseline as time intervals increase HeLiPR’s KAIST 06 sequences), aligned using the LT￾SLAM module from LT-mapper [8]. Notably, our dataset exhibits TCRsym values that meet or even surpass the changes observed in the KAIST sequences. This highlights the ca…
Figure 6
Figure 6. Figure 6: Precision–Recall curves of each baseline for Metropolis ability to handle significant transformations effectively. In contrast, RING++ stands out with superior performance compared to the other methods. This can be explained by its use of Bird’s Eye View (BEV) images, …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 6 canonical work pages

  1. [1]

    Nothing stands still: A spatiotemporal benchmark on 3d point cloud registration under large geometric and temporal change,

    T. Sun, Y . Hao, S. Huang, S. Savarese, K. Schindler, M. Pollefeys, and I. Armeni, “Nothing stands still: A spatiotemporal benchmark on 3d point cloud registration under large geometric and temporal change,” 2025. [Online]. Available: https://arxiv.org/abs/2311.09346

    arXiv 2025

  2. [2]

    Carla: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning. PMLR, 2017, pp. 1–16

    2017

  3. [3]

    University of michigan north campus long-term vision and lidar dataset,

    N. Carlevaris-Bianco, A. K. Ushani, and R. M. Eustice, “University of michigan north campus long-term vision and lidar dataset,” Intl. J. of Robot. Research , vol. 35, no. 9, pp. 1023–1035, 2016

    2016

  4. [4]

    1 year, 1000 km: The oxford robotcar dataset,

    W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 year, 1000 km: The oxford robotcar dataset,” Intl. J. of Robot. Research, vol. 36, no. 1, pp. 3–15, 2017

    work page 2017

  5. [5]

    Boreas: A multi-season autonomous driving dataset,

    K. Burnett, D. J. Yoon, Y . Wu, A. Z. Li, H. Zhang, S. Lu, J. Qian, W.-K. Tseng, A. Lambert, K. Y . Leunget al., “Boreas: A multi-season autonomous driving dataset,” Intl. J. of Robot. Research, vol. 42, no. 1-2, pp. 33–42, 2023

    2023

  6. [6]

    Mulran: Multimodal range dataset for urban place recognition,

    G. Kim, Y . S. Park, Y . Cho, J. Jeong, and A. Kim, “Mulran: Multimodal range dataset for urban place recognition,” in Proc. IEEE Intl. Conf. on Robot. and Automat. IEEE, 2020, pp. 6246–6253

    2020

  7. [7]

    Helipr: Heterogeneous lidar dataset for inter-lidar place recognition under spatiotemporal variations,

    M. Jung, W. Yang, D. Lee, H. Gil, G. Kim, and A. Kim, “Helipr: Heterogeneous lidar dataset for inter-lidar place recognition under spatiotemporal variations,” Intl. J. of Robot. Research, vol. 43, no. 12, pp. 1867–1883, 2024

    work page 2024

  8. [8]

    Lt-mapper: A modular framework for lidar-based lifelong mapping,

    G. Kim and A. Kim, “Lt-mapper: A modular framework for lidar-based lifelong mapping,” in Proc. IEEE Intl. Conf. on Robot. and Automat. , 2022, pp. 7995–8002

    work page 2022

Show all 12 references
  1. [9]

    Scan context++: Structural place recognition robust to rotation and lateral variations in urban environments,

    G. Kim, S. Choi, and A. Kim, “Scan context++: Structural place recognition robust to rotation and lateral variations in urban environments,” IEEE Trans. Robot. and Automat. , vol. 38, no. 3, pp. 1856–1874, 2021

    2021

  2. [10]

    Narrowing your fov with solid: Spatially organized and lightweight global descriptor for fov-constrained lidar place recognition,

    H. Kim, J. Choi, T. Sim, G. Kim, and Y . Cho, “Narrowing your fov with solid: Spatially organized and lightweight global descriptor for fov-constrained lidar place recognition,” IEEE Robot. and Automat. Lett. , 2024

    2024

  3. [11]

    Ring++: Roto-translation invariant gram for global localization on a sparse scan map,

    X. Xu, S. Lu, J. Wu, H. Lu, Q. Zhu, Y . Liao, R. Xiong, and Y . Wang, “Ring++: Roto-translation invariant gram for global localization on a sparse scan map,” IEEE Trans. Robot. and Automat., vol. 39, no. 6, pp. 4616–4635, 2023

    2023

  4. [12]

    Btc: A binary and triangle combined descriptor for 3-d place recognition,

    C. Yuan, J. Lin, Z. Liu, H. Wei, X. Hong, and F. Zhang, “Btc: A binary and triangle combined descriptor for 3-d place recognition,” IEEE Trans. Robot. and Automat. , vol. 40, pp. 1580–1599, 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.