{"id":"cb043e68-c6b0-49ac-b42a-d73dfe539380","arxiv_id":"2509.07362","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A large-scale dataset and benchmark for localizing ground images against aerial ALS point clouds in three cities, with ground-truth poses generated by MLS-ALS registration.","lead":"New benchmark pairs ground-level city images with aerial laser scan maps in Wuhan, Hong Kong, and San Francisco, and provides 6-DoF poses for the images via a LiDAR-to-aerial alignment pipeline. Tests show most existing image-to-point-cloud localization methods struggle across the aerial-ground gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth accuracy claim is not independently validated: Section 5.2 checkpoints use the same MLS–ALS alignment that the optimization (Eq. 8) produces, so systematic ALS georeferencing errors—larger than the claimed 9–16 cm—are invisible.","rationale":"I agree with the reader's weakest assumption: the checkpoint validation is not an independent ground truth. The concern is load-bearing because the dataset's primary contribution is providing reliable GT for benchmarking; if the ALS georeferencing has a global bias, all GT poses and all benchmark metrics inherit that bias. The paper's own accuracy figures for the ALS maps make this concrete: the reported checkpoint errors are smaller than or comparable to the ALS horizontal accuracy, which is impossible for absolute accuracy unless the ALS is locally far more accurate than its global error. The fix is straightforward: add an external validation with ground control points, or at least compare against the original SPAN-CPT reference trajectories to quantify the shift introduced by ALS alignment. The fine-localization benchmark substitution (Section 6.2) is a secondary concern, but the GT validation issue is more fundamental. Therefore the CONDITIONAL verdict is appropriate; the paper is not rejected but the accuracy claim must be either independently validated or reworded as 'relative to ALS'.","tokens_in":18406,"tokens_out":5128,"duration_ms":53717,"concrete_test":"For at least one sequence (e.g., Wuhan Loop 1), survey 10–20 well-distributed ground control points with a total station or RTK GNSS using a local base station, and compute the 3D residuals between the GCP coordinates and (a) the ALS point cloud and (b) the GT poses derived from the optimized trajectory. If the RMS of GT pose errors relative to the GCPs approaches the reported 0.09 m, the validation is confirmed; if it exceeds ~0.2–0.3 m (consistent with the stated ALS horizontal accuracy), the claimed accuracy is not independent. An alternative computational test is to shift the ALS point cloud by a known 0.3 m global offset and re-run the pose graph optimization and checkpoint evaluation; if the checkpoint errors remain unchanged while the whole trajectory shifts by ~0.3 m, the evaluation is insensitive to absolute georeferencing errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the dataset provides 6-DoF ground-truth poses with 0.09–0.16 m accuracy (Abstract, Table 2). This rests on the checkpoint evaluation in Section 5.2, where manually selected points are aligned between MLS and ALS point clouds after optimization. However, the pose graph optimization already minimizes aerial-ground residuals (Eq. 8) that register MLS submaps to the same ALS point cloud. Thus the checkpoints measure the consistency of that same alignment, not an independent absolute reference. The paper reports ALS horizontal georeferencing accuracy of 0.12 m (SF), 0.2 m (Wuhan), and 0.3 m (HK) (Section 3.3). The claimed GT accuracy (0.09–0.16 m) is at or below these values, so any global ALS shift or bias would propagate into the GT poses and remain undetected by the checkpoint residual. The evaluation therefore only demonstrates local self-consistency between MLS and ALS; it cannot establish absolute localization accuracy in a global frame. Without an external reference (e.g., surveyed ground control points), the benchmark's GT may be systematically biased, which would directly undermine any conclusions drawn from the localization benchmarks in Section 6.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a dataset for aerial-ground cross-modal localization, combining ground-level imagery from mobile mapping systems with airborne laser scanning (ALS) point clouds in Wuhan, Hong Kong, and San Francisco. The ground-truth 6-DoF image poses are generated indirectly: mobile LiDAR (MLS) submaps are aligned to ALS point clouds via ground segmentation, façade reconstruction, and multi-sensor pose-graph optimization that fuses loop-closure, aerial-ground ICP, IMU, GNSS, and odometry factors. The optimized MLS trajectory is then transferred to the camera stream through extrinsic calibration. The authors report average checkpoint errors of 0.09–0.16 m for three evaluation sequences (Table 2) and use the dataset to benchmark three global I2P localization methods (AE-Spherical, LIPLoc, SaliencyI2PLoc) and three fine-registration methods (DeepI2P, CorrI2P, CoFiI2P). LIPLoc performs best on global retrieval; the fine-registration baselines fail, so an SfM+VINS+ICP pipeline is used as an alternative baseline.","tokens_in":18735,"tokens_out":5300,"duration_ms":61173,"significance":"If the ground-truth accuracy claim is trustworthy, this dataset would be a valuable resource for evaluating image-to-point-cloud localization across aerial and ground platforms, a setting that is underrepresented in existing benchmarks. The paper contributes a large-scale multi-city dataset, an indirect alignment pipeline that avoids laborious ground surveying, and a reproducible evaluation of several state-of-the-art methods. The main significance, however, hinges on the reliability of the claimed 9–16 cm pose accuracy. The current validation is not independent of the optimization that generated the poses, so the absolute accuracy of the dataset remains unestablished. If the validation is strengthened or the claim appropriately qualified, the benchmark would still be useful for comparing methods, but the paper's central quantitative claim currently exceeds what the evidence supports.","major_comments":[{"comment":"The quantitative validation of the ground-truth poses is not independent of the optimization that produced them. The checkpoint evaluation in Section 5.2 manually aligns selected feature points between the same MLS and ALS point clouds that define the aerial-ground constraints in Eq. (8). If the ALS frame carries a global translation or rotation error, or if the façade-completion procedure biases the extracted ALS features, both the optimized trajectory and the checkpoint alignment are affected in the same way, so the residuals in Table 2 cannot detect such errors. The reported 0.09–0.16 m errors should therefore be described as self-consistency residuals, not absolute accuracy. To support the abstract's accuracy claim, the authors need an independent reference (surveyed ground control points, total-station measurements, or an independent high-accuracy GNSS/INS trajectory) and a report o","section":"§5.2 and Eq. (8)"},{"comment":"The claimed ground-truth accuracy is below the stated accuracy of the ALS reference. Table 2 reports average checkpoint errors of 0.16 m (California), 0.11 m (Hong Kong), and 0.09 m (Wuhan), while Section 3.3 gives ALS horizontal accuracies of 0.12 m, 0.3 m, and 0.2 m for the same sites. Since the pose-graph optimization references the MLS trajectory to the ALS frame (Eq. 8), and since the GNSS factor (Eq. 12) in dense urban canyons is typically too weak to override a global ALS shift, the absolute pose error cannot in general be smaller than the ALS georeferencing error. The checkpoint metric as defined in Section 5.2 excludes such global shifts. The paper should either provide external absolute measurements or explicitly restate the claimed accuracy as 'relative to the ALS frame' and quantify the propagation of ALS georeferencing uncertainty into the final poses.","section":"§3.3 and Table 2"},{"comment":"The checkpoint evaluation is under-reported. The number of checkpoints per sequence, the selection criteria, the manual alignment procedure, and the operator variability are not given; only average/min/max errors for three sequences are listed in Table 2. With such a small and unquantified sample, a 9–16 cm average error is not a statistically robust accuracy certificate. This matters because Table 2 is the only quantitative support for the central ground-truth claim.","section":"§5.2"}],"minor_comments":[{"comment":"The table is garbled: the column headers repeat 'HK' and the train/evaluation counts for the three datasets are not aligned with the rows. Please reformat it so that the split sizes for California, Hong Kong, and Wuhan are unambiguous.","section":"Table 3"},{"comment":"The phrase 'According to2' (before 'in the dense urban datasets') should be a proper cross-reference to Table 2.","section":"§5.2"},{"comment":"The definition of 'patches of 100 m2' is ambiguous; please specify whether this is 100 m × 100 m or another shape, and clarify how patches overlap between adjacent images.","section":"§3.4"},{"comment":"The fine-localization section reports that all three learning-based methods 'failed to produce reliable correspondences' but then does not report their quantitative results; the SfM+VINS+ICP baseline in Table 6 is then presented as the benchmark. Please make explicit that Table 6 is a baseline sanity check rather than a comparison of the three selected methods.","section":"§6.2"},{"comment":"The AE-Spherical training uses positive/negative distance thresholds of 20 m and 40 m without sensitivity analysis; given the strong effect of these thresholds on place-recognition recall, this is worth a sentence of justification.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the ground-truth validation is the main obstacle. If the authors can obtain even a small set of surveyed control points in a subset of the three cities and re-report errors against that reference, the paper would be substantially stronger. If external validation is impossible, the accuracy claim should be softened to 'consistency relative to the ALS frame' and the benchmark results in Section 6 should be interpreted accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: this is the first dataset I know of that pairs ground-level imagery with public ALS point clouds for 6-DoF localization, and that alone makes it worth having. The GT generation idea—align MLS submaps to ALS via ground segmentation, façade completion, and a pose graph with GNSS/IMU/odometry/loop closure—is a sensible indirect way to avoid direct image-to-ALS registration, and the qualitative projections look convincing.\n\nThe soft spot is exactly where the reader put their finger. Section 5.2 validates the GT by manually aligning checkpoints on the same MLS and ALS clouds that the aerial-ground factors (Eq. 8) were fitting. That tells you the optimization converged to a self-consistent MLS-ALS alignment, not that the poses are accurate in an absolute sense. The reported 9–16 cm errors sit at or below the stated ALS horizontal accuracy (0.12–0.3 m), so a systematic georeferencing shift would be invisible to this check. The GNSS factors in the graph provide some independent anchor, but the paper overstates the claim: 'consistent with ALS within 9–16 cm' is honest, 'ground-truth accuracy' is not. Add surveyed ground control points or at least substantiate the GNSS contribution.\n\nTwo smaller things. The fine-localization 'benchmark' is not really a benchmark: the three learned methods all failed, so the reported numbers are for a substituted SfM+VINS+ICP pipeline. That's a reasonable fallback, but it does not benchmark I2P methods. And the paper has editorial roughness—Table 3 is confusing, and the sequence name '4701' flips to '4706' between prose and Table 4.\n\nOverall, the dataset is the contribution, and it is a good one. The GT validation needs an independent reference before the headline accuracy numbers are taken at face value, but that is addressable. Send it to a venue with serious referees; it deserves their time.","headline":"A genuinely useful ground-image-to-ALS dataset, but the 9–16 cm ground-truth accuracy claim is a self-consistency measure until validated against an independent reference.","tokens_in":19248,"tokens_out":4647,"would_cite":true,"duration_ms":51347,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces a large-scale benchmark that matches ground-level street imagery to airborne laser scanning point clouds in Wuhan, Hong Kong, and San Francisco, using an indirect ground-truth pipeline reported at 9–16 cm checkpoint acc","keywords":["aerial-ground cross-modal localization","airborne laser scanning (ALS)","mobile mapping system","image-to-point-cloud registration","6-DoF ground-truth poses","pose graph optimization","urban localization benchmark","multi-sensor fusion"],"falsifier":"Take one sequence from each city and survey a set of camera locations with an independent instrument not involved in the optimization—total-station ground-control targets, or high-accuracy RTK-GNSS positions collected after the fact—then compare those surveyed poses to the published ground-truth poses. If disagreement substantially exceeds the reported 0.09–0.16 m average, the claimed ground-truth accuracy is not supported; if it matches, the indirect MLS-to-ALS pipeline is validated as an independent check.","tokens_in":18309,"feed_emoji":"🗺️","tokens_out":4093,"duration_ms":51762,"temperature":0.7,"pith_summary":"The authors set out to establish that airborne laser scanning (ALS) data—publicly available over many cities—can serve as a reliable prior map for visual localization, provided researchers have a dataset with trustworthy image poses. Their load-bearing contribution is a new large-scale dataset pairing ground-level mobile-mapping images with ALS point clouds across three dense urban areas. Because direct image-to-ALS registration is too hard, they generate ground-truth image poses indirectly: align vehicle-mounted LiDAR submaps to the ALS via ground segmentation, façade reconstruction, and pose-graph optimization, then transfer the refined trajectory to the camera through a rigid calibration. Checkpoint evaluations report average errors of 0.09–0.16 m. On this benchmark, current image-to-point-cloud localization methods perform poorly, especially the state-of-the-art fine-registration methods, showing that aerial-ground cross-modal localization is a live open problem.","feed_headline":"New benchmark pairs street photos with aerial lidar in three cities","feed_subtitle":"Public dataset supplies 6-DoF image poses at 9–16 cm error and shows today's I2P methods still struggle.","key_machinery":"The load-bearing mechanism is the tight multi-sensor pose graph whose nodes are full LiDAR-frame states (pose, velocity, IMU biases) and whose edges include an aerial-ground constraint: ICP registration of MLS submaps to ALS-derived façade and ground features. That aerial-ground residual, together with loop, IMU, odometry, and GNSS factors, converts the globally referenced ALS into an absolute anchor that corrects drift in the ground trajectory. Compatibility between MLS and ALS is manufactured by extracting planar ground seeds and by projecting roof boundaries onto the ground to complete façades barely visible from above.","core_discovery":"The central claim is that accurate 6-DoF ground-truth poses for ground images relative to ALS point clouds can be obtained without direct image-to-point-cloud registration. The method first registers mobile laser scanning (MLS) submaps to pre-georeferenced ALS data using ground planes and reconstructed building façades as cross-modal features, then solves a multi-sensor pose graph that fuses loop closures, aerial-ground ICP residuals, IMU pre-integration, odometry, and GNSS constraints. The rigid LiDAR-camera mounting then transfers the optimized MLS trajectory to the image stream. Across the three cities, average checkpoint errors are 0.09 m in Wuhan, 0.11 m in Hong Kong, and 0.16 m in San","pith_inferences":["Because the manual checkpoints are selected on the same MLS and ALS clouds that the pose graph already aligned, the reported 9–16 cm figures are best read as consistency checks rather than independent accuracy bounds; true absolute error could be larger if the ALS georeferencing itself carries systematic bias.","The benchmark’s fine-registration failures might not be purely a modality-gap problem: the learning-based baselines were originally trained on denser, structure-rich vehicle LiDAR, so the benchmark could also be used to isolate how much of the failure is due to scene geometry versus platform discrepancy.","A testable extension is change-aware evaluation: several ALS datasets were collected years apart from the ground imagery, so temporal change can be measured per sequence, and descriptors or registration methods could be scored separately in changed versus stable regions.","The ground-truth generation strategy generalizes beyond this dataset: any city with public ALS and a mobile-mapping trajectory could receive the same treatment, making ALS-based localization benchmarks feasible government-data scale rather than research-collection scale."],"forward_implications":["The public dataset gives researchers a standardized way to train and evaluate image-to-point-cloud localization across a genuine aerial-ground platform gap, not just same-platform scenarios.","If the reported checkpoint accuracy holds, the dataset supplies ground-truth image poses at centimeter-to-decimeter level for three dense urban environments, enabling reliable benchmarking and supervision.","Global I2P methods that use projected range images as a proxy bridge the modality gap better than methods consuming raw point clouds, indicating a productive design direction.","Fine-grained I2P registration methods that work on vehicle-borne point clouds do not transfer to ALS-based maps, so the benchmark defines a concrete unsolved subproblem.","The trajectory refinement also improves the underlying MLS point-cloud quality, meaning the dataset can support point-cloud quality assessment beyond localization.","The performance gap between ALS and vehicle-borne input suggests that airborne prior maps remain under-exploited: with better cross-modal features, ALS-based localization could scale to any city with government LiDAR coverage."],"supporting_citations":[{"why":"Supplies the Hong Kong UrbanNav ground sequences (Medium, Deep, Harsh) with LiDAR, camera, IMU, GNSS, and initial reference trajectories that are re-optimized into ground truth.","marker":"(Hsu et al., 2023)"},{"why":"Supplies the San Francisco UrbanLoco ground sequences with a 32-line LiDAR, six cameras, and SPAN-CPT reference trajectories used as raw material for the California subset.","marker":"(Wen et al., 2020)"},{"why":"Supplies the WHU-Helmet helmet-based mobile mapping system used to collect the three Wuhan trajectory loops.","marker":"(Li et al., 2023a)"},{"why":"Provides the public Hong Kong ALS point clouds that serve as the aerial reference map, with stated vertical and horizontal accuracies.","marker":"(CEDD, Hong Kong, 2020)"},{"why":"Provides the San Francisco ALS point clouds used as the aerial reference, with reported checkpoint-based accuracy figures.","marker":"(USGS, U.S.A., 2023)"},{"why":"Defines the planarity feature used to identify initial ground plane seeds and building roof points in the MLS and ALS feature extraction.","marker":"(Weinmann et al., 2015)"},{"why":"Defines the verticality feature used to distinguish building façades and roofs from other non-ground points for cross-modal correspondence.","marker":"(Demantké et al., 2012)"},{"why":"Provides CoFiI2P, one of the three fine I2P registration baselines retrained and evaluated on the benchmark.","marker":"(Kang et al., 2024)"},{"why":"Provides SaliencyI2PLoc, a global I2P localization baseline whose saliency-guided contrastive architecture is the strongest raw-point-cloud baseline in the benchmark.","marker":"(Li et al., 2025b)"},{"why":"Provides DeepI2P, the classification-based fine I2P registration baseline that fails under the aerial-ground cross-modal setting.","marker":"(Li and Lee, 2021)"}],"fun_headline_variants":["Cross-modal localization benchmark: ground truth without registration","New dataset pairs street photos with aerial lidar in 3 cities","Aerial-ground localization benchmark: I2P methods still fail","6-DoF ground truth for image-to-point-cloud without direct registration"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that manual checkpoint alignment on the same MLS and ALS clouds used to build the pose-graph constraints gives an independent accuracy estimate; if that alignment or the underlying ALS georeferencing is biased, the claimed 9–16 cm ground-truth error inherits the bias.","fun_headline_variants_meta":{"raw":{"variants":["Cross-modal localization benchmark: ground truth without registration","New dataset pairs street photos with aerial lidar in 3 cities","Aerial-ground localization benchmark: I2P methods still fail","6-DoF ground truth for image-to-point-cloud without direct registration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1060,"prompt_tokens":721,"completion_tokens":339,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":465,"completion_tokens_details":{"reasoning_tokens":268}},"tokens_in":465,"tokens_out":339,"duration_ms":4791,"temperature":1.0,"reasoning_tokens":268,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:18:54.090227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one sequence from each city and survey a set of camera locations with an independent instrument not involved in the optimization—total-station ground-control targets, or high-accuracy RTK-GNSS positions collected after the fact—then compare those surveyed poses to the published ground-truth poses. If disagreement substantially exceeds the reported 0.09–0.16 m average, the claimed ground-truth accuracy is not supported; if it matches, the indirect MLS-to-ALS pipeline is validated as an independent check.","supporting_citations":[{"cited_title":", author Huang, F","cited_arxiv_id":null,"evidence_quote":"Supplies the Hong Kong UrbanNav ground sequences (Medium, Deep, Harsh) with LiDAR, camera, IMU, GNSS, and initial reference trajectories that are re-optimized into ground truth."},{"cited_title":", author Zhou, Y","cited_arxiv_id":null,"evidence_quote":"Supplies the San Francisco UrbanLoco ground sequences with a 32-line LiDAR, six cameras, and SPAN-CPT reference trajectories used as raw material for the California subset."},{"cited_title":"title Hong Kong LiDAR Data","cited_arxiv_id":null,"evidence_quote":"Provides the public Hong Kong ALS point clouds that serve as the aerial reference map, with stated vertical and horizontal accuracies."},{"cited_title":", year 2023","cited_arxiv_id":null,"evidence_quote":"Provides the San Francisco ALS point clouds used as the aerial reference, with reported checkpoint-based accuracy figures."},{"cited_title":", author Jutzi, B","cited_arxiv_id":null,"evidence_quote":"Defines the planarity feature used to identify initial ground plane seeds and building roof points in the MLS and ALS feature extraction."},{"cited_title":", author Liao, Y","cited_arxiv_id":null,"evidence_quote":"Provides CoFiI2P, one of the three fine I2P registration baselines retrained and evaluated on the benchmark."}],"review_version":1}