{"id":"97919168-5440-402d-8f2a-f1b235e64f22","arxiv_id":"2505.11046","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Street view coverage deviates from uniform road coverage in most of the 28 studied cities, and binary coverage percentages do not reveal these distributional biases.","lead":"Street view imagery is not collected evenly across cities, even in densely covered ones, and common coverage statistics hide the problem. The authors measured this uneven distribution in 28 cities and show how it can bias AI systems trained on street view data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GSV retrieval step likely returns only the closest panorama per grid query, so C_Real is a density-dependent thinning of true coverage and the quantitative bias metrics rest on an unvalidated sample.","rationale":"The reader's conditional verdict is appropriate, but I locate the pivotal weakness earlier in the pipeline. The uniform-prior concern is real, though the prior is explicitly motivated by the fixed-interval sampling assumption in [25,37], so it is defensible as a benchmark. The GSV metadata retrieval, in contrast, is an unvalidated sampling step. If each grid query returns only one panorama, the density estimate is thinned by a factor that depends on local density. The paper uses this density to compute KL/EMD, to rank cities, and to produce C_delta maps; a density-dependent thinning would compress high-density areas, making cities look more uniform than they are. The qualitative Amsterdam case study is unaffected, and the general point that coverage is not uniform likely survives, but the quantitative support for the central claim would need to be redone. The condition for acceptance should therefore be an audit of the retrieval step (or code release showing it retrieves all panorama IDs), not only a stress test of the uniform prior. This keeps the verdict at CONDITIONAL.","tokens_in":22968,"tokens_out":9406,"duration_ms":98932,"concrete_test":"Select a test area in one GSV city (e.g., Los Angeles) and obtain a ground-truth list of all panorama IDs from Google's internal endpoints or the Street View Publish API. Emulate the paper's 20m grid with 100m radius and record the unique returned IDs. Compute the fraction of ground-truth IDs returned and check whether the missing fraction is larger in areas with high local panorama density. If the fraction is significantly below 1 or density-dependent, the reported EMD/KL and C_delta results must be recomputed from an unthinned retrieval, or the claims restricted to coverage presence rather than distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is not the uniform prior itself but that the density estimate C_Real faithfully represents the distribution of all available street view imagery. Section 3.1 states that for Google Street View, a 20m grid is overlaid and 'API requests are then made at every point for the closest images within 100m.' The standard Google Street View Static API metadata endpoint, which is the approach used in the cited [17], returns a single panorama per request. Consequently, for each 20m grid cell only the nearest panorama ID is retrieved; other panoramas at the same or nearby location are invisible unless they are the closest to some other grid point. Repeated captures, which are exactly the oversampling the paper wants to measure (e.g., the 12-14 vs 4-5 images per street reported in Figure 6), are therefore collapsed toward one observation per ~20m interval. C_Real is not the PDF of all panoramas; it is a spatially thinned subset whose density is capped by the query grid. Because the thinning rate depends on local panorama density, the EMD/KL values in Tables 1/3 and the C_delta maps systematically compress the very biases they claim to quantify. The paper filters duplicate IDs but never checks for IDs that are never returned. This is an internal validity problem for the central quantitative claim, independent of whether the uniform prior is the right fairness benchmark.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method to evaluate the spatial distribution of street view imagery coverage relative to a uniform road-coverage prior, using the KL divergence and Earth Mover's Distance (approximated by Sinkhorn divergence) to quantify deviations. The authors apply this method to 28 cities for Google Street View and Mapillary, compare the distributional metrics with binary coverage percentages, and present a qualitative case study of Amsterdam based on six semi-structured interviews with people involved in the municipal panorama collection process. The central claim is that street view data shows distributional biases even in densely covered cities, that these biases are not captured by binary coverage measures, and that they trace to human idiosyncrasies in the collection process. The paper concludes that datasets sampled from street view services inherit these distributional biases and that domain knowledge is needed when constructing AI datasets.","tokens_in":23255,"tokens_out":4922,"duration_ms":50019,"significance":"If the quantitative results were fully valid, this paper would make an important contribution to the growing literature on street view data bias, moving the discussion beyond binary coverage to distributional coverage. The proposed pipeline (metadata retrieval, density estimation, distribution comparison) is simple and potentially reusable, and the interview-based case study is a valuable and relatively rare source of evidence linking collection practices to measurable spatial artifacts. The paper is clearly organized and the core intuition is compelling. However, the quantitative claims are currently compromised by a likely retrieval-induced thinning of the Google Street View data, by the absence of reported hyperparameters and uncertainty estimates for the metrics, and by an untested sensitivity to the uniform road prior. These issues bear directly on the rankings and C_delta maps that form the quantitative backbone of the paper.","major_comments":[{"comment":"The Google Street View metadata retrieval procedure is likely to return only the closest panorama per 20m grid cell, because the standard GSV metadata endpoint returns a single result per request. The paper states that 'API requests are then made at every point for the closest images within 100m' and that duplicate panorama IDs are filtered, but it does not account for panoramas that are never returned because they are not the closest panorama to any grid point. This means C_Real is a density-dependent thinning of the full panorama set, not an estimate of the true density of all available imagery. Repeated captures, which are exactly the oversampling the paper aims to measure (e.g., the 12-14 vs 4-5 images per street in Figure 6), are collapsed unless they are the closest to separate grid points. The thinning is more severe in high-density areas, so the KL/EMD values in Tables 1 and 3 and the C_delta maps systematically compress the very biases they claim to quantify. The authors must either show that their retrieval retrieves all panoramas (e.g., by validating against a known city with dense repeated captures or by using a pagination/offset endpoint), or explicitly model and bound the effect of this censoring on the reported metrics.","section":"Section 3.1"},{"comment":"The paper does not report the Gaussian KDE bandwidth used for density estimation, the value of k used in the k-nearest-neighbor KL divergence estimator, or the Sinkhorn regularization parameter and iteration count used to approximate the EMD. These are free parameters that materially affect the values in Tables 1 and 3, the ranking of cities, and the C_delta maps. Without them, the central quantitative results cannot be reproduced, and no statement is made about how sensitive the rankings are to these choices. The authors should report all hyperparameter values and include a sensitivity analysis over a reasonable range, together with bootstrap or other uncertainty estimates for the KL/EMD scores and their ranking.","section":"Sections 3.2 and 3.3"},{"comment":"The uniform road-coverage prior assumes that every road segment should receive equal image density per unit length. The paper acknowledges in Section 6.1 that OpenStreetMap road classifications have limitations, but it never tests how sensitive the KL/EMD deviations and city rankings are to this central modeling choice. If multi-lane divided roads, slow-moving segments, or certain OSM road classes are systematically imaged more often for mechanical reasons, then part of the measured 'bias' reflects the mismatch between the prior and real collection mechanics rather than human idiosyncrasy. As a concrete test, the authors should reweight OSM road classes by functional class or road type, or use an alternative prior (e.g., one based on road length weighted by expected capture frequency), and show whether the main conclusions and the ranking of cities are preserved.","section":"Sections 3.2 and 6.1"},{"comment":"The MANOVA significance scores reported in Tables 5-8 are effectively all zero because of the very large sample sizes and thus provide no discriminating information about whether the observed distributional differences are meaningful beyond statistical noise. The authors should instead report effect sizes (e.g., standardized mean differences or overlap coefficients) or bootstrap confidence intervals for the KL and EMD differences between C_Real and C_Uniform, which would give a more honest sense of the strength of the deviations.","section":"Section 4.1"}],"minor_comments":[{"comment":"The formula for W_p is written as an infimum over permutations of n elements, which is only valid for equal-size empirical measures; the notation is also ambiguous about the value of p and the placement of the exponent. Please clarify whether p=1 or p=2 is used and give the exact Sinkhorn divergence formula used in the experiments.","section":"Section 3.3"},{"comment":"The phrase 'squares of400m^2 meters' is ambiguous; presumably the authors mean 400m x 400m squares, but the notation suggests 400 square meters. Please correct the unit description.","section":"Section 3.1"},{"comment":"There is a typo: 'which is is labour-intensive' should be 'which is labour-intensive'.","section":"Section 2.1"},{"comment":"The interpretations of the KL and EMD (e.g., 'A low KL indicates all roads have been covered' and 'A low EMD indicates all parts of the city are being covered equally') are not formally derived from the definitions. Please provide a brief justification or separate these descriptive statements from formal interpretation.","section":"Section 4.1"},{"comment":"The paper reports R^2 = 0.07 for the Mapillary EMD-coverage relationship but does not give Pearson or Spearman correlation coefficients, nor the information needed to assess whether the linear fit is even appropriate. Please report the underlying correlation coefficients and, if relevant, note whether the R^2 comes from a linear or nonlinear fit.","section":"Section 4.2"},{"comment":"The word 'idiosyncracy' is consistently misspelled; the correct spelling is 'idiosyncrasy'.","section":"Abstract and throughout"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the GSV metadata retrieval thinning issue in Section 3.1, which is load-bearing for the quantitative rankings and C_delta maps. The qualitative interview study and the general framing are valuable, and the paper is potentially salvageable if the authors can empirically validate the completeness of their retrieval or quantitatively bound the censoring effect. The lack of hyperparameter reporting and prior-sensitivity analysis also needs to be addressed before the quantitative claims can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper does two genuinely useful things. It extends street view coverage research from binary “covered/not covered” to distributional bias at scale: 28 cities, automated KL/EMD comparisons against a uniform road prior, with C_delta maps that actually reveal neighborhood-level oversampling patterns. That’s a real advance on Smith et al.’s two-neighborhood manual audit, and the method is simple enough to reuse. Second, the Amsterdam interview case study is worth reading on its own: six interviews across drivers, collection managers, and data managers show how route choices and emotional state create systematic oversampling in the city center. That is new, honest, and clearly separated from the quantitative claims.\n\nThe soft spots are real. The stress-test note is correct: the GSV retrieval step asks for “the closest images within 100m” at each 20m grid point, and if the API returns only the single nearest panorama per request, then C_Real is a density-dependent thinning of all available imagery. Repeated captures, the exact oversampling the paper wants to measure, get collapsed into one ID unless multiple panoramas happen to be nearest to different grid points. The paper filters duplicate IDs but never checks for panoramas that are never returned. That means the EMD/KL rankings in Tables 1 and 3 likely compress the very biases they quantify. The uniform road prior is a reasonable normative benchmark, and the paper notes OSM limitations, but it never tests sensitivity to that prior. Add to that the missing hyperparameters (KDE bandwidth, k for the KL estimator, Sinkhorn epsilon), no code or data release, and no uncertainty estimates on the rankings. None of these are fatal to the qualitative conclusion, but they make the quantitative rankings provisional.\n\nThe paper is otherwise careful. It flags its own limitations on OSM boundaries, gated communities, and low-coverage cities. The interviews are clearly described with access limitations acknowledged. This is not a sloppy or overclaimed paper.\n\nWho is this for? Anyone auditing street view data for fairness or building urban vision datasets. It deserves a serious referee, but with major revision: validate what the GSV retrieval actually returns, report the hyperparameters, and ideally release the retrieval code. The qualitative Amsterdam findings stand regardless; the quantitative ranks need more work before I’d rely on them.","headline":"A useful 28-city distributional audit with real interview value, but the GSV retrieval method probably thins the very density signal the rankings claim to measure.","tokens_in":23802,"tokens_out":2883,"would_cite":false,"duration_ms":33108,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Street view data skews inside cities even when coverage looks complete","keywords":["Street View Data","Coverage Distribution","Spatial Bias","Urban AI Datasets","Google Street View","Mapillary","Data Collection Artifacts"],"falsifier":"Re-run the KL-divergence, EMD, and $C_\\Delta$ analysis against a prior that weights roads by lane count or OpenStreetMap road class instead of weighting every 20-metre point equally; if the measured deviations mostly vanish or shrink dramatically, the claimed idiosyncratic bias is largely an artifact of the uniform prior rather than a property of collection.","tokens_in":22750,"feed_emoji":"🚗","tokens_out":4085,"duration_ms":41092,"temperature":0.7,"pith_summary":"Street view imagery is usually treated as a systematic sample of a city, especially when coverage looks complete. This paper argues that even densely covered cities carry a hidden distributional bias: some neighbourhoods are imaged far more often than others, so any dataset sampled from the imagery inherits that skew. For 28 cities it measures the gap between real panorama locations and a uniform road-coverage prior, using KL-divergence and Earth Mover's distance, and shows that binary coverage percentages barely predict these gaps. A case study of Amsterdam with six interviews traces the skew to human choices in the collection chain, from routes and driver discretion to camera hardware and seasons. The practical point is that auditing a street view dataset by 'was every street covered?' is not enough to know what biases are baked into it.","feed_headline":"Street view data skews inside cities even when coverage looks complete","feed_subtitle":"Binary coverage reports miss where imagery is over- or under-sampled; 28 cities show the gap.","key_machinery":"The evaluation is carried by a comparison between two densities over the city: $C_{\\text{Uniform}}$, a Gaussian kernel density estimate of evenly spaced points at 20 metre intervals along OpenStreetMap roads, and $C_{\\text{Real}}$, the same estimate over retrieved panorama locations. Their difference $C_\\Delta = C_{\\text{Real}} - C_{\\text{Uniform}}$ produces maps of over- and under-sampled $1\\,\\text{km}^2$ cells. Two distances quantify the gap: a $k$-nearest-neighbour estimator of Kullback-Leibler divergence, which is sensitive to roads being skipped, and Earth Mover's distance approximated with the debiased Sinkhorn divergence, which captures how far the centres of mass of over- and under-sampling are separated. The case study adds a qualitative mechanism: semi-structured interviews with six people across the Amsterdam collection chain, from data managers to contracted collectors.","core_discovery":"The paper's central claim is that the spatial distribution of street view coverage deviates from uniform road coverage in most of the 28 cities examined, and that these deviations are artifacts of the collection process rather than of the city itself. It finds cities like Kiev, Almaty, and Reykjavik with near-uniform distributions, while Los Angeles, Auckland, and Sydney show high Earth Mover's distance with relatively low KL-divergence, meaning coverage exists nearly everywhere but its mass is concentrated in certain neighbourhoods. Coverage percentage and distribution metrics do not track each other, so a city can look fully covered by binary measures yet be strongly skewed in where the images actually are. The Amsterdam interviews show that the same city centre gets revisited after traffic obstructions, that different contracted providers drive different distances per day, use different cameras, and collect in different months, and that deduplication only filters images taken on the same day, so the oversampling survives in the database. The paper concludes that domain knowledge of both city and collection process is necessary before treating street view imagery as representative.","pith_inferences":["The same vehicle-based collection mechanism applies to other geospatial data streams, so $C_\\Delta$-style distribution checks could audit mobile LiDAR, dashcam, and similar fleets for the same obstruction-driven oversampling.","The uniform prior is a normative benchmark; re-running the metrics with a road-class-weighted prior would separate 'human idiosyncrasy' from mechanical differences between highway and residential driving, which would sharpen the paper's causal interpretation.","A concrete next test is to use the $C_\\Delta$ maps as a stratified sampling guide and measure whether model performance becomes more equal across neighbourhoods, which would directly connect distribution bias to downstream AI behaviour."],"forward_implications":["Machine learning datasets sampled at fixed intervals from street view services carry a neighbourhood-level skew that persists even in cities with near-total coverage.","Binary coverage audits are insufficient validation for street view based models; distributional metrics such as EMD and KL should accompany them.","In a single city, Google Street View and Mapillary can have opposite bias patterns, so the choice of provider changes what a model learns.","The Amsterdam case implies that collection contracts that require equal coverage and cross-day deduplication could remove some artifacts at their source.","Researchers building urban perception datasets should treat each city as having its own coverage idiosyncrasy rather than assuming one sampling recipe generalises."],"supporting_citations":[{"why":"Supplies the road network and administrative boundary polygons used to build the uniform road-coverage prior.","marker":"[41]"},{"why":"Provides the k-nearest-neighbour estimator of KL-divergence used to compare real and uniform coverage distributions.","marker":"[43]"},{"why":"Provides the debiased Sinkhorn divergence used to approximate Earth Mover's distance at scale.","marker":"[15]"},{"why":"Establishes the prior binary coverage framing and the claim that certain cities are considered fully covered, which this paper argues is insufficient.","marker":"[44]"},{"why":"Supplies the spatial-grid API query approach for retrieving Google Street View metadata across a city.","marker":"[17]"},{"why":"Demonstrates on a small scale that street view image availability varies over time and neighbourhood, motivating a scalable distribution-level method.","marker":"[50]"},{"why":"Articulates the assumption of sampling at fixed spatial intervals to obtain a contiguous visual representation of the city, which the paper challenges.","marker":"[37]"},{"why":"Another example of a street view dataset construction that assumes fixed-interval sampling, used here as part of the mechanistic-collection assumption.","marker":"[25]"}],"fun_headline_variants":["Street view coverage skews by city layout even when dense","City layout skews street view images in 28 cities","Street view data hides uneven sampling within cities","Coverage maps lie: street view density varies inside cities","Street view bias traced to city form and collection quirks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark for 'no bias' is that every road segment should receive the same image density per unit length, so the measured deviations partly reflect departures from that specific uniform ideal rather than from the true collection process.","fun_headline_variants_meta":{"raw":{"variants":["Street view coverage skews by city layout even when dense","City layout skews street view images in 28 cities","Street view data hides uneven sampling within cities","Coverage maps lie: street view density varies inside cities","Street view bias traced to city form and collection quirks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1276,"prompt_tokens":910,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":526,"tokens_out":366,"duration_ms":3996,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:58:55.685153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the KL-divergence, EMD, and $C_\\Delta$ analysis against a prior that weights roads by lane count or OpenStreetMap road class instead of weighting every 20-metre point equally; if the measured deviations mostly vanish or shrink dramatically, the claimed idiosyncratic bias is largely an artifact of the uniform prior rather than a property of collection.","supporting_citations":[{"cited_title":"In Proceedings of the IEEE International Conference on Computer Vision (ICCV)","cited_arxiv_id":null,"evidence_quote":"Establishes the prior binary coverage framing and the claim that certain cities are considered fully covered, which this paper argues is insufficient."},{"cited_title":"Mooney, Daniel A","cited_arxiv_id":null,"evidence_quote":"Supplies the spatial-grid API query approach for retrieving Google Street View metadata across a city."},{"cited_title":"PanorAMS: Automatic Annotation for Detecting Objects in Urban Context","cited_arxiv_id":"2208.14295","evidence_quote":"Another example of a street view dataset construction that assumes fixed-interval sampling, used here as part of the mechanistic-collection assumption."}],"review_version":1}