{"id":"553224b6-ef0d-47f6-8331-7926c2af8478","arxiv_id":"2412.11061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Misclassification, not localization, drives most of the geographic bias in Eurocentric street-scene segmentation models.","lead":"This paper shows that when vision models trained mostly on European street scenes are tested worldwide, most of the geographic performance gap comes from the models mislabeling objects, such as calling a minibus a car, rather than from drawing boxes or masks inaccurately. The authors find that grouping fine-grained classes like 'bus' and 'truck' into coarser categories such as 'four-wheeler' removes a large share of the bias.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribution depends on a post hoc class-merging diagnostic and a recall-only IoU metric that ignores false positives; a standard-AP and class-agnostic re-analysis would test whether classification truly drives geo-bias.","rationale":"The reader identified the post hoc class-merging groups as the weakest assumption; I agree that this is a serious threat, and I add that the nonstandard per-instance IoU metric (Eq. 1) excludes false positives, so the dependent variable itself may not capture all localization errors. For the central claim to hold, the class-merging diagnostic must isolate classification errors and the continent-IoU metric must faithfully represent detection/segmentation quality. Both conditions are currently supported only by the authors' choices, not by comparison with standard metrics or label-agnostic matching. The paper has real strengths: it uses multiple architectures, a globally distributed evaluation set, and concrete qualitative examples of misclassification, and it openly discloses the manual nature of class-merging. That is why the correct verdict is conditional rather than rejection: the central claim is plausible but not yet established. A single re-analysis of the existing predictions with standard AP and class-agnostic localization scores would either confirm the attribution or show that it is an artifact of the chosen metric and grouping. Until that check is run, the quantitative ranges in the abstract should not be treated as robust estimates of classification-error contribution.","tokens_in":11881,"tokens_out":8357,"duration_ms":81646,"concrete_test":"Re-run the full evaluation on the saved predictions from all four models using (a) standard COCO-style AP at IoU 0.5:0.95 per class and continent, which penalizes false positives, and (b) label-agnostic instance matching that computes localization-only IoU without any class-merging groups. Compare the resulting continent disparities and the implied classification-share percentages with Tables 2, 7, and 8. If AP reveals large geographic disparities, or if the label-agnostic localization disparity does not shrink to near-zero for bicycle and motorcycle, the claim that localization is geographically robust and the 10-90% attribution fail; if both agree with the class-merging results, the central conclusion is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 attributes 19-88% (detection) and 10-90% (segmentation) of geo-disparity to classification errors by measuring the drop in geo-disparity (Disp = sigma/mu) after merging car-bus-truck, motorcycle-bicycle, and person-rider. Two assumptions are load-bearing. First, the merge groups are chosen after inspecting the failure modes (Section 4.2, and acknowledged in Limitations), so the diagnostic removes exactly the label mismatches the authors already know about; it does not independently establish that these mismatches are the causal classification contribution. Second, the continent-IoU metric in Eq. (1) averages only over ground-truth instances and ignores all false-positive detections; a model could hallucinate many objects in one continent and still show no localization geo-bias. The central claim that localization is geographically robust is therefore conditional on a recall-style measure. The reported percentages also carry no uncertainty; for classes with small instance counts (e.g., bicycle Africa n=24, motorcycle Oceania n=37 in Table 5), the 10-19% reductions are within plausible sampling noise. These issues are addressable but make the headline ranges not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates geographic bias in instance segmentation and detection. Four models (Mask-RCNN and OneFormer with Swin-L, ConvNext-L, ConvNext-XL backbones) trained on the Eurocentric Cityscapes dataset are evaluated on Mapillary Vistas images from six continents. The authors define a continent-IoU metric that averages per-instance IoU over ground-truth instances, and a geo-disparity measure Disp = sigma/mu over the six continent means. To isolate the contribution of classification errors, they merge visually similar classes (car-bus-truck, motorcycle-bicycle, person-rider) and compute the percentage reduction in geo-disparity after merging. They find that person and car are geographically robust, while rider, bus, truck, motorcycle, and bicycle are geo-biased, and report that classification errors contribute 19-88% of geo-disparity in detection and 10-90% in segmentation. The paper concludes that geo-bias in these Eurocentric models is driven primarily by classification errors rather than localization errors, and suggests that coarser class labels could mitigate the bias.","tokens_in":12097,"tokens_out":5468,"duration_ms":49271,"significance":"If the quantitative claim were established, this would be a useful and timely result: it is the first study of geo-bias on a task combining classification and localization in driving scenes, and it offers a simple practical mitigation (coarser classes). The analysis is transparent, the release of image-location metadata is a reproducible contribution, and the qualitative finding that specific cross-class confusions (mini-buses versus cars in Africa, motorcycles versus bicycles in Asia) drive much of the disparity in most models is well supported by the examples and by the consistency across several architectures. However, the headline numerical ranges are not yet established because the class-merging groups are chosen post hoc, the continent-IoU metric ignores false positives, and no uncertainty is reported for the six-continent disparity estimates. The qualitative conclusion about the importance of classification errors is likely robust; the specific quantitative attribution is not.","major_comments":[{"comment":"The abstract's ranges '19-88%' and '10-90%' are presented as general findings, but they are the ranges across classes for the Swin-L model only. Table 7 shows, for example, a -7.99% detection reduction for bicycle under ConvNext-L, and Table 8 shows a +32.55% increase for bus under Faster-RCNN. The paper should report the per-model ranges, account for the increase in geo-disparity for Faster-RCNN bus, and qualify the abstract so that it does not overstate the universality of the 10-90% and 19-88% figures.","section":"Abstract and Section 4.3, Tables 2, 7, 8"},{"comment":"The class-merging groups (car-bus-truck, motorcycle-bicycle, person-rider) are selected after inspecting the misclassifications in Section 4.2, and the Limitations section acknowledges that the classes were manually identified. Because the merge groups encode exactly the confusions the authors already observed, the percentage reduction in Disp computed in Eqs. (2) and (3) measures the effect of those specific confusions rather than an independent estimate of the total classification-error contribution. The paper's causal phrasing, such as 'classification errors caused geo-bias,' overstates what this post hoc diagnostic can establish; the numbers should be described as an upper-end estimate for the chosen merge groups.","section":"Section 4.3 and Limitations"},{"comment":"The continent-IoU metric averages only over ground-truth instances, ignores duplicate predictions, and never penalizes predictions with no ground-truth overlap. As a result, a model that hallucinates many objects in one continent can still show zero geo-bias, so the conclusion that 'localization (including detection and segmentation) is geographically robust' is conditional on a recall-style metric. The authors should re-analyze with a metric that includes false positives (e.g., standard AP or a precision-aware matching) or explicitly justify why false positives are irrelevant to the geo-bias question.","section":"Section 3.4, Eq. (1)"},{"comment":"All geo-disparity values are computed from six continent-level means, and no error bars or confidence intervals are provided. The small instance counts in Table 5, such as 24 bicycle instances in Africa and 37 motorcycle instances in Oceania, mean that the 10-19% reductions reported for bicycle in Table 2 could be within sampling noise. A bootstrap or per-image re-sampling procedure for Disp and for the percentage change would be needed before the headline ranges can be interpreted quantitatively.","section":"Section 4.3 and Table 5"}],"minor_comments":[{"comment":"The supplementary text says that the Table 4 total of 10,547 images is 'lesser' than the 11,300 evaluation images 'because a few images were eliminated in pre-processing.' If images were eliminated, the final count should be smaller than 11,300, not larger; please clarify whether 11,300 is the count before or after pre-processing.","section":"Section B"},{"comment":"There are minor typographical issues: 'as4-wheeler' should be 'as 4-wheeler', and the supplementary section title 'Dataset Comparision' should be 'Dataset Comparison'.","section":"Abstract and Supplementary title"},{"comment":"Each box plot is built from only six continent-level data points, making interquartile ranges visually unstable; overlaying the individual continent values as a strip or beeswarm plot would let readers verify the spread directly.","section":"Figures 2 and 3"},{"comment":"The notation 'Dispdet-det-corrected' is confusing because it denotes a percentage change rather than a disparity value; renaming it as something like 'PercentChangeDet' would improve readability.","section":"Section 4.3, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"This is an honest and readable empirical study with a plausible qualitative conclusion. The main risk is that the headline numbers are presented as a general quantitative attribution when they depend on post hoc merge choices, a recall-only metric, and one model's class-level range. I do not see a fundamental flaw that would require rejection, but the abstract and conclusion need to be re-scoped and the load-bearing metric/uncertainty issues need to be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing to know: this is the first paper to look at geo-bias in instance segmentation on driving datasets, and it introduces a clean diagnostic — merging visually confusable classes — to separate classification from localization errors. That framing is new and worth taking seriously. The abstract's headline numbers (10-90%, 19-88%) are real, but they are model- and class-specific ranges, and the claim that localization is geographically robust is stronger than the data support.\n\nWhat is good: the authors evaluate four instance segmentation models plus two object detectors on Mapillary Vistas, the only driving dataset with global coverage and instance labels, and they publish the location metadata. The qualitative examples are convincing: mini-buses in Africa get called cars, thin-wheel motorcycles in Asia get called bicycles. Class-merging does shrink the disparity for most geo-biased classes across most models, so the effect is reproducible. The related work is fair and the citations are on point.\n\nWhere it is soft. The class-merging groups were chosen after inspecting the failures; the authors acknowledge this in the Limitations, but it means the contribution percentages are not an independent estimate of causal effect. They measure the drop after fixing exactly the confusions the authors already knew about. Second, the continent-IoU metric averages over ground-truth instances only and ignores false positives. That is a recall-style measure. A model that hallucinates extra objects in one continent would not be penalized, so 'localization is geographically robust' is conditional on a metric that never looks at precision. A standard AP re-analysis is the obvious check. Third, there are no error bars, and for classes like bicycle (24 instances in Africa) or motorcycle (37 in Oceania) the disparity numbers are within sampling noise. Fourth, the title overclaims: for bicycle, class-merging cuts only about 10-20% of the disparity, so localization is the bigger contributor there. The 'classification drives geo-bias' story holds for bus, truck, motorcycle, and rider, not for bicycle.\n\nBottom line: this paper is useful for anyone working on geographic robustness or driving perception. It deserves a serious referee, but the headline claims need tightening and the metric needs a precision-aware sibling analysis. I would send it to review with major revisions expected.","headline":"First geo-bias study on instance segmentation in driving data; useful diagnostic but the headline claims outrun the evidence on metric choices and class-merging.","tokens_in":12616,"tokens_out":3785,"would_cite":true,"duration_ms":32239,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Misclassification, not localization, drives the geographic gap in street-scene models.","keywords":["geographic bias","geo-bias","instance segmentation","class-merging","classification errors","localization","Cityscapes","Mapillary Vistas"],"falsifier":"Re-run the class-merging analysis on randomly paired classes that are visually dissimilar and rarely confused (e.g., person-car, bus-bicycle); if geo-disparity falls by as much as with the cited groups, the conclusion that similarity-driven misclassification causes the bias is not supported.","tokens_in":11675,"feed_emoji":"🚗","tokens_out":4443,"duration_ms":36105,"temperature":0.7,"pith_summary":"The paper asks whether instance-segmentation models trained on European street scenes (Cityscapes) perform unevenly across continents when applied to globally diverse driving images (Mapillary Vistas), and if so, why. It finds clear geographic bias for five classes—rider, bicycle, motorcycle, bus, and truck—and argues the bias comes from classification errors, not from how well the model localizes objects. Using a class-merging strategy that re-labels visually similar classes during evaluation, the authors estimate classification errors alone account for 19-88% of the geo-bias in detection and 10-90% in segmentation. The practical upshot is that coarse class labels such as \"four-wheeler\" or \"two-wheeler\" substantially reduce geographic performance gaps for Eurocentric models used globally.","feed_headline":"Misclassification drives up to 90% of street-scene geo-bias","feed_subtitle":"Coarse class labels like \"four-wheeler\" shrink geographic gaps for Eurocentric models across six continents.","key_machinery":"The load-bearing instrument is the class-merging strategy: during evaluation, ground-truth and predicted labels are collapsed from fine classes into structural super-classes (four-wheelers, two-wheelers, humans), then the IoU earned by the merged class is remapped to the original fine class as a \"corrected IoU.\" Paired with a scale-invariant geo-disparity measure, Disp = sigma/mu over continent IoUs, the before-and-after comparison isolates how much of the geographic performance gap is attributable to misclassification versus localization. The strategy works because the merged groups are precisely the pairs the model confuses, so the corrected IoU rescues instances that were localized correctly but labeled wrongly.","core_discovery":"The central claim is that geographic bias in Eurocentric instance-segmentation models is driven by classification errors while localization is geographically robust. On six continents, models pre-trained on Cityscapes performed markedly worse for buses, trucks, motorcycles, bicycles, and riders in Africa and Asia, largely because objects there resemble different European classes—African mini-buses look like cars, Asian motorcycles look like bicycles. When the authors merge confusable classes (car-bus-truck, motorcycle-bicycle, person-rider) at evaluation time, the inter-continental spread of per-class IoU shrinks dramatically, and the reduction in geo-disparity quantifies the classification-error share: 19-88% in detection and 10-90% in segmentation across the geo-biased classes. This is the first demonstration for a localization task that geo-bias resides in the classifier rather than the localizer.","pith_inferences":["A diagnostic the authors leave implicit: the same class-merging procedure could serve as a general tool to decompose any domain-shift performance gap into classification versus localization components, not just geographic bias.","Because the merged classes are also a hierarchy, the results suggest that fine-grained taxonomies themselves encode geographic priors—a model trained only in Europe can localize a vehicle but cannot learn that \"bus\" in Africa is a smaller van-like object.","A testable extension would be to apply geography-aware classifiers (which the paper mentions) and check whether geo-disparity after class-merging falls further, which would confirm the residual gap is localization rather than remaining label noise."],"forward_implications":["If the claim holds, improving classification alone—without touching localization—can close most of the geographic gap for these classes.","Deploying region-specific models globally with coarse labels (four-wheeler, two-wheeler, human) is a practical mitigation that requires no retraining.","Fine-grained applications still need region-specific training data or geography-aware classification to recover the lost class distinctions.","Localization errors, though a smaller share, remain a residual source of geo-bias and are the next target for mitigation."],"supporting_citations":[{"why":"Supplies the Eurocentric Cityscapes training data used for all models.","marker":"[4]"},{"why":"Supplies the globally covered Mapillary Vistas evaluation data with location metadata.","marker":"[12]"},{"why":"Establishes prior evidence that object recognition performance degrades across income levels and geographies, motivating the geo-bias question.","marker":"[5]"},{"why":"Shows geographic bias in classifiers trained on Western-centric datasets, the baseline phenomenon this paper extends to localization tasks.","marker":"[15]"},{"why":"Source of the OneFormer ViT-based segmentation models (Swin-L, ConvNext-L, ConvNext-XL).","marker":"[1]"},{"why":"Source of the Mask-RCNN pretrained model used as the detect-and-segment baseline.","marker":"[3]"}],"fun_headline_variants":["Geo-bias in segmentation traces to classification errors","Street-scene bias? It's misclassification, not mislocalization","Coarse labels cut geographic bias in Eurocentric models","Split by continent, street-scene bias lives in the classifier","Why Eurocentric models fail abroad: the classifier's fault"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole attribution rests on the assumption that merging car-bus-truck, motorcycle-bicycle, and person-rider removes exactly the classification errors that cause geographic disparity; if other confusions or interactions with the metric contribute, the reported 10-90% shares are overstated.","fun_headline_variants_meta":{"raw":{"variants":["Geo-bias in segmentation traces to classification errors","Street-scene bias? It's misclassification, not mislocalization","Coarse labels cut geographic bias in Eurocentric models","Split by continent, street-scene bias lives in the classifier","Why Eurocentric models fail abroad: the classifier's fault"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001103,"raw_usage":{"total_tokens":4581,"prompt_tokens":905,"completion_tokens":3676,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":3594}},"tokens_in":521,"tokens_out":3676,"duration_ms":23717,"temperature":1.0,"reasoning_tokens":3594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:19:38.876409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the class-merging analysis on randomly paired classes that are visually dissimilar and rarely confused (e.g., person-car, bus-bicycle); if geo-disparity falls by as much as with the cited groups, the conclusion that similarity-driven misclassification causes the bias is not supported.","supporting_citations":[{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the Eurocentric Cityscapes training data used for all models."},{"cited_title":"The mapillary vistas dataset for semantic understanding of street scenes","cited_arxiv_id":null,"evidence_quote":"Supplies the globally covered Mapillary Vistas evaluation data with location metadata."},{"cited_title":"Does object recognition work for ev- eryone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 52–59, 2019","cited_arxiv_id":null,"evidence_quote":"Establishes prior evidence that object recognition performance degrades across income levels and geographies, motivating the geo-bias question."},{"cited_title":"com / SHI - Labs / OneFormer,","cited_arxiv_id":null,"evidence_quote":"Source of the OneFormer ViT-based segmentation models (Swin-L, ConvNext-L, ConvNext-XL)."}],"review_version":1}