{"id":"700605ab-bc37-4651-bba4-14686b334884","arxiv_id":"2511.18011","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 9,121-case benchmark of road-marking tasks shows most multimodal LLMs perform near or below simple rule-based baselines in fine-grained urban spatial reasoning.","lead":"RoadBench is a new benchmark that tests how well multimodal AI models understand road markings and lane structure in satellite and dashboard-camera images. Across six tasks and about 9,100 hand-verified cases, most models score near or below simple rule-based guesses, suggesting current systems lack fine-grained spatial reasoning in real urban scenes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels from Amap are unvalidated; systematic map errors could invalidate the finding that MLLMs underperform simple baselines.","rationale":"The paper's central claim is that RoadBench is a challenging benchmark and that MLLMs underperform simple baselines on fine-grained spatial understanding. The strongest assumption behind this claim is that the ground-truth labels are accurate. The authors rely on a commercial map database (Amap) for lane counts, lane designations, junction positions, and road types, and the quality-control step rejects data that is hard to recognize instead of independently verifying labels. This is explicitly stated in Section 3.3: 'The rare erroneous or unrecognizable data were simply rejected rather than relabeled to avoid introducing manual errors.' That sentence is a self-acknowledged limitation: it means no independent check on the external labels exists. The paper provides no verification statistics, no inter-annotator agreement, and no rejection rate. Without knowing the error rate of the Amap labels, we cannot be sure that the reported MLLM failures reflect genuine shortcomings rather than label noise or systematic map errors. If the map data is wrong, the benchmark could be measuring map errors rather than MLLM capabilities, and the conclusion that 'MLLMs fall short of simple rule-based baselines' could be an artifact. This concern is load-bearing because it affects every quantitative conclusion in the paper. The reader's weakest_assumption correctly identified this. I agree with the reader's conditional verdict: the artifact shows promise, but the missing verification of ground-truth labels prevents full confidence. The abstract/body inconsistency (3,040 vs. 9,121 cases; 20 vs. 14 MLLMs) is concerning but secondary; it can be fixed without altering the core argument. Therefore, I recommend no change to the reader's CONDITIONAL verdict, with the condition being the necessity of publishing verified labels and code.","tokens_in":25455,"tokens_out":3715,"duration_ms":39264,"concrete_test":"Independently re-label a random sample of 100 test cases per task from raw images, with two annotators blinded to Amap labels and to each other, following the task definitions. Compute inter-annotator agreement (e.g., Cohen's kappa) and the disagreement rate with the released ground truth. Then recompute best-MLLM and rule-based baseline scores on the subset of cases where both annotators confirm the provided label. If (a) human-Amap agreement is below ~95% for lane counting/designation, or (b) the rule-based baseline no longer beats the best MLLM on the confirmed subset, the central claim is not supported. Also report the number of cases rejected during quality control to assess selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that RoadBench reveals MLLMs' poor fine-grained spatial understanding and that MLLMs fall short of simple baselines—rests entirely on the correctness of ground-truth labels sourced from the Amap database (Section 3.3). The quality-control stage rejects rather than relabels ambiguous cases, so there is no independent verification of these external labels. If Amap contains systematic errors (outdated lane counts, incorrect junction topology, misaligned coordinates), the reported metrics and baseline comparisons are not measuring the intended capabilities. For example, the lane designation rule-based baseline (Appendix 7.2) uses only the provided lane count; if that count is wrong, the baseline's apparent superiority could be an artifact. The paper provides no inter-annotator agreement, no re-labeling of a holdout subset, and no statistics on rejection rates. This absence of ground-truth validation directly threatens the benchmark's validity and the paper's headline conclusion, making it more load-bearing than the abstract/body number mismatch, which is a presentation issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RoadBench, a benchmark for evaluating MLLMs on fine-grained spatial understanding and reasoning about road markings in urban settings. Six tasks are proposed: BEV lane counting, BEV lane designation recognition, BEV road network correction, FPV lane counting, FPV lane designation recognition, and FPV road type classification, with test cases sourced from satellite imagery and in-vehicle camera images. Labels are derived from an online map service database (Amap), with human proofreading and rejection of ambiguous cases. The authors evaluate 14 MLLMs on these tasks and report that performance is generally low, and that in several tasks MLLMs underperform simple rule-based or random baselines. The paper also includes analyses of prompt-format effects, environmental conditions, and failure cases.","tokens_in":25721,"tokens_out":4656,"duration_ms":44811,"significance":"If the empirical claims hold, RoadBench addresses a genuine gap in spatial benchmarks for MLLMs: existing urban benchmarks mostly target whole-image understanding or isolated objects, whereas RoadBench focuses on fine-grained, interdependent road markings and tasks that scale from local perception to global reasoning. The benchmark design is thoughtful, the task set is systematic, and the inclusion of rule-based baselines is a useful diagnostic. The paper also provides detailed per-task results, prompt templates, and failure-case analyses, which strengthen its value as a resource. However, the current manuscript contains internal inconsistencies in the benchmark statistics and does not provide sufficient validation of the external ground-truth labels. These issues are load-bearing because the central conclusion—that MLLMs are weak at fine-grained urban spatial reasoning and even fall below simple baselines—depends directly on the correctness of the labels and on the reliability of the reported numbers.","major_comments":[{"comment":"The paper's metadata-level abstract states that RoadBench comprises eight tasks, 3,040 test cases, and 20 evaluated MLLMs, while the body abstract, Section 3.1, and Section 4.1 consistently describe six tasks, 9,121 test cases, and 14 evaluated MLLMs. The contribution size and the set of evaluated models are central to the benchmark claim, so this contradiction must be resolved and the correct values reported consistently in the camera-ready version.","section":"Abstract versus Sections 3.1 and 4.1"},{"comment":"The ground-truth labels for lane counts, lane designations, junction positions, and road types are taken from the Amap database without independent verification. The quality-control stage rejects erroneous or unrecognizable cases rather than re-labeling them, and the paper reports no inter-annotator agreement, no rejection statistics, and no holdout re-labeling check. If the external map database contains systematic errors—outdated lane counts, incorrect junction topology, or coordinate misalignment—the reported model scores and, in particular, the conclusion that rule-based baselines beat MLLMs could be artifacts of label noise. The authors should provide a validation protocol, e.g., re-annotating a random sample by independent annotators and reporting agreement, along with rejection rates and any coordinate-matching error analysis.","section":"Section 3.3 (Quality Control) and Tables 2/71-76"},{"comment":"The claim that 'none of the MLLMs can outperform the baseline' in lane designation rests on very small margins in some cases. For example, in the FPV lane designation task, the rule-based baseline has Hamming Loss 0.1282 and accuracy 0.6019, while GPT-5 has 0.1287 and 0.5932, and GPT-5-Mini has 0.1301 and 0.5941. These differences could lie within sampling variability. No confidence intervals, bootstrap estimates, or significance tests are reported for model-versus-baseline comparisons. The authors should report per-example bootstrap intervals or an appropriate paired test to support the strong wording that MLLMs 'fall short of' simple baselines.","section":"Section 4.2 and Tables 72/75"},{"comment":"Table 74 lists F1 = 0.0000 for LLaMA-3.2-11B-Vision and LLaMA-3.2-90B-Vision in FPV lane counting, despite positive Precision and Recall values and despite Table 2 reporting F1 = 0.273 and F1 = 0.305 for those same models. Since F1 cannot be zero when both precision and recall are positive, these entries are numerically inconsistent. The authors need to correct the appendix table and audit the underlying computation.","section":"Appendix Table 74 and Table 2"}],"minor_comments":[{"comment":"The phrase 'randomly select two lanes from {2,3,4}' should read 'randomly select a lane count from {2,3,4}'.","section":"Section 4.1"},{"comment":"The text says adverse lighting conditions caused only a 0.2%–1.9% drop for GPT-5, but Table 810 shows F1 improving by 0.2% and accuracy dropping by 0.8%. Please express changes consistently with respect to metric direction (higher is better for F1/accuracy, lower is better for RMSE/Hamming loss).","section":"Section 8.1 / Table 810"},{"comment":"The use of star symbols and cell placeholders in Table 1 makes the comparison hard to read. Consider replacing with textual descriptors or a cleaner symbol legend.","section":"Table 1"},{"comment":"No URL or release plan for the benchmark data and evaluation code is provided. Since the benchmark is a main contribution, a public release link should be included in the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important gap and the core idea is sound, but the abstract/body mismatch and the lack of ground-truth validation are serious. I would be willing to reconsider after the authors align the statistics, add an independent validation of labels, and support the baseline-comparison claims with uncertainty estimates. If the ground-truth validation reveals substantial label error, the headline conclusion could change, so this is not merely a presentation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: RoadBench is a genuine contribution to MLLM benchmark literature, and the central qualitative claim—that current MLLMs perform poorly on fine-grained road-marking tasks and often fall short of simple baselines—is likely correct, even if the details need fixing.\n\nWhat's new: existing urban benchmarks look at whole-image understanding or isolated objects. RoadBench focuses on interdependent road markings across BEV and FPV, with six tasks from lane counting to network correction. The dataset is large (9,121 cases), the model coverage is solid (14 mainstream MLLMs), and the rule-based/random baselines are a useful reality check. The failure-case analysis is valuable.\n\nSoft spots, in order of severity:\n1. The arXiv metadata says 8 tasks/3,040 cases/20 models; the body says 6/9,121/14. That's not a typo—it suggests the abstract/body were produced from different runs. Similarly, Table 74 in the appendix lists F1=0.0000 for two LLaMA models while precision/recall are nonzero; either a formatting error or a calculation bug. These need to be cleaned before anyone trusts the numbers.\n2. The ground-truth labels come from Amap. The quality-control stage rejects unrecognizable cases but doesn't independently re-label a holdout, report rejection rates, or provide inter-annotator agreement. If Amap has systematic errors in lane counts or junction topology, model scores and baseline comparisons could shift. I don't think this kills the paper—the effect sizes are large, and a noisy-label robustness check would probably hold—but the authors should provide validation stats or a small manually re-annotated subset.\n3. No dataset/code release. For a benchmark, that's a serious limitation. It should be a condition of acceptance.\n\nWhat the reader's stress-test missed: the stress-test worried the Amap labels could invalidate the 'rule-based beats MLLMs' conclusion. On reading, the rule-based baseline in lane designation (task 2/5) uses only lane count and common-sense mapping; if lane count is wrong, the baseline's advantage could indeed be partially an artifact. But the same conclusion appears in road type classification where random choice is near 0.5 and several models barely beat it. The core finding is overdetermined. So the label concern is real but not fatal.\n\nCitable? Yes—it's a benchmark people will use. Peer review? Send it, but the authors should fix the inconsistencies and release the artifact. Serious thinker: yes—the work is coherent and honest, just currently messy.","headline":"RoadBench fills a real gap in evaluating MLLMs on fine-grained road-marking spatial reasoning, but the paper needs cleanup and the unvalidated external labels deserve scrutiny before the headline claim is taken on faith.","tokens_in":26196,"tokens_out":3928,"would_cite":true,"duration_ms":36530,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoadBench: A benchmark built on road markings shows multimodal models fail fine-grained spatial reasoning under urban road scenarios.","keywords":["road markings","fine-grained spatial understanding","multimodal large language models","benchmark","bird's-eye view","first-person view","lane counting","urban scenarios"],"falsifier":"Independently re-annotate a random sample (say 300 cases) of RoadBench with a second set of human labelers and compare against the provided ground truth; if the disagreement rate on lane counts or junction positions exceeds the gap between the best MLLM and the rule-based baseline, the headline conclusion collapses. Alternatively, run the same six tasks on images from left-hand-traffic countries; if MLLM scores improve dramatically there, the failures are specific to China's marking conventions rather than general fine-grained spatial ability.","tokens_in":25372,"feed_emoji":"🛣️","tokens_out":4665,"duration_ms":41736,"temperature":0.7,"pith_summary":"The paper proposes RoadBench, a benchmark of 9,121 hand-verified test cases that probes multimodal large language models (MLLMs) on fine-grained spatial understanding and reasoning in urban road scenes, using road markings as the central test object. The six tasks range from local lane counting and lane-designation recognition on satellite (bird's-eye view) and dash-cam (first-person view) images to global road-network correction and road-type classification. Evaluating 14 mainstream MLLMs, the authors find that the best models achieve low F1 scores on lane counting, none beat a traffic-commonsense rule-based baseline on lane designation recognition, and several fall below random-choice baselines. The paper argues this shows current MLLMs lack the fine-grained spatial perception and domain-knowledge integration needed for real-world urban applications such as HD map generation and autonomous driving.","feed_headline":"AI vision models can't beat simple rules on road markings","feed_subtitle":"A 9,121-case benchmark shows MLLMs fail lane counting, junction detection, and road-type reasoning.","key_machinery":"The central object is the road marking system—lane dividers, directional arrows, and related painted symbols—treated as fine-grained spatial elements with strong inter-dependency. The benchmark's mechanism is a hierarchy of six tasks that escalate from local recognition (lane counting, lane designation) to global reasoning (road-network correction, road-type classification), requiring the model to integrate image content with traffic domain knowledge. Evaluation uses multi-class metrics (Precision/Recall/F1, RMSE), Hamming Loss and accuracy for multi-label lane designations, and RMSE and Fréchet distance for geometric outputs, with rule-based and random baselines as reference points.","core_discovery":"RoadBench is presented as the first benchmark that centers on road markings—thin, elongated, interdependent spatial elements—as the unit of evaluation. It comprises two viewpoints (BEV satellite and FPV vehicle-camera images), six tasks organized by spatial scope, and 9,121 manually verified test cases from multiple Chinese cities. On it, 14 closed- and open-source MLLMs are tested; the central finding is that all models underperform on fine-grained road-marking tasks, with the best lane-counting F1 score at 0.369 and no model exceeding a rule-based lane-designation baseline (which achieves Hamming Loss 0.141 vs best model 0.152). The authors conclude that MLLMs fall short of simple non-visu","pith_inferences":["Editorial note: the provided abstract states eight tasks and 3,040 test cases, while the body, tables, and conclusion state six tasks and 9,121 test cases; this internal inconsistency should be reconciled before the benchmark is cited as a fixed resource.","Because all images come from five Chinese cities with right-hand traffic, the benchmark's baselines and difficulty may not transfer directly to left-hand-traffic regions or cities with different marking conventions; a left-hand-traffic extension would test whether the failures are perceptual or convention-dependent.","The quality-control stage rejects unrecognizable cases instead of re-labeling them, so the benchmark measures performance on human-legible images; an adversarial evaluation with harder, partially occluded cases might produce even lower model scores.","A targeted probe—varying image resolution or adding synthetic occlusions—could determine whether the BEV weakness is a resolution artifact or a structural reasoning limit, informing whether better cameras or better models are the fix."],"forward_implications":["If RoadBench is a fair instrument, then existing MLLMs cannot be trusted for fine-grained spatial annotations in HD-map generation or end-to-end driving pipelines without substantial improvement.","The benchmark separates perceptual failure (reading thin markings in BEV images) from reasoning failure (integrating domain knowledge such as bidirectional-road conventions), giving developers a diagnostic ladder.","The result that a rule-based, image-free baseline beats MLLMs on lane-designation recognition implies that models have not yet absorbed basic traffic-layout common sense, pointing to a concrete training-data gap.","The coordinate-output task shows MLLMs produce geometrically unreliable junction points and polylines, suggesting structured numeric output and spatial grounding need dedicated treatment."],"fun_headline_variants":["MLLMs fall short of rule-based baselines on road-marking benchmark","RoadBench: AI vision models flunk fine-grained road spatial reasoning","AI models can't count lanes better than simple rules, new benchmark finds","New benchmark exposes MLLMs' weak spatial skills in urban roads","RoadBench reveals MLLMs worse than random on some road tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The ground-truth labels (lane counts, designations, junction positions, road types) are taken from a commercial map provider's database without independent re-labeling; if that database contains systematic errors—stale lane counts, wrong junction topology, or coordinate mismatches—the reported model scores and the 'rule-based beats MLLMs' conclusion may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs fall short of rule-based baselines on road-marking benchmark","RoadBench: AI vision models flunk fine-grained road spatial reasoning","AI models can't count lanes better than simple rules, new benchmark finds","New benchmark exposes MLLMs' weak spatial skills in urban roads","RoadBench reveals MLLMs worse than random on some road tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1500,"prompt_tokens":845,"completion_tokens":655,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":561}},"tokens_in":589,"tokens_out":655,"duration_ms":6042,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:48:33.485053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently re-annotate a random sample (say 300 cases) of RoadBench with a second set of human labelers and compare against the provided ground truth; if the disagreement rate on lane counts or junction positions exceeds the gap between the best MLLM and the rule-based baseline, the headline conclusion collapses. Alternatively, run the same six tasks on images from left-hand-traffic countries; if MLLM scores improve dramatically there, the failures are specific to China's marking conventions rather than general fine-grained spatial ability.","supporting_citations":[],"review_version":1}