{"id":"ee06479a-1415-46c6-8a68-b04b3f95d138","arxiv_id":"2606.09644","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Presents a benchmark of 122 conflict-centric QA pairs from 73 NuScenes scenes to evaluate view-level evidence identification in multi-view MLLMs.","lead":"This paper introduces a benchmark using NuScenes multi-view driving scenes to test whether MLLMs identify the correct camera view when answering questions. A smart generalist might read it to understand gaps in how current AI models ground answers in safety-critical multi-camera settings.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Ground-truth view labels rest on an unquantified automatic conflict-mining pipeline plus manual verification whose error rate is unreported.","rationale":"The reader’s weakest assumption is exactly the load-bearing precondition for the benchmark’s validity. Because the full text was referenced but the provided input contained only the abstract, the UNVERDICTED verdict is appropriate; once the verification statistics are examined, the verdict can be updated to CONDITIONAL or ACCEPT depending on the measured label reliability.","tokens_in":1695,"tokens_out":368,"duration_ms":10491,"concrete_test":"Sample 30 of the 122 QA pairs, have two additional annotators independently label the supporting view(s) without seeing the original pipeline output, and compute Cohen’s kappa and the fraction of pairs where at least one annotator disagrees with the published golden view; if disagreement exceeds 15 % or kappa < 0.75, the label noise is large enough to weaken the benchmark’s diagnostic power.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that separating view identification from answer correctness reveals grounding failures missed by accuracy-only metrics—requires that each of the 122 QA pairs has a unique, correctly identified supporting camera view. The paper states that views are proposed by an automatic conflict-mining pipeline and then manually verified, yet provides no inter-annotator agreement, no error analysis on the pipeline outputs, and no count of how many pairs required correction during verification. If the pipeline systematically under- or over-identifies conflicts (e.g., when an answer is supported by two overlapping views or when scene geometry makes the conflict ambiguous), the resulting “golden view” labels become noisy; models flagged as grounding failures may simply be disagreeing with an imperfect label rather than exhibiting a true grounding error.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a benchmark for evaluating multi-view MLLMs on visual evidence identification in autonomous driving scenes. Using 122 conflict-centric QA pairs from 73 NuScenes scenes spanning causality, counterfactual reasoning, and intent prediction, models must select the supporting camera view and answer the question. The benchmark evaluates three settings (camera-view selection, oracle QA on the golden view, and joint prediction) in both multiple-choice and free-form formats, with the goal of exposing grounding failures missed by answer-accuracy metrics alone. View labels are generated by an automatic conflict-mining pipeline followed by manual verification.","tokens_in":1843,"tokens_out":406,"duration_ms":13688,"significance":"If the ground-truth view labels prove reliable, the benchmark would offer a targeted evaluation protocol that separates visual-source identification from answer correctness, addressing a practical gap in assessing MLLM reliability for safety-critical multi-view perception tasks.","major_comments":[{"comment":"The automatic conflict-mining pipeline and manual verification process for the 122 QA pairs are described only at a high level with no reported inter-annotator agreement, pipeline error rate, or count of corrections applied during verification. Because the central claim—that the benchmark reveals grounding failures missed by accuracy-only metrics—depends on the correctness of these golden-view labels, the absence of quantitative validation on label quality is load-bearing.","section":"Benchmark Construction"}],"minor_comments":[{"comment":"The evaluation protocol states that free-form answers are judged by an LLM but provides no details on the judge model, prompt template, or calibration against human judgments.","section":"Evaluation Metrics"},{"comment":"The abstract and methods would benefit from an explicit statement of how many of the 122 pairs were supported by multiple overlapping views and how such cases were resolved in the conflict-mining step.","section":"Data Statistics"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback highlighting the importance of validating the ground-truth view labels. We address the major comment on benchmark construction below.","responses":[{"response":"We agree that a more detailed account of the label generation process would strengthen the paper. In the revision we will expand Section 3.2 with additional specifics on the automatic conflict-mining pipeline (including any measurable error characteristics), the exact verification protocol used by the annotators, and the number of corrections applied. We note that formal inter-annotator agreement statistics were not computed because verification was performed by a small group of domain experts who reached consensus through discussion; we will explicitly state this and report all available process statistics to support label reliability.","revision_made":"yes","referee_comment":"[Benchmark Construction] The automatic conflict-mining pipeline and manual verification process for the 122 QA pairs are described only at a high level with no reported inter-annotator agreement, pipeline error rate, or count of corrections applied during verification. Because the central claim—that the benchmark reveals grounding failures missed by accuracy-only metrics—depends on the correctness of these golden-view labels, the absence of quantitative validation on label quality is load-bearing."}],"tokens_in":1296,"tokens_out":266,"duration_ms":21607,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper introduces a benchmark to test whether MLLMs identify the correct camera view in six-view NuScenes scenes rather than just producing a right answer. It creates 122 conflict-centric QA pairs covering causality, counterfactuals, and intent, then runs three settings: view selection alone, oracle QA with the golden view, and joint prediction.\n\nIt does a couple of things cleanly. The separation of evidence source from final answer is a direct response to a real gap in current VQA-style evals for autonomous driving. Using exact match plus an LLM judge for free-form answers is straightforward, and the data comes from an external source rather than synthetic generation.\n\nThe soft spot is the label creation process. Views are proposed by an automatic conflict-mining pipeline and then manually verified, yet the abstract gives no inter-annotator agreement, no error count on the pipeline, and no breakdown of how many pairs needed fixes. With only 73 scenes and 122 pairs, any systematic bias in the mining step would make the reported grounding failures hard to interpret.\n\nThis is for people working on MLLM evaluation or deployment in driving. Readers who want concrete test cases for visual grounding will get value from the setup. It deserves a serious referee because the core distinction it targets is worth checking, even if the current label details need more scrutiny.","headline":"The benchmark separates view selection from answer accuracy in multi-view driving MLLMs, but the ground-truth labels come from an unquantified pipeline.","tokens_in":2327,"tokens_out":349,"would_cite":false,"duration_ms":15781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A benchmark requires models to identify the supporting camera view before answering multi-view driving questions.","keywords":["multi-view visual question answering","visual evidence identification","autonomous driving","multimodal large language models","camera view selection","grounding evaluation","NuScenes scenes","conflict-centric questions"],"falsifier":"A large-scale re-annotation showing that human judges disagree with the supplied view labels on more than a small fraction of the 122 pairs would show that the benchmark does not accurately measure visual grounding.","tokens_in":2600,"feed_emoji":"🚗","tokens_out":690,"duration_ms":9582,"temperature":0.7,"pith_summary":"The paper creates a benchmark that tests multimodal models on both selecting the correct camera view and giving the right answer in six-view driving scenes. Standard benchmarks only score the final answer, allowing models to succeed by using the wrong visual input. The new test set uses 122 question-answer pairs drawn from 73 scenes chosen so that different views produce conflicting information on causality, counterfactuals, and intent. Three evaluation modes compare view selection alone, answering with the correct view supplied, and doing both in one step. This separation shows grounding mistakes that answer accuracy alone conceals, which matters when models are used for decisions that depend on specific visual evidence.","feed_headline":"Benchmark reveals MLLMs often answer from the wrong camera view","feed_subtitle":"122 questions across 73 driving scenes test whether models name the supporting view before giving their answer.","key_machinery":"Conflict-centric question-answer pairs with automatically mined and manually verified view labels that require explicit identification of one supporting camera among six synchronized views.","core_discovery":"The paper presents a multi-view VQA benchmark of 122 conflict-centric question-answer pairs from 73 NuScenes scenes. Models receive six synchronized camera views and must both name the supporting view and answer the question. View labels come from an automatic conflict-mining pipeline followed by manual verification. The benchmark runs in three settings—view selection, oracle QA with the golden view, and joint view-plus-answer prediction—and scores answers in multiple-choice and free-form formats. By measuring visual-source identification separately from answer correctness, the benchmark exposes cases where plausible answers rest on incorrect camera evidence.","pith_inferences":["The same separation of source identification from answer correctness could be applied to any multi-camera setting to make model decisions more auditable.","Explicit training signals for view selection might reduce the frequency of answers that ignore the relevant visual input.","In safety-critical domains, requiring models to report their evidence source could serve as an additional check before acting on an answer."],"forward_implications":["Answer-only metrics overestimate reliability because models can produce correct answers from the wrong view.","Joint view-and-answer prediction is strictly harder than either task performed separately.","Oracle performance with the golden view supplied gives an upper bound on what is possible once view selection is solved.","The benchmark covers causality, counterfactual reasoning, and intent prediction, so grounding failures appear across these task types."],"fun_headline_variants":["Benchmark tests MLLM camera view selection in driving scenes","MLLMs evaluated on supporting view identification for QA","New multi view benchmark checks evidence source in NuScenes","Joint view and answer prediction scored in 122 conflict pairs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The automatic conflict-mining pipeline combined with manual verification correctly identifies which camera view supports each of the 122 question-answer pairs.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark tests MLLM camera view selection in driving scenes","MLLMs evaluated on supporting view identification for QA","New multi view benchmark checks evidence source in NuScenes","Joint view and answer prediction scored in 122 conflict pairs"]},"model":"grok-4.3","cost_usd":0.004684,"raw_usage":{"total_tokens":2330,"prompt_tokens":698,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":46837000,"prompt_tokens_details":{"text_tokens":698,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1569,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":698,"tokens_out":63,"duration_ms":12030,"temperature":1.0,"reasoning_tokens":1569,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T16:39:18.803709+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A large-scale re-annotation showing that human judges disagree with the supplied view labels on more than a small fraction of the 122 pairs would show that the benchmark does not accurately measure visual grounding.","supporting_citations":[],"review_version":1}