{"id":"e1c93059-0c2f-47d9-b5de-cba4cfd55ebe","arxiv_id":"2508.11058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors build a multi-view 3D QA benchmark and a 1M-triplet 2D-3D-text pretraining corpus, and show a model trained on them achieves state-of-the-art results on several 3D vision-language benchmarks.","lead":"This paper introduces MV-ScanQA, a question-answering benchmark where most questions require a model to combine information from multiple camera views of a 3D scene. It also introduces TripAlign, a million-sample pretraining corpus that aligns 2D views with groups of 3D objects and text, and a model, LEGO, that improves state-of-the-art results on several 3D vision-language tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 68% multi-view statistic and the TripAlign/LEGO object filter both rest on an unvalidated IoSA visibility rule that ignores occlusion; a depth-aware re-derivation is needed.","rationale":"The paper has real strengths: a concrete benchmark construction, an automated pretraining corpus, and a baseline that improves on many metrics. The solvability analysis is, however, the hinge on which the novelty claim turns, and it is computed with a proxy that has not been validated. The reader's CONDITIONAL verdict already anticipates this by flagging the IoSA threshold; my stress test broadens the flag to occlusion and projection accuracy, which makes the concern more concrete but does not escalate it. A depth-aware re-derivation is feasible and would settle the question, so I leave the verdict at CONDITIONAL/UNCHANGED. No claim of misconduct is made; the issue is an unvalidated operationalization of 'visibility.'","tokens_in":12744,"tokens_out":6601,"duration_ms":71633,"concrete_test":"Build an occlusion-aware visibility oracle using ScanNet depth maps (or mesh raycasting) for the views used in Sec. 3.1 and Sec. 5; classify an object as witnessed only if at least some of its 3D surface is depth-visible. Compare this oracle with the IoSA>0.5 rule on the question-object-view triples, sweeping τ over {0.3,0.5,0.7}. Recompute the percentages of ScanQA/ScanRefer/Nr3D instructions solvable from one view and the 68% multi-view fraction for MV-ScanQA under the oracle. If the existing-benchmark single-view percentages change by more than 10 points, or the MV-ScanQA multi-view fraction falls below about 50%, the paper's motivating statistics—and the filtering used for TripAlign and LEGO—are not robust to a reasonable visibility model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines a question as solvable from a view set when every relevant object is 'witnessed,' where an object is witnessed if the IoSA between its projected 2D box and the image area exceeds 0.5. The same IoSA with threshold τ is the in-view object filter in Section 5 and thus controls which objects are aligned with text in TripAlign and LEGO. Two properties of this metric are load-bearing but unvalidated. (i) The 0.5 threshold is arbitrary: no calibration, ablation, or human agreement study is reported, and the paper does not state τ for the Section 5 filter; sweeping τ would change both the 68% MV-ScanQA statistic and the training-signal filtering. (ii) IoSA is a projection-overlap test, not a visibility test: it does not check occlusion by walls, furniture, or other objects, and it assumes camera intrinsic/extrinsic correctness. In ScanNet, an object behind a wall can project inside the image rectangle and be counted as witnessed, so single-view solvability of ScanQA/ScanRefer/Nr3D may be overestimated and the claimed 'single-view bottleneck' overstated, while MV-ScanQA's multi-view fraction could shift in either direction. Because the same flawed oracle is used to construct the benchmark's headline statistic and to filter LEGO's training triplets, the paper's two central claims are coupled to this unvalidated geometric proxy. The 100-image manual verification checks QA text quality, not visibility, so it does not address this gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper makes three contributions: MV-ScanQA, a 3D question-answering benchmark built by LLM-based composition of ScanQA questions, with 68% of questions claimed to require multiple egocentric views; TripAlign, an automatically generated corpus of 1M <2D view, set of 3D objects, text> triplets; and LEGO, a 2D-3D large vision-language model built on Fuyu that uses informative 2D views and a view-dependent object filter. The authors report a solvability analysis in which 94%, 96%, and 93% of ScanQA, ScanRefer, and Nr3D instructions are solvable from a single view, and report state-of-the-art results for LEGO on ScanRefer, Nr3D, ScanQA, and MV-ScanQA. The empirical improvements are consistent and the datasets/code are promised for release, but the central multi-view statistics and a key training-signal filter both depend on an unvalidated IoSA visibility proxy.","tokens_in":13040,"tokens_out":5827,"duration_ms":61118,"significance":"If the multi-view requirement and downstream gains hold, the paper supplies a more demanding benchmark, a scalable multi-object pre-training signal, and a strong baseline; the release of datasets and code is a concrete community asset. The solvability analysis is a useful diagnostic that goes beyond prior benchmark papers. However, the two headline claims—the 68% multi-view statistic and the value of TripAlign's object filtering—rest on a single unvalidated visibility proxy, so the significance cannot be fully assessed until that proxy is checked.","major_comments":[{"comment":"The IoSA visibility criterion (IoSA > 0.5 in Section 3.1) is load-bearing for both central claims: it produces the 94%/96%/93% single-view solvability rates and the 68% multi-view rate for MV-ScanQA, and it is reused in Section 5 as the in-view object filter for TripAlign and LEGO. The threshold is introduced without calibration or ablation, the tau used for the Section 5 filter is not stated, and the projection-overlap test ignores occlusion by walls or furniture and camera pose error. Because the same proxy drives the benchmark statistic and the training-signal filter, any bias in it moves both claims together. Please (i) report tau in every use, (ii) sweep tau and show the sensitivity of the solvability percentages and of Table 6, and (iii) validate 'witnessed' labels against depth-based occlusion tests or human annotation on a stratified sample.","section":"Section 3.1 and Section 5"},{"comment":"All main result tables report single point estimates with no standard deviations, confidence intervals, or significance tests. Several reported advantages are small (e.g., Table 4, ScanQA test EM 28.4 vs 27.0; Table 3, ScanRefer C@0.25 84.66 vs 81.94), and without variance over seeds the 'state-of-the-art on all 9 metrics' claim is not yet supported. Please provide at least three seeds with mean and standard deviation for the main comparisons, or a paired significance test for the ablated variants.","section":"Tables 2–6"},{"comment":"The dataset quality statement rests on 'Manual verification on 100 images shows 94% questions are synthesized correctly,' but the manuscript gives no sampling protocol, annotator instructions, inter-annotator agreement, or error taxonomy. This check validates question-answer text, not the visibility labels behind the 68% statistic. Please report a detailed verification protocol, and consider verifying a larger stratified sample of question-view assignments.","section":"Section 3.2"}],"minor_comments":[{"comment":"The concluding paragraph is grammatically incomplete ('more than 10×Second'); please fix the sentence.","section":"Section 7"},{"comment":"The text uses 'CiDER' and 'CiDER metrics'; the standard name is CIDEr, and the metric should be defined consistently.","section":"Section 6.3"},{"comment":"The sentence 'we investigate the effectiveness two data pipelines of of TripAlign' contains a typo and an incomplete phrase; please rewrite.","section":"Table 6"},{"comment":"The claim of state-of-the-art on 'all 9 metrics' is unclear because Tables 3 and 4 together report 8 metrics (3 in Table 3, 5 in Table 4); please clarify the ninth metric or correct the count.","section":"Section 6.3"},{"comment":"The columns 'w/ obj' and 'w/o obj' are used without definition in the text; please state what these conditions mean.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The artifacts and empirical study are valuable, and the work is within the scope of a multimedia venue. My main concern is that the benchmark's headline statistic and the training-signal filter share an unvalidated geometric criterion; this is fixable with additional validation experiments and sensitivity analyses. I would not reject the paper, but I cannot recommend acceptance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper ships a genuinely useful artifact: MV-ScanQA is a new compositional QA benchmark that is clearly harder than existing single-view 3D QA, and TripAlign's 1M 2D-view/3D-object/text triplets are a plausible way to densify pretraining signal. Second, the paper's load-bearing statistics—68% of MV-ScanQA questions requiring multiple views, and the claim that over 93% of existing benchmark questions are single-view solvable—are computed with an IoSA projection-overlap heuristic that ignores occlusion. That metric is used both to label the benchmark and to filter objects during LEGO training, so a flaw there couples both central claims.\n\nWhat the paper does well: the compositional question synthesis from shared-anchor QA pairs is principled, the TripAlign construction via image captioning plus retrieval is a reasonable low-cost pipeline, and LEGO's design (2D LVLM backbone, 3D detector features, view-dependent object filtering) is a sensible baseline. The ablations in Tables 5 and 6 are consistent: both the view extension and the multi-object alignment improve results across ScanRefer, Nr3D, and ScanQA. The reported SOTA numbers on existing benchmarks, if reproducible, would be a solid contribution.\n\nSoft spots, in rough order of severity. First, IoSA with τ=0.5 is arbitrary and untested—no ablation, no human agreement, no comparison against depth-based visibility. An object behind a wall can project into the image rectangle and count as witnessed, so the single-view solvability of ScanQA/ScanRefer/Nr3D could be overestimated and the 68% figure for MV-ScanQA could shift. The paper does not even state τ for the Section 5 filter. Second, there are no error bars or significance tests anywhere in Tables 2–6; for a new benchmark and a new training set, that matters. Third, the manual verification is only on 100 samples with no selection details, which is thin for a ~10K question dataset. Fourth, the novelty claim of being “first” to construct tri-modal triplets is slightly overstated given SceneLLM and SceneVerse already use 2D views, even if not at this scale. Datasets and code are promised but were not available at review time.\n\nThese issues do not sink the paper. The central direction is sound, and even if the exact multi-view percentages change, MV-ScanQA will still be a more demanding benchmark than its predecessors. But the authors need to re-derive the solvability analysis with depth-aware occlusion checks and ablate τ, and they should release data and code before anyone builds on the specific numbers.\n\nWho is this for? Anyone working on 3D vision-language pretraining or benchmark design. It deserves a serious referee: I'd send it to review, but ask the authors to address the metric validation and add error bars before acceptance.","headline":"Useful new 3D VLM benchmark and pretraining corpus, but the headline multi-view statistics rest on an unvalidated projection overlap metric that should be fixed before the numbers are cited.","tokens_in":13607,"tokens_out":2127,"would_cite":true,"duration_ms":24780,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that existing 3D vision-language benchmarks are mostly single-view solvable, and answers with MV-ScanQA, a 68% multi-view benchmark, the TripAlign pretraining corpus, and the LEGO model that tops those benchmarks.","keywords":["3D vision-language learning","multi-view reasoning","question answering","dataset construction","2D-3D pretraining","multimodal alignment","scene understanding"],"falsifier":"Re-run the solvability analysis on ScanQA, ScanRefer, Nr3D, and MV-ScanQA with the visibility threshold set to 0.3 and to 0.7, keeping the same camera projections; if MV-ScanQA's 68% multi-view share drops toward the single digits at either setting, the multi-view bottleneck is an artifact of the threshold rather than a property of the questions. Complementarily, if human annotators answered MV-ScanQA questions correctly while seeing only the best single view, the questions would not truly need multiple views.","tokens_in":12537,"feed_emoji":"🧩","tokens_out":12426,"duration_ms":118383,"temperature":0.7,"pith_summary":"Existing 3D vision-language benchmarks are far easier than they appear: under the paper's solvability analysis, 94% of ScanQA, 96% of ScanRefer, and 93% of Nr3D instructions can be answered from a single egocentric view. To remove that single-view bottleneck, the paper builds MV-ScanQA, a question-answering benchmark in which 68% of questions require integrating at least two views and 13% require three or more, synthesized by an LLM from pairs of questions that share an anchor object but need complementary information. It also contributes TripAlign, a fully automatic one-million-triplet corpus of (2D view, set of visible 3D objects, text), whose grouped-object supervision is denser than existing single-object annotations. Trained on TripAlign, the LEGO model built atop a frozen 2D vision-language model reaches state-of-the-art results on ScanQA, ScanRefer, and Nr3D, and improves markedly on MV-ScanQA when given multiple views. The value of the package is a harder, more honest 3D evaluation and a scalable way to train for it.","feed_headline":"68% of new 3D QA questions need multiple views to answer","feed_subtitle":"It forces view integration; a 1M-triplet corpus backs a model that tops existing 3D QA and captioning.","key_machinery":"The load-bearing mechanism is the Intersection-over-Smaller-Area (IoSA) visibility metric, $\\mathrm{IoSA}(A,B)=\\frac{\\mathrm{Area}(A\\cap B)}{\\min(\\mathrm{Area}(A),\\mathrm{Area}(B))}$, applied to a 3D object's projected 2D box against an egocentric view. IoSA with threshold 0.5 does three jobs: it defines which objects are witnessed in the solvability analysis that yields the 94/96/93% single-view figures for existing benchmarks; the same analysis marks MV-ScanQA's 68% multi-view share; and LEGO's view-dependent multi-object alignment uses the same projection and threshold to filter 3D object proposals down to those actually visible in the paired view. Around this metric, the other machinery is data construction: LLM-based compositional question synthesis for MV-ScanQA, and the TripAlign dual pipeline that produces 1M triplets by generating captions from egocentric views with image captioners, filtering them by image-text retrieval, and attaching informative views to existing QA and captioning samples.","core_discovery":"The paper's central discovery is a measurement plus a remedy. Measured with an Intersection-over-Smaller-Area visibility rule, existing 3D vision-language benchmarks are single-view solvable almost all the time, so their scores do not certify genuine multi-view 3D understanding. MV-ScanQA fixes the measurement by composing pairs of existing single-view questions into new ones that share an anchor object but require complementary information, pushing the multi-view requirement to 68% of questions (13% needing three or more views); manual checking of 100 samples finds 94% correctly formed. TripAlign fixes the data side by generating, from egocentric views of ScanNet scenes, over one million triplets that couple a 2D view, the set of 3D objects visible in it, and a caption, using image captioners and retrieval filters, plus an extension that attaches informative views to existing QA and captioning samples. LEGO then shows the remedy works: built on a frozen 2D vision-language model with a 3D detector and LoRA, using the same visibility projection to mask out-of-view objects, it achieves state-of-the-art scores on ScanQA, ScanRefer, and Nr3D, and on MV-ScanQA its multi-view variant scores 34.1 exact-match versus 30.0 for single-view, with the largest gain on questions needing four or more views.","pith_inferences":["The paper does not calibrate the IoSA threshold of 0.5 against human visibility judgments; if that threshold drifts, both the 68% difficulty statistic and LEGO's object filtering would move, so the benchmark's difficulty gradient is best read as conditional on that choice.","The same triplet structure could be reused for tasks that also need viewpoint-dependent object grouping, such as referring expression segmentation or embodied instruction following, since it already provides the alignment between text, visible objects, and a specific view.","A direct control would compare TripAlign's view-grouped triplets against random object groups of the same size; if random groups transfer nearly as well, the value may come from scale and caption quality rather than from view-based grouping.","Because TripAlign is generated automatically from ScanNet, it could be extended to other RGB-D scan collections or to synthetic scenes, which would test whether the approach depends on egocentric human-captured views or works with any view distribution."],"forward_implications":["MV-ScanQA resets the bar for 3D QA: a model given one informative view scores 30.0 exact-match, while the same model with four spliced views scores 34.1, with the largest gap on questions needing four or more views (26.5 to 30.9).","TripAlign's two data pipelines each add measurable value: view extension alone improves ScanRefer C@0.25 by +4.4 and ScanQA exact-match by +2.1, and adding generated triplets yields cumulative gains of +10.5, +8.4, and +3.3 on ScanRefer, Nr3D, and ScanQA.","The view-dependent multi-object alignment mask is beneficial: removing it lowers ScanRefer C@0.25 by +3.6, Nr3D C@0.5 by +0.4, and ScanQA exact-match by +0.5, showing that filtering out-of-view objects improves 2D-to-3D transfer.","Because LEGO reaches state-of-the-art results across ScanQA, ScanRefer, and Nr3D in addition to MV-ScanQA, multi-object, view-grounded pretraining transfers to tasks whose annotations remain single-object.","The LLM-guided composition recipe, which pairs questions with a shared anchor and non-nested object sets, is a general procedure that can convert other single-view QA datasets into multi-view evaluations."],"supporting_citations":[{"why":"Supplies the Nr3D benchmark whose 7% multi-view solvability is measured and on which LEGO is evaluated.","marker":"[1]"},{"why":"Supplies ScanQA, the base dataset whose 6% multi-view rate motivates MV-ScanQA and whose split and metrics are used for QA evaluation.","marker":"[2]"},{"why":"Fuyu is the frozen 2D vision-language model backbone whose visual-language alignment LEGO transfers to the 3D domain.","marker":"[3]"},{"why":"Supplies ScanRefer, the dense-captioning benchmark analyzed and evaluated in the paper.","marker":"[4]"},{"why":"Vote2Cap-DETR is the 3D detector that extracts the object features fed into LEGO.","marker":"[6]"},{"why":"Defines the CIDEr-style metrics used to score dense captioning on ScanRefer and Nr3D.","marker":"[8]"},{"why":"ScanNet provides the 3D scenes and egocentric views that both MV-ScanQA and TripAlign are built from.","marker":"[9]"},{"why":"As a pre-trained captioner, generates the view-specific captions that become text in TripAlign triplets.","marker":"[22]"},{"why":"As the other pre-trained captioner, also generates egocentric-view descriptions for TripAlign.","marker":"[26]"},{"why":"As the image-text retrieval filter, keeps only captions that genuinely match the view in TripAlign.","marker":"[29]"}],"fun_headline_variants":["Multi-view 3D QA dataset forces distant object reasoning","TripAlign: 1M triplets pre-train 3D VLMs for multi-view QA","New 3D QA benchmark: 68% of questions need multiple views","LEGO tops 3D QA and captioning after TripAlign pre-training","MV-ScanQA: 68% of questions require view integration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the 0.5 visibility threshold in the IoSA metric, the rule that an object counts as seen only if its projected box covers more than half of the view, because the same threshold sets the 68% multi-view statistic and decides which objects LEGO aligns with text.","fun_headline_variants_meta":{"raw":{"variants":["Multi-view 3D QA dataset forces distant object reasoning","TripAlign: 1M triplets pre-train 3D VLMs for multi-view QA","New 3D QA benchmark: 68% of questions need multiple views","LEGO tops 3D QA and captioning after TripAlign pre-training","MV-ScanQA: 68% of questions require view integration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001008,"raw_usage":{"total_tokens":4357,"prompt_tokens":1137,"completion_tokens":3220,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":753,"completion_tokens_details":{"reasoning_tokens":3130}},"tokens_in":753,"tokens_out":3220,"duration_ms":24736,"temperature":1.0,"reasoning_tokens":3130,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:27:58.313088+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the solvability analysis on ScanQA, ScanRefer, Nr3D, and MV-ScanQA with the visibility threshold set to 0.3 and to 0.7, keeping the same camera projections; if MV-ScanQA's 68% multi-view share drops toward the single digits at either setting, the multi-view bottleneck is an artifact of the threshold rather than a property of the questions. Complementarily, if human annotators answered MV-ScanQA questions correctly while seeing only the best single view, the questions would not truly need multiple views.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ScanRefer, the dense-captioning benchmark analyzed and evaluated in the paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Vote2Cap-DETR is the 3D detector that extracts the object features fed into LEGO."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CIDEr-style metrics used to score dense captioning on ScanRefer and Nr3D."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Nr3D benchmark whose 7% multi-view solvability is measured and on which LEGO is evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ScanQA, the base dataset whose 6% multi-view rate motivates MV-ScanQA and whose split and metrics are used for QA evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fuyu is the frozen 2D vision-language model backbone whose visual-language alignment LEGO transfers to the 3D domain."}],"review_version":2}