{"id":"8ec73d8d-37e7-474c-8dfa-2f34ea2e3acf","arxiv_id":"2512.18735","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark tests whether large multimodal models can compare paired 'before and after' videos to detect scene changes, and finds current models perform near random.","lead":"This paper introduces M3-Verse, a benchmark that tests AI models' ability to spot differences between two videos of the same room before and after a change. It shows that even the best AI models score far below humans, suggesting current video AI struggles with tracking object and scene changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central gap claim assumes questions are visually answerable and the human baseline is visual-only; neither is verified—QA is generated from simulator metadata with no observability check, and the human protocol is unspecified.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the benchmark's construct validity depends on questions being visually answerable from the paired videos and on the human baseline being collected under visual-only conditions. I agree with this diagnosis and find no additional independent objection that would change the verdict. The paper does not provide per-question observability checks, and the human protocol is under-specified, so the central quantitative claim (45.14% vs 89.75%) is currently not fully supported. However, the benchmark data and construction pipeline are publicly released, making the required validation straightforward; the authors could add a visual-answerability audit and a detailed human-protocol description. These are addressable revisions rather than fundamental flaws, so conditional acceptance remains the appropriate verdict. I do not escalate to rejection because the benchmark's core contribution—a paired-video, multi-state QA dataset—is potentially valuable even if some questions need to be pruned or relabeled, and the authors have open-sourced the materials needed for community verification.","tokens_in":24246,"tokens_out":4070,"duration_ms":44349,"concrete_test":"Re-run the human evaluation on a stratified random sample (e.g., 200 questions, oversampling metric/temporal templates) with strict visual-only conditions—participants see only the paired egocentric videos, no metadata or simulator access—and compare per-question accuracy to the published 89.75% and to metadata-based answers. If visual-only human accuracy is materially lower (e.g., >10 points) on scene-area/distance/visit-count questions, the benchmark's construct validity fails and the reported gap is not a pure measure of multi-state visual reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states QA pairs are generated from scene metadata, and Appendix A.4's two-stage filtering only removes text-answerable questions (Stage 1) and questions a lightweight LMM can answer (Stage 2). No step verifies that the visual content of the paired egocentric videos is sufficient to determine the ground-truth answer. Many templates in Tables 4–6 ask for quantities not reliably recoverable from a first-person video: scene area, room area, metric distances (e.g., 'distance between yellow vase and statue'), exact visit counts, and total observation times. If these questions are unanswerable from vision, low LMM scores reflect ungroundable queries, not a deficit in state-change reasoning. Moreover, the human baseline of 89.75% is presented without a protocol: Appendix A.4 only mentions 12 reviewers 're-examine and screen the data,' not whether they watched the videos or had access to simulator metadata. If humans used metadata, the human–model gap is inflated. This is the load-bearing assumption: the interpretation of every headline result depends on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces M3-Verse, a benchmark for multi-state video understanding. For 270 ProcTHOR indoor scenes, the authors create paired egocentric videos of a scene before and after a programmatic state change (540 videos total), and automatically generate 2,932 multiple-choice QA pairs from simulator metadata using template-based generation. The questions are divided into intra-state and inter-state categories, span 50+ subtasks across four capabilities, and include 23.2% hallucination-type questions. After a two-stage LLM/LMM filtering pipeline and human review, the authors evaluate 16 LMMs, 6 text-only LLMs, a random baseline, and a human baseline. The best LMM (GPT-5) reaches 45.14% overall, compared with 89.75% for humans. The paper also proposes Hierarchical Captioning and Text-based Reasoning (HCTR), which improves several base models by converting video QA into caption-based text QA.","tokens_in":24393,"tokens_out":4578,"duration_ms":53522,"significance":"If the benchmark is valid, it fills a real gap: current benchmarks mostly test static or single-state video understanding, while M3-Verse targets the ability to compare two observations of the same scene and reason about state transitions. The construction has notable strengths: answers are grounded in simulator state rather than human annotations; the authors release both the construction pipeline and the full benchmark data; the evaluation covers a wide range of models including text-only controls; and the frame-sampling analysis is a useful sanity check. However, the central claim—that the human–model gap reflects a deficit in multi-state visual reasoning—depends on two unverified assumptions: that every question is visually answerable from the paired videos, and that the human baseline was collected under the same visual-only protocol. The presence of text-only models scoring far above random also indicates possible residual textual leakage. These issues are fixable but load-bearing.","major_comments":[{"comment":"The benchmark generation never verifies that the ground-truth answer is visually recoverable from the paired egocentric videos. Stage 1 (Fig. 15) only removes questions answerable from text, question, options, and answer; Stage 2 removes questions a lightweight LMM can answer; neither step checks observability. Many templates in Tables 4–6 ask for quantities that a first-person video cannot reliably support: scene area, room area, metric distances (e.g., 'distance between object1 and object2'), object height/length, room visit counts, exact visit order, and total observation times. If a substantial fraction of the 2,932 questions are not answerable from the provided frames, then low model scores conflate ungroundable queries with failures of state-change reasoning, and the headline human–model gap is not interpretable. I request an observability audit (per-template, with examples) or a r","section":"Section 3.2; Appendix A.4; Tables 4–6"},{"comment":"The human baseline of 89.75% is presented without a protocol. The appendix only says that 12 reviewers were recruited to 're-examine and screen the data' and to 'evaluate human performance,' but does not state whether the reviewers watched the paired videos, whether they had access to simulator metadata or backend logs, what options/format they saw, or how disagreement was resolved. Since the human number anchors the central 'stark performance gap' conclusion, this is load-bearing. The authors should specify the exact human evaluation protocol and, ideally, report a separate visual-only human result collected under the same constraints as the models (same frames, same options, no metadata).","section":"Appendix A.4; Table 2"},{"comment":"The text-only LLMs perform substantially above random even though Stage 1 is supposed to remove text-answerable questions. For example, Qwen3-14B reaches 33.78% and Qwen3-32B reaches 33.76% versus a random baseline of 24.14% (Table 2). Since these models receive no video input, any above-random accuracy indicates residual textual cues, option-order patterns, or language priors that the vision-free filtering (Fig. 15) did not remove. This weakens the claim that the benchmark isolates visual understanding and the conclusion that 'vision is essential.' I recommend measuring a stronger text-only ceiling (e.g., an LLM prompted with all options and the question, with option order ablated) and reporting per-template leakage statistics.","section":"Table 2; Appendix A.4 Stage 1"}],"minor_comments":[{"comment":"Model names are inconsistent/misspelled: 'LLaV A-OneVision-1.5-8B' should be 'LLaVA-OneVision-1.5-8B'. Also 'InternVL3.5-2B-Instruct*' etc. are fine, but the footnote markers are not explained in the caption.","section":"Table 2"},{"comment":"Typo: 'folllwing' should be 'following'. In Section 4.1 the bullet list contains 'r4we', likely a stray artifact. Please proofread.","section":"Section 4.2, finding 6"},{"comment":"The HCTR comparisons may not be apples-to-apples in compute/frame budget. The base models are evaluated with a fixed number of sampled frames (e.g., 200 per video), while HCTR uses clip captions with different and potentially larger total frame usage. Please state the total number of frames consumed by each HCTR variant and, if possible, compare against a base model with matched frame count.","section":"Section 5; Figures 5–6"}],"recommendation":"major_revision","confidential_remarks":"The benchmark has clear value and the authors have open-sourced the pipeline and data, which is commendable. The main risk is validity: the key result (human 89.75 vs. best LMM 45.14) is only interpretable if the questions are visually answerable and the human baseline is visual-only. Both need to be demonstrated, and the above-random text-only LLM scores suggest leakage that should be quantified. These are fixable with additional experiments and analysis rather than a full redesign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this benchmark fills a real gap — paired before/after videos testing multi-state change perception — and the authors did the work properly in many ways. But the central claim that SOTA LMMs are near-random while humans are at 90% depends on two assumptions that are not verified in the paper: that every question is visually answerable from the egocentric videos, and that the human baseline was collected under the same visual-only conditions. If either fails, the headline gap is misleading.\n\nWhat's new: the two-state paired-video format, 50+ subtasks across four capability axes, hallucination controls (23% \"no correct option\" items), and the backend metadata/depth/segmentation. That combination is not in the cited benchmarks. The construction pipeline is transparent, the data and code are public, and the evaluation covers 16 LMMs plus text-only LLMs with a standardized protocol. HCTR — hierarchical captioning then text reasoning — is a sensible baseline and shows real gains. The released assets deserve credit.\n\nSoft spots, in order: (1) Visual answerability is never checked. QA is generated from simulator metadata, and the filtering pipeline only removes text-answerable questions (Stage 1) and questions a lightweight LMM can answer (Stage 2). Nothing verifies that the correct answer can be determined from the frames alone. Many templates ask for metric distances, room/scene areas, exact visit counts, and total observation times — quantities that are not reliably recoverable from a first-person video. If a chunk of the 2,932 questions are ungroundable, low model scores reflect bad queries, not missing state-change reasoning. (2) Human protocol is underspecified. “12 reviewers re-examine and screen the data” and “evaluate human performance” — but did they watch the videos, or have access to simulator metadata? An 89.75% human score with questions about distances and areas seems inflated if metadata was available. (3) A milder issue: the multimodal filtering stage intentionally removes questions a lightweight LMM answers correctly, which biases the benchmark toward that model's weaknesses. That's a design choice, but it should be reported as a selection effect.\n\nNone of these are fatal. The benchmark is released, so the community can check answerability question-by-question. I'd send it to peer review with a request for a per-question observability audit and a detailed human-protocol description. If the authors close those gaps, this becomes a standard evaluation tool for multi-state understanding.","headline":"Useful paired-video benchmark with real novelty; headline gap rests on unverified question answerability and human protocol — fixable, worth reviewing.","tokens_in":24992,"tokens_out":2724,"would_cite":true,"duration_ms":29197,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current multimodal models fail at spotting changes across paired videos","keywords":["multimodal benchmark","video question answering","state change reasoning","egocentric video","spatial intelligence","hallucination evaluation","temporal reasoning"],"falsifier":"Give a fresh group of human annotators the same 2,932 questions with access only to the videos and not to any simulator metadata, and compare their accuracy to the reported 89.75%; a significant drop would show that the human baseline and the model gap are partly an artifact of metadata, not pure visual understanding.","tokens_in":24064,"feed_emoji":"🎥","tokens_out":3744,"duration_ms":37009,"temperature":0.7,"pith_summary":"The paper introduces M3-Verse, a benchmark that tests whether large multimodal models can detect and reason about changes in an indoor scene by comparing two first-person videos taken before and after a state change. It comprises 270 scenes, 2,932 multiple-choice questions, and over 50 subtasks spanning spatial, temporal, attribute, and reasoning skills. The central finding is a large gap: the best model scores 45.14%, while humans score 89.75%. The paper argues this gap reveals a missing capability in current models that is essential for embodied intelligence, and proposes a simple captioning-and-text-reasoning baseline that raises scores by several points.","feed_headline":"Best AI model scores 45% on spot-the-difference video benchmark","feed_subtitle":"New paired-video benchmark shows large multimodal models struggle to compare before/after states.","key_machinery":"The central object is the paired two-state video: each scene is recorded by an embodied agent exploring it once before and once after an object-level change, so answering inter-state questions requires comparing the two observations rather than recognizing a single state. Questions are generated from simulator metadata through a template pipeline with hierarchical entity grounding to avoid ambiguous references, then filtered in two stages (text-only answerability and multimodal triviality). The proposed HCTR baseline works by turning each video into time-stamped clip captions, summarizing them, and answering from text alone.","core_discovery":"On its own terms, the paper establishes that state-of-the-art large multimodal models cannot reliably compare two observations of the same scene to identify what changed. With paired egocentric videos of 270 indoor scenes before and after a programmatic change, and 2,932 questions, the best evaluated model reaches 45.14% accuracy versus 89.75% for human evaluators. The benchmark also shows that models are particularly prone to hallucination-type questions, that larger scale does not guarantee better performance, and that converting videos into hierarchical text captions before answering improves inter-state reasoning scores. The paper treats M3-Verse as a shared testbed for a capability exis","pith_inferences":["A key unstated risk is that the ground-truth answers are derived from simulator metadata, and the filtering pipeline only removes questions answerable from text, not questions unanswerable from the video pixels; if many questions ask for metric distances or areas that a first-person video cannot convey, the human baseline may overstate what is visually recoverable.","The benchmark's release of depth maps, instance masks, and operation logs invites future work that uses these as supervision; a model trained on them might be expected to far exceed the reported 45%.","The intra-state tasks being harder than inter-state tasks for many models is counterintuitive and may reflect a 'needle in a haystack' localization problem; a fair test of state-change reasoning would be to control for localization difficulty.","One testable extension: measure human performance on the same videos without any access to simulator metadata or question templates; if human accuracy drops substantially, the benchmark's visual-grounding claim would need qualification."],"forward_implications":["If the gap is real, current multimodal models are not reliable for embodied applications where an agent must check what changed after an action.","The finding that inter-state reasoning improves under hierarchical captioning suggests that explicitly serializing video content into text helps models track change.","Hallucination-type questions expose a systematic tendency of models to commit to an option even when the correct answer is 'no correct option is listed'.","Denser frame sampling improves performance, implying state-change perception is partly limited by temporal resolution rather than pure reasoning.","Model scale alone does not close the gap; architectural and instruction-tuning choices matter more."],"fun_headline_variants":["Multimodal models fail to spot video changes: best scores 45%","AI vs humans: 45% vs 90% on new video change benchmark","New benchmark exposes AI weakness in tracking scene changes","Video spot-the-difference: models lag far behind humans","Only 45% accuracy: LLMs stumble on video state changes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark assumes every question can be answered from the paired videos alone, but the QA pairs come from simulator metadata and the filtering only removes text-answerable questions, never verifying that the correct answer is visually recoverable from the frames.","fun_headline_variants_meta":{"raw":{"variants":["Multimodal models fail to spot video changes: best scores 45%","AI vs humans: 45% vs 90% on new video change benchmark","New benchmark exposes AI weakness in tracking scene changes","Video spot-the-difference: models lag far behind humans","Only 45% accuracy: LLMs stumble on video state changes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1063,"prompt_tokens":783,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":190}},"tokens_in":527,"tokens_out":280,"duration_ms":3476,"temperature":1.0,"reasoning_tokens":190,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:53:53.991165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give a fresh group of human annotators the same 2,932 questions with access only to the videos and not to any simulator metadata, and compare their accuracy to the reported 89.75%; a significant drop would show that the human baseline and the model gap are partly an artifact of metadata, not pure visual understanding.","supporting_citations":[],"review_version":1}