DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

Guirong Wang; Huixuan Zhang; Hu Wei; Junzhe Zhang; Lin Qu; Pei Liu; Xiaojun Wan; Xingyao Zhang

REVIEW 3 major objections 2 minor 19 references

DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

T0 review · 3 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A benchmark of real-world videos shows current multimodal large language models still lack causal counterfactual reasoning.

desk verdict Real-world video benchmark for MLLM causal reasoning with dynamic graph intervention, but thin on construction and result details. read the letter →

arxiv 2605.29339 v1 pith:Q62WBSKT submitted 2026-05-28 cs.CV

Junzhe Zhang , Huixuan Zhang , Guirong Wang , Xingyao Zhang , Pei Liu , Lin Qu , Hu Wei , Xiaojun Wan This is my paper · ORCID

classification cs.CV

keywords multimodallargelanguagemodelscausalreasoningcounterfactualquestionsbenchmarkconstructiondynamicevaluationreal-worldvideosdatacontaminationgraphintervention

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds DMC-CF-Static by collecting real-world videos and turning causal events into counterfactual questions to test whether multimodal models grasp cause and effect rather than surface patterns. It then introduces the Dynamic Graph Intervention framework, which encodes events as causal graphs and generates fresh question variants on the fly, creating DMC-CF-Dynamic to reduce the risk that models simply memorize training data. Results across both parts of the benchmark indicate that existing models perform poorly, suggesting statistical training alone does not produce genuine understanding of real-world causality. A sympathetic reader would care because reliable decision-making in changing environments requires models to handle interventions they have never seen before.

What carries the argument

The Dynamic Graph Intervention (DGI) framework, which encodes causal events as graphs and generates dynamic question variants to create contamination-resistant evaluation sets from the static benchmark.

What would settle it

A multimodal model that scores above 80 percent accuracy on the dynamic benchmark after training only on the static videos without exposure to the graph interventions would indicate that the benchmark does not fully test causal reasoning.

Watch

Extended reading notes

Core claim

The authors construct DMC-CF-Static from real-world videos as a large-scale benchmark for multimodal causal counterfactual reasoning and apply the Dynamic Graph Intervention framework to derive DMC-CF-Dynamic from causal graphs; experiments on the combined benchmark demonstrate that the multimodal causal reasoning capabilities of current multimodal large language models in real-world scenarios still require substantial improvement.

Load-bearing premise

Real-world videos collected and represented via causal graphs provide an unbiased and contamination-free measure of causal reasoning capabilities.

Editorial extensions

If this is right

Existing multimodal models trained by statistical learning do not reliably capture underlying causal relationships in videos.
Synthetic or cartoon-based causal datasets are insufficient proxies for real-world performance.
Future model development must incorporate mechanisms that handle dynamic interventions rather than static pattern matching.
The DMC-CF benchmark supplies a concrete testbed for measuring progress toward causal understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the DGI method successfully blocks contamination, similar graph-based dynamic generation could be applied to other video or image reasoning tasks.
Persistent failure on the benchmark raises the possibility that scaling model size alone will not close the gap without changes to training objectives.
The benchmark could be extended by adding new video domains or by measuring how well models generalize to unseen causal graphs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The paper's main contribution is DMC-CF, a large-scale benchmark drawn from real-world videos for testing multimodal causal counterfactual reasoning, along with the Dynamic Graph Intervention framework for a dynamic evaluation set.

This moves past the synthetic or small-scale datasets that have been common. Representing events with causal graphs and intervening on them for dynamic tests is a distinct way to handle contamination.

The paper does a solid job of motivating the need for more realistic data and proposing a framework to address evaluation issues.

The central claim is that current MLLMs still require substantial improvement in real-world causal reasoning, based on results from both the static and dynamic parts of the benchmark.

Where it is thinner is in the details we can see. The abstract mentions the collection and construction but provides no information on methodology for building the graphs, ensuring quality, or the actual model results and error analysis. That makes it hard to judge whether the benchmark truly measures what it claims or if the reported limitations in MLLMs hold up. Without seeing how they collected the videos or verified the causal graphs, it's difficult to know if the benchmark avoids its own biases or if the DGI framework actually works as intended in practice.

The assumption that real videos plus graph interventions give an unbiased test also needs scrutiny once the full construction is available.

This paper is for researchers working on multimodal large language models and causal reasoning benchmarks. Anyone building evaluations in this space could find the dynamic approach worth examining.

It should go through peer review so the construction and results can be properly checked.

Referee Report

3 major / 2 minor

Summary. The paper introduces DMC-CF, a benchmark for multimodal counterfactual QA on causal reasoning. DMC-CF-Static is constructed from real-world videos represented via causal graphs; DMC-CF-Dynamic is derived from it via the Dynamic Graph Intervention (DGI) framework to enable dynamic evaluation that mitigates data contamination. Experiments on the combined benchmark conclude that current MLLMs still require substantial improvement in real-world multimodal causal reasoning.

Significance. If the benchmark construction, validation, and experimental results hold, the work supplies a large-scale, real-world resource that improves on prior synthetic or small-scale causal-reasoning datasets and introduces a dynamic intervention mechanism to reduce contamination. This could become a useful standard for diagnosing limitations of statistical learning in MLLMs.

major comments (3)

[Abstract / Results] The central claim rests on experimental results showing MLLM limitations, yet the manuscript provides no quantitative tables, model names, accuracy numbers, or error analysis in the abstract or visible sections; without these, the strength of the conclusion cannot be evaluated.
[DGI framework description] The claim that DGI effectively addresses data contamination (weakest assumption) is load-bearing for DMC-CF-Dynamic; the description does not include concrete validation such as contamination-rate measurements before/after intervention or comparison against static baselines.
[Benchmark construction] The assertion that real-world videos yield an unbiased measure of causal reasoning requires supporting statistics (e.g., diversity of scenes, causal-graph complexity distribution, or inter-annotator agreement on graph construction); these are not referenced.

minor comments (2)

[Notation / §2] Notation for causal graphs and intervention operators should be defined explicitly with an example figure early in the paper.
[Abstract / Dataset statistics] The abstract states the benchmark is 'large-scale' but supplies no size statistics (number of videos, QA pairs, graph nodes); these should appear in a table in the main text.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which identify opportunities to strengthen the presentation of quantitative results and the empirical validation of our methods. We respond to each major comment below.

read point-by-point responses

Referee: [Abstract / Results] The central claim rests on experimental results showing MLLM limitations, yet the manuscript provides no quantitative tables, model names, accuracy numbers, or error analysis in the abstract or visible sections; without these, the strength of the conclusion cannot be evaluated.

Authors: We agree that the abstract would benefit from explicit quantitative highlights to allow immediate evaluation of the central claims. Detailed results, including the specific MLLMs evaluated, accuracy numbers, and error analysis, appear in Section 4 and the accompanying tables. We will revise the abstract to include a concise summary of key performance metrics. revision: yes
Referee: [DGI framework description] The claim that DGI effectively addresses data contamination (weakest assumption) is load-bearing for DMC-CF-Dynamic; the description does not include concrete validation such as contamination-rate measurements before/after intervention or comparison against static baselines.

Authors: Section 3.3 describes the DGI mechanism for generating dynamic counterfactual questions via graph interventions. We acknowledge that explicit before-and-after contamination measurements and static-baseline comparisons would provide stronger support. We will add a dedicated validation subsection with these quantitative analyses in the revised manuscript. revision: yes
Referee: [Benchmark construction] The assertion that real-world videos yield an unbiased measure of causal reasoning requires supporting statistics (e.g., diversity of scenes, causal-graph complexity distribution, or inter-annotator agreement on graph construction); these are not referenced.

Authors: Section 3.1 outlines the real-world video collection and causal-graph construction process. To directly address the request for supporting statistics, we will add a summary table and explicit references to scene diversity, graph-complexity distributions, and inter-annotator agreement in the main text. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

This is an empirical benchmark paper focused on data collection from real-world videos, causal graph construction, and introduction of the DGI framework for dynamic evaluation. No mathematical derivations, equations, fitted parameters presented as predictions, or self-citation chains appear in the abstract or described content. The central claim (MLLMs require improvement on DMC-CF) rests on experimental results from the new benchmark rather than any reduction to inputs by construction. Self-citations, if present, are not load-bearing for any derivation. This matches the expected non-finding for benchmark construction work.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review supplies no information on free parameters, axioms, or invented entities underlying the benchmark construction or evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning." pith.science (2026). https://pith.science/paper/Q62WBSKT

@misc{pith2026260529339,
  author       = {Pith},
  title        = {Pith review of: DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q62WBSKT}},
  note         = {Machine review of arXiv:2605.29339}
}

read the original abstract

With the rapid advancement of multimodal large language models (MLLMs), models have demonstrated increasingly powerful multimodal capabilities. However, whether MLLMs trained through statistical learning can truly understand the causal relationships underlying the real world remains a key research question. In recent years, numerous multimodal causal reasoning datasets have been proposed. Nevertheless, these datasets are either limited in scale or constructed from synthetic images and videos, cartoon-based content, or other non-realistic multimodal sources. To address these limitations, we collect real-world videos and construct DMC-CF-Static, a large-scale benchmark for multimodal causal counterfactual reasoning. Furthermore, to mitigate issues such as data contamination in traditional static evaluation, we represent causal events using causal graphs and propose the Dynamic Graph Intervention (DGI) framework to build the dynamic evaluation benchmark DMC-CF-Dynamic from DMC-CF-Static. Experimental results on the overall DMC-CF, which includes both static and dynamic evaluation benchmarks, demonstrate that the multimodal causal reasoning capabilities of current multimodal large language models in real-world scenarios still require substantial improvement.

Figures

Figures reproduced from arXiv: 2605.29339 by the authors.

**Figure 2.** Distribution of Causality Types. we are inspired to construct a multimodal causal counterfactual benchmark capable of supporting dynamic evaluation. 3 DMC-CF-Static 3.1 Video Collection We first manually select real-world videos from public accessible video streaming platform such as YouTube that satisfy our following proposed requirements. We require the videos to originate from real-world sources rather than AI-ge… view at source ↗

**Figure 3.** An example of our annotated sample. timately gathered 1614 distinct videos that satisfy our requirements. 3.2 Data Annotation Annotators are first asked to identify the video segments corresponding to causal events and annotate the associated metadata, such as video descriptions and lists of causal events. They are then required to label the start and end timestamps corresponding to each causal event in the video. … view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: An example of Causal Graph. The left side of the figure illustrates the static causal graph corresponding to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Model performance across video types. gories, demonstrating the strongest overall multimodal causal reasoning capability. The secondbest overall performance is achieved by Qwen3VLThinking, which is consistent with the results reported in [PITH_FULL_IMAGE:figures/f…

**Figure 6.** Figure 6: Video collection guideline. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Video annotation guideline. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Video counterfactual QA quality evaluation guideline [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: Static causality graph extraction prompt. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: Intervention causality graph extraction prompt. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

**Figure 11.** Figure 11: Prompt for generating L2_N. Prompt for Generating L2_Y We conducted a counterfactual intervention experiment based on the video. Here, we provide the video clip and the corresponding causal graph extracted after the intervention. Please perform the required interventi…

**Figure 12.** Figure 12: Prompt for generating L2_Y . 15 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 2 canonical work pages

[1]

InProceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974

Intentqa: Context-aware video intent reason- ing. InProceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974. Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, and Weidong Cai. 2025. Mul- timodal causal reasoning benchmark: Challenging multimodal large language models to discern causal links across mo...

work page arXiv 2025
[2]

Junzhe Zhang, Huixuan Zhang, and Xiaojun Wan

Kbe-dme: Dynamic multimodal evaluation via knowledge enhanced benchmark evolution.Preprint, arXiv:2510.21182. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. Dyval: Dy- namic evaluation of large language models for rea- soning tasks.Preprint, arXiv:2309.17167. A Human Annotation Details A.1 Human Annotation Guidel...

work page arXiv 2024
[3]

If multiple identical objects exist and cannot be unambiguously referred to or distinguished linguistically, the video should not pass screening

Objects involved in the causal event must be clearly identifiable in the video and referable through language (e.g., wooden blocks, wooden boards, etc., should correspond accurately to visible objects in the video). If multiple identical objects exist and cannot be unambiguously referred to or distinguished linguistically, the video should not pass screening
[4]

Click to view counterexamples (e.g., in the referenced video, the coin collision can only be explained through force analysis, and the angles are difficult to describe clearly)

The causal relationship should be intuitive, and the causal event should be easy to describe. Click to view counterexamples (e.g., in the referenced video, the coin collision can only be explained through force analysis, and the angles are difficult to describe clearly)
[5]

Content involving gore, violence, soldiers, or other sensitive material must not appear
[6]

Prefer videos that depict the cause and result of a specific event/action, rather than simple production or workflow processes
[7]

Description,

Videos must be understandable without relying on subtitles. Do not select videos that require subtitles for comprehension. III.Scene Categories 3.1 Physical Causality Collisions and Reactions / Gravity and Motion / Fluid Dynamics / Friction and Resistance / Mechanical Devices 3.2 Social Causality Emotional Reactions / Group Behavior / Conflict and Reconci...
[8]

The degree to which answering the question depends on video information

Show all 19 references

[9]

Whether the counterfactual QA contains irrelevant hallucinations Degree of Dependence on Video Information for Answering the Counterfactual QA (Score: 1–5)
[10]

1: Almost no dependence on video information, The question can be answered entirely based on common sense without watching the video
[11]

2: Slight dependence on video information, Basic knowledge of the objects or scene in the video is required, but the answer still mainly relies on common sense
[12]

3: Moderate dependence on video information, Understanding part of the behaviors, events, or relationships in the video is necessary
[13]

4: High dependence on video information, A relatively deep understanding of event progres- sion, character interactions, or action logic in the video is required
[14]

Did not actually happen

5: Complete dependence on video information, The question can only be answered by watching and fully understanding the video. Whether the Counterfactual QA Contains Irrelevant Hallucinations (Score: 1–5) Evaluate whether the counterfactual question and answer are constructed b...
[15]

There is little to no meaningful connection to the video

1: Severe Hallucination: The question or answer is largely disconnected from the video content, introducing objects, actions, or events that are completely absent from the video. There is little to no meaningful connection to the video
[16]

There are substantial unsupported inferences

2: Major Hallucination: Some content is related to the video, but the QA contains clearly fabricated or incorrect information. There are substantial unsupported inferences
[17]

There is slight over-interpretation or speculation

3: Minor Hallucination / Partially Ungrounded: The QA is generally related to the video, but includes some unsupported inferences or extrapolations. There is slight over-interpretation or speculation
[18]

4: Mostly Grounded: The question and answer are largely based on the video content, with only very minor imprecision or ambiguous inference
[19]

{res_node}

5: Fully Grounded: The counterfactual QA is constructed strictly based on actual video content. It only introduces reasonable modifications to existing events in the video and contains no unsupported fabrication. Figure 8: Video counterfactual QA quality evaluation guideline 1...

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.

[1] [1]

InProceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974

Intentqa: Context-aware video intent reason- ing. InProceedings of the IEEE/CVF international conference on computer vision, pages 11963–11974. Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, and Weidong Cai. 2025. Mul- timodal causal reasoning benchmark: Challenging multimodal large language models to discern causal links across mo...

work page arXiv 2025

[2] [2]

Junzhe Zhang, Huixuan Zhang, and Xiaojun Wan

Kbe-dme: Dynamic multimodal evaluation via knowledge enhanced benchmark evolution.Preprint, arXiv:2510.21182. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024. Dyval: Dy- namic evaluation of large language models for rea- soning tasks.Preprint, arXiv:2309.17167. A Human Annotation Details A.1 Human Annotation Guidel...

work page arXiv 2024

[3] [3]

If multiple identical objects exist and cannot be unambiguously referred to or distinguished linguistically, the video should not pass screening

Objects involved in the causal event must be clearly identifiable in the video and referable through language (e.g., wooden blocks, wooden boards, etc., should correspond accurately to visible objects in the video). If multiple identical objects exist and cannot be unambiguously referred to or distinguished linguistically, the video should not pass screening

[4] [4]

Click to view counterexamples (e.g., in the referenced video, the coin collision can only be explained through force analysis, and the angles are difficult to describe clearly)

The causal relationship should be intuitive, and the causal event should be easy to describe. Click to view counterexamples (e.g., in the referenced video, the coin collision can only be explained through force analysis, and the angles are difficult to describe clearly)

[5] [5]

Content involving gore, violence, soldiers, or other sensitive material must not appear

[6] [6]

Prefer videos that depict the cause and result of a specific event/action, rather than simple production or workflow processes

[7] [7]

Description,

Videos must be understandable without relying on subtitles. Do not select videos that require subtitles for comprehension. III.Scene Categories 3.1 Physical Causality Collisions and Reactions / Gravity and Motion / Fluid Dynamics / Friction and Resistance / Mechanical Devices 3.2 Social Causality Emotional Reactions / Group Behavior / Conflict and Reconci...

[8] [8]

The degree to which answering the question depends on video information