Pith. sign in

REVIEW 4 major objections 6 minor 17 references

MuSciClaims: Multimodal Scientific Claim Verification

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MuSciClaims, a 1,515-instance multimodal benchmark, shows current vision-language models are poor at scientific claim verification (most 0.3–0.5 F1, best 0.72) and biased toward judging claims supported.

desk verdict A useful new multimodal claim-verification benchmark whose headline numbers are credible, but the NEUTRAL class needs validation before the per-class scores can be fully trusted. read the letter →

arxiv 2506.04585 v2 pith:OLVJ3335 submitted 2025-06-05 cs.CL

Yash Kumar Lal , Manikanta Bandham , Mohammad Saqib Hasan , Apoorva Kashi , Mahnaz Koupaee , Niranjan Balasubramanian This is my paper
classification cs.CL
keywords scientificclaimverificationmultimodalbenchmarkvision-languagemodelsevidencelocalizationcross-modalaggregationepistemicsensitivityfigurecomprehension
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MuSciClaims is a new benchmark for multimodal scientific claim verification: given a claim, a multi-panel figure from a research article, and its caption, a system must say whether the claim is supported, contradicted, or unrelated. The paper constructs 1,515 balanced (claim, figure) instances—supported claims extracted from real papers, manually-perturbed contradictions, and neutral pairings with unrelated figures—plus four diagnostic tasks. Benchmarking twelve vision-language models, it finds that most are poor (macro F1 around 0.3–0.5), the best reach 0.72, and models share a bias toward 'supported'. Diagnostics attribute the failures to weak evidence localization, basic figure misreading, poor cross-modal aggregation, and low epistemic sensitivity. If the benchmark is sound, it provides both a reusable evaluation target and a concrete failure map for improving VLM reasoning over scientific figures.

What carries the argument

The load-bearing objects are the dataset construction pipeline and the four diagnostic tests. The pipeline automatically extracts sentences with explicit figure references from Results sections, keeps only clear, unambiguous supported claims via manual filtering, and applies manual perturbations in four designed categories—Qualitative Inference, Qualitative Relationship Inference, Quantitative Reasoning, and Epistemic Mismatch—each targeting a specific verification capability. NEUTRAL instances are made by pairing a claim with an unassociated figure from the same paper. The diagnostics—Basic Visual Understanding (per-claim one-word reading questions), Evidence Localization (gold panel annotations; panel prediction; and hint experiments where gold panels are supplied as text or cropped images), Cross-Modal Aggregation (compare full input vs figure-only vs caption-only), and Epistemic Sensitivity (whether predictions flip across original/perturbed pairs)—convert the benchmark from a scoreboard into a mechanism for localizing where verification fails.

What would settle it

Have a panel of domain experts independently rate whether each figure in a random sample of NEUTRAL pairs is actually related to the paired claim; if a non-trivial share are implicitly relevant, the NEUTRAL labels and the resulting F1 numbers are unreliable.

Watch

Extended reading notes

Core claim

The paper's central claim is that claim verification over information-rich figures is a distinct and currently unmet multimodal task, and that contemporary vision-language models are far from solving it. To show this, the authors build MuSciClaims: 505 supported claims extracted from the Results sections of peer-reviewed articles in physics, chemistry, and biology; 505 contradicted claims produced by manually perturbing the supported claims so that the same figure contradicts them (flipping qualitative terms, reversing comparisons, altering quantities, or breaking the link between a visually true observation and a domain-knowledge inference); and 505 neutral instances where each claim is paired with an unrelated figure from the same paper. Across 12 models, most score only 0.3–0.5 macro F1, with the best (o3 and o4-mini in a reasoning mode) at 0.72. The models' high recall and low precision on SUPPORT reveal a systematic 'supported' bias, and the diagnostic tasks show that models cannot reliably locate the relevant panel, answer simple reading questions about the figure, blend figure and caption information, or change their judgment when the observation versus the inference is perturbed. The result is a reproducible measurement that current models are not ready for multimodal scientific claim verification.

Load-bearing premise

The load-bearing premise is that a claim paired with a figure from the same paper that is not explicitly referenced truly has 'no connection' to that figure; this unverified assumption underwrites the entire NEUTRAL third of the benchmark, and any implicit relation would mislabel those instances and distort all downstream scores.

Editorial extensions

If this is right

  • Progress on multimodal claim verification can be measured directly against MuSciClaims, instead of being inferred from chart QA or text-only fact-checking numbers.
  • Because supplying the correct panel as a hint (textual or visual) improves claim-verification performance above the models' own localization, better evidence localization is a concrete lever for improving verification.
  • Model performance with only the figure or only the caption is close to performance with both, implying that current systems are not truly aggregating cross-modal information.
  • The low epistemic-sensitivity scores mean models will tend to accept claims where a visually true observation is paired with a wrong inference, a failure mode that matters for real scientific reading.
  • Domain-specific results (best on biology, worst on physics for several models) show the task is not generic visual reasoning and that domain-sensitive evidence selection is part of the challenge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the NEUTRAL pairing assumption holds, MuSciClaims could double as a visual-entailment probe: a model must reject a figure merely because it is irrelevant, not because it is visually implausible.
  • The diagnostic scaffolding (localization hints, modality ablations, epistemic flips) could be ported to other evidence-dense fields such as medical imaging or engineering diagrams, where the same failure classes would likely appear.
  • An obvious next stress test would be to vary caption wording while keeping the figure fixed; if model judgments swing with paraphrase, part of the 'verification' is riding on textual priors rather than the figure.
  • The study only uses English-language articles from three venues; extending to non-English papers and a wider variety of figure types (maps, schematics, spectra) would tell whether the measured weaknesses generalize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. MuSciClaims is a new benchmark for multimodal scientific claim verification over information-rich figures from physics, chemistry, and biology papers. The dataset contains 1515 (claim, figure) instances balanced across SUPPORT, NEUTRAL, and CONTRADICT, together with per-instance diagnostic annotations for evidence localization, basic visual understanding, cross-modal aggregation, and epistemic sensitivity. The authors automatically extract figure-referencing sentences from Results sections, manually filter them to choose supported claims, manually perturb them to create contradicted claims, and create NEUTRAL instances by pairing claims with unassociated figures from the same paper. They evaluate 12 VLMs in zero-shot decision and reasoning-then-decision settings, reporting per-class and overall precision/recall/F1. Most models score in the ~0.3-0.5 F1 range in the decision-only setting, while o3 and o4-mini reach 0.72 F1 in the reasoning setting; models show high SUPPORT recall and low SUPPORT precision. Diagnostic experiments on the biology subset find that models localize relevant panels poorly, that providing localization hints improves verification, and that cross-modal and basic reading abilities are weak.

Significance. If the label construction is sound, MuSciClaims fills a genuine gap: it is the first readily usable benchmark that directly tests claim verification over heterogeneous, real scientific figures with balanced labels and diagnostic subtasks. The resource brings together three disciplines, uses real articles, includes exact prompts and detailed per-domain results (Tables 6-9), and the CONTRADICT perturbation design is thoughtful, with a second annotation round showing 100% agreement that perturbations contradict the original claims. The evaluation is not circular: labels are human-constructed, and the self-citation (Verma et al., 2023) is used only as background for epistemic sensitivity. However, the validity of the headline results depends on label quality, particularly the unverified NEUTRAL construction, and on accurate reporting of the performance distribution; both need attention before the benchmark can be used as a reliable testbed.

major comments (4)
  1. [Section 3.4] The NEUTRAL class is constructed by pairing each claim with an 'unassociated figure from the same paper,' where 'unassociated' is operationalized only as the absence of an explicit figure reference in the extracted claim sentence. This is not sufficient: in the source journals, a single result is often presented across multiple figures or shared panels, so a claim explicitly tied to Figure 2A may also be supported, partially supported, or contradicted by Figure 3 from the same paper. No manual verification, expert annotation, or inter-annotator agreement is reported for these 505 instances, which are one third of the benchmark. Because the per-class precision/recall/F1 numbers in Table 1 and the 'bias towards SUPPORT' finding both depend on the correctness of these labels, the paper should either validate the NEUTRAL pairs (e.g., by expert screening of all same-paper pairs, or by constructing NEUTRAL from different papers) or report the expected label-noise rate and its effect on the reported metrics.
  2. [Sections 3.1 and 3.2] The SUPPORT labels rest on two unquantified steps: the claim-extraction filter and the manual filtering 'to only retain claims that are clearly and unambiguously supported by the figures.' The paper does not report the number of extracted sentences, the number discarded by the manual filter, the number of annotators, or any agreement measure for this filter. Since CONTRADICT instances are generated by perturbing the retained SUPPORT claims, any mislabeled SUPPORT instance propagates directly into CONTRADICT and contaminates the class balance assumptions. Please document the annotation protocol for the SUPPORT filter and provide a validation sample with expert judgments for both SUPPORT and CONTRADICT labels.
  3. [Section 5 and Abstract] The abstract's claim that 'most vision-language models are poor (~0.3-0.5 F1), with even the best model only achieving 0.72 F1' is not supported by Table 1. In the D setting, the overall macro-F1 scores are 0.43 (4o-mini), 0.51 (4o), 0.64 (Sonnet), 0.43 (Phi-4), 0.33 (Llava-Next), 0.45 (Llama-3.2), 0.43 (Molmo), 0.70 (InternVL3), 0.63 (Qwen2.5), and 0.58 (DeepSeek). Thus several models score above 0.5 in F1, and the stated ~0.3-0.5 range omits them. The same discrepancy appears in Section 5's first observation. The abstract and Section 5 should either report the actual distribution or use a qualifier such as 'many' or 'the majority of open-source models.'
  4. [Section 5 / Table 1] There is no human performance baseline anywhere in the paper. Without expert human performance on the claim-verification task (especially on the NEUTRAL class) or on the diagnostic tasks, the reader cannot calibrate whether an F1 of 0.72 indicates that the benchmark is too easy, whether the labels are too noisy, or whether the models are actually poor. Reporting a human ceiling on a random sample of each class, together with human agreement on the SUPPORT and NEUTRAL labels, would address both calibration and the NEUTRAL validation concern directly.
minor comments (6)
  1. [Section 6] The diagnostic experiments are performed only on the BIOLOGY subset (Section 6) and only on four models (4o-mini, 4o, Sonnet, InternVL3); the general statements in the conclusion ('models are bad at localizing', etc.) should be scoped accordingly, and the absence of o3/o4-mini from diagnostics should be noted in the corresponding tables.
  2. [Table 3] The EVIDENCE LOCALIZATION evaluation does not define how predicted panel names are matched to gold panel names; please state whether exact-set matching, subset matching, or token-level F1 is used.
  3. [Figure 2] The hint experiments are run on a random sample of n=101 class-balanced data points; please report variance or confidence intervals, and clarify whether the 'Weighted F1' axis is weighted by class frequency or by panel frequency.
  4. [Appendix G] The prompts in Figures 8-13 use slightly different output formats for Sonnet and InternVL3; this is probably intentional, but please note in the text whether the same prompt template was used for all models in a given setting to rule out format-driven differences.
  5. [Appendices A and E.3] There are several typos in the appendices: 'cpation' (Appendix E.3), 'seperately' and 'resoultion' (Appendix A), and 'modalities' (Appendix A). Please proofread.
  6. [Appendix B] The annotation appendix states that annotators 'were not paid for annotations' and are graduate students; please clarify whether these annotators are authors/volunteers and whether formal consent or IRB review was obtained, since the data is being publicly released.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MuSciClaims is an externally grounded benchmark with human/rule-constructed labels; the only self-citation is background and is not load-bearing.

full rationale

The paper's central outputs—VLM F1 scores, the support-bias observation, and the diagnostic failings—are empirical measurements over a benchmark whose labels are produced by extraction, manual filtering, perturbation plus a 100%-agreement verification round, and same-paper figure pairing (Sections 3.1, 3.2, 3.4). No model output is used to define any label, and no parameter is fitted and then 'predicted,' so the headline results are not forced by construction. The sole self-citation, Verma et al. (2023) co-authored by Lal, appears in Section 6 only as background for the notion of epistemic relationships; it supplies no uniqueness theorem, ansatz, or fitted value, so it does not carry the argument. The NEUTRAL construction (Section 3.4) rests on an unverified assumption that a same-paper figure not explicitly referenced has 'no connection' to the claim; this is a real label-validity threat that could bias the reported per-class scores, but it is not circularity because the label is not defined in terms of the model predictions being evaluated. Under the required evidence standard of quoting a specific reduction to the paper's own inputs, no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark labels depend on three domain assumptions about the relationship between published text, figures, and perturbed claims. These assumptions are stated but not independently verified.

assumptions (3)
  • domain assumption Published articles' claims are supported by their associated figures.
    Section 3, footnote 2: 'We rely on the scientific integrity of the published articles and assume that evidence support the asserted claims.'
  • domain assumption A perturbation that contradicts the supported claim yields a claim contradicted by the same figure.
    Section 3.2: 'Our perturbations ensure that the modified claim is a contradiction of the supported claim. This means that the figure which supports the original claim will, by extension, not support the modified claim.' This assumes the original claim is fully supported by the figure.
  • domain assumption A claim paired with an unassociated figure from the same paper is neutral with respect to that figure.
    Section 3.4: 'we pair each claim with an unassociated figure from the same paper to obtain data where there is no connection between them.' No verification that the claim is indeed unrelated to the figure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MuSciClaims: Multimodal Scientific Claim Verification." pith.science (2026). https://pith.science/paper/OLVJ3335

@misc{pith2026250604585,
  author       = {Pith},
  title        = {Pith review of: MuSciClaims: Multimodal Scientific Claim Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OLVJ3335}},
  note         = {Machine review of arXiv:2506.04585}
}
read the original abstract

Assessing scientific claims requires identifying, extracting, and reasoning with multimodal data expressed in information-rich figures in scientific literature. Despite the large body of work in scientific QA, figure captioning, and other multimodal reasoning tasks over chart-based data, there are no readily usable multimodal benchmarks that directly test claim verification abilities. To remedy this gap, we introduce a new benchmark MuSciClaims accompanied by diagnostics tasks. We automatically extract supported claims from scientific articles, which we manually perturb to produce contradicted claims. The perturbations are designed to test for a specific set of claim verification capabilities. We also introduce a suite of diagnostic tasks that help understand model failures. Our results show most vision-language models are poor (~0.3-0.5 F1), with even the best model only achieving 0.72 F1. They are also biased towards judging claims as supported, likely misunderstanding nuanced perturbations within the claims. Our diagnostics show models are bad at localizing correct evidence within figures, struggle with aggregating information across modalities, and often fail to understand basic components of the figure.

Figures

Figures reproduced from arXiv: 2506.04585 by the authors.

Figure 1
Figure 1. Each data point from MUSCICLAIMS contains a claim, its associated figure and caption and annota￾tions about its relevant panels. Each claim, both original and perturbed, is also labeled with its relationship to the figure (SUPPORT, NEUTRAL, CONTRADICT). It also enables performing diagnostic tests. Models must identify the relevant panel as part of EVIDENCELOCALIZATION and answer a question about the figure for BASIC… view at source ↗
Figure 2
Figure 2. Model performance on CLAIMVERIFICATION when prompted for (or provided) localization. Providing models with hints about the relevant panels of figures im￾proves their claim verification. Textual hints (TH) guide models to focus on the correct part of the full figure, showing higher performance than R→D and I→R→D. Models using visual hints (VH; relevant panel as visual input instead of full figure) perform even better… view at source ↗
Figure 3
Figure 3. Model performance on BASICVISUALUN￾DERSTANDING. Models fail to answer basic questions about components of figures associated with claims. Models need to improve on basic visual reading. Each claim in MUSCICLAIMS is also accompa￾nied by a question about the figure that is relevant for the verification process. These questions test ba￾sic visual reading abilities (e.g., “How many days does the data span?" in [PITH_FU… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Instructions and UI of the application used to collect perturbations of claims from manual annotators. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Instructions and UI of the application used to collect perturbations of claims from manual annotators [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Instructions and UI designed to collect a second round of manual annotation to verify that the perturbed [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Each claim is accompanied by a diagnostic [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Prompt for Sonnet for the D experiment You are an AI model tasked with verifying claims related to visual evidence using zero-shot learning. Your job is to analyze a given image(s) and its provided caption(s) to decide whether it SUPPORT or CONTRADICT or NEU￾TRAL the p…
Figure 10
Figure 10. Figure 10: Prompt for Sonnet for the I→R→D experi￾ment This is an image from a scientific paper. The following is the caption of the image. IMAGE CAPTION(S): IMAGE_CAPTIONS Using this image, analyze whether the following claim is supported, contradicted or neutral according to t…
Figure 9
Figure 9. Figure 9: Prompt for Sonnet for the R→D experiment You are an AI model tasked with verifying claims related to visual evidence using zero-shot learning. Your job is to analyze a given image(s) and its provided caption(s) to decide whether it SUPPORT or CONTRADICT or NEUTRAL the …
Figure 12
Figure 12. Figure 12: Prompt for InternVL3 for the R→D experi￾ment This is an image, with multiple panels, from a scientific paper. The following is the caption of this image. IMAGE CAPTION(S): IMAGE_CAPTIONS Using this image, analyze whether the following claim is supported, contradicted …
Figure 13
Figure 13. Figure 13: Prompt for InternVL3 for the I→R→D experiment 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 14 canonical work pages

  1. [1]

    Domain Expertise (27%) - Models lack domain expertise, knowledge of related work and com- mon practices of representing data specific to the scientific field

    work page

  2. [2]

    Visual Understanding (23%) - Models are un- able to make the correct inference from the information that they perceive from the figure

    work page

  3. [3]

    In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 244–257, Marseille, France

    CoVERT: A corpus of fact-checked biomed- ical COVID-19 tweets. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 244–257, Marseille, France. European Language Resources Association. Shraman Pramanick, Rama Chellappa, and Subhashini Venugopalan. 2024. Spiqa: A dataset for multi- modal question answering on scientific ...

    arXiv 2024

  4. [4]

    Cross Modal Aggregation (17%) - Models in- correctly weight information from one modality over another (such as focus on the caption more than the figure), coming to the wrong conclu- sion

    work page

  5. [5]

    Using the methodology described in sec- tion 3, we create 45 claims from Cell papers 7

    on BIOLOGY claims for the decision-only (D) experiments using a subset of closed-source models. Using the methodology described in sec- tion 3, we create 45 claims from Cell papers 7. The 7These papers are not part of the MUSCICLAIMS evalua- tion set. 14 Figure 5: Instructions and UI of the application used to collect perturbations of claims from manual a...

    work page

  6. [8]

    Visual Perception (23%) - Models either miss or pick up data presented in the figure, which can lead to the correct judgment

    work page

  7. [10]

    For in- stance, models simply misunderstand the cap- tion (textual understanding), or fail to aggregate information from multiple panels (multi-panel aggregation)

    Others (10%) - This category contains infre- quent error types bucketed together. For in- stance, models simply misunderstand the cap- tion (textual understanding), or fail to aggregate information from multiple panels (multi-panel aggregation). G Prompts Used We present the exact prompts used for different experiments with Sonnet in Figure 8, Figure 9 an...

    work page

  8. [13]

    decision

    Analyze the broader context and scope of the image(s) and caption(s) in relation to the claim. After completing your analysis, output exactly one JSON object with exactly one key: “decision”. - For “decision”, output exactly one word — either “SUP- PORT” or “CONTRADICT” or “NEUTRAL” (uppercase, no extra text). Do NOT add markdown formatting, code fences, ...

    work page

Show all 17 references
  1. [14]

    reasoning

    Evaluate the claim’s plausibility based on visual ele- ments within the image(s). 2. Consider the relevance, meaning, and implications of both the depicted content and the caption(s). 3. Analyze the broader context and scope of the image(s) and caption(s) in relation to the cl...

  2. [15]

    Evaluate the claim’s plausibility based on visual elements within the image(s)

  3. [16]

    Consider the relevance, meaning, and implications of both the depicted content and the caption(s)

  4. [17]

    Analyze the broader context and scope of the image(s) and caption(s) in relation to the claim

  5. [18]

    Identify which specific panels (e.g., Panel A, Panel B, Panel C, etc.) are necessary to evaluate the claim

  6. [19]

    figure_panels

    Think step by step to reach your conclusion and provide it in a concise manner in the output. After completing your analysis, output exactly one JSON object with exactly three keys in this order: “figure_panels”, “reasoning”, and “decision”. - For “figure_panels”, list ONLY th...

  7. [2022]

    In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 1–16, Seattle, United States

    PubHealthTab: A public health table-based dataset for evidence-based fact checking. In Find- ings of the Association for Computational Linguistics: NAACL 2022, pages 1–16, Seattle, United States. As- sociation for Computational Linguistics. 9 Mubashara Akhtar, Oana Cocarascu, ...

    2022

  8. [2023]

    Results” and “Discussion

    Vistext: A benchmark for semantically rich chart captioning. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (V olume 1: Long Papers), pages 7268–7298. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 20...

    2018 arXiv

  9. [2024]

    In Findings of the Associa- tion for Computational Linguistics: ACL 2024 , pages 13921–13937, Bangkok, Thailand

    ChartCheck: Explainable fact-checking over real-world chart images. In Findings of the Associa- tion for Computational Linguistics: ACL 2024 , pages 13921–13937, Bangkok, Thailand. Association for Computational Linguistics. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.