{"id":"e389a7cf-b6cf-412e-88bd-64521ea2b29d","arxiv_id":"2507.18932","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MMESGBench provides 933 validated multimodal QA pairs from 45 ESG documents and shows that multimodal and retrieval-augmented models outperform text-only baselines on ESG document reasoning.","lead":"This paper introduces MMESGBench, a benchmark of 933 question-answer pairs built from 45 real ESG documents to test how well AI models understand text, tables, charts, and layout in sustainability reports. Initial tests on 15 models show that multimodal models and retrieval-augmented pipelines beat text-only baselines, especially on questions that need visual or cross-page reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity rests on gold labels and model rankings that are not independent of the generating/verifying model family (Qwen); no quantified human agreement is reported, so top results may reflect self-consistency rather than task mastery.","rationale":"I read the paper in good faith: it releases a much-needed multimodal ESG QA collection with 933 pairs, diverse document types, and a reasonable evaluation protocol. The expert-in-the-loop step is a genuine effort, and the public GitHub release is positive. However, the paper's own pipeline creates an independence problem: the same model family (Qwen) generates the questions, verifies the answers, and then tops the benchmark. Section 3.3 only flags QA pairs where the verifier disagrees with the generated answer, so shared systematic errors are invisible to the automated QC, and expert review is limited to flagged items. Without inter-annotator agreement or an expert revision count, the reader's weakest assumption (gold-label correctness) remains unverified. My emphasis differs slightly: I see the more load-bearing issue as the combination of same-family generation, verification, and evaluation, which threatens not only label correctness but also the ranking's validity. The reader's rationale already mentions self-agreement, so this is a partial agreement. The appropriate action is to require independent expert re-validation and a controlled evaluation before treating the leaderboard as definitive; since the paper is currently CONDITIONAL, my recommendation is unchanged. I am not recommending rejection because the dataset itself is plausibly useful and the reported concerns are addressable with additional evidence rather than being fatal flaws in the resource.","tokens_in":11962,"tokens_out":7088,"duration_ms":67851,"concrete_test":"Take a stratified random sample of 150 QA pairs from the released dataset (covering text, table, chart, image, layout; single-page, cross-page, unanswerable). Have two ESG experts independently re-answer each question using only the cited evidence pages, blind to the gold answer and to the model that generated it, and adjudicate disagreements. Compute inter-annotator agreement and the rate at which the published gold answer is judged incorrect. Then recompute Table 1's overall ACC/F1 on the subset of items with confirmed labels and compare Qwen-VL-Max's margin over ColBERT+Qwen-Max.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that MMESGBench is a reliable evaluation resource and that the Table 1 ranking shows multimodal/RAG superiority depends on gold labels being correct and on evaluated models not being advantaged by the data-creation process. In Section 3.2, Qwen-VL-Max generates candidate QA pairs; in Section 3.3, an automated QC step re-infers answers with a multimodal LLM and flags only mismatches, after which experts review only flagged items. Because the generator and verifier are the same model family, systematic errors (e.g., consistent misreading of a chart or table) will appear in both the generated gold answer and the verification answer, so those items can pass without expert scrutiny. The paper reports no inter-annotator agreement, no count of expert revisions, and no error-rate estimate on non-flagged items. Section 4 then reports Qwen-VL-Max and ColPali+Qwen-VL-Max as the top performers (Table 1), so the headline 'multimodal and retrieval-augmented models substantially outperform text-only baselines' may partly reflect self-agreement with the generator's answer style rather than general ESG document understanding. The text-only baselines also use truncated OCR with different, generally weaker models, adding a further confound, but the label-independence gap is the more load-bearing issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MMESGBench, a multimodal question-answering benchmark for ESG documents, built from 45 ESG reports and standards and containing 933 QA pairs labeled as single-page, cross-page, or unanswerable. The construction pipeline uses Qwen-VL-Max to generate candidate QA pairs from rendered document pages, Qwen-Max to perform automated quality control and flag mismatches, and ESG experts to review flagged items. The authors evaluate 15 text-only, multimodal, and retrieval-augmented models, reporting that multimodal and RAG pipelines substantially outperform text-only baselines, with ColPali+Qwen-VL-Max reaching 51.8% accuracy. The dataset and code are released publicly.","tokens_in":12167,"tokens_out":3805,"duration_ms":39897,"significance":"If the gold labels and rankings are trustworthy, MMESGBench would be a useful first benchmark for multimodal, long-document ESG reasoning, with clear practical applications in compliance, disclosure validation, and RAG evaluation. Strengths include the diversity of document sources and types, the fine-grained evidence and modality annotations, the inclusion of unanswerable questions, and the public release of the dataset. However, the current evaluation loop is not independent of the data-generation process: the same model family that generates and verifies the QA pairs also achieves the top scores. The paper also lacks the inter-annotator agreement, filtering statistics, and significance testing needed to support the reliability claims. The benchmark's value depends on establishing label independence and fairer comparisons, so the central claims are not yet established.","major_comments":[{"comment":"The data-creation and evaluation loop is not independent. Qwen-VL-Max generates the candidate QA pairs, Qwen-Max performs the automated quality control, and only flagged items are reviewed by experts; then Qwen-VL-Max achieves the best non-RAG accuracy (40.0) and ColPali+Qwen-VL-Max the best overall (51.8). Because the verifier belongs to the same model family, systematic errors in reading tables or charts or in answer style will tend to agree with the generator and therefore escape flagging, so the gold labels and the leaderboard partly measure self-consistency with Qwen's answer distribution rather than general ESG understanding. The paper reports no inter-annotator agreement, no count of expert revisions or discards, and no error-rate estimate on non-flagged items. Please report these statistics, audit a random sample of non-flagged QA pairs with independent annotators, use multiple diverse verifier models, and report results for models outside the generating family separately.","section":"§3.3 and Table 1"},{"comment":"The construction of unanswerable questions is not adequately validated. The paper states that these items are generated 'while ensuring no corresponding evidence exists,' but this guarantee comes only from an LLM prompt; no document-level retrieval or expert verification is described to confirm that evidence is absent from the full document, which can exceed 2,000 pages. Since unanswerable items constitute 15.9% of the benchmark and Table 1 shows large differences across models on the UNA column, any mislabeled negatives would directly distort accuracy and F1. Please describe and implement a validation protocol for unanswerable items using exhaustive or retrieval-assisted evidence search with human confirmation, and report the outcome.","section":"§3.2, unanswerable QA"},{"comment":"The headline comparison between text-only and multimodal or RAG pipelines is confounded by differences in input construction. Text-only LLMs receive truncated OCR up to 120 pages with no retrieval, while multimodal models receive page images and RAG pipelines receive retrieved pages; this varies the context budget, access to retrieval, and the decoder simultaneously. Therefore the claims that 'multimodal models substantially outperform their text-only counterparts' and that 'ColPali+Qwen-VL-Max improves overall accuracy by 30%' are not cleanly attributable to multimodality or retrieval. Please evaluate matched settings, such as OCR with ColBERT retrieval for text decoders, the same decoder in text-only and multimodal modes, and equal context lengths, and report confidence intervals or significance tests for the accuracy differences.","section":"§4.1 and Table 1"},{"comment":"The paper gives no quantitative account of the quality-control pipeline: how many candidate QA pairs were generated, how many were removed by no-context inference, how many were flagged by the automated verifier, how many were discarded or revised by experts, and how many experts or annotators participated. Without these numbers, the claim that the dataset is 'high-fidelity' and 'evaluation-ready' cannot be assessed. Please report these pipeline statistics and, if possible, expert agreement coefficients on a sample of the final QA pairs.","section":"§3.3, quality-control pipeline"}],"minor_comments":[{"comment":"There is a typo in 'the fellowing three interrelated dimensions'; it should be 'following'.","section":"§1"},{"comment":"In the Mistral-Instruct-v0.1 row, the values appear as '10.684.522.2' instead of separate entries; please fix the formatting so all numbers are readable as distinct metrics.","section":"Table 1"},{"comment":"Figure 2(a) is referenced as showing dataset statistics, but the panel content is not displayed or described in the text; please ensure the panel and its contents are included and explained.","section":"Figure 2"},{"comment":"The paper mentions 'Qwen-VL-7B' in the evaluation but cites Qwen2-VL; please clarify the exact model version and configuration for reproducibility.","section":"§4.1"},{"comment":"The automatic short-form answer extraction and rule-based score computation are described only briefly; please provide details or pseudocode for how partial matches are scored and how abstentions are handled in the F1 metric.","section":"§4.1"},{"comment":"The phrase 'stakeholder-orientated' should be 'stakeholder-oriented'; similar minor language issues appear throughout and should be corrected.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is relevant to the conference's multimedia and document-understanding scope, and the public release is a positive aspect. The main risk is circularity between data generation and evaluation; I would ask the authors to address the independence issue with concrete statistics and re-evaluations before the paper is considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper delivers something the field actually needs. MMESGBench is the first multimodal QA benchmark for ESG documents, and the dataset is public—933 QA pairs over 45 long ESG reports spanning corporate reports, standards, and government material, with modality tags (text, table, chart, image, layout) and scope tags (single-page, cross-page, unanswerable). That is a concrete contribution, and the construction pipeline is sensible: generate with a vision-language model, use semantic clustering for cross-page questions, filter obvious easy/out-of-context items, then have experts review flagged cases. Releasing evidence localization is a plus.\n\nThe soft spot is load-bearing, though not fatal. Qwen-VL-Max generated the QA pairs, Qwen-Max did the automated QC, and Qwen-VL-Max tops the leaderboard. Only flagged examples go to experts. So if the Qwen family makes a systematic error—say, consistently misreads a certain chart type—that error can survive in both the gold answer and the verification step, and never hit a human. The paper reports no inter-annotator agreement, no count of expert revisions, and no error rate on unflagged items. Without those, part of the ranking may reflect self-consistency with the generator's answer style rather than general ESG document understanding.\n\nThe evaluation also gives text-only baselines a poor deal: truncated OCR with no retrieval, while multimodal models see page images, and the winning model is a large proprietary API rather than the open-weight models that most users would deploy. The broad claim that 'multimodal and RAG substantially outperform text-only' conflates modality, retrieval, and model scale. And there are no error bars or significance tests, so the Table 1 gaps should not be over-read.\n\nThese issues are addressable, and the authors seem to have built the dataset carefully. The fix is transparency: publish human-validation statistics, ideally from an independent random sample of non-flagged items; run controlled comparisons that match model size; and soften the overclaims. I would still send this to peer review—benchmark papers with public resources deserve referee time even when flawed—but I'd ask for those numbers before accepting. If I worked on long-document or ESG QA, I'd cite it.","headline":"MMESGBench is a genuinely useful new resource—the first multimodal QA benchmark for ESG documents—but the leaderboard should be read with caution because the same model family generated, verified, and topped it.","tokens_in":12744,"tokens_out":2889,"would_cite":true,"duration_ms":30748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMESGBench is a new 933-question benchmark for multimodal reasoning over ESG documents, and the strongest system tested answers only about 52 percent of it correctly.","keywords":["ESG","multimodal document understanding","benchmark dataset","document question answering","retrieval-augmented generation","layout-aware reasoning","long-document reasoning","vision-language models"],"falsifier":"Re-annotate a random sample of 100 of the 933 QA pairs with two independent ESG-specialist teams, giving them full access to the source documents, and measure agreement on answers, evidence pages, and unanswerable decisions; if agreement is low, or if re-annotation finds a meaningful share of answers or 'unanswerable' labels is wrong, the benchmark cannot support trustworthy model comparisons.","tokens_in":11713,"feed_emoji":"📊","tokens_out":7482,"duration_ms":69403,"temperature":0.7,"pith_summary":"ESG reports matter for sustainability, compliance, and investment, but they are long, multimodal, and structurally varied, and no benchmark was designed for the reasoning they demand. The paper introduces MMESGBench to fill that gap: 933 expert-validated question-answer pairs drawn from 45 real ESG documents covering corporate reports, standards and frameworks, and government and international-organization publications. Each question is labeled as single-page, cross-page, or unanswerable, with fine-grained evidence-type tags covering text, tables, charts, images, and layout. On this testbed, the paper finds that multimodal models and retrieval-augmented pipelines clearly outperform text-only baselines, yet the best configuration, retrieving five pages with ColPali and answering with Qwen-VL-Max, reaches only 51.8 percent accuracy, so the benchmark has substantial room for improvement.","feed_headline":"ESG AI exam: 933 questions, best score 52%","feed_subtitle":"Multimodal and retrieval-augmented models lead the pack, yet charts and cross-page reasoning remain unsolved.","key_machinery":"The load-bearing object is the MMESGBench dataset: 933 QA pairs, each attached to evidence pages, modality tags, and a difficulty score, built so that a model must jointly read text, tables, charts, images, and layout. The generation pipeline uses Qwen-VL-Max to produce candidate pairs from rendered page images, Qwen-Max to check answerability and consistency, and expert reviewers to revise or discard flagged items; cross-page pairs are generated by embedding pages with PaliGemma-3B, grouping them via FAISS-based clustering, and prompting for aggregation, comparison, or causal-linkage questions. Unanswerable items are included deliberately to test whether a model knows when the answer is not in the document.","core_discovery":"The central claim is that no existing benchmark captures the full multimodal and structural complexity of ESG documents, and that MMESGBench is the first dataset to do so, enabling reliable measurement of document-level reasoning. The paper substantiates this with a human-AI collaborative pipeline: a multimodal LLM generates candidate QA pairs from layout-aware page images, an LLM verifier filters weakly grounded or inconsistent items, and ESG experts calibrate the survivors; cross-page questions are produced by embedding pages, clustering them semantically, and prompting for multi-hop synthesis, while unanswerable questions probe hallucination. Initial experiments show that rendering pages as images and adding retrieval substantially improves accuracy over OCR-based text inputs, especially on layout and cross-page questions, and that chart-based reasoning remains the weakest modality.","pith_inferences":["A natural follow-up the paper does not run is to have two independent expert teams re-label a random subset of the 933 pairs and measure inter-annotator agreement; that would tell how much of the benchmark's validity rests on the single LLM-flag-to-expert review chain.","Because the construction pipeline is domain-agnostic, the same generate-verify-calibrate recipe could be transplanted to other long regulated documents, such as financial filings or clinical trial reports, where layout and tables also carry meaning.","The large gap between open-source multimodal models and the proprietary Qwen-VL-Max points to a concrete development target: improving open models' numeric and spatial reasoning on charts, which the paper identifies as the weakest area.","The unanswerable-question design could be reused as a cheap hallucination probe for arbitrary long-document QA systems, not just ESG ones."],"forward_implications":["Multimodal models and retrieval-augmented pipelines become the necessary baseline for ESG document QA, while text-only OCR baselines fall far behind and should be treated as the floor.","The benchmark's modality and location splits let researchers pinpoint failure modes: charts and layout-sensitive questions are the hard cases, while pure text and layout cues are comparatively easy.","Unanswerable questions turn hallucination resistance into a measurable quantity, and models that simply guess 'not answerable' inflate accuracy but drop in F1, so rankings must consider both metrics.","With the strongest system at about 52 percent accuracy, the benchmark offers a concrete yardstick for progress in long-context, layout-aware, multimodal document reasoning."],"supporting_citations":[{"why":"supplies the three-stage evaluation protocol of generation, extraction, and rule-based scoring used to compare all models","marker":"[25]"},{"why":"the multimodal LLM that generates candidate QA pairs and also serves as a strong baseline model","marker":"[40]"},{"why":"encodes document pages into embeddings for semantic clustering in cross-page QA generation","marker":"[4]"},{"why":"builds the vector index that makes page-group clustering for cross-page questions possible","marker":"[9]"},{"why":"text retriever used in the RAG baselines that beat non-retrieval models","marker":"[21]"},{"why":"multimodal retriever whose combination with Qwen-VL-Max gives the best overall result","marker":"[10]"},{"why":"existing text-only ESG QA benchmark that the paper contrasts with to motivate a multimodal resource","marker":"[16]"},{"why":"expert-designed financial-report QA benchmark that requires manual evaluation and lacks multimodal grounding, positioning MMESGBench as complementary","marker":"[19]"}],"fun_headline_variants":["First ESG multimodal benchmark: 933 QA pairs","ESG AI: multimodal beats text, charts still trip up","New benchmark tests AI on ESG reports' charts and pages","MMESGBench: 933 questions to stress-test ESG AI","AI struggles with ESG charts and cross-page reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gold answers, including the 'unanswerable' labels, are correct after expert calibration; the paper reports no inter-annotator agreement, no count of expert revisions, and relies on an LLM to flag errors, so if many answers are wrong or some unanswerable items do have evidence, the benchmark's model rankings could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["First ESG multimodal benchmark: 933 QA pairs","ESG AI: multimodal beats text, charts still trip up","New benchmark tests AI on ESG reports' charts and pages","MMESGBench: 933 questions to stress-test ESG AI","AI struggles with ESG charts and cross-page reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2615,"prompt_tokens":1004,"completion_tokens":1611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1531}},"tokens_in":620,"tokens_out":1611,"duration_ms":9955,"temperature":1.0,"reasoning_tokens":1531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:04:25.199175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of 100 of the 933 QA pairs with two independent ESG-specialist teams, giving them full access to the source documents, and measure agreement on answers, evidence pages, and unanswerable decisions; if agreement is low, or if re-annotation finds a meaningful share of answers or 'unanswerable' labels is wrong, the benchmark cannot support trustworthy model comparisons.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the three-stage evaluation protocol of generation, extraction, and rule-based scoring used to compare all models"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"multimodal retriever whose combination with Qwen-VL-Max gives the best overall result"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"existing text-only ESG QA benchmark that the paper contrasts with to motivate a multimodal resource"}],"review_version":1}