{"id":"e0854767-e5a4-4bd4-a629-751106d5d8c8","arxiv_id":"2510.24870","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MiRAGE scores multimodal RAG outputs at the subclaim level, and human-applied MiRAGE matches human quality judgments better than text-only metrics do.","lead":"This paper introduces MiRAGE, a claim-by-claim scoring system for judging AI-written articles that draw on videos, audio, and images, checking both factual accuracy and whether citations support each statement. It compares these scores with human quality ratings and with text-only evaluation metrics, and finds the human-applied version aligns more closely with quality judgments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human validation supports only reference-based INFOF1, not MIRAGE as a whole; GJ–EQJ agreement is near zero and the sample is only 3 systems.","rationale":"The reader's weakest assumption was the decomposition of all modality information into unimodal linguistic subclaims. That is a fundamental premise of the framework, but it is not the most direct threat to the paper's headline claim. Even if decomposition is incomplete, a metric can still correlate with human quality judgments. The more immediate problem is that the paper's own data contradict the scope of the claim: the human validation shows alignment only for the INFOF1-reference component, while the human grounding judgments for the citation component (GJ) show near-zero or negative correlation with extrinsic quality judgments. Additionally, the empirical basis is thin--only three systems--so the reported taus may reflect system-level ranking rather than robust topic-level agreement. These concerns are concrete, grounded in the paper's tables, and directly target the abstract's central assertion. They do not change the reader's CONDITIONAL verdict, but they sharpen the condition: the claim must be narrowed, and the citation component needs separate validation. The reader's rationale did mention the small scale of human validation, so there is partial agreement, but the weakest-assumption framing (decomposition) is not the same as the evidence-claim mismatch identified here.","tokens_in":25566,"tokens_out":9363,"duration_ms":81700,"concrete_test":"Recompute the Kendall's tau in Table 2 at the topic level after stratifying by system (i.e., compute per-system topic-level taus and then average, or fit a mixed-effects model with system as a random effect). If the within-system tau is not significantly positive across the 10 topics, the apparent alignment is driven by the three system-level rankings, and the claim should be revised to refer only to system ranking. Separately, report the GJ-EQJ agreement per system; if it remains near zero for all three systems, the abstract should not attribute alignment to CITEF1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim--'MIRAGE, when applied by humans, strongly aligns with extrinsic judgments of output quality'--is not supported for the framework as a whole. Table 2 shows strong-to-moderate agreement only between human INFOF1 judgments (ICJ) and EQJs (tau between 0.49 and 0.81). The human grounding judgments (GJ), which are the human-applied version of CITEF1, agree near zero or negatively with EQJs (18.2, -5.5, 23.0). Since MIRAGE comprises both INFOF1 and CITEF1, the headline overstates the evidence. Additionally, the agreement is computed over only 3 systems (LLM-only, CAG-2 Oracle, CAG-2 RAG) across 10 topics; with three systems, a rank correlation can be dominated by coarse system-level differences rather than topic-level quality discrimination. The paper's own Section 5 concedes 'the agreement across the board for these automatic metrics is low,' and Appendix H shows the automatic VLM verifier is near chance (ZS F1=41.7). Thus the evidence supports a narrower claim: reference-based, human-annotated INFOF1 correlates with reference-based quality judgments; it does not validate CITEF1 or the full multimodal framework.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MiRAGE, a claim-level evaluation framework for multimodal retrieval-augmented generation (RAG). MiRAGE decomposes generated and reference text into subclaims and computes INFOF1 (factuality and information coverage) and CITEF1 (citation support and attribution), with reference-based and reference-free (collection-based) variants. The authors collect three kinds of human judgments on 10 WikiVideo topics from three generation systems (LLM-only, CAG-2 Oracle, CAG-2 RAG), compare MiRAGE and three adapted text-based RAG metrics (ALCE, ARGUE, RAGAS) against these judgments using Kendall's tau, and report experiments on automatic LLM/VLM claim verification. They conclude that human-applied INFOF1 aligns strongly with extrinsic quality judgments, that text-centric metrics fail to capture grounding, and that current automatic verifiers are poorly calibrated. The paper releases open-source implementations and provides detailed appendices with metric formulations, annotation instructions, and additional correlation tables.","tokens_in":25810,"tokens_out":6058,"duration_ms":53761,"significance":"If the central claim is narrowed appropriately, this is a useful contribution. The claim-level decomposition is well motivated, the adaptation of existing text-based RAG metrics to multimodal sources is clearly described, and the release of open-source code plus detailed annotation protocols supports reproducibility. The paper is also honest in reporting that automatic verifiers are not yet calibrated (Appendix H, Table 10), and the circularity concern about metric validation does not materialize: the automatic verifier is trained on 47 held-out topics, and the human metric judgments are collected by independent annotation protocols rather than fitted to the extrinsic judgments. However, the headline claim that 'MIRAGE, when applied by humans, strongly aligns with extrinsic judgments of output quality' is only directly supported for the INFOF1 component, not for CITEF1, and the correlation evidence rests on a very small sample without significance testing. The framework itself is a reasonable scaffold for future work, but the validation section needs to be re-scoped and the recommendations reconciled with the reported automatic-metric performance.","major_comments":[{"comment":"The abstract states that 'MIRAGE, when applied by humans, strongly aligns with extrinsic judgments of output quality,' but the evidence in Table 2 supports only the INFOF1 component. The intrinsic claim judgments (ICJ), which mirror INFOF1-Ref, show Kendall's tau between 49.0 and 81.2 against the extrinsic quality judgments (EQJ), while the grounding judgments (GJ), which mirror CITEP, show tau of 18.2, -5.5, and 23.0, i.e., near zero or negative. Since CITEF1 is part of MiRAGE, the headline claim should be narrowed to INFOF1, or the abstract and Section 5 need to report separately the components that are validated and the components that are not.","section":"Abstract and §5, Table 2"},{"comment":"The human-validation analysis is computed over only 10 topics and 3 systems, and no confidence intervals or significance tests are reported for any Kendall's tau value. With 3 systems, the rank correlation is heavily quantized and can be dominated by coarse system-level differences rather than by the ability of a metric to discriminate quality at the topic level. Since the central claim of the paper is the strong alignment between human-applied MiRAGE and extrinsic quality judgments, the absence of any uncertainty quantification is load-bearing. I would like to see at least a bootstrap confidence interval, a per-topic breakdown, or a paired significance test.","section":"§5 Evaluation Setup and Tables 1-5"},{"comment":"Section 5 states that 'the agreement across the board for these automatic metrics is low, outside of ROUGE and BERTScore,' and Appendix H (Table 10) shows that a zero-shot VLM verifier achieves F1=41.7, below chance on a balanced test set. Nevertheless, Section 6 recommends that evaluators report automatic INFOF1 and CITEF1 (both reference and collection variants) as part of both the quick and comprehensive evaluation suites. This recommendation is inconsistent with the paper's own evidence that the automatic implementations are not calibrated for support judgments. The recommendations should be re-scoped to human-annotated or explicitly calibrated variants, or the paper should provide a concrete decision rule for when automatic MiRAGE scores are reliable enough to use.","section":"§5, Table 1, and §6"},{"comment":"The text in Section 5 claims that 'claim-based precision metrics, RAGAS Faith and INFOP, not only align with EQJs and ICJs, but also with GJs' and that 'CITEF1 has the highest agreement for both EQJs and GJs.' The values in Table 1 show INFOP agreement with GJ of 6.6, 21.5, and 1.8, and RAGAS Faith agreement with GJ of 24.8; these are weak correlations, not clear alignment. In Table 3, the highest EQJ value appears under the ALCE Citation Quality column (54.3), not under CITEF1. The claims need to be re-derived from the actual table values, and the table headers/labels need to be clarified so that the comparison is interpretable.","section":"§5, Tables 1 and 3"}],"minor_comments":[{"comment":"There are numerous typos and misspellings, including 'mulitmodal' in the abstract, 'effectievly' and 'groudns' in §4.2, 'asseses' in the Figure 1 caption, 'alterantive' in §4.2.1, 'wills tart' in Appendix B.2, and 'agrement' in §5. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The weighted precision and recall formulations are written as 1/|C| * sum(s * I_i). If I_i is intended as an importance weight, the denominator should be sum(I_i) rather than the number of claims; otherwise the weighted score depends on the arbitrary scale of I and is not a proper weighted proportion. Please clarify the intended normalization.","section":"Appendix B, Eqs. (8) and (11)"},{"comment":"Table 3 has four data rows but the caption says 'Row 1: EQJ, Row 2: GJ,' and the column header 'R C R CQ SS CR' is ambiguous. Please give a clear two-row header that identifies each metric variant and enumerates the annotator rows explicitly.","section":"Table 3"},{"comment":"For RAGAS Context Relevance, the paper says the denominator is simulated by eliciting a detailed VLM summary, but the connection to the original formula 'extracted sentences / total sentences' is not made explicit. Please provide the exact formula used for the multimodal adaptation.","section":"Appendix F"},{"comment":"The balanced test set used to report verifier F1 is described, but it is not stated whether the reported F1 is macro-averaged or micro-averaged, or how the 787-claim evaluation set was balanced relative to the original label distribution. Please specify this for reproducibility.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid framework proposal with a genuinely useful decomposition and a considerable amount of careful annotation work, but the abstract and Section 5 currently claim more than the data support. The narrow claim regarding INFOF1 is worth publishing; the broader claim about MiRAGE, and the Section 6 recommendations for using automatic MiRAGE, need to be reconciled with the reported correlations and the poor VLM verifier performance. I would suggest a thorough revision rather than rejection, because the core framework and the empirical study are substantial enough to be valuable once the claims are scoped accurately."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it moves RAG evaluation to the claim level for multimodal sources, defines INFOF1 and CITEF1 with reference-based and collection-based variants, and ports ALCE, ARGUE, and RAGAS to video. The negative result that ROUGE and BERTScore capture information but not grounding is worth having, and the authors ship code and are transparent about the verifier's weakness.\n\nThe soft spots are real and two are serious. First, the headline claim that human-applied MIRAGE strongly aligns with extrinsic quality judgments is only supported for INFOF1. The grounding judgments (the human version of CITEF1) show near-zero or negative agreement with EQJ in Table 2 (18.2, -5.5, 23.0). The abstract says 'MIRAGE' as a whole, which overstates the evidence.\n\nSecond, and more worrying, the reported Kendall's tau values look impossible for the setup described. With only 3 systems being ranked, standard Kendall's tau can only produce a small set of discrete values (e.g., -1, -1/3, 1/3, 1 without ties; slightly more with ties). Values like 0.49, 0.61, 0.81 cannot come from ranking 3 items with the standard estimator. Either the paper is using a different statistic or there is a computation error. The authors need to explain exactly what was computed, ideally with per-topic agreement or a different formulation. This undercuts the central quantitative result, not just a minor detail.\n\nThe automatic metrics also show low agreement with human judgments, and Appendix H shows the zero-shot VLM verifier is near chance. Section 6 still recommends using these automatic tools for comprehensive evaluation. The paper admits the limitation, but the recommendation is more optimistic than the numbers justify.\n\nThe human validation is small: 10 topics, 3 systems, 3 annotators for EQJ/ICJ. That is acceptable for a first look, but it cannot carry strong claims.\n\nWho is this for? Researchers working on multimodal RAG evaluation or claim-level factuality metrics. The framework and code are a useful starting point, and the negative results about text-centric metrics are informative. But the paper should not be accepted as is. It needs a statistical fix and a more carefully scoped claim.\n\nI would send it to peer review with the expectation of major revision. The core idea is worth engaging with, and the mistakes look fixable.","headline":"Claim-level multimodal RAG evaluation framework worth engaging with, but the human-alignment claim rests on a statistically impossible Kendall's tau and an overbroad abstract.","tokens_in":26373,"tokens_out":5226,"would_cite":true,"duration_ms":45367,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MiRAGE evaluates multimodal RAG by scoring every claim a system makes against sources.","keywords":["multimodal retrieval augmented generation","claim decomposition","factuality evaluation","citation support","video-based RAG","INFOF1","CITEF1","human evaluation"],"falsifier":"Take a video where the decisive fact is a continuous, non-linguistic property—for example, that a temperature trend rose steadily over 24 hours with no single numeric claim being independently supported—generate an output that states that fact, and ask humans for extrinsic quality scores while MiRAGE scores the subclaims. If humans rate the output high but MiRAGE rates it low because the fact cannot be decomposed into discrete supported subclaims, the paper's decomposition premise is falsified.","tokens_in":25370,"feed_emoji":"🎯","tokens_out":5068,"duration_ms":41222,"temperature":0.7,"pith_summary":"This paper introduces MiRAGE, an evaluation framework for retrieval-augmented generation from multimodal sources such as video, audio, and images. Its central claim is that all information in any modality can be decomposed into textual subclaims, and that scoring those subclaims for factual support (INFOF1) and citation support (CITEF1) captures output quality better than text-centric metrics. The paper reports that when humans apply MiRAGE, its scores align strongly with extrinsic judgments of quality on the WikiVideo article-generation task, while three leading text-based RAG metrics show poor or negative agreement. If this holds, MiRAGE gives the field a way to judge whether a system's answer is factually grounded and properly cited, rather than merely lexically similar to a reference.","feed_headline":"Claim-level scoring outdoes text metrics for multimodal RAG","feed_subtitle":"Human-scored INFOF1 tracks quality on video-based RAG; text-only metrics often disagree.","key_machinery":"The machinery is subclaim decomposition plus a binary support scoring function $s(p,h)$. Sentences in both the prediction and the reference are broken into declarative subclaims, and each subclaim is scored as supported or not by the cited or collection evidence for INFOF1, or by a reference or citation proxy for CITEF1. The metrics are precision-recall pairs—INFOF1-P and INFOF1-R for factuality and coverage, CITEF1-P and CITEF1-R for citation support and attribution—with F1 as the summary. Claim-level granularity is what lets the metric verify each atomic proposition against a video instead of requiring an entire multi-claim sentence to be supported at once.","core_discovery":"The central discovery is that claim-level, source-grounded evaluation transfers to multimodal RAG: decomposing generated and reference sentences into subclaims, then measuring support precision and recall against either the evidence collection or a reference, produces scores that track human judgments of quality, whereas sentence-level TextRAG metrics fail. On WikiVideo, human-annotated MiRAGE agrees with extrinsic quality judgments at Kendall's tau between roughly 0.49 and 0.81, while ROUGE and BERTScore capture what information is present but not whether it is grounded in cited sources. The paper also shows that automatic verification is the bottleneck: zero-shot vision-language models are not calibrated for subclaim-versus-video support judgments, with near-chance performance in one 7-billion-parameter configuration, and supervised next-token training improves calibration.","pith_inferences":["A testable extension: if the decomposition premise is right, the same subclaim-scoring pipeline should rank systems on image-only and audio-only RAG without retuning; a cross-modal consistency check would compare MiRAGE ranks to extrinsic quality judgments on each modality.","The framework implicitly defines a research program: train a single vision-language or language verifier on subclaim grounding across modalities. The paper's appendix suggests supervised next-token training is a promising baseline, and confidence-calibrated scalar judgments would sharpen the ranking.","Because CITEF1-recall takes the max over cited videos, a system could use it as a training signal to suppress citations that do not actually contain the claimed information, making citation generation optimizable end-to-end.","The reference-precision variant of INFOF1 conflates factuality with salience, because a true but non-salient claim gets penalized; evaluators who want pure factuality should prefer collection precision, a distinction the paper leaves as a practical choice for the user."],"forward_implications":["On WikiVideo, human-annotated MiRAGE correlates with extrinsic quality judgments at Kendall tau 0.49–0.81, well above ROUGE and BERTScore, so it can rank systems by usefulness rather than surface similarity.","ALCE, ARGUE, and RAGAS, when adapted to video, show low or negative agreement with human judgments because they verify at sentence level, assume concatenable sources, or assume exhaustive extraction from video.","ROUGE and BERTScore track what information is present, but not whether it is grounded in cited sources; a system can score high on them while citing nothing.","Zero-shot vision-language claim verification is near chance for subclaim-video pairs, so automatic MiRAGE is currently unreliable; supervised next-token training raises the reported F1 substantially, suggesting calibration is achievable.","For practical evaluation the paper recommends a quick reference-based suite (ROUGE, BERTScore, INFOF1-ref, CITEF1-ref) and a comprehensive collection-based suite that adds RAGAS-Faithfulness for full grounding checks."],"supporting_citations":[{"why":"Supplies the WikiVideo task, the video-based article generation setting, and the subclaim grounding annotations used for evaluation.","marker":"Martin et al., 2025"},{"why":"Provides the claim-decomposition idea (FActScore) that MiRAGE generalizes to multimodal evidence.","marker":"Min et al., 2023"},{"why":"Defines the ALCE baseline whose Claim Recall and Citation Quality are adapted to video and compared against MiRAGE.","marker":"Gao et al., 2023"},{"why":"Defines the ARGUE baseline with nugget coverage and sentence support that MiRAGE contrasts with its subclaim-level scoring.","marker":"Mayfield et al., 2024"},{"why":"Defines the RAGAS baseline with faithfulness, answer relevance, and context relevance, adapted here to multimodal input.","marker":"Es et al., 2024"},{"why":"Supplies Kendall's tau, the rank-correlation statistic used to measure agreement between metrics and human judgments.","marker":"Kendall, 1938"},{"why":"Defines ROUGE, the surface-level n-gram metric compared against MiRAGE in the agreement analysis.","marker":"Lin, 2004"},{"why":"Defines BERTScore, the semantic similarity metric compared against MiRAGE in the agreement analysis.","marker":"Zhang et al., 2019"}],"fun_headline_variants":["Video RAG quality? Score subclaims, not sentences","Claim-level scores win for multimodal RAG","MiRAGE: RAG eval that tracks human judgment","For multimodal RAG, claim-level evaluation is the way"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all information in a video, image, or audio source can be decomposed into discrete textual subclaims whose support can be judged by a scoring function; if some content (a continuous visual trend, a melody, a non-linguistic relational fact) is not expressible as such a subclaim, INFOF1 and CITEF1 will simply miss it.","fun_headline_variants_meta":{"raw":{"variants":["Video RAG quality? Score subclaims, not sentences","Claim-level scores win for multimodal RAG","MiRAGE: RAG eval that tracks human judgment","For multimodal RAG, claim-level evaluation is the way"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00104,"raw_usage":{"total_tokens":4341,"prompt_tokens":875,"completion_tokens":3466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":3402}},"tokens_in":491,"tokens_out":3466,"duration_ms":24791,"temperature":1.0,"reasoning_tokens":3402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:40:18.661316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video where the decisive fact is a continuous, non-linguistic property—for example, that a temperature trend rose steadily over 24 hours with no single numeric claim being independently supported—generate an output that states that fact, and ask humans for extrinsic quality scores while MiRAGE scores the subclaims. If humans rate the output high but MiRAGE rates it low because the fact cannot be decomposed into discrete supported subclaims, the paper's decomposition premise is falsified.","supporting_citations":[],"review_version":2}