MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

Dang Quang Thien Tran; Franck Dernoncourt; Koustava Goswami; Nedim Lipka; Quang V. Dang; Ryan A. Rossi; Sai Soorya Rao Veeravalli; Samyadeep Basu; Trang Nguyen; Vinamra Tyagi

Attention from a few cross-modal retrieval heads can attribute multimodal answers in long documents without training, faster and more accurately than prompting the same model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 08:49 UTC pith:YW4VFL3L

load-bearing objection Solid systems paper: training-free multimodal attribution from prefill attention plus a first long-doc free-form MultAttrEval benchmark; gains and latency look real on this setup, with the main soft spot being small same-pipeline probe calibration. the 3 major comments →

arxiv 2607.01420 v3 pith:YW4VFL3L submitted 2026-07-01 cs.CL cs.AIcs.CV

MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

Dang Quang Thien Tran , Quang V. Dang , Vinamra Tyagi , Sai Soorya Rao Veeravalli , Trang Nguyen , Ryan A. Rossi , Franck Dernoncourt , Nedim Lipka

show 2 more authors

Koustava Goswami Samyadeep Basu

This is my paper

classification cs.CL cs.AIcs.CV

keywords multimodal attributionlong document QAattention headstraining-free methodsretrieval headscausal mediationMultAttrEvalvision-language models

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When AI assistants answer questions over long PDFs that mix text and images, users need to know which passages and figures actually support each claim. MultAttnAttrib shows that this localization need not require fine-tuning or a second generative pass: a single prefill pass, a small set of attention heads that act as cross-modal retrieval circuits, and calibrated score thresholds are enough to cite text spans and image regions. The authors also release MultAttrEval, a long-document benchmark with fine-grained ground-truth attributions across text-only, image-only, and joint regimes. On that benchmark the method beats strong prompting and retrieval-augmented baselines on the same open backbone, stays competitive with a frontier closed model, and runs at roughly one-seventh the latency while using less peak memory. The practical stake is clear: trustworthy multimodal grounding becomes cheaper and more precise if the model already knows where it looked.

Core claim

MultAttnAttrib claims that multimodal answer attribution in long interleaved documents can be recovered, without any training, by identifying a sparse set of cross-modal retrieval heads via causal mediation on a small probe set, averaging their attention over image patches and text windows, and applying modality-specific thresholds learned by an F1 sweep; the resulting citations outperform prompting-based attribution on the same backbone and remain competitive with frontier models while costing a single prefill pass.

What carries the argument

Cross-modal retrieval heads selected by causal mediation analysis (clean vs. corrupted evidence) plus min-max-normalized attention scores thresholded for images and text; attribution is a single forward prefill pass that scores image slots and sliding text windows jointly.

Load-bearing premise

A small labeled probe set of about ninety question-answer-attribution triples is enough to pick heads and thresholds that still work on the held-out long documents.

What would settle it

Re-run head identification and threshold calibration on a disjoint probe sample drawn from different documents or domains; if the new heads and thresholds produce a large drop in test F1 relative to the original probe, the generalization claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid systems paper: training-free multimodal attribution from prefill attention plus a first long-doc free-form MultAttrEval benchmark; gains and latency look real on this setup, with the main soft spot being small same-pipeline probe calibration.

read the letter

Punchline: this is a useful systems contribution for multimodal long-document attribution. MultAttnAttrib takes the retrieval-head idea (Basu, Wu et al.), extends it with multimodal CMA-style corruption, joint image-slot + text-window scoring, and F1-calibrated modality thresholds, then extracts citations from one prefill pass. MultAttrEval is the other real piece: free-form localization over full interleaved PDFs rather than citation selection from a pre-curated pool.

What they do well is clear and proportionate. On the same Qwen3-VL-30B backbone they beat prompting, captioning, and RAG variants across text-only, image-only, and joint regimes (Table 2), with especially strong image precision and better text recall after thresholding. They are competitive with prompted GPT-5.4 on several axes, particularly visual grounding, while reporting ~7× lower latency and ~15GB lower peak VRAM than direct VLM prompting on non-OOM inputs. The head analysis is honest: top ranks are somewhat modality-specific, the broader population is shared, CMA favors cross-modal heads more than mean attention, and retrieval heads are sparse. Related work correctly distinguishes candidate-pool citation benches from true localization. Algorithms and evaluation details are spelled out enough to follow.

Soft spots, in proportion. The 90-item probe (30 per regime) is used both for CMA head ID and for threshold sweeps; probe and test come from the same MLLM-generated MultAttrEval pipeline, so the stress-test concern about distribution-specific calibration is real and under-quantified—no probe-size or label-source sensitivity. Fuzzy text tiers and exact image match can under-credit near-duplicates, which the Limitations section already flags. Labels are synthetic under strict filters, not human-authored free-form attributions. None of that collapses the central claim on this benchmark; it does mean external transfer is still open.

Who it’s for: people building grounded multimodal assistants, attribution tooling, or long-doc VQA eval. Math is light (attention aggregation + thresholds), data and citation pattern look solid for an empirical systems paper. I would send it to peer review; it deserves referee time, ideally with a request for probe ablations and public data/code. Worth engaging if you care about practical multimodal grounding.

Referee Report

3 major / 0 minor

Summary. The paper proposes MultAttnAttrib, a training-free multimodal attribution method for long-document QA that identifies cross-modal retrieval heads via causal mediation analysis (CMA) on a small labeled probe set, calibrates modality-specific thresholds by F1 sweep, and extracts text-span and image-slot citations from a single prefill attention pass (Alg. 1–4, §3). It also introduces MultAttrEval, a synthetic QAA benchmark over long PDFs spanning text-only, image-only, and joint regimes across five domains (§4, App. A). On a held-out MultAttrEval test split, MultAttnAttrib on Qwen3-VL-30B outperforms same-backbone VLM/LLM/RAG prompting baselines in F1 across regimes (Table 2), is competitive with prompted GPT-5.4 especially on image metrics (Fig. 3, App. F), and reports ~7× lower latency and ~15 GB lower peak VRAM than direct VLM prompting (Table 1).

Significance. If the gains hold beyond this benchmark, the work is a useful contribution: multimodal attribution in long interleaved documents is under-served relative to unimodal citation work, MultAttrEval is the first fine-grained localization benchmark of its kind rather than candidate-pool citation selection, and a single-pass attention method that matches frontier prompting at a fraction of the cost is practically valuable for grounded assistants. Strengths include explicit head-agreement analyses (IoU/Spearman, layer distributions; §6.3, Figs. 4–5), latency/memory measurements, multi-regime tables on both open and frontier models, and an LLM-as-judge supplement. The main significance risk is that both method calibration and evaluation live on the same synthetically generated MultAttrEval distribution.

major comments (3)

§5.1 and §3.2–3.3: Head identification (CMA on MultAttrEval ground-truth positions) and F1-maximizing threshold calibration (Alg. 4) both use a 90-item probe drawn from the same MultAttrEval generation pipeline as the 608-item test set. Table 2’s large same-backbone F1 gains and GPT-competitive image results therefore rest on an untested transfer assumption. The Limitations section notes the labeled-probe requirement but does not report sensitivity to probe size, random split, or label source. At minimum, ablations (probe-size curves; leave-one-domain-out calibration; thresholds fixed without F1 sweep) are needed before the central claim can be treated as robust rather than MultAttrEval-specific.
§4 and App. K: MultAttrEval QAAs are MLLM-generated under strict synthetic filters (entity co-occurrence, 12–25-word extractive spans, cross-modal grounding and circularity rubrics). Evaluation then uses exact image match and fuzzy substring tiers with length-ratio penalties (App. C). Limitations already notes near-duplicate images depress image metrics; more load-bearing is whether the benchmark’s generation artifacts (single-image image-only items, short extractive text spans, entity-verified multimodal pairs) systematically favor attention localization over generative citation. Without a human-authored or external-document subset, it is hard to separate method quality from benchmark style.
Table 2 vs. App. F / Fig. 3: The abstract and §6.2 claim MultAttnAttrib “matches” or is competitive with GPT-5.4, but the strongest GPT Cohere+LLM text F1 (0.888) and multimodal F1 (0.663) still exceed Cohere+MultAttnAttrib (0.665 / 0.601), while MultAttnAttrib leads mainly on image precision/F1 and recall-oriented text metrics. The claim should be restated with regime-specific caveats (stronger on visual localization and efficiency; mixed or weaker on high-precision text citation) so the contribution is not oversold.

Circularity Check

0 steps flagged

No by-construction circularity: MultAttnAttrib is supervised head/threshold calibration on a held-out MultAttrEval split, not a first-principles prediction forced by its inputs.

full rationale

This is an empirical methods paper, not a derivation of a physical or mathematical quantity from axioms. MultAttnAttrib (1) scores attention heads on a labeled probe set via CMA/MAS against MultAttrEval ground-truth evidence positions, (2) sweeps F1-maximizing modality thresholds on those probe score distributions, and (3) attributes held-out test QAAs with the selected heads and thresholds in one prefill pass. That is standard labeled calibration plus held-out evaluation, not a self-definitional loop: the test attributions are not algebraically or statistically forced to equal the probe fits. Baselines (prompting, captioning, RAG) are run without those probe-derived heads/thresholds, so Table 2 F1 gains are comparative empirical results, not tautologies. Overlap with Basu et al. (2025) is ordinary methodological inheritance for CMA-style head scoring, not a load-bearing uniqueness theorem that forbids alternatives. The only mild circularity-adjacent concern is that probe and test share the same MultAttrEval generation pipeline and domain mix, so calibration may overfit that synthetic label style—this is a generalization/overfitting risk, not reduction of the central claim to its inputs by construction. Limitations already note the labeled-probe requirement. Score 1 reflects that minor same-family calibration dependency without elevating it to fitted-input-as-prediction or self-definitional circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The central claim rests on empirical ML assumptions: that a sparse set of attention heads mediates multimodal evidence retrieval, that clean/corrupt attention differences identify those heads, and that F1-max thresholds on a small probe transfer. Free parameters are the discrete design choices (k, W, T_img, T_txt) and the probe labels themselves. No new physical entities are postulated.

free parameters (4)

top-k retrieval heads H
Number and identity of heads retained after CMA/MAS scoring; chosen from probe scores and treated as fixed at test time.
text sliding-window length W
Set to median chunk token length on the probe set; directly shapes text span scores.
modality thresholds T_img and T_txt
Chosen by F1-maximizing sweep on probe attribution scores; decision boundary for citing images vs text.
probe set size and composition (90 QAAs)
Labeled MultAttrEval probes used for both head identification and calibration; composition affects which heads and thresholds are selected.

axioms (4)

domain assumption A sparse subset of attention heads causally mediates retrieval of supporting evidence from long multimodal context.
Inherited from text retrieval-head literature and extended multimodally in §3; underpins single-pass attribution.
ad hoc to paper Clean-vs-corrupted mean attention difference (CMA), entropy-weighted, ranks heads by causal retrieval contribution across modalities.
Method choice in §3.2; corruption replaces GT text spans and resizes images while preserving shapes.
ad hoc to paper Min-max normalized head-averaged attention over image slots and text windows, thresholded, yields valid citations (with max-score fallback).
Algorithm 1 operational definition of attribution.
domain assumption MultAttrEval filtered MLLM-generated QAA labels are sufficiently accurate ground truth for ranking methods.
Dataset construction and multi-stage rubrics in §4 and Appendix K; evaluation validity depends on this.

invented entities (2)

MultAttnAttrib pipeline (cross-modal retrieval heads + calibrated multimodal thresholds) no independent evidence
purpose: Produce modality-aware text/image citations from one prefill pass without training.
Methodological construct, not a physical entity; independent evidence is the empirical MultAttrEval comparison.
MultAttrEval benchmark no independent evidence
purpose: Provide fine-grained multimodal attribution labels on long documents for evaluation.
New dataset entity; value depends on external adoption and label quality audits not fully externalized here.

pith-pipeline@v1.1.0-grok45 · 29509 in / 3213 out tokens · 27145 ms · 2026-07-12T08:49:30.005790+00:00 · methodology

0 comments

read the original abstract

As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.

Figures

Figures reproduced from arXiv: 2607.01420 by Dang Quang Thien Tran, Franck Dernoncourt, Koustava Goswami, Nedim Lipka, Quang V. Dang, Ryan A. Rossi, Sai Soorya Rao Veeravalli, Samyadeep Basu, Trang Nguyen, Vinamra Tyagi.

**Figure 1.** Figure 1: MULTATTNATTRIB: We identify signals for each attention head, then filter to select cross-modal heads. We then calibrate the threshold to maximize F1 scores on the probe set from MULTATTREVAL. For attribution, we use our top k heads to generate attention spans and return the final results using our calibrated thresholds. et al., 2025). However, these approaches focus on text-only QA, leaving image and multi… view at source ↗

**Figure 2.** Figure 2: MULTATTREVAL: Overview of the QAA generation process used to construct MultAttrEval from processed MINT-1T PDFs across text-only, image-only, and combined text-image attribution settings. then use an MLLM to generate QA using only our selected images or text-chunk spans, thereby creating unimodal attributions for our input. Text + Image. This case warrants a different treatment from the previous cases, a… view at source ↗

**Figure 3.** Figure 3: MULTATTNATTRIB closely matches and is competitive with latest frontier models such as GPT-5.4. Comparing all GPT variants with the Cohere + MULTATTNATTRIB (Ours) variant. the circuits are shared, a single joint head set is preferable and reduces the cost of modality-specific head identification. We score all L × H = 1536 heads under both CMA and MAS, then measure cross-modal agreement via IoU and Spearman’… view at source ↗

**Figure 4.** Figure 4: Crossmodal retrieval head agreement under CMA and Mean Attention Scoring. The usage of CMA results in higher overlap between image and text head sets in comparison to using Mean Attention. The broader head population is largely crossmodal with specialization at the very top ranks. (a) Layer distribution of heads in the CMA top-20. (b) Min-max normalized CMA head score distribution per modality [PITH_FULL_… view at source ↗

**Figure 5.** Figure 5: Layer distribution and score sparsity of CMA top-20 heads. Image heads concentrate at mid-to-late layers while text heads span early to late layers; crossmodal heads cluster in the transition zone. A small proportion of heads scored above 0.6 in any modality, indicating that retrieval heads are scarce for both text and images. across modalities, improving IoU and reducing anticorrelation at small k. In co… view at source ↗

**Figure 7.** Figure 7: Distribution of MultAttrEval QAA items by [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗

**Figure 8.** Figure 8: Distribution of MultAttrEval QAA items by [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗

**Figure 9.** Figure 9: Domain Difficulty Chart F1 scores for each of our regimes, grouped by document domain and method used, ordered from hardest to easiest domains (all modalities pooled) Domains have no tangible performance impact on intra-baseline relationships. Generally, the VLM and LLM baselines perform the worst, with Cohere + VLM and Cohere + LLM being similarly better, and MULTATTNATTRIB, along with its Cohere variant… view at source ↗

**Figure 12.** Figure 12: CMA attribution score heatmap for heads that attend to text sources. Circles mark the top-20 text heads, and diamonds mark the top-20 cross-modal heads. 0 5 10 15 20 25 30 Head 0 10 20 30 40 Layer Image Text Crossmodal 0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png] view at source ↗

**Figure 13.** Figure 13: Relative modality specialization of CMA [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

19 extracted references · 1 linked inside Pith

[1]

Luyu Gao, Zhuyun Dai, Panupong Pasupat, An- thony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y

Benchmarking retrieval-augmented multi- modal generation for document question answer- ing.Advances in Neural Information Processing Systems, 38. Luyu Gao, Zhuyun Dai, Panupong Pasupat, An- thony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y . Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. Rarr: 9 Researching and revising what lang...

arXiv 2023
[2]

InProceedings of the 2025 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 47–54

Vericite: Towards reliable citations in retrieval-augmented generation via rigorous ver- ification. InProceedings of the 2025 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 47–54. Pritika Ramu, Koustava Goswami, Apoorv Saxena, and Balaji Vasan Srinivasan. 2024. Enhancing p...

2025
[3]

InProceedings of the AAAI Confer- ence on Artificial Intelligence, volume 40, pages 33028–33037

Mavis: A benchmark for multimodal source attribution in long-form visual question 10 answering. InProceedings of the AAAI Confer- ence on Artificial Intelligence, volume 40, pages 33028–33037. David Wan, Han Wang, Ziyang Wang, Elias Stengel-Eskin, Hyunji Lee, and Mohit Bansal
[4]

in the image

Multimodal fact-level attribution for verifiable reasoning.arXiv preprint arXiv:2602.11509. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. In- terpretability in the wild: a circuit for indirect ob- ject identification in gpt-2 small.arXiv preprint arXiv:2211.00593. Yanting Wang, Runpeng Geng, Ying Chen, and Jin...

Pith/arXiv arXiv 2022
[5]

Produce ‘domain_grounding‘ (2--4 sentences) summarizing subject matter and terminology from **raster-legible** content, aligned with grounding chunks for domain only
[6]

Set ‘is_relevant‘ false only for blank/decorative/unusably degraded content
[7]

visual"‘ (specific values/labels/readouts in the raster are **essential** to justify the answer) or ‘

If relevant, emit all strong non-redundant ‘qa_pairs‘. Each pair **must** include: - ‘question‘, ‘answer‘, ‘type‘ in \{relational, inferential, procedural, quantitative\} - ‘answer_evidence‘: one of ‘"visual"‘ (specific values/labels/readouts in the raster are **essential** to justify the answer) or ‘"visual_plus_general"‘ (answer combines one raster-spec...
[8]

The answer MAY be paraphrased (it does not need to be copied verbatim)
[9]

Do NOT add facts not present in the paragraph

The answer MUST be fully supported by the paragraph. Do NOT add facts not present in the paragraph
[10]

The answer MUST be between 12 and 25 words long (inclusive)
[11]

The question MUST require reading comprehension of the paragraph, not just simple word or name lookup
[12]

Each question MUST be answerable solely from the given paragraph, without any external knowledge
[13]

Triplets must be diverse: do NOT ask multiple questions that can be answered with nearly the same statement
[14]

this page

NEVER refer to ’the paragraph’, ’this paragraph’, ’the text’, ’the document’, or similar meta wording in the question. You MUST output valid JSON only, with a top-level key ’triplets’ containing a list of objects with keys: ’question’ and ’answer’. L.1.3 Multimodal Listing 7: Multimodal QAA System Prompt You are an expert at creating challenging, non-triv...
[15]

**Scan document images** and identify indices whose visual content (figures, tables, charts, diagrams, schematics, photos, labeled components, layouts) is relevant and supportive of the reference answer
[17]

L.3.3 LLM Preamble When using captions instead of raw raster content, we can apply the modification below, followed by the general prompt

Use whichever evidence is actually supportive: - text only, - images only, or - both text and images. L.3.3 LLM Preamble When using captions instead of raw raster content, we can apply the modification below, followed by the general prompt. You are given **document image captions** (for labels ‘[Image 0]‘, ‘[Image 1]‘, ... in order) and **document text**....
[18]

**Scan image captions** tied to ‘[Image k]‘ and identify indices whose described visual content is relevant and supportive of the reference answer
[19]

**Scan document text** and find the passage that most directly states or supports the key fact(s) in the reference answer
[20]

Use whichever evidence is actually supportive: - text only, - images only, or - both text and images. 25

[1] [1]

Luyu Gao, Zhuyun Dai, Panupong Pasupat, An- thony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y

Benchmarking retrieval-augmented multi- modal generation for document question answer- ing.Advances in Neural Information Processing Systems, 38. Luyu Gao, Zhuyun Dai, Panupong Pasupat, An- thony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y . Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023a. Rarr: 9 Researching and revising what lang...

arXiv 2023

[2] [2]

InProceedings of the 2025 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 47–54

Vericite: Towards reliable citations in retrieval-augmented generation via rigorous ver- ification. InProceedings of the 2025 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 47–54. Pritika Ramu, Koustava Goswami, Apoorv Saxena, and Balaji Vasan Srinivasan. 2024. Enhancing p...

2025

[3] [3]

InProceedings of the AAAI Confer- ence on Artificial Intelligence, volume 40, pages 33028–33037

Mavis: A benchmark for multimodal source attribution in long-form visual question 10 answering. InProceedings of the AAAI Confer- ence on Artificial Intelligence, volume 40, pages 33028–33037. David Wan, Han Wang, Ziyang Wang, Elias Stengel-Eskin, Hyunji Lee, and Mohit Bansal

[4] [4]

in the image

Multimodal fact-level attribution for verifiable reasoning.arXiv preprint arXiv:2602.11509. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. In- terpretability in the wild: a circuit for indirect ob- ject identification in gpt-2 small.arXiv preprint arXiv:2211.00593. Yanting Wang, Runpeng Geng, Ying Chen, and Jin...

Pith/arXiv arXiv 2022

[5] [5]

Produce ‘domain_grounding‘ (2--4 sentences) summarizing subject matter and terminology from **raster-legible** content, aligned with grounding chunks for domain only

[6] [6]

Set ‘is_relevant‘ false only for blank/decorative/unusably degraded content

[7] [7]

visual"‘ (specific values/labels/readouts in the raster are **essential** to justify the answer) or ‘

If relevant, emit all strong non-redundant ‘qa_pairs‘. Each pair **must** include: - ‘question‘, ‘answer‘, ‘type‘ in \{relational, inferential, procedural, quantitative\} - ‘answer_evidence‘: one of ‘"visual"‘ (specific values/labels/readouts in the raster are **essential** to justify the answer) or ‘"visual_plus_general"‘ (answer combines one raster-spec...

[8] [8]

The answer MAY be paraphrased (it does not need to be copied verbatim)

[9] [9]

Do NOT add facts not present in the paragraph

The answer MUST be fully supported by the paragraph. Do NOT add facts not present in the paragraph

[10] [10]

The answer MUST be between 12 and 25 words long (inclusive)

[11] [11]

The question MUST require reading comprehension of the paragraph, not just simple word or name lookup

[12] [12]

Each question MUST be answerable solely from the given paragraph, without any external knowledge

[13] [13]

Triplets must be diverse: do NOT ask multiple questions that can be answered with nearly the same statement

[14] [14]

this page

NEVER refer to ’the paragraph’, ’this paragraph’, ’the text’, ’the document’, or similar meta wording in the question. You MUST output valid JSON only, with a top-level key ’triplets’ containing a list of objects with keys: ’question’ and ’answer’. L.1.3 Multimodal Listing 7: Multimodal QAA System Prompt You are an expert at creating challenging, non-triv...

[15] [15]

**Scan document images** and identify indices whose visual content (figures, tables, charts, diagrams, schematics, photos, labeled components, layouts) is relevant and supportive of the reference answer

[16] [17]

L.3.3 LLM Preamble When using captions instead of raw raster content, we can apply the modification below, followed by the general prompt

Use whichever evidence is actually supportive: - text only, - images only, or - both text and images. L.3.3 LLM Preamble When using captions instead of raw raster content, we can apply the modification below, followed by the general prompt. You are given **document image captions** (for labels ‘[Image 0]‘, ‘[Image 1]‘, ... in order) and **document text**....

[17] [18]

**Scan image captions** tied to ‘[Image k]‘ and identify indices whose described visual content is relevant and supportive of the reference answer

[18] [19]

**Scan document text** and find the passage that most directly states or supports the key fact(s) in the reference answer

[19] [20]

Use whichever evidence is actually supportive: - text only, - images only, or - both text and images. 25