{"id":"4d6f4759-5921-4d6b-83d6-18a344684515","arxiv_id":"2607.01420","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training-free multimodal attribution from calibrated retrieval-head attention on long documents, with a new MultAttrEval benchmark and large latency gains over prompting.","lead":"MultAttnAttrib finds which text spans and images support a multimodal QA answer by reading selected attention heads from one prefill pass, without extra training. MultAttrEval is a new long-document benchmark for this task; the method beats same-backbone prompting and is competitive with GPT-5.4 at much lower latency.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Probe-set calibration may overfit MultAttrEval, so reported gains over same-backbone prompting may not transfer beyond this synthetic distribution.","rationale":"The reader's weakest assumption is exactly the load-bearing hinge: without transfer of the 90-probe CMA heads and F1 thresholds, the headline empirical superiority over same-backbone prompting and the GPT-competitive story on MultAttrEval do not stand. The paper's own design (single shared MultAttrEval distribution, supervised calibration, synthetic QAA pipeline) makes this the most concrete risk, more so than latency claims (which are mechanical) or novelty relative to unimodal retrieval heads. I agree with CONDITIONAL rather than REJECT: the method is training-free at inference, the attention-based localization idea is sound, and results are consistent across regimes; the missing piece is external validation of calibration stability and label quality. No stronger internal inconsistency appears in the algorithms or metrics. Keeping CONDITIONAL with the same probe-overfit concern is the honest adjustment; confidence remains moderate until the proposed split or human-label check is run.","tokens_in":25396,"tokens_out":650,"duration_ms":7256,"concrete_test":"Re-run head identification and threshold calibration on a random 45-item half of the probe (or on a held-out human-relabeled 30-per-regime subset if available), freeze those heads/thresholds, and re-evaluate Table 2 Full Document and Cohere rows on the full 608-item test set. If any regime F1 drops by more than ~0.05 absolute or the Δ vs VLM falls below ~10%, the generalization claim for the reported gains is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that MultAttnAttrib consistently outperforms same-backbone prompting/RAG and is competitive with GPT-5.4 rests on heads and thresholds chosen from a 90-item probe (30 per regime) drawn from the same MultAttrEval pipeline as the 608-item test set (§5.1, §3.2–3.3). CMA scores heads by clean-vs-corrupted attention to MultAttrEval ground-truth positions, then F1-maximizing sweeps set T_img and T_txt on those same probe score distributions (Alg. 4). Because MultAttrEval QAAs are MLLM-generated under strict synthetic filters (entity co-occurrence, 12–25-word extractive spans, cross-modal grounding rubrics; §4, App. K), probe and test share the same generation artifacts and domain mix. If selected heads or thresholds are tuned to that label style rather than to general cross-modal retrieval, Table 2 F1 gains (e.g. Full Document +22.9% text, +25.8% image vs VLM) and GPT-competitive image precision can be inflated relative to human-authored or out-of-distribution documents. The Limitations section acknowledges the labeled-probe requirement but does not quantify sensitivity to probe size, split, or label source.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes MultAttnAttrib, a training-free multimodal attribution method for long-document QA that identifies cross-modal retrieval heads via causal mediation analysis (CMA) on a small labeled probe set, calibrates modality-specific thresholds by F1 sweep, and extracts text-span and image-slot citations from a single prefill attention pass (Alg. 1–4, §3). It also introduces MultAttrEval, a synthetic QAA benchmark over long PDFs spanning text-only, image-only, and joint regimes across five domains (§4, App. A). On a held-out MultAttrEval test split, MultAttnAttrib on Qwen3-VL-30B outperforms same-backbone VLM/LLM/RAG prompting baselines in F1 across regimes (Table 2), is competitive with prompted GPT-5.4 especially on image metrics (Fig. 3, App. F), and reports ~7× lower latency and ~15 GB lower peak VRAM than direct VLM prompting (Table 1).","tokens_in":25800,"tokens_out":928,"duration_ms":7732,"significance":"If the gains hold beyond this benchmark, the work is a useful contribution: multimodal attribution in long interleaved documents is under-served relative to unimodal citation work, MultAttrEval is the first fine-grained localization benchmark of its kind rather than candidate-pool citation selection, and a single-pass attention method that matches frontier prompting at a fraction of the cost is practically valuable for grounded assistants. Strengths include explicit head-agreement analyses (IoU/Spearman, layer distributions; §6.3, Figs. 4–5), latency/memory measurements, multi-regime tables on both open and frontier models, and an LLM-as-judge supplement. The main significance risk is that both method calibration and evaluation live on the same synthetically generated MultAttrEval distribution.","major_comments":[{"comment":"§5.1 and §3.2–3.3: Head identification (CMA on MultAttrEval ground-truth positions) and F1-maximizing threshold calibration (Alg. 4) both use a 90-item probe drawn from the same MultAttrEval generation pipeline as the 608-item test set. Table 2’s large same-backbone F1 gains and GPT-competitive image results therefore rest on an untested transfer assumption. The Limitations section notes the labeled-probe requirement but does not report sensitivity to probe size, random split, or label source. At minimum, ablations (probe-size curves; leave-one-domain-out calibration; thresholds fixed without F1 sweep) are needed before the central claim can be treated as robust rather than MultAttrEval-specific.","section":null},{"comment":"§4 and App. K: MultAttrEval QAAs are MLLM-generated under strict synthetic filters (entity co-occurrence, 12–25-word extractive spans, cross-modal grounding and circularity rubrics). Evaluation then uses exact image match and fuzzy substring tiers with length-ratio penalties (App. C). Limitations already notes near-duplicate images depress image metrics; more load-bearing is whether the benchmark’s generation artifacts (single-image image-only items, short extractive text spans, entity-verified multimodal pairs) systematically favor attention localization over generative citation. Without a human-authored or external-document subset, it is hard to separate method quality from benchmark style.","section":null},{"comment":"Table 2 vs. App. F / Fig. 3: The abstract and §6.2 claim MultAttnAttrib “matches” or is competitive with GPT-5.4, but the strongest GPT Cohere+LLM text F1 (0.888) and multimodal F1 (0.663) still exceed Cohere+MultAttnAttrib (0.665 / 0.601), while MultAttnAttrib leads mainly on image precision/F1 and recall-oriented text metrics. The claim should be restated with regime-specific caveats (stronger on visual localization and efficiency; mixed or weaker on high-precision text citation) so the contribution is not oversold.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a useful systems contribution for multimodal long-document attribution. MultAttnAttrib takes the retrieval-head idea (Basu, Wu et al.), extends it with multimodal CMA-style corruption, joint image-slot + text-window scoring, and F1-calibrated modality thresholds, then extracts citations from one prefill pass. MultAttrEval is the other real piece: free-form localization over full interleaved PDFs rather than citation selection from a pre-curated pool.\n\nWhat they do well is clear and proportionate. On the same Qwen3-VL-30B backbone they beat prompting, captioning, and RAG variants across text-only, image-only, and joint regimes (Table 2), with especially strong image precision and better text recall after thresholding. They are competitive with prompted GPT-5.4 on several axes, particularly visual grounding, while reporting ~7× lower latency and ~15GB lower peak VRAM than direct VLM prompting on non-OOM inputs. The head analysis is honest: top ranks are somewhat modality-specific, the broader population is shared, CMA favors cross-modal heads more than mean attention, and retrieval heads are sparse. Related work correctly distinguishes candidate-pool citation benches from true localization. Algorithms and evaluation details are spelled out enough to follow.\n\nSoft spots, in proportion. The 90-item probe (30 per regime) is used both for CMA head ID and for threshold sweeps; probe and test come from the same MLLM-generated MultAttrEval pipeline, so the stress-test concern about distribution-specific calibration is real and under-quantified—no probe-size or label-source sensitivity. Fuzzy text tiers and exact image match can under-credit near-duplicates, which the Limitations section already flags. Labels are synthetic under strict filters, not human-authored free-form attributions. None of that collapses the central claim on this benchmark; it does mean external transfer is still open.\n\nWho it’s for: people building grounded multimodal assistants, attribution tooling, or long-doc VQA eval. Math is light (attention aggregation + thresholds), data and citation pattern look solid for an empirical systems paper. I would send it to peer review; it deserves referee time, ideally with a request for probe ablations and public data/code. Worth engaging if you care about practical multimodal grounding.","headline":"Solid systems paper: training-free multimodal attribution from prefill attention plus a first long-doc free-form MultAttrEval benchmark; gains and latency look real on this setup, with the main soft spot being small same-pipeline probe calibration.","tokens_in":26402,"tokens_out":577,"would_cite":true,"duration_ms":6284,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Attention from a few cross-modal retrieval heads can attribute multimodal answers in long documents without training, faster and more accurately than prompting the same model.","keywords":["multimodal attribution","long document QA","attention heads","training-free methods","retrieval heads","causal mediation","MultAttrEval","vision-language models"],"falsifier":"Re-run head identification and threshold calibration on a disjoint probe sample drawn from different documents or domains; if the new heads and thresholds produce a large drop in test F1 relative to the original probe, the generalization claim fails.","tokens_in":26312,"feed_emoji":"📄","tokens_out":613,"duration_ms":5379,"temperature":0.7,"pith_summary":"When AI assistants answer questions over long PDFs that mix text and images, users need to know which passages and figures actually support each claim. MultAttnAttrib shows that this localization need not require fine-tuning or a second generative pass: a single prefill pass, a small set of attention heads that act as cross-modal retrieval circuits, and calibrated score thresholds are enough to cite text spans and image regions. The authors also release MultAttrEval, a long-document benchmark with fine-grained ground-truth attributions across text-only, image-only, and joint regimes. On that benchmark the method beats strong prompting and retrieval-augmented baselines on the same open backbone, stays competitive with a frontier closed model, and runs at roughly one-seventh the latency while using less peak memory. The practical stake is clear: trustworthy multimodal grounding becomes cheaper and more precise if the model already knows where it looked.","feed_headline":"Attention heads alone cite text and images in long PDFs","feed_subtitle":"One prefill pass beats prompting on the same model and matches frontier systems at one-seventh the latency","key_machinery":"Cross-modal retrieval heads selected by causal mediation analysis (clean vs. corrupted evidence) plus min-max-normalized attention scores thresholded for images and text; attribution is a single forward prefill pass that scores image slots and sliding text windows jointly.","core_discovery":"MultAttnAttrib claims that multimodal answer attribution in long interleaved documents can be recovered, without any training, by identifying a sparse set of cross-modal retrieval heads via causal mediation on a small probe set, averaging their attention over image patches and text windows, and applying modality-specific thresholds learned by an F1 sweep; the resulting citations outperform prompting-based attribution on the same backbone and remain competitive with frontier models while costing a single prefill pass.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Sparse attention heads attribute text and images without training","One prefill pass cites multimodal evidence in long documents","Selected heads plus thresholds locate answer sources in PDFs","Training-free MultAttnAttrib matches frontier multimodal attribution","Prefill attention recovers fine-grained multimodal citations fast"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"A small labeled probe set of about ninety question-answer-attribution triples is enough to pick heads and thresholds that still work on the held-out long documents.","fun_headline_variants_meta":{"raw":{"variants":["Sparse attention heads attribute text and images without training","One prefill pass cites multimodal evidence in long documents","Selected heads plus thresholds locate answer sources in PDFs","Training-free MultAttnAttrib matches frontier multimodal attribution","Prefill attention recovers fine-grained multimodal citations fast"]},"model":"grok-4.5","effort":"low","cost_usd":0.004792,"raw_usage":{"total_tokens":1376,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":47920000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":547,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":59,"duration_ms":5341,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T08:49:30.005790+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run head identification and threshold calibration on a disjoint probe sample drawn from different documents or domains; if the new heads and thresholds produce a large drop in test F1 relative to the original probe, the generalization claim fails.","supporting_citations":[],"review_version":2}