Pith. sign in

REVIEW 2 major objections 5 minor 17 references

Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When multimodal retrieval is polluted, selective trust beats unconditional fusion for multi-sentence factual QA.

desk verdict Solid controlled stress-test: unconditional multimodal fusion is brittle under polluted text, and training-free source-aware selection (Field-Selector) recovers the clean/polluted trade-off mainly via text reliability. read the letter →

arxiv 2607.10798 v1 pith:YLGILXVL submitted 2026-07-12 cs.CL

Saadeldine Eletter , Owais Aijaz , Preslav Nakov This is my paper
classification cs.CL
keywords multimodalRAGretrievalpollutionsource-awaretrustresolutionQIMG-7SATRfactualQAselectiveimageattacks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal retrieval-augmented generation is usually tested with clean evidence, but real retrieval can return topically relevant yet unreliable text and images. This paper builds QIMG-7, a controlled stress-test for that failure mode in multi-sentence factual QA: four datasets, seven image-attack families, and sixteen paired clean/polluted regimes. Across generator stacks, naive full multimodal fusion is brittle—under polluted text its support collapses, often below answering from parametric knowledge alone. The authors introduce source-aware trust resolution (SATR), a training-free method that generates Parametric, Text-only, and Full-MM candidates, judges source reliability, and selects, drops a channel, or falls back. Field-Selector is the strongest variant; ablations show the gains come mainly from explicit text-reliability modeling. The practical claim is simple: in text-first factual QA with multimodal conflict, trust sources selectively rather than fuse everything by default.

What carries the argument

Source-aware trust resolution (SATR), especially Field-Selector: isolate Parametric, Text-only, and Full-MM answers; emit structured reliability and conflict fields; then deterministically fall back if text is untrusted, drop the image channel if the image is weak or suspicious, otherwise pick the highest-scored candidate.

What would settle it

On a held-out set of real web retrieval failures with natural text/image pollution—not synthetic minimal edits or constructed image attacks—measure whether Field-Selector still beats Full-MM on balanced factual support under the same clean-evidence judge protocol.

Watch

Extended reading notes

Core claim

In text-first multi-sentence factual QA with multimodal retrieval conflict, unconditional Full-MM fusion is unsafe under polluted text (main stack: clean-text support 0.908 falls to 0.490), while training-free source-aware trust resolution—especially Field-Selector SATR—raises balanced clean/polluted support to 0.816, driven primarily by explicit text-reliability modeling rather than surface consensus or unconditional multimodal fusion.

Load-bearing premise

Synthetic paired pollution on a controlled 110-question set, scored by an LLM judge against trusted clean evidence, is a good enough stand-in for real multimodal retrieval pollution that “selective trust over fusion” should transfer to deployment.

Editorial extensions

If this is right

  • Multimodal RAG systems for long-form factual QA should treat retrieval as optionally trustworthy and keep a parametric fallback path.
  • Answer-level surface consensus among candidates is not a sufficient defense when evidence is polluted.
  • In text-first settings, defenses should prioritize text-reliability signals over image-only or generic aggregation tricks.
  • Prompt-based trust routing only helps when the resolver can actually assess evidence reliability; weak gates can erase the gains.
  • Controlled paired clean/polluted regimes are a useful evaluation standard for multimodal retrieval robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Production multimodal RAG may need cheaper learned or distilled reliability routers so the multi-call SATR pattern does not dominate latency and cost.
  • The same selective-trust pattern likely matters more in vision-critical tasks (charts, OCR, multi-image docs) where images are necessary evidence rather than optional noise.
  • If text pollution remains the dominant harm channel online, hybrid pipelines that first gate text quality before spending on vision fusion could be the practical default.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies multimodal retrieval pollution for multi-sentence factual QA. It introduces QIMG-7, a controlled paired clean/polluted benchmark over four datasets, seven image-attack families, and 16 regimes (1,760 rows per method). Empirically, naive Full-MM fusion is brittle under polluted text (main gpt-4o-mini stack: clean-text support 0.908 → polluted-text 0.490), often worse than Parametric fallback. The authors propose training-free source-aware trust resolution (SATR) over isolated Parametric, Text-only, and Full-MM candidates; Field-Selector reaches balanced score 0.816 (+11.7 over Full-MM, +2.7 over Cascaded Router). Field ablations attribute gains mainly to explicit text_reliability modeling. Cross-model stacks, attack-family breakdowns, atomic claim audit, multi-judge sensitivity, bootstrap CIs, and a small human audit support the controlled finding that selective trust is preferable to unconditional fusion in this text-first setting.

Significance. If the controlled result holds, the paper makes a useful systems contribution: it supplies a reproducible stress-test benchmark for multimodal retrieval pollution and a simple, training-free design principle—compare isolated source branches and resolve by reliability rather than fuse by default. Strengths include released artifacts, paired clean/polluted regimes, multi-stack evaluation, field ablation isolating text_reliability, bootstrap intervals, multi-judge and human audits, and an atomic-claim check that preserves method ranking. The work is scoped as a stress test rather than a deployment guarantee, which is appropriate. Its main value is diagnostic and methodological: it shows that multimodal RAG evaluation under clean evidence can overstate robustness, and that answer-aware source selection can recover a better clean/polluted trade-off when the resolver can assess text reliability.

major comments (2)
  1. The central claim is carefully scoped to a controlled text-first stress test and is well supported by Tables 2–5, the LongFact field ablation (text_reliability removal drops polluted performance 0.958→0.720), decision distributions, bootstrap CIs, multi-judge audit, and human validation. I do not find an internal inconsistency that overturns that controlled finding. The remaining load-bearing limitation is external validity of the proxy: synthetic minimal-edit text families plus seven constructed image attacks on 110 questions, scored by LLM-as-judge against trusted clean evidence (§3, §5, Limitations). The manuscript already states this bound; for journal publication it should be made more operational in the main text by (i) stating more clearly that image attacks mainly create conflict while polluted text dominates harm, and (ii) adding one concrete external check or failure-mode discu
  2. Table 3 shows that SATR is resolver-quality dependent: Field-Selector is best balanced on three stacks but not on Llama-3.2-11B-Vision, where Full-MM remains slightly better balanced and polluted-regime routing degrades. This is acknowledged, but the abstract and conclusion still read as a fairly general design recommendation. The paper should elevate this dependence into a primary result condition rather than a secondary caveat, and state more explicitly when selective trust is expected to help versus when a weak gate can erase the gain.
minor comments (5)
  1. Abstract and §1 should more clearly foreground that this is a text-first factual QA setting in which polluted text is the dominant harmful channel; the current framing can over-suggest equal text/image threat.
  2. Table 2 and related tables would benefit from a short note that Parametric answers are regenerated per row and that small cross-regime differences are generation noise, as stated later in §5.
  3. Human validation of image attacks (n=56) and the 96-output human audit are useful but author-only; state this more prominently near the main claims rather than only in Ethical Considerations / Appendix.
  4. Appendix route-accuracy numbers are low; a one-sentence interpretation in the main decision-behavior section would help readers avoid over-reading exact branch matching.
  5. Minor polish: consistent naming of Field-Selector / Soft-Conductor, and a clearer pointer from Algorithm 1 to the exact reliability labels used in the field screen.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical systems evaluation with independent baselines, not a derivation that redefines its target.

full rationale

This paper is a controlled empirical systems study of multimodal RAG under synthetic retrieval pollution, not a first-principles derivation. The central claim—that unconditional Full-MM fusion is brittle under polluted text and that training-free source-aware trust resolution (especially Field-Selector SATR) improves the clean/polluted balanced support score—is established by comparing methods on QIMG-7 against Parametric, Text-only, Full-MM, and Answer-Consensus baselines, with the primary metric being LLM-as-judge support against trusted clean evidence that is never shown to the generators or routers. Pollution operators, regime construction, and candidate branches are defined independently of the reported scores; ablations (text_reliability removal), cross-model stacks, multi-judge audits, atomic claim audits, and human validation further test rather than tautologically force the result. There is no self-definitional equation, no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, and no load-bearing self-citation chain that substitutes for the evaluation. Mild shared-model risk (gpt-4o-mini often used for generation, gating, and primary judging) is an evaluator-dependence concern, not circularity by construction, and is partially mitigated by the paper’s own multi-judge and human audits. Score 0 is therefore appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

Load-bearing content is mostly experimental design choices and domain assumptions about pollution and evaluation, not fitted physical constants. The central claim rests on synthetic pollution operators, a fixed support-score judge protocol, and the premise that isolated Parametric/Text-only/Full-MM branches plus reliability fields are the right decision space. Free parameters are implementation hyperparameters rather than values fitted to manufacture the headline delta.

free parameters (3)
  • top-k text passages / max clean images = k=5, m=5
    Fixed at k=5 and m=5; retrieval breadth can change how often pollution is visible and how strong clean evidence is.
  • support-score label mapping = {1.0, 0.5, 0.0, 0.0}
    Judge labels supported/partial/unsupported/uncertain mapped to 1.0/0.5/0.0/0.0; this scoring convention affects absolute levels and balanced averages.
  • Field-Selector reliability screen thresholds = categorical screen rules in Algorithm 1
    Deterministic fallback when text_reliability ∈ {conflicting, suspicious, weak} and image drop when image_reliability ∈ {suspicious, weak, irrelevant}; these discrete rules are design choices that drive the headline method.
assumptions (4)
  • domain assumption Threat model: adversary may pollute retrieved text/image evidence but not the query, answer functions, or judge.
    Stated in §3 and Appendix A; defines what robustness means in QIMG-7.
  • domain assumption LLM-as-judge support against trusted clean evidence is a valid primary measure of answer quality under pollution.
    §5 evaluation metric; all headline rankings depend on this proxy, with partial external checks via multi-judge and human audits.
  • ad hoc to paper Synthetic minimal-edit text pollution and seven constructed image-attack families adequately instantiate multimodal retrieval pollution for stress testing.
    §3 pollution design and Limitations; without this, the benchmark does not support the deployment-facing design conclusion.
  • ad hoc to paper Isolating Parametric, Text-only, and Full-MM candidates is a sufficient decision space for selective trust.
    §4 methods; SATR never searches broader actions such as re-retrieval, claim-level editing of evidence, or multi-image forensics.
invented entities (2)
  • QIMG-7 benchmark independent evidence
    purpose: Controlled paired clean/polluted multimodal RAG stress test with 16 regimes and seven image-attack families.
    New evaluation construct introduced by the paper; independent use requires the released artifacts rather than prior public benchmarks alone.
  • SATR / Field-Selector
    purpose: Training-free source-aware resolver that selects among candidate answers or falls back using structured reliability fields.
    Method family defined in §4; performance is measured only inside this paper’s pipeline, though the idea is falsifiable on other polluted-RAG suites.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG." pith.science (2026). https://pith.science/paper/YLGILXVL

@misc{pith2026260710798,
  author       = {Pith},
  title        = {Pith review of: Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YLGILXVL}},
  note         = {Machine review of arXiv:2607.10798}
}
read the original abstract

Multimodal retrieval-augmented generation (RAG) is often evaluated with clean evidence, yet real retrieval can return topically relevant but unreliable content: false text and misleading images from corrupted metadata, entity swaps, typographic overlays, semantic edits, adversarial patches, blends, or style transfer. We introduce QIMG-7, a controlled benchmark for multimodal retrieval pollution in multi-sentence factual QA, spanning four datasets, seven image-attack families, and 16 paired clean/polluted regimes, for 1,760 evaluation rows per method. Across four generator/gate stacks, naive multimodal fusion is brittle: in the main gpt-4o-mini stack, Full-MM support drops from 0.908 with clean text to 0.490 with polluted text, often making Parametric fallback safer than retrieval. We propose source-aware trust resolution (SATR), a training-free approach that compares Parametric, Text-only, and Full-MM candidate answers and selects among candidate answers or falls back based on source reliability. The Field-Selector variant achieves the best balanced score, 0.816, improving over Full-MM by 11.7 points and over the Cascaded Router by 2.7 points. Ablations show that, in this text-first setting, explicit text-reliability modeling is the dominant driver of these gains. Overall, in text-first factual QA with multimodal retrieval conflict, our results support selective trust rather than unconditional fusion. Artifacts are available at https://github.com/SaadElDine/Trust_Before_Fusion.

Figures

Figures reproduced from arXiv: 2607.10798 by the authors.

Figure 1
Figure 1. Motivating failure case for multimodal re [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the QIMG-7 benchmark construction and SATR evaluation pipeline. QIMG-7 constructs [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The seven QIMG-7 image pollution families, shown alongside the clean retrieved image. Caption flip [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: SATR routing decisions across regimes. Both [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Polluted-text support score by image attack [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Cost-quality trade-off on QIMG-7. Field [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 linked inside Pith

  1. [1]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S

    Qwen2.5-VL technical report.arXiv preprint, arXiv:2502.13923. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S. Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. InAdvances in Neural Information Processing Systems, volume 36, ...

    arXiv 2023

  2. [2]

    Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dim- itrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, and Heng Ji

    MIRAGE: Defending long-form RAG against misinformation pollution.arXiv preprint, arXiv:2607.05069. Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dim- itrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, and Heng Ji. 2025. MM-PoisonRAG: Disrupting multimodal RAG with local and global poisoning attacks.arXiv preprint, arXiv:2502.17832...

    arXiv 2025

  3. [3]

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen

    Curran Associates, Inc. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucina- tion in large vision-language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305, Sin- gapore. Association for Computational Linguistics. Sheng-Chieh Lin, Luyu Gao, B...

    2023

  4. [4]

    InAdvances in Neural Informa- tion Processing Systems, volume 37, pages 115588– 115614

    FLAME: Factuality-aware alignment for large language models. InAdvances in Neural Informa- tion Processing Systems, volume 37, pages 115588– 115614. Yinuo Liu, Zenghui Yuan, Guiyao Tie, Jiawen Shi, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2025. Poisoned-MRAG: Knowledge poisoning attacks to multimodal retrieval augmented generation.arXiv preprint, ar...

    arXiv 2025

  5. [5]

    InAdvances in Neural Information Processing Systems, volume 37, pages 80756–80827

    Long-form factuality in large language mod- els. InAdvances in Neural Information Processing Systems, volume 37, pages 80756–80827. Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. Certifiably robust RAG against retrieval corruption. InICML 2024 Workshop on Next Generation of AI Safety. Shi-Qi Yan, Jia-Chen Gu, Yun Z...

    2024

  6. [6]

    Marie Curie won the Nobel Prize in Chemistry in 1911

    Corrective retrieval augmented generation. arXiv preprint, arXiv:2401.15884. Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Jun- hao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. Vis- RAG: Vision-based retrieval-augmented generation on multi-modality documents. InProceedings of the Thirteenth International Conference on ...

    arXiv 2025

  7. [7]

    Choose FULL_MM only if text and image evidence both look relevant and trustworthy

  8. [8]

    Choose TEXT_ONLY if text looks useful but image evidence looks suspicious, weak, irrelevant, or manipulated

Show all 17 references
  1. [9]

    Choose FALLBACK if text evidence itself looks inconsistent, polluted, or unreliable

  2. [10]

    decision

    Ignore your own world knowledge; judge only evidence quality. Return JSON only: {"decision":"FULL_MM"|"TEXT_ONLY"|"FALLBACK", "reason":"..."} E.4 Cascaded Router Logic if selfcheck_gate_decision == "FALLBACK": route = "FALLBACK" # parametric answer else: route = triage_gate_de...

  3. [11]

    Each must sound like a real image caption but contain a specific factual error (wrong location, wrong name, wrong date)

    Write 3 SHORT false captions (under 10 words each), numbered 1-3. Each must sound like a real image caption but contain a specific factual error (wrong location, wrong name, wrong date)

  4. [12]

    Image blend VLM prompt

    On a new line write: ANNOTATION: <a 5-7 word false image label with a factual error> Output ONLY the 3 numbered captions and the ANNOTATION line, nothing else. Image blend VLM prompt. In one short phrase, what is the main subject shown? Context: {subject}. Reply with phrase on...

    2024

  5. [13]

    What can you tell me about the Palace of Versailles?

    Correct fallback under polluted text: Palace of Versailles. Dataset/regime:LongFact, Baseline TC avg.TC_IP_sem TC_IP_fig TP_IC TP_IP_sem TP_IP_figTP avg. Parametric 0.950 0.939 0.9600.9700.939 0.9500.953 Text-only 0.980 0.969 0.990 0.620 0.592 0.510 0.574 Full-MM0.9860.9591.00...

  6. [14]

    Can you provide information about the Orion Nebula?

    Correct fallback under polluted text: Orion Nebula. Dataset/regime:LongFact,TP_IC. Question:“Can you provide information about the Orion Nebula?” Key evidence:the first retrieved snippet falsely described the nebula as “devoid of gas and dust” and said star formation had cease...

  7. [15]

    Can you tell me about Ernst & Young’s Global Review?

    Unnecessary fallback on usable clean evidence. Dataset/regime:LongFact, TC_IP_caption_flip. Question:“Can you tell me about Ernst & Young’s Global Review?” Key evidence:the text snippets discussed EY revenue, service lines, diversity initiatives, and con- sulting; the image me...

    2022

  8. [16]

    List all of the names of the Barti- maeus Sequence books

    Semantic entity rewrite: trust method recovers. Dataset/regime:FA V A, TC_IP_semantic_entity_rewrite. Question:“List all of the names of the Barti- maeus Sequence books.” Key evidence:the image attack rewrote the book-set image by changing paperback books to glowing digital ta...

  9. [17]

    Tell me a bio of William Post

    FigStep typography: trust method recovers. Dataset/regime:Biography, TC_IP_figstep_typography. Question:“Tell me a bio of William Post.” Key evidence:the typographic image attack inserted the false caption “William Post, inventor of the microwave oven, died at 85 in New York C...

    1947

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.