Pith. sign in

REVIEW 4 major objections 7 minor 52 references

Forcing a vision-language model to localize, read, and extract relations from a dense page in one step triggers a boundary-driven visual attention sink; decoupling those jobs with a global semantic anchor and region crops restores graph qua

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 11:45 UTC pith:G5YDV35U

load-bearing objection Real mechanism work plus a usable describe→crop→extract pipeline; the +12.5 SPR headline is softer than the abstract implies because PaperTab was never human-calibrated. the 4 major comments →

arxiv 2607.24554 v1 pith:G5YDV35U submitted 2026-07-27 cs.IR cs.CV

DeCoRAG: Cognitive Decoupling and Semantic-Aware Cropping for Complex Document Understanding

Shuo Wang , Kai Zhang , Wenyuan Huang , Yizheng Yu , Xia Liao , Junming Su , Qing Wang , Fang Xi This is my paper
classification cs.IR cs.CV
keywords multimodal RAGGraph RAGvisual attention sinkcognitive decouplingsemantic anchorregion-aware croppingcomplex document understandingvision-language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal Graph RAG systems often ask a vision-language model to find sparse evidence, interpret a page, and emit knowledge triples in a single pass. On charts, tables, and slides this coupled stage fails: attention collapses onto image boundaries (especially the top-left), relations become shallow or hallucinated, and full-page high-resolution encoding wastes tokens on empty background. The paper shows the failure is driven by layout boundaries and structural sparsity, not by any special content in the corner, and that a brief global description can redirect attention. DeCoRAG therefore rebuilds graph construction as cognitive decoupling: first write a macroscopic semantic anchor from a cheap downsampled view, then use that anchor plus simple visual density cues to crop only clusters of interest (RAP-Crop), then extract triples from the purified high-resolution patches. The resulting graphs improve end-to-end semantic pass rate by up to 12.5 points over strong baselines and cut offline prompt tokens by about 41 percent without hurting answer quality.

Core claim

When a vision-language model must simultaneously localize sparse visual evidence, interpret page semantics, and induce open-schema relations, its attention mass sinks to boundary tokens rather than information-bearing regions. That sink is boundary-driven, systematically correlates with collapsed extraction yield, and is mitigated by an explicit semantic prior. Reorganizing construction into global anchoring, dual-stream region pruning, and local high-resolution extraction therefore recovers richer graphs and cheaper encoding.

What carries the argument

Cognitive Decoupling via a Semantic Anchor plus RAP-Crop: a downsampled global description redirects attention and supplies keywords; dual-stream (morphology + OCR) proposals are greedily merged into padded clusters-of-interest that discard background before local triple extraction.

Load-bearing premise

The cheap global description written in the first phase is reliable enough to steer attention and cropping; if that anchor is badly wrong, later stages inherit the error.

What would settle it

Run the same controlled top-left mask, blank-margin, and layout-shift tests on a held-out dense-document set and check whether semantic anchoring still cuts corner over-concentration and raises valid triple yield while RAP-Crop keeps end-to-end semantic pass rate flat as prompt tokens fall ~40 percent; if anchoring fails to move attention or pass rate drops with the crops, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Multimodal Graph RAG builders should stop treating full-page coupled extraction as the default graph-construction stage.
  • A short global description can serve as a reusable control signal that redirects VLM attention away from sink regions across model families.
  • Dual-stream crop-and-prune can cut offline graph-construction prompt tokens by roughly two-fifths while preserving answer quality on high-resolution pages.
  • The same describe-then-crop pattern generalizes from sparse charts and tables to broader document VQA layouts.
  • Hybrid retrieval over the recovered graphs yields higher semantic pass rates than dense visual retrievers or prior multimodal graph pipelines on the reported benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If anchor errors cascade, the natural next control loop is a cheap verify-and-rewrite step on the global description before cropping, rather than larger end-to-end VLMs.
  • Boundary-driven sinks may appear in other high-resolution sparse canvases (schematics, UI screenshots, maps), so the same decoupling pattern is a candidate outside financial and scientific documents.
  • Token savings concentrated in offline construction suggest indexing cost, not online latency, is the first place this design pays off at industrial scale.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper argues that monolithic multimodal Graph RAG construction fails on visually dense, structurally sparse documents because coupling localization, semantic interpretation, and relation extraction in one VLM pass triggers a "Visual Attention Sink" — attention mass escaping to boundary tokens (top-left), producing shallow, hallucinated graphs. Evidence: attention visualizations on 500 SPIQA images, a sink-severity vs. extraction-yield scatter, and a 100-page paired intervention study (Table 1) showing the sink is boundary-driven rather than content-driven and that a generated global description ("Semantic Anchor") reduces COCR by 77% and TL mass by 69%. The proposed system, DeCoRAG, operationalizes "Cognitive Decoupling": Phase I generates a Semantic Anchor from a downsampled view; Phase II (RAP-Crop) fuses morphological density boxes with OCR-grounded semantic boxes to crop Clusters-of-Interest; Phase III performs anchor-guided high-resolution extraction. Reported results: Semantic Pass Rate gains over text-only RAG, ColPali/ColQwen2, and MMGraphRAG on SPIQA/SlideVQA/PaperTab (up to +12.5 points on PaperTab), DocVQA generalization, and a 40.8% reduction in offline graph-construction prompt tokens with stable end-to-end SPR.

Significance. If the central claims hold, the paper contributes both a reusable diagnosis (the Visual Attention Sink as a boundary-driven failure of coupled multimodal graph construction) and a practical system (DeCoRAG) with a credible efficiency story: a 40.8% offline prompt-token reduction on FetaTab at essentially unchanged request counts and stable SPR (Table 8). Particular strengths worth naming: (1) the paired controlled interventions with a sign test (p≈1.9e-15) rather than anecdotal attention maps; (2) the honest reporting that masking controls can amplify rather than mitigate the sink; (3) cross-architecture ablation of Phase I over six VLMs (Table 6), showing the anchor is not a Qwen-specific prompt trick; (4) a branch-level RAP-Crop ablation (Table 7) that transparently shows the visual-density stream alone matches dual-stream QA Pass, giving a recall floor when the anchor errs — which partially mitigates the cascade risk flagged in §6. Weaknesses in the evaluation protocol (uncalibrated judge on the headline benchmark, rubric weights favoring verbose systems, no released code or prompts) currently prevent full confidence in the magnitude of the reported gains, though the direction (

major comments (4)
  1. [§5.1–5.3, Tables 3–4] The headline +12.5-point claim comes from PaperTab (79.3% vs ColQwen2's 66.8%), but the evaluator calibration in Table 3 covers only SPIQA (95% agreement) and SlideVQA (86%) — PaperTab has no human-agreement evidence at all, so the benchmark carrying the largest gain is exactly the one where the judge is unvalidated. This is fixable: extend the 100-sample human calibration to PaperTab (and ideally FetaTab/DocVQA). Relatedly, the 86% SlideVQA agreement implies roughly 1-in-7 judge verdicts diverge from humans; several reported gaps (e.g., SPIQA 89.2% vs 86.8%, a 2.4-point margin) are within plausible judge noise. Please report binomial confidence intervals or per-run variance for all SPR numbers in Table 4, and disclose the judge model's identity — §5.1 says only '32B LLM-as-a-judge'; if it is the same Qwen3-VL-32B family used as the system backbone, that self-evaluation risk must be addr
  2. [§5.1, Evaluation Protocol] The SPR rubric weights Information Completeness (30%) and Expression Quality (10%) at 40% combined. DeCoRAG is structurally advantaged on these dimensions: it produces denser graphs (Table 2: RDI 70.5% vs 57.3%; Table 7: 6.2K nodes/11.2K edges dual-stream vs 2.2K/3.3K semantic-only) and richer intermediate descriptions, and LLM judges have documented verbosity/completeness biases. The SPR gain could therefore partly reflect answer style rather than factual correctness, independently of the authors' intent. A load-bearing check: report the dimension-wise judge scores per system (Fact Consistency separately from Completeness/Expression), or a fact-only pass-rate variant, to show the Table 4 margins survive under a correctness-dominated weighting.
  3. [§5.4, Table 5] The abstract and §5.4 claim generalization to conventional document VQA, but Table 5 reports DeCoRAG's DocVQA numbers (0.968 ANLS, 97.0% QA Pass) with no baseline whatsoever. Without ColQwen2, MMGraphRAG, or at minimum the raw backbone VLM on the same 100-sample split, the result cannot be attributed to cognitive decoupling rather than to the underlying 32B model, which is strong on DocVQA-style inputs out of the box. n=100 also warrants a confidence interval. Please add baseline rows to Table 5.
  4. [§3.5, Table 1] The boundary-driven (vs content-specific) causal diagnosis rests on this table, and two entries are underexplained. The control mask (masking a non-TL corner) does not merely fail to mitigate the sink — it doubles COCR from 3.49 to 6.62, and TL shift raises it to 12.57. The text glosses this as 'generic masking does not mitigate the sink,' but a twofold amplification under an off-target intervention is itself a mechanistic fact (e.g., attention renormalization over reduced visual area?) that bears on the interpretation of the anchor condition's COCR reduction (3.49→0.80). Please provide an explanation or additional controls (e.g., mask area matched to the anchor's effective information reduction) so the semantic-anchor effect can be separated from a generic 'reduce visual entropy' effect.
minor comments (7)
  1. [§3.4, Fig. 2c] Fig. 2c claims a 'negative trend' between sink density and extraction yield but reports no correlation coefficient or regression fit; please quantify (Spearman/Pearson with p-value) or soften the language.
  2. [§5.6, Threshold Robustness] The ρ sweep reports foreground/OCR/reference recalls 'at 1.00' for all three threshold values — perfect recall across the board is surprising and suggests the recall metric may be too coarse to detect degradation; please clarify the metric definition and report at least one precision-side statistic.
  3. [§3.5, Table 1] 'Structured-output validity 75%→85%' is cited as evidence of improved graph quality but the validity criterion is never defined (schema conformance? parseability?). Please define it.
  4. [Table 2] DSSR (Dense Semantic Subsumption Rate) is defined as core L3 fact retention, but Table 2 reports it only for the anchored condition ('-' for baseline), making it uninterpretable as a comparison; either give both values or explain why only one side is meaningful.
  5. [Algorithm 1] Algorithm 1, lines 9–16: after popping the largest box b and merging a single nearby b′, the merged box is appended to B_CoI without re-checking it against remaining candidates, so the result depends on pop order and does not implement full transitive merging. Either clarify that single-pass merging is intended or fix the pseudocode.
  6. [General] Presentation: 'intoGraph RAG' (§1) missing space; 'DeC-oRAG' line-break hyphenation throughout reads as a different system name; Fig. 4 contains garbled OCR-like placeholder text ('ounodone cndgh bondbo...') that should be cleaned before publication; the footnote about zeroing the bottommost attention row (§3.4) deserves to be in the main text since it affects interpretation of all attention visualizations.
  7. [General] No code, prompts, or judge rubric files are released. Given that the central metric is author-defined and the pipeline has several free parameters (ρ, padding α, SPR threshold), an artifact release would substantially strengthen reproducibility; at minimum please release the judge prompt and the SPR rubric.

Circularity Check

0 steps flagged

No significant circularity: empirical systems claims tested on external benchmarks, not forced by definitional identity or self-citation chains.

full rationale

DeCoRAG is an empirical multimodal Graph-RAG systems paper. Its load-bearing chain is (1) diagnose Visual Attention Sink via attention maps and paired interventions on SPIQA, (2) design Cognitive Decoupling + RAP-Crop, (3) measure end-to-end SPR and token cost against external baselines (ColPali, ColQwen2, MMGraphRAG) on SPIQA, SlideVQA, PaperTab, DocVQA, and FetaTab. None of these steps reduces a claimed prediction to its own fitted input or to a self-citation uniqueness theorem. Controlled interventions (Table 1) modify boundaries/content independently of the proposed fix; ablations (Tables 6–8) remove Phase I or RAP-Crop branches and report drops/gains rather than tautologies. SPR is an author-chosen LLM-as-judge protocol, which raises evaluation-validity questions, but that is not circular derivation: pass/fail is not algebraically identical to the system’s construction inputs. No self-definitional loop, fitted-as-prediction, or load-bearing same-author uniqueness import appears. Score 0 is the honest finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 4 invented entities

Load-bearing content is mostly engineering assumptions and hand-chosen routing knobs, plus named mechanisms built on prior attention-sink work. No physical constants; the claim rests on VLM behavioral regularities, heuristic crop fusion, and an LLM judge protocol.

free parameters (4)
  • RAP-Crop merge ratio ρ = 0.08
    Dynamic merge threshold τ=ρ min(w,h); default chosen by authors and lightly swept (0.06–0.10) without formal fitting to maximize SPR.
  • RAP-Crop padding ratio α = 0.1
    Expands each CoI box before crop; stated default without derivation from data.
  • SPR pass threshold = 0.5
    Binary success if weighted LLM-judge score ≥0.5; defines the headline metric.
  • LLM-as-judge rubric weights = 0.4/0.3/0.2/0.1
    Fact 40%, Completeness 30%, Logic 20%, Expression 10%—author-chosen mixture that produces SPR.
axioms (5)
  • domain assumption Standard transformer attention maps (final-layer, head-averaged, temporally aggregated) are a valid proxy for visual grounding during graph extraction.
    §3.1 builds the entire sink diagnosis on this measurement choice.
  • domain assumption Complex document pages exhibit structural visual sparsity: semantics are localized while most pixels are non-informative background.
    §3.3 motivates RAP-Crop and token pruning from this layout prior.
  • ad hoc to paper A downsampled global caption/description can inject priors that redirect attention away from boundary sinks before fine-grained extraction.
    Phase I design premise; supported empirically in-paper but not guaranteed for all VLMs/layouts (§4.2, §6).
  • domain assumption Open-schema triple extraction quality can be compared via LLM-as-judge SPR aligned with human pass/fail on sampled answers.
    §5.1–5.2 evaluation protocol; calibration reported but still an assumption for headline claims.
  • domain assumption Hybrid dense retrieval + Personalized PageRank over the built graph is an adequate backend so graph fidelity differences show up in QA.
    §4.5 system integration; end-to-end gains attributed to construction quality under this fixed retriever.
invented entities (4)
  • Visual Attention Sink (in coupled multimodal graph construction) independent evidence
    purpose: Name the failure mode where attention mass concentrates on boundary/sink tokens, yielding shallow or hallucinated graphs.
    Extends known LLM/ViT sink phenomena to this pipeline setting; evidenced by maps and interventions but is a descriptive mechanism, not an independent physical object.
  • Semantic Anchor no independent evidence
    purpose: Macroscopic global description used as cognitive prior and crop guidance.
    Pipeline-specific intermediate representation introduced in Phase I.
  • RAP-Crop (Region-Aware Pruning and Cropping) no independent evidence
    purpose: Dual-stream heuristic to form Clusters-of-Interest and drop background tokens.
    Core algorithmic artifact of the paper (Algorithm 1).
  • Cognitive Decoupling paradigm (Describe-then-Extract) no independent evidence
    purpose: Organize graph construction into grounding, routing, and local extraction stages.
    Framing device for the system; success is measured only via DeCoRAG outcomes.

pith-pipeline@v1.2.0-grok45-kimik3 · 22211 in / 3630 out tokens · 67672 ms · 2026-07-31T11:45:39.618808+00:00 · methodology

0 comments
read the original abstract

Advancing multimodal retrieval-augmented generation (RAG) for complex document understanding presents a formidable dual dilemma of accuracy and efficiency, particularly in graph RAG. Processing structurally sparse yet visually dense layouts, such as extracting a tiny data marker from a financial chart, often incurs computationally prohibitive token overhead while still triggering catastrophic hallucination. However, multimodal Graph RAG pipelines rely on graph-construction stages that assume Vision-Language Models (VLMs) can resolve sparse semantics within high-density layouts. We challenge this assumption, revealing that forcing VLMs to localize visual evidence, interpret semantics, and extract relations triggers a "Visual Attention Sink," a mechanism driving catastrophic semantic loss, while full-page processing incurs massive computational overhead. Controlled interventions verify that this failure is boundary-driven rather than content-specific and that semantic anchoring mitigates it. To fundamentally correct this flawed paradigm, we introduce DeCoRAG, a multimodal Graph RAG pipeline that shifts knowledge processing from coupled visual-semantic reasoning to "Cognitive Decoupling." Rather than passively processing raw pixels, its graph-construction stage establishes a macroscopic Semantic Anchor to neutralize the attention sink. This anchor subsequently drives our Region-Aware Pruning and Cropping (RAP-Crop) mechanism, shifting the reasoning space from dense, noisy backgrounds to purified, intent-driven semantic clusters. The resulting graph supports hybrid retrieval and answer generation. Across complex document benchmarks, DeCoRAG improves the semantic pass rate by up to 12.5 percentage points over the strongest baseline and generalizes to DocVQA. RAP-Crop reduces offline graph-construction prompt tokens by 40.8% without sacrificing end-to-end accuracy.

Figures

Figures reproduced from arXiv: 2607.24554 by Fang Xi, Junming Su, Kai Zhang, Qing Wang, Shuo Wang, Wenyuan Huang, Xia Liao, Yizheng Yu.

Figure 1
Figure 1. Figure 1: Comparison of Attention Patterns and Graph Generation. (a) Baseline: The attention is disproportionately attracted to the top-left corner (“Attention Sink”), despite scattered focus on other regions. This lack of coherent focus results in a Shallow Spatial Graph (bottom), capturing only fragmented positional relations while missing structural logic. (b) Ours: Our method effectively anchors attention to sem… view at source ↗
Figure 2
Figure 2. Figure 2: Quantitative Validation of the Attention Sink Phenomenon. (a) The aggregated attention map over 500 sampled SPIQA images reveals a persistent hotspot at the top-left patch during baseline inference. (b) Our Visual Grounding method suppresses this sink, redistributing attention towards the bottom-center regions. (c) The scatter plot illustrates an observable negative trend between sink severity (X-axis) and… view at source ↗
Figure 3
Figure 3. Figure 3: Semantic Distribution of Generated Triplets. Explicit Semantic Anchoring effectively suppresses L0 and L1 noise while significantly expanding the density of high-order logical triples. Ratio: 42.1%). In stark contrast, explicit anchoring substan￾tially suppresses this noise and boosts the Relational Depth Index (RDI)—the proportion of L2 and L3 triplets—from 57.3% to 70.5%. The Dense Semantic Subsumption R… view at source ↗
Figure 4
Figure 4. Figure 4: The Cognitive-Decoupling Graph-Construction Subsystem within DeCoRAG. This subsystem separates semantic planning, evidence routing, and local execution under a “Describe-then-Extract” paradigm. (Phase I) A VLM first generates a dense Global Description (Semantic Anchor) from a downsampled view, mitigating the attention sink. (Phase II) Guided by this semantic anchor and visual morphology, dual-stream RAP-C… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 13 linked inside Pith

  1. [1]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P . Albertet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Retrieval-augmented generation for knowledge- intensive nlp tasks,

    P . Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge- intensive nlp tasks,” inAdvances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  4. [4]

    REALM: Retrieval-augmented language model pre-training,

    K. Guu, K. Lee, Z. Tung, P . Pasupat, and M.-W. Chang, “REALM: Retrieval-augmented language model pre-training,” inInterna- tional Conference on Machine Learning (ICML), 2020

    2020

  5. [5]

    Retrieval-augmented generation for large language models: A survey,

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,”arXiv preprint arXiv:2312.10997, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. Oguz, S. Min, P . Lewis, L. Wuet al., “Dense passage retrieval for open-domain question answering,” inPro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020

    2020

  7. [7]

    Leveraging passage retrieval with gener- ative models for open domain question answering,

    G. Izacard and E. Grave, “Leveraging passage retrieval with gener- ative models for open domain question answering,” inProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021

    2021

  8. [8]

    Rag-anything: All-in-one rag framework,

    Z. Guo, X. Ren, L. Xu, J. Zhang, and C. Huang, “Rag-anything: All-in-one rag framework,”arXiv preprint arXiv:2510.12323, 2025

    arXiv 2025

  9. [9]

    From local to global: A Graph RAG approach to query-focused summarization,

    D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Tru- itt, D. Metropolitansky, R. O. Ness, and J. Larson, “From local to global: A Graph RAG approach to query-focused summarization,” arXiv preprint arXiv:2404.16130, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    G-Retriever: Retrieval-augmented generation for textual graph understanding and question answering,

    X. He, Y. Tian, Y. Sun, N. V . Chawla, T. Laurent, Y. LeCun, X. Bres- son, and B. Hooi, “G-Retriever: Retrieval-augmented generation for textual graph understanding and question answering,” in Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  11. [11]

    LayoutLMv3: Pre- training for document ai with unified text and image masking,

    Y. Huang, T. Lv, L. Cui, Y. Lu, and F. Wei, “LayoutLMv3: Pre- training for document ai with unified text and image masking,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022

    2022

  12. [12]

    Ocr-free document understanding transformer,

    G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park, “Ocr-free document understanding transformer,” inEuropean Conference on Computer Vision (ECCV), 2022. 11

    2022

  13. [13]

    Nougat: Neural optical understanding for academic documents,

    L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic, “Nougat: Neural optical understanding for academic documents,”arXiv preprint arXiv:2308.13418, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Docvqa: A dataset for vqa on document images,

    M. Mathew, D. Karatzas, and C. V . Jawahar, “Docvqa: A dataset for vqa on document images,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021

    2021

  15. [15]

    Towards vqa models that can read,

    A. Singh, V . Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  16. [16]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  17. [17]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inInternational Conference on Machine Learning (ICML), 2023

    2023

  18. [18]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P . Luc, A. Miechet al., “Flamingo: a visual language model for few-shot learning,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  19. [19]

    Instructblip: Towards general-purpose vision-language models with instruc- tion tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhaoet al., “Instructblip: Towards general-purpose vision-language models with instruc- tion tuning,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  20. [20]

    Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,

    Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, and N. Zhang, “Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,”World Wide Web, 2024

    2024

  21. [21]

    Mitigating object hallucinations in large vision-language mod- els through visual contrastive decoding,

    S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Mitigating object hallucinations in large vision-language mod- els through visual contrastive decoding,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  22. [22]

    Woodpecker: Hallucination correction for multimodal large language models,

    S. Yin, C. Fu, S. Zhao, T. Xu, H. Wang, D. Sui, Y. Shen, K. Li, X. Sun, and E. Chen, “Woodpecker: Hallucination correction for multimodal large language models,”Science China Information Sciences, 2024

    2024

  23. [23]

    Hallusionbench: An advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models,

    T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, D. Manocha, and T. Zhou, “Hallusionbench: An advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  24. [24]

    Evaluating object hallucination in large vision-language models,

    Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” inProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  25. [25]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” inInternational Conference on Learning Repre- sentations (ICLR), 2021

    2021

  26. [26]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021

    2021

  27. [27]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” in Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  28. [28]

    Vision trans- formers need registers,

    T. Darcet, M. Oquab, J. Mairal, and P . Bojanowski, “Vision trans- formers need registers,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  29. [29]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  30. [30]

    Lm-infinite: Zero-shot extreme length generalization for large language models,

    C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, and S. Wang, “Lm-infinite: Zero-shot extreme length generalization for large language models,”arXiv preprint arXiv:2308.16137, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    H 2O: Heavy- hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. R ´e, C. Barrett, Z. Wang, and B. Chen, “H 2O: Heavy- hitter oracle for efficient generative inference of large language models,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  32. [32]

    Colpali: Efficient document retrieval with vision language models,

    M. Faysse, H. Sibille, T. Wu, B. Omrani, G. Viaud, C. Hudelot, and P . Colombo, “Colpali: Efficient document retrieval with vision language models,” inInternational Conference on Learning Represen- tations (ICLR), 2025

    2025

  33. [33]

    Mmgraphrag: Bridging vision and language with interpretable multimodal knowledge graphs,

    X. Wan and H. Yu, “Mmgraphrag: Bridging vision and language with interpretable multimodal knowledge graphs,”arXiv preprint arXiv:2507.20804, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    VisRAG: Vision-based retrieval-augmented generation on multi-modality documents,

    S. Yu, C. Tang, B. Xu, J. Cui, J. Ran, Y. Yan, Z. Liu, S. Wang, X. Han, Z. Liu, and M. Sun, “VisRAG: Vision-based retrieval-augmented generation on multi-modality documents,” inInternational Confer- ence on Learning Representations (ICLR), 2025

    2025

  35. [35]

    Megarag: Multimodal knowledge graph-based retrieval aug- mented generation,

    C.-H. Hsiao, Y.-C. Wang, T.-S. Lin, Y.-R. Yeh, and C.-S. Chen, “Megarag: Multimodal knowledge graph-based retrieval aug- mented generation,”arXiv preprint arXiv:2512.20626, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Image cropping with spatial-aware feature and rank consistency,

    C. Wang, L. Niu, B. Zhang, and L. Zhang, “Image cropping with spatial-aware feature and rank consistency,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  37. [37]

    Reliable and efficient image cropping: A grid anchor based approach,

    H. Zeng, L. Li, Z. Cao, and L. Zhang, “Reliable and efficient image cropping: A grid anchor based approach,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  38. [38]

    Cropper: Vision-language model for image cropping through in-context learning,

    S. H. Lee, J. Jiang, Y. Xu, Z. Li, J. Ke, Y. Li, J. He, S. Hickson, K. Datsenko, S. Kim, M.-H. Yang, I. Essa, and F. Yang, “Cropper: Vision-language model for image cropping through in-context learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

    2025

  39. [39]

    mPLUG-DocOwl: Modularized multimodal large language model for document understanding,

    J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, Y. Dan, C. Zhao, G. Xu, C. Li, J. Tian, Q. Qi, J. Zhang, and F. Huang, “mPLUG-DocOwl: Modularized multimodal large language model for document understanding,”arXiv preprint arXiv:2307.02499, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P . Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond,”arXiv preprint arXiv:2308.12966, 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Au- toschemakg: Autonomous knowledge graph construction through dynamic schema induction from web-scale corpora,

    J. Bai, W. Fan, Q. Hu, Q. Zong, C. Li, H. T. Tsang, H. Luo, Y. Yim, H. Huang, X. Zhou, F. Qin, T. Zheng, X. Peng, X. Yao, H. Yang, L. Wu, Y. Ji, G. Zhang, R. Chen, and Y. Song, “Au- toschemakg: Autonomous knowledge graph construction through dynamic schema induction from web-scale corpora,”arXiv preprint arXiv:2505.23628, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Spiqa: A dataset for multimodal question answering on scientific papers,

    S. Pramanick, R. Chellappa, and S. Venugopalan, “Spiqa: A dataset for multimodal question answering on scientific papers,” inAd- vances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  43. [43]

    Deformable detr: Deformable transformers for end-to-end object detection,

    X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” in International Conference on Learning Representations (ICLR), 2021

    2021

  44. [44]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning,

    A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” inFindings of the Association for Computational Linguistics: ACL, 2022

    2022

  45. [45]

    Deplot: One-shot visual language reasoning by plot-to-table translation,

    F. Liu, J. M. Eisenschlos, F. Piccinno, S. Krichene, C. Pang, K. Lee, M. Joshi, W. Chen, N. Collier, and Y. Altun, “Deplot: One-shot visual language reasoning by plot-to-table translation,” inFindings of the Association for Computational Linguistics: ACL, 2023

    2023

  46. [46]

    Plotqa: Reasoning over scientific plots,

    N. Methani, P . Ganguly, M. M. Khapra, and P . Kumar, “Plotqa: Reasoning over scientific plots,” inProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2020

    2020

  47. [47]

    Slidevqa: A dataset for document visual question an- swering on multiple images,

    R. Tanaka, K. Nishida, K. Nishida, T. Hasegawa, I. Saito, and K. Saito, “Slidevqa: A dataset for document visual question an- swering on multiple images,” inProceedings of the AAAI Conference on Artificial Intelligence, 2023

    2023

  48. [48]

    UDA: A benchmark suite for retrieval augmented generation in real-world document analysis,

    Y. Hui, Y. Lu, and H. Zhang, “UDA: A benchmark suite for retrieval augmented generation in real-world document analysis,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  49. [49]

    TabFact: A large-scale dataset for table-based fact verification,

    W. Chen, H. Wang, J. Chen, Y. Zhang, H. Wang, S. Li, X. Zhou, and W. Y. Wang, “TabFact: A large-scale dataset for table-based fact verification,” inInternational Conference on Learning Representations (ICLR), 2020

    2020

  50. [50]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution,

    P . Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Yin, “Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Judging LLM-as-a-Judge with MT-Bench and Chat- bot Arena,

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P . Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chat- bot Arena,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023. 12

    2023

  52. [52]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,”arXiv preprint arXiv:2303.16634, 2023

    Pith/arXiv arXiv 2023