HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

Gyuho Shim; Heuiseok Lim; Jaehyung Seo; Jeongbae Park; Joongmin Shin

arxiv: 2605.29606 · v1 · pith:X4D5MLTQnew · submitted 2026-05-28 · 💻 cs.AI · cs.IR

HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

Joongmin Shin , Gyuho Shim , Jeongbae Park , Jaehyung Seo , Heuiseok Lim This is my paper

Pith reviewed 2026-06-29 07:37 UTC · model grok-4.3

classification 💻 cs.AI cs.IR

keywords hierarchical retrievalmultimodal RAGdocument question answeringopen-domain QAevidence retrievaldocument parsing

0 comments

The pith

HiKEY treats document hierarchy as a first-class retrieval signal to fix routing failures and evidence fragmentation in large-scale document QA.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that flat chunking or page-level approaches fail to locate the right document among thousands and to connect scattered multimodal evidence like tables and figures. HiKEY instead parses documents into a logical tree with explicit parent-child links, then uses a coarse-to-fine process: first routing at the hierarchy level to shrink the search space, then ranking sections with multimodal fusion, and finally packing a compact evidence subgraph. If this hierarchy signal works, retrieval recall rises and end-to-end QA accuracy improves on industrial benchmarks compared with page- or chunk-based baselines.

Core claim

HiKEY reconstructs documents into a logical heterogeneous graph via Document Hierarchical Parsing, performs global routing with hierarchical indexing to prune candidates, applies fine-grained multimodal retrieval to rank sections, and assembles a token-efficient evidence subgraph through hybrid structural-semantic packing, yielding up to 12.9 percent higher retrieval recall and 6.8 percent better QA performance than page- and chunk-based methods.

What carries the argument

Document Hierarchical Parsing (DHP), which builds a tree of parent-child relationships to enable hierarchical coarse-to-fine retrieval and multimodal fusion.

If this is right

Global routing at the hierarchy level reduces the number of candidates that must be examined in detail.
Multimodal fusion inside sections allows tables and figures to be scored together with surrounding text.
Hybrid packing produces shorter evidence sets that still preserve cross-section connections.
The same tree structure can be reused for both retrieval and later generation steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The method may generalize to any corpus that already contains explicit section nesting, such as technical manuals or legal filings.
If parsing errors are common, adding a lightweight human correction step at the tree-construction stage could preserve most of the gains.
The coarse-to-fine split suggests a natural way to combine fast symbolic indexes with slower neural rerankers.

Load-bearing premise

Document Hierarchical Parsing can reliably recover accurate logical parent-child relationships in varied industrial documents.

What would settle it

Run the system on a corpus of industrial documents whose logical hierarchy has been deliberately scrambled or removed and measure whether the reported recall and QA gains disappear.

Figures

Figures reproduced from arXiv: 2605.29606 by Gyuho Shim, Heuiseok Lim, Jaehyung Seo, Jeongbae Park, Joongmin Shin.

**Figure 1.** Figure 1: Overview of HiKEY, showing 1. offline hierarchy-aware graph and index construction; 2. online hierarchical coarse-to-fine retrieval (Stage-1 document routing followed by Stage-2 section scoring); 3. ancestryaware evidence subgraph assembly for an LVLM reader. methods mainly rely on text nodes and cannot use multimodal evidence (tables/figures) as first-class retrieval units. Recent work proposes multimoda… view at source ↗

**Figure 2.** Figure 2: Schematic of the three packing phases in Al [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

**Figure 3.** Figure 3: The LVLM QA prompt template, where the [Context] slot is filled with the serialized ancestry-aware evidence subgraph assembled under the token budget. visualize the final evidence subgraph serialized for the LLM Reader. G.1 System Walkthrough When a user asks: “Who were the crew members of Apollo 11, and when did they land on the Moon?”, HiKEY generates an answer via the following internal steps. Inferenc… view at source ↗

read the original abstract

Retrieval-augmented generation (RAG) for document-based Open-domain Question Answering (ODQA) on large-scale industrial corpora faces two critical bottlenecks: routing failure in locating the correct document and evidence fragmentation in integrating scattered information. Existing approaches relying on flat text chunks or page-level images inherently struggle to (i) precisely pinpoint the target document among thousands of candidates and (ii) organically connect multimodal evidence, such as tables and figures, within a limited token budget. To address these challenges, we propose HiKEY, a hierarchical tree-based multimodal retrieval framework that elevates document hierarchy to a first-class retrieval signal. Instead of simple chunking, HiKEY reconstructs a logical heterogeneous graph via Document Hierarchical Parsing (DHP), explicitly encoding parent-child relationships. Adopting a hierarchical coarse-to-fine strategy, the framework (1) performs global routing to rapidly prune the search space using hierarchical indexing, and (2) conducts fine-grained retrieval to rank sections by employing a multimodal fusion strategy that captures the most discriminative evidence. Finally, HiKEY assembles a token-efficient evidence subgraph via a hybrid structural-semantic packing strategy. Experiments on ODQA benchmarks demonstrate that HiKEY significantly outperforms page- and chunk-based baselines, improving retrieval recall by up to 12.9% and end-to-end QA performance by up to 6.8%.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HiKEY's tree-based hierarchy via DHP and coarse-to-fine routing is the clear novelty for document RAG, but the abstract leaves the experimental backing too thin to assess the reported gains.

read the letter

The main takeaway is that HiKEY builds a hierarchical multimodal retrieval system on top of document structure rather than flat chunks or pages. It parses documents into a logical tree with parent-child links using their Document Hierarchical Parsing step, then routes coarsely at the global level to cut the search space before doing fine multimodal retrieval and hybrid structural-semantic packing to keep evidence within token limits.

What is actually new is elevating the reconstructed hierarchy to a first-class signal for both routing and evidence assembly. The coarse-to-fine split plus the fusion strategy for tables and figures addresses two practical bottlenecks in large industrial corpora that standard RAG setups still hit.

The paper does a reasonable job framing those bottlenecks and outlining a coherent pipeline that tries to solve them without exploding compute. The hybrid packing idea in particular looks like a pragmatic engineering choice for token efficiency.

The soft spots sit mostly in the evaluation. The claimed lifts of 12.9% recall and 6.8% end-to-end QA rest on DHP producing reliable hierarchies, yet the abstract supplies no accuracy numbers for the parser, no failure cases on messy layouts, and no ablation that isolates the hierarchy signal. Without those, it is difficult to tell whether the gains are robust or tied to particular datasets where parsing happens to work well. Baselines and statistical details are also missing from what is visible.

This is aimed at groups working on scalable RAG over complex document collections, especially multimodal ones. A reader focused on retrieval architecture would find the structural ideas worth examining even if the numbers require verification.

I would send it to peer review. The problem is real and the framing is distinct enough that a full methods and results section could make a useful contribution once the evidential gaps are addressed.

Referee Report

2 major / 1 minor

Summary. The paper proposes HiKEY, a hierarchical tree-based multimodal retrieval framework for open-domain document QA. It reconstructs a logical heterogeneous graph via Document Hierarchical Parsing (DHP) to encode parent-child relationships, then applies a coarse-to-fine strategy with global routing to prune search space and fine-grained multimodal fusion for ranking sections, followed by hybrid structural-semantic packing. Experiments on ODQA benchmarks are claimed to show up to 12.9% retrieval recall improvement and 6.8% end-to-end QA gains over page- and chunk-based baselines.

Significance. If the central claims hold, the explicit use of document hierarchy as a first-class signal could meaningfully address routing failures and evidence fragmentation in large industrial corpora, offering a structured alternative to flat chunking. The framework is presented as a new construction evaluated against external benchmarks rather than self-referential fitting, which is a positive aspect.

major comments (2)

[Abstract] Abstract: The claims of up to 12.9% recall lift and 6.8% QA improvement are stated without any description of datasets, baselines, error bars, or statistical tests. This is load-bearing for the central claim of significant outperformance, as the abstract supplies no experimental details to support verification.
[Document Hierarchical Parsing (DHP)] Document Hierarchical Parsing (DHP) section: The strongest performance claims rest on DHP reliably reconstructing parent-child hierarchies to enable effective global routing and fine-grained retrieval. No quantitative DHP accuracy metrics, failure-mode analysis on industrial layouts (e.g., tables spanning pages), or ablation removing the hierarchy signal are provided; high parsing error rates would add noise rather than signal and could make reported gains dataset-specific artifacts.

minor comments (1)

[Abstract] The abstract could include a short statement of the specific ODQA benchmarks used to ground the performance numbers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important aspects of clarity and validation that we will address in revision. We respond point-by-point below.

read point-by-point responses

Referee: [Abstract] Abstract: The claims of up to 12.9% recall lift and 6.8% QA improvement are stated without any description of datasets, baselines, error bars, or statistical tests. This is load-bearing for the central claim of significant outperformance, as the abstract supplies no experimental details to support verification.

Authors: We agree that the abstract would benefit from additional context to make the performance claims more verifiable at a glance. In the revised manuscript, we will update the abstract to briefly reference the ODQA benchmarks used, note that comparisons are against page- and chunk-based baselines, and indicate that the gains are supported by experiments that include error bars and statistical tests (with full details provided in the experimental section). This revision will be made while preserving abstract length constraints. revision: yes
Referee: [Document Hierarchical Parsing (DHP)] Document Hierarchical Parsing (DHP) section: The strongest performance claims rest on DHP reliably reconstructing parent-child hierarchies to enable effective global routing and fine-grained retrieval. No quantitative DHP accuracy metrics, failure-mode analysis on industrial layouts (e.g., tables spanning pages), or ablation removing the hierarchy signal are provided; high parsing error rates would add noise rather than signal and could make reported gains dataset-specific artifacts.

Authors: We acknowledge that direct validation of the DHP step strengthens the claims. We will add an ablation study comparing HiKEY against a non-hierarchical variant to isolate the contribution of the parent-child signal. For quantitative DHP accuracy metrics and failure-mode analysis on layouts such as spanning tables, we will incorporate a discussion of parsing reliability (drawing on available annotations where present) and potential error propagation; if comprehensive per-layout metrics require new annotations beyond the current scope, we will explicitly note this as a limitation while emphasizing the end-to-end gains. revision: partial

Circularity Check

0 steps flagged

No circularity; new framework evaluated on external benchmarks

full rationale

The paper introduces HiKEY as a novel hierarchical multimodal retrieval system built around Document Hierarchical Parsing (DHP) to create a heterogeneous graph, followed by coarse-to-fine routing and multimodal fusion. All performance claims (recall +12.9%, QA +6.8%) are presented as empirical outcomes measured against page- and chunk-based baselines on ODQA benchmarks. No equations, fitted parameters renamed as predictions, self-citations used as load-bearing uniqueness theorems, or ansatzes smuggled via prior work appear in the provided text. The derivation chain consists of explicit construction steps whose outputs are independently tested rather than reducing to the inputs by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

The central claim rests on the effectiveness of newly introduced components such as DHP and the hierarchical strategy, with no free parameters, axioms, or invented entities quantified in the abstract.

invented entities (1)

Document Hierarchical Parsing (DHP) no independent evidence
purpose: Reconstructs logical heterogeneous graph encoding parent-child relationships
Core new component introduced to elevate document hierarchy as retrieval signal

pith-pipeline@v0.9.1-grok · 5782 in / 1109 out tokens · 40807 ms · 2026-06-29T07:37:19.595031+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages · 2 internal anchors

[1]

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

Llava-onevision-1.5: Fully open framework for democratized multimodal training.Preprint, arXiv:2509.23661. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the ACL workshop on intrinsic and extrinsic evalu- ation measures for machine translation and/or s...

work page internal anchor Pith review Pith/arXiv arXiv 2005
[2]

We build upon

Development of a liver disease-specific large language model chat interface using retrieval aug- mented generation.medRxiv. Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. 2020. Recurrent chunking mechanisms for long-text machine reading comprehension. pages 6751–6761. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V . Chawla, Thomas Laurent, Yan...

work page arXiv 2020
[3]

RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Raptor: Recursive abstractive processing for tree-organized retrieval.Preprint, arXiv:2401.18059. Mohammad Shoeybi Sheng-Chieh Lin, Chankyu Lee

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

Mm-embed: Universal multimodal retrieval with multimodal llms.Preprint, arXiv:2411.02571. Joongmin Shin, Chanjun Park, Jeongbae Park, Jaehyung Seo, and Heuiseok Lim. 2025. MultiDocFusion : Hierarchical and multimodal chunking pipeline for enhanced RAG on long industrial documents. InProceedings of the 2025 Conference on Empir- ical Methods in Natural Lang...

work page arXiv 2025
[5]

under which section does X fall?

MoC: Mixtures of text chunking learners for retrieval-augmented generation system. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5172–5189, Vienna, Austria. Associa- tion for Computational Linguistics. Xu Zhong, Elaheh ShafieiBavani, and Antonio Ji- meno Yepes. 2020. Image-base...

2020

[1] [1]

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

Llava-onevision-1.5: Fully open framework for democratized multimodal training.Preprint, arXiv:2509.23661. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the ACL workshop on intrinsic and extrinsic evalu- ation measures for machine translation and/or s...

work page internal anchor Pith review Pith/arXiv arXiv 2005

[2] [2]

We build upon

Development of a liver disease-specific large language model chat interface using retrieval aug- mented generation.medRxiv. Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. 2020. Recurrent chunking mechanisms for long-text machine reading comprehension. pages 6751–6761. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V . Chawla, Thomas Laurent, Yan...

work page arXiv 2020

[3] [3]

RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Raptor: Recursive abstractive processing for tree-organized retrieval.Preprint, arXiv:2401.18059. Mohammad Shoeybi Sheng-Chieh Lin, Chankyu Lee

work page internal anchor Pith review Pith/arXiv arXiv

[4] [4]

Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

Mm-embed: Universal multimodal retrieval with multimodal llms.Preprint, arXiv:2411.02571. Joongmin Shin, Chanjun Park, Jeongbae Park, Jaehyung Seo, and Heuiseok Lim. 2025. MultiDocFusion : Hierarchical and multimodal chunking pipeline for enhanced RAG on long industrial documents. InProceedings of the 2025 Conference on Empir- ical Methods in Natural Lang...

work page arXiv 2025

[5] [5]

under which section does X fall?

MoC: Mixtures of text chunking learners for retrieval-augmented generation system. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5172–5189, Vienna, Austria. Associa- tion for Computational Linguistics. Xu Zhong, Elaheh ShafieiBavani, and Antonio Ji- meno Yepes. 2020. Image-base...

2020