Advanced ingestion process powered by LLM parsing for RAG system

Arnau Perez; Xavier Vizcaino

REVIEW 3 major objections 6 minor 22 references

Advanced ingestion process powered by LLM parsing for RAG system

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A multi-strategy, LLM-driven ingestion pipeline—combining fast extraction, OCR, and image descriptions into node hierarchies—improves RAG answer relevancy and faithfulness.

desk verdict A clearly-described RAG ingestion pipeline with a plausible architecture, but the evaluation never compares it to anything, so the central improvement claim is unsubstantiated. read the letter →

arxiv 2412.15262 v1 pith:A6MXJY6T submitted 2024-12-16 cs.CL cs.AIcs.IR

Arnau Perez , Xavier Vizcaino This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords RetrievalAugmentedGenerationLLM-poweredOCRmultimodaldocumentparsingnode-basedchunkingcontext-awaremetadatahierarchicalstructureanswerrelevancyfaithfulness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-Augmented Generation (RAG) systems depend on how documents are ingested, and standard chunking ignores structure like images, tables, headers, and pages. This paper claims that a multi-strategy parsing pipeline removes that bottleneck: fast text extraction, a dedicated OCR service, and a multimodal large language model that reads pages as images all feed into an assembler that produces page-level markdown. The markdown is then broken into a hierarchy of Header, Text, Table, Image, Page, and Document nodes, each linked to others, summarized, and embedded in a type-appropriate way. Evaluated on academic papers, corporate slides, and a mixed corpus, the system reports higher answer relevancy and faithfulness. If the results hold, document-heavy knowledge bases would answer questions more accurately without changing the underlying language model.

What carries the argument

The load-bearing mechanism is the node-based extraction and contextualization process. A Multimodal Assembler Agent merges the page snapshot, described images, and text from the FAST, OCR, and LLM strategies into a page-level markdown file; a Metadata Extractor Agent adds topic, keywords, and summary; and the processing phase splits the markdown into Header, Text, Table, Image, Page, and Document nodes connected by next, previous, parent, and child links, with only Header nodes allowed to have children. Question Generator and Summary Generator agents produce context-aware summaries for Table and Header nodes, and each node type gets its own embedding strategy, turning a flat chunk into a structured object that carries its own context and place in the document hierarchy.

What would settle it

Run a held-out set of fifty documents through two ingestion pipelines—this node-based one and a standard recursive chunker—with the same retriever, answer generator, and judge, and compute score distributions over the coarse 0-to-1 grid. If the node-based pipeline's answer-relevancy and faithfulness margins do not exceed the judge's run-to-run variance on repeated scoring, the claimed improvement is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that replacing generic chunking with a node-based, hierarchy-aware ingestion process makes retrieved context more usable, so downstream RAG answers are more relevant and more faithful to the source. The pipeline merges page snapshots, described images, and text from three parse strategies into a synthesized markdown file, then extracts nodes with relationships and context-aware summaries, and embeds each node type differently: text directly, images through description, tables through contextualized description, and headers, pages, and documents through summaries. The authors support this with evaluations across three knowledge bases—five arXiv papers, more than ten corporate documents, and a mixed set—and report that the system excels at Answer Relevancy and Faithfulness while retrieving a larger amount of context. They also observe that over 80% of retrieved nodes are Page and Header nodes, and that contextual precision is lower in the mixed-topic base, which they attribute to relevant context appearing deeper in the ranking.

Load-bearing premise

The load-bearing premise is that the evaluation scores are meaningful: a single large language model judges answers and contexts on a coarse 0-to-1 grid, while the questions and ground truths were produced by another large language model without human validation, and no standard ingestion baseline was scored under identical conditions.

Editorial extensions

If this is right

Scanned PDFs and image-heavy slides become first-class RAG inputs: when fast extraction finds no text, the OCR and LLM strategies supply exact words while the LLM recovers reading order.
Retrieval can be steered by node type, because Page and Header nodes dominate retrieval (over 80% of retrieved nodes) and carry summaries; numeric queries could target Table nodes specifically.
Mixed-topic knowledge bases will likely need a reranker, since the paper reports lower contextual precision there because the first relevant context sits deeper in the retrieved list.
Answer relevancy and faithfulness improve at the cost of retrieving more context, so downstream latency and prompt length grow unless a node filter or reranker is added.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the decisive test the authors do not run is a head-to-head against a standard recursive or semantic chunker using the same retriever, generator, and judge; without that baseline the reported scores measure a pipeline, not an advantage.
Beyond the paper: the node hierarchy suggests a cheap extension—filtering retrieval by node type per query, such as Table nodes for numeric questions or Header nodes for section-level questions—which would likely raise contextual precision in mixed corpora.
Beyond the paper: because each node's embedding is chosen by type, the same question-generation and summary agents could be reused as a synthetic evaluation harness for any new corpus, making ingestion improvements measurable without hand-labeled data.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is an engineering report about a RAG ingestion pipeline that combines three parsing strategies (fast text extraction, LLM OCR, AWS Textract), assembles per-page markdown, and builds a node hierarchy (Header/Text/Table/Image/Page/Document) with summary-based embeddings. The architecture is coherent and the writing is unusually honest: Section IV.B openly says contextual relevancy is low and that most retrieved nodes are not relevant. That honesty cuts against the abstract, which claims improvements in answer relevancy and information faithfulness without ever running a baseline condition.

What's new: I haven't seen the exact integration of three parsing strategies plus hierarchical nodes plus summary embeddings in the cited related work. As a system description it is a reasonable engineering contribution, and the appendix showing markdown output from Euro NCAP reports and two-column papers gives a useful qualitative sense of what the assembler produces. The authors also correctly identify that OCR flattens two-column layouts and that the LLM assembler can restore logical reading order.

Where it falls down: the evaluation. Three knowledge bases, questions and ground truths generated by Claude Haiku 3.5, answers by Claude Sonnet 3.5, scores by Llama 3.1 405B on a coarse 0.2 grid. There is no control condition: no recursive, semantic, or hierarchical chunking baseline run through the same retriever, embedder, and answer generator. Absolute scores cannot support a comparative claim. The free parameter K=5 is never swept. No variance estimates, no human validation, no per-query breakdowns. Section IV.B concedes contextual relevancy is low, so the conclusion that the system 'excels' rests entirely on answer relevancy and faithfulness numbers that are not measured against anything. The paper itself flags that most retrieved nodes are Page and Header summaries containing largely irrelevant information, which is a real signal that retrieval is not working as well as the headline suggests.

The metric definitions are also sloppy: the same symbol mCR is used for contextual relevancy and contextual recall, and the equations mix ra→q, ra→c, re→c without explaining who scores whom. Minor, but it should be cleaned up.

Verdict: this deserves a serious referee, not because the evaluation is acceptable, but because the underlying system is plausible and the authors show honest reporting. A revision with an A/B test against standard chunking, human-judged spot checks, and released artifacts would make this a useful reference for RAG practitioners. As is, I would not cite it as evidence of improvement; I'd cite it, if at all, as an example of a multi-strategy ingestion architecture.

Referee Report

3 major / 6 minor

Summary. The manuscript proposes an ingestion architecture for RAG systems in which PDF/DOCX/PPTX files are parsed by three strategies (FAST, OCR, and a multimodal LLM), assembled into page-level markdown by a Multimodal Assembler Agent, and then decomposed into typed nodes (Header, Text, Table, Image, Page, Document, Q&A). Tables and headers receive LLM-generated questions and summaries, and each node type is embedded in a different way. The evaluation constructs three knowledge bases (arXiv papers, corporate slides, and mixed files), uses Claude Haiku 3.5 to generate questions/expected answers/ground truths, Claude Sonnet 3.5 v2 to answer, and Llama 3.1 405B to score answer relevancy, faithfulness, contextual relevancy, precision, and recall. The abstract claims that the approach demonstrates improvements in answer relevancy and information faithfulness.

Significance. The engineering pipeline is clearly described and includes several sensible components: fusing FAST/OCR/LLM extraction, using an assembler agent, typed nodes with hierarchical links, and separate embedding strategies per node type. Using a different LLM for judgment than for answer generation is a reasonable precaution. However, the empirical section does not deliver the comparative evidence promised by the abstract. Without a baseline ingestion condition, human-validated ground truth, or variance estimates, the reported absolute LLM-judged scores cannot establish improvement. The paper's own observation that contextual relevancy is low in all test cases further weakens the claim of enhanced retrieval. The contribution therefore remains an architecture proposal of potential practical interest, not a validated improvement.

major comments (3)

[Section IV, Figure 3] The central claim of 'improvements in answer relevancy and information faithfulness' (Abstract) is comparative, but Section IV reports only absolute scores for the proposed system. There is no condition in which the same question set, retriever, embedding model, and answer generator are run on output from the standard chunking alternatives named in Section I (recursive, semantic, hierarchical). Without that control, the reported scores cannot support 'improvements'; the headline rests on an implicit comparison that is never performed. Please add a controlled baseline and report per-query paired differences.
[Section IV, Equations (1)-(5)] All headline metrics are computed as LLM judgments by Llama 3.1 405B on a coarse 0-1 grid, while the questions, expected answers, and ground truths were generated by Claude Haiku 3.5. There is no human validation, no inter-annotator agreement, and no variance estimate. Because Eqs. (1)-(5) all depend on these uncalibrated judgments, the numerical values in Figure 3 cannot be interpreted as reliable estimates of the claimed properties. Report N, score distributions, a random human-audited subset, and ideally a calibrated benchmark.
[Section IV.B] The paper reports that 'the context relevancy is quite low in all test cases' and that over 80% of retrieved nodes were Page and Header nodes. Since faithfulness is judged against retrieved context, low contextual relevancy does not support the Abstract's claim that the system 'enhances document comprehension and retrieval capabilities.' At minimum, explain the relation between the low contextual relevancy and the high answer relevancy/faithfulness claims, and analyze why low-level nodes are rarely retrieved if the hierarchy is meant to improve retrieval.

minor comments (6)

[Section III.C] The sentence 'This metric can be using the following metric' is ungrammatical; it should be 'can be computed using the following metric.'
[Section III.E] Equation (5) uses the symbol mCR, which is already used for Contextual Relevancy in Eq. (3); use a distinct symbol such as mCRec or mCL to avoid ambiguity.
[Section IV.C] The in-text claim that contextual precision is 'remarkably high' for corporate documentation is not supported by a numeric table; Figure 3 should be accompanied by the exact values and per-query distributions.
[Section II.A] The acronym 'F AST' appears with a stray space in several places; define it on first use and render it consistently as FAST.
[Section IV.A] The qualitative observations about parsing strategies are not quantified; if they are intended as evidence, provide error measurements on a sample of pages.
[Section IV.C and References] Reference [13] appears as '[ ? ]' in the text; complete the citation to the Cohere rerank model documentation.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the proposed ingestion pipeline is described independently of the evaluation scores, and the only self-citation is a non-load-bearing illustrative reference.

full rationale

Walking the paper's derivation chain, the claimed contribution is the multi-strategy LLM-powered parsing and node-based ingestion architecture described in Section II (parsing, assembling, metadata extraction, node types, contextualization, embedding choices). These design choices are stated directly and are not derived from, nor fitted to, the evaluation metrics in Section IV. The abstract's comparative wording ('improvements in answer relevancy and information faithfulness') is an empirical claim, but the evaluation section reports only absolute LLM-judged scores on three knowledge bases and does not include a baseline ingestion condition; this is an evidentiary gap about whether improvement was demonstrated, not a circular reduction in which the predicted quantity is defined in terms of the inputs or fitted parameters are renamed as predictions. The evaluation does rely on LLM-generated questions, expected answers, and ground truths (Claude Haiku 3.5) and an LLM judge (Llama 3.1 405B) assigning scores on a coarse 0–1 grid; that is a measurement-validity concern (no human validation, no variance estimates), not a self-definitional circularity, because the scores are not used to define or fit the ingestion method. The only self-referential element is reference [16], a prior paper coauthored by the advisor, used solely as an illustrative source page in Figure 2; it carries no argumentative weight. Notably, Section IV.B and IV.C report low contextual relevancy and lower mixed-topic precision, so the paper does not conceal negative results. No equation reduces to another by construction, and no load-bearing premise rests on a self-citation chain. Consequently, the appropriate finding is no significant circularity, with a low score reflecting only the minor non-load-bearing self-citation and an evaluation design that raises external-validity questions rather than circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim depends on LLM-generated evaluation data and on the unstated merge quality of the assembler; the only hand-set parameter reported is the retrieval limit K=5. No fitted parameters appear.

free parameters (1)

maximum retrieved nodes K = 5
Section IV sets the retrieval limit to 5 nodes per query for all knowledge bases; the central metrics (contextual precision, recall) depend on this choice, with no sensitivity analysis.

assumptions (3)

domain assumption The LLM evaluator (Llama 3.1 405B) assigns valid relevance and faithfulness scores on the 0-1 grid.
Section IV states metrics are scored by an LLM according to guidelines without human validation or inter-annotator agreement.
domain assumption The Claude Haiku 3.5 generated questions, expected answers, and ground truths are correct for each document.
Section IV: 'an LLM was used to generate a number of questions, expected answers, and ground truths' with no quality check.
domain assumption The Multimodal Assembler Agent can merge FAST, OCR, and LLM extractions into a markdown representation without losing content needed for retrieval.
Section II.A describes assembly as an implicit reliable operation; no ablation verifies the merge.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Advanced ingestion process powered by LLM parsing for RAG system." pith.science (2026). https://pith.science/paper/A6MXJY6T

@misc{pith2026241215262,
  author       = {Pith},
  title        = {Pith review of: Advanced ingestion process powered by LLM parsing for RAG system},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A6MXJY6T}},
  note         = {Machine review of arXiv:2412.15262}
}

read the original abstract

Retrieval Augmented Generation (RAG) systems struggle with processing multimodal documents of varying structural complexity. This paper introduces a novel multi-strategy parsing approach using LLM-powered OCR to extract content from diverse document types, including presentations and high text density files both scanned or not. The methodology employs a node-based extraction technique that creates relationships between different information types and generates context-aware metadata. By implementing a Multimodal Assembler Agent and a flexible embedding strategy, the system enhances document comprehension and retrieval capabilities. Experimental evaluations across multiple knowledge bases demonstrate the approach's effectiveness, showing improvements in answer relevancy and information faithfulness.

Figures

Figures reproduced from arXiv: 2412.15262 by the authors.

**Figure 1.** FIG. 1. Preprocessing pipeline for document ingestion in RAG system. The flowchart illustrates the parsing and assembling [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** FIG. 2. The illustration shows the assembling of the page 2 of the paper [16]. There are 5 kind of nodes represented: Header, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** FIG. 3. Comparative Analysis of Knowledge Base Performance Across Different Metrics. The figure illustrates the percentage [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 17 canonical work pages

[1]

F AST: Utilizes Python libraries to extract text and images from each document page

work page
[2]

It describes images without text or extracts text from images

LLM : Employs a multimodal LLM for OCR task, extracting text, table content, and image informa- tion. It describes images without text or extracts text from images. In this case it was used Sonnet 3.5 v2 model from Anthropic Claude family [5]

work page
[3]

Advanced ingestion process powered by LLM parsing for RAG system

OCR: Leverages external and dedicated machine learning models for OCR task. In this case it was used the A WS Textract service [6]. Before assembly, images are analyzed and described based on content type. For plots, the system extracts axis values, legends, labels, and provides a visualization description. Flowcharts are described in terms of process rel...

work page Pith review arXiv 2024
[4]

Gemini 2.000.000 context window model, https://blog.google/technology/developers/gemini-gemma-developer-updates- may-2024/, (December 12th, 2024)

work page 2024
[5]

Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023a

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023a

arXiv
[6]

Anthropic prompt caching, https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching (December 9th, 2024)

work page 2024
[7]

Renyi Qu, Ruixuan Tu, Forrest Bao Is Semantic Chunking Worth the Computational Cost?, arXiv:2410.13070

arXiv
[8]

Anthropic models, https://docs.anthropic.com/en/docs/about-claude/models, (November 4th, 2024)

work page 2024

Show all 22 references

[9]

A WS Textract service, https://aws.amazon.com/textract/, (December 12th, 2024)

2024
[10]

Llamaindex splitters, https://docs.llamaindex.ai/en/stable/api reference/node parsers/(December 9th, 2024)

2024
[11]

Anthropic tokens equivalency, https://docs.anthropic.com/en/docs/resources/glossary, (November 4th, 2024)

2024
[12]

Gemini models, https://ai.google.dev/gemini-api/docs/models/gemini#token-size, (November 4th, 2024)

2024
[13]

(December 2th, 2024)

Pinecone Metadata, https://docs.pinecone.io/guides/data/understanding-metadata. (December 2th, 2024)

2024
[14]

(December 2th, 2024)

VoyageAI, https://docs.voyageai.com/docs/embeddings. (December 2th, 2024)

2024
[15]

(December 2th, 2024)

Cohere embedding models, https://docs.cohere.com/v2/docs/cohere-embed. (December 2th, 2024)

2024
[16]

(December 2th, 2024)

Cohere reranking models, https://docs.cohere.com/v2/docs/rerank-2. (December 2th, 2024)

2024
[17]

Anthropic contextual retrieval, https://www.anthropic.com/news/contextual-retrieval (September 26th, 2024)

2024
[18]

Pinecone chunking strategies, https://www.pinecone.io/learn/chunking-strategies/ (July 17th, 2024)

2024
[19]

Diego Martn, Jordi Sanchez and Xavier Vizcano, ”Human Interaction Classifier for llm based chatbot.” arXiv, 2024, arXiv:2407.21647

2024 arXiv
[20]

In- context retrieval-augmented language models.arXiv preprint arXiv:2302.00083, 2023

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In- context retrieval-augmented language models.arXiv preprint arXiv:2302.00083, 2023

2023 arXiv
[21]

Ishneet Sukhvinder Singh*, Ritvik Aggarwal*, Ibrahim Allahverdiyev, Muhammad Taha, Aslihan Akalin, Kevin Zhu, Sean OBrien, ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systems, arXiv:2410.19572

arXiv
[22]

APPENDIX This appendix presents a curated selection of markdown files generated during the pre-processing stage of the system

Zijie Zhong, Hanwen Liu, Xiaoya Cui, Xiaofan Zhang, Zengchang Qin, Mix-of-Granularity: Optimize the Chunking Gran- ularity for Retrieval-Augmented Generation, arxiv: arXiv:2406.00456 6 Barcelona, November 2024 Advanced ingestion process powered by LLM parsing for RAG system Ar...

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

F AST: Utilizes Python libraries to extract text and images from each document page

work page

[2] [2]

It describes images without text or extracts text from images

LLM : Employs a multimodal LLM for OCR task, extracting text, table content, and image informa- tion. It describes images without text or extracts text from images. In this case it was used Sonnet 3.5 v2 model from Anthropic Claude family [5]

work page

[3] [3]

Advanced ingestion process powered by LLM parsing for RAG system

OCR: Leverages external and dedicated machine learning models for OCR task. In this case it was used the A WS Textract service [6]. Before assembly, images are analyzed and described based on content type. For plots, the system extracts axis values, legends, labels, and provides a visualization description. Flowcharts are described in terms of process rel...

work page Pith review arXiv 2024

[4] [4]

Gemini 2.000.000 context window model, https://blog.google/technology/developers/gemini-gemma-developer-updates- may-2024/, (December 12th, 2024)

work page 2024

[5] [5]

Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023a

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023a

arXiv

[6] [6]

Anthropic prompt caching, https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching (December 9th, 2024)

work page 2024

[7] [7]

Renyi Qu, Ruixuan Tu, Forrest Bao Is Semantic Chunking Worth the Computational Cost?, arXiv:2410.13070

arXiv

[8] [8]

Anthropic models, https://docs.anthropic.com/en/docs/about-claude/models, (November 4th, 2024)

work page 2024