{"id":"dffeb145-c538-4dff-8b2c-b0b1d1c56807","arxiv_id":"2412.15262","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-strategy LLM-based document parsing pipeline for RAG is described, but the evaluation lacks a baseline comparison, so the claimed improvements are unsubstantiated.","lead":"This paper describes a RAG ingestion pipeline that parses documents with a mix of Python libraries, an LLM OCR step, and AWS Textract, then organizes the content into hierarchical nodes with summaries and metadata. The claimed improvement in answer relevancy and faithfulness is not backed by any comparison against a standard baseline in the reported experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The results in Section IV report absolute LLM-judged scores on three knowledge bases but include no baseline ingestion condition, so the abstract's claim that the approach shows 'improvements in answer relevancy and information faithfulness' is not supported by the presented evidence.","rationale":"The paper is a plausible engineering report, and the reader's rejection is appropriate. My stress-test identifies the same weakest assumption: the evaluation in Section IV contains no baseline ingestion condition, so the abstract's comparative claim of 'improvements' is asserted rather than demonstrated. The LLM-judge scoring on a coarse 0.2 grid without human validation or variance estimates is a secondary concern, but the missing baseline is the load-bearing issue because it makes the headline claim undefined. The paper's own observation that Contextual Relevancy is low in all test cases underscores that the positive conclusion rests on an implicit comparison that never appears in the text. A concrete A/B test against standard chunking, with the same retriever and answer generator and with human validation of a subsample, would settle whether the proposed pipeline actually improves answer relevancy and faithfulness. Until such a comparison exists, REJECT remains the correct verdict.","tokens_in":6331,"tokens_out":3261,"duration_ms":29463,"concrete_test":"Run a controlled A/B evaluation on the same three knowledge bases: condition A uses the proposed node-based ingestion; condition B uses a standard chunker, e.g., LlamaIndex's recursive splitter with a comparable chunk size, keeping the same embedding model, vector store, retriever (top-5 nodes), answer generator, and question set. Score both conditions with Llama 3.1 405B using the paper's metric prompts, and also have blind human annotators rate a random subset of at least 50 answers per condition on relevancy and faithfulness. Report per-knowledge-base means, standard deviations, and 95% confidence intervals for Answer Relevancy, Faithfulness, and Contextual Relevancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is comparative: the proposed multi-strategy LLM-parsing ingestion 'demonstrate[s] ... improvements in answer relevancy and information faithfulness' over existing ingestion. The evaluation in Section IV measures only the proposed system. Three knowledge bases are built (5 arXiv papers, 10+ corporate files, up to 10 mixed files); an LLM generates questions, expected answers, and ground truths; retrieval returns up to 5 nodes of any type; answers are generated by Claude Sonnet 3.5 v2; and Llama 3.1 405B judges Answer Relevancy, Faithfulness, Contextual Relevancy, Precision, and Recall on a 0-1 grid in steps of 0.2. There is no condition in which the same question set, retriever, embedder, and answer generator are run on output from standard chunking (recursive, semantic, or hierarchical, the methods named in Section I as the alternatives). Without that control, absolute scores cannot establish improvement: the system might have high answer relevancy and faithfulness relative to nothing, and the paper's own Section IV.B reports low Contextual Relevancy, so the positive headline depends entirely on an implicit comparison that is never performed. The LLM-judge design also lacks human validation, variance estimates, and per-query breakdowns; a judge from a different family than the answer generator is a reasonable precaution but does not calibrate the scores. Thus the load-bearing assumption is not just that the scores are meaningful but that they were measured against the appropriate baseline, which the paper never states.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes an ingestion architecture for RAG systems in which PDF/DOCX/PPTX files are parsed by three strategies (FAST, OCR, and a multimodal LLM), assembled into page-level markdown by a Multimodal Assembler Agent, and then decomposed into typed nodes (Header, Text, Table, Image, Page, Document, Q&A). Tables and headers receive LLM-generated questions and summaries, and each node type is embedded in a different way. The evaluation constructs three knowledge bases (arXiv papers, corporate slides, and mixed files), uses Claude Haiku 3.5 to generate questions/expected answers/ground truths, Claude Sonnet 3.5 v2 to answer, and Llama 3.1 405B to score answer relevancy, faithfulness, contextual relevancy, precision, and recall. The abstract claims that the approach demonstrates improvements in answer relevancy and information faithfulness.","tokens_in":6609,"tokens_out":4839,"duration_ms":42855,"significance":"The engineering pipeline is clearly described and includes several sensible components: fusing FAST/OCR/LLM extraction, using an assembler agent, typed nodes with hierarchical links, and separate embedding strategies per node type. Using a different LLM for judgment than for answer generation is a reasonable precaution. However, the empirical section does not deliver the comparative evidence promised by the abstract. Without a baseline ingestion condition, human-validated ground truth, or variance estimates, the reported absolute LLM-judged scores cannot establish improvement. The paper's own observation that contextual relevancy is low in all test cases further weakens the claim of enhanced retrieval. The contribution therefore remains an architecture proposal of potential practical interest, not a validated improvement.","major_comments":[{"comment":"The central claim of 'improvements in answer relevancy and information faithfulness' (Abstract) is comparative, but Section IV reports only absolute scores for the proposed system. There is no condition in which the same question set, retriever, embedding model, and answer generator are run on output from the standard chunking alternatives named in Section I (recursive, semantic, hierarchical). Without that control, the reported scores cannot support 'improvements'; the headline rests on an implicit comparison that is never performed. Please add a controlled baseline and report per-query paired differences.","section":"Section IV, Figure 3"},{"comment":"All headline metrics are computed as LLM judgments by Llama 3.1 405B on a coarse 0-1 grid, while the questions, expected answers, and ground truths were generated by Claude Haiku 3.5. There is no human validation, no inter-annotator agreement, and no variance estimate. Because Eqs. (1)-(5) all depend on these uncalibrated judgments, the numerical values in Figure 3 cannot be interpreted as reliable estimates of the claimed properties. Report N, score distributions, a random human-audited subset, and ideally a calibrated benchmark.","section":"Section IV, Equations (1)-(5)"},{"comment":"The paper reports that 'the context relevancy is quite low in all test cases' and that over 80% of retrieved nodes were Page and Header nodes. Since faithfulness is judged against retrieved context, low contextual relevancy does not support the Abstract's claim that the system 'enhances document comprehension and retrieval capabilities.' At minimum, explain the relation between the low contextual relevancy and the high answer relevancy/faithfulness claims, and analyze why low-level nodes are rarely retrieved if the hierarchy is meant to improve retrieval.","section":"Section IV.B"}],"minor_comments":[{"comment":"The sentence 'This metric can be using the following metric' is ungrammatical; it should be 'can be computed using the following metric.'","section":"Section III.C"},{"comment":"Equation (5) uses the symbol mCR, which is already used for Contextual Relevancy in Eq. (3); use a distinct symbol such as mCRec or mCL to avoid ambiguity.","section":"Section III.E"},{"comment":"The in-text claim that contextual precision is 'remarkably high' for corporate documentation is not supported by a numeric table; Figure 3 should be accompanied by the exact values and per-query distributions.","section":"Section IV.C"},{"comment":"The acronym 'F AST' appears with a stray space in several places; define it on first use and render it consistently as FAST.","section":"Section II.A"},{"comment":"The qualitative observations about parsing strategies are not quantified; if they are intended as evidence, provide error measurements on a sample of pages.","section":"Section IV.A"},{"comment":"Reference [13] appears as '[ ? ]' in the text; complete the citation to the Cohere rerank model documentation.","section":"Section IV.C and References"}],"recommendation":"reject","confidential_remarks":"The system description is coherent and the paper may be salvageable with a rigorous controlled evaluation, but the current evidence does not support the advertised comparative claim. The absence of a baseline, the use of LLM-generated questions and judgments without human validation, and the reported low contextual relevancy are load-bearing deficiencies that cannot be fixed by modest edits. I would not consider this for publication in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an engineering report about a RAG ingestion pipeline that combines three parsing strategies (fast text extraction, LLM OCR, AWS Textract), assembles per-page markdown, and builds a node hierarchy (Header/Text/Table/Image/Page/Document) with summary-based embeddings. The architecture is coherent and the writing is unusually honest: Section IV.B openly says contextual relevancy is low and that most retrieved nodes are not relevant. That honesty cuts against the abstract, which claims improvements in answer relevancy and information faithfulness without ever running a baseline condition.\n\nWhat's new: I haven't seen the exact integration of three parsing strategies plus hierarchical nodes plus summary embeddings in the cited related work. As a system description it is a reasonable engineering contribution, and the appendix showing markdown output from Euro NCAP reports and two-column papers gives a useful qualitative sense of what the assembler produces. The authors also correctly identify that OCR flattens two-column layouts and that the LLM assembler can restore logical reading order.\n\nWhere it falls down: the evaluation. Three knowledge bases, questions and ground truths generated by Claude Haiku 3.5, answers by Claude Sonnet 3.5, scores by Llama 3.1 405B on a coarse 0.2 grid. There is no control condition: no recursive, semantic, or hierarchical chunking baseline run through the same retriever, embedder, and answer generator. Absolute scores cannot support a comparative claim. The free parameter K=5 is never swept. No variance estimates, no human validation, no per-query breakdowns. Section IV.B concedes contextual relevancy is low, so the conclusion that the system 'excels' rests entirely on answer relevancy and faithfulness numbers that are not measured against anything. The paper itself flags that most retrieved nodes are Page and Header summaries containing largely irrelevant information, which is a real signal that retrieval is not working as well as the headline suggests.\n\nThe metric definitions are also sloppy: the same symbol mCR is used for contextual relevancy and contextual recall, and the equations mix ra→q, ra→c, re→c without explaining who scores whom. Minor, but it should be cleaned up.\n\nVerdict: this deserves a serious referee, not because the evaluation is acceptable, but because the underlying system is plausible and the authors show honest reporting. A revision with an A/B test against standard chunking, human-judged spot checks, and released artifacts would make this a useful reference for RAG practitioners. As is, I would not cite it as evidence of improvement; I'd cite it, if at all, as an example of a multi-strategy ingestion architecture.","headline":"A clearly-described RAG ingestion pipeline with a plausible architecture, but the evaluation never compares it to anything, so the central improvement claim is unsubstantiated.","tokens_in":7135,"tokens_out":1971,"would_cite":false,"duration_ms":18259,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multi-strategy, LLM-driven ingestion pipeline—combining fast extraction, OCR, and image descriptions into node hierarchies—improves RAG answer relevancy and faithfulness.","keywords":["Retrieval Augmented Generation","LLM-powered OCR","multimodal document parsing","node-based chunking","context-aware metadata","hierarchical document structure","answer relevancy","faithfulness"],"falsifier":"Run a held-out set of fifty documents through two ingestion pipelines—this node-based one and a standard recursive chunker—with the same retriever, answer generator, and judge, and compute score distributions over the coarse 0-to-1 grid. If the node-based pipeline's answer-relevancy and faithfulness margins do not exceed the judge's run-to-run variance on repeated scoring, the claimed improvement is not established.","tokens_in":6120,"feed_emoji":"📄","tokens_out":7337,"duration_ms":63891,"temperature":0.7,"pith_summary":"Retrieval-Augmented Generation (RAG) systems depend on how documents are ingested, and standard chunking ignores structure like images, tables, headers, and pages. This paper claims that a multi-strategy parsing pipeline removes that bottleneck: fast text extraction, a dedicated OCR service, and a multimodal large language model that reads pages as images all feed into an assembler that produces page-level markdown. The markdown is then broken into a hierarchy of Header, Text, Table, Image, Page, and Document nodes, each linked to others, summarized, and embedded in a type-appropriate way. Evaluated on academic papers, corporate slides, and a mixed corpus, the system reports higher answer relevancy and faithfulness. If the results hold, document-heavy knowledge bases would answer questions more accurately without changing the underlying language model.","feed_headline":"LLM-OCR document parsing lifts RAG answer quality","feed_subtitle":"Merging fast text extraction, OCR, and image descriptions into node hierarchies makes answers more relevant and faithful","key_machinery":"The load-bearing mechanism is the node-based extraction and contextualization process. A Multimodal Assembler Agent merges the page snapshot, described images, and text from the FAST, OCR, and LLM strategies into a page-level markdown file; a Metadata Extractor Agent adds topic, keywords, and summary; and the processing phase splits the markdown into Header, Text, Table, Image, Page, and Document nodes connected by next, previous, parent, and child links, with only Header nodes allowed to have children. Question Generator and Summary Generator agents produce context-aware summaries for Table and Header nodes, and each node type gets its own embedding strategy, turning a flat chunk into a structured object that carries its own context and place in the document hierarchy.","core_discovery":"On the paper's own terms, the central claim is that replacing generic chunking with a node-based, hierarchy-aware ingestion process makes retrieved context more usable, so downstream RAG answers are more relevant and more faithful to the source. The pipeline merges page snapshots, described images, and text from three parse strategies into a synthesized markdown file, then extracts nodes with relationships and context-aware summaries, and embeds each node type differently: text directly, images through description, tables through contextualized description, and headers, pages, and documents through summaries. The authors support this with evaluations across three knowledge bases—five arXiv papers, more than ten corporate documents, and a mixed set—and report that the system excels at Answer Relevancy and Faithfulness while retrieving a larger amount of context. They also observe that over 80% of retrieved nodes are Page and Header nodes, and that contextual precision is lower in the mixed-topic base, which they attribute to relevant context appearing deeper in the ranking.","pith_inferences":["Beyond the paper: the decisive test the authors do not run is a head-to-head against a standard recursive or semantic chunker using the same retriever, generator, and judge; without that baseline the reported scores measure a pipeline, not an advantage.","Beyond the paper: the node hierarchy suggests a cheap extension—filtering retrieval by node type per query, such as Table nodes for numeric questions or Header nodes for section-level questions—which would likely raise contextual precision in mixed corpora.","Beyond the paper: because each node's embedding is chosen by type, the same question-generation and summary agents could be reused as a synthetic evaluation harness for any new corpus, making ingestion improvements measurable without hand-labeled data."],"forward_implications":["Scanned PDFs and image-heavy slides become first-class RAG inputs: when fast extraction finds no text, the OCR and LLM strategies supply exact words while the LLM recovers reading order.","Retrieval can be steered by node type, because Page and Header nodes dominate retrieval (over 80% of retrieved nodes) and carry summaries; numeric queries could target Table nodes specifically.","Mixed-topic knowledge bases will likely need a reranker, since the paper reports lower contextual precision there because the first relevant context sits deeper in the retrieved list.","Answer relevancy and faithfulness improve at the cost of retrieving more context, so downstream latency and prompt length grow unless a node filter or reranker is added."],"supporting_citations":[{"why":"Supplies the multimodal LLM used for OCR parsing and for answer generation during evaluation.","marker":"[5]"},{"why":"Supplies the dedicated OCR service that extracts verbatim text from scanned images in the parsing stage.","marker":"[6]"},{"why":"Frames the problem by arguing that standard chunking strategies introduce computational challenges and ignore document structure, motivating the node-based ingestion approach.","marker":"[4]"}],"fun_headline_variants":["Node-based ingestion lifts RAG answer relevance","LLM-OCR node parsing makes RAG answers more faithful","Multimodal node extraction improves RAG answer relevancy","Hierarchical nodes from LLM OCR boost RAG retrieval","Context-aware node embedding boosts RAG answer quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation scores are meaningful: a single large language model judges answers and contexts on a coarse 0-to-1 grid, while the questions and ground truths were produced by another large language model without human validation, and no standard ingestion baseline was scored under identical conditions.","fun_headline_variants_meta":{"raw":{"variants":["Node-based ingestion lifts RAG answer relevance","LLM-OCR node parsing makes RAG answers more faithful","Multimodal node extraction improves RAG answer relevancy","Hierarchical nodes from LLM OCR boost RAG retrieval","Context-aware node embedding boosts RAG answer quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001282,"raw_usage":{"total_tokens":5185,"prompt_tokens":833,"completion_tokens":4352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":4285}},"tokens_in":449,"tokens_out":4352,"duration_ms":25065,"temperature":1.0,"reasoning_tokens":4285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:09:52.399396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a held-out set of fifty documents through two ingestion pipelines—this node-based one and a standard recursive chunker—with the same retriever, answer generator, and judge, and compute score distributions over the coarse 0-to-1 grid. If the node-based pipeline's answer-relevancy and faithfulness margins do not exceed the judge's run-to-run variance on repeated scoring, the claimed improvement is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dedicated OCR service that extracts verbatim text from scanned images in the parsing stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames the problem by arguing that standard chunking strategies introduce computational challenges and ignore document structure, motivating the node-based ingestion approach."}],"review_version":1}