{"id":"a2521d8d-378f-4b08-9163-10e9c5bde898","arxiv_id":"2506.23136","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":13,"one_line_summary":"A RAG pipeline with OCR, table and image to text conversion, and a RAFT-tuned reranker reports high QA scores, but its 50-question evaluation overlaps with its training manuals and its baseline comparison uses only 5 questions.","lead":"This paper builds a question-answering system that reads technical documents with tables, images, and scanned pages by combining text search with a language model trained to pick the correct context. The authors report high accuracy, but the tests are small, partly drawn from the same manuals used for training, and the code and data links are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training and test draw on the same three technical manuals; the 'no exact question reused' caveat does not rule out passage-level memorization, so the headline faithfulness/relevancy scores may measure document familiarity, not retrieval quality.","rationale":"The reader's REJECT rests on the credibility of the evaluation, and I find the same weakest assumption to be load-bearing. The paper's evidence requires the test set to be independent of the RAFT training corpus, but §4.2.2 and §5.3.1 create overlapping document sources. The statement that no exact question was reused does not address passage-level memorization, so the headline scores and the outside-context advantage may reflect document familiarity rather than retrieval quality. This is an internal-consistency concern, not a disagreement with community consensus, and it directly undermines the central quantitative claims. The five-question comparison to a general RAG pipeline is statistically thin and would support a conditional verdict at best, but the source overlap is sufficient to keep the verdict at REJECT until a document-level held-out evaluation is provided. The engineering contribution could still be useful, but the scientific claim as presented is not established.","tokens_in":15678,"tokens_out":5185,"duration_ms":54762,"concrete_test":"Retrain the reranker with a document-level split: remove from the RAFT corpus all chunks derived from the three testing manuals (or, minimally, the Power Transformer Testing manual), add comparable unseen technical manuals, and then evaluate on the existing 50 questions with identical hyperparameters. If the RAGAS/DeepEval faithfulness and relevancy scores drop materially (e.g., more than 5 percentage points) relative to the reported 94/96 and 87/93, the original scores are inflated by train/test document overlap. Report per-question scores and confidence intervals on the held-out technical subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the proposed pipeline achieves roughly 94-96% faithfulness, 87-93% answer relevancy, and superiority on table-based and outside-context questions. The load-bearing assumption is that the 50-question test set measures generalization to unseen content. That assumption is not met as described. Section 4.2.2 builds the RAFT training corpus from the Circuit Breaker, Power Cable, and Power Transformer testing manuals, generating one oracle question and one CoT answer from each of 240 chunks. Section 5.3.1 constructs the technical part of the test set from transformer-testing documents, and all five displayed technical questions in Table 6 are transformer-manual content. A question-level check that no exact question was reused does not rule out passage-level leakage: the reranker was fine-tuned on chunks containing the same sentences, tables, and paraphrased answers that the test queries target, so it can identify the correct context through memorized lexical overlap even for a novel surface question. This would inflate faithfulness, answer relevancy, context precision, context recall, and the outside-context advantage, since recognizing that an answer is absent is also easier when the model has memorized the coverage of the training manuals. Without a document-level split, the reported numbers cannot be attributed to the retrieval pipeline rather than to content familiarity. The five-question comparison to a general RAG pipeline adds a second weakness, but the source overlap is the more fundamental threat because it contaminates the main evidence itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Retrieval-Augmented Generation (RAG) pipeline for technical documents that handles scanned PDFs, tables, and images. The pipeline converts scanned documents via OCR, extracts tables and images into descriptive text, performs two-stage retrieval (vector similarity search followed by a Gemma-2-9b-it reranker fine-tuned with RAFT), and generates answers with a fine-tuned Gemma model. The authors report high RAGas and DeepEval scores on a 50-question self-created test set (94% and 96% faithfulness; 87% and 93% answer relevancy) and claim superiority over general RAG pipelines on table-based and outside-context questions, supported by a 5-question comparison. The paper provides code, notebooks, and fine-tuned weights for reproducibility.","tokens_in":15931,"tokens_out":6763,"duration_ms":68544,"significance":"If the headline results held, the pipeline would be a practically useful contribution for question answering over industrial technical manuals, especially in settings with scanned documents and tabular content. The paper is commendably concrete about the pipeline components and makes its artifacts available. However, the central empirical claims are not supported by the evaluation as designed: the RAFT training data and the technical test questions are drawn from overlapping source manuals, the comparison with general RAG uses only five questions, and all scores are LLM-as-judge point estimates without human validation or statistical testing. The claimed quantitative advantages could reflect content memorization rather than retrieval quality, so the significance of the work is currently unverified.","major_comments":[{"comment":"The training and test sets overlap at the document level. The RAFT corpus is generated from the Circuit Breaker Testing Guide, Power Cable Testing, and Power Transformer Testing manuals, while the technical portion of the test set is built from transformer-testing documents; all five technical questions shown in Table 6 are transformer-manual content. Since the reranker and the generator (Section 3.3, Table 3) are fine-tuned on chunks of these same manuals, the statement that no exact question was reused does not rule out passage-level memorization of the relevant sentences, tables, and paraphrased answers. A document-level split, such as fine-tuning on two manuals and testing on a held-out manual, is necessary before the reported faithfulness, relevancy, and outside-context advantages can be attributed to the retrieval pipeline rather than to content familiarity.","section":"§4.2.2 and §5.3.1"},{"comment":"The claim of superiority over general RAG pipelines rests on only five questions, and the two headline sub-claims (table-based questions and outside-context questions) each depend on a single instance, Q1 and Q5 respectively. No confidence intervals or significance tests are provided. In addition, context precision and context recall for Q5 are marked N/A, so the average values of 1.0 in Table 10 are computed over only four questions. A single example in each category is insufficient evidence for a general architectural claim.","section":"§5.4, Table 10"},{"comment":"The headline scores are produced by an LLM-as-a-judge (Llama-3.3-70B) on a self-created 50-question set, with no human validation, no variance estimate, and no comparison against an existing benchmark. The authors themselves note in Section 5.3 that evaluation is subjective and that different evaluator LLMs give different scores. Reporting point estimates such as 94% and 96% without confidence intervals or human-audited examples does not establish that these are stable properties of the pipeline.","section":"§5.3.2 and §5.3.3"},{"comment":"The reported context precision (0.94) and context recall (0.97) are computed under a setup where RAGas uses the same chunk size as the proposed pipeline, and the paper's own explanation states that because the correct context and retrieved context are similar in size, these metrics become very high whenever the retrieved chunk is correct. This makes the metrics heavily dependent on the authors' chunk-level ground-truth assignments rather than being independent measures of retrieval quality, and the resulting near-perfect scores should not be presented as strong evidence of retrieval superiority.","section":"§5.3.2"}],"minor_comments":[{"comment":"The two example tables are rendered identically, so the claimed loss of table structure in the vector database is not visible to the reader; showing a flattened representation with lost borders would make the illustration clearer.","section":"Table 2"},{"comment":"The statement that RAGas 'evaluates exactly word by word' is inaccurate, since RAGas faithfulness and answer relevancy are based on LLM judgments rather than exact string matching.","section":"§5.3.2"},{"comment":"The sentence 'From earlier evaluations, it is proven that our model performs better' overstates what the evidence supports; a more cautious formulation such as 'suggests' or 'indicates' would be appropriate.","section":"§5.5"},{"comment":"The composition of the 1,040 chunks should be stated explicitly; the current text requires the reader to infer that 500 of the 800 additional chunks are technical to reconcile the reported totals of 740 technical and 300 general chunks.","section":"§4.2.2"},{"comment":"Several reproducibility links appear as '[link]' placeholders rather than full URLs or DOIs; these should be completed in the references for the artifacts to be accessible.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper describes a coherent pipeline and provides useful engineering details, but the evaluation design is the central issue: the training and test corpora are not document-disjoint, the comparative evaluation is extremely small, and the headline metrics are LLM-judged point estimates. These are not presentation issues but load-bearing problems that require a fundamentally reworked evaluation, ideally with a held-out document split, a larger and ideally public benchmark, and statistical testing. If the authors can provide such an evaluation, the work may be worth reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fast read: this is a competent engineering write-up, but the headline faithfulness and relevancy scores are not trustworthy. The test set draws on the same three testing manuals used to build the RAFT fine-tuning corpus; the reranker was trained on chunks containing exactly the passages the test questions target. The 'no exact question reused' statement only checks surface question text, not passage-level overlap, so the scores likely reflect content familiarity rather than retrieval quality. That is a load-bearing flaw.\n\nWhat is actually new: the pipeline is a clean assembly of known parts—OCR, YOLO table detection, LLM table summarization, VLM image captioning, bge-small embeddings, FAISS, and a Gemma-2-9b-it reranker fine-tuned with RAFT. The one real twist is using RAFT-style data to train a reranker rather than the generator. That is a sensible transfer, clearly explained. The extraction examples are illustrative, and the writing is straightforward.\n\nSoft spots, in order:\n\n1. Train/test overlap. Section 4.2.2 shows the three technical manuals used for RAFT training; Section 5.3.1 builds the technical part of the test set from transformer-testing documents, and all five displayed test questions in Table 6 are transformer-manual content. Without a document-level split, the 94–96% faithfulness and 87–93% relevancy numbers cannot be attributed to the retrieval pipeline. This is the main threat.\n2. The general-RAG comparison in Table 10 uses five questions, no error bars, no significance testing. The proposed pipeline wins on some questions, but that is anecdotal.\n3. Reproducibility: the GitHub, dataset, weight, and document links are placeholders. As it stands, the experiments cannot be independently checked.\n4. The evaluation uses LLaMA-3.3-70B as an LLM judge. That is common, but it is a confound the authors acknowledge only in passing.\n\nOne small point: the reader's note mentions a factual error in the table extraction example. I checked the parsed rows against Table 4; the numbers match, so that specific criticism does not land. The broader claim that table summaries are 'always precise' is still overconfident given a single example.\n\nWho is this for? Practitioners building RAG for technical documents may find the modular recipe useful. As a scientific claim about retrieval quality, the evidence is not there yet.\n\nRecommendation: I would not send this to a serious venue in its current form. A revised version with a document-level split, a larger baseline set, and real links to code and data would deserve a proper review. As is, treat the headline numbers as unvalidated.","headline":"Competent RAG engineering write-up, but the train/test overlap between the RAFT corpus and the evaluation set makes the headline faithfulness/relevancy numbers unreliable.","tokens_in":16581,"tokens_out":7144,"would_cite":false,"duration_ms":65101,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A structured-data-aware RAG pipeline reports 94–96% faithfulness and 87–93% answer relevancy on technical-document QA.","keywords":["LLM","Hallucination","Fine-tuning","RAG","Structured data","RAFT","Question answering","Document understanding"],"falsifier":"Run the full pipeline on a technical corpus that shares no text with the three manuals used to build the RAFT training data, with a fresh set of questions built from that corpus; if faithfulness and answer relevancy fall materially below the reported 94–96% and 87–93%, or if the system starts answering out-of-context questions with fabricated content, the central claim of general technical-document capability is refuted.","tokens_in":15400,"feed_emoji":"📄","tokens_out":13829,"duration_ms":125433,"temperature":0.7,"pith_summary":"This paper claims that a RAG pipeline can answer questions about technical documents containing tables, images, and scanned pages with a faithfulness score of 94% (RAGas) and 96% (DeepEval), and an answer-relevancy score of 87% (RAGas) and 93% (DeepEval). The wager is that the bottleneck in technical-document QA is not the generator but the retrieval: tables lose their structure when flattened into chunks, images never enter the vector store, and vector search can pull chunks that are similar but wrong. The pipeline therefore converts scanned pages to searchable text, turns tables into descriptive sentences and figures into VLM-generated captions, and reranks retrieved chunks with an LLM fine-tuned with RAFT to tell relevant from irrelevant contexts. If the claim is right, engineers could get faithful, source-grounded answers from equipment manuals and test reports without retraining the generator for each new document. The paper further claims that this architecture beats a general RAG pipeline on table-based questions and on questions whose answers lie outside the document, where the general pipeline hallucinates.","feed_headline":"RAG pipeline hits 94–96% faithfulness on technical PDFs","feed_subtitle":"Tables become sentences, images become captions, and a reranker trained to reject wrong contexts keeps answers faithful.","key_machinery":"The argument is carried by two coupled mechanisms. The first is a structured-data preprocessing layer: OCR converts scanned PDF pages into searchable text; a table detector locates tables, which are converted to HTML and then paraphrased by an LLM into row-by-row descriptive sentences; and a vision-language model turns figures into detailed captions. These descriptions sit alongside the original text so the vector store can retrieve them. The second mechanism is the reranker: after first-stage semantic search over fixed 512-token chunks retrieves the top 10 contexts, a Gemma-2-9b-it model fine-tuned with RAFT scores them and keeps the top 3. RAFT (Retrieval-Augmented Fine-Tuning) matters because the training data pairs each question with an oracle chunk carrying the answer and two distractor chunks, teaching the reranker to prefer contexts that actually answer and to stay silent when none does. The generator, also fine-tuned Gemma-2-9b-it, answers from the reranked context.","core_discovery":"The core discovery is that making a RAG system both structured-data aware and context-aware is what moves the metrics: first, tables and images are extracted and re-expressed as natural-language descriptions so that semantic search can index them; second, a reranker fine-tuned with RAFT on oracle-plus-distractor chunks learns to reject retrieved passages that are topically adjacent but do not contain the answer. With these two changes, the system reports a faithfulness of 0.94 (RAGas) and 0.96 (DeepEval) and an answer relevancy of 0.87 (RAGas) and 0.93 (DeepEval) on a 50-question test set drawn from technical and general documents. In the head-to-head comparison, the proposed pipeline answered a table-derived question correctly where the general pipeline stumbled, and on a question outside the document it answered 'This document doesn't contain the answer' while the general pipeline fabricated a response. The authors present this as evidence that the pipeline improves retrieval and generation on technical documents without requiring the generator to be retrained on each new corpus.","pith_inferences":["Beyond the paper, the same preprocessing could be tested on engineering drawings and datasheets by pairing the table detector with a stronger vision-language model, since the paper itself flags open-source VLM accuracy as a current limit.","Beyond the paper, the outside-context advantage could be stress-tested on documents whose topic differs entirely from the fine-tuning manuals, which would separate the RAFT reranker's 'do not answer' behavior from topic familiarity.","Beyond the paper, the table-to-sentence conversion suggests a testable extension: measure whether retrieval precision degrades for tables with ambiguous headers, merged cells, or multi-row entries, and whether retaining the HTML structure would preserve more information."],"forward_implications":["Questions about column-and-row table data in manuals can be answered because tables are re-expressed as descriptive sentences before embedding.","Scanned technical manuals become usable in RAG without manual text extraction, since OCR makes their pages searchable.","Questions whose answers are not in the supplied documents should produce an explicit 'not found' answer instead of a hallucinated one.","The reranker can be fine-tuned once on mixed technical and general chunks and then applied to new corpora without retraining the generator.","The reported scores suggest that entirely open-source components are sufficient for a technical-document QA pipeline in a resource-constrained setting."],"supporting_citations":[{"why":"Supplies the RAFT training recipe (oracle chunks plus distractor chunks) that the paper uses to fine-tune the reranker.","marker":"[41]"},{"why":"Provides the RAGas faithfulness, answer-relevancy, context-precision, and context-recall metrics used for the headline scores.","marker":"[42]"},{"why":"Defines the retrieval-augmented generation framework that the proposed pipeline extends and uses as the general RAG baseline for comparison.","marker":"[26]"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to adapt the generator and reranker without updating the full model.","marker":"[38]"},{"why":"Shows that specialized table-parsing models exist but do not integrate naturally into RAG, motivating the paper's table-to-sentence extraction step.","marker":"[34]"}],"fun_headline_variants":["Table-aware RAG reranker hits 96% faithfulness","Structured-data RAG: tables become text, answers stay faithful","RAG that turns tables into sentences beats general pipelines","Context-aware reranker lifts RAG faithfulness to 94–96%","RAG rejects wrong contexts, scores 96% faithfulness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the 50-question test set being an independent measure of generalization, but the technical test questions come from transformer testing documents and the RAFT training data was generated from the same three testing manuals, so content overlap rather than retrieval quality could explain the high scores and the outside-context advantage.","fun_headline_variants_meta":{"raw":{"variants":["Table-aware RAG reranker hits 96% faithfulness","Structured-data RAG: tables become text, answers stay faithful","RAG that turns tables into sentences beats general pipelines","Context-aware reranker lifts RAG faithfulness to 94–96%","RAG rejects wrong contexts, scores 96% faithfulness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001053,"raw_usage":{"total_tokens":4455,"prompt_tokens":1011,"completion_tokens":3444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":3359}},"tokens_in":627,"tokens_out":3444,"duration_ms":27749,"temperature":1.0,"reasoning_tokens":3359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:48:39.908835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a technical corpus that shares no text with the three manuals used to build the RAFT training data, with a fresh set of questions built from that corpus; if faithfulness and answer relevancy fall materially below the reported 94–96% and 87–93%, or if the system starts answering out-of-context questions with fabricated content, the central claim of general technical-document capability is refuted.","supporting_citations":[{"cited_title":"In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp","cited_arxiv_id":null,"evidence_quote":"Provides the RAGas faithfulness, answer-relevancy, context-precision, and context-recall metrics used for the headline scores."},{"cited_title":"Advances in neural information processing systems 33, 9459–9474 (2020)","cited_arxiv_id":null,"evidence_quote":"Defines the retrieval-augmented generation framework that the proposed pipeline extends and uses as the general RAG baseline for comparison."},{"cited_title":": Lora: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Provides LoRA, the parameter-efficient fine-tuning method used to adapt the generator and reranker without updating the full model."}],"review_version":1}