{"id":"6f9d5e68-8674-4a69-ba8a-2d404bf61833","arxiv_id":"2507.15863","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"An enterprise RAG module with 1000-token chunks, hybrid retrieval, and a verifier reports small retrieval gains and low hallucination, but several headline numbers conflict with its own appendix tables.","lead":"This report describes DEREK, a commercial retrieval-augmented generation pipeline for enterprise document QA that combines chunk tuning, hybrid search, and a citation-verification loop. A generalist might read it to see whether an off-the-shelf RAG stack can deliver measurable grounding gains in legal and financial question answering.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'limits unsupported statements to less than 3%' is asserted for strict-grounding mode, but Table 5 reports only default-mode hallucination rates of 14–27%, and no strict-grounding results are shown.","rationale":"The Reader's verdict is REJECT with high correctness risk, and this stress-test identifies a concrete, load-bearing gap that supports that verdict: the headline '<3% unsupported statements' claim is not represented in the paper's own evaluation table. The Reader's weakest_assumption focused on the general validity and representativeness of the internal evaluations (ground-truth construction, grader qualifications, error bars, strict-grounding configuration). My concern is more targeted: regardless of how representative the 100 RAGtruth questions are, the strict-grounding configuration that supposedly produces the headline number is never measured, so the central claim is unsupported even on the paper's own evaluation setup. This is a partial agreement because the Reader correctly noted the missing strict-grounding configuration in Appendix A.2/B.1, but the sharper issue is that no strict-grounding result appears in Table 5 at all. I also note the Reader's additional observations: missing Precision@10, default hallucination rates of 14–27%, and utilization above 0.50 for only three of five models. My concern is not about external consensus or novelty; it is about internal consistency between the abstract and the appendix data. If strict-grounding results were supplied and showed <3% hallucination, the production-readiness claim would be substantially stronger; absent that, the evidence does not support the abstract. Because the Reader already assigned REJECT, my recommended verdict is unchanged. I am not recommending ACCEPT or CONDITIONAL, since the missing evaluation is central and the paper currently provides no way to check it without additional author-supplied data.","tokens_in":8024,"tokens_out":2427,"duration_ms":27392,"concrete_test":"Run the same 100-question RAGtruth evaluation described in Appendix B with the strict-grounding preset enabled for all five models, and report hallucination rate, TRACe utilization, and refusal rate. If the measured hallucination rate exceeds 3% for any model, the abstract's 'less than 3%' claim is false in its current form. If strict-grounding code or logs are unavailable, require the authors to provide the missing strict-grounding rows from Table 5 or explicitly soften the abstract claim to match the default-mode numbers shown.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central production-readiness claim rests on the verifier both raising TRACe Utilization above 0.50 and limiting unsupported statements to below 3%. Appendix B, Table 5, reports default DEREK-pipeline hallucination rates of 18.23% (GPT-4o), 27.29% (GPT-4o-mini), 14.03% (Claude 3.7), 17.12% (Gemini 1.5 Pro), and 14.86% (DeepSeek-R1). These are 14–27%, not below 3%. Section B.3 says a strict-grounding preset 'lowers hallucination risk,' but no strict-grounding metrics are reported anywhere in the paper: there is no table row, appendix section, or quantitative evaluation for that configuration. Utilization also exceeds 0.50 for only GPT-4o (0.5224) and Gemini 1.5 Pro (0.5179), not for all five models as the abstract appears to imply. The strongest quantitative support for 'accurate, traceable, and production-ready document QA' is therefore a promissory note rather than a measured result. This is not a claim that the system is bad; it is a claim that the paper's headline numbers are not backed by the data it presents. The case-study numbers in Section 5.2 are also self-reported and unaccompanied by methodology, error bounds, or independent verification, but the more decisive gap is the missing strict-grounding evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes DEREK, a retrieval-augmented generation pipeline for enterprise document QA, with four stages (ingestion/chunking, hybrid HNSW+BM25 indexing, retrieval/reranking, and a LangGraph-based citation verifier). It reports retrieval experiments on four LegalBench subsets comparing 500- and 1000-token chunks, and TRACe generation-quality results for five LLMs against a FAISS baseline. The central claims are that 1000-token chunks improve Recall@50 by roughly 1 pp, hybrid+rerank improves Precision@10 by roughly 7 pp, and the verifier pushes TRACe Utilization above 0.50 while limiting unsupported statements to below 3%. A VC due-diligence case study is presented as evidence of production readiness.","tokens_in":8259,"tokens_out":6090,"duration_ms":61460,"significance":"If the headline numbers were fully supported, the paper would be a useful engineering validation of a standard RAG pipeline with a verification loop, offering concrete comparisons of chunk size, retrieval configuration, and five LLMs that are of interest to practitioners in legal and financial QA. The paper's strengths are the comparatively clean framing of the chunk-size experiment and the inclusion of a multi-model TRACe evaluation; these are potentially useful data points. However, the paper is a self-assessment by the vendor, ships no code, data, or evaluation harness, and its most important claims—the below-3% unsupported-statement rate and utilization above 0.50—are not substantiated by the appendix tables. The contribution is therefore better described as an engineering report than as a fully supported scientific result.","major_comments":[{"comment":"The abstract's claim that 'the verifier raises TRACe Utilization above 0.50 and limits unsupported statements to less than 3%' is not supported by the reported data. Table 5 reports default-mode hallucination rates of 18.23% (GPT-4o), 27.29% (GPT-4o-mini), 14.03% (Claude 3.7), 17.12% (Gemini 1.5 Pro), and 14.86% (DeepSeek-R1), and utilization exceeds 0.50 for only GPT-4o (0.5224) and Gemini 1.5 Pro (0.5179). Section B.3 describes a strict-grounding preset that 'lowers hallucination risk,' but no strict-grounding metrics are reported anywhere in the paper. The headline numbers are therefore either unsubstantiated or contradicted by Table 5, and the production-readiness conclusion in Section 5.1 rests on a promissory note.","section":"Abstract; §5.1; Table 5"},{"comment":"The abstract and Section 5.1 state that 'hybrid+rerank boosts Precision@10 by approximately 7 pp,' but Tables 3 and 4 report Precision only at k = 1, 2, 4, 8, 16, and 50, and no comparison condition isolating hybrid+rerank is presented. There is no k = 10 column and no baseline without hybrid/rerank, so this claim cannot be checked from the manuscript.","section":"Abstract; Tables 3 and 4"},{"comment":"The TRACe evaluation is the only quantitative support for the generation-quality and production-readiness claims, yet the methodology is under-specified: it uses 100 random RAGtruth questions, but no ground-truth construction protocol, grader qualifications, inter-annotator agreement, error bars, or exact strict-grounding configuration are described. Human-rated accuracy and hallucination are central metrics, and the unreported variance makes it impossible to know whether the 1 pp chunk-size difference or any of the 7 pp precision differences are meaningful.","section":"Appendix B"},{"comment":"The VC due-diligence case study reports large business outcomes (e.g., 25-40% faster deal closing, up to 10x more opportunities, 20-30% confidence improvement) with no data-collection procedure, no measured metrics, no error bounds, and no independent verification. These numbers appear in Table 1 as estimated time savings but are not tied to any instrumented evaluation, so they should not be described as results or used to support the production-readiness conclusion.","section":"§5.2"},{"comment":"Tables 3 and 4 report only point estimates, with no confidence intervals or significance tests. The central chunk-size conclusion rests on a difference between 51.82% and 52.54% Recall@50, which is within plausible noise for averaged aggregates; the paper should state the number of queries per dataset, provide standard errors or bootstrap intervals, and specify how the 'ALL' row is averaged.","section":"Appendix A.3"}],"minor_comments":[{"comment":"The reference list contains corrupted author names (e.g., 'Aleksandara Piktus', 'Heinrich Paux', and 'Ishan Underscore' in the Gorilla entry), which must be corrected before any publication.","section":"References"},{"comment":"Table 2 labels a column 'Q–A Pairs' but does not clarify whether these counts refer to the full dataset, a train split, or the evaluation subset; please specify the exact evaluation protocol for each LegalBench subset.","section":"Table 2"},{"comment":"The Figure 1 caption says the process is divided into four stages, but the diagram numbers 15 individual steps; the mapping between stages and steps is unclear and should be made explicit.","section":"Figure 1"},{"comment":"Section 2 reads as a marketing motivation rather than a technical problem definition; the acceptance-criteria statements would be more appropriate in the evaluation section, where they can be directly checked against data.","section":"Section 2"},{"comment":"The phrase 'approximately 1 pp' and 'approximately 7 pp' is used without identifying the exact baseline; the paper should state precisely which table row or configuration yields each claimed improvement.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"This paper is a vendor whitepaper whose central quantitative claims are not backed by the tables in its own appendix. The most serious issue is the abstract's below-3% unsupported-statement claim, for which no strict-grounding evaluation is provided. I would require added strict-grounding results, correction of the abstract and Section 5.1, full methodology details for the TRACe evaluation, and public release of the evaluation harness before acceptance. If the authors cannot supply the missing strict-grounding numbers, the appropriate outcome would be rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a product-oriented preprint, not a research contribution. The system is a standard RAG stack (LangChain loaders, Elasticsearch HNSW+BM25, GPT-4o query rewrite, Cohere rerank, LangGraph verifier). The one solid empirical finding is that 1000-token chunks beat 500-token chunks on Recall@50 by ~0.7pp across four LegalBench subsets. That's a legitimate configuration data point, though without error bars it's not strong.\n\nWhat the paper does well: it's clearly written, the architecture is sensible, and the authors are transparent about some limitations. Also, the admission that their default pipeline hallucinates more than a strict FAISS baseline (14-27% vs 9-15%) is honest.\n\nThe soft spots are significant. The abstract claims Precision@10 improves by ~7pp, but no k=10 precision column appears in any table. It claims the verifier limits unsupported statements to <3%, but Table 5 reports default-mode hallucination rates of 14-27%; the strict-grounding mode that supposedly achieves <3% is never evaluated. Utilization exceeds 0.50 for three of five models (GPT-4o, Gemini, DeepSeek), not all. The stress-test note I saw incorrectly says only two; check the table. Also missing: ground-truth construction protocol, grader qualifications, inter-annotator agreement, and any release of code or data.\n\nThe central production-readiness claim rests on a promissory note, not measured results. That's a load-bearing flaw, because the abstract's whole hook is the sub-3% hallucination guarantee.\n\nWho this is for: practitioners looking for a default RAG configuration and a cautionary example of how to (not) present internal benchmarks. It is not a paper that reshapes the field. With a revised version that reports strict-grounding metrics, error bars, and evaluator details, it could be a useful industry-scale evaluation note.\n\nRecommendation: I'd let a serious referee look at it, because the underlying question—how to make grounded enterprise QA work in practice—is real and the paper's failures are mostly missing evidence rather than wrong architecture. But I would not cite it for any quantitative claim as currently written.","headline":"A clear product report on an assembled RAG stack whose abstract promises more than its own appendix tables deliver.","tokens_in":8862,"tokens_out":2317,"would_cite":false,"duration_ms":22876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a retrieval-augmented generation pipeline whose answers are regenerated until every claim cites retrieved text can make enterprise document QA traceable enough for legal and financial use.","keywords":["retrieval-augmented generation","enterprise document QA","hybrid retrieval","citation verification","hallucination control","chunk size","legal and finance applications","production RAG pipeline"],"falsifier":"An independent run of the same pipeline on a separately constructed document-QA test set with documented ground truth and blinded human grading would settle the central claim; the claim fails if unsupported statements exceed 3% in strict-grounding mode or if context utilization drops below 0.50. A quicker check is to compare hybrid search plus reranking against vector-only search at the same chunk size, since the paper's own tables only compare 500- and 1,000-token chunks.","tokens_in":7757,"feed_emoji":"⚖️","tokens_out":11884,"duration_ms":122470,"temperature":0.7,"pith_summary":"This paper is a system report for DEREK, an enterprise module that answers questions over contracts, manuals, and other documents by retrieving passages and making a large language model cite them. It argues that the module's shipped defaults—1,000-token chunks, a hybrid vector-and-keyword index with reranking, and a verification loop that regenerates any draft sentence that lacks a supporting citation—are what make the answers traceable. On internal evaluations, 1,000-token chunks raised recall at the top 50 retrieved passages by about one percentage point over 500-token chunks, and the verifier kept unsupported statements below 3 percent in strict-grounding mode while pushing context utilization above 0.50. If these results hold, the practical consequence is that careful pipeline configuration, rather than a new model or training method, is enough to make document QA usable in legal and financial settings.","feed_headline":"Verifier loop keeps unsupported legal answers under 3 percent","feed_subtitle":"A citation-checking loop turns ordinary AI document search into auditable legal and finance answers.","key_machinery":"The load-bearing mechanism is the verification loop: after the LLM drafts an answer, a second pass checks each sentence for an overlapping citation in the retrieved snippets and regenerates the draft, with refined instructions, until every sentence passes. This is what turns retrieved passages into auditable claims. The same loop is supported by two design choices: 1,000-token chunks with 150-token overlap, which preserve enough surrounding context for citations, and a hybrid vector-plus-keyword index followed by reranking, which supplies the verifier with cleaner top-50 context than keyword search or vector search alone.","core_discovery":"The paper's central claim is that the architecture, not the generator, is what makes an LLM usable for enterprise document QA. In DEREK, a user question is expanded, used to pull 200 candidate passages from a hybrid vector-plus-keyword index, reranked to the 50 most relevant, and handed to a large language model instructed to answer from context. A verifier then checks each sentence of the draft against the retrieved text and regenerates the answer until every sentence carries an overlapping citation. The paper reports that on four legal question sets the default 1,000-token chunks raise recall at the top 50 from 51.82% to 52.54% versus 500-token chunks, and that hybrid search plus reranking adds roughly 7 percentage points of precision at the top 10; on 100 random questions from a public QA set, the verifier pushes context utilization above 0.50 and keeps unsupported statements below 3% in strict-grounding mode, with human-graded accuracy at or near 4.0 out of 5 across several large language models.","pith_inferences":["Editorial inference: the verification loop is separable from the rest of the stack, so the same mechanism could be applied to other retrieval-augmented systems; traceability would become a configuration choice rather than a property of any one LLM.","Editorial inference: a natural extension is to measure the refusal rate that strict grounding introduces, since the paper mentions this tradeoff only qualitatively; that number would tell enterprises the real cost of the under-3% guarantee.","Editorial inference: the architecture's portability across large language models suggests the next test is whether the verification loop also controls unsupported statements with smaller or open-weight models, which would matter for cost-sensitive deployments.","Editorial inference: in high-stakes settings the more relevant threshold may be the verifier's ability to catch subtle synthesis errors that are logically implied but not stated verbatim, which the paper lists as a limitation and leaves to future work."],"forward_implications":["A production team could adopt the module's shipped defaults as a baseline: the paper reports the 1,000-token chunk setting gains roughly one point of recall at the top 50 over 500-token chunks without added latency.","The verifier, not the choice of LLM, is the main hallucination control: the claim of under 3% unsupported statements applies when strict-grounding mode is enabled alongside the verification loop.","The precision gain from hybrid search and reranking gives the generator cleaner context, which is what the paper connects to higher context relevance compared with a simpler vector-only baseline.","The reported accuracy scores across several large language models suggest the pipeline is portable; an enterprise could plug in a preferred generator without rebuilding retrieval or verification.","The strict-grounding tradeoff is explicit: the baseline pipeline hallucinated less because it refused to answer without evidence, while DEREK allows limited abstraction and adds a refuse-to-answer path only in strict mode."],"supporting_citations":[{"why":"Defines retrieval-augmented generation, the paradigm the module operationalizes in a production pipeline.","marker":"Lewis et al., 2020"},{"why":"Supplies the retrieve-then-read formulation that motivates the coupled retrieval and generation stages.","marker":"Izacard and Grave, 2021"},{"why":"Provides the orchestration layer used for loaders, chunking, and prompt flow in the implementation.","marker":"Chase, 2022"}],"fun_headline_variants":["Verifier loop keeps unsupported legal answers under 3 percent","Citation-checking verifier keeps legal LLM answers grounded","Legal QA verifier ensures every claim is cited","Grounding loop for legal AI: unsupported below 3%","Hybrid retrieval plus verifier makes legal answers auditable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that internal evaluations on four legal question sets and 100 randomly drawn questions, with undisclosed ground-truth construction, human-grading protocol, and the exact strict-grounding settings, are representative enough to support claims of production readiness.","fun_headline_variants_meta":{"raw":{"variants":["Verifier loop keeps unsupported legal answers under 3 percent","Citation-checking verifier keeps legal LLM answers grounded","Legal QA verifier ensures every claim is cited","Grounding loop for legal AI: unsupported below 3%","Hybrid retrieval plus verifier makes legal answers auditable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000951,"raw_usage":{"total_tokens":4094,"prompt_tokens":1019,"completion_tokens":3075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2993}},"tokens_in":635,"tokens_out":3075,"duration_ms":27547,"temperature":1.0,"reasoning_tokens":2993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:53:38.483864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent run of the same pipeline on a separately constructed document-QA test set with documented ground truth and blinded human grading would settle the central claim; the claim fails if unsupported statements exceed 3% in strict-grounding mode or if context utilization drops below 0.50. A quicker check is to compare hybrid search plus reranking against vector-only search at the same chunk size, since the paper's own tables only compare 500- and 1,000-token chunks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the paradigm the module operationalizes in a production pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the retrieve-then-read formulation that motivates the coupled retrieval and generation stages."}],"review_version":1}