{"id":"9907f0b0-aa91-4dad-babb-39da199fa9f3","arxiv_id":"2506.16768","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"eSapiens combines text-to-SQL and hybrid retrieval-augmented generation with citation verification, and its evaluation on RAGTruth shows better contextual relevance than a FAISS baseline across most LLMs, at the cost of more hallucination.","lead":"This paper describes eSapiens, an enterprise system that answers plain-language questions by combining database queries with document search and AI generation. Tests against a simpler FAISS-based pipeline show higher contextual relevance and readability on most large language models, but also higher hallucination rates, which the paper acknowledges.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's hallucination scores contradict the conclusion: eSapiens reports higher hallucination and lower completeness than FAISS across LLMs, so 'outperforming in factual grounding' is unsupported.","rationale":"The reader's weakest assumption focuses on benchmark reliability and lack of confidence intervals, which is a valid concern. However, the more decisive issue is internal: Table 3 and the TRACe section directly contradict the conclusion's claim of superiority in factual grounding. Even if every metric were perfectly reliable, eSapiens shows higher hallucination and lower completeness than FAISS for all five LLMs. The reader's rationale does note this ('the system's own table shows more hallucination and less completeness than the FAISS baseline'), but the formal weakest_assumption field points to statistical generalizability rather than this contradiction. My agreement is therefore partial. The appropriate action is still a conditional acceptance with required revisions: temper or re-scope the headline claim, release the evaluation artifacts, and either provide strict-grounding results or describe the factual-grounding behavior as a trade-off rather than an outperformance. The unresolved internal contradiction is addressable and does not by itself invalidate the entire system description, but it does mean the central claim cannot be accepted as stated.","tokens_in":7259,"tokens_out":2669,"duration_ms":32324,"concrete_test":"Re-run the RAGTruth TRACe evaluation on the exact question subset used for Table 3, reporting per-model hallucination and completeness with bootstrap 95% confidence intervals, and include the eSapiens 'strict grounding' preset. If strict grounding keeps hallucination above the FAISS baseline, the conclusion must drop 'factual grounding' and describe a trade-off instead; if strict grounding lowers hallucination below FAISS, that configuration should be the basis for the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is not external benchmark reliability but an internal contradiction. Table 3 reports hallucination (lower is better) for eSapiens-gpt4o at 0.1823 versus faiss+top-2+short+gpt4o at 0.0875, and the gap persists across all five LLMs. Completeness is also lower for eSapiens in four of five rows. The TRACe section explicitly states that FAISS has 'consistently lower hallucination rates' and offers 'stricter factuality guarantees.' Yet the Conclusion claims eSapiens outperforms FAISS 'in relevance, fluency, and factual grounding.' The factual-grounding component is contradicted by the paper's own quantitative results; only contextual relevance and human-rated accuracy/fluency are consistently better. The absence of confidence intervals, annotator agreement, and question counts compounds the issue, but even the reported point estimates refute the conclusion as written. The optional 'strict grounding' preset is mentioned as a mitigation, but no strict-grounding results are reported, so the paper cannot currently support the factual-grounding superiority claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces eSapiens, an enterprise question-answering system that combines a Text-to-SQL module for structured databases with a hybrid retrieval-augmented generation (RAG) module for unstructured documents. The RAG pipeline uses dense HNSW and sparse BM25 retrieval, a commercial reranker, and a citation verification loop intended to enforce grounded generation. The authors evaluate retrieval quality on four LegalBench datasets at two chunk sizes, compare answer quality against a FAISS baseline on the RAGTruth benchmark across five LLMs using TRACe-style metrics, and include an appendix with seven anecdotal Text-to-SQL queries. The central reported findings are that eSapiens improves contextual relevance and human-rated accuracy/fluency, while the FAISS baseline has lower hallucination rates and generally higher completeness. The paper concludes, however, that eSapiens outperforms FAISS in 'relevance, fluency, and factual grounding,' a claim that conflicts with the paper's own Table 3.","tokens_in":7448,"tokens_out":2790,"duration_ms":31193,"significance":"A production-oriented system that unifies structured and unstructured enterprise data access and provides citation-aware generation is a useful engineering contribution, and the paper deserves credit for evaluating across five LLMs and using external benchmarks (LegalBench and RAGTruth) rather than only self-reported examples. The explicit reporting of hallucination and completeness numbers, even when unfavorable, is a strength in transparency. That said, the core comparative claim of factual-grounding superiority is contradicted by the paper's own quantitative results, and the absence of statistical detail (confidence intervals, annotator agreement, question counts) makes the remaining positive claims difficult to assess. The paper is best read as a system description with preliminary evaluation; in its current form it does not support the stronger conclusions stated in the abstract and conclusion.","major_comments":[{"comment":"The conclusion that eSapiens outperforms FAISS 'in relevance, fluency, and factual grounding' is contradicted by Table 3. For every LLM, the eSapiens pipeline reports higher hallucination (lower is better) than the corresponding FAISS baseline; for example, eSapiens-gpt4o has hallucination 0.1823 versus 0.0875 for faiss+top-2+short+gpt4o, and the same pattern holds for all five models. Completeness is also lower for eSapiens in four of five rows. The TRACe Evaluation section itself acknowledges that FAISS has 'consistently lower hallucination rates' and 'stricter factuality guarantees.' The factual-grounding component of the conclusion must be removed or replaced with results from the 'strict grounding' preset, which is mentioned but never evaluated.","section":"Table 3 and Conclusion"},{"comment":"The TRACe evaluation reports only point estimates with no confidence intervals, no annotator agreement statistics, and no number of test questions. The text describes the question set as 'consistent and representative' but does not specify its size, sampling method, or relation to the full RAGTruth benchmark. Without this information, the observed differences in contextual relevance and human-rated accuracy between eSapiens and FAISS cannot be distinguished from noise, especially given that hallucination and completeness favor the baseline.","section":"TRACe Evaluation, Table 3"},{"comment":"The paper states that the citation verification loop 'ensures factual integrity and traceability' and that the system regenerates until all cited statements are supported, yet Table 3 shows that eSapiens has higher hallucination than a simple FAISS baseline across all five LLMs. This indicates that the verification loop either does not operate as described or is ineffective at preventing unsupported claims. The authors should report the verification loop's pass/fail statistics or an ablation comparing eSapiens with and without the loop; currently the architecture's central grounding mechanism is unsupported by the presented evidence.","section":"RAG Module Architecture"},{"comment":"The LegalBench retrieval analysis (Tables 1 and 2) reports recall and precision for eSapiens alone, without any baseline or downstream QA evaluation. The absolute numbers are low, especially for MAUD (Recall@50 of 22.60% at chunk 500 and 13.60% at chunk 1000), which undermines the claim that the results 'affirm that the design of the eSapiens RAG module ensures robust and generalizable performance.' A comparison against at least one standard retrieval baseline, or a task-level metric such as answer accuracy, is needed before drawing conclusions about robustness.","section":"Retrieval Performance on Long-form Legal QA"}],"minor_comments":[{"comment":"The text lists 'citation faithfulness, contextual relevance, factual correctness, and information completeness' as the four key metrics, but Table 3 reports 'completeness, utilization, context relevance, hallucination, and accuracy.' The metric names should be aligned.","section":"Experiment"},{"comment":"The benchmark name is written inconsistently as 'RAGtruth' in Table 3 and 'RAGTruth' elsewhere; please standardize.","section":"Tables 1-3"},{"comment":"The T2S evaluation is based on only seven hand-picked queries with anonymized baselines and no quantitative success/failure counts; it should be labeled as a case study or pilot rather than an evaluation.","section":"Appendix A"},{"comment":"The 'large-scale embedding model' and 'commercial reranking model' are never named, which prevents reproducibility; please provide model names or versions.","section":"RAG Module Architecture"},{"comment":"The abstract and architecture describe 'approximately 1000 tokens' chunks, and the experiments compare chunk sizes 500 and 1000, but the exact tokenization and overlap settings are not stated; please specify them.","section":"Retrieval Performance on Long-form Legal QA"}],"recommendation":"major_revision","confidential_remarks":"The paper's own Table 3 refutes the abstract's factual-grounding claim, so the revision must either present strict-grounding results or substantially soften the claims. The T2S appendix is more promotional than evaluative and should be recast as a qualitative case study. Given the system's breadth, the authors may also consider submitting to a systems or demo venue rather than a research track."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a credibly engineered enterprise RAG/T2S system, and the paper gives you enough architecture detail to build on, but the abstract and conclusion overstate what their own Table 3 shows. The hallucination numbers contradict the \"factual grounding\" claim.\n\nWhat's actually new: no single component — hybrid retrieval, reranking, citation verification, LangGraph orchestration are all known — but the specific integration and the chunk-size sensitivity study on LegalBench are documented carefully. Running the same pipeline across five LLMs on RAGTruth is a reasonable empirical exercise, and the retrieval recall/precision tables are useful for practitioners picking chunk sizes.\n\nCredit where due: the paper is unusually explicit about the tradeoff. The TRACe section plainly says FAISS has \"consistently lower hallucination rates\" and \"stricter factuality guarantees,\" and even explains why (strict reliance vs. limited abstraction). That transparency is good.\n\nThe soft spot is a load-bearing contradiction. Table 3 shows eSapiens hallucinates more than FAISS on all five models, and is less complete on four of five. The conclusion nevertheless claims eSapiens \"outperforms FAISS-only baselines ... in relevance, fluency, and factual grounding.\" The first two are defensible; factual grounding is not supported by their own numbers. The optional \"strict grounding\" preset is never evaluated, so it can't rescue the claim. That needs fixing before publication.\n\nBeyond the contradiction: the eval is thin in standard ways — no confidence intervals, no annotator agreement, no size or composition of the RAGTruth subset, and no comparison against a hybrid pipeline beyond their own. The T2S evaluation is seven hand-picked queries, which shows the mechanism but proves nothing. No code or data, so the numbers aren't checkable.\n\nMy take: the engineering is real and the architecture description would help people building similar systems, but the central claim needs to be rewritten to match the evidence, and the artifact release and uncertainty reporting need to be addressed. The reader's conditional verdict is fair; the stress-test note is right on the factual-grounding contradiction.\n\nRecommendation: send it to review. The internal contradiction is fixable, and a referee can push for honest claims and artifacts. I'd tell the authors to revise, release the evaluation setup, and rerun the conclusion. Worth a reading group if you want a case study in how benchmark tables can tell a different story than the abstract.","headline":"A well-described enterprise RAG/T2S system whose own Table 3 contradicts the abstract's factual-grounding claim — the architecture is worth a look, the headline needs revision.","tokens_in":7994,"tokens_out":2572,"would_cite":false,"duration_ms":25733,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid retrieval pipeline with citation checks beats vector-only search on enterprise QA.","keywords":["eSapiens","enterprise question answering","retrieval-augmented generation","hybrid retrieval","citation verification","Text-to-SQL","RAGTruth","hallucination"],"falsifier":"Re-run the comparison on a large, publicly fixed sample of RAGTruth questions with multiple annotators, pre-registered scoring, and confidence intervals; the claim is settled if eSapiens no longer shows higher context relevance and accuracy, or if its hallucination rate is not higher than FAISS. A negative result would also follow if removing the citation-verification loop leaves answer quality unchanged.","tokens_in":7043,"feed_emoji":"🔍","tokens_out":7913,"duration_ms":72580,"temperature":0.7,"pith_summary":"eSapiens is an enterprise question-answering system that routes natural-language questions either to a Text-to-SQL planner or to a retrieval-augmented generation pipeline. The paper's central claim is that the RAG pipeline, combining dense and sparse retrieval with reranking and a citation-verification loop, selects more relevant passages and produces answers that human raters find more accurate and natural than those of a vector-search-only baseline. This is reported across five LLMs on the RAGTruth benchmark, alongside the finding that the baseline hallucinates less while eSapiens offers a strict-grounding preset for compliance-sensitive use. The broader claim is that a production-grade, citation-aware RAG system can bridge structured and unstructured enterprise data.","feed_headline":"Dense plus sparse retrieval and citation checks beat vector search","feed_subtitle":"Enterprise QA system eSapiens claims higher relevance and accuracy across five LLMs on RAGTruth.","key_machinery":"The load-bearing machinery is the hybrid retrieval and grounding pipeline: a stateful execution graph in which a supervisor agent routes each query, an index combining HNSW dense vectors and BM25 sparse keyword search returns the top 200 candidate passages, a commercial reranker narrows these to the top 50 snippets, and a citation-verification loop checks every generated sentence against the retrieved snippets, triggering regeneration when a cited statement is unsupported. The T2S module contributes a bounded retry loop that reads SQL error messages and regenerates queries, which the paper credits for recovering from schema and syntax failures.","core_discovery":"On its own terms, the paper demonstrates that eSapiens' hybrid retrieval pipeline—dense vector search plus sparse keyword search, reranked by a commercial model—and its citation-verification loop supply the generation stage with more semantically targeted passages, and that this translates into higher context-relevance scores and higher human-rated accuracy than a FAISS top-k pipeline on the RAGTruth question set across GPT-4o, GPT-4o-mini, Claude 3.7, Gemini 1.5 Pro, and DeepSeek R1. The paper also reports that the FAISS baseline exhibits lower hallucination, and positions eSapiens' optional strict-grounding mode as the remedy for that tradeoff.","pith_inferences":["The citation-verification loop could in principle be extracted and used as a standalone hallucination checker for any RAG system, independent of eSapiens.","The reported tradeoff (lower hallucination for FAISS, higher relevance and accuracy for eSapiens) suggests a controllable fidelity-fluency dial; whether that dial can be tuned without losing relevance is a testable question.","Because the paper compares pipelines with the same LLM held fixed, its results imply retrieval design and post-processing matter as much as model choice for enterprise answer quality.","The legal-domain retrieval tables suggest chunk size interacts with document structure, so a document-type-aware chunking policy could extend eSapiens beyond fixed 1000-token windows."],"forward_implications":["If eSapiens works as claimed, enterprise users can get more contextually relevant answers from the same underlying LLM simply by changing the retrieval and verification pipeline.","The strict-grounding preset gives a compliance-mode answer generation that refuses to produce unsupported sentences, potentially making RAG usable in regulated industries.","The hybrid retrieval design is reported to generalize across five different LLMs, suggesting the pipeline gains do not depend on a single model.","The T2S self-healing loop implies that many natural-language SQL failures can be recovered automatically without human intervention."],"supporting_citations":[],"fun_headline_variants":["Hybrid retrieval plus citation loop beats FAISS on RAGTruth","Reranked dense plus sparse retrieval tops FAISS across five LLMs","eSapiens: citation-checked RAG improves relevance over vector search","Strict grounding counters FAISS hallucination edge in QA system","Dense plus sparse with reranking outranks FAISS in enterprise QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the RAGTruth question subset and the TRACe metrics—including human accuracy ratings and the hallucination measure—are reliable and representative of real enterprise workloads, but the paper reports point estimates without confidence intervals, annotator agreement, or the number of test questions.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid retrieval plus citation loop beats FAISS on RAGTruth","Reranked dense plus sparse retrieval tops FAISS across five LLMs","eSapiens: citation-checked RAG improves relevance over vector search","Strict grounding counters FAISS hallucination edge in QA system","Dense plus sparse with reranking outranks FAISS in enterprise QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1161,"prompt_tokens":846,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":462,"tokens_out":315,"duration_ms":3449,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:37:44.127114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on a large, publicly fixed sample of RAGTruth questions with multiple annotators, pre-registered scoring, and confidence intervals; the claim is settled if eSapiens no longer shows higher context relevance and accuracy, or if its hallucination rate is not higher than FAISS. A negative result would also follow if removing the citation-verification loop leaves answer quality unchanged.","supporting_citations":[],"review_version":1}