{"id":"2c1ec991-201f-452a-b60e-a015d1766068","arxiv_id":"2507.09588","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The abstract promises benchmark wins for the eSapiens RAG platform, but the appendix tables do not contain those numbers and show higher hallucination rates than the baseline.","lead":"eSapiens is an enterprise platform that combines document retrieval, vector search, and large language models into a no-code AI assistant, evaluated on legal question answering and generation quality. The paper's specific performance numbers are inconsistent between its abstract and its appendices, so the evidence for those claims is weak.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's headline numbers are absent from, and partly contradicted by, the appendix; the eSapiens-vs-FAISS comparison in Appendix B is confounded and shows higher hallucination for eSapiens.","rationale":"Reader verdict REJECT. I concur. The central claim rests on Appendix B, but Appendix B is not a controlled comparison and, where numbers are reported, it partially contradicts the abstract. This is not a question of differing external consensus; it is an internal-evidence failure. The missing 91.3% Top-3 and 23% factual-alignment figures could be recovered by a manuscript audit; showing they are absent or inconsistent would be decisive. If they were found in an omitted table, the confounding of prompt and architecture would still invalidate attribution. Thus no adjustment to the verdict is needed; the paper should remain rejected, or at best be classified as an unsubstantiated product document.","tokens_in":13059,"tokens_out":9045,"duration_ms":96467,"concrete_test":"Run a controlled replication of Appendix B on the same 100 RAGtruth questions, holding the LLM, temperature, system prompt, and post-processing identical across arms, and toggling only one eSapiens component at a time (query rewrite, hybrid BM25+dense retrieval, reranking, CO-STAR formatting), with two independent annotators blind to condition and pre-registered rubrics for context relevance and factual alignment, reporting bootstrap 95% CIs. In parallel, perform a text/table audit to verify whether '91.3% Top-3' and '23% factual alignment' appear in or can be recomputed from any appendix table; if they cannot, the abstract's quantitative claims fail regardless of the replication outcome.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that eSapiens beats a FAISS RAG pipeline, the reader must be able to attribute the Appendix B result to the platform's architecture. That attribution is blocked in two independent ways. First, the strong quantitative claims in the abstract are not present in the appendices. Appendix A compares only chunk sizes 500 and 1000 using Recall@k/Precision@k; there is no chunk size 512 condition, no Top-3 accuracy column, and no 91.3% value anywhere in the paper. Its prose concludes that chunk=1000 is 'better aligned with production needs,' undermining the abstract's '512 tokens yields highest retrieval precision.' Second, Appendix B's Table 5 does not report 'factual alignment' or a 23% improvement; instead it shows eSapiens with higher hallucination than the FAISS baseline for all five models (GPT-4o: 18.23% vs 8.75%) and lower completeness for all five models. The largest relative gain in the table is about 22.5% in Context Relevance for GPT-4o-mini, not factual alignment. Even that gain cannot be isolated: eSapiens differs from 'faiss+top-2+short' in query rewriting, CO-STAR prompting, multi-agent validation, and post-processing, none of which are ablated; human Accuracy is graded by authors without blinding and no confidence intervals are reported. The abstract's numerical claims are therefore neither derived from nor supported by the reported evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents eSapiens, an enterprise AI-as-a-service platform for retrieval-augmented generation over private documents and SQL data, with two main modules: DEREK for document QA and THOR for text-to-SQL. It describes the architecture, workflows, and security features, and reports two evaluation studies: a retrieval benchmark on LegalBench subsets comparing chunk sizes 500 and 1000, and a TRACe-based generation quality study on 100 RAGtruth questions comparing eSapiens with a FAISS-based pipeline across five LLMs. The abstract claims that a 512-token chunk achieves 91.3% Top-3 retrieval accuracy and that eSapiens delivers up to a 23% improvement in factual alignment. These headline claims are not present in, and are partly contradicted by, the appendices, and the system-versus-baseline comparison is confounded by untested differences in query rewriting, prompt format, validation, and post-processing.","tokens_in":13429,"tokens_out":6033,"duration_ms":59682,"significance":"The paper addresses a genuine industrial need for auditable enterprise RAG, and the architecture description may be useful to practitioners. The decision to use external evaluation datasets (LegalBench and RAGtruth) and to report raw numeric tables in appendices is a mild positive. However, the scientific contribution rests entirely on the evaluation claims, and those claims are not supported by the reported data. Appendix A does not include the 512-token or Top-3 condition, and Appendix B's Table 5 shows eSapiens with a higher hallucination rate and lower completeness than the FAISS baseline on every model tested, with only modest gains in context relevance and human-rated accuracy. No code, data, or artifact is provided, so the evaluation is not reproducible. If the claimed results were valid and properly attributed through ablations, the contribution would be noteworthy; on the current evidence, the central claims fail.","major_comments":[{"comment":"The abstract's headline claim 'a chunk size of 512 tokens yields the highest retrieval precision (Top-3 accuracy: 91.3%)' is absent from the experimental section. Appendix A tests only chunk sizes 500 and 1000 and reports Recall@k and Precision@k at k = 1, 2, 4, 8, 16, 50 (Tables 3 and 4); there is no chunk size of 512, no Top-3 column, and no 91.3% value anywhere. The prose in A.4 further states that chunk = 1000 is 'better aligned with production needs', which is not consistent with the abstract's implication that 512 is best. The headline result cannot be derived from the reported experiments.","section":"Abstract / Appendix A"},{"comment":"The abstract's claim of 'up to a 23% improvement in factual alignment' is not a measured quantity anywhere in the paper. Table 5 reports Completeness, Utilization, Context Relevance, pc hallucinated, and Accuracy, but no metric named 'factual alignment' appears. The table shows eSapiens with a higher hallucination rate than the FAISS baseline for all five models (e.g., GPT-4o: 0.1823 vs 0.0875; Claude 3.7: 0.1403 vs 0.0860) and lower Completeness for all five models (e.g., GPT-4o: 0.4307 vs 0.4450). Finding 1 in B.3 explicitly concedes that the FAISS baseline has lower hallucination. The only value close to 23% is the relative gain in Context Relevance for GPT-4o-mini (approximately 22.5%), not factual alignment. The body claim in Section 5.6 that eSapiens 'significantly outperforms traditional FAISS-based methods' is contradicted on these key dimensions.","section":"Appendix B, Table 5"},{"comment":"The comparison is confounded. The eSapiens pipeline differs from the 'faiss+top-2+short' baseline not only in retrieval but also in query rewriting, CO-STAR prompt format, multi-agent validation via LangGraph, and post-processing. No ablation separates these components, so the observed gains in Context Relevance (e.g., GPT-4o-mini: 0.3785 vs 0.3090) and Accuracy cannot be attributed to eSapiens' architecture. The evaluation uses only 100 randomly selected questions from RAGtruth, graded by the authors without a blinding protocol, and no confidence intervals, inter-annotator agreement, or statistical tests are reported, so the reported differences may be within noise.","section":"Appendix B.1-B.3"},{"comment":"The selective reporting is load-bearing. Section 5.6 states that eSapiens 'significantly outperforms traditional FAISS-based methods' in Context Relevance, Utilization, Accuracy, and Naturalness, but Appendix B does not show such consistent outperformance, and its own summary in B.4 frames the result as a tradeoff. The abstract and body present only the favorable side while the appendix reports the unfavorable results. Because the central claim of the paper is that eSapiens improves over a standard RAG pipeline, this mismatch between text and data is not a presentation issue but a substantive flaw.","section":"Section 5.6 / Appendix B"}],"minor_comments":[{"comment":"The DEREK description says documents are chunked into 1,000 tokens with a 150-token overlap, but Appendix A evaluates chunk sizes 500 and 1000 without stating the overlap for either condition; please specify the exact splitter configuration so the experiments are reproducible.","section":"Section 5.6 / Section 6"},{"comment":"Section 7.2 introduces 'eSapiens-claude-3.7-extended' as a fine-tuned variant, but Appendix B evaluates 'eSapiens-claude-3.7' and Section 6 lists Anthropic's Claude 3.7 Sonnet; clarify whether these refer to the same model and provide details if any fine-tuning was performed.","section":"Section 7.2 / Table 5"},{"comment":"The column header 'pc hallucinated' should be spelled out and defined (e.g., percentage of hallucinated tokens), and the scale for the Accuracy column should be stated, including whether higher scores always indicate better output.","section":"Table 5 / Appendix B.1"},{"comment":"The Appendix C examples refer to 'Product X', 'Product Y', and 'Product Z' without defining these products or describing the comparison protocol, which makes the SQL comparison difficult to interpret.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"This manuscript reads more like a product whitepaper than a scientific paper. The architecture sections are extensive, but the evaluation is sparse and internally contradictory: the abstract's headline numbers are absent from the appendices, and the appendix's own table shows the proposed system underperforming the baseline on hallucination and completeness. The authors would need a fundamentally redesigned evaluation with ablations, blinded human evaluation, and a fully specified baseline to make the central claim credible. I do not see a straightforward revision path short of that."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Skip this one unless you want an example of how to bury your own negative results. The abstract promises a 91.3% Top-3 accuracy at chunk size 512 and a 23% gain in factual alignment; the appendices test 500 and 1000 only, never report Top-3 or 91.3%, and show eSapiens with higher hallucination and lower completeness than a plain FAISS baseline on every model tested.\n\nWhat's good: The platform description is detailed and coherent, and the decision to use the TRACe framework for output quality is sensible. The appendices include the actual numbers, and they don't hide the fact that the FAISS baseline wins on hallucination and completeness. That transparency is worth a nod, even if the abstract undoes it. The THOR SQL examples are concrete, though ungrounded in any evaluation.\n\nSoft spots: The headline claims are unsupported. Appendix A compares chunk sizes 500 and 1000, so the 512 claim has no basis in the reported data. Appendix B shows eSapiens loses on hallucination (e.g., GPT-4o 18.2% vs 8.8%) and completeness across all five models; the claimed 23% improvement in factual alignment appears nowhere. The only favorable difference is context relevance, and that's confounded: eSapiens differs from the baseline in query rewriting, prompting, validation, and post-processing, none ablated. No error bars, no released code, and the human accuracy scores are unblinded. The paper also mentions a fine-tuned model, eSapiens-claude-3.7-extended, in the architecture diagram but gives no details and no evaluation.\n\nWho it's for: Possibly someone scoping enterprise RAG deployment options, but not a research audience. It's a product brochure with an evaluation section appended.\n\nRecommendation: I'd desk-reject this. The mismatch between the abstract and the appendix is a correctness issue, not a style issue. A referee might catch it, but the paper doesn't contain enough original research to justify the time. If the authors want a preprint to share with customers, fine, but it's not a research paper.","headline":"A product pitch that misreports its own results: the abstract claims numbers the appendix never shows and, in places, contradicts.","tokens_in":13899,"tokens_out":3007,"would_cite":false,"duration_ms":32390,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"eSapiens claims that a full-stack hybrid RAG platform, chunked at 512 tokens and routing across any LLM, achieves 91.3% top-3 retrieval precision and up to 23% better factual alignment than FAISS.","keywords":["retrieval-augmented generation","enterprise AI platform","hybrid vector retrieval","legal question answering","chunk size optimization","model-agnostic LLM","no-code orchestration","auditability"],"falsifier":"Re-run Appendix B's comparison with a blinded, externally graded set of several hundred questions and ablate hybrid retrieval, reranking, CO-STAR prompting, and post-processing one at a time; if the eSapiens advantage over FAISS disappears under any single-component removal, the central attribution claim fails. Separately, reproduce the abstract's 91.3% top-3 accuracy with 512-token chunks on the same legal corpora; failure to reproduce would undercut the headline metric.","tokens_in":12904,"feed_emoji":"⚖️","tokens_out":11328,"duration_ms":112397,"temperature":0.7,"pith_summary":"The paper is trying to establish that the bottleneck in enterprise AI is not the LLM but the missing middle layer between proprietary data and model output. eSapiens is proposed as that layer: unified document ingestion, hybrid retrieval, citation-aware answer generation, a SQL-style THOR agent, and no-code workflow orchestration wrapped in access control and audit logging. The evaluation claims that a 512-token chunk size yields the best retrieval precision (Top-3 accuracy 91.3%) and that the platform improves context relevance and factual alignment by up to 23% over a FAISS-based pipeline across five LLMs. If those claims hold, the practical consequence is that organizations in high-stakes fields can get more trustworthy, traceable answers without waiting for a single better model.","feed_headline":"Hybrid RAG stack claims 91.3% legal-retrieval precision","feed_subtitle":"An enterprise platform says 512-token chunks plus hybrid search beat a FAISS baseline on legal answers.","key_machinery":"The DEREK engine (Deep Extraction and Reasoning Engine for Knowledge) is the central mechanism: it chunks documents, embeds them in a hybrid Elasticsearch index that combines BM25 keyword search with dense vector similarity, rewrites queries through an LLM, retrieves top-50 snippets, and generates answers via the CO-STAR prompt format (a structured template specifying context, objective, style, tone, audience, and response constraints) with a LangGraph multi-agent validation loop. The THOR agent is its structured-data counterpart, turning natural-language questions into executable SQL with a self-correction and re-execution loop. Together they implement the paper's core identity: every generated answer is meant to be traceable to retrievable source passages, with retrieval quality and generation quality as separable, tunable variables.","core_discovery":"On its own terms, the paper claims that eSapiens is a complete AI-as-a-service platform whose value proposition rests on three connected pieces: proprietary data control, operational workflow automation, and model-agnostic LLM support. The retrieval experiments on legal corpora are offered as evidence for the chosen chunk size, and the generation-quality experiments report gains in context relevance, utilization, and human-rated accuracy relative to a FAISS-based DEREK pipeline. The authors read these results as showing that auditable, modular infrastructure rather than a single stronger model is what makes RAG trustworthy in legal and financial settings.","pith_inferences":["Editorial inference: the abstract's 512-token optimum and the appendix's preference for 1000-token chunks in production point to a recall-versus-precision tradeoff the paper does not resolve; a direct benchmark across 512, 1000, and intermediate sizes with error bars would settle it.","Editorial inference: because the evaluation compares whole pipelines, the individual contributions of query rewriting, hybrid retrieval, reranking, CO-STAR formatting, and post-processing are unidentified; an ablation study would show which component actually carries the reported gains.","Editorial inference: the THOR SQL examples suggest a separable research target, natural-language-to-SQL with schema awareness, unit conversion, and fuzzy matching, that could be benchmarked independently of document RAG quality.","Editorial inference: if auditable governance is the real differentiator, the decisive test is a compliance-focused adoption study with security and audit teams, not another retrieval or generation benchmark."],"forward_implications":["If 512-token chunks really deliver 91.3% top-3 accuracy on legal QA, then chunk-size tuning becomes a first-order retrieval lever independent of model choice.","If the up-to-23% factual-alignment gain reproduces across GPT-4o, Claude, Gemini, and DeepSeek, then platform-level retrieval and prompting decisions can matter as much as the underlying LLM.","If hybrid retrieval plus reranking is the source of the context-relevance advantage, the same pipeline should transfer to finance, insurance, and life-science document workloads.","The paper's own data show the FAISS baseline hallucinating less and covering more points, so a defensible corollary is that eSapiens wins on naturalness and intent alignment while losing on strict grounding, a deployment tradeoff rather than a clean sweep."],"supporting_citations":[{"why":"It defines retrieval-augmented generation, the paradigm eSapiens operationalizes for enterprise document QA.","marker":"(Lewis and et al., 2020)"},{"why":"It provides the LangChain orchestration layer eSapiens uses for prompt templating, tool calls, and workflow execution.","marker":"(Chase, 2022)"},{"why":"It presents LlamaIndex as an alternative modular RAG toolchain that eSapiens differentiates from on enterprise hardening.","marker":"(Liu, 2022)"},{"why":"It supplies ChatLaw, a legal-domain QA specialist that motivates eSapiens' broader multi-domain platform design.","marker":"(Gao and et al., 2023)"},{"why":"It supplies Lawyer-LLM, another legal-domain baseline that eSapiens claims to generalize beyond.","marker":"(Fan et al., 2023)"}],"fun_headline_variants":["Legal RAG hits 91.3% top-3 precision with 512-token chunks","eSapiens: secure RAG with 91.3% legal retrieval accuracy","RAG platform reports 23% factual alignment gain over FAISS","Enterprise RAG claims 91.3% legal precision, 23% factual gain","Auditable AI platform: hybrid RAG beats FAISS on legal data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that comparing the full eSapiens pipeline against a home-built FAISS pipeline on 100 author-graded RAGtruth questions, with no blinding and no component-by-component ablation, isolates the platform's contribution.","fun_headline_variants_meta":{"raw":{"variants":["Legal RAG hits 91.3% top-3 precision with 512-token chunks","eSapiens: secure RAG with 91.3% legal retrieval accuracy","RAG platform reports 23% factual alignment gain over FAISS","Enterprise RAG claims 91.3% legal precision, 23% factual gain","Auditable AI platform: hybrid RAG beats FAISS on legal data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000905,"raw_usage":{"total_tokens":3879,"prompt_tokens":915,"completion_tokens":2964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":2859}},"tokens_in":531,"tokens_out":2964,"duration_ms":23425,"temperature":1.0,"reasoning_tokens":2859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:52:24.341964+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Appendix B's comparison with a blinded, externally graded set of several hundred questions and ablate hybrid retrieval, reranking, CO-STAR prompting, and post-processing one at a time; if the eSapiens advantage over FAISS disappears under any single-component removal, the central attribution claim fails. Separately, reproduce the abstract's 91.3% top-3 accuracy with 512-token chunks on the same legal corpora; failure to reproduce would undercut the headline metric.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines retrieval-augmented generation, the paradigm eSapiens operationalizes for enterprise document QA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the LangChain orchestration layer eSapiens uses for prompt templating, tool calls, and workflow execution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It presents LlamaIndex as an alternative modular RAG toolchain that eSapiens differentiates from on enterprise hardening."},{"cited_title":"Pointwise modulus of continuity of the Lyapunov exponent and integrated density of states for analytic multi-frequency quasiperiodic $M(2, \\mathbb{C})$ cocycles","cited_arxiv_id":"2310.10472","evidence_quote":"It supplies Lawyer-LLM, another legal-domain baseline that eSapiens claims to generalize beyond."}],"review_version":1}