{"id":"698d3e95-524d-4e36-8783-3639aa39f06f","arxiv_id":"2502.15723","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding more schema descriptions and examples to retrieved documents improves table retrieval but increases SQL query errors in a Text2SQL model, with the best balance at medium document richness.","lead":"This paper tests how much database detail should be included in the reference material given to an AI that writes SQL queries. It finds that adding more descriptions and examples helps the system find the right tables, but causes the AI to produce more wrong queries once the material gets too long.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RAG evaluation in S5.1–6.1 measures score dispersion without comparing retrieved tables to ground-truth tables, so the claim that richer documents improve retrieval is unverified and may undermine the central trade-off.","rationale":"The reader's weakest assumption already identifies the most load-bearing concern, and I agree. The paper's central narrative depends on an independently validated retrieval benefit; absent any ground-truth comparison of retrieved tables, the RAG half of the trade-off story is unestablished. The score-spread metrics are heuristics, not measures of correct retrieval, and the non-standard DCG weakens them further. Nevertheless, the end-to-end execution comparison (Figure 7) could in principle support the preference for moderate document enrichment, so the paper is not beyond repair. The correct verdict remains CONDITIONAL: the authors should redo the RAG evaluation with recall-based or intersection-based metrics, report statistical uncertainty on all headline comparisons, and clarify how queries requiring more than k tables are handled. This would test whether the central claim actually lands.","tokens_in":14809,"tokens_out":3920,"duration_ms":38170,"concrete_test":"For the 719-query subset, parse each ground-truth SQL to extract the set of referenced tables using the SPIDER schema. For each of the seven document sets, run the RAG retriever with k=3 and compute recall@3: the fraction of queries where all referenced tables appear among the top-3 retrieved documents, while separately noting queries that reference more than 3 tables. Compare recall@3 trends across document sets to the reported range/StdDev/DCG trends. Also recompute DCG with binary relevance (1 if a referenced table is in the top-k, 0 otherwise) to see whether the claimed RAG improvement survives a ground-truth-based evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 7 items 1 and 5) rests on two legs: richer documents help RAG retrieve better, and larger prompts hurt Text2SQL. The first leg is supported only by 'broader spread of relevance scores' (range, standard deviation, and a non-standard DCG that uses raw RAG scores as 'relevance'). Sections 5.1 and 6.1 never compare retrieved tables to the tables actually referenced in the SPIDER ground-truth SQL. A retriever could assign widely separated scores to entirely wrong tables and still look good on these metrics. Moreover, the DCG formula uses the raw score (where lower is better) in place of a graded relevance label, so the reported 'downward DCG trend' is just a weighted average of that score and says nothing about ranking quality. Consequently, the statement 'increasing content size benefits the RAG component independently' is unsupported, and the causal interpretation of why Document Set 4 is best overall lacks its retrieval half. Even the end-to-end execution results (Figure 7) are presented without exact numbers, error bars, or significance tests, making the 'best overall' claim fragile as well.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how the amount of content in schema documents affects a RAG + Text2SQL pipeline built on a subset of the SPIDER benchmark. Seven document sets are constructed by adding normalized schemas, example INSERT statements, and textual descriptions. The authors evaluate retrieval with spread-based scores (range, standard deviation, DCG) and evaluate end-to-end SQL generation with similarity, component-match, and execution-comparison metrics. They conclude that moderately enriched documents (Document Set 4) give the best overall trade-off, and that richer documents help retrieval but hurt generation.","tokens_in":15012,"tokens_out":3842,"duration_ms":34144,"significance":"If the empirical claims were fully supported, the paper would offer practical, low-cost guidance for practitioners choosing how much schema detail to store in a RAG corpus, and it would quantify a real trade-off between retrieval quality and prompt size. The study is clearly described, uses a standard benchmark, and releases code. Its current significance is limited by evaluation choices that do not measure retrieval against ground truth and by the absence of statistical rigor; these are fixable within the manuscript's scope.","major_comments":[{"comment":"The RAG evaluation metrics (AvgRange, AvgStdDev, AvgDCG) quantify only the spread of raw relevance scores among the top-k retrieved documents. They never compare the retrieved tables with the tables actually referenced in the SPIDER ground-truth SQL, even though every query has a ground-truth SQL that the authors already use for end-to-end evaluation. As a result, the central claim in §7 item 1 that increasing the content size of documents benefits the RAG component independently is not supported: a retriever assigning well-separated scores to entirely wrong tables would appear to improve on all three metrics. Please add a ground-truth-based retrieval measure (e.g., recall@3 or MRR over tables referenced by the ground-truth SQL) for each document set.","section":"§5.1, §6.1"},{"comment":"The thresholds for normalized edit distance (0.5), embedding cosine similarity (0.85), and fuzzy matching (75) are described as determined empirically, but no justification, sensitivity analysis, or cross-metric agreement is reported. The end-to-end results, including Figure 7, contain no exact percentages, error bars, confidence intervals, or significance tests. The §7 item 5 claim that Document Set 4 exhibited the best overall performance is therefore based on visual inspection of point estimates, and the drop between sets 4 and 5 could be within noise. Please report per-metric numeric values, a threshold-ablation study, and either confidence intervals or paired statistical tests across queries.","section":"§5.2"},{"comment":"The DCG formula uses the raw RAG relevance score (where lower scores are more relevant) in place of rel(i), which in standard DCG must be a higher-is-better graded relevance label. With this substitution, the reported downward trend of AvgDCG is just a weighted average of the raw distance scores and carries no information about ranking quality. Either define a proper DCG using binary or graded ground-truth relevance, or remove DCG from the retrieval evaluation.","section":"§5.1, Eq. (1)"},{"comment":"The document-set design changes multiple factors at once: sets 1–4 vary schema formatting and the number of INSERT examples, while sets 5–7 additionally add textual descriptions. The claim that the utility of one-shot examples diminishes when documents already contain substantial content is therefore not identifiable from these data, since the effect of the INSERT statement is confounded with the presence of textual descriptions. A factorial design (crossing description presence and number of examples) would be needed to support this finding.","section":"§4.2, §6.2, §7 item 3"}],"minor_comments":[{"comment":"The prompt template example appears truncated; it ends inside the assistant header rather than showing the completed output format.","section":"§4.4.1"},{"comment":"The text references key thresholds where performance degradation occurs, but the only numeric thresholds in the paper are the similarity thresholds in §5.2; the degradation thresholds should be stated explicitly or rephrased.","section":"Abstract, §6.1"},{"comment":"Figures 2–4 would benefit from error bars or per-query variance, even for the retrieval metrics.","section":"Figures 2–4"},{"comment":"Table 1 lists domains and query counts but not the number of tables per domain; adding that column would help readers assess retrieval difficulty.","section":"Table 1"},{"comment":"The reference Observations on Building RAG Systems for Technical Documents has an unusual author string and should be formatted consistently with the other entries.","section":"References"},{"comment":"Minor typos: LLama should be Llama, and the title on the first line has unintended spacing (RAG-T EXT 2SQL S YSTEM).","section":"Title and text"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an empirical study with modest novelty; its value depends on the reliability of the comparisons. I would encourage the editor to treat the Section 7 claims as tentative until retrieval is validated against ground truth. The code link is included but was not executed in review; if the repository is in good shape, that is a point in the authors' favor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take this as a conditional pass. The paper asks a real question and the qualitative answer — moderate schema enrichment beats maximal detail for RAG+Text2SQL — is plausible and worth taking seriously. The authors also do a clean job of building seven document sets on SPIDER and they ship code, which is more than many papers at this level do. I also appreciate that the limitations section is honest about the single-dataset and truncation issues. So there is genuine value here.\n\nThe soft spots are where the stress-test note lands. The retrieval half of the central claim is not actually supported. Section 5.1 defines 'better RAG' as a broader spread of relevance scores, and Figures 2-4 report range, std dev, and DCG of those scores. But the paper never checks whether the retrieved tables match the tables referenced in the SPIDER ground-truth SQL. A retriever that confidently ranks three wrong tables would look great on these metrics. And the DCG as used is not a ranking-quality measure: they plug the raw score (where lower is better) into a formula designed for graded relevance, so the 'downward DCG trend' is just a weighted average of distance scores. That does not tell you anything about ranking quality. So when Section 7 says richer documents benefit RAG independently, that statement is unverified.\n\nThe end-to-end side is weaker than it should be. Figure 7 shows the execution-match percentages, but there are no exact numbers, no error bars, no significance tests, and the similarity thresholds in Section 5.2 are all empirically chosen without sensitivity analysis. The conclusion also mentions zero-shot, one-shot, and two-shot learning as if they were tested, but the paper only varies the number of example INSERT statements inside documents — that is not a few-shot learning experiment.\n\nNone of this kills the practical direction. It does mean the specific recommendation that Document Set 4 is best is fragile. A referee should ask for retrieval validation against ground truth, exact numbers, and at least basic error bars. The code and data being public makes that feasible.\n\nMy recommendation: send it to peer review, but flag that the retrieval metric needs real validation before the central trade-off can be believed. It is a useful empirical datapoint, not a definitive result.","headline":"Plausible practical trade-off, but the retrieval evaluation never checks ground truth, so the central claim rests on a weak first half.","tokens_in":15570,"tokens_out":2357,"would_cite":false,"duration_ms":22340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that richer schema documents improve RAG retrieval on their own but degrade the Text2SQL generation as prompts grow, putting the best end-to-end balance at a moderate content level.","keywords":["Text-to-SQL","Retrieval-Augmented Generation","document size","schema documents","hallucination","prompt engineering","SPIDER benchmark","trade-off"],"falsifier":"Run the same seven document sets and compare each query's top-3 retrieved tables against the tables referenced in the SPIDER ground-truth SQL; if the richer document sets do not retrieve the correct tables more often than Document Set 4, or if score spread and table-retrieval accuracy diverge, the claimed retrieval benefit and the trade-off narrative are unsupported.","tokens_in":14578,"feed_emoji":"🗄️","tokens_out":10400,"duration_ms":91524,"temperature":0.7,"pith_summary":"The paper argues that document content in a retrieval-augmented generation (RAG) plus natural-language-to-SQL (Text2SQL) pipeline has opposite effects on the two halves of the system. Making each per-table schema document larger, by adding textual descriptions, example INSERT statements, or both, improves the RAG retriever's ability to spread out and rank its top candidates, but the same extra content inflates the prompt given to the Text2SQL model and makes its generated SQL less faithful. Using a subset of the SPIDER benchmark, the authors build seven document sets ranging from bare CREATE TABLE statements to schema-plus-description documents and measure both retrieval and end-to-end SQL quality. Their central finding is that the combined system peaks at Document Set 4, a normalized schema plus two example inserts, and that adding descriptive text beyond that point raises hallucination rates and lowers query similarity and execution match. A reader should care because it challenges the common assumption that richer retrieval context is always better for deployed systems.","feed_headline":"Richer documents: good for RAG, bad for SQL","feed_subtitle":"In a SPIDER-based benchmark, moderately enriched schema documents beat heavily detailed ones end-to-end.","key_machinery":"The central object is the document set, a corpus in which each document represents one SQL table; the authors create seven progressively richer variations of the same SPIDER subset. The mechanism that carries the argument is the coupling between document content and the two stages: the same content moves the spread of RAG relevance scores, measured by range, standard deviation, and DCG, and the size and noise of the prompt consumed by the Text2SQL model. This coupling is what produces the observed curve, where retrieval improves with content while end-to-end SQL quality peaks earlier and then declines.","core_discovery":"The paper's central claim is that document size and the performance of a retrieval-augmented generation (RAG) + Text2SQL system form a trade-off, not a monotonic curve. Retrieval quality by the paper's measures improves as document content increases: the range and standard deviation of top-3 relevance scores grow, and DCG improves as lower scores indicate higher relevance. Yet when the same documents are passed into the prompt of the Text2SQL model, performance rises only through Document Set 4, a normalized CREATE TABLE statement plus two example INSERT queries, and then drops when textual table and column descriptions are added in Document Set 5. The drop shows up as lower similarity under normalized edit distance, embedding matching, and fuzzy matching, more column-selection and operation mismatches, and worse agreement on database execution. The paper's conclusion is that what serves the retriever is not what serves the generator, so document design must be optimized for the combined system rather than for RAG alone.","pith_inferences":["The paper's first half is only about score spread; a quantitative table-recall comparison against the ground-truth SQL would turn 'retrieval improves with content' into a directly testable claim rather than an inferred one.","If large-prompt hallucination is the mechanism, then a dynamic content filter that retrieves a rich document but passes only the referenced table and its relevant columns to the generator should dominate any fixed document set; the paper proposes the idea in its optimization section but does not test it.","For stronger Text2SQL models, the degradation point should move to larger document sizes; that gives a concrete calibration experiment for the paper's trade-off."],"forward_implications":["Deployments that tune the RAG layer alone, by maximizing retrieval score spread, will not automatically get better SQL; they may be selecting a configuration that hurts the generator.","Schema documents with a normalized CREATE TABLE plus one or two example INSERT statements appear to be a low-cost sweet spot for an 8B-parameter SQL model on SPIDER-like schemas.","If the degradation is prompt-size driven, then raising the number of retrieved documents should be accompanied by shortening each document to keep total prompt size constant.","Hallucination analysis should be reported per SQL component, because the paper shows column selection is the most error-prone component under larger prompts."],"supporting_citations":[{"why":"Supplies the SPIDER benchmark, the ground-truth SQL queries and table schemas that all document sets and evaluations are built from.","marker":"Yu et al. [2019]"},{"why":"Provides SQLCoder-8B, the Text2SQL model whose generated queries are compared against ground truth across document sets.","marker":"Defog [2024]"},{"why":"Provides FAISS, the vector store used for embedding search in the RAG retriever.","marker":"Douze et al. [2024]"},{"why":"Supplies the all-MiniLM-L12-v2 embedding model used for both retrieval and SQL query similarity comparisons.","marker":"Aperdannier et al. [2024]"},{"why":"Provides evidence that excessive or irrelevant context degrades LLM outputs, supporting the hallucination-side interpretation of larger prompts.","marker":"Liu et al. [2023]"}],"fun_headline_variants":["RAG wants more, SQL wants less: find the sweet spot","Document size trade-off: RAG gains, SQL suffers","Too much context hurts Text2SQL: moderate wins","Balancing act: RAG likes detail, SQL likes lean","More docs, more noise: RAG-Text2SQL balance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on treating the spread of RAG relevance scores, range, standard deviation, and DCG, as proof that retrieval quality improved; the paper never checks whether the retrieved tables are the tables the correct SPIDER query actually references.","fun_headline_variants_meta":{"raw":{"variants":["RAG wants more, SQL wants less: find the sweet spot","Document size trade-off: RAG gains, SQL suffers","Too much context hurts Text2SQL: moderate wins","Balancing act: RAG likes detail, SQL likes lean","More docs, more noise: RAG-Text2SQL balance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1470,"prompt_tokens":972,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":413}},"tokens_in":588,"tokens_out":498,"duration_ms":4453,"temperature":1.0,"reasoning_tokens":413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:08:42.332958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same seven document sets and compare each query's top-3 retrieved tables against the tables referenced in the SPIDER ground-truth SQL; if the richer document sets do not retrieve the correct tables more often than Document Set 4, or if score spread and table-retrieval accuracy diverge, the claimed retrieval benefit and the trade-off narrative are unsupported.","supporting_citations":[],"review_version":1}