{"id":"376384cb-1c93-4eb0-8ee3-58ec6557adb7","arxiv_id":"2602.07086","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In a 631-case enterprise benchmark, iterative retrieval (CoRAG) achieved 10.29% exact match vs 7.45% for standard RAG on combined SQL/API generation with hybrid documentation, while no-RAG gives 0% exact match.","lead":"This paper evaluates three retrieval-augmented generation (RAG) variants for translating natural language into SQL queries and REST API calls in an enterprise setting, using SAP Transactional Banking documentation. It finds that RAG is essential (0% exact match without retrieval) and that iterative CoRAG slightly outperforms standard RAG when database and API documentation are mixed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoRAG's advantage in hybrid settings may stem from a larger retrieval budget, not from iterative decomposition","rationale":"The reader's weakest assumption was self-generation circularity: GPT-5 generated the reference outputs, which could bias exact-match comparisons. That is a real concern, but the retrieval-budget confound is more immediate and more directly undermines the paper's central comparative claim. The paper's own method description creates an internal inconsistency: all variants are said to use a fixed top-5 retrieval, yet CoRAG's iterative design necessarily retrieves more than five chunks. Without equating the total retrieved context, the statistically significant CoRAG advantages in hybrid documentation cannot be attributed to iterative query decomposition. This is testable by a simple top-k control, and it affects the headline conclusion about retrieval-policy design. The self-generation issue may also matter, but it is secondary because it would need to interact with retrieval strategy to explain relative differences. The RAG-essential finding (0% exact match without RAG) is robust and does not depend on this confound. Therefore the overall verdict remains CONDITIONAL, but with an additional specific experimental requirement beyond the reader's original concern.","tokens_in":12145,"tokens_out":5569,"duration_ms":62854,"concrete_test":"Run standard RAG with top-k set to the average total number of chunks CoRAG actually retrieves (e.g., top-15) on the 631 test cases, keeping all other settings identical. If standard RAG top-15 matches or exceeds CoRAG's hybrid exact-match accuracy (10.29% combined, 15.32% SQL), the CoRAG advantage is due to a larger retrieval budget, not iterative decomposition. Alternatively, constrain CoRAG to a single retrieval round (5 chunks total) and compare; if the advantage disappears, the same conclusion holds. Report the mean and median number of retrieved chunks per variant to verify the imbalance.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparative claim—that CoRAG's iterative query decomposition provides robustness under hybrid documentation—is confounded by an unequal retrieval budget. §3.1 states 'Retrieval size was fixed at top-5 chunks' for all variants, but CoRAG's implementation retrieves top-5 chunks per sub-query and repeats until completion (§3.1). Standard RAG and Self-RAG receive at most 5 chunks total, while CoRAG can access 10, 15, or more. In hybrid contexts, this matters: standard RAG's single top-5 may be dominated by API documentation, whereas CoRAG can retrieve database schema chunks in later rounds. The reported exact-match advantages (combined p=0.0006; SQL-hybrid p=0.0026) could therefore be an artefact of greater context size rather than a better retrieval policy. The paper does not report the total number of retrieved chunks per variant, and future-work (§6.2) only acknowledges CoRAG's extra LLM calls, not this retrieval-budget imbalance. Thus the paper's conclusion that 'iterative query decomposition outperforms both top-k retrieval and binary relevance filtering' is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates three RAG variants—standard RAG, Self-RAG, and CoRAG—for natural-language-to-SQL and REST-API-call generation in an enterprise setting based on SAP Transactional Banking documentation. It introduces a new 631-case dataset (346 SQL, 285 API) built with an adapted APIGen pipeline, using GPT-5 to generate reference outputs, humanizing inputs, validating outputs against mock SQLite/Postman environments, and expert review. The authors report 18 experimental configurations across database-only, API-only, and hybrid documentation contexts. The central claims are: (i) RAG is functionally essential, with 0% exact-match accuracy for the no-RAG baseline across all tasks and execution accuracy rising to 71–79% with retrieval; and (ii) CoRAG provides statistically significant robustness advantages in hybrid documentation settings, especially for SQL generation (e.g., combined-task EM 10.29% vs. 7.45% for standard RAG, p=0.0006; SQL-hybrid EM 15.32% vs. 11.56%, p=0.0091/0.0026). The discussion frames retrieval-policy design as a primary determinant of production viability.","tokens_in":12453,"tokens_out":2538,"duration_ms":30218,"significance":"If the comparative findings hold, this would be a useful contribution: it addresses a real gap by jointly evaluating SQL and API generation under heterogeneous documentation, and it provides an execution-validated, publicly released test set. The RAG-essential result appears robust because the no-RAG baseline achieves 0% exact match while retrieval lifts execution accuracy to roughly 70–79%, measured against mock servers rather than only string matching. The paper also ships a concrete dataset and reports extensive experimental configurations. However, the headline comparative claim about CoRAG is currently not adequately supported. Two confounds—an unequal retrieval budget and reference outputs generated by the same model family being scored—undermine the attribution of the observed exact-match differences to iterative query decomposition. The practical significance of the paper therefore depends on resolving these issues.","major_comments":[{"comment":"The central CoRAG comparison is confounded by an unequal retrieval budget. Section 3.1 states that retrieval size was fixed at top-5 chunks, but CoRAG retrieves top-5 chunks per sub-query and repeats until the LLM signals completion, so it can access 10, 15, or more chunks while standard RAG and Self-RAG receive at most 5 chunks total. The reported hybrid-SQL and combined-task advantages (Table 2: 15.32% vs. 11.56%; Table 3: 10.29% vs. 7.45%) could therefore reflect a larger context window rather than a better retrieval policy. The paper does not report the total number of retrieved chunks per variant. To support the conclusion that iterative decomposition itself helps, the authors need an ablation that controls retrieval budget (e.g., standard RAG with top-10 or top-15 chunks, or CoRAG limited to 5 total chunks) and should report the average number of chunks used by each variant.","section":"§3.1, §4.2, §4.3"},{"comment":"The evaluation is partially self-referential. The reference SQL queries and API calls used for exact-match scoring were generated by GPT-5, and GPT-5 is also the backbone model whose RAG variants are being scored. Since exact match is a strict single-reference string comparison, any stylistic or structural bias in the reference set toward GPT-5 output patterns will disproportionately benefit variants whose outputs most resemble GPT-5's own generation style. This is especially relevant for CoRAG, whose multi-step reasoning may produce outputs more aligned with the generator's internal style. The limitations section (§5.5) lists single-reference and single-backbone issues but does not acknowledge this circularity. The authors should either add human-validated alternative references, evaluate with execution-based semantic equivalence as the primary metric, or analyze the degree of stylistic","section":"§3.3 step 1, §4.2, §5.5"},{"comment":"The statistical significance claim rests on very small numbers of additional correct cases. In the combined task, CoRAG's exact-match advantage over standard RAG is 10.29% vs. 7.45%; over the full 631-case set this difference corresponds to roughly 18 cases. The SQL-subset difference of 15.90% vs. 11.56% corresponds to about 15 of 346 cases. These are plausible, but the paper reports only p-values from a paired t-test on binary outcomes and does not report the number of discordant pairs, effect sizes, or confidence intervals. Given the small absolute counts and the confounding in the previous comment, the authors should report McNemar's test details, exact confidence intervals, and ideally bootstrap results to show stability before the superiority claim can be accepted.","section":"§4.3, Table 3"}],"minor_comments":[{"comment":"The abstract states CoRAG's SQL-hybrid exact match as 15.32%, while §4.3 reports the SQL subset as 15.90%. The relationship between these numbers and the combined-task SQL subset should be clarified.","section":"Abstract vs. §4.2 and §4.3"},{"comment":"The description of Self-RAG's relevance threshold (≥0.2) is vague—what exactly is being thresholded, and how was this value chosen? The paper should provide the prompt or a precise definition.","section":"§3.1"},{"comment":"Paired t-tests on binary outcomes are unconventional; McNemar's test is more standard for paired binary data. At minimum, the reported p-values should be accompanied by the discordant-pair counts.","section":"§3.5"},{"comment":"The limitations section is missing the two most consequential threats to validity identified above: the self-generated reference outputs and the unequal retrieval budget. These should be acknowledged explicitly.","section":"§5.5"},{"comment":"The API-only and hybrid baselines are identical (0.00, 20.31, 0.00, 28.45), which makes sense because baseline uses no retrieval, but the table could state this to avoid confusion.","section":"§4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The RAG-essential finding seems solid and is a useful practical result. The paper's more distinctive claim—that CoRAG's iterative decomposition is superior under hybrid documentation—is currently undercut by the retrieval-budget confound and the self-generated-reference issue. I would advise the editor that the paper is salvageable with a controlled-budget ablation and a more careful treatment of the reference-output circularity, but as written the central comparative conclusion is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing to know: this paper gives you a useful, reproducible demonstration that RAG is not optional for enterprise SQL/API generation—no-retrieval baselines score 0% exact match across the board, while RAG lifts execution accuracy to 70%+. That finding holds up. But the headline that CoRAG is more robust under hybrid documentation is not yet supported; the experimental design leaves two uncontrolled variables that could explain the effect.\n\nWhat's new: a combined SQL+API benchmark with 631 cases on SAP Transactional Banking, plus a systematic comparison of standard RAG, Self-RAG, and CoRAG under DB-only, API-only, and hybrid documentation. The dataset is released, and execution validation against mock servers is a real step beyond pure text matching. The paper is clearly written and the setup is described carefully enough to reproduce.\n\nSoft spots, in order of severity. First, the retrieval budget confound. Section 3.1 says all variants use top-5 chunks, but CoRAG retrieves top-5 per sub-query and iterates until completion. Standard RAG gets exactly 5 chunks total; CoRAG can access 10, 15, or more. In hybrid settings, standard RAG's single shot may be dominated by API docs while CoRAG fetches schema chunks in later rounds. That alone could explain the exact-match gains in the SQL hybrid condition. The paper never reports total retrieved chunks per variant, and future work only acknowledges CoRAG's extra LLM calls. This is load-bearing for the 'iterative decomposition outperforms' conclusion.\n\nSecond, the reference outputs are generated by GPT-5, the same model family being scored. That makes the exact-match comparisons, including the CoRAG advantage, partially self-referential. Execution accuracy against mock servers is an external check, so the RAG-essential finding is not circular. But the fine-grained p-values (0.0006, 0.0026) rest on a small number of additional correct cases against a GPT-5-generated gold set. The limitations section lists other threats but not this one.\n\nThird, minor: the statistical tests treat each test case as independent, but cases come from a small number of business objects; there may be clustering effects.\n\nOverall, I'd treat this as a conditional accept: the main empirical finding is valuable and the dataset is a contribution, but the comparative claims need either a re-run with matched retrieval budgets or substantially hedged language. Worth sending to peer review. I'd cite the dataset and the RAG-essential result.","headline":"Solid proof that RAG is essential for enterprise SQL/API generation, but the CoRAG advantage is confounded by unequal retrieval budgets and GPT-5-generated references.","tokens_in":12902,"tokens_out":2863,"would_cite":true,"duration_ms":26568,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Without retrieval augmentation, an LLM produces zero correctly matching enterprise SQL or API calls; with iterative retrieval, it maintains accuracy even when database and API documentation are mixed.","keywords":["retrieval-augmented generation","text-to-SQL","API call generation","CoRAG","Self-RAG","enterprise software","LLM evaluation","documentation retrieval"],"falsifier":"A concrete way to test the central claim: replace the GPT-5-generated reference outputs with references written by a different model family or by human experts, then rerun CoRAG vs standard RAG on the same 631 cases. If CoRAG's statistically significant exact-match advantage disappears or reverses, the claimed robustness is an artifact of self-generated references; if it persists, the claim survives.","tokens_in":12054,"feed_emoji":"⚙️","tokens_out":4160,"duration_ms":43241,"temperature":0.7,"pith_summary":"The paper sets out to show that for enterprise systems, retrieval-augmented generation is a functional prerequisite: with no retrieval, a state-of-the-art LLM scores 0% exact match on every task, while any RAG variant jumps execution accuracy to 71–79%. Among the three retrieval policies tested, CoRAG — which decomposes a user request into iterative sub-queries and retrieves iteratively — is the most robust in hybrid documentation settings, keeping SQL exact-match at 15.32% when standard RAG falls to 11.56% (p<0.01). The combined SQL-plus-API task shows the same pattern: CoRAG reaches 10.29% exact match versus 7.45% for standard RAG (p=0.0006), driven mainly by SQL. These findings matter because production assistants must handle mixed schema and endpoint documentation, and retrieval-policy choice is shown to be as important as the documentation itself.","feed_headline":"RAG lifts enterprise SQL/API accuracy from 0% to 79%","feed_subtitle":"Iterative CoRAG retrieval keeps SQL exact matches stable when schemas and API docs mix.","key_machinery":"CoRAG (Chain-of-Retrieval Augmented Generation) is the central mechanism: it iteratively generates a sub-query from the user's request, retrieves top-5 documentation chunks for that sub-query, performs an aggregation-and-sufficiency check, and repeats until the LLM signals completion. This is contrasted with standard RAG (single top-5 retrieval, one generation call) and Self-RAG (top-5 retrieval followed by LLM-based relevance filtering). The paper's evaluation harness also matters: 631 execution-validated test cases derived from SAP Transactional Banking, with SQL and API gold outputs, scored by exact match, component match, execution accuracy, endpoint retrieval, and classification accurac","core_discovery":"The central discovery is that retrieval policy, not just retrieval presence, determines whether enterprise structured generation survives documentation heterogeneity. In database-only or API-only contexts the three RAG variants perform comparably, but when database schemas and API specifications are both retrievable, standard RAG and Self-RAG degrade by 20–30% relative on SQL exact match, while CoRAG's iterative query decomposition holds essentially flat. CoRAG's advantage is statistically significant in the combined task (p=0.0006 vs standard RAG) and in hybrid SQL (p<0.01 vs both other variants). The paper attributes this to SQL's compositional structure: schema elements are spread across","pith_inferences":["The paper's single-reference, single-backbone design likely understates all variants' true correctness; a multi-reference or semantic-equivalence metric could shrink the gap between exact match and execution accuracy, and might change the CoRAG-vs-RAG ranking.","Because the gold references were generated by the same model family (GPT-5) used as the generation backbone, the exact-match advantage could partly reflect stylistic self-similarity rather than functional superiority; re-testing with references written by humans or by a different model family would isolate this effect.","CoRAG's robustness in hybrid settings suggests a testable extension: feeding CoRAG only the final aggregated context, or using a fixed number of retrieval rounds, could separate the benefit of iteration from the benefit of query decomposition."],"forward_implications":["Any production natural-language assistant for enterprise SQL/API should treat retrieval infrastructure as mandatory, not optional, since the no-RAG baseline is 0% exact match.","When documentation is homogeneous or partitionable by task type, standard RAG is sufficient; when it is mixed, CoRAG's extra two-to-three LLM calls are justified by statistically significant robustness.","Execution accuracy (68–79%) far exceeds exact match (10–15%), so production systems should use execution success as the primary correctness signal and treat structural match as a secondary check.","Unified assistants that route between SQL and API should consider task-specific retrieval policies, e.g., CoRAG for SQL, standard RAG for API, because API generation showed no significant variant differences.","Current systems remain best deployed in assisted mode: even the best configuration reaches only ~15% exact match, so interactive refinement and human fallback are necessary."],"fun_headline_variants":["Retrieval strategy decides SQL/API success in mixed docs","CoRAG beats RAG in combined SQL/API generation","Retrieval policy, not presence, boosts enterprise code gen","Mixed schemas and API specs? CoRAG holds SQL accuracy","RAG strategy key for enterprise SQL and API calls"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper assumes its self-generated gold SQL/API calls are neutral ground truth, but since the same LLM family writes both the references and the candidates, stylistic self-similarity could inflate exact-match differences.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval strategy decides SQL/API success in mixed docs","CoRAG beats RAG in combined SQL/API generation","Retrieval policy, not presence, boosts enterprise code gen","Mixed schemas and API specs? CoRAG holds SQL accuracy","RAG strategy key for enterprise SQL and API calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1242,"prompt_tokens":808,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":552,"tokens_out":434,"duration_ms":4495,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:06:02.705978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete way to test the central claim: replace the GPT-5-generated reference outputs with references written by a different model family or by human experts, then rerun CoRAG vs standard RAG on the same 631 cases. If CoRAG's statistically significant exact-match advantage disappears or reverses, the claimed robustness is an artifact of self-generated references; if it persists, the claim survives.","supporting_citations":[],"review_version":1}