{"id":"4f7fc9b8-e330-4b45-8925-d8d5768cd721","arxiv_id":"2507.09592","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"THOR is an enterprise Text-to-SQL system using LLM agents, schema injection, and a five-retry self-correction loop, evaluated only on seven anecdotal prompts.","lead":"THOR is described as a multi-agent system that turns plain-English business questions into read-only SQL queries against relational databases. The paper is a system write-up with anecdotal smoke tests and no quantitative evaluation, useful only as a snapshot of how LLM agent stacks are being commercialized.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retry-loop success-rate claim rests entirely on seven hand-picked prompts with no baseline identification; the paper's own Limitations concede the core failure modes the evaluation claims to solve.","rationale":"The reader's weakest_assumption (seven selected prompts are not a representative or quantitative benchmark) is the same load-bearing concern I identify, so I agree. My attack is more pointed: Table 1 is not even a smoke test with counts; it is a qualitative anecdote table. The paper's own Limitations section explicitly walks back the strongest claim by conceding unvalidated complex queries, scalability limits, and potential contextually incorrect results. That makes the central 'verified' and 'significantly improving success rate' claims self-undermined. I checked the Appendix SQL: several baselines use clearly bogus generated SQL (e.g., Prompt 2 Product X groups by 'invitation_token' to answer a channel question, and Prompt 3 Product Y computes income_per_mile with distance in meters not converted—both are plausible stumbles, but they are presented anonymously with no evidence that these are state-of-the-art or representative agents; they could be weak or deliberately misconfigured baselines. The comparison is therefore not a fair evaluation of 'baseline agents' in general. The paper does describe a plausible enterprise system with concrete guardrails (read-only, retry loop, rating), which has practitioner value, but as a scientific claim of superior success rate it is unsupported. Given the paper's own admission that its contribution is not a novel SQL algorithm, the lack of a quantitative comparison is fatal to the empirical claim. I recommend REJECT, consistent with the reader, though a revised version with a proper benchmark could change that.","tokens_in":6857,"tokens_out":1681,"duration_ms":17330,"concrete_test":"Reconstruct the evaluation on a reproducible benchmark: fix the 7 prompts, the schemas (logistics/retail/Chinook), and named baselines (e.g., GPT-4o with schema-only prompts, DAIL-SQL-style exemplars) and record, for N>=50 runs per prompt per system: success rate (defined as exact-match or execution-pass with human-verified correctness), number of retries until success, and rows-returned correctness. If THOR's success rate minus the best baseline is not statistically significant (e.g., overlapping 95% CIs), the 'significantly improving' claim fails. Also run a broader sample of >=100 enterprise-style prompts to test representativeness of the 7.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim ('Self-healing pipeline... significantly improving the success rate over baseline agents') is unsupported by the evaluation in Section 5. Table 1 presents only qualitative summaries of 7 prompts, with anonymous baselines ('Agent-A/B/C') and no definition of 'success rate'—no per-prompt pass/fail counts, no retry counts, no baseline success counts, no error bars, and no dataset. The prompts themselves are cherry-picked to showcase the system's heuristics: each of the four cited challenges (nonexistent status codes, unit conversion, fuzzy match, future-dated rows) corresponds to a specific guardrail the paper advertises, so success on these prompts is near-tautological. Moreover, the paper's Limitations section (Section 7) admits the system is 'not yet fully validated' on complex relational logic, has schema scalability issues, and may produce 'logically valid but contextually incorrect results'—directly undermining the abstract's 'verified' promise. The internal contradiction between the claimed 'verified, read-only SQL analytics' and the admitted possibility of unvalidated, contextually wrong results is a correctness risk, not just an evaluation gap. Because the paper explicitly disclaims a novel SQL generation algorithm, the architecture (LLM + schema injection + retry + guardrails) is a composition of known components; without a quantitative comparative evaluation, no empirical contribution is established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents THOR, a multi-agent text-to-SQL system comprising a supervisor agent, dynamic schema retrieval, a SQL generation agent, a self-correction and rating loop, and a result interpretation agent. The authors claim that THOR transforms natural-language questions into \"verified, read-only SQL analytics\" and that its bounded retry loop \"significantly improving the success rate over baseline agents.\" The evaluation consists of seven hand-picked prompts compared against three anonymous baseline agents, plus a logistics case study with estimated time savings. The paper explicitly states that it does not propose a novel SQL generation algorithm, positioning its contribution instead as a holistic system architecture.","tokens_in":7061,"tokens_out":3689,"duration_ms":40289,"significance":"If the empirical claims were substantiated, THOR would be a practically useful enterprise text-to-SQL system. However, the manuscript provides no quantitative metrics anywhere: Table 1 is a qualitative summary of seven prompts, Table 2 is an unsubstantiated set of time-savings estimates, and Appendix A contains only the seven prompts and generated SQL. The central claim that THOR \"significantly improving the success rate\" over baselines is not supported by any measurement. The abstract's promise of \"verified\" results is directly contradicted by Section 7, which concedes that the system \"may produce logically valid but contextually incorrect results.\" Because the paper explicitly disclaims a novel algorithm, the contribution reduces to a composition of known components, and the evaluation is too thin and circular to establish an empirical contribution. The authors deserve credit for clearly describing the architecture and for including the full prompts and SQL listings in an appendix, but these do not compensate for the absence of a rigorous evaluation.","major_comments":[{"comment":"The claim in the Contributions section that the self-healing pipeline \"significantly improving the success rate over baseline agents\" is unsupported. There is no definition of success rate, no per-prompt pass/fail counts, no retry counts, no baseline success rates, no error bars, and no dataset. The seven prompts in Appendix A are hand-picked and each appears to target a specific THOR guardrail (e.g., nonexistent status codes, unit conversion, fuzzy matching, future-dated rows), so success on these prompts is close to a tautology. The authors need to provide a quantitative evaluation on a standard benchmark such as Spider or on a realistically sampled enterprise workload, with identified baselines and statistical significance.","section":"Section 5, Table 1 and Appendix A"},{"comment":"The abstract says THOR produces \"verified, read-only SQL analytics\" and Section 6 says users \"receive verified, read-only results,\" but Section 7 admits that the system \"may produce logically valid but contextually incorrect results\" and that performance on complex relational logic is \"not yet fully validated.\" This is not a minor wording issue: the core value proposition of verified correctness is explicitly disclaimed. The authors should either remove the word \"verified\" or provide a precise operational definition and evidence that the system meets it.","section":"Abstract and Section 6 vs. Section 7"},{"comment":"The time-savings estimates in Table 2 (e.g., \"↓ 90%\", \"↓ 100%\", \"↓ 70–90%\") are presented without any methodology, measurement, or source. There is no description of how manual effort was measured, how AI query times were recorded, or whether the percentages are averages, anecdotal, or hypothetical. If these are illustrative estimates, they should be explicitly labeled as such; if they are experimental findings, the protocol must be described.","section":"Section 5.1.3, Table 2"},{"comment":"The evaluation is circular: the only judge of output quality is THOR's own rating loop, and the baselines are anonymous. Success is therefore measured against the system's internal notion of a good answer, not against an external standard. The authors should include human evaluators or a gold set of SQL queries with execution-based equivalence checks, and the baselines should be named with their configurations.","section":"Section 4.2 and Appendix A"},{"comment":"The paper explicitly states that \"Rather than proposing a novel SQL generation algorithm, its primary contribution lies in the holistic system architecture.\" This places a higher, not lower, burden on the empirical evaluation: a composition of known components (LLM, schema injection, retries, guardrails) needs a rigorous comparison or ablation to demonstrate that the architecture, heuristics, and retry loop add measurable value. The current smoke test with seven prompts cannot bear this weight.","section":"Section 1 and Section 3"}],"minor_comments":[{"comment":"The phrase \"theTHOR Module\" appears to have a missing space; it should read \"the THOR Module.\"","section":"Abstract"},{"comment":"The prompt \"What are the different channels through which drivers learned about someone\" is unclear; the word \"someone\" appears to be a placeholder or a typo for \"a job\" or \"driving opportunities.\" The ambiguous wording makes it difficult to judge the correctness of the generated SQL.","section":"Appendix A, Prompt 2"},{"comment":"The comment in the THOR SQL listing, \"-- id en ti ca l query ; eSapiens adds a na rr at iv e e x p l a n a t i o n,\" contains spacing artifacts and a semicolon misuse; this should be copyedited for readability.","section":"Appendix A, Prompt 4"},{"comment":"Several references are incompletely formatted, such as \"Jing-Jing Gao and 1 others\" and \"Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingqing Yao, Shanelle Roman, and 1 others\"; the author lists should be completed.","section":"References"},{"comment":"The \"AI Agent\" column mixes units (minutes per query, hours per week, etc.) and the \"Time Saved\" column is not defined consistently; consider providing a clearer breakdown of measurement units and assumptions.","section":"Section 5.1.3, Table 2"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as a product description for eSapiens rather than a research paper. The empirical evaluation is essentially absent, the baselines are anonymous, and the central claims are contradicted by the authors' own limitations section. The paper could in principle be revised into a system description with a rigorous benchmark, but as submitted it does not meet the standard for a research contribution in a database venue. I also note that all authors are from the commercial entity eSapiens, which may be relevant for the editor in assessing potential conflicts of interest regarding the promotional nature of the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the THOR paper. What you should know: this is a vendor's systems description, not a research paper. It says as much in the related work: \"Rather than proposing a novel SQL generation algorithm, its primary contribution lies in the holistic system architecture.\" Take that seriously.\n\nWhat is good: the architecture is described clearly and the system is a sensible assembly of known components — a supervisor agent, on-demand schema retrieval, a bounded retry loop, read-only enforcement, and a natural-language interpretation layer. The write-up is honest enough to include a Limitations section that admits complex queries aren't validated, schema scaling is unsolved, and domain misreadings are possible. For a practitioner building an enterprise text-to-SQL pipeline, this is a useful reference design.\n\nThe problem is the evaluation, and it is a load-bearing problem. The central claim is that the retry loop \"significantly improves the success rate over baseline agents,\" but Table 1 is qualitative, the baselines are anonymous, and the entire test set is seven hand-picked prompts in the appendix. There is no definition of success rate, no per-prompt counts, no retry counts, no error bars, and no released data. The prompts look constructed to trigger each advertised heuristic, so success on them is near-tautological. Table 2 is an unsubstantiated estimate of time savings. The abstract promises \"verified\" results; the Limitations section says the system may return \"logically valid but contextually incorrect results.\" That mismatch is a real over-claim.\n\nThe reader's REJECT is right. The paper does not establish an empirical contribution. The architecture may be useful, but that is a systems note, not a scientific result. I would not desk-reject a demo or an industry-track submission outright; the description alone could help people, but a serious research venue would need quantitative evaluation, identified baselines, and a realistic workload. I would not cite it for any measured result. I wouldn't bring it to a reading group either — there is nothing new to argue about.\n\nRecommendation: if this lands on your desk as a research paper, return it. If you are chairing an applied systems track, you could send it out with a strong request for real evaluation and release of prompts and results. As it stands, it is a marketing document with appendices.","headline":"A clear, honest systems description with an evaluation that is a smoke test: seven hand-picked prompts and anonymous baselines cannot support the 'significantly improving success rate' claim.","tokens_in":7628,"tokens_out":2801,"would_cite":false,"duration_ms":31275,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"THOR claims a self-healing, read-only pipeline turns natural-language questions into correct SQL answers, beating baseline agents on realistic enterprise queries.","keywords":["Text-to-SQL","natural language to SQL","multi-agent LLM pipeline","self-correction retry loop","read-only SQL guardrail","schema retrieval","enterprise data analytics","semantic guardrails"],"falsifier":"A concrete check: run the same seven prompts with the Self-Correction & Rating loop disabled; if the success rate does not drop, the central claim that the loop significantly improves the success rate has no support. A broader falsifier would be a sample of unseen enterprise queries: if THOR's success on the seven prompts is much higher than on a larger, independently drawn set, the smoke test does not generalize.","tokens_in":6613,"feed_emoji":"📊","tokens_out":7218,"duration_ms":71290,"temperature":0.7,"pith_summary":"This paper presents THOR, a multi-agent system that turns natural-language business questions into SQL queries against relational databases, runs those queries read-only, and returns concise natural-language answers. Its central claim is that a bounded self-correction loop—triggered by empty results, execution errors, or low-quality output, with up to five regeneration attempts—salvages queries that simpler text-to-SQL agents get wrong. The authors test seven prompts spanning logistics and retail schemas and report that THOR handled unit conversion, fuzzy genre matching, future-dated rows, and multi-step joins while baseline agents returned empty tables or wrong values. If the approach holds, it offers a practical path for non-technical users to interrogate live enterprise data without writing SQL or waiting days for reports. The paper positions the contribution as a system architecture rather than a new generation algorithm.","feed_headline":"Retry loop turns business questions into safe SQL, paper claims","feed_subtitle":"Enterprise users could query live data read-only; seven smoke-test prompts show recoveries baseline agents missed.","key_machinery":"The load-bearing mechanism is the bounded Self-Correction & Rating loop: after each SQL execution, the system checks for empty results, execution errors, or a low quality score, and if any trigger fires it uses LLM feedback to regenerate the SQL and re-execute, with a hard cap of five attempts to prevent infinite loops. Around that loop sit two supporting mechanisms: dynamic Schema Retrieval, which pulls relevant table and column metadata into the prompt, and a read-only guardrail that forces generated statements to be single SELECT queries. The loop is what turns a single-shot text-to-SQL system into one that can notice and repair its own mistakes, and it is the component the paper credits for the improved success rate over baseline agents.","core_discovery":"The paper's central discovery, stated on its own terms, is that a decoupled orchestration/execution pipeline with dynamic schema injection and a retry-and-rating loop is enough to make LLM-generated SQL reliable on realistic business queries. THOR routes each request through a Supervisor Agent, injects relevant table and column metadata into the prompt, and has a SQL Generation Agent emit single-statement SELECT queries behind a read-only guardrail. When a query returns nothing, errors, or scores poorly, the Self-Correction & Rating loop re-generates the SQL and re-executes it up to five times. In the seven smoke-test prompts, the loop recovered from failures such as nonexistent status codes, unit mismatches, exact-match genre filters, and silently included future timestamps, producing correct results where baseline agents did not. The paper also claims that an appended narrative and the read-only, schema-aware design make the results immediately usable and compliant for enterprise settings.","pith_inferences":["The paper leaves the retry loop's actual success-rate uplift unquantified; a direct extension would be to measure execution success with the loop disabled versus enabled over the same seven prompts and report the per-attempt recovery distribution.","The same loop suggests a general reliability recipe for LLM-based tools: spend bounded extra LLM calls to verify and repair outputs, rather than seeking a single perfect generation, and characterize that trade-off as cost per recovered query.","The future-work items—synonym dictionaries, graph-aware join planning, and a nightly regression suite from real execution errors—point toward a testable path: if those are added, the system should improve on a held-out set of enterprise queries with complex joins, a prediction not tested here."],"forward_implications":["Non-technical staff could run ad-hoc analytic questions directly against live databases, cutting routine reporting and diagnostics from hours to minutes.","Enterprises can enforce compliance by construction: all queries are read-only and unauthorized column requests are refused, so the natural-language interface does not weaken data governance.","The bounded retry loop converts common failure modes—empty tables, wrong units, fuzzy text, future dates—into recoverable events rather than dead ends.","Because the pipeline uses a stock LLM for generation, the architecture can adopt newer models and improve without changing the orchestration layer."],"supporting_citations":[{"why":"Shows that GPT-4 with condensed exemplars pushes execution accuracy beyond 85%, the generation capability THOR wraps.","marker":"(Gao et al., 2023)"},{"why":"Establishes LLM-based text-to-SQL as competitive with careful schema and example selection, the baseline THOR builds on.","marker":"(Rajkumar et al., 2022)"},{"why":"Introduces execution-guarded constrained decoding, a precedent for using execution feedback to eliminate faulty SQL.","marker":"(Scholak et al., 2021)"},{"why":"Survey that identifies agent-style pipelines, cost control, and compliance as production challenges THOR addresses.","marker":"(Aiweiya and Li, 2024)"},{"why":"Early work rewarding executable queries, the lineage behind execution-aware generation.","marker":"(Zhong et al., 2017)"}],"fun_headline_variants":["Retry loop rescues failed SQL queries","AI SQL self-corrects in enterprise smoke tests","Up to five retries turn LLM SQL into valid queries","Retry and rate loop boosts SQL query success","Self-fixing SQL model passes seven business tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole enterprise-readiness claim rests on seven hand-picked prompts in Appendix A being representative of real ad-hoc enterprise workloads; the paper's Limitations section itself concedes that complex relational logic and very large schemas are not yet validated.","fun_headline_variants_meta":{"raw":{"variants":["Retry loop rescues failed SQL queries","AI SQL self-corrects in enterprise smoke tests","Up to five retries turn LLM SQL into valid queries","Retry and rate loop boosts SQL query success","Self-fixing SQL model passes seven business tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1267,"prompt_tokens":920,"completion_tokens":347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":273}},"tokens_in":536,"tokens_out":347,"duration_ms":3984,"temperature":1.0,"reasoning_tokens":273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:52:02.953276+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: run the same seven prompts with the Self-Correction & Rating loop disabled; if the success rate does not drop, the central claim that the loop significantly improves the success rate has no support. A broader falsifier would be a sample of unseen enterprise queries: if THOR's success on the seven prompts is much higher than on a larger, independently drawn set, the smoke test does not generalize.","supporting_citations":[{"cited_title":"Singularities of scattering matrix","cited_arxiv_id":"2308.05389","evidence_quote":"Shows that GPT-4 with condensed exemplars pushes execution accuracy beyond 85%, the generation capability THOR wraps."},{"cited_title":"Convergence of a generalized Riemann problem scheme for the Burgers equation","cited_arxiv_id":"2401.03714","evidence_quote":"Survey that identifies agent-style pipelines, cost control, and compliance as production challenges THOR addresses."}],"review_version":1}