{"id":"91d18b88-a160-4135-b8b3-c6bb8f36fdd1","arxiv_id":"2412.07214","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TiInsight automates cross-domain data exploration by summarizing database schemas with LLMs, generating SQL, and recommending charts, achieving 86.3% execution accuracy on Spider and state-of-the-art results on Bird.","lead":"This paper presents TiInsight, a system that uses large language models to automatically explore relational databases by turning natural language questions into SQL queries and charts. It reports near-top accuracy on standard text-to-SQL benchmarks and claims production deployment at PingCAP, which makes it a concrete example of LLMs automating data analysis workflows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HDC's contribution is never isolated: without an ablation or schema-linking recall analysis, the reported Spider/Bird numbers do not establish that LLM schema summaries, rather than GPT-4's raw capability, drive the claimed cross-domain generalization.","rationale":"The reader's weakest_assumption identifies exactly the same issue: HDC fidelity is unmeasured, and the absence of an ablation makes the central mechanism untestable. I agree. The concern is load-bearing because the paper's novelty and the generalization claim rest on HDC. Without an ablation or schema-linking recall, the reported accuracy could be explained by GPT-4's prior knowledge and the refinement loop. This is not an internal inconsistency; the numbers may be correct. But the paper's own §6.2 attributes the gain to HDC without evidence. The proposed test is feasible with the released code and would settle the issue. Credit: the paper does open-source the APIs and evaluation code, so this is checkable. The verdict remains CONDITIONAL; if the ablation reveals HDC is inert, it should be REJECT or at least substantially revised.","tokens_in":22825,"tokens_out":7243,"duration_ms":72809,"concrete_test":"Run the open-source TiSQL pipeline on the Spider dev set in three configurations: (1) full HDC as in the paper; (2) raw schema only—real table/column names, types, and foreign keys, no LLM summaries; and (3) full HDC minus table-relationship inference. Report EX for each, plus schema-linking recall (fraction of gold tables/columns present in the filtered set). If (2) is within ~1–2 EX points of (1), or if schema-linking recall is low for multi-join questions, the paper's central claim that HDC drives cross-domain accuracy is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HDC—LLM-generated hierarchical schema summaries—enables cross-domain text-to-SQL without fine-tuning, evidenced by 86.3% EX on Spider and 60.98% on Bird. The paper never isolates HDC's contribution. §6.2 attributes the results 'primarily to HDC' (p. 9, col. 1) but reports no ablation (e.g., TiSQL with raw schemas, or with HDC components disabled), no HDC fidelity metric, and no schema-linking recall (whether gold tables/columns survive the coarse-grained and fine-grained filters in Algorithm 4). The margin over strong baselines is small (86.3 vs 86.2 for DAIL-SQL+GPT-4 on Spider; on Bird, R-VES is below CodeS-15B), so without an ablation the observed accuracy may reflect GPT-4's strong zero-shot capability and the self-refinement chain rather than HDC. Moreover, the HDC construction itself is lossy by design—map-reduce summarization of wide tables, random 3-row sampling per column (§3.1.1), and coarse-grained relationship search limited to top-20 similar tables (§3.1.2) can omit or distort the very columns and joins needed for correct SQL. If the summaries drop a key column or a foreign-key relationship, subsequent retrieval and generation cannot recover it, undercutting both the absolute numbers on complex schemas and the claimed generalization to new domains. The paper's user study (20 participants, 60 minutes, no statistical rigor) does not mitigate this; it addresses overall usability, not schema-summary fidelity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TiInsight, an LLM-based end-to-end system for cross-domain exploratory data analysis. The system has four stages: hierarchical data context (HDC) generation, which uses LLMs to produce column, table, and database summaries; question clarification and decomposition; TiSQL, a zero-shot text-to-SQL method that performs coarse- and fine-grained schema filtering and then applies a self-refinement chain; and TiChart, a rule-based visualization recommender. The central empirical claims are that TiSQL achieves 86.3% execution accuracy on Spider with GPT-4, that it achieves state-of-the-art execution accuracy on Bird (60.98% test EX), that it shows an 82.3% execution success rate in six months of public testing at PingCAP, and that a user study finds TiInsight comparable to human experts. The authors also report open-sourcing the system's APIs and evaluation code.","tokens_in":23165,"tokens_out":5405,"duration_ms":50179,"significance":"If the central claim holds, this is a practically significant systems result: it suggests that LLM-generated hierarchical schema summaries can make a no-fine-tuning text-to-SQL pipeline competitive with specialized and fine-tuned systems across domains, in a deployed enterprise setting. The paper's strengths are its evaluation on external public benchmarks (Spider and Bird), its production deployment evidence, and its release of code and APIs, which makes independent verification feasible. However, the contribution attribution is not yet established: the paper never isolates HDC's effect, and the accuracy margins over strong baselines are small. Because the cross-domain generalization claim rests on HDC, the missing ablation and missing schema-linking recall analysis are load-bearing gaps rather than cosmetic issues.","major_comments":[{"comment":"The paper attributes TiSQL's benchmark results 'primarily to HDC' (p. 9, col. 1) and the abstract credits HDC for cross-domain generalization, but no experiment isolates HDC's contribution. There is no ablation comparing TiSQL with HDC against TiSQL with raw schemas (or with HDC components disabled), and no schema-linking recall statistic showing that gold tables and columns survive the coarse-grained and fine-grained filters of Algorithm 4. Given the small margins over strong baselines (86.3% vs. 86.2% for DAIL-SQL + GPT-4 on Spider; 60.98% vs. 60.37% for SFT CodeS-15B on Bird test EX; and R-VES below CodeS-15B), the observed accuracy could plausibly reflect GPT-4's zero-shot capability and the self-refinement chain rather than HDC. This is load-bearing for the central cross-domain generalization claim and should be addressed with an ablation and/or a recall analysis before the attribution is made.","section":"§6.2, Tables 1 and 3"},{"comment":"HDC is lossy by construction: column summaries are generated on groups of 40-80 columns with only three randomly sampled rows per column (§3.1.1), table relationships are identified from at most the top-20 similar tables (§3.1.2), and TiSQL subsequently filters to the top-N retrieved tables (Algorithm 4). If a key column or foreign-key relationship is omitted or distorted in these summaries, downstream retrieval and SQL generation cannot recover it. The paper reports no fidelity metric for the summaries and no recall of gold tables/columns/joins on Spider or Bird. Since the stated motivation is complex real-world schemas with thousands of columns, the paper should measure information loss (e.g., gold-table/column recall after HDC + TabColFilter) or at least compare against a raw-schema condition on schemas that fit in the context window.","section":"§3.1.1, §3.1.2, and Algorithm 4"},{"comment":"The 82.3% figure reported from six months of public testing is an 'execution success rate' only, i.e., the generated SQL executed without error. It is not a measure of whether the SQL answered the user's question, and the paragraph does not define what counts as an EDA task, how failures were classified, or what baseline was used. As presented, this number does not support the abstract's claim of success on 'real-world EDA tasks' beyond syntactic executability; it should be relabeled or supplemented with a correctness or user-confirmation measure.","section":"§6.2, public testing paragraph"},{"comment":"The user study has 20 participants, 60 minutes, three datasets, and no statistical significance testing; the text states TiInsight is 'comparable' to the human expert based on overlapping confidence intervals. With this sample size and no paired tests or effect sizes, the claim in the abstract of 'remarkable performance compared to human experts' is not supported. At minimum, report per-participant paired comparisons and a significance test (or explicitly recast the finding as a descriptive usability observation).","section":"§6.3, Figures 6 and 7"}],"minor_comments":[{"comment":"The 'state-of-the-art performance on the Bird dataset' claim should be qualified: on R-VES (Table 4), TiSQL + GPT-4 (56.06) is below SFT CodeS-15B (56.73), so SOTA holds only for EX, not for R-VES.","section":"Abstract and Table 4"},{"comment":"Cost reporting is internally inconsistent and hard to read: the text says GPT-4 costs 'about $300' while Figure 10 shows values of 260.0 and 280.0, and Table 5's pricing cell is garbled (e.g., '$1.25 / 1M cached$0.15 / 1M tokens').","section":"§6.5 and Figure 10"},{"comment":"The statement that 'the prompt used in this section is not included in the paper' is a reproducibility gap for the core HDC component; because the code is open-sourced, please either include the prompt in an appendix or point to the exact file and version in the repository.","section":"§3.1"},{"comment":"The random three-row sampling per column is not described with a seed or stability check; a short sensitivity analysis would assure readers that summaries do not depend on the particular sampled rows.","section":"§3.1.1"},{"comment":"Latency results are reported as averages without variance or a per-component breakdown, making it hard to assess stability under concurrency.","section":"Figure 9"},{"comment":"Several references reuse the ACM template defaults (e.g., 'Received 20 February 2007') and contain 'Retrived' typos; these should be cleaned before camera-ready.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems paper with external benchmark evaluation, a deployment narrative, and open-sourced code. The main risk is attribution: the paper credits HDC for the gains without any ablation or schema-linking recall analysis, and the margins over strong baselines are small. Both concerns are addressable with the released code, so I would ask the authors for an HDC ablation and a table/column recall study rather than rejecting. I have no concerns about misconduct; the reported numbers are consistent with published baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible industrial EDA system paper with plausible numbers on Spider and Bird, open-sourced APIs, and a real production deployment. The specific contribution the authors claim—HDC, the hierarchical schema summaries—is never isolated, and the 'SOTA' phrasing is softer than the paper's own tables. I'd send it to review with a request for an ablation.\n\nWhat's actually new: the HDC design (map-reduce summarization from columns to tables to databases, stored in a vector store and used for retrieval) plus its integration into a four-stage pipeline: clarification/decomposition, TiSQL text-to-SQL, and TiChart rule-based visualization. The benchmark measurements are new and look internally consistent—86.3% on Spider is 0.3 below DAIL-SQL+GPT-4+SC and 0.1 above DAIL-SQL+GPT-4, and the Bird numbers line up with the baselines they cite. The paper also reports cost and latency, which is rare and useful. The evaluation is against external public benchmarks, so there's no circularity problem; the hand-set hyperparameters aren't fitted to the results.\n\nSoft spots, in order of importance. First, no ablation of HDC. The paper says the gains come 'primarily' from HDC (Section 6.2), but there is no run with raw schemas, no disabled HDC components, and no recall check on whether the gold tables/columns survive the coarse- and fine-grained filtering. Given the margin over the strongest baseline is 0.1–0.3 points, GPT-4's zero-shot ability plus the self-refinement chain could explain the result just as well. This is the load-bearing weakness. Second, the 'SOTA' language overstates: on Spider, DAIL-SQL+GPT-4+SC does 86.6; on Bird R-VES, CodeS-15B does 56.73 while TiSQL does 56.06. The authors acknowledge both in the text, so it's a wording problem, but it will annoy referees. Third, the user study (20 participants, 60 minutes, no significance tests) is a usability check, not evidence for 'remarkable performance compared to human experts'. That claim should be toned down. Fourth, minor: HDC is lossy by construction—random 3-row samples per column and top-20 similar tables for relationships—so a fidelity metric or analysis of failure cases would strengthen the paper. Also, the references contain formatting typos, but that's trivial.\n\nWho it's for: anyone building LLM-based data analysis systems or doing text-to-SQL with wide schemas. It deserves a proper peer review, not a desk reject, with the ablation and recall measurement made conditions.","headline":"Credible industrial EDA system with plausible benchmark numbers, but HDC's contribution is never isolated and the 'SOTA' claim overstates the paper's own tables.","tokens_in":23783,"tokens_out":3175,"would_cite":true,"duration_ms":29611,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TiInsight claims that LLM-generated hierarchical schema summaries, not fine-tuning, are enough to reach near-state-of-the-art text-to-SQL and cross-domain exploratory data analysis.","keywords":["Exploratory data analysis","Text-to-SQL","Large language models","Database schema summarization","Hierarchical data context","SQL generation","Data visualization","Cross-domain generalization"],"falsifier":"A schema with a deliberately misleading or abbreviated column (for example, dod standing for day-over-day) where the column summary omits the expansion; if TiSQL cannot produce the correct SQL even though the raw schema comment contains the meaning, the HDC faithfulness assumption is falsified.","tokens_in":22592,"feed_emoji":"📊","tokens_out":6080,"duration_ms":56361,"temperature":0.7,"pith_summary":"TiInsight is an end-to-end SQL-based exploratory data analysis system that claims to generalize across data domains without per-domain fine-tuning. The paper's central claim is that condensing a complex database schema into three levels of LLM-written summaries—columns, tables, and databases—plus vector-database retrieval, gives a text-to-SQL component that reaches 86.3% execution accuracy on Spider with GPT-4 and state-of-the-art results on Bird. The same pipeline is reported to answer real-world EDA questions with an 82.3% execution success rate during six months of public use. The paper argues this matters because existing EDA and text-to-SQL methods are benchmark-oriented, need labeled data for fine-tuning, and cannot handle wide tables with hundreds of columns or ambiguous user questions.","feed_headline":"LLM schema summaries hit 86.3% text-to-SQL accuracy on Spider","feed_subtitle":"A no-fine-tuning pipeline reaches state-of-the-art on Bird and answers real EDA queries 82.3% of the time.","key_machinery":"The central object is the hierarchical data context (HDC): three nested LLM-generated summaries of a database, at column, table, and database level, stored in a vector database. Column summaries are produced in parallel groups with retrieved domain knowledge and sample rows; table summaries combine a map-reduce description of wide tables with two-stage relationship detection; database summaries are built from the top-N most-connected tables and their inferred entities. HDC carries the argument because it replaces the raw schema in prompts, letting TiSQL retrieve only relevant tables and columns, fit inside context windows, and transfer across domains without fine-tuning.","core_discovery":"The paper's discovery is that an open-world EDA system can be built from four LLM-driven stages—hierarchical data context (HDC) generation, question clarification and decomposition, text-to-SQL (TiSQL), and rule-based visualization (TiChart)—where HDC is the load-bearing novelty. HDC uses LLMs to summarize each column, then each table, then the database as a whole, storing these summaries in a vector database. TiSQL retrieves relevant tables and columns from these summaries, builds few-shot chain-of-thought prompts, and applies a self-refinement chain using EXPLAIN and execution feedback. In the paper's evaluation, this design without fine-tuning reaches 86.3% execution accuracy on Spider with GPT-4, outperforms DAIL-SQL on Bird, and achieves 82.3% execution success on real-world tasks.","pith_inferences":["If HDC summaries are the reason for the gains, the same summarization-and-retrieval pattern could be lifted into any NL2SQL or data-agent pipeline, not just TiSQL; a direct ablation comparing HDC against raw-schema prompting on the same backbone would test this.","The 82.3% real-world execution success rate is not directly comparable to benchmark accuracy because it counts executability, not correctness against gold SQL; a deployment study with labeled answers would be the natural next measurement.","The map-reduce summarization scheme suggests a scaling rule of thumb for enterprise schemas: grouping of 40 columns for GPT-4 and 80 for other models, with retrieval over summaries, could be adopted as a standard preprocessing step for wide tables.","Since HDC generation is one-time per database, its latency cost can be amortized; the paper's latency numbers imply that the economic bottleneck of such systems is LLM token cost, not summarization time."],"forward_implications":["A text-to-SQL component that never sees the raw schema can match or beat fine-tuned systems: TiSQL reaches 86.3% on Spider and state-of-the-art execution accuracy on Bird with GPT-4.","Cross-domain EDA becomes feasible without per-domain retraining: HDC summarization plus vector retrieval is domain-agnostic, and the system reports 82.3% execution success across finance, retail, and gaming tasks in public testing.","Ambiguous or multi-step user questions can be handled by a clarification-and-decomposition stage that turns them into explicit sub-questions for SQL generation.","Chart selection can be rule-based and LLM-checked, so end-to-end text-to-visualization requires no specialized visualization language or reinforcement learning."],"supporting_citations":[{"why":"Spider benchmark provides the main test set for TiSQL's 86.3% execution accuracy claim.","marker":"[76]"},{"why":"Bird benchmark provides the dev and test sets and the R-VES metric for the state-of-the-art claim.","marker":"[45]"},{"why":"DAIL-SQL is the primary LLM-based baseline that TiSQL is compared against and slightly exceeds in the paper.","marker":"[33]"},{"why":"DIN-SQL supplies the decomposed in-context learning and self-correction baseline that TiSQL contrasts with.","marker":"[57]"},{"why":"SFT CodeS is the fine-tuned open-source system that TiSQL compares against on Bird, supporting the no-fine-tuning claim.","marker":"[43]"},{"why":"Chat2Query is the prior LLM-based EDA system used as a baseline in the user study and the system TiInsight builds from.","marker":"[81]"}],"fun_headline_variants":["TiInsight: LLM-driven EDA hits 86.3% text-to-SQL accuracy","Hierarchical context boosts LLM SQL accuracy to 86.3% on Spider","Cross-domain EDA with LLMs: 86.3% SQL accuracy, no fine-tuning","TiInsight automates EDA with hierarchical LLM context and 86.3% SQL accuracy","LLM-based EDA system achieves 86.3% on Spider, SOTA on Bird"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the LLM-generated summaries of columns, tables, and databases preserve all the information needed to answer later questions, so anything the summaries omit or mistranslate is unrecoverable by the SQL stage.","fun_headline_variants_meta":{"raw":{"variants":["TiInsight: LLM-driven EDA hits 86.3% text-to-SQL accuracy","Hierarchical context boosts LLM SQL accuracy to 86.3% on Spider","Cross-domain EDA with LLMs: 86.3% SQL accuracy, no fine-tuning","TiInsight automates EDA with hierarchical LLM context and 86.3% SQL accuracy","LLM-based EDA system achieves 86.3% on Spider, SOTA on Bird"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4297,"prompt_tokens":1060,"completion_tokens":3237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":3117}},"tokens_in":676,"tokens_out":3237,"duration_ms":23824,"temperature":1.0,"reasoning_tokens":3117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:47.431823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A schema with a deliberately misleading or abbreviated column (for example, dod standing for day-over-day) where the column summary omits the expansion; if TiSQL cannot produce the correct SQL even though the raw schema comment contains the meaning, the HDC faithfulness assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Spider benchmark provides the main test set for TiSQL's 86.3% execution accuracy claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DIN-SQL supplies the decomposed in-context learning and self-correction baseline that TiSQL contrasts with."}],"review_version":1}