{"id":"ceb2cf83-b754-46f4-9a6c-189c88c64339","arxiv_id":"2505.21575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A locally deployed LLM-based distributed data system converts natural language to SQL, and its fine-tuned Llama-3.1-8B model reaches 86.9% exact match and 89.7% execution accuracy on the Spider dev set.","lead":"StreamLink is a distributed data system that uses locally fine-tuned language models to turn plain-English questions into SQL commands and then checks those commands for errors or injection attacks. The authors report that their best fine-tuned Llama-3.1-8B model reaches 89.7% execution accuracy on the Spider benchmark, a few points above the baselines they list.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'over 10%' Spider result hinges on an unspecified training split: §3.1 says only 'Spider' was used in fine-tuning, so dev-set leakage cannot be ruled out and no artifacts are provided to check.","rationale":"The reader's weakest assumption was that the Spider results were produced by an apples-to-apples, leakage-free evaluation. My reading agrees: the manuscript's own wording in §3.1 is ambiguous about which Spider split was used, and §4.1 gives no training/evaluation protocol details. This is not a manufactured concern; it is a missing-support issue in the exact place the central empirical claim rests. The numbers in Table 1 are internally plausible for a modern 8B model fine-tuned on Spider, and the paper includes a comparative table and security experiments, but none of that substitutes for the ability to verify the dev-set evaluation. I did not find an internal contradiction that would force rejection; the conditional verdict remains appropriate, with release of artifacts and a clear split specification as the condition. The most useful single check is to inspect the training corpus for dev leakage and reproduce Table 1 with only Spider train. If the authors provide that, the central claim can be confirmed or refuted directly.","tokens_in":9781,"tokens_out":6544,"duration_ms":72641,"concrete_test":"Request the exact fine-tuning corpus and training/evaluation scripts. Run a verification script that checks none of the 10,181 Spider dev questions or SQL statements generated from dev schemas appear in the training file, then re-run the SSQLG3.1-8B LoRA recipe using only the official Spider train split (spider_train.json) and re-evaluate with the official Spider evaluation script. If dev examples are found in the training corpus, or if EM/EA drop by more than 2 points when only the train split is used, the headline claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that the fine-tuning corpus 'combined the domain-specific dataset with publicly available NL-2-SQL datasets like WikiSQL and Spider.' Section 4.1 then evaluates on the Spider dev set, reporting 86.9% EM and 89.7% EA for SSQLG3.1-8B. The paper never states whether 'Spider' means the official Spider training split or the full dataset, and it does not describe how the template-augmented patent-domain data was mixed with Spider or whether any dev-set schemas or questions were used during augmentation. If the fine-tuning corpus included Spider dev examples, or SQL templates derived from dev-set schemas, the reported dev-set numbers are contaminated and the central 'over 10% improvement' claim collapses. Because no code, model weights, training corpus, or evaluation scripts are released, this cannot be checked externally. This is the most load-bearing uncertainty: the headline result depends entirely on an apples-to-apples, leakage-free evaluation that the manuscript neither specifies nor makes verifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"StreamLink is presented as an LLM-driven distributed data engineering system built on Spark and Hadoop, with a WebUI cluster, a Central Control Unit, and a locally deployed fine-tuned LLM (Llama-3.1-8B via LoRA) that translates natural-language requests into SQL. The paper reports 86.9% exact match and 89.7% execution accuracy on the Spider dev set, claims improvements of over 10% over baselines, describes zero-shot Llama-based SQL syntax and security checkers evaluated on a Kaggle SQL-injection dataset, and claims end-to-end retrieval from 180 million patents in under 6 seconds.","tokens_in":10036,"tokens_out":11938,"duration_ms":121261,"significance":"The practical scenario is relevant and the local-LLM privacy design is a sensible contribution, as is the use of standard public benchmarks (Spider dev, Kaggle SQL injection) for falsifiable headline numbers. The paper's strengths are its deployment scale (180 million patents, 15.3 TB, 280 cores), the template-based domain augmentation idea, and the zero-shot security-checker evaluation. Those strengths do not yet establish the central claims, because no code, model weights, prompts, training corpus, or evaluation harness are provided; the Spider fine-tuning split is unspecified, so dev-set leakage cannot be ruled out; and the latency claim is presented without a measurement protocol.","major_comments":[{"comment":"The training corpus for SSQLG3.1-8B is not specified at the level needed to validate the headline Spider result. Section 3.1 says only that the domain-specific dataset was combined with publicly available NL-2-SQL datasets like WikiSQL and Spider, and Section 4.1 evaluates on the Spider dev set without stating whether Spider means the official training split or the full dataset. Since the dev set is public, any use of dev questions, dev schemas, or templates derived from dev schemas during fine-tuning would contaminate the 86.9% EM and 89.7% EA numbers, and the absence of released code, model weights, training corpus, and evaluation scripts makes this impossible to check. The paper must specify the exact split, describe the template-augmented data, support the claimed optimal 1:1 hybrid ratio with the missing ablation, and either release artifacts or provide an independent reproduction before the central claim can be accepted.","section":"§3.1, §4.1, Table 1"},{"comment":"The comparison is not apples-to-apples. The baselines (T5-3B, BERT_LARGE+RAT-SQL, PICARD, RASAT, and others) are from earlier model families and were evaluated under their own pipelines, while SSQLG3.1-8B is a LoRA-adapted Llama-3.1-8B with undisclosed hyperparameters (LoRA rank, alpha, learning rate, epochs, batch size, prompt format, and decoding settings). The text claims the model exceeded the baseline by over 10% in both execution accuracy and exact match, but Table 1 shows an absolute execution-accuracy lead of 9.2 percentage points over the best baseline (89.7 vs. 80.5); the claim is only true under a relative-improvement interpretation. Please report controlled experiments in a single harness, include contemporary LLM baselines, and provide variance estimates or error bars.","section":"§4.1, Table 1"},{"comment":"The malicious-SQL evaluation is reported without a reproducible protocol: the zero-shot prompts, classification threshold, sampling scheme, and number of repetitions are not given, and it is unclear whether the syntax checker and security checker were evaluated jointly or separately. All metrics in Table 2 are point estimates without confidence intervals, so the deployment recommendation of SSQLC3-8B is not statistically supported. The prose interpretation that Llama-3.1's training data is not as well-optimized for SQL interception is unsupported speculation and should be removed or replaced with evidence.","section":"§4.2, Table 2, Figures 4 and 5"},{"comment":"The headline system-level claim that natural-language requests retrieve items from 180 million patents in under 6 seconds is not supported by any experimental protocol. The paper does not define the query workload, the number of runs, the hardware configuration used for the measurement, the latency percentiles, or the comparison point, so the claim cannot be verified or reproduced. A system paper that advertises this as a contribution needs at least a defined benchmark, a latency distribution, and a description of what is included in the measured interval.","section":"§1, §3.2"},{"comment":"The Central Control Unit (CCU), listed as a contribution, is never specified. The text states that it schedules tasks across the distributed LLM, UI, and storage clusters, but gives no information about its implementation, scheduling policy, interfaces, or failure behavior, and Figure 1 only shows it as a box. Without this, the claimed architectural contribution cannot be assessed.","section":"§1, §3.2, Figure 1"}],"minor_comments":[{"comment":"The word 'correspoding' should be 'corresponding', and the probability notation should be unified between the two equations (lowercase p in Equation (1) versus uppercase P in Equation (2)).","section":"§3.1, Equations (1) and (2)"},{"comment":"In the metric definitions, the second 'FN (False Negative)' bullet is repeated; the intended entry appears to be 'TN (True Negative) - Negative in the label, and predicted negative'.","section":"§4.2, Equations (3)-(6)"},{"comment":"The column header 'Exact Match Accuracy' is ambiguous because the table has two numeric columns; label them explicitly as 'Exact Match' and 'Execution Accuracy'.","section":"Table 1"},{"comment":"The caption calls the panels 'Precision-Recall-Escape-Misintercept Curves', but each panel plots a metric against the number of tests; either change the plot type to standard curves or rename the figure.","section":"Figure 4"},{"comment":"References [23] and [24] are not cited in the text, and no reference is provided for Llama-3 or Llama-3.1, which are the bases for the main results.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is a system description whose empirical core is currently unverifiable. The deployment context is interesting, but the experimental and reproducibility gaps are substantial, and the 'over 10%' claim is at best ambiguous when compared against Table 1. I would consider a revised version if the authors release artifacts and rerun the evaluation in a controlled harness; otherwise the manuscript does not meet the journal's evidence bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read for you. StreamLink is a workmanlike integration of local LLMs (Llama-3.1-8B fine-tuned with LoRA) with Spark/Hadoop for NL-to-SQL over a large patent warehouse. The architecture is sensible: a CCU schedules tasks across LLM and Spark clusters, a Nginx WebUI cluster balances load, and a Llama-based syntax/security checker sits in front of execution. Deploying this over 180 million patents and reporting sub-6-second average latency is a real engineering achievement, and the privacy rationale for local models is sound. Credit where due: the template-augmentation for domain adaptation is a practical trick, and the 1:1 hybrid ratio with open-domain sets is a reasonable choice.\n\nThe soft spots are where the claims outrun the evidence. The headline result—86.9% EM / 89.7% EA on Spider dev—is reported in Table 1 with no code, no model weights, no hyperparameters, no evaluation harness, and no error bars. More worrying, Section 3.1 says the fine-tuning corpus combined domain data with 'WikiSQL and Spider' but never states that 'Spider' means the official training split. If dev-set schemas or questions leaked into the augmentation, the over-10% claim collapses. The paper doesn't have to prove a negative, but it has to specify the split; it doesn't. The baselines are also dated (T5-3B, PICARD, etc.), so an apples-to-apples comparison with a current 8B model is missing. And the 'over 10%' phrasing is ambiguous—percentage points vs. relative improvement.\n\nThe security checker section is internally coherent but compares only its own variants; no existing SQL injection detector. The Kaggle dataset is fine for a smoke test, not for a claim about real-world security. Several typos (duplicated FN definition, inconsistent equation fonts) are minor.\n\nBottom line: the system story is credible, the benchmark story is not checkable. This deserves a serious referee only if the authors commit to releasing the training/evaluation artifacts and explicitly state the Spider split. As a desk decision, I'd send it out with that condition, expecting heavy revision. If the authors won't release anything, reject. For you personally: maybe worth a quick skim for the system integration, but don't cite the Spider numbers yet.","headline":"Plausible system, unverifiable headline numbers: the Spider result may be real but the paper does not provide enough to check.","tokens_in":10584,"tokens_out":2576,"would_cite":false,"duration_ms":26818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A locally hosted 8-billion-parameter LLM reaches 86.9% exact match and 89.7% execution accuracy on the Spider dev set, the best results in the paper's comparison table.","keywords":["Distributed Database","Large Language Model","SQL Generation","LLM-Driven SQL Checker","Natural Language to SQL","LoRA Fine-Tuning","Patent Data Retrieval","Data Privacy"],"falsifier":"Run the published model, if one is released, on the Spider dev set with the standard exact-match and execution-accuracy evaluation and compare the results to the reported 86.9% and 89.7%; also inspect the fine-tuning mixture for any overlap with Spider dev databases or questions.","tokens_in":9603,"feed_emoji":"🤖","tokens_out":8256,"duration_ms":78349,"temperature":0.7,"pith_summary":"StreamLink is a data system that lets users query very large databases by typing ordinary English instead of writing SQL. The paper's central claim is that a small language model fine-tuned locally, rather than a cloud AI service, can generate accurate SQL from natural language: on the dev set of the standard Spider text-to-SQL benchmark, its best model, SSQLG3.1-8B, scores 86.9% exact match and 89.7% execution accuracy, surpassing all listed baselines. The same system, deployed over 180 million patent records across three storage nodes, returns a user's requested patent in an average of under six seconds. A Llama-based syntax and security checker filters generated SQL to block malformed or malicious statements before execution. If true, this makes large private databases directly queryable by non-programmers without sending data to an outside service.","feed_headline":"On-site LLM beats text-to-SQL baselines by over 10%","feed_subtitle":"StreamLink turns 180 million patents into natural-language queries in under six seconds.","key_machinery":"The central mechanism is the fine-tuned local language model used as an NL-to-SQL generator (SSQLG), trained with LoRA on a 1:1 hybrid of domain-augmented query templates and public datasets such as WikiSQL and Spider. The generator is paired with a Llama-based SQL syntax and security checker (SSQLC) that validates each statement before execution. Around these models, StreamLink layers a distributed stack, with Spark for computing, HDFS-backed storage, a load-balanced WebUI cluster, and a Central Control Unit, so that a natural-language request becomes SQL, is checked, and runs over 180 million patents quickly.","core_discovery":"On the paper's own terms, the discovery is that a modestly sized, locally hosted LLM can be adapted to structured-data querying well enough to beat specialized text-to-SQL systems on a standard benchmark. The best system, SSQLG3.1-8B, fine-tuned from Llama-3.1-8B, reaches 86.9% exact match (verbatim agreement with a gold SQL query) and 89.7% execution accuracy (the query returns the right result) on the Spider dev set, the best numbers in Table 1. The same deployment, running on three storage nodes totaling 15.3 TB with 280 cores, answers natural-language queries over roughly 180 million patents in an average of under six seconds, and its Llama-3-based checker, SSQLC3-8B, catches 98.09% of malicious SQL statements at 79.31% precision. The paper frames these results as evidence that private-by-design, domain-adapted LLM assistants can make large distributed databases accessible to non-specialists without sacrificing security or scale.","pith_inferences":["If the reported accuracy holds up, the 1:1 hybrid ratio is a transferable recipe: a domain with only a few dozen query templates could be made queryable by generating a synthetic parallel corpus instead of hiring SQL experts.","The checker results suggest a practical follow-up: since SSQLC3-8B has 79% precision, a production system would want a low-cost second-pass filter or a human-in-the-loop confirmation for queries the checker flags, otherwise roughly one in five legitimate queries may be blocked.","A natural extension beyond patents is to apply the same template-propagation fine-tuning to other governed data, such as clinical records or financial reporting, where the privacy argument for local LLMs is even stronger.","The end-to-end six-second latency mixes LLM generation, checking, and Spark execution; splitting those stages in a benchmark would show whether the bottleneck is model inference or data scan, and where a faster deployment should invest."],"forward_implications":["Users who do not know SQL can retrieve specific records from huge private databases using natural language, removing a major barrier to data-driven work.","Keeping the LLM on local infrastructure avoids sending proprietary or sensitive data to a public AI service, which is the paper's stated privacy design.","The two-part generation-plus-checking pipeline makes LLM-produced SQL safe enough to run automatically, since malformed or injection-style statements are filtered before execution.","Because the system continues to use Spark and HDFS underneath, the approach inherits distributed scalability and can be pointed at other large datasets with new schema templates.","Domain adaptation can be rapid: given a small set of query templates, the bi-directional propagation method grows a training set and fine-tunes a model for that database."],"supporting_citations":[{"why":"Spider dev set is the benchmark where the headline exact match and execution accuracy numbers are measured.","marker":"[21]"},{"why":"WikiSQL is one of the public NL-to-SQL datasets mixed into the fine-tuning data alongside domain templates.","marker":"[20]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to produce the SSQLG generator from a pre-trained model.","marker":"[19]"},{"why":"Llama models are the base family for both the LLM-driven SQL checker and the fine-tuned generators, including the Llama-3.1-8B model.","marker":"[13]"},{"why":"Spark provides the distributed compute layer that executes generated SQL over the 180-million-patent dataset.","marker":"[2]"},{"why":"The USPTO Patent Assignment Dataset is the data source behind the scale and latency claims for the deployed system.","marker":"[8]"},{"why":"PICARD, at 75.5 exact match and 79.3 execution accuracy, is one of the strongest listed baselines the reported results are compared against.","marker":"[28]"},{"why":"GRAPPA gives the comparison a leading baseline for the exact-match metric.","marker":"[26]"}],"fun_headline_variants":["Local LLM beats text-to-SQL by 10%+ on benchmark","StreamLink queries 180M patents via natural language in <6s","Private LLM improves SQL accuracy by 10%+ on Spider","Natural-language SQL that's 10% more accurate and private"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim holds only if the reported Spider scores come from a fair, leak-free comparison, meaning the same evaluation setup for every model and no Spider dev-set example in the training data, and the paper gives no code, model, or evaluation script to verify that.","fun_headline_variants_meta":{"raw":{"variants":["Local LLM beats text-to-SQL by 10%+ on benchmark","StreamLink queries 180M patents via natural language in <6s","Private LLM improves SQL accuracy by 10%+ on Spider","Natural-language SQL that's 10% more accurate and private"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2933,"prompt_tokens":994,"completion_tokens":1939,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1862}},"tokens_in":610,"tokens_out":1939,"duration_ms":13531,"temperature":1.0,"reasoning_tokens":1862,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:46:27.416201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the published model, if one is released, on the Spider dev set with the standard exact-match and execution-accuracy evaluation and compare the results to the reported 86.9% and 89.7%; also inspect the fine-tuning mixture for any overlap with Spider dev databases or questions.","supporting_citations":[{"cited_title":"Spark: Cluster computing with working sets","cited_arxiv_id":null,"evidence_quote":"Spark provides the distributed compute layer that executes generated SQL over the 180-million-patent dataset."},{"cited_title":"The uspto patent assignment dataset: Descriptions and analysis","cited_arxiv_id":null,"evidence_quote":"The USPTO Patent Assignment Dataset is the data source behind the scale and latency claims for the deployed system."}],"review_version":1}