{"arxiv_id":"2611.17093","title":"FlinkEval: A Synthetic Corpus for Evaluating Natural Language Interfaces to Streaming SQL","abstract":"Existing natural-language-to-SQL benchmarks primarily evaluate queries over bounded relational data, leaving streaming semantics underexplored. We introduce FlinkEval, an execution-grounded synthetic benchmark comprising 3,903 question--SQL pairs across 45 industry domains and 45 semantic labels. We create diverse schemas and domain-plausible, temporally consistent data through iterative generation and validation. SQL patterns collected from the human-written NL2SQL pairs and vendor documentation drive quota-balanced generation, and execution with a repair loop on Confluent Cloud Apache Flink produces reference queries covering windowing, temporal joins, pattern matching, nested types, and data manipulation. The corpus includes paired schema-explicit and business-language requests, with annotations for analysis by query pattern, complexity, and execution mode. A seven-level Question Specification Ladder examines semantics-preserving reformulation and underspecified requests augmented with oracle-derived clues. Baseline evaluations combine execution outcomes with adjudicated LLM judgments, revealing challenges in pattern composition and request specification.","primary_cat":"cs.DB","cross_cats":[],"submitted_at":"2026-11-12T16:54:35+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"ACCEPT","confidence":"HIGH","one_line_summary":"FlinkEval provides an execution-grounded benchmark for natural language to streaming SQL, revealing that current LLMs struggle with pattern composition and underspecified requests.","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0.0,"inbound_count":0,"_links":{"html":"https://pith.science/paper/2611.17093","verdict":"/api/v1/verdicts/2611.17093"}}