{"id":"fc021191-c647-4e5c-b923-2f0ca46e72a4","arxiv_id":"2611.17093","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"FlinkEval provides an execution-grounded benchmark for natural language to streaming SQL, revealing that current LLMs struggle with pattern composition and underspecified requests.","lead":"FlinkEval is a synthetic benchmark of 3,903 question-SQL pairs for streaming Flink SQL, grounded in execution on a live Confluent Cloud cluster. It evaluates how well large language models can generate streaming SQL queries and introduces a seven-level specification ladder to study how request underspecification affects performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"LLM-generated L2.1 requests and oracle clues may systematically underestimate real-world underspecification, biasing the Ladder results.","rationale":"The reader's weakest_assumption correctly identifies that the synthetic corpus and LLM-generated requests may not be representative of real-world streaming NL2SQL tasks. The reader notes that the paper acknowledges this limitation. My stress-test focuses on a specific, load-bearing consequence of this assumption: the Ladder study's central finding (27% pass rate on unclarified requests, partial recovery by oracle clues) could be an artifact of LLM-generated underspecification rather than a genuine property of natural user intents. This is the most load-bearing concern because the Ladder is a key contribution (one of three claimed contributions) and its quantitative results are cited as evidence for the 'specification bottleneck.' The paper's own limitation statement (b134) flags this, but does not assess its impact on the Ladder conclusions. The concrete test I propose is feasible (small human annotation study) and would directly settle whether the concern lands. I agree with the reader that the paper is otherwise well-constructed and transparent; the concern does not warrant changing the verdict from ACCEPT, as the paper acknowledges the limitation and the benchmark remains useful even if the Ladder numbers shift. However, the reader's verdict of ACCEPT with HIGH confidence may be slightly too confident given this unresolved threat to the Ladder's validity; I would keep ACCEPT but note that the Ladder results should be interpreted with caution until human validation is performed. The agreement_with_reader is 'agree' because the reader identified the same weakest assumption (representativeness of LLM-generated content).","tokens_in":21337,"tokens_out":835,"duration_ms":10050,"concrete_test":"Conduct a small human annotation study: have 3–5 domain experts write 50–100 unclarified business-language requests (L2.1 equivalent) for a subset of FlinkEval schemas, following the same guidelines used for the LLM generator. Then run GPT-6-Astra on these human-authored requests (same protocol as Section 5.2) and compare the judge-pass rate to the 27% reported for LLM-generated L2.1 requests. If the human-authored rate is significantly higher (e.g., >40%), the LLM-generated requests are artificially hard and the Ladder gap is inflated. If it is similar or lower, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that the Question Specification Ladder reveals a 'specification bottleneck' (Section 5.2) rests on the premise that the L2.1 (unclarified) requests and L2.2–L2.4 oracle clues are representative of natural user underspecification. However, the authors acknowledge (Section 6, Limitations) that these are 'LLM-generated and not human-authored, which may introduce systematic biases toward phrasings the generator itself favours.' This is not merely a scope limitation; it is a load-bearing threat to the Ladder study's internal validity. The L2.1 requests are generated by the same LLM family (GPT-5.6-Sol) that later serves as the generator in the Ladder evaluation (GPT-6-Astra). If the L2.1 requests are systematically easier or harder than human-authored underspecified requests, the reported 27% pass rate and the recovery pattern from oracle clues may not generalize. The paper's strongest claim—that the Ladder reveals challenges in request specification—depends on the L2.1–L2.4 conditions being meaningful proxies for real underspecification. The authors flag this as a limitation but do not assess its direction or magnitude. The concern is not that the data is synthetic per se, but that the generator's own biases could create an artificial gap between L0/L1 and L2 levels that would not hold with human-authored requests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces FlinkEval, a synthetic benchmark for evaluating natural language interfaces to streaming SQL (specifically Apache Flink/Confluent Cloud). It comprises 3,903 question-SQL pairs across 45 domains, with execution-grounded validation on a live cluster. A seven-level Question Specification Ladder studies underspecification. Baseline evaluations of five LLMs show 65-80% judge-pass rate on schema-explicit requests but only 27% on unclarified requests, revealing challenges in pattern composition and request specification.","tokens_in":21751,"tokens_out":1799,"duration_ms":18671,"significance":"If the benchmark is valid, it fills a clear gap: existing NL2SQL benchmarks (Spider, BIRD) focus on bounded relational data and do not cover streaming semantics, temporal joins, windowing, pattern matching, etc. The execution-grounded validation on a live cluster is a strength. The Question Specification Ladder is a novel contribution for studying underspecification. The paper is likely to be useful to the community working on NL2SQL for streaming systems.","major_comments":[{"comment":"The Ladder study's central claim—that unclarified requests reveal a 'specification bottleneck'—rests on the L2.1 requests being representative of natural user underspecification. The authors acknowledge (§6) that these requests are LLM-generated and may have systematic biases. This is a load-bearing threat to internal validity. The paper does not assess the direction or magnitude of this bias. For example, if the LLM generates L2.1 requests that are systematically easier (e.g., by omitting only easily recoverable details) or harder (e.g., by omitting critical details that a human would typically include), the reported 27% pass rate may not generalize. The authors should either (a) provide a human annotation study on a sample to calibrate the L2.1 requests, or (b) explicitly temper the claim about the 'specification bottleneck' to reflect that it is an artifact of the generation pipeline, not necessarily a property of real-world underspecification. As it stands, the paper's strongest claim (the Ladder reveals challenges in request specification) is undermined by this acknowledged limitation.","section":"§6 (Limitations) and §5.2","authors_acknowledge":"Specification validity.The unclarified base re- quests (L2.1) and oracle clues (L2.2–L2.4) are LLM-generated and not human-authored, which may introduce systematic biases toward phrasings the generator itself favours.","authors_acknowledge_p":0.94},{"comment":"The model comparison excludes 115 test questions that use TUMBLE or HOP windows due to a Confluent Cloud defect (SQL-5248). This is a significant exclusion: windowed aggregation is a core streaming SQL construct. The authors note that the Window TVF group in the evaluation contains only session_window_tvf and cumulate_tvf. This means the benchmark does not evaluate the most common window types (tumbling, hopping) in the model comparison. The paper should discuss how this exclusion affects the generalizability of the results, and ideally provide some evidence (e.g., from the full corpus evaluation on v2) that the relative model rankings hold for these window types. Without this, the claim that the benchmark covers windowing is only partially supported.","section":"§5.1, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The evaluation uses LLM judges (Claude Opus 5.5, GPT-6-Astra, Gemini 3.6 Flash) which overlap with the generator models (GPT-6-Astra, Claude Opus 5, etc.). The authors acknowledge this as a limitation (§6, [b139], [b310]). However, the potential circularity is more severe than acknowledged: the same model families are used for both generation and evaluation. For the Ladder study, GPT-6-Astra is both the generator under evaluation and one of the judges. While the judge panel uses a different version (Opus 5.5 vs Opus 5 for Claude), the overlap could bias judgments in favor of the generator's outputs. The paper should at minimum report the agreement rates between judges and the adjudication outcomes broken down by generator model, to allow readers to assess the extent of this bias. Currently, only aggregate adjudication rates are given (14.5% overall).","section":"§5.1, Table 3 and §5.2, Table 4","authors_acknowledge":"The panel invokes adjudication on a non-negligible share of predictions (Section 5.1), and the evaluator pool overlaps with the generator families; face-value judgments are not calibrated against human annotations.","authors_acknowledge_p":0.96},{"comment":"The Ladder study uses only a single model (GPT-6-Astra) and a single run. The authors acknowledge this limits conclusions about model differences and run-to-run variability (§6, [b142]). However, the central claim about the 'specification bottleneck' is based entirely on this single-model, single-run experiment. Without replication on at least one other model (e.g., Claude Opus 5), it is unclear whether the observed pattern (sharp drop from L1.2 to L2.1, partial recovery with clues) is a general property of LLMs or specific to GPT-6-Astra. The paper should either replicate the Ladder study on a second model or explicitly state that the results are preliminary and may not generalize.","section":"§5.2, Table 4","authors_acknowledge":"These choices limit conclusions about small model differences and run-to-run variability.","authors_acknowledge_p":0.92}],"minor_comments":[{"comment":"The abstract states '3,903 question–SQL pairs across 45 industry domains and 45 semantic labels.' However, Table 2 shows 45 domains and 45 semantic labels, but the text in §4.2 says '45 industry domains' and '45 semantic labels.' The numbers are consistent, but the abstract could be clearer about what '45 semantic labels' refers to (it is the number of observed patterns, not the number of groups).","section":"§1, Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 6 lists 14 groups (G1-G15 with G11 absent). The text says '46 SQL patterns' but Table 6 shows 46 patterns (including time_series_math which is not generated). The count is correct, but the presentation is slightly confusing because G11 is intentionally absent. Consider renumbering or explicitly noting the gap.","section":"§4.1, Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The row 'Pass & ran' is defined as 'share of questions with score 1.0 and no execution error.' However, for background statements, all fail to execute, so 'Pass & ran' is 0 for all models on background. This is correct but could be clarified in the table caption.","section":"§5.1, Table 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The table caption says 'Clues are ranked oracle additions, not clarification turns.' This is important context that should also appear in the main text discussion of the Ladder results, not just in the table caption.","section":"§5.2, Table 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The limitations section is thorough and well-structured. However, the sentence 'Broad paraphrasing confounds linguistic changes with semantic omissions' ([b135]) is a key point that could be expanded: the L2.1 requests differ from L1.2 in both wording and omitted details, so the drop in performance cannot be attributed solely to underspecification.","section":"§6, Limitations","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The pattern 'time_series_math' is listed but marked as not generated. This is fine, but the reason given ('requires synthetic signal generation, which yields no stable reference result') could be clarified: does this mean the pattern is excluded from the corpus entirely, or just that it is not a generation target?","section":"Appendix B.1.1, Table 6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The per-domain analysis from release v2 is informative but the paper should note that these results are from an earlier release and may not be directly comparable to the v3.1 results in the main text.","section":"Appendix D, Table 12","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is the first benchmark that actually runs streaming SQL on a live cluster and checks that the reference queries execute and return rows. That alone makes it worth knowing about. The corpus construction is careful—quota-balanced patterns, a repair loop, SME audit, integrity checks on the data generation. The Question Specification Ladder is a sensible framework for separating identifier abstraction from genuine underspecification, and the face-value vs. ground-truth comparison is informative.\n\nThe soft spots are real but mostly acknowledged. The biggest one is the Ladder study: the L2.1 unclarified requests and the oracle clues are all LLM-generated, and the same model family (GPT-5.6-Sol) that generates them also serves as the generator in the evaluation (GPT-6-Astra). The authors flag this in the limitations section, but they don't assess its direction or magnitude. The 27% pass rate on L2.1 could be an artifact of the generator's own phrasing biases rather than a genuine measure of underspecification difficulty. This is not a fatal flaw—the paper is transparent about it—but it means the Ladder results should be read as a lower bound on what a human-authored version might show, not as a calibrated measurement.\n\nOther concerns: single-cluster grounding (acknowledged), LLM-as-judge overlap with generator families (acknowledged), exclusion of TUMBLE/HOP windows due to a cluster defect (acknowledged), and the fact that the Ladder study uses only one model (GPT-6-Astra) in a single run. These limit the precision of the claims but don't invalidate the central contribution.\n\nWho this is for: anyone working on NL2SQL for streaming systems, or on benchmarks for temporal/operational semantics. The corpus and evaluation protocol are reusable.\n\nRecommendation: accept for peer review. The paper is honest about its limitations, the engineering is solid, and the benchmark fills a genuine gap. The Ladder results need a caveat about the LLM-generated underspecification, but the authors already wrote that caveat. A serious referee would want to see a human-authored subset for calibration, but that's a request for future work, not a reason to desk-reject.","headline":"First execution-grounded streaming NL2SQL benchmark; solid engineering but the Ladder study's load-bearing assumption about LLM-generated underspecification is acknowledged but uncalibrated.","tokens_in":22287,"tokens_out":553,"duration_ms":7012},"pith_extraction":null,"created_at":"2026-11-13T10:10:30.692583+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}