{"id":"acce5617-3b65-4054-b15d-b96dd10bc814","arxiv_id":"2608.09254","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A rule-gated 7B agent with deterministic post-execution verification outperforms a direct-prompted 32B baseline on a reliability benchmark where half of correct responses are not SQL answers.","lead":"This paper builds a 400-task benchmark where roughly half of correct responses are clarifications, abstentions, or refusals, and tests a 7B agent that decides behavior by rules and verifies SQL after execution. The agent beats a direct-prompted 32B model on correct business answers at lower cost per correct answer, though the test split had a disclosed contamination issue.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +0.237 headline rests on a test split the author admits was seen during development; because the gain is concentrated in exactly the strata where prior label knowledge most helps, the central comparison is not a confirmed measurement until it replicates on an uncontaminated split.","rationale":"I agree with the reader's identification of the weakest assumption. The single most load-bearing concern is that the test split was exposed during development. The paper discloses the five contaminated lexicon phrases and removes them, but that cannot undo the possibility that other design decisions—the task taxonomy, the semantic-layer definitions, the rule set—were shaped by knowledge of the held-out split. The disclosed contamination is therefore not a minor blemish; it undermines the confirmatory status of the only measurement supporting the headline comparison. The concern is amplified by where the reported advantage lives: QueryProof's separation from the baselines is almost entirely in unanswerable, schema-drift, and ambiguous strata, which are exactly the strata where knowing the intended behavior label makes rule-based correct behavior easy to engineer. The family-level bootstrap interval including zero is consistent with the effect being driven by a small number of test families. The paper's own limitation statement says the test split should be read as a single frozen evaluation with disclosed prior exposure, not as an uncontaminated confirmatory test; that is precisely the right framing, and it means the central claim is conditionally supported rather than established. Secondary issues—the missing matched-scaffold baseline and the absence of a component ablation—affect the attribution of the gain to the deterministic layer, but they are less fundamental than the validity of the comparison itself. Because the reader already arrived at a CONDITIONAL verdict and my analysis does not move that verdict, I recommend no change. The concrete path forward is replication on a genuinely unseen split; only that can settle whether the +0.237 effect is a property of the system or an artifact of prior exposure.","tokens_in":20320,"tokens_out":4092,"duration_ms":45401,"concrete_test":"Construct a fresh WarehouseReliabilityBench-style split from the same generator with previously unused template families (or a second seed/domain), freeze QueryProof and all baselines exactly as shipped, and run the same pre-registered one-pass protocol once. If the BTR advantage over 32B direct replicates (especially in unanswerable/drift strata) and the paired interval excludes zero, the contamination concern is largely settled. If the advantage shrinks toward zero, the leaked test split was load-bearing. A complementary check is a git-history audit of the repository showing that all semantic-layer and rule decisions affecting the two test ambiguous families predate the author's first access to the test split; this would bound, though not eliminate, the contamination risk.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that QueryProof beats a direct 32B baseline by +0.237 BTR and that the gain comes mostly from the deterministic layer—depends entirely on the 80-task test split. Section 3.5 discloses that five hand-authored lexicon phrases matched the entire test ambiguous stratum and could only have been written from the held-out split. Deleting them before the run does not undo the exposure: the author had test labels during development, so the task taxonomy, the semantic-layer metric definitions, and the rule set may all encode knowledge of the test families. This is not a hypothetical; the paper itself concedes that 'any development decision taken with that knowledge... cannot be unwound by removing five strings.' The danger is sharpened by Table 4: QueryProof's advantage is almost entirely in the non-answerable strata (unanswerable 0.438 vs 0.000; schema drift 1.000 vs 0.000; ambiguous 0.250 vs 0.000), precisely the strata where knowing the intended behavior label makes rule-based correct responses easy to construct. The family-level bootstrap interval, which widens to include zero, reinforces that the effect may be concentrated in a few test families. The paper is admirably transparent and the pre-registration is real, but transparency does not convert a contaminated split into a confirmatory test. For the headline claim to stand as a general result, it must be reproduced on data the author has never seen.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WarehouseReliabilityBench (WRB), a 400-task synthetic benchmark over two warehouses in which roughly half of the correct responses are clarifications, abstentions, or refusals, and defines reliability-focused metrics (Business Truth Rate, False Success Rate, coverage, cost per correct answer) with pinned denominators and a pre-registered paired bootstrap. It also presents QueryProof, a 7B analytics agent that uses a deterministic rule set over a semantic layer and physical catalog to decide behavior, and gates answers on static and post-execution verification. On an 80-task test split evaluated once, QueryProof is reported to outperform a direct-prompted 32B baseline by +0.237 in Business Truth Rate and to cut false success to 0.351 of returned answers. The paper also reports negative results: the routing layer does not transfer from validation to test, the fitted confidence model is worse than the heuristic it replaced, and post-training was declined by a pre-registered gate. The manuscript is transparent about its limitations, including test-split contamination, single-author labeling, and weak family-level bootstrap intervals.","tokens_in":20610,"tokens_out":4980,"duration_ms":48912,"significance":"If the empirical claims were reliable, this paper would make a useful contribution by demonstrating that deterministic semantic-layer and verification machinery can substantially improve the reliability of small analytics models on tasks that require clarification, abstention, or refusal, and by providing a benchmark and evaluation framework for such tasks. The paper is notable for its strengths: a real pre-registered protocol, a one-run rule on the test split, fully committed and replayable scoring artifacts, explicit denominator definitions for every metric, and unusually candid reporting of negative results and limitations. These strengths make the benchmark and the evaluation framework potentially reusable by the community. However, the central comparative result is undermined by the disclosed test-split contamination, and the family-level bootstrap intervals include zero, so the headline effect is not confirmed as a general result.","major_comments":[{"comment":"The central comparison is not a valid confirmatory test because the test split was exposed during development. The paper admits that five hand-authored lexicon phrases matched the entire 16-task test ambiguous stratum and could only have been written from the held-out split. Table 4 shows that QueryProof's advantage over the 32B baseline is concentrated in the ambiguous and unanswerable strata (0.250 vs 0.000 and 0.438 vs 0.000, respectively), exactly the strata whose labels the contaminated lexicon encoded. Removing the strings before the run does not undo the potential influence of prior test knowledge on taxonomy, rule design, or semantic-layer choices. The abstract's '+0.237 [+0.112, +0.375]' therefore should not be reported as an 'outperforms' result unless either replicated on an uncontaminated split or explicitly reframed as a hypothesis-generating observation from a single frozen evaluation with known exposure.","section":"Section 3.5 and Section 6.1"},{"comment":"The family-level resampling interval includes zero for both headline paired comparisons (+0.237 [−0.125, +0.562] against 32B; +0.262 [−0.075, +0.575] against few-shot). The text acknowledges that the task-level interval is optimistic and that the cluster bootstrap over 10 families is itself unreliable, but it nevertheless concludes that 'the direction of the effect is supported.' Given the acknowledged optimism of the task-level interval and the unreliability of the 10-cluster bootstrap, the evidence does not establish a directional effect. The paper should either provide a more reliable interval (for example, by generating more template families) or restrict the claim to a point estimate on this specific split.","section":"Section 6.3 / Table 8"},{"comment":"The claim that 'the gain tracks the deterministic layer' is not supported by the reported experiments. No component ablation is run; the only ablation (routing on/off, Table 10) shows the routed variant slightly underperforms the base variant on test (0.537 vs 0.562). The comparison between QueryProof and the baselines confounds the semantic layer, the verification checks, the repair loop, and the model prompting scheme, so the difference cannot be attributed to any specific deterministic component. The abstract and conclusion should either report that the attribution is currently unsupported by direct experimental evidence or add an ablation that isolates the deterministic components.","section":"Section 7 and Table 10"},{"comment":"The benchmark's labels, task taxonomy, semantic layer, and rules were all authored by a single person, and the re-adjudication (Table 1, κ = 0.887) is a test–retest measurement, not inter-annotator agreement. The paper states this limitation, but its central claim depends on the benchmark's validity as a fair measurement; single-author co-design plus the disclosed leakage means the benchmark has not been independently validated. A second annotator, or a review by a separate expert, would be needed to support the external-validity claims that the comparison implicitly makes.","section":"Section 3.4 and Section 7"}],"minor_comments":[{"comment":"The title and abstract emphasize '7B' versus '32B,' but the paper correctly states that the comparison is between systems rather than model sizes. Consider rewording the title to avoid implying a model-size comparison, since the 32B baseline receives none of the scaffolding.","section":"Title and Abstract"},{"comment":"The column header 'Agreed n' is ambiguous; clarify that it is the number of labels on which the two adjudication passes agreed, and note that 'Agreement' is the raw proportion.","section":"Table 1"},{"comment":"The pre-registration is described as fixing artifact hashes and claim verbs, but the paper does not state whether the pre-registration document is publicly time-stamped or deposited; if it is, provide a link or repository path for verification.","section":"Section 5.3"},{"comment":"The 'Phase' column values (e.g., '6', '10', '11') are not defined in the table; consider adding a short explanation of what a phase is or aligning the labels with the text.","section":"Appendix A, Table 5"}],"recommendation":"major_revision","confidential_remarks":"This manuscript is unusually transparent, and the reproducibility infrastructure is a genuine strength. However, the central comparative claim rests on a test split the author acknowledges was exposed during development, and the family-level bootstrap intervals include zero. The editor may wish to consider whether the paper should be reframed as a benchmark and evaluation-framework contribution with an illustrative case study, since the empirical comparison cannot be repaired within the current manuscript without an uncontaminated replication. The single-author co-design further compounds the validity risk. If the authors can provide a new uncontaminated evaluation or explicitly downgrade the comparative claim, the paper could be acceptable; in its current form the headline result is not confirmatory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper is worth your time for its evaluation framework and its honesty, not for its headline number. The +0.237 BTR win over a direct 32B baseline is real as measured, but the test split it comes from was contaminated, and the author says so plainly in Section 3.5. Five lexicon phrases matched the entire test ambiguous stratum, which means the author had test labels during development. Removing the strings doesn't undo that. So read the result as a single frozen evaluation with disclosed prior exposure, exactly as the paper asks, and not as a confirmed effect.\n\nWhat's genuinely new: WarehouseReliabilityBench, where roughly half the tasks have no correct SQL and are scored by behavior contracts (clarify, abstain, refuse). Business Truth Rate and False Success Rate with pinned denominators are a real improvement over execution-match metrics for production-style failures. The pre-registration is real, the paired bootstrap verb rule is sensible, and the reproducibility story is unusually complete: committed outputs, one command to regenerate every table, sealed cost values. The negative results are also refreshing: routing helped on validation but hurt on test, the fitted confidence model lost to the heuristic it replaced, and post-training was declined by a pre-registered gate. That kind of reporting is rare.\n\nThe soft spots, in order. The contamination is the big one. The gain is concentrated in exactly the strata where knowing the intended label makes rule-based behavior easy to construct: unanswerable 0.438 vs 0.000, schema drift 1.000 vs 0.000, ambiguous 0.250 vs 0.000. The family-level bootstrap interval includes zero. So the direction of the effect is plausible, the magnitude is not established. Second, the headline comparison is confounded: QueryProof vs 32B direct differs in scale and scaffolding, and there's no matched-scaffold baseline or component ablation, so the claim that the deterministic layer is the source is inferred, not demonstrated. Third, the benchmark is synthetic, single-author, and small: 80 test tasks from 10 families. External validity is limited.\n\nI agree with the reader's conditional verdict and the stress-test note. Still, the framework and the honest failure reporting are exactly what this area needs. This paper deserves a serious referee, not a desk reject. I'd send it out with the expectation of heavy revision, and I'd want to see the headline claim reproduced on uncontaminated data before treating it as a result.\n\nFor your own work: the metrics and the benchmark design are worth citing; the specific performance claim I'd be cautious about.","headline":"The evaluation framework and honest negative results are the contribution; the headline +0.237 win rests on a test split the author admits was contaminated.","tokens_in":21093,"tokens_out":2777,"would_cite":true,"duration_ms":27713,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B analytics agent that decides behavior by rules and verifies results after execution outperforms a direct-prompted 32B baseline on business-answer reliability, with most of the gain coming from the deterministic layer rather than…","keywords":["text-to-SQL","business truth rate","analytics agent","ambiguity detection","abstention","semantic layer","post-execution verification","reliability benchmark"],"falsifier":"Run the same pre-registered protocol on a freshly generated, uncontaminated warehouse with a new seed and no author knowledge of the questions; if the 7B rule-gated agent no longer beats the direct-prompted 32B baseline, or if the advantage falls inside noise when template families are resampled, the central claim is refuted. A second decisive check: scaffold the 32B model with the same semantic-layer rules and verification gates; if it then matches or exceeds the 7B agent, the credited source of the gain shifts from deterministic rules to scale.","tokens_in":20112,"feed_emoji":"📊","tokens_out":6680,"duration_ms":59745,"temperature":0.7,"pith_summary":"The paper argues that analytics agents should be judged on whether the business number they return is correct and appropriate, not on whether their SQL matches a reference query. To make that measurable, it introduces WarehouseReliabilityBench, 400 frozen tasks over two synthetic warehouses in which roughly half of the correct responses are a clarification, abstention, or refusal. It then reports that QueryProof, a 7B agent that decides behavior by deterministic rules over a semantic layer and verifies results after execution, reaches a higher Business Truth Rate than a direct-prompted 32B baseline on an 80-task test split, at lower cost per correct answer. The author's claim is that most of that reliability gain comes from the deterministic layer, not from model size; the paper also discloses that the test ambiguous stratum was exposed during development via contaminated lexicon phrases, so the test figures are a frozen evaluation with prior exposure, not a clean confirmatory test.","feed_headline":"7B rule-gated agent beats 32B baseline on business answers","feed_subtitle":"On tasks where the right move is to clarify, abstain or refuse, deterministic verification—not model size—carries the gain.","key_machinery":"The central object is the deterministic behavior-and-verification layer: an ordered rule set over a semantic layer (metric definitions) and physical catalog (tables, columns, coverage) that decides each terminal behavior, plus static AST validation and post-execution checks on the result. It carries the argument by making the reliability decision auditable and independent of model judgment; the model is used only to propose candidate metric resolutions and SQL, and its outputs are re-checked before they can change the outcome.","core_discovery":"QueryProof is a state machine that calls a 7B language model twice but never lets the model decide behavior. From a semantic layer and physical catalog, rules choose REFUSE, ABSTAIN, or CLARIFY before the model is asked anything; the model only proposes which metric is meant and what SQL computes it, and every candidate is checked by static validation and post-execution verification. On the 80-task frozen test split, this system scores 0.537 Business Truth Rate routed (0.562 unrouted) against 0.300 for the 32B direct baseline, a paired difference of +0.237 [+0.112, +0.375]; it cuts false success to 0.351 of returned answers and returns no wrong number on an answerable task (0 of 24). The paper does not claim model size explains the gap; it claims the deterministic behavior and verification layer is where the reliability comes from, and it reports that the learned components — routing, confidence model, and post-training — did not transfer.","pith_inferences":["Editorial inference: a component ablation is the decisive missing experiment; wrapping the 32B model in the same scaffold, or stripping the 7B agent of its verification gates, would cleanly separate scale from scaffolding.","Editorial inference: because the test ambiguous stratum was exposed during development, the benchmark's confirmatory value is limited; a pre-registered replication on a fresh seed-generated warehouse with no author knowledge would settle whether the direction of the effect survives.","Editorial inference: if the gain really lives in the deterministic layer, the practical bottleneck shifts to maintaining the semantic layer over time; organizations with unstable metric definitions may not reproduce the result without equivalent curation effort.","Editorial inference: the cost-per-correct-answer framing could be extended to count a correct abstention or refusal as a paid outcome, which would reward appropriate caution rather than only correct SQL."],"forward_implications":["If the paper is right, text-to-SQL evaluation that relies on execution-match accuracy alone will overstate production readiness, because the expensive failures are wrong business numbers returned by valid SQL.","The measured gain is concentrated in strata where the correct response is not an answer; on the 32 standard questions the systems are near parity, so benchmark composition materially decides the apparent advantage.","A deterministic verification layer can absorb much of the reliability headroom that post-training would target: the pre-registered gate was not met because the remaining model-capability failure pool was 20 against a threshold of 100.","Learned reliability components may not transfer: routing tuned on validation over-abstains on test, and the fitted confidence model loses to the heuristic it replaced on both calibration and selective prediction.","The direction of the effect is better supported than its magnitude: resampling template families rather than tasks widens both accuracy intervals to include zero, so the ordering is evidence but the size of the benefit is poorly determined."],"supporting_citations":[{"why":"Defines the Spider text-to-SQL benchmark and execution-match scoring that WarehouseReliabilityBench deliberately departs from.","marker":"Yu et al., 2018"},{"why":"Introduces BIRD as a realistic text-to-SQL standard; the paper argues such suites skip tasks with no correct SQL.","marker":"Li et al., 2023"},{"why":"TrustSQL is the closest prior position, scoring reliability with penalties for confident wrong answers rather than coverage alone.","marker":"Lee et al., 2024"},{"why":"Execution-guided decoding is the ancestor of execution grounding that QueryProof moves downstream to post-execution result checks.","marker":"Wang et al., 2018"},{"why":"Supplies the semantic-layer and MetricFlow concept of externally maintained metric definitions that deterministic ambiguity detection rests on.","marker":"dbt Labs, 2026b,a"},{"why":"Formal selective classification underpins the calibrated-abstention component that the paper reports as a negative result.","marker":"El-Yaniv and Wiener, 2010"},{"why":"Qwen2.5-Coder is the model family shared by all six systems, so the comparison isolates scaffolding rather than pretraining.","marker":"Hui et al., 2024"},{"why":"Provides the bootstrap methodology behind the pre-registered paired intervals, with the caveat that a ratio of sums cannot be bootstrapped as a mean.","marker":"Efron and Tibshirani, 1993"},{"why":"Preregistration protocol fixes the claim verbs, one-run rule, and artifact hashes that govern the test evaluation.","marker":"Nosek et al., 2018"}],"fun_headline_variants":["Rules beat scale: 7B agent outperforms 32B on business truth","Small model wins when it clarifies, abstains, refuses: 7B beats 32B","Not SQL accuracy: 7B agent beats 32B on business answers","Deterministic checks, not model size: 7B tops 32B on truth rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen test set is a fair yardstick; the paper discloses that five hand-written synonym rules matched every test question about ambiguous terms and could only have been written after seeing the held-out questions, so the +0.237 gap may partly reflect prior knowledge rather than the system's design.","fun_headline_variants_meta":{"raw":{"variants":["Rules beat scale: 7B agent outperforms 32B on business truth","Small model wins when it clarifies, abstains, refuses: 7B beats 32B","Not SQL accuracy: 7B agent beats 32B on business answers","Deterministic checks, not model size: 7B tops 32B on truth rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1804,"prompt_tokens":1119,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":735,"completion_tokens_details":{"reasoning_tokens":593}},"tokens_in":735,"tokens_out":685,"duration_ms":6972,"temperature":1.0,"reasoning_tokens":593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:43:05.713297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pre-registered protocol on a freshly generated, uncontaminated warehouse with a new seed and no author knowledge of the questions; if the 7B rule-gated agent no longer beats the direct-prompted 32B baseline, or if the advantage falls inside noise when template families are resampled, the central claim is refuted. A second decisive check: scaffold the 32B model with the same semantic-layer rules and verification gates; if it then matches or exceeds the 7B agent, the credited source of the gain shifts from deterministic rules to scale.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formal selective classification underpins the calibrated-abstention component that the paper reports as a negative result."}],"review_version":1}