{"id":"31a277a0-1aa1-4654-930d-ec57f9956499","arxiv_id":"2610.02739","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An external, mutable question pool that agents must empty or deliberately prune before submitting SQL improves ambiguity coverage and cuts silent failures in interactive text-to-SQL.","lead":"An AI agent that turns natural language into database queries often stops asking questions too early, so it makes unverified guesses that happen to be correct. This paper introduces PlanPool, a question checklist the agent must resolve or deliberately remove before answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"PlanPool's headline gains (ambiguity recall, silent failures) are scored by the user-simulator encoder that maps questions to annotated ambiguities, a metric unvalidated here for compositional multi-ambiguity settings; for a method that mechanically forces more questions, a permissive mapping can…","rationale":"I read the paper in good faith first. The central claim has two pillars: (1) a diagnostic that agents identify ambiguities yet abandon them, and (2) a method claim that PLANPOOL improves groundedness. Pillar (1) is largely encoder-free: the planned-vs-asked evidence (4.1 vs 3.3 questions on BIRD-Lite, Figure 2c-d) is a count of questions, and the question-budget accuracy scaling (64.9% to 88.9%, Figure 2a) is execution-based; both are reasonably secure. Pillar (2), the contribution, is measured by recall, grounded success, and silent failures, and all three flow from the encoder's ambiguity mapping. This is precisely the reader's weakest assumption, and I agree it is the most load-bearing concern. I confirmed it concretely: the encoder both answers and evaluates; PLANPOOL's mechanism forces additional questions, so the metric and the mechanism are confounded by construction. When I checked the mitigations, I found two that partially soften the reader's reading: Figure 2(b) shows the encoder does not map everything (it classifies many questions as non-critical or unanswerable), and the protocol is cited as validated in Huo et al. (2026). These reduce but do not remove the concern, because the validation was for user simulation, not for coverage ground truth on compositional multi-ambiguity instances; this is a transferred use outside the source's stated hypotheses, so requesting a check is appropriate under my rules. A separate concern, the absence of error bars and significance tests, is real and the reader flagged it; on Spider the deltas are small enough that noise is plausible, but it is secondary to the validity of the metric itself. A biased metric defeats even perfectly precise statistics, which is why I treat the mapping audit as the decisive check. Because the concern is empirically testable and the reader already assigned CONDITIONAL with the same remedy in view, I do not move the verdict; I recommend UNCHANGED, with the concrete test specifying exactly how the mapping would be validated.","tokens_in":21213,"tokens_out":7649,"duration_ms":55619,"concrete_test":"Stratified human-annotation audit: take ~50 BIRD-Lite and ~50 Spider trajectories from both PLANPOOL and Plan-Prompt. Two to three human annotators, blinded to method and to the encoder's mapping, mark which annotated ambiguity (if any) each asked question addresses. Recompute recall, grounded success, and silent-failure for both methods using only human judgments. If encoder-human agreement is high and balanced across methods, the groundedness claim holds; if the encoder flags substantially more of PLANPOOL's recovered (extra) questions as addressing ambiguities than humans do, the measured coverage and silent-failure gains are partly instrumental and the central claim must be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's value-add is groundedness, not execution accuracy: the contribution rests on ambiguity recall rising from 80.8% to 87.5% and silent failures falling from 10.5% to 8.2% on BIRD-Lite (Table 2). These numbers are produced by the user simulator's encoder: an annotated ambiguity counts as 'addressed' iff the encoder maps the agent's clarification question to it (Section 2, Appendix B). The same model family (Claude-Opus-4.8) generates the questions as the system agent and judges whether they address the annotations. Because PLANPOOL's mechanism forbids submission while any pool item is pending, it by construction asks nearly every question it plans; the encoder is then the only instrument standing between 'more mechanical questions' and 'more genuinely resolved ambiguities.' A lenient mapping would inflate recall and, via GS = EA AND full coverage, deflate silent failures, and it would do so differentially for the higher-asking method. Mitigations exist: Figure 2(b) shows the encoder does mark many questions 'non-critical' or 'unanswerable,' so it is not trivially permissive, and the two-stage protocol is cited as validated in Huo et al. (2026). But that validation targeted user-simulation fidelity, not the use of the mapping as the coverage ground truth for the compositional multi-ambiguity instances retained here (171/300 and 225/600 after filtering). The transfer is new and unchecked. The missing error bars compound this: on Spider the deltas are only +1.1 recall and -0.7 silent failures (n=297), plausibly noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper studies interactive agentic Text-to-SQL under underspecified user queries and argues that execution correctness is not the same as grounded resolution: an agent can return the correct execution result while leaving annotated ambiguities unresolved (ungrounded success) or fail while leaving them unresolved (silent failure). The authors diagnose premature clarification termination with a question-budget study, in which execution accuracy on BIRD-Lite rises from 64.9% at k=2 to 88.9% at k=10 but later questions increasingly concern non-critical or unanswerable issues, and with a plan-prompt study, in which agents plan more clarification questions than they ultimately ask. They propose PlanPool, which materializes the clarification plan as external state and forbids SQL submission while the pool is non-empty. Using solvability-filtered subsets of BIRD-Interact-Lite and BIRD-Interact-Full and a newly constructed multi-ambiguity Spider benchmark, PlanPool is compared against six baselines and consistently improves ambiguity recall and grounded success while reducing silent failures at somewhat higher interaction cost. The paper is explicit that its BIRD evaluation is conditional on solvability after oracle clarification, and the method-specific prompts, simulator, and construction pipelines are documented in the appendix.","tokens_in":21834,"tokens_out":12149,"duration_ms":147309,"significance":"The contribution is a useful and currently timely distinction between asking good questions and maintaining a commitment to resolve them. The controlled budget analysis, the disclosed solvability filter, the decomposition into grounded success, ungrounded success, and silent failure, and the detailed appendix-level protocol descriptions are genuine strengths; the method is not fitted to the ambiguity annotations, so the headline comparisons are not circular in the fitted-parameter sense. If the clarifying analyses below confirm that the measurement instrument is not permissive and that the gains are not merely a question-count effect, the paper would be a solid empirical contribution to agentic text-to-SQL. As it stands, the central empirical evidence is weaker than the abstract's wording suggests, but it is defensible and the requested checks are within the scope of a revision.","major_comments":[{"comment":"The main groundedness evidence is generated by a single LLM-based instrument. In Section 2 and Appendix B, the set A(τ) is defined operationally as the annotated ambiguities to which the user-simulator encoder maps the agent's clarification questions, and the same encoder also decides whether each question is answerable. PlanPool's empty-pool precondition causes it to ask more questions by construction, so a permissive or biased mapping would inflate its measured recall and deflate its silent-failure rate differentially relative to baselines. The two-stage protocol is validated in the cited Huo et al. (2026) work for user-simulation fidelity, but that validation does not establish the encoder's mapping as coverage ground truth on the compositional multi-ambiguity instances retained here (171/300 and 225/600). Please add: (i) inter-annotator agreement on a stratified sample of question-to-ambiguity mappings, especially for instances with several simultaneous ambiguities; (ii) an alternative scoring rule that does not use the same encoder, such as exact matching of the annotated ambiguity phrase or human judgment on a sample; and (iii) a per-method examination of false-positive mappings. Without these, the Table 2 recall and silent-failure deltas are not independently verified.","section":"Section 2 and Appendix B","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"PlanPool's advantages are partially confounded with the number of questions asked. Table 2 shows PlanPool asks 4.2, 3.6, and 2.5 questions versus 3.3, 2.9, and 2.2 for Plan-Prompt, and Table 3 shows that additions occur in 30.4% and 32.9% of BIRD-Lite and BIRD-Full instances while drops are rare; thus the mechanism's observable effect is mostly a higher question count. The controlled budget study in Fig. 2(a)-(b) demonstrates that execution accuracy responds strongly to the question budget, but the paper does not report ambiguity recall or silent failure for the k=4, k=6, and k=10 conditions, so the reader cannot tell whether PlanPool's groundedness gains reflect commitment management or simply about one additional question per query. Please add a matched-question-count comparison (for example, an enforced-k version of Plan-Prompt or Naive whose realized number of asks equals PlanPool's) or per-question-position recall curves, otherwise the central \"identify versus maintain\" distinction is not separately identified.","section":"Section 5.2, Section 5.3, Table 2, Table 3, Fig. 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table 2 contains no error bars or significance tests, and several headline deltas are small relative to sample sizes. On BIRD-Lite, silent failure drops from 10.5% to 8.2% over 171 instances, corresponding to roughly 18 versus 14 cases; on Spider, the recall gain is 94.3% versus 93.2% over 297 instances and the silent-failure gain is 4.0% versus 4.7%. Please report bootstrap confidence intervals or per-instance matched tests (e.g., McNemar for GS, US, and SF, and bootstrap intervals for ambiguity recall), and qualify the abstract and Section 5.2 claims accordingly; this also matters for the \"competitive execution accuracy\" statement, given that PlanPool's BIRD-Full EA of 67.1% is below Self-Reflection's 70.7%.","section":"Table 2","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The solvability filter is disclosed and the conditional scope is stated, but the abstract's \"across three benchmarks\" should at first mention carry the qualification that the BIRD benchmarks are solvability-filtered subsets, to avoid a reader inferring behavior on the unfiltered BIRD-Interact sets.","section":"Section 5.1 and Appendix A.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Spider construction would benefit from a histogram or table of the resulting ambiguity-count distribution (e.g., how many instances have two, three, or four injected ambiguities) and a breakdown by corruption operator, since the current Table 1 only reports the mean of 2.08 per task.","section":"Appendix A.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The appendix gives the prompts and pipelines, but there is no repository link or release commitment for the filtered datasets and runner code; given that the benchmarks are new and the harness is method-specific, such a link is important for verifying the Table 2 numbers.","section":"Reproducibility statement","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The legend \"annotated non-critical unanswerable\" would be clearer as labeled series or a legend that visually separates the three question categories; currently the category boundaries are difficult to read without the surrounding text.","section":"Figure 2(b)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper's real contribution is the mechanism, and it isn't prompt engineering in costume. PlanPool externalizes the clarification plan as persistent state that must be explicitly asked, dropped, or added to before submission, with submission gated on the pool being empty. That's a concrete commitment-maintenance device, and I can't find prior work that enforces discharge of planned questions before SQL generation. The diagnosis it builds on is also well-supported internally: the planned-versus-asked gap (4.1 planned vs 3.3 asked on BIRD-Lite), trajectory inspection showing the agent silently skips identified questions once it gains confidence, and the question-budget study showing brute-force questioning has diminishing returns. The case study in Appendix D actually demonstrates the add/drop behavior rather than asserting it. The authors also deserve credit for disclosing the solvability filter (171/300 and 225/600 retained) and stating outright that implementation-level ambiguities are normalized away by execution-based evaluation. They know their scope.\n\nThe soft spot is the one the stress-test flags, and it's real: the groundedness metrics — ambiguity recall and silent failure, which are the paper's value-add over ordinary text-to-SQL — are scored by the user simulator's encoder mapping the agent's questions to annotated ambiguities. The same model family (Claude-Opus-4.8) generates the questions as system agent and judges whether they address the annotations. The cited validation of the two-stage protocol (Huo et al. 2026) targeted user-simulation fidelity, not this mapping's use as coverage ground truth on compositional multi-ambiguity instances. So the main evidence is not independently verified. That said, this isn't rigging: Figure 2(b) shows the encoder does classify many questions as non-critical or unanswerable, so it isn't trivially permissive. The honest fix is external validation of the mapping plus significance-aware reporting. Missing error bars is a genuine weakness on its own — on Spider the deltas are +1.1 recall and -0.7 silent failure over 297 instances, which is plausibly noise, and the BIRD-Lite silent-failure drop from 10.5% to 8.2% on 171 instances is thin.\n\nBottom line: the conceptual claim — correct execution doesn't imply resolution of underspecification, and identification doesn't imply maintenance — holds up, and PlanPool is a real design option for interactive agents. The quantitative foundation is weaker than the narrative. I wouldn't take the headline numbers at face value until the mapping is validated independently.\n\nThis is for people building interactive text-to-SQL or clarification-based agents, and anyone studying agent commitment behavior. It deserves a serious referee, with a demand for significance testing and independent grounding validation before it's accepted in a form that keeps these numbers.","headline":"Genuinely new mechanism — enforced, externalized clarification commitments with add/drop/next actions — built on a real abandonment diagnosis, but the headline groundedness numbers are scored by the same user-simulator encoder that produces the evidence, and there are no error bars anywhere.","tokens_in":22168,"tokens_out":2400,"duration_ms":18605},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A correct SQL result can still leave the user's intent unresolved.","keywords":["agentic text-to-sql","underspecification","clarification planning","groundedness","LLM agents","premature clarification termination","question pool","silent failures"],"falsifier":"Re-evaluate ambiguity coverage on the same interaction trajectories with an independent labeler — human annotators or a second, more conservative encoder — and check whether PLANPOOL's recall and silent-failure advantages over Plan-Prompt persist; if the gains shrink or the ranking flips under a stricter mapping, part of the reported improvement is an artifact of the measurement encoder rather than of the pool mechanism.","tokens_in":22700,"feed_emoji":"💬","tokens_out":5175,"duration_ms":32266,"temperature":0.7,"pith_summary":"Agentic Text-to-SQL systems can ask users to clarify underspecified requests, but this paper argues that a correct SQL result is no proof that the underlying ambiguities were actually resolved: an agent can silently assume an interpretation that happens to match the expected answer. The paper shows that this gap is driven partly by premature clarification termination, and that merely forcing the agent to ask more questions helps only with sharply diminishing returns, because later questions drift toward non-critical or unanswerable issues. It further shows that instructing an agent to plan its clarification does not fix the problem: the agent abandons questions it had already flagged as relevant. The paper's central claim is that identifying missing information is not sufficient — the agent must also maintain and resolve its identified ambiguities before committing — and it introduces PLANPOOL to enforce exactly that discipline.","feed_headline":"Forcing agents to track their planned questions lifts ambiguity recall","feed_subtitle":"PlanPool makes every flagged ambiguity be asked, dropped, or added before SQL submission, cutting silent failures.","key_machinery":"The central object is PLANPOOL, a mutable question pool that externalizes the clarification plan. After exploring the datasource, the agent generates a ranked list of clarification questions (pool_generate), then iteratively asks the top question (pool_next), inspects the pool (pool_view), removes a question that has become redundant or irrelevant (pool_drop), or adds newly discovered ambiguities (pool_add). Submission is blocked while any question remains in the pool, so every identified clarification must be explicitly discharged rather than silently forgotten. The pool turns planning from soft text the model may ignore into enforceable state that is also adaptive, since roughly a third of BIRD instances modify the pool during interaction.","core_discovery":"The paper's central claim is that in interactive Text-to-SQL, execution correctness and resolution groundedness are distinct: a submitted query counts as a grounded success only if it both executes correctly and explicitly addresses every annotated ambiguity, while a correct query that leaves an ambiguity unresolved is an ungrounded success and an incorrect one is a silent failure. On this accounting, even in interactive settings capable agents leave a meaningful share of ambiguities untouched, and the failure is not primarily a shortage of questions. Forcing a fixed question budget raises execution accuracy but spends the later questions on non-critical and unanswerable items, and prompt-based planning identifies relevant questions yet fails to keep asking them. PLANPOOL makes the clarification plan into persistent external state that must be emptied — by asking, explicitly dropping, or adding items — before SQL submission, and on three benchmarks it raises ambiguity recall and grounded success while cutting silent failures at competitive execution accuracy.","pith_inferences":["A natural robustness test is to sweep the user simulator's mapping strictness and verify that the relative ordering of methods — not just absolute numbers — is stable; the paper's own evidence does not settle this.","The pool_add action lets the agent surface ambiguities not in the annotation, and the paper does not measure the precision of these added questions; some of the added interaction cost likely contributes nothing to recall.","The underlying principle — deferred obligations must be externally maintained rather than left in conversational context — plausibly extends beyond clarification to other agentic settings where an agent identifies work it later forgets, such as verification steps or multi-step tool use."],"forward_implications":["Execution accuracy and grounded success should be reported as separate metrics wherever user requests are underspecified, because a correct result can rest on an unverbalized assumption.","Brute-force clarification budgets are inefficient: later questions increasingly target non-critical or unanswerable issues, so the goal is resolving the important ambiguities rather than maximizing question count.","Prompt-based planning alone is insufficient: an agent that can identify ambiguities will still abandon them absent a mechanism that forces explicit accounting before submission.","The pool mechanism is adaptive rather than a rigid checklist, with added questions roughly three to seven times more frequent than dropped ones across the benchmarks."],"supporting_citations":[{"why":"Supplies the BIRD-Interact benchmarks, the datasource interaction tools, the base agent prompt, and the function-driven user-simulation protocol that the paper adopts.","marker":"Huo et al. (2026)"},{"why":"Provides Spider, the source benchmark that the paper corrupts into its multi-ambiguity evaluation set.","marker":"Yu et al. (2018)"},{"why":"Supplies the PRACTIQ corruption operators that the paper reuses and composes to introduce multiple ambiguities per Spider instance.","marker":"Dong et al. (2025)"},{"why":"Provides the ReAct-style reasoning-and-acting harness that all compared methods share as their base interaction loop.","marker":"Yao et al. (2023)"},{"why":"Motivates the Plan-Prompt plan-then-execute baseline, the closest comparison to PlanPool.","marker":"Wang et al. (2023a)"},{"why":"Grounds the decision to filter BIRD-Interact down to intent- and knowledge-level ambiguities by showing that implementation-level variations are normalized away by execution evaluation.","marker":"Liu & Erdelt (2026)"},{"why":"Identifies Claude-Opus-4.8, the backbone LLM used for both the system agent and the user simulator across all compared methods.","marker":"Anthropic (2026)"}],"fun_headline_variants":["PlanPool forces agents to ask or drop every planned question before SQL","External question pool makes agents resolve every planned ambiguity","PlanPool: track clarification questions, cut silent SQL failures","Instead of prompting PlanPool externalizes the question plan to reduce silent errors"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The groundedness measurement depends on a single LLM encoder inside the user simulator: it decides whether a clarification question is answerable and maps that question to annotated ambiguities, and the paper's recall and silent-failure numbers count an ambiguity as addressed only when that encoder makes the mapping, so those results are not independently verified.","fun_headline_variants_meta":{"raw":{"variants":["PlanPool forces agents to ask or drop every planned question before SQL","External question pool makes agents resolve every planned ambiguity","PlanPool: track clarification questions, cut silent SQL failures","Instead of prompting PlanPool externalizes the question plan to reduce silent errors"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"cost":0.000439488,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000439488,"upstream_inference_prompt_cost":0.000103488,"upstream_inference_completions_cost":0.000336},"total_tokens":2144,"prompt_tokens":944,"completion_tokens":1200,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1128}},"tokens_in":944,"tokens_out":1200,"cost_source":"openrouter_usage","duration_ms":7261,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1128,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-05T19:02:12.060250+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-evaluate ambiguity coverage on the same interaction trajectories with an independent labeler — human annotators or a second, more conservative encoder — and check whether PLANPOOL's recall and silent-failure advantages over Plan-Prompt persist; if the gains shrink or the ranking flips under a stricter mapping, part of the reported improvement is an artifact of the measurement encoder rather than of the pool mechanism.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}