{"id":"b3ab81b4-c742-4ffc-814f-168c3105a580","arxiv_id":"2507.17896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VeriMinder detects cognitive biases in NL2SQL questions and generates refined, more robust analytical questions, outperforming baselines in human and LLM evaluations.","lead":"VeriMinder is a new web tool that checks natural-language database questions for cognitive biases, like asking for the largest loans instead of the riskiest ones, and suggests safer reformulations. The paper reports that users and analysts rate the tool's refined questions as more accurate, concrete, and comprehensive than several baseline approaches.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20% superiority claim is measured against self-implemented baselines that are not yet released; if the baseline prompts are weak or unfaithful operationalizations, the comparison is invalidated.","rationale":"The reader's weakest assumption is that the crafted BIRD-DEV evaluation, rated by two professional analysts and an LLM judge, generalizes to actual NLIDB users. I agree external validity is a concern, but I identify a distinct internal-validity risk: the comparison baselines are self-defined and unreleased. This is more load-bearing because even on the reported benchmark, the 20% figure is only meaningful if the baselines are fair and faithful implementations of the cited alternatives. The paper explicitly says baselines will be released in the future, so the current evidence is not independently checkable. My concrete test would re-run the evaluation with a faithful PerQS and equal compute; if the gap persists, the claim is robust, and if it disappears, the claim fails. This does not require changing the reader's CONDITIONAL verdict, but it highlights that the primary condition for trust is baseline transparency and fairness, not only larger rater pools or unseen benchmarks.","tokens_in":12364,"tokens_out":9106,"duration_ms":103345,"concrete_test":"Download the repository at reproducibility.link/veriminder and verify whether exact prompts and a runnable evaluation harness for all four baselines are included. Then re-run the §3.3 comparative evaluation with the PerQS baseline replaced by a faithful implementation from its cited source (Zhu et al., 2024a), giving each baseline the same number of candidate generations and self-reflection iterations as VeriMinder. If the mean accuracy or comprehensiveness gap against PerQS falls below 20% or loses significance at p<0.05, the claimed superiority is an artifact of the baseline configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; §3.3) is that VeriMinder is 'at least 20% better' than four baselines on accuracy, concreteness, and comprehensiveness. These baselines are not established systems: §3.1 states that the authors operationalized Decision-Focused Query Generation, Question Perturbation (PerQS), and Critic-Agent Feedback themselves, using the same LLM (Gemini Flash 2.0), and 'plan to release them as part of our code release.' The linked repository is described only as VeriMinder's code and prompts, not the baseline harness. Thus the headline comparison depends entirely on the faithfulness and strength of unverified, self-defined baselines. The asymmetry is concrete: VeriMinder's pipeline generates twelve candidate templates, evaluates them with a panel of three LLM critics, and applies a self-reflection pass (§2.2.4–2.2.6). If the baselines are implemented as simple single-pass prompts, the reported improvements (22–87% depending on baseline) are an expected product of asymmetric engineering effort, not of the analytical framework. This is especially acute for PerQS, the strongest baseline: the cited work (Zhu et al., 2024a) is PromptRobust, which addresses adversarial prompt robustness, not 'question perturbation' for analytical quality; the mapping is not validated. Section 3.6's acknowledgment of BIRD-DEV leakage and crafted vulnerability-prone scenarios (§3.1) further limits scope, but the baseline problem is the most load-bearing because it threatens the internal validity of the 20% number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VeriMinder, an interactive system that sits between a user and an NL2SQL engine, detecting cognitive biases in analytical questions and suggesting refinements before query execution. It introduces a three-stage architecture (Data Preparation, Analytical Validation, Refinement Synthesis), a Hard-to-Vary-inspired HV score, and an LLM prompt-generation pipeline with twelve candidate templates, three critics, and self-reflection. Evaluation comprises a user study with 63 Prolific participants, a comparative rating by two data analysts on 59 scenarios, and an automated LLM-judge evaluation on 100 scenarios; the paper claims VeriMinder outperforms four baselines by at least 20% on accuracy, concreteness, and comprehensiveness.","tokens_in":12694,"tokens_out":5097,"duration_ms":50092,"significance":"If the claims hold, VeriMinder addresses a genuine, under-explored gap in NL2SQL research: ensuring the analytical question is sound, not just the SQL. The paper is transparent about many limitations, ships an MIT-licensed codebase with prompts, and includes a synthetic validation suite for the HV score. However, the current evidence is weaker than the abstract suggests because the comparative evaluation rests on self-implemented, unreleased baselines, a two-rater human study, and an LLM judge calibrated on 15 examples; the reported effect sizes therefore should be treated as upper bounds until the baselines and evaluators are independently validated.","major_comments":[{"comment":"The headline claim of 'at least 20% better' rests on three self-implemented baselines (Decision-Focused Query Generation, Question Perturbation, and Critic-Agent Feedback) whose prompts are not included and are only 'planned' for release. The paper does not specify how these systems were operationalized, how much engineering effort went into them, or whether the cited work supports the chosen design; notably, Zhu et al. (2024a) is a study of adversarial prompt robustness, not of question perturbation for analytical quality. Because VeriMinder uses a twelve-candidate ensemble with three critics and a self-reflection pass, the reported 22–87% improvements could be an artifact of asymmetric prompt engineering rather than of the analytical framework. To make the comparison valid, the authors should release the exact baseline prompts and, ideally, compare against publicly available systems or published prompts.","section":"§3.1, §3.3"},{"comment":"The external validity of the quantitative evaluation is limited by two design choices. First, the comparative study used only two data analysts, one from each of two US companies, which provides little evidence about how the system performs across user populations; the reported Gwet's AC1 values do not compensate for the tiny rater pool. Second, the automated evaluation uses Gemini Flash 2.0 as judge, the same model family used for the NL2SQL component, and the critique prompts in VeriMinder are generated by Claude 3.7 Sonnet, the same model used as critic; the calibration for the LLM judge is based on only 15 examples with r=0.74. The paper should either use a held-out, independently labeled set for calibration or clearly restrict the 'at least 20%' claim to the specific evaluation setting.","section":"§3.3, §3.4"},{"comment":"The dataset is explicitly hand-crafted to be vulnerability-rich and is derived from BIRD-DEV, which the authors acknowledge LLMs may have seen during training. This makes the measured improvements a best-case estimate rather than a measure of real-world impact: real users may ask questions with fewer or different vulnerabilities, and the system's benefits may be smaller outside this benchmark. The paper should hedge the abstract's general claim and, if possible, include an evaluation on unseen or newly collected questions.","section":"§3.1, §3.6"},{"comment":"The HV score in Eq. (1) is presented as the theoretical target, but the deployed system uses LLM critic scores as a proxy for I(T;S) and a 'structured analytical process' as a proxy for DL(S). The synthetic Bayesian-network validation validates only the formal HV score with DL(S)=|S|, not these proxies. Since the paper openly acknowledges that the proxies are not formal equivalents, the claims 'operationalizes the Hard-to-Vary principle' and 'analytical framework' should be treated as design inspiration rather than as verified instantiations; the authors should either add a validation of the proxies or soften the framing in Sections 1 and 3.3.","section":"§2.2.2, §2.2.3"}],"minor_comments":[{"comment":"The sentence 'The system using generates a diverse set of candidates using twelve prompt templates' contains a typo ('using generates'); it should read 'The system generates a diverse set of candidates using twelve prompt templates.'","section":"§2.2.4"},{"comment":"The phrase 'with paired t_test' should be 'with a paired t-test', and the sentence should be rephrased for grammatical completeness.","section":"§3.3"},{"comment":"The calibration statement 'finding a m (Pearson’s r = 0.74' has a truncated metric name; please specify what 'm' stands for (e.g., 'a moderate correlation').","section":"§3.4"},{"comment":"The word-cloud analysis in Figure 8 is qualitative; if it is meant to support claims about bias mitigation, consider reporting quantitative counts or inter-annotator agreement.","section":"§3.5"},{"comment":"The SLOW framework is abbreviated without being defined in the main text; please define 'Sure, Look, Opposite, Worst' at first use.","section":"Appendix B.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about limitations, but the abstract overclaims relative to the evidence. I would ask the authors to release the baselines and re-run the evaluation before considering acceptance; the novelty is real and the system is well-engineered, so this is a revision, not a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea in VeriMinder is a real improvement over the usual NL2SQL focus: it stops the query at the question stage, checks for cognitive biases, and suggests refinements grounded in an explicit analytical framework. The authors integrate a bias taxonomy, Toulmin argument evaluation, counter-argument checks, and a multi-candidate LLM prompt pipeline into an interactive system, and they ship the code and prompts under an open license. The synthetic validation of the HV score on Bayesian networks is a good faith check. They also honestly flag the big limitations: BIRD-DEV leakage, the heuristic nature of the proxies, and the fact that the evaluation scenarios were deliberately vulnerability-rich. That level of candor is unusual and earns real credit.\n\nThe soft spot is the headline number. The paper claims at least 20% improvement over four baselines, and those baselines are self-implemented, unreleased, and likely not as engineered as VeriMinder's twelve-template, three-critic, self-reflection pipeline. If the baselines are simple single-pass prompts, the improvement is an expected product of asymmetric effort, not of the analytical framework. The mapping of the strongest baseline, Question Perturbation, to PromptRobust—which is about adversarial robustness—looks wrong on its face. The comparative evaluation itself used only two human raters, and the LLM judge was calibrated on 15 examples with r=0.74. With a deliberately bias-heavy dataset, the measured effect size is hard to trust as a real-world expectation.\n\nNone of this kills the contribution. The problem is genuine, the system is thoughtfully built, and the paper is honest about what it does not claim. But the magnitude of the benefit is not yet established. A revision that releases the baseline prompts, adds more raters, and tests on naturally-occurring analyst questions would settle the question.\n\nThis paper is for people building NLIDBs or analytical assistant tools, and for anyone working on LLM-assisted question refinement. It is worth a serious referee. My recommendation: send it to review, and in the revision require the baseline harness and a stronger evaluation design. Treat the 20% figure as provisional.","headline":"A genuinely useful system for bias mitigation in NL2SQL, but its headline 20% improvement rests on self-implemented baselines that need to be released and strengthened before the number can be trusted.","tokens_in":13184,"tokens_out":2838,"would_cite":false,"duration_ms":30945,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim: detecting cognitive biases in the user's question—not just SQL generation accuracy—is what verifiably improves analytical outcomes in NL2SQL workflows.","keywords":["NL2SQL","cognitive bias mitigation","analytical question formulation","natural language interfaces to databases","Hard-to-Vary principle","LLM-based refinement","decision contexts","text-to-SQL"],"falsifier":"Run the same comparative evaluation on decision scenarios assembled from databases released after the LLM's training cutoff, and if VeriMinder no longer beats Question Perturbation by at least 20%, the benchmark-memorization concern is confirmed.","tokens_in":12160,"feed_emoji":"🛡️","tokens_out":7734,"duration_ms":72029,"temperature":0.7,"pith_summary":"VeriMinder is a web tool that tries to fix the question, not just the SQL. The paper's central claim is that text-to-SQL systems, even when they generate correct SQL, can produce misleading analyses when the user's original question is biased or framed around the wrong variables; VeriMinder detects such analytical vulnerabilities and suggests refinements before the query runs. The authors report that in a comparative evaluation with two data analysts, VeriMinder beat Direct NL2SQL by 60.4% in accuracy, 63.2% in concreteness, and 86.9% in comprehensiveness, and beat the strongest baseline, Question Perturbation, by 21–28% on the same metrics. They also report that 82.5% of 63 study participants said the tool improved the quality of the analysis. If true, the work shifts attention from SQL generation accuracy to the analytical quality of the question itself.","feed_headline":"VeriMinder catches biased data questions before the query runs","feed_subtitle":"Rewrites biased analytical questions, beating direct text-to-SQL by up to 86.9% in analyst ratings.","key_machinery":"The central mechanism is the Hard-to-Vary score, defined as mutual information between the selected analytical variables and the decision target divided by description length, $HV(S) = I(T;S)/DL(S)$. The paper uses this as a theoretical ideal from the Hard-to-Vary principle—an explanation is good if changing any part of it weakens or invalidates it—and then implements it through a three-stage pipeline: twelve prompt templates generate candidate refinements, a panel of three LLM critics (two sampled per candidate) scores them for insight, logic, and bias, and a self-reflection pass fuses the feedback into a final suggestion. The system also connects bias detection to SQL schema patterns and argument-structure checks, including Toulmin-style component analysis and counter-argument frameworks. The score is the object that ties the normative principle to the LLM prompts.","core_discovery":"On the paper's own terms, the discovery is that 'wrong question' errors in NL2SQL are a separate, addressable failure mode: a user asking for 'clients with the largest loans' when the decision is about 'loan accounts at risk' displays similarity, framing, and selection biases, and even a perfect SQL translator will answer the wrong question. VeriMinder's claim is that a structured, LLM-based refinement pipeline can detect these biases, using a taxonomy of 53 cognitive biases, schema patterns, and Toulmin-style argument checks, and produce a hard-to-vary analytical question that is more likely to inform the decision. The paper formalizes a Hard-to-Vary score $HV(S)=I(T;S)/DL(S)$ as the idealized target, then shows how a multi-candidate prompt-generation pipeline with critic feedback approximates it. The authors state that this approach yields 'at least 20% better' analytical outcomes across accuracy, concreteness, and comprehensiveness compared with baseline approaches.","pith_inferences":["A reader should treat the 20%+ figures as upper bounds on a crafted benchmark: the paper itself warns that BIRD-DEV may be memorized by LLMs, and the scenarios were deliberately designed to be vulnerability-prone.","The HV score $I(T;S)/DL(S)$ is a form of normalized mutual-information feature selection; the paper's real contribution is showing an LLM pipeline can act as its surrogate, not the formula itself, so the sharpest next test is comparing against simpler feature-selection-inspired prompts.","The paper's bias taxonomy is drawn from Western psychology literature (a point it concedes), so extending VeriMinder to other cultural or domain-specific taxonomies could change which vulnerabilities are caught—an untested direction.","The same refinement loop should carry over to Python/pandas analytical code generation and to business-intelligence dashboards, but that is an inference from the architecture, not something the paper evaluates."],"forward_implications":["If VeriMinder's stated improvements hold, NL2SQL systems should be evaluated on the analytical quality of the user's question—accuracy, concreteness, comprehensiveness relative to the decision—not only on execution accuracy.","Users without statistical training would receive targeted, explainable suggestions before the query runs, potentially reducing misleading conclusions that arise from well-formed SQL on biased questions.","The multi-candidate, critic-scored prompt pipeline would demonstrate that LLM-based refinement can approximate an information-theoretic objective in an interactive setting.","Because the system isolates question formulation, its gains would mean the same NL2SQL engine produces more decision-relevant results when paired with VeriMinder.","The stated improvements over Decision-Focused, Question Perturbation, and Critic-Agent baselines would establish a new comparison point for question-refinement research."],"supporting_citations":[{"why":"Supplies the Hard-to-Vary principle that the system operationalizes into the HV score.","marker":"(Deutsch, 2011)"},{"why":"Provides the BIRD-DEV benchmark questions from which all evaluation scenarios were derived.","marker":"(Li et al., 2023)"},{"why":"Supplies the Case Study Method used to craft the 164 decision scenarios.","marker":"(Ellet, 2007)"},{"why":"Defines the accuracy, concreteness, and comprehensiveness metrics used for evaluation.","marker":"(Zhu et al., 2024b)"},{"why":"The Task Alignment strategy that the experimental NL2SQL component extends.","marker":"(Qu et al., 2024)"},{"why":"Provides the SLOW framework used to structure the automated LLM evaluation.","marker":"(O’Sullivan and Schofield, 2019)"},{"why":"Grounds the description-length term in the Hard-to-Vary score.","marker":"(Rissanen, 1978; Grünwald, 2007)"},{"why":"Supplies the mutual-information definition used in the HV score.","marker":"(Cover and Thomas, 2006)"}],"fun_headline_variants":["VeriMinder spots biased questions before SQL runs","Detect and fix analytical bias in NL2SQL with VeriMinder","VeriMinder: guard against wrong analytical queries","Rewriting biased NL2SQL questions with VeriMinder","VeriMinder targets 'wrong question' errors in NL2SQL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benefit rests on the assumption that improvements measured on 164 hand-crafted decision scenarios from BIRD-DEV, rated by two professional analysts and an LLM judge, generalize to the questions real NLIDB users actually ask.","fun_headline_variants_meta":{"raw":{"variants":["VeriMinder spots biased questions before SQL runs","Detect and fix analytical bias in NL2SQL with VeriMinder","VeriMinder: guard against wrong analytical queries","Rewriting biased NL2SQL questions with VeriMinder","VeriMinder targets 'wrong question' errors in NL2SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4282,"prompt_tokens":1034,"completion_tokens":3248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":3161}},"tokens_in":650,"tokens_out":3248,"duration_ms":22750,"temperature":1.0,"reasoning_tokens":3161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:40.970318+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same comparative evaluation on decision scenarios assembled from databases released after the LLM's training cutoff, and if VeriMinder no longer beats Question Perturbation by at least 20%, the benchmark-memorization concern is confirmed.","supporting_citations":[],"review_version":1}