{"id":"4681a31c-9e7a-4229-9a2b-6d872d6d37e9","arxiv_id":"2509.10963","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.","lead":"This paper introduces a statistical test that compares an AI model's answer distribution for a new query against a group of slightly different but semantically equivalent queries. It aims to prevent small wording changes from causing false alarms about whether the model's behavior changed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm C cannot deliver the claimed consistency: its fixed ε-grid keeps ε>0 and m bounded, so power against alternatives within ε of P0 does not converge to 1.","rationale":"The paper's stated contribution is an asymptotically valid and consistent test. Corollary 2 supports validity of Algorithm C under the authors' assumptions, but no result proves consistency of Algorithm C. The only consistency statement, Corollary 1, imposes ε→0 and m→∞, which Algorithm C's fixed grid cannot satisfy: the smallest positive grid point is selected for large budgets, leaving ε bounded away from 0 and m bounded. For alternatives at distance ε*/2 from P0, the ideal-test power is (1−ε*/(2(b−a)))^{m*}, a constant below 1, so the realistic test is not consistent. This is an internal gap, not a disagreement with consensus. The reader's weakest assumption—uniformity of null Bernoulli parameters—is also a genuine limitation and is acknowledged in Section 6, but it concerns robustness of Type I error under real data; the Algorithm C consistency failure is more load-bearing because it contradicts the central claim even when all assumptions hold. A concrete trace and Monte Carlo at p′=0.3975 settles the issue. Verdict remains CONDITIONAL: the paper should either add a shrinking-η schedule with proof that Corollary 1's conditions are met, or revise the abstract and consistency claims.","tokens_in":23207,"tokens_out":11049,"duration_ms":136705,"concrete_test":"Instantiate Algorithm C with α=0.1, oracle (a,b)=(0.4,0.6), η=0.005, ν=10^6,10^8,10^10. Record (ε*,m*,r*). If ε* stays 0.005 and m* stays ≈91, evaluate Lemma 3 at p′=0.3975=a−ε*/2: power limit = (1−0.005/(2·0.2))^91 ≈ 0.33. Then run 10^4 Monte Carlo trials with r=r*, m=m*, ε=ε* and p′ fixed at 0.3975, p_j~Unif[0.4,0.6], and Bernoulli estimates; confirm the empirical rejection probability is ≈0.33 for all three budgets, not tending to 1. If instead the authors intend η_ϵ to shrink with ν, the test is to verify that the chosen sequence satisfies ε→0, m→∞, r=ω(m^2) and to prove Corollary 1 applies.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract's 'consistent' is not established for Algorithm C. Corollary 1 gives consistency only under ε→0, m→∞, r=ω(m^2). Algorithm C (Section 3.1) searches ε over a fixed grid {0, η, 2η, ..., ε_max} and sets m=ceil(|log α|/|log(1−ε/(b̂−â))|), r=ν/m. As ν→∞ with η fixed, the approximate power objective H is decreasing in ε once the validity constraint binds ((1−ε/c)^m ≈ α at the boundary), so the selected ε* is the smallest positive grid value, i.e., ε*≈η; m* is bounded (O(c/η log(1/α))) and r*→∞. Thus the Corollary 1 conditions ε*→0 and m*→∞ fail. For p′=a−ε*/2 (which lies in P1), Lemma 3 gives ideal-test power (1−ε*/(2(b−a)))^{m*}, a constant strictly less than 1; the realistic test inherits this limit by Theorem 3. Hence P(reject|p′) does not approach 1. Corollary 2 proves only asymptotic validity, so the consistency half of the central claim is unsupported and is in fact false for Algorithm C as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of testing whether the binary response probability p' of a new query belongs to an unknown set P0 of probabilities induced by semantically irrelevant perturbations of a base query. Since the query-to-probability map is unknown, the test samples m null queries, estimates each p_j from r responses, forms the statistic T_{m,r} = min_j |p_hat_j - p_hat'|, and rejects H0: p' in P0 if T_{m,r} > epsilon. Under Assumptions 1-3, the paper proves: a finite-sample closeness lemma between the realistic statistic and an oracle statistic (Lemma 1, Theorem 1), an upper bound on size (Theorem 2), a lower bound on power (Theorem 3), an average-power formula (Theorem 4), and consistency/validity conditions (Corollary 1). Algorithm C selects (epsilon,m,r) under a budget constraint by maximizing an estimated power lower bound subject to an estimated validity constraint. Experiments on LLaMA-3-8B illustrate the method on the 'R.A. Fisher' motivating example.","tokens_in":23560,"tokens_out":9167,"duration_ms":112442,"significance":"If the claims held as stated, the paper would provide a practical template for composite-null testing when the null set is unknown but can be sampled, with clean Hoeffding-type bounds and explicit budget allocation. The manuscript is transparent about experimental cost and acknowledges limitations. However, two load-bearing issues must be addressed: (i) Algorithm C as written does not implement the consistency conditions of Corollary 1, so the abstract's consistency claim is unsupported for the proposed procedure; and (ii) the key uniformity assumption (Assumption 2) is contradicted by the paper's own motivating null set, so the Type I error guarantee does not apply to the flagship application.","major_comments":[{"comment":"The abstract claims the proposed test is asymptotically valid and consistent, but Algorithm C does not deliver consistency. Corollary 1 requires epsilon->0, m->infty, and r=omega(m^2). Algorithm C searches epsilon over a fixed grid {0, eta, 2eta, ..., epsilon_max} independent of the budget nu; for each fixed epsilon, m is set by line 6 and r=nu/m. As nu->infty with eta fixed, the selected epsilon* converges to the smallest positive grid value eta, m* stays bounded at about |log alpha|/|log(1-eta/(b-a))|, and only r*->infty. For p'=a-eta/2 in P1, the ideal-test power equals (1-eta/(2(b-a)))^{m*} <1, and by Theorem 3 the realistic power has the same constant limit. Thus P(reject|p') does not approach 1. Corollary 2 proves only asymptotic validity, not consistency. The algorithm needs to let the grid spacing (or epsilon*) tend to 0 with nu while maintaining m*->infty and r=omega(m^2), or th","section":"§3.1 (Algorithm C), §4.3 (Corollary 1)"},{"comment":"The size guarantee of Theorem 2 and the validity constraint in Algorithm C rely on Assumption 2, p_1,...,p_m i.i.d. Uniform[a,b]. The paper's own motivating data violate this assumption: the histogram in Fig. 4 (left) shows estimated null probabilities concentrated near 0.8-1.0, and Section 6 explicitly acknowledges the non-uniformity of the Bernoulli parameters. Therefore the Type I error control that motivates the paper is not guaranteed for the actual LLM experiment. The authors should either restrict the claims to the uniform model or provide robustness analysis/empirical calibration that addresses non-uniform null distributions before claiming the method solves the motivating problem.","section":"§4.2 (Theorem 2), §6 (Discussion), Fig. 4"},{"comment":"Corollary 2's proof is a one-line convergence argument for (b_hat-a_hat) to (b-a). This gives asymptotic validity only if the selected triple (epsilon,m,r) also satisfies the true size bound in the limit; the current proof does not account for the fact that the algorithm's validity constraint is checked with plug-in estimates and the objective is maximized over a grid. This is likely repairable, but as written it is not a complete argument.","section":"§3.1 (Algorithm C), §4.4 (Corollary 2)"}],"minor_comments":[{"comment":"The grid includes epsilon=0, where the denominator |log(1-epsilon/(b_hat-a_hat))| is zero. The algorithm should either start the grid at eta or handle epsilon=0 separately.","section":"§3.1 (Algorithm C, line 6)"},{"comment":"Section 5.2 says the unbiased estimates are 'provided in Algorithm B', but Algorithm B returns raw sample extremes, while Remark 1 proposes bias-corrected versions. Please reconcile the implementation description with the estimator actually used.","section":"§5.2, Algorithm B, Remark 1"},{"comment":"The expression '2/1-(b_hat-a_hat)' should read 2/(1-(b_hat-a_hat)) for clarity; the current typesetting is easy to misread.","section":"§3.1 (Algorithm C, line 10)"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core under Assumptions 1-3 is sound and the paper is likely to be of interest to the journal's readership. The main revision needed is to fix the consistency claim: either modify Algorithm C so that epsilon* can tend to 0 as the budget grows (e.g., by letting the grid spacing depend on nu) or explicitly limit the paper's claims to asymptotic validity plus consistency under the idealized conditions of Corollary 1. The uniformity violation in the motivating data should also be discussed more prominently, perhaps with an empirical size check under the non-uniform null distribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core: they formalize a real problem—classical two-sample tests reject semantically equivalent query variants—and set up a composite null over semantic perturbations, with a min-distance test on estimated Bernoulli parameters. The finite-sample bounds (Lemmas 1–4) are standard Hoeffding-plus-union-bound work and not circular; plugging in (a_hat,b_hat) is honest. Corollary 1's asymptotic validity/consistency under ε→0, m→∞, r=ω(m²) is plausible.\n\nBut the paper advertises this for the proposed test, and Algorithm C is the test. Algorithm C fixes an ε-grid with mesh η and chooses m only as small as validity allows, so as ν→∞, ε* stays at the smallest grid value and m* is O(c/η log(1/α))—a constant. For p' just outside [a,b], e.g. p'=a−ε*/2, the ideal power is (1−ε*/(2c))^m*, a constant bounded away from 1, and Theorem 3 gives the same limit for the realistic test. So P(reject|p') does not go to 1. The stress-test note is right: consistency fails for Algorithm C as written. Corollary 2 proves only validity, with a one-line sketch, so the abstract's consistency claim is unsupported.\n\nSecond soft spot: Assumption 2, uniform null probabilities, is doing the work for size control, and the paper's own Figure 4 histogram is clearly non-uniform. The authors acknowledge this in Section 6, which is honest, but it means the Type I error guarantee does not cover the motivating LLM setting. That is a limitation, not a fixable proof detail.\n\nMinor point: the reported experiment with (a_hat,b_hat)=(0.898,1) and ε**=0.085 seems to violate Algorithm C's own ε_max = min(a_hat, b_hat-a_hat, 1-b_hat) rule, which would cap the grid at 0. Needs clarification, probably a typo, but currently the numbers don't reproduce.\n\nFor a reader in statistical evaluation of generative models, this is still a genuinely new formulation and a decent template. It deserves a serious referee, but the revision must fix the consistency claim (either drop it for Algorithm C or let ε shrink with ν) and get the uniformity caveat into the main claims. I'd bring it to reading group and would cite it in an LLM-eval methods context, with the consistency caveat attached.","headline":"The paper has a genuinely useful new framing and the core math is sound under its assumptions, but the algorithm's advertised consistency claim is not proven and is in fact false as written; the key uniformity assumption also conflicts with the paper's own data.","tokens_in":24001,"tokens_out":7047,"would_cite":true,"duration_ms":77941,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62F05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A new test decides whether a query's response distribution belongs to the family induced by semantically equivalent queries, proving asymptotic validity and consistency under a fixed sampling budget.","keywords":["LLM response distributions","hypothesis testing","composite null","perturbation robustness","Bernoulli parameters","Type I error control","power","budget-constrained testing"],"falsifier":"Build a synthetic family of null queries whose true 'yes' probabilities sit at just two points, say 0.3 and 0.7, with nothing in between, and choose a test query whose probability is 0.5 — inside the interval but not equal to any null query's probability. Run the proposed algorithm with a large budget and simulate many replicates. If the observed rejection frequency exceeds the nominal alpha as the budget grows, the uniform-spread assumption is doing the work; this directly tests the claimed size bound, which the paper's own non-uniform histogram already suggests will fail.","tokens_in":23131,"feed_emoji":"⚖️","tokens_out":7005,"duration_ms":80555,"temperature":0.7,"pith_summary":"The paper addresses a mismatch between statistical significance and user intent when testing responses from generative models: a tiny, semantically irrelevant edit to a query (for example, 'RA' versus 'R.A.') can measurably shift the probability of a 'yes' answer, so a classical test would reject equality even though no user would care. The proposed remedy is to test against a composite null, the whole family of response distributions produced by a user-chosen set of semantically equivalent queries, even though the mapping from queries to distributions is unknown and must be estimated under a fixed sampling budget. Restricting to binary responses, the paper defines a minimum-distance test, derives finite-sample bounds for its size and power, and shows that choosing the threshold and sample sizes via its algorithm yields an asymptotically valid and consistent test. This gives practitioners a principled way to ask whether an observed change in LLM behavior is real relative to harmless variation in phrasing, rather than merely statistically detectable. The main caveat is the paper's reliance on a uniformity assumption for the null probabilities, which its own motivating data visibly violate.","feed_headline":"New test separates real LLM response shifts from wording noise","feed_subtitle":"Controls false alarms when a small edit changes the model's answer odds, within a fixed query budget.","key_machinery":"The test statistic is the minimum absolute difference T_{m,r} = min_j |hat p_j - hat p'|; the decision rule rejects when T_{m,r} > epsilon. The load-bearing analytic tool is a pair of bounds: Lemma 1 shows |T_{m,r} - Ttilde_m| is small with high probability (using Hoeffding-type concentration on Bernoulli sums), so the realistic test inherits the ideal test's behavior. The size bound (Theorem 2) and the average-power bound (Theorem 4) both depend on the uniformity of the null probabilities on [a,b]; Algorithm C uses pilot estimates (hat a, hat b) to turn these bounds into a concrete choice of (m,r,epsilon).","core_discovery":"The central claim is that, despite not knowing the map from semantically similar queries to response probabilities, one can still test whether a new query's response distribution belongs to the induced null set with asymptotic Type I error control and asymptotic power one, provided responses are binary and the total sampling budget is fixed. The authors propose a minimum-distance rule: estimate the Bernoulli parameter of the test query and of m sampled null queries, and reject H0: p' in P0 when the smallest of the m absolute differences exceeds a threshold epsilon. They prove that this realistic test tracks an ideal test that knows the true parameters, with estimation error bounded by 2m/sqr","pith_inferences":["If the uniform assumption fails, the Type I error control could be rescued by first estimating the null distribution's support and shape; a mixture-of-Betas prior, as the paper proposes, would supply a richer family but would require new bounds.","The method's reliance on a pilot estimate of [a,b] means the effective size depends on pilot quality; a double-bootstrap or a conservative widening of the interval would make the test more robust at the cost of power.","For non-binary responses, the same minimum-distance idea could be applied to embeddings, but the concentration argument would need a replacement for the Hoeffding bound; the energy-test comparison in the paper's Figure 5 suggests that a distribution-free version is still open.","A direct practical test: choose a null family with tightly clustered probabilities (as in Figure 4) and measure the empirical rejection rate at nominal alpha; if the rate exceeds alpha, the stated guarantee is not operative at realistic budgets."],"forward_implications":["With binary responses and a fixed budget, one can run a valid test that rejects only when the test query's probability is far from all sampled null query probabilities, accounting for estimation noise.","The Type I error is asymptotically controlled at the nominal level even though the null set is unknown; the unknown interval [a,b] can be replaced by pilot estimates without breaking validity as the budget grows.","Power converges to 1 for any test query whose true probability is bounded away from the null interval, provided r = omega(m^2) and epsilon -> 0.","The procedure reduces unwanted rejections in the motivating example: changing 'RA' to 'R.A.' no longer triggers rejection, while changing 'statistician' to 'eugenicist' still does.","Algorithm C provides a concrete recipe for choosing m, r, and epsilon by maximizing an estimated lower bound on average power subject to an approximate validity constraint."],"fun_headline_variants":["Test LLM responses despite harmless query rewording","New hypothesis test for LLM outputs, robust to wording tweaks","Binary-outcome test separates LLM response shift from noise","LLM response test with fixed budget: robust to phrasing changes","No false alarms when LLM query is reworded: new test"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The false-rejection guarantee rests on the assumption that the null queries' 'yes' probabilities are spread uniformly across an unknown interval; the paper's own motivating data show a strongly non-uniform spread, so the guarantee is not assured for the very scenario the test was built for.","fun_headline_variants_meta":{"raw":{"variants":["Test LLM responses despite harmless query rewording","New hypothesis test for LLM outputs, robust to wording tweaks","Binary-outcome test separates LLM response shift from noise","LLM response test with fixed budget: robust to phrasing changes","No false alarms when LLM query is reworded: new test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1328,"prompt_tokens":711,"completion_tokens":617,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":455,"tokens_out":617,"duration_ms":8038,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:18:30.998981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a synthetic family of null queries whose true 'yes' probabilities sit at just two points, say 0.3 and 0.7, with nothing in between, and choose a test query whose probability is 0.5 — inside the interval but not equal to any null query's probability. Run the proposed algorithm with a large budget and simulate many replicates. If the observed rejection frequency exceeds the nominal alpha as the budget grows, the uniform-spread assumption is doing the work; this directly tests the claimed size bound, which the paper's own non-uniform histogram already suggests will fail.","supporting_citations":[],"review_version":1}