Pith. sign in

REVIEW 3 major objections 3 minor 27 references

A new test decides whether a query's response distribution belongs to the family induced by semantically equivalent queries, proving asymptotic validity and consistency under a fixed sampling budget.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:18 UTC pith:U6W6SM2E

load-bearing objection The paper has a genuinely useful new framing and the core math is sound under its assumptions, but the algorithm's advertised consistency claim is not proven and is in fact false as written; the key uniformity assumption also conflicts with the paper's own data. the 3 major comments →

arxiv 2509.10963 v1 pith:U6W6SM2E submitted 2025-09-13 math.ST cs.AIstat.MEstat.TH

Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations

Aranyak Acharyya , Carey E. Priebe , Hayden S. Helm This is my paper
classification math.ST cs.AIstat.MEstat.TH MSC 62F0362F05
keywords LLM response distributionshypothesis testingcomposite nullperturbation robustnessBernoulli parametersType I error controlpowerbudget-constrained testing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses a mismatch between statistical significance and user intent when testing responses from generative models: a tiny, semantically irrelevant edit to a query (for example, 'RA' versus 'R.A.') can measurably shift the probability of a 'yes' answer, so a classical test would reject equality even though no user would care. The proposed remedy is to test against a composite null, the whole family of response distributions produced by a user-chosen set of semantically equivalent queries, even though the mapping from queries to distributions is unknown and must be estimated under a fixed sampling budget. Restricting to binary responses, the paper defines a minimum-distance test, derives finite-sample bounds for its size and power, and shows that choosing the threshold and sample sizes via its algorithm yields an asymptotically valid and consistent test. This gives practitioners a principled way to ask whether an observed change in LLM behavior is real relative to harmless variation in phrasing, rather than merely statistically detectable. The main caveat is the paper's reliance on a uniformity assumption for the null probabilities, which its own motivating data visibly violate.

Core claim

The central claim is that, despite not knowing the map from semantically similar queries to response probabilities, one can still test whether a new query's response distribution belongs to the induced null set with asymptotic Type I error control and asymptotic power one, provided responses are binary and the total sampling budget is fixed. The authors propose a minimum-distance rule: estimate the Bernoulli parameter of the test query and of m sampled null queries, and reject H0: p' in P0 when the smallest of the m absolute differences exceeds a threshold epsilon. They prove that this realistic test tracks an ideal test that knows the true parameters, with estimation error bounded by 2m/sqr

What carries the argument

The test statistic is the minimum absolute difference T_{m,r} = min_j |hat p_j - hat p'|; the decision rule rejects when T_{m,r} > epsilon. The load-bearing analytic tool is a pair of bounds: Lemma 1 shows |T_{m,r} - Ttilde_m| is small with high probability (using Hoeffding-type concentration on Bernoulli sums), so the realistic test inherits the ideal test's behavior. The size bound (Theorem 2) and the average-power bound (Theorem 4) both depend on the uniformity of the null probabilities on [a,b]; Algorithm C uses pilot estimates (hat a, hat b) to turn these bounds into a concrete choice of (m,r,epsilon).

Load-bearing premise

The false-rejection guarantee rests on the assumption that the null queries' 'yes' probabilities are spread uniformly across an unknown interval; the paper's own motivating data show a strongly non-uniform spread, so the guarantee is not assured for the very scenario the test was built for.

What would settle it

Build a synthetic family of null queries whose true 'yes' probabilities sit at just two points, say 0.3 and 0.7, with nothing in between, and choose a test query whose probability is 0.5 — inside the interval but not equal to any null query's probability. Run the proposed algorithm with a large budget and simulate many replicates. If the observed rejection frequency exceeds the nominal alpha as the budget grows, the uniform-spread assumption is doing the work; this directly tests the claimed size bound, which the paper's own non-uniform histogram already suggests will fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • With binary responses and a fixed budget, one can run a valid test that rejects only when the test query's probability is far from all sampled null query probabilities, accounting for estimation noise.
  • The Type I error is asymptotically controlled at the nominal level even though the null set is unknown; the unknown interval [a,b] can be replaced by pilot estimates without breaking validity as the budget grows.
  • Power converges to 1 for any test query whose true probability is bounded away from the null interval, provided r = omega(m^2) and epsilon -> 0.
  • The procedure reduces unwanted rejections in the motivating example: changing 'RA' to 'R.A.' no longer triggers rejection, while changing 'statistician' to 'eugenicist' still does.
  • Algorithm C provides a concrete recipe for choosing m, r, and epsilon by maximizing an estimated lower bound on average power subject to an approximate validity constraint.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the uniform assumption fails, the Type I error control could be rescued by first estimating the null distribution's support and shape; a mixture-of-Betas prior, as the paper proposes, would supply a richer family but would require new bounds.
  • The method's reliance on a pilot estimate of [a,b] means the effective size depends on pilot quality; a double-bootstrap or a conservative widening of the interval would make the test more robust at the cost of power.
  • For non-binary responses, the same minimum-distance idea could be applied to embeddings, but the concentration argument would need a replacement for the Hoeffding bound; the energy-test comparison in the paper's Figure 5 suggests that a distribution-free version is still open.
  • A direct practical test: choose a null family with tightly clustered probabilities (as in Figure 4) and measure the empirical rejection rate at nominal alpha; if the rate exceeds alpha, the stated guarantee is not operative at realistic budgets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper addresses the problem of testing whether the binary response probability p' of a new query belongs to an unknown set P0 of probabilities induced by semantically irrelevant perturbations of a base query. Since the query-to-probability map is unknown, the test samples m null queries, estimates each p_j from r responses, forms the statistic T_{m,r} = min_j |p_hat_j - p_hat'|, and rejects H0: p' in P0 if T_{m,r} > epsilon. Under Assumptions 1-3, the paper proves: a finite-sample closeness lemma between the realistic statistic and an oracle statistic (Lemma 1, Theorem 1), an upper bound on size (Theorem 2), a lower bound on power (Theorem 3), an average-power formula (Theorem 4), and consistency/validity conditions (Corollary 1). Algorithm C selects (epsilon,m,r) under a budget constraint by maximizing an estimated power lower bound subject to an estimated validity constraint. Experiments on LLaMA-3-8B illustrate the method on the 'R.A. Fisher' motivating example.

Significance. If the claims held as stated, the paper would provide a practical template for composite-null testing when the null set is unknown but can be sampled, with clean Hoeffding-type bounds and explicit budget allocation. The manuscript is transparent about experimental cost and acknowledges limitations. However, two load-bearing issues must be addressed: (i) Algorithm C as written does not implement the consistency conditions of Corollary 1, so the abstract's consistency claim is unsupported for the proposed procedure; and (ii) the key uniformity assumption (Assumption 2) is contradicted by the paper's own motivating null set, so the Type I error guarantee does not apply to the flagship application.

major comments (3)
  1. [§3.1 (Algorithm C), §4.3 (Corollary 1)] The abstract claims the proposed test is asymptotically valid and consistent, but Algorithm C does not deliver consistency. Corollary 1 requires epsilon->0, m->infty, and r=omega(m^2). Algorithm C searches epsilon over a fixed grid {0, eta, 2eta, ..., epsilon_max} independent of the budget nu; for each fixed epsilon, m is set by line 6 and r=nu/m. As nu->infty with eta fixed, the selected epsilon* converges to the smallest positive grid value eta, m* stays bounded at about |log alpha|/|log(1-eta/(b-a))|, and only r*->infty. For p'=a-eta/2 in P1, the ideal-test power equals (1-eta/(2(b-a)))^{m*} <1, and by Theorem 3 the realistic power has the same constant limit. Thus P(reject|p') does not approach 1. Corollary 2 proves only asymptotic validity, not consistency. The algorithm needs to let the grid spacing (or epsilon*) tend to 0 with nu while maintaining m*->infty and r=omega(m^2), or th
  2. [§4.2 (Theorem 2), §6 (Discussion), Fig. 4] The size guarantee of Theorem 2 and the validity constraint in Algorithm C rely on Assumption 2, p_1,...,p_m i.i.d. Uniform[a,b]. The paper's own motivating data violate this assumption: the histogram in Fig. 4 (left) shows estimated null probabilities concentrated near 0.8-1.0, and Section 6 explicitly acknowledges the non-uniformity of the Bernoulli parameters. Therefore the Type I error control that motivates the paper is not guaranteed for the actual LLM experiment. The authors should either restrict the claims to the uniform model or provide robustness analysis/empirical calibration that addresses non-uniform null distributions before claiming the method solves the motivating problem.
  3. [§3.1 (Algorithm C), §4.4 (Corollary 2)] Corollary 2's proof is a one-line convergence argument for (b_hat-a_hat) to (b-a). This gives asymptotic validity only if the selected triple (epsilon,m,r) also satisfies the true size bound in the limit; the current proof does not account for the fact that the algorithm's validity constraint is checked with plug-in estimates and the objective is maximized over a grid. This is likely repairable, but as written it is not a complete argument.
minor comments (3)
  1. [§3.1 (Algorithm C, line 6)] The grid includes epsilon=0, where the denominator |log(1-epsilon/(b_hat-a_hat))| is zero. The algorithm should either start the grid at eta or handle epsilon=0 separately.
  2. [§5.2, Algorithm B, Remark 1] Section 5.2 says the unbiased estimates are 'provided in Algorithm B', but Algorithm B returns raw sample extremes, while Remark 1 proposes bias-corrected versions. Please reconcile the implementation description with the estimator actually used.
  3. [§3.1 (Algorithm C, line 10)] The expression '2/1-(b_hat-a_hat)' should read 2/(1-(b_hat-a_hat)) for clarity; the current typesetting is easy to misread.

Circularity Check

0 steps flagged

No significant circularity: the derivations are self-contained under stated assumptions; the main gap is a correctness/consistency issue in Algorithm C, not a circular one.

full rationale

I traced the claimed derivation chain: Assumptions 1–3 feed Lemma 1, Theorems 1–3, Corollary 1, and finally Algorithm C. The ideal test is explicitly defined using the true Bernoulli parameters p_j, and the realistic test is shown to be close to it via concentration inequalities; no parameter is fitted to the alternative and then renamed as a prediction. The plug-in estimates (â,b̂) from Algorithm B are used as inputs to the validity constraint and to the approximate power objective Ĥ, but this is an estimation step in the construction of the decision rule, not a definitional reduction of the conclusion to the data. The size and power bounds follow from the stated distributional assumptions rather than from the observed rejection outcomes. Section 6 openly acknowledges that the uniform assumption on the null Bernoulli parameters is violated in the motivating experiment, which is a limitation on validity, not evidence that the derivation is circular. The more serious issue is that the abstract's 'consistent' claim is not actually delivered by Algorithm C as written: its fixed ε-grid and the validity constraint keep ε* bounded away from 0 and m* bounded as the budget grows, so Corollary 1's conditions ε→0, m→∞ are not met. That is a mathematical correctness gap, not a circularity. The self-citations (Helm et al. 2023, Helm et al. 2025, Ness et al. 2024) are used for motivation and context, not as load-bearing support for the central test construction. I therefore find no circular step that would warrant a nonzero circularity score.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The test's validity rests on the three explicit assumptions in Section 4.1, especially the uniformity of null Bernoulli parameters. These are domain assumptions about the LLM response mapping and the user's query set, not consequences of the math. The paper uses standard concentration inequalities (Hoeffding) for the bounds.

free parameters (2)
  • Range estimation sample sizes (m_tilde, r_tilde) = m_tilde=20, r_tilde=50 in experiment
    User-chosen for Algorithm B; controls precision of a_hat and b_hat used in the validity constraint and power objective.
  • Threshold grid step (eta_epsilon) = 0.005 in experiment
    User-chosen discretization in Algorithm C; affects which (epsilon,m,r) is selected, not the asymptotic validity.
axioms (6)
  • standard math Hoeffding's inequality for bounded random variables
    Used in Lemma 1's proof to bound |T_{m,r} - T_tilde_m|.
  • domain assumption Assumption 1: p_1, ..., p_m independent of p'
    Required for Theorem 1 and Theorem 2 to separate estimation error of the null set from the test query.
  • domain assumption Assumption 2: p_1, ..., p_m i.i.d. Uniform[a,b]
    Gives the exact probabilities in Lemma 2 and the size bound in Theorem 2; questionable in practice.
  • domain assumption Assumption 3: p' ~ Uniform(0,1)
    Prior for the average power calculation in Theorem 4.
  • domain assumption The response space is binary, so each query has a Bernoulli response distribution
    Restricts the analysis; stated in Section 3.
  • domain assumption P0 is the smallest interval [a,b] containing all null Bernoulli parameters
    Defines the null region used in all theoretical results.

pith-pipeline@v1.3.0-alltime-deepseek · 22999 in / 18623 out tokens · 188527 ms · 2026-08-04T17:18:30.998981+00:00 · methodology

0 comments
read the original abstract

Given an input query, generative models such as large language models produce a random response drawn from a response distribution. Given two input queries, it is natural to ask if their response distributions are the same. While traditional statistical hypothesis testing is designed to address this question, the response distribution induced by an input query is often sensitive to semantically irrelevant perturbations to the query, so much so that a traditional test of equality might indicate that two semantically equivalent queries induce statistically different response distributions. As a result, the outcome of the statistical test may not align with the user's requirements. In this paper, we address this misalignment by incorporating into the testing procedure consideration of a collection of semantically similar queries. In our setting, the mapping from the collection of user-defined semantically similar queries to the corresponding collection of response distributions is not known a priori and must be estimated, with a fixed budget. Although the problem we address is quite general, we focus our analysis on the setting where the responses are binary, show that the proposed test is asymptotically valid and consistent, and discuss important practical considerations with respect to power and computation.

Figures

Figures reproduced from arXiv: 2509.10963 by Aranyak Acharyya, Carey E. Priebe, Hayden S. Helm.

Figure 1
Figure 1. Figure 1: Empirical distributions for p-values when testing for equality of the binary re￾sponse distributions for q1 = “RA Fisher was a statistician. Was he great?” against itself (control = black) or against the semantically irrelevant perturba￾tion q2 = “R.A. Fisher was a statistician. Was he a great man?” (condition = red). The figure presents p-values, for various values of r, from Fisher’s exact test for 100 M… view at source ↗
Figure 2
Figure 2. Figure 2: Analytical upper bounds for the Type I Error (left) and lower bounds for the average power (right) of the proposed test for various thresholds (ϵ) and bud￾gets. The maximal average power for a given budget is highlighted by a red star. Algorithm C selects m, r, and ϵ such that the test is asymptotically valid (P[type-I error] → α), and an estimated lower bound on the average power is maximized. 9 [PITH_FU… view at source ↗
Figure 3
Figure 3. Figure 3: The derived upper bound on Type I Error (left) and the derived lower bound on average power (right) compared to the simulated probability of rejection (green) for different rejection thresholds and various budgets. We include both the an￾alytical bound (black) – where all population parameters are known, and the estimated bound (blue) – where we use plug-in estimates . The derived analyt￾ical and estimated… view at source ↗
Figure 4
Figure 4. Figure 4: The histogram of estimated Bernoulli parameters of the sampled null queries (left) and the empirical probability of rejecting H0 : p ′ ∈ P0 vs. HA : p ′ ∈ P / 0 for q0 = “RA Fisher was a statistician. Was he a great man?” using the test described in Alg. C for various q ′ (right). The proposed test greatly reduces the undesirable rejections in our motivating example (e.g., when changing “RA” to “R.A.”), ma… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of p-values for tests for semantic equivalence of queries, in the setting of general (non-binary) responses. The large language model used is google/gemma/2-2b-it and the embedding function g is nomic-ai/nomic-embed-text-v2-moe. For every query, we bootstrap k = 100 responses from a pool of 1000 randomly generated responses, and implement Szekely’s Energy Test on any pair of queries, obtaining… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 15 linked inside Pith

  1. [1]

    A., and Sheikh, J

    Damseh, R., Alrazak, S. A., and Sheikh, J. (2023). Large language models in medical education: opportunities, challenges, and future directions.JMIR medical education, 9(1):e48291

    2023

  2. [2]

    Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). GPT-4 technical report.arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  3. [3]

    Agarwal, A., Xiao, M., Barter, R., Ronen, O., Fan, B., and Yu, B. (2025). PCS-UQ: Un- certainty quantification via the predictability-computability-stability framework.arXiv preprint arXiv:2505.08784

    Pith/arXiv arXiv 2025

  4. [4]

    Alexander, C. H. (1980). Simultaneous confidence bounds for the tail of an inverse distri- bution function.The Annals of Statistics, 8(6):1391–1394. Anthropic (2024). Introducing the next generation of Claude.https://www.anthropic. com/news/claude-3-family. Accessed: 2025-08-26

    1980

  5. [5]

    Senn, S. (2021). The outstanding scientist, R.A. Fisher: his views on eugenics and race. Heredity, 126(4):565–576

    2021

  6. [6]

    A., Wang, X., and Zhou, D

    Chen, X., Chi, R. A., Wang, X., and Zhou, D. (2024). Premise order matters in reasoning with large language models.arXiv preprint arXiv:2402.08939. D’Antonoli, T. A., Stanzione, A., Bluethgen, C., Vernuccio, F., Ugga, L., Klontzas, M. E.,

    Pith/arXiv arXiv 2024

  7. [7]

    Cuocolo, R., Cannella, R., and Ko¸ cak, B. (2024). Large language models in radiology: fundamentals, applications, ethical considerations, risks, and future directions.Diagnostic and Interventional Radiology, 30(2):80

    2024

  8. [8]

    Hartvigsen, T., and Bitterman, D. (2024). Language models are surprisingly fragile to drug names in biomedical benchmarks.arXiv preprint arXiv:2406.12066

    Pith/arXiv arXiv 2024

  9. [9]

    Mathur, A., Schelten, A., Vaughan, A., et al. (2024). The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  10. [10]

    Helm, H., Acharyya, A., Park, Y., Duderstadt, B., and Priebe, C. (2025). Statistical inference on black-box generative models in the data kernel perspective space. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T., editors,Findings of the Association for Computational Linguistics: ACL 2025, pages 3955–3970, Vienna, Austria. Association for Computa...

    2025

  11. [11]

    E., and Yang, W

    Helm, H., Priebe, C. E., and Yang, W. (2023). A statistical Turing test for generative models.arXiv preprint arXiv:2309.08913. 20 Testing For LLM Response Differences

    Pith/arXiv arXiv 2023

  12. [12]

    Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., Gao, Z., Zhang, X., Li, J., and Xiong, H. (2023). From CLIP to DINO: Visual encoders shout in multi-modal large language models.arXiv preprint arXiv:2310.08825

    Pith/arXiv arXiv 2023

  13. [13]

    Khan, M. S. and Umer, H. (2024). ChatGPT in finance: Applications, challenges, and solutions.Heliyon, 10(2)

    2024

  14. [14]

    Kim, S., Lee, C.-k., and Kim, S.-s. (2024). Large language models: a guide for radiologists. Korean Journal of Radiology, 25(2):126

    2024

  15. [15]

    Lo, C. K. (2023). What is the impact of chatGPT on education? A rapid review of the literature.Education sciences, 13(4):410

    2023

  16. [16]

    Miller, E. (2024). Adding error bars to evals: A statistical approach to language model evaluations.arXiv preprint arXiv:2411.00640

    Pith/arXiv arXiv 2024

  17. [17]

    O., Matton, K., Helm, H., Zhang, S., Bajwa, J., Priebe, C

    Ness, R. O., Matton, K., Helm, H., Zhang, S., Bajwa, J., Priebe, C. E., and Horvitz, E. (2024). MedFuzz: Exploring the robustness of large language models in medical question answering.arXiv preprint arXiv:2406.06573

    Pith/arXiv arXiv 2024

  18. [18]

    T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al

    Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., et al. (2023). Can generalist foundation models outcompete special-purpose tuning? Case study in medicine.arXiv preprint arXiv:2311.16452

    Pith/arXiv arXiv 2023

  19. [19]

    M., Terano, H

    Rahman, M. M., Terano, H. J., Rahman, M. N., Salamzadeh, A., and Rahaman, M. S. (2023). Chatgpt and academic research: A review and recommendations based on prac- tical examples.Rahman, M., Terano, HJR, Rahman, N., Salamzadeh, A., Rahaman, S.(2023). ChatGPT and Academic Research: A Review and Recommendations Based on Practical Examples. Journal of Educati...

    2023

  20. [20]

    J., Ryan, P

    Schuemie, M. J., Ryan, P. B., DuMouchel, W., Suchard, M. A., and Madigan, D. (2014). Interpreting observational studies: why empirical calibration is needed to correct p-values. Statistics in medicine, 33(2):209–218

    2014

  21. [21]

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettle- moyer, L. (2023). Detecting pretraining data from large language models.arXiv preprint arXiv:2310.16789

    Pith/arXiv arXiv 2023

  22. [22]

    Siino, M., Falco, M., Croce, D., and Rosso, P. (2025). Exploring LLMs applications in law: A literature review on current legal NLP approaches.IEEE Access, 13:18253–18276

    2025

  23. [23]

    Sun, Z. (2023). A short survey of viewing large language models in legal aspect.arXiv preprint arXiv:2303.09136. Sz´ ekely, G. J. and Rizzo, M. L. (2004). Testing for equal distributions in high dimension. InterStat: Statistics on the Internet, 5(16.10)

    Pith/arXiv arXiv 2023

  24. [24]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023). Gemini: a family of highly capable multi- modal models.arXiv preprint arXiv:2312.11805. 21 A. Acharrya, C.E. Priebe, H.S. Helm

    Pith/arXiv arXiv 2023

  25. [25]

    J., Ting, D

    Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W. (2023). Large language models in medicine.Nature medicine, 29(8):1930–1940. ¨Ust¨ un, A., Aryabumi, V., Yong, Z.-X., Ko, W.-Y., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., et al. (2024). Aya model: An instruction finetuned ope...

    Pith/arXiv arXiv 2023

  26. [26]

    and Barter, R

    Yu, B. and Barter, R. L. (2024).Veridical data science: The practice of responsible data analysis and decision making

    2024

  27. [27]

    rX k=1 X (j) k −X ′ k r −E rX k=1 X (j) k −X ′ k r ! < t # ≥1−2e − rt2 2 =⇒P

    Zhang, K., Li, J., Li, G., Shi, X., and Jin, Z. (2024). CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges.arXiv preprint arXiv:2401.07339. 7 Appendix: Proofs of results Lemma 1.Suppose that for every queryq∈ {q 1, . . . , qm, q′}we observe iid replicates of responses denoted byf(q) 1, . . ....

    Pith/arXiv arXiv 2024