Pith. sign in

REVIEW 3 major objections 4 minor 19 references

Statistical Hypothesis Testing for Auditing Robustness in Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM's response change under any prompt edit can be tested as a frequentist hypothesis, with valid p-values from a permutation test on cosine similarities.

desk verdict A clean, valid permutation-test framework for auditing LLM perturbations, undercut by an unvalidated semantic-similarity layer that its own appendix data show can reverse conclusions. read the letter →

arxiv 2506.07947 v1 pith:LGS4NBXU submitted 2025-06-09 cs.CL

Paulius Rauba , Qiyao Wei , Mihaela van der Schaar This is my paper
classification cs.CL MSC 62F0362G1062G0968T50
keywords distribution-basedperturbationanalysisLLMauditinghypothesistestingpermutationtestsemanticsimilarityMonteCarlosamplingeffectsizefamily-wiseerrorrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models give different answers to the same prompt on every draw, so comparing one output under an original input with one output under a perturbed input cannot separate a real intervention effect from sampling noise. This paper argues that the right question is whether the whole output distribution has moved, and that this question can be answered with a frequentist hypothesis test. The proposed method, distribution-based perturbation analysis (DBPA), draws $k$ outputs from each condition, maps every pair of outputs to a cosine similarity between their embeddings, builds two empirical similarity distributions, and compares them with a permutation test on the pooled samples. The output is a p-value for the hypothesis $H_0 : D_x = D_{x'}$, where $D_x$ is the output distribution for input $x$ and $x'$ is the perturbed input, alongside a scalar effect size and an extension that controls error rates when many perturbations are tested at once. If the method works as claimed, auditors gain a principled way to say whether a prompt edit, role instruction, or model swap changed the behavior of any black-box LLM.

What carries the argument

The load-bearing mechanism is a permutation test on two empirical distributions of pairwise cosine similarities. From $k$ baseline outputs and $k$ perturbed outputs, the method builds $P_0 = \{s(y_i, y_j) : 1 \le i < j \le k\}$ and $P_1 = \{s(y_i, y'_j) : 1 \le i, j \le k\}$, where $s$ is cosine similarity between deterministic, semantic-preserving embeddings. Under the null $H_0 : D_x = D_{x'}$ the pooled $2k$ outputs are exchangeable, so randomly re-splitting the pool and recomputing the Jensen-Shannon divergence $\omega(P_0, P_1)$ produces a valid null distribution; the p-value is the fraction of permuted discrepancies at least as large as the observed one, and the observed discrepancy serves as the effect size. The machinery requires only that the similarity function be bounded and meaning-preserving, not that it be a metric.

What would settle it

Build a set of LLM answer pairs that human raters judge semantically identical (near-paraphrases) and a second set that raters judge meaningfully different; run DBPA on both with the paper's default embedding and similarity function. If the test yields small p-values for a large share of the identical pairs, or large p-values for a large share of the different pairs, the semantic-preservation assumption fails and the p-values are measuring something other than semantic change.

Watch

Extended reading notes

Core claim

The paper's central claim is that an arbitrary intervention on a black-box LLM can be audited by testing $H_0 : D_x = D_{x'}$ against $H_1 : D_x \neq D_{x'}$, where $D_x$ is the output distribution for input $x$ and $x' = \Delta x(x)$ is the perturbed input. Because the full output space is exponentially large, DBPA approximates the two distributions with $k$ Monte Carlo samples each and summarizes them by pairwise cosine similarities in a semantic embedding space: the within-condition multiset $P_0 = \{s(y_i, y_j) : i < j\}$ captures baseline variability, while the cross-condition multiset $P_1 = \{s(y_i, y'_j)\}$ captures the distance induced by the intervention. The discrepancy $T^{\text{obs}} = \omega(P_0, P_1)$ is measured by Jensen-Shannon divergence, and because the pooled outputs are exchangeable under $H_0$, a permutation test with $B$ random splits yields a valid frequentist p-value $\hat{p} = (1+\text{count})/(1+B)$; the observed discrepancy itself is the scalar effect size. The paper argues that this procedure is model-agnostic, supports any user-defined perturbation, controls family-wise error under multiple testing, and enables TPR/FPR analyses for user-defined classes of perturbations.

Load-bearing premise

The method only works if the embedding function used for cosine similarity reflects what humans mean by 'same meaning' — otherwise the test flags superficial rewording and misses real shifts.

Editorial extensions

If this is right

  • An auditor can test whether a specific prompt edit, role instruction, or model swap changes an LLM's answer distribution, with a calibrated false-positive rate instead of a subjective read of a few outputs.
  • Testing a family of perturbations with Bonferroni-style adjustment keeps the family-wise error rate at a user-chosen level, and raising the Monte Carlo sample size $k$ restores power.
  • For a user-defined partition of perturbations into 'should change' and 'should not change' classes, DBPA yields TPR/FPR curves and ROC-AUC, turning model selection for robustness into a quantitative trade-off.
  • Treating the model swap itself as the intervention gives a distribution-level measure of alignment between a candidate model and a reference model.
  • A small p-value means a detectable semantic shift under the chosen similarity measure; a large p-value means no detectable shift, not identity of the full text-generation distributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same two-distribution permutation machinery could audit any stochastic generative system, not only LLMs, wherever a meaning-preserving similarity function is available.
  • Beyond the paper: the semantic-preservation assumption is itself testable; a benchmark of human-rated paraphrase and meaning-change pairs would reveal which embedding choices keep DBPA's decisions aligned with human judgment, complementing the paper's own embedding-sensitivity experiments.
  • Beyond the paper: the paper's finding that longer contexts make perturbations harder to detect suggests that auditors should report the minimum detectable effect size at a given prompt length, rather than treating one sample size $k$ as universally adequate.
  • Beyond the paper: an adversary who knows the chosen embedding could in principle craft perturbations that preserve cosine similarity while changing meaning, so deployment of DBPA as a compliance tool should pair it with a secret or rotated embedding choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces distribution-based perturbation analysis (DBPA), a framework for testing whether an intervention (input perturbation, system change, or reference model comparison) changes the output distribution of a black-box LLM. The method samples k outputs under baseline and perturbed conditions, maps all pairs of outputs to scalar similarities via an embedding-based similarity function (cosine similarity with ada-002 by default), constructs within-sample and cross-sample similarity distributions P0 and P1, and compares them with a discrepancy measure ω (Jensen-Shannon divergence by default). Significance is assessed by a permutation test that re-randomizes the pooled outputs under an exchangeability assumption, yielding a frequentist p-value; the observed discrepancy T_obs serves as the effect size. The paper reports three main case studies (persona perturbations, TPR/FPR trade-offs, and model alignment) plus additional studies on prompt length, alternative distance measures, embedding costs, text-similarity metrics, and alternative embedding models. The central claim is that DBPA provides a reliable, model-agnostic, and interpretable frequentist framework for auditing LLM robustness.

Significance. If the framework's advertised reliability held, DBPA would be a useful addition to the LLM-auditing toolkit: the permutation-test core is statistically sound under the stated exchangeability assumption, the procedure is clearly and reproducibly specified, the framework makes no parametric distributional assumptions, and the authors correctly avoid permuting the correlated similarity scores and instead permute the pooled outputs. The availability of code, the explicit formulation of hypothesis tests and effect sizes, and the inclusion of multiple case studies are strengths. However, the paper's own experimental results show that both p-values and effect sizes depend strongly on the choice of similarity function and embedding model, and in some cases the significance decisions reverse. Because the framework's advertised interpretability and reliability rest on the unvalidated semantic-preserving assumption for embeddings, the current evidence supports the permutation-test methodology but does not yet support the broader claim that DBPA yields a reliable auditing tool independent of arbitrary design choices.

major comments (3)
  1. [Sec. 4.4, Table 4 (Appendix C.2), Table 1] The paper's claim that significance decisions are consistent across distance measures is contradicted by its own data. Table 1 states that significance decisions are consistent across measures, and Sec. 4.4 repeats that evaluations 'stay comparatively consistent on the significance of the effect sizes.' However, Table 4 shows that for the Comedian persona, JSD gives 0.31 with no star (nonsignificant) while Energy distance gives p<0.001, and similarly for Robot From The Future, JSD gives 0.22 with no star while Energy distance gives p<0.001. Since the choice of ω is left to the user without principled guidance, two auditors using the same data and the same default embedding can reach opposite conclusions about the same perturbation. This directly undermines the advertised reliability and interpretability of the p-values.
  2. [Sec. 4.4, Table 7 (Appendix C.5)] The choice of embedding model changes qualitative conclusions for the same LLM, same persona, and same perturbation. In Table 7, Doctor and Nurse are nonsignificant with ada-002 (0.21 and 0.20, no stars) but are significant at p<0.001 under Jasper, Stella, and Kalm; the Robot From The Future persona also changes significance across embeddings. Section 4.4 concedes that results 'are generally sensitive to the embedding for calculating p-values and effect sizes,' and Sec. 6 defers clearer selection guidelines to future work. This is not a minor robustness caveat: without a validated criterion for choosing the embedding function, the p-values and effect sizes produced by DBPA are not stable audit quantities, and the central claim of a reliable frequentist hypothesis testing framework is therefore underdetermined.
  3. [Sec. 3 (beginning of the section)] The load-bearing semantic-preserving assumption on the embedding function is stated but never validated. The paper assumes that if two outputs are judged equally similar by humans, their embedding distance is equal up to a monotone transform, yet no experiment compares embedding-based similarities with human similarity judgments, and no falsifiable criterion is provided for when this assumption holds. If this assumption fails, the test can both miss real semantic shifts and flag superficial rewording, and all downstream p-values and effect sizes inherit that distortion. Since the paper's own appendices show that different embeddings produce opposite significance decisions, demonstrating that the chosen embedding actually tracks the intended semantic notion is necessary to support the framework's advertised interpretability.
minor comments (4)
  1. [Appendix C.4] The phrase 'creative personas like ‘comedia’ and ‘child’' contains a typo: 'comedia' should be 'comedian'.
  2. [Appendix A, Eq. (11)] Equation (11) has a mismatched bracket: the text reads 'E[ω(P′0, P′1]' but should be 'E[ω(P′0, P′1)]'.
  3. [Table 1 and Appendix C.2] Table 1 references 'Table A.2' for alternative distance measures, while the actual table appears in Appendix C.2 as Table 4; the cross-reference should be corrected.
  4. [Algorithm 1] The algorithm provides no guidance on choosing the number of permutations B, and the text only cites general references; a brief practical recommendation (for example, the resolution of the resulting p-value and the trade-off with computational cost) would improve usability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DBPA's permutation test is self-contained, and the semantic-similarity layer is an explicit assumption rather than a fitted or self-referential input.

full rationale

The paper's central derivation is the permutation procedure in Algorithm 1. The validity of the resulting p-value rests on Assumption 3.1: under H0, the pooled outputs are exchangeable, so any fixed similarity function s and discrepancy omega yield a valid permutation null. The paper states this directly: 'Under the strict null hypothesis H0 : D_x = D_x', any function of the pooled outputs is exchangeable, so our permutation-based p-value is valid.' This is a standard exchangeability argument, not a claim that the test detects every possible distributional difference; the paper explicitly concedes that non-rejection 'does not imply that the full text-generation distribution is identical.' No parameter is fitted to the case-study data and then renamed as a prediction or effect size. The 'semantic-preserving' embedding assumption at the start of Section 3 is an input assumption, explicitly flagged, and its consequences are acknowledged in Section 4.4 ('results are generally sensitive to the embedding for calculating p-values and effect sizes') and in the Limitations ('clearer selection guidelines that should be established in future work'). The TPR/FPR case study uses user-specified target and control partitions, which are evaluative labels rather than quantities derived from or fitted by the method. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The observed inconsistency of significance decisions across distance measures (e.g., Table 4) is a robustness or correctness concern, not a circularity concern, because the method does not claim to be invariant to those choices. Overall, the derivation is self-contained and the claims are appropriately scoped, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The ledger is small because DBPA is an empirical procedure rather than a parametric model. The only hand-chosen quantities are the sampling budget k and permutation count B; the stated assumptions are exchangeability, i.i.d. sampling, and semantic fidelity of the embedding. No new latent entities are introduced.

free parameters (2)
  • Monte Carlo sample size k = Not reported in main experiments
    Controls the variance of the empirical similarity distributions and the power of the permutation test. The paper treats k as a user-chosen hyperparameter but does not state its value in the case studies, so the reported p-values and effect sizes are conditional on an unreported experimental choice.
  • Number of permutations B = Not reported in main experiments
    The permutation p-value is computed as (1+count)/(1+B), so B sets the resolution and stability of the p-value. The paper does not report B for the case studies, which affects reproducibility of the exact significance stars.
assumptions (3)
  • domain assumption Exchangeability of pooled outputs under H0 (Assumption 3.1)
    The permutation p-value is valid only if the pooled outputs are exchangeable under the null. The paper states this as Assumption 3.1; it follows from i.i.d. sampling when the two output distributions are identical, but it is the formal basis of the test.
  • domain assumption Semantic-preserving, deterministic, stable embedding function
    Section 3 begins by assuming the embedding is deterministic and stable and that cosine similarity in this space tracks human semantic similarity up to a monotone transform. All interpretability claims for p-values and effect sizes rest on this.
  • domain assumption i.i.d. Monte Carlo draws from S(x) and S(x')
    The empirical distributions treat repeated LLM generations as independent samples from a fixed output distribution. This is standard for stochastic decoding but is an assumption about sampling that the procedure relies on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Statistical Hypothesis Testing for Auditing Robustness in Language Models." pith.science (2026). https://pith.science/paper/LGS4NBXU

@misc{pith2026250607947,
  author       = {Pith},
  title        = {Pith review of: Statistical Hypothesis Testing for Auditing Robustness in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LGS4NBXU}},
  note         = {Machine review of arXiv:2506.07947}
}
read the original abstract

Consider the problem of testing whether the outputs of a large language model (LLM) system change under an arbitrary intervention, such as an input perturbation or changing the model variant. We cannot simply compare two LLM outputs since they might differ due to the stochastic nature of the system, nor can we compare the entire output distribution due to computational intractability. While existing methods for analyzing text-based outputs exist, they focus on fundamentally different problems, such as measuring bias or fairness. To this end, we introduce distribution-based perturbation analysis, a framework that reformulates LLM perturbation analysis as a frequentist hypothesis testing problem. We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling, enabling tractable inference without restrictive distributional assumptions. The framework is (i) model-agnostic, (ii) supports the evaluation of arbitrary input perturbations on any black-box LLM, (iii) yields interpretable p-values; (iv) supports multiple perturbations via controlled error rates; and (v) provides scalar effect sizes. We demonstrate the usefulness of the framework across multiple case studies, showing how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models. Above all, we see this as a reliable frequentist hypothesis testing framework for LLM auditing.

Figures

Figures reproduced from arXiv: 2506.07947 by the authors.

Figure 1
Figure 1. TPR/FPR trade-offs for three selected language models. The figure shows three panels: A, B, and C, de￾scribing the relationship between TPR and FPR for three models together with the AUC score evaluated based on the methodology in Sec. 4.2. The ROC-AUC describes the changing performance of the model as we vary α ∈ [0, 1]. Models with higher AUC are better because they can detect true changes and fail to answer diffe… view at source ↗
Figure 3
Figure 3. Distance of responses from a reference lan￾guage model. We collect k responses from a base model (GPT-4) and evaluate the responses from multiple other lan￾guage models reported on the left, compute their ω and p-values using Alg. 1. Higher distance implies greater dif￾ferences in similarity distributions. • Goal. We aim to show that we can understand how well two language models answer across their response distrib… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 4 canonical work pages

  1. [1]

    Re-evaluating evaluation in text summarization.arXiv preprint arXiv:2010.07100,

    Bhandari, M., Gour, P., Ashfaq, A., Liu, P., and Neubig, G. Re-evaluating evaluation in text summarization.arXiv preprint arXiv:2010.07100,

    arXiv 2010

  2. [7]

    J., Shlens, J., and Szegedy, C

    Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples.arXiv preprint arXiv:1412.6572,

    arXiv

  3. [12]

    H., Shin, J., and Fung, P

    Park, J. H., Shin, J., and Fung, P. Reducing gender bias in abusive language detection.arXiv preprint arXiv:1808.07231,

    arXiv

  4. [14]

    T., Zhao, C., Feng, S., Daumé III, H., and Boyd-Graber, J

    Si, C., Goyal, N., Wu, S. T., Zhao, C., Feng, S., Daumé III, H., and Boyd-Graber, J. Large language models help hu- mans verify truthfulness–except when they are convinc- ingly wrong.arXiv preprint arXiv:2310.12558,

    arXiv

  5. [15]

    M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y ., Guo, H., Gan, R., Ni, Z., Yang, J., et al

    Wang, Z. M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y ., Guo, H., Gan, R., Ni, Z., Yang, J., et al. Rolellm: Bench- marking, eliciting, and enhancing role-playing abilities of large language models.arXiv preprint arXiv:2310.00746,

    arXiv

  6. [16]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y . React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629,

    arXiv

  7. [18]

    M., and Eger, S

    Zhao, W., Peyrard, M., Liu, F., Gao, Y ., Meyer, C. M., and Eger, S. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622,

    arXiv 1909

  8. [2003]

    Q., and Artzi, Y

    Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., and Artzi, Y . Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675,

    arXiv 1904

Show all 19 references
  1. [2004]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilac- qua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilac- qua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172,

    arXiv

  2. [2009]

    Better zero-shot reasoning with role-play prompting.arXiv preprint arXiv:2308.07702,

    Kong, A., Zhao, S., Chen, H., Li, Q., Qin, Y ., Sun, R., Zhou, X., Wang, E., and Dong, X. Better zero-shot reasoning with role-play prompting.arXiv preprint arXiv:2308.07702,

    arXiv

  3. [2010]

    and Guven, E

    Renze, M. and Guven, E. The effect of sampling tempera- ture on problem solving in large language models.arXiv preprint arXiv:2402.05201,

    arXiv

  4. [2014]

    J., Wang, Z., Wang, D

    Hao, S., Gu, Y ., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model.arXiv preprint arXiv:2305.14992,

    arXiv

  5. [2018]

    Accountability of ai under the law: The role of explanation.arXiv preprint arXiv:1711.01134,

    Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gersh- man, S., O’Brien, D., Scott, K., Schieber, S., Waldo, J., Weinberger, D., et al. Accountability of ai under the law: The role of explanation.arXiv preprint arXiv:1711.01134,

    arXiv

  6. [2019]

    The oscars of ai theater: A survey on role-playing with language models

    Chen, N., Wang, Y ., Deng, Y ., and Li, J. The oscars of ai theater: A survey on role-playing with language models. arXiv preprint arXiv:2407.11484,

    arXiv

  7. [2020]

    Nuanced metrics for measuring unintended bias with real data for text classification

    Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias with real data for text classification. InCompanion pro- ceedings of the 2019 world wide web conference, pp. 491–500,

    2019

  8. [2021]

    H., and Beutel, A

    Garg, S., Perot, V ., Limtiaco, N., Taly, A., Chi, E. H., and Beutel, A. Counterfactual fairness in text classifi- cation through robustness. InProceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 219–226,

    2019

  9. [2023]

    Inherent trade-offs in the fair determination of risk scores.arXiv preprint arXiv:1609.05807,

    Kleinberg, J., Mullainathan, S., and Raghavan, M. Inherent trade-offs in the fair determination of risk scores.arXiv preprint arXiv:1609.05807,

    arXiv

  10. [2024]

    Measuring and mitigating unintended bias in text classification

    Dixon, L., Li, J., Sorensen, J., Thain, N., and Vasserman, L. Measuring and mitigating unintended bias in text classification. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67–73,

    2018

  11. [2025]

    Dollar amounts are shown in parentheses for costs exceeding 100¢ (1 USD)

    Values show the total cost for processing all tokens required for the specified number of perturbations, assuming 200 tokens per perturbation on average. Dollar amounts are shown in parentheses for costs exceeding 100¢ (1 USD). OpenAI Embedding Models # of Perturbations Total ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.