{"id":"a64d9408-d4c8-4e2e-b069-2843c75ab68d","arxiv_id":"2506.07947","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A permutation-based hypothesis test on pairwise semantic similarities detects whether LLM outputs shift under arbitrary input or model perturbations.","lead":"The paper introduces a statistical test for whether a language model's answers change when you tweak the input, for example by changing the prompt or swapping in a different model. This gives auditors and regulators a principled way to distinguish real behavior shifts from random variation in high-stakes LLM applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DBPA's p-values and effect sizes reverse under the paper's own design choices (JSD vs Energy; ada-002 vs alternative embeddings), so the central claim of a reliable auditing tool is underdetermined without a validated semantic-similarity choice.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my stress test reinforces it rather than overturning it. The statistical machinery of DBPA — Monte Carlo estimation of output distributions, pairwise cosine similarities, permutation testing under exchangeability — is internally valid for any fixed similarity function and discrepancy measure; I do not claim a mathematical error in the hypothesis test. The weakest point is the semantic-similarity layer: the embedding and the discrepancy ω are free parameters, and the paper's own tables show these parameters reverse significance decisions. Table 4 (Appendix C.2) shows JSD missing Comedian and Robot from the Future while Energy distance rejects at p<0.001 for the same persona experiment; Table 1's assertion of cross-measure consistency is contradicted by that data. Table 7 (Appendix C.5) shows medical personas flipping from non-significant under ada-002 to p<0.001 under three alternative embedders. The paper explicitly acknowledges sensitivity in Section 4.4 and defers selection guidelines in the Limitations paragraph, but this internal evidence goes beyond an external philosophical worry: it demonstrates that the headline claim of 'interpretable p-values and effect sizes' for auditing is not yet supported. My agreement with the reader is partial because they located the concern primarily in the semantic-preserving embedding assumption, while the additional, more concrete failure mode is the distance-measure inconsistency, which is checked against the paper's own reported results. The correct verdict remains CONDITIONAL: DBPA is a promising framework and the permutation test is valid, but acceptance of the central claim requires either validation of a default (s,ω) against human similarity judgments, or a sensitivity analysis that bounds the flip rate and provides practical selection guidance. I therefore see no reason to change the reader's verdict.","tokens_in":18113,"tokens_out":8241,"duration_ms":102261,"concrete_test":"Re-run Case Study 1 (eight personas, GPT-4, 5 seeds, k as in the paper) on the same pooled outputs with two configurations: (s,ω) = (ada-002, JSD) and (s,ω) = (all-MiniLM-L6-v2, Energy distance). Count significance flips at α=0.05 across the eight personas, and also cross-tabulate the significance decisions already reported in Tables 4 and 7 (JSD vs Energy; Ada vs Jasper/Stella/Kalm). If any of the four medical personas flips from non-significant to significant, or any creative persona flips from significant to non-significant, DBPA's conclusions are design-choice dependent and the central claim needs an explicit sensitivity analysis or a selection rule backed by human judgments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The permutation-test core is statistically sound: under H0 the pooled outputs are exchangeable, so permuting labels gives a valid null for any fixed similarity function s and discrepancy ω. The load-bearing gap is the semantic-similarity layer, and the paper's own experiments show it changes qualitative conclusions. Appendix C.2 (Table 4) reports the default JSD as non-significant for Comedian (0.31, no star) and Robot from the Future (0.22, no star), while Energy distance gives p<0.001 for both; Table 1's claim that 'significance decisions [are] consistent across measures' is directly contradicted. Appendix C.5 (Table 7) shows medical personas that are non-significant with ada-002 (Doctor 0.21, no star; Nurse 0.20, no star) become p<0.001 under Jasper/Stella/Kalm, and effect sizes for some personas change by an order of magnitude. Section 4.4 concedes results 'are generally sensitive to the embedding for calculating p-values and effect sizes,' and Section 6's Limitations defer 'clearer selection guidelines' to future work. Without a validated criterion for choosing s and ω, a user auditing the same LLM can reach opposite conclusions about the same perturbation merely by switching embedders. This does not invalidate the permutation test, but it undercuts the advertised interpretability and reliability of DBPA's p-values and effect sizes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces distribution-based perturbation analysis (DBPA), a framework for testing whether an intervention (input perturbation, system change, or reference model comparison) changes the output distribution of a black-box LLM. The method samples k outputs under baseline and perturbed conditions, maps all pairs of outputs to scalar similarities via an embedding-based similarity function (cosine similarity with ada-002 by default), constructs within-sample and cross-sample similarity distributions P0 and P1, and compares them with a discrepancy measure ω (Jensen-Shannon divergence by default). Significance is assessed by a permutation test that re-randomizes the pooled outputs under an exchangeability assumption, yielding a frequentist p-value; the observed discrepancy T_obs serves as the effect size. The paper reports three main case studies (persona perturbations, TPR/FPR trade-offs, and model alignment) plus additional studies on prompt length, alternative distance measures, embedding costs, text-similarity metrics, and alternative embedding models. The central claim is that DBPA provides a reliable, model-agnostic, and interpretable frequentist framework for auditing LLM robustness.","tokens_in":18314,"tokens_out":2994,"duration_ms":40092,"significance":"If the framework's advertised reliability held, DBPA would be a useful addition to the LLM-auditing toolkit: the permutation-test core is statistically sound under the stated exchangeability assumption, the procedure is clearly and reproducibly specified, the framework makes no parametric distributional assumptions, and the authors correctly avoid permuting the correlated similarity scores and instead permute the pooled outputs. The availability of code, the explicit formulation of hypothesis tests and effect sizes, and the inclusion of multiple case studies are strengths. However, the paper's own experimental results show that both p-values and effect sizes depend strongly on the choice of similarity function and embedding model, and in some cases the significance decisions reverse. Because the framework's advertised interpretability and reliability rest on the unvalidated semantic-preserving assumption for embeddings, the current evidence supports the permutation-test methodology but does not yet support the broader claim that DBPA yields a reliable auditing tool independent of arbitrary design choices.","major_comments":[{"comment":"The paper's claim that significance decisions are consistent across distance measures is contradicted by its own data. Table 1 states that significance decisions are consistent across measures, and Sec. 4.4 repeats that evaluations 'stay comparatively consistent on the significance of the effect sizes.' However, Table 4 shows that for the Comedian persona, JSD gives 0.31 with no star (nonsignificant) while Energy distance gives p<0.001, and similarly for Robot From The Future, JSD gives 0.22 with no star while Energy distance gives p<0.001. Since the choice of ω is left to the user without principled guidance, two auditors using the same data and the same default embedding can reach opposite conclusions about the same perturbation. This directly undermines the advertised reliability and interpretability of the p-values.","section":"Sec. 4.4, Table 4 (Appendix C.2), Table 1"},{"comment":"The choice of embedding model changes qualitative conclusions for the same LLM, same persona, and same perturbation. In Table 7, Doctor and Nurse are nonsignificant with ada-002 (0.21 and 0.20, no stars) but are significant at p<0.001 under Jasper, Stella, and Kalm; the Robot From The Future persona also changes significance across embeddings. Section 4.4 concedes that results 'are generally sensitive to the embedding for calculating p-values and effect sizes,' and Sec. 6 defers clearer selection guidelines to future work. This is not a minor robustness caveat: without a validated criterion for choosing the embedding function, the p-values and effect sizes produced by DBPA are not stable audit quantities, and the central claim of a reliable frequentist hypothesis testing framework is therefore underdetermined.","section":"Sec. 4.4, Table 7 (Appendix C.5)"},{"comment":"The load-bearing semantic-preserving assumption on the embedding function is stated but never validated. The paper assumes that if two outputs are judged equally similar by humans, their embedding distance is equal up to a monotone transform, yet no experiment compares embedding-based similarities with human similarity judgments, and no falsifiable criterion is provided for when this assumption holds. If this assumption fails, the test can both miss real semantic shifts and flag superficial rewording, and all downstream p-values and effect sizes inherit that distortion. Since the paper's own appendices show that different embeddings produce opposite significance decisions, demonstrating that the chosen embedding actually tracks the intended semantic notion is necessary to support the framework's advertised interpretability.","section":"Sec. 3 (beginning of the section)"}],"minor_comments":[{"comment":"The phrase 'creative personas like ‘comedia’ and ‘child’' contains a typo: 'comedia' should be 'comedian'.","section":"Appendix C.4"},{"comment":"Equation (11) has a mismatched bracket: the text reads 'E[ω(P′0, P′1]' but should be 'E[ω(P′0, P′1)]'.","section":"Appendix A, Eq. (11)"},{"comment":"Table 1 references 'Table A.2' for alternative distance measures, while the actual table appears in Appendix C.2 as Table 4; the cross-reference should be corrected.","section":"Table 1 and Appendix C.2"},{"comment":"The algorithm provides no guidance on choosing the number of permutations B, and the text only cites general references; a brief practical recommendation (for example, the resolution of the resulting p-value and the trade-off with computational cost) would improve usability.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The permutation-test core is valid, but the paper's own Appendix C.5 shows that significance decisions reverse under alternative embeddings, which undercuts the central reliability claim. I recommend major revision with a requirement to either provide a principled validation of the semantic-similarity layer or substantially rescope the claims to acknowledge that DBPA's conclusions are contingent on design choices that are not yet specified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nWhat's actually new: the paper reformulates LLM perturbation analysis as a two-sample permutation test on pairwise cosine-similarity distributions, and packages it with a TPR/FPR/AUC decision layer. That combination is not in the cited related work. The test itself is valid: under the null of equal output distributions, the pooled outputs are exchangeable, so permutation p-values are correct for any fixed similarity function. The method is genuinely black-box and model-agnostic, and the procedure is described precisely enough to reimplement. Credit also for the TPR/FPR setup, which gives auditors a comparative tool rather than just a single p-value.\n\nThe soft spot is the semantic-similarity layer, and it is load-bearing. The authors assume embeddings are semantic-preserving but never validate that assumption. Worse, their own appendix contradicts the main text. Table C.2 shows JSD giving non-significant results for Comedian and Robot from the Future (p≈0.31 and 0.22) while Energy distance gives p<0.001 for both; Table C.5 shows medical personas flipping from non-significant under ada-002 to p<0.001 under Jasper, Stella, and Kalm. The body claims “significance decisions are consistent across measures” (Table 1); the appendix data say otherwise. The authors concede in Section 4.4 that results are “generally sensitive to the embedding,” so they are aware. This is not a minor caveat: for a given model and perturbation, switching embedders can reverse the audit conclusion. The permutation machinery is fine; the inferential target is underdetermined.\n\nMinor issues: the stars in Tables 2 and 3 are uncorrected for multiple testing despite the FWER discussion; and the “first approach” claim in the Limitations is overstated given the broader literature on embedding-based distribution tests. These are fixable with tighter writing.\n\nBottom line: the paper is a legitimate framework proposal with an honest limitations section, but the central reliability claim needs either a validated choice of similarity function or explicit guidance on how a user should select one. It deserves a serious referee: the idea is good, the statistics are checkable, and the inconsistency between claims and appendix data is exactly what review should catch. I'd recommend accept-in-principle after revision, not a desk reject.\n\nRecommendation: send it to review.","headline":"A clean, valid permutation-test framework for auditing LLM perturbations, undercut by an unvalidated semantic-similarity layer that its own appendix data show can reverse conclusions.","tokens_in":18910,"tokens_out":3321,"would_cite":true,"duration_ms":36958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62G10","62G09","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM's response change under any prompt edit can be tested as a frequentist hypothesis, with valid p-values from a permutation test on cosine similarities.","keywords":["distribution-based perturbation analysis","LLM auditing","hypothesis testing","permutation test","semantic similarity","Monte Carlo sampling","effect size","family-wise error rate"],"falsifier":"Build a set of LLM answer pairs that human raters judge semantically identical (near-paraphrases) and a second set that raters judge meaningfully different; run DBPA on both with the paper's default embedding and similarity function. If the test yields small p-values for a large share of the identical pairs, or large p-values for a large share of the different pairs, the semantic-preservation assumption fails and the p-values are measuring something other than semantic change.","tokens_in":17850,"feed_emoji":"📊","tokens_out":18061,"duration_ms":160198,"temperature":0.7,"pith_summary":"Large language models give different answers to the same prompt on every draw, so comparing one output under an original input with one output under a perturbed input cannot separate a real intervention effect from sampling noise. This paper argues that the right question is whether the whole output distribution has moved, and that this question can be answered with a frequentist hypothesis test. The proposed method, distribution-based perturbation analysis (DBPA), draws $k$ outputs from each condition, maps every pair of outputs to a cosine similarity between their embeddings, builds two empirical similarity distributions, and compares them with a permutation test on the pooled samples. The output is a p-value for the hypothesis $H_0 : D_x = D_{x'}$, where $D_x$ is the output distribution for input $x$ and $x'$ is the perturbed input, alongside a scalar effect size and an extension that controls error rates when many perturbations are tested at once. If the method works as claimed, auditors gain a principled way to say whether a prompt edit, role instruction, or model swap changed the behavior of any black-box LLM.","feed_headline":"Statistical test tells when LLM answers really change","feed_subtitle":"Turns LLM auditing into a hypothesis test with p-values, effect sizes, and error control.","key_machinery":"The load-bearing mechanism is a permutation test on two empirical distributions of pairwise cosine similarities. From $k$ baseline outputs and $k$ perturbed outputs, the method builds $P_0 = \\{s(y_i, y_j) : 1 \\le i < j \\le k\\}$ and $P_1 = \\{s(y_i, y'_j) : 1 \\le i, j \\le k\\}$, where $s$ is cosine similarity between deterministic, semantic-preserving embeddings. Under the null $H_0 : D_x = D_{x'}$ the pooled $2k$ outputs are exchangeable, so randomly re-splitting the pool and recomputing the Jensen-Shannon divergence $\\omega(P_0, P_1)$ produces a valid null distribution; the p-value is the fraction of permuted discrepancies at least as large as the observed one, and the observed discrepancy serves as the effect size. The machinery requires only that the similarity function be bounded and meaning-preserving, not that it be a metric.","core_discovery":"The paper's central claim is that an arbitrary intervention on a black-box LLM can be audited by testing $H_0 : D_x = D_{x'}$ against $H_1 : D_x \\neq D_{x'}$, where $D_x$ is the output distribution for input $x$ and $x' = \\Delta x(x)$ is the perturbed input. Because the full output space is exponentially large, DBPA approximates the two distributions with $k$ Monte Carlo samples each and summarizes them by pairwise cosine similarities in a semantic embedding space: the within-condition multiset $P_0 = \\{s(y_i, y_j) : i < j\\}$ captures baseline variability, while the cross-condition multiset $P_1 = \\{s(y_i, y'_j)\\}$ captures the distance induced by the intervention. The discrepancy $T^{\\text{obs}} = \\omega(P_0, P_1)$ is measured by Jensen-Shannon divergence, and because the pooled outputs are exchangeable under $H_0$, a permutation test with $B$ random splits yields a valid frequentist p-value $\\hat{p} = (1+\\text{count})/(1+B)$; the observed discrepancy itself is the scalar effect size. The paper argues that this procedure is model-agnostic, supports any user-defined perturbation, controls family-wise error under multiple testing, and enables TPR/FPR analyses for user-defined classes of perturbations.","pith_inferences":["Beyond the paper: the same two-distribution permutation machinery could audit any stochastic generative system, not only LLMs, wherever a meaning-preserving similarity function is available.","Beyond the paper: the semantic-preservation assumption is itself testable; a benchmark of human-rated paraphrase and meaning-change pairs would reveal which embedding choices keep DBPA's decisions aligned with human judgment, complementing the paper's own embedding-sensitivity experiments.","Beyond the paper: the paper's finding that longer contexts make perturbations harder to detect suggests that auditors should report the minimum detectable effect size at a given prompt length, rather than treating one sample size $k$ as universally adequate.","Beyond the paper: an adversary who knows the chosen embedding could in principle craft perturbations that preserve cosine similarity while changing meaning, so deployment of DBPA as a compliance tool should pair it with a secret or rotated embedding choice."],"forward_implications":["An auditor can test whether a specific prompt edit, role instruction, or model swap changes an LLM's answer distribution, with a calibrated false-positive rate instead of a subjective read of a few outputs.","Testing a family of perturbations with Bonferroni-style adjustment keeps the family-wise error rate at a user-chosen level, and raising the Monte Carlo sample size $k$ restores power.","For a user-defined partition of perturbations into 'should change' and 'should not change' classes, DBPA yields TPR/FPR curves and ROC-AUC, turning model selection for robustness into a quantitative trade-off.","Treating the model swap itself as the intervention gives a distribution-level measure of alignment between a candidate model and a reference model.","A small p-value means a detectable semantic shift under the chosen similarity measure; a large p-value means no detectable shift, not identity of the full text-generation distributions."],"supporting_citations":[{"why":"Supplies the few-permutations p-value estimation method that Algorithm 1 builds on.","marker":"Knijnenburg et al., 2009"},{"why":"Justifies computing the permutation p-value as $(1+\\text{count})/(1+B)$ so it is never zero under random permutation sampling.","marker":"Phipson & Smyth, 2010"},{"why":"Supports the choice of cosine similarity as a bounded, meaning-preserving comparison despite not being a metric.","marker":"Schubert, 2021"},{"why":"Documents how temperature and sampling parameters make LLM outputs stochastic, motivating distribution-level rather than single-output comparison.","marker":"Renze & Guven, 2024"},{"why":"Provides the resampling-based null-distribution approach from which the permutation test is distinguished.","marker":"Yu, 2003"}],"fun_headline_variants":["A statistical test for LLM output consistency","Auditing LLM robustness with p-values and effect sizes","When does an LLM really change? A hypothesis test","Permutation test exposes LLM sensitivity to inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method only works if the embedding function used for cosine similarity reflects what humans mean by 'same meaning' — otherwise the test flags superficial rewording and misses real shifts.","fun_headline_variants_meta":{"raw":{"variants":["A statistical test for LLM output consistency","Auditing LLM robustness with p-values and effect sizes","When does an LLM really change? A hypothesis test","Permutation test exposes LLM sensitivity to inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1399,"prompt_tokens":1066,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":271}},"tokens_in":682,"tokens_out":333,"duration_ms":3796,"temperature":1.0,"reasoning_tokens":271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:21:57.905858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a set of LLM answer pairs that human raters judge semantically identical (near-paraphrases) and a second set that raters judge meaningfully different; run DBPA on both with the paper's default embedding and similarity function. If the test yields small p-values for a large share of the identical pairs, or large p-values for a large share of the different pairs, the semantic-preservation assumption fails and the p-values are measuring something other than semantic change.","supporting_citations":[],"review_version":1}