Pith. sign in

REVIEW 3 major objections 4 minor 11 references

Reference-side nearest-neighbor coincidences yield a two-sample test that is normal under the null and consistent under alternatives.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ASK-NN, a one-sided nearest-neighbor coincidence test, detects distribution drift between reference and query samples with theoretical guarantees and competitive empirical performance on LLM hallucination and artificial-text detection.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A clean, modest extension of Henze's NN coincidence test that works well empirically; the theory is inherited, the calibration caveat is real but narrower than the stress-test suggests. the 3 major comments →

arxiv 2607.15607 v1 pith:HC6WMCNY submitted 2026-07-17 cs.LG stat.ML

ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language

Sergey Zakharov , Rodion Oblovatny , Alexey Zaytsev This is my paper
classification cs.LG stat.ML MSC 62G1062G20
keywords asymmetric two-sample testnearest-neighbor graphdistribution drifthallucination detectionartificial text detectionhigh-dimensional embeddingspermutation testasymptotic normality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ASK-NN is a two-sample test built for asymmetric settings where one distribution is the anchor and the other is under scrutiny. Its statistic counts reference points whose k nearest neighbors in the pooled sample are also reference points. Under the permutation null, the statistic has exact finite-sample mean and variance, is asymptotically normal, and converges to a limit functional that is minimized if and only if the reference and query distributions coincide. This makes the test consistent against fixed alternatives, and it works with modest computation. On real tasks, the test is competitive with kernel and graph baselines for artificial-text detection and LLM hallucination detection from token-level hidden states.

Core claim

The paper's central claim is that a one-sided version of the classical nearest-neighbor coincidence test retains the strong statistical guarantees of its symmetric counterpart while matching the reference–query structure of real detection tasks. Specifically, the statistic T_n has exact conditional mean and variance under the permutation null, is asymptotically normal (Theorem 3.1), and converges under fixed alternatives to the functional \tilde{D}(f,g,\tau)=∫ τ²f²/(τf+(1−τ)g) dx, which is shown to be minimized if and only if f equals g (Theorem 3.2 and Proposition 3.3). The proof follows by relating \tilde{D} to the symmetric divergence D through \tilde{D}=(D+2τ−1)/2. As a consequence, the

What carries the argument

The central object is the statistic T_n, defined on the directed k-nearest-neighbor graph of the pooled sample: it sums, over all reference points, how many of their k nearest neighbors are also reference points. This is the asymmetric counterpart of the symmetric nearest-neighbor coincidence count. Under the null, its finite-sample conditional distribution is tractable, with exact mean k n1(n1−1)/(n−1) and variance involving the graph's in-degree variation and reciprocal-neighbor counts. The limiting separation functional \tilde{D}(f,g,\tau)=∫ τ²f²/(τf+(1−τ)g) dx is the key identity that carries the consistency argument, via the exact relation \tilde{D}(f,g,\tau)=(D(f,g,\tau)+2τ−1)/2, where

Load-bearing premise

The theoretical guarantees assume the reference and query samples are i.i.d. draws from two densities; token-level hidden states from LLMs are strongly dependent across tokens, and the paper itself notes that extending the theory to dependence would align the method with LLM applications.

What would settle it

Compute the statistic's rejection rate under the null on dependent token-level hidden states drawn from the same prompt–response distribution; if the type I error under the asymptotic normal calibration substantially exceeds the nominal level (as the paper's own Section 5 suggests), the asymptotic-calibration claim fails in the application regime that motivated the method.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the asymptotic normality and consistency results hold, ASK-NN becomes a parameter-free, computationally light two-sample test suited to high-dimensional embedding spaces, with a closed-form p-value from the analytic mean and variance.
  • In hallucination detection, the test can be used as a training-free score on token-level hidden states, matching or beating kernel and optimal-transport baselines on several benchmarks.
  • The limit functional \tilde{D} gives an interpretable measure of asymmetric divergence; minimizing it identifies exactly the null distribution.
  • For real text embeddings, label-shuffling calibration (rather than the asymptotic formula) keeps type I error near nominal while preserving power, indicating the test's practical usefulness depends on resampling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to replace permutation resampling with block or sequential resampling that respects the token-level dependence of hidden states; the paper's own limitation note suggests this and it should restore calibration on LLM data.
  • Because \tilde{D} relates to D via a one-line identity, other known bounds and properties of symmetric nearest-neighbor divergences may transfer directly to the asymmetric measure, potentially simplifying analysis in other reference–query problems.
  • The test could be adapted for online drift detection by maintaining a fixed reference window and sliding query window; the closed-form conditional variance under the null might give a cheap control chart.
  • The paper's admission that asymptotic calibration fails on real embeddings implies the asymptotic results are best seen as qualitative guidance, while uncertainty quantification for LLM monitoring should rely on permutation calibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ASK-NN, an asymmetric nearest-neighbor two-sample test. The statistic counts, for each reference point, how many of its k nearest neighbors in the pooled sample are also reference points. Under the permutation null the authors derive exact finite-sample conditional mean and variance (Proposition 3.1), prove asymptotic normality (Theorem 3.1) and consistency under fixed alternatives via a limiting functional (Theorem 3.2, Proposition 3.3). Empirically, ASK-NN is compared with MMD, Sinkhorn, Hotelling's T^2, and the symmetric Henze test on Gaussian benchmarks, artificial-text detection (RAID), and LLM hallucination detection using token-level hidden states. For real embedding data the asymptotic calibration is reported to be inaccurate, and a label-shuffling variant (ASK-NN (LS)) is used instead.

Significance. If the theoretical results are correct and properly scoped, ASK-NN is a useful asymmetric alternative to existing graph-based two-sample tests. The exact conditional moments and the consistency characterization are genuine contributions, and the computational advantage over kernel and MST methods is clearly demonstrated. The paper is honest about the failure of asymptotic calibration on real embeddings and provides code. The main application to LLM hallucination detection is promising, although the connection between the theory and the dependent token-level setting remains incomplete.

major comments (3)
  1. [Sections 3.1, 4.4, 5] The theoretical guarantees in Theorem 3.1 assume i.i.d. samples from densities, but the primary natural-language applications use token-level hidden states from single prompt-response pairs, which are strongly dependent. The paper itself admits in Section 5 that 'extending the theory to dependent token-level samples... would make the method better aligned with LLM applications.' Section 4.4 confirms that the asymptotic calibration is not valid on real text embeddings. The label-shuffling variant ASK-NN (LS) is then used, but label shuffling only yields a valid null distribution under exchangeability of the pooled sample; with within-sequence dependence, permuting labels across sequences breaks the dependence structure. No theory or simulation is provided to show that LS controls Type I error on dependent token-level data. This is a load-bearing gap: the claim of a calibrated drift test f
  2. [Theorem 3.1 and its proof] The proof of asymptotic normality is only sketched by invoking Bloemena (1964) and Henze (1988). Since ASK-NN is an asymmetric statistic and the graph a+_ij is data-dependent, it is not immediate that the central limit theorem of Bloemena applies without verifying additional conditions. The authors should either state the relevant theorem, show how its conditions are satisfied for the k-NN graph, or provide a more self-contained argument. As written, the derivation relies heavily on external results and is difficult to verify.
  3. [Section 4.4, Figure 3] The original ASK-NN's Type I error deviates from the nominal level on real text embeddings, which means the test as proposed with its asymptotic p-values is not valid in the artificial-text detection setting. The label-shuffling variant is presented as a fix, but its Type I control is not quantified with confidence intervals or compared statistically against the nominal level. More importantly, no justification is given for why label shuffling is appropriate when the underlying samples are dependent. The manuscript should either provide such evidence or significantly temper the claim that ASK-NN is a calibrated test for natural-language drift.
minor comments (4)
  1. [Abstract and Section 5] The abstract says 'directed 1-nearest-neighbor graph,' but the statistic is defined for general k and experiments use k>1. Section 5 repeats 'directed 1-nearest-neighbor coincidences.' Please make the notation consistent.
  2. [Throughout] There are several typos: 'Saint-Peterburg' should be 'Saint-Petersburg'; the corresponding author placeholder 'Firstname1 Lastname1 <first1.last1@xxx.edu>' appears to be unanonymized filler.
  3. [Section 3.3] In the definition of σ^2, the dependence on the norm |·| is not made explicit; please clarify that c_∞ and v_∞ depend on the underlying norm and dimension.
  4. [Section 4.5] The hallucination detection experiment uses ASK-NN only as a score (ROC AUC), not as a calibrated test. This should be stated explicitly in the experimental setup to avoid confusion with the testing framework of Section 3.

Circularity Check

0 steps flagged

No significant circularity: ASK-NN is a directly defined asymmetric variant of Henze's NN test, with null moments and limit behavior derived from external results rather than from its own predictions.

full rationale

The paper's central object, T_n = sum_i sum_r A(Z_i) A(N_r(Z_i)), is defined directly from the pooled sample, and its claimed properties are proved rather than imported as conclusions. The exact conditional mean and variance under the permutation null (Proposition 3.1) are taken from Bloemena (1964) through the same adjacency-matrix notation used by Henze (1988); asymptotic normality (Theorem 3.1) and the consistency functional Dtilde (Theorem 3.2) are explicitly adapted from Henze with the needed limiting constants c_inf and v_inf cited as external results. Proposition 3.3 then proves the equality condition for Dtilde directly from the identity Dtilde(f,g,tau) = [D(f,g,tau) + 2tau - 1]/2 and Henze's Proposition 4.3; the conclusion that Dtilde is minimized iff f=g is a consequence of that algebra, not an input. There is no fitted parameter later renamed as a prediction: the statistic uses no estimated nuisance parameters, and the theoretical p-value calibration follows from the derived moments. The self-citations (Oblovatny et al. 2025, Bazarova et al. 2025) are contextual for LLM hallucination evaluation and do not carry the mathematical load; the theory rests on the external works of Henze and Bloemena, and the empirical claims are benchmarked against independent baselines such as MMD, Sinkhorn divergence, Hotelling's T^2, and Henze's symmetric test. The paper's own Section 5 limitation that asymptotic calibration can be inaccurate for real hidden-state embeddings and that dependent token-level samples are not covered is a validity caveat about assumptions, not a circularity: it does not turn any output into an input. The label-shuffling variant is presented as an empirical resampling calibration, not derived from the asymptotic theory, so it is not a case of calling a fit a prediction, though it is correctly flagged as lacking theoretical backing for dependent data. Overall, no step in the derivation chain reduces by construction to its own input.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The only free parameter is k, the neighbor count. The main axioms are the i.i.d. density model and the borrowed convergence conditions from Henze. No new physical entities are introduced.

free parameters (1)
  • k (number of nearest neighbors) = not specified; likely 1 in experiments
    The statistic depends on k, but the paper does not state the value used in the empirical sections except in the baseline name for Henze's 1-NN test. This is a user-chosen hyperparameter.
axioms (4)
  • domain assumption X and Y samples are independent with densities f and g
    The theoretical framework assumes i.i.d. draws from densities, but token-level hidden states are dependent, so this is a modeling assumption.
  • standard math The permutation null conditional on pooled locations is valid
    Under H0, the labels are exchangeable given the pooled points.
  • domain assumption The k-NN graph is unique (no ties)
    The adjacency matrix a_ij is implicitly assumed well-defined; ties would require a tie-breaking rule.
  • standard math Henze's conditions for convergence of c_n and v_n to c_inf and v_inf hold
    The paper refers to Henze (1988) for these limits, which require regularity conditions on f and g.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language." pith.science (2026). https://pith.science/paper/HC6WMCNY

@misc{pith2026260715607,
  author       = {Pith},
  title        = {Pith review of: ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HC6WMCNY}},
  note         = {Machine review of arXiv:2607.15607}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions. Since prompts or retrieved contexts typically serve as reference samples and responses as query samples, with major differences in length, these asymmetries motivate the use of change test statistics that treat the two samples differently. We consider an asymmetric two-sample test ASK-NN based on the directed k-nearest-neighbor graph. Our statistic counts reference points whose nearest neighbor in the pooled sample is also a reference point. Under the permutation null, it admits an exact finite-sample conditional mean and variance; we further establish asymptotic normality and consistency under fixed alternatives. ASK-NN is computationally effective and easy to implement. Empirically, it is competitive with kernel and graph-based baselines on synthetic benchmarks, artificial-text detection, and LLM hallucination detection from token-level hidden states.

Figures

Figures reproduced from arXiv: 2607.15607 by Alexey Zaytsev, Rodion Oblovatny, Sergey Zakharov.

Figure 1
Figure 1. Figure 1: Observed wall-clock runtime scaling of the proposed ASK-NN statistic and baseline two-sample statistics on an NVIDIA A100 GPU for dimensions d = 128 (Left) and d = 1024 (Right). Runtime is measured only for statistic computation, excluding data generation. Both axes use logarithmic scales. Missing points correspond to configurations that exceeded the available GPU memory. mean-shift experiments, we use 20 … view at source ↗
Figure 2
Figure 2. Figure 2: Empirical power on synthetic Gaussian two-sample benchmarks at a given significance level α = 0.05. Top: Gaus￾sians having the same variance and different means. Bottom: Gaussians having the same mean and different variances. the GPT-4-generated subset. Texts are embedded using the multilingual-e5-large sentence encoder (Wang et al., 2024), and the resulting embedding samples are compared using the conside… view at source ↗
Figure 3
Figure 3. Figure 3: Type II error and Type I error on the RAID artificial-text detection dataset as a function of sample size. The label-shuffling version of the proposed asymmetric statistic maintains Type I error close to the nominal level while retaining strong power. Results [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 6 linked inside Pith

  1. [6]

    Fact-checking the output of large language models via token-level uncertainty quantifica- tion

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shel- manov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. Fact-checking the output of large language models via token-level uncertainty quantifica- tion. InFindings of the Association for Computational Linguistics: ACL 2024, pages 9367–9385,

    2024

  2. [9]

    Probabilistic distances-based hal- lucination detection in llms with rag.arXiv preprint arXiv:2506.09886,

    Rodion Oblovatny, Alexandra Kuleshova, Konstantin Polev, and Alexey Zaytsev. Probabilistic distances-based hal- lucination detection in llms with rag.arXiv preprint arXiv:2506.09886,

    arXiv

  3. [12]

    Multilingual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672,

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report.arXiv preprint arXiv:2402.05672,

    Pith/arXiv arXiv

  4. [1964]

    Lookback lens: De- tecting and mitigating contextual hallucinations in large language models using only attention maps

    Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. Lookback lens: De- tecting and mitigating contextual hallucinations in large language models using only attention maps. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1419–1436,

    2024

  5. [2012]

    How close is chatgpt to human experts? compari- son corpus, evaluation, and detection.arXiv preprint arXiv:2301.07597,

    Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jin- ran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? compari- son corpus, evaluation, and detection.arXiv preprint arXiv:2301.07597,

    Pith/arXiv arXiv

  6. [2014]

    Liam Dugan, Alyssa Hwang, Filip Trhl ´ık, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch

    URL https://arxiv.org/abs/ 1408.5404. Liam Dugan, Alyssa Hwang, Filip Trhl ´ık, Andrew Zhu, Josh Magnus Ludan, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. RAID: A shared benchmark for robust evaluation of machine-generated text detec- tors. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    Pith/arXiv arXiv

  7. [2019]

    Hallucination detection: A probabilistic framework using embeddings distance analysis.arXiv preprint arXiv:2502.08663,

    Emanuele Ricco, Lorenzo Cima, and Roberto Di Pietro. Hallucination detection: A probabilistic framework using embeddings distance analysis.arXiv preprint arXiv:2502.08663,

    Pith/arXiv arXiv

  8. [2020]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Pith/arXiv arXiv

  9. [2023]

    org/papers/v24/21-1289.html

    URL http://jmlr. org/papers/v24/21-1289.html. CH-Wang Sky, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. Do androids know they’re only dreaming of electric sheep? InFindings of the Association for Computational Linguistics: ACL 2024, pages 4401–4420,

    2024

  10. [2024]

    URL https://aclanthology.org/2024

    Association for Computational Linguis- tics. URL https://aclanthology.org/2024. acl-long.674. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. InProceedings of the 18th Confer- ence of the European Chapter of the Association for Com- putational Linguistics: System Demonstrat...

    2024

  11. [2025]

    URL https:// arxiv.org/abs/2504.10063. A. R. Bloemena.Sampling from a Graph, volume 2 of Mathematical Centre Tracts. Mathematisch Centrum, Amsterdam,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.