REVIEW 5 major objections 6 minor 27 references
Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that LLM code attribution can be solved by distribution testing, and that its tool Anubis outperforms detectGPT in zero-shot code attribution.
desk verdict Practical heuristic for zero-shot LLM attribution with strong AUROC numbers, but the EVAL+ oracle breaks the theoretical link, so the distribution-testing guarantees do not carry over. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the pair of testing algorithms Global and Local applied over a bucketing of the token-sequence domain. Buckets are probability shells $\{x : 2^{-j} < L^*(x) \le 2^{-j+1}\}$ under the suspect model; Global compares the empirical CDFs of sample counts per bucket using the Dvoretzky–Kiefer–Wolfowitz inequality, and Local applies a chi-square-type statistic within each bucket. The load-bearing practical piece is EVAL+, which estimates $L(s)$ by calling the model's next-token EVAL oracle over tokenizer collision sequences; getCollision enumerates all depth-$d$ token sequences decoding to the same text, and for long texts the computation splits recursively.
What would settle it
A decisive experiment would be to compare EVAL+ with true model probabilities on a white-box model: if the relative error on typical code samples is large, or if Anubis's acceptance decision flips when EVAL+ is replaced by exact $L(s)$, then the statistical test is not operating on the distributions the paper claims.
Extended reading notes
Core claim
The authors claim that attribution reduces to a test between two hypotheses about the $\ell_1$ distance between the empirical distribution of the sample set and the target model's distribution, and that the test can be run with oracle access combining samples and density evaluations. The algorithm partitions the domain into buckets by target-model probability, checks the empirical bucket CDFs against a DKW-based threshold, then runs a chi-square-type local test inside each bucket. The practical contribution is an implementation whose EVAL+ procedure estimates $L(s)$ by summing over tokenizer collision sequences, and experiments showing this pipeline reaches AUROC $\ge 0.9$ distinguishing DeepSeek-Coder and CodeGemma from Stable-Code with roughly 2000 samples while outperforming detectGPT.
Load-bearing premise
The whole pipeline depends on EVAL+ returning an accurate estimate of the probability that the target model assigned to each sample, but the paper describes EVAL+ as a lower bound that searches only depth-2 tokenizer collisions and gives no error guarantee matching the approximate EVAL oracle needed by the theory.
Editorial extensions
If this is right
- If the reported AUROC transfers, organizations can audit code provenance without training a detector or modifying generation.
- The approach is agnostic to post-processing such as formatting or dead-code removal, because it queries the suspect model's probabilities rather than matching surface text.
- The method's sample complexity becomes a practical design parameter: the paper finds roughly 1000 samples already give AUROC around 0.8–0.9 depending on the model and gap.
- Because the test is zero-shot, a new suspect model only requires access to its log-probabilities; no labeled dataset of that model's outputs is needed.
Reading between the lines
- Editorial inference: the experiments do not separate the value of the bucketing-and-testing idea from the error introduced by EVAL+ with depth $d=2$; replacing EVAL+ with exact probabilities on a white-box model would isolate that.
- Editorial inference: the same bucket-and-test scheme could apply to natural-language text, but the collision enumeration in EVAL+ would become far more expensive as vocabulary and subword ambiguity grow.
- Editorial inference: a minimally fair baseline would be a detector using the same probability oracle but a simpler statistic, such as mean log-probability; if that baseline matches Anubis, the bucketing machinery is not the source of the gain.
- Editorial inference: for pairs of models whose output distributions are close in $\ell_1$, the theoretical sample bound scales with vocabulary size, so applying Anubis to larger models or near-twin checkpoints may require many more samples than the experiments used.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Anubis, a zero-shot attribution method for code samples generated by LLMs, framing attribution as a distribution testing problem. The authors present a theoretical tester (Algorithm 1) adapted from prior work, then propose a practical implementation (EVAL+ in Algorithm 5) that estimates the probability of a text under a target LLM using token-level conditional probabilities. Experiments on code datasets from HumanEval compare Anubis against detectGPT, reporting AUROC above 0.9 in several settings and above 0.8 with as few as 1000 samples. The paper includes proofs in the appendix and claims a strong theoretical foundation for the method.
Significance. If the reported empirical results are reliable, Anubis is a useful practical tool for zero-shot code attribution, and the paper's open-source release is a valuable contribution to a timely problem. The experimental comparison against detectGPT is a reasonable first step, and the evaluation across contamination levels and sample sizes is informative. However, the theoretical claims are not supported by the practical algorithm: the distribution-testing guarantee in Theorem 3.1 applies only under an exact or η-approximate EVAL oracle and a condition on the reference distribution that is not verified, while the implemented EVAL+ deviates substantially from that oracle. The paper is more convincing as an empirical heuristic study than as an instance of the distribution-testing framework, and the central claim that the method's success 'reflects the strong theoretical foundation' is not justified by the evidence presented.
major comments (5)
- [Section 4.1, Algorithm 5] EVAL+ as specified is not a valid approximation to the oracle required by Definition 2.1. In the recursive case (|σ|>d), the second factor EVAL+(σ_{i+1}...σ_n,d) is computed as a standalone token sequence, so the first token of the right part is not conditioned on the left context; the product is therefore not the probability of the concatenated sequence under the autoregressive model. Additionally, different branches of the sum in line 8 double-count the same tokenization; for example, with n=4 and d=2, the full sequence σ1σ2σ3σ4 is counted in both the i=1 and i=2 branches. This contradicts the claimed 'lower bound' output in the algorithm header. Since Anubis uses these values as oracle probabilities in Algorithms 2-4, the theoretical guarantee of Algorithm 1 does not transfer to the implemented method. The authors must either (a) provide a corrected EVAL+ with a proven η-approximation property, or (b) explicitly state that the practical algorithm is a heuristic and remove the distribution-testing guarantee claims.
- [Section 3, Theorem 3.1 and Section 4] The condition ||P*||_2 ≤ sqrt(2/|Ω|) is not verified for language models, and the paper does not discuss whether code LLMs such as DeepSeek-Coder or CodeGemma satisfy it. If this condition fails, the theoretical testing guarantee is void. Furthermore, the sample complexity in Theorem 3.1 depends on |Ω|, the support size of the token-sequence domain, which is astronomically large for LLMs; the reported success with ~1000 samples is in apparent conflict with this bound. The authors should either verify the ℓ2-norm condition for their models and reconcile the sample complexity, or explain why the theory is nonetheless predictive for the empirical regime.
- [Section 4.1 and Algorithm 6] getCollision(tok, σ, d) enumerates all token sequences in Σ_tok^d, which for d=2 and a typical vocabulary size of ~5×10^4 is about 2.5×10^9 sequences per input sample. The paper does not describe how this enumeration is made tractable in the reported experiments, nor does it specify whether the implementation actually follows this pseudocode. If the implementation does not enumerate all such sequences, the pseudocode is not a faithful description of the evaluated algorithm; if it does, the computational cost appears prohibitive. This is a reproducibility issue that must be resolved.
- [Section 5, evaluation methodology] The paper does not explain how an AUROC curve is obtained from Anubis, which is described as a test that outputs a binary accept/reject decision. To compute AUROC, a continuous score must be used to rank positive and negative datasets; the authors should specify whether they use an internal statistic (e.g., the global or local test statistics) or vary a threshold in some other way. Without this, the reported AUROC values are not reproducible.
- [Section 3, Algorithm 4 and Appendix A.1] The Local algorithm computes Z = Σ (z1−z2)^2−z1−z2 over max(z1+z2,1), which is not the chi-squared statistic used in the theoretical analysis referenced in the appendix (where the bound is based on an identity-testing result from Canonne et al. 2021). The thresholds in Algorithm 1 (Thresh2) are derived from the theoretical analysis of a different statistic, so applying them to this ad-hoc statistic is not justified. The authors should either prove the correctness of this statistic under the stated guarantees or align the implementation with the analyzed test.
minor comments (6)
- [Title and Section 2] The heading 'Preliminiaries' contains a typo; it should be 'Preliminaries'.
- [Section 2, Definition 2.1] The definition of an approximate EVAL oracle is stated correctly, but the notation η-estimate could be made more explicit about whether the guarantee is per-query or uniform; the paper later uses the oracle in a way that assumes consistency across queries, which is not guaranteed by the definition as written.
- [Section 3, Theorem 3.1] The theorem's sample complexity expression has unbalanced parentheses and the role of the tilde in eO is not explained; please reformat for clarity and state which algorithm achieves the bound.
- [Appendix A.1] In the proof, the sentence 'P ∗(∆j)≥ c4ε2 2ℓ ≥ 2c3ε2 ℓ ,2 then we can assume' is garbled; please fix the typesetting and the logical structure. There is also a typo 'fo the buckets considered'.
- [Section 2, Language Models as Distributions] The definition of the tokenizer's decoder is inconsistent: Definition 2.3 says dec:Σ_tok→Σ_w, but it is later used on sequences; this should be stated uniformly as dec on token sequences.
- [Section 5, Figures 2 and 3] The number of independent runs used to produce the shaded regions and averaged AUROC is not reported; please include this information for statistical interpretability.
Circularity Check
No significant circularity: the central AUROC claim is an external benchmark comparison, and the self-cited distribution-testing theorem is independent support.
full rationale
The claimed derivation chain is not circular. The theoretical guarantee in Theorem 3.1 is explicitly adapted from Canonne et al. (2022), a prior distribution-testing result whose stated assumptions (a fully specified reference distribution and access to an EVAL oracle) do not include the paper's attribution target; it is therefore independent evidence rather than a conclusion that is assumed into existence. Although the first author is a co-author of that cited work, the theorem is not invoked as an unverifiable uniqueness principle and does not forbid competing approaches. The practical pipeline replaces the idealized EVAL oracle with EVAL+, and the empirical evaluation compares Anubis against detectGPT on code samples derived from the external HumanEval benchmark, so the reported AUROC does not reduce to a fitted parameter or to the paper's own test thresholds. The paper does not report the numerical values of epsilon_1, epsilon_2, and c_1 through c_4, but there is no quoted evidence that these were tuned on the evaluation set, so hidden calibration remains only a speculative concern. The apparent invalidity of EVAL+ as a true probability oracle is a correctness and robustness issue, not a circularity: the test decisions may lose their statistical grounding, but they are not equivalent to their inputs by construction.
Assumptions & free parameters
free parameters (3)
- tau_Delta0 (leftover fraction) =
0.05
- d (EVAL+ depth) =
2
- Threshold constants epsilon1, epsilon2, c1, c2, c3, c4 =
not reported
assumptions (4)
- standard math DKW inequality bounds the l-infinity error of the empirical CDF for i.i.d. samples (Lemma 2.2).
- domain assumption The LLM token distribution factorizes via the chain rule as Ltok(sigma) = Prod_i Ltok(sigma_i | sigma_<i).
- ad hoc to paper The reference distribution satisfies ||P*||_2 <= sqrt(2/|Omega|), and the bucketing approximation preserves the testing guarantee.
- ad hoc to paper EVAL+ provides an eta-approximate EVAL oracle (Definition 2.1).
Cite this review
Pith. "Pith review of Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach." pith.science (2026). https://pith.science/paper/E32HWFM5
@misc{pith2026250620197,
author = {Pith},
title = {Pith review of: Zero-Shot Attribution for Large Language Models: A Distribution Testing Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/E32HWFM5}},
note = {Machine review of arXiv:2506.20197}
}
abstract
A growing fraction of all code is sampled from Large Language Models (LLMs). We investigate the problem of attributing code generated by language models using hypothesis testing to leverage established techniques and guarantees. Given a set of samples $S$ and a suspect model $\mathcal{L}^*$, our goal is to assess the likelihood of $S$ originating from $\mathcal{L}^*$. Due to the curse of dimensionality, this is intractable when only samples from the LLM are given: to circumvent this, we use both samples and density estimates from the LLM, a form of access commonly available. We introduce $\mathsf{Anubis}$, a zero-shot attribution tool that frames attribution as a distribution testing problem. Our experiments on a benchmark of code samples show that $\mathsf{Anubis}$ achieves high AUROC scores ( $\ge0.9$) when distinguishing between LLMs like DeepSeek-Coder, CodeGemma, and Stable-Code using only $\approx 2000$ samples.
Figures
Reference graph
Works this paper leans on
-
[1]
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv preprint arXiv:2310.05130, 2023
arXiv 2023
-
[2]
The complexity of approximating the entropy
Tugkan Batu, Sanjoy Dasgupta, Ravi Kumar, and Ronitt Rubinfeld. The complexity of approximating the entropy. SIAM J. Comput. , 35 0 (1): 0 132--150, 2005
work page 2005
-
[3]
Efficient distance approximation for structured high-dimensional distributions via learning
Arnab Bhattacharyya, Sutanu Gayen, Kuldeep S Meel, and NV Vinodchandran. Efficient distance approximation for structured high-dimensional distributions via learning. Advances in Neural Information Processing Systems, 33: 0 14699--14711, 2020
work page 2020
-
[4]
Cl\'ement L. Canonne and Ronitt Rubinfeld. Testing probability distributions underlying aggregated data. In Proceedings of ICALP, pp.\ 283--295, 2014
work page 2014
-
[5]
Canonne, Dana Ron, and Rocco A
Cl\'ement L. Canonne, Dana Ron, and Rocco A. Servedio. Testing equivalence between distributions using conditional samples. In Proceedings of SODA, pp.\ 1174--1192. Society for Industrial and Applied Mathematics (SIAM), 2014. ISBN 978-1-611973-38-9. URL http://dl.acm.org/citation.cfm?id=2634074.2634161
-
[6]
The Price of Tolerance in Distribution Testing
Cl \' e ment L. Canonne, Ayush Jain, Gautam Kamath, and Jerry Li. The price of tolerance in distribution testing. CoRR, abs/2106.13414, 2021
work page Pith review arXiv 2021
-
[7]
Canonne, Ayush Jain, Gautam Kamath, and Jerry Li
Cl \' e ment L. Canonne, Ayush Jain, Gautam Kamath, and Jerry Li. The price of tolerance in distribution testing. In COLT , volume 178 of Proceedings of Machine Learning Research, pp.\ 573--624. PMLR , 2022. Full version at CanonneJKL22:arxiv
work page 2022
-
[8]
Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winte...
arXiv 2021
Show all 27 references
-
[9]
Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator
Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, 27 0 (3): 0 642--669, 1956
1956
-
[10]
The foundations of tokenization: Statistical and computational concerns
Juan Luis Gastaldi, John Terilla, Luca Malagutti, Brian DuSell, Tim Vieira, and Ryan Cotterell. The foundations of tokenization: Statistical and computational concerns. arXiv preprint arXiv:2407.11606, 2024
2024 arXiv
-
[11]
Watermarking pre-trained language models with backdooring
Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsieh. Watermarking pre-trained language models with backdooring. arXiv preprint arXiv:2210.07543, 2022
2022 arXiv
-
[12]
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming -- the rise of code intelligence, 2024. URL https://arxiv....
2024 arXiv
-
[13]
On the reliability of watermarks for large language models
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023
2023 arXiv
-
[14]
J-guard: Journalism guided adversarially robust detection of ai-generated news
Tharindu Kumarage, Amrita Bhattacharjee, Djordje Padejski, Kristy Roschke, Dan Gillmor, Scott Ruston, Huan Liu, and Joshua Garland. J-guard: Journalism guided adversarially robust detection of ai-generated news. arXiv preprint arXiv:2309.03164, 2023
2023 arXiv
-
[15]
The tight constant in the D voretzky-- K iefer-- W olfowitz inequality
Pascal Massart. The tight constant in the D voretzky-- K iefer-- W olfowitz inequality. The Annals of Probability, 18 0 (3): 0 1269--1283, 07 1990. doi:10.1214/aop/1176990746. URL http://dx.doi.org/10.1214/aop/1176990746
1990
-
[16]
Detectgpt: Zero-shot machine-generated text detection using probability curvature
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pp.\ 24950--24962. PMLR, 2023
2023
-
[17]
Probability-revealing samples
Krzysztof Onak and Xiaorui Sun. Probability-revealing samples. In AISTATS , volume 84 of Proceedings of Machine Learning Research, pp.\ 2018--2026. PMLR , 2018
2018
-
[18]
Stable code 3b, 2023
Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. Stable code 3b, 2023. URL [https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b)
2023
-
[19]
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019
2019
-
[20]
Neural machine translation of rare words with subword units
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015
2015 arXiv
-
[21]
Did you train on my dataset? towards public dataset protection with cleanlabel backdoor watermarking
Ruixiang Tang, Qizhang Feng, Ninghao Liu, Fan Yang, and Xia Hu. Did you train on my dataset? towards public dataset protection with cleanlabel backdoor watermarking. ACM SIGKDD Explorations Newsletter, 25 0 (1): 0 43--53, 2023
2023
-
[22]
Codegemma: Open code models based on gemma
CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette-Choo, Jingyue Shen, Joe Kelley, et al. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409, 2024
2024 arXiv
-
[23]
Authorship attribution for neural text generation
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp.\ 8384--8395, 2020
2020
-
[24]
Topformer: Topology-aware authorship attribution of deepfake texts with diverse writing styles
Adaku Uchendu, Thai Le, and Dongwon Lee. Topformer: Topology-aware authorship attribution of deepfake texts with diverse writing styles. In ECAI 2024, pp.\ 1446--1454. IOS Press, 2024
2024
-
[25]
Estimating the unseen: An n/ n -sample estimator for entropy and support size, shown optimal via new clts
Gregory Valiant and Paul Valiant. Estimating the unseen: An n/ n -sample estimator for entropy and support size, shown optimal via new clts. In Symposium on Theory of Computing Conference, STOC'11, pp.\ 685--694, 2011
2011
-
[26]
A survey on llm-generated text detection: Necessity, methods, and future directions
Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, pp.\ 1--66, 2025
2025
-
[27]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.