{"id":"b9543569-25b5-4597-a0b0-0dae0975b947","arxiv_id":"2506.20197","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Anubis re-frames LLM attribution as a distribution testing problem with EVAL access, and reports AUROC above 0.9 on code benchmarks with around 2000 samples, beating detectGPT.","lead":"This paper introduces Anubis, a tool that attributes batches of code samples to a specific large language model by treating the model as a probability distribution and testing whether the sample set looks like it was drawn from that distribution. The authors report that Anubis separates code from different LLMs with AUROC above 0.9 in some settings using about two thousand samples, outperforming the detectGPT baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 5's EVAL+ is not a valid probability oracle: its recursive split ignores cross-segment conditioning and counts the same tokenization multiple times, so the distribution-testing guarantee in Theorem 3.1 does not transfer to Anubis.","rationale":"The reader identified EVAL+ as the weakest assumption, citing the missing error guarantee and the lower-bound behavior. My stress-test finds a stronger, more specific failure: EVAL+ as written is not merely unvalidated, it is internally inconsistent. The recursive multiplication ignores the prefix context on the right-hand segment, so the product is not the probability of any concatenated token sequence, and the overlapping split branches double-count tokenizations, so the output can exceed the true L(s) despite being labeled a lower bound. These are not matters of consensus or tuning; they are checkable properties of the pseudocode. The central contribution of the paper is to port the distribution-testing framework (Theorem 3.1) into practice by supplying an EVAL oracle. If the supplied oracle is not an oracle, the theoretical guarantees are void, and the empirical AUROC values, while possibly reproducible, are not evidence for the stated framework unless the implementation uses a corrected estimator. The reader's conditional verdict assumed the issue was a missing derivation or threshold reporting; the concrete double-counting and context-loss errors show the submitted algorithm cannot implement the required oracle. The paper could be reconsidered after replacing or correcting EVAL+ and re-running the experiments with a valid oracle, but as submitted the central methodological claim fails. Therefore the verdict should move from CONDITIONAL to REJECT for the current version.","tokens_in":14074,"tokens_out":11908,"duration_ms":136162,"concrete_test":"Run EVAL+ from the released code on a small controlled case where exact enumeration is possible: use a tiny tokenizer (4–8 tokens) or the GPT2 tokenizer restricted to a short code snippet of length 4 tokens, enumerate all token sequences of lengths 1–6 decoding to the same text, and compute L(s) = Σ_{σ: dec(σ)=s} L_tok(σ). Compare with EVAL+(s, d=2). The concern lands if EVAL+ exceeds L(s) (double counting) or deviates by more than the η in Definition 2.1 for any allowed η<1; in particular, the σ=σ1σ2σ3σ4 example should already show EVAL+ ≥ 2·L(σ), falsifying the lower-bound claim.","verdict_should_be":"REJECT","load_bearing_attack":"The practical pipeline uses EVAL+ (Algorithm 5) as the EVAL oracle of Definition 2.1, but EVAL+ is not an approximation to L(s) and is not even a lower bound. Two concrete defects. First, for |σ|>d the recursive return is Σ_{i=1}^d EVAL+(σ_1...σ_i, i) · EVAL+(σ_{i+1}...σ_n, d). The second factor is evaluated as a standalone token sequence, so the product is not the probability of the concatenated token sequence under the prefix-dependent LLM: it conditions the right part only on its own internal prefix, ignoring the left context. Thus the computed value is not L(s), nor an η-estimate of L(s). Second, different branches double-count the same tokenization. For n=4, d=2, the branch i=1 yields the decomposition σ1 | σ2 | σ3σ4 and the branch i=2 yields σ1σ2 | σ3σ4; both include the original token sequence σ=σ1σ2σ3σ4, so EVAL+(σ,2) ≥ 2·EVAL(L_tok, σ). This directly contradicts the 'lower bound' description in Algorithm 5 line 2. Since bucketing (Algorithm 2) and both global and local tests consume these values as oracle probabilities, the theoretical guarantee of Algorithm 1 does not apply to the implemented Anubis. The reported AUROC may reflect a heuristic score rather than the claimed distribution-testing method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Anubis, a zero-shot attribution method for code samples generated by LLMs, framing attribution as a distribution testing problem. The authors present a theoretical tester (Algorithm 1) adapted from prior work, then propose a practical implementation (EVAL+ in Algorithm 5) that estimates the probability of a text under a target LLM using token-level conditional probabilities. Experiments on code datasets from HumanEval compare Anubis against detectGPT, reporting AUROC above 0.9 in several settings and above 0.8 with as few as 1000 samples. The paper includes proofs in the appendix and claims a strong theoretical foundation for the method.","tokens_in":14501,"tokens_out":7853,"duration_ms":76593,"significance":"If the reported empirical results are reliable, Anubis is a useful practical tool for zero-shot code attribution, and the paper's open-source release is a valuable contribution to a timely problem. The experimental comparison against detectGPT is a reasonable first step, and the evaluation across contamination levels and sample sizes is informative. However, the theoretical claims are not supported by the practical algorithm: the distribution-testing guarantee in Theorem 3.1 applies only under an exact or η-approximate EVAL oracle and a condition on the reference distribution that is not verified, while the implemented EVAL+ deviates substantially from that oracle. The paper is more convincing as an empirical heuristic study than as an instance of the distribution-testing framework, and the central claim that the method's success 'reflects the strong theoretical foundation' is not justified by the evidence presented.","major_comments":[{"comment":"EVAL+ as specified is not a valid approximation to the oracle required by Definition 2.1. In the recursive case (|σ|>d), the second factor EVAL+(σ_{i+1}...σ_n,d) is computed as a standalone token sequence, so the first token of the right part is not conditioned on the left context; the product is therefore not the probability of the concatenated sequence under the autoregressive model. Additionally, different branches of the sum in line 8 double-count the same tokenization; for example, with n=4 and d=2, the full sequence σ1σ2σ3σ4 is counted in both the i=1 and i=2 branches. This contradicts the claimed 'lower bound' output in the algorithm header. Since Anubis uses these values as oracle probabilities in Algorithms 2-4, the theoretical guarantee of Algorithm 1 does not transfer to the implemented method. The authors must either (a) provide a corrected EVAL+ with a proven η-approximation property, or (b) explicitly state that the practical algorithm is a heuristic and remove the distribution-testing guarantee claims.","section":"Section 4.1, Algorithm 5"},{"comment":"The condition ||P*||_2 ≤ sqrt(2/|Ω|) is not verified for language models, and the paper does not discuss whether code LLMs such as DeepSeek-Coder or CodeGemma satisfy it. If this condition fails, the theoretical testing guarantee is void. Furthermore, the sample complexity in Theorem 3.1 depends on |Ω|, the support size of the token-sequence domain, which is astronomically large for LLMs; the reported success with ~1000 samples is in apparent conflict with this bound. The authors should either verify the ℓ2-norm condition for their models and reconcile the sample complexity, or explain why the theory is nonetheless predictive for the empirical regime.","section":"Section 3, Theorem 3.1 and Section 4"},{"comment":"getCollision(tok, σ, d) enumerates all token sequences in Σ_tok^d, which for d=2 and a typical vocabulary size of ~5×10^4 is about 2.5×10^9 sequences per input sample. The paper does not describe how this enumeration is made tractable in the reported experiments, nor does it specify whether the implementation actually follows this pseudocode. If the implementation does not enumerate all such sequences, the pseudocode is not a faithful description of the evaluated algorithm; if it does, the computational cost appears prohibitive. This is a reproducibility issue that must be resolved.","section":"Section 4.1 and Algorithm 6"},{"comment":"The paper does not explain how an AUROC curve is obtained from Anubis, which is described as a test that outputs a binary accept/reject decision. To compute AUROC, a continuous score must be used to rank positive and negative datasets; the authors should specify whether they use an internal statistic (e.g., the global or local test statistics) or vary a threshold in some other way. Without this, the reported AUROC values are not reproducible.","section":"Section 5, evaluation methodology"},{"comment":"The Local algorithm computes Z = Σ (z1−z2)^2−z1−z2 over max(z1+z2,1), which is not the chi-squared statistic used in the theoretical analysis referenced in the appendix (where the bound is based on an identity-testing result from Canonne et al. 2021). The thresholds in Algorithm 1 (Thresh2) are derived from the theoretical analysis of a different statistic, so applying them to this ad-hoc statistic is not justified. The authors should either prove the correctness of this statistic under the stated guarantees or align the implementation with the analyzed test.","section":"Section 3, Algorithm 4 and Appendix A.1"}],"minor_comments":[{"comment":"The heading 'Preliminiaries' contains a typo; it should be 'Preliminaries'.","section":"Title and Section 2"},{"comment":"The definition of an approximate EVAL oracle is stated correctly, but the notation η-estimate could be made more explicit about whether the guarantee is per-query or uniform; the paper later uses the oracle in a way that assumes consistency across queries, which is not guaranteed by the definition as written.","section":"Section 2, Definition 2.1"},{"comment":"The theorem's sample complexity expression has unbalanced parentheses and the role of the tilde in eO is not explained; please reformat for clarity and state which algorithm achieves the bound.","section":"Section 3, Theorem 3.1"},{"comment":"In the proof, the sentence 'P ∗(∆j)≥ c4ε2 2ℓ ≥ 2c3ε2 ℓ ,2 then we can assume' is garbled; please fix the typesetting and the logical structure. There is also a typo 'fo the buckets considered'.","section":"Appendix A.1"},{"comment":"The definition of the tokenizer's decoder is inconsistent: Definition 2.3 says dec:Σ_tok→Σ_w, but it is later used on sequences; this should be stated uniformly as dec on token sequences.","section":"Section 2, Language Models as Distributions"},{"comment":"The number of independent runs used to produce the shaded regions and averaged AUROC is not reported; please include this information for statistical interpretability.","section":"Section 5, Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The theoretical framework in Section 3 is largely a restatement of Canonne et al. (2022), with the first author of the current paper as a co-author, and the appendix provides a detailed derivation. The novelty of the paper lies in the practical adaptation and experiments. However, the practical adaptation (EVAL+) appears to be internally inconsistent and computationally unrealistic as described, and the link between the distribution-testing theory and the implemented algorithm is not established. I recommend requiring a major revision in which the authors either (a) fix and validate EVAL+ as an η-approximate oracle, or (b) explicitly reframe the paper as a heuristic approach with empirical support only and remove the theoretical-guarantee claims. The editor may also wish to ask the authors to release the code and clarify the exact implementation details for reproducibility. The comparison to detectGPT is only one baseline; additional baselines such as log-likelihood thresholding would strengthen the empirical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper ports a distribution-testing algorithm to zero-shot LLM attribution and reports strong AUROC numbers against detectGPT on code. That empirical comparison is the new content, and it is worth looking at. But the practical oracle, EVAL+, does not implement the approximate EVAL oracle the algorithm needs, so the theoretical guarantees in Theorem 3.1 do not transfer to Anubis as implemented.\n\nWhat is good: the problem is real and timely. Framing attribution as closeness between distributions, with access to both samples and logprobs, is a sensible abstraction. The authors are honest that the theory is adapted from Canonne et al. (2022) — they say so explicitly. They also run a nontrivial evaluation on code LLMs (DeepSeek-Coder, CodeGemma, Stable-Code) and compare against detectGPT, the standard zero-shot baseline. The EVAL+ attempt to handle tokenizer non-invertibility is a genuine practical obstacle they identify and try to address.\n\nWhere it falls down: EVAL+ (Algorithm 5) is not a lower bound on L(s), and it is not an eta-estimate either. When |sigma|>d, the recursion splits the sequence and multiplies EVAL+ of the prefix with EVAL+ of the suffix evaluated as a standalone sequence. That ignores the autoregressive conditioning: the suffix's probability should be conditioned on the prefix, not computed marginally. The product can double-count or mis-count tokenizations; the stress-test note's example of overlapping decompositions is real. So the bucketing, global and local tests are consuming numbers with no proven relationship to the true probabilities. The paper does not analyze this, and the threshold constants and AUROC computation are under-specified. The condition ||P*||_2 <= sqrt(2/|Omega|) is not checked for the LLM distributions, so even the adapted theorem is not connected to the experiments.\n\nThis matters because the paper's selling point is that it inherits distribution-testing guarantees. Without a valid oracle, the AUROC numbers are those of a heuristic. That heuristic may still work — the experiments suggest it does — but the paper does not tell us why, and it does not give enough detail to reproduce the exact thresholds. The citation pattern is fine; the first author's prior work is the natural basis.\n\nBottom line: this is a promising heuristic with a mislabeled theoretical foundation. The empirical comparison is worth refereeing, and the EVAL+ flaw is fixable in revision — either by a proper accounting of the recursion or by explicitly reframing the method as heuristic. I would send it to review, but with major revision required.","headline":"Practical heuristic for zero-shot LLM attribution with strong AUROC numbers, but the EVAL+ oracle breaks the theoretical link, so the distribution-testing guarantees do not carry over.","tokens_in":14948,"tokens_out":5070,"would_cite":false,"duration_ms":50388,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62G10","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM code attribution can be solved by distribution testing, and that its tool Anubis outperforms detectGPT in zero-shot code attribution.","keywords":["zero-shot attribution","distribution testing","LLM code detection","EVAL oracle","tokenizer collision","AUROC","Anubis","hypothesis testing"],"falsifier":"A decisive experiment would be to compare EVAL+ with true model probabilities on a white-box model: if the relative error on typical code samples is large, or if Anubis's acceptance decision flips when EVAL+ is replaced by exact $L(s)$, then the statistical test is not operating on the distributions the paper claims.","tokens_in":13908,"feed_emoji":"🤖","tokens_out":5388,"duration_ms":51424,"temperature":0.7,"pith_summary":"Anubis is a zero-shot method for attributing a set of code samples to a suspect language model: it decides, with high confidence, whether at least a large fraction of the samples came from that model, using no training and no watermarking. The paper's central claim is that this task can be recast as a hypothesis-testing problem over the probability mass function of the model, provided the model allows both sampling and per-sequence probability queries. On code generated by DeepSeek-Coder, CodeGemma, and Stable-Code, Anubis is reported to separate target-model samples from others with AUROC consistently above 0.9 when the acceptance/rejection gap is at least 70 percentage points, and to remain useful with as few as about 1000 samples. The authors argue this matters because attribution of machine-written code is becoming a practical need, and existing zero-shot statistical detectors such as detectGPT degrade on code.","feed_headline":"Distribution testing attributes AI code with AUROC above 0.9","feed_subtitle":"Anubis tells which LLM wrote a code sample set using only samples and probability queries.","key_machinery":"The central object is the pair of testing algorithms Global and Local applied over a bucketing of the token-sequence domain. Buckets are probability shells $\\{x : 2^{-j} < L^*(x) \\le 2^{-j+1}\\}$ under the suspect model; Global compares the empirical CDFs of sample counts per bucket using the Dvoretzky–Kiefer–Wolfowitz inequality, and Local applies a chi-square-type statistic within each bucket. The load-bearing practical piece is EVAL+, which estimates $L(s)$ by calling the model's next-token EVAL oracle over tokenizer collision sequences; getCollision enumerates all depth-$d$ token sequences decoding to the same text, and for long texts the computation splits recursively.","core_discovery":"The authors claim that attribution reduces to a test between two hypotheses about the $\\ell_1$ distance between the empirical distribution of the sample set and the target model's distribution, and that the test can be run with oracle access combining samples and density evaluations. The algorithm partitions the domain into buckets by target-model probability, checks the empirical bucket CDFs against a DKW-based threshold, then runs a chi-square-type local test inside each bucket. The practical contribution is an implementation whose EVAL+ procedure estimates $L(s)$ by summing over tokenizer collision sequences, and experiments showing this pipeline reaches AUROC $\\ge 0.9$ distinguishing DeepSeek-Coder and CodeGemma from Stable-Code with roughly 2000 samples while outperforming detectGPT.","pith_inferences":["Editorial inference: the experiments do not separate the value of the bucketing-and-testing idea from the error introduced by EVAL+ with depth $d=2$; replacing EVAL+ with exact probabilities on a white-box model would isolate that.","Editorial inference: the same bucket-and-test scheme could apply to natural-language text, but the collision enumeration in EVAL+ would become far more expensive as vocabulary and subword ambiguity grow.","Editorial inference: a minimally fair baseline would be a detector using the same probability oracle but a simpler statistic, such as mean log-probability; if that baseline matches Anubis, the bucketing machinery is not the source of the gain.","Editorial inference: for pairs of models whose output distributions are close in $\\ell_1$, the theoretical sample bound scales with vocabulary size, so applying Anubis to larger models or near-twin checkpoints may require many more samples than the experiments used."],"forward_implications":["If the reported AUROC transfers, organizations can audit code provenance without training a detector or modifying generation.","The approach is agnostic to post-processing such as formatting or dead-code removal, because it queries the suspect model's probabilities rather than matching surface text.","The method's sample complexity becomes a practical design parameter: the paper finds roughly 1000 samples already give AUROC around 0.8–0.9 depending on the model and gap.","Because the test is zero-shot, a new suspect model only requires access to its log-probabilities; no labeled dataset of that model's outputs is needed."],"supporting_citations":[{"why":"Supplies the tolerance-testing theorem that Algorithm 1 adapts.","marker":"Canonne et al. (2022)"},{"why":"Provides the DKW inequality used to set the Global test threshold.","marker":"Dvoretzky et al. (1956)"},{"why":"Fixes the optimal constant in the DKW inequality used in the analysis.","marker":"Massart (1990)"},{"why":"Defines detectGPT, the zero-shot baseline that Anubis is compared against.","marker":"Mitchell et al. (2023)"},{"why":"Supplies the HumanEval prompts used to generate the code datasets.","marker":"Chen et al. (2021)"},{"why":"Provides DeepSeek-Coder, one of the target LLMs in the experiments.","marker":"Guo et al. (2024)"},{"why":"Provides CodeGemma, the other target LLM in the experiments.","marker":"Team et al. (2024)"},{"why":"Provides Stable-Code, the source model used for non-target and adversarial samples.","marker":"Pinnaparaju et al. (2023)"},{"why":"Establishes the dimensionality barrier that motivates oracle-enhanced testing.","marker":"Valiant & Valiant (2011)"},{"why":"Introduces the evaluation-oracle model that the paper ports to LLM attribution.","marker":"Canonne et al. (2014)"}],"fun_headline_variants":["Anubis: zero-shot attribution of code to LLMs via distribution testing","Zero-shot LLM attribution: Anubis hits AUROC 0.9+ on code","Zero-shot code attribution via distribution testing scores AUROC 0.9"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline depends on EVAL+ returning an accurate estimate of the probability that the target model assigned to each sample, but the paper describes EVAL+ as a lower bound that searches only depth-2 tokenizer collisions and gives no error guarantee matching the approximate EVAL oracle needed by the theory.","fun_headline_variants_meta":{"raw":{"variants":["Anubis: zero-shot attribution of code to LLMs via distribution testing","Zero-shot LLM attribution: Anubis hits AUROC 0.9+ on code","Zero-shot code attribution via distribution testing scores AUROC 0.9"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3845,"prompt_tokens":865,"completion_tokens":2980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":2912}},"tokens_in":481,"tokens_out":2980,"duration_ms":18378,"temperature":1.0,"reasoning_tokens":2912,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:54:41.105991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive experiment would be to compare EVAL+ with true model probabilities on a white-box model: if the relative error on typical code samples is large, or if Anubis's acceptance decision flips when EVAL+ is replaced by exact $L(s)$, then the statistical test is not operating on the distributions the paper claims.","supporting_citations":[{"cited_title":"Stable code 3b, 2023","cited_arxiv_id":null,"evidence_quote":"Provides Stable-Code, the source model used for non-target and adversarial samples."},{"cited_title":"Estimating the unseen: An n/ n -sample estimator for entropy and support size, shown optimal via new clts","cited_arxiv_id":null,"evidence_quote":"Establishes the dimensionality barrier that motivates oracle-enhanced testing."}],"review_version":1}