Pith. sign in

REVIEW 2 major objections 4 minor 1 cited by

One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions

T0 review · 2 major / 4 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A large language model can be identified and verified from the distribution of its answers to trivial one-word prompts, each costing a single output token.

desk verdict Solid empirical fingerprinting paper: single-token answer distributions really do separate models at low cost, with strong artifacts and a real commercial use case; main soft spot is unmeasured paraphrase robustness against adaptive providers. read the letter →

arxiv 2607.10252 v1 pith:R457MV2G submitted 2026-07-11 cs.CR cs.CLcs.LG

Tomas Bruckner This is my paper · ORCID
classification cs.CRcs.CLcs.LG
keywords largelanguagemodelsmodelfingerprintingattributionAPIauditingblack-boxverificationinformationforensicsJensen-Shannondivergencebehavioralbiometrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Clients of LLM APIs usually cannot check that the model answering is the one that was advertised; providers can quietly substitute cheaper or quantized variants. This paper shows that the empirical distribution of answers to everyday one-word questions—“name a random number between 1 and 100,” “name a favorite color,” and similar prompts in four languages—is already a stable, model-specific behavioral fingerprint. Split halves of the same model sit an order of magnitude closer in Jensen–Shannon distance than different models, lineage is recovered well above chance, and a biometric-style verification protocol reaches a 7.3% equal-error rate with a 40-cell battery (or under 11% with eight cells). The same measurements surface real ecosystem anomalies, including a proprietary flagship endpoint that is distributionally indistinguishable from an open-weight Qwen model. Because each query returns only one token and the prompts look like ordinary traffic, continuous auditing becomes cheap and hard to special-case.

What carries the argument

The single-token behavioral fingerprint: the tuple of empirical categorical distributions obtained by repeatedly querying a model on a battery of ten trivial tasks (random/favorite number, letter, word, color, animal, city, coin flip) in four languages, then comparing fingerprints by average Jensen–Shannon divergence.

What would settle it

Collect the same 40-cell battery on a claimed model X and on a deliberately substituted cheaper model Y under realistic serving conditions; if their fingerprints cannot be reliably separated (EER near 50% or within-model and between-model JSDs overlapping), the central claim fails.

Watch

Extended reading notes

Core claim

Empirical distributions of single-token answers to a fixed battery of trivial one-word prompts form stable, model-specific behavioral fingerprints: within-model split-half Jensen–Shannon divergence is an order of magnitude smaller than between-model divergence, enabling both family assignment (59.5% leave-one-out 1-NN accuracy versus 18.4% chance) and biometric-style verification at 7.3% equal-error rate with the full battery.

Load-bearing premise

That an adaptive provider cannot cheaply filter or special-case the probes, because the probe space is an open-ended family of everyday paraphrases sampled at audit time rather than a fixed magic string.

Editorial extensions

If this is right

  • A client can verify a claimed model identity with roughly a hundred single-token queries at a cost of fractions of a cent for typical models.
  • Distance between fingerprints recovers documented model lineage and can flag unlabeled or overwritten ancestry.
  • The same protocol surfaces deployment anomalies (cross-provider divergence beyond the impostor range) without needing logits or long generations.
  • Continuous re-auditing of opaque aggregator endpoints becomes economically trivial and can be interleaved with ordinary traffic.
  • Providers that merely swap models or heavy quantizations become detectable unless they reproduce the claimed model’s full answer distributions on everyday prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fingerprint is largely invariant to serving-stack details yet sensitive to weight changes, it could serve as a lightweight canary for silent model updates or quantization drift over time.
  • Shared cultural priors that appear across languages in the fingerprint may also explain homogenized behavior among independently trained models, linking identity leakage to coordination risk.
  • Extending the method to post-reasoning answer channels (under a relaxed token budget) would cover the growing class of mandatory-thinking endpoints currently excluded.
  • A public, regularly refreshed fingerprint registry of first-party vendor endpoints would turn the protocol into a standard integrity check for any third-party reseller.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper defines a black-box behavioral fingerprint of an LLM as the empirical distribution of single-token answers to a fixed battery of trivial one-word prompts (10 tasks × 4 languages), collected at temperature 1.0 with a hard completion cap. On a pre-registered census of 165 models served via OpenRouter (~326k responses, cost under $35), it shows that these distributions are highly non-uniform and model-specific (median split-half cell JSD 0.075 vs 0.489 between models), that mean JSD recovers documented family lineage at 59.5% LOO 1-NN accuracy against an 18.4% chance baseline, and that a biometric-style verification protocol achieves 7.3% EER with the full 40-cell battery (AUC 0.971) and 10.6% EER with eight cells. Cross-provider natural experiments and ecosystem anomalies (including a proprietary flagship endpoint distributionally indistinguishable from an open-weight Qwen model) are reported; prompts, raw responses with serving metadata, and analysis code are released.

Significance. If the measurements hold, the work supplies a practical, near-zero-cost forensic instrument for the model-substitution problem that has been documented in commercial LLM APIs. Strengths that raise the contribution above a pure measurement note include: pre-registration of hypotheses and analysis plan, a large multi-family census with full raw-data release, an explicit reliability–cost curve (EER vs number of probe cells), and a threat model that distinguishes oblivious, filtering, and emulating adversaries. The protocol needs neither logits nor long generations, making it usable against text-only production endpoints. The main operational caveat (paraphrase robustness under a filtering adversary) is acknowledged and deferred rather than over-claimed.

major comments (2)
  1. Sec. III and Sec. VII-B: the claim that a T2 (filtering) adversary cannot cheaply special-case the probes rests on the assertion that the probe space is an open-ended paraphrase family sampled at audit time. The paper explicitly leaves a dedicated paraphrase-invariance experiment to future work. Because the verification protocol is offered for operational use, at least a modest empirical check (e.g., synonym/rewording variants of a subset of cells, or cross-language transfer already partially present) is needed to bound how much EER degrades under rewording; without it the T2 claim remains by-construction rather than measured.
  2. Sec. VI-A / RQ1 and Sec. VII-C: temperature-0 determinism drops from 90.4% within-provider to 84.5% when pooled across providers, and 10 of 34 same-model provider pairs exceed the 5th-percentile impostor distance. The paper correctly flags these as anomaly candidates, yet the main verification numbers (AUC 0.971, EER 7.3%) are computed on split halves that largely share serving context. A clearer separation of same-provider vs cross-provider genuine trials in the primary ROC/EER tables would let readers judge how much of the reported reliability survives the serving-stack variance that real audits will encounter.
minor comments (4)
  1. Table II: the heterogeneous “other” family (n=17) has precision/recall 0.20/0.06 and dilutes the flat-cut ARI; a short note on how labels were curated and whether re-labeling singletons changes LOO accuracy would help.
  2. Fig. 2 dendrogram leaf labels are dense; the electronic zoomable version is mentioned, but a supplementary large-format or interactive rendering should be explicitly linked in the caption for print readers.
  3. Sec. IV-B: validity is high overall (97.6%), yet the minimum per-model rate is 60.6%. A one-sentence statement of whether low-validity models were retained in the EER/LOO analyses (they appear to be) would remove ambiguity.
  4. The token-cap change from the pre-registered 12 to 16 is transparently documented; a brief confirmation that no measured cell used more than one answer token would close the loop for readers who check the pre-registration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical measurement study with held-out splits, external family labels, and pre-registered analyses.

full rationale

The paper defines a behavioral fingerprint as the empirical categorical distribution of single-token answers to a fixed battery of trivial prompts (Sec. IV-A), then measures model-specificity via split-half Jensen–Shannon divergence, recovers lineage via LOO 1-NN against externally documented family labels, and evaluates verification as a biometric ROC/EER on genuine (same-model split halves) versus impostor trials (Sec. VI). None of these quantities is obtained by fitting a free parameter to a subset of the same data and then “predicting” a statistically forced related quantity; the distance, clustering, and error-rate figures are direct computations on held-out partitions of the released census. Family labels are taken from developer documentation, not inferred from the fingerprints themselves. There are no load-bearing self-citations of uniqueness theorems or ansatzes, no self-definitional equations, and no renaming of a known closed-form result. Pre-registration and pilot exclusion further separate design choices from confirmatory statistics. The only unmeasured claim (paraphrase robustness under a T2 adversary) is explicitly deferred to future work and does not enter the reported EER or accuracy numbers. The derivation chain is therefore self-contained empirical measurement, not circular.

Assumptions & free parameters 4 free parameters · 6 assumptions · 1 invented entities

Empirical security measurement: almost no free physical constants. Load-bearing content is operational definitions (fingerprint = tuple of empirical cell distributions; distance = mean JSD), design choices for the probe battery and sample sizes, and domain assumptions about black-box API access and adversary tiers. No new particles or forces; the “behavioral fingerprint” is a defined statistic, not an unobserved entity requiring independent evidence beyond the measurements themselves.

free parameters (4)
  • probe battery composition (10 tasks × 4 languages)
    Hand-chosen task set and languages; results (EER, lineage accuracy) depend on this design. Pilot fixed the battery before the main run, but the set is still a free design parameter of the method.
  • samples per cell (n=30 / n=15 for frontier; split-half evaluation)
    Sample size and temperature (T=1.0 primary, T=0 secondary) are chosen by the experimenters and affect histogram stability and reported EER.
  • verification threshold τ and cell-subset size k
    Operating point on the ROC and the query-budget curve; EER is reported across τ and random k-subsets, so not a single hidden fit, but still free operational parameters.
  • minimum valid samples per cell (≥10) and validity/normalization rules
    Filtering and canonicalization choices that define which responses enter ˆp; documented but analyst-chosen.
assumptions (6)
  • domain assumption Jensen–Shannon divergence (base 2), averaged over cells with ≥10 valid samples, is an appropriate symmetric distance on sparse categorical answer distributions.
    Sec. IV-C; standard choice but not uniquely forced; alternative distances could reorder neighbors.
  • domain assumption A client can obtain a trusted reference fingerprint of claimed model X (e.g., vendor first-party API) for enrollment.
    Threat model Sec. III and verification protocol Sec. IV-D; without a trusted reference, verification reduces to relative clustering only.
  • domain assumption Direct single-pass one-token completions (reasoning disabled, short token cap) sample the model’s conditional answer distribution in a way comparable across endpoints.
    Sec. IV-A; motivates exclusion of mandatory/hidden-reasoning endpoints so fingerprints are not mixed generative regimes.
  • ad hoc to paper T2 adversaries cannot cheaply special-case an open-ended paraphrase family of everyday questions sampled at audit time; T3 matching of conditional distributions collapses to running X.
    Sec. III and VII-B; central to claimed robustness, with paraphrase invariance left untested experimentally.
  • domain assumption Documented base-model lineage labels from public developer documentation are accurate enough for family classification evaluation.
    Sec. IV-C, VII-D; label noise would depress, not inflate, reported accuracy.
  • standard math Standard categorical entropy/JSD and biometric ROC/EER methodology apply to these histograms.
    Used throughout Sec. IV–VI without novel probabilistic claims.
invented entities (1)
  • behavioral fingerprint F(M) as the tuple of empirical single-token answer distributions over the probe battery independent evidence
    purpose: Operational object used for distance, lineage recovery, and verification scoring.
    Defined in Sec. IV-A from measurements; not a latent physical entity. independent_evidence is effectively the released census itself (falsifiable by re-querying endpoints).

how reviews work

0 comments
Cite this review

Pith. "Pith review of One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions." pith.science (2026). https://pith.science/paper/R457MV2G

@misc{pith2026260710252,
  author       = {Pith},
  title        = {Pith review of: One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R457MV2G}},
  note         = {Machine review of arXiv:2607.10252}
}
read the original abstract

Large language models (LLMs) are increasingly consumed through opaque serving chains - API aggregators, resellers, and inference providers - in which the client has no technical means to confirm that the model answering is the model advertised, and recent audits show that a substantial fraction of commercial endpoints deviate from the vendor's reference weights. Existing identification techniques require long generated texts, token-level log-probabilities, adversarially crafted prompts, or the model owner's cooperation. We show that far weaker evidence suffices. We define a behavioral fingerprint of an LLM as the empirical distribution of its answers to trivial one-word prompts - "name a random number between 1 and 100" - collected across four languages at a cost of one output token per query. Measuring 165 models served via a large commercial aggregator (OpenRouter), we find that (i) these distributions are highly non-uniform (median cell entropy 1.0 bit) and model-specific: split halves of the same model's samples lie an order of magnitude closer than samples of different models; (ii) Jensen-Shannon divergence between fingerprints recovers model lineage, assigning a model to its documented family with 59.5% leave-one-out accuracy against an 18.4% chance rate; and (iii) a biometric-style verification protocol achieves a 7.3% equal error rate with the full 40-cell battery, and below 11% with eight probe cells - roughly a hundred single-token queries per audit. We further report ecosystem anomalies, including a proprietary-branded flagship endpoint distributionally indistinguishable from an open-weight Qwen model. The protocol, prompts, raw data, and analysis code are released for reproduction and operational use.

Figures

Figures reproduced from arXiv: 2607.10252 by the authors.

Figure 1
Figure 1. The raw fingerprint signal: answer distributions for [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Hierarchical clustering (UPGMA) of 165 served models on mean Jensen–Shannon divergence between single-token fingerprints; leaf labels are colored [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Verification ROC: split-half fingerprint distance separates genuine [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Reliability–cost trade-off: EER as a function of the number of probe [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Random-generation probes plus a pilot-fitted budget let a text-only auditor detect model substitution, estimate the routing dilution fraction, and attribute the served backend across LLM gateways.

Reference graph

Works this paper leans on

27 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    Model equality testing: Which model is this API serving?

    I. Gao, P. Liang, and C. Guestrin, “Model equality testing: Which model is this API serving?” inThe Thirteenth International Conference on Learning Representations (ICLR), 2025. [Online]. Available: https://openreview.net/forum?id=QCDdI7X3f9

    2025

  2. [2]

    Are you getting what you pay for? Auditing model substitution in LLM APIs,

    W. Cai, T. Shi, X. Zhao, and D. Song, “Are you getting what you pay for? Auditing model substitution in LLM APIs,” arXiv preprint arXiv:2504.04715, 2025. [Online]. Available: https: //arxiv.org/abs/2504.04715

    arXiv 2025

  3. [3]

    Auditing black-box LLM APIs with a rank-based uniformity test,

    X. Zhu, Y . Ye, T. Qiu, H. Zhu, S. Tan, A. Mannan, J. Michala, R. A. Popa, and W. Neiswanger, “Auditing black-box LLM APIs with a rank-based uniformity test,” arXiv preprint arXiv:2506.06975, 2025. [Online]. Available: https://arxiv.org/abs/2506.06975

    arXiv 2025

  4. [4]

    A watermark for large language models,

    J. Kirchenbauer, J. Geiping, Y . Wen, J. Katz, I. Miers, and T. Goldstein, “A watermark for large language models,” inProceedings of the 40th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 17 061–17 084. [Online]. Available: https://proceedings.mlr.press/ v202/kirchenbauer23a.html 9

    2023

  5. [5]

    Instruc- tional fingerprinting of large language models,

    J. Xu, F. Wang, M. D. Ma, P. W. Koh, C. Xiao, and M. Chen, “Instruc- tional fingerprinting of large language models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics, 2024,...

    2024

  6. [6]

    FIT-Print: Toward false-claim-resistant model ownership verification via targeted fingerprint,

    S. Shao, H. Zhu, Y . Li, H. Yao, T. Zhang, and Z. Qin, “FIT-Print: Toward false-claim-resistant model ownership verification via targeted fingerprint,”IEEE Transactions on Information Forensics and Security, vol. 21, pp. 5612–5626, 2026

    2026

  7. [7]

    Authorship attribution for neu- ral text generation,

    A. Uchendu, T. Le, K. Shu, and D. Lee, “Authorship attribution for neu- ral text generation,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020, pp. 8384–8395

    2020

  8. [8]

    Idiosyncrasies in large language models,

    M. Sun, Y . Yin, Z. Xu, J. Z. Kolter, and Z. Liu, “Idiosyncrasies in large language models,” inProceedings of the 42nd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 57 854–57 885. [Online]. Available: https://proceedings.mlr.press/v267/sun25z.html

    2025

Show all 27 references
  1. [9]

    LLMmap: Fingerprinting for large language models,

    D. Pasquini, E. M. Kornaropoulos, and G. Ateniese, “LLMmap: Fingerprinting for large language models,” inProceedings of the 34th USENIX Security Symposium (USENIX Security 25). Seattle, W A, USA: USENIX Association, 2025. [Online]. Available: https: //www.usenix.org/conference...

    2025

  2. [10]

    TRAP: Targeted random adversarial prompt honeypot for black-box identification,

    M. Gubri, D. Ulmer, H. Lee, S. Yun, and S. J. Oh, “TRAP: Targeted random adversarial prompt honeypot for black-box identification,” in Findings of the Association for Computational Linguistics: ACL 2024. Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 1...

    2024

  3. [11]

    Can LLMs generate random numbers? Evaluating LLM sampling in controlled domains,

    A. Renda, A. Hopkins, and M. Carbin, “Can LLMs generate random numbers? Evaluating LLM sampling in controlled domains,” in ICML 2023 Workshop: Sampling and Optimization in Discrete Space (SODS), 2023. [Online]. Available: https://openreview.net/forum?id= Vhh1K9LjVI

    2023

  4. [12]

    How random is random? Evaluating the randomness and humaness of LLMs’ coin flips,

    K. Van Koevering and J. Kleinberg, “How random is random? Evaluating the randomness and humaness of LLMs’ coin flips,” arXiv preprint arXiv:2406.00092, 2024. [Online]. Available: https: //arxiv.org/abs/2406.00092

    2024 arXiv

  5. [13]

    A comparison of large language model and human performance on random number generation tasks,

    R. M. Harrison, “A comparison of large language model and human performance on random number generation tasks,” arXiv preprint arXiv:2408.09656, 2024. [Online]. Available: https://arxiv.org/abs/2408. 09656

    2024 arXiv

  6. [14]

    Deterministic or probabilistic? The psychology of LLMs as random number generators,

    J. Coronado-Bl ´azquez, “Deterministic or probabilistic? The psychology of LLMs as random number generators,” arXiv preprint arXiv:2502.19965, 2025. [Online]. Available: https: //arxiv.org/abs/2502.19965

    2025 arXiv

  7. [15]

    BOSC: A backdoor-based frame- work for open set synthetic image attribution,

    J. Wang, B. Tondi, and M. Barni, “BOSC: A backdoor-based frame- work for open set synthetic image attribution,”IEEE Transactions on Information Forensics and Security, vol. 20, pp. 8043–8058, 2025

    2025

  8. [16]

    AdaParse: Personalized fingerprinting for visual generative model reverse engineering,

    Y . Zheng, Z. Li, B. Yu, J. Zhou, and J. Lu, “AdaParse: Personalized fingerprinting for visual generative model reverse engineering,”IEEE Transactions on Information Forensics and Security, vol. 21, pp. 2682– 2697, 2026

    2026

  9. [17]

    A soft-contrastive pseudo learning approach toward open-world forged speech attribution,

    Q. Zhang, X. Zhang, M. Sun, and J. Yang, “A soft-contrastive pseudo learning approach toward open-world forged speech attribution,”IEEE Transactions on Information Forensics and Security, vol. 20, pp. 1135– 1148, 2025

    2025

  10. [18]

    BDMMT: Backdoor sample detection for language models through model mutation testing,

    J. Wei, M. Fan, W. Jiao, W. Jin, and T. Liu, “BDMMT: Backdoor sample detection for language models through model mutation testing,”IEEE Transactions on Information Forensics and Security, vol. 19, pp. 4285– 4300, 2024

    2024

  11. [19]

    Playing repeated games with large language models,

    E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz, “Playing repeated games with large language models,”Nature Human Behaviour, vol. 9, pp. 1380–1390, 2025

    2025

  12. [20]

    Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?

    R. Bommasani, K. A. Creel, A. Kumar, D. Jurafsky, and P. Liang, “Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?” inAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. [Online]. Available: https://proceedings.neur...

    2022

  13. [21]

    Algorithmic monoculture and social welfare,

    J. Kleinberg and M. Raghavan, “Algorithmic monoculture and social welfare,”Proceedings of the National Academy of Sciences, vol. 118, no. 22, p. e2018340118, 2021

    2021

  14. [22]

    T. C. Schelling,The Strategy of Conflict. Cambridge, MA: Harvard University Press, 1960

    1960

  15. [23]

    The nature of salience: An experimental investigation of pure coordination games,

    J. Mehta, C. Starmer, and R. Sugden, “The nature of salience: An experimental investigation of pure coordination games,”American Economic Review, vol. 84, no. 3, pp. 658–673, 1994. [Online]. Available: https://www.jstor.org/stable/2118074

    1994

  16. [24]

    Single-token output distributions as behavioral fingerprints of large language models,

    T. Bruckner, “Single-token output distributions as behavioral fingerprints of large language models,” Zenodo dataset, Jul. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.21278557

    2026 doi

  17. [25]

    Single-token output distributions as behavioral fingerprints of large language models — software,

    T. Bruckner, “Single-token output distributions as behavioral fingerprints of large language models — software,” Zenodo software archive, Jul

  18. [26]

    Available: https://doi.org/10.5281/zenodo.21278793

    [Online]. Available: https://doi.org/10.5281/zenodo.21278793

    doi

  19. [27]

    Single-token output distributions as behavioral fingerprints of large language models: model verification and tacit coordination

    T. Bruckner, “Single-token output distributions as behavioral fingerprints of large language models: model verification and tacit coordination.” OSF pre-registration, Jul. 2026. [Online]. Available: https://doi.org/10. 17605/OSF.IO/KXAHM

    2026

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.