Pith. sign in

REVIEW 3 major objections 4 minor 59 references

Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A text-only audit can detect when an LLM gateway serves a cheaper model, name the substitute, and estimate the diluted fraction before spending a query budget it sizes itself.

desk verdict IRIS nails the text-only dilution-detection task on well-separated pairs, but the abstract oversells the self-sizing budget and the pooled false-positive rate; the authors themselves disclose the gaps in the appendix. read the letter →

arxiv 2607.20860 v1 pith:L6ICYQUS submitted 2026-07-23 cs.CR

Yuewei Zhang , Zhi-Hai Zhang , Hanzhang Qin This is my paper · ORCID
classification cs.CR
keywords LLMgatewayauditingmodelsubstitutionroutingdilutionblack-boxfingerprintingrandom-generationprobesquerybudgetestimationvisible-stringfeatures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces IRIS, an audit that uses only the returned text of random-generation probes to check whether an LLM gateway is truly serving the advertised model. It claims to be the first method that combines, in one text-only audit, detection of whole-stream substitution and fractional dilution, attribution of the served backend, estimation of the routing fraction, and a query budget it sizes itself from a cheap pilot. If correct, a client could verify model identity and quantify dilution without privileged signals like log-probabilities or token ranks, and without fixing the query count in advance. The central validation is that on a commercial OpenRouter library, IRIS catches epsilon=0.3 dilution on margin-qualified pairs at 0.85 mean power with a 0.017 false-positive rate, and recovers the dilution rate to within 0.04 for enrolled diluents.

What carries the argument

The central object is the random-generation probe family c_{n,L} (e.g., c_{2,16} for 16 random bits, c_{10,8} for 8 random digits) paired with a 179-dimensional visible-string feature vector and a multiclass classifier's negative log-posterior score. The load-bearing mechanism is the 'estimate-then-budget' pilot: a cheap labeled pilot (m <= 4 queries) fits the exponential rank-error decay log(1-AUROC) ~ const - I_auc * m, and the fitted exponent (with a lower-confidence bound) is used to freeze the live-query budget and select the probe before any suspect traffic is queried. For dilution, response-level 'tell' rates (reference-atypical responses) are calibrated on a separate split, and the t

What would settle it

A concrete observation that would settle the central claim: take a fresh honest endpoint of the same model not used in calibration, run the frozen IRIS plan (same probe, same budget, same thresholds), and measure the realized false-positive rate across many independent audit runs. If the per-pair FPR exceeds the claimed 0.017 (or the pre-registered alpha) on a material fraction of pairs, or if the realized power at the pre-committed budget falls below the target on a substantial minority of margin-qualified pairs, then the calibration-based budget guarantee fails. The paper itself reports such

Watch

Extended reading notes

Core claim

The paper's central claim is that visible-string biases in responses to random-generation prompts—such as asking for digits, bits, or short sequences—are backend-specific fingerprints that can be used to audit LLM gateways. IRIS turns each response into a 179-dimensional feature vector, trains a multiclass classifier on enrolled endpoints, and scores responses by the reference negative log-posterior. It then calibrates telling thresholds and, critically, estimates the query budget before any suspect query is issued: a pilot fits the exponential decay of rank error (1 - AUROC) and the dilution tell-rate regime, yielding a pre-committed budget. The paper argues that evidence accumulates across

Load-bearing premise

The honest reference calibration window is representative and large enough that the thresholds and budgets frozen on it control type-I error at the stated level and meet the power target on a new audit stream.

Editorial extensions

If this is right

  • A client can audit an LLM gateway unilaterally from returned text alone, with no privileged access, and with a budget that is estimated before the audit begins.
  • The audit can not only decide whether dilution is happening but also name the substitute and estimate the routing fraction, making the result actionable for billing disputes or price renegotiation.
  • The measured 1/epsilon budget law means that even low-rate dilution (e.g., epsilon=0.1) is detectable within a realistic number of queries when a separating tail exists, and the paper's tail-exponent measurement can decide which pairs fall into the cheap 1/epsilon regime versus the expensive 1/epsilon^2 wall.
  • Because the probes are reusable and task-agnostic, the same audit pipeline transfers across endpoints without target-specific probe design, unlike prior knowledge-boundary or border-input audits.
  • A real cross-provider audit of the same nominal model served by different providers found 14 of 15 pairs distinguishable, indicating that the method can flag genuine quantization/kernel deviations in the wild.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If IRIS's calibration guarantees hold in deployment, a practical consequence is that gateway providers can be held accountable for silent downgrades: a customer could routinely audit a fraction of their traffic and obtain evidence that stands up enough to contest billing, even without provider cooperation.
  • The paper's distinction between a pure temperature retune (second-order, near-invisible) and a base-model swap (first-order, detectable) suggests that the audit's 'blind spot' is intentionally aligned with economic incentives: tempering alone does not create extra gateway profit under token-count billing, so the audit focuses on the substitution threat that matters financially.
  • A testable extension is to turn the audit into an anytime-valid sequential test using the same tell-count machinery; the paper's own appendix compares a betting e-process and SPRT, so a deployment could stop early on clearly honest or clearly diluted streams while preserving the pre-committed budget contract.
  • The measured tail exponent kappa is reported as a pair-specific property; if a gateway adaptively switches between multiple diluents or uses sticky routing, the i.i.d. tell-count interval degrades, and the paper's block-bootstrap repair suggests that per-session or longer audit horizons would be needed to maintain calibrated uncertainty on the routing fraction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents IRIS, an output-only (visible-string) black-box auditor for LLM gateways. It uses random-generation probes, a 179-dimension visible-string feature vector, and a random-forest posterior to detect whole-stream substitution and fractional dilution, attribute the served backend, estimate the routing fraction ε, and pre-commit a query budget estimated from a cheap pilot. The theoretical core (Section 5, App. S28) proves exponential decay of mean-score verification error, a response-level tell-rate characterization of the dilution budget, and a Θ(ε^{-2}) all-test lower bound under a χ²-regularity condition, with an interpolating tail-exponent law (Prop. 6). Experiments on a local Qwen3 ladder, a 17- and 45-endpoint OpenRouter library, and a live cross-provider audit report strong detection, attribution, and ε-recovery numbers, supported by a released code/data bundle. The appendices are unusually candid about finite-sample limitations, including per-pair FPR tails, under-coverage of the δ-method interval, and the fact that the prescribed budget meets the power target on only a minority of pairs at the calibrated sample size.

Significance. If the claims hold, IRIS is a substantial advance: a text-only auditor that detects dilution, names the diluent, estimates ε, and predicts its own query cost before issuing suspect queries would close a real gap in gateway auditing. The paper's strengths are concrete: machine-checkable-style complete proofs of the main propositions in App. S28; a large frozen response dataset and code release; head-to-head comparisons against FLIPS, MET, and RUT on a shared probe; and a clear identification of the 1/ε versus 1/ε² regimes as a measured tail-exponent phenomenon rather than an unconditional law. The paper also deserves credit for explicitly labeling its own reliability limits in App. S5 (Table 3: the prescribed budget is a 'feasibility diagnostic, not a coverage guarantee') and for reporting FPR tails and held-out margin-selection checks rather than only pooled means. However, the central abstract claims—'a query budget it sizes itself' and a '0.017 false-positive rate'—are stronger than what the paper's own finite-calibration results support, and this is load-bearing for the paper's main novelty.

major comments (3)
  1. [Abstract; §1 Contributions; App. S5] The central reliability claims are not supported as stated. App. S5's end-to-end check at (α=0.05, δ=0.1, ε=0.3) shows realized power averaging 0.73 but meeting the pre-registered 0.9 target on only 17.8% of feasible pairs (12.6% at ε=0.2), and per-pair type-I FPR with mean 0.013 but maximum 0.475 before hardening and 0.39 after exact-binomial hardening. Table 3 of the appendix explicitly labels the prescribed budget a 'feasibility diagnostic, not a coverage guarantee.' The abstract's '0.017 false-positive rate' is the pooled mean, not a per-audit guarantee, and 'a query budget it sizes itself' is not a reliability contract. Revision must either provide a calibration-window size/composition rule that restores the (α, δ) guarantee at the stated level, or re-scope the claim to 'feasibility diagnostic' and report the per-pair FPR distribution rather than the pooled mean.
  2. [§4.1 Eq. (7), Eq. (12); App. S4] The estimate-then-budget pilot is not calibration-robust. App. S4 shows that a 1000-sample reference calibration overshoots the realized total FPR to 0.09–0.14 on c2,16 (≤0.04 on c10,8), and the certifiable budget is only m≤16 at N=1000 and m≤50 at N=3000 for α=0.05. Since the deployed pipeline calibrates on roughly 33 held-out responses per pair (App. S5), the m⋆ produced by Eqs. (7) and (12) does not in general meet the stated (α, δ) targets. The paper should state the honest-window size required to certify a given m⋆ and qualify the 1/ε law as finite-ε and calibration-bounded, exactly as App. S4 already does.
  3. [§6.2; App. S3] The headline detection figure is conditioned on margin-qualified pairs in a way not fully transparent in the body. App. S3 reports that over the 218 margin-qualified pairs the per-pair honest FPR has median 0.000, 95th percentile 0.090, maximum 0.557 (0.39 after exact-binomial hardening), with 7.8% of pairs above 0.05; the body reports only the pooled mean 0.017. The held-out margin-selection power drops to 0.66 vs. 0.85 in-sample; the matched control attributes this to bank size, but it still shows sensitivity to the calibration split. The body should present the FPR distribution, the overall reliable-detection rate (≈0.51 over all 272 ordered pairs), and the caveat that a substantial minority of pairs are returned as indeterminate/low-margin rather than flagged.
minor comments (4)
  1. [App. S15, S26, S25] Several works cited in the appendices are missing from the reference list: Bruckner (2026), Richter et al. (2025), Shekhar and Ramdas (2023), Dima et al. (2025), Huber (1964), Donoho and Jin (2004), and Ong et al. (2025). Please complete the bibliography.
  2. [§1, App. S15] The phrase 'first to combine' should be slightly moderated in the main text. The paper itself identifies Bruckner (2026) as concurrent single-token fingerprinting and FLIPS as sharing the random-generation signal; the defensible novelty is the specific combination (reusable probe, budget estimation, dilution ε-estimation, attribution) rather than the underlying signal.
  3. [App. S5, Table 3] Table 3's 'which statistic backs which claim, and at what guarantee level' is an excellent self-assessment. Consider moving a condensed version of it into the main text so that the abstract's guarantees are immediately qualified.
  4. [App. S2] The δ-method interval for ε is mildly anti-conservative (empirical coverage 0.86–0.87 at nominal 90%, 0.92–0.93 at 95%). The paper describes this accurately as 'slightly optimistic,' but the main text should at least footnote this so readers do not treat the 90% interval as exact.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: IRIS's detection, attribution, epsilon-estimation, and budget-size claims are validated on held-out and external data, with the paper's own calibration caveats disclosed as limitations rather than hidden reductions.

full rationale

The paper's central derivation is self-contained. Thresholds and budgets are frozen during Audit-Plan Construction on labeled reference/candidate data, then applied to a disjoint suspect stream; the headline power/FPR numbers are measured on held-out diluted or honest streams, not computed from the same fit that set the thresholds. The epsilon estimator Eq. (14) is a method-of-moments inversion of the mixture model Q_epsilon, and its accuracy is evaluated on controlled mixtures with disjoint enrollment and audit splits. The 1/epsilon dilution budget is derived from a Chernoff bound (Thm. 1) and then empirically measured, with the paper explicitly citing Huber contamination and higher criticism as the classical lineage rather than claiming a new phenomenon. The pilot exponent bIauc is honestly labeled a 'budgeting fit, not an information-theoretic lower bound' (Def. 1), and its predictive accuracy is checked on held-out points. The only possible self-citation (Zhang et al. 2026) is motivational in the introduction and is not load-bearing for any technical claim. The calibration-robustness concerns in App. S4/S5 — power met on 17.8% of feasible pairs at the prescribed budget, FPR tail up to 0.39 post-hardening, and the 1000-sample calibration overshoot — are explicitly disclosed and are correctness/scope limitations, not circularity; Table 3 even labels the prescribed budget a 'feasibility diagnostic, not a coverage guarantee.' No equation in the paper reduces to its own input by construction.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The audit rests on ~5 hand-chosen or fitted quantities (pilot exponent, tail exponent, margin threshold, threshold grid, anchor level) plus three domain assumptions (stable fingerprint, trusted reference, i.i.d. routing). No invented entities in the graviton sense: IRIS introduces statistical constructs (the visible-string score s_c(y), tell rates, the pilot exponent) but postulates no new physical or system entity, and every construct is computable from observable returned text. The fitted exponents are treated honestly by the paper as measured quantities — which is exactly why the 1/ε law is presented as behavior, not a law (App. S26).

free parameters (5)
  • pilot rank-error exponent bIauc = probe-dependent; e.g. near 0 for c2,1 to >1 for c2,16 (Fig. 2a, Fig. 12)
    OLS slope of log(1−AUROC(m)) on m≤4 pilot data; enters the substitution budget directly (Eq. 7). The paper states it is a budgeting fit not guaranteed to be below the information-theoretic rate Isc, and can exceed it (App. S24).
  • tail exponent κ = 0.00–0.97 depending on pair/probe (Table 22; distribution in Table 23)
    Measured log-log slope of the tell rate q_α vs tail level α. Decides whether the audit operates in the near-1/ε regime or the 1/ε² wall (Prop. 6). The paper treats it as a measured finite-sample property, not an asymptotic constant.
  • tell-rate margin threshold = q − p₀ ≥ 0.15
    Hand-chosen eligibility cutoff selecting the 218/272 'margin-qualified' pairs on which the headline dilution power (0.85) and FPR (0.017) are computed (App. S3).
  • honest-tail grid A_resp = as implemented: a₀ ∈ {0.02, 0.05, 0.1, 0.15} (App. S5)
    Discrete set of reference quantiles searched for the tell threshold τ. The winner's-curse minimization over this grid is identified in App. S5 as a source of the type-I tail.
  • null-tell anchor level α₀ = 0.1
    Fixed reference quantile defining τ in the ε-estimation protocol 'so there is no max-gap snooping' (App. S2).
assumptions (6)
  • domain assumption Random-generation outputs expose backend-specific sampling biases that are stable and discriminative across endpoints.
    The entire fingerprint signal rests on this premise, grounded in prior work (Hopkins et al. 2023; Zhao et al. 2026; FLIPS) and validated empirically in §6, but it is not proven; GhostPrint-style spoofing (Zhang, Li, Wang 2026) is cited as a known break.
  • domain assumption A trusted reference endpoint P serves the advertised model faithfully and can be sampled under known labels during Audit-Plan Construction.
    Section 3 model. Detection compares the suspect stream to P; if the 'trusted reference' is itself mislabeled, the audit decides between two wrong hypotheses. App. S20 flags this trust assumption; App. S3 relies on a fresh honest window of P.
  • domain assumption Independent API calls to the suspect endpoint are i.i.d. draws from Q_ε (no sticky or content-adaptive routing).
    Eq. (1)/(16) and all budget theorems assume i.i.d. routing. App. S13 shows sticky routing does not inflate type-I but collapses CI coverage (0.92→0.35), and probe-aware routing is left as an open threat (App. S12).
  • ad hoc to paper The (reference, substitute) pair is measurably separated at a usable threshold (q > α₁ with margin).
    Low-margin pairs are excluded as indeterminate (54/272 pairs at ε=0.3); the audit's guarantees apply only to margin-qualified pairs (App. S3, Table 1).
  • standard math Standard large-deviations machinery: Chernoff information theorem, Cramér, CLT with Lindeberg, Hölder, and the TV ≤ √(1−A²) bound.
    Invoked in Prop. 1, Thm. 1, Prop. 6 proofs (App. S28) with stated regularity conditions (bounded score, χ²<∞, atomless score CDF in Prop. 6).
  • ad hoc to paper A pure decoding-temperature retune is second-order and out of audit scope (Prop. 5).
    Derived from the exponential-family Fisher quadratic, but the scope exclusion is an audit choice: a gateway profiting from temperature retunes alone (no model swap) is not an auditing target (§3, App. S25).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways." pith.science (2026). https://pith.science/paper/L6ICYQUS

@misc{pith2026260720860,
  author       = {Pith},
  title        = {Pith review of: Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6ICYQUS}},
  note         = {Machine review of arXiv:2607.20860}
}
abstract

Commercial LLM gateways mediate access to hosted models, but the served backend may not match the advertised one: it may substitute a cheaper model on every request or route only a fraction $\epsilon$ of requests to it. Prior black-box auditors often need a privileged signal (log-probabilities, token ranks, or reference samples) or a target-specific probe, fix the query budget in advance, and return a yes/no verdict. We present $\mathrm{IRIS}$, an audit that needs only the returned text: it asks endpoints to generate random numbers or strings, fingerprints the backend, and is the first to combine, in one text-only audit, detection of whole-stream substitution and fractional dilution, attribution of the served backend, routing-fraction ($\epsilon$) estimation, and a query budget it sizes itself. A cheap pilot fits the exponential query-error decay and freezes that budget before any suspect query is issued. On an intra-family Qwen3 ladder $\mathrm{IRIS}$ verifies the backend at $0.99$ AUROC and sharpens attribution as queries accumulate; across a commercial OpenRouter library it catches $\epsilon{=}0.3$ dilution on margin-qualified pairs at $0.85$ mean power ($0.017$ false-positive rate) and recovers $\epsilon$ to within $0.04$ for enrolled diluents; and a live cross-provider audit flags $14$ of $15$ same-model provider pairs by genuine quantization and kernel deviations, corroborated on third-party MET traces. Against comparable black-box auditors, $\mathrm{IRIS}$ matches or beats detection on shared tasks, and adaptive allocation lifts the matched-budget target-hit rate from $73$% to $87$%. Further experiments cover adversarial gateways, knob identifiability, unseen diluents, and false-positive control.

Figures

Figures reproduced from arXiv: 2607.20860 by the authors.

Figure 1
Figure 1. IRIS audits gateway routing from visible strings only: a user task may be served by the intended model or by a weaker substitute; IRIS turns the returned strings into fingerprints, estimates the endpoint information rate, sizes the audit budget, and detects routing dilution. and fractional replacement dilution. Either may arise through misconfiguration, cost control, or misrepresentation, and ei￾ther requires an aud… view at source ↗
Figure 2
Figure 2. Experimental overview. (a) Verification error falls as e −Ibaucm: the ladder exemplar c100,1, the same decay on 17 commercial APIs (c10,8, dashed), and the entropy-capped c2,1 (flat). (b) Dilution evidence accumulates for IRIS but not for RUT (c2,16; MET and FLIPS cannot audit dilution, Ta￾ble 1). (c) Margin-qualified commercial dilutions cost near￾1/ϵ queries to detect. (d) Deployed m=80 gateway audit: known and le… view at source ↗
Figure 1
Figure 1. Pilot rank-error rate Ibauc vs. sampling temperature T for the entropy-ladder probes: Ibauc → 0 under greedy decoding, peaks near T≈1, and the onset temperature scales with output entropy (c2,1 only starts at T=2). S1 Temperature Gating Proposition 1 (Distributional separation). For a single-draw probe at temperature T with categorical laws p T P , pT R on n symbols: 1. One draw carries at most the visible-output in… view at source ↗
Figures from the paper (13 more)
Figure 2
Figure 2. Figure 2: Routing-fraction estimation on controlled mixtures [PITH_FULL_IMAGE:figures/full_fig_p010_2.png]
Figure 3
Figure 3. Figure 3: Deployed dilution audit on the real 17-endpoint gateway. (a) Detection power vs. routing fraction ϵ at m=80 under the fixed-p0 two-proportion test, pooled over pairs passing the margin check (band = 25–75% over pairs): the unknown-diluent leave-one-out curve nearly coi…
Figure 4
Figure 4. Figure 4: External validation on 17 commercial and open APIs (chance 1/17, dotted). (a) attribution accuracy and (b) reference-vs-suspect verification AUROC vs. indepen￾dent queries m; shaded bands are 95% bootstrap intervals over 40 splits. The e −Im law and entropy gating repl…
Figure 6
Figure 6. Figure 6: Scaling to 45 models via OpenRouter (chance 1/45=2.2%, dotted). (a) attribution accuracy and (b) reference-vs-suspect verification AUROC vs. indepen￾dent queries m, per probe; shaded bands are 95% bootstrap intervals over 10 splits. The e −Im law and entropy gating hol…
Figure 7
Figure 7. Figure 7: Which serving knobs leave a visible-string signa [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Method comparison on the shared random￾generation probe (c2,16), verification AUROC vs. queries m. (a) Substitution: IRIS ties a FLIPS-style classifier (AUROC ≈ 0.99 at m=1, both saturating to 1.0 by m=8): the shared random probe ties, and IRIS’s contribution is the bu…
Figure 9
Figure 9. Figure 9: Budget-from-Ibauc is predictive (K=6). Predicted verification error from an m≤4 pilot vs. realized error at held-out larger m, per (pair, probe); the dashed line is perfect prediction and the band is a 2× tolerance. Across 224 vali￾dation points the median | log-ratio|…
Figure 10
Figure 10. Figure 10: Anytime-valid sequential auditing on the separa [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: A library of visible-string fingerprints for [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Calibration-estimated Ibauc (from an m≤4 pilot) predicts realized verification AUROC at m=8, one point per probe family. The estimate orders probes by efficiency before any audit budget is spent: high-Ibauc sequence probes saturate at m=8 while c2,1 (Ibauc ≈0) lags. I…
Figure 13
Figure 13. Figure 13: Why a temperature retune is second order while [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Dilution (referenced from Section 6). (a) AUROC [PITH_FULL_IMAGE:figures/full_fig_p033_14.png]
Figure 15
Figure 15. Figure 15: The tell-rate continuum on c100,1 (ref qwen3:8b@T=1). Here q is the per-query tell rate Py∼R[s(y)>τ ] and α the per-query false-positive floor. (a) Per-query tell rate q per substitute, ascending; bars below the false-positive floor α (greedy and near-greedy twins) ar…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 7 linked inside Pith

  1. [2]

    Cai, W.; Shi, T.; Zhao, X.; and Song, D. 2025. Are You Getting What You Pay For? Auditing Model Substitution in LLM API s. arXiv preprint arXiv:2504.04715

    arXiv 2025

  2. [3]

    M.; and Thomas, J

    Cover, T. M.; and Thomas, J. A. 2006. Elements of Information Theory. Wiley-Interscience, 2nd edition. ISBN 9780471241959

    2006

  3. [5]

    Donoho, D.; and Jin, J. 2004. Higher Criticism for Detecting Sparse Heterogeneous Mixtures. The Annals of Statistics, 32(3): 962--994

    2004

  4. [6]

    Gao, I.; Liang, P.; and Guestrin, C. 2025. Model Equality Testing: Which Model Is This API Serving? In International Conference on Learning Representations (ICLR)

    2025

  5. [7]

    K.; Renda, A.; and Carbin, M

    Hopkins, A. K.; Renda, A.; and Carbin, M. 2023. Can LLM s Generate Random Numbers? Evaluating LLM Sampling in Controlled Domains. In ICML Workshop on Sampling and Optimization in Discrete Space (SODS)

    2023

  6. [8]

    Huber, P. J. 1964. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1): 73--101

    1964

  7. [9]

    Nikolic, I.; Baluta, T.; and Saxena, P. 2025. Model Provenance Testing for Large Language Models. In Advances in Neural Information Processing Systems, volume 38, 34126--34153

    2025

  8. [10]

    M.; Di Ferrante, M.; Pazdera, A.; Garner, R.; Jaghouar, S.; Basra, M.; Ryabinin, M.; and Hagemann, J

    Ong, J. M.; Di Ferrante, M.; Pazdera, A.; Garner, R.; Jaghouar, S.; Basra, M.; Ryabinin, M.; and Hagemann, J. 2025. TOPLOC : A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference. arXiv preprint arXiv:2501.16007

    arXiv 2025

Show all 59 references
  1. [11]

    M.; and Ateniese, G

    Pasquini, D.; Kornaropoulos, E. M.; and Ateniese, G. 2025. LLMmap : Fingerprinting For Large Language Models. In 34th USENIX Security Symposium (USENIX Security 25), 299--318

    2025

  2. [12]

    Richardeau, G.; Dashyan, G.; Le Merrer, E.; and Tredan, G. 2026. FLIPS : Instance-Fingerprinting for LLM s via Pseudo-random Sequences. In International Conference on Machine Learning (ICML)

    2026

  3. [13]

    Richter, L.; He, X.; Minervini, P.; and Kusner, M. J. 2025. An Auditing Test to Detect Behavioral Shift in Language Models. In International Conference on Learning Representations (ICLR)

    2025

  4. [14]

    Shekhar, S.; and Ramdas, A. 2023. Nonparametric Two-Sample Testing by Betting. IEEE Transactions on Information Theory

    2023

  5. [15]

    Z.; and Liu, Z

    Sun, M.; Yin, Y.; Xu, Z.; Kolter, J. Z.; and Liu, Z. 2025. Idiosyncrasies in Large Language Models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, 57854--57885. PMLR

    2025

  6. [16]

    Zhao, M.; Du, Y.; and Wang, M. 2026. Large Language Models Are Bad Dice Players: LLM s Struggle to Generate Random Numbers from Statistical Distributions. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 22942-...

    2026

  7. [17]

    Your ``Pro''

    Zhang, Jiahao and Li, Xiuyu and Wang, Suhang , journal=. Your ``Pro''. 2026 , url =

    2026

  8. [18]

    Token-Efficient Change Detection in

    Chauvin, Timoth\'ee and Lalanne, Cl\'ement and Le Merrer, Erwan and Loubes, Jean-Michel and Ta\"iani, Fran. Token-Efficient Change Detection in. International Conference on Machine Learning (ICML) , year =

  9. [19]

    2026 , url =

    Richardeau, Gurvan and Dashyan, Gohar and Le Merrer, Erwan and Tredan, Gilles , booktitle =. 2026 , url =

    2026

  10. [20]

    2026 , url =

    Fang, Yijia and Feng, Yiqing and Li, Bingyu and Zhou, Mingxun , journal=. 2026 , url =

    2026

  11. [21]

    Auditing Black-Box

    Zhu, Xiaoyuan and Ye, Yaowen and Qiu, Tianyi and Zhu, Hanlin and Tan, Sijun and Mannan, Ajraf and Michala, Jonathan and Popa, Raluca Ada and Neiswanger, Willie , booktitle =. Auditing Black-Box. 2026 , url =

    2026

  12. [22]

    Model Equality Testing: Which Model Is This

    Gao, Irena and Liang, Percy and Guestrin, Carlos , booktitle =. Model Equality Testing: Which Model Is This. 2025 , url =

    2025

  13. [23]

    and Ateniese, Giuseppe , booktitle =

    Pasquini, Dario and Kornaropoulos, Evgenios M. and Ateniese, Giuseppe , booktitle =. 2025 , url =

    2025

  14. [24]

    doi:10.18653/v1/2024.findings-acl.683 , year =

    Gubri, Martin and Ulmer, Dennis and Lee, Hwaran and Yun, Sangdoo and Oh, Seong Joon , booktitle =. doi:10.18653/v1/2024.findings-acl.683 , year =

    2024 doi

  15. [25]

    Advances in Neural Information Processing Systems , volume =

    Model Provenance Testing for Large Language Models , author =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

    2025

  16. [26]

    and Renda, Alex and Carbin, Michael , booktitle =

    Hopkins, Aspen K. and Renda, Alex and Carbin, Michael , booktitle =. Can. 2023 , url =

    2023

  17. [27]

    Are You Getting What You Pay For? Auditing Model Substitution in

    Cai, Will and Shi, Tianneng and Zhao, Xuandong and Song, Dawn , journal=. Are You Getting What You Pay For? Auditing Model Substitution in. 2025 , url =

    2025

  18. [28]

    The Annals of Mathematical Statistics , volume =

    A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the Sum of Observations , author =. The Annals of Mathematical Statistics , volume =. 1952 , doi =

    1952

  19. [29]

    The Annals of Mathematical Statistics , volume =

    Asymptotically Optimal Tests for Multinomial Distributions , author =. The Annals of Mathematical Statistics , volume =. 1965 , doi =

    1965

  20. [30]

    2006 , doi =

    Elements of Information Theory , author =. 2006 , doi =

    2006

  21. [31]

    Large Language Models Are Bad Dice Players:

    Zhao, Minda and Du, Yilun and Wang, Mengyu , booktitle =. Large Language Models Are Bad Dice Players:. 2026 , url =

    2026

  22. [32]

    Behavioral Consistency and Transparency Analysis on Large Language Model

    Lin, Guanjie and Wan, Yinxin and Pei, Shichao and Xu, Ting and Xu, Kuai and Xue, Guoliang , booktitle =. Behavioral Consistency and Transparency Analysis on Large Language Model. doi:10.1145/3777912.3809156 , year =

  23. [33]

    Real Money, Fake Models: Deceptive Model Claims in Shadow

    Zhang, Yage and Jiang, Yukun and Chen, Zeyuan and Backes, Michael and Shen, Xinyue and Zhang, Yang , journal=. Real Money, Fake Models: Deceptive Model Claims in Shadow. 2026 , url =

    2026

  24. [34]

    Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the

    Liu, Hanzhi and Shou, Chaofan and Wen, Hongbo and Chen, Yanju and Fang, Ryan Jingyang and Feng, Yu , journal=. Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the. 2026 , url =

    2026

  25. [35]

    Incompressible Knowledge Probes: Estimating Black-Box

    Li, Bojie , journal=. Incompressible Knowledge Probes: Estimating Black-Box. 2026 , url =

    2026

  26. [36]

    doi:10.18653/v1/2026.findings-eacl.273 , year =

    Yan, Yuliang and Tang, Haochun and Yan, Shuo and Dai, Enyan , booktitle =. doi:10.18653/v1/2026.findings-eacl.273 , year =

    2026 doi

  27. [38]

    , journal=

    Bradley, William F. , journal=. 2024 , url =

    2024

  28. [39]

    Log Probability Tracking of

    Chauvin, Timoth\'ee and Le Merrer, Erwan and Ta\"iani, Fran. Log Probability Tracking of. International Conference on Learning Representations (ICLR) , year =

  29. [40]

    Proceedings of the 42nd International Conference on Machine Learning , series =

    Idiosyncrasies in Large Language Models , author =. Proceedings of the 42nd International Conference on Machine Learning , series =. 2025 , url =

    2025

  30. [41]

    arXiv preprint arXiv:2607.10252 , year =

    One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions , author =. arXiv preprint arXiv:2607.10252 , year =

    arXiv

  31. [42]

    International Conference on Learning Representations (ICLR) , year =

    An Auditing Test to Detect Behavioral Shift in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  32. [43]

    arXiv preprint arXiv:2504.12335 , year =

    You've Changed: Detecting Modification of Black-Box Large Language Models , author =. arXiv preprint arXiv:2504.12335 , year =

    arXiv

  33. [44]

    2025 , url =

    Ong, Jack Min and Di Ferrante, Matthew and Pazdera, Aaron and Garner, Ryan and Jaghouar, Sami and Basra, Manveer and Ryabinin, Max and Hagemann, Johannes , journal=. 2025 , url =

    2025

  34. [45]

    The Annals of Mathematical Statistics , volume =

    Robust Estimation of a Location Parameter , author =. The Annals of Mathematical Statistics , volume =. 1964 , doi =

    1964

  35. [46]

    The Annals of Statistics , volume =

    Higher Criticism for Detecting Sparse Heterogeneous Mixtures , author =. The Annals of Statistics , volume =. 2004 , doi =

    2004

  36. [47]

    IEEE Transactions on Information Theory , year =

    Nonparametric Two-Sample Testing by Betting , author =. IEEE Transactions on Information Theory , year =

  37. [48]

    Bradley, W. F. 2024. LLM s and the Madness of Crowds. arXiv preprint arXiv:2411.01539

    2024 arXiv

  38. [49]

    Chauvin, T.; Lalanne, C.; Le Merrer, E.; Loubes, J.-M.; Ta\"iani, F.; and Tredan, G. 2026 a . Token-Efficient Change Detection in LLM API s. In International Conference on Machine Learning (ICML)

    2026

  39. [50]

    Chauvin, T.; Le Merrer, E.; Ta\"iani, F.; and Tredan, G. 2026 b . Log Probability Tracking of LLM API s. In International Conference on Learning Representations (ICLR)

    2026

  40. [51]

    Chernoff, H. 1952. A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the Sum of Observations. The Annals of Mathematical Statistics, 23(4): 493--507

    1952

  41. [52]

    Fang, Y.; Feng, Y.; Li, B.; and Zhou, M. 2026. KBF : Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing. arXiv preprint arXiv:2605.29524

    2026 arXiv

  42. [53]

    Gubri, M.; Ulmer, D.; Lee, H.; Yun, S.; and Oh, S. J. 2024. TRAP : Targeted Random Adversarial Prompt Honeypot for Black-Box Identification. In Findings of the Association for Computational Linguistics: ACL 2024, 11496--11517. Bangkok, Thailand: Association for Computational L...

    2024

  43. [54]

    Hoeffding, W. 1965. Asymptotically Optimal Tests for Multinomial Distributions. The Annals of Mathematical Statistics, 36(2): 369--401

    1965

  44. [55]

    Li, B. 2026. Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity. arXiv preprint arXiv:2604.24827

    2026 arXiv

  45. [56]

    Lin, G.; Wan, Y.; Pei, S.; Xu, T.; Xu, K.; and Xue, G. 2026. Behavioral Consistency and Transparency Analysis on Large Language Model API Gateways. In Proceedings of the 2026 ACM Internet Measurement Conference (IMC '26)

    2026

  46. [57]

    J.; and Feng, Y

    Liu, H.; Shou, C.; Wen, H.; Chen, Y.; Fang, R. J.; and Feng, Y. 2026. Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain. arXiv preprint arXiv:2604.08407

    2026 arXiv

  47. [58]

    P e zik, P.; Kaczy \'n ski, K.; Szyma \'n ska, M.; \.Z arnecki, F.; Deckert, Z.; Kwiatkowski, J.; and Janowski, W. 2025. LLMLagBench : Identifying Temporal Training Boundaries in Large Language Models. arXiv preprint arXiv:2511.12116

    2025

  48. [59]

    Yan, Y.; Tang, H.; Yan, S.; and Dai, E. 2026. DuFFin : A Dual-Level Fingerprinting Framework for LLM s IP Protection. In Findings of the Association for Computational Linguistics: EACL 2026, 5168--5184. Rabat, Morocco: Association for Computational Linguistics

    2026

  49. [60]

    Zhang, J.; Li, X.; and Wang, S. 2026. Your ``Pro'' LLM Subscription May Actually Be ``Free'': Exposing Fingerprint Spoofing Risks in LLM Inference Services. arXiv preprint arXiv:2606.16100

    2026

  50. [61]

    Zhang, Y.; Jiang, Y.; Chen, Z.; Backes, M.; Shen, X.; and Zhang, Y. 2026. Real Money, Fake Models: Deceptive Model Claims in Shadow API s. arXiv preprint arXiv:2603.01919

    2026

  51. [62]

    A.; and Neiswanger, W

    Zhu, X.; Ye, Y.; Qiu, T.; Zhu, H.; Tan, S.; Mannan, A.; Michala, J.; Popa, R. A.; and Neiswanger, W. 2026. Auditing Black-Box LLM API s with a Rank-Based Uniformity Test. In International Conference on Learning Representations (ICLR)

    2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.