EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint

Meng Han; Wenpeng Xing; Zhenhua Xu

REVIEW 3 major objections 4 minor 1 cited by

EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EverTracer claims that a stolen language model can be identified from the ordinary sentences it has memorized, using only API-level access.

desk verdict EverTracer's memorization-based fingerprint is a real idea, but the paper's robustness claims outrun the evidence until adaptive erasure is actually studied. read the letter →

arxiv 2509.03058 v1 pith:ZQ64VWY3 submitted 2025-09-03 cs.CR

Zhenhua Xu , Meng Han , Wenpeng Xing This is my paper · ORCID

classification cs.CR

keywords LLMfingerprintingmembershipinferenceattackmemorizationprobabilityvariationgray-boxverificationmodelprovenanceLoRAfine-tuningfingerprintrobustness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EverTracer claims that a model owner can prove an LLM was stolen by checking whether the suspect model shows unusually high probability variation around a set of private natural-language sentences that were memorized during fine-tuning. The check needs only gray-box access: send the sentences and semantically altered variants, read their log-probabilities, and compare the suspect model's probability variation against a reference model trained on the same distribution. If the calibrated signal is high for the private sentences and low for unrelated ones, the model is judged to have memorized them and therefore to descend from the owner's fine-tuned model. This would matter because it replaces trigger words and abnormal outputs with ordinary sentences, making fingerprints hard for perplexity filters to detect while surviving pruning, merging, and further fine-tuning.

What carries the argument

The load-bearing quantity is the calibrated probability variation $\Delta\widehat{p}(x) = \widehat{p}_{\theta_U}(x) - \widehat{p}_{\psi_{\text{ref}}}(x)$, where $\widehat{p}_\theta(x)$ approximates the expected second-order directional derivative of the model's log-likelihood at $x$: the average log-probability of semantically perturbed neighbors, $\frac{1}{2K}\sum_{k=1}^K [p_\theta(x^+_k) + p_\theta(x^-_k)] - p_\theta(x)$. Memorized fingerprint samples sit near local likelihood maxima, so their neighbors drop in probability, making $\widehat{p}$ large; a reference model fine-tuned on a distribution-matched sibling set provides the baseline that cancels generic frequency effects. The verification decision is a threshold on this difference, reported as true-positive rate at a false-positive rate of 5 percent (FSR) plus AUC, with $K=5$ token-level semantic perturbations generated by a paraphrasing text-to-text model per fingerprint sample.

What would settle it

Take a fingerprinted model, run the same calibrated probability-variation scoring over a wide text pool to identify which fingerprint strings score highest, then apply a fine-tuning or unlearning procedure that specifically lowers log-likelihood on those top-ranked strings; if FSR falls to near zero and AUC to about 0.5 while general task performance stays roughly level, the paper's central robustness claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that memorization itself is a usable fingerprint: a model owner fine-tunes the protected model on any private natural-language corpus, with no trigger–output pair, and later verifies ownership by detecting elevated calibrated probability variation on those exact strings in a suspect model. The paper argues that a finite-difference probability-variation estimator, subtracted from the same estimator computed by a reference model trained on a distribution-matched corpus, separates genuine memorization from frequency artifacts. Empirically, on Falcon-7B, LLaMA-2-7B, Mistral-7B, and LLaMA-3-8B with AG News and XSum fingerprints, the signal gives roughly 97 to 100 percent fingerprint success at a false-positive rate of 5 percent or less on unmodified models, and it persists through input perturbation, structural pruning, model merging with low weight on the fingerprinted model, and incremental fine-tuning, often where trigger-based baselines collapse.

Load-bearing premise

The threat model assumes the adversary applies generic modifications such as pruning, fine-tuning on unrelated data, or merging, and never tries to detect and erase the specific memorized fingerprint strings; an attacker who uses the same probability-variation method to identify those strings and then fine-tunes the model to lower their probabilities could remove the signal.

Editorial extensions

If this is right

A defender with only API access and token-level log-probabilities can trace stolen models that retain memorized private strings, without needing white-box access or predefined trigger responses.
Fingerprint inputs are fluent natural language, so perplexity-based input filters are unlikely to single them out, unlike optimization-based adversarial prompts.
The fingerprint survives generic post-hoc modifications including structural pruning, model merging with small weight on the fingerprinted model, and incremental fine-tuning on instruction data, in settings where backdoor baselines often drop to near zero success.
Reference-model calibration removes frequency-induced false positives, so the same distribution-matched corpus split provides both the fingerprint and the baseline needed for reliable verification.
Because any natural-language corpus can be used as fingerprint data, the approach is compatible with arbitrary private or domain-specific text chosen by the model owner.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same probability-variation metric that creates the fingerprint can be turned against it—an attacker who ranks candidate strings by the same signal and fine-tunes the model to lower their likelihoods could erase the fingerprint, so the robustness claim likely does not cover that fingerprint-aware attacker.
Editorial inference: choosing private or organization-specific natural-language data as the fingerprint would make removal harder, because deleting those strings would degrade genuinely useful capabilities; the trade-off is that the fingerprint itself is potentially extractable training data, so defenders using sensitive text must weigh privacy.
Editorial inference: if memorization persists through light downstream fine-tuning as the experiments suggest, a natural extension is transferability—the same memorized strings could identify a family of models sharing a pretraining origin, letting one injection trace multiple derivative models; the paper lists this as an open question.
Editorial inference: because verification needs token-level log-probabilities, a strict black-box API that rounds or hides logits would blunt the signal; a testable extension is to check whether the probability-variation signal survives quantization or logit rounding before deployment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

EverTracer is worth reading, and it deserves a serious referee. The core move—verifying ownership by detecting memorized natural-language strings via reference-calibrated probability variation, rather than by eliciting trigger outputs—is a genuine reframing of LLM fingerprinting. The paper is the first to repurpose MIA signals defensively in this setting, and it evaluates the idea across four architectures, under pruning, merging, and incremental fine-tuning, with public code and data.

The unmodified-model results are strong, and the method holds up well in several adversarial settings: Taylor pruning, low-alpha merging, and some incremental tuning regimes. The harmlessness study is more thorough than what most fingerprinting papers run. The citation pattern is honest about the MIA lineage (Fu et al., Mattern et al.) and the limitations paragraph is candid.

The soft spots are real, though. First, the robustness claims outrun the numbers: under random/L1/L2 pruning on LLaMA2, FSR drops to 13–27%, and an AUC of 0.74 is described as a strong and reliable signal; 5% character deletion on LLaMA2 cuts FSR to 49%. Those are useful data points, not failures, but they do not support "robust against adaptive adversaries" as stated. Second, the adaptive-adversary gap is load-bearing. The verification signal is an MIA score; an attacker who knows EverTracer can run the same PV calibration on their own reference model to locate likely memorized strings, then fine-tune or unlearn them. Because the fingerprint data are sampled from public corpora, the attacker does not even need access to the defender's private data. The paper explicitly leaves this to future work, but until it is tested, the security claim should be scoped to non-adaptive modifications. Third, the FSR metric is not apples-to-apples across methods: EverTracer's FSR is TPR at FPR≤5%, while the baselines report trigger-success rates. Side-by-side tables therefore overstate the comparison. Finally, most results have no error bars; the one place with repetitions reports no variance.

The central idea holds up as a proof of concept. The paper is for researchers working on LLM IP protection and model provenance; it will likely set off follow-up work on adaptive erasure and on the MIA-as-defense direction. For peer review, I would recommend conditional acceptance, with the robustness language tempered, error bars added, and at least one adaptive erasure experiment using the same PV signal—or an explicit, defended scope limitation if that experiment is infeasible.

Referee Report

3 major / 4 minor

Summary. EverTracer proposes a gray-box LLM fingerprinting method that injects memorization of natural-language strings via LoRA fine-tuning and verifies ownership by comparing a calibrated probability-variation signal, computed from neighborhood perturbations, between the suspect model and a reference model trained on a held-out split of the fingerprint corpus. The paper reports high FSR and AUC on unmodified models, low perplexity of fingerprint inputs, and evaluates robustness under input character deletion, pruning, model merging, and incremental fine-tuning across Falcon, LLaMA2, Mistral, and LLaMA3. It claims state-of-the-art effectiveness, stealthiness, and resilience, including against adaptive adversaries.

Significance. If the result holds, EverTracer is a meaningful step for LLM ownership verification: it avoids trigger-output overfitting, uses natural-language memorization, requires only token-level log-probabilities, and controls false positives via a held-out background set and reference-model calibration. The method's verification signal is not obviously overfitted to the reported numbers: the PV signal is adopted from prior work, the threshold is selected to control FPR at 5%, and the reference model is independently trained. The paper also provides a broad evaluation across four model families and several modification scenarios, and it releases code and data. However, the robustness claims are substantially stronger than the evidence, particularly under input perturbation, pruning, and fingerprint-aware erasure, so the central claim needs revision.

major comments (3)

[Limitations] The Limitations section explicitly states that "explicit adaptive attacks that aim to erase memorized content ... have yet to be systematically studied" and that resilience to MEraser remains unclear. This is not a peripheral caveat: EverTracer's verification signal is itself a membership-inference score, so an adversary who knows the verification procedure can compute the same calibrated PV signal on candidate strings drawn from the same public distribution as the fingerprint data (AG News, XSum), identify the highest-scoring memorized strings, and fine-tune or unlearn those exact strings. The current experiments in §5.4 only consider non-adaptive modifications such as pruning, merging, and unrelated fine-tuning. Because ownership tracing depends on the suspect model retaining an elevated PV signal on precisely the fingerprinted strings, this gap is load-bearing for the central robustness claim. I recommend adding an explicit adaptive-erasure experiment in which the attacker uses the public PV/MIA procedure to rank a pool of candidate strings, erases the top-ranked ones, and reports FSR and AUC on the remaining fingerprint set.
[§5.4.1, Table 3] Under 5% character deletion on LLaMA2, EverTracerAG achieves FSR 49% with AUC 0.67; at 10% deletion, FSR drops to 37% with AUC 0.47, which is essentially a coin flip. This contradicts the abstract's claim that the approach "remains robust against adaptive adversaries, including input level modification." The text acknowledges the drop but attributes the model-dependent behavior to a hypothesis about weaker models and defers further investigation to future work. Since input-level perturbation is one of the explicitly claimed adaptive scenarios, the robustness claim should be qualified to specific model families and perturbation levels, and the proposed mechanism should be supported by evidence rather than left as speculation.
[§5.4.2, Table 4] Under Random, L1, and L2 pruning on LLaMA2, EverTracerAG reports FSR of 27%, 13%, and 16%, respectively; the text states that AUC values around 0.74–0.84 are "sufficient to serve as a strong and reliable fingerprint signal." Under the paper's own definition of FSR as TPR at FPR≤5% (§4.3), these FSR values mean the fingerprint is detected in only a small fraction of cases at the controlled false-positive rate. The claim that this constitutes a reliable signal is not supported by the operational metric, and this is load-bearing for the robustness claims in the abstract and conclusion.

minor comments (4)

[Introduction and §4.3] There are typos in the technical description: "naural language dataset" should be "natural language dataset," and "fingerprint mermbers" should be "fingerprint members."
[§4.3] The notation for the probability-variation estimator is inconsistent: the equation defines $\hat{e}_{p_\theta}(x)$, but subsequent text uses "ep" without the hat. Please use one consistent symbol throughout.
[§5.4.2, Tables 4 and 5] The method name is rendered both as "EverTracer AG" and "EverTracerAG" in different tables; please standardize the notation.
[Abstract] The claim of being "the first" to repurpose MIAs for defensive use is strong; consider softening it to "to our knowledge" and citing any prior MIA-based defensive methods for LLMs to make the novelty statement precise.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: verification uses an independent published MIA signal, threshold is controlled on background data, and self-citations are not load-bearing.

full rationale

EverTracer's verification signal is imported from published MIA work (Fu et al., 2024; Mattern et al., 2023), not derived from the ownership claim itself. The decision threshold is selected on the background set Dunseen to hold FPR at 5%, and FSR/AUC are then measured on Dtr, so no parameter is fitted to the reported success rates. The reference model is independently fine-tuned on Dref, which is distribution-matched but separate from Dtr, providing an external calibration anchor. The paper's many self-citations are to related work, baselines, and its own acknowledged limitation (MEraser), and none is used as the load-bearing justification for the memorization signal. The admitted absence of an adaptive attacker who would use the same PV/MIA score to erase the exact memorized strings is a threat-model scope gap, not a circular definition: the paper's equations do not reduce the fingerprint claim to an input assumption. No self-definitional step, fitted-input-called-prediction step, or self-citation chain was found.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or abstract entities are introduced. The five hand-chosen hyperparameters are fixed, not fitted to maximize reported metrics. The main assumptions are inherited from the MIA literature on which the method rests.

free parameters (5)

Fingerprint training set size (Dtr) = 100 samples
Chosen as a fixed size for efficient fingerprint injection; not optimized to reported results.
Reference set size (Dref) = 1000 samples
Chosen to balance calibration and data representativeness.
Perturbation count K = 5
Number of paraphrases generated per fingerprint sample; fixed hyperparameter.
Token perturbation ratio = 30%
Fraction of tokens randomly selected for semantic replacement via T5-Base.
LoRA fine-tuning epochs = 10 (Mistral) or 20 (others) for victim; 4 for reference
Fixed by hand; no sweep performed to maximize reported FSR.

assumptions (4)

domain assumption Memorized training samples occupy local maxima or positive-curvature regions of the model's likelihood landscape.
Inherited from van den Burg and Williams (2021) and Fu et al. (2024), used in Appendix B to justify the PV signal.
domain assumption A reference model fine-tuned on the same data distribution calibrates out frequency-related false positives.
Based on Watson et al. (2022) and Mireshghallah et al. (2022); used in Section 4.3 and 5.6.
domain assumption T5-Base rephrasing preserves semantic locality so that perturbed variants approximate small shifts in likelihood space.
Used in Section 5.1 and Appendix C; if paraphrases are noisy or off-manifold, the PV estimate degrades.
domain assumption The attacker cannot access Dtr and does not perform fingerprint-aware fine-tuning to erase known memorized strings.
Stated in the threat model (Section 3) and flagged as future work in Limitations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint." pith.science (2026). https://pith.science/paper/ZQ64VWY3

@misc{pith2026250903058,
  author       = {Pith},
  title        = {Pith review of: EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQ64VWY3}},
  note         = {Machine review of arXiv:2509.03058}
}

read the original abstract

The proliferation of large language models (LLMs) has intensified concerns over model theft and license violations, necessitating robust and stealthy ownership verification. Existing fingerprinting methods either require impractical white-box access or introduce detectable statistical anomalies. We propose EverTracer, a novel gray-box fingerprinting framework that ensures stealthy and robust model provenance tracing. EverTracer is the first to repurpose Membership Inference Attacks (MIAs) for defensive use, embedding ownership signals via memorization instead of artificial trigger-output overfitting. It consists of Fingerprint Injection, which fine-tunes the model on any natural language data without detectable artifacts, and Verification, which leverages calibrated probability variation signal to distinguish fingerprinted models. This approach remains robust against adaptive adversaries, including input level modification, and model-level modifications. Extensive experiments across architectures demonstrate EverTracer's state-of-the-art effectiveness, stealthness, and resilience, establishing it as a practical solution for securing LLM intellectual property. Our code and data are publicly available at https://github.com/Xuzhenhua55/EverTracer.

Figures

Figures reproduced from arXiv: 2509.03058 by the authors.

**Figure 2.** MDARE task and MDARE ties visualisations showing trends for different α values. Detailed numerical results can be found in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Summary of average task performance and variations for each method vided in Appendix F. In particular, we apply different values of α for different merging strategies to merge fingerprinted Mistral with benign Mistral7B-Instruct-v0.3 (Jiang et al., 2024). The corresponding results are presented in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Overall comparasion of input-output patterns accross different baseline fingerprinting methods [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

**Figure 6.** Figure 6: Ablation study on EverTracer’s performance [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

**Figure 5.** Figure 5: Mtask and Mties visualizations showing trends under various α values. F Model Merging Strategies F.1 Task Arithmetic Task Arithmetic (Ilharco et al., 2022) synthesizes a unified model by aggregating parameter deviations between expert models and the base model. Let θ0 …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor
cs.CL 2025-09 conditional novelty 6.0 of 10

CTCC embeds LLM ownership fingerprints in cross-turn semantic contradictions: the model fires a secret response only when a user contradicts an earlier statement, with higher robustness and stealth than single-turn triggers.

Reference graph

Works this paper leans on

83 extracted references · 8 canonical work pages · cited by 1 Pith paper

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. 2018. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX security symposium (USENIX Security 18), pages 1615--1631

2018
[4]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B : an open large language model with state-of-the-art performance

2023
[5]

Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. 2024. Here's a free lunch: Sanitizing backdoored models with model merge. arXiv preprint arXiv:2402.19334

arXiv 2024
[6]

Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. arXiv preprint arXiv:2402.11746

arXiv 2024
[7]

Andrew P Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145--1159

1997
[8]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020

Show all 83 references

[9]

Jiacheng Cai, Jiahao Yu, Yangguang Shao, Yuhang Wu, and Xinyu Xing. 2024. Utf: Undertrained tokens as fingerprints a novel approach to llm identification. arXiv preprint arXiv:2410.12318

2024 arXiv
[10]

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21),...

2021
[11]

Jialuo Chen, Jingyi Wang, Tinglan Peng, Youcheng Sun, Peng Cheng, Shouling Ji, Xingjun Ma, Bo Li, and Dawn Song. 2022. Copy, right? a testing framework for copyright protection of deep learning models. In 2022 IEEE symposium on security and privacy (SP), pages 824--841. IEEE

2022
[12]

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, pages 2924--2936

2019
[13]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

2018 arXiv
[14]

Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. 2024. Have you merged my model? on the robustness of large language model ip protection methods against model merging. arXiv preprint arXiv:2404.05188

2024 arXiv
[15]

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free dolly: Introducing the world's ...

2023
[16]

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107--124

2019
[17]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: efficient finetuning of quantized llms (2023). arXiv preprint arXiv:2305.14314, 52:3982--3992

2023 arXiv
[18]

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2402.07841 Do Membership Inference Attacks Work on Large Language Models ?

2024 arXiv
[19]

Vitaly Feldman. 2020. https://doi.org/10.1145/3357713.3384290 Does learning require memorization? a short tale about a long tail . In Proccedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, Chicago, IL, USA, June 22-26, 2020 , pages 954--959. ACM

2020
[20]

Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2024. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[21]

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

2024 doi
[22]

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1--9

2007
[23]

Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. https://doi.org/10.18653/v1/2024.emnlp-industry.36 Arcee ' s M erge K it: A toolkit for merging large language models . In Proceedi...

2024 doi
[24]

Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. 2024. https://openreview.net/forum?id=9k0krNzvlV On the learnability of watermarks for language models . In The Twelfth International Conference on Learning Representations

2024
[25]

Martin Gubri, Dennis Ulmer, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. 2024. Trap: Targeted random adversarial prompt honeypot for black-box identification. arXiv preprint arXiv:2402.12991

2024 arXiv
[26]

Jia Guo and Miodrag Potkonjak. 2018. Watermarking deep neural networks for embedded systems. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 1--8. IEEE

2018
[27]

hiyouga. 2023. Llama factory. https://github.com/hiyouga/LLaMA-Factory

2023
[28]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[29]

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

2022 arXiv
[30]

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

2023 arXiv
[31]

Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger,...

2024
[32]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[33]

Heng Jin, Chaoyu Zhang, Shanghao Shi, Wenjing Lou, and Y Thomas Hou. 2024. Proflingo: A fingerprinting-based intellectual property protection scheme for large language models. In 2024 IEEE Conference on Communications and Network Security (CNS), pages 1--9. IEEE

2024
[34]

Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp...

2018
[35]

John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. A watermark for large language models. In International Conference on Machine Learning, pages 17061--17084. PMLR

2023
[36]

Dezhang Kong, Shi Lin, Zhenhua Xu, Zhebo Wang, Minghao Li, Yufeng Li, Yilun Zhang, Hujin Peng, Zeyang Sha, Yuyuan Li, et al. 2025. A survey of llm-driven ai agent communication: Protocols, security risks, and defense countermeasures. arXiv preprint arXiv:2506.19676

2025
[37]

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519--3529. PMLR

2019
[38]

Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning

2012
[39]

Huiying Li, Emily Wenger, Shawn Shan, Ben Y Zhao, and Haitao Zheng. 2019 a . Piracy resistant watermarks for deep neural networks. arXiv preprint arXiv:1910.01226

2019 arXiv
[40]

Peixuan Li, Pengzhou Cheng, Fangqi Li, Wei Du, Haodong Zhao, and Gongshen Liu. 2023. Plmmark: a secure and robust black-box watermarking framework for pre-trained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14991--14999

2023
[41]

Shen Li, Liuyi Yao, Jinyang Gao, Lan Zhang, and Yaliang Li. 2024. Double-i watermark: Protecting model copyright for llm fine-tuning. arXiv preprint arXiv:2402.14883

2024 arXiv
[42]

Zheng Li, Chengyu Hu, Yang Zhang, and Shanqing Guo. 2019 b . How to prove your model belongs to you: A blind-watermark based framework to protect intellectual property of dnn. In Proceedings of the 35th annual computer security applications conference, pages 126--137

2019
[43]

Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021. Logiqa: a challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial I...

2021
[44]

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

2023
[45]

M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. In Proceedings of the ARPA Workshop on Human Language Technology, pages 114--119. Association for Computational Linguistics

1993
[46]

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Sch \"o lkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick . 2023. https://arxiv.org/abs/2305.18462 Membership Inference Attacks against Language Models via Neighbourhood Comparison

2023 arXiv
[47]

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381--2391

2018
[48]

Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick. 2022. https://aclanthology.org/2022.emnlp-main.119 An empirical analysis of memorization in fine-tuned autoregressive language models . In Proceedings of the 2022 Conference on E...

2022
[49]

Cohen, and Mirella Lapata

Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natu...

2018 doi
[50]

Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI : A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computat...

2020
[51]

Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

2019
[52]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8)

2019
[53]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

2020
[54]

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series

2011
[55]

Mark Russinovich and Ahmed Salem. 2024. Hey, that's my model! introducing chain & hash, an llm fingerprinting technique. arXiv preprint arXiv:2407.10887

2024 arXiv
[56]

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

2021
[57]

Shenghao, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhur...

2024 arXiv
[58]

Haonan Shi, Tu Ouyang, and An Wang. 2024. Learning-based difficulty calibration for enhanced membership inference attacks. In 2024 IEEE 9th European Symposium on Security and Privacy ( EuroS & P ) , pages 62--77. IEEE Computer Society

2024
[59]

shibing624. 2024. Sharegpt gpt4 dataset on hugging face hub. https://huggingface.co/datasets/shibing624/sharegpt_gpt4. Accessed: 2025-02-04

2024
[60]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

2023
[61]

Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization Without Overfitting : Analyzing the Training Dynamics of Large Language Models . Advances in Neural Information Processing Systems, 35:38274--38290

2022
[62]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[63]

Gerrit van den Burg and Chris Williams. 2021. On memorization in probabilistic deep generative models. Advances in neural information processing systems, 34:27916--27928

2021
[64]

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

2017
[65]

Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. 2022. https://openreview.net/forum?id=3eIrli0TwQ On the importance of difficulty calibration in membership inference attacks . In The Tenth International Conference on Learning Representations, ICLR 2022, Vi...

2022
[66]

Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94--106

2017
[67]

Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024 a . Instructional fingerprinting of large language models. arXiv preprint arXiv:2401.12255

2024 arXiv
[68]

Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, and Ting Wang. 2024 b . https://arxiv.org/abs/2411.13144 Copyrightmeter: Revisiting copyright protection in text-to-image models . Preprint, arXiv:2411.13144

2024 arXiv
[69]

Naen Xu, Jinghuai Zhang, Changjiang Li, Zhi Chen, Chunyi Zhou, Qingming Li, Tianyu Du, and Shouling Ji. 2025 a . https://arxiv.org/abs/2508.15314 Videoeraser: Concept erasure in text-to-video diffusion models . Preprint, arXiv:2508.15314

2025 arXiv
[70]

Zhenhua Xu, Meng Han, Xubin Yue, and Wenpeng Xing. 2025 b . https://doi.org/10.1360/SSI-2025-0022 Insty: a robust multi-level cross-granularity fingerprint embedding algorithm for multi-turn dialogue in large language models . SCIENTIA SINICA Informationis, 55(8):1906--

2025 doi
[71]

Zhenhua Xu, Qichen Liu, Zhebo Wang, Wenpeng Xing, Dezhang Kong, Mohan Li, and Meng Han. 2025 c . https://arxiv.org/abs/2409.08846 Fingerprint vector: Enabling scalable and efficient model fingerprint transfer via vector addition . Preprint, arXiv:2409.08846

2025 arXiv
[72]

Zhenhua Xu, Zhebo Wang, Maike Li, Wenpeng Xing, Chunqiang Hu, Chen Zhi, and Meng Han. 2025 d . https://arxiv.org/abs/2505.06304 Rap-sm: Robust adversarial prompt via shadow models for copyright verification of large language models . Preprint, arXiv:2505.06304

2025 arXiv
[73]

Zhenhua Xu, Zhaokun Yan, Binhan Xu, Xin Tong, Haitao Xu, Yourong Chen, and Meng Han. 2025 e . https://arxiv.org/abs/2509.00820 Unlocking the effectiveness of lora-fp for seamless transfer implantation of fingerprints in downstream models . Preprint, arXiv:2509.00820

2025 arXiv
[74]

Zhenhua Xu, Xubin Yue, Zhebo Wang, Qichen Liu, Xixiang Zhao, Jingxuan Zhang, Wenjun Zeng, Wengpeng Xing, Dezhang Kong, Changting Lin, and Meng Han. 2025 f . https://arxiv.org/abs/2508.11548 Copyright protection for large language models: A survey of methods, challenges, and tr...

2025 arXiv
[75]

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

2024
[76]

Zhiguang Yang and Hanzhou Wu. 2024. A fingerprint for large language models. arXiv preprint arXiv:2407.01235

2024
[77]

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

2024
[78]

Xubin Yue, Zhenhua Xu, Wenpeng Xing, Jiahui Yu, Mohan Li, and Meng Han. 2025. https://arxiv.org/abs/2509.00918 Pree: Towards harmless and adaptive fingerprint editing in large language models via knowledge prefix enhancement . Preprint, arXiv:2509.00918

2025 arXiv
[79]

Boyi Zeng, Chenghu Zhou, Xinbing Wang, and Zhouhan Lin. 2023. Huref: Human-readable fingerprint for large language models. arXiv preprint arXiv:2312.04828

2023 arXiv
[80]

Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph Stoecklin, Heqing Huang, and Ian Molloy. 2018. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia conference on computer and communications security, pages 159--172

2018
[81]

Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. 2024. Reef: Representation encoding fingerprints for large language models. arXiv preprint arXiv:2410.14273

2024 arXiv
[82]

Jingxuan Zhang, Zhenhua Xu, Rui Hu, Wenpeng Xing, Xuhong Zhang, and Meng Han. 2025. https://doi.org/10.18653/v1/2025.acl-long.1455 ME raser: An effective fingerprint erasure approach for large language models . In Proceedings of the 63rd Annual Meeting of the Association for C...

2025 doi
[83]

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html Character-level convolutional networks for text classification . In Advances in Neural Information Processing Systems 28: Annual Co...

2015

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. 2018. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In 27th USENIX security symposium (USENIX Security 18), pages 1615--1631

2018

[4] [4]

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B : an open large language model with state-of-the-art performance

2023

[5] [5]

Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. 2024. Here's a free lunch: Sanitizing backdoored models with model merge. arXiv preprint arXiv:2402.19334

arXiv 2024

[6] [6]

Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. arXiv preprint arXiv:2402.11746

arXiv 2024

[7] [7]

Andrew P Bradley. 1997. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145--1159

1997

[8] [8]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020