{"id":"b8c01c66-b647-499f-a25d-c5bb58dae0a6","arxiv_id":"2509.03058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A gray-box LLM fingerprinting method that detects memorized private text via calibrated probability variation, instead of using backdoor triggers.","lead":"EverTracer fingerprints stolen language models by fine-tuning them on private natural-language text and later checking whether a suspect model still 'remembers' those strings. It turns the usual membership-inference attack into a defensive tool that works through API-level access.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed robustness against adaptive adversaries is untested against an attacker who uses the same MIA/PV signal to locate and erase the memorized fingerprint strings; the paper's own Limitations section admits this gap, and the experimental fingerprint data is drawn from public corpora that…","rationale":"The paper's contribution is coherent and the experiments are extensive: the LoRA injection is cheap, the PV signal is derived from established MIA work, and the reference-model calibration is a sensible way to control distributional bias. The effectiveness, PPL stealthiness, and many robustness results are genuinely useful. The central weakness is not the empirical FSR/AUC values under non-adaptive modifications (though the LLaMA2 RP-10% AUC of 0.47 and an AUC of 0.74 described as 'strong' are concerning), but the absence of any evaluation of an adversary who actively attempts to erase the memorized content. Because the fingerprint is a set of natural-language strings and the verification score is a public MIA score, the attacker can invert the verification process: query the suspect model with a large candidate corpus, rank by PV signal, and target the top-ranked strings with fine-tuning or unlearning. The paper's own Limitations section explicitly identifies this as an open problem and mentions MEraser as a plausible erasure technique. The fact that Dtr is drawn from public benchmark data in all experiments makes the search space concrete; if the intended deployment uses truly private data, that secrecy assumption should be stated and tested, since it is not the setting evaluated. This gap does not make the paper unserious, but it means the headline claim 'robust against adaptive adversaries' is broader than the evidence supports. A conditional acceptance requiring either an adaptive-erasure evaluation or an explicit restriction of the threat model to non-fingerprint-aware attackers is appropriate; hence the reader's CONDITIONAL verdict is unchanged.","tokens_in":26063,"tokens_out":10511,"duration_ms":104282,"concrete_test":"Reproduce the LLaMA2-AGNews EverTracer model from the released code. Simulate an adaptive attacker with API access who knows the EverTracer algorithm but not Dtr: (1) sample 10,000 documents from the public AG News corpus; (2) train a reference model on a random 1,000-document split with the same LoRA recipe; (3) compute the PV signal for all 10,000 candidates under the suspect model; (4) select the 300 highest-scoring candidates; (5) fine-tune the suspect model with LoRA for one epoch using gradient ascent (or an unlearning method like MEraser) to decrease likelihood on those candidates while preserving performance on a held-out task; (6) run EverTracer verification on Dtr and a fresh Dunseen. If FSR falls below 50% or AUC below 0.8, the claimed robustness against adaptive model-level modification does not hold and the threat model must be restricted to non-fingerprint-aware attackers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"EverTracer's central security property is that a stolen model retains a detectable memorization signal that survives model-level modification. The experiments only consider non-adaptive modifications (pruning, merging, fine-tuning on unrelated corpora). They do not consider an attacker who knows EverTracer's public verification procedure and uses the same calibrated PV/MIA score to discover which natural-language strings the suspect model has memorized, then fine-tunes or unlearns those exact strings. Because the reported fingerprint datasets are sampled from public corpora (AG News, XSum), the attacker can enumerate candidates from the same distribution, compute PV scores against their own reference model, and select the top-scoring strings; the private-data claim in the threat model is not what is evaluated. The Limitations section explicitly states that 'explicit adaptive attacks that aim to erase memorized content ... have yet to be systematically studied' and that resilience to MEraser remains unclear. Since the verification signal is itself a membership-inference score, the method hands the attacker the same detector used for verification. If this attack succeeds, ownership tracing fails even though the method performs well under generic fine-tuning or pruning. This is a scope gap in the threat model rather than an internal inconsistency, but it is the load-bearing assumption for the paper's robustness claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EverTracer proposes a gray-box LLM fingerprinting method that injects memorization of natural-language strings via LoRA fine-tuning and verifies ownership by comparing a calibrated probability-variation signal, computed from neighborhood perturbations, between the suspect model and a reference model trained on a held-out split of the fingerprint corpus. The paper reports high FSR and AUC on unmodified models, low perplexity of fingerprint inputs, and evaluates robustness under input character deletion, pruning, model merging, and incremental fine-tuning across Falcon, LLaMA2, Mistral, and LLaMA3. It claims state-of-the-art effectiveness, stealthiness, and resilience, including against adaptive adversaries.","tokens_in":26290,"tokens_out":4048,"duration_ms":35349,"significance":"If the result holds, EverTracer is a meaningful step for LLM ownership verification: it avoids trigger-output overfitting, uses natural-language memorization, requires only token-level log-probabilities, and controls false positives via a held-out background set and reference-model calibration. The method's verification signal is not obviously overfitted to the reported numbers: the PV signal is adopted from prior work, the threshold is selected to control FPR at 5%, and the reference model is independently trained. The paper also provides a broad evaluation across four model families and several modification scenarios, and it releases code and data. However, the robustness claims are substantially stronger than the evidence, particularly under input perturbation, pruning, and fingerprint-aware erasure, so the central claim needs revision.","major_comments":[{"comment":"The Limitations section explicitly states that \"explicit adaptive attacks that aim to erase memorized content ... have yet to be systematically studied\" and that resilience to MEraser remains unclear. This is not a peripheral caveat: EverTracer's verification signal is itself a membership-inference score, so an adversary who knows the verification procedure can compute the same calibrated PV signal on candidate strings drawn from the same public distribution as the fingerprint data (AG News, XSum), identify the highest-scoring memorized strings, and fine-tune or unlearn those exact strings. The current experiments in §5.4 only consider non-adaptive modifications such as pruning, merging, and unrelated fine-tuning. Because ownership tracing depends on the suspect model retaining an elevated PV signal on precisely the fingerprinted strings, this gap is load-bearing for the central robustness claim. I recommend adding an explicit adaptive-erasure experiment in which the attacker uses the public PV/MIA procedure to rank a pool of candidate strings, erases the top-ranked ones, and reports FSR and AUC on the remaining fingerprint set.","section":"Limitations"},{"comment":"Under 5% character deletion on LLaMA2, EverTracerAG achieves FSR 49% with AUC 0.67; at 10% deletion, FSR drops to 37% with AUC 0.47, which is essentially a coin flip. This contradicts the abstract's claim that the approach \"remains robust against adaptive adversaries, including input level modification.\" The text acknowledges the drop but attributes the model-dependent behavior to a hypothesis about weaker models and defers further investigation to future work. Since input-level perturbation is one of the explicitly claimed adaptive scenarios, the robustness claim should be qualified to specific model families and perturbation levels, and the proposed mechanism should be supported by evidence rather than left as speculation.","section":"§5.4.1, Table 3"},{"comment":"Under Random, L1, and L2 pruning on LLaMA2, EverTracerAG reports FSR of 27%, 13%, and 16%, respectively; the text states that AUC values around 0.74–0.84 are \"sufficient to serve as a strong and reliable fingerprint signal.\" Under the paper's own definition of FSR as TPR at FPR≤5% (§4.3), these FSR values mean the fingerprint is detected in only a small fraction of cases at the controlled false-positive rate. The claim that this constitutes a reliable signal is not supported by the operational metric, and this is load-bearing for the robustness claims in the abstract and conclusion.","section":"§5.4.2, Table 4"}],"minor_comments":[{"comment":"There are typos in the technical description: \"naural language dataset\" should be \"natural language dataset,\" and \"fingerprint mermbers\" should be \"fingerprint members.\"","section":"Introduction and §4.3"},{"comment":"The notation for the probability-variation estimator is inconsistent: the equation defines \\(\\hat{e}_{p_\\theta}(x)\\), but subsequent text uses \"ep\" without the hat. Please use one consistent symbol throughout.","section":"§4.3"},{"comment":"The method name is rendered both as \"EverTracer AG\" and \"EverTracerAG\" in different tables; please standardize the notation.","section":"§5.4.2, Tables 4 and 5"},{"comment":"The claim of being \"the first\" to repurpose MIAs for defensive use is strong; consider softening it to \"to our knowledge\" and citing any prior MIA-based defensive methods for LLMs to make the novelty statement precise.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is solid, but the adaptive-attack limitation is not merely an edge case: it is central to the claimed robustness. The revision should add an adaptive-erasure experiment or substantially narrow the robustness claims in the abstract and conclusion. Without that change, I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EverTracer is worth reading, and it deserves a serious referee. The core move—verifying ownership by detecting memorized natural-language strings via reference-calibrated probability variation, rather than by eliciting trigger outputs—is a genuine reframing of LLM fingerprinting. The paper is the first to repurpose MIA signals defensively in this setting, and it evaluates the idea across four architectures, under pruning, merging, and incremental fine-tuning, with public code and data.\n\nThe unmodified-model results are strong, and the method holds up well in several adversarial settings: Taylor pruning, low-alpha merging, and some incremental tuning regimes. The harmlessness study is more thorough than what most fingerprinting papers run. The citation pattern is honest about the MIA lineage (Fu et al., Mattern et al.) and the limitations paragraph is candid.\n\nThe soft spots are real, though. First, the robustness claims outrun the numbers: under random/L1/L2 pruning on LLaMA2, FSR drops to 13–27%, and an AUC of 0.74 is described as a strong and reliable signal; 5% character deletion on LLaMA2 cuts FSR to 49%. Those are useful data points, not failures, but they do not support \"robust against adaptive adversaries\" as stated. Second, the adaptive-adversary gap is load-bearing. The verification signal is an MIA score; an attacker who knows EverTracer can run the same PV calibration on their own reference model to locate likely memorized strings, then fine-tune or unlearn them. Because the fingerprint data are sampled from public corpora, the attacker does not even need access to the defender's private data. The paper explicitly leaves this to future work, but until it is tested, the security claim should be scoped to non-adaptive modifications. Third, the FSR metric is not apples-to-apples across methods: EverTracer's FSR is TPR at FPR≤5%, while the baselines report trigger-success rates. Side-by-side tables therefore overstate the comparison. Finally, most results have no error bars; the one place with repetitions reports no variance.\n\nThe central idea holds up as a proof of concept. The paper is for researchers working on LLM IP protection and model provenance; it will likely set off follow-up work on adaptive erasure and on the MIA-as-defense direction. For peer review, I would recommend conditional acceptance, with the robustness language tempered, error bars added, and at least one adaptive erasure experiment using the same PV signal—or an explicit, defended scope limitation if that experiment is infeasible.","headline":"EverTracer's memorization-based fingerprint is a real idea, but the paper's robustness claims outrun the evidence until adaptive erasure is actually studied.","tokens_in":26871,"tokens_out":3252,"would_cite":true,"duration_ms":29642,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EverTracer claims that a stolen language model can be identified from the ordinary sentences it has memorized, using only API-level access.","keywords":["LLM fingerprinting","membership inference attack","memorization","probability variation","gray-box verification","model provenance","LoRA fine-tuning","fingerprint robustness"],"falsifier":"Take a fingerprinted model, run the same calibrated probability-variation scoring over a wide text pool to identify which fingerprint strings score highest, then apply a fine-tuning or unlearning procedure that specifically lowers log-likelihood on those top-ranked strings; if FSR falls to near zero and AUC to about 0.5 while general task performance stays roughly level, the paper's central robustness claim is falsified.","tokens_in":25801,"feed_emoji":"🕵️","tokens_out":6717,"duration_ms":56944,"temperature":0.7,"pith_summary":"EverTracer claims that a model owner can prove an LLM was stolen by checking whether the suspect model shows unusually high probability variation around a set of private natural-language sentences that were memorized during fine-tuning. The check needs only gray-box access: send the sentences and semantically altered variants, read their log-probabilities, and compare the suspect model's probability variation against a reference model trained on the same distribution. If the calibrated signal is high for the private sentences and low for unrelated ones, the model is judged to have memorized them and therefore to descend from the owner's fine-tuned model. This would matter because it replaces trigger words and abnormal outputs with ordinary sentences, making fingerprints hard for perplexity filters to detect while surviving pruning, merging, and further fine-tuning.","feed_headline":"Private sentences memorized during fine-tuning trace stolen LLMs","feed_subtitle":"A gray-box probability check exposes whether a suspect model still memorized your sentences, even after pruning, merging, or fine-tuning","key_machinery":"The load-bearing quantity is the calibrated probability variation $\\Delta\\widehat{p}(x) = \\widehat{p}_{\\theta_U}(x) - \\widehat{p}_{\\psi_{\\text{ref}}}(x)$, where $\\widehat{p}_\\theta(x)$ approximates the expected second-order directional derivative of the model's log-likelihood at $x$: the average log-probability of semantically perturbed neighbors, $\\frac{1}{2K}\\sum_{k=1}^K [p_\\theta(x^+_k) + p_\\theta(x^-_k)] - p_\\theta(x)$. Memorized fingerprint samples sit near local likelihood maxima, so their neighbors drop in probability, making $\\widehat{p}$ large; a reference model fine-tuned on a distribution-matched sibling set provides the baseline that cancels generic frequency effects. The verification decision is a threshold on this difference, reported as true-positive rate at a false-positive rate of 5 percent (FSR) plus AUC, with $K=5$ token-level semantic perturbations generated by a paraphrasing text-to-text model per fingerprint sample.","core_discovery":"The central claim is that memorization itself is a usable fingerprint: a model owner fine-tunes the protected model on any private natural-language corpus, with no trigger–output pair, and later verifies ownership by detecting elevated calibrated probability variation on those exact strings in a suspect model. The paper argues that a finite-difference probability-variation estimator, subtracted from the same estimator computed by a reference model trained on a distribution-matched corpus, separates genuine memorization from frequency artifacts. Empirically, on Falcon-7B, LLaMA-2-7B, Mistral-7B, and LLaMA-3-8B with AG News and XSum fingerprints, the signal gives roughly 97 to 100 percent fingerprint success at a false-positive rate of 5 percent or less on unmodified models, and it persists through input perturbation, structural pruning, model merging with low weight on the fingerprinted model, and incremental fine-tuning, often where trigger-based baselines collapse.","pith_inferences":["Editorial inference: the same probability-variation metric that creates the fingerprint can be turned against it—an attacker who ranks candidate strings by the same signal and fine-tunes the model to lower their likelihoods could erase the fingerprint, so the robustness claim likely does not cover that fingerprint-aware attacker.","Editorial inference: choosing private or organization-specific natural-language data as the fingerprint would make removal harder, because deleting those strings would degrade genuinely useful capabilities; the trade-off is that the fingerprint itself is potentially extractable training data, so defenders using sensitive text must weigh privacy.","Editorial inference: if memorization persists through light downstream fine-tuning as the experiments suggest, a natural extension is transferability—the same memorized strings could identify a family of models sharing a pretraining origin, letting one injection trace multiple derivative models; the paper lists this as an open question.","Editorial inference: because verification needs token-level log-probabilities, a strict black-box API that rounds or hides logits would blunt the signal; a testable extension is to check whether the probability-variation signal survives quantization or logit rounding before deployment."],"forward_implications":["A defender with only API access and token-level log-probabilities can trace stolen models that retain memorized private strings, without needing white-box access or predefined trigger responses.","Fingerprint inputs are fluent natural language, so perplexity-based input filters are unlikely to single them out, unlike optimization-based adversarial prompts.","The fingerprint survives generic post-hoc modifications including structural pruning, model merging with small weight on the fingerprinted model, and incremental fine-tuning on instruction data, in settings where backdoor baselines often drop to near zero success.","Reference-model calibration removes frequency-induced false positives, so the same distribution-matched corpus split provides both the fingerprint and the baseline needed for reliable verification.","Because any natural-language corpus can be used as fingerprint data, the approach is compatible with arbitrary private or domain-specific text chosen by the model owner."],"supporting_citations":[{"why":"Supplies the low-rank adaptation procedure used to fine-tune both the fingerprinted and reference models.","marker":"(Hu et al., 2021)"},{"why":"Supplies the self-prompt probability-variation estimator that EverTracer adapts as its verification signal.","marker":"(Fu et al., 2024)"},{"why":"Provides the neighborhood-comparison membership inference insight that member records sit near likelihood maxima.","marker":"(Mattern et al., 2023)"},{"why":"Documents false positives from high-frequency records, motivating the reference-model calibration design.","marker":"(Watson et al., 2022)"},{"why":"Supports the premise that memorization can be necessary for near-optimal learning, justifying persistent injected fingerprints.","marker":"(Feldman, 2020)"},{"why":"Grounds the theoretical foundation that generative models assign higher probability near training samples.","marker":"(van den Burg and Williams, 2021)"},{"why":"Provides the text-to-text model used to generate the semantically perturbed variants for probability-variation estimation.","marker":"(Raffel et al., 2020)"},{"why":"Supplies the XSum corpus used as one of the two fingerprint datasets in the experiments.","marker":"(Narayan et al., 2018)"}],"fun_headline_variants":["Memorized phrases become silent fingerprints that survive pruning","Gray-box LLM fingerprint rides on natural memorization, not triggers","EverTracer turns membership inference into an LLM theft alarm","Stealthy LLM fingerprinting via probability variation on memorized text","Fingerprinting stolen LLMs with memorization, no artificial triggers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The threat model assumes the adversary applies generic modifications such as pruning, fine-tuning on unrelated data, or merging, and never tries to detect and erase the specific memorized fingerprint strings; an attacker who uses the same probability-variation method to identify those strings and then fine-tunes the model to lower their probabilities could remove the signal.","fun_headline_variants_meta":{"raw":{"variants":["Memorized phrases become silent fingerprints that survive pruning","Gray-box LLM fingerprint rides on natural memorization, not triggers","EverTracer turns membership inference into an LLM theft alarm","Stealthy LLM fingerprinting via probability variation on memorized text","Fingerprinting stolen LLMs with memorization, no artificial triggers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3094,"prompt_tokens":934,"completion_tokens":2160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":2072}},"tokens_in":550,"tokens_out":2160,"duration_ms":14884,"temperature":1.0,"reasoning_tokens":2072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:33:34.814506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fingerprinted model, run the same calibrated probability-variation scoring over a wide text pool to identify which fingerprint strings score highest, then apply a fine-tuning or unlearning procedure that specifically lowers log-likelihood on those top-ranked strings; if FSR falls to near zero and AUC to about 0.5 while general task performance stays roughly level, the paper's central robustness claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents false positives from high-frequency records, motivating the reference-model calibration design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounds the theoretical foundation that generative models assign higher probability near training samples."}],"review_version":1}