{"id":"685e0586-3c30-4d9b-ab86-94a33fe176f5","arxiv_id":"2608.08139","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"A token-overlap fingerprint on fixed probes tracks model lineage and shared training data without weight access, with calibrated similarity levels across 32 models.","lead":"The authors introduce a training-free fingerprint that compares language models by the top words they produce on 250 fixed questions, using Jaccard overlap of decoded tokens. It shows the similarity matches documented model relationships across 32 open-weight models and can retrieve the base model of fine-tuned distillations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The shared-base rung of the calibration ladder is computed from only three distill-to-base pairs, excluding documented base-to-instruct pairs in the same pool (e.g., Qwen2.5-32B to Instruct at 0.81), so the headline ladder order may depend on category selection.","rationale":"The reader's ACCEPT is understandable: the method is lightweight, evaluated on a broad pool, and the null exchangeability limitation is explicitly disclosed. However, the central claim is a calibrated similarity ladder with specific numeric values, and that ladder is only meaningful if relationship categories are complete and consistently assigned. The paper's own data contain documented shared-base pairs with much higher similarity, 0.81, than the three distill-to-base pairs used to define the \"fine-tuned from documented base\" rung. Omitting these pairs without a stated uniform assignment rule makes the headline ordering potentially an artifact of category selection. This is not a disagreement with the field's consensus or an attack on the method's mechanism; it is an internal consistency issue in how a headline statistic is computed. The proposed check is a re-derivation with a complete pair-assignment rule. If the re-derived ladder still separates identical-data from shared-base fine-tunes, the current ACCEPT can stand; if not, the paper needs a revised framing, for example treating exact base-to-instruct post-training as its own rung. Because the required re-analysis affects the paper's headline calibration claim, the appropriate disposition is conditional acceptance rather than unchanged acceptance.","tokens_in":22305,"tokens_out":15148,"duration_ms":150817,"concrete_test":"Recompute Table 1 and Figure 2 after placing every pair in the pool whose documented relationship is \"fine-tuned or instruction-tuned from a base checkpoint in the pool\" into the shared-base category, including Qwen2.5-32B to Qwen2.5-32B-Instruct and Llama-3.1-8B to Llama-3.1-8B-Instruct, and pre-register the category-assignment rule before viewing the values. If the shared-base raw mean rises to about 0.48 or above, or the excess mean comes within 0.02 of the identical-data mean, the abstract's ladder ordering is not robust; if it stays below with a stated rationale for excluding base-to-instruct pairs, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing problem is not the null exchangeability assumption but the definition of the \"fine-tuned from documented base\" rung in Table 1 and Figure 2. That row has n=3 and contains only R1 distillation-to-base pairs: R1-Q-32B to Qwen2.5-32B (~0.57), R1-Q-7B to Qwen2.5-Math-7B (~0.31), and R1-L-8B to Llama-3.1-8B (~0.29), averaging 0.39. Yet the same calibration pool contains at least two directly documented shared-base fine-tune pairs that are excluded from this row: Qwen2.5-32B to Qwen2.5-32B-Instruct, reported in Section 4.1 and Figure 3 as the strongest non-identical pair at raw 0.81, and Llama-3.1-8B to Llama-3.1-8B-Instruct. The caption defines the row only as \"the three distill-base pairs,\" without stating why these exact-base fine-tunes are excluded. Because the excluded pairs score far above the three included pairs, the headline ordering \"identical data (0.48) followed by shared-base fine-tunes (0.39)\" depends on which documented fine-tune pairs are admitted. Re-including the two base-to-instruct pairs moves the shared-base mean to roughly 0.45-0.50, erasing or inverting the gap to the identical-data rung. The excess/corrected values inherit the same selection. This is an internal consistency issue in the central calibration deliverable, not merely a matter of null-model uncertainty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TokenPrint proposes a training-free token-space fingerprint for language-model provenance: for each of 250 fixed knowledge probes, the top-20 decoded tokens are read out from late hidden states (L/4, L/2, 3L/4, L-2) and from the output distribution via the logit lens, and pairwise similarity is the mean Jaccard overlap of these decoded token sets, with a mismatched-probe null subtracted to obtain an 'excess' score. On a 32-model calibration pool with documented relationships, the paper reports a similarity ladder (identical-data 0.48 raw/0.35 excess; fine-tuned from documented base 0.39/0.33; same lineage group 0.38/0.28; same-developer next generation 0.37/0.25; unrelated 0.22/0.17), same-group vs. cross-group AUC of 0.859 at near-final depth, top-2 retrieval of the documented base for all five R1 distillations, a same-corpus witness signal across organizations, tokenizers, and architecture classes (excess 0.35-0.43 vs. a 0.15 reference), emergence of the signal within the first 1% of Pythia training before measurable factual competence, and stability under pool perturbations, probe removal, top-k choice, and int8/int4 quantization. The paper argues that token-space similarity tracks documented training relatedness rather than capability alone, while explicitly disclaiming mechanism identification.","tokens_in":22579,"tokens_out":26908,"duration_ms":237198,"significance":"If the main claims hold, TokenPrint is a valuable, low-cost provenance-screening tool: extraction is one forward pass per probe with no gradients, the released probes/code/fingerprints make the results independently checkable, and the evaluation is unusually careful in several respects. The probe suite was fixed before any cross-model similarity was computed (§3.2, Appendix B); the glitch-token filter is shown to be numerically inert; significance is assessed at the model level via permutation (p<10^-4); the robustness section includes leave-one-lineage-group-out (AUC 0.820-0.903), block removals, leave-one-model-out jackknife, a capability/scale confound regression (Eq. 1), and quantized-checkpoint stability; and Appendix D's Proposition 2 honestly delimits that behavioral scores cannot certify the mechanism of an observed relationship. The most novel scientific observation is the early-training emergence (Figure 4): same-data excess similarity of 0.25 at step 1000 while factual probe accuracy is 0.02. The principal weakness I find is in the calibration-ladder category assignment (detailed in the major comment); the broad three-regime conclusion (strongly related vs.","major_comments":[{"comment":"The headline ordering of the calibration ladder is not robust to how 'fine-tuned from documented base' is defined, because that row of Table 1 is populated by an unstated pair-assignment rule. The row is assigned only the three cross-lineage R1-distill to base pairs (R1-Q-32B to Qwen2.5-32B, R1-Q-7B to Qwen2.5-Math-7B, R1-L-8B to Llama-3.1-8B), even though the same calibration pool contains two documented same-developer base-to-Instruct fine-tune pairs: Qwen2.5-32B to Qwen2.5-32B-Instruct, reported as the strongest non-identical pair in the pool at 0.81 (Figure 3), and Llama-3.1-8B to Llama-3.1-8B-Instruct, which is in the pool and is used in §4.5 as a 'same-base Instruct variant.' Because the rows are stated to cover 490 of the 496 pairs and the 'Same developer, next generation' row accounts for exactly the 30 Qwen3 times Qwen2.5 pairs, the two base-to-Instruct pairs must therefore be counted in the 'Same lineage group, different size' row (n=36), despite that row's 'different size' label. Under the alternative and equally natural assignment in which every documented shared-base fine-tune populates the fine-tuned rung, the raw rung mean becomes (3x0.39 + 0.81 + s)/5, which exceeds 0.45 and reaches or surpasses the identical-data mean of 0.48 for plausible values of the un-reported Llama base-to-Instruct score s; the excess-score rung is affected even more strongly, since the Qwen pair's excess is roughly 0.68 after subtracting the Qwen2.5 null of 0.128 (Table 7). The abstract's claim that 'independently trained models on identical data score 0.48 ... followed by shared-base fine-tunes (0.39/0.33)' is therefore a consequence of admitting only the three distill-to-base pairs. Please state the pair-assignment rule explicitly, relabel or split the rows (for example, 'cross-developer distillation from documented base' versus 'same-developer post-training from base'), report the Llama base-to-Instruct similarity, and present the ladder under both assignments. The three-regime conclusion in §4.1 survives; the specific top-rung ordering does not.","section":"§4.1, Table 1, Figure 3"}],"minor_comments":[{"comment":"The probe-bootstrap 95% CI (+/-0.013) and the §4.1 CI for the same-developer difference ([+0.010,+0.019]) resample the 250 probes while holding model pairs fixed; for rungs with n=3 pairs this materially overstates precision. Please add model-level (cluster) bootstrap or permutation-based intervals for the category means.","section":"Table 1 footnote, §4.1"},{"comment":"The excess-score interpretation rests on the exchangeability of matched and mismatched probe pairings under content-independent top-k sets; the assumption is disclosed but never directly tested. A direct validation (for example, showing that excess is near zero for models with matched vocabularies but unrelated training, or comparing the null against a probe-content-scrambled control) would firm up the corrected ladder values. Because all main conclusions also hold for raw scores, I regard this as a robustness gap rather than a correctness error.","section":"Appendix D (§3.3)"},{"comment":"The lineage-retrieval headline (mean rank 1.8, MRR 0.60) is based on five queries, all from the DeepSeek R1 distillation family; a single query change moves MRR by 0.2. The limitation is acknowledged in §6, but the abstract should carry the n=5 context or an interval.","section":"§4.5, Table 4"},{"comment":"The early-training claim rests on a single model pair (Pythia-1.4B vs. 6.9B). Reporting the pairwise trajectories of all three Pythia sizes, and the deduped-suite checkpoints if available, would show whether the step-1000 signal is pair-specific.","section":"§4.2, Figure 4"},{"comment":"The 'no documented relationship' reference (0.22/0.17) is computed within the calibration pool, where absence of a documented relationship does not imply absence of corpus overlap; the control analysis itself shows GPT-2 to OPT at 0.40 and OPT to Pythia at 0.34-0.37 with documented corpus overlap (§5). The floor is best read as an upper bound on a truly unrelated baseline, so absolute gap sizes should be interpreted accordingly.","section":"§5"},{"comment":"The 'dip' annotation at half depth (AUC 0.72 then 0.70 then 0.77) is unexplained in the caption; either briefly account for the non-monotonicity or remove the annotation.","section":"Figure 5"},{"comment":"Small typographical issues: 'PMlR' in the Kornblith et al. entry should be 'PMLR', and the reference title 'Huref: Human-readable fingerprint for large language models' is inconsistent with the body text's 'HuRef.'","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the reader's accept is defensible in spirit, since the empirical work is careful and honest, but the top-rung ordering claim in the abstract and §4.1 needs a transparency fix before publication. I am not concerned about circularity: the probes were fixed up front, and the validation targets externally documented relationships. The main residual risks after revision are scope (the lineage-retrieval evaluation is tiny) and the cleanliness of the unrelated floor. If the authors adopt the category-sensitivity revision and report the ladder under both assignment rules, I would support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTokenPrint is a genuinely useful addition to the model-provenance toolkit: a training-free, forward-pass-only fingerprint based on top-k decoded token overlap on 250 fixed probes, with a mismatched-probe null to correct for vocabulary effects. The evaluation is careful—permutation tests at the model level, leave-one-group-out, jackknife, external witness models across organizations, tokenizers, and architectures, and a capability-confound regression. The lineage-retrieval result (base in top-2 for all five R1 distillations) and the early-training signal (excess similarity 0.25 at step 1000 with probe accuracy 0.02) are both solid. The paper also openly acknowledges that behavioral scores cannot distinguish weight inheritance from corpus overlap (Proposition 2), which is the right kind of honesty.\n\nThe soft spot is not the null exchangeability assumption—that is disclosed and reasonable. It is the definition of the \"fine-tuned from documented base\" rung in Table 1. As defined, it contains only the three distill↔base pairs, averaging 0.39. But the same pool contains documented base→instruct fine-tunes (Qwen2.5-32B→Instruct at 0.81, Llama-3.1-8B→Instruct), which are excluded without justification. If you include them, the shared-base mean rises to roughly 0.46–0.50, which erases or inverts the gap to the identical-data rung (0.48). The caption says \"the three distill↔base pairs,\" but the category label says \"fine-tuned from documented base.\" That is an internal consistency problem in the central calibration deliverable, and it needs fixing—either redefine the category or justify the exclusion. This does not sink the paper's other contributions, but the headline ladder is not as robust as it looks.\n\nSecondary, smaller gaps: lineage retrieval covers five distillations from one ecosystem; the same-data early-training evidence is limited to Pythia checkpoints; and the exchangeability assumption, while disclosed, is not directly tested. These are minor.\n\nThis paper deserves a serious referee. The method is cheap, reproducible (they ship probes, code, and fingerprints), and the core findings are likely to hold up after the category issue is addressed. I would bring it to a reading group and would cite it, but I would want the ladder fixed in the version I cite.","headline":"A useful calibrated fingerprint for model provenance, with a real selection problem in the headline similarity ladder that needs fixing.","tokens_in":23189,"tokens_out":2578,"would_cite":true,"duration_ms":24328,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free fingerprint of top-20 decoded token sets on 250 probes orders models by documented relatedness and retrieves the base checkpoint of five DeepSeek-R1 distillations.","keywords":["language model provenance","token-space fingerprint","Jaccard similarity","lineage retrieval","training distribution overlap","top-k tokens","model forensics","quantization robustness"],"falsifier":"Train two models from scratch on the same corpus but with deliberately different tokenizers, and two models on disjoint corpora with the same tokenizer; if the same-corpus/different-tokenizer pair shows excess similarity at the unrelated baseline (≈0.17) rather than near the reported 0.35, the claim that the fingerprint tracks shared training data rather than vocabulary overlap is falsified. Alternatively, explicitly test the exchangeability assumption by comparing the null estimated from content-independent prompts with the mismatched-probe null; a systematic difference would invalidate the corrected ladder.","tokens_in":22008,"feed_emoji":"🧬","tokens_out":11159,"duration_ms":93670,"temperature":0.7,"pith_summary":"The paper sets out to show that a cheap, training-free fingerprint—the set of top-20 decoded token strings a model produces on 250 fixed knowledge probes, read from late internal layers—carries a graded signal of how models are related. Across 32 open-weight models with documented relationships, fingerprint similarity forms a ladder: models trained independently on identical data look most alike, followed by models fine-tuned from a shared base, same-developer relatives, and finally unrelated models. The signal persists across tokenizers, architectures, and training organizations; appears within the first 1% of training, before the models can answer the probes correctly; and, as a nearest-neighbor method, ranks the documented base checkpoint first or second for all five DeepSeek-R1 distillations tested. The authors argue that token-choice behavior, calibrated against known relationships, can generate provenance hypotheses about a model's base and training-data overlap without needing access to weights, gradients, or training logs.","feed_headline":"Token fingerprints trace a model's training ancestry","feed_subtitle":"Top-20 token sets on 250 probes rank identical-data pairs above unrelated ones.","key_machinery":"The central object is a token-space fingerprint: for each of 250 fixed probes, the model's top-20 decoded token strings, obtained by applying the output head to late hidden states (default depth $L-2$, plus the final output distribution). Pairwise similarity is the mean Jaccard overlap of these token sets across probes. Because globally frequent tokens can inflate overlap regardless of probe content, the paper subtracts a pair-specific mismatched-probe null—model a's response to probe i paired with model b's response to a different probe j—to obtain an excess similarity that isolates probe-content-locked overlap. This calibration converts token-choice behavior into a graded relatedness score, which the ladder and lineage-retrieval results then validate against documented model relationships.","core_discovery":"The central claim is that fingerprint similarity, defined as the mean Jaccard overlap of top-20 decoded token strings at near-final depth on 250 fixed probes, tracks documented model relatedness. Identical-data pairs (Pythia models with no shared weights) average 0.48 raw and 0.35 vocabulary-corrected excess similarity; shared-base fine-tunes average 0.39/0.33; same-developer pairs 0.37–0.38/0.25–0.28; and pairs with no documented relationship 0.22/0.17. This ordering yields a same-group versus cross-group ROC AUC of 0.859 at near-final depth, rising to 0.90 at the final output distribution, and, using only the top 5 output tokens, 0.87. As lineage retrieval, the fingerprint places the exact documented base in the top two of 33 candidates for all five R1 distillations (mean rank 1.8, MRR 0.60), including a math-specialized base that coarse metadata does not identify. The paper also reports that the identical-data signal emerges within the first 1% of training while probe accuracy is still near zero, and that the fingerprint is stable under quantization: Jaccard 0.92 under int8 and 0.82–0.85 under int4, above the largest cross-model similarity (0.81) in the calibration pool.","pith_inferences":["Because the score depends only on token rank order, the fingerprint could transfer to black-box APIs that expose ranked top-k tokens without log-probabilities, an extension the paper notes in principle but does not evaluate.","The category distributions overlap, so the ladder is calibrated at the pool level; turning the method into a pair-level decision tool would require explicit thresholds or abstention rules, which the paper leaves as future work.","The early-emergence result suggests token-space overlap might detect corpus reuse or contamination independently of task accuracy, a use case beyond provenance.","The exact-parent versus sibling ambiguity in four of five retrievals might be reduced by rank-weighted Jaccard or by combining the token-space score with weight-space evidence when weights are available."],"forward_implications":["Provenance triage becomes possible with forward passes alone: 250 probes per model, no training, no gradients, no weight access.","A model's base checkpoint can be recovered or narrowed to a small lineage neighborhood even when metadata is missing or misleading, as in the math-specialized 7B case.","The fingerprint detects overlap in training corpora even without shared parameters, extending forensics to independently trained models that reuse data.","The signal's presence before capability emerges means early training checkpoints already carry lineage information, useful for auditing partially released models.","Quantization robustness (int8 stable, int4 usable) means deployed quantized models can still be fingerprinted without dequantizing."],"supporting_citations":[{"why":"supplies the Pythia suite, the identical-data, no-shared-weights pairs that anchor the top of the similarity ladder.","marker":"Biderman et al., 2023"},{"why":"documents the five DeepSeek-R1 distillations and their base checkpoints, the ground truth for lineage retrieval.","marker":"Guo et al., 2025"},{"why":"defines the Pile corpus that the external witness models share with Pythia.","marker":"Gao et al., 2020"},{"why":"provides GPT-NeoX-20B as a same-corpus, different-project witness.","marker":"Black et al., 2022"},{"why":"provides Cerebras-GPT as the same-corpus witness with a different tokenizer, organization, and recipe.","marker":"Dey et al., 2023"},{"why":"provides RWKV-4-Pile as the same-corpus witness with a different architecture class.","marker":"Peng et al., 2023"},{"why":"supplies the probability-overlap baseline that the fingerprint is compared against on the same pool.","marker":"Yax et al., 2025"},{"why":"underpins the tuned-lens projection that reads token distributions from late hidden states.","marker":"Belrose et al., 2023"}],"fun_headline_variants":["Training-free token prints reveal a model's lineage","Fingerprinting LLMs without retraining: token overlap does it","Token overlap on 250 probes identifies a model's base","Provenance from token sets: how to trace an LLM's roots"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's corrected scores rest on the assumption that, when a model's top-k tokens do not depend on the probe, matched and mismatched probe pairings are equally likely to overlap, so the mismatched-probe null cleanly isolates probe-content-locked similarity; the documented relationships in the model cards are likewise assumed to be accurate ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Training-free token prints reveal a model's lineage","Fingerprinting LLMs without retraining: token overlap does it","Token overlap on 250 probes identifies a model's base","Provenance from token sets: how to trace an LLM's roots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2243,"prompt_tokens":1220,"completion_tokens":1023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":836,"completion_tokens_details":{"reasoning_tokens":952}},"tokens_in":836,"tokens_out":1023,"duration_ms":8872,"temperature":1.0,"reasoning_tokens":952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:20:41.097110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two models from scratch on the same corpus but with deliberately different tokenizers, and two models on disjoint corpora with the same tokenizer; if the same-corpus/different-tokenizer pair shows excess similarity at the unrelated baseline (≈0.17) rather than near the reported 0.35, the claim that the fingerprint tracks shared training data rather than vocabulary overlap is falsified. Alternatively, explicitly test the exchangeability assumption by comparing the null estimated from content-independent prompts with the mismatched-probe null; a systematic difference would invalidate the corrected ladder.","supporting_citations":[],"review_version":1}