REVIEW 3 major objections 6 minor 43 references
Stemma: Induced Decision Regions Reveal LLM Provenance
T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A language model's private decision regions — the choices it makes when responses are mapped to a finite answer set — are inherited by its fine-tunes, adapters, merges, quantisations, and distillations, and this inheritance lets a black-box
desk verdict Stemma is a genuinely useful provenance-fingerprinting paper whose headline results are strong but rest on unverified model-card lineage labels; worth serious review with a demand for label validation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Induced decision regions: for a finite decision space D (here, the answer options of multiple-choice questions), a model's discrete decision map h_M(q) assigns each query to a region R^M_d, partitioning the query space into a decision geometry. The carrier of the argument is the empirical claim that this geometry is inherited — descendants of a source preserve region assignments more often than unrelated models, even when response surface forms shift. Three probe-selection principles make the inheritance observable: stability (the source decision persists across cyclic option permutations), robustness (the decision sits well inside its region, with a large log-probability margin over the run
What would settle it
Take model pairs whose derivation is certified by independent training logs or weight-difference analysis rather than model cards, hold out one model family, and measure Stemma's alignment scores on those certified pairs: if AUC drops well below 0.967, the headline number rests on unverified labels. Alternatively, fine-tune a suspect to minimise agreement on the source's recorded non-gold probes and check whether the alignment score falls; if it stays high, the non-adaptive assumption is doing the work.
Extended reading notes
Core claim
The central claim is that induced decision region inheritance is a reliable provenance signal. For a model M and finite decision space D, the discrete decision map h_M(q) partitions the query space into induced decision regions R^M_d = {q : h_M(q)=d}; provenance is measured by how much of S's region geometry survives in T. Stemma makes this observable with a finite probe set: it calibrates a prompt template, selects questions whose source decisions are stable under cyclic option permutations, lie well inside their region, and disagree with both the gold answer and a background model, then computes fingerprint alignment A(S,T), the fraction of probe–permutation observations that preserve the
Load-bearing premise
The benchmark's provenance labels are taken at face value from public model-card documentation — every checkpoint labelled a fine-tune, adapter, merge, quantisation, or distillation of a source is genuinely derived from that source, and cross-group checkpoints are genuinely unrelated; if any label is wrong, the reported AUC values partly measure mislabelled pairs.
Editorial extensions
If this is right
- Provenance testing can be conducted with black-box query access only — no weights, activations, logits, training data, or development history from the suspect.
- The signal survives interface mismatch: source fingerprints created under raw completion prompts still separate suspects queried through chat templates, where response-based baselines degrade toward random.
- The method stays near-perfect across deployment variations — system prompts, role-play, chain-of-thought, retrieval augmentation, and decoding changes — because decisions abstract away surface form.
- Ablations show the selection principles matter: random probes give 0.751 AUC, the full stability–robustness–specificity ranking reaches 0.967 AUC, with pAUC jumping from 0.614 to 0.944.
- One-time fingerprint construction takes roughly 4.5 minutes and verification 38 seconds, making the approach practical for audits of many suspects.
Reading between the lines
- The paper instantiates induced decision regions with multiple-choice questions, but the formalism only requires a mapping from open-ended outputs to a finite decision space; refusal/escape categories, classification-style outputs, or verifier-defined behavioural tests are natural extensions that should carry the same inheritance signal.
- The deliberate focus on stable, non-gold, background-specific decisions suggests that a model's idiosyncratic confident errors are its most diagnostic fingerprints — a property worth testing on privately constructed or adversarially assembled question pools, where an adaptive suspect would not know the probes.
- If the reported persistence holds across families, provenance scores could be paired with licensing audits: a small set of source-elicited probes queried through a suspect API would give statistical lineage evidence even when the suspect is served behind a deployment wrapper or retrieval pipeline.
- The stated non-adaptive threat model leaves a direct test standing: fine-tune a suspect to avoid agreement on the public probe distribution and measure how much the alignment score drops; the authors note this risk, and the outcome would bound the method's real-world evasion resistance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces induced decision regions as a formalization for black-box LLM provenance testing. The key idea is to map open-ended model outputs into a finite decision space, instantiated here with multiple-choice questions, so that provenance is assessed by whether a suspect model preserves the source's decision-region assignments rather than by surface-form response similarity. Stemma operationalizes this by selecting probes on the source model that are stable under answer-option permutations, robust in log-probability margin, and specific against a background model, then scores a suspect by the alignment of source decisions across cyclic permutations. The authors report AUC 0.967, pAUC 0.944, and TPR 0.878 at 1% FPR across 770 source-suspect pairs from 56 public checkpoints, and 0.995 AUC on 1,260 deployment-variant pairs, outperforming four black-box baselines in the main setting. The paper includes extensive ablations on probe selection, pool size, probe budget, probe dataset, and background model, and releases an open-source implementation.
Significance. If the empirical claims hold, this is a useful contribution to black-box model provenance testing. The formalization is clean and it leads to a concrete, practical method with modest query cost (40 probes times 4 permutations) and strong separation in the evaluated benchmark. The paper is unusually thorough in its ablations, and the release of an open-source implementation is valuable for reproducibility. The main caveats are that the benchmark's ground-truth lineage labels are taken from model-card metadata without independent verification, and that the headline numbers are single-run point estimates. A control setting (all-raw prompts) also shows that one baseline (MPT) achieves higher overall AUC than Stemma when the prompt-format mismatch is removed, which tempers the claim of superiority over baselines. These issues are addressable with additional verification and reporting, but they affect the strength of the central empirical claim.
major comments (3)
- [§5.1 / Appendix D (Table 3)] The provenance labels that define the 770 pairs are taken from Hugging Face model cards and repository names without independent verification. Several checkpoints labelled 'Distillation' appear to name a different teacher than the assigned group source (e.g., 'prithivMLmods/Regulus-Qwen3-R1-Llama-Distill-1.7B', 'eganwo/mistral7b-distilled-from-deepseek-r1-qwen32b', 'RedaAlami/Falcon3-7B-Instruct-Distill-DS-v1'). If these positive pairs are not true derivatives, or if cross-group pairs are secretly related, the reported AUC/pAUC/TPR are computed against mislabelled ground truth. No label audit or sensitivity analysis to label perturbation is reported. Please verify lineage from README/training records or add a perturbation analysis and report metrics on a verified subset.
- [§5.2 (Table 4)] In the all-raw control setting, MPT achieves higher overall AUC (0.988 vs 0.965) and higher pretrained-source AUC (0.993 vs 0.968) than Stemma; Stemma has better pAUC (0.952 vs 0.938) and TPR at 1% FPR (0.898 vs 0.816). The manuscript describes Stemma as 'competitive' here, but this result directly qualifies the abstract's claim that Stemma 'substantially outperforms four representative baselines.' Because the all-raw setting removes the prompt-format mismatch that explains baseline failure in the main benchmark, this control should be reported prominently and the comparative claim should be rephrased or explained.
- [§5.2–§5.3 / Appendix I] All headline results are single-run point estimates. Generation uses stochastic decoding (temperature 0.7, top-p 0.9), and Appendix I attributes small cross-run differences to stochastic decoding and hardware nondeterminism. No error bars, confidence intervals, or significance tests are reported for the 14-source comparisons, so it is unclear whether differences such as Stemma's 0.944 pAUC vs MPT's 0.613 (Table 1) are stable. Please report results over multiple seeds/fingerprint draws with intervals, or provide bootstrap CIs over source-suspect pairs.
minor comments (6)
- [Appendix D, Table 3] Repository ID 'Qwen/Qwen3-1.7B' is listed with model type 'Instruct'; if this is intended to be the instruct checkpoint, the ID should be 'Qwen/Qwen3-1.7B-Instruct'. Please check that all repository IDs match the listed model types.
- [Figure 1] The figure would benefit from explicit axis labels and a legend. The caption defines the green/orange coloring, but the rendered figure has no visible legend, making it hard to interpret without referring back to the caption.
- [Table 4] The formatting of the MPT row is broken (e.g., 'MPT0.9930.946 0.878 3.258 0.991 0.972 0.939 3.333 0.9880.938 0.816 2.713'). Please fix the spacing and bold/underline markers.
- [Equation (7)] Treating invalid outcomes as non-aligned means that a suspect that systematically fails to produce parseable option labels will receive a lower score. This is a reasonable choice, but it should be stated as a limitation or at least discussed in the verification stage, since it interacts with prompt calibration quality.
- [Appendix C] The exact filtering thresholds and ranking weights (0.5, 0, 0.5, 1.0, 0.5) are only given in prose. A small configuration table would improve reproducibility and make the method easier to reimplement.
- [§6 / Limitations] The limitations discussion does not mention that the empirical evaluation is limited to models in the 1.7B–14B range. Given the paper's general title and conclusion, this scope restriction should be stated explicitly.
Circularity Check
No significant circularity: Stemma's probe selection uses only source and background information, and the headline results are empirical measurements over held-out external checkpoints, not consequences of fitted inputs.
full rationale
Walking the paper's derivation chain: induced decision regions are defined in Eqs. (1)-(2) from a score extraction rule and a finite decision space; inheritance is defined in Eqs. (3)-(4) and (7). Probe selection (Section 4.3) uses source-model next-token logits, cyclic-permutation stability, log-probability margin robustness, non-gold decisions, and a background-model reproduction rate. None of these quantities involves the suspect model or the provenance labels, and the background model (Phi-3.5-mini-instruct) is not one of the 56 benchmark checkpoints, so negative-pair scores are not forced by construction. The fingerprint alignment score in Eq. (7) compares suspect decisions with recorded source decisions, which is the intended measurement rather than a hidden reuse of the target result. The headline AUC/TPR claims are empirical evaluations over 770 source-suspect pairs drawn from public checkpoints, with ablations showing that random probes already reveal the signal and that each selection stage adds discriminative power. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter that is renamed as a prediction. The skeptic's concern about Hugging Face model-card provenance labels is a ground-truth validity and robustness issue, not circularity of the derivation; likewise, the paper's stated limitation about adaptive attackers is an external threat-model concern. The derivation is therefore self-contained against external benchmarks, and the honest finding is no significant circularity.
Assumptions & free parameters
free parameters (6)
- stability threshold tau_s =
0.5
- robustness threshold tau_r =
0
- background alignment threshold tau_b =
0.5
- ranking weights (w_r, w_b) =
1.0, -0.5
- fingerprint size K =
40
- candidate pool size =
3000
assumptions (5)
- domain assumption The suspect model is accessible only through a black-box text interface that returns generated responses, with no logits or weights exposed.
- domain assumption The provider is non-adaptive: neither the model nor its interface is modified to detect or evade fingerprint queries.
- domain assumption The evaluated models can reliably process multiple-choice questions and produce parseable option labels.
- domain assumption The provenance labels of the benchmark checkpoints (derived vs unrelated) are correct.
- domain assumption A single unrelated background model (Phi-3.5-mini-instruct) provides a reliable estimate of how uncommon a source decision is among all unrelated models.
invented entities (1)
-
induced decision region
Cite this review
Pith. "Pith review of Stemma: Induced Decision Regions Reveal LLM Provenance." pith.science (2026). https://pith.science/paper/2EBLYPPU
@misc{pith2026260725880,
author = {Pith},
title = {Pith review of: Stemma: Induced Decision Regions Reveal LLM Provenance},
year = {2026},
howpublished = {\url{https://pith.science/paper/2EBLYPPU}},
note = {Machine review of arXiv:2607.25880}
}
read the original abstract
LLM provenance testing asks whether a suspect LLM belongs to the same lineage as a source. Existing black-box methods largely infer this relationship from response-level characteristics, but these characteristics may shift under adaptation or deployment even when the underlying meaning remains unchanged, weakening the reliability of provenance evidence. To address this limitation, we introduce induced decision regions by mapping open-ended outputs into a finite decision space, thereby abstracting away surface-form variation and reframing provenance testing as measuring the inheritance of decision regions. Empirical analysis shows that the source's induced regions are preserved more strongly in related models than in unrelated models. Building on this signal, we propose Stemma, a practical black-box LLM fingerprinting method that operationalises stability, robustness, and specificity as complementary probe-selection principles for reliably estimating induced decision region inheritance. Across 770 source-suspect pairs drawn from 56 public checkpoints and spanning diverse model-weight transformations, Stemma achieves 0.967 AUC and 87.8% TPR at 1% FPR, substantially outperforming four representative baselines. It further achieves 0.995 AUC and 93.5% TPR at 1% FPR on 1,260 pairs covering 91 deployment instances, demonstrating robustness to diverse inference-time deployment settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Turning Your Weakness into a Strength: Watermarking Deep Neural Networks by Backdooring , booktitle =
Adi, Yossi and Baum, Carsten and Cisse, Moustapha and Pinkas, Benny and Keshet, Joseph , year = 2018, series =. Turning Your Weakness into a Strength: Watermarking Deep Neural Networks by Backdooring , booktitle =
2018
-
[2]
, year = 2016, month = sep, journal =
Andrews, Tara L. , year = 2016, month = sep, journal =. Analysis of Variation Significance in Artificial Traditions Using. doi:10.1093/llc/fqu072 , urldate =
-
[3]
Cao, Xiaoyu and Jia, Jinyuan and Gong, Neil Zhenqiang , year = 2021, month = jun, series =. Proceedings of the 2021. doi:10.1145/3433210.3437526 , urldate =
arXiv 2021
-
[4]
Advances in
Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , year = 2023, volume =. Advances in
2023
-
[5]
KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing
Fang, Yijia and Feng, Yiqing and Li, Bingyu and Zhou, Mingxun , year = 2026, month = may, number =. doi:10.48550/arXiv.2605.29524 , urldate =. 2605.29524 , primaryclass =
work page Pith review arXiv doi:10.48550/arxiv.2605.29524 2026
-
[6]
Gloaguen, Thibaud and Staab, Robin and Jovanovi. The
-
[7]
Queries, Representation & Detection: The next 100 Model Fingerprinting Schemes , shorttitle =
Godinot, Augustin and Le Merrer, Erwan and Penzo, Camilla and Ta. Queries, Representation & Detection: The next 100 Model Fingerprinting Schemes , shorttitle =. Proceedings of the. doi:10.1609/aaai.v39i16.33848 , urldate =
-
[8]
Guan, Jiyang and Liang, Jian and He, Ran , year = 2022, volume =. Are. Advances in
2022
Show all 43 references
-
[9]
Findings of the
Gubri, Martin and Ulmer, Dennis and Lee, Hwaran and Yun, Sangdoo and Oh, Seong Joon , editor =. Findings of the. doi:10.18653/v1/2024.findings-acl.683 , urldate =
2024 doi
-
[10]
Hammoud, Hasan Abed Al Kader and Michieli, Umberto and Pizzati, Fabio and Torr, Philip and Bibi, Adel and Ghanem, Bernard and Ozay, Mete , editor =. Model. Findings of the. doi:10.18653/v1/2024.findings-emnlp.762 , urldate =
2024 doi
-
[11]
Thematic
Heikkil. Thematic. Digital Scholarship in the Humanities , volume =. doi:10.1093/llc/fqw038 , urldate =
-
[12]
Measuring
Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , year = 2020, month = oct, urldate =. Measuring. International
2020
-
[13]
Fingerprinting
Hu, Yuepeng and Jiang, Zhengyuan and Li, Mengyuan and Ahmed, Osama and Huang, Zhicong and Hong, Cheng and Gong, Neil Zhenqiang , editor =. Fingerprinting. Proceedings of the 64th. doi:10.18653/v1/2026.acl-long.541 , urldate =
2026 doi
-
[14]
Hugging Face , urldate =
Hugging Face Hub , author =. Hugging Face , urldate =
-
[15]
Entangled
Jia, Hengrui and. Entangled. 30th
-
[16]
Engineering Applications of Artificial Intelligence , volume =
Intellectual Property Protection for Deep Learning Model and Dataset Intelligence , author =. Engineering Applications of Artificial Intelligence , volume =. doi:10.1016/j.engappai.2025.113024 , urldate =
2025
-
[17]
Thomas , year = 2024, month = sep, pages =
Jin, Heng and Zhang, Chaoyu and Shi, Shanghao and Lou, Wenjing and Hou, Y. Thomas , year = 2024, month = sep, pages =. 2024. doi:10.1109/CNS62487.2024.10735575 , urldate =
2024
-
[18]
Kojima, Takeshi and Gu, Shixiang (Shane) and Reid, Machel and Matsuo, Yutaka and Iwasawa, Yusuke , year = 2022, month = dec, journal =. Large
2022
-
[19]
Watermarking
Li, Linyang and Jiang, Botian and Wang, Pengyu and Ren, Ke and Yan, Hang and Qiu, Xipeng , editor =. Watermarking. Findings of the. doi:10.18653/v1/2023.findings-emnlp.220 , urldate =
2023 doi
-
[20]
Nikolic, Ivica and Baluta, Teodora and Saxena, Prateek , year = 2025, volume =. Model. Advances in
2025
-
[21]
and Ateniese, Giuseppe , year = 2025, pages =
Pasquini, Dario and Kornaropoulos, Evgenios M. and Ateniese, Giuseppe , year = 2025, pages =. 34th
2025
-
[22]
Pezeshkpour, Pouya and Hruschka, Estevam , editor =. Large. Findings of the. doi:10.18653/v1/2024.findings-naacl.130 , urldate =
2024 doi
-
[23]
Reading between the Lines:
Shao, Shuo and Li, Yiming and Yao, Hongwei and Chen, Yifei and Yang, Yuchen and Qin, Zhan , year = 2026, series =. Reading between the Lines:. Proceedings of the. doi:10.1145/3774904.3792196 , isbn =
2026
-
[24]
doi:10.48550/arXiv.2508.19843 , urldate =
Shao, Shuo and Li, Yiming and He, Yu and Yao, Hongwei and Yang, Wenyuan and Tao, Dacheng and Qin, Zhan , year = 2025, month = nov, number =. doi:10.48550/arXiv.2508.19843 , urldate =. 2508.19843 , primaryclass =
2025 doi
-
[25]
Shi, Chufan and Yang, Haoran and Cai, Deng and Zhang, Zhisong and Wang, Yifan and Yang, Yujiu and Lam, Wai , editor =. A. Proceedings of the 2024. doi:10.18653/v1/2024.emnlp-main.489 , urldate =
2024 doi
-
[26]
and Di Penta, Massimiliano and German, Daniel M and Poshyvanyk, Denys , year = 2025, month = nov, journal =
Stalnaker, Trevor and Wintersgill, Nathan and Chaparro, Oscar and Heymann, Laura A. and Di Penta, Massimiliano and German, Daniel M and Poshyvanyk, Denys , year = 2025, month = nov, journal =. An. doi:10.1145/3776739 , urldate =
2025 doi
-
[27]
Tang, Chen and Zhang, Lan and Zhao, Qi and Zhuang, Xirong and Li, Xiang-Yang , year = 2025, month = apr, journal =. Model. doi:10.1609/aaai.v39i19.34292 , urldate =
2025 doi
-
[28]
Llama 2:
Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and Bikel, Dan and Blecher, Lukas and Ferrer, Cristian Canton and Chen, Moya and Cucu...
- [29]
-
[30]
Embedding
Uchida, Yusuke and Nagai, Yuki and Sakazawa, Shigeyuki and Satoh, Shin'ichi , year = 2017, month = jun, series =. Embedding. Proceedings of the 2017. doi:10.1145/3078971.3078974 , urldate =
2017
-
[31]
Plan-and-
Wang, Lei and Xu, Wanyu and Lan, Yihuai and Hu, Zhiqiang and Lan, Yunshi and Lee, Roy Ka-Wei and Lim, Ee-Peng , editor =. Plan-and-. Proceedings of the 61st. doi:10.18653/v1/2023.acl-long.147 , urldate =
2023 doi
-
[32]
Wang, Noah and Peng, Z.y. and Que, Haoran and Liu, Jiaheng and Zhou, Wangchunshu and Wu, Yuhan and Guo, Hongcheng and Gan, Ruitong and Ni, Zehao and Yang, Jian and Zhang, Man and Zhang, Zhaoxiang and Ouyang, Wanli and Xu, Ke and Huang, Wenhao and Fu, Jie and Peng, Junran , edi...
2024 doi
-
[33]
Xu, Tianlong and Wang, Chen and Liu, Gaoyang and Yang, Yang and Peng, Kai and Liu, Wei , year = 2024, volume =. United. Advances in. doi:10.52202/079017-2213 , urldate =
2024 doi
-
[34]
Findings of the
Yan, Yuliang and Tang, Haochun and Yan, Shuo and Dai, Enyan , editor =. Findings of the. doi:10.18653/v1/2026.findings-eacl.273 , urldate =
2026 doi
-
[35]
Securing
Ye, Peigen and Ren, Huali and Li, Zhengdao and Yan, Anli and Yan, Hongyang and Wang, Shaowei and Li, Jin , year = 2026, month = may, journal =. Securing. doi:10.1145/3773028 , urldate =
2026 doi
-
[36]
Language
Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin , year = 2024, month = jul, pages =. Language. Proceedings of the 41st
2024
-
[37]
Yu, Runpeng and Wang, Xinchao , year = 2024, month = jun, pages =. Neural. 2024. doi:10.1109/CVPR52733.2024.00459 , urldate =
2024
-
[38]
Advances in
Zang, Chuanchao and Meng, Xiangtao and Chen, Wenyu and Cong, Tianshuo and Yaxing, Zha and Qi, Dong and Li, Zheng and Guo, Shanqing , year = 2025, volume =. Advances in
2025
-
[39]
Proceedings of the 2024 on
Zhang, Jinghuai and Chi, Jianfeng and Li, Zheng and Cai, Kunlin and Zhang, Yang and Tian, Yuan , year = 2024, series =. Proceedings of the 2024 on. doi:10.1145/3658644.3690284 , isbn =
2024
-
[40]
Zhang, Jie and Liu, Dongrui and Qian, Chen and Zhang, Linfeng and Liu, Yong and Qiao, Yu and Shao, Jing , year = 2025, month = may, journal =
2025
-
[41]
Zheng, Chujie and Zhou, Hao and Meng, Fandong and Zhou, Jie and Huang, Minlie , year = 2024, month = may, journal =. Large
2024
-
[42]
Zheng, Mingqian and Pei, Jiaxin and Logeswaran, Lajanugen and Lee, Moontae and Jurgens, David , editor =. When ``. Findings of the. doi:10.18653/v1/2024.findings-emnlp.888 , urldate =
2024 doi
-
[43]
The Fourteenth International Conference on Learning Representations , author =
Auditing Black-Box. The Fourteenth International Conference on Learning Representations , author =
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.