REVIEW 3 major objections 4 minor 37 references
Reliable Hierarchical Operating System Fingerprinting via Conformal Prediction
T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Hierarchical OS fingerprinting can carry formal coverage guarantees via conformal prediction, but only if operators accept a trade-off between tight sets and taxonomically consistent sets.
desk verdict A solid applied CP-for-OS-fingerprinting paper with honest framing, but the formal coverage claim for P-CP is overstated for ragged hierarchies; worth peer review after a targeted fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is split conformal prediction applied to a ragged K-level label tree. A nonconformity score of 1 minus the softmax probability is computed for each level; the calibration quantile at level k is the ceiling((n_k+1)(1-alpha_k))/(n_k+1) empirical quantile of calibration scores. L-CP thresholds each level separately, giving P(Y_k in Gamma_k) >= 1-alpha_k per level but no cross-level coherence. P-CP thresholds only the leaf level and defines coarse sets as the set of ancestors with at least one surviving leaf, which forces nesting and transfers leaf coverage upward. The novel Hierarchical Inconsistency Rate counts samples with at least one orphan (child without parent) or sterile (p
What would settle it
Run the same L-CP pipeline on a corpus whose OS labels are verified by controlled installation or manual audit rather than User-Agent parsing, and check empirical family-level coverage at alpha=0.10 over many splits; if L-CP coverage falls below 90% or P-CP family coverage falls below 90%, the paper's central validity claim fails. A simpler check: re-parse a random sample of the dataset's User-Agent strings by hand and measure label disagreement; large disagreement would show the reported coverage is not coverage of true OSs.
Extended reading notes
Core claim
Both level-wise CP and projection-based CP provide marginal coverage guarantees at each level of an OS taxonomy, but they cannot simultaneously deliver maximal level-wise efficiency and structural consistency. L-CP calibrates a separate conformal threshold at each hierarchy level, yielding prediction sets close to the nominal size but that frequently violate parent-child logic (up to roughly 37% HIR in experiments). P-CP applies conformal prediction only at the leaves and lifts the resulting set to ancestors, which by construction gives nested, HIR=0 sets and inherits leaf coverage at coarser levels, yet inflates coarse-level sets because any retained rare leaf keeps its whole family. Empiri
Load-bearing premise
The coverage guarantees are only as trustworthy as the labels used for calibration: the dataset's OS labels come from HTTP User-Agent strings that may contain parsing errors, so if those labels are wrong the reported coverage overstates performance against the true OS; exchangeability of calibration and test traffic is also assumed.
Editorial extensions
If this is right
- Network operators can now select prediction sets with guaranteed coverage instead of point predictions; at alpha=0.10, family-level sets hover around one label while maintaining at least 90% coverage.
- For automated policy enforcement, P-CP is the usable option: nested sets make policy checks like 'block all Android except 11+' unambiguous because HIR is zero.
- For human forensic triage, L-CP's tighter sets give more useful granularity, and analysts can tolerate or resolve occasional taxonomic contradictions.
- Empty-set behavior differs: L-CP abstains more often at coarse levels than P-CP; both abstain more as alpha grows, signaling out-of-distribution traffic.
- The trade-off framework and HIR metric extend to any hierarchical classification task with cost-asymmetric errors.
Reading between the lines
- A hybrid could recover P-CP's lost efficiency: use a coarse-level model to prune families whose leaf mass is negligible, then project remaining leaves upward; this would preserve nesting while tightening coarse sets.
- The HIR curve's non-monotonic shape suggests HIR could double as a drift or calibration monitor: if HIR rises at moderate alpha in deployment, the base leaf model may be degrading even if marginal coverage still holds.
- The same level-wise/projection trade-off likely appears in other hierarchical security classification domains, such as malware family or device-type identification, not just OS fingerprinting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies split conformal prediction (CP) to hierarchical operating-system fingerprinting. It defines two strategies: level-wise CP (L-CP), which calibrates each taxonomy level independently, and projection-based CP (P-CP), which calibrates only at the leaf level and projects the resulting set upward to coarser levels. The central claim is that both methods provide marginal coverage guarantees at every taxonomy level, with a trade-off: L-CP yields tighter but taxonomically inconsistent sets, while P-CP yields nested, consistent sets at the cost of larger coarse-level sets. The empirical evaluation uses a public 109,663-flow dataset with family/major/minor labels, sweeping α over 50 Monte Carlo splits. Results show that L-CP tracks nominal coverage and has a non-zero hierarchical inconsistency rate, while P-CP over-covers at coarse levels and has HIR=0 by construction. The paper argues P-CP is preferable for automated policy enforcement and L-CP for forensic triage.
Significance. If the claims hold, this is a useful application of established conformal-prediction ideas to a network-security problem where uncertainty quantification is genuinely needed. The paper includes a complete open-source implementation, 50-run Monte Carlo evaluation, and a clear operational discussion of set-valued predictions in hierarchical labeling. The two compared schemes are not new in themselves—L-CP and P-CP correspond to known hierarchical CP constructions, and the paper cites the relevant prior work—but the OS-fingerprinting domain, the HIR metric, and the systematic coverage-efficiency-consistency evaluation are useful contributions. The formal CP derivations are standard and, in the complete-hierarchy case, correct. The main impediments are a gap in the P-CP coverage proof for ragged hierarchies and an evaluation that relies on a single noisy-label dataset and lacks a flat-CP baseline.
major comments (3)
- [§4.1 / §5.1] The empirical coverage numbers are all relative to HTTP User-Agent-derived labels, which the paper itself acknowledges 'may contain residual noise inherent to User-Agent parsing.' If these labels are wrong for some fraction of the data, the reported coverage is coverage of the noisy label, not necessarily of the true operating system. Since the paper's main applied claim is reliable OS fingerprinting, this label-quality issue is a threat to external validity. A sensitivity analysis, a comparison on a subset with independently verified labels, or at least a more prominent caveat that the guarantees are label-conditional would strengthen the paper. This does not invalidate the CP theory, but it does affect what the empirical evaluation supports.
- [§5.1 / Table 2] The evaluation compares only L-CP and P-CP; there is no flat-CP baseline. The introduction motivates hierarchical CP by contrasting it with a direct application of CP to flat classification, but the paper never quantifies the efficiency or validity differences between either structured method and flat CP. Without that baseline, the claimed 'limitations' of flat CP and the operational advantage of the hierarchical framework are not empirically tested. This is a significant evaluation gap, although it does not undermine the internal L-CP/P-CP trade-off analysis.
- [§4.3.2 / Eq. (10)] Equation (10) writes P{Y_k ∈ Γ_k | Y_k} ≥ 1−α, but the conditioning object 'Y_k' appears to be a typographical artifact. The standard split-CP guarantee is marginal over the test point and calibration data, not conditional on the label random variable. Please restate the equation as P(Y_k ∈ Γ_k(X)) ≥ 1−α (or with the appropriate subpopulation conditioning, depending on the intended meaning).
minor comments (4)
- [§5.1 / Table 2] Table 2 reports coverage and set size only for the family and leaf levels, while the text says both methods were evaluated at all three levels and Figure 6 is said to show all levels. The major-version level should appear in the summary table, or its omission should be explained.
- [§4.4 / §5.1] P-CP's HIR=0 is a logical consequence of Eq. (12)–(13), not an empirical finding. The paper correctly notes this 'by construction' in the implementation, but the Results section still presents HIR as a 'key differentiator.' It would be clearer to state that HIR=0 is a guaranteed property of P-CP, not a measured outcome.
- [§4.3.4] The experimental implementation converts the ragged hierarchy into an effectively full-depth hierarchy by adding generic 'empty' leaf nodes. This choice should be more explicitly tied to the theoretical section: either the theory should be presented for full-depth hierarchies, or the implementation should be described as an approximation for the ragged case.
- [§3.1 / §4.3.1] The notation for the ancestor map π_{k←leaf} is informal; the paper says 'whenever the level-k ancestor exists' but does not define the domain of π. Formalizing the map as a partial function would help avoid the ambiguity in Eq. (15).
Circularity Check
P-CP's HIR=0 is a definitional tautology (explicitly labeled 'by construction'), but it is not load-bearing; coverage, set-size, and L-CP's nonzero HIR are independent empirical findings.
-
self definitional
[Section 4.3.4 / Eq. (12)-(13) / Section 4.4 Eq. (18)-(21) / Table 2]
"In the case of P-CP, the implementation strictly enforces consistency by deriving the upper-level sets solely from the valid leaf prediction set. Specifically, the major set is generated as the unique set of parents of all retained leaves, and the family set as the unique set of grandparents, ensuring HIR=0 by construction."
P-CP defines Γ^(B)_k (Eq. 12) as exactly the set of ancestors of retained leaves, and HIR (Eqs. 18-21) counts orphan/sterile violations of exactly that parent-child containment. Therefore HIR(P-CP)=0 follows from the method's definition; reporting it in Table 2 as a comparative differentiator is not an independent empirical result. The paper itself marks it 'by construction,' so this is a minor definitional tautology rather than a hidden circularity, and it does not affect the measured coverage or set-size comparisons.
full rationale
No significant circularity beyond the acknowledged definitional point. The two CP variants are applied to a held-out test partition (15% split, 50 Monte Carlo iterations) with calibration quantiles computed on a separate calibration set, so the reported marginal coverage and mean set sizes are genuine out-of-sample measurements, not fitted values renamed as predictions. L-CP's nonzero HIR is an empirical observation, as are P-CP's inflated coarse-level sets; the claimed efficiency-consistency trade-off therefore rests on independent evidence. Self-citations (Pérez-Jove et al., 2025 and related fingerprinting work) are background context, not load-bearing support for the coverage theorems. The separate concern raised about P-CP's coarse-level coverage in ragged hierarchies—marginal leaf coverage need not control coverage conditional on the level-k ancestor being defined—is a correctness/assumption gap in Eq. (15), not a circularity, and is not counted here.
Assumptions & free parameters
free parameters (3)
- Per-level MLP architecture =
e.g., [64,32] family; [1024,512] leaf
- Mutual-information feature count k
- Class-weight / SMOTE settings
assumptions (5)
- domain assumption Train/calibration/test data are exchangeable
- domain assumption User-Agent-derived OS labels are accurate ground truth
- domain assumption The family→major→minor tree is the correct label hierarchy
- standard math 1−softmax is a valid nonconformity score
- standard math Split CP marginal coverage theorem
Cite this review
Pith. "Pith review of Reliable Hierarchical Operating System Fingerprinting via Conformal Prediction." pith.science (2026). https://pith.science/paper/YWT4CPG6
@misc{pith2026260212825,
author = {Pith},
title = {Pith review of: Reliable Hierarchical Operating System Fingerprinting via Conformal Prediction},
year = {2026},
howpublished = {\url{https://pith.science/paper/YWT4CPG6}},
note = {Machine review of arXiv:2602.12825}
}
read the original abstract
Operating System (OS) fingerprinting is critical for network security, but conventional methods do not provide formal uncertainty quantification mechanisms. Conformal Prediction (CP) could be directly wrapped around existing methods to obtain prediction sets with guaranteed coverage. However, a direct application of CP would treat OS identification as a flat classification problem, ignoring the natural taxonomic structure of OSs and providing brittle point predictions. This work addresses these limitations by introducing and evaluating two distinct structured CP strategies: level-wise CP (L-CP), which calibrates each hierarchy level independently, and projection-based CP (P-CP), which ensures structural consistency by projecting leaf-level sets upwards. Our results demonstrate that, while both methods satisfy validity guarantees, they expose a fundamental trade-off between level-wise efficiency and structural consistency. L-CP yields tighter prediction sets suitable for human forensic analysis but suffers from taxonomic inconsistencies. Conversely, P-CP guarantees hierarchically consistent, nested sets ideal for automated policy enforcement, albeit at the cost of reduced efficiency at coarser levels.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...
-
[2]
Accurate TLS Fingerprinting using Destination Context and Knowledge Bases
author Anderson, B. , & author McGrew , D. ( year 2020 ). title Accurate TLS fingerprinting using destination context and knowledge bases . :10.48550/arxiv.2009.01939
work page Pith review arXiv doi:10.48550/arxiv.2009.01939 2020
-
[3]
author Angelopoulos, A. N. , & author Bates, S. ( year 2023 ). title Conformal prediction: A gentle introduction . journal Foundations and Trends® in Machine Learning \/ , volume 16 \/ , pages 494--591 . :10.1561/2200000101
-
[4]
title FinisTerrae III User Guide - CESGA Technical Documentation
author CESGA ( year 2022 ). title FinisTerrae III User Guide - CESGA Technical Documentation . https://cesga-docs.gitlab.io/ft3-user-guide/index.html
2022
-
[5]
, author Zhang, Q
author Chang, D. , author Zhang, Q. , & author Li, X. ( year 2015 ). title Study on os fingerprinting and nat/tethering based on dns log analysis . In booktitle IRTF & ISOC Workshop on Research and Applications of Internet Measurements ( RAIM ) \/ (pp. pages 1--4 )
2015
-
[6]
author Fan, X. , author Gou, G. , author Kang, C. , author Shi, J. , & author Xiong, G. ( year 2019 ). title Identify OS from encrypted traffic with TCP / IP stack fingerprinting . In booktitle 2019 IEEE 38th International Performance Computing and Communications Conference ( IPCCC ) \/ (pp. pages 1--7 ). publisher IEEE . :10.1109/IPCCC47392.2019.8958772
arXiv 2019
-
[7]
author Hagos, D. H. , author Yazidi, A. , author Kure, O. , & author Engelstad, P. E. ( year 2021 ). title A machine-learning-based tool for passive OS fingerprinting with TCP variant as a novel feature . journal IEEE Internet of Things Journal \/ , volume 8 \/ , pages 3534--3553 . :10.1109/JIOT.2020.3024293
arXiv 2021
-
[8]
author Hulák, M. , author Bartoš, V. , & author Čejka, T. ( year 2023 ). title Evaluation of passive OS fingerprinting methods using TCP / IP fields . In booktitle 2023 8th International Conference on Smart and Sustainable Technologies, SpliTech 2023 \/ . publisher IEEE . :10.23919/SpliTech58164.2023.10192974
arXiv 2023
Show all 37 references
-
[9]
, author Čermák, M
author Husák, M. , author Čermák, M. , author Jirsík, T. , & author Čeleda, P. ( year 2016 ). title HTTPS traffic analysis and client identification using passive SSL / TLS fingerprinting . journal EURASIP Journal on Information Security \/ , volume 2016 \/ , pages 6 . :10.118...
2016 doi
-
[10]
, author Barford, P
author Kline, J. , author Barford, P. , author Cahn, A. , & author Sommers, J. ( year 2017 ). title On the structure and characteristics of user agent string . In booktitle Proceedings of the 2017 Internet Measurement Conference \/ (pp. pages 184--190 ). publisher ACM . :10.11...
2017
-
[11]
, author Jirsik, T
author Lastovicka, M. , author Jirsik, T. , author Celeda, P. , author Spacek, S. , & author Filakovsky, D. ( year 2018 ). title Passive os fingerprinting methods in the jungle of wireless networks . In booktitle NOMS 2018 - 2018 IEEE / IFIP Network Operations and Management S...
2018
-
[12]
, author Dufka, A
author Laštovička, M. , author Dufka, A. , & author Komárková, J. ( year 2018 ). title Machine learning fingerprinting methods in cyber security domain: Which one to use? In booktitle 2018 14th International Wireless Communications & Mobile Computing Conference ( IWCMC ) \/ (p...
2018
-
[13]
, & author Filakovsky, D
author Laštovička, M. , & author Filakovsky, D. ( year 2018 ). title Passive os fingerprinting prototype demonstration . In booktitle NOMS 2018 - 2018 IEEE / IFIP Network Operations and Management Symposium \/ (pp. pages 1--2 ). :10.1109/NOMS.2018.8406128
2018
-
[14]
, author Husák, M
author Laštovička, M. , author Husák, M. , author Velan, P. , author Jirsík, T. , & author Čeleda, P. ( year 2023 a ). title Dataset - passive operating system fingerprinting revisited - network flows dataset . :10.5281/ZENODO.7635138
2023 doi
-
[15]
, author Husák, M
author Laštovička, M. , author Husák, M. , author Velan, P. , author Jirsík, T. , & author Čeleda, P. ( year 2023 b ). title Passive operating system fingerprinting revisited: Evaluation and current challenges . journal Computer Networks \/ , (p. pages 109782 ). :10.1016/j.com...
2023
-
[16]
, author Špaček, S
author Laštovička, M. , author Špaček, S. , author Velan, P. , & author Čeleda, P. ( year 2020 ). title Using TLS fingerprints for OS identification in encrypted traffic . In booktitle NOMS 2020 - 2020 IEEE / IFIP Network Operations and Management Symposium \/ (pp. pages 1--6 ...
2020
-
[17]
, author Wei, Z
author Li, J. , author Wei, Z. , & author Chen, S. ( year 2023 ). title Passive OS identification in imbalanced dataset . In booktitle 2023 International Conference on Electrical, Computer and Energy Technologies ( ICECET ) \/ (pp. pages 1--6 ). :10.1109/ICECET58911.2023.10389569
2023
-
[18]
( year 2020 )
author Li, R. ( year 2020 ). title Overview: Fingerprinting services on the internet . journal Network \/ , . :10.2313/NET-2020-11-1_14
2020 doi
-
[19]
author Lyon, G. F. ( year 2009 ). title Nmap Network Scanning: The Official Nmap Project Guide to Network Discovery and Security Scanning \/ . publisher Insecure.Com LLC . https://nmap.org/book/
2009
-
[20]
, author Rysavý, O
author Matousek, P. , author Rysavý, O. , author Grégr, M. , & author Vymlátil, M. ( year 2014 ). title Towards identification of operating systems from the internet traffic - ipfix monitoring with fingerprinting and clustering. In booktitle DCNET \/ (pp. pages 21--27 ). https...
2014 doi
-
[21]
, author Javanmardi, A
author Mortier, T. , author Javanmardi, A. , author Sale, Y. , author Hüllermeier, E. , & author Waegeman, W. ( year 2025 ). title Conformal prediction in hierarchical classification . :10.48550/arXiv.2501.19038. arXiv:2501.19038 [stat] http://arxiv.org/abs/2501.19038 [stat]
-
[22]
title NetworkMiner - the NSM and network forensics analysis tool
author Netresec ( year 2022 ). title NetworkMiner - the NSM and network forensics analysis tool . https://www.netresec.com/?page=NetworkMiner
2022
-
[23]
title Ettercap project
author Ornaghi, Alberto , author Valleri, Marco , author Escobar, Emilio , author Costamagna, Gianfranco , author Koeppe, Alexander , & author Abdulkadir, Ali ( year 2001 ). title Ettercap project
2001
-
[24]
, author Georgiou, N
author Papadopoulos, H. , author Georgiou, N. , author Eliades, C. , & author Konstantinidis, A. ( year 2018 ). title Android malware detection with unbiased confidence guarantees . journal Neurocomputing \/ , volume 280 \/ , pages 3--12 . :10.1016/j.neucom.2017.08.072
2018 doi
-
[25]
, author Munteanu, C
author P \'e rez-Jove , R. , author Munteanu, C. R. , author Pazos, A. , & author V \'a zquez-Naya , J. ( year 2025 ). title Application of Tabular Transformer Architectures for Operating System Fingerprinting . :10.48550/arXiv.2502.09084. arXiv:2502.09084 http://arxiv.org/abs...
-
[26]
( year 1981 a )
author Postel, J. ( year 1981 a ). title Internet protocol . https://www.rfc-editor.org/rfc/rfc791
1981
-
[27]
( year 1981 b )
author Postel, J. ( year 1981 b ). title Transmission control protocol . https://www.rfc-editor.org/rfc/rfc793
1981
-
[28]
, author Munteanu, C
author Pérez-Jove, R. , author Munteanu, C. R. , author Dorado, J. , author Pazos, A. , & author Vázquez-Naya, J. ( a ). title Network traffic foundation models: A systematic review . journal Computer Networks \/ , volume 276 \/ , pages 111998 . :10.1016/j.comnet.2026.111998
2026
-
[29]
, author Munteanu, C
author Pérez-Jove, R. , author Munteanu, C. R. , author Dorado, J. , author Pazos, A. , & author Vázquez-Naya, J. ( b ). title Operating System Fingerprinting Tool Based on Classical Machine Learning Algorithms . In booktitle 2023 JNIC Cybersecurity Conference ( JNIC ) \/ (pp....
2023
-
[30]
, author Cline, D
author Shamsi, Z. , author Cline, D. B. H. , & author Loguinov, D. ( year 2021 ). title Faulds: A non-parametric iterative classifier for internet-wide OS fingerprinting . journal IEEE / ACM Transactions on Networking \/ , volume 29 \/ , pages 2339--2352 . :10.1109/TNET.2021.3088333
2021
-
[31]
, author Courtay, O
author Veysset, F. , author Courtay, O. , & author Heen, O. ( year 2002 ). title New tool and technique for remote operating system fingerprinting . journal Intranode Software Technologies \/ , (p. pages 13 )
2002
-
[32]
, & author Gnezdilov, K
author Voronov, I. , & author Gnezdilov, K. ( year 2021 ). title Determining OS and applications by DNS traffic analysis . In booktitle 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering ( ElConRus ) \/ (pp. pages 72--76 ). :10.1109/ElCo...
2021
-
[33]
, author Lindsay, D
author Vovk, V. , author Lindsay, D. , author Nouretdinov, I. , & author Gammerman, A. ( year 2003 ). title Mondrian Confidence Machine \/ . type Technical Report Royal Holloway, University of London
2003
-
[34]
, author Issac, B
author Yumlembam, R. , author Issac, B. , author Jacob, S. M. , & author Yang, L. ( year 2024 ). title Comprehensive botnet detection by mitigating adversarial attacks, navigating the subtleties of perturbation distances and fortifying predictions with conformal layers . journ...
2024
-
[35]
( year 2005 )
author Zalewski, M. ( year 2005 ). title p0f v2: Passive OS fingerprinting tool . https://lcamtuf.coredump.cx/p0f.shtml
2005
- [36]
-
[37]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.