{"id":"6c14aeae-80bc-4bcb-ba0f-3a904746c138","arxiv_id":"2602.12825","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Two conformal prediction variants for hierarchical OS fingerprinting trade set tightness against taxonomic consistency, both achieving marginal coverage.","lead":"This paper applies two known conformal prediction methods to hierarchical operating system fingerprinting, producing prediction sets with guaranteed coverage at each taxonomy level. It shows a trade-off: level-wise calibration gives tighter but logically inconsistent outputs, while projection-based calibration gives consistent nested sets that are larger at coarse levels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P-CP's coarser-level coverage guarantee fails for ragged hierarchies because it relies on marginal leaf coverage, not coverage conditional on the level being defined.","rationale":"The reader's weakest assumption (User-Agent label noise) is a reasonable empirical limitation, but it does not directly threaten the central theoretical claim that both methods satisfy validity guarantees: even with noisy labels, CP guarantees coverage of the noisy labels, and the trade-off is structural. The more load-bearing concern is that P-CP's coarser-level coverage guarantee, as stated in Eq. (15), is not actually implied by leaf-level marginal coverage when the hierarchy is ragged. The paper's own framework admits ragged hierarchies, so this is an internal theoretical gap, not merely a data-quality caveat. The experimental evaluation uses 'empty' nodes to make the tree full-depth, so the reported numbers may be valid for that specific setup, but the general claim overreaches. This warrants a conditional verdict: the authors should either restrict the guarantee to full-depth hierarchies or prove/achieve conditional coverage (e.g., via Mondrian CP) for ragged trees.","tokens_in":16565,"tokens_out":18148,"duration_ms":167819,"concrete_test":"Construct a synthetic ragged-hierarchy dataset (e.g., 50% of leaves at depth 2, 50% at depth 3). Calibrate P-CP on all leaf labels at a target α=0.10, then measure the empirical coverage of Y_2 among test samples where Y_2 exists. If the coverage is below 0.90, the claim in Eq. (15) fails for the ragged case, confirming that the guarantee requires a full-depth tree or an additional conditional-calibration step.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.3.3 claims that P-CP inherits leaf-level coverage at coarser levels: Eq. (14) implies Eq. (15), i.e., P{Y_k ∈ Γ^(B)_k,α(X)} ≥ 1−α for all levels k with Y_k defined. The derivation uses P{Y_leaf ∈ Γ^(B)_leaf,α(X)} ≥ 1−α, but this is an unconditional marginal guarantee over the entire test distribution. When the hierarchy is ragged, the subpopulation of test points for which Y_k is defined is a strict subset; standard split CP does not control coverage on arbitrary subsets. Thus P{Y_leaf ∈ Γ_leaf | Y_k defined} can be below 1−α, and the lower bound in Eq. (15) does not follow. The paper's own Section 3.1 explicitly allows ragged hierarchies, and Section 4.3.1 defines C_k as containing only samples for which the level-k ancestor exists, but P-CP calibrates only on leaves without conditioning on the presence of upper-level ancestors. The experimental implementation sidesteps this by adding generic 'empty' leaf nodes so every sample has a level-3 leaf (Section 4.2), making the hierarchy effectively full-depth in the experiments. However, the formal claim (15) is stated for general K-level trees and is not proven for ragged hierarchies. This is a gap in the central validity guarantee for P-CP.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies split conformal prediction (CP) to hierarchical operating-system fingerprinting. It defines two strategies: level-wise CP (L-CP), which calibrates each taxonomy level independently, and projection-based CP (P-CP), which calibrates only at the leaf level and projects the resulting set upward to coarser levels. The central claim is that both methods provide marginal coverage guarantees at every taxonomy level, with a trade-off: L-CP yields tighter but taxonomically inconsistent sets, while P-CP yields nested, consistent sets at the cost of larger coarse-level sets. The empirical evaluation uses a public 109,663-flow dataset with family/major/minor labels, sweeping α over 50 Monte Carlo splits. Results show that L-CP tracks nominal coverage and has a non-zero hierarchical inconsistency rate, while P-CP over-covers at coarse levels and has HIR=0 by construction. The paper argues P-CP is preferable for automated policy enforcement and L-CP for forensic triage.","tokens_in":16954,"tokens_out":7870,"duration_ms":71536,"significance":"If the claims hold, this is a useful application of established conformal-prediction ideas to a network-security problem where uncertainty quantification is genuinely needed. The paper includes a complete open-source implementation, 50-run Monte Carlo evaluation, and a clear operational discussion of set-valued predictions in hierarchical labeling. The two compared schemes are not new in themselves—L-CP and P-CP correspond to known hierarchical CP constructions, and the paper cites the relevant prior work—but the OS-fingerprinting domain, the HIR metric, and the systematic coverage-efficiency-consistency evaluation are useful contributions. The formal CP derivations are standard and, in the complete-hierarchy case, correct. The main impediments are a gap in the P-CP coverage proof for ragged hierarchies and an evaluation that relies on a single noisy-label dataset and lacks a flat-CP baseline.","major_comments":[{"comment":"The empirical coverage numbers are all relative to HTTP User-Agent-derived labels, which the paper itself acknowledges 'may contain residual noise inherent to User-Agent parsing.' If these labels are wrong for some fraction of the data, the reported coverage is coverage of the noisy label, not necessarily of the true operating system. Since the paper's main applied claim is reliable OS fingerprinting, this label-quality issue is a threat to external validity. A sensitivity analysis, a comparison on a subset with independently verified labels, or at least a more prominent caveat that the guarantees are label-conditional would strengthen the paper. This does not invalidate the CP theory, but it does affect what the empirical evaluation supports.","section":"§4.1 / §5.1"},{"comment":"The evaluation compares only L-CP and P-CP; there is no flat-CP baseline. The introduction motivates hierarchical CP by contrasting it with a direct application of CP to flat classification, but the paper never quantifies the efficiency or validity differences between either structured method and flat CP. Without that baseline, the claimed 'limitations' of flat CP and the operational advantage of the hierarchical framework are not empirically tested. This is a significant evaluation gap, although it does not undermine the internal L-CP/P-CP trade-off analysis.","section":"§5.1 / Table 2"},{"comment":"Equation (10) writes P{Y_k ∈ Γ_k | Y_k} ≥ 1−α, but the conditioning object 'Y_k' appears to be a typographical artifact. The standard split-CP guarantee is marginal over the test point and calibration data, not conditional on the label random variable. Please restate the equation as P(Y_k ∈ Γ_k(X)) ≥ 1−α (or with the appropriate subpopulation conditioning, depending on the intended meaning).","section":"§4.3.2 / Eq. (10)"}],"minor_comments":[{"comment":"Table 2 reports coverage and set size only for the family and leaf levels, while the text says both methods were evaluated at all three levels and Figure 6 is said to show all levels. The major-version level should appear in the summary table, or its omission should be explained.","section":"§5.1 / Table 2"},{"comment":"P-CP's HIR=0 is a logical consequence of Eq. (12)–(13), not an empirical finding. The paper correctly notes this 'by construction' in the implementation, but the Results section still presents HIR as a 'key differentiator.' It would be clearer to state that HIR=0 is a guaranteed property of P-CP, not a measured outcome.","section":"§4.4 / §5.1"},{"comment":"The experimental implementation converts the ragged hierarchy into an effectively full-depth hierarchy by adding generic 'empty' leaf nodes. This choice should be more explicitly tied to the theoretical section: either the theory should be presented for full-depth hierarchies, or the implementation should be described as an approximation for the ragged case.","section":"§4.3.4"},{"comment":"The notation for the ancestor map π_{k←leaf} is informal; the paper says 'whenever the level-k ancestor exists' but does not define the domain of π. Formalizing the map as a partial function would help avoid the ambiguity in Eq. (15).","section":"§3.1 / §4.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid application of known hierarchical CP constructions to OS fingerprinting, with reproducible code and a sensible evaluation framework. The main blocking issue is the unproven (and in general false) coverage guarantee for P-CP on ragged hierarchies, which is a central claim. The label-noise and no-flat-baseline concerns are secondary but should be addressed in revision. I would not reject the paper; the technical framework can be repaired by tightening the theoretical statement and adding appropriate experiments or conditional-calibration guarantees."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a competent, clearly written application paper. It takes two known hierarchical CP schemes, applies them to OS fingerprinting, adds HIR as an evaluation metric, and gives an honest empirical comparison on a public dataset. The methods are not new and the paper does not pretend they are; the citations to Mortier et al. and Zhang et al. are explicit. If you care about hierarchical set-valued prediction or OS fingerprinting with uncertainty quantification, this is worth reading.\n\nWhat is genuinely new: the HIR metric is simple and useful; the empirical comparison on 109k flows with three taxonomy levels is careful; the 50-run Monte Carlo evaluation is a plus; code and data are public. The L-CP/P-CP trade-off is real and intuitive: L-CP gives tighter sets per level and pays in inconsistent outputs; P-CP is consistent by construction and pays in coarse-level set size. Coverage numbers behave as expected.\n\nSoft spots:\n\n1. The P-CP coverage claim for ragged hierarchies, Eq. (15), is overstated. The proof uses marginal leaf coverage and then says this implies coverage at every coarser level where the ancestor exists. For ragged trees, the subset of points with a defined level-k ancestor is not the whole test distribution, and marginal CP does not control coverage on arbitrary subsets. The paper explicitly allows ragged hierarchies in Sec. 3.1. The implementation avoids the issue by adding generic empty leaf nodes, so the empirical results stand, but the formal statement should be restricted to full-depth hierarchies or changed to a Mondrian/group-conditional guarantee.\n\n2. HIR=0 for P-CP is true by construction, so it is not an empirical discovery. The authors say this themselves, but the abstract and discussion present it as a headline result. Fine as a property; less fine as evidence.\n\n3. No flat-CP baseline. Since the intro dismisses flat CP, they should quantify what hierarchy-aware CP buys over it. Minor, but easy to add.\n\n4. Ground-truth labels come from User-Agent parsing. They admit residual noise, and it is a standard limitation, but it means coverage is measured against possibly wrong labels, and the caveat should be more than a parenthetical.\n\nOverall this is not a paper with a load-bearing flaw. The theory used is standard, the experiments are reproducible, and the central applied claim holds for the full-depth hierarchy used in the experiments. I would send it to peer review, with a request to fix the ragged-hierarchy statement and to add the flat baseline if feasible. It deserves referee time, not a desk rejection.","headline":"A solid applied CP-for-OS-fingerprinting paper with honest framing, but the formal coverage claim for P-CP is overstated for ragged hierarchies; worth peer review after a targeted fix.","tokens_in":17351,"tokens_out":4246,"would_cite":true,"duration_ms":40969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hierarchical OS fingerprinting can carry formal coverage guarantees via conformal prediction, but only if operators accept a trade-off between tight sets and taxonomically consistent sets.","keywords":["Operating system fingerprinting","Conformal prediction","Hierarchical classification","Uncertainty quantification","Network security","Prediction sets","Coverage guarantee","Taxonomy consistency"],"falsifier":"Run the same L-CP pipeline on a corpus whose OS labels are verified by controlled installation or manual audit rather than User-Agent parsing, and check empirical family-level coverage at alpha=0.10 over many splits; if L-CP coverage falls below 90% or P-CP family coverage falls below 90%, the paper's central validity claim fails. A simpler check: re-parse a random sample of the dataset's User-Agent strings by hand and measure label disagreement; large disagreement would show the reported coverage is not coverage of true OSs.","tokens_in":16536,"feed_emoji":"🖥️","tokens_out":4517,"duration_ms":39780,"temperature":0.7,"pith_summary":"Operating-system fingerprinting currently outputs a single guessed OS, which can be wrong in ways that matter more at some taxonomic levels than others. The paper wraps conformal prediction around hierarchical OS classifiers so each prediction becomes a set of plausible OS labels with a formal coverage guarantee. It introduces two ways to do this: calibrating each level of the OS tree independently (L-CP), and calibrating leaves then projecting the leaf set upward (P-CP). On 109,663 real traffic flows, both methods hit their nominal coverage targets, but they split on a second property: L-CP produces tighter sets that are often taxonomically inconsistent, while P-CP always produces nested, consistent sets at the cost of larger coarse-level sets. The paper's contribution is making this efficiency-versus-consistency trade-off explicit and measurable through a new Hierarchical Inconsistency Rate.","feed_headline":"Conformal OS fingerprinting: tight sets or consistent sets","feed_subtitle":"Level-wise tuning yields smaller, sometimes contradictory sets; leaf-projection yields nested, policy-safe sets.","key_machinery":"The machinery is split conformal prediction applied to a ragged K-level label tree. A nonconformity score of 1 minus the softmax probability is computed for each level; the calibration quantile at level k is the ceiling((n_k+1)(1-alpha_k))/(n_k+1) empirical quantile of calibration scores. L-CP thresholds each level separately, giving P(Y_k in Gamma_k) >= 1-alpha_k per level but no cross-level coherence. P-CP thresholds only the leaf level and defines coarse sets as the set of ancestors with at least one surviving leaf, which forces nesting and transfers leaf coverage upward. The novel Hierarchical Inconsistency Rate counts samples with at least one orphan (child without parent) or sterile (p","core_discovery":"Both level-wise CP and projection-based CP provide marginal coverage guarantees at each level of an OS taxonomy, but they cannot simultaneously deliver maximal level-wise efficiency and structural consistency. L-CP calibrates a separate conformal threshold at each hierarchy level, yielding prediction sets close to the nominal size but that frequently violate parent-child logic (up to roughly 37% HIR in experiments). P-CP applies conformal prediction only at the leaves and lifts the resulting set to ancestors, which by construction gives nested, HIR=0 sets and inherits leaf coverage at coarser levels, yet inflates coarse-level sets because any retained rare leaf keeps its whole family. Empiri","pith_inferences":["A hybrid could recover P-CP's lost efficiency: use a coarse-level model to prune families whose leaf mass is negligible, then project remaining leaves upward; this would preserve nesting while tightening coarse sets.","The HIR curve's non-monotonic shape suggests HIR could double as a drift or calibration monitor: if HIR rises at moderate alpha in deployment, the base leaf model may be degrading even if marginal coverage still holds.","The same level-wise/projection trade-off likely appears in other hierarchical security classification domains, such as malware family or device-type identification, not just OS fingerprinting."],"forward_implications":["Network operators can now select prediction sets with guaranteed coverage instead of point predictions; at alpha=0.10, family-level sets hover around one label while maintaining at least 90% coverage.","For automated policy enforcement, P-CP is the usable option: nested sets make policy checks like 'block all Android except 11+' unambiguous because HIR is zero.","For human forensic triage, L-CP's tighter sets give more useful granularity, and analysts can tolerate or resolve occasional taxonomic contradictions.","Empty-set behavior differs: L-CP abstains more often at coarse levels than P-CP; both abstain more as alpha grows, signaling out-of-distribution traffic.","The trade-off framework and HIR metric extend to any hierarchical classification task with cost-asymmetric errors."],"fun_headline_variants":["OS fingerprinting: tight vs consistent prediction sets","Hierarchical conformal prediction: efficiency vs consistency","Tight OS sets or consistent ones? CP trade-off","Conformal prediction for OS: pick tight or nested","Level-wise CP gives smaller sets, projection gives safe sets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The coverage guarantees are only as trustworthy as the labels used for calibration: the dataset's OS labels come from HTTP User-Agent strings that may contain parsing errors, so if those labels are wrong the reported coverage overstates performance against the true OS; exchangeability of calibration and test traffic is also assumed.","fun_headline_variants_meta":{"raw":{"variants":["OS fingerprinting: tight vs consistent prediction sets","Hierarchical conformal prediction: efficiency vs consistency","Tight OS sets or consistent ones? CP trade-off","Conformal prediction for OS: pick tight or nested","Level-wise CP gives smaller sets, projection gives safe sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00052,"raw_usage":{"total_tokens":2328,"prompt_tokens":688,"completion_tokens":1640,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":1564}},"tokens_in":432,"tokens_out":1640,"duration_ms":10448,"temperature":1.0,"reasoning_tokens":1564,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:41:14.351840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same L-CP pipeline on a corpus whose OS labels are verified by controlled installation or manual audit rather than User-Agent parsing, and check empirical family-level coverage at alpha=0.10 over many splits; if L-CP coverage falls below 90% or P-CP family coverage falls below 90%, the paper's central validity claim fails. A simpler check: re-parse a random sample of the dataset's User-Agent strings by hand and measure label disagreement; large disagreement would show the reported coverage is not coverage of true OSs.","supporting_citations":[],"review_version":1}