{"id":"1eb0657e-5f8a-4f39-a07b-f64e9fe70026","arxiv_id":"2502.09084","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"FT-Transformer outperforms kNN, Random Forest, MLP, and TabTransformer on most OS fingerprinting tasks across three public datasets, though the reported gains are weakened by pre-split SMOTE resampling.","lead":"This paper applies two existing Transformer models for tabular data, TabTransformer and FT-Transformer, to the task of identifying operating systems from network traffic. The authors report that FT-Transformer usually beats classical machine learning baselines on three public datasets, but the evaluation leaks synthetic training samples into the test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage from SMOTE and undersampling before the train/test split invalidates the reported F1 scores; a pristine held-out test set is required to support the FT-Transformer superiority claim.","rationale":"The reader's weakest assumption identifies exactly the load-bearing flaw: resampling before the train/test split contaminates the evaluation set with synthetic and undersampled samples. I independently confirmed this from Section 4.2, where class-imbalance handling is described as part of data preparation followed by the split, and Section 4.4, where the test set is said to use the same preprocessing pipeline. This is not a disagreement with external consensus or a stylistic complaint; it is an internal methodological inconsistency with the paper's own stated evaluation protocol of a hold-out test set. Because the reported F1 scores are the sole quantitative evidence for the central claim, the results are not trustworthy as support for real-world superiority. The concrete test is straightforward and computationally feasible with the promised code: preserve a pristine test split and move all resampling into training. Until that is done, the manuscript should not be accepted as establishing that FT-Transformer outperforms existing methods on genuine network traffic. The verdict therefore remains REJECT, and no change to the reader's assessment is needed.","tokens_in":19428,"tokens_out":2281,"duration_ms":25035,"concrete_test":"Re-run the DAT1 family, DAT2 major, and DAT3 family experiments with a corrected protocol: (1) split the original, unmodified data into stratified train/test sets once; (2) apply random undersampling and SMOTE only inside the training portion, preferably within each cross-validation fold; (3) evaluate all models on the untouched test set using the same metrics and report confidence intervals or paired tests. If FT-T retains its margin over RF, MLP, and TabT on all three tasks, the central claim survives. If any margin shrinks materially, reverses, or becomes statistically indistinguishable, the reported benchmark is an artifact of test-set leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FT-Transformer generally outperforms traditional ML models, previous approaches, and TabTransformer rests on the metrics in Tables 7-9. Section 4.2 applies random undersampling and SMOTE as part of data preparation and then states that \"data is split into training and test sets using stratified sampling.\" Section 4.4 explicitly says the hold-out test set is evaluated using \"the same preprocessing pipeline.\" Therefore, the test set contains SMOTE-generated synthetic minority samples and excludes majority-class real samples removed by undersampling. It is not a sample of unmodified network traffic. This has two consequences. First, absolute accuracies and weighted F1 scores are inflated because the test distribution matches the artificially balanced training distribution rather than the real deployment distribution. Second, the comparison to previous work in Table 10 is unfair: prior methods were evaluated on original data while FT-T and the in-paper baselines are scored on resampled data. The relative ordering among models trained and tested on the same contaminated pipeline could survive a corrected split, but the paper's \"new benchmark\" and \"generally outperforms\" statements are not supported until the test set is held out before any resampling. The paper lists the code repository, but no commit hash or artifact verification is provided, so the exact pipeline order cannot be independently confirmed from the manuscript alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies two tabular Transformer architectures, TabTransformer and FT-Transformer, to OS fingerprinting on three public datasets (DAT1, DAT2, DAT3) at family, major, and minor version granularities. It compares them with kNN, Random Forest, and MLP baselines and with previously published results, reporting that FT-Transformer generally outperforms the baselines and prior work. The paper claims to be the first application of Transformer architectures to OS fingerprinting and provides a public code repository. The core evaluation, however, is compromised by the data preparation order: resampling is applied before the train/test split, so the reported test-set metrics do not reflect performance on unmodified traffic.","tokens_in":19653,"tokens_out":7965,"duration_ms":78169,"significance":"If the central claim were supported by a clean evaluation, the paper would be a useful empirical contribution: it would be one of the first studies of tabular Transformer models for OS fingerprinting, it uses three diverse public datasets, and it reports multiple classification granularities. The open-source code and fixed library versions are strengths. However, the current evaluation protocol invalidates the reported performance numbers and the comparison with prior work, so the significance of the contribution cannot be assessed from the manuscript as written.","major_comments":[{"comment":"The test set is contaminated by resampling. Section 4.2 applies random undersampling and SMOTE to the full dataset during data preparation and only afterwards describes the stratified train/test split, while Section 4.4 states that the hold-out test set is evaluated with 'the same preprocessing pipeline.' Consequently, the test set contains SMOTE-generated synthetic minority samples and excludes real majority-class samples removed by undersampling. The accuracies and F1 scores in Tables 7-9 therefore do not describe performance on unmodified network traffic, and the comparisons in Table 10 are unfair because prior methods were evaluated on original data. The central claim in the Abstract and Section 5 is unsupported until the test set is held out before any resampling and resampling is performed only inside training folds. The code link does not resolve this issue because no commit hash is provided, so the exact pipeline order cannot be independently confirmed from the manuscript.","section":"Sections 4.2 and 4.4"},{"comment":"Even setting aside the leakage issue, the claim that FT-Transformer 'generally outperforms ... previous approaches' is contradicted by the paper's own comparison table. For DAT2 minor classification, the proposed FT-T achieves 68.52% accuracy and 69.76% F1, while the previously published kNN result [67] is listed with 92.10% accuracy (F1 not reported). The text calls this 'competitive,' but the only quantitative comparison available for that task shows a large accuracy gap. The authors should either provide the missing F1 for [67] under a comparable protocol or explicitly qualify the superiority claim so that it does not extend to this task.","section":"Table 10, DAT2 minor row"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported, and some of the comparisons that support the 'generally outperforms' claim rest on very small margins. For example, in DAT3 family classification FT-T achieves F1 92.23% versus MLP 91.62%, and in DAT3 minor classification TabT achieves 75.90% versus FT-T 75.74%. A single stratified split with fixed seeds cannot establish superiority at these margins. At minimum, the authors should report repeated cross-validation or repeated seeded splits on a pristine test set, together with the resulting variability.","section":"Tables 7-9"}],"minor_comments":[{"comment":"The contribution 'First application of the Transformer architecture to OS fingerprinting' is internally contradicted by Section 3.1.3, which cites a preliminary version of this work [45] applying the Transformer architecture to the Nmap database. The novelty claim should be rephrased, for example to 'first detailed comparison of tabular Transformer architectures for OS fingerprinting.'","section":"Abstract and Section 3.1.3"},{"comment":"The sentence 'One-Hot Encoding is applied to categorical target variables' is presumably a typo for 'categorical features'; the target variables are not one-hot encoded for the classifiers used in the study.","section":"Section 4.2"},{"comment":"Table 4 states that use_shared_categ_embed is 'Applicable to TabTransformer only,' but Table 6 reports this hyperparameter for FT-T runs. This inconsistency should be resolved.","section":"Tables 4 and 6"},{"comment":"The 'predetermined removal percentages' for random undersampling and the fixed random seed values are not given in the text. Without these values, the reproducibility claim cannot be checked independently of the unversioned repository.","section":"Section 4.2"},{"comment":"Reference [22] appears to be unrelated to OS fingerprinting; the cited paper by Matoušek and Wagner is in computational geometry. The reference intended for flow-based passive fingerprinting should be corrected.","section":"References"},{"comment":"In the DAT1 family comparison, the previous DT method has higher accuracy (93.12%) but lower F1 (78.96%) than the proposed FT-T (90.69% accuracy, 90.80% F1). The text should acknowledge this metric trade-off explicitly rather than reporting only the F1 improvement.","section":"Table 10, DAT1 row"},{"comment":"The order of SMOTE relative to one-hot encoding of categorical features is not stated. If SMOTE is applied after one-hot encoding, the synthetic minority samples may not correspond to valid categorical configurations; this should be clarified.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The reader's report recommends rejection, and I agree with the diagnosis of test-set leakage. I recommend major revision rather than rejection because the problem is fixable: the authors can rerun the pipeline with a pristine hold-out test set and resampling restricted to training folds. If they cannot produce such a rerun, the paper's central claims should be withdrawn. The novelty overstatement and the DAT2 minor comparison in Table 10 also need to be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a decent application paper undone by a classic data-leakage error. SMOTE and random undersampling are applied before the train/test split (Section 4.2), and Section 4.4 confirms the hold-out test set goes through \"the same preprocessing pipeline.\" So the test set contains synthetic minority samples and excludes undersampled majority samples. The scores in Tables 7–9 are not measurements on unmodified network traffic. The abstract's claim that FT-Transformer \"generally outperforms\" prior methods is therefore not supported, and the comparison to previous work in Table 10 is unfair because prior methods were evaluated on original data.\n\nWhat is genuinely new is the breadth: no prior work has compared FT-Transformer and TabTransformer across three datasets and three granularities (family, major, minor). The writing is clear, the datasets are real and public, seeds are fixed, hyperparameters are reported, and code is promised. That is real effort. The Table 10 survey of prior benchmarks is useful, even if the leakage makes the new results unreliable.\n\nThe soft spots, in order of severity: the leakage is load-bearing. There are also no confidence intervals or significance tests; several margins are small (DAT2 family: RF beats FT-T on F1, 95.33 vs 95.09, and the authors admit this). The \"first application\" novelty is weak because their own preliminary paper [45] already applied TabTransformer to the Nmap database. The code repository has no commit hash or artifact verification, so the exact pipeline order cannot be independently checked from the manuscript.\n\nThis paper deserves peer review, not desk rejection. The question—do tabular Transformers actually help OS fingerprinting?—is worth a careful answer, and the flaw is fixable: move resampling inside cross-validation and leave a pristine held-out test set. A corrected evaluation could shrink, reverse, or preserve the margins. I would send it to a serious referee with the expectation of major revision.","headline":"A well-written empirical study whose central claim is undermined by test-set leakage: resampling before the split means the reported F1 scores are not measurements on real traffic.","tokens_in":20199,"tokens_out":1631,"would_cite":false,"duration_ms":16645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FT-Transformer, a tabular Transformer, identifies operating systems from network traffic more accurately than classical machine-learning baselines across three public datasets.","keywords":["operating system fingerprinting","tabular transformers","FT-Transformer","TabTransformer","network traffic classification","self-attention","deep learning","OS identification"],"falsifier":"Repeat the experiments with resampling applied only inside cross-validation folds and the final test set left as the original real traffic; if FT-Transformer's weighted F1 drops by more than a few points or no longer beats the random forest and MLP baselines, the paper's main claim would be refuted.","tokens_in":19212,"feed_emoji":"🖥️","tokens_out":5341,"duration_ms":46040,"temperature":0.7,"pith_summary":"This paper argues that a Transformer architecture designed for tabular data, the FT-Transformer, can fingerprint a device's operating system from network-flow features more accurately than established machine-learning models and most previously published methods. It brings attention-based deep learning to a domain that has relied on rule-based tools such as Nmap and p0f or on classical classifiers. The authors evaluate on three public datasets with labels at OS family, major, and minor version levels, reporting that FT-Transformer achieves the best weighted F1 in five of seven tasks, including 90.80% on the DAT1 family task and 79.32% on the challenging DAT2 major-version task. If the result holds on unmodified traffic, it would make Transformer models a strong foundation for adaptive OS identification in dynamic networks.","feed_headline":"FT-Transformer beats classic ML at OS fingerprinting","feed_subtitle":"Attention-based tabular model tops kNN, random forest, and MLP on three public datasets across OS family, major, and minor levels.","key_machinery":"The load-bearing object is the FT-Transformer architecture: it embeds each feature—both categorical and numerical—into a token, then runs stacked Transformer layers with multi-head self-attention over the full set of feature tokens, so the model can weight and combine every feature based on the others. The paper pairs it with TabTransformer, which applies self-attention only to categorical embeddings, to isolate the value of handling numerical features in the attention mechanism. Both are trained on tabular network-flow records (IPFIX flows, packet captures, and Nmap OS signatures) with hyperparameters tuned by a genetic sampler under stratified 10-fold cross-validation.","core_discovery":"The paper's central claim is that FT-Transformer, which applies multi-head self-attention to all input features—categorical and numerical alike—generally outperforms TabTransformer, k-nearest neighbors, random forests, multi-layer perceptrons, and prior AI fingerprinting studies when classifying operating systems from tabular network data. Across seven dataset–granularity combinations, FT-Transformer attains the highest weighted F1 in five: DAT1 family (90.80%), DAT2 major (79.32%) and minor (69.76%), and DAT3 family (92.23%), while random forest edges it on DAT2 family and TabTransformer wins on DAT3 minor. The authors interpret the result as evidence that self-attention captures complex feature interactions in TCP/IP and TLS parameters that classical models miss, and they position FT-Transformer as a new benchmark for DL-based OS fingerprinting.","pith_inferences":["Because SMOTE and random undersampling are applied before the train/test split, the reported F1 scores likely overstate performance on raw network traffic; re-running with resampling restricted to the training folds could reduce FT-Transformer's margins.","The same tabular-Transformer pipeline could plausibly transfer to neighboring problems such as device-type identification, botnet detection, or TLS fingerprinting, but the paper does not test those tasks, so that remains a hypothesis.","The paper's comparison to prior work is complicated by inconsistent class groupings across studies; a head-to-head re-run on identical class definitions would be needed to confirm the claimed ~12-point gain over the TLS decision tree on DAT1.","FT-Transformer's gains may come partly from its ability to exploit the high-cardinality TLS JA3 fingerprint feature in DAT2; ablating features one group at a time would reveal whether the advantage is attention dynamics or simply access to richer inputs."],"forward_implications":["If FT-Transformer's advantage holds, network administrators can identify OS family and version from passive traffic without maintaining signature databases that go stale as operating systems update.","The method's strong performance on TLS-derived features suggests it can fingerprint devices even when payloads are encrypted, a scenario where rule-based tools degrade.","The attention weights give a natural route to explaining which TCP/IP and TLS parameters drive an OS decision, something opaque classical ensembles offer only indirectly.","The open-source release means other researchers can benchmark their own fingerprinting models against FT-Transformer on the same three datasets with identical preprocessing.","At the major-version level, where prior ML methods often struggled, FT-Transformer's larger margin suggests attention-based interaction modeling is most valuable precisely when classes are many and similar."],"supporting_citations":[{"why":"Supplies the FT-Transformer architecture that is the paper's main model and its claim of superiority.","marker":"[15]"},{"why":"Supplies the TabTransformer architecture used as the second attention-based model.","marker":"[16]"},{"why":"Provides the Transformer/self-attention mechanism that both tabular models build on.","marker":"[4]"},{"why":"Provides DAT1, the TLS-enriched flow dataset used for family-level fingerprinting.","marker":"[63]"},{"why":"Provides DAT2, the passive-flow dataset with family, major, and minor OS labels.","marker":"[64]"},{"why":"Provides DAT3, the Nmap OS signature database with family and minor labels.","marker":"[65]"},{"why":"The prior TLS-based decision-tree study on DAT1 that FT-Transformer is compared against.","marker":"[33]"},{"why":"The survey/benchmark study used as context for DAT2 comparisons and prior results.","marker":"[3]"}],"fun_headline_variants":["FT-Transformer tops classic ML in OS fingerprinting","Attention model wins most OS fingerprinting tasks","FT-Transformer beats kNN, RF, MLP in OS classification","Tabular attention excels at OS fingerprinting","FT-Transformer sets new standard for OS fingerprinting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes the test data is genuine network traffic, but the authors balance the data by discarding real samples and fabricating synthetic ones before splitting into train and test, so the test set is contaminated.","fun_headline_variants_meta":{"raw":{"variants":["FT-Transformer tops classic ML in OS fingerprinting","Attention model wins most OS fingerprinting tasks","FT-Transformer beats kNN, RF, MLP in OS classification","Tabular attention excels at OS fingerprinting","FT-Transformer sets new standard for OS fingerprinting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000893,"raw_usage":{"total_tokens":3818,"prompt_tokens":883,"completion_tokens":2935,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":2859}},"tokens_in":499,"tokens_out":2935,"duration_ms":20461,"temperature":1.0,"reasoning_tokens":2859,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T22:41:52.399457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the experiments with resampling applied only inside cross-validation folds and the final test set left as the original real traffic; if FT-Transformer's weighted F1 drops by more than a few points or no longer beats the random forest and MLP baselines, the paper's main claim would be refuted.","supporting_citations":[{"cited_title":"Dataset - using TLS fingerprints for OS identification in encrypted traffic","cited_arxiv_id":null,"evidence_quote":"Provides DAT1, the TLS-enriched flow dataset used for family-level fingerprinting."},{"cited_title":"Dataset - passive operating system fingerprinting revisited - network flows dataset","cited_arxiv_id":null,"evidence_quote":"Provides DAT2, the passive-flow dataset with family, major, and minor OS labels."},{"cited_title":"nmap OS DB - revision 38950: /nmap-releases/nmap-7.94","cited_arxiv_id":null,"evidence_quote":"Provides DAT3, the Nmap OS signature database with family and minor labels."},{"cited_title":"Using TLS fingerprints for OS identification in encrypted traffic,","cited_arxiv_id":null,"evidence_quote":"The prior TLS-based decision-tree study on DAT1 that FT-Transformer is compared against."},{"cited_title":"Passive operating system fingerprinting revisited: Evaluation and current challenges,","cited_arxiv_id":null,"evidence_quote":"The survey/benchmark study used as context for DAT2 comparisons and prior results."}],"review_version":1}