{"id":"6bf529bf-8728-4896-917b-d0f148069567","arxiv_id":"2505.20866","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An instruction-tuned LLM with flow-graph alignment classifies encrypted traffic under distribution shift, reporting F1 improvements of up to 18 points over baselines on out-of-distribution benchmarks.","lead":"This paper presents ETooL, a system that combines a large language model with graph representations of encrypted network traffic to classify apps and botnets when the traffic distribution shifts over time or across versions. It reports large F1-score gains over existing methods on several out-of-distribution benchmarks and introduces a new adjustable synthetic dataset called NETD.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed O.O.D. robustness is not independently validated: the BURST graph threshold γ (Section V, Algorithm 1) is selected using target test performance (Section VIII-F), and the graph component is essential (Table VI, row 3), so the reported gains may reflect test-set selection bias rather…","rationale":"I read the paper in good faith: the method is coherent, the two-stage frozen-backbone tuning is clearly described, and the ablation study (Table VI) genuinely supports the importance of the graph representation and the LLM. The central claim, however, is that ETooL exhibits O.O.D. robustness because TRG captures distribution-invariant flow interaction patterns. The weakest link in that argument is not the architecture but the evidence for invariance: the single most important graph hyperparameter, γ, is chosen by observing testing performance on the target shift, per Section VIII-F. Since BURST membership is defined entirely by γ in Algorithm 1, tuning γ on the O.O.D. test data leaks information from the target distribution into the representation, making the reported 16.66% average improvement impossible to interpret as pure generalization. The ablation’s 22.13-point drop without graph structural tuning shows the graph is not a decorative component, which sharpens the concern: a small, test-tuned parameter can shift the headline result substantially. The reader’s weakest assumption focused on the stability of BURST under shifts; my concern is complementary and more specific—the paper never measures that stability independently of the end-task, and the γ analysis actively selects on the target test set. This warrants the same CONDITIONAL verdict rather than REJECT, because the issue is fixable by target-free hyperparameter selection and multi-seed reporting, and the method may well survive such a test. I did not treat the private NETD dataset or the label corrections as the primary obstacle, since they mainly affect external comparability, not the internal validity of the comparison shown in Tables IV–VI.","tokens_in":25334,"tokens_out":4447,"duration_ms":49874,"concrete_test":"Re-run the full pipeline on APP53⇒TIME and APP53⇒VERSION with γ chosen without access to target test data: hold out a validation split from the I.I.D. source (e.g., 20% of APP53-TIME training flows), sweep γ ∈ {0.1, 0.5, 1, 2, 5, 10} s, select the γ maximizing source-validation macro-F1, then freeze all other settings and evaluate on the target-shift test sets. Compare the resulting F1 values (mean ± std over at least 3 seeds) against the reported 74.88% and 72.13%. If the source-selected γ reproduces the reported target F1 within a few points, the stability claim is credible; if F1 drops materially (e.g., more than 5 points) or varies widely across γ, the O.O.D. gains are partly explained by test-set hyperparameter selection rather than invariant BURST structure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that TRG/BURST encodes transferable flow-interaction structure that survives time and version shifts—rests on the stability of flow-level BURST grouping. That stability is not demonstrated directly. In Algorithm 1, BURST membership is controlled by γ; Section VIII-A5 fixes γ = 1 s, and Section VIII-F reports sweeping γ and choosing the value with the best 'testing performance' on the target O.O.D. tasks. No held-out validation split or target-free selection rule is described. This is load-bearing because the graph path is not a minor component: removing graph structural tuning (Table VI, model 3) cuts average F1 by 22.13 points, so the representation's sensitivity to γ is coupled to the headline 16.66% average gain. If γ had been tuned against the same target distributions on which ETooL is then evaluated, part of the observed O.O.D. invariance could be an artifact of selection on test data rather than a property of BURST structure. The paper also reports single-run numbers without error bars, so it is unclear whether the remaining gaps exceed training noise. This does not refute the method; it means the invariance assumption is currently supported only by test-selected numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ETooL, a two-stage instruction-tuning framework that combines a traffic relation graph (TRG) built from flow-level BURST structures with a frozen Vicuna-7B LLM and a frozen graph encoder, training only a projection layer and a task head. The first tuning stage is a self-supervised BURST graph matching task, and the second is task-specific instruction tuning. The authors evaluate ETooL on APP53 I.I.D./O.O.D., ISCX-Botnet, and a newly constructed NETD dataset, reporting large F1 gains over statistical, deep-learning, and pre-trained baselines, plus ablation, efficiency, and hyperparameter-sensitivity studies. They also document label-correction decisions for APP53 and ISCX-Botnet.","tokens_in":25548,"tokens_out":5072,"duration_ms":52503,"significance":"If the reported results are robust, the paper would make a useful contribution: it demonstrates a concrete way to inject domain-specific graph structure into an LLM through instruction tuning with a small trainable parameter budget, and it is one of few works explicitly targeting non-I.I.D. encrypted traffic with a tunable O.O.D. dataset. The candid reporting of label inconsistencies, the OOM comparison in Table VII, and the explicit statement that the BURST threshold and learning rate were chosen by testing performance are transparency strengths. However, the central invariance claim—that TRG/BURST structure survives time and version shifts—is not directly validated, and the current evidence relies on test-selected hyperparameters and single-run numbers. These issues are fixable but load-bearing for the headline O.O.D. results.","major_comments":[{"comment":"The BURST time threshold gamma is load-bearing: removing graph structural tuning reduces average F1 by 22.13 points (Table VI, model 3), yet Section VIII-F states that gamma was chosen after 'observing the testing performance' on the target I.I.D. and O.O.D. tasks, and the learning rate is selected in the same way. No held-out validation split or target-free selection rule is described. If gamma and the learning rate were tuned against the same target distributions on which ETooL is then evaluated, part of the observed O.O.D. invariance could reflect selection on test data rather than a property of BURST structure, and the comparison would be unfair to baselines that did not receive such tuning. Please specify a validation protocol that does not use target test labels (e.g., source-only validation or a fixed domain-motivated gamma), or report how much of the gains survive under a target-free choice.","section":"Section VIII-F, Algorithm 1, Table VI"},{"comment":"All reported AC/PR/RC/F1 numbers appear to come from a single run with no error bars, confidence intervals, or multiple seeds. The headline claims (16.66%, 29.79%, and 33.42% average F1 improvements in Section VIII-B.2, and the 9.16% and 12.08% gains on ISCX-Botnet) are differences between single runs; without variance estimates it is unclear whether the gaps are beyond training noise, especially for smaller-margin comparisons such as the I.I.D. results against ET-BERT. Please report means and standard deviations over at least three seeds and, where possible, a paired significance test.","section":"Section VIII-B, Tables IV and V"},{"comment":"The paper corrects the APP53 label set from 22 to 25 classes and supplements two botnet classes in ISCX-Botnet multi-class, which changes the benchmark relative to the original FDAN-APP53 and ISCX-Botnet releases. If baselines were not re-tuned on the corrected label sets, their reported numbers may be lower than what those methods would achieve on the corrected data, inflating ETooL's relative gains. Please state explicitly whether all baselines were re-trained and re-tuned on the same corrected splits, and report the number of label-mapping changes per dataset.","section":"Section VIII-A2, Table III"},{"comment":"The NETD generalization claims are supported only by a figure with no numerical table, no error bars, and no statistical test; the figure is also hard to read because labels appear as corrupted or unicode strings. Please add a table with exact F1 and accuracy values for all methods on NETD-1 through NETD-4, including variance, so the claim that ETooL 'significantly outperforms' other methods can be verified.","section":"Section VIII-D, Figure 4"}],"minor_comments":[{"comment":"The first paragraph states that a too-small BURST time threshold aggregates flows serving different functionalities; this appears to be the opposite of the mechanism described, since a too-small threshold should split BURSTs rather than merge them. Please correct or rephrase.","section":"Section VIII-F"},{"comment":"The contrastive loss uses a similarity matrix and a target y described as a vector; please clarify the dimensions, the role of the temperature coefficient tau, and what constitutes positive and negative pairs in this alignment.","section":"Section VI-A, Eqs. (3)-(4)"},{"comment":"The edge-insertion procedure is hard to follow from the pseudocode alone; please add a textual step-by-step explanation of how burst edges and adjacency edges are formed, including how isolated flows are handled.","section":"Algorithm 1, lines 9-15"},{"comment":"The table contains placeholder or unicode strings in several rows and an incomplete latency entry; please regenerate the table cleanly.","section":"Table VII"},{"comment":"The axis labels and legends contain unicode escape sequences (e.g., '/uni0000002f/...'); please replace them with proper labels such as 'BURST Time Threshold' and 'Learning Rate'.","section":"Figure 5"},{"comment":"The abstract reports significant F1 improvements without mentioning that the APP53 label set was corrected and that NETD is private; please add a reproducibility statement in the experimental settings specifying data availability and label versions.","section":"Abstract and Section VIII-A2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong empirical systems paper for a security journal. My main concern is that the central O.O.D. invariance claim is supported by test-selected hyperparameters and single-run numbers, which is fixable with additional experiments. I would not recommend rejection because the method and dataset are of interest if the robustness checks pan out. I would also encourage the editor to ask the authors to release code and, if possible, the NETD construction pipeline, since the label corrections and private dataset currently make exact reproduction difficult."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-motivated empirical paper on using frozen LLMs plus flow-graph alignment for out-of-distribution encrypted traffic classification. The architecture is new in this domain: a traffic relation graph (TRG) built from flow-level BURSTs, a self-supervised BURST graph matching task, and two-stage instruction tuning with a frozen Vicuna backbone. The paper also introduces NETD, a synthetically shifted dataset, and an efficiency study showing that freezing the LLM cuts training cost roughly 50x. The evaluation is broad: seven datasets, IID and OOD tasks, ablations, and hyperparameter sweeps. The ablation study genuinely supports the design: removing graph structural tuning costs about 22 F1 points on average, and removing the LLM costs about 13. On its face, the method works.\n\nThe soft spots are real but not fatal. First, the central hyperparameter—the BURST time threshold γ—is chosen by sweeping and observing testing accuracy on the target OOD tasks (Section VIII-F). Since the graph component is essential, the reported OOD gains may partly reflect selection on the test set. This is the load-bearing weakness, and it is not addressed. Second, all results are single-run, no error bars, so we don't know if the gaps are outside noise. Third, the new dataset is private, and the APP53 label corrections (25 classes instead of 22, plus supplemented botnet classes) change the public benchmark; the authors are transparent about this, but it makes independent replication harder. Fourth, TrafficLLM is cited but not included as a baseline, which is a miss for an LLM-based traffic paper.\n\nThe stress-test note about γ selection holds up on reading. It doesn't refute the method; it means the invariance claim is currently supported only by test-selected numbers. The paper would be much stronger with a validation-based selection rule for γ, multiple seeds, a released dataset/code, and a direct comparison to TrafficLLM.\n\nWho this is for: network security researchers working on encrypted traffic classification and anyone interested in how OOD evaluation can be accidentally optimistic. I'd give it a serious referee: the method is coherent, the writing is clear, and the issues are addressable in revision. I would not yet cite the numbers as established.\n\nRecommendation: send to peer review with major revision requests: independent validation of γ, error bars, and code/data release.","headline":"A coherent LLM+graph method for OOD encrypted traffic classification with plausible gains, but test-set-tuned hyperparameters and missing reproducibility make the headline numbers not yet trustworthy.","tokens_in":26130,"tokens_out":3562,"would_cite":false,"duration_ms":35230,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that instruction-tuning a large language model on flow-interaction graphs, rather than on single-flow features, keeps encrypted traffic classification accurate when application versions, time windows, or botnet types…","keywords":["Encrypted traffic classification","Out-of-distribution generalization","Large language models","Instruction tuning","Traffic relation graph","BURST structure","Self-supervised learning","Network security"],"falsifier":"Train ETooL on one major version of an application and evaluate it zero-shot on the next version whose network implementation has been substantially rewritten, for instance switching from HTTP/2 to QUIC or changing connection-pooling behavior; if its F1 drops to the level of the best single-flow baseline rather than degrading only mildly, the claim that BURST graphs are transferable across version shifts is falsified. A complementary check is to shuffle the order of flows inside each BURST while preserving all marginal flow statistics; if accuracy stays high, graph topology is not carrying the improvement.","tokens_in":25114,"feed_emoji":"🔒","tokens_out":10921,"duration_ms":97057,"temperature":0.7,"pith_summary":"Encrypted traffic classifiers usually assume the traffic they see at test time comes from the same distribution they trained on, an assumption that breaks when applications update, time passes, or new malware types appear. This paper tries to establish that an instruction-tuned large language model can absorb that drift instead of fighting it, provided the input is a graph of how network flows form BURSTs, which are clusters of flows that start within roughly one second of one another. The proposed model, ETooL, turns traffic into a Traffic Relation Graph, aligns graph structure with the language model's token space through contrastive learning, then tunes only a small projection layer using a self-supervised BURST graph matching task followed by task-specific instructions, with the LLM backbone frozen. The paper reports F1 scores of 93.19% on same-distribution app classification, 74.88% and 72.13% on time-shift and version-shift app tasks, and 95.03% on binary botnet detection, with the largest margins over baselines appearing exactly in the out-of-distribution settings. If correct, this means accurate traffic identification after distribution shifts is achievable without collecting labels from the new distribution.","feed_headline":"Flow graphs keep encrypted traffic ID accurate through app updates","feed_subtitle":"A frozen LLM taught on burst graphs lifts out-of-distribution F1 scores by up to 33 points.","key_machinery":"The central object is the Traffic Relation Graph (TRG), built from flow-level BURST structures: each node is a network flow carrying a raw datagram byte-pair sequence and a directed packet-size sequence, burst edges connect flows whose start timestamps fall within a small threshold γ, chosen as 1 second, and adjacency edges connect the last flow of one BURST to the first and last flows of the next. The load-bearing mechanism is the two-stage instruction tuning: a contrastive objective aligns the graph encoder's structure-level representation with a flow encoder's node-level representation, a self-supervised BURST graph matching task teaches the LLM to reorder disrupted BURST traffic features by matching them to graph nodes, and a final task-specific instruction tune adapts the small structure-aware projection head to classification. All LLM and graph-encoder parameters stay frozen, so only about 131 million projection parameters are tuned, which the paper identifies as the reason the approach trains in reasonable time and GPU memory.","core_discovery":"On the paper's own terms, ETooL's discovery is that flow interaction topology is the transferable signal for encrypted traffic classification under distribution shift. The TRG represents each flow as a node carrying a raw datagram byte-pair sequence and a directed packet-size sequence, with burst edges linking flows that start within the time threshold γ and adjacency edges linking consecutive BURST structures. A two-stage instruction-tuning procedure teaches a 7-billion-parameter LLM to read these graphs: first, contrastive learning aligns the graph encoder's representations with flow encodings, and a self-supervised BURST graph matching task asks the model to reorder shuffled BURST features by graph topology; second, task-specific instructions fine-tune only the projection layer while both the LLM and graph encoder remain frozen. In the evaluation, ETooL holds F1 at 93.19% and 92.11% on the same-distribution APP53 tasks and at 74.88% and 72.13% on the time-shift and version-shift tasks, while reaching 95.03% and 81.95% on binary and multi-class ISCX-Botnet detection; the improvements over the best baselines in the shifted settings are 18.17, 15.15, 9.16, and 12.08 F1 points respectively.","pith_inferences":["Editorial inference: the same BURST-graph recipe could be carried to protocols and settings the paper did not test, such as QUIC traffic or non-mobile platforms, because the claimed invariant is flow interaction topology rather than app-specific bytes.","Editorial inference: the central role of γ means deliberate jittering of flow start times by a network middlebox or adversary could break the claimed invariance; measuring ETooL under such timing perturbation would probe the mechanism directly.","Editorial inference: the self-supervised BURST graph matching task is a general pattern, and any domain with recurring co-occurrence structure could be instruction-tuned into an LLM the same way, using unlabeled graph structure instead of task labels.","Editorial inference: the reported gains are averaged against several baselines, and a controlled comparison that matches model size and input length against the strongest baseline, while ablating the graph structure, would isolate how much of the improvement comes from the graph representation itself."],"forward_implications":["In the APP53 time-shift and version-shift tasks, ETooL shows the smallest performance degradation among all compared methods, with average F1 improvements of 16.66%, 29.79%, and 33.42% over representative baselines.","On ISCX-Botnet, the same model detects traffic from botnet types never seen in training, reaching 95.03% F1 binary and 81.95% multi-class, beating the strongest baseline by 9.16 and 12.08 points.","Because the LLM and graph encoder are frozen, tuned parameters drop by more than 50 times compared with full-parameter LLM fine-tuning, making a 7-billion-parameter instruction-tuned classifier practical on one 80 GB GPU.","The generalization is zero-shot with respect to the shifted distribution: task tuning is done only on source-distribution data, and the shifted data is classified without retraining or relabeling.","The new NETD dataset, built by controlling proportional and compositional bias inside ISCX-VPN, positions the method to be stress-tested under adjustable degrees of distribution shift rather than only on fixed splits."],"supporting_citations":[{"why":"Supplies the flow-level BURST/graph representation idea and the APP53 dataset with its time- and version-shift splits.","marker":"[40]"},{"why":"ET-BERT is both the strongest pre-training baseline and the flow encoder used inside ETooL for node-level representations.","marker":"[12]"},{"why":"ISCX-Botnet provides the out-of-distribution malicious-traffic tasks with unseen botnet types.","marker":"[27]"},{"why":"ISCX-VPN is the public dataset from which NETD constructs controlled proportional and compositional distribution shifts.","marker":"[29]"},{"why":"PERT is the pre-training baseline that represents transformer-based traffic classification without O.O.D. handling.","marker":"[24]"},{"why":"AppScanner is the statistical-feature baseline whose packet-size random forest degrades sharply under shift.","marker":"[39]"},{"why":"FS-Net is the recurrent-network baseline on directed packet-size sequences, one of the main comparison points.","marker":"[22]"},{"why":"GraphDApp is the graph-neural-network baseline for application identification, the closest structural competitor.","marker":"[7]"},{"why":"CETP supplies the byte-pair token expansion used to represent raw datagrams as LLM input tokens.","marker":"[30]"},{"why":"Supplies the Non-I.I.D. Index (NI) used to quantify distribution shift when evaluating NETD and existing datasets.","marker":"[33]"}],"fun_headline_variants":["Burst graphs keep encrypted traffic ID stable across app shifts","Frozen LLM reads flow topology to beat traffic distribution drift","ETooL: flow burst topology transfers for out-of-distribution traffic ID","Flow interaction graphs, not payloads, fight encrypted traffic drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that flow-level BURST structure, meaning the pattern of which flows start within about one second of each other, stays similar enough when an application updates, when time passes, or when a new botnet type appears that the Traffic Relation Graph built from it means the same thing in the new distribution as in the training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Burst graphs keep encrypted traffic ID stable across app shifts","Frozen LLM reads flow topology to beat traffic distribution drift","ETooL: flow burst topology transfers for out-of-distribution traffic ID","Flow interaction graphs, not payloads, fight encrypted traffic drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":4096,"prompt_tokens":1152,"completion_tokens":2944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":768,"completion_tokens_details":{"reasoning_tokens":2871}},"tokens_in":768,"tokens_out":2944,"duration_ms":21498,"temperature":1.0,"reasoning_tokens":2871,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:45:36.053682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ETooL on one major version of an application and evaluate it zero-shot on the next version whose network implementation has been substantially rewritten, for instance switching from HTTP/2 to QUIC or changing connection-pooling behavior; if its F1 drops to the level of the best single-flow baseline rather than degrading only mildly, the claim that BURST graphs are transferable across version shifts is falsified. A complementary check is to shuffle the order of flows inside each BURST while preserving all marginal flow statistics; if accuracy stays high, graph topology is not carrying the improvement.","supporting_citations":[{"cited_title":"Accurate mobile-app fingerprinting using flow-level relationship with graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the flow-level BURST/graph representation idea and the APP53 dataset with its time- and version-shift splits."},{"cited_title":"ET-BERT: A contextualized datagram representation with pre-training transformers for encrypted traffic classification,","cited_arxiv_id":null,"evidence_quote":"ET-BERT is both the strongest pre-training baseline and the flow encoder used inside ETooL for node-level representations."},{"cited_title":"Unbiased look at dataset bias,","cited_arxiv_id":null,"evidence_quote":"ISCX-Botnet provides the out-of-distribution malicious-traffic tasks with unseen botnet types."},{"cited_title":"Towards effective feature selection in machine learning-based botnet detection approaches,","cited_arxiv_id":null,"evidence_quote":"ISCX-VPN is the public dataset from which NETD constructs controlled proportional and compositional distribution shifts."},{"cited_title":"PERT: payload encoding representation from transformer for encrypted traffic classification,","cited_arxiv_id":null,"evidence_quote":"PERT is the pre-training baseline that represents transformer-based traffic classification without O.O.D. handling."},{"cited_title":"Robust smartphone app identification via encrypted network traffic analysis,","cited_arxiv_id":null,"evidence_quote":"AppScanner is the statistical-feature baseline whose packet-size random forest degrades sharply under shift."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GraphDApp is the graph-neural-network baseline for application identification, the closest structural competitor."},{"cited_title":"& Xiong, G","cited_arxiv_id":null,"evidence_quote":"CETP supplies the byte-pair token expansion used to represent raw datagrams as LLM input tokens."},{"cited_title":"Towards non-i.i.d. image classification: A dataset and baselines,","cited_arxiv_id":null,"evidence_quote":"Supplies the Non-I.I.D. Index (NI) used to quantify distribution shift when evaluating NETD and existing datasets."}],"review_version":1}