REVIEW 4 major objections 5 minor 18 references
MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A 0.66-billion-parameter mixture-of-experts classifier built from distilled three-layer GPT-2-base models matches or beats a 7-billion-parameter generative baseline on ten encrypted-traffic benchmarks while using 85-90% less inference…
desk verdict The 0.66B parameter claim is contradicted by the paper's own component counts, but the new combination of distilled per-dataset experts with hard gating is worth a serious look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a deterministic one-hot gating network implemented as a distilled, pruned GPT-2-base model, together with a set of three-layer distilled GPT-2-base experts. The gating network takes the contextual feature embedding of a traffic instance and outputs a binary vector with exactly one nonzero entry, forcing a single expert to produce the classification. This hard-routing design converts the mixture-of-experts idea into a conditional computation scheme where the active parameter count per inference is small and fixed, and that is what delivers the resource reduction while retaining specialized accuracy.
What would settle it
Evaluate the trained framework on a held-out mixture of traffic drawn from two or more of the ten datasets with labels shuffled across sources, and compare its gating choices against an oracle that always activates the expert with lowest loss on that sample; if the gate's agreement with the oracle is near chance, the one-hot routing has not generalized.
Extended reading notes
Core claim
The central discovery is that compact models distilled from GPT-2-base, each reduced to three transformer layers, retain most of the classification accuracy of the full model, and that organizing these distilled models as specialized experts under a deterministic one-hot gating network yields a single framework that is competitive or superior to a 7-billion-parameter baseline on all ten datasets. Each expert is trained on one dataset with soft labels from a fine-tuned teacher and hard labels from ground truth; the gating network selects exactly one expert per incoming traffic instance. Because only one three-layer expert runs at inference time, the computational cost stays on the order of a small model while performance remains at the level of a much larger generative classifier. The paper reports F1 gains over the large baseline on six datasets, parity on the rest, and an 85-90% reduction in inference time and memory use.
Load-bearing premise
The entire scalability claim rests on the assumption that the one-hot gating network routes each incoming traffic instance to the genuinely appropriate expert; the paper specifies neither the gating training objective nor an evaluation against unseen or mixed flow types, so the routing could in principle be memorizing which dataset a sample came from rather than learning traffic semantics.
Editorial extensions
If this is right
- Deploying LLM-based encrypted traffic classification on resource-constrained edge devices becomes plausible, since the 0.66-billion-parameter framework uses 85-90% less inference time and memory than the 7-billion-parameter baseline.
- Replacing generative, prompt-based classification with direct final-token classification removes the overhead of prompt construction and response parsing, lowering latency without sacrificing measured accuracy.
- Distilling teachers down to three-layer students keeps performance stable, while pruning further to a single layer causes a notable accuracy decline, setting a practical compression limit.
- Adding contextual metadata such as protocol types and IP addresses as natural-language prompts improves precision, recall, and F1 on the evaluated datasets.
- A single unified large model is not necessary for diverse encrypted traffic; a collection of specialized experts with one-hot routing can match or exceed it per task.
Reading between the lines
- Editorial inference: The gating network's generalization is untested across mixed or unseen flow types, so a natural extension is to measure routing agreement with an oracle expert on held-out traffic distributions.
- Editorial inference: The same distillation-plus-one-hot-gating recipe could apply to other domains with separable input types, such as intrusion detection across protocols or malware family classification.
- Editorial inference: Because routing is deterministic, auditing which metadata tokens drive expert selection could provide explainability for encrypted-traffic decisions, a useful property where labels are scarce.
- Editorial inference: Relaxing the one-hot constraint to a learned top-k budget would only modestly raise inference cost and could improve robustness on mixed flows, though the paper does not test this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MERLOT, a mixture-of-experts framework for encrypted traffic classification in which several distilled GPT-2-base models act as specialized experts and a gating network selects one expert per instance. Each expert is trained by knowledge distillation from a task-specific fine-tuned teacher, and input flows are augmented with natural-language metadata such as protocol and IP addresses. The authors report results on 10 public encrypted-traffic benchmarks, claim superiority or competitiveness against the 7-billion-parameter TrafficLLM, and state an 85-90% reduction in inference time and memory for a 0.66-billion-parameter MERLOT variant. Ablations cover the number of student layers, contextual feature embedding, and distillation hyperparameters.
Significance. If the central claims were established, MERLOT would be a practically valuable contribution: a relatively small, distilled MoE classifier that approaches the accuracy of a much larger LLM-based baseline while reducing resource consumption. The paper has useful strengths: it evaluates on 10 public datasets, compares against strong baselines (ET-BERT and TrafficLLM), and provides ablation evidence for distillation and contextual feature embedding. However, the headline quantitative claims are internally inconsistent, and the gating mechanism that is central to the MoE design lacks a specified training objective and generalization test. As written, the paper does not support its main conclusions, so the significance of the contribution cannot be assessed from the presented evidence.
major comments (4)
- [Section IV.B; Abstract; Table II] The parameter-count arithmetic contradicts the paper's central efficiency claim. Section IV.B states that MERLOT has 10 distilled experts of about 85 million parameters each and a gating network of 935 million parameters; this totals approximately 1.785 billion, not the 0.66 billion stated in the Abstract or the 660M (or 600M, as Section IV.B also says) used in Table II. Even if '935 million' were a typo for '93.5 million', the total would be 943.5 million, still not 660 million. Furthermore, Section III.A establishes that GPT-2-base has about 117 million parameters, so a 'distilled, pruned GPT-2-base' gating network cannot have 935 million parameters. Because the abstract's headline '0.66-billion-parameter MERLOT' and the 85-90% resource reduction claim depend directly on this count, the efficiency conclusion is not supported.
- [Table II; Section IV.B] The claim that MERLOT achieves 'superior precision, recall, and F1 scores on 6 of 10 datasets than the 7-billion-parameter TrafficLLM' is not supported by the table. For the 660M variant, the F1 values are higher than TrafficLLM on ISCX Tor 2016, CSTNET 2023, USTC TFC 2016, CIC DoHBrw 2020, and CSIC 2010 (five datasets), equal on ISCX Botnet 2014, and lower on ISCX VPN 2016, APP-53 2023, CW-100 2018, and DAPT 2020. Thus the evidence shows at best 5 wins and 4 losses, not superiority on 6 of 10, and the statement that MERLOT performs 'comparably' on the remaining datasets is inaccurate for four datasets where the gap is substantial (e.g., APP-53: 0.8601 vs. 0.9320). The claim should be restated and the observed gaps analyzed, with error bars or repeated-seed statistics if available.
- [Section III.C, Eq. (3); Figure 3; Section IV.A] The gating network, which is load-bearing for the MoE claim, has no specified training objective or training procedure. Eq. (3) defines the one-hot selection vector but does not say how G(x; theta_G) is optimized, how hard (non-differentiable) routing is trained, or what supervision signal is used. This omission matters because the framework as described in Section III.B trains a separate teacher (and hence a separate expert) for each individual dataset; if the gate learns to recognize the dataset identity from metadata such as IP addresses, the MoE architecture would reduce to a dataset selector rather than a general traffic classifier. Figure 3(a) shows distinct embedding clusters for different datasets, which is consistent with this concern. No experiment tests routing on mixed or unseen traffic types, so the generalization and scalability claims are unsupported.
- [Section IV.B; Conclusion] The paper claims an 85-90% reduction in inference time and memory usage, but no runtime, memory, or throughput measurements are reported. Section IV.B gives only an asymptotic complexity argument O(L * L_seq * D_hidden^2) and says the reduction is 'corroborated by empirical results as well,' yet no table or figure shows wall-clock time, peak memory, or FLOPs for MERLOT versus TrafficLLM. The conclusion repeats the resource-efficiency claim as though it had been demonstrated. If the parameter count were corrected to 1.785B as per Section IV.B, the claimed reduction would also need to be re-derived; empirical measurements are essential.
minor comments (5)
- [Table I; Section IV.A] The dataset name 'CSIS2010' in Section IV.A is inconsistent with 'CSIC 2010' in Table I and elsewhere; please correct the typo.
- [Section IV.B] The text uses '600-million-parameter MERLOT' while the Abstract says 0.66 billion and Table II says 660M; unify the size description across all occurrences.
- [Figure 3] The right panel is labeled 'Predicted labels by the gating function and expert models in MoE,' but it is unclear whether these are predicted traffic classes or predicted routing choices; clarify the caption and the y-axis labels.
- [Section IV.A] No random seed, number of runs, or standard deviation is reported for the experiments, and no code or data-split details are supplied; adding reproducibility information would strengthen the empirical claims.
- [Section III.C] The reference to 'deterministic routing [10]' is appropriate for hard gating, but the remark that soft gating systems cause 'fluctuating activation patterns' should cite the specific evidence for that claim.
Circularity Check
No significant circularity: MERLOT's distilled experts are trained by standard teacher-student distillation against external task teachers and evaluated against external baselines; the only mild concern is hyperparameter selection on one dataset used in reporting that same dataset.
full rationale
MERLOT's derivation chain is self-contained. Each expert is a distilled GPT-2-base student trained with Eq. (2), a composite hard cross-entropy plus KL loss against a teacher fine-tuned on that dataset, and the reported accuracy numbers are measured against ET-BERT and TrafficLLM, both external baselines. No load-bearing premise is justified by a self-citation: the cited references are standard external work, and TrafficLLM [8] is an external repository. The MoE routing in Eqs. (3)-(4) is an architectural choice, not a quantity derived from the benchmark outputs. The only mild concern is that the distillation hyperparameters (alpha = 0.5, temperature = 2.0) are described after Table IV, which is computed on USTC TFC 2016, and the same dataset appears in the main comparison of Table II. This is a tuning/selection concern and a correctness risk, but not circularity, because the expert parameters are not fitted to the reported benchmark F1 values; the hard-label loss still uses the dataset labels, and the comparison against TrafficLLM is external. Separately, Section IV.B's parameter arithmetic (10 x 85M + 935M = 1.785B, and 935M exceeding the stated 117M GPT-2-base size) contradicts the claimed 0.66B/660M/600M sizes, but an arithmetic inconsistency is not a circular definition or a fitted-input-called-prediction step. Overall, the central claim rests on independent, externally validated comparisons.
Assumptions & free parameters
free parameters (6)
- distillation coefficient alpha =
0.5
- distillation temperature =
2.0
- student transformer layers =
3
- number of experts K =
10
- train/test split ratio =
95:5
- training epochs =
5
assumptions (5)
- standard math The final decoder token aggregates global context from all preceding tokens in an autoregressive transformer.
- domain assumption Encrypted traffic metadata (IPs, ports, timing, protocol headers) retains implicit correlation with the traffic class despite encryption.
- domain assumption Tshark-extracted features and template-based natural-language augmentation preserve the semantics needed for classification.
- ad hoc to paper A hard one-hot gating mechanism provides deterministic routing that prevents fluctuating activation patterns and improves accuracy.
- domain assumption The same hyperparameters and 5-epoch training protocol transfer across all 10 datasets.
Cite this review
Pith. "Pith review of MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification." pith.science (2026). https://pith.science/paper/UDAKYYIA
@misc{pith2026241113004,
author = {Pith},
title = {Pith review of: MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/UDAKYYIA}},
note = {Machine review of arXiv:2411.13004}
}
read the original abstract
We present MERLOT, a scalable mixture-of-expert (MoE) based refinement of distilled large language model optimized for encrypted traffic classification. By applying model distillation techniques in a teacher-student paradigm, compact models derived from GPT-2-base retain high classification accuracy while minimizing computational costs. These models function as specialized experts in an MoE architecture, dynamically assigned via a gating network. Unlike generation-based methods, our approach directly classifies encrypted traffic using the final decoder token with contextual feature embedding as input. Experiments on 10 datasets show superior or competitive performance over the state-of-the-art models while significantly reducing resource demands, underscoring its effectiveness and robustness.
Figures
Reference graph
Works this paper leans on
-
[1]
Network traffic classification for data fusion: A survey,
J. Zhao, X. Jing , et al. , “Network traffic classification for data fusion: A survey,” Inf. Fusion , vol. 72, pp. 22–47, 2021
work page 2021
-
[2]
Optimizing deep packet inspection for high-speed traffic analysis,
N. Cascarano, L. Ciminiera , et al. , “Optimizing deep packet inspection for high-speed traffic analysis,” J. Netw. Syst. Manag. , vol. 19, pp. 7–31, 2011
work page 2011
-
[3]
Deep learning and zero-day traffic classification: Lessons learned from a commercial-grade dataset,
L. Yang, A. Finamore , et al. , “Deep learning and zero-day traffic classification: Lessons learned from a commercial-grade dataset,” IEEE Trans. Netw. Serv. Manag. , vol. 18, no. 4, pp. 4103–4118, 2021
work page 2021
-
[4]
Network traffic classification: Tech- niques, datasets, and challenges,
A. Azab, M. Khasawneh , et al. , “Network traffic classification: Tech- niques, datasets, and challenges,” Digit. Commun. Netw. , vol. 10, no. 3, pp. 676–692, 2024
work page 2024
-
[5]
Language models are unsupervised multitask learners,
A. Radford, J. Wu , et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
work page 2019
-
[6]
X. Lin, G. Xiong , et al. , “ET-BERT: A contextualized datagram rep- resentation with pre-training transformers for encrypted traffic classifi- cation,” in Proc. ACM Web Conf. 2022 , Lyon, France, April 2022, pp. 633–642
work page 2022
-
[7]
NetGPT: Generative pretrained transformer for network traffic,
X. Meng, C. Lin , et al. , “NetGPT: Generative pretrained transformer for network traffic,” arXiv preprint arXiv:2304.09513 , 2023
arXiv 2023
-
[8]
TrafficLLM: A universal LLM adaptation framework for network traffic analysis,
ZGC-LLM-Safety, “TrafficLLM: A universal LLM adaptation framework for network traffic analysis,” accessed: November 15, 2024. [Online]. Available: https://github.com/ZGC-LLM-Safety/TrafficLLM
work page 2024
Show all 18 references
-
[9]
NetLLM: Adapting large language models for networking,
D. Wu, X. Wang , et al. , “NetLLM: Adapting large language models for networking,” in Proc. ACM SIGCOMM Conf. 2024 , New York, NY , USA, August 2024, pp. 661–678
2024
-
[10]
Adaptive mixtures of local experts,
R. A. Jacobs, M. I. Jordan , et al. , “Adaptive mixtures of local experts,” Neural Comput. , vol. 3, no. 1, pp. 79–87, 1991
1991
-
[11]
Distilling the knowledge in a neural network,
G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015
2015 arXiv
-
[12]
Generative adversarial classification network with application to network traffic classification,
R. Ghanavi, B. Liang , et al. , “Generative adversarial classification network with application to network traffic classification,” in Proc. IEEE GLOBECOM. Madrid, Spain: IEEE, December 2021, pp. 1–6
2021
-
[13]
Mixture-of-experts with expert choice routing,
Y . Zhou, T. Lei , et al. , “Mixture-of-experts with expert choice routing,” in Proc. NeurIPS, vol. 35. New Orleans, LA, USA: Curran Associates, Inc., December 2022, pp. 7103–7114
2022
-
[14]
Harder tasks need more experts: Dynamic routing in moe models,
Q. Huang, Z. An , et al. , “Harder tasks need more experts: Dynamic routing in moe models,” arXiv preprint arXiv:2403.07652 , 2024
2024
-
[15]
Wireshark Network Protocol Analyzer,
“Wireshark Network Protocol Analyzer,” https://www.wireshark.org/, accessed: Nov. 16, 2024
2024
-
[16]
C. J. van Rijsbergen, Information Retrieval , 2nd ed. London, UK: Butterworth-Heinemann, 1979, accessed: Nov. 17, 2024. [Online]. Available: http://www.dcs.gla.ac.uk/Keith/Preface.html
1979
-
[17]
Attention is all you need,
A. Vaswani, N. Shazeer , et al. , “Attention is all you need,” in Proc. NeurIPS, vol. 30. Curran Associates, Inc., 2017, pp. 5998–6008
2017
-
[18]
Visualizing data using t-SNE
L. Van der Maaten and G. Hinton, “Visualizing data using t-SNE.” J. Mach. Learn. Res. , vol. 9, no. 11, 2008
2008
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.