REVIEW 4 major objections 6 minor 38 references
Learning QoE from Packet-Level Measurements in Encrypted Video Conferencing Traffic
T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper claims that a self-attention-augmented CNN can infer BRISQUE and FPS from encrypted video-conferencing packet sizes alone, reaching 2.14% and 7.39% mean absolute error percentage on its WhatsApp dataset.
desk verdict Plausible architecture, but the headline MAEP numbers are not yet believable because the paper never explains how BRISQUE/FPS labels were computed or aligned to windows, and the 10-fold split ignores call boundaries. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core mechanism is qAttCNN, a pipeline that turns a 350-sample packet-size vector into an N×N matrix via a 1×1 convolution embedding, applies masked multi-head self-attention to weight global feature relevance while respecting causality, and lets a convolutional head extract local patterns before a fully connected layer outputs the QoE value. The same pipeline is trained with a custom MAEP loss, a cyclical learning-rate schedule, scheduled dropout, and additive-noise/row-shuffle augmentation, with the input normalized and transformed via FFT before attention.
What would settle it
Re-run the experiment with labels computed independently per 350ms window and with windows grouped by call into folds instead of randomly split; if the reported 2.14% and 7.39% errors rise substantially, the original numbers reflect label leakage rather than predictive skill.
Extended reading notes
Core claim
The central claim is that qAttCNN—an embedding layer, a masked multi-head self-attention module, and a convolutional head—can infer BRISQUE and FPS from a 350-sample vector of packet sizes in encrypted WhatsApp video traffic. On the authors' dataset, qAttCNN achieves 2.14% MAEP for BRISQUE with a ResNet34 head and 7.39% MAEP for FPS with a ResNet18 head, the best results among all compared models. Ablation results support the design: removing the attention module raises BRISQUE error to 2.90% and FPS error to 10.58%, while removing the CNN head causes much larger degradation. The authors interpret this as evidence that packet size alone, transformed into a 2D representation and processed by
Load-bearing premise
The result depends on the ground-truth BRISQUE and FPS labels being correctly computed and aligned with each 350-packet-size window, and on random cross-validation splits not putting windows from the same call into both training and test sets; the paper does not describe either procedure.
Editorial extensions
If this is right
- ISPs could monitor BRISQUE and FPS in encrypted video calls using only passive packet-size observations.
- qAttCNN outperforms classical machine-learning baselines (linear regression, SVM, boosting, random forest) and the earlier QoENet1D on both predicted metrics.
- The ablation results indicate that both the attention module and the FFT preprocessing contribute to the reported accuracy, since removing them increases MAEP.
- The best head depth depends on the label type: a deeper residual head works better for continuous BRISQUE, while a shallower head works better for discrete FPS.
- The approach is relatively simple to implement and does not require specialized traffic inspection hardware.
Reading between the lines
- The main open question is cross-platform generalization: since training used only WhatsApp traffic, testing on Zoom, Meet, or Telegram calls would reveal whether packet-size patterns are platform-specific or transferable.
- The tiny standard deviations across folds (e.g., ±0.025 MAEP) may understate real-world variability because the dataset appears to come from a single controlled session; a multi-session, multi-device dataset would give more meaningful confidence intervals.
- The reported MAEP could hide calibration problems, since predicting the mean label would already land near the mode of the Gaussian-like BRISQUE distribution; plotting residuals against true BRISQUE would clarify how much of the 2.14% reflects genuine signal rather than averaging.
- Because the paper does not specify how adjacent 350-sample windows are constructed or how the ground-truth labels are aligned with them, a public benchmark with non-overlapping windows and explicit label-generation code would make the comparison reproducible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes qAttCNN, a hybrid self-attention/CNN model that predicts two no-reference QoE metrics, BRISQUE and FPS, from 350-dimensional packet-size measurements extracted from encrypted WhatsApp video-call traffic. The authors evaluate qAttCNN on a custom dataset of 51,341 1-ms line measurements using 10-fold cross-validation and report MAEP of 2.14% for BRISQUE and 7.39% for FPS, outperforming linear regression, SVM, boosting, random forest, and their previous QoENet1D models. An ablation study indicates that the CNN head is essential, while removing FFT or attention degrades performance moderately. The paper also compares ResNet-18/34/50 heads and proposes cyclical learning-rate and dropout schedules. The main claims are the two MAEP numbers and the superiority over baselines.
Significance. If the reported numbers are valid, the contribution is practically relevant: a relatively simple, packet-size-only QoE predictor for encrypted conferencing traffic, useful for ISPs and reproducible with moderate resources. Strengths include using the same MAEP loss and evaluation metric, a sensible ablation design, comparison against multiple baselines, and an explicit statement of the single-platform limitation. The significance is conditional, however, because the ground-truth label generation and alignment are not described, and the cross-validation protocol does not prevent temporal leakage. These issues affect every quantitative result in the paper.
major comments (4)
- [Section 3, Section 3.1.2, Table 7] The dataset section describes only raw line measurements (features in Table 1). It never states how BRISQUE and FPS ground-truth labels were computed for each 350-sample window, what window stride/overlap was used, or how labels were synchronized with packet timestamps. Since all reported MAEP values in Table 7 are computed against these labels, the central quantitative claim (2.14% BRISQUE, 7.39% FPS) is not reproducible and could reflect misalignment or label leakage. This is load-bearing and must be fixed by a precise label-generation protocol.
- [Section 5.3.1] The 10-fold CV splits are described as 'random, non-overlapping subset of the test data' with no grouping by session or call. Section 3 says the data come from 'a WhatsApp video call' (singular); if windows from the same call appear in both training and test folds, adjacent windows are autocorrelated and the model can memorize temporal structure rather than learn a general mapping. Please report results with grouped-by-call or temporal hold-out splits, including a gap between train and test windows, and per-call metrics.
- [Section 3, Table 2; Section 5.3.1] The input feature is defined inconsistently: Table 2 calls 'packet size 1' the 'Number of arriving packets in the first time interval,' while Section 3.1 and Section 4 describe 'instantaneous packet size (PCKT)' and 'sizes of data packets.' Clarify whether the model input is per-ms packet count, per-packet byte length, or another quantity, and specify the exact preprocessing pipeline (normalization, FFT, embedding, attention order) because Section 4.1 and Section 5.3.1 appear to disagree on whether embedding precedes attention.
- [Section 5.3.2] Pretrained ImageNet ResNet weights are used on a 350×350 single-channel input, but the manuscript does not explain how the first convolutional layer is adapted for single-channel input or how the 350×350 resolution is reconciled with the ResNet input size. Without this detail, the transfer-learning setup is not reproducible and the 'almost stock CNN' claim is unsupported.
minor comments (6)
- [Throughout] Typos and grammar: 'fo ISPs', 'conjuncture', 'who’s', 'tomet', 'an its'.
- [Table 4] Table 4 header says ResNet x16/x34/x50 whereas text and Table 7 say x18/x34/x50; standardize architecture names.
- [Eq. (11)] The sum runs from i=0, but vectors are presumably 1-indexed; define N and the index range explicitly.
- [Section 5.3.1] The text says 'tested on an unseen test set, which constituted 10% of the data' and then describes 10-fold CV; reword to avoid ambiguity.
- [Section 2] The related-work survey includes applications in electricity pricing, carbon emissions, TEC, and silk weaving (Refs. [14,15,36,38]) that are only loosely connected; consider trimming to keep focus.
- [Section 3] State explicitly how many 350-sample windows were formed from the 51,341 1-ms records and what overlap was used; this is needed to interpret sample size and CV.
Circularity Check
No circularity: results are measured on held-out CV folds; self-citations are not used to define the prediction target.
full rationale
This paper reports an empirical CNN model evaluation; there is no mathematical derivation chain claimed. The reported MAEP values (2.14% for BRISQUE, 7.39% for FPS) are obtained by training on 90% of the dataset and evaluating on held-out 10% folds using 10-fold CV (Section 5.3.1). The loss in Eq. (13) is the same MAEP criterion used in Eq. (11); this is appropriate optimization/evaluation alignment, not circular. BRISQUE and FPS labels are described only via distributions (Section 3.1.2); the paper does not define them as functions of the packet-size inputs, so the prediction target is not self-referential. Self-citations to [27], [28], and [29] supply architectural inspiration, baseline models, the MAEP metric, and the dataset; none is invoked as a uniqueness theorem or as external proof of the reported accuracy. The stated limitations (long training time, large-data dependence, single-platform data) concern generalization and do not indicate that the reported numbers are imposed by construction. Potential label-alignment and temporal-leakage issues are correctness risks, not circularity. Therefore, no load-bearing step reduces to its inputs by the paper's own equations or citations.
Assumptions & free parameters
free parameters (6)
- Input sequence length N =
350
- Number of attention heads =
35
- Learning-rate schedule parameters =
Initial 5e-3, period 4×50 epochs, specific decay steps
- Dropout schedule parameters =
p increases by 0.1 every 50 epochs to max 0.5
- Augmentation thresholds =
n∈[0,0.3], s∈[0,0.4]
- Head model depth selection =
ResNet34 for BRISQUE, ResNet18 for FPS
assumptions (4)
- domain assumption Ground-truth BRISQUE and FPS labels are correctly computed and aligned with each 350-ms packet-size window.
- domain assumption The 51,341 samples are independent enough for random 10-fold CV to yield unbiased performance.
- ad hoc to paper ImageNet-pretrained ResNet weights can be applied to the 350×350 single-channel input.
- domain assumption MAEP is an appropriate metric when true labels can be zero.
Cite this review
Pith. "Pith review of Learning QoE from Packet-Level Measurements in Encrypted Video Conferencing Traffic." pith.science (2026). https://pith.science/paper/ADWG3ZIZ
@misc{pith2026260106862,
author = {Pith},
title = {Pith review of: Learning QoE from Packet-Level Measurements in Encrypted Video Conferencing Traffic},
year = {2026},
howpublished = {\url{https://pith.science/paper/ADWG3ZIZ}},
note = {Machine review of arXiv:2601.06862}
}
read the original abstract
The quality of the user experience has become one of the most important aspects in todays world, as it directly influences individuals willingness to continue using or abandon a product or service. In this context, video conferencing applications (VCAs), which experienced widespread adoption following the COVID-19 pandemic, must deliver excellent performance to remain competitive in an increasingly crowded market. Although content providers (CPs) such as Zoom, WhatsApp, Telegram, and Google Meet can assess conversation quality by comparing transmitted and received data. The widespread use of end-to-end encryption in VCAs makes quality-of-experience (QoE) evaluation by internet service providers (ISPs) far more challenging. Since ISPs do not have access to the encrypted content, they must rely on passive measurements of unencrypted traffic characteristics on the data path. In this work, we present a simple yet effective QoE prediction framework based on an almost stock convolutional neural network (CNN) architecture that uses only the packet sizes extracted from the communication between two participants in a video conferencing (VC) call to predict two QoE metrics: BRISQUE and MOS. The proposed framework is simple, easy to implement, and does not require high-end computational resources, yet it provides superior prediction performance, as shown in our experiments on two custom datasets collected from WhatsApp and Zoom, which achieve substantial improvements over previous models for the QoE prediction task.
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
Mobile encrypted traffic classification using deep learning
Giuseppe Aceto et al. “Mobile encrypted traffic classification using deep learning”. In:2018 Network traffic measurement and analysis confer- ence (TMA). IEEE. 2018, pp. 1–8
2018
-
[2]
Bandwidth prediction in low-latency chun- ked streaming
Abdelhak Bentaleb et al. “Bandwidth prediction in low-latency chun- ked streaming”. In:Proceedings of the 29th ACM workshop on network and operating systems support for digital audio and video. 2019, pp. 7– 13
2019
-
[3]
Inferring streaming video quality from en- crypted traffic: Practical models and deployment experience
Francesco Bronzino et al. “Inferring streaming video quality from en- crypted traffic: Practical models and deployment experience”. In:Pro- ceedings of the ACM on Measurement and Analysis of Computing Sys- tems3.3 (2019), pp. 1–25
2019
-
[4]
Seq2img: A sequence-to-image based approach towards ip traffic classification using convolutional neural networks
Zhitang Chen et al. “Seq2img: A sequence-to-image based approach towards ip traffic classification using convolutional neural networks”. In:2017 IEEE International conference on big data (big data). IEEE. 2017, pp. 1271–1276
2017
-
[5]
Distribution of the estimators for autoregressive time series with a unit root
David A Dickey and Wayne A Fuller. “Distribution of the estimators for autoregressive time series with a unit root”. In:Journal of the American statistical association74.366a (1979), pp. 427–431
1979
-
[6]
Measuring video QoE from encrypted traf- fic
Giorgos Dimopoulos et al. “Measuring video QoE from encrypted traf- fic”. In:Proceedings of the 2016 Internet Measurement Conference. 2016, pp. 513–526
2016
-
[7]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy. “An image is worth 16x16 words: Transformers for image recognition at scale”. In:arXiv preprint arXiv:2010.11929 (2020)
arXiv 2010
-
[8]
Why (and how) networks should run themselves
Nick Feamster and Jennifer Rexford. “Why (and how) networks should run themselves”. In:arXiv preprint arXiv:1710.11583(2017)
arXiv 2017
Show all 38 references
-
[9]
Requet: Real-time QoE detection for encrypted YouTube traffic
Craig Gutterman et al. “Requet: Real-time QoE detection for encrypted YouTube traffic”. In:Proceedings of the 10th ACM Multimedia Systems Conference. 2019, pp. 48–59
2019
-
[10]
Reducing the dimen- sionality of data with neural networks
Geoffrey E Hinton and Ruslan R Salakhutdinov. “Reducing the dimen- sionality of data with neural networks”. In:science313.5786 (2006), pp. 504–507. 30
2006
-
[11]
Unsupervised traffic flow classification using a neu- ral autoencoder
Jonas H¨ ochst et al. “Unsupervised traffic flow classification using a neu- ral autoencoder”. In:2017 IEEE 42Nd Conference on local computer networks (LCN). IEEE. 2017, pp. 523–526
2017
-
[12]
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “Imagenet classification with deep convolutional neural networks”. In:Advances in neural information processing systems25 (2012)
2012
-
[13]
Backpropagation applied to handwritten zip code recognition
Yann LeCun et al. “Backpropagation applied to handwritten zip code recognition”. In:Neural computation1.4 (1989), pp. 541–551
1989
-
[14]
TEC Prediction Based on Att-CNN-BiLSTM
Haijun Liu et al. “TEC Prediction Based on Att-CNN-BiLSTM”. In: IEEE Access12 (2024), pp. 68471–68484
2024
-
[15]
Prediction of carbon emissions in Zhejiang province based on ATT-CNN-LSTM model
Xiaohu Liu et al. “Prediction of carbon emissions in Zhejiang province based on ATT-CNN-LSTM model”. In:2023 8th Asia Conference on Power and Electrical Engineering (ACPEE). IEEE. 2023, pp. 1918– 1922
2023
-
[16]
Network traffic classifier with convolu- tional and recurrent neural networks for Internet of Things
Manuel Lopez-Martin et al. “Network traffic classifier with convolu- tional and recurrent neural networks for Internet of Things”. In:IEEE access5 (2017), pp. 18042–18050
2017
-
[17]
Deep packet: A novel approach for en- crypted traffic classification using deep learning
Mohammad Lotfollahi et al. “Deep packet: A novel approach for en- crypted traffic classification using deep learning”. In:Soft Computing 24.3 (2020), pp. 1999–2012
2020
-
[18]
Neural adap- tive video streaming with pensieve
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. “Neural adap- tive video streaming with pensieve”. In:Proceedings of the conference of the ACM special interest group on data communication. 2017, pp. 197– 210
2017
-
[19]
The cost of the
David Naylor et al. “The cost of the” s” in https”. In:Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies. 2014, pp. 133–140
2014
-
[20]
A machine learning approach to classifying YouTube QoE based on encrypted network traffic
Irena Orsolic et al. “A machine learning approach to classifying YouTube QoE based on encrypted network traffic”. In:Multimedia tools and ap- plications76.21 (2017), pp. 22267–22301
2017
-
[21]
Qoe estimation method with time-series features extracted from packet flows for video streaming
Junki Oura et al. “Qoe estimation method with time-series features extracted from packet flows for video streaming”. In:2024 IEEE 21st Consumer Communications & Networking Conference (CCNC). IEEE. 2024, pp. 1–6. 31
2024
-
[22]
Website Fingerprinting at Internet Scale
Andriy Panchenko et al. “Website Fingerprinting at Internet Scale.” In:NDSS. Vol. 1. 2016, p. 23477
2016
-
[23]
On leveraging machine and deep learning for through- put prediction in cellular networks: Design, performance, and chal- lenges
Darijo Raca et al. “On leveraging machine and deep learning for through- put prediction in cellular networks: Design, performance, and chal- lenges”. In:IEEE Communications Magazine58.3 (2020), pp. 11–17
2020
-
[24]
Deep learning for encrypted traffic clas- sification: An overview
Shahbaz Rezaei and Xin Liu. “Deep learning for encrypted traffic clas- sification: An overview”. In:arXiv preprint arXiv:1810.07906(2018)
2018 arXiv
-
[25]
How to achieve high classification accu- racy with just a few labels: A semi-supervised approach using sampled packets
Shahbaz Rezaei and Xin Liu. “How to achieve high classification accu- racy with just a few labels: A semi-supervised approach using sampled packets”. In:arXiv preprint arXiv:1812.09761(2018)
2018 arXiv
-
[26]
DeepQoE: Real-time measurement of video QoE from encrypted traffic with deep learning
Meng Shen et al. “DeepQoE: Real-time measurement of video QoE from encrypted traffic with deep learning”. In:2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS). IEEE. 2020, pp. 1–10
2020
-
[27]
Revisiting informa- tion cascades in online social networks
Michael Sidorov, Ofer Hadar, and Dan Vilenchik. “Revisiting informa- tion cascades in online social networks”. In:Mathematics13.1 (2024), p. 77
2024
-
[28]
Estimating QoE from Encrypted Video Confer- encing Traffic
Michael Sidorov et al. “Estimating QoE from Encrypted Video Confer- encing Traffic”. In:Sensors (Basel, Switzerland)25.4 (2025), p. 1009
2025
-
[29]
Reliable QoE Prediction in IMVCAs Using an LMM-Based Agent
Michael Sidorov et al. “Reliable QoE Prediction in IMVCAs Using an LMM-Based Agent”. In:Sensors25.14 (2025), p. 4450
2025
-
[30]
Cyclical learning rates for training neural networks
Leslie N Smith. “Cyclical learning rates for training neural networks”. In:2017 IEEE winter conference on applications of computer vision (WACV). IEEE. 2017, pp. 464–472
2017
-
[31]
Dropout: a simple way to prevent neural net- works from overfitting
Nitish Srivastava et al. “Dropout: a simple way to prevent neural net- works from overfitting”. In:The journal of machine learning research 15.1 (2014), pp. 1929–1958
2014
-
[32]
An ionospheric TEC forecasting model based on a CNN-LSTM-attention mechanism neural network
Jun Tang et al. “An ionospheric TEC forecasting model based on a CNN-LSTM-attention mechanism neural network”. In:Remote Sensing 14.10 (2022), p. 2433
2022
-
[33]
Unsupervised deep learning for real-time assessment of video stream- ing services
Maria Torres Vega, Decebal Constantin Mocanu, and Antonio Liotta. “Unsupervised deep learning for real-time assessment of video stream- ing services”. In:Multimedia Tools and Applications76.21 (2017), pp. 22303– 22327. 32
2017
-
[34]
Attention is all you need
Ashish Vaswani et al. “Attention is all you need”. In:Advances in neural information processing systems30 (2017)
2017
-
[35]
A deep learning based method for handling imbalanced problem in network traffic clas- sification
Ly Vu, Cong Thanh Bui, and Quang Uy Nguyen. “A deep learning based method for handling imbalanced problem in network traffic clas- sification”. In:Proceedings of the 8th international symposium on in- formation and communication technology. 2017, pp. 333–339
2017
-
[36]
Process Quality Prediction Algorithm of Multi output Workshop Based on ATT-CNN-TCN
Bin Yi et al. “Process Quality Prediction Algorithm of Multi output Workshop Based on ATT-CNN-TCN”. In:Proceedings of the 2023 6th International Conference on Machine Vision and Applications. 2023, pp. 120–127
2023
-
[37]
Wide activation for efficient and accurate image super- resolution
Jiahui Yu et al. “Wide activation for efficient and accurate image super- resolution”. In:arXiv preprint arXiv:1808.08718(2018)
2018 arXiv
-
[38]
Electricity Price Prediction in Electricity Mar- kets Using an ATT-CNN-LSTM Model Based on the Sparrow Search Algorithm
Hongcheng Zhao et al. “Electricity Price Prediction in Electricity Mar- kets Using an ATT-CNN-LSTM Model Based on the Sparrow Search Algorithm”. In:2024 3rd Asia Power and Electrical Technology Con- ference (APET). IEEE. 2024, pp. 589–594. 33
2024
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.