Pith. sign in

REVIEW 3 major objections 4 minor 18 references

Intelligent Detection of Non-Essential IoT Traffic on the Home Gateway

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A machine-learning model on the home gateway can tell essential from non-essential IoT destinations using only aggregate traffic statistics, and scores 100% on destinations it has never seen.

desk verdict The paper's key generalization claim is undermined by a degenerate test set containing only non-essential destinations; the rest is a useful, honestly-reported systems paper that deserves a major-revision path. read the letter →

arxiv 2504.18571 v1 pith:WJ7JZH63 submitted 2025-04-22 cs.CR cs.LG

Fabio Palmese , Anna Maria Mandalari , Hamed Haddadi , Alessandro Enrico Cesare Redondi This is my paper · ORCID
classification cs.CRcs.LG
keywords IoTprivacynon-essentialtrafficclassificationmachinelearninghomegatewaysmartallow-listnetworkstatistics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a home gateway can tell which network destinations an IoT device genuinely needs from those that merely carry telemetry, ads, or other non-essential traffic, using only statistical properties of the traffic rather than domain names or IP addresses. The authors build labeled data by blocking each destination in turn and checking whether the device still performs its expected functions, then train a global binary classifier over 204 header-derived features. On their testbed of eight consumer devices from five categories, the classifier reaches near-perfect F1 scores, and destinations that first appear after the training period are flagged as non-essential with 100% accuracy. The payoff is a privacy-aware traffic-control layer that can block unwanted connections locally and in real time, without stale allow-lists or cloud-based analysis.

What carries the argument

The load-bearing object is the 204-feature windowed traffic descriptor. For each device, DNS mappings associate IPs with domain names, packets are grouped by destination, and non-overlapping time windows of 10, 60, 600, or 3600 seconds produce counts, packet-size statistics, interarrival-time statistics, protocol ratios, port counts, and flow counts split by protocol and direction. This descriptor carries the argument because it removes all identity information yet still allows a Random Forest classifier to separate essential from non-essential endpoints. The other half is the labeling procedure: blocking a destination and observing whether the device's functions still succeed turns 'essential' into a behavioral property, and repeating until at least 80% of iterations agree produces the ground truth used for training.

What would settle it

Hold out a deliberately mixed set of unseen destinations that contains known essential and known non-essential endpoints for the same devices, and report the confusion matrix; if an always-non-essential classifier matches ML-IoTrim's accuracy on that set, the 100% unseen-destination result is an artifact of test-set composition rather than evidence about the model.

Watch

Extended reading notes

Core claim

ML-IoTrim's central claim is that the essentiality of a network destination is learnable from the shape of the traffic itself: a destination is essential if blocking it stops a device's expected operations, and non-essential otherwise, and this property can be predicted from aggregate statistics of packet sizes, packet timing, protocol mix, and flow counts computed over fixed time windows. The paper demonstrates this with a global Random Forest classifier that scores above 99.6% F1 across aggregation windows and a neural network that stays above 98%. It further claims that newly contacted destinations absent from the training set are classified as non-essential with 100% accuracy, and that the full pipeline runs on an access point in time to block before the next collection cycle, scaling to hundreds of devices when processing is multi-threaded.

Load-bearing premise

The load-bearing premise is that the held-out unseen destinations include both essential and non-essential cases; the paper reports only non-essential counts for that set, so a classifier that always answers 'non-essential' could match the 100% accuracy without any learning.

Editorial extensions

If this is right

  • A home gateway can block non-essential IoT traffic automatically and locally, in near-real time, without sending DNS names, IP addresses, or payload data to a cloud service.
  • Destinations that first appear after the training period are recognized as non-essential with 100% accuracy in the paper's evaluation, removing the main weakness of allow-lists that must be updated by hand.
  • One global model trained on several device categories keeps F1 scores above 90% for each evaluated device and stays stable over roughly three months of unseen data after training on the first 30 days.
  • The same global model does not transfer to device types absent from training: for a device with no similar brand or type in the training set, classification is near random, so practical coverage requires models per device category or more diverse training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the all-vs-one test shows near-random accuracy only for devices with no similar counterpart in training, a deployed privacy service would likely ship a library of per-category models plus a device-identification front-end rather than a single universal model; the paper lists per-category models only as future work.
  • Editorial inference: using only header statistics means the same classifier could run on other edge hardware, such as an ISP router or set-top box, without adding deep-packet inspection; the paper demonstrates only a mini-PC access point.
  • Editorial inference: the claimed ability to flag unseen non-essential destinations suggests a behavioral complement to block-list and DNS defenses: a gateway that catches newly registered or previously unseen tracker domains because their traffic shape resembles known non-essential traffic, even before any list contains them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents ML-IoTrim, an edge-based system that classifies network destinations contacted by smart-home IoT devices as either essential or non-essential for device operation. The authors build a labeled dataset by observing eight consumer devices, blocking candidate destinations one at a time, and checking whether device functions still succeed; they then extract 204 statistical features from packet headers over time windows and train Random Forest and neural-network classifiers. The main claims are that a single global model can distinguish essential from non-essential destinations using only statistical traffic features, that it can recognize previously unseen non-essential destinations with 100% accuracy, and that the system can run in near-real-time on a home gateway while scaling to hundreds of devices. The paper also reports an all-vs-one experiment showing poor generalization to entirely new devices, which is acknowledged as a limitation.

Significance. If the central claims were supported, ML-IoTrim would be a useful contribution: it targets a real privacy problem, avoids cloud-based inspection, and replaces static allow-lists with a learned classifier that can operate locally. The authors deserve credit for releasing the code and data, for building a real testbed with eight devices across five categories, and for honestly reporting the all-vs-one generalization failure. However, the signature claim that new destinations are recognized as non-essential with 100% accuracy is not established by the presented evaluation: the held-out set, as described and plotted, appears to contain only non-essential destinations, so a constant classifier achieves the same perfect result. The global-model results are also reported only as aggregate F1 scores despite a strongly imbalanced class distribution. These issues are load-bearing for the paper's central contribution and require substantial additional analysis.

major comments (3)
  1. [Section 5, Figure 5] The claim that "new destinations are properly recognized as non-essential with 100% accuracy" is an artifact of the evaluation-set composition. Figure 5 is described in its caption as the distribution of "unique non-essential destinations contacted over time," and the text explicitly excludes Switchbot Hub Mini and Yeelight Bulb because they have only essential traffic, and excludes Tapo Plug and Kasa Bulb because all their destinations were contacted at boot. The four remaining devices are evaluated on a held-out period from which only non-essential destinations are reported; there is no indication that any essential destination appears for the first time in that period. Under this setup, a constant classifier that always predicts "non-essential" achieves 100% accuracy, so the experiment does not test whether the model can distinguish essential from non-essential destinations it has never seen. To support the stated generalization claim, the authors must construct a held-out destination set containing both classes, report per-class performance on that set, and provide evidence that the model does not simply exploit the class prior.
  2. [Table 2 and Figure 3] The global-model evaluation is reported only as aggregate F1 scores, which is insufficient for a dataset as imbalanced as the one described in Table 1: across the eight devices, the authors list 11 essential versus 255 non-essential destinations. A constant non-essential predictor would already achieve a high F1 score for the majority class, so near-100% aggregate F1 does not demonstrate that the classifier has learned to identify essential destinations. Please report per-class precision and recall, a confusion matrix, and the class distribution of the time-windowed samples, and consider precision-recall or ROC curves. Without this, the claim that "one single classifier trained on all the devices can effectively distinguish the destinations as essential/non-essential" is not fully supported.
  3. [Figure 4] The long-term consistency experiment is interpreted as showing "the model's capacity to handle new data over time," but the evaluation appears to be on the same set of destinations present in the training period, since Figure 5 shows that most destinations are contacted within the first days. The 5-day test windows therefore likely re-encounter known destinations, which is closer to a stability check than to a test of generalization to new destinations. Please clarify whether the evaluation excludes training-period destinations, and if it does not, soften the claim accordingly. The missing per-class metrics noted above apply to this experiment as well.
minor comments (4)
  1. [Section 5 (text)] In the paragraph describing Figure 5, "We excude the Switchbot hub mini" should read "We exclude the Switchbot Hub Mini."
  2. [Table 1] The column header "Cathegory" is misspelled as "Category." Also, the "Non-Required Traffic" percentages should be defined explicitly, since the denominator (packets, bytes, or time) is not stated.
  3. [Figure 5] The caption says "unique non-essential destinations" while the text refers to "unique destinations contacted by the devices." These are different quantities, and the discrepancy should be resolved in favor of the more precise caption.
  4. [Section 6, Table 3] The scalability experiment reports average processing times, but not the variance across runs, the number of runs, or how the 50-camera traffic was generated and whether it resembles the eight-device testbed. Please add methodological detail so that the real-time claim can be assessed.

Circularity Check

1 steps flagged · score 7.0 of 10

Unseen-destination claim is forced by a held-out set composed entirely of non-essential destinations.

  1. self definitional [Section 5, final paragraph ('new destinations' evaluation); Figure 5 caption]
    "The plot highlights that most of the destinations are contacted within the first days of traffic collection. After inspecting the distribution, we selected the first 15 days of traffic for the training set, while we used the remaining part for the evaluation. Destinations present in the training set have been excluded from the evaluation set. The results are optimal for all four devices, as new destinations are properly recognized as non-essential with 100% accuracy. ... Figure 5. Distribution of unique non-essential destinations contacted over time by the different IoT devices."

    Figure 5, the sole evidence for the unseen-destination test, plots only 'unique non-essential destinations,' and the text reports no essential destination appearing in the held-out period for the four tested devices. The evaluation set is therefore composed entirely of non-essential labels. On such a set, any constant classifier that always predicts 'non-essential' achieves 100% accuracy, so the reported 100% is a property of the test-set composition, not of the model's learned ability to distinguish unseen essential from unseen non-essential destinations. The claim that new destinations are 'properly recognized as non-essential' is thus self-fulfilling: the evaluation set is defined in terms of the very label being predicted.

full rationale

The central generalization claim of the paper — that new destinations not present in the training set are recognized as non-essential with 100% accuracy — reduces by construction to the composition of the held-out set. The paper's own Figure 5 caption identifies the plotted destinations as 'unique non-essential destinations,' and no essential destination is shown or described for the evaluation period. With only one class present in the test set, the reported perfect accuracy does not demonstrate any discriminative ability; it is exactly what a trivial always-non-essential predictor would achieve. This is a genuine circularity in the evaluation of the paper's headline allow-list-replacement capability. The rest of the paper is not circular: the ground-truth labels come from an external, empirical blocking methodology adapted from prior work, and the global time-split results, per-device F1 scores, and the honest negative all-vs-one result are independent measurements. The score is 7 rather than 8 because the circularity is confined to the unseen-destination evaluation, while other reported results retain independent content; nevertheless, the unsupported claim is central to the paper's stated advantage over allow-lists.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the behavioral labeling procedure from [16], a completeness assumption about tested device functions, the stability of traffic features over months, and the choice of aggregation and consensus parameters. The model itself is a supervised fit, so the main burden is on the ground-truth definition and the evaluation set composition, not on any invented physical entity.

free parameters (4)
  • Aggregation window w = 10, 60, 600, 3600 s; best at 60-600 s
    The time window for aggregating packets into samples is evaluated over four values and the best-performing windows are used in later experiments.
  • Label consensus threshold = 80%
    The blocking procedure repeats at least 30 times and requires 80% consensus to assign essential/non-essential labels; this threshold is chosen without independent justification.
  • ANN architecture widths = 128-64-32-1
    The neural network layer sizes are hand-designed and no architecture search or comparison is reported.
  • Rotation period r = 60-600 s
    System timing parameter evaluated over several values; used to estimate scalability.
assumptions (3)
  • domain assumption A destination is essential if and only if blocking it disrupts at least one tested function; the tested functions are a complete representation of device operations.
    Section 3.1: the labeling procedure marks a destination essential if any of the probe functions fails, and non-essential otherwise. If the subset of triggered functions misses some device operation, a destination needed for that operation is mislabeled.
  • domain assumption Statistical traffic features for a destination are sufficiently stationary over months so that time-split evaluation is valid.
    Section 5: the model is trained on the first 30 days (or 15 days) and evaluated on later data; this assumes the feature distribution for each destination does not drift over the collection period.
  • domain assumption The per-device feature normalization computed from training data remains applicable at inference time.
    Section 4.2: features are normalized per device to improve generalization; the paper does not specify how normalization statistics are maintained when new traffic arrives.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Intelligent Detection of Non-Essential IoT Traffic on the Home Gateway." pith.science (2026). https://pith.science/paper/WJ7JZH63

@misc{pith2026250418571,
  author       = {Pith},
  title        = {Pith review of: Intelligent Detection of Non-Essential IoT Traffic on the Home Gateway},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJ7JZH63}},
  note         = {Machine review of arXiv:2504.18571}
}
read the original abstract

The rapid expansion of Internet of Things (IoT) devices, particularly in smart home environments, has introduced considerable security and privacy concerns due to their persistent connectivity and interaction with cloud services. Despite advancements in IoT security, effective privacy measures remain uncovered, with existing solutions often relying on cloud-based threat detection that exposes sensitive data or outdated allow-lists that inadequately restrict non-essential network traffic. This work presents ML-IoTrim, a system for detecting and mitigating non-essential IoT traffic (i.e., not influencing the device operations) by analyzing network behavior at the edge, leveraging Machine Learning to classify network destinations. Our approach includes building a labeled dataset based on IoT device behavior and employing a feature-extraction pipeline to enable a binary classification of essential vs. non-essential network destinations. We test our framework in a consumer smart home setup with IoT devices from five categories, demonstrating that the model can accurately identify and block non-essential traffic, including previously unseen destinations, without relying on traditional allow-lists. We implement our solution on a home access point, showing the framework has strong potential for scalable deployment, supporting near-real-time traffic classification in large-scale IoT environments with hundreds of devices. This research advances privacy-aware traffic control in smart homes, paving the way for future developments in IoT device privacy.

Figures

Figures reproduced from arXiv: 2504.18571 by the authors.

Figure 1
Figure 1. Sketch of the ML-IoTrim framework architecture [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data processing pipeline from raw traffic to Machine Learning [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. F1 Score of the ANN classification evaluated on single devices [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Distribution of unique non-essential destinations contacted over [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: All vs. One RFC classification F1-Score 6. System Implementation We implement the proposed framework in an Intel NUC mini PC acting as an access point for an IoT network. We install the MonIoTr framework to collect and control the traffic, the feature extraction and mo…
Figure 7
Figure 7. Figure 7: Number of supported devices when using different number of [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages

  1. [16]

    Blocking without Breaking: Identification and Mitigation of Non-Essential IoT Traffic

    A.M. Mandalari et al., “Blocking without Breaking: Identifica- tion and Mitigation of Non-Essential IoT Traffic,” arXiv preprint arXiv:2105.05162, 2021

    work page Pith review arXiv 2021

  2. [1]

    A Systematic Content Review of Artificial Intelligence and the Internet of Things Applications in Smart Home,

    S. Sepasgozar et al., “A Systematic Content Review of Artificial Intelligence and the Internet of Things Applications in Smart Home,” Applied Sciences , vol. 10, no. 9, p. 3074, 2020

    work page 2020

  3. [2]

    A Survey on IoT Security: Application Areas, Security Threats, and Solution Architectures,

    V . Hassija et al., “A Survey on IoT Security: Application Areas, Security Threats, and Solution Architectures,” IEEe Access, vol. 7, pp. 82 721–82 743, 2019

    work page 2019

  4. [3]

    Protected or Porous: A Comparative Analysis of Threat Detection Capability of IoT Safeguards,

    A. M. Mandalari et al., “Protected or Porous: A Comparative Analysis of Threat Detection Capability of IoT Safeguards,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2023, pp. 3061–3078

    work page 2023

  5. [4]

    SunBlock: Cloudless Protection for IoT Sys- tems,

    V . Safronov et al. , “SunBlock: Cloudless Protection for IoT Sys- tems,” in International Conference on Passive and Active Network Measurement. Springer, 2024, pp. 322–338

    work page 2024

  6. [5]

    Can Allowlists Capture the Variability of Home IoT Device Network Behavior?

    W. He et al., “Can Allowlists Capture the Variability of Home IoT Device Network Behavior?” in 2024 IEEE 9th European Symposium on Security and Privacy (EuroS&P) . IEEE, 2024, pp. 114–138

    work page 2024

  7. [6]

    Heimdall: Mitigating the Internet of Insecure Things,

    J. Habibi et al., “Heimdall: Mitigating the Internet of Insecure Things,” IEEE Internet of Things Journal , vol. 4, no. 4, pp. 968– 978, 2017

    work page 2017

  8. [7]

    Protecting Home Networks From Insecure IoT Devices,

    E. Lastdrager et al., “Protecting Home Networks From Insecure IoT Devices,” in NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium . IEEE, 2020, pp. 1–6

    work page 2020

Show all 18 references
  1. [8]

    PRASH: A Framework for Privacy Risk Analysis of Smart Homes,

    J. Bugeja et al., “PRASH: A Framework for Privacy Risk Analysis of Smart Homes,” Sensors, vol. 21, no. 19, p. 6399, 2021

    2021

  2. [9]

    A capability-oriented approach to assessing privacy risk in smart home ecosystems,

    J. Sturgess et al., “A capability-oriented approach to assessing privacy risk in smart home ecosystems,” in Living in the Internet of Things: Cybersecurity of the IoT-2018 . IET, 2018, pp. 1–8

    2018

  3. [10]

    [Online]

    Bitdefender for IoT. [Online]. Available: https://bitdefender.com/ iot/

  4. [11]

    [Online]

    Agent-less IoT Security: Any Device, Any Threat, Any Scale. [Online]. Available: https://shieldiot.io/

  5. [12]

    [Online]

    Fing. [Online]. Available: https://fing.com

  6. [13]

    [Online]

    Kaspersky Embedded Security. [Online]. Avail- able: https://www.kaspersky.com/enterprise-security/ embedded-security-internet-of-things

  7. [14]

    [Online]

    Firewalla. [Online]. Available: https://firewalla.com/

  8. [15]

    [Online]

    RATtrap: A super-smart firewall for your home WiFi network. [Online]. Available: https://www.myrattrap.com/

  9. [17]

    Rfc 8520: Manufacturer usage description specification,

    E. Lear, R. Droms, and D. Romascanu, “Rfc 8520: Manufacturer usage description specification,” 2019

    2019

  10. [18]

    Information Exposure From Consumer IoT Devices: A Multidimensional, Network-Informed Measurement Approach,

    J. Ren et al., “Information Exposure From Consumer IoT Devices: A Multidimensional, Network-Informed Measurement Approach,” in Proceedings of the Internet Measurement Conference , 2019

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.