{"id":"3a4a4e7b-f55f-4524-b0a1-05a37d543f8e","arxiv_id":"2509.08485","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A flow-based ML system reports over 99% accuracy for known IoT camera identification and claims strong zero-day camera detection, but feature-selection leakage and unreleased data weaken the zero-day evidence.","lead":"This paper presents zCamInspector, a machine-learning system that identifies known IoT cameras from network flows and uses one-class classifiers to flag never-seen cameras. It reports very high accuracy, but the zero-day evaluation appears optimistic because the selected flow features were chosen using the very camera flows being tested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-day claim is not supported as evaluated: feature selection uses the test class, and abstract accuracies conflict with Table 5; maintain REJECT.","rationale":"The reader's weakest assumption identified exactly this leakage, and the paper's own text confirms it: Section 5.1's ExtraTree feature selection is explicitly supervised on Others vs IoTCam, and Section 5.5 says all one-class models are trained on Others using those top-10 features. Because the test class labels are used before training the one-class detectors, the reported DeepSVDD accuracy of 98.17% on Set I & II cannot be taken as evidence that an administrator with no camera labels could detect zero-day cameras. The supervised identification results in Section 6 are a separate, more plausible contribution, so this is not a wholesale rejection of everything in the paper; it is a rejection of the central zero-day claim in its current form. The abstract/table mismatch strengthens the case that the zero-day numbers need re-verification. Maintaining the reader's REJECT verdict is appropriate; a revised version with leakage-free feature selection, clear reporting, and released data could be reconsidered.","tokens_in":25900,"tokens_out":7341,"duration_ms":64601,"concrete_test":"Re-run the zero-day experiments in Section 5.5 with feature selection that never sees IoTCam labels: for example, select features by an unsupervised criterion (variance or reconstruction error) on the Set III training split only, or evaluate all 62 features directly, keeping the same train/test splits and one-class models. Compare DeepSVDD's testing accuracy on Set I & II to Table 5's 98.17%. If it drops materially (e.g., below 75%) or if the abstract's numbers cannot be reproduced from the released data, the reported zero-day performance is an artifact of label leakage rather than genuine detection of unseen cameras.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central new claim, zero-day IoT camera detection, rests on an evaluation that is not blind. Section 5.1 selects the top-10 features using an ExtraTree classifier trained on two classes, 'Others' (Set III) and 'IoTCam' (Set I + Set II). Section 5.5 then trains every one-class detector on only Set III but uses exactly those top-10 features. Thus the labels of the 'unseen' test cameras participate in feature selection. In the intended deployment no IoTCam labels would exist before the zero-day camera appears, and Figure 3(b) shows the two-class ExtraTree nearly perfectly separates these classes, so the selected features are known to be discriminative for the test class. The result is an optimistic estimate of one-class detection. Separately, the abstract's zero-day accuracies (93.20%, 96.55%, 78.65%, 92.16%) do not match Table 5's Set I & II testing means (27.2%, 19.7%, 14.35%, 98.17%), and the 'all cameras zero-day' DeepSVDD figure of 74.51% is actually the mean of the one-known-camera rows in Table 6. These issues jointly mean the paper's headline contribution is not established as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes zCamInspector, a system for identifying known IoT cameras with supervised classifiers (zCamClassifier) and detecting zero-day IoT cameras with one-class classifiers (zCamDetector), using flow-based features extracted with CICFlowmeter and avoiding IP addresses and transport ports. The authors use about 40 GB of traffic: Set I from six commercial cameras in their own laboratory, Set II from five open-source cameras (UNSW dataset), and Set III from non-camera video-conferencing and video-sharing applications. They evaluate seven supervised models and four one-class models, reporting that XGB achieves over 99% accuracy for known camera identification and that one-class detectors achieve zero-day accuracies of 93.20% (OCSVM), 96.55% (SGDOCSVM), 78.65% (Isolation Forest), and 92.16% (DeepSVDD) when trained on Set III and tested on Sets I and II. The paper also reports a scenario where all eleven cameras are treated as zero-day, with DeepSVDD giving a mean testing accuracy of 74.51%.","tokens_in":26178,"tokens_out":6315,"duration_ms":54261,"significance":"If the zero-day detection results were valid, the paper would make a useful contribution to IoT security, and the release of a new IoT camera traffic dataset (BITSPHC) would benefit the community. The paper is also commendable for making part of the code and sample traces available. However, the central zero-day claim is not established by the current evaluation: the feature selection step uses the labels of the 'unseen' camera class, the headline zero-day accuracies in the abstract conflict with the numbers in Table 5, a key all-zero-day result is actually the mean of a different experimental scenario, and hyperparameters appear to have been tuned against the test class. These issues jointly invalidate the paper's main contribution as reported.","major_comments":[{"comment":"The zero-day evaluation is not blind because the top-10 feature subset is selected using labels from both 'Others' (Set III) and 'IoTCam' (Set I and Set II). Section 5.1 explicitly states that the ExtraTree classifier is trained on two classes, 'Others' and 'IoTCam', and Section 5.5 trains every one-class detector on Set III using exactly these top-10 features. Since the feature selection step has already seen the labels of the cameras that are supposed to be unseen, the detector is evaluated on features known to discriminate the test class from the training class. Figure 3(b) confirms that the two-class ExtraTree separates these classes almost perfectly, so the selected features are strongly informative for the target class by construction. The reported zero-day accuracies in Table 5 and the abstract are therefore optimistically biased and do not estimate performance on genuinely unseen camera traffic. To support the zero-day claim, feature selection must be performed using only the training-class data (e.g., Set III only) or with an unsupervised method applied before any camera labels are used.","section":"§5.1, §5.5"},{"comment":"The abstract reports zero-day accuracies of 93.20%, 96.55%, 78.65%, and 92.16% for OneClassSVM, SGDOneClassSVM, IsolationForest, and DeepSVDD when trained on Set III and tested on Set I and Set II. Table 5, however, reports the mean testing accuracies on Set I & Set II as 27.2%, 19.7%, 14.35%, and 98.17% for the same models and the same training condition. These numbers are mutually inconsistent; for example, the abstract's 92.16% for DeepSVDD does not match Table 5's 98.17%, and the abstract's 93.20% for OneClassSVM is far from Table 5's 27.2%. The authors must clarify which set of numbers is correct, define the exact metric used (e.g., outlier detection rate versus classification accuracy), and reconcile the abstract with the experimental tables.","section":"Abstract, §5.5"},{"comment":"The abstract and conclusion state that when all IoT cameras are zero-day, DeepSVDD achieves a mean testing accuracy of 74.51%. This value is, in fact, the mean of the 11 testing accuracies in Table 6, which is the 'All But one Zero Day' scenario where each model is trained on the flows of exactly one known camera and tested on the remaining ten cameras. The true all-zero-day scenario of Section 5.5 trains on Set III and tests on Set I and Set II, for which Table 5 reports a DeepSVDD mean testing accuracy of 98.17%. The reported 74.51% is therefore attributed to the wrong experimental setting, and the claim that DeepSVDD is best when all devices are zero-day is not supported by the cited number.","section":"§5.6, Abstract"},{"comment":"The hyperparameters of the one-class models appear to have been tuned against the test class. Section 5.5 states that the number of outliers is shown 'by tuning different hyperparameter in Fig. 10 indicating maximum outliers in IoTCam class.' If the final hyperparameter choices (Table 3) were selected by maximizing outlier detection on the IoTCam test set, then the test set has influenced model selection, further invalidating the zero-day evaluation. The authors should specify how the hyperparameters were chosen, and they must use a validation set drawn from the training distribution or fix hyperparameters a priori before any contact with the zero-day class.","section":"§5.5, Fig. 10"},{"comment":"The supervised classification results, which support the secondary claim that XGB identifies known cameras with over 99% accuracy, are reported as single 'Maximum Accuracy' values without cross-validation, confidence intervals, or multiple runs. This makes it impossible to assess whether the differences among models (e.g., XGB versus Random Forest) are statistically meaningful or whether the reported numbers are sensitive to the train/test split. The authors should provide cross-validated results with standard deviations, especially because the comparison with prior work (Section 6.3) relies on these point estimates.","section":"§6.1, Table 8"}],"minor_comments":[{"comment":"The text says CICFlowmeter extracts 77 flow-based features, while the abstract and Section 1 state 62 features; these numbers should be reconciled.","section":"§3.4.1"},{"comment":"The section heading in the introduction is printed as 'SySetm Design and Implementation'; this appears to be a typo for 'System'.","section":"§1"},{"comment":"The ROC curve described in the text is referenced as 'Fig.??'; the figure reference is missing and should be fixed.","section":"§6.1"},{"comment":"Table 6 contains two rows labeled 'Netatmo: 10 IoT Cams.' with different testing accuracies, which is confusing because one Netatmo belongs to Set I and the other to Set II; the rows should be labeled distinctly (e.g., Netatmo (BITSPHC) and Netatmo (UNSW)).","section":"§5.6"},{"comment":"The evaluation protocol for Table 5 is not fully specified: the paper states that 90% of Set III is used for training and 10% for testing, but the details of how the standard scaler is fitted and whether the Set I/II test data undergo the same preprocessing are not described; these details should be stated explicitly.","section":"§5.5"},{"comment":"The comparison in Figure 12 reports TPR and FPR for zCamInspector, DeWiCam, and SCamF, but the text does not explain how the zCamInspector TPR/FPR values were obtained or under which test scenario they were measured; this should be clarified.","section":"§6.3"}],"recommendation":"reject","confidential_remarks":"The central zero-day detection claim is not supported as evaluated because the feature selection uses the labels of the supposedly unseen cameras, and the abstract numbers conflict with the tables. These are load-bearing issues rather than presentation problems. The authors would need to redo the evaluation with a genuinely blind protocol and correct the numerical inconsistencies before the paper could be reconsidered. The novel dataset and the supervised identification results may still be of interest in a revised submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the supervised identification half is legit: XGB recognizes known IoT cameras with >99% accuracy across their own BITSPHC data and UNSW, and the misclassification tables are informative. Second, the zero-day half, the actual new claim, is not supported by the evaluation. The feature selection contaminates the one-class experiment.\n\nHere's the specific issue. Section 5.1 picks the top-10 features with an ExtraTree classifier trained on two classes: 'Others' and 'IoTCam' (all cameras). Section 5.5 then trains OCSVM, SGDOCSVM, IsolationForest, and DeepSVDD on only 'Others' but uses those same top-10 features. So the labels of the unseen cameras participated in choosing the features. In deployment you wouldn't have camera labels before the zero-day camera shows up. That makes the reported zero-day accuracies optimistic. The paper even shows the two-class ExtraTree separates the classes almost perfectly, so the selected features are known to be discriminative for exactly the test class.\n\nThere's also a reporting problem. The abstract gives zero-day accuracies of 93.20%, 96.55%, 78.65%, 92.16% for OCSVM, SGDOCSVM, IF, DeepSVDD. Those match neither Table 5's means (27.2%, 19.7%, 14.35%, 98.17%) nor anything else I can reconcile. The 'all cameras zero-day' DeepSVDD number 74.51% looks like it comes from the 'one known camera' experiments in Table 6, not from a truly all-unseen evaluation. That kind of discrepancy is exactly what a referee should catch.\n\nWhat the paper does well: the dataset is a real contribution (14.96GB of six consumer cameras, plus using UNSW), the flow-based features are standard and well-motivated, they avoid IP/port dependence, and the supervised classification is straightforward and reproducible in style. The comparison with prior work is fair on the supervised side. The GitHub repo exists with sample traces.\n\nThe soft spots beyond the zero-day evaluation: hyperparameters for the one-class models appear tuned with the test set in view (Figure 10), and the full dataset is withheld until acceptance, so the numbers aren't independently checkable right now. Both are fixable.\n\nWho this is for: someone working on IoT device fingerprinting or network-level spy-camera detection. The supervised part is usable as a known-camera identifier. The zero-day part should not be cited as a demonstrated capability yet.\n\nMy recommendation: send it to peer review, but with a clear directive that the zero-day evaluation must be redone—feature selection using only training-class data, cross-validated, and the abstract reconciled with the tables. Fixable in a major revision, but not acceptable as is.","headline":"Known-camera identification is solid and useful, but the zero-day claim is undermined by target-aware feature selection and inconsistent numbers.","tokens_in":26688,"tokens_out":2284,"would_cite":false,"duration_ms":20023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that flow statistics alone, without IP addresses, ports, or decryption, can identify known IoT cameras with over 99% accuracy and flag unseen zero-day cameras as outliers.","keywords":["IoT camera detection","zero-day detection","flow-based features","one-class classification","DeepSVDD","XGBoost","network traffic classification","outlier detection"],"falsifier":"Re-run the zero-day pipeline with feature selection performed on the non-camera training split only, holding out all camera flows until testing; if the mean DeepSVDD testing accuracy falls well below the reported 74.51%, the optimistic estimate caused by target-aware feature selection is confirmed.","tokens_in":25670,"feed_emoji":"📷","tokens_out":9231,"duration_ms":75310,"temperature":0.7,"pith_summary":"This paper sets out to solve a practical problem for network administrators: detecting that a streaming IoT camera has joined a network, even when the camera model has never been seen before. It proposes zCamInspector, which turns captured traffic into flow-level statistics and runs two kinds of models: supervised classifiers that name a known camera, and one-class classifiers that treat any unseen camera as an outlier against ordinary non-camera traffic. On eleven cameras plus conferencing and video-sharing applications, the paper reports that XGB identifies known cameras with more than 99% accuracy and a false-negative rate as low as 0.3%. For zero-day cameras, it reports detection accuracy up to 96.55% and identifies DeepSVDD as the most consistent one-class detector when all test cameras are new. If these results hold, camera presence can be inferred from generic flow statistics, without decrypting traffic or relying on IP addresses or ports.","feed_headline":"Traffic flows alone expose hidden IoT cameras, known and new","feed_subtitle":"A flow-based system identifies known cameras at over 99% accuracy and flags unseen ones as outliers.","key_machinery":"The mechanism that carries the argument is the per-flow feature vector: 62 bidirectional flow statistics such as packet lengths, flag counts, inter-arrival times, window sizes, and throughput ratios, reduced to the ten most important features by an ExtraTree classifier. The same ten features feed both the supervised camera-identity model and the one-class zero-day detectors. The one-class models learn a boundary around ordinary non-camera traffic; DeepSVDD does this by mapping flows into a low-dimensional latent space and enclosing normal traffic in a minimal hypersphere, so a flow from a never-seen camera lands outside the boundary and is flagged as an outlier.","core_discovery":"The central claim is that streaming IoT cameras leave a distinguishable trace in generic flow statistics, and that an unseen camera can be detected as an outlier by a one-class model trained purely on non-camera traffic. zCamClassifier, using XGB, identifies each known camera with over 99% accuracy in the combined dataset; zCamDetector, using DeepSVDD, learns a hypersphere around non-camera flows in a latent space and reports strong zero-day detection, with a mean testing accuracy of 74.51% when every test camera is unseen. The system deliberately avoids IP addresses and transport ports, so it is agnostic to NAT and to cameras that share cloud infrastructure with ordinary applications.","pith_inferences":["The reported zero-day accuracy is probably optimistic because the top-ten features were selected by a classifier that saw both the non-camera and camera classes; retraining the feature selector on the non-camera training data alone would give a fairer estimate.","The most confusing traffic is likely other continuous video streams, such as conferencing and video-sharing applications; a stress test would pit cameras against long, high-bitrate video calls to see how much of the outlier signal remains.","The same flow-feature design could plausibly detect other streaming IoT devices, but the paper only claims cameras; testing on doorbells, baby monitors, and smart displays would be a natural extension."],"forward_implications":["A network administrator could be alerted the first time an unseen camera streams, without a vendor whitelist or prior knowledge of the device.","Because detection uses flow statistics only, it works on encrypted traffic and behind NAT, as long as the router or switch can mirror packets.","The same ten features serve both identification and zero-day detection, so deployment could sit at a gateway with modest computing resources.","If the method transfers to other streaming IoT devices, such as doorbells and baby monitors, the same outlier framework could flag them without retraining on each model."],"supporting_citations":[{"why":"Supplies the open-source IoT camera traffic used as the second camera set and a prior IoT-device classification baseline.","marker":"[14]"},{"why":"Baseline camera-flow classifier whose accuracy and design zCamInspector extends and compares against.","marker":"[18]"},{"why":"Provides the flow-feature extraction method that produces the 62 per-flow features.","marker":"[20]"},{"why":"Earlier spy-camera detection method compared for TPR and FPR on camera identification.","marker":"[22]"},{"why":"Earlier hidden-wireless-camera detection method compared for TPR and FPR.","marker":"[23]"},{"why":"Defines the one-class SVM objective used for the OCSVM zero-day detector.","marker":"[48]"},{"why":"Provides the stochastic-gradient one-class SVM variant used for zero-day detection.","marker":"[49]"},{"why":"Defines Isolation Forest, one of the one-class detectors evaluated.","marker":"[50]"},{"why":"Defines DeepSVDD, the deep one-class method that gives the strongest zero-day results in the paper.","marker":"[51]"}],"fun_headline_variants":["Flows alone reveal both known and unseen IoT cameras","Zero-day cameras detected from traffic statistics only","One-class classifiers flag unseen IoT cameras via flows","Unseen cameras detected via flow-based outlier models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that features chosen after looking at both camera and non-camera traffic can still measure how well a system detects cameras it has never seen; if feature selection is the part that leaks information about the target class, the zero-day results are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Flows alone reveal both known and unseen IoT cameras","Zero-day cameras detected from traffic statistics only","One-class classifiers flag unseen IoT cameras via flows","Unseen cameras detected via flow-based outlier models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000972,"raw_usage":{"total_tokens":4157,"prompt_tokens":993,"completion_tokens":3164,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3114}},"tokens_in":609,"tokens_out":3164,"duration_ms":21913,"temperature":1.0,"reasoning_tokens":3114,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:08:06.132713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the zero-day pipeline with feature selection performed on the non-camera training split only, holding out all camera flows until testing; if the mean DeepSVDD testing accuracy falls well below the reported 74.51%, the optimistic estimate caused by target-aware feature selection is confirmed.","supporting_citations":[{"cited_title":"Classifying iot devices in smart environments using network traffic characteristics,","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source IoT camera traffic used as the second camera set and a prior IoT-device classification baseline."},{"cited_title":"icamin- spector: Classify video traffic and detect iot (spy) camera flows,","cited_arxiv_id":null,"evidence_quote":"Baseline camera-flow classifier whose accuracy and design zCamInspector extends and compares against."},{"cited_title":"Cicflowmeter-v4.0 (formerly known as is- cxflowmeter) is a network traffic bi-flow generator and analyser for anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Provides the flow-feature extraction method that produces the 62 per-flow features."},{"cited_title":"Are there wireless hidden cameras spying on me?","cited_arxiv_id":null,"evidence_quote":"Earlier spy-camera detection method compared for TPR and FPR on camera identification."},{"cited_title":"Estimating the support of a high-dimensional distri- bution,","cited_arxiv_id":null,"evidence_quote":"Defines the one-class SVM objective used for the OCSVM zero-day detector."},{"cited_title":"Toward su- pervised anomaly detection,","cited_arxiv_id":null,"evidence_quote":"Provides the stochastic-gradient one-class SVM variant used for zero-day detection."},{"cited_title":"Isolation forest,","cited_arxiv_id":null,"evidence_quote":"Defines Isolation Forest, one of the one-class detectors evaluated."},{"cited_title":"Deep one- class classification,","cited_arxiv_id":null,"evidence_quote":"Defines DeepSVDD, the deep one-class method that gives the strongest zero-day results in the paper."}],"review_version":2}