{"id":"4ecc71ea-1475-47ce-bb6b-4d295706addc","arxiv_id":"2607.17035","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An FL autoencoder plus LoRA-tuned LLM on MQTT/TLS reports perfect separation on self-generated IoT attacks, without external validation or baselines.","lead":"This paper assembles federated learning, a fine-tuned large language model, and zero-trust network controls into one IoT security pipeline, reporting F1 0.909 and ROC-AUC 1.0 for anomaly detection. The design is coherent, but the near-perfect results come from synthetic traffic generated from the same normal data the model was trained on.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: synthetic 'attacks' are perturbations of the same normal data used to train the autoencoder and set τ, so AUC 1.0 and perfect recall are artifacts of the generation procedure, not measurements of real-world detection.","rationale":"The central claim is a measured performance result, so for it to be meaningful the evaluation must measure the framework against realistic attack/normal distributions. The attack dataset is generated by perturbing the same pooled normal data used for training and thresholding (Eqs. 2-5), which makes high reconstruction-error separation likely by construction. Bootstrapping virtual devices from the same D_G (Eq. 3) further risks train/test leakage, though even without leakage the synthetic generation does not establish real-world efficacy. The internal inconsistencies in reporting (e.g., F1 values in Table I vs. IV) are symptomatic but not the primary reason the central claim fails. Because the existing evidence is an artifact of the evaluation design, the rejection verdict stands; no adjustment is needed.","tokens_in":8226,"tokens_out":5486,"duration_ms":53086,"concrete_test":"Run the trained federated autoencoder (or retrain with the same procedure) on a public real IoT attack corpus such as IoT-23 or CICIDS2017 filtered to MQTT/TLS flows, using the same 22 features and τ; report AUC, recall, and F1 per attack class. If AUC drops below 1.0 or recall is below 1.0, the headline numbers are artifacts of the synthetic generation procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline performance (F1 0.9091, ROC-AUC 1.0, perfect recall) rests on an evaluation set whose 'attacks' are synthetic modifications of the same normal-traffic distribution used to train the autoencoder and set the threshold. In §IV.A.2-3, attack samples are created by modifying a synthetic baseline, and virtual devices are generated by bootstrapping the pooled dataset D_G (Eq. 3) with multiplicative shifts and noise (Eqs. 2-5). The normal evaluation samples appear to be the same D_G used to compute τ (95th percentile) and possibly the training set. No independent train/test split, no real attack traces, and no held-out data are described. Under this construction, an autoencoder trained on D_G will assign high reconstruction error to samples that are deliberately moved away from D_G; perfect separation (AUC=1.0) is an artifact of the generation procedure, not evidence that the framework detects real port scans, DDoS, or TLS downgrades. The LLM classification numbers (Table IV) inherit the same synthetic pipeline. The absence of any baseline or external dataset makes the central claim unfalsifiable in its current form.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-trust IoT architecture that combines federated learning (FL) with an autoencoder-based anomaly detector, a LoRA fine-tuned LLM (Llama 3.2-3B) for threat classification and response generation, and automated iptables enforcement on an MQTT/TLS broker. The system is evaluated on a testbed of five Raspberry Pis extended with statistically bootstrapped virtual devices. The paper reports an F1 score of 0.9091, ROC-AUC of 1.0, perfect recall across all attack classes, and perfect LLM classification accuracy, and claims that this demonstrates effective privacy-preserving anomaly detection for resource-constrained IoT devices.","tokens_in":8563,"tokens_out":5920,"duration_ms":53811,"significance":"If the reported results were valid, the architecture would be a useful integration of FL, LLM reasoning, and zero-trust enforcement for IoT. However, the empirical evaluation is constructed in a way that makes the headline performance essentially an artifact of the data-generation procedure. Attacks are synthetic modifications of the same normal-traffic distribution used to train the autoencoder and set the anomaly threshold; virtual devices are bootstrapped from the same pooled dataset; and no independent train/test split, real attack traces, or baselines are provided. The contribution is therefore currently a plausible system design rather than a validated detection framework. The paper does not provide code or data, so the results are not independently reproducible.","major_comments":[{"comment":"The central detection claim (ROC-AUC 1.0, recall 1.0, F1 0.9091) is an artifact of the evaluation design. Attack traffic is created by taking a synthetic baseline of normal MQTT/TLS traffic and modifying selected features to emulate malicious behavior. The autoencoder is trained on normal traffic, and the threshold τ is set as the 95th percentile of reconstruction MSE over normal samples. Under this construction, any deterministic feature perturbation moves attack samples away from the training manifold, guaranteeing high reconstruction error and near-perfect separation. No independent attack dataset or held-out normal set is used. The reported per-class F1 of 0.909 is simply a consequence of the fixed 5% false-positive rate and the balanced one-vs-rest evaluation, not a measured property of the method. The evaluation must be redone with real attack traces or a standard public IoT attack","section":"§IV.A.2, §IV.B"},{"comment":"Virtual devices are generated by bootstrapping the pooled dataset D_G and applying per-device multiplicative shifts and Gaussian noise. Because D_G is the same data used to train the autoencoder and to compute the threshold, the bootstrapped evaluation samples are near-duplicates of training samples. No device-wise, temporal, or other independent split is described. This creates a serious risk of data leakage: the reported FL convergence, resource measurements, and anomaly-detection scores may reflect memorization of training samples rather than generalization to genuinely new devices. The authors must describe an independent split and quantify the overlap between bootstrapped samples and training samples.","section":"§IV.A.3, Eqs. (1)–(6)"},{"comment":"The LLM is fine-tuned on 500 instruction-response pairs generated for the same four attack classes used in the evaluation (TLS downgrade, DDoS SYN flood, port scan, plain MQTT) and then evaluated on anomalies produced by the same synthetic generator. Perfect precision and recall on a self-generated, in-distribution test set do not establish that the LLM can classify real attacks or handle out-of-distribution anomalies. At minimum, an evaluation on held-out attack descriptions, a different attack taxonomy, or a public dataset is required. The current Table IV is not informative about generalization.","section":"§IV.D, Table IV"},{"comment":"No baseline or ablation is reported. The paper does not compare the proposed FL autoencoder against a centralized autoencoder, an isolation forest, a supervised classifier, or a non-federated variant. Similarly, the LLM component is not compared with a generic prompted LLM or with a simpler rule-based classifier. Without such comparisons, the specific contributions of FL and the LLM are not quantified, and the resource-overhead tables (Tables II and III) cannot be interpreted. The paper should include baselines and ablations to support the claim that the proposed framework is effective.","section":"§IV.B"}],"minor_comments":[{"comment":"In the text, the F1-score is reported as '0' for each attack class ('F1-score of 0'); this is clearly a typo and should be 0.909.","section":"§IV.B, Table I"},{"comment":"The abstract reports F1 = 0.9091, which is the per-class macro average, while Table I also reports an Overall F1 of 0.976. Please clarify which F1 is meant in the abstract and conclusions.","section":"Abstract, Table I"},{"comment":"Feature engineering states that the feature space is reduced to 22 discriminative features, but the autoencoder is described with an input dimension of d=36. This inconsistency should be resolved; if one-hot expansions or additional features are used, they should be described explicitly.","section":"§III.B and §III.C"},{"comment":"The introduction mentions 25 heterogeneous clients, while the physical testbed consists of five Raspberry Pis. Please clarify how the virtual devices map to the 25 clients and whether each virtual device corresponds to a separate FL client.","section":"§IV.A"},{"comment":"Resource measurements are reported as single averages without variance or confidence intervals. Since virtual-device generation is stochastic, reporting the spread across multiple random seeds would substantially improve interpretability.","section":"§IV.B, Tables II–III"},{"comment":"The end-to-end LLM latency of about 41 seconds relies on a Google Colab backend and an ngrok tunnel. This is a deployment limitation, not a property of the proposed architecture; it should be stated as such, and the feasibility of real-time enforcement should be discussed.","section":"§IV.D"}],"recommendation":"reject","confidential_remarks":"I agree with the reader's assessment. The evaluation is circular: the attack samples are perturbations of the same normal data used to train the detector and set the threshold, and the virtual devices are bootstrapped from the same pooled dataset. The headline ROC-AUC of 1.0 and perfect recall are therefore forced by construction. Fixing this would require a fundamentally new empirical study with real attack data, independent splits, and baselines, which is beyond a revision of the current manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clear and honest assembly of known pieces: a federated autoencoder for anomaly detection, a LoRA-fine-tuned Llama 3.2 for threat classification, and mTLS/MQTT with automatic iptables enforcement. The testbed with five Raspberry Pis and 25 federated clients is real, and the resource measurements (inference latency, CPU/RAM, communication overhead) are useful. The related work is adequate, and the writing is readable. I believe the authors actually built this system.\n\nThe problem is the evaluation. The stress-test note is correct and lands squarely. Attack samples in Section IV.A.2 are generated by modifying features of a synthetic baseline built from the same normal traffic distribution; virtual devices are bootstrapped from the pooled dataset D_G (Eq. 3) with multiplicative shifts and noise. The threshold τ is the 95th percentile of the reconstruction MSE computed from all normal samples, and no train/test split is described. Under that construction, an autoencoder fitted on D_G will naturally assign higher reconstruction error to samples deliberately moved away from D_G. Perfect separation (AUC 1.0) and perfect recall are forced by the generation procedure, not measured on real attacks. The LLM evaluation inherits the same pipeline: the fine-tuning data and the test taxonomy are the same four attack classes, and there is no held-out set or real-world attack trace. Without baselines or external datasets, the central claims are unfalsifiable in the current form.\n\nThe numbers also have internal inconsistencies: the text says an F1-score of 0 in one place, the table says 0.909, the abstract says 0.9091, and Table IV gives an overall F1 of 0.9231. The 41-second LLM latency is worth flagging for real-time enforcement, though the enforcement step itself is fast.\n\nThese flaws are load-bearing. As a prototype description, the paper has value; as a research claim about a privacy-preserving zero-trust IoT defense, it fails. The authors should redo the evaluation with real attack traffic or a properly held-out public dataset, add baselines, and fix the inconsistency. I would not cite the numbers, and I would not send this to peer review in its current form, since the circularity is identifiable at the desk and the revision needed is substantial.","headline":"An honest engineering assembly whose headline numbers are artifacts of a circular evaluation; the architecture is worth reading as a prototype, not as a validated defense.","tokens_in":9001,"tokens_out":2112,"would_cite":false,"duration_ms":21672,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that combining a federated autoencoder with a fine-tuned language model yields a privacy-preserving zero-trust architecture for IoT that detects every tested attack with perfect recall and automatically blocks the attacker.","keywords":["federated learning","zero trust architecture","IoT security","anomaly detection","autoencoder","LLM threat intelligence","MQTT/TLS","automated response"],"falsifier":"Run the trained autoencoder and LLM pipeline on a live packet capture of a real port-scan or SYN-flood campaign against an MQTT broker and check whether recall stays at 1.0; or measure the fraction of virtual test samples that are near-duplicates of training samples (e.g., by Euclidean distance in feature space) and see whether AUC drops when those near-duplicates are removed.","tokens_in":8099,"feed_emoji":"🛡️","tokens_out":4809,"duration_ms":40976,"temperature":0.7,"pith_summary":"This paper argues that zero trust can be made practical for IoT by layering a federated autoencoder, a fine-tuned large language model, and automated firewall enforcement into a single pipeline. The autoencoder learns only normal traffic patterns on each device and flags anomalies without sharing raw data; the LLM converts each anomaly into a structured threat report with severity and remediation commands; and the broker automatically blocks malicious source IPs. The authors report perfect recall and a ROC-AUC of 1.0 across four attack types (port scan, DDoS SYN flood, TLS downgrade, plain MQTT) on a testbed of single-board computers and sensors extended with bootstrapped virtual devices. They interpret this as evidence that privacy-preserving anomaly detection and autonomous response can coexist on resource-constrained IoT hardware.","feed_headline":"Zero-trust IoT pipeline catches every tested attack","feed_subtitle":"Privacy-preserving federated detector scores perfect recall and AUC 1.0 on synthetic attacks, with automated IP blocking.","key_machinery":"The load-bearing mechanism is the combination of (1) a federated autoencoder that learns only normal traffic patterns and flags anything whose reconstruction error exceeds the p95 threshold; (2) FedAvg aggregation so raw sensor and network data never leave the devices; and (3) an LLM fine-tuned on 500 instruction–response pairs that converts anomaly feature vectors into structured JSON threat reports, which the system turns into iptables rules to block malicious IPs. The p95 threshold is the fulcrum: it sets a fixed 5% false-positive rate and, together with perfect reconstruction-error separation, yields the reported precision and recall.","core_discovery":"On the paper's own terms, the central discovery is that a federated-learning autoencoder, aggregated with FedAvg over 25 clients and thresholded at the 95th percentile of reconstruction error, separates all tested attack traffic from normal MQTT/TLS traffic with zero false negatives, and that a LoRA-fine-tuned LLM maps the resulting anomaly features to accurate, actionable decisions (verdict, severity, remediation commands) that the broker can execute via firewall rules in under a second after the LLM responds. The authors see this as evidence that a zero-trust IoT architecture—mutual TLS on every MQTT connection, no implicit trust, continuous identity verification, and automated threat isol","pith_inferences":["The reported ROC-AUC of 1.0 and perfect recall are only meaningful if the synthetic attack samples (built by modifying features of normal traffic) and the bootstrapped virtual devices are representative; if they share near-duplicates with the training distribution, the numbers could be optimistic.","A natural testable extension: replace the synthetic attacks with packet captures from real IoT botnets or published intrusion datasets and check whether recall and AUC survive.","The roughly 41-second LLM response delay, while stable, is larger than typical IoT alert latency budgets; a future design could cache responses, run a smaller local model, or limit LLM invocation to high-severity anomalies.","Because the autoencoder is trained only on normal traffic, the approach might detect novel (zero-day) attacks as anomalies, but the LLM's ability to classify them depends on how similar they are to the four trained classes."],"forward_implications":["If the claim holds, zero trust for IoT does not require heavyweight per-device security: a broker-side monitor plus local autoencoders can enforce 'never trust, always verify' at the communication layer.","Privacy-preserving anomaly detection is feasible on low-power devices: clients share only weight updates, and per-sample inference latency is about 0.0065 ms.","Automated detection-to-response is achievable without human intervention: the LLM decides, and firewall enforcement completes in under a second; the main latency (about 41 seconds) comes from remote LLM inference.","The fixed 5% false-positive rate is a tunable design point: at high imbalance (up to 100:1 attack scarcity), recall stays perfect, so no attacks are missed, though precision drops.","A direct corollary is that the same pipeline generalizes across four distinct attack classes with a single shared feature set, suggesting the anomaly detector is attack-agnostic."],"fun_headline_variants":["Federated learning + LLM catches every IoT attack in test","Zero-trust IoT: FL autoencoder nails every attack, AUC 1.0","FL+LLM threat response blocks all attacks in zero-trust IoT","Perfect detection: federated autoencoder + LLM for zero-trust IoT"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation hinges on the assumption that the synthetic attack traffic and bootstrapped virtual devices behave like real malicious IoT traffic; if that assumption fails, the perfect recall and AUC of 1.0 are artifacts of how the test data was manufactured.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning + LLM catches every IoT attack in test","Zero-trust IoT: FL autoencoder nails every attack, AUC 1.0","FL+LLM threat response blocks all attacks in zero-trust IoT","Perfect detection: federated autoencoder + LLM for zero-trust IoT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001261,"raw_usage":{"total_tokens":4975,"prompt_tokens":690,"completion_tokens":4285,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":4203}},"tokens_in":434,"tokens_out":4285,"duration_ms":25305,"temperature":1.0,"reasoning_tokens":4203,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:12:11.371138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained autoencoder and LLM pipeline on a live packet capture of a real port-scan or SYN-flood campaign against an MQTT broker and check whether recall stays at 1.0; or measure the fraction of virtual test samples that are near-duplicates of training samples (e.g., by Euclidean distance in feature space) and see whether AUC drops when those near-duplicates are removed.","supporting_citations":[],"review_version":1}