{"id":"220f725c-3370-489e-adc4-0f8d09685caf","arxiv_id":"2505.06636","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A federated semi-supervised intrusion detection framework that trains robot-client encoders with contrastive learning and blends them on a server via EMA claims top accuracy on NSL-KDD.","lead":"This paper combines federated learning, contrastive learning, and semi-supervised training to detect network intrusions in robot fleets without requiring labeled data on the robots. It reports better accuracy and a smaller model than several baselines on the NSL-KDD benchmark, but the evaluation relies on test-set hyperparameter tuning and lacks code or error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy is selected on KDDTest+: hyperparameters tuned on the test set (Tables II–III) invalidate the claimed outperformance in Tables V–VI.","rationale":"The reader's weakest_assumption focuses on whether tabular augmentations preserve attack-class semantics. That is a real concern, but the test-set tuning is more directly load-bearing because it invalidates the headline empirical claim regardless of whether the augmentations are semantically sound. The paper's own Section IV-B and IV-C show that hyperparameters were chosen by maximizing KDDTest+ metrics, and the chosen configuration is exactly what appears in the final comparison tables. This is textbook selection on the test set, and it biases every reported advantage. The gaps over the strongest baseline are small, and no error bars or code are provided, so the observed margins could easily be artifacts of tuning. The concrete test proposed above would settle whether the advantage survives a clean evaluation protocol. Since the central claim is an empirical superiority claim, and the evidence for it is tainted by test-set selection, I agree with the reader's REJECT verdict, while noting that the most load-bearing flaw is the evaluation protocol rather than the augmentation semantics.","tokens_in":11052,"tokens_out":4880,"duration_ms":44635,"concrete_test":"Hold out 10,000 of the server's 50,000 labeled NSL-KDD samples as a validation set. Tune B, τ, BN, learning rate, and local epochs for CFedSSL-NID and for the strongest baseline FedProx+Fixmatch on this validation set, using identical search budgets and the same augmentation/noise settings. Then evaluate each selected configuration exactly once on KDDTest+, repeating with 5 random seeds and reporting mean±std. If CFedSSL-NID's test accuracy/F1 does not exceed FedProx+Fixmatch by more than the run-to-run standard deviation, the claimed outperformance is not supported and the paper should be revised.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that CFedSSL-NID outperforms existing federated semi-supervised and fully supervised methods rests on Tables V and VI, whose CFedSSL-NID rows use the configuration (B=1024, τ=0.5, BN=0) selected by hyperparameter tuning directly on KDDTest+ (Section IV-C, Tables II and III). The paper states in Section IV-B that 'All of the following evaluations were conducted on KDDTest+,' and the tuning tables report Acc/Pre/Recall/F1 on that same test set. The final numbers are therefore the maximum of a small grid search over B, τ, and BN, not the performance of a predetermined method. The baselines in Tables V and VI are not reported with equivalent test-set tuning, so the comparison is biased in CFedSSL-NID's favor. With accuracy gaps of only 2.08% (binary) and 1.23% (multi-class) over the strongest baseline FedProx+Fixmatch, and with no error bars or code, the selection effect alone could account for the reported advantage. This concern is independent of the augmentation-semantics question; even if the weak/strong augmentations preserve class semantics, the empirical evidence as presented does not establish outperformance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CFedSSL-NID, a federated semi-supervised learning framework for network intrusion detection in the Internet of Robotic Things. Robot clients with unlabeled data apply weak and strong augmentations and an InfoNCE-style contrastive loss on a lightweight CNN encoder; the server aggregates client encoders, applies an EMA update, and trains a supervised classification head on its own labeled data. Experiments on NSL-KDD use an artificial split of 50,000 labeled server samples and 69,070 unlabeled client samples, with binary and multi-class comparisons against FedAvg/FedProx combined with CR, UDA, and FixMatch, plus fully supervised baselines; the paper also reports a model-complexity comparison centered on a lw-CNN. The central claim is that CFedSSL-NID outperforms existing federated semi-supervised and fully supervised methods while having lower resource requirements.","tokens_in":11273,"tokens_out":7481,"duration_ms":79301,"significance":"If the empirical claims were established, the framework would be a useful practical contribution: it addresses a realistic federated setting in which robot clients lack labels, it combines contrastive pretraining with a lightweight encoder, and it includes an ablation study separating the contributions of augmentation, contrastive learning, and EMA update. The complexity comparison in Table VIII is a useful sanity check for deployability. However, the current evidence does not support the headline outperformance claim. The reported final configuration is selected by tuning on KDDTest+, the same test set used for all reported comparisons, and no code, error bars, or confidence intervals are provided. The paper also does not verify the key semantic-preservation assumption behind its contrastive positive pairs. These issues affect the central empirical claim rather than merely the presentation.","major_comments":[{"comment":"The final CFedSSL-NID configuration is selected by hyperparameter tuning on KDDTest+, and Section IV.B states that all following evaluations were conducted on KDDTest+. Table III shows that accuracy falls from 80.82% at B=1024, tau=0.5, BN=0 to roughly 76–78% for other batch sizes, so the value reported in Tables V and VI is the maximum of a grid search on the test set. The baselines are not shown to have received equivalent test-set tuning, and no confidence intervals are reported despite the statement that results are averaged over more than five runs. With accuracy margins of only 2.08 points (binary) and 1.23 points (multi-class) over the best baseline, the selection effect alone could account for the reported advantage. The central claim of outperformance is therefore not supported by the experiments as presented.","section":"Section IV.B–IV.C, Tables II, III, V, VI"},{"comment":"The method assumes that weak and strong augmentations preserve class semantics: the text states that the augmented pair (ai, bi) is a positive pair because its semantic information remains unchanged. No experiment verifies this assumption for tabular NSL-KDD features. Additive noise can alter class-discriminative features such as service types or count-based attributes, in which case the positive pair in Eq. (1) may contain samples of different classes and the contrastive objective could pull together unrelated representations. The t-SNE visualization in Fig. 6 only shows cluster separation without labels; it does not measure label consistency between augmented and original samples. The ablation gain of 4.15% accuracy attributed to contrastive learning in Table IV is only meaningful if semantic preservation actually holds.","section":"Section III.A, Eq. (1)"},{"comment":"NSL-KDD is a network intrusion benchmark, not IoRT traffic, and the paper does not justify why its distribution represents robotic or vehicular network connections. The artificial split of KDDTrain+ into 50,000 labeled server samples and 69,070 unlabeled client samples also sidesteps the real label-availability process in IoRT, and no sensitivity analysis is given for different label ratios, client heterogeneity, or non-IID data. These choices limit the external validity of the IoRT-specific claims, including the runtime and complexity claims in Table VIII.","section":"Section IV.B and IV.D"}],"minor_comments":[{"comment":"The denominator sums over k=1 to B including the case k=i, which adds exp(sim(z_ai,z_ai)/tau)=1 and, together with the second sum, counts the positive pair twice when i=k. Please clarify whether self-similarity is deliberately included or define the negative set as k≠i.","section":"Eq. (1)"},{"comment":"The configuration text says servers aggregate 10 times and clients update for 5 epochs, but these numbers are not mapped to the notation R_s and P_c introduced in Section III; please specify the exact server rounds and local epochs used for the tables.","section":"Section IV.A"},{"comment":"The baselines are described only as combinations such as FedAvg+Fixmatch; please provide the exact hyperparameters used for each baseline, including learning rate, local epochs, augmentation magnitudes, and any pseudo-label thresholds, so the comparison is reproducible.","section":"Section IV.D"},{"comment":"There are several typos and unclear expressions, including 'In pratical IoRT system' in the Fig. 1 caption, 'Futhermore' in the Conclusion, and 'network comunication' in the Conclusion. A careful proofread is needed.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The test-set tuning issue in Section IV.C is decisive for me. The reported superiority in Tables V and VI is not a fair comparison because CFedSSL-NID's hyperparameters were selected on KDDTest+, while the baselines were not given equivalent tuning and no error bars are reported. I would only reconsider if the authors redo the evaluation with a properly held-out validation split, report confidence intervals, release code, and verify the semantic-preservation assumption of their augmentations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a plausible FedSSL framework for IoRT intrusion detection, but the headline numbers are tuned directly on KDDTest+, so the claim that CFedSSL-NID outperforms existing methods does not hold as written. The architecture is a sensible assembly: SimCLR-style contrastive loss with weak/strong augmentation on robot clients, FedAvg aggregation, and a server-side EMA blend of supervised and unsupervised parameters, using a small 1D CNN. That combination is not deeply novel—FedCon [17] is cited as a contrastive FedSSL method—but the paper is the first to package it specifically for unlabeled robot clients in IoRT, and the motivation is clear: no local labels, privacy constraints, resource limits.\n\nWhat it does well: the method section is readable and mostly reproducible; the ablation in Table IV shows each component (augmentation+dropout, contrastive learning, EMA) contributes, and the complexity table at least reports params/FLOPs for the proposed lw-CNN.\n\nThe soft spot is not subtle. Section IV-B says \"All of the following evaluations were conducted on KDDTest+,\" and Section IV-C reports hyperparameter sweeps over batch size B, temperature τ, and BN count with Acc/F1 measured on that same KDDTest+. The best configuration from that sweep (B=1024, τ=0.5, BN=0) is then used in Tables V and VI. So the reported superiority is a fitted maximum, not the performance of a predetermined method. The gaps over the strongest baselines are small—2.08% binary accuracy, 1.23% multiclass accuracy—and the baselines are not given equivalent test-set tuning. Selection effect alone could account for the gap. The authors also do not report variance despite saying runs were averaged, provide no code, and omit FedCon from the comparison tables even though it is the most closely related prior work. Those are real omissions, not nitpicks.\n\nSecondary concerns: the assumption that random noise augmentations preserve class semantics in tabular NSL-KDD is unverified; if augmentation distorts attack-relevant features, the contrastive pretraining gain could vanish. And NSL-KDD is a 1999 dataset, not IoRT-specific, which limits the external-validity claim. Both are worth mentioning but neither is the main problem.\n\nBottom line: the framework is plausible and the evaluation is fixable, but the evidence as presented does not support the abstract's claim. I would not cite the numbers, and I would not spend referee time on this version. If the author re-runs the experiments with a proper validation split, adds variance and code, includes FedCon, and updates to a current dataset, it could become a reasonable workshop or minor-conference paper. For now, treat it as a useful negative example of test-set tuning.","headline":"Plausible FedSSL framework for IoRT intrusion detection, but the reported gains are the product of tuning on KDDTest+, so the empirical claim does not survive contact with the paper.","tokens_in":11839,"tokens_out":3848,"would_cite":false,"duration_ms":38031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A federated framework for robotic IoT intrusion detection trains on unlabeled robot data plus one labeled server set and beats fully supervised baselines on NSL-KDD.","keywords":["Internet of Robotic Things","federated semi-supervised learning","contrastive learning","intrusion detection","NSL-KDD","exponential moving average","lightweight CNN","privacy-preserving learning"],"falsifier":"Run a neighborhood audit on KDDTest+: apply the paper's weak and strong augmentations to each test sample, then check whether a majority of each augmented sample's nearest neighbors in the original feature space share the original sample's class label; if strong augmentation regularly relocates samples into neighborhoods dominated by a different attack class, the semantic-preservation premise that makes contrastive positives meaningful is empirically false.","tokens_in":10804,"feed_emoji":"🤖","tokens_out":10664,"duration_ms":95515,"temperature":0.7,"pith_summary":"This paper claims that a network intrusion detector for the Internet of Robotic Things can be trained well even when robot clients hold no labeled data. The proposed framework, CFedSSL-NID, lets each robot client learn traffic representations by contrastive learning on unlabeled local data—pulling together weak and strong augmentations of the same sample—while a server, which alone holds labels, fine-tunes the aggregated model through an exponential moving average. On the NSL-KDD benchmark the framework reports 85.33% binary accuracy and 80.82% multi-class accuracy, ahead of federated semi-supervised and even fully supervised baselines that use all labeled training data. It also uses the smallest model in the comparison: a 49,469-parameter one-dimensional CNN taking 0.636 ms per sample. If these results hold, IoRT deployments could get privacy-preserving, real-time intrusion detection without asking robots to label their own traffic.","feed_headline":"Unlabeled robot data beats labeled baselines in intrusion detection","feed_subtitle":"Client contrastive learning plus server-side moving averages beats fully supervised baselines in a 49k-parameter model.","key_machinery":"The load-bearing machinery is the client–server training loop with an EMA bridge. On each robot client, every unlabeled sample is perturbed twice—weakly and strongly—to make a positive pair; a lightweight one-dimensional CNN encoder plus a projection head maps the pair into a latent space, where a temperature-scaled contrastive loss (cosine similarity, $\\tau=0.5$, batch size 1024) pulls those two representations together and pushes every other sample's augmentations apart. Clients upload only encoder parameters; the server aggregates them as in FedAvg, then updates the global model with $\\theta_{t+1}^{\\mathrm{Glo}} = \\xi\\,\\theta_t^{\\mathrm{Glo}} + (1-\\xi)\\,\\theta_t^{\\mathrm{Agg}}$ and refines it with cross-entropy loss on the server's labeled data. The EMA term is what carries the supervised signal back into the contrastively pretrained encoder, and the contrastive loss is what lets unlabeled robot traffic shape the feature space at all.","core_discovery":"CFedSSL-NID's central claim is that combining unsupervised contrastive representation learning on unlabeled robot clients with a small supervised fine-tuning step on a label-holding server can outperform federated and centralized methods that train on fully labeled data, while reducing model cost. The paper attributes the gain to three coordinated mechanisms: random weak and strong augmentations that preserve sample semantics, a contrastive loss in a projection-head latent space that organizes the encoder's features without labels, and an EMA update on the server that blends the contrastively trained client model with the supervised global model. On NSL-KDD's KDDTest+, the framework reaches 85.33% binary and 80.82% multi-class accuracy, and its lightweight CNN has 49,469 parameters, roughly one-quarter to one-fiftieth the size of compared models. For minority attack classes—R2L and U2R—the framework still struggles (F1 45.23 and 18.47), but it beats all listed baselines on aggregate weighted metrics.","pith_inferences":["Because the method's only real assumption is that noise-based augmentations preserve class semantics, the same recipe should transfer to other tabular traffic benchmarks, such as CICIDS or BoT-IoT, where feature scales and noise magnitudes differ; the paper provides no evidence for that transfer, so this is a testable extension rather than a claim.","The EMA weight $\\xi$ is a hidden dial between the two learning signals: near 1 it makes the server ignore client contrastive updates, and near 0 it makes the server discard its own supervised history; tuning it per round could adapt to shifts in either data distribution.","A protocol-aware augmentation scheme, with noise scaled by per-feature variance rather than by a fixed magnitude, would likely make the positive-pair assumption hold on real IoRT captures, whose feature scales differ from NSL-KDD's."],"forward_implications":["Robot clients with zero local labels can still improve a global intrusion detector: CFedSSL-NID reports 85.33% binary accuracy and 80.82% multi-class accuracy on KDDTest+, exceeding every listed baseline, including fully supervised ones that train on all 125,973 labeled NSL-KDD samples.","The detector can run on constrained robot hardware: the lw-CNN has 49,469 parameters and 729,000 FLOPs, and the paper measures 0.636 ms per sample, the smallest model in the comparison.","Privacy is preserved structurally: raw traffic stays on robot clients and only encoder parameters are uploaded and averaged on the server.","The gains are carried by the combination of contrastive pretraining and EMA fine-tuning; removing the contrastive component drops multi-class accuracy from 80.82% to 76.67% in the paper's ablation."],"supporting_citations":[{"why":"Supplies the FedAvg aggregation rule used to combine client encoder updates and the backbone for the supervised federated baselines.","marker":"[10]"},{"why":"Supplies UDA, one of the consistency-regularization baselines (FedUDA) the framework must outperform.","marker":"[11]"},{"why":"Supplies FixMatch, the pseudo-labeling semi-supervised method used in the FedAvg+Fixmatch and FedProx+Fixmatch baselines.","marker":"[13]"},{"why":"Supplies the contrastive learning recipe—projection head, temperature-scaled cosine similarity, and augmented positive pairs—that the client-side pretraining adapts.","marker":"[18]"},{"why":"Supplies the NSL-KDD benchmark and KDDTest+ split on which every comparison is run.","marker":"[22]"},{"why":"Supplies the consistency-regularization baseline (CR) used in FedAvg+CR and FedProx+CR.","marker":"[30]"},{"why":"Supplies FedProx, the proximal federated algorithm used in the supervised and semi-supervised baselines.","marker":"[31]"}],"fun_headline_variants":["Federated contrastive model beats labeled baselines on robot data","Semi-supervised federated learning tops supervised intrusion detection","Unlabeled robot data suffices for effective IoRT security","Lightweight federated framework outperforms fully supervised methods","Contrastive semi-supervised nets win on NSL-KDD for IoRT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the assumption that the paper's weak and strong random noise added to NSL-KDD traffic samples always preserves the attack's class, so that contrastive positive pairs are genuinely similar; if that fails, robot-client pretraining would teach the encoder to merge different attack classes and the reported gains would vanish.","fun_headline_variants_meta":{"raw":{"variants":["Federated contrastive model beats labeled baselines on robot data","Semi-supervised federated learning tops supervised intrusion detection","Unlabeled robot data suffices for effective IoRT security","Lightweight federated framework outperforms fully supervised methods","Contrastive semi-supervised nets win on NSL-KDD for IoRT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2273,"prompt_tokens":912,"completion_tokens":1361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1274}},"tokens_in":528,"tokens_out":1361,"duration_ms":12304,"temperature":1.0,"reasoning_tokens":1274,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:37:23.352551+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a neighborhood audit on KDDTest+: apply the paper's weak and strong augmentations to each test sample, then check whether a majority of each augmented sample's nearest neighbors in the original feature space share the original sample's class label; if strong augmentation regularly relocates samples into neighborhoods dominated by a different attack class, the semantic-preservation premise that makes contrastive positives meaningful is empirically false.","supporting_citations":[{"cited_title":"Communication- efficient learning of deep networks from decentralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the FedAvg aggregation rule used to combine client encoder updates and the backbone for the supervised federated baselines."},{"cited_title":"Unsupervised data augmentation for consistency training,","cited_arxiv_id":null,"evidence_quote":"Supplies UDA, one of the consistency-regularization baselines (FedUDA) the framework must outperform."},{"cited_title":"Fixmatch: Simplifying semi- supervised learning with consistency and confidence,","cited_arxiv_id":null,"evidence_quote":"Supplies FixMatch, the pseudo-labeling semi-supervised method used in the FedAvg+Fixmatch and FedProx+Fixmatch baselines."},{"cited_title":"Temporal ensembling for semi-supervised learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the consistency-regularization baseline (CR) used in FedAvg+CR and FedProx+CR."},{"cited_title":"On convergence of fedprox: Local dissimilarity invariant bounds, non-smoothness and beyond,","cited_arxiv_id":null,"evidence_quote":"Supplies FedProx, the proximal federated algorithm used in the supervised and semi-supervised baselines."}],"review_version":1}