{"id":"9f483d57-3a4a-4207-a0e7-b1df7743211b","arxiv_id":"2502.06099","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A server-edge fine-tuning federated learning framework claims up to 42% memory savings and 75% training time savings with 99.2% intrusion detection accuracy on NSL-KDD, though the evaluation design uses the test set for pre-training.","lead":"The paper proposes FedFT, a federated learning design in which a central server pre-trains an intrusion detection model and participating edge devices only fine-tune the final output layers on their own data. It claims large savings in memory and training time with competitive accuracy, which matters for running security monitoring on resource-limited connected vehicles.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Server pre-trains on NSL-Test and accuracy is measured on that same set; reported 99.2% is a test-set-leakage artifact, not evidence of competitive performance.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the server pre-trains on the full NSL-Test set, and the model is then evaluated on that same set. This is a textbook test-set contamination that invalidates the central accuracy claim. The paper's own text in Section IV-A confirms the protocol, and the experimental section gives no indication that a different evaluation split was used. Because the headline contribution is an IDS framework with \"competitive accuracy of up to 99.2%,\" losing that accuracy claim removes the main evidence for the framework's viability. The resource-savings numbers are also internally inconsistent (abstract 42%/75% vs. Table III 47%/58%), but that is a reporting problem, not a fundamental invalidation. The test-set leakage is fundamental: no amount of re-labeling or hyperparameter tuning can justify evaluating on data used for pre-training. Therefore the reader's REJECT verdict is appropriate, and my stress-test does not change it. I would still recommend rejection unless the authors can demonstrate that the accuracy was measured on a truly held-out set not used in any phase of training.","tokens_in":8516,"tokens_out":2649,"duration_ms":24515,"concrete_test":"Re-run the FedFT-3 experiment under the same hyperparameters and client partitioning, but pre-train the server model only on a random subset of NSL-Train (or on the original KDDTrain+ set) and never touch any NSL-Test record during training. Then evaluate the final federated model on NSL-Test. If the resulting accuracy drops materially below 99.2% or below the FL-IDS baseline, the reported competitive accuracy was an artifact of test-set leakage. Also recompute the memory and time reductions from Table III and compare them to the abstract's 42% and 75% figures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of competitive accuracy (up to 99.2%) rests on an evaluation protocol that leaks the test set into training. Section IV-A states: \"we utilize the entire testing data as proxy data for pre-training on the server,\" while clients fine-tune on partitions of NSL-Train. The only \"testing data\" in the paper is NSL-Test, and standard practice (plus the paper's own language, which calls this split the testing data) implies the accuracy in Table III is computed on NSL-Test. Thus the server's pre-training has already seen the features and labels of the evaluation set. Even though clients later fine-tune on NSL-Train, the feature extractor has been fit to the test distribution, so the reported 99.2% is not an honest estimate of generalization to unseen data. This invalidates the main empirical support for FedFT's claimed advantage over the centralized and FL-IDS baselines. If the authors actually evaluated on a held-out portion of NSL-Train, the paper does not say so, and no code or data are released to check. Secondary issues, such as the abstract claiming 42% and 75% reductions while Table III implies 47% and 58%, further weaken the presentation, but the test-set leakage alone is sufficient to undermine the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FedFT, a hybrid server-edge federated learning framework for intrusion detection in connected and autonomous vehicle networks. The server pre-trains an IDS model on proxy data, then clients fine-tune only the classification head on local data. Experiments on NSL-KDD claim competitive accuracy (up to 99.2%) with reduced memory usage (up to 42%) and training time (up to 75%) compared to centralized and FL-IDS baselines, plus scalability to 8 clients.","tokens_in":8854,"tokens_out":2537,"duration_ms":22798,"significance":"If the results were sound, the framework would be a useful engineering contribution for resource-constrained FL deployments: it provides a concrete recipe (server pre-training plus client-side fine-tuning of the classifier head) and reports detailed resource measurements from real Raspberry Pi 4 clients. The scalability analysis is a reasonable first step. However, the central accuracy claim is undermined by a test-set leakage problem in the experimental protocol, and the abstract's efficiency numbers are inconsistent with the reported table. As published, the paper does not provide trustworthy evidence for its main claims.","major_comments":[{"comment":"The evaluation protocol leaks the test set into pre-training. Section IV-A states, \"we utilize the entire testing data as proxy data for pre-training on the server,\" and the only testing data in the paper is NSL-Test. The same NSL-Test set is then used for evaluating the final model (the paper reports no other held-out set). The global model therefore sees the features and labels of the evaluation data before fine-tuning, rendering the reported 99.2% accuracy and the comparison with baselines in Table III invalid as estimates of generalization. A correct evaluation must hold out a separate test set that is not touched during either server pre-training or client fine-tuning, or partition NSL-Train into client and test portions.","section":"IV-A"},{"comment":"The abstract claims memory usage reductions \"up to 42%\" and training time decreases \"up to 75%\". Table III, however, shows FedFT-1 reducing memory from 8.1 MB to 4.3 MB, which is a 47% reduction relative to the centralized baseline (or 49% relative to FL-IDS), and execution time from 1339 s to 564 s, a 58% reduction. No configuration in Table III achieves a 75% training time reduction. The abstract's numbers are not supported by the presented data and should be corrected.","section":"Abstract and Table III"},{"comment":"Section IV-A explicitly states \"We assume that the dataset is independently and identically distributed (IID).\" This assumption is particularly strong for CAV networks, where traffic patterns, attack types, and device conditions are naturally heterogeneous. The paper nevertheless claims the framework is suitable for large-scale CAV networks. This mismatch is acknowledged only in the future-work sentence about \"data heterogeneity.\" For the paper's central deployment claim to be credible, the experiments need to include at least one non-IID partition, or the claims must be scaled back accordingly.","section":"IV-A and VI"}],"minor_comments":[{"comment":"The paper states each experiment is repeated three times and averaged, but no standard deviations, variance, or confidence intervals are reported. Adding these would strengthen the reliability of the resource and accuracy comparisons.","section":"IV-B"},{"comment":"The scalability analysis in Table IV reports only accuracy, loss, memory, and time, but does not report the accuracy of the centralized or FL-IDS baselines at the same client counts, nor does it describe whether the \"virtual Flower client instances\" run on the same Raspberry Pis or on the server. This makes the scalability claim hard to interpret.","section":"V-B"},{"comment":"The classification is binary (normal versus attack), yet NSL-KDD has four attack classes. The paper should justify why a binary decision is appropriate for the CAV intrusion detection use case, especially since the dataset description in Section IV-A lists four attack categories.","section":"III-B"},{"comment":"The paper compares FedFT against only one FL baseline ([28]). Given the large literature on communication-efficient and resource-aware federated learning, a broader comparison (or a justification for the single baseline) would strengthen the resource-efficiency claims.","section":"V-A"}],"recommendation":"reject","confidential_remarks":"The test-set leakage in Section IV-A is a fundamental experimental-design flaw that invalidates the paper's headline accuracy number. The resource-efficiency measurements are independent and may be salvageable, but the accuracy claim is central to the paper's thesis. Re-running the experiments with a proper train/test split is feasible in principle, but the results could change materially, so the paper is not publishable in its current form. The abstract-versus-table inconsistency further suggests that the numbers have not been carefully checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the one number that matters most, the 99.2% accuracy, is an artifact. Section IV-A says the server pre-trains on \"the entire testing data\" (i.e., NSL-Test), and evaluation is done on the same test set. So the feature extractor has already been fit to the evaluation distribution. That undermines the central claim of competitive accuracy and the comparison against centralized and FL-IDS baselines.\n\nWhat the paper does well: it's a real deployment, not just simulation. Flower server on a workstation, Raspberry Pi 4s as clients, three runs averaged. The resource measurements (memory, execution time) are concrete and independent of the accuracy problem — FedFT-1 at 4.3 MB versus 8.1 MB for centralized is a genuine saving. The comparison with FL-IDS [28] is useful. The related work is honest about transfer learning and TinyTL, so the novelty is an incremental combination, but that's fine if the evaluation were clean.\n\nSoft spots, in order of severity. First, the leakage. Even if the clients fine-tune on NSL-Train partitions, the server's pre-training on the full test set means the model has seen the test features and labels. The paper doesn't report a held-out split, and no code is released to check. Second, the abstract/conclusion numbers don't match Table III: 42% and 75% don't appear in the table; the table shows about 47% and 58% for FedFT-1 versus centralized. That's a presentation error that needs fixing. Third, the IID assumption is optimistic for CAV networks, and the scalability study only goes from 4 to 8 clients, so \"large-scale\" is a stretch. Fourth, hyperparameters like communication rounds and PCA components aren't fully specified.\n\nI don't think the central idea is wrong; it's a standard transfer-learning-plus-FL recipe, and the resource savings are plausible. But as written, the accuracy evidence doesn't support the headline. This is a paper for readers working on edge FL for IDS who want a concrete benchmark; they'd get value from the deployment details but should ignore the 99.2% until it's re-measured on a clean split. For peer review: I'd send it to a referee, not desk-reject, because the system and resource data are worth scrutiny and the fix (re-run on a proper held-out split) is straightforward. But the current version should be rejected and revised.","headline":"The reported 99.2% accuracy is a test-set leakage artifact — pre-training on NSL-Test and evaluating on the same data — but the resource-efficiency measurements are real and the paper deserves a corrective review.","tokens_in":9326,"tokens_out":3182,"would_cite":false,"duration_ms":27141,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid server-edge federated learning framework achieves up to 99.2% intrusion detection accuracy with up to 42% less memory and 75% less training time.","keywords":["Federated Learning","Intrusion Detection","Connected and Autonomous Vehicles","Transfer Learning","Fine-Tuning","Resource Constraints","Scalability","NSL-KDD"],"falsifier":"Run the same FedFT pipeline but with server pre-training restricted to a proxy dataset that is disjoint from the evaluation set, then measure accuracy on the untouched NSL-Test split. If accuracy stays near 99.2%, the framework's accuracy claim generalizes; if it drops sharply, the headline accuracy is an artifact of pre-training on the test set.","tokens_in":8268,"feed_emoji":"🚗","tokens_out":7850,"duration_ms":65238,"temperature":0.7,"pith_summary":"This paper claims that the resource bottleneck of federated intrusion detection on connected vehicles can be broken by splitting the work between a central server and edge devices. The server pre-trains a shared model on a proxy dataset, and each client freezes the feature-extraction layers and fine-tunes only a small classification head on its local data. On the NSL-KDD benchmark the authors measure up to 99.2% accuracy, up to 42% lower training-memory use, and up to 75% shorter training time than the baselines they compare against. They also report that accuracy degrades by less than one percent as the number of clients grows from four to eight. If these results are correct, the framework offers a practical way to run intrusion detection training on vehicles with tight memory and compute budgets.","feed_headline":"Federated intrusion detection cuts memory 42%, training time 75%","feed_subtitle":"Hybrid server-edge framework reaches up to 99.2% accuracy while keeping edge devices lightweight.","key_machinery":"The load-bearing mechanism is the split of training labor between server-side pre-training and client-side layer-wise fine-tuning. A server first trains the entire CNN-MLP model on proxy data; clients then freeze all CNN feature-extraction layers and update only a small number of fully connected layers in the MLP head using their private local data. Because the frozen layers produce no new gradients or activations on the client, memory and computation are dominated by the head, which is what yields the reported 42% memory reduction and 75% training-time reduction. The number of fine-tuned layers, one versus three, acts as a knob controlling the accuracy-versus-resource trade-off.","core_discovery":"The central discovery is that a hybrid server-edge federated learning framework, named FedFT, makes on-device intrusion-detection training lightweight without giving up accuracy. The server pre-trains a CNN-plus-MLP model on proxy data and then distributes it; clients freeze the CNN module and fine-tune only the MLP classification head on their local data, sending the updated head parameters back for federated averaging. This keeps the expensive feature extractor shared and frozen, so each client only allocates memory for the head's parameters, activations, and gradients. The authors report that FedFT-3, which fine-tunes three fully connected layers, reaches 99.2% accuracy with 20% lower memory use and 5% shorter execution time than a centralized baseline, while FedFT-1, which fine-tunes one layer, reaches 94.3% accuracy with 42% lower memory use and 75% shorter execution time. Scalability runs with four, six, and eight clients show accuracy between 93.9% and 94.8%, a drop of less than one percentage point.","pith_inferences":["If the server were pre-trained on a proxy set actually disjoint from the test set, the 99.2% accuracy figure would likely fall, because the current setup lets the server see the evaluation data during pre-training.","The eight-client maximum in the scalability study is far below real connected-vehicle fleet sizes, so the 'minimal degradation' claim has only been demonstrated for small groups, not for deployment-scale networks.","The layer-freezing recipe is transportable: any IoT sensing task with a suitable server-side proxy dataset could adopt the same server-pre-train, client-fine-tune-head pattern.","The paper assumes IID data across clients; real fleets generate non-IID data, so the resource and accuracy trade-offs under data heterogeneity remain open questions."],"forward_implications":["The same framework could let vehicles with very different hardware join the same federated training round, since each vehicle only needs to run the small classification head update.","Operators can tune the number of fine-tuned layers per vehicle, trading a few accuracy points for large resource savings in the most constrained devices.","The design keeps the global model up to date across many clients because only the head parameters are exchanged, and the scalability runs suggest a small accuracy drop as fleet size grows.","Applying this to the NSL-KDD benchmark positions the approach for network-based attack detection in V2X traffic, not just the laboratory dataset."],"supporting_citations":[{"why":"Provides the federated averaging strategy that the server uses to aggregate client updates and the baseline decentralized training concept the paper builds on.","marker":"[7]"},{"why":"Supplies the on-device fine-tuning memory-reduction insight that motivates freezing layers to cut memory rather than shrinking the whole model.","marker":"[24]"},{"why":"Provides the federated learning orchestration library used to run the server-client simulations.","marker":"[25]"},{"why":"Provides the NSL-KDD dataset used for all intrusion-detection experiments and its train/test split.","marker":"[26]"},{"why":"Defines the FL-IDS baseline that FedFT is compared against in the head-to-head performance table.","marker":"[28]"}],"fun_headline_variants":["Lightweight federated IDS: 42% less memory, 75% faster training","Hybrid FL cuts IDS training time 75%, memory 42%","Fine-tune just the head: 42% less memory, 75% faster","Server pre-train, edge fine-tune: 42% less memory, 75% faster IDS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The server pre-trains the model on the same NSL-Test data that is later used to measure accuracy, so the reported accuracy assumes it is acceptable for the evaluation set to be seen during pre-training.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight federated IDS: 42% less memory, 75% faster training","Hybrid FL cuts IDS training time 75%, memory 42%","Fine-tune just the head: 42% less memory, 75% faster","Server pre-train, edge fine-tune: 42% less memory, 75% faster IDS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000896,"raw_usage":{"total_tokens":3892,"prompt_tokens":1007,"completion_tokens":2885,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2790}},"tokens_in":623,"tokens_out":2885,"duration_ms":18515,"temperature":1.0,"reasoning_tokens":2790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:46:16.108769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same FedFT pipeline but with server pre-training restricted to a proxy dataset that is disjoint from the evaluation set, then measure accuracy on the untouched NSL-Test split. If accuracy stays near 99.2%, the framework's accuracy claim generalizes; if it drops sharply, the headline accuracy is an artifact of pre-training on the test set.","supporting_citations":[{"cited_title":"Tinytl: Reduce memory, not parameters for efficient on-device learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the on-device fine-tuning memory-reduction insight that motivates freezing layers to cut memory rather than shrinking the whole model."},{"cited_title":"A study on nsl-kdd dataset for intrusion detection system based on classification algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the NSL-KDD dataset used for all intrusion-detection experiments and its train/test split."},{"cited_title":"Fl- ids: Federated learning-based intrusion detection system using edge devices for transportation iot,","cited_arxiv_id":null,"evidence_quote":"Defines the FL-IDS baseline that FedFT is compared against in the head-to-head performance table."}],"review_version":1}