{"id":"3a115ed3-a0b9-4478-ba8f-1611e6155963","arxiv_id":"2506.03207","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An attacker can classify CNN vs RNN traffic in a simulated federated learning setup with up to 100% accuracy, but the experiment conflates model architecture with dataset and uses only 39 traffic captures.","lead":"This paper tests whether an attacker can tell which deep learning model, a CNN or an RNN, is being trained inside a federated learning system just by sniffing network traffic. In a small localhost experiment, machine learning classifiers distinguished the two model types with up to 100% accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.4 perfectly confounds architecture with dataset/task: CNN runs use CIFAR-10/Fashion-MNIST while RNN runs use Sunspot time series, so the classifiers may be detecting data modality or task rather than the DL architecture.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing flaw: the architecture classes in Section 3.4 are perfectly confounded with dataset and task, so the fingerprinting classifiers may be detecting data modality or update size rather than architecture identity. This is not a minor cosmetic issue; it directly undermines the abstract's claim that 'we can identify specific architectures running within the subsection of the network traffic.' I agree with the REJECT verdict because the experiment as designed cannot support the central claim. The paper does have some merits: it is a clearly written feasibility study, it uses a local controlled testbed to isolate FL traffic, and it honestly lists limitations in Section 5.1, including the small dataset and the assumption of a single architecture per traffic instance. However, the confound is not listed as a limitation, and it is the most fundamental issue. The concrete test I propose would settle the question: a crossed design with both architectures trained on both datasets would separate architecture effects from data-modality effects. Without such a control, the reported 95-100% accuracy is equally consistent with the much weaker claim that an observer can distinguish image-classification FL workloads from time-series-forecasting FL workloads. Therefore, no verdict adjustment is needed; the reader's REJECT stands.","tokens_in":9099,"tokens_out":2324,"duration_ms":23382,"concrete_test":"Run a 2x2 factorial control: train both a CNN and an RNN on the same dataset (e.g., both on CIFAR-10 and both on Sunspot), collect FL traffic traces for each combination, and retrain the Random Forest fingerprint classifier on architecture labels while holding the dataset fixed. If test accuracy drops to near chance (or the classifier fails to generalize across datasets), the original result was driven by dataset/task or model size rather than architecture. Separately report parameter counts and per-round update payload sizes for each model pair to confirm whether packet-size features are proxies for model size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that an adversary can identify the deep learning architecture from FL network traffic — rests on a two-class experiment in which architecture and data modality are perfectly confounded. Section 3.4 states: 'For each DL architecture, we use distinct datasets and tasks to ensure variational workloads.' CNN models are trained on CIFAR-10 and Fashion-MNIST images; RNN models are trained on the Sunspot time-series dataset. The classification target is 'CNN vs. RNN', but the two classes differ simultaneously in model family, data modality, dataset, task, and likely model parameter count. A classifier trained on these traces can achieve high accuracy by distinguishing image training from time-series training — or simply by distinguishing large update payloads from small ones — without learning anything architecture-specific. The reported traffic features support this concern: Figure 4 shows mean interarrival time separating the classes, and packet-size distributions differ substantially, which in FL are dominated by model update sizes. Because model sizes are not reported or controlled, the observed traffic differences may be driven entirely by model size or data characteristics, not by architectural family. Additionally, the test set is only 23 traffic traces (11 RNN, 12 CNN); the difference between 100% and 95.65% accuracy is exactly one misclassified instance, so the headline numbers are statistically fragile. The authors do acknowledge limitations about dataset size and overfitting in Section 5.1, but the abstract and discussion nevertheless assert that 'we can identify specific architectures.' This assertion is not supported by the experimental design without a control condition that varies architecture while holding dataset, task, and model size fixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a passive network-traffic fingerprinting attack against federated learning systems, aiming to identify whether a CNN or RNN is being trained by analyzing packet sizes, directions, and interarrival times. The authors build a localhost FL testbed, collect traffic from CNN training on image datasets and RNN training on a time-series dataset, extract statistical features, and train Random Forest, SVM, and XGBoost classifiers. They report 100% accuracy for Random Forest and 95.65% for SVM and XGBoost on a held-out set of 23 packet captures, concluding that DL architectures can be fingerprinted from encrypted FL traffic.","tokens_in":9377,"tokens_out":3536,"duration_ms":34865,"significance":"If the finding is robust, it would identify a new privacy risk in FL and motivate network-level defenses. The topic is timely, and the paper is clearly written with a reproducible-looking pipeline. However, the significance is currently conditional: the experiment perfectly confounds architecture family with dataset/task/modality, and the test set is too small to support the claimed precision. The evidence does not yet isolate architecture as the cause of the observable traffic differences.","major_comments":[{"comment":"The central claim that an adversary can identify the DL architecture is not supported because architecture and data modality are perfectly confounded. CNN models are trained on CIFAR-10 and Fashion-MNIST images, while RNN models are trained on the Sunspot time-series dataset, so the two classes differ simultaneously in model family, data modality, dataset, task, and likely model size. A classifier can achieve high accuracy by detecting image training versus time-series training, or by detecting large versus small update payloads, without learning anything architecture-specific. The reported features support this concern: Figure 4 shows that mean_frame (packet size) separates the classes, and in FL, packet size is dominated by model update size, which is not reported or controlled. Please add control conditions that train both architectures on the same data modality and the same architecture on different data modalities, and report model parameter counts and update sizes for every run.","section":"Section 3.4"},{"comment":"The test set consists of only 23 packet captures (12 CNN, 11 RNN), and the difference between 100% and 95.65% accuracy is exactly one misclassified instance. No confidence intervals, repeated train/test splits, or per-class confusion counts are reported, so the headline accuracy figures are statistically fragile. The paper should report exact confusion matrices, bootstrap or exact binomial confidence intervals, and, ideally, a larger number of independent captures to demonstrate that the result is not driven by a single instance.","section":"Section 4.2, Table 1"},{"comment":"The testbed uses localhost loopback traffic on a single machine with client instances simulated on unique ports. This controlled setting is acknowledged in Section 5.1, but the abstract and introduction claim a general network-layer vulnerability. Since the discriminating features include interarrival times, it is important to show that the observed separation persists under realistic network conditions such as packet loss, jitter, or non-loopback transport. At minimum, the paper should quantify how much of the discriminative signal is carried by timing features versus packet size and direction, and should temper the abstract's claim to match the controlled environment.","section":"Sections 3.3 and 5.1"}],"minor_comments":[{"comment":"The heading 'Feature Section' appears to be a typo for 'Feature Selection', and the sentence 'From here, I took the highest-ranked features' uses first person; please change to 'we selected'.","section":"Section 3.5.3"},{"comment":"The Random Forest row reads '100%RNN' without a space; please fix the formatting for readability.","section":"Table 1"},{"comment":"The caption says 'KL divergence plots', but the panels show density plots and no KL divergence values are reported in the text or figure. Please either report the calculated KL values or revise the caption.","section":"Figure 4"},{"comment":"The sentence 'For each DL architecture, we use distinct datasets and tasks to ensure variational workloads' is precisely the source of the confounding described above. Please at least justify this design choice and explicitly discuss how it limits the interpretation of the results.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an interesting question, but the current experimental design cannot distinguish architecture fingerprinting from dataset/task fingerprinting. I recommend major revision rather than rejection because the confounding and small-sample issues are addressable with additional control experiments, more captures, and more careful reporting of model sizes and uncertainty. The paper would be a useful workshop contribution if these issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. It asks a genuinely new question — can a passive observer infer the DL architecture from FL network traffic — and it attempts a controlled testbed with standard classifiers. But the experiment confounds architecture with dataset and task: every CNN run uses images (CIFAR-10/Fashion-MNIST), every RNN run uses a time series (Sunspot). So the classifiers might be detecting data modality or model size, not architectural family.\n\nWhat is good: the setup is simple and reproducible, they use standard classifiers with grid-search tuning and report per-class metrics, and the limitations section in Section 5.1 is candid. They also try adding browsing noise. The feature analysis with Fisher scores and KL divergence is a reasonable sanity check, and the paper is clearly written.\n\nThe soft spots are serious. The confounding is load-bearing. The test set is 23 captures, so one misclassification moves accuracy from 100% to 95.65%. Model sizes are not reported, so the packet-size differences in Figure 4 could be driven entirely by update size. The abstract and conclusion assert that 'we can identify specific architectures,' but the evidence doesn't support that. The authors acknowledge some limitations, but they don't name the confound itself — they say they used distinct datasets 'to ensure variational workloads,' which is exactly the problem.\n\nWho this is for: researchers working on FL privacy or traffic fingerprinting. They might get ideas for a follow-up study, but the paper as is does not establish the phenomenon. It could become a reasonable workshop paper after adding a control condition where CNN and RNN are trained on the same data type, plus a same-architecture-different-size condition to separate model-size effects.\n\nRecommendation: this deserves a serious referee because the question is important and the first-pass result is a clean failure mode worth correcting. I would not desk-reject it, but I'd expect the review to ask for a redo of the experiment. The authors are honest and the idea is worth taking seriously; the execution just doesn't yet support the claim.","headline":"A genuinely new question undermined by perfect confounding: CNN and RNN runs differ in dataset and task, so the classifiers may be detecting data modality, not architecture.","tokens_in":9938,"tokens_out":2138,"would_cite":false,"duration_ms":20630,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A passive observer can identify a deep learning architecture being trained in a federated learning system just from network packet sizes, directions, and timings.","keywords":["federated learning","network traffic fingerprinting","deep learning architecture","CNN","RNN","passive adversary","traffic metadata","privacy leakage"],"falsifier":"Capture traffic from an FL testbed where the identical dataset and task are trained with a CNN and an RNN, and separately where each architecture trains on several datasets; if the trained fingerprints no longer separate the architectures or instead separate datasets, the central claim is refuted.","tokens_in":8882,"feed_emoji":"🕵️","tokens_out":10714,"duration_ms":83311,"temperature":0.7,"pith_summary":"Federated learning is a training setup where many devices improve a shared model without uploading their raw data, on the theory that the server never sees private inputs. This paper asks whether an outsider who only watches the network can still learn something sensitive: specifically, which deep learning architecture the system is training. Using a small testbed with CNN and RNN models, it derives statistical features from packet sizes, directions, and interarrival times, and shows that standard classifiers separate the two architectures nearly perfectly—Random Forest reached 100 percent accuracy on the test captures, SVM and XGBoost about 95.7 percent. The authors' conclusion is that architecture fingerprinting is feasible with only network-layer metadata, so a passive adversary could identify the model and then tailor attacks to it. That would matter because it undermines the privacy promise of federated learning even when model updates are encrypted.","feed_headline":"Packet metadata reveals CNN vs RNN in federated learning","feed_subtitle":"Passive observers can learn the model architecture, turning FL's privacy promise into an attack surface.","key_machinery":"The mechanism is a statistical traffic fingerprint built from packet-level metadata. For each captured session, the paper computes packet-length statistics, direction statistics, interarrival-time statistics, and peak counts; Fisher scores, which rank features by how far class means are apart relative to within-class variance, select the most discriminative ones. Those features feed three standard classifiers—Random Forest, SVM, and XGBoost—which learn the CNN-versus-RNN boundary. The load-bearing idea is that architectural differences in compute and communication show up in the size and rhythm of packets exchanged between server and clients, even though the payloads themselves are encrypted.","core_discovery":"The paper's central claim is that deep learning architectures leave distinguishable signatures in the network-layer traffic of federated learning, and that a passive adversary with no access to payloads, flow-level data, or model updates can recover the architecture from packet metadata alone. Concretely, the local training and update exchange of a CNN versus an RNN produce different packet-size distributions, transmission directions, and interarrival-time patterns; statistics computed from these patterns (means, spreads, peak counts, uplink/downlink proportions) are enough for classifiers to tell the two apart. The authors report near-perfect separation on their test data and read this as evidence that FL systems have a previously unexplored indirect privacy vulnerability.","pith_inferences":["A control experiment that trains both architectures on the same dataset would clarify whether the fingerprint is architectural or driven by data modality, since the current design couples the two.","If workload size is the true signal, the same features might estimate model size, batch size, or local epoch count, giving an adversary finer-grained information than architecture alone.","A defense experiment follows directly: padding packets and regularizing transmission timing should flatten interarrival-time features and lower fingerprinting accuracy, which can be tested on the same testbed.","The mechanism is not FL-specific; any periodic client-server update protocol would expose the same metadata, so the threat may extend to other distributed training schemes."],"forward_implications":["An FL adversary who can sniff layer-3 traffic can infer the trained architecture without decrypting anything, which is a concrete privacy leak beyond the usual membership or inversion attacks.","The fingerprinting survives at least some background noise, since the paper includes web-browsing traffic during data collection and still gets high accuracy.","Knowing the architecture lets an attacker select model-specific exploits, such as adversarial perturbations tuned to CNNs or RNNs, making the FL system more exposed.","Because the distinguishing signal lives in packet metadata, defenses would have to change traffic shape (padding, rate smoothing, aggregation obfuscation) rather than rely on encryption alone.","Extending the same pipeline to modern architectures (transformers, GANs, RL policies) is a stated next step, with the same passive threat model."],"supporting_citations":[{"why":"Defines federated learning and the server-client update exchange that the attack passively observes.","marker":"[12]"},{"why":"Random Forest, the classifier that achieved 100 percent fingerprinting accuracy in the evaluation.","marker":"[2]"},{"why":"XGBoost, the gradient-boosting classifier evaluated for fingerprinting.","marker":"[3]"},{"why":"Support Vector Machine, the classifier evaluated for fingerprinting.","marker":"[4]"},{"why":"Shows unintended feature leakage in collaborative learning, the line of work this attack extends to model architecture.","marker":"[13]"},{"why":"Demonstrates device fingerprinting in FL using network traffic feature engineering, a precedent for the method.","marker":"[5]"},{"why":"Analyzes user-level privacy attacks against federated learning via network information, motivating the threat model.","marker":"[17]"}],"fun_headline_variants":["Federated learning model type leaked via network traffic","Passive traffic analysis identifies FL model architectures","Traffic patterns fingerprint deep learning models in FL","Model architecture leaks via federated traffic metadata","Network-layer traffic betrays DL architecture in FL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that traffic differences come from the choice of CNN versus RNN, but since CNNs were trained on image datasets and RNNs on a time-series dataset, the differences could instead come from the dataset or task; the paper never runs the same data through both architectures.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning model type leaked via network traffic","Passive traffic analysis identifies FL model architectures","Traffic patterns fingerprint deep learning models in FL","Model architecture leaks via federated traffic metadata","Network-layer traffic betrays DL architecture in FL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2646,"prompt_tokens":889,"completion_tokens":1757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1687}},"tokens_in":505,"tokens_out":1757,"duration_ms":11627,"temperature":1.0,"reasoning_tokens":1687,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:26:39.331611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Capture traffic from an FL testbed where the identical dataset and task are trained with a CNN and an RNN, and separately where each architecture trains on several datasets; if the trained fingerprints no longer separate the architectures or instead separate datasets, the central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Support Vector Machine, the classifier evaluated for fingerprinting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates device fingerprinting in FL using network traffic feature engineering, a precedent for the method."}],"review_version":1}