{"id":"16feceb3-2c8e-47c8-8a82-1445bfce5a30","arxiv_id":"2501.12112","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A smart-contract-orchestrated federated learning framework for financial bot detection reports 99% binary and 97% multiclass F1-scores on a SMOTE-expanded version of a 270-address Ethereum dataset.","lead":"This paper proposes BotDetect, a federated learning system run through smart contracts that aims to detect financial bots on Ethereum-like blockchains without sharing raw transaction data. It reports very high detection accuracy on a synthetic dataset, and benchmarks the on-chain aggregation cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.99/0.97 F1 is not a valid estimate of real-world detection accuracy: Section IV-A applies SMOTE before the train/test split, so the test set is mostly synthetic points generated from training neighborhoods.","rationale":"The reader's verdict identifies the same weak point: the SMOTE-before-split protocol makes the test set partly synthetic and correlated with training data. I agree that this is the most load-bearing concern because the paper's main quantitative claim is the detection F1; the framework architecture and blockchain benchmark cannot compensate if that number is not a valid estimate. The issue is concrete: with 270 original addresses and 5000 synthetic samples per class, the test set is almost entirely synthetic, and SMOTE's construction guarantees proximity to training samples. Re-evaluating on real held-out addresses is a straightforward, decisive check. I would keep the reader's REJECT verdict, since the central accuracy claim is currently unsupported. I am not raising objections about the permissioned consensus or the reliability assumption; those are stated assumptions. The blockchain performance results are a useful, separate contribution.","tokens_in":7832,"tokens_out":6527,"duration_ms":68319,"concrete_test":"Split the original 270 addresses into 70% train / 30% test (or stratified 5-fold cross-validation) before any SMOTE; then apply SMOTE to the training split only and evaluate the DFL model on the untouched real test addresses. In the same protocol, retrain Random Forest, Gradient Boosting, and AdaBoost on the SMOTE-augmented training data with the same features and a short hyperparameter search. If DFL's F1 falls toward the 0.80-0.85 range or a simple baseline matches it, the reported 0.99/0.97 is an artifact of the SMOTE-before-split evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A states: 'Given the limited number of samples in the dataset... we expand the dataset using SMOTE [18] with κ=5. We generated 5000 samples for each data class... The dataset is then split into 70% training and 30% testing.' Because the split occurs after SMOTE expansion, the test set is dominated by synthetic samples: for binary classification, roughly 10,000 synthetic samples plus 270 real addresses means a 30% test slice is about 97% synthetic. SMOTE creates each synthetic point by interpolating among κ=5 nearest neighbors in the original feature space, so synthetic test points are by construction close to training points; the split therefore does not provide independent test data. The central accuracy claim (F1 0.99 binary, 0.97 multiclass in Tables II-IV) is measured on this non-independent test set and says little about detecting previously unseen real bot addresses. The 270 real labels are too few to support the claim by themselves, and no repeated-seed confidence intervals are reported. Comparisons against Random Forest, Gradient Boosting, and AdaBoost use numbers from [1] trained on the original dataset, not on the same SMOTE-expanded data, so the comparison conflates method with augmentation. The blockchain throughput/latency benchmark is real and useful, but it validates infrastructure performance, not detection accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BotDetect, a decentralized federated learning (DFL) framework for detecting financial bots on EVM blockchains. In the proposed design, clients train a fully connected neural network on local transaction-derived features, and model updates are aggregated on-chain by a smart contract using a permissioned QBFT consensus. The authors evaluate detection accuracy using the 270-address dataset of Niedermayer et al., augmented via SMOTE, and report binary F1 of 0.99 and multiclass F1 of 0.97 with 3-5 clients. They also benchmark the smart-contract functions submitUpdate and aggregateModel with Hyperledger Caliper, reporting a maximum throughput of about 255 TPS for submitUpdate at 1s block time.","tokens_in":8099,"tokens_out":6907,"duration_ms":74036,"significance":"The problem is timely and relevant: financial bots on EVM blockchains cause measurable harm, and a privacy-preserving decentralized detection framework is a worthwhile goal. The smart-contract orchestration of federated learning and the on-chain aggregation design are architecturally interesting, and the throughput/latency benchmark in Section IV-C is a useful empirical contribution. However, the paper's headline claim of near-99% detection accuracy is not supported by the current experimental protocol, because the SMOTE-based augmentation is applied before the train/test split, contaminating the test set with synthetic samples generated from training neighborhoods. As a result, the reported F1 scores are not credible estimates of performance on unseen real addresses. The comparison against centralized baselines is also not apples-to-apples. The core contribution therefore rests on an invalid evaluation, and the paper needs a substantial revision of its experimental methodology.","major_comments":[{"comment":"The dataset is expanded with SMOTE before the 70/30 split. Because each SMOTE sample is generated by interpolating among its k=5 nearest neighbors in the original feature space, the synthetic test samples lie close to training samples and do not provide an independent test of generalization. For the binary case, the expanded dataset contains 10,000 synthetic examples plus 270 real addresses, so a 30% test slice is roughly 97% synthetic. The F1 scores of 0.99 (Table III) and 0.97 (Table IV) are therefore measured on a non-independent test set and do not support the claim that BotDetect detects previously unseen real bot addresses. The split must be performed on the original 270 addresses first, SMOTE applied only to the training partition, and metrics reported on the untouched real test fold, ideally with multiple random seeds and confidence intervals.","section":"Section IV-A, Tables II-IV"},{"comment":"The comparison in Tables III and IV is not apples-to-apples: the Random Forest, Gradient Boosting, and AdaBoost numbers are taken from [1], where they were obtained on the original 270-address dataset without SMOTE augmentation, while 'Our Work' is trained on SMOTE-expanded data and evaluated on a synthetic-heavy test set. This conflates the detection method with the data-augmentation protocol. To support the claimed superiority, the authors must train all baselines on the same augmented training partition and evaluate them on the same untouched real test partition.","section":"Tables III and IV"},{"comment":"The centralized baseline in Table II is insufficiently specified. The text says 'identical setups with 4 and 5 clients' but does not state whether the centralized model was trained on the entire training portion or on only a single client's subset. Since DFL aggregates updates from all clients, a centralized model trained on all data would ordinarily be expected to perform at least comparably; the large gap (0.877 vs 0.984) suggests the centralized model may have been trained on a smaller subset or with different hyperparameters. The authors should specify the exact training data and hyperparameters for the centralized baseline, and if it was intentionally limited, explain why the comparison is fair.","section":"Table II"},{"comment":"The abstract and conclusion claim that the framework maintains 'robustness', but no experiment addresses robustness: there are no tests with non-IID client data, client dropout, poisoning attacks, or varying client participation. Either add such experiments or remove the robustness claim from the abstract and conclusion.","section":"Abstract and Section V"}],"minor_comments":[{"comment":"Equation (5) presents unweighted averaging and Eq. (6) presents data-size-weighted aggregation, but Algorithm 1 implements simple averaging. Please clarify which aggregation rule is actually used and under what conditions the weighted variant applies.","section":"Section III-B, Eq. (5)-(6), Algorithm 1"},{"comment":"Reference [5], cited for graph-theoretic metrics for classifying Ethereum wallets, is a semi-supervised learning survey by Xiaojin Zhu; it does not appear to support the stated claim. Please verify the intended citation.","section":"Section II"},{"comment":"The two-panel figure with both throughput and latency curves is hard to read, especially the latency y-axis labels. Consider using four separate subfigures or larger fonts.","section":"Figure 4"},{"comment":"The multiclass labels 'Arbitrage, Liquidation, Sandwich, and Non-MEV' are introduced only in the preprocessing section; please introduce them before the multiclass results in Table IV, and define what 'Non-MEV' includes.","section":"Section IV-A"},{"comment":"No link to code or smart-contract artifacts is provided. Making these available would substantially improve reproducibility.","section":"General"},{"comment":"There is a minor typo in the conclusion: 'transaction' history should be 'transaction history'.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"I considered reject, because the SMOTE-before-split protocol invalidates the central accuracy claim as presented. I ultimately chose major_revision because the flaw is fixable in principle by re-executing the experiments with a proper split, SMOTE applied only to training, and fair baselines trained and tested on the same partitions. If the authors cannot provide valid results under that protocol, the paper should not be published. The relationship to the already-accepted ICC version should also be clarified in any journal version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe systems part of this paper is solid and the blockchain benchmark is real. The detection accuracy numbers, however, do not survive contact with the evaluation protocol. Section IV-A applies SMOTE before the 70/30 split, so the test set is dominated by synthetic samples generated from training neighborhoods. That alone invalidates the claimed F1 of 0.99/0.97 as an estimate of real-world performance.\n\nThe genuinely new piece is the application of federated learning to EVM bot detection, with aggregation mediated by a smart contract. The system model is clearly described, and the on-chain benchmark with Hyperledger Caliper gives a credible throughput/latency picture for submitUpdate and aggregateModel. The authors also honestly note the assumption that participants are reliable and cooperative. That is a reasonable starting point.\n\nThe soft spots are concentrated in the detection evaluation, and they are load-bearing. The SMOTE-before-split issue is the main one. Roughly 10,000 synthetic binary samples plus 270 real addresses means a 30% test slice is about 97% synthetic, and SMOTE interpolation makes those test points near neighbors of training points. So the test set is not independent. Additionally, the comparisons against Random Forest/Gradient Boosting/AdaBoost in Tables III and IV quote numbers from [1] trained on the original data, not on the SMOTE-expanded data, so the gain could be mostly augmentation, not the FL framework. The claim that DFL outperforms a centralized model in Table II is also not apples-to-apples, since the centralized model appears to be trained on a subset rather than the full pooled data.\n\nThe blockchain benchmark does not rescue the detection claim; it only validates the infrastructure.\n\nWho is this paper for? Readers working on decentralized FL for blockchain security will find the architecture and benchmark useful as a proof-of-concept. As a bot detector with reliable accuracy, it is not there yet. It deserves a serious referee because the systems work is non-trivial and the flaw is fixable with a re-evaluation on real held-out data, error bars, and fair baselines. I would not cite the accuracy numbers, but I would cite the on-chain aggregation benchmark if I were working on that specific problem.","headline":"Plausible decentralized FL architecture and real on-chain benchmark, but the reported detection accuracy is unsupported because SMOTE is applied before the train/test split.","tokens_in":8646,"tokens_out":2460,"would_cite":false,"duration_ms":23770,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BotDetect reports that a federated learning framework coordinated by smart contracts detects financial bots on EVM blockchains with a binary F1 of 0.99 and multiclass F1 of 0.97, outperforming centralized baseline models while keeping…","keywords":["financial bot detection","federated learning","decentralized federated learning","EVM blockchains","smart contract aggregation","MEV bots","SMOTE","transaction analysis"],"falsifier":"Re-run the binary and multiclass evaluations on the original 270 labeled addresses, or on newly labeled bot addresses, without SMOTE expansion; if F1 drops materially below 0.99/0.97, the headline accuracy is an artifact of the synthetic test set. A second observation: submit a deliberately poisoned weight update from one client and check whether the on-chain average remains clean.","tokens_in":7627,"feed_emoji":"🤖","tokens_out":6169,"duration_ms":64305,"temperature":0.7,"pith_summary":"This paper tries to establish that bot detection on EVM blockchains can be run as a decentralized federated learning process, in which nodes train locally on their own transaction histories and a smart contract aggregates the model updates. The authors report that this design detects financial bots with a binary F1-score of 0.99 and a multiclass F1-score of 0.97, beating centralized machine learning baselines from the dataset they build on. If this result holds, it would give DeFi networks a privacy-preserving way to identify arbitrage, sandwich, and liquidation bots without collecting user data into a central repository. The reader should care because bot exploitation currently costs users over a billion dollars through front-running and similar attacks, and existing rule-based tools cannot adapt to new bot strategies.","feed_headline":"Decentralized learning catches Ethereum bots at 0.99 F1 score","feed_subtitle":"Smart-contract aggregation keeps transaction data private while beating centralized detectors.","key_machinery":"The central mechanism is on-chain federated averaging. Each training client pulls the current global model from the blockchain, trains a fully connected neural network locally for several epochs using SGD, then submits its weight vector to the AG smart contract; once K updates arrive, the contract computes the average (optionally weighted by each client's dataset size) and publishes the new global model for the next round. This replaces a central parameter server with a transparent, auditable smart contract, and the paper's Algorithm 1 specifies the aggregation logic. The same mechanism, together with a permissioned QBFT consensus layer, is what the paper argues makes bot detection both decentralized and private.","core_discovery":"The paper's central claim is that a decentralized federated learning framework, orchestrated by a smart contract, can detect financial bots on EVM blockchains more accurately than centralized training while preserving data privacy. On the Niedermayer et al. dataset of 270 labeled Ethereum accounts, the framework reports 0.99 accuracy and F1 in binary bot/human classification and 0.96 accuracy with 0.97 F1 in multiclass classification across arbitrage, liquidation, sandwich, and non-MEV categories. The authors attribute the improvement over centralized models to data diversity: each client trains on a different private subset, and the aggregated global model generalizes better than a model trained on any single copy. They also report on-chain performance with a QBFT permissioned network, reaching a maximum throughput of 255 transactions per second for the update submission function.","pith_inferences":["The 0.99/0.97 accuracy figures are likely optimistic because the test set was built after SMOTE expansion, so synthetic test points share neighborhoods with synthetic training points; a holdout evaluation on the original 270 labeled addresses would give a more realistic number.","A natural next stress test would be to inject Byzantine or malicious updates; the presented aggregation contract averages every submitted update equally, so it has no defense against a poisoned client.","The framework's assumption that all participating nodes are reliable and cooperative excludes the most adversarial scenario in public blockchains, so the design as presented fits permissioned settings better than open ones."],"forward_implications":["A working BotDetect would let DeFi nodes cooperate on bot detection without ever sharing raw transaction histories, since only model weights cross the network.","The reported accuracy suggests that the behavioral features in the underlying dataset carry enough signal for a small neural network, so the approach could be extended to other EVM chains like BNB Smart Chain and Solana with similar feature pipelines.","Since aggregation happens on-chain, the global model's provenance is public and tamper-evident, which fits the transparency requirements of permissioned consortia.","The measured throughput of roughly 255 TPS for submitting updates indicates the on-chain overhead is not prohibitive for periodic retraining, provided model sizes stay small."],"supporting_citations":[{"why":"Supplies the labeled Ethereum dataset, the extracted behavioral features, and the centralized baselines (Random Forest, Gradient Boosting, AdaBoost) that BotDetect compares against.","marker":"[1]"},{"why":"Provides SMOTE, the oversampling method used to expand the 270-address dataset before federated training, which is the basis of the reported test-set accuracy.","marker":"[18]"},{"why":"Defines the QBFT/BFT consensus mechanism used in the permissioned blockchain setup and performance evaluation.","marker":"[17]"},{"why":"Supplies the decentralized federated learning background that motivates replacing a central aggregator with blockchain coordination.","marker":"[13]"}],"fun_headline_variants":["Federated learning on Ethereum catches bots with 0.99 F1","Privacy-preserving federated learning nails Ethereum bot detection","Smart-contract federated learning beats centralized bot detection","On-chain federated learning spots Ethereum bots with 0.99 F1","Decentralized federated learning detects bots without sharing data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on treating SMOTE-generated synthetic examples as genuine bot transactions, since the reported test accuracy is measured on data that was synthesized, not on newly observed real bots.","fun_headline_variants_meta":{"raw":{"variants":["Federated learning on Ethereum catches bots with 0.99 F1","Privacy-preserving federated learning nails Ethereum bot detection","Smart-contract federated learning beats centralized bot detection","On-chain federated learning spots Ethereum bots with 0.99 F1","Decentralized federated learning detects bots without sharing data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000888,"raw_usage":{"total_tokens":3814,"prompt_tokens":909,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2828}},"tokens_in":525,"tokens_out":2905,"duration_ms":20141,"temperature":1.0,"reasoning_tokens":2828,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:29:07.427123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the binary and multiclass evaluations on the original 270 labeled addresses, or on newly labeled bot addresses, without SMOTE expansion; if F1 drops materially below 0.99/0.97, the headline accuracy is an artifact of the synthetic test set. A second observation: submit a deliberately poisoned weight update from one client and check whether the on-chain average remains clean.","supporting_citations":[{"cited_title":"Detecting financial bots on the ethereum blockchain","cited_arxiv_id":null,"evidence_quote":"Supplies the labeled Ethereum dataset, the extracted behavioral features, and the centralized baselines (Random Forest, Gradient Boosting, AdaBoost) that BotDetect compares against."},{"cited_title":"Decentralized federated learning: A survey and perspective","cited_arxiv_id":null,"evidence_quote":"Supplies the decentralized federated learning background that motivates replacing a central aggregator with blockchain coordination."}],"review_version":1}