{"id":"49b7e8ef-3c23-43c2-b627-c7f9631c5695","arxiv_id":"2504.17684","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Simple single-feature perturbations, such as shifted timestamps and altered values, sharply reduce the accuracy of Random Forest, Decision Tree, and KNN Ethereum phishing detectors, with adversarial training reported to restore performance.","lead":"This paper tests whether simple changes to transaction data, like shifted timestamps or changed amounts, can fool machine learning classifiers that spot Ethereum phishing scams. It reports severe accuracy drops and claims adversarial retraining restores performance, but its experimental reporting is internally inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Dataset-1 counts equal the full dataset despite the stated 80/20 split, so the core vulnerability numbers are not held-out test results.","rationale":"The reader's weakest_assumption focuses on FGSM applied to non-differentiable models, which is a valid concern but one that mainly affects the gradient-based attack branch and the adversarial-training defense. The more fundamental issue is that all Dataset-1 tables report counts summing to 23,472, the full dataset size, contradicting the stated 80/20 split. This means the core empirical evidence for the central claim—rule-based timestamp, value, and address manipulations degrading RF, DT, and KNN—is not measured on a held-out test set. That is load-bearing because the paper's central claim is empirical: simple single-feature perturbations substantially degrade detection. Without a valid held-out evaluation, those numbers do not establish the claim. The reader's rationale does list this as one of several issues, so there is partial agreement, but the weakest_assumption field identifies FGSM as the primary weak point. I would place the full-data evaluation first because it undermines a wider set of results, including the very manipulations named in the strongest_claim. The verdict should remain REJECT; the disclosed inconsistencies are severe enough that no amount of rewriting of the FGSM section alone would make the empirical claim trustworthy.","tokens_in":25865,"tokens_out":4663,"duration_ms":47858,"concrete_test":"Ask the authors to rerun the Dataset-1 experiments with the stated 80/20 split and report per-class counts on the test fold only. If the test fold contains roughly 4,698 instances (not 23,472), recompute the baseline and adversarial rows of Tables 2, 3, 4, and 5 on that test fold. If the accuracy drops differ materially from the published tables, the central vulnerability claim is not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 promises an 80/20 train/test split, yet every Dataset-1 table reports class counts summing to 23,472, the full dataset size. For example, Table 2's baseline row lists 15,989 benign + 7,483 phishing = 23,472, and Table 10 repeats exactly these counts after adversarial training. Tables 3, 4, and 5 likewise all sum to 23,472. An 80/20 split of 23,472 would yield about 4,694 test instances, so these tables cannot be reporting metrics on a held-out test fold. If accuracy and degradation are computed on the training data, the headline vulnerability numbers for timestamp, value, and address manipulations reflect memorization or training-set fit, not generalization to unseen transactions. This is the most load-bearing concern because the paper's central claim rests on these rule-based single-feature perturbations; even if the FGSM non-differentiability issue were resolved, the empirical foundation for the claim would still be missing without a proper held-out evaluation. The FGSM problem remains real, but the full-data issue is broader and affects all Dataset-1 results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates the robustness of Random Forest, Decision Tree, and K-Nearest Neighbors classifiers for Ethereum phishing detection against simple single-feature manipulations (timestamp, value, sender/receiver address) and against FGSM-style gradient-based attacks. Using two Ethereum transaction datasets, it reports accuracy, precision, recall, and F1 degradations under these perturbations, then proposes adversarial training and feature-selection guidance as mitigation strategies. The central claims are that simple manipulations substantially degrade detection performance, that model vulnerability varies across algorithms, and that adversarial training restores most lost performance.","tokens_in":26115,"tokens_out":3509,"duration_ms":34430,"significance":"If the empirical claims were sound, the paper would offer a useful cautionary result for practitioners building Ethereum phishing detectors, showing that even single-feature changes can bypass non-neural classifiers, and that adversarial training can partially mitigate this. The simple perturbation model is realistic and the comparison across three classic classifiers is informative. The paper also openly builds on a prior WISA workshop version and uses public datasets, which supports reproducibility in principle. However, as submitted, the experimental reporting has multiple serious internal inconsistencies that undermine the validity of the headline numbers, so the significance currently rests on claims that are not adequately supported.","major_comments":[{"comment":"The paper states an 80/20 train/test split, but every Dataset-1 table reports class counts that sum to 23,472, the full dataset size. For example, Table 2's baseline row gives 15,989 benign + 7,483 phishing = 23,472, and Table 10 repeats these exact counts. An 80/20 split would yield about 4,694 test instances, so these tables cannot be reporting metrics on a held-out test fold. If the reported accuracy and degradation numbers are computed on the training set or on the full dataset, the headline vulnerability and recovery results do not demonstrate generalization to unseen transactions. This is load-bearing because the paper's central claim rests on these measurements.","section":"Section 4.2 and Tables 2–5, 10, 11"},{"comment":"FGSM is defined for models whose loss is differentiable with respect to the input, but the paper applies it to Random Forest, Decision Tree, and K-Nearest Neighbors, which are piecewise-constant functions with gradients that are zero or undefined almost everywhere. The manuscript presents the standard FGSM formula in Section 4.2 but never specifies a surrogate gradient, a smoothing procedure, or a gradient-estimation method. Without such a mechanism, the FGSM-based attack results (Section 5.3, Figures 2–4) and the adversarial-training claims that rely on FGSM (Section 6.1 and Table 12) are not supported. The paper must either describe and justify an approximate-gradient approach or replace FGSM with a well-defined gradient-free attack.","section":"Section 4.2 (FGSM) and Section 5.3"},{"comment":"Section 5.4 reports that manipulating address features (from_address and to_address) caused phishing and scamming precision and recall to drop to zero for all models (Table 7), and the same section states that temporal feature manipulation nearly nullified phishing detection (Table 9). Section 8, however, claims that address features are 'less affected' and are the 'most resistant' to adversarial attacks, and that temporal features are 'resistant' and have smaller impact than value manipulations. These statements directly contradict the reported results. This internal contradiction means the discussion and the feature-selection recommendations at the end of Section 8 are not grounded in the paper's own experimental findings.","section":"Section 5.4 versus Section 8"},{"comment":"The adversarial-training evaluation is circular in design: the training augmentation is generated with the same perturbation families (timestamp shifts, value changes, FGSM) that are then used for the attack evaluation, and the post-retraining results in Table 12 are measured on those same perturbation types. The near-total recovery of accuracy to 0.98–0.99 under this protocol may reflect the model having been trained on the exact test distribution rather than genuine robustness to unseen manipulations. To support the claimed robustness benefit, the authors need to evaluate on attack types, magnitudes, or feature subsets that were not included in the adversarial training augmentation.","section":"Sections 6–7 and Table 12"},{"comment":"There are internal reporting inconsistencies that make several metrics unreliable. In Table 3, the 'Uniform' row for RF reports accuracy 0.69 with counts of 23,353 benign and 119 phishing; the per-class precision and recall shown (benign precision 0.96, phishing precision 0.68, phishing recall 0.02) are not consistent with the confusion-matrix sizes implied by those counts. In Table 6, the 'Instance Counts Before and After Attacks' row for KNN shows 'Fake ICO' as 2,189, equal to the Scamming count, whereas Section 4.1.1 states that the second dataset contains only one Fake ICO transaction; Table 12 also lists Fake ICO count as 2,189. Such inconsistencies indicate that the tabulated numbers cannot be taken at face value without clarification.","section":"Tables 3 and 6"}],"minor_comments":[{"comment":"The paper contains numerous typos and duplicated phrases, e.g., 'Despite advancements in machine learning and Despite significant progress' in Section 2.4, and 'probability' / 'propability' inconsistencies. A careful copyedit is needed before resubmission.","section":"Throughout"},{"comment":"Figure 7 is described only as 'Combined model accuracy across 2-6 figures' and appears to be a collage of bar charts; the reader cannot discern which subpanel corresponds to which attack or metric. Each subpanel needs a clear caption and consistent axis labeling.","section":"Figure 7"},{"comment":"Table 12 has merged cells that obscure which values are shared across models; for instance, the row 'Accuracy Before and After Adversarial Training' shows 0.99/0.98 once but is meant to apply to all three models. The table should be restructured so that each model's pre/post accuracy and misclassification rates are explicit.","section":"Table 12"},{"comment":"The text says 'DT's recall for phishing was especially affected, falling to just 0.01%.' The corresponding Table 3 does not clearly show a 0.01% value; the reported recall for phishing under uniform value manipulation is 0.02. The numerical discrepancy between the text and the table should be reconciled.","section":"Section 5.1"},{"comment":"A few references are cited inconsistently: [47] is listed as Szegedy et al. in the reference list but is also attributed to Goodfellow et al. in the introduction when discussing FGSM. The authors should verify that citation numbers match the intended works.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript is a revised and extended version of a WISA 2024 workshop paper, and the authors have made an effort to address a broader set of attacks and defenses. However, the load-bearing experimental issues (full-dataset counts instead of held-out splits, undefined FGSM gradients for tree/KNN models, and the internal contradiction between Section 5.4 and Section 8) are not local fixes; they require re-running the experiments with a proper evaluation protocol and a well-defined attack methodology. Given the journal context, I cannot recommend acceptance or minor revision in its current form. If the authors redo the experiments, a revised submission could be considered anew, but I would not hold this version to the standard of a major revision because the empirical foundation for the central claim is currently absent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper tackles a real problem: simple, realistic feature edits can beat Ethereum phishing detectors, and that threat deserves attention. Second, as submitted the numbers do not hold up. The Dataset-1 tables all sum to 23,472, the full dataset, even though Section 4.2 promises an 80/20 split. So the headline degradation figures are probably not held-out test results. That alone sinks the current version.\n\nWhat is genuinely useful is the attack taxonomy: timestamp shifts, uniform value changes, address replacement, plus the RF/DT/KNN comparison across two datasets. If the results were sound, they would be a practical warning for fraud-detection engineers. The paper also honestly acknowledges its WISA 2024 predecessor, and the FGSM and adversarial-training additions are the right direction.\n\nNow the soft spots, in order. The split problem is the most serious. Every count in Tables 2-5, 10, and 11 sums to 23,472; an 80/20 split would leave about 4,694 test points. Either the split was not used or the tables report training-set metrics, and either way the central vulnerability claim is unsupported. Second, FGSM is applied to RF, DT, and KNN, which are piecewise-constant functions; the gradient is zero or undefined almost everywhere. The paper states the standard FGSM formula but never specifies a surrogate gradient or smoothing, so the FGSM results and the adversarial-training recovery built on them do not follow. Third, Section 8 says address and temporal features are the most resistant to attack, while Section 5.4 says address manipulation drops phishing/scamming detection to zero and timestamp manipulation nearly nullifies it. A discussion contradicting its own results is a red flag. Fourth, adversarial training is evaluated on the same perturbation families used to generate the augmented training set, so the near-total recovery in Table 12 may be contamination rather than robustness. Minor issues include swapped or inconsistent per-class metrics in Tables 3 and 6.\n\nI would not accept this in current form. It deserves a serious referee, but the referee's message should be clear: redo the evaluation on a true held-out split, either specify a gradient surrogate or drop the FGSM story, reconcile the discussion with the results, and test adversarial training against attacks not seen in training. The idea is worth fixing; the current evidence is not.","headline":"A real problem and a sensible attack taxonomy, but as submitted the experiments do not support the claims: the split is contradicted by the table counts, FGSM is used on non-differentiable models without a surrogate, and the discussion contradicts the results.","tokens_in":26606,"tokens_out":3029,"would_cite":false,"duration_ms":31311,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Single-feature edits break Ethereum phishing detectors trained on Random Forest, Decision Tree, and KNN.","keywords":["Ethereum phishing detection","adversarial perturbations","single-feature attacks","Fast Gradient Sign Method","Random Forest","Decision Tree","K-Nearest Neighbors","adversarial training"],"falsifier":"Run the gradient-based attack on the same three classifiers using the models' own loss gradient rather than an external surrogate. For piecewise-constant classifiers like RF, DT, and KNN, $\\nabla_x J(\\theta,x,y)$ is zero almost everywhere, so the FGSM update $x' = x + \\epsilon \\cdot \\text{sign}(\\nabla_x J(\\theta,x,y))$ would leave inputs unchanged and accuracy should stay at baseline. If the reported accuracy collapses are reproduced only when a surrogate gradient supplies nonzero directions, then the single-feature FGSM claim depends on an attack the paper does not specify.","tokens_in":25696,"feed_emoji":"🛡️","tokens_out":10515,"duration_ms":87114,"temperature":0.7,"pith_summary":"The paper argues that Ethereum phishing and scam detection built on Random Forest, Decision Tree, and K-Nearest Neighbors can be evaded by changing one transaction feature at a time, such as shifting the timestamp, altering the value, or swapping the sender or receiver address. Under these simple manipulations, phishing and scamming recall collapses for all three classifiers in several scenarios, with Decision Tree and KNN often becoming almost useless. The paper also claims that retraining on adversarially modified examples restores most of the lost accuracy, and that combining temporal features with address or financial features makes models harder to fool. If true, this matters because a real attacker can often edit only one field of a transaction, so detectors that rely on these standard classifiers need explicit defenses rather than assuming clean data.","feed_headline":"Simple tweaks defeat Ethereum phishing detectors","feed_subtitle":"Changing one transaction field drops some detector accuracies below 0.70; retraining on modified examples recovers them.","key_machinery":"The paper's core instrument is a set of single-feature perturbation operators applied to two Ethereum transaction datasets, together with the Fast Gradient Sign Method (FGSM), a one-step attack that perturbs features in the direction that most increases the loss. FGSM computes $x' = x + \\epsilon \\cdot \\text{sign}(\\nabla_x J(\\theta, x, y))$ and is applied to transaction value, gas, gas_price, and block_timestamp. The perturbation operators are timestamp shifts, uniform and proportional value changes, and sender or receiver address replacement; these define the attack surface being tested. The same operators double as the adversarial training recipe: retraining on the modified examples is the defense whose accuracy restoration is measured. The machinery carries the argument because every result is a before-and-after comparison of accuracy, precision, recall, and F1 under one of these operators.","core_discovery":"The central discovery is that the evaluated ML detectors are not robust to single-feature perturbations that stay within realistic transaction ranges. On the first dataset, a one-day timestamp shift drops KNN accuracy from 0.94 to 0.83, a 1% uniform value change drops RF and DT accuracy from about 0.99 and 0.98 to 0.69, and replacing the recipient address in all transactions drops RF phishing recall from 1.00 to 0.51. On the second dataset, targeted rule-based changes push phishing detection accuracy for RF and DT from 0.96 to 0.01 and scamming accuracy from roughly 0.98 to 0.14, while gradient-based FGSM attacks reduce DT overall accuracy to about 0.09 and KNN's phishing and scamming metrics to zero. The paper further claims that adversarial training on timestamp- and value-perturbed samples returns RF, DT, and KNN to roughly their original accuracy. The authors interpret the inconsistency across algorithms as an opportunity: model choice and feature composition can be tuned for resilience.","pith_inferences":["The paper does not test whether an attacker needs a surrogate gradient to run FGSM on tree-based or nearest-neighbor classifiers; if one is required, the gradient-based results overstate how easily a real attacker can produce those perturbations.","A testable extension is to apply the same timestamp, value, and address perturbations to graph-based phishing detectors, since the paper's address results suggest those models may be even more sensitive to identity changes.","The uniform value perturbation's catastrophic effect on RF and DT may be partly an artifact of how categorical or scaled features are encoded; re-running with standardized feature encoders would isolate whether the vulnerability is in the model or the representation.","The paper's adversarial training used only timestamp and value perturbations, so an immediate extension is to include address replacement in the training mix and measure whether the restored accuracy generalizes to other single-feature edits."],"forward_implications":["A phishing detector that relies on transaction value and timestamp alone can be evaded by an attacker who edits a single field, so production detectors should monitor feature consistency.","Overall accuracy can stay high while phishing recall collapses, so reporting class-level precision and recall is necessary to see whether a detector is still safe.","KNN's phishing and scamming metrics drop to zero under FGSM in the paper's experiments, suggesting KNN should not be the sole classifier in adversarial settings unless retrained.","Adversarial training on timestamp and value perturbations returns RF and DT to near-baseline accuracy, giving a concrete defense that does not require changing the model family.","Combining temporal features with address or financial features reduces vulnerability, making feature selection a defense lever independent of training."],"supporting_citations":[{"why":"Supplies the binary phishing/benign Ethereum dataset of 23,472 transactions used for timestamp, value, and address manipulations.","marker":"[26]"},{"why":"Supplies the multi-class Ethereum dataset of 71,250 transactions used for targeted and untargeted attacks.","marker":"[9]"},{"why":"Provides the FGSM perturbation formula that Section 4.2 uses to craft gradient-based adversarial examples.","marker":"[47]"},{"why":"Cited alongside [47] as the FGSM source for generating adversarial examples.","marker":"[7]"},{"why":"Cited in Section 6.1 as the method used to craft FGSM adversarial examples for the retraining phase.","marker":"[16]"},{"why":"Supplies the adversarial-training defense used to retrain models against timestamp and value attacks.","marker":"[28]"}],"fun_headline_variants":["Single feature shift dismantles Ethereum phishing detectors","One-field tweak drops detector accuracy to near zero","Minimal edits expose Ethereum phishing ML blind spots","Adversarial training restores Ethereum scam detection","Ethereum fraud AI fails under one-feature attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the Fast Gradient Sign Method can be meaningfully applied to Random Forest, Decision Tree, and K-Nearest Neighbors, even though those models do not provide the gradient of the error with respect to input features that the method assumes.","fun_headline_variants_meta":{"raw":{"variants":["Single feature shift dismantles Ethereum phishing detectors","One-field tweak drops detector accuracy to near zero","Minimal edits expose Ethereum phishing ML blind spots","Adversarial training restores Ethereum scam detection","Ethereum fraud AI fails under one-feature attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1314,"prompt_tokens":845,"completion_tokens":469,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":461,"tokens_out":469,"duration_ms":4873,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:33:30.398851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the gradient-based attack on the same three classifiers using the models' own loss gradient rather than an external surrogate. For piecewise-constant classifiers like RF, DT, and KNN, $\\nabla_x J(\\theta,x,y)$ is zero almost everywhere, so the FGSM update $x' = x + \\epsilon \\cdot \\text{sign}(\\nabla_x J(\\theta,x,y))$ would leave inputs unchanged and accuracy should stay at baseline. If the reported accuracy collapses are reproduced only when a surrogate gradient supplies nonzero directions, then the single-feature FGSM claim depends on an attack the paper does not specify.","supporting_citations":[{"cited_title":"Sanjalawe, and Selvakumar Manickam","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-class Ethereum dataset of 71,250 transactions used for targeted and untargeted attacks."}],"review_version":1}