{"id":"8347f19e-4ad1-471b-8b40-6c2605a3a3e6","arxiv_id":"2504.17953","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A graph neural network with handcrafted temporal features identifies more Ethereum phishing addresses than one using raw transaction fields, though the phishing-class F1 is 0.28, not the reported 0.95.","lead":"This paper compares machine learning models that detect phishing on the Ethereum blockchain, pitting raw transaction fields against hand-built behavioral and network features. The behavioral features detect more phishing addresses, but the paper's headline numbers overstate how well the model catches actual phishing addresses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported train/test split is transaction-level, not address-level, so GCN label propagation and aggregate implicit features leak test information; the central implicit-vs-explicit comparison is unsupported as reported.","rationale":"The paper's stated contribution is that a small set of implicit behavioral and graph features outperforms explicit transactional features for address-level phishing detection. For that claim to hold, the evaluation must measure generalization to unseen addresses. Section 4.1 claims an address-level split, but the numbers in Table 3 match transaction counts, and the graph construction in Section 3.4 uses addresses as nodes and transactions as edges, so a transaction-level split is the most natural reading of the reported procedure. Under that reading, nodes are shared across splits. This is not a minor reporting issue: GCN propagates labels through the graph, and implicit features are aggregate statistics over all of an address's transactions, so test information enters training in two independent ways. The reader's weakest assumption identifies exactly this mechanism, and I agree that it is the most load-bearing issue. I also note the metric inconsistencies in Tables 5 and 6 (e.g., a class cannot have precision 0.76 and recall 0.00; the weighted averages do not follow from the per-class rows and listed supports), which independently undermine the quantitative support but are secondary to the leakage concern. Section 6 concedes low phishing recall and non-benchmarked comparisons, but the central contribution is still stated in terms of the inflated metrics. Since the core empirical evidence is invalid as reported, the reader's REJECT verdict is unchanged; if the address-disjoint re-run reproduces the qualitative pattern, the paper should be re-evaluated.","tokens_in":20112,"tokens_out":11519,"duration_ms":115806,"concrete_test":"Re-run both experiments with an 80/20 split of unique Ethereum addresses rather than transaction rows: assign every transaction of a chosen address to that address's split, recompute the implicit features of Section 3.3 using only training-set transactions, train and test the GCN on the resulting address-disjoint graph, and report phishing-class precision, recall, and F1 for the explicit and implicit feature sets. If the implicit-feature phishing F1 (reported as 0.28) drops toward the explicit-feature result, the claimed advantage is an artifact of label leakage through shared addresses; if it remains substantially higher, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison between implicit and explicit features is not supported because the reported evaluation appears to split transactions, not addresses, creating label leakage. Section 4.1 says the data were divided by an 80/20 split of phishing and benign addresses, but Table 3 lists 'Phishing Nodes' 537,492/134,373 and 'Benign Nodes' 2,149,968/537,492, which are exactly the 80/20 splits of the 671,865 phishing and 2,687,460 benign transactions given in Section 4.1, not of the 7,915 phishing addresses or of any address-level node set consistent with Section 3.4 (nodes are addresses, edges are transactions). If the split is at the row/transaction level, the same Ethereum address appears as a node in both training and test. A GCN trained with node-level labels can propagate a known phishing label from a node's training edges to its test edges, and the implicit features of Section 3.3 are aggregated over each address's full transaction history, so test-period behavior is already present in the training features. Both effects inflate the reported metrics and, more importantly, can inflate the apparent advantage of implicit over explicit features. The explicit-feature table (Section 5.2) is also arithmetically inconsistent (phishing precision 0.76 with recall 0.00 is impossible), which further weakens confidence, but the split problem alone is sufficient to invalidate the central claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a GCN-based phishing detection system for Ethereum transactions, comparing two feature sets: explicit transactional features (value, gas, timestamp) and implicit graph/behavioral features (transaction counts, inter-transaction timing, weekend ratios, etc.). The authors report that implicit features substantially outperform explicit features, with accuracy rising from 0.79 to 0.95 and phishing recall from 0.00 to 0.33, and they further claim an F1-score of 0.95 for their approach. The paper also includes a Random Forest feature-importance analysis, a weighted loss for class imbalance, and a qualitative comparison with prior work.","tokens_in":20348,"tokens_out":5509,"duration_ms":52740,"significance":"If the empirical claims were supported, the paper would make a moderate contribution: a systematic per-feature-type comparison on a large Ethereum dataset, a compact set of temporal/behavioral implicit features, and a weighted-loss treatment of class imbalance. However, the evaluation as presented contains load-bearing inconsistencies: the reported split appears to be at the transaction level rather than the address level, which can leak test information into training; the explicit-feature metrics in Table 5 are arithmetically impossible; and the headline F1 of 0.95 in Table 1 and Section 5.4 is a weighted average that hides the much lower phishing-class scores. The paper does not provide code, full hyperparameters, or exact class weights, so the experiments are not reproducible as described. The central claim that implicit features outperform explicit features is therefore not established by the current evidence.","major_comments":[{"comment":"The reported data split appears to be transaction-level rather than address-level, which leaks test information into training. Section 3.4 defines nodes as Ethereum addresses, and Section 3.1 states the dataset contains 7,915 unique phishing addresses, yet Table 3's 'Phishing Nodes' and 'Benign Nodes' totals (671,865 and 2,687,460) exactly match the phishing and benign transaction counts given in Section 4.1. If the 80/20 split was applied to transactions, the same address appears as a node in both training and test data; because the GCN is transductive and the implicit features in Section 3.3 are aggregated over each address's full transaction history, the reported implicit-feature gains in Section 5.3 can be inflated by label propagation and by features that already contain test-period information. The central comparison between implicit and explicit features is therefore unsupported by the evaluation as described.","section":"4.1 and Table 3"},{"comment":"The phishing row of Table 5 reports precision 0.76, recall 0.00, and F1 0.01. With zero true positives, precision is 0 or undefined rather than 0.76, and the F1-score would be 0 rather than 0.01. These internally inconsistent numbers mean the explicit-feature baseline is not reliably characterized, which directly undermines the comparison in Section 5.4 and the claim that implicit features outperform explicit features.","section":"5.2, Table 5"},{"comment":"Table 1 and Section 5.4 report F1 0.95, precision 0.96, and recall 0.95 for 'Our work,' but the phishing-class results in Table 6 are precision 0.25, recall 0.33, and F1 0.28. The high reported numbers are weighted averages dominated by the benign class. Presenting them as phishing detection performance overstates the method's ability to detect phishing addresses and makes the comparison with prior work in Table 1 misleading, since the prior rows report per-class metrics.","section":"5.4 and Table 1"},{"comment":"The Random Forest feature-importance analysis is described without any mention of a train/test split, and Section 3.3 applies Min-Max scaling using dataset-wide minima and maxima. If feature selection and scaling use the full dataset, then test-period information contributes to the construction of the training features. Combined with the transaction-level split identified above, this is an additional channel through which the reported improvement of implicit features could be inflated.","section":"4.2 and 5.1"},{"comment":"The benign-node 'Avg time between tx' value of -5.16×10^4 seconds is impossible for a time interval. This indicates a computation or reporting error in the feature statistics used to justify the proposed feature set, and it reduces confidence in the surrounding statistical analysis.","section":"5.1, Table 4"}],"minor_comments":[{"comment":"The phrase 'how feature affect model resilience' should read 'how features affect model resilience.'","section":"Abstract"},{"comment":"The dataset composition is described as 'around 2M benign and around 600K phishing transactions'; exact counts are given later in Section 4.1, so the approximate language should be replaced with precise numbers throughout.","section":"3.1"},{"comment":"Reference [20] lists 'Author Name Kimber' as the author, which is a placeholder that should be corrected before submission.","section":"References"},{"comment":"The 'Edge Attributes' item says edges carry a phishing/benign label, but the task is node classification; this ambiguity should be clarified, since it is unclear whether the graph edges are labeled separately from node labels.","section":"3.4"},{"comment":"The GCN hyperparameters (number of layers, hidden dimensions, dropout rate, learning rate, epochs) and the exact class weights and decision threshold are not reported. Without these details the experiments are not reproducible.","section":"4.2"},{"comment":"Min-Max scaling should be fitted on the training set only and then applied to the test set; using dataset-wide minima and maxima as described leaks test-set statistics into training.","section":"3.3"}],"recommendation":"reject","confidential_remarks":"This appears to be an early arXiv version with placeholder reference [20] and a not-yet-polished presentation. The more serious issue is that the experimental evaluation, as written, cannot support the central claim: the split is described at the transaction level despite the graph being defined at the address level, and the explicit-feature metrics are internally inconsistent. These are load-bearing problems that would require re-running the experiments with an address-level split, re-checking all reported numbers, and rewriting the comparison with prior work. I would encourage the authors to address these issues and resubmit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. The central comparison is not supported as reported because the data split appears to be transaction-level, not address-level, and the headline F1 is the weighted average, not phishing-class performance. The actual phishing-class F1 is 0.28, not 0.95.\n\nWhat is genuinely new: a set of 16 handcrafted temporal and behavioral features (inter-transaction intervals, weekend ratios, hour-of-day statistics) and a two-phase evaluation that runs a GCN on explicit features and then on implicit features. That is a concrete, useful contribution. The qualitative conclusion—that implicit features beat raw transaction fields—is plausible and consistent with the prior work they cite.\n\nThe problems are in the evaluation. Section 4.1 describes an 80/20 split of phishing and benign addresses, but Table 3 lists 'Phishing Nodes' and 'Benign Nodes' whose counts exactly match the transaction counts. Since the GCN operates on address nodes and the implicit features are aggregated over each address's full transaction history, a transaction-level split means the same address appears in both training and test. That lets the GCN propagate labels across the split, and it embeds test-period behavior into training features. This is not a nitpick; it invalidates the comparison as reported. The same section also reports RF feature importance, apparently run before the split, which adds another leakage channel.\n\nThe reporting is also misleading. The abstract, contributions, Table 1, and Section 5.4 present weighted-average metrics (F1 0.95, recall 0.95) as the model's phishing detection performance. Table 6 shows the phishing-class F1 is 0.28 with recall 0.33. Table 5 contains an internal impossibility: phishing precision 0.76 with recall 0.00. A model that predicts no phishing positives cannot have precision 0.76.\n\nTo their credit, the authors are honest in Section 6: they acknowledge low recall, say the comparisons are not direct benchmarks, and state that phishing detection remains hard. The qualitative finding may survive a corrected analysis. But the quantitative claims as stated do not.\n\nWho is this for? People building Ethereum phishing features might borrow the feature list. Nobody should rely on the reported numbers without a re-run.\n\nMy recommendation: send it to peer review only if the venue is willing to demand a re-analysis. The flaws are identifiable and fixable, but the paper as written does not support its central claim. If a referee can check that the split is address-level and the metrics are per-class, the comparison may be worth having. Otherwise, reject.","headline":"The comparison is unsupported as reported: the split looks transaction-level, and the headline F1 is the weighted average, not phishing-class performance.","tokens_in":20951,"tokens_out":4225,"would_cite":false,"duration_ms":40364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a small set of implicit behavioral and graph features outperforms raw transactional fields for graph-convolutional phishing detection in Ethereum, lifting overall accuracy from 0.79 to 0.95 while per-class phishing…","keywords":["Ethereum","phishing detection","graph convolutional network","implicit features","transaction analysis","class imbalance","feature engineering","blockchain security"],"falsifier":"Count unique Ethereum addresses in the graph and retrain the implicit-feature GCN under a strict address-level 80/20 split. The paper's own data table reports 'phishing nodes' and 'benign nodes' equal to the transaction counts, so the graph may treat transactions as nodes; if a correct address-level split leaves phishing-class recall near 0.33 rather than near the reported F1, the current result is partly label leakage.","tokens_in":19870,"feed_emoji":"🎣","tokens_out":12287,"duration_ms":110571,"temperature":0.7,"pith_summary":"Phishing detection on Ethereum usually feeds raw transaction fields—value, gas price, timestamp—into a classifier. This paper tries to establish that a graph convolutional network trained on a small set of implicit behavioral features (how many transactions an address sends or receives, total value moved, average gas, transaction-hour statistics, inter-transaction intervals, weekend-activity ratios) detects phishing addresses far better than the same model trained on raw explicit fields, with accuracy and weighted F1 rising from about 0.79 and 0.70 to 0.95. The authors isolate the two feature classes in separate experiments and show the implicit set wins despite being smaller, supporting the general claim that feature design matters more than feature quantity in adversarial settings. The paper also applies a weighted loss to handle the heavy class imbalance, and it reports per-class phishing precision, recall, and F1 (0.25, 0.33, 0.28) rather than only weighted averages. If the claim holds, it would shift detection practice toward temporal and relational fingerprints that are harder for attackers to fake than surface-level transaction attributes.","feed_headline":"Graph behavior features beat raw transaction fields for phishing","feed_subtitle":"A graph-convolution model reaches 0.95 overall accuracy, but phishing-class recall stays at 0.33","key_machinery":"The machinery is a directed transaction graph $G=(V,E)$ whose nodes are Ethereum addresses and whose edges are transactions, combined with a GCN whose propagation rule is $H^{(l+1)} = \\sigma(D^{-1/2} A D^{-1/2} H^{(l)} W^{(l)})$, where $A$ is the adjacency matrix, $D$ the degree matrix, and $W^{(l)}$ the learned weights. The node feature matrix is built from the 16 implicit behavioral features, including total value sent and received, average gas for sends and receives, mean and standard deviation of transaction hour, average, minimum, and maximum time between transactions, transaction duration, and weekend-activity ratios. A weighted cross-entropy loss $L = -\\sum_i w_{y_i} \\log p(y_i)$ counteracts the class imbalance (phishing is about 7.6% of transactions). Random Forest feature-importance ranking and statistical comparisons justify which implicit features are kept. The GCN's neighborhood aggregation lets an address's classification depend on the behavior of the addresses it transacts with.","core_discovery":"The paper's central discovery, stated on its own terms, is that implicit graph-derived features are the primary carriers of phishing signal for a GCN on Ethereum's transaction network. In the explicit-feature experiment (value, gas used, gas price, timestamp), the model collapses to essentially always predicting benign, with phishing recall of 0.00 and F1 of 0.01. In the implicit-feature experiment (16 handcrafted temporal and behavioral statistics per address), the same architecture reaches accuracy 0.95 and weighted F1 0.95, with per-class phishing precision 0.25, recall 0.33, and F1 0.28. Feature-importance analysis ranks total value sent, average gas used when sending, and inter-transaction timing as the strongest signals. The paper interprets this as evidence that relational and temporal behavior, not raw transaction values, distinguishes phishers, and that a small focused feature set can outperform a larger set of basic features—an advantage it attributes to adversarial settings where attackers can mimic obvious transaction attributes but not statistical behavioral patterns.","pith_inferences":["An implication the paper leaves implicit is that the same behavioral features could transfer to other automated Ethereum fraud, such as token spamming or Ponzi-style payout patterns, which also produce burst activity and irregular inter-transaction timing.","The reported weighted-average F1 of 0.95 is carried almost entirely by the benign majority class; a deployable system would need a cost-sensitive threshold tuned on per-class phishing precision-recall, which the paper itself flags as future work.","Because the implicit features are computed from address histories, they could be updated incrementally as new blocks arrive, offering a streaming detection path that avoids re-training the full graph each time."],"forward_implications":["Feature design should be studied separately for each feature class; the paper shows that a 16-feature implicit set outperforms raw explicit fields for a GCN on the same data.","Detection systems should favor temporal statistics such as inter-transaction intervals, hour-of-day variability, and weekend ratios over raw value and gas fields when constructing node features.","The weighted loss function is a low-cost way to recover minority-class recall without extra hyperparameter tuning, which matters in networks where phishing is under 10% of activity.","Graph-based models that propagate labels through address connectivity can surface suspicious addresses that explicit, per-transaction classifiers miss."],"supporting_citations":[{"why":"Baseline implicit graph-based phishing detection method; supplies the relational-feature approach this work builds from.","marker":"[8]"},{"why":"Representative explicit-feature ML method with high F1 on small data; the key contrast to the implicit-feature result.","marker":"[18]"},{"why":"Publicly compiled list of verified phishing addresses used to label the dataset.","marker":"[20]"},{"why":"Predecessor using temporal transaction aggregation, motivating the inter-transaction timing features.","marker":"[21]"},{"why":"Implicit graph contrast-network baseline whose embeddings and dataset scale this work compares against.","marker":"[22]"},{"why":"Network-embedding phishing detection that the implicit graph-feature rationale extends.","marker":"[43]"},{"why":"Attributed ego-graph embedding baseline supporting the use of network context for phishing detection.","marker":"[44]"},{"why":"Edge-featured graph attention baseline that uses combined explicit and implicit features and is compared in the results.","marker":"[49]"}],"fun_headline_variants":["Graph features beat raw transaction fields for phishing detection","Implicit graph features outperform explicit transaction fields for phishing","Ethereum phishing: graph features improve accuracy, but recall stays at 0.33","GCN with implicit features: 95% accuracy, but phishing recall only 0.33"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 80/20 split separates whole Ethereum addresses, so no address's transactions appear in both training and testing sets; if the split is actually at transaction level, the GCN can silently transfer labels between an address's training and test transactions, inflating the reported performance.","fun_headline_variants_meta":{"raw":{"variants":["Graph features beat raw transaction fields for phishing detection","Implicit graph features outperform explicit transaction fields for phishing","Ethereum phishing: graph features improve accuracy, but recall stays at 0.33","GCN with implicit features: 95% accuracy, but phishing recall only 0.33"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00106,"raw_usage":{"total_tokens":4413,"prompt_tokens":881,"completion_tokens":3532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":3453}},"tokens_in":497,"tokens_out":3532,"duration_ms":24202,"temperature":1.0,"reasoning_tokens":3453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:28:39.345149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count unique Ethereum addresses in the graph and retrain the implicit-feature GCN under a strict address-level 80/20 split. The paper's own data table reports 'phishing nodes' and 'benign nodes' equal to the transaction counts, so the graph may treat transactions as nodes; if a correct address-level split leaves phishing-class recall near 0.33 rather than near the reported F1, the current result is partly label leakage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline implicit graph-based phishing detection method; supplies the relational-feature approach this work builds from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Representative explicit-feature ML method with high F1 on small data; the key contrast to the implicit-feature result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Publicly compiled list of verified phishing addresses used to label the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Predecessor using temporal transaction aggregation, motivating the inter-transaction timing features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Implicit graph contrast-network baseline whose embeddings and dataset scale this work compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Network-embedding phishing detection that the implicit graph-feature rationale extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Attributed ego-graph embedding baseline supporting the use of network context for phishing detection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Edge-featured graph attention baseline that uses combined explicit and implicit features and is compared in the results."}],"review_version":1}