{"id":"bcae7c3e-37e2-4057-a4af-b01c0278a9c6","arxiv_id":"2607.27350","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper reports that XGBoost beats Transformer and BiLSTM models for Ethereum actor classification after masking certain high-signal contracts, and that sequence order adds little signal.","lead":"Ethereum's airdrop system is under attack from Sybil bots that fake human activity. This paper tests whether expensive sequence models are needed to catch them, and claims cheap tree models work better—but its leakage controls have holes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Blind-Spot protocol does not remove the label-source Hop contracts, so the 'leakage-aware' comparison is contaminated; XGBoost's margin may collapse under a truly leakage-free protocol.","rationale":"The reader's weakest assumption—that Hop contracts are not removed despite being the label source—is exactly the load-bearing point. This is not a matter of consensus or preference: if true, the evaluation is not leakage-aware, and the paper's strongest claim (XGBoost superiority under leakage-aware conditions) is unsupported. I also noted the class-label-based intent vocabulary as a secondary leak, but the Hop contract issue is the most fundamental because it directly contaminates the label-generation mechanism. A concrete test is to verify the presence of Hop interactions and rerun the comparison without them. The reader's REJECT verdict remains appropriate; no adjustment is needed.","tokens_in":7792,"tokens_out":4311,"duration_ms":45149,"concrete_test":"First, check whether any transaction to Hop Protocol contracts (e.g., the Hop airdrop/bridge/router addresses from the hop-airdrop repository) remains in the post-filtering dataset. If they do, re-run the full benchmark with these contracts added to the Blind-Spot removal list and with the Intent vocabulary built from training-set labels only. Compare XGBoost vs. Transformer MCC under this corrected protocol; if the gap narrows by more than ~0.05 or the ordering changes, the headline claim is compromised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that under leakage-aware evaluation XGBoost outperforms Transformers—depends entirely on the Blind-Spot protocol actually removing high-signal shortcuts. Section 4.2 states that Organic and Sybil labels come from Hop Protocol's official investigation list, but Table 2's Blind-Spot removals include only Uniswap Router, OpenSea, ENS, and WETH9. Hop Protocol contracts are not listed as removed. Because the dataset is built from Hop airdrop participants, transactions to/from Hop's airdrop and bridge contracts are a near-perfect class proxy: Sybil wallets are precisely the addresses that farmed the Hop airdrop, while Organic wallets are verified Hop users. XGBoost's Bag-of-Grammar features can directly count these interactions, so its reported MCC 0.7535 versus the Transformer's 0.6602 may reflect exploitation of the label-generating contract rather than a genuine advantage of aggregate statistics over sequence modeling. The paper also constructs the Intent vocabulary (Section 3.1) using class labels to select top-k selectors, which is an additional target-leakage pathway. If Hop interactions and class-selected tokens were masked, the performance gap—and the 'Bag-of-Grammar' conclusion drawn from it—could vanish. This is the single most load-bearing flaw because the headline result is explicitly framed as leakage-aware.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'Blind-Spot' protocol and a 'Transaction Grammar' representation for classifying Ethereum wallets as Organic, Sybil, or MEV bot. Labels for Organic/Sybil are taken from Hop Protocol's official airdrop investigation list; MEV bots are identified via Dune Analytics. The authors compare Transformer and BiLSTM sequence models against XGBoost and linear SVM on an 80/20 split. The headline claim is that, under leakage-aware evaluation, XGBoost outperforms Transformer-based sequence models while being faster and more energy-efficient, and that blockchain activity behaves more like a 'Bag-of-Grammar' than a natural-language-like sequence.","tokens_in":8180,"tokens_out":5924,"duration_ms":64603,"significance":"The research question is timely and practically important: if the leakage-aware comparison were valid, the paper would provide a valuable counterpoint to the trend of applying sequence models to blockchain data and would support simpler, deployable classifiers for Sybil detection. The paper contributes a common representation across model families, an entropy/LZ complexity analysis, an order-vs-timing ablation, and a deployment-oriented efficiency comparison. However, the central 'leakage-aware' claim is not currently supported because label information enters feature construction, the reported evaluation split doubles as a validation set, and the label-source contract family is not removed. These issues directly affect the XGBoost-versus-sequence comparison and the 'Bag-of-Grammar' conclusion.","major_comments":[{"comment":"The Intent vocabulary is built by merging the top-k selectors from each class into a global vocabulary of size |V|=100. This is feature selection using ground-truth labels: the function selectors that define model inputs are chosen by class-specific frequency. This is a target-leakage pathway independent of contract removal. Any classifier, especially XGBoost's count-based Bag-of-Grammar, can exploit tokens selected for their class discriminability. To support the leakage-aware claim, the vocabulary must be constructed without labels (e.g., from unlabeled training frequencies), and the whole pipeline should be re-evaluated with selection nested inside cross-validation folds.","section":"§3.1 (Intent vocabulary)"},{"comment":"Hyperparameters are 'selected based on validation Matthews Correlation Coefficient' (Table 1), and the classification results in Table 5 are reported on the same 80/20 validation split. There is no held-out test set. Thus the reported MCCs are the result of model selection on the evaluation set, which is optimistic and can differentially favor the model that received more tuning. This invalidates the quantitative comparison. The authors should use nested cross-validation or a separate test set untouched during tuning and report the selected models' performance on that test set.","section":"§3.3 and §4.1, Tables 1 and 5"},{"comment":"The Blind-Spot protocol removes Uniswap Router, OpenSea, ENS, and WETH9, but the Organic/Sybil labels come from Hop Protocol's official investigation list, and the dataset is built from Hop airdrop participants. Hop's own bridge/airdrop contracts are not removed. For this population, interactions with Hop contracts are almost a class-defining shortcut: Sybil addresses are precisely the addresses that farmed the Hop airdrop, while Organic addresses are verified Hop users. XGBoost's aggregate token-count features can directly count such interactions, so the reported MCC 0.7535 for XGBoost versus 0.6602 for the Flattened Transformer may reflect this shortcut rather than a genuine advantage of aggregate statistics over sequence modeling. This is the single most load-bearing flaw for the 'leakage-aware' framing. The authors must remove all Hop Protocol addresses/tokens from the transaction hi","section":"§4.2 and Table 2"}],"minor_comments":[{"comment":"The table is titled 'Examples of Top Leakage Sources Removed'. The protocol does not state whether the list is exhaustive or how leakage sources were identified. Please give the complete removal list or a reproducible procedure for constructing it.","section":"Table 2"},{"comment":"The energy claim is derived from asymptotic complexity (O(L^2) vs. linear) and described as 'estimated', but no hardware or energy measurement is reported. The 30x figure should be softened or supported by actual measurements.","section":"§4.6"},{"comment":"Kruskal-Wallis p-values and Cliff's delta are reported without confidence intervals or correction for multiple comparisons across the several entropy metrics.","section":"§4.3"},{"comment":"The term 'validation set' is used for the split on which final results are reported. To avoid confusion, use 'test set' for the held-out evaluation split and reserve 'validation' for the split used during model selection.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"I considered reject because the leakage issues are severe, but I chose major_revision because the research questions are interesting and the flaws are in principle fixable by re-analysis: remove Hop contracts, construct features without labels, and use a proper held-out test set. If the authors cannot re-run under these conditions, the paper should not claim to be leakage-aware."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. It does real work: a new Ethereum dataset, a Transaction Grammar representation, and a side-by-side comparison of tree and sequence models under a designed leakage-control protocol. But the headline result — that under leakage-aware evaluation XGBoost beats Transformers — does not hold as written, because the leakage-aware protocol leaks.\n\nThe Transaction Grammar idea is genuinely useful. Encoding rhythm, EVM execution structure, and intent into tokens, then feeding the same representation to both sequential and tabular models, is a clean experimental design. The Blind-Spot protocol is a good concept, and the ablation showing the Transformer loses almost nothing when the full sequence is shuffled is a nice, honest result. That part supports the Bag-of-Grammar view.\n\nThe problems are load-bearing. Section 4.2 says labels come from Hop Protocol's investigation list, but Table 2 removes only Uniswap Router, OpenSea, ENS, and WETH9. Hop contracts remain in the transaction histories. Since the dataset consists of Hop airdrop participants, and Sybil labels are precisely the wallets that farmed the airdrop, Hop interactions are nearly a class indicator. XGBoost's count-based features can exploit that directly. The 0.75 vs 0.66 MCC gap could easily be an artifact of this leftover leakage rather than a genuine tabular advantage.\n\nSecond, the intent vocabulary is built using class labels. Section 3.1 merges the top-k selectors from each class into a global vocabulary. That is target leakage in feature construction. It affects all models, so the ranking may survive with corrections, but the absolute numbers and the claim about 'authentic behavioral signals' don't stand.\n\nThird, hyperparameters are selected on the validation set and results are reported on that same validation split. There is no test set. That's a smaller sin because all models share the protocol, but the reported numbers are optimistic, and the paper's confidence in them is overstated.\n\nThe efficiency comparison also mixes A100 and CPU runs, but that's a minor point; the qualitative ordering is likely fine.\n\nBottom line: solid framework, weak execution on leakage control. The fix is straightforward — remove all Hop-related contracts, build the vocabulary class-agnostically, and hold out a test set. This deserves a serious referee because the topic is important and the framework is worth iterating on. I'd recommend sending it out with an expectation of major revision.","headline":"The framework and representation are useful, but the leakage-aware protocol misses the label-source contracts, so the headline tree-vs-sequence result is not yet supported.","tokens_in":8607,"tokens_out":3864,"would_cite":false,"duration_ms":33770,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Once leaky contract shortcuts are removed, XGBoost beats Transformer sequence models on Ethereum Sybil bot detection, and transaction order adds little signal.","keywords":["Sybil detection","Ethereum","blockchain analytics","label leakage","XGBoost","Transformer","transaction grammar","bot detection"],"falsifier":"Run the benchmark again after removing all transactions touching Hop Protocol contracts (or masking their contract addresses) and recompute MCC for XGBoost and the Flattened Transformer. If XGBoost's lead over the Transformer disappears or reverses, the claim that tabular models win under leakage control is not established.","tokens_in":7704,"feed_emoji":"🤖","tokens_out":5098,"duration_ms":45717,"temperature":0.7,"pith_summary":"The paper tries to establish that reported performance of deep sequence models for blockchain Sybil detection is inflated by label leakage from high-signal contracts, and that once leakage is controlled, a simple tree-based model on aggregate behavioral statistics beats Transformer and BiLSTM models. The authors build a 'Transaction Grammar' encoding each wallet as rhythm, EVM structure, and intent tokens, evaluate under a 'Blind-Spot' protocol that removes shortcut contracts, and find XGBoost achieves higher MCC, macro-F1, and accuracy with much lower latency and energy. The larger claim is that Ethereum actor behavior behaves like a Bag-of-Grammar, not natural language, so preserving transaction order adds little predictive value. This matters because it points to cheaper, real-time Sybil monitoring without specialized GPU infrastructure.","feed_headline":"Leak-aware test: XGBoost tops transformers on Ethereum bots","feed_subtitle":"A simple tree model on grammar statistics is more accurate than sequence networks and far cheaper to run.","key_machinery":"The Transaction Grammar represents a wallet as an ordered sequence of transactions, each encoded as a triple of tokens: Rhythm (discretized inter-arrival time), Structure (nine EVM trace properties such as call depth and error flags), and Intent (the 4-byte function selector projected to a vocabulary of 100). Sequential models consume the order; tabular baselines consume aggregate counts, Shannon entropy, and Lempel-Ziv complexity from the same grammar. The Blind-Spot protocol removes interactions with high-signal contracts such as OpenSea, Uniswap Router, ENS, and WETH9 to block shortcuts. This shared representation is what makes the inductive-bias comparison clean within the paper's own te","core_discovery":"The central empirical discovery is that, under a leakage-aware evaluation protocol, XGBoost on Bag-of-Grammar aggregated features achieves MCC 0.7535 versus 0.6602 for the best Transformer and 0.6187 for a hierarchical BiLSTM, while running roughly 100x faster with estimated energy costs over 30x lower. An ablation that fully shuffles the token sequence drops MCC by only 0.0248, leaving the Transformer with over 95% of its predictive power, which the paper interprets as evidence that the model is mainly using token frequency distributions rather than long-range dependencies. The authors conclude that blockchain activity is closer to a Bag-of-Transactions than to a natural-language-like seque","pith_inferences":["The Blind-Spot protocol does not remove Hop Protocol contracts even though the labels come from Hop's investigation list; if Hop interactions are class-correlated, the reported XGBoost advantage could shrink. This is a testable next step rather than a claim the paper makes.","If the Bag-of-Grammar result generalizes, Sybil detection could move to client-side, low-energy inference, which would alter the decentralization calculus for airdrop and governance security.","The same leakage-aware protocol could be applied to graph neural networks; the paper names them as promising but does not evaluate them, and graph structure may recover some of the order signal the ablation destroys."],"forward_implications":["For Ethereum actor classification, transaction order and timing carry little signal once high-signal contracts are removed; distributional features are enough.","Tree-based models on grammar statistics are both more accurate and cheaper, making them deployable in validators, light clients, and wallet extensions.","Sequence models trained on transaction histories under standard protocols may have inflated past results; future benchmarks should apply leakage control.","Organic wallets show measurably higher structural entropy and grammar complexity than MEV and Sybil bots, while timing entropy does not separate the two bot types."],"fun_headline_variants":["XGBoost beats transformers in leak-aware Ethereum bot test","Tree model beats deep learning for Ethereum Sybil detection","Leakage-aware eval: XGBoost wins over transformers on Ethereum bots","Simple tree beats big transformers for Ethereum bot spotting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire leakage-aware comparison rests on the assumption that the Blind-Spot protocol removed all class-defining contract shortcuts; it removes OpenSea, Uniswap Router, ENS, and WETH9, but leaves in Hop Protocol contracts even though Hop's investigation list generated the labels, so Hop interactions could still leak the class.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost beats transformers in leak-aware Ethereum bot test","Tree model beats deep learning for Ethereum Sybil detection","Leakage-aware eval: XGBoost wins over transformers on Ethereum bots","Simple tree beats big transformers for Ethereum bot spotting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1173,"prompt_tokens":776,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":520,"tokens_out":397,"duration_ms":4761,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:59:00.284578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the benchmark again after removing all transactions touching Hop Protocol contracts (or masking their contract addresses) and recompute MCC for XGBoost and the Flattened Transformer. If XGBoost's lead over the Transformer disappears or reverses, the claim that tabular models win under leakage control is not established.","supporting_citations":[],"review_version":1}