Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Once leaky contract shortcuts are removed, XGBoost beats Transformer sequence models on Ethereum Sybil bot detection, and transaction order adds little signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:59 UTC pith:QKIBSMRB

load-bearing objection The framework and representation are useful, but the leakage-aware protocol misses the label-source contracts, so the headline tree-vs-sequence result is not yet supported. the 3 major comments →

arxiv 2607.27350 v1 pith:QKIBSMRB submitted 2026-07-29 cs.LG

Modeling Decisions in Blockchain Analytics: A Leakage-Aware Evaluation of Tree-Based vs. Sequential Models

Micha{\l} Bartnicki , Jaros{\l}aw A. Chudziak This is my paper
classification cs.LG
keywords Sybil detectionEthereumblockchain analyticslabel leakageXGBoostTransformertransaction grammarbot detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that reported performance of deep sequence models for blockchain Sybil detection is inflated by label leakage from high-signal contracts, and that once leakage is controlled, a simple tree-based model on aggregate behavioral statistics beats Transformer and BiLSTM models. The authors build a 'Transaction Grammar' encoding each wallet as rhythm, EVM structure, and intent tokens, evaluate under a 'Blind-Spot' protocol that removes shortcut contracts, and find XGBoost achieves higher MCC, macro-F1, and accuracy with much lower latency and energy. The larger claim is that Ethereum actor behavior behaves like a Bag-of-Grammar, not natural language, so preserving transaction order adds little predictive value. This matters because it points to cheaper, real-time Sybil monitoring without specialized GPU infrastructure.

Core claim

The central empirical discovery is that, under a leakage-aware evaluation protocol, XGBoost on Bag-of-Grammar aggregated features achieves MCC 0.7535 versus 0.6602 for the best Transformer and 0.6187 for a hierarchical BiLSTM, while running roughly 100x faster with estimated energy costs over 30x lower. An ablation that fully shuffles the token sequence drops MCC by only 0.0248, leaving the Transformer with over 95% of its predictive power, which the paper interprets as evidence that the model is mainly using token frequency distributions rather than long-range dependencies. The authors conclude that blockchain activity is closer to a Bag-of-Transactions than to a natural-language-like seque

What carries the argument

The Transaction Grammar represents a wallet as an ordered sequence of transactions, each encoded as a triple of tokens: Rhythm (discretized inter-arrival time), Structure (nine EVM trace properties such as call depth and error flags), and Intent (the 4-byte function selector projected to a vocabulary of 100). Sequential models consume the order; tabular baselines consume aggregate counts, Shannon entropy, and Lempel-Ziv complexity from the same grammar. The Blind-Spot protocol removes interactions with high-signal contracts such as OpenSea, Uniswap Router, ENS, and WETH9 to block shortcuts. This shared representation is what makes the inductive-bias comparison clean within the paper's own te

Load-bearing premise

The entire leakage-aware comparison rests on the assumption that the Blind-Spot protocol removed all class-defining contract shortcuts; it removes OpenSea, Uniswap Router, ENS, and WETH9, but leaves in Hop Protocol contracts even though Hop's investigation list generated the labels, so Hop interactions could still leak the class.

What would settle it

Run the benchmark again after removing all transactions touching Hop Protocol contracts (or masking their contract addresses) and recompute MCC for XGBoost and the Flattened Transformer. If XGBoost's lead over the Transformer disappears or reverses, the claim that tabular models win under leakage control is not established.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For Ethereum actor classification, transaction order and timing carry little signal once high-signal contracts are removed; distributional features are enough.
  • Tree-based models on grammar statistics are both more accurate and cheaper, making them deployable in validators, light clients, and wallet extensions.
  • Sequence models trained on transaction histories under standard protocols may have inflated past results; future benchmarks should apply leakage control.
  • Organic wallets show measurably higher structural entropy and grammar complexity than MEV and Sybil bots, while timing entropy does not separate the two bot types.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Blind-Spot protocol does not remove Hop Protocol contracts even though the labels come from Hop's investigation list; if Hop interactions are class-correlated, the reported XGBoost advantage could shrink. This is a testable next step rather than a claim the paper makes.
  • If the Bag-of-Grammar result generalizes, Sybil detection could move to client-side, low-energy inference, which would alter the decentralization calculus for airdrop and governance security.
  • The same leakage-aware protocol could be applied to graph neural networks; the paper names them as promising but does not evaluate them, and graph structure may recover some of the order signal the ablation destroys.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a 'Blind-Spot' protocol and a 'Transaction Grammar' representation for classifying Ethereum wallets as Organic, Sybil, or MEV bot. Labels for Organic/Sybil are taken from Hop Protocol's official airdrop investigation list; MEV bots are identified via Dune Analytics. The authors compare Transformer and BiLSTM sequence models against XGBoost and linear SVM on an 80/20 split. The headline claim is that, under leakage-aware evaluation, XGBoost outperforms Transformer-based sequence models while being faster and more energy-efficient, and that blockchain activity behaves more like a 'Bag-of-Grammar' than a natural-language-like sequence.

Significance. The research question is timely and practically important: if the leakage-aware comparison were valid, the paper would provide a valuable counterpoint to the trend of applying sequence models to blockchain data and would support simpler, deployable classifiers for Sybil detection. The paper contributes a common representation across model families, an entropy/LZ complexity analysis, an order-vs-timing ablation, and a deployment-oriented efficiency comparison. However, the central 'leakage-aware' claim is not currently supported because label information enters feature construction, the reported evaluation split doubles as a validation set, and the label-source contract family is not removed. These issues directly affect the XGBoost-versus-sequence comparison and the 'Bag-of-Grammar' conclusion.

major comments (3)
  1. [§3.1 (Intent vocabulary)] The Intent vocabulary is built by merging the top-k selectors from each class into a global vocabulary of size |V|=100. This is feature selection using ground-truth labels: the function selectors that define model inputs are chosen by class-specific frequency. This is a target-leakage pathway independent of contract removal. Any classifier, especially XGBoost's count-based Bag-of-Grammar, can exploit tokens selected for their class discriminability. To support the leakage-aware claim, the vocabulary must be constructed without labels (e.g., from unlabeled training frequencies), and the whole pipeline should be re-evaluated with selection nested inside cross-validation folds.
  2. [§3.3 and §4.1, Tables 1 and 5] Hyperparameters are 'selected based on validation Matthews Correlation Coefficient' (Table 1), and the classification results in Table 5 are reported on the same 80/20 validation split. There is no held-out test set. Thus the reported MCCs are the result of model selection on the evaluation set, which is optimistic and can differentially favor the model that received more tuning. This invalidates the quantitative comparison. The authors should use nested cross-validation or a separate test set untouched during tuning and report the selected models' performance on that test set.
  3. [§4.2 and Table 2] The Blind-Spot protocol removes Uniswap Router, OpenSea, ENS, and WETH9, but the Organic/Sybil labels come from Hop Protocol's official investigation list, and the dataset is built from Hop airdrop participants. Hop's own bridge/airdrop contracts are not removed. For this population, interactions with Hop contracts are almost a class-defining shortcut: Sybil addresses are precisely the addresses that farmed the Hop airdrop, while Organic addresses are verified Hop users. XGBoost's aggregate token-count features can directly count such interactions, so the reported MCC 0.7535 for XGBoost versus 0.6602 for the Flattened Transformer may reflect this shortcut rather than a genuine advantage of aggregate statistics over sequence modeling. This is the single most load-bearing flaw for the 'leakage-aware' framing. The authors must remove all Hop Protocol addresses/tokens from the transaction hi
minor comments (4)
  1. [Table 2] The table is titled 'Examples of Top Leakage Sources Removed'. The protocol does not state whether the list is exhaustive or how leakage sources were identified. Please give the complete removal list or a reproducible procedure for constructing it.
  2. [§4.6] The energy claim is derived from asymptotic complexity (O(L^2) vs. linear) and described as 'estimated', but no hardware or energy measurement is reported. The 30x figure should be softened or supported by actual measurements.
  3. [§4.3] Kruskal-Wallis p-values and Cliff's delta are reported without confidence intervals or correction for multiple comparisons across the several entropy metrics.
  4. [Throughout] The term 'validation set' is used for the split on which final results are reported. To avoid confusion, use 'test set' for the held-out evaluation split and reserve 'validation' for the split used during model selection.

Circularity Check

2 steps flagged

Intent vocabulary is built from class labels and the Blind-Spot protocol leaves the label-source Hop contracts in place, so the headline 'leakage-aware' comparison is self-referential rather than clean.

specific steps
  1. self definitional [Section 3.1 (Transaction Grammar, Intent token); evaluated in Sections 4.2-4.4]
    "Intent (νt): We encode semantic intent using the 4-byte function selector. To control vocabulary size and reduce class imbalance, we merge the top-k selectors from each class into a global vocabulary of size |V|=100, mapping rare selectors to [UNK]."

    The Intent vocabulary is selected using the ground-truth class labels ('from each class'). Model inputs are therefore defined from the target variable, and the same labeled wallets are then classified (Table 5). Tokens that separate Organic/Sybil/MEV are chosen before training, so the 'leakage-aware' evaluation is not blind: the feature space already encodes class-discriminative selectors. This inflates all reported MCC/F1 values and makes the XGBoost-vs-Transformer comparison a comparison under target-derived features, not a clean test of inductive bias.

  2. other [Section 4.2 (Dataset and Preprocessing) and Table 2 (Blind-Spot removals)]
    "Labels for Organic (verified legitimate human users) and Sybil (coordinated bot rings eliminated for reward farming) were sourced from the Hop Protocol’s official investigation list [10]... our Blind-Spot protocol removes interactions with entities like OpenSea or Uniswap Router, reducing total volume by 21.8% (Table 2)."

    The paper's own label definition makes Hop Protocol the source of the Organic/Sybil distinction, yet Table 2's Blind-Spot removals list only Uniswap Router, OpenSea, ENS Controller, and WETH9; Hop contracts are not removed. Sybil wallets are, by construction, wallets that farmed the Hop airdrop, and Organic wallets are verified Hop users, so Hop interactions are a nearly direct class proxy. Leaving them in the transaction history means the 'leakage-aware evaluation' still contains the label-defining shortcut, and the reported XGBoost advantage may largely reflect this residual Hop signal rather than a genuine superiority of bag-of-grammar features over sequence models.

full rationale

The strongest circularity is in the feature definition: Section 3.1 constructs the Intent vocabulary by merging top-k selectors from each ground-truth class, so label information flows directly into model inputs before any classifier is trained. This is not a full equivalence-to-input case—the models still learn weights and could fail—but it is a concrete target-leakage loop that invalidates the 'leakage-aware' framing and inflates the reported numbers. A second, related gap is that the Blind-Spot protocol does not remove the Hop Protocol contracts that generated the labels, even though the paper defines Sybil/Organic precisely via Hop's investigation list; the remaining Hop interactions are an obvious shortcut. These two issues jointly undermine the central claim that XGBoost beats Transformers 'under leakage-aware evaluation.' The paper's self-citations ([1], [11], [23]) are not load-bearing to the main claim, and there is no imported uniqueness theorem. The result is not literally equivalent to its inputs, so a score of 6 (partial circularity) is appropriate rather than 8-10.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The central empirical claim depends on a hand-picked leakage-removal list, a label-derived feature vocabulary, and unvalidated assumptions about ground-truth labels. These choices are not justified independently and directly affect the reported comparison.

free parameters (6)
  • Intent vocabulary size |V| = 100
    Hand-chosen in Section 3.1; built from class-specific selectors, which introduces label leakage.
  • Temporal bin count = 16
    Discretization of inter-arrival times into 16 bins (Section 3.1).
  • Minimum transaction threshold L_min = 10
    Filter to exclude low-activity wallets (Section 4.2).
  • Top-k selectors per class (k) = not specified
    Used to build the intent vocabulary in Section 3.1; value not reported.
  • Model hyperparameters = Table 1
    Transformer and XGBoost configs selected on validation MCC; no separate test set.
  • Blind-Spot removed contract set = Uniswap Router, OpenSea, ENS, WETH9
    Hand-picked leakage sources in Table 2; excludes the label-source contract Hop.
axioms (4)
  • domain assumption Labels from Hop Protocol investigation and Dune Analytics are accurate ground truth.
    Section 4.2 relies on external label sources without independent validation.
  • domain assumption Removing OpenSea, Uniswap Router, ENS, and WETH9 removes all significant leakage.
    Table 2 lists removed contracts, but the Hop contract is not removed despite being the label source.
  • domain assumption The nine EVM trace properties fully capture execution structure.
    Section 3.1 defines Structure tokens from nine unspecified EVM properties; no evidence they are sufficient.
  • ad hoc to paper Building the intent vocabulary from class labels does not leak label information.
    Section 3.1: 'merge the top-k selectors from each class into a global vocabulary'—explicitly uses labels in feature construction.
invented entities (2)
  • Transaction Grammar no independent evidence
    purpose: Represent wallets as sequences of Rhythm, Structure, and Intent tokens for model input.
    New representation without external validation that it captures behavior better than alternatives.
  • Blind-Spot protocol no independent evidence
    purpose: Remove high-signal contract interactions to reduce label leakage.
    Protocol is defined ad hoc (Table 2); no principled criterion for which contracts to remove, and it omits the label-source contract.

pith-pipeline@v1.3.0-daily-deepseek · 7538 in / 13493 out tokens · 129297 ms · 2026-08-01T08:59:00.284578+00:00 · methodology

0 comments
read the original abstract

Sybil bots are Ethereum actors that imitate legitimate users to extract airdrop rewards or influence governance. Recent Sybil detection methods increasingly use deep learning and treat blockchain activity as a quasi-linguistic sequence. However, complex sequence models are computationally expensive for real-time monitoring, and their reported performance may be inflated by label leakage from high-signal smart contracts. We ask whether and how organic users, Sybil bots, and MEV bots differ in the structural complexity of their transaction histories; whether sequential models outperform tree-based tabular models once leakage is reduced; whether transaction order or timing provides the stronger behavioral signal; and whether the resulting models are practical for low-latency deployment. Our approach to leakage-aware Sybil bot detection consists of a Blind-Spot protocol and a Transaction Grammar representation of wallet behavior. The former eliminates shortcuts associated with high-signal contracts, whereas the latter models wallets using rhythm, EVM execution structure, and intent. We evaluate this approach on Ethereum actor classification by comparing Transformer and BiLSTM sequence models against XGBoost and SVM baselines. We contribute a framework for leakage-aware Ethereum actor classification and a Transaction Grammar representation of wallet behavior. Our results demonstrate that, under leakage-aware evaluation, XGBoost outperforms Transformer-based sequence models while providing lower latency and estimated energy use.

Figures

Figures reproduced from arXiv: 2607.27350 by Jaros{\l}aw A. Chudziak, Micha{\l} Bartnicki.

Figure 1
Figure 1. Figure 1: End-to-end workflow for leakage-aware Ethereum actor classification [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Impact of sequence ablations on Transformer performance (MCC). 4.6 Deployment Viability and Efficiency [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Efficiency vs. Accuracy. XGBoost (top left) achieves the most optimal trade-off on the efficiency-accuracy curve. 5 Discussion and Future Work Our results show a mismatch between attention-based inductive biases and blockchain transactions. Because transaction histories lack the compositional semantics of natural language, global sequencing provides less signal than lo￾cal frequency and execution patterns.… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    In: Proc

    Bańka, F., Chudziak, J.: Applying informer for option pricing: A transformer- based approach. In: Proc. ICAART. p. 1270–1277. SCITEPRESS - Science and Technology Publications (2025). https://doi.org/10.5220/0013320900003890

    work page doi:10.5220/0013320900003890 2025

  2. [2]

    IEEE Trans

    Borisov, V., et al.: Deep neural networks and tabular data: A sur- vey. IEEE Trans. Neural Netw. Learn. Syst.35, 7499–7519 (2024). https://doi.org/10.1109/tnnls.2022.3229161

    arXiv 2024

  3. [3]

    In: Proc

    Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proc. 22nd KDD. pp. 785–794 (2016). https://doi.org/10.1145/2939672.2939785

    arXiv 2016

  4. [4]

    Fan, S., et al.: Al-spsd: Anti-leakage smart ponzi schemes de- tection in blockchain. Inf. Process. Manage.58, 102587 (2021). https://doi.org/10.1016/j.ipm.2021.102587

    arXiv 2021

  5. [5]

    Geirhos, R., et al.: Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020). https://doi.org/10.1038/s42256-020-00257-z

    doi:10.1038/s42256-020-00257-z 2020

  6. [6]

    https://doi.org/10.48550/arXiv.2106.11959

    Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data (2021). https://doi.org/10.48550/arXiv.2106.11959

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.11959 2021

  7. [7]

    36th NeurIPS (2022)

    Grinsztajn, L., Oyallon, E., Varoquaux, G.: Why do tree-based models still out- perform deep learning on typical tabular data? In: Proc. 36th NeurIPS (2022). https://doi.org/10.48550/arXiv.2207.08815

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.08815 2022

  8. [8]

    Neural Comput.9, 1735–1780 (1997)

    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput.9, 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    doi:10.1162/neco.1997.9.8.1735 1997

  9. [9]

    In: NeurIPS 2022 1st Table Rep

    Hollmann, N., et al.: TabPFN: A transformer that solves small tabular classifi- cation problems in a second. In: NeurIPS 2022 1st Table Rep. Workshop (2022), https://openreview.net/forum?id=eu9fVjVasr4

    2022

  10. [10]

    https://github.com/hop- protocol/hop-airdrop (2022)

    Hop Protocol: Hop Protocol: Hop Airdrop distribution. https://github.com/hop- protocol/hop-airdrop (2022)

    2022

  11. [11]

    In: Proc

    Kwiatkowski, J., Chudziak, J.A.: On evaluating loss functions for stock ranking: An empirical analysis with transformer model. In: Proc. 34th ACM CIKM. pp. 4910–4914 (2025). https://doi.org/10.1145/3746252.3760812

    arXiv 2025

  12. [12]

    In: 2025 IEEE ICBC

    Liu, Q., et al.: Detecting sybil addresses in blockchain airdrops. In: 2025 IEEE ICBC. pp. 1–7 (2025). https://doi.org/10.1109/ICBC64466.2025.11185061

    arXiv 2025

  13. [13]

    Liu, T., et al.: The role of transformer models in advancing blockchain tech- nology: A systematic survey. Eng. Appl. Artif. Intell.163, 112968 (2026). https://doi.org/10.1016/j.engappai.2025.112968

    arXiv 2026

  14. [14]

    IEEE Trans

    Materwala, H., et al.: Maximal extractable value in decentralized finance: Taxon- omy, detection, and mitigation. IEEE Trans. Serv. Comput.18, 4386–4407 (2025). https://doi.org/10.1109/tsc.2025.3620604

    arXiv 2025

  15. [15]

    https://doi.org/10.48550/arXiv.2312.02752

    Messias, J., Yaish, A., Livshits, B.: Airdrops: Giving money away is harder than it seems (2023). https://doi.org/10.48550/arXiv.2312.02752

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.02752 2023

  16. [16]

    In: Companion Proc

    Niedermayer, T., Saggese, P., Haslhofer, B.: Detecting financial bots on the ethereum blockchain. In: Companion Proc. Web Conf. 2024. pp. 1742–1751 (2024). https://doi.org/10.1145/3589335.3651959

    arXiv 2024

  17. [17]

    Machine Learning on Blockchain Data: A Systematic Mapping Study

    Palaiokrassas, G., Bouraga, S., Tassiulas, L.: Machine learning on blockchain data: A systematic mapping study (2024). https://doi.org/10.48550/arXiv.2403.17081

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2403.17081 2024

  18. [18]

    Schwartz, R., Dodge, J., Smith, N.A., Etzioni, O.: Green ai. Commun. ACM 63(12), 54–63 (Nov 2020). https://doi.org/10.1145/3381831

    doi:10.1145/3381831 2020

  19. [19]

    Shwartz-Ziv, R., Armon, A.: Tabular data: Deep learning is not all you need. Inf. Fusion81, 84–90 (2022). https://doi.org/10.1016/j.inffus.2021.11.011

    doi:10.1016/j.inffus.2021.11.011 2022

  20. [20]

    https://doi.org/10.48550/arXiv.2106.01342

    Somepalli, G., et al.: Saint: Improved neural networks for tab- ular data via row attention and contrastive pre-training (2021). https://doi.org/10.48550/arXiv.2106.01342

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.01342 2021

  21. [21]

    Strubell, E., Ganesh, A., Mccallum, A.: Energy and policy considerations for deep learning in nlp. pp. 3645–3650 (01 2019). https://doi.org/10.18653/v1/P19-1355

    doi:10.18653/v1/p19-1355 2019

  22. [22]

    Sun, J., et al.: Ethereum fraud detection via joint transaction language model and graph representation learning. Inf. Fusion120, 103074 (2025). https://doi.org/10.1016/j.inffus.2025.103074

    arXiv 2025

  23. [23]

    https://doi.org/10.1109/ICTAI66417.2025.00147

    Tokajuk, A., Chudziak, J.: Partial multivariate transformer as a tool for cryptocurrencies time series prediction (2025). https://doi.org/10.1109/ICTAI66417.2025.00147

    arXiv 2025

  24. [24]

    Vaswani, A., et al.: Attention is all you need. In: Adv. Neural Inf. Process. Syst. vol. 30 (2017). https://doi.org/10.48550/arXiv.1706.03762

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.03762 2017

  25. [25]

    In: Proc

    Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proc. AAAI. vol. 35, pp. 11106–11115 (2021). https://doi.org/10.1609/aaai.v35i12.17325

    doi:10.1609/aaai.v35i12.17325 2021