{"id":"33b0cccb-97a5-4f54-8b54-927848cf1e7a","arxiv_id":"2505.09313","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper reports that a LightGBM model with two-layer subgraph features detects Sybil airdrop addresses with precision 0.94, recall 0.92, F1 0.93, and AUC 0.98 on Binance BAB data.","lead":"This paper trains a machine learning model on blockchain transaction histories to identify fake addresses, known as Sybil addresses, that create multiple accounts to abuse token airdrops. The method builds a two-layer transaction subgraph around each address, extracts timing, amount, and network features, and reports very high detection accuracy on data from Binance's BAB airdrop.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported >0.9 metrics may reflect learning the undisclosed clustering heuristic used to create ground-truth labels, since label generation and baseline share graph-clustering logic.","rationale":"The stress-test pass identifies the same load-bearing assumption as the reader: the ground-truth sybil labels must be valid and independent of the features used by the model. This assumption is the linchpin of the paper's central performance claim. The manuscript provides no detail on the clustering used to generate the 23,240 confirmed sybil labels, and the feature set described in Section III-A is drawn from the same transaction graph attributes—timing, amounts, and topology—that any graph-based clustering heuristic would use. The comparison with 'Clustering-based Trusta' is especially concerning because if the label-generation clustering resembles Trusta's community detection, the reported superiority over Trusta is not meaningful; the model would simply be a more flexible function approximator of the latent clustering. The absence of a disclosed train/test split compounds this: subgraph feature propagation across two layers means random address-level splits can share cluster members across partitions. The concrete temporal holdout test would settle whether the model generalizes or merely memorizes the labeling heuristic. Because the paper lacks the essential details to rule out circularity, the reader's REJECT verdict is well-founded; no verdict adjustment is needed.","tokens_in":8996,"tokens_out":3067,"duration_ms":33402,"concrete_test":"Run a temporal holdout evaluation: sort the 193,701-address dataset by first-transaction time, train on the first 12 months (approximately Jan 2023–Dec 2023) and test on the remaining period (Jan 2024–May 2024), recomputing all 75 subgraph features using only transactions up to each address's cutoff time. If F1 drops below roughly 0.85 or AUC below 0.95, the Table I metrics are inflated by label leakage from the shared clustering heuristic. If the authors cannot provide label-confirmation dates, use a cluster-based split instead so that all addresses in a connected sybil cluster are kept entirely in either train or test, and report F1 and AUC for that split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—Subgraph-based lightGBM achieves precision 0.9428, recall 0.9182, F1 0.9303, AUC 0.9806—depends entirely on the validity and independence of the ground-truth labels. Section IV-A states that confirmed sybil addresses were identified through 'comprehensive manual analysis and clustering methods,' but the clustering algorithm, its features, and its parameters are never disclosed. The model's features (Section III-A) are lifecycle time features, amount statistics, and two-layer transaction topology. Any clustering performed on the same transaction graph will produce labels strongly correlated with these features; the supervised model may then be learning to imitate the labeling heuristic rather than detect real sybils. This concern is reinforced by the fact that the only non-classical baseline, 'Clustering-based Trusta,' uses community detection on asset transfer graphs and K-means refinement—precisely the kind of method that could have generated the labels. If so, the comparison is circular. Additionally, the paper reports no train/test split or cross-validation for Table I. A random address-level split leaks information through the two-layer subgraph feature propagation because addresses in the same sybil cluster are likely split across train and test, inflating the metrics. Without holding out entire clusters or a time period, the reported metrics are not evidence of generalization to previously unseen sybil campaigns.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a supervised method, Subgraph-based LightGBM, for detecting Sybil addresses in blockchain airdrops. The method builds a two-layer transaction subgraph around each address, extracts time, amount, and network-structure features, and fuses features propagated from the subgraph layers. The authors evaluate on a dataset of 193,701 addresses from the BAB airdrop, of which 23,240 are labeled Sybil, and report that their model achieves precision 0.9428, recall 0.9182, F1 0.9303, and AUC 0.9806, outperforming SVM, decision trees, plain LightGBM, and the Clustering-based Trusta baseline. The paper also includes a feature importance analysis and a brief discussion of limitations.","tokens_in":9216,"tokens_out":5017,"duration_ms":50120,"significance":"If the central result holds, the paper would demonstrate that a supervised, subgraph-feature approach can outperform unsupervised clustering for Sybil detection in long-term airdrop campaigns, with a clear and interpretable feature set. The problem is practically important, and the dataset is real and industry-sourced. However, the evaluation as presented lacks the experimental rigor needed to support the claimed >0.9 performance: the ground-truth labeling procedure is not disclosed, no train/test split or cross-validation is reported, baselines are not feature-matched, and the data cleaning removes a meaningful address population. The paper also does not provide code or data, limiting reproducibility. These issues are central to the paper's contribution and require substantial revision.","major_comments":[{"comment":"The ground-truth Sybil labels are described as obtained through 'comprehensive manual analysis and clustering methods,' but the clustering algorithm, its input features, and its parameters are never disclosed. Because the proposed model's features (Section III-A) include transaction graph structure, amount statistics, and temporal lifecycle features, and because the Clustering-based Trusta baseline (Section IV-B) uses community detection and K-means on asset transfer graphs, the reported metrics in Table I may simply measure how well the supervised model reproduces the labeling heuristic rather than its independent detection capability. The authors must disclose the label-generation method or otherwise demonstrate that the labels are independent of the features used by the model, for example by ablating the clustering-derived features and showing that detection remains strong.","section":"Section IV-A"},{"comment":"No train/test split, cross-validation procedure, or variance estimates are reported for any model in Table I. This is particularly problematic because the two-layer subgraph feature propagation described in Section III-B means that an address's feature vector contains aggregated statistics from its two-hop neighbors; if those neighbors appear in the training set, the model can exploit label information through the graph structure. A random address-level split will place many addresses from the same Sybil cluster in both training and test sets, inflating all metrics. The authors should report evaluation with cluster-level or time-based held-out data and specify the number of Sybil clusters used for splitting.","section":"Section IV-C and Table I"},{"comment":"The comparison in Table I is not feature-matched. SVM, DT, and LightGBM are trained on 'first-order features' from Farrugia et al. [24], while the proposed model uses 75 features that include two-layer propagated subgraph features. The reported improvements over the baselines therefore cannot be attributed to the subgraph propagation and fusion method; they may simply reflect a richer feature set. A fair comparison would feed the same feature set to all classifiers, or ablate the propagation module to isolate its contribution.","section":"Section IV-B"},{"comment":"The preprocessing step that removes addresses with lifecycles exceeding one year (2.6% of the dataset) defines away a population that the model is never evaluated on. The authors justify this by arguing that Sybil attackers exhibit high address abandonment, but the consequence is that the reported precision and recall apply only to short-lived addresses. This limitation is not acknowledged in Section V, and it weakens the general claim that the model identifies Sybil addresses in airdrops broadly. The paper should at least discuss the excluded population and whether the model can be expected to handle long-lived Sybil addresses.","section":"Section IV-A"}],"minor_comments":[{"comment":"Figures 1, 2, and 3 are referenced in the text but are not included in the submitted manuscript; please add the figures or remove the references.","section":"General"},{"comment":"The feature fusion formula is ambiguous: the union operator S is not defined, and the amount-feature count of '2×6×5' is inconsistent with the four statistics (min, max, avg, var) listed in the same section, though Section III-A includes five statistics (adding median). Please clarify the notation and the exact feature count.","section":"Section III-B"},{"comment":"Reference [22] is cited twice with different attributions: 'Ramalingam et al. [22]' and 'Chen et al. [22]', while the introduction says the work is inspired by Chen et al. [1]. In addition, references [13] and [24] are duplicates of the same Farrugia et al. paper.","section":"References"},{"comment":"The description of the Clustering-based Trusta baseline is incomplete: the authors state that 'time features were prioritized during clustering' but do not specify how the baseline was adapted to this dataset, what parameters were used, or how its risk scores were thresholded to produce labels. Without this, the Trusta comparison is not reproducible.","section":"Section IV-B"},{"comment":"The contribution claim of 'the first application of a supervised machine learning method to sybil address identification' is too strong without a systematic survey to establish novelty; either provide evidence or weaken the claim.","section":"Section I"},{"comment":"The paper's own limitation statement that 'the current model relies on features specific to certain datasets' should be reflected in the abstract, which currently claims broad applicability to 'broader blockchain security areas' without this caveat.","section":"Section V"},{"comment":"The claim of 'lower computational overhead compared to traditional methods' is not supported by any runtime or complexity measurements; please provide experimental evidence or remove the claim.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper is written by researchers affiliated with Binance's Risk Department and uses a proprietary internal dataset. This raises two concerns for the editor: (1) the ground-truth label generation may be based on undisclosed internal clustering methods that could compromise the validity of the evaluation, and (2) even with revision, the absence of code and data may make it impossible for readers to independently verify the claimed >0.9 performance. The paper would be more suitable for an applied/industry venue or as a case study, provided the authors can address the experimental rigor issues. The current version is not ready for a security journal with a typical methodological bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a reasonable feature-engineering paper from the Binance risk team, and it is likely the first supervised treatment of airdrop-specific sybil detection. The central claim — subgraph-based LightGBM beats everything with F1 0.93 and AUC 0.98 — is not supportable from the evaluation as written. The problem is not the method; it's the ground truth and the split.\n\nWhat's new and useful: the lifecycle timing features (first gas, first tx, first airdrop participation, last tx) are a sensible addition to the usual amount and degree features. The two-layer subgraph propagation is not novel by itself — it's Chen et al.'s cascading trick — but applying it to airdrop sybil detection is new, and the interpretability analysis (top features, star/chain/tree patterns) is a nice touch. The data is real and sizable: 193,701 addresses, 23,240 labeled sybils, 58M transactions from the BAB event. That alone gives the paper practical value.\n\nWhere it falls down: Section IV-A says the labels came from \"comprehensive manual analysis and clustering methods,\" but the clustering algorithm, its features, and its parameters are never disclosed. Meanwhile, the strongest baseline, Trusta, is itself community detection on asset transfer graphs plus K-means. If the labeling step used anything close to that, the supervised model is partly learning to imitate the labeler. The appeal process adds a human check, which mitigates the concern somewhat, but it doesn't remove it. You need to know the label-generation features to rule out circularity, and they aren't given.\n\nSecond, Table I has no train/test split, no cross-validation, no error bars. With two-layer subgraph features, a random address-level split leaks information: addresses in the same sybil cluster end up on both sides, so the model can peek at neighbors it has already seen. They should hold out whole connected components or a time period. Absent that, the reported metrics do not tell you how the model will generalize to a new sybil campaign.\n\nThird, the baselines are not feature-matched. Plain LightGBM gets first-order features from Farrugia et al.; the proposed model gets two-layer subgraph features. That's a fair comparison of feature extraction, but the paper never ablates to show how much the subgraph features add on top of the same base model.\n\nThese are load-bearing flaws, not cosmetic. If I were refereeing, I'd ask for a rewrite with full disclosure of the labeling procedure, a leakage-aware held-out evaluation, and matched baselines. The idea is sound enough to deserve another round. This is not a desk-reject; it's a major-revision candidate.\n\nWho is it for: people working on sybil detection in airdrops and blockchain security practitioners who care about deployable features. Skip if you need a rigorous benchmark.\n\nRecommendation: send to peer review, but expect heavy revision.","headline":"Plausible first supervised airdrop-sybil method, but the evaluation's undisclosed label generation and missing train/test split keep the >0.9 claims unproven; deserves a careful major-revision round.","tokens_in":9758,"tokens_out":3305,"would_cite":false,"duration_ms":32371,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Subgraph-based LightGBM detects Sybil airdrop addresses with all metrics above 0.9.","keywords":["sybil attack","airdrop fraud","subgraph feature extraction","LightGBM","feature propagation","transaction graph analysis","blockchain security","supervised classification"],"falsifier":"Take a fresh airdrop dataset, label sybils through an independent process that does not use the same transaction features or clustering heuristics, split into train and test, and rerun the 75-feature LightGBM pipeline; if precision, recall, F1, and AUC drop below the reported range, the original numbers reflected label leakage or overfitting to the labeling method rather than general Sybil detection.","tokens_in":8776,"feed_emoji":"🕸️","tokens_out":6165,"duration_ms":56657,"temperature":0.7,"pith_summary":"The paper claims that a supervised model built on transaction-subgraph features can identify Sybil addresses in airdrops more reliably than the unsupervised clustering methods currently used for that task. The model first builds a two-layer transaction subgraph around each address, then extracts time features tied to the Sybil lifecycle (first gas receipt, first transaction, first airdrop participation, last transaction) together with amount and network-structure features, and propagates and fuses those features through the subgraph. On a dataset of 193,701 addresses, including 23,240 confirmed Sybil addresses, the resulting LightGBM classifier reports precision 0.9428, recall 0.9182, F1 0.9303, and AUC 0.9806, outperforming standard classifiers and a clustering-based Sybil detector. If correct, this makes supervised detection practical for long-running airdrops and reduces reliance on manually tuned clustering thresholds.","feed_headline":"Sybil airdrop addresses caught at 94% precision by subgraph model","feed_subtitle":"Temporal lifecycle and network features push every metric above 0.9, beating clustering-based detection.","key_machinery":"The load-bearing object is the two-layer transaction subgraph representation with cascade feature propagation and fusion. For a target address at level 0, the subgraph includes two layers of senders above it and two layers of recipients below it; features computed at each level are aggregated toward the target—amounts by min/max/mean/variance, degrees by summation, time features directly from the target's transactions—to form a 75-dimensional feature vector (7 time, 60 amount, 8 network features). This representation converts an unbounded transaction graph into a fixed-size, interpretable feature set that a gradient-boosted tree model can train on efficiently.","core_discovery":"The central claim is that Sybil addresses are detectable from the shape and timing of their local transaction neighborhoods, not just from fund-flow clusters. For each address the paper builds a two-hop subgraph above and below it, derives 75 features, and learns a LightGBM classifier. The lifecycle time features capture the compressed, coordinated timing of Sybil creation and activity, while the propagated amount and degree features capture star, chain, and tree topologies characteristic of one controller operating many addresses. The paper reports that this approach beats all baselines on every metric, with all values above 0.9, and that its probability scores refine medium- and low-risk cases that clustering-based methods leave ambiguous.","pith_inferences":["If the confirmed-Sybil labels were produced by clustering that resembles the baseline and consumes the same transaction features, the reported metrics may partly measure how well the model reproduces that labeling heuristic; an independent held-out label set would test this.","The lifecycle features are likely transferable to other coordinated-identity problems, such as vote manipulation in DAOs or artificial liquidity in token markets, where the same just-in-time creation pattern appears.","Replacing the hand-set fusion operations (min/max/mean/variance, sum) with learned aggregation, as in graph neural networks, is the natural next step and could push performance beyond the reported AUC while sacrificing some interpretability.","The two-hop window is an implicit assumption about how far Sybil controllers operate from their seed address; checking whether three-hop or cross-chain expansion changes predictions would bound that assumption."],"forward_implications":["Long-term airdrop campaigns, where labeled Sybil addresses accumulate over time, can move from manually tuned clustering to supervised detection with reported accuracy high enough for production use.","The same 75-feature extraction can be applied to other account-based, EVM-compatible chains without retraining the graph construction, only the classifier.","Feature importance analysis shows that lifecycle timing and total balances drive detection, so attackers who randomize timing or maintain larger balances should become harder to catch.","Because the subgraph is limited to two hops, the method avoids expanding the full transaction graph and keeps computational cost manageable on datasets with tens of millions of transactions."],"supporting_citations":[{"why":"Supplies the cascade subgraph feature extraction idea on which the proposed feature propagation and fusion is built.","marker":"[1]"},{"why":"Describes the clustering-based airdrop Sybil identification method that serves as the main baseline and prior unsupervised approach.","marker":"[20]"},{"why":"Supports the effectiveness of graph-based features and motivates the subgraph feature extraction.","marker":"[22]"},{"why":"Supplies the first-order address features used to train the SVM, decision tree, and LightGBM baselines.","marker":"[24]"}],"fun_headline_variants":["Subgraph features sniff out Sybil airdrop fraud","LightGBM on subgraphs flags Sybil airdrop addresses","Feature fusion catches Sybil addresses in airdrops at 0.9+ metrics","Airdrop Sybils exposed by temporal subgraph features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on the ground-truth Sybil labels being valid and independent of the features the model learns; the paper's labeling process is only summarized, and its relationship to the clustering baseline is not fully disclosed.","fun_headline_variants_meta":{"raw":{"variants":["Subgraph features sniff out Sybil airdrop fraud","LightGBM on subgraphs flags Sybil airdrop addresses","Feature fusion catches Sybil addresses in airdrops at 0.9+ metrics","Airdrop Sybils exposed by temporal subgraph features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1684,"prompt_tokens":884,"completion_tokens":800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":724}},"tokens_in":500,"tokens_out":800,"duration_ms":7211,"temperature":1.0,"reasoning_tokens":724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:34:21.371626+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh airdrop dataset, label sybils through an independent process that does not use the same transaction features or clustering heuristics, split into train and test, and rerun the 75-feature LightGBM pipeline; if precision, recall, F1, and AUC drop below the reported range, the original numbers reflected label leakage or overfitting to the labeling method rather than general Sybil detection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the cascade subgraph feature extraction idea on which the proposed feature propagation and fusion is built."},{"cited_title":"Fighting Sybils in Airdrops","cited_arxiv_id":"2209.04603","evidence_quote":"Describes the clustering-based airdrop Sybil identification method that serves as the main baseline and prior unsupervised approach."},{"cited_title":"Ramalingam and V","cited_arxiv_id":null,"evidence_quote":"Supports the effectiveness of graph-based features and motivates the subgraph feature extraction."},{"cited_title":"Farrugia, J","cited_arxiv_id":null,"evidence_quote":"Supplies the first-order address features used to train the SVM, decision tree, and LightGBM baselines."}],"review_version":1}