{"id":"381cacf6-f140-40fb-a700-bcd14df42b1e","arxiv_id":"2505.04094","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A rule-based detector finds three Solana-specific phishing transaction patterns, producing a first academic dataset and estimates of eight thousand cases and about $1.1 million in losses.","lead":"This paper identifies three phishing transaction patterns that exploit Solana's unique account and transaction design, and builds a rule-based detector that flags 8,058 such transactions. It also releases the first academic dataset for Solana phishing and estimates victim losses near $1.1 million.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8,058 count and 93.96% precision rest on unvalidated assignment of TDPA transactions to phishing status; no independent sample of the positives was checked, so the headline figures may be materially overstated.","rationale":"The paper makes a genuinely useful contribution by defining three Solana-specific phishing transaction types and providing clear, reproducible detection rules. The taxonomy and the qualitative findings about temporal evolution, token targets, and gang structures are plausible and well supported by the detected data. The central claim, however, is explicitly quantitative: 8,058 SolPhish instances and roughly 93.96% precision. That claim depends on treating the presence of a seed-labeled phishing account as sufficient evidence that a flagged transaction is a true positive. The authors manually validated the small set of TDNA false positives but did not report any manual validation of the 8,058 TDPA positives. Because the seed list itself comes from community reports and Solscan labels, and because no dataset link is provided, the count and precision are not independently checkable. This is a real soft spot, but it is addressable through dataset release and a modest manual annotation study, so it does not invalidate the paper's contribution; it does mean the quantitative headline should be treated as conditional until verified. The reader identified essentially the same weakest assumption, and the recommended conditional verdict remains appropriate.","tokens_in":17786,"tokens_out":6549,"duration_ms":70364,"concrete_test":"Release SolPhishDataset with transaction hashes and randomly sample 384 of the 8,058 TDPA positives, plus all 513 flagged-but-excluded transactions. Have two independent annotators label each using full transaction context, program logs, and available victim reports, with disagreement adjudicated. Recompute precision and the 8,058 total using the verified positive rate; if the adjusted precision falls below roughly 90% or the corrected count deviates by more than 10%, the headline detection and loss figures must be revised. As a secondary check, apply the rules to a random sample of transactions from unlabeled Solana accounts and manually validate any hits to estimate recall loss from seed-list incompleteness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section IV-B3, SolPhishHunter flags 8,571 suspicious transactions in TDPA and 5 in TDNA. The authors then 'label and classify' the TDPA hits: 8,058 are called phishing, 130 mutual transfers between labeled phishers, and 383 transfers from labeled phishers to other entities. The 8,058 positives are accepted as true positives without any independent per-transaction verification; they are positive because the transaction involves a known phishing account as beneficiary, or an account whose authority is transferred, which is exactly the seed-derived ground truth used to assemble TDPA. The 5 TDNA false positives receive expert review, but the 8,058 TDPA positives do not. Consequently the claimed precision (8058/8576 = 93.96%) and the count '8,058 instances' are not measured against an independent standard; they assume that every non-mutual/non-money-laundering flagged transaction from a seed account is real phishing. If even a small fraction of these are legitimate transfers, token movements, or misclassified flows, both the count and the $1.1M loss figure change materially. The lack of a dataset link prevents external checking, and the seed-list completeness (Chainabuse/Solscan, Section IV-B1) bounds recall in a way the paper never quantifies.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines three Solana-specific phishing transaction types (STMT, AAT, ISA), proposes a rule-based detector called SolPhishHunter, and evaluates it on transactions of 169 seed phishing accounts (TDPA) and 200 high-asset normal accounts (TDNA). It reports detecting 8,058 SolPhish transactions with 93.96% precision, estimates nearly $1.1 million in victim losses, analyzes temporal and gang-level patterns, and announces a public dataset, SolPhishDataset.","tokens_in":18043,"tokens_out":4481,"duration_ms":47148,"significance":"If the evaluation were independently validated, this would be a useful first systematic study of Solana transaction-level phishing: the three proposed types are concrete, checkable on-chain, and the empirical observations about AAT's outsized financial impact and the existence of phishing gangs are informative for the security community. The paper is also commendable for giving explicit detection rules and for manually reviewing the five TDNA false positives. However, the headline metrics (8,058 instances and 93.96% precision) are not yet supported because the positive labels come from the same seed account list used to build TDPA, the loss figures use current token prices rather than prices at attack time, and the detection thresholds are not sensitivity-tested.","major_comments":[{"comment":"The precision estimate is circular with respect to the ground-truth construction. TDPA is built from 169 accounts already labeled as phishing on Chainabuse/Solscan, and the 8,058 'true positives' are classified by checking whether the beneficiary or authority recipient of a flagged transaction is one of these same labeled accounts (or a labeled phishing program). No independent per-transaction verification is reported for the TDPA positives, in contrast to the five TDNA positives that receive expert review. Consequently, the 93.96% precision and the 8,058 count are not measured against an independent standard; they are partly an artifact of the seed list. The authors should manually review a random sample of the positives (stratified by type), report per-type precision with confidence intervals, and state how seed-list incompleteness bounds recall.","section":"IV-B1/IV-B3, Table IV"},{"comment":"The financial loss estimates are computed with current token prices ('we query the latest prices of various tokens using the token query interface provided by Oklink') rather than prices at the time each transaction occurred. For volatile tokens, including the meme coins highlighted in Table VII, current prices can differ by orders of magnitude from attack-time prices, so the 'nearly $1.1 million' figure is not a reliable measure of actual victim loss. For AAT transactions the loss is also taken as the current total value of tokens in the transferred accounts. The authors should recompute losses using historical prices at the transaction block timestamp (e.g., via a price oracle with historical data) or provide a sensitivity analysis bracketing the estimates with attack-time and current-price valuations.","section":"V-B, Table VI"},{"comment":"The STMT rule's thresholds ('more than two transfer instructions' and 'depletion of two or more types of tokens') and the ISA address-pattern rule ('Compu.*' or '.*1111') are presented without justification or sensitivity analysis. Since the headline count of 8,058 and the derived loss and gang statistics all depend on these thresholds, the paper should include an ablation or threshold sweep showing how the detected counts and precision change as these parameters vary, or justify the thresholds by a distributional analysis of benign versus phishing transactions.","section":"IV-A2-IV-A4, Table II"},{"comment":"The paper claims to publicly release SolPhishDataset and to report results to Solscan and GoPlus, but no dataset URL, repository, or artifact link is provided anywhere in the manuscript. Without a link or clear access instructions, the 'first Solana phishing dataset in academia' contribution and the reproducibility of the empirical analysis cannot be verified. A public artifact link and a description of the dataset schema should be added.","section":"VI-A"}],"minor_comments":[{"comment":"The prerequisite row writes 'tx.from == tx.to', but the text in Section IV-A1 states that a transaction is deemed non-phishing when the beneficiary and loser are the same. The table appears to invert the condition; it should be 'tx.from != tx.to'.","section":"Table II"},{"comment":"The dataset name is inconsistently spelled 'TDFA' in the 'Results on TDFA' heading and in the opening of Section V; it should be 'TDPA'.","section":"IV-B3"},{"comment":"The suspicious-transaction counts are inconsistent: the text reports 8,571 suspicious TDPA transactions, then later refers to 'these 8,573 suspicious transactions', and the summary uses 8,576 total suspicious transactions including TDNA. These numbers should be reconciled.","section":"IV-B3"},{"comment":"Finding 1 says 'from ISA to ATMT to AAT'; 'ATMT' should be 'STMT'.","section":"V-A, Finding 1"},{"comment":"The loss values in Table VI contain formatting corruptions (e.g., '$150 , 19.12', '$751 , 88.51', '$106 , 34.25'); these should be corrected to standard decimal notation.","section":"Table VI"},{"comment":"The gang analysis would benefit from a precise definition of what constitutes a 'gang' (e.g., minimum number of interactions or thresholds for edge creation) and from reporting how the 11 gangs were extracted from Figure 8.","section":"V-D"}],"recommendation":"major_revision","confidential_remarks":"The topic is timely and the manuscript addresses a genuine gap, but the evaluation section needs substantial rework before the headline claims can be trusted. The circular precision estimate and current-price loss calculations are the main blockers; both are fixable within the manuscript's scope through manual sampling, historical pricing, and sensitivity analysis. I would also encourage the editor to require the dataset artifact to be available at revision time, since the dataset is advertised as a core contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SolPhishHunter is the first academic attempt to map phishing on Solana, and that alone is worth something. The three transaction patterns (STMT, AAT, ISA) are genuinely new to the literature, the detection rules are explicit and reproducible, and the empirical analysis - temporal trends, phisher life cycles, gang structures - gives a useful picture of how Solana-specific abuse works. The authors also report findings to Solscan and GoPlus, a real community contribution. The soft spots are real but addressable. The main one is the 93.96% precision figure. The 8,058 transactions are counted as true positives simply because they involve accounts from the seed phishing list; there is no manual verification of the TDPA positives. That is not a measured precision, it is an assumption. The five TDNA false positives get expert review, but the 8,058 do not. If even a small fraction are legitimate transfers or misclassified flows, the count and the $1.1M loss figure change materially. The loss calculation also uses current token prices rather than prices at attack time, which distorts historical losses in a volatile market. There are also numerical inconsistencies: 8,571 vs 8,573 vs 8,576 suspicious transactions, and the Table VI loss values look garbled. The promised dataset is not linked anywhere, a problem for a paper whose main contribution is a dataset. None of this is fatal. The detection rules are independently defined and the taxonomy stands on its own. The paper would benefit from a major revision: release the dataset, manually validate a random sample of positives, recompute losses with historical prices, and clean up the inconsistencies. Recall bounds from the seed list should also be discussed, since the paper never quantifies missed phishing. Who is this for? Blockchain security researchers, especially Solana defenders and web3 abuse analysts. Not a broad breakthrough, but a useful empirical foundation. I would send it to peer review, not desk reject, with requests for the revisions above.","headline":"First academic dataset and taxonomy for Solana phishing, but the headline counts and precision rest on an unvalidated labeling step that needs fixing before the numbers are trusted.","tokens_in":727,"tokens_out":1635,"would_cite":true,"duration_ms":30365,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Solana phishing has three transaction-level signatures that a simple rule engine can detect, catching 8,058 frauds worth about $1.1 million.","keywords":["Solana","phishing detection","transaction-level detection","account authority transfer","system account impersonation","multiple transfers","blockchain security","phishing gangs"],"falsifier":"Take a random sample of roughly 386 of the 8,058 flagged transactions, reconstruct each one from Solana's public records, and independently classify whether it actually drained victim funds or transferred account authority to a phisher-controlled program, without relying on the source account's label. If independently verified precision falls well below 93.96%, or if the audit cannot reproduce the transaction-type distribution, the paper's central precision claim and count are not supported.","tokens_in":1527,"feed_emoji":"🎣","tokens_out":1860,"duration_ms":68234,"temperature":0.7,"pith_summary":"The paper sets out to show that phishing on Solana is not just Ethereum phishing transplanted: Solana's support for multiple instructions in one transaction and its owner-based account model have produced at least three new transaction-level fraud patterns, which the authors call SolPhish. They define the three types—single transaction with multiple transfers, account authority transfer, and impersonation of system accounts—and build a small rule-based detector, SolPhishHunter, around them. Applied to the historical transactions of 169 labeled phishing accounts, the tool flags 8,058 transactions as SolPhish at roughly 93.96% precision, and those transactions caused about $1.1 million in victim losses. The claim matters because it is the first systematic, transaction-level picture of Solana phishing and because the rules are simple enough to run inside wallets before a user signs.","feed_headline":"8,058 Solana phishing transactions found by three simple rules","feed_subtitle":"Solana's multi-instruction and account-ownership design enables new fraud; simple rules catch it before signing.","key_machinery":"The load-bearing object is a set of four prerequisite filters plus three type-specific rule patterns. The prerequisite filters remove benign market activity by requiring that neither beneficiary nor loser is a trading market, that the transaction log lacks buy/sell/purchase keywords, and that sender and receiver differ. Then STMT requires more than two transfer instructions and at least two token balances falling to zero; AAT checks for an Assign instruction or an account-owner SetAuthority instruction; ISA checks for a transfer that empties SOL or a token into an address matching \"Compu.*\" or \".*1111\". These rules are the whole engine: they turn Solana's instruction, log, and balance records directly into labels, and the paper's empirical analysis is built entirely on their outputs.","core_discovery":"The paper claims that Solana's unique transaction and account design creates three concrete phishing transaction types that can be detected with explicit rules: STMT, a single transaction containing more than two transfer instructions that drains at least two token types; AAT, a transaction carrying an Assign instruction or a SetAuthority instruction with authorityType set to \"account owner\", transferring ownership of a wallet or token account to a phisher; and ISA, a transaction whose beneficiary address mimics a system account by starting with \"Compu\" or ending with \"11111\" while draining tokens. Using these rules, SolPhishHunter detects 8,058 phishing transactions in the historical records of 169 known phishing accounts, with 93.96% precision after excluding market transactions, and the authors report those transactions caused nearly $1.1 million in losses. They also construct what they describe as the first Solana phishing transaction dataset and report 64 phishing accounts to security services in the ecosystem.","pith_inferences":["The 93.96% precision likely overstates true precision for the whole network, because a rule match on a pre-labeled phishing account is counted as a true positive without independent verification; on 200 normal accounts the tool already produced five false positives.","If the same rules were run over all Solana transactions rather than only known-phisher histories, false positives from legitimate multi-token sweeps, vault assignments, and vanity addresses could grow, so the 8,058 figure is best read as a floor, not the true prevalence of SolPhish.","Loss estimates use current token prices applied to historical transfers, so the reported $1.1 million may differ from what victims actually lost at the time; illiquid meme coins in AAT cases could inflate the figure.","Phishers can likely evade detection by generating vanity addresses with prefixes outside the two patterns or by splitting multi-token drains into separate transactions, which makes the detector a moving target rather than a permanent solution."],"forward_implications":["A wallet-integrated version of the tool could warn users before signing, because all three signatures are visible in the transaction before execution.","Security scanners and block explorers can add Assign and account-owner SetAuthority to high-risk instruction lists; AAT, although the least frequent type, accounts for about 74% of the measured losses.","The released transaction dataset gives graph and machine-learning researchers a labeled Solana set for training phishing-account and phishing-gang detection models.","Blacklist coverage improves: 46 of the 64 reported phishing accounts were not already flagged by the block explorer, so transaction-level rules find accounts that reputation feeds miss.","The three explicit transaction shapes can be encoded directly into real-time alerts by wallet and security services, not just retrospective scanners."],"supporting_citations":[{"why":"Supplies the seed phishing-account labels from user victim reports; without it there is no ground-truth account set.","marker":"[17]"},{"why":"Supplies Solana block-explorer phishing flags as the second seed source and the baseline for false-positive comparison.","marker":"[18]"},{"why":"Provides the RPC-based parallel data-crawling method used to build the transaction datasets from account histories.","marker":"[16]"},{"why":"The Ethereum payload-based transaction-phishing detector that the paper contrasts, showing why Solana's multi-instruction design requires new rules.","marker":"[10]"},{"why":"Documents prior Solana wallet-drainer attacks and victim losses, motivating the existence and scale of the STMT pattern.","marker":"[4]"},{"why":"The tip-router code repository used to explain four of the five false positives as MEV fee distribution rather than phishing.","marker":"[24]"}],"fun_headline_variants":["8,058 Solana phishing transactions caught by three rules","Solana's unique design enables new phishing—8,058 found","SolPhishHunter flags $1.1M in novel Solana scams","Three Solana transaction patterns reveal 8,058 phishing cases","First dataset maps Solana phishing with simple detection"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The entire result rests on the assumption that a transaction matching a rule and touching an account already labeled as phishing is itself phishing; if the seed label list from user reports and the block explorer is incomplete or contains mistaken labels, both the 93.96% precision and the 8,058 count drift.","fun_headline_variants_meta":{"raw":{"variants":["8,058 Solana phishing transactions caught by three rules","Solana's unique design enables new phishing—8,058 found","SolPhishHunter flags $1.1M in novel Solana scams","Three Solana transaction patterns reveal 8,058 phishing cases","First dataset maps Solana phishing with simple detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1293,"prompt_tokens":909,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":298}},"tokens_in":525,"tokens_out":384,"duration_ms":4013,"temperature":1.0,"reasoning_tokens":298,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:37:34.474667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly 386 of the 8,058 flagged transactions, reconstruct each one from Solana's public records, and independently classify whether it actually drained victim funds or transferred account authority to a phisher-controlled program, without relying on the source account's label. If independently verified precision falls well below 93.96%, or if the audit cannot reproduce the transaction-type distribution, the paper's central precision claim and count are not supported.","supporting_citations":[{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Supplies the seed phishing-account labels from user victim reports; without it there is no ground-truth account set."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Supplies Solana block-explorer phishing flags as the second seed source and the baseline for false-positive comparison."},{"cited_title":"Know your transactions: Real-time and generic transaction semantic representation on blockchain & web3 ecosystem,","cited_arxiv_id":null,"evidence_quote":"Provides the RPC-based parallel data-crawling method used to build the transaction datasets from account histories."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Documents prior Solana wallet-drainer attacks and victim losses, motivating the existence and scale of the STMT pattern."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"The tip-router code repository used to explain four of the five false positives as MEV fee distribution rather than phishing."}],"review_version":1}