{"id":"461cb7a1-ea02-45cb-bd67-806436f39d03","arxiv_id":"2504.18055","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A large-scale study of 1.5 billion failed Solana transactions finds that price or profit not met and invalid status errors dominate, that bot-run accounts fail at 58.43%, and that failures concentrate in a few AMM and DEX aggregator programs.","lead":"An empirical study of 1.5 billion failed Solana transactions shows that nearly 60% of bot-driven transactions fail, mostly because expected prices or profits are not met. It also introduces a taxonomy of failure types and links them to specific programs and initiator classes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finding 1 is computed only for the 26.9% of sampled transactions from accounts passing the 0.9 bot/human confidence threshold; the remaining 73.1% (70% of failures) are unclassified, so the bot-driven conclusion does not cover most failures unless unknown accounts are shown to be bot-like.","rationale":"The reader's CONDITIONAL verdict already flags classifier issues, but its weakest_assumption is sampling representativeness. I find the sampling design less threatening: 53 days spread weekly over 12 months with 10.4M blocks (about 14.5% of blocks) gives broad temporal coverage, and Figure 4 shows the sampled hours tracking the year. The more load-bearing weakness is the coverage of Finding 1. Even if the classifier is 96.91% accurate, thresholding at 0.9 excludes 10.55M accounts and 73% of transactions; the bot/human failure-rate gap is only observed for the classified tail. Since 70% of failures are from unclassified accounts with a 49.9% failure rate, the paper's own data leave open that a large fraction of failures are not from identified bots. This does not refute the paper, because the replication package may allow reweighting, but it means the central 'bot-driven' claim is conditional on an unstated assumption. The other candidate concerns are less central: the error-type percentages appear to be relative to the 1.145B failed transactions with explicit error logs rather than all 1.510B failures, which is a precision issue for exact percentages but not for the qualitative error-mix finding; and the ten-versus-nine error-type inconsistency is minor. Since the identified concern is addressable and does not, on current evidence, break the central claim, I keep the verdict at CONDITIONAL, matching the reader.","tokens_in":22937,"tokens_out":8126,"duration_ms":82983,"concrete_test":"Using the replication package, do not threshold at 0.9: for every account, weight its transactions by the random-forest probability of being a bot (P_bot) and of being a human (P_human), then compute weighted failure rates and the weighted share of failed transactions attributable to each class. If the bot-weighted failure rate remains approximately 58% and the human-weighted rate remains approximately 6%, Finding 1 generalizes beyond the high-confidence tail. If the human-weighted failure rate moves toward the 49.9% rate of unknown accounts, or the bot-attributable share of all failures stays near 30%, the paper's bot-driven conclusion should be weakened to 'high-confidence bot accounts fail often.' Also report the same table with threshold 0.5 as a robustness check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's classifier thresholds the random-forest probability at 0.9, labeling only 803,136 bot and 1,359,772 human accounts out of 12,712,516. Table 1's headline rates (58.43% vs. 6.22%) are computed only over transactions from these labeled accounts, which sum to 779.3M of the 2,898.2M sampled non-vote transactions (26.9%). The other 2,118.8M transactions, including 1,057.1M failed transactions (70.0% of all failures), come from accounts below the threshold; their failure rate is 49.9%, close to the 52.1% overall rate. Nothing in the paper establishes the initiator type of these unknown-account failures. If they are predominantly bot-initiated, the bot-driven narrative is strengthened; if a substantial share are human or mixed, Finding 1's contrast is an artifact of selecting high-confidence bot extremes and excluding everything else. The 96.91% validation accuracy on 194 accounts tests label agreement, not the representativeness of the high-confidence subset. The ground truth of 200 accounts (96 human, 104 bot) also looks unlike the classified population: classified human accounts average only 2.3 transactions per account-year, so the training distribution may not reflect real transaction behavior. Consequently, the central claim that Solana's failure problem is largely bot-driven rests on an unverified assumption about the 73.1% of transactions that are unclassified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale empirical study of failed non-vote transactions on the Solana blockchain. The authors sample 53 days (one per week) between August 1, 2023 and July 31, 2024, collect 2,898,175,006 non-vote transactions, classify initiator accounts as bot or human using a random forest with a 0.9 confidence threshold, characterize failures by program, time, block position, and cost, extract error messages, and derive a ten-category error taxonomy via thematic analysis. The main findings are that bot accounts have a 58.43% failure rate versus 6.22% for human accounts; the top ten programs account for 77.95% of failures; price/profit-not-met, invalid-status, and validity-expiration errors account for 84.90% of failures; and DEX aggregators and AMMs exhibit distinct error profiles. The paper concludes with ecosystem-level recommendations and a public replication package.","tokens_in":23227,"tokens_out":5798,"duration_ms":58596,"significance":"If the results are robust, this is a valuable first systematic account of failed transactions on Solana, with direct relevance to protocol design, DeFi program development, and user tooling. The dataset is unusually large, the data-processing pipeline is described in detail, the error taxonomy is built through manual coding with consensus procedures, and a replication package is provided. The headline conclusion that Solana's transaction-failure problem is largely bot-driven is plausible and practically important, but it currently rests on the classified subset of accounts, and several statistical claims are under-supported. I also note that the random forest is a labeling tool rather than a model of the outcome being estimated, so I do not see a circularity problem here; the issue is coverage and validation of the labels, not circularity.","major_comments":[{"comment":"Finding 1 is computed only from accounts classified with random-forest probability above 0.9. These accounts produce 779.3M transactions, i.e., 26.9% of the 2,898.2M sampled non-vote transactions. The remaining 2,118.8M transactions—including 1,057.1M failed transactions, or 70.0% of all failures in the sample—come from 'unknown' accounts and are excluded from the bot-versus-human comparison. The unknown-account failure rate is 49.9%, close to the overall sample failure rate of 52.1%, so the claim that Solana's failures are largely bot-driven depends on an unverified assumption about the initiator type of unknown-account failures. Since §7.3 does not discuss this coverage threat, please report a sensitivity analysis (e.g., lower the threshold, or manually label a sample of unknown-account failures) and bound Finding 1 under worst-case assumptions about the unknown set.","section":"§3.2, Table 1"},{"comment":"The claim that stratified weekly sampling achieves a '99.999% confidence level with a margin of error of ±0.06%' is asserted without derivation. The failure rate is highly non-stationary—the paper itself notes memecoin-mania episodes and the June 10, 2024 validator update—so a textbook simple-random-sampling formula cannot be applied to one randomly selected day per week. Please provide the exact estimator, its variance, and the assumptions; otherwise, the aggregate percentages in Findings 2 and 5 are not shown to represent the full year.","section":"§3.1"},{"comment":"The classifier is trained on 200 manually labeled accounts and validated on 194 sampled accounts, yielding 96.91% accuracy. This measures agreement on a hand-picked validation set, not whether the high-confidence subsets are representative of bots and humans at large. The imbalance is striking: classified human accounts average about 2.3 transactions per account-year, while classified bot accounts average about 966, so the random forest may be separating high-activity from low-activity accounts rather than automated from human initiators. Please report per-class precision and recall, class-balance information, and validation results for accounts near the 0.9 threshold.","section":"§3.2"},{"comment":"The Wilcoxon rank-sum tests are applied to billions of transactions, so p<0.001 is essentially guaranteed and does not convey the magnitude of the observed differences. Please report standardized effect sizes (e.g., rank-biserial correlation) and confidence intervals for the block-position, fee, compute-unit, and cost-efficiency comparisons, so readers can gauge whether the reported differences are practically meaningful.","section":"§4.2.2"},{"comment":"Finding 3 states that failure rates exhibit a strong positive correlation with the volume of failed transactions, but the surrounding text describes correlation with hourly transaction volume, and no correlation coefficient is reported. Please state the exact statistic (e.g., Spearman's r), its lag, and its confidence interval; otherwise the 'strong positive correlation' claim is not quantitatively supported.","section":"§4.2.1, Finding 3"}],"minor_comments":[{"comment":"The reference '[73? ]' is a broken citation placeholder and should be corrected.","section":"§7.1"},{"comment":"The sentence 'A Wilcoxon rank-sum test confirms the statistical significance of the difference in block positions...' appears twice; the duplicate in the Temporal Trends paragraph should be removed.","section":"§4.2.1 and §4.2.2"},{"comment":"Finding 5 lists the top three error types as 'price or profit not met, validity expiration, and invalid status', whereas the preceding text and Figure 6 present the second and third types in the order invalid status then validity expiration. Please make the order consistent.","section":"§5.2, Finding 5"},{"comment":"The phrase 'the failed transactions have be outcompeted' is a grammatical error; it should read 'have been outcompeted'.","section":"§4.2.2"},{"comment":"The abstract says the dataset spans 'more than 72 million blocks', but the sampling procedure selected 10,458,452 blocks from the 72,123,900-block range; please clarify that 72 million is the full-year block range, not the sampled block count.","section":"Abstract and §3.1"},{"comment":"Consider adding a row for unknown accounts in Table 1, since their exclusion is material to the interpretation of the bot/human comparison.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the empirical software engineering scope of the venue despite its blockchain topic. My main concern is that the headline bot-vs-human claim is presented at the population level while being computed on a 26.9% subset; this is fixable with sensitivity analyses and should not require a full re-collection of the data. The authors should also be asked to justify or temper the sampling-confidence claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the Solana failed-transactions paper. The headline bot-vs-human split is the most publicized result, but it is also the weakest part of the study. The authors classify accounts with a random forest trained on 200 hand-labeled accounts, then keep only predictions above 0.9 confidence. That covers 803K bot + 1.36M human accounts out of 12.7M, i.e., about 27% of sampled transactions. The failure rate for the unclassified 73% (which contains 70% of all failures) is 49.9%, close to the overall 52.1%. So the 58.43% vs 6.22% contrast describes only the extreme high-confidence accounts, and we have no evidence that the unlabeled majority behaves like those groups. The stress-test note is right: the central \"bot-driven\" story is an unverified assumption about most failures.\n\nThat said, the paper is genuinely new and useful. It provides the first large-scale quantitative picture of failed Solana transactions: 1.5B failures across 72M blocks, a systematic error taxonomy (nine? ten?—they claim ten but only describe nine—need to fix), and program-level associations. The top-ten programs account for 78% of failures, and the split between AMMs (invalid status) and DEX aggregators (price/profit not met) is a solid, reproducible finding. The replication package is linked, the data collection pipeline is described in detail, and the manual coding process is reasonable.\n\nThe other soft spots are minor relative to the classifier issue. The \"99.999% confidence, ±0.06%\" for the sampling strategy appears without derivation—suggesting it comes from the binomial formula for a proportion, but they should show it. The Wilcoxon tests on billions of rows are statistically trivially significant; better to report effect sizes. And the tenth error type needs to be identified or the claim changed to nine.\n\nWho is this for? Researchers studying blockchain transaction behavior, Solana tooling designers, and protocol economists. The error taxonomy alone is worth having. The paper deserves serious peer review: the methodological concern about the classifier coverage is fixable (e.g., classify all accounts with uncertainty propagation, or present the bot/human claim only for the high-confidence subset with appropriate caveats). I would send it to review with a request for major revision rather than desk reject.","headline":"Worth taking seriously, but the bot-vs-human headline is built on a classifier that labels only a quarter of the data; the error taxonomy alone justifies a revised publication.","tokens_in":23821,"tokens_out":3075,"would_cite":true,"duration_ms":28218,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bots fail 58.43% of Solana transactions; humans 6.22%","keywords":["Solana","failed transactions","blockchain","DeFi","bot accounts","error classification","empirical study","automated market maker"],"falsifier":"Recompute the six headline statistics (bot failure rate, human failure rate, top-ten program share, and the three largest error-type shares) from all 365 days of August 2023 through July 2024 rather than the 53 sampled days; if any figure moves outside the paper's stated ±0.06% margin, or materially away from the reported values, the characterization fails.","tokens_in":22736,"feed_emoji":"🤖","tokens_out":6724,"duration_ms":57541,"temperature":0.7,"pith_summary":"This paper establishes that the flood of failed transactions on Solana is overwhelmingly a bot problem, not a user problem: across 53 sampled days spanning a year, bot-initiated transactions failed 58.43% of the time while human-initiated transactions failed only 6.22%. It further shows the failures concentrate in a small set of programs: the top ten failure-triggering programs account for 77.95% of all failed transactions, led by the Raydium AMM and Jupiter DEX aggregators. By classifying the error messages attached to more than a billion failed transactions, the paper finds that three error types—price or profit not met, invalid status, and validity expiration—explain 84.90% of failures. If these numbers hold, they reframe Solana's congestion and reliability debate: the problem is high-frequency automated trading racing slim margins, not ordinary users making mistakes.","feed_headline":"Bots fail 58.43% of Solana transactions; humans 6.22%","feed_subtitle":"A year of 1.5 billion failed transactions shows the failures concentrate in a few DeFi programs and bot spam.","key_machinery":"The load-bearing machinery is a large curated dataset plus two classification layers. The dataset joins on-chain block and transaction records retrieved through an RPC provider with off-chain program names and source code, covering 2,898,175,006 non-vote transactions. On top of it, the paper builds a random-forest account classifier that labels 12,712,516 accounts as bot or human from transaction frequency and volume features, validated at 96.91% accuracy on a manually labeled sample, and an error taxonomy constructed by extracting 2,311 unique error messages and grouping them through thematic analysis into ten error types. The account classifier makes the bot-versus-human failure-rate contrast possible; the error taxonomy carries the finding that most failures reduce to price, state, and validity checks.","core_discovery":"The paper's central discovery is a quantitative map of why Solana transactions fail at the ecosystem scale. Mining 1,510,834,168 failed non-vote transactions from 72,123,900 blocks, it reports that bots experience a transaction failure rate of 58.43% versus 6.22% for humans, that the top ten programs responsible for failures account for 77.95% of all failed transactions, and that just three error categories—price or profit not met (47.99%), invalid status (19.19%), and validity expiration (17.72%)—account for 84.90% of all failures. The paper also reports that failed transactions sit deeper in blocks (median position 592 versus 529) while paying higher fees per compute unit, that DEX aggregators fail mainly on price or profit conditions, and that AMMs fail mainly on invalid status checks, often because sniper bots trade before a pool is initialized. The intended upshot is that Solana's failure problem is a bot-driven, DeFi-concentrated phenomenon rather than a general user-experience defect.","pith_inferences":["If the 47.99% price-or-profit-not-met share reflects rational bot strategy rather than user error, then a large share of 'failed' transactions are actually expected costs of a search process: bots intentionally fire unprofitable transactions and let rejection filter them, so a testable extension is measuring how failure rates respond when the fee for repeated submissions from one account rises.","The drop in failure rates after the June 2024 validator update, noted in the paper, suggests a quasi-natural experiment; comparing failure rates around that update with and without bot-account filtering could isolate how much of the reduction comes from protocol changes versus reduced spam.","The paper's taxonomy could be applied prospectively: if invalid-status errors spike around new pool creations, real-time detection of sniper-bot waves could be built from the error logs alone."],"forward_implications":["If bots are the main source of failed transactions, then reducing bot spam—for example through dynamic fees on rapid repeated submissions—should lower both failure counts and network congestion.","Failure diagnosis can be prioritized: a transaction that fails on a DEX aggregator should first be checked for unmet price or profit conditions, and a transaction that fails on an AMM should first be checked for pool state readiness.","Human-focused tooling should target out-of-funds checks and input validation, since humans fail less often and on a narrower set of errors.","The 24-hour periodicity in failure rates means failure load is partly predictable, opening the door to time-aware fee or retry strategies.","Architecture-level comparison: Solana's low fees and parallel execution appear to enable the bot behavior; ecosystems with gas auctions show lower bot-driven failure rates."],"supporting_citations":[{"why":"Supplies the RPC-provider block data from which the transaction dataset is built.","marker":"[1]"},{"why":"Supplies program public names and source-code links used to identify failure-triggering programs.","marker":"[83]"},{"why":"Provides the bot-detection features, transaction frequency and volume, on which the account classifier is based.","marker":"[71]"},{"why":"Provides prior behavioral characterization of on-chain accounts used to define bot versus human behavior.","marker":"[62]"},{"why":"Supplies the thematic-synthesis steps used to build the ten-type error taxonomy.","marker":"[53]"},{"why":"Gives the Ethereum transaction-confirmation and failure baseline against which Solana failure rates are compared.","marker":"[73]"},{"why":"Supplies cross-chain daily failure-rate data for Ethereum, Optimism, and Arbitrum used in the ecosystem comparison.","marker":"[44]"},{"why":"Documents the validator update that the paper links to the post-June-2024 drop in failure rates.","marker":"[21]"}],"fun_headline_variants":["Solana: bots fail 58% of txns, humans only 6%","Why Solana fails: 84.9% from just 3 error types","Solana failure rate: 58% bot vs 6% human","1.5B failed Solana txns map to bot spam","Solana's failed txns: 78% from top 10 programs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every headline percentage is computed from 53 sampled days—one randomly chosen day per week—and the paper asserts, without derivation, that the sample represents the whole year at a 99.999% confidence level; if those days miss memecoin surges or protocol-upgrade periods, all of the aggregate figures shift.","fun_headline_variants_meta":{"raw":{"variants":["Solana: bots fail 58% of txns, humans only 6%","Why Solana fails: 84.9% from just 3 error types","Solana failure rate: 58% bot vs 6% human","1.5B failed Solana txns map to bot spam","Solana's failed txns: 78% from top 10 programs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1756,"prompt_tokens":1061,"completion_tokens":695,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":594}},"tokens_in":677,"tokens_out":695,"duration_ms":6115,"temperature":1.0,"reasoning_tokens":594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:25:47.926748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the six headline statistics (bot failure rate, human failure rate, top-ten program share, and the three largest error-type shares) from all 365 days of August 2023 through July 2024 rather than the 53 sampled days; if any figure moves outside the paper's stated ±0.06% margin, or materially away from the reported values, the characterization fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies program public names and source-code links used to identify failure-triggering programs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides prior behavioral characterization of on-chain accounts used to define bot versus human behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Ethereum transaction-confirmation and failure baseline against which Solana failure rates are compared."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies cross-chain daily failure-rate data for Ethereum, Optimism, and Arbitrum used in the ecosystem comparison."},{"cited_title":"Mainnet Beta Validators: Please upgrade to","cited_arxiv_id":null,"evidence_quote":"Documents the validator update that the paper links to the post-June-2024 drop in failure rates."}],"review_version":1}