{"id":"75d83d12-84f6-4f1c-a044-f5ca183573ef","arxiv_id":"2504.15948","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MuSe injects six vulnerability types into 47,398 real smart contracts, producing 350,493 mutated contracts, and Slither detects only 59.7% of the injected flaws.","lead":"This paper presents MuSe, a tool that automatically plants six well-known security flaws into real Ethereum smart contracts to produce large test collections for security analyzers. The study measures how often the injections succeed and how often the Slither analyzer spots the planted flaws.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels are pattern-adherent, not exploitability-validated; Table 3's recall may measure pattern detection rather than vulnerability detection.","rationale":"The paper's strongest contribution is the 0.597 recall figure and the 350k-contract benchmark, so the empirical validity of the ground-truth labels is the hinge. The Section 3.3 manual validation verifies that code changed the way the operator intended and still compiles, but a pattern-adherent mutation is not necessarily a vulnerability: an unchecked call is only exploitable if the call can fail and the failure has harmful consequences; tx.origin is only exploitable if a trusted caller can be induced to interact with an adversary contract; a delegatecall is only exploitable if an attacker can control the target. Section 5.1 contains an explicit admission that most DTU injection sites, namely constructors, cannot yield exploitable instances, which shows the gap is real rather than hypothetical. The CL operator's loop injection also needs gas and DoS semantics to count. Thus Table 3's per-type recall is at risk of measuring Slither's syntactic pattern coverage, not vulnerability detection. A local-EVM exploit audit on the validation sample would settle this: if a high percentage of mutants have a demonstrated exploit path, the reader's conditional concern is answered; if not, the recall claim and the '350,493 vulnerable contracts' assertion need re-scoping to 'pattern-adherent mutants.' This is not a rejection because the artifact is public and the experiment is reproducible; a conditional revision requesting exploitability evidence is proportionate. The reader's weakest assumption already identified this exploitability gap, and I agree; the verdict should remain conditional.","tokens_in":16302,"tokens_out":5328,"duration_ms":51191,"concrete_test":"Run a local-EVM exploitability audit on the 384-contract validation sample (or a fresh random sample of 50-100 per operator): for UC/US, make the recipient call fail and check whether the transaction succeeds while leaving the contract in a harmed state; for TX, invoke the protected function from a malicious caller contract and check for privilege bypass; for DTU, set delegate to attacker-controlled code and confirm attacker state changes; for CL, execute the mutated loop under the block gas limit and check whether it actually causes denial of service; for UR, trace whether the discarded return value leads to an unsafe subsequent state. Report the share of pattern-adherent mutants with a demonstrated exploit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 0.597 recall in Table 3. Its denominator is every mutant that a MuSe operator touched and that compiled. The Section 3.3 validation checks only compilation, SuMo logs, pattern adherence, and modification correctness; it never checks whether the injected pattern is actually exploitable in the contract's runtime context. For DTU the paper itself concedes in Section 5.1 that most delegatecall sites are in constructors, where 'it is impossible to inject an instance of the vulnerability into the constructor that is exploitable,' yet such mutants are still counted as injected vulnerabilities. Similarly, the CL operator wraps a single external call in a for loop: this creates a pattern Slither calls 'calls-loop', but whether it is a DoS vulnerability depends on gas limits, loop bounds, and whether an attacker can force the loop to run. If a large share of the 350,493 mutants match a syntactic pattern but have no exploit path, then Slither's 40.2% false-negative rate is a statement about pattern-recognition coverage, not about failure to detect real vulnerabilities, and the benchmark labels would overstate vulnerability prevalence. The side-effect analysis in Section 5.2 compounds this: mutations add or remove other Slither warnings, so the before/after diff at mutated lines does not cleanly isolate the injected flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MuSe, a mutation-based tool that injects six vulnerability patterns into Solidity smart contracts from the SmartBugs-wild dataset, producing 350,493 mutated contracts. RQ1 measures how often each operator can inject its target pattern, yielding injection rates from 71.5% (UR) down to 0.23% (DTU). RQ2 evaluates Slither's ability to detect the injected vulnerabilities by comparing its output before and after mutation, reporting an overall recall of 0.597 (Table 3) and per-type recall from 1.000 (UC, US) to 0.100 (DTU). The authors manually validate a statistically sampled subset of 384 mutants for compilation, SuMo-log agreement, pattern adherence, and modification correctness, but not for runtime exploitability. The paper claims that MuSe expands the dataset by over 840% and that static analysis tools fail to detect all injected vulnerabilities.","tokens_in":16507,"tokens_out":7967,"duration_ms":69670,"significance":"If the ground-truth labels were valid, MuSe would be a useful contribution: it provides a publicly available tool, a large mutated contract dataset, a replication package, and a concrete empirical finding that even on a set of vulnerability types preselected for Slither's detectors, Slither does not detect 40.2% of the generated mutants. The per-type injection-rate analysis (Table 2) is also informative for understanding pattern prevalence in real-world contracts. However, the central measurement (Table 3 recall) is only as credible as the ground-truth labels, and the paper does not establish that the injected patterns are exploitable vulnerabilities. The paper itself concedes that DTU mutants in constructors are not exploitable, yet those mutants are counted in Table 3; side effects in 30-75% of mutations confound the before/after comparison; and the vulnerability types were chosen partly because Slither already detects them, limiting generalizability. These issues affect the load-bearing empirical claim, so the significance of the contribution will be fully realized only after the measurement is repaired or its scope is clearly delimited.","major_comments":[{"comment":"The manual validation in Section 3.3 checks compilation, SuMo logs, pattern adherence, and modification correctness, but it does not check whether the injected mutation is exploitable in the contract's runtime context. This is load-bearing for RQ2 because the denominator of Table 3 includes all analyzed mutants, not only those with a potential exploit path. The paper itself states in Section 5.1 that most delegatecall sites are in constructors, where \"it is impossible to inject an instance of the vulnerability into the constructor that is exploitable,\" yet such mutants are counted as DTU false negatives (134 of 149 in Table 3). The overall recall of 0.597 is therefore not a measurement of detection of exploitable vulnerabilities. I recommend either restricting the ground truth to reachable, potentially exploitable mutations (e.g., excluding constructor DTU and CL loops without attacker-controlled bounds and sufficient gas), or explicitly re-labeling the dependent variable as 'pattern-detection recall' and adjusting the abstract and conclusion accordingly.","section":"Section 3.3 and Section 5.1/Table 3"},{"comment":"The vulnerability types are selected \"based on their relevance in the literature and the ability of Slither to detect them with at least medium confidence.\" This makes the 0.597 overall recall a conditional estimate: it measures Slither on a set of types for which Slither already provides detectors, so it is not representative of the general failure rate across the space of smart contract vulnerabilities. The abstract states that \"static analysis tools fail to detect all vulnerabilities injected using pattern-based mutations,\" which goes beyond what the design can support. I suggest either adding vulnerability types without existing Slither detectors to test generality, or explicitly framing RQ2 as an evaluation of Slither's detectors on mutation-injected instances of six targeted patterns rather than a general statement about static analysis tools.","section":"Section 3, first paragraph"},{"comment":"The TP/FN classification in Section 4.2 is based on comparing Slither output before and after mutation, filtered to the mutated lines. Section 5.2 reports that 30-75% of mutations introduce side effects that add or remove other Slither warnings, including removal of pre-existing vulnerabilities (e.g., reentrancy-benign -5,340, reentrancy-no-eth -3,191, reentrancy-eth -830, divide-before-multiply -3,913, incorrect-equality -3,851). Because a mutation can remove a warning at the mutated line or change the code so that a pre-existing warning disappears, the before/after diff does not cleanly isolate the injected vulnerability; Table 3's counts may conflate \"Slither does not detect the injected pattern\" with \"Slither no longer detects a different pattern affected by the mutation.\" Please provide a sensitivity analysis that excludes mutants with interfering side effects, or a per-case reconciliation of the diff, before the recall values can be interpreted as detection of the injected vulnerability.","section":"Section 4.2 with Section 5.2"},{"comment":"The manual validation found that 20 of 384 sampled mutants (5.21%) were not correctly injected, giving examples such as syntax truncation on semicolons and variable-scope conflicts in the CL operator. The paper does not state that these failed mutants are excluded from Table 3; since the table's denominator is the 335,163 successfully analyzed mutants, it likely includes a comparable fraction that are not actual instances of the target vulnerability. If Slither does not flag a failed injection, that mutant is mislabeled as a false negative, deflating the reported recall. I recommend filtering out known-failed injections where detectable from SuMo logs, applying a conservative correction based on the observed 5.21% failure rate, or adding a quantitative threat-to-validity discussion of the impact on the reported recall.","section":"Section 3.3 vs Table 3"}],"minor_comments":[{"comment":"The number of mutated contracts is given as 350,716 in Section 3.3 but 350,493 in Section 4 and Table 2; additionally, Section 4.2 says Slither successfully analyzed 335,234 mutants, while Table 3's TP+FN totals sum to 335,163. Please reconcile these counts.","section":"Section 3.3 / Section 4 / Table 3"},{"comment":"The text says six vulnerabilities are selected but then lists only five: \"Unchecked call return value, Unchecked send, Authentication through tx.origin, Delegatecall to untrusted callee, and Unused return.\" \"Multiple calls in a loop\" is missing from the list.","section":"Section 3, first paragraph"},{"comment":"The prose says the UR recall is 0.63, but Table 3 reports 0.605; update the prose to match the table.","section":"Section 4.2, RQ2 text"},{"comment":"The illustrative code for the CL operator contains an unbalanced parenthesis in \"require (member.send(0.1 ether);\" in both the before and after snippets; fix the example's syntax.","section":"Section 3.2, CL operator code example"},{"comment":"The paper uses \"vulnerability,\" \"code smell,\" and \"warning\" inconsistently; for example, events-math and events-access are described as vulnerabilities while Slither labels them as informational or optimization issues. Please use consistent terminology that matches Slither's detector categories.","section":"Section 5.2"},{"comment":"The CL row describes \"Multiple calls in a loop\" with the hedged text \"might lead to a denial-of-service attack,\" but RQ2 and Table 3 count all CL mutants as true vulnerabilities. The hedged description should be reflected in the interpretation of the recall numbers.","section":"Table 1"},{"comment":"Reference [17] has a placeholder title (the citation reads \"Ghaleb and Pattabiraman. 2020. '. In Proceedings...\"). Please complete the bibliographic entry.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of an empirical software engineering venue and the tool artifact is a positive feature. The main risk is that the headline recall value is presented as measuring detection of vulnerabilities when the ground truth is only pattern-adherent and, by the authors' own admission, includes non-exploitable DTU mutants. The authors also selected types for which Slither has detectors, so the external-validity claims need to be narrowed. These are fixable with targeted re-analysis and more careful claim framing, hence major revision rather than rejection. I would also check the reference list carefully; at least one entry is malformed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on smart contract security benchmarks. MuSe extends SuMo with six security-specific mutation operators and runs them over 47k real contracts to produce 350k mutants. The tool is public, the method is clear, and the per-operator injection rates are a genuinely useful data point. The manual validation of 384 mutants—checking compilation, logs, pattern adherence, and intended modification—is real work, not a box-ticking exercise.\n\nThe soft spot is the one the stress-test flags: pattern-adherent is not the same as exploitable. The paper counts a mutant as a vulnerability if it compiles and matches the mutation pattern. It never checks whether the injected flaw can be exploited in context. Section 5.1 admits that most delegatecall sites are in constructors, where it is impossible to inject an exploitable instance, yet those mutants still appear in Table 3’s denominator. The CL operator does something similar: wrapping a call in a for loop produces Slither’s calls-loop pattern, but whether that is a DoS depends on gas limits and loop bounds. So the 0.597 recall is really about how often Slither recognizes the synthetic pattern, not how often it misses real vulnerabilities. That weakens the central claim.\n\nTwo smaller issues. The vulnerability set was chosen partly because Slither can detect those types at medium confidence, which biases recall upward. And the abstract and conclusion generalize to “static analysis tools” after testing only Slither; the threats section notes this, but the framing still overstates. The side-effect analysis in Section 5.2 also shows that mutations add and remove other warnings, so the before/after diff at mutated lines is not clean—though the authors do discuss this at length.\n\nNone of this kills the contribution. The tool, the dataset, and the injection-rate statistics are useful to the community as a benchmark generation approach, as long as consumers treat the labels as pattern-based, not exploit-proven. A serious referee should engage with it and push for either exploitability validation on a sample or softer conclusions. I’d bring it to a reading group, cite the tool, and conditionally accept after revision.","headline":"Useful mutation tool and large benchmark, but labels are pattern-adherent rather than exploitability-checked, so the 59.7% recall measures pattern detection more than real vulnerability detection.","tokens_in":17060,"tokens_out":2106,"would_cite":true,"duration_ms":20545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pattern-based mutation operators can inject six known vulnerability types into real-world Solidity contracts at scale, and the static analyzer Slither detects only 59.7% of the injected flaws.","keywords":["smart contract security","vulnerability injection","mutation testing","Solidity","static analysis","benchmark generation","Slither","Ethereum"],"falsifier":"Deploy a random sample of mutants that Slither flagged as true positives, roughly 50 per vulnerability type, in a local Ethereum VM; run the attack that each vulnerability class is supposed to enable, such as draining funds via an unchecked call, spoofing the owner via tx.origin, or redirecting delegatecall to a malicious contract; and count how many mutants are actually exploitable. If that fraction is far below 1, the 59.7% recall measures detection of code patterns rather than detection of exploitable vulnerabilities.","tokens_in":16091,"feed_emoji":"🧬","tokens_out":7296,"duration_ms":61655,"temperature":0.7,"pith_summary":"The paper tries to establish that mutation seeding can turn ordinary Solidity contracts into a large, labeled benchmark of vulnerable ones, and that doing so exposes measurable gaps in current static analysis. It introduces MuSe, a tool that extends an existing mutation tester with six pattern-based operators, each of which rewrites code so it matches a known vulnerability pattern. On 47,398 contracts from an Ethereum-derived dataset, MuSe generates 350,493 mutants, an increase above 840%; manual review of 384 sampled mutants found 94.79% correctly injected. Running Slither on the mutants, the paper measures an overall recall of 59.7%, ranging from 100% for unchecked low-level calls and unchecked sends down to 10% for delegatecall-to-untrusted-callee. If the label 'injected vulnerability' is taken at face value, the benchmark gives detector builders a concrete stress test with known ground truth.","feed_headline":"Injecting known flaws into 47k contracts: Slither misses 40%","feed_subtitle":"From 47,398 real contracts to 350,493 vulnerable ones; Slither catches only 59.7% of injected flaws.","key_machinery":"The load-bearing mechanism is the pattern-based mutation operator: a syntactic rewrite rule that scans an abstract syntax tree for a code shape and rewrites it into a shape associated with a known vulnerability. MuSe pairs one operator with each of the six vulnerabilities, using the parser's AST to locate target statements and the mutation engine to apply the rewrite. The operators carry the argument because benchmark validity rests on them: a mutant counts as an injected vulnerability if and only if the operator's pattern was applied, and detection is measured by comparing Slither's findings before and after that rewrite on the mutated lines.","core_discovery":"The central discovery, on the paper's own terms, is that pattern-based mutation operators can automatically produce vulnerable smart contracts and that Slither cannot detect all of them. The six operators remove return-value checks on low-level call and send, replace msg.sender with tx.origin in authorization checks, drop unused return values, wrap external calls in loops, and replace delegatecall targets with a user-settable address. Across the real-world dataset, 41,337 of 47,398 contracts were mutated at least once; the most common injections were unused return (213,912 mutants) and tx.origin authorization (65,825), while delegatecall injection was rare (149 mutants). Slither's reported recall on the injected flaws is 59.7%, with per-type recall of 1.000 for unchecked low-level call and unchecked send, 0.810 for calls in a loop, 0.605 for unused return, 0.336 for tx.origin authorization, and 0.100 for delegatecall to untrusted callee.","pith_inferences":["A next step the paper leaves implicit is to check exploitability dynamically, for example with a test harness or symbolic execution, which would separate 'the code matches a vulnerable pattern' from 'an attacker can actually cause harm'.","The per-type recall spread suggests detector authors could use the mutants as regression tests for the specific missed patterns, such as tx.origin inside modifiers, unused returns from inherited functions, and settable delegatecall targets.","Because only contracts containing the relevant constructs can be mutated, mutation-based benchmarks inherit the base dataset's blind spots; combining mutation with audit-based or generative datasets may cover rare types like delegatecall.","The side effects described in the paper imply that training machine-learning detectors on this benchmark without filtering to the mutated lines could inject label noise; line-scoped logs from the mutation engine are the obvious mitigation."],"forward_implications":["Researchers can expand a 47,398-contract dataset into 350,493 mutants, an increase above 840%, with labels tied to six injected vulnerability types.","Detection-tool evaluation can expose per-type blind spots: Slither catches 100% of unchecked call and send mutants but only 10% of delegatecall mutants, so the benchmark separates easy from hard vulnerability classes.","Because operators are pattern-driven, adding a new vulnerability type is a matter of writing a new operator, so the benchmark can grow with the vulnerability taxonomy.","Consumers of the benchmark should expect side effects: mutations can add or remove unrelated warnings, so labels need line-level filtering or manual review."],"supporting_citations":[{"why":"The mutation-testing tool MuSe extends; its parser and mutation engine carry the injection logic.","marker":"[2]"},{"why":"The dataset of real Ethereum contracts that supplies the 47,398 base contracts for mutation.","marker":"[15]"},{"why":"The static analyzer whose before/after findings define the recall measurements.","marker":"[16]"},{"why":"The execution framework used to run Slither in parallel and parse its output.","marker":"[12]"},{"why":"The prior Solidity bug-injection work whose code-transformation strategy is extended.","marker":"[17]"},{"why":"The systematic literature review used to choose the six injected vulnerability types.","marker":"[38]"}],"fun_headline_variants":["Auto-seed 350k vulnerable contracts; Slither catches only 60%","Mutation injection creates 6 flaw types at scale; static tools lag","Slither misses 40% of automatically injected smart contract flaws","Pattern-based mutation seeds vulnerabilities; Slither fails 40%","Auto-inject 6 flaw types into real contracts; static tools catch only 60%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on treating a mutant that compiles and matches the intended code pattern as an actual instance of the vulnerability it is meant to inject.","fun_headline_variants_meta":{"raw":{"variants":["Auto-seed 350k vulnerable contracts; Slither catches only 60%","Mutation injection creates 6 flaw types at scale; static tools lag","Slither misses 40% of automatically injected smart contract flaws","Pattern-based mutation seeds vulnerabilities; Slither fails 40%","Auto-inject 6 flaw types into real contracts; static tools catch only 60%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002384,"raw_usage":{"total_tokens":9182,"prompt_tokens":959,"completion_tokens":8223,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":8126}},"tokens_in":575,"tokens_out":8223,"duration_ms":49504,"temperature":1.0,"reasoning_tokens":8126,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:13:37.487121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Deploy a random sample of mutants that Slither flagged as true positives, roughly 50 per vulnerability type, in a local Ethereum VM; run the attack that each vulnerability class is supposed to enable, such as draining funds via an unchecked call, spoofing the owner via tx.origin, or redirecting delegatecall to a malicious contract; and count how many mutants are actually exploitable. If that fraction is far below 1, the 59.7% recall measures detection of code patterns rather than detection of exploitable vulnerabilities.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The mutation-testing tool MuSe extends; its parser and mutation engine carry the injection logic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The dataset of real Ethereum contracts that supplies the 47,398 base contracts for mutation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The systematic literature review used to choose the six injected vulnerability types."}],"review_version":1}