{"id":"ad5d232b-f0d8-4bb0-b2ef-aa5116056ead","arxiv_id":"2506.19624","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuned Llama-3.2-3B model plus a three-address-code intermediate representation decompiles EVM bytecode into Solidity with an average semantic similarity of 0.82 on held-out functions.","lead":"This paper builds a decompiler that turns raw Ethereum smart contract bytecode into readable Solidity source code, using a fine-tuned large language model guided by static analysis. The tool could help security auditors and incident responders understand the roughly 99% of on-chain contracts that are not open source.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'average semantic similarity of 0.82' is not assessable because Section 4 never defines the semantic-similarity metric; the quantitative claim is unfalsifiable until the metric and its implementation are specified.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already lists the undefined semantic similarity metric as a material gap. My read agrees with that concern and elevates it to the single most load-bearing issue: the central numeric claim of the paper is the 0.82 average semantic similarity, and that number cannot be checked, interpreted, or compared against baselines without a precise metric definition. The reader's named weakest assumption, that the TAC stage 'preserves the semantic content of the original bytecode,' is also a real concern and is supported by the staking-rewards case study, but it is secondary in one sense: even if the TAC were perfectly semantic-preserving, the paper still does not tell us what its headline similarity score means. Conversely, if the metric is a weak proxy such as embedding cosine, the TAC preservation issue becomes even more damaging because the metric may not penalize semantic loss. The two concerns compound, but the undefined metric is the one that must be settled first for the paper's central claim to be verifiable. I agree with the CONDITIONAL verdict: the idea and pipeline are plausible and the failure cases are honestly reported, but the evaluation needs a precise, reproducible metric before the headline claim can be accepted. No change to the reader's verdict is needed, hence UNCHANGED.","tokens_in":15218,"tokens_out":3641,"duration_ms":39213,"concrete_test":"Require the authors to publish the exact definition and implementation of the semantic-similarity metric (e.g., CodeBLEU, CodeBERTScore, tree-sitter AST similarity, or an execution-based equivalence oracle) and rerun the evaluation on the 9,731-function test set. As a sanity check, take a random sample of 100 functions and compare the claimed metric against a behavioral oracle (e.g., symbolic execution or differential testing of original vs. decompiled function under random or symbolic inputs). If the 0.82 average moves materially under a behavior-based metric, or if standard decompilers (Gigahorse, Panoramix) score comparably on the same metric, the headline claim of a 'significant leap' needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim rests on a metric that is never defined. Section 4.1 promises to evaluate 'semantic preservation' with 'multiple complementary metrics,' but no concrete formula is given even for normalized edit distance. Figures 3-5 report 'semantic similarity' distributions, thresholds (0.8, 0.9), and comparisons to traditional decompilers ('typically achieve such scores for only 40-50% of functions', Fig. 4 caption), yet no definition, equation, or reference for 'semantic similarity' appears anywhere. Without knowing whether this is token overlap, a code-embedding cosine, an AST match, or an execution-equivalence oracle, the number 0.82 cannot be interpreted or reproduced. The stakes are concrete: the staking-rewards case study (Section 5.2) shows a decompiled function that is functionally wrong (loses 1e18 scaling and time-weighted reward logic) yet still receives a 'semantic similarity' score of 0.52; a metric that gives 0.52 to a function returning 0 instead of the correct reward is already questionable. The comparison to baselines is likewise unverifiable because no baseline scores are reported in a table with the same metric. This is not a dispute about consensus; it is a missing definition in the paper itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid smart-contract decompiler that first converts EVM bytecode into three-address code (TAC) via static analysis and then uses a LoRA-fine-tuned Llama-3.2-3B model to generate Solidity source. The authors construct a dataset of 238,446 TAC/Solidity function pairs, evaluate on 9,731 held-out functions, and report an average 'semantic similarity' of 0.82 with the original source, together with improved readability over traditional decompilers. They also present two case studies (one successful, one showing substantial semantic loss), an ablation showing the value of fine-tuning, entropy measurements over representations, and two security-audit examples. The main quantitative claim, however, is built on an undefined similarity metric, and the baseline comparison is not described.","tokens_in":15505,"tokens_out":5615,"duration_ms":52719,"significance":"If the quantitative claims survive a properly specified evaluation, the work is a useful contribution: it demonstrates that a small specialized LLM operating on a structured IR can produce readable Solidity from bytecode, and it provides a public dataset, a public system, and an honest failure analysis. The held-out evaluation, the ablation, and the real-world audit examples are strengths. The current bottleneck is evaluation rigor: until the semantic-similarity metric is defined, baseline decompilers are run on the same test set, and behavioral equivalence is checked, the headline 0.82 cannot be taken as evidence of semantic preservation. The approach is nonetheless plausible, and the artifacts make the requested fixes feasible.","major_comments":[{"comment":"The paper's headline result ('average semantic similarity of 0.82') rests on a metric that is never defined. Sections 4.1–4.2 state that 'multiple complementary metrics' are used, and Figures 3–5 report similarity distributions, thresholds, and comparisons, but no formula, implementation, or reference for 'semantic similarity' (or for the 'normalized edit distance') is given. The number cannot be reproduced or interpreted: it is unknown whether it is token overlap, an embedding-based score, an AST match, or an execution-equivalence oracle. I request a precise definition, the computation code, and a worked example; without this, the central quantitative claim is unfalsifiable.","section":"§4.1–4.2, Figs. 3–5"},{"comment":"The claimed advantage over traditional decompilers is not verifiable. The text says traditional decompilers 'typically achieve such scores for only 40% to 50% of functions' (Fig. 4 caption) and exhibit edit distances 'concentrated in the 0.6–0.8 range' (§4.2), but no decompiler names, versions, configuration, or per-tool scores are provided, and no baseline results are reported on the same 9,731-function test set with the same metric. Without this, the comparison is anecdotal. Please run at least one standard smart-contract decompiler on the same held-out functions and report a table of scores under the same metric.","section":"§4.2, Fig. 4"},{"comment":"The staking-rewards case study undermines the validity of the semantic-similarity metric: the decompiled function returns 1 or 0 instead of computing the time-weighted reward, yet it receives a similarity score of 0.52. A metric that scores a behaviorally wrong function this high is not measuring semantic preservation in the security-relevant sense. This does not disprove the approach, but it shows the headline 0.82 must be complemented by an execution-equivalence or behavioral test on representative functions before the paper can claim semantic fidelity.","section":"§5.2, Fig. 9"},{"comment":"The pipeline's upper bound is set by the bytecode-to-TAC stage, which the paper asserts 'preserves the semantic content of the original bytecode' (§3.1) without proof or evaluation. The paper itself documents a failure mode in §5.2 where the TAC-derived output lost 1e18 scaling and time-weighted logic, and §2.3 lists type recovery, storage patterns, and arithmetic precision as known losses during compilation. I ask for an explicit evaluation of the TAC stage (e.g., round-trip execution equivalence between bytecode and TAC, or a manual audit of a random sample) so readers can separate TAC losses from LLM generation errors.","section":"§3.1, §2.3"},{"comment":"The evaluation reports point estimates over 9,731 functions without confidence intervals, and it is not clear whether the holdout was performed at the contract level or the function level. If two functions from the same contract appear in both training and test splits, the model can memorize contract-specific idioms, inflating the reported similarity. Please report bootstrap intervals for the averages and clarify (or implement) a contract-disjoint split.","section":"§4.1"}],"minor_comments":[{"comment":"Figure 3 (left) states that 82.5% of functions have normalized edit distance below 0.4, while Figure 4 (left) says 60% of functions achieve distances under this threshold; these statements are mutually inconsistent and should be reconciled.","section":"Figs. 3 and 4"},{"comment":"The entropy analysis in Section 8 is not load-bearing but is presented without methodological details (tokenizer, corpus subset, estimation method); please add a short description or clearly label it as preliminary and speculative.","section":"§8"},{"comment":"The paper claims the evaluation dataset and metrics calculation code are publicly available, but no URL or repository identifier is given; please provide a permanent link.","section":"§4.1"},{"comment":"The code listings in Figures 10 and 11 contain line-break and arrow artifacts (e.g., '→' and stray commas) that make the examples hard to read; please replace them with clean listings.","section":"Figs. 10 and 11"},{"comment":"The phrases 'for the first time' and 'pioneering' should be qualified, since the related-work section itself lists LLM-based decompilers such as LLM4Decompile and DeGPT; a systematic comparison with these systems is needed before claiming priority.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"The approach is promising and the artifacts appear to exist, but the central metric must be defined and baselines must be run on the same test set. I do not see grounds for rejection; the evaluation gap is substantial but fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the pipeline: static analysis to three-address code, then a fine-tuned Llama-3.2-3B to generate Solidity. That is a sensible division of labor, and the 238,446 paired TAC-Solidity functions are a real contribution. The paper also earns credit for showing a blunt failure case (the staking rewards function) and for the ablation showing the base model is much worse. Those parts are honest and useful.\n\nThe soft spot is the one the stress-test flags, and it is not minor: the paper's central quantitative claim, \"average semantic similarity of 0.82,\" is never defined. Section 4 says \"multiple complementary metrics\" but gives no formula, no reference, no implementation details for semantic similarity. The figures report thresholds and distributions, and the text compares against traditional decompilers, but a reader cannot tell if this is token overlap, embedding cosine, AST matching, or execution equivalence. That makes the headline number unfalsifiable as written. The staking case deepens the worry: a decompiled function that omits the 1e18 scaling and time-weighted reward logic, and returns 0 in the failing branch, still scores 0.52. A metric that calls that 0.52 may be measuring something, but it is not measuring behavioral equivalence in a security-relevant sense.\n\nThe other gaps are in proportion. The traditional-decompiler baselines are described only as \"typically\" achieving 40-50%, with no table of same-metric scores. No error bars or confidence intervals appear for any of the reported numbers. The paper promises public release of the evaluation dataset and metrics code but gives only a website URL, and no model weights or repository link. That is a reproducibility problem for a paper whose main evidence is empirical. The entropy analysis in Section 8 is speculative and could be cut without loss.\n\nNone of this kills the paper. The direction is right, the dataset is useful, and the qualitative case studies show the system can produce auditable code in practice. But the quantitative claims need to be made concrete before the 0.82 number can be trusted. This deserves a serious referee: the authors should be required to define the metric, release the code and data, and report baseline scores with the same metric. I would not cite the 0.82 in its current form, but I would point people at the approach and the dataset.","headline":"A useful hybrid decompiler with a large dataset, but the headline 0.82 similarity rests on a metric the paper never defines; the idea is sound and worth refereeing.","tokens_in":87,"tokens_out":1331,"would_cite":true,"duration_ms":36466,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage pipeline turns EVM bytecode into readable, semantically faithful Solidity.","keywords":["smart contract decompilation","EVM bytecode","three-address code","large language models","Solidity","semantic similarity","security auditing","low-rank fine-tuning"],"falsifier":"Compile a set of verified DeFi functions that use $10^{18}$ fixed-point scaling, nested storage mappings, and time-weighted reward calculations, decompile them with this pipeline, compile the generated Solidity, and run differential tests against the original bytecode on random and adversarial inputs; if behavioral mismatches recur and semantic similarity on that set stays near the 0.52 level the paper reports for its own staking case, the general claim of semantic fidelity fails for a common contract class.","tokens_in":15049,"feed_emoji":"🔓","tokens_out":7408,"duration_ms":73128,"temperature":0.7,"pith_summary":"Ethereum has more than 78 million deployed smart contracts, yet under one percent are source-verified, so security review routinely has to start from raw bytecode. This paper argues that decompilation becomes tractable when split into two stages: static analysis that lifts EVM bytecode into three-address code, followed by a three-billion-parameter language model, fine-tuned on 238,446 function pairs, that rewrites the three-address code as natural Solidity. On a held-out set of 9,731 functions the pipeline achieves an average semantic similarity of 0.82 with the original source, with 78.3% of outputs above 0.8 similarity, and it preserves readability markers that conventional decompilers miss. If the claim holds, auditors, incident responders, and security researchers could read unverified contracts almost as fluently as they read open-source ones, closing the transparency gap on the largest blockchain.","feed_headline":"LLM decompiler turns bytecode into readable Solidity","feed_subtitle":"A two-stage pipeline recovers auditable Solidity from opaque EVM bytecode for unverified contracts.","key_machinery":"The load-bearing object is three-address code (TAC), an intermediate representation in which each instruction has at most three operands and data flow is explicit, for example temp1 = a + b; temp2 = c - d; result = temp1 * temp2. The paper treats TAC as the semantic bridge between bytecode and Solidity: it has intermediate entropy, preserves operand relationships and structured control flow, and gives the language model a translation task rather than a reconstruction task. The second mechanism is low-rank fine-tuning of a three-billion-parameter model; the ablation shows this stage is essential, because the base model's average semantic similarity drops by 45 percent and its output reverts to goto-heavy, generic code.","core_discovery":"The paper's central claim is that combining a structured intermediate representation with a small specialized language model beats both rule-based decompilers and direct neural translation. The bytecode-to-TAC stage does the hard structural work of resolving stack operations, control flow, and function boundaries; the fine-tuned model then supplies what static analysis cannot, namely idiomatic variable names, signatures, and high-level constructs. The reported result is an average semantic similarity of 0.82 to original source and an edit-distance distribution concentrated below 0.4, with security-critical tokens such as require and msg.sender preserved within two percent of their original frequency. Two case studies show the practical effect: one recovers a token-enumeration function nearly unchanged, and another exposes a logic flaw in an unverified token-locking contract that allowed repeated withdrawals before the lock expiry.","pith_inferences":["Because the paper's own staking-reward example scores only 0.52 and loses $10^{18}$ scaling and time-weighted reward logic, I infer that the pipeline is weakest precisely on high-value DeFi math; building a training subset that oversamples fixed-point arithmetic is a direct test and likely remedy.","The entropy argument implies the same TAC-plus-LLM recipe should transfer to other virtual machines with a readable high-level target, such as other EVM-compatible chains or WebAssembly; the paper leaves that as a natural next step rather than a demonstrated result.","A behavioral check, compiling the generated Solidity and differentially testing it against the original bytecode, would convert similarity scores into a guarantee of equivalent execution, which is the property auditors actually need.","The near-perfect frequency match of keywords like require may partly reflect token-level mimicry rather than verified semantics, so numerical constants and storage layouts still deserve manual review."],"forward_implications":["Unverified contracts become practical audit targets: the decompiled output exposes require guards, access-control checks, and function signatures that auditors need to spot vulnerabilities.","The two-stage design decouples structural recovery from language generation, so improvements in static analysis should directly raise the quality ceiling of the neural stage.","Small, carefully fine-tuned models can perform specialized technical translation as well as or better than much larger generic models, reducing the compute needed for decompilation services.","Preserving security-critical tokens at near-original frequencies means automated scanners could run on decompiled code as a first-pass filter for dangerous patterns.","The approach suggests a reusable recipe: lift a low-level bytecode to a structured, low-entropy intermediate form and let a domain-tuned language model render it in the target language."],"supporting_citations":[{"why":"Disclosure of the token-locking vulnerability that motivates the security-auditing use case.","marker":"[1]"},{"why":"Earlier EVM reverse-engineering tool whose readability limits the paper positions its LLM output against.","marker":"[8]"},{"why":"Declarative decompiler that lifts bytecode to three-address code; the representation this pipeline builds on.","marker":"[9]"},{"why":"Advanced traditional decompiler used as a baseline for semantic accuracy and readability comparisons.","marker":"[10]"},{"why":"Nearest LLM-based decompilation baseline, against which the TAC-guided two-stage design is situated.","marker":"[18]"}],"fun_headline_variants":["LLM decompiler turns opaque bytecode into readable Solidity","Two-stage LLM pipeline recovers source-like Solidity from bytecode","Semantic similarity 0.82: LLM decompiler beats traditional tools","Fine-tuned Llama-3.2-3B decompiles EVM bytecode to Solidity","Reading the unreadable: LLM decompiler opens 99% of contracts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conversion from EVM bytecode to three-address code preserves the semantic content of the original bytecode; anything the static-analysis stage drops, such as type information, storage structure, or arithmetic precision, cannot be recovered by the language model.","fun_headline_variants_meta":{"raw":{"variants":["LLM decompiler turns opaque bytecode into readable Solidity","Two-stage LLM pipeline recovers source-like Solidity from bytecode","Semantic similarity 0.82: LLM decompiler beats traditional tools","Fine-tuned Llama-3.2-3B decompiles EVM bytecode to Solidity","Reading the unreadable: LLM decompiler opens 99% of contracts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3856,"prompt_tokens":1007,"completion_tokens":2849,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2745}},"tokens_in":623,"tokens_out":2849,"duration_ms":19225,"temperature":1.0,"reasoning_tokens":2745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:29:59.030981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile a set of verified DeFi functions that use $10^{18}$ fixed-point scaling, nested storage mappings, and time-weighted reward calculations, decompile them with this pipeline, compile the generated Solidity, and run differential tests against the original bytecode on random and adversarial inputs; if behavioral mismatches recur and semantic similarity on that set stays near the 0.52 level the paper reports for its own staking case, the general claim of semantic fidelity fails for a common contract class.","supporting_citations":[{"cited_title":"Dx Protocol Vulnerability Disclosure: Mitigation of a $5,200,000 Smart Contract Exploit,","cited_arxiv_id":null,"evidence_quote":"Disclosure of the token-locking vulnerability that motivates the security-auditing use case."},{"cited_title":"Erays: reverse engineering ethereum’s opaque smart contracts,","cited_arxiv_id":null,"evidence_quote":"Earlier EVM reverse-engineering tool whose readability limits the paper positions its LLM output against."},{"cited_title":"Gigahorse: thorough, declarative decompilation of smart contracts,","cited_arxiv_id":null,"evidence_quote":"Declarative decompiler that lifts bytecode to three-address code; the representation this pipeline builds on."},{"cited_title":"Elipmoc: Advanced decompilation of ethereum smart contracts,","cited_arxiv_id":null,"evidence_quote":"Advanced traditional decompiler used as a baseline for semantic accuracy and readability comparisons."}],"review_version":1}