{"id":"a6a896da-6c6c-419c-b22b-4938ac91f213","arxiv_id":"2506.15790","paper_version":3,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ETrace proposes detecting smart-contract attacks by having an LLM analyze transaction event logs instead of source code, with a four-example feasibility study only.","lead":"ETrace uses large language models to read the event logs of blockchain transactions and guess whether a smart contract was attacked, without ever looking at the contract's source code. The authors tested it on four known hacks and found the model could describe the likely attack, but the evidence is too thin to call it a working detector.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The experiments cannot validate the effectiveness claim: with only the four known incidents, prompts that define the same four attack patterns, and no negative controls, the LLM's correct labels are consistent with mere prompt-following.","rationale":"The reader correctly identifies the ABI dependency as a real unstated premise, but the more load-bearing gap is that the experiments do not test discriminative power at all. ETrace's own description shows the prompts encode the vulnerability conditions for the four target classes, and the evaluation set contains exactly one positive instance of each class, hand-selected from incidents used to motivate the method. Table 5's outputs are hedged ('could potentially be') and mirror the prompt language, so they demonstrate that an LLM can paraphrase the supplied conditions, not that ETrace detects vulnerabilities in unseen contracts. The admitted false-positive issue in Section 4.2 reinforces this: without a value threshold, the method flags large but legitimate transfer amounts as integer overflows. A balanced benchmark with benign controls would settle whether the method has any actual detection ability. Since the reader already rejected the paper on closely related evidential grounds, my read does not change the verdict; it strengthens the same conclusion by pinpointing prompt circularity rather than only missing ABIs. This is a New Idea paper, so the framework may be worth pursuing, but the stated 'effectiveness' is not currently supported.","tokens_in":7009,"tokens_out":4373,"duration_ms":52516,"concrete_test":"Build a balanced benchmark from Etherscan: the four paper incidents plus additional attack transactions, matched with an equal number of benign transactions from the same protocols that emit the same event names (Transfer, Swap, FlashLoan, Sync, Withdrawal, Approval, repeated lendGM-like calls from normal arbitrage or MEV activity). Run ETrace's exact pipeline and prompts, blinded to true labels, and compute per-class precision, recall, and false-positive rate. If benign traces are flagged as attacks at a rate comparable to attack traces, or if only the four hand-picked incidents are detected, the claimed effectiveness is not discriminative beyond keyword cues already present in the prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ETrace's effectiveness is confirmed by preliminary experiments is unsupported because the evaluation cannot distinguish genuine detection from prompt-following. Section 4.1 uses exactly the four motivating incidents, one per target vulnerability; Section 3.2 states that the prompts embed explicit vulnerability conditions for these same four attack types; and Table 5 reports comprehensive judgments that essentially restate those conditions (e.g., 'The FlashLoan event followed by Swap events ... could potentially indicate a FlashLoanAttack event'). There are no negative controls, no benign transactions, no precision or recall metrics, and no out-of-sample attacks. The one admitted failure in Section 4.2 — that no value threshold caused integer-overflow false positives during reentrancy analysis — confirms that the method over-triggers, yet its rate is not quantified. Under these conditions, correct labels on the four known incidents are equally consistent with (a) ETrace genuinely inferring intent from events and (b) the LLM rephrasing cues already supplied in the prompt. A secondary unstated dependency compounds this: Section 3.1 requires the ABI of each contract address to decode event topics, so the 'without source code' framing actually assumes interface metadata whose availability is not guaranteed for the motivating code-less contracts. The combination leaves the headline effectiveness claim unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ETrace, a framework for detecting smart-contract vulnerabilities without source code access. ETrace extracts and decodes event logs from transaction traces using contract ABIs, feeds the decoded events to an LLM with chain-of-thought prompting and hand-crafted vulnerability conditions for reentrancy, integer overflow, flash-loan attacks, and denial-of-service, and then applies a pattern-matching stage to produce a final judgment. The authors claim that preliminary experiments on four real-world incidents confirm the effectiveness of the approach.","tokens_in":7271,"tokens_out":4308,"duration_ms":45313,"significance":"The problem is timely and the event-centric, code-free detection idea is worth exploring. The paper presents a clear pipeline and is transparent about some limitations, such as the lack of a value threshold in Section 4.2. However, the reported evaluation is too weak to support the central effectiveness claim: it uses only the four motivating incidents, all positive cases, with no negative controls, no benign transactions, no quantitative metrics, and no baselines. Because the vulnerability conditions in the prompts are hand-crafted from the same four incidents, the results are equally consistent with the LLM rephrasing cues already supplied in the prompt as with ETrace genuinely inferring attack intent. The claimed contribution is therefore not established by the current evidence.","major_comments":[{"comment":"The evaluation is limited to exactly the four motivating incidents, one per target vulnerability, and reports only qualitative 'comprehensive judgments' in Table 5. There are no negative cases, no benign transactions, no precision/recall metrics, and no comparison against any baseline detector. The judgments in Table 5 largely restate the vulnerability conditions given in the prompts (e.g., 'The FlashLoan event followed by the Swap events with abnormal price fluctuations could potentially indicate a FlashLoanAttack event'). Therefore the results in Section 4.2 cannot substantiate the Abstract's claim that 'preliminary experiments have confirmed the effectiveness of ETrace'; correct labels on the training-like examples are also consistent with mere prompt-following.","section":"Section 4.1 and 4.2"},{"comment":"The prompt-engineering stage embeds vulnerability conditions for the same four attack types that are motivated in Section 2 and used as the evaluation set in Section 4.1. For example, the reentrancy condition in Section 3.2 is derived from the XSURGE pattern of repeated calls and alternating transfers shown in Table 1. This circular setup means the evaluation cannot show generalization to unseen attacks, and it does not test whether ETrace can detect a vulnerability whose event signature was not already encoded in the prompt.","section":"Section 3.2 and Section 2"},{"comment":"The manuscript admits that, because no value threshold was set, the LLM incorrectly identified integer overflows during reentrancy analysis when values were too high. This is an acknowledged false-positive mode, but its frequency is never quantified, no error analysis is provided, and no mitigation is evaluated. The existence of an unquantified over-triggering behavior further weakens the claim that the method is effective.","section":"Section 4.2"},{"comment":"The data-preprocessing stage decodes event topics 'using the ABI of the corresponding contract address.' This is a load-bearing dependency: for contracts without available source code, the ABI may also be unavailable or must be reconstructed, and the paper does not discuss how ETrace handles such cases. The motivating scenario of code-less contracts therefore relies on an unstated assumption about metadata availability.","section":"Section 3.1"},{"comment":"The pattern-matching stage is described only conceptually as 'a judgment to the pattern analysis results' and is not given an algorithmic specification or an independent evaluation. Table 5 reports only the LLM's comprehensive judgments, so the reader cannot tell what pattern matching adds beyond the LLM output, or whether it is implemented at all.","section":"Section 3.4"}],"minor_comments":[{"comment":"There is a typo in Section 4.2 ('Table .5' should be 'Table 5'), and the caption of Table 5 reads 'Compehensive Judgement' instead of 'Comprehensive Judgment'.","section":"Section 4.2 and Table 5"},{"comment":"References [5] and [13] are the same paper (Liu et al., 'Phantom events'); this duplication should be removed and the citation numbering adjusted.","section":"References"},{"comment":"The phrase 'None code requirement' should be rephrased as a 'no-code requirement' or 'without-source-code requirement' for clarity.","section":"Section 3.3"},{"comment":"The row 'Swap None →None out0=0, out1=0' is unclear: the 'None' addresses and zero-valued outputs need explanation, as they are not self-explanatory from the raw event data.","section":"Table 3"},{"comment":"The sentence 'As a result, the trading volume of DEXs continues to grow steadily' is not logically connected to the preceding sentence about blockchain logs; the transition should be clarified.","section":"Introduction"}],"recommendation":"reject","confidential_remarks":"The main concern is the circularity between the hand-crafted prompt conditions and the evaluation set, combined with the absence of any negative controls or quantitative metrics. These issues are load-bearing for the paper's central claim and cannot be fixed by minor revisions within a four-page paper. I would encourage the authors to resubmit with an independent dataset, adversarial or benign cases, and quantitative evaluation, but the current manuscript does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: ETrace is exactly what it looks like—a four-page New Idea paper with a plausible new combination and an evaluation that cannot back up the effectiveness claim it makes. I'd give the idea a fair hearing at a workshop-style venue, but the authors need to be told the experiments as presented support 'illustrated' not 'validated.'\n\nWhat's new: I agree with the reader's novelty score. Prior work uses events for execution, logging, and testing, and LLMs for code analysis; nobody has pointed LLMs at decoded event traces to infer attack intent. That is a real gap, especially for proxy contracts and third-party dependencies without source code. The pipeline—decode logs, prompt with vulnerability conditions, chain-of-thought analysis, pattern match—is clearly described, and the four real incidents (XSURGE, Beauty Chain, MEVBOT, GovernMental) give it concrete grounding.\n\nWhere it falls down: the stress-test has it right. The vulnerability conditions in Section 3.2 are hand-crafted from the same four incidents used as the entire evaluation set in Section 4.1. Table 5 shows the LLM essentially restating those conditions in hedged language ('could potentially be'). There are no negative controls, no precision/recall, no out-of-sample cases, no baseline comparison. The one admitted failure—the integer-overflow false positive during reentrancy analysis—confirms over-triggering, but the rate is never quantified. On top of that, Section 3.1 requires the ABI to decode topics, which undercuts the 'code-free' selling point when ABIs are absent; the paper doesn't acknowledge that dependency.\n\nWhat's solid: the examples are real, the tables match on-chain events, and the conclusion is honest about the work being preliminary. The citation pattern looks fine—the related work cites recent event and LLM papers and doesn't pad.\n\nBottom line: I wouldn't cite this as evidence for the effectiveness of event-trace LLM detection, and I'd want a blinded evaluation with negatives, released prompts, and an ABI-availability discussion before treating it as more than a proposal. But as a New Idea paper, it articulates a gap and a plausible approach, and a serious referee could help the authors shape it into something testable. So: send it out, but with a clear expectation of major revision.","headline":"ETrace is a plausible new idea for code-free smart-contract vulnerability detection, but the preliminary experiments don't support the effectiveness claim; it deserves a New Idea track referee, not full-paper acceptance.","tokens_in":7788,"tokens_out":2530,"would_cite":false,"duration_ms":27093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ETrace claims that event logs, decoded and interpreted by an LLM, can expose smart-contract attacks even when the contract's source code is unavailable.","keywords":["smart contracts","vulnerability detection","event detection","large language models","transaction logs","chain-of-thought prompting","reentrancy","blockchain security"],"falsifier":"Run ETrace on a held-out set of labeled attack transactions covering the four vulnerability classes, with the ABIs of the involved contracts withheld, and require the final pattern-matching verdict to match the documented incident type; if decoding fails for a material share of inputs or the verdicts match ground truth no better than chance, the central claim collapses.","tokens_in":6807,"feed_emoji":"🔍","tokens_out":8852,"duration_ms":96069,"temperature":0.7,"pith_summary":"The paper is trying to establish that a smart contract's emitted events—the function names, addresses, and values recorded in a transaction's log—carry enough semantic information to reveal attack behavior, even when the contract's source code is not available. ETrace is a framework built on that idea: it decodes transaction-log topics with the contract ABI, asks a large language model to explain each event and then judge the whole sequence, and matches the judgment against four predefined attack patterns. The claim matters because many real-world DeFi attacks run through third-party contracts whose code is inaccessible, which defeats conventional code-based analysis. The authors validate the framework on four documented incidents and report that the LLM judgments align with the actual attack types.","feed_headline":"LLM reads event logs to detect smart-contract attacks without code","feed_subtitle":"ETrace decodes on-chain events and uses an LLM to flag reentrancy, overflow, flash loans, and DoS.","key_machinery":"The central object is the fine-grained event sequence: transaction-log topics are decoded through the contract ABI into triples of function name, address, and value, then sliced by function name. The argument is carried by an LLM used as a semantic interpreter with chain-of-thought prompting: it first explains each event in plain terms, then forms a holistic judgment, and a pattern-matching stage attaches that judgment to one of four predefined attack patterns. This two-step design is what converts raw log entries into an attack verdict, and it is also what produces interpretable output, since every verdict comes with the event-level reasoning behind it.","core_discovery":"On the paper's own terms, the discovery is that fine-grained event sequences extracted from transaction logs can be semantically interpreted to reconstruct the intent of a smart-contract transaction without reading its code. Each event is reduced to a function name, an address, and a value; a large language model, prompted with the characteristics of known attacks and asked to reason step by step, produces an event-by-event explanation followed by a comprehensive judgment in the form Summary–Pattern Analysis–Further Recommendation. A final pattern-matching stage checks that judgment against hand-specified patterns for reentrancy, integer overflow, flash-loan attacks, and denial of service. The paper reports that this pipeline identified the correct attack type in all four real-world case studies it examines.","pith_inferences":["An implication the paper leaves implicit: the 'no code' promise is conditional on ABI availability, since topics are decoded with the ABI of each involved contract; extending ETrace to contracts without public ABIs would require recovering interface metadata from bytecode or registries, which is a testable next step.","The reported confusion between large values in reentrancy traces and integer overflow suggests that adding value-range thresholds or cross-event consistency checks would sharpen the LLM's judgments, and this can be measured on the same four incidents.","If the approach scales, event-trace analysis could serve as a forensic layer: given a transaction hash, an auditor would get a structured attack narrative without needing the contract code or a live chain.","Because events are data that contracts choose to emit, a deployment should cross-check event authenticity against state-transition evidence; otherwise forged or spoofed logs could mislead the LLM, a risk the paper's own cited work on phantom events raises."],"forward_implications":["Transactions emitting telltale event sequences can be flagged as reentrancy, integer overflow, flash-loan, or denial-of-service attacks even when the contract's source code is unpublished.","Security monitoring becomes possible from on-chain logs alone, so attacks that pass through third-party dependencies such as oracles, bridges, or unaudited DEX components become visible to the detector.","Every verdict is interpretable: the chain-of-thought output narrates the events and the pattern analysis explains how the attack unfolded.","Because the input is a transaction hash and recorded logs, the method can be applied retroactively to historical incidents without re-executing the contract."],"supporting_citations":[{"why":"Shows event information is the core communication mechanism for triggering contract logic, motivating logs as a semantic signal.","marker":"[4]"},{"why":"Analyzes phantom-event issues such as log forgery and contract impersonation, defining the reliability concerns for log-based reasoning.","marker":"[5]"},{"why":"Uses contract log files to test smart-contract behavior against expected outputs, supporting the use of event data for behavioral analysis.","marker":"[6]"},{"why":"Provides efficient-logging results for blockchain applications, cited as evidence that event information in transaction logs is valuable.","marker":"[7]"},{"why":"Demonstrates LLM-based analysis of smart-contract vulnerabilities with static analysis, the direction ETrace extends to code-free traces.","marker":"[15]"},{"why":"Applies LLM reasoning to detect imminent attacks from adversarial contracts, motivating semantic interpretation of event behavior.","marker":"[16]"}],"fun_headline_variants":["No code? No problem: ETrace spots contract attacks via logs","ETrace reads transaction events to flag smart contract exploits","LLM decodes on-chain logs to catch smart contract vulnerabilities","ETrace leverages LLM to find contract flaws from event traces"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline depends on having an ABI for every contract involved in a transaction, because event topics are decoded through that ABI into the function names and values the LLM analyzes; without this interface metadata, the event sequence the whole method relies on cannot be built.","fun_headline_variants_meta":{"raw":{"variants":["No code? No problem: ETrace spots contract attacks via logs","ETrace reads transaction events to flag smart contract exploits","LLM decodes on-chain logs to catch smart contract vulnerabilities","ETrace leverages LLM to find contract flaws from event traces"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2856,"prompt_tokens":839,"completion_tokens":2017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1959}},"tokens_in":455,"tokens_out":2017,"duration_ms":16040,"temperature":1.0,"reasoning_tokens":1959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:50:55.679795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ETrace on a held-out set of labeled attack transactions covering the four vulnerability classes, with the ABIs of the involved contracts withheld, and require the final pattern-matching verdict to match the documented incident type; if decoding fails for a material share of inputs or the verdicts match ground truth no better than chance, the central claim collapses.","supporting_citations":[{"cited_title":"An event driven framework for smart contract execution,","cited_arxiv_id":null,"evidence_quote":"Shows event information is the core communication mechanism for triggering contract logic, motivating logs as a semantic signal."},{"cited_title":"A novel method based on log files for smart contract testing,","cited_arxiv_id":null,"evidence_quote":"Uses contract log files to test smart-contract behavior against expected outputs, supporting the use of event data for behavioral analysis."},{"cited_title":"Efficient Logging for Blockchain Applications","cited_arxiv_id":"2001.10281","evidence_quote":"Provides efficient-logging results for blockchain applications, cited as evidence that event information in transaction logs is valuable."},{"cited_title":"Advscanner: Generating adversarial smart contracts to exploit reentrancy vulnerabilities using llm and static analysis,","cited_arxiv_id":null,"evidence_quote":"Demonstrates LLM-based analysis of smart-contract vulnerabilities with static analysis, the direction ETrace extends to code-free traces."},{"cited_title":"Skyeye: Detecting imminent attacks via analyzing adversarial smart contracts,","cited_arxiv_id":null,"evidence_quote":"Applies LLM reasoning to detect imminent attacks from adversarial contracts, motivating semantic interpretation of event behavior."}],"review_version":1}