Intent2Tx shows that LLMs often generate syntactically valid but functionally incorrect Ethereum transactions, especially on multi-step and out-of-distribution intents, despite gains from scaling and retrieval augmentation.
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://anonymous.4open.science/r/bsc_quest_bench-A9CF/.
fields
cs.AI 2years
2026 2verdicts
UNVERDICTED 2representative citing papers
Persona prompting trades expertise depth for reduced clarity in LLM answers and works best on advisory questions in medicine and psychology.
citing papers explorer
-
Intent2Tx: Benchmarking LLMs for Translating Natural Language Intents into Ethereum Transactions
Intent2Tx shows that LLMs often generate syntactically valid but functionally incorrect Ethereum transactions, especially on multi-step and out-of-distribution intents, despite gains from scaling and retrieval augmentation.
-
When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs
Persona prompting trades expertise depth for reduced clarity in LLM answers and works best on advisory questions in medicine and psychology.