{"id":"7cf33528-1a39-4bef-b842-82b7f1bb918c","arxiv_id":"2507.23087","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An open benchmark shows top LLMs generate functionally correct smart contracts from BPMN process models in about 80 to 92 percent of checks, below the reliability needed for blockchain deployment.","lead":"The authors built an automated benchmark that asks seven large language models to turn business process diagrams into Solidity smart contracts, then checks each contract by replaying correct and incorrect execution traces. The best model scored about 92 percent on this check, a result the paper argues is still not reliable enough for blockchain contracts that hold real money.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1 headline rests on hand-filtered 'non-conforming' traces whose counts and removal procedure are unreported; if mislabeled traces remain, the claimed correctness levels (0.918/0.862/0.797) are not reproducible from the paper alone.","rationale":"The central empirical contribution is the automated benchmark and the Table 2 F1 numbers. The paper itself acknowledges a manual step inside the supposedly automated oracle: Section 5.3 says accidentally conforming non-conforming traces must be manually removed, but no counts, procedure, or agreement metrics are given. This is exactly the kind of self-admitted limitation that the review rules require flagging. The concern is load-bearing because every precision/recall/F1 value is computed relative to trace labels, and those labels are the ground truth for functional correctness. If the negative labels are noisy, the headline numbers are not reproducible, and the paper's stated contribution of an 'automated evaluation framework' is weakened because the oracle is not fully automated. The qualitative conclusion that LLMs are not reliable enough for production blockchain contracts is likely robust to this issue, since even a substantially corrected F1 would still be below 100 percent and the paper's policy argument does not depend on exact digits. However, the exact ranking and the 'most of the time' phrasing do depend on the oracle. A concrete re-validation of the retained non-conforming traces via alignment-based conformance checking would settle whether this is a reporting gap or a numerical validity threat. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":11783,"tokens_out":6554,"duration_ms":81078,"concrete_test":"Download the released execution logs and trace files from the archived repository (doi 10.5281/zenodo.16616694), identify every non-conforming trace that was retained in the final benchmark for each of the 165 models, and re-validate it against the Chorpiler interaction net using an automated alignment-based conformance check (e.g., pm4py alignments) instead of manual inspection. Also count, from the logs or reconstruction, how many generated non-conforming traces were removed per model. Then recompute macro-F1 for each row of Table 2 with only verified non-conforming traces. If the recomputed F1 values move by more than about 0.02, or the removal counts are large relative to the 50 negative traces per model, the reported performance magnitudes are materially affected; if they move only within sampling noise, the concern is a reporting gap rather than a validity threat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is an empirical statement: top LLMs produce functionally correct contracts 'most of the time,' with macro-F1 up to 0.918. These F1 scores are only as good as the trace oracle. Section 5.3 states that for models whose conforming-trace set exceeds the 2,500-trace search threshold, the non-conforming trace generator can produce traces that are actually conforming, and that '[t]hese must be manually removed.' The paper reports neither how many traces were removed, per model or in total, nor how the manual removal was performed, nor any reliability check on that step. If any accidentally conforming trace remained in the negative set, a contract that correctly accepts it is counted as a false positive, deflating precision and F1; conversely, if a contract rejects such a mislabeled trace, the rejection is counted as a true negative, inflating recall. The two errors do not cancel, and their net direction depends on model behavior. Because Table 2 reports no confidence intervals or repeated runs, the exact differences among grok-3-beta (0.918), claude-sonnet-4 (0.862), and gpt-4.1 (0.797) cannot be separated from oracle-label noise. The qualitative conclusion that LLMs are not perfectly reliable would likely survive, but the specific claim of 'syntactically and functionally correct most of the time' is not independently verifiable from the paper as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an open-source benchmarking framework that evaluates LLMs' ability to generate Solidity smart contracts from BPMN 2.0 choreography models. Using 165 pre-processed models from the SAP-SAM dataset, the framework generates conforming and non-conforming execution traces, prompts seven LLMs (proprietary and open-source) with one-shot and two-shot variants, compiles and deploys the generated contracts in a simulated EVM, replays the traces, and reports macro-F1 and compilability. The headline results are that the best models reach macro-F1 of 0.918 (grok-3-beta), 0.862 (claude-sonnet-4), and 0.797 (gpt-4.1) in the one-shot setting, with high compilability. The authors conclude that current LLMs generate correct contracts only 'most of the time' and, because blockchain applications require perfect reliability, recommend future work on integrating LLMs into rule-based generation tools rather than using LLM output directly.","tokens_in":12114,"tokens_out":7013,"duration_ms":80096,"significance":"If the results hold, this is a useful contribution: it is, to my knowledge, the first open, large-scale benchmark for LLM-based smart contract generation from process models, and it shifts evaluation from compilability-only checks to functional trace replay. The framework is concrete, the artifact is publicly available with an archived version, and the use of an independent trace oracle rather than LLM self-assessment is methodologically sound. The paper also provides cost, token, and gas data, which is valuable for practitioners. However, the central quantitative claims rest on the quality of the trace oracle and on a single run per configuration; the current manuscript does not report enough detail to make the headline F1 values reproducible or to separate them from oracle-label noise and stochastic variation.","major_comments":[{"comment":"The trace-generation footnote states that for models whose conforming-trace count exceeds the 2,500-trace search threshold, non-conforming traces that are actually conforming can be generated and 'must be manually removed.' The paper reports neither how many traces were removed (per model or in total), nor how the manual removal was performed, nor any inter-rater reliability check. This is load-bearing because these traces form the negative oracle: if a mislabeled trace remains, a correct contract that accepts it is scored as a false positive, deflating precision and F1, while a contract that incorrectly rejects it is scored as a true negative, inflating measured performance; the net effect depends on model behavior. Without these statistics, the headline F1 values in Table 2 are not independently reproducible from the paper alone.","section":"§5.2, footnote 15"},{"comment":"Each model and prompt variant is evaluated in a single run with temperature set to 0, which the paper itself describes as only 'quasi-deterministic.' No confidence intervals, repeated runs, or per-case score distributions are reported. Given the paper's own citation that LLM outputs are non-deterministic [30], the differences among grok-3-beta (0.918), claude-sonnet-4 (0.862), and gpt-4.1 (0.797), as well as the 'most of the time' quantification, cannot be separated from run-to-run variation. Please report repeated runs, variance measures, or at least the distribution of per-process F1 scores.","section":"§5.2, Table 2"},{"comment":"The pre-processing step inserts synthetic boolean conditions for all non-default outgoing XOR flows and arbitrarily sets a default flow when none is marked. This means that the evaluation of the 'data-based conditions' property, and the claim of a 'realistic and diverse dataset,' are based on models containing artificial decision logic rather than the original or recovered conditions. This limits the external validity of the F1 scores for real-world process models. The paper should state how many models had original conditions, discuss the impact of the synthetic conditions, and ideally report results separately for models with and without original gateway data.","section":"§5.1"},{"comment":"The prompts were refined in pre-runs conducted on sets of five to twenty process models, but the paper does not state whether those pre-run models were excluded from the final 165-model benchmark set. If there is overlap, the reported F1 values could be optimistically biased by prompt selection on the evaluation cases. Please clarify the disjointness of the pre-run and evaluation sets, or confirm that no pre-run model is included in the benchmark sample.","section":"§5.2"}],"minor_comments":[{"comment":"The classification framework defines 'accepted' and 'rejected' traces but does not specify how the replay distinguishes between a transaction that reverts (and is therefore rejected) and a transaction that executes without a state change. Please clarify the replay semantics.","section":"§5.3"},{"comment":"The statement that the reliability issue 'cannot be resolved with current LLM architectures' is broader than the evidence from seven models and one prompting scheme; consider rephrasing to 'in our experiments' or 'based on current evidence' to avoid overgeneralization.","section":"§6"},{"comment":"Reporting F1 macro to three decimal places suggests a precision that is not supported by the single-run, manually filtered evaluation; consider rounding or adding variance measures.","section":"Table 2"},{"comment":"The phrase 'most of the time' is used without a quantitative definition; please tie it to the F1 values or define an explicit threshold.","section":"§1 and §6"}],"recommendation":"major_revision","confidential_remarks":"The core benchmarking framework is a solid contribution and the artifact availability is a strength. The main empirical claims are currently not fully verifiable because of the unreported manual trace-filtering step and the absence of repeated runs. These issues appear fixable within the manuscript's scope, so I recommend major revision rather than rejection. I do not see a novelty or scope concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first open, execution-based benchmark for turning BPMN choreographies into smart contracts, and it is mostly well done. The authors ship a framework, data, and results across 165 SAP-SAM models and seven LLMs. The headline macro-F1 scores (0.918 for grok-3-beta, 0.862 for claude-sonnet-4, 0.797 for gpt-4.1) are the main event. They are believable as order-of-magnitude measurements, but I would not treat the exact numbers as final until a couple of reporting gaps are closed.\n\nWhat is genuinely new: prior work used small samples, manual inspection, or compile checks. This paper evaluates functional correctness by replaying conforming and non-conforming traces against the generated contracts. It also releases everything. That is real evidence and a useful community asset.\n\nThe core qualitative conclusion—that today's LLMs are not reliable enough for production smart contracts in an unforgiving blockchain environment—holds up. Even if the F1 numbers move by a few points under more careful measurement, the gap to 100% is not close. The paper's suggestion to use LLMs inside a verifiable toolchain is sensible.\n\nSoft spots, in proportion: Section 5.3 admits that non-conforming traces that are actually conforming must be manually removed, but does not report how many were removed, how the removal was done, or any reliability check. That is a real gap. If mislabeled traces remained, the F1 values could be deflated or inflated in ways that are not predictable from the paper alone. Also, the runs are single-shot at temperature 0; the authors acknowledge quasi-determinism, but no confidence intervals or repeated runs are reported. The synthetic boolean conditions for XOR gateways (Section 5.1) are a simplification, though necessary given the source data. One more thing I would want clarified: whether the 5-20 model pre-runs used for prompt tuning overlap with the final 165. The paper does not say, and if they overlap, that is a minor contamination.\n\nNone of this changes the central argument. The authors are transparent about the manual step, and the framework is designed for reuse. I would accept this for peer review with the expectation that the authors disclose the trace-removal counts and ideally some variance estimates.\n\nFor you: worth bringing to a reading group if you care about empirical evaluation methodology for LLM code generation or blockchain process execution. I would cite it if I were working in this area.","headline":"A genuinely useful open benchmark for LLM-to-smart-contract generation, with solid-if-not-yet-fully-reported numbers that support the paper's core conclusion.","tokens_in":12627,"tokens_out":2229,"would_cite":true,"duration_ms":26219,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that current LLMs can turn BPMN choreographies into mostly correct smart contracts, yet argues that anything below 100% correctness is unsafe for blockchains.","keywords":["LLM code generation","smart contracts","BPMN choreographies","process conformance","blockchain-based process execution","trace replay benchmarking","Solidity","automated evaluation"],"falsifier":"Take one of the 165 process models, replace the 2,500-trace sampling cap with exhaustive enumeration of every possible execution path, and rerun the one-shot grok-3-beta benchmark: if any contract that scored F1 1.0 accepts a non-conforming trace that the larger trace set can now generate, or if the reported F1 drops, the paper's headline reliability estimate is too optimistic.","tokens_in":11603,"feed_emoji":"🤖","tokens_out":7330,"duration_ms":83688,"temperature":0.7,"pith_summary":"This paper establishes an automated, open benchmark for a specific task: given a BPMN 2.0 choreography model, can an LLM produce a Solidity smart contract that actually executes the process correctly? The authors run 165 realistic process models through seven LLMs, in one-shot and two-shot settings, and check each generated contract by replaying conforming and non-conforming traces against it. The best model, grok-3-beta, reaches an F1 macro of 0.918 with 100% compilability; Claude Sonnet 4 reaches 0.862 and GPT-4.1 0.797. Because a blockchain cannot tolerate a 2% error rate, let alone 8-20%, the paper concludes that standalone LLM generation falls short of the perfect reliability required, and recommends integrating LLMs into rule-based tools with formal verification rather than replacing them.","feed_headline":"Best LLM gets 92% of smart contracts right; that's still not enough","feed_subtitle":"Top models compiled almost always on 165 process models, yet missed execution traces too often for immutable blockchains.","key_machinery":"The load-bearing mechanism is a trace-replay conformance oracle. The framework converts a BPMN choreography into an interaction net via Chorpiler, uses pm4py's playout routine to generate up to 2,500 conforming traces plus 50 manipulated non-conforming traces per process, then compiles and deploys the LLM-generated Solidity contract on a local EVM and replays the traces: a conforming trace must lead to the end event, and a non-conforming trace must be rejected somewhere. Precision, recall, and F1 macro are computed from that replay for each process, with a bitmask-encoding prompt pushing the LLM toward token-based execution. This oracle is what turns the question from 'does it compile?' into 'does it behave correctly?'.","core_discovery":"The central claim, stated on the paper's own terms, is a measured capability gap. Current LLMs can transform executable choreography models into syntactically and functionally correct smart contracts most of the time, even on a diverse and realistic dataset, but not reliably enough for deployment. Concretely, the top-performing model achieves F1 macro 0.918 and 100% compilability in the one-shot setting, while open-source models lag behind; and because public blockchains are unforgiving, immutable, and exposed to exploit, the authors argue that scores below a reliable 100% are not suitable. They further state that this is a fundamental issue with current LLM architectures, not merely a prompt-tuning problem, and they show that two-shot prompting does not consistently improve results.","pith_inferences":["I infer the benchmark's F1 numbers are probably conservative: the 2,500-trace cap and the manual removal of accidentally conforming traces mean the oracle covers only a subset of a model's behavior, so a contract that passes the sampled traces could still violate an unsampled execution path.","A testable extension would be to run the same framework with exhaustive reachability on small models; if F1 drops, the reported gap is even larger than the paper states.","The paper's own limitation note implies the manual trace-removal step is a source of measurement noise, and quantifying inter-rater agreement would let future work attach confidence intervals to the F1 scores.","I infer the framework could be repurposed to localize errors by reporting which construct (gateway, participant, or data condition) first fails on a trace, turning the benchmark from a score into a diagnostic for prompt or template design."],"forward_implications":["Top proprietary LLMs generate contracts that compile almost always and pass the trace oracle in roughly 80-92% of checks on average per process.","Two-shot prompting does not consistently beat one-shot prompting, so simply adding examples is not a reliable route to production-quality contracts.","For blockchain process enactment, even a hypothetical 98% average F1 is presented as inadequate, so the required bar is 100% trace conformance rather than high accuracy.","A viable direction is to use LLMs to propose code or tests, then vet the output with formal specifications or theorem provers before deployment.","The open framework and data allow repeated benchmarking of future model versions and of hybrid rule-based/LLM tools."],"supporting_citations":[{"why":"Defines blockchain-based process enactment and the replay of conforming and non-conforming traces as the correctness test for generated smart contracts; the benchmark's oracle method rests on it.","marker":"[39]"},{"why":"Supplies the SAP Signavio Academic Models dataset from which the 165 BPMN choreography models are sampled.","marker":"[37]"},{"why":"Introduces Chorpiler, the tool that is extended with simulation capabilities to transform choreographies into contracts and generate trace encodings.","marker":"[40]"},{"why":"Provides the pm4py playout functionality used to generate conforming traces from the interaction net.","marker":"[6]"},{"why":"Defines the transactional and autonomous-gateway execution semantics that the generated smart contracts are expected to implement.","marker":"[24]"},{"why":"Motivates the bitmask token-based encoding that the prompt asks the LLM to use for efficient contract state.","marker":"[16]"},{"why":"The directly related prior case study on BPMN-to-smart-contract LLM generation, evaluated on a single process; the paper contrasts its larger benchmark against this baseline.","marker":"[15]"}],"fun_headline_variants":["92% smart contract accuracy still too low for blockchains","Best LLM gets most contracts right, but not all: unsafe for chain","LLM smart contract code: compiles often, executes rarely correctly","Immutability demands 100%: LLMs achieve 92% F1 max"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on whether the generated test traces are a correct ground truth: if the trace suite is wrong or incomplete, a correct contract can be marked incorrect, or an incorrect contract can look correct.","fun_headline_variants_meta":{"raw":{"variants":["92% smart contract accuracy still too low for blockchains","Best LLM gets most contracts right, but not all: unsafe for chain","LLM smart contract code: compiles often, executes rarely correctly","Immutability demands 100%: LLMs achieve 92% F1 max"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":1914,"prompt_tokens":903,"completion_tokens":1011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":932}},"tokens_in":519,"tokens_out":1011,"duration_ms":11102,"temperature":1.0,"reasoning_tokens":932,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:04:02.365024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the 165 process models, replace the 2,500-trace sampling cap with exhaustive enumeration of every possible execution path, and rerun the one-shot grok-3-beta benchmark: if any contract that scored F1 1.0 accepts a non-conforming trace that the larger trace set can now generate, or if the reported F1 drops, the paper's headline reliability estimate is too optimistic.","supporting_citations":[{"cited_title":"In: Process Mining Workshops at ICPM 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the SAP Signavio Academic Models dataset from which the 165 BPMN choreography models are sampled."},{"cited_title":"In: BPM, vol","cited_arxiv_id":null,"evidence_quote":"Defines the transactional and autonomous-gateway execution semantics that the generated smart contracts are expected to implement."},{"cited_title":"In: BPM, pp","cited_arxiv_id":null,"evidence_quote":"Motivates the bitmask token-based encoding that the prompt asks the LLM to use for efficient contract state."},{"cited_title":"IEEE Software (2025)","cited_arxiv_id":null,"evidence_quote":"The directly related prior case study on BPMN-to-smart-contract LLM generation, evaluated on a single process; the paper contrasts its larger benchmark against this baseline."}],"review_version":1}