{"id":"78bedbe5-c736-45b4-8d93-db6281ecd808","arxiv_id":"2411.11469","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Deegen automatically generates a two-tier VM (interpreter plus baseline JIT) from C++ bytecode semantics; a Lua 5.1 VM built with it runs 179% faster than the reference interpreter and approaches LuaJIT.","lead":"This paper presents Deegen, a tool that automatically generates a fast interpreter and a just-in-time compiler for a programming language from simple C++ descriptions of what each bytecode does. It could let language developers build high-performance VMs without hand-writing assembly, as demonstrated by a Lua VM that beats the reference interpreter.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The omitted correctness argument for the type-based optimization (Section 5.1, Algorithm A) is the linchpin risk; a differential fuzz test against an unoptimized reference can settle it.","rationale":"The stress-test pass identified the correctness of Algorithm A (Section 5.1) as the single most load-bearing concern. The paper's headline claims—automatic generation of a JIT-capable VM, interpreter outperforming state of the art, baseline JIT near optimizing-JIT performance—all presuppose that the generated VMs execute user programs correctly. Algorithm A is the mechanism by which Deegen removes or strength-reduces type checks and splits hot/cold paths; if it is unsound, Deegen-generated VMs can silently miscompile. The paper explicitly says 'We omit an argument of correctness due to space,' which is a red flag precisely because the analysis is non-trivial: it runs SCCP under |T|^n substituted type assignments and unions per-block type sets. I examined the algorithm and could not exhibit a concrete unsoundness under the stated model (operands are immutable boxed values, type checkers are pure predicates on type membership). However, the paper neither states nor enforces the purity contract, and the fixed-point computation's over-approximation property is not proven. Given that the entire performance evaluation rests on LJR, a differential test against PUC Lua or an unoptimized reference would be the decisive check. Other concerns—lack of error bars, single-machine benchmarks, baseline-vs-optimizing JIT comparison—are real but secondary: the performance deltas (179%, 360%, 33%) are large enough that modest noise would not overturn them, and the paper already frames the LuaJIT comparison as not apple-to-apple. The engineering-cost claim is supported by the artifact and the 1500-LLOC LJR implementation. Therefore, the reader's CONDITIONAL verdict is appropriate; I recommend UNCHANGED, with the concrete test above as the condition for full acceptance.","tokens_in":52257,"tokens_out":9357,"duration_ms":95846,"concrete_test":"Implement a differential fuzzer for LJR: for each bytecode variant using EnableTypeBasedCodeSplitting, sample operand values covering all base-type combinations (NaN, non-double boxed values, nil, strings, tables, functions, boundary constants), run the generated fast/slow paths with their runtime guards, and compare outputs/state changes against a reference compiled from the original unoptimized C++ semantics. Also run a unit-level test of Algorithm A on small random functions with random predicates p, checking that A(f,p) is observationally equivalent to f on inputs satisfying p. If any mismatch appears, the omitted correctness argument is a real bug, not just a missing proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Deegen automatically generates a correct two-tier VM from C++ bytecode semantics—rests on the soundness of Algorithm A in Section 5.1, which removes or strength-reduces type checks and splits fast/slow paths. The paper explicitly states 'We omit an argument of correctness due to space' (Section 5.1, page 10). This is not merely a stylistic gap: A works by replacing user-supplied type-check API calls with true/false constants and running SCCP to compute per-block possible type sets M(b,i). Soundness requires (i) each type checker c in ⟨S,c,d,e⟩ is a pure function of the operand's type with no side effects or dependence on global state, and (ii) the type hierarchy is a faithful abstraction of the boxing scheme. Neither requirement is verified or even stated as an API contract. If a user writes a checker that, e.g., consults a hidden-class cache or has a side effect, replacing calls with constants during SCCP changes program behavior, and the resulting fast/slow paths can silently execute wrong code while still being fast. Even under the purity assumption, the paper gives no proof that the SCCP-based fixed-point computation of M(b,i) over-approximates reachable types; a subtle bug here (e.g., in undoing modifications, or handling of type checks on non-operand values) would also produce incorrect code. Since the performance claims are all measured on LJR, whose correctness is asserted but not differentially tested against PUC Lua in the paper, the absence of a correctness argument for the central optimization is the most load-bearing risk.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Deegen, a build-time meta-compiler that takes C++ descriptions of bytecode execution semantics and automatically generates a two-tier VM consisting of a register-pinned, continuation-passing interpreter and a Copy-and-Patch based baseline JIT, together with profiling, tier-up, and OSR-entry logic. The user writes bytecode semantics as C++ functions using Deegen APIs, and Deegen performs LLVM-IR-level transformations for type-based optimization, inline caching, tag-register optimization, and other dynamic-language optimizations. The paper validates the approach with LuaJIT Remake (LJR), a Lua 5.1 VM, reporting interpreter performance 179% faster than PUC Lua and 31% faster than LuaJIT's interpreter, and baseline JIT performance 360% faster than PUC Lua and 33% slower than LuaJIT's optimizing JIT, with JIT compilation throughput of 1.62 GiB/s.","tokens_in":52533,"tokens_out":2727,"duration_ms":30677,"significance":"If the claims are correct, Deegen is a substantial advance in VM construction: it offers a single source of truth for bytecode semantics and automatically produces an interpreter and a baseline JIT whose generated assembly is competitive with hand-written VM code. The paper provides concrete evidence in the form of real disassembly listings, 44 benchmarks, an open-source artifact, and direct performance measurements rather than fitted models. The main substantive gap is the absence of a correctness argument for the type-based optimization algorithm in Section 5.1, which is load-bearing for the central claim that Deegen automatically generates a correct VM.","major_comments":[{"comment":"The paper states 'We omit an argument of correctness due to space' immediately after describing Algorithm A, which replaces type-checker calls with true/false constants under SCCP and then splits code into fast and slow paths. This is a load-bearing omission: the generated VM's correctness depends on (i) each user-supplied type checker being a pure predicate over the operand's type with no side effects or hidden state, and (ii) the user-declared type hierarchy faithfully modeling the actual boxing scheme. Neither requirement is stated as an API contract or verified by the framework. A type checker that, for example, consults a hidden-class cache or an environment flag would change behavior when replaced by a constant, and a wrong type lattice could silently produce incorrect code on the fast path. The paper should either provide a formal soundness statement for Algorithm A under explicit assumptions about the type-checker API, or supply a differential fuzz test that compares executions of the optimized and unoptimized VM on a broad set of programs; without one of these, the 'automatically generated correct VM' claim is not supported.","section":"§5.1, Algorithm A"},{"comment":"The evaluation measures performance but never verifies that LJR produces the same observable results as PUC Lua or LuaJIT on the 44 benchmarks. Since the paper claims LJR is 'standard-compliant' and uses that compliance to argue for Deegen's correctness, the absence of any output comparison or conformance testing is a gap. Even if Algorithm A were proven correct, the implementation could still have lowering bugs in the interpreter, JIT, or inline-cache machinery. Adding a differential test harness that runs the same benchmarks under LJR and PUC Lua and compares outputs would directly address this concern and is feasible within the manuscript's scope.","section":"§8, Evaluation"},{"comment":"Even setting aside the proof, the paper should specify the exact obligations of the user when defining type checkers and strength-reduction rules. The current text describes syntactic tuples ⟨S,c,d,e⟩ but does not state that c must be a pure function of the boxed value's type, nor that d/e must be inverses on the corresponding domain, nor that the type hierarchy must be a partition refinement of the concrete value representation. Without these contracts, the 'user-provided cost estimation' and rule selection in Algorithm A are not enough to guarantee that a chosen strength reduction preserves semantics. This is fixable by adding a short 'soundness requirements' subsection or a formal lemma.","section":"§5.1, type-checker API contract"}],"minor_comments":[{"comment":"Performance numbers are reported as averages of three runs on a single machine with no error bars, variation, or per-run data; given the small run count, adding at least min/max or standard deviation would make the headline comparisons more robust.","section":"§8.3"},{"comment":"Algorithm A runs SCCP |T|^n times, where n is the number of bytecode operands; the paper notes this is acceptable at build time, but it would be useful to report the actual build-time cost for the 42 bytecode definitions in LJR, since this affects the 'engineering cost similar to a simple interpreter' claim.","section":"§5.1, Algorithm A complexity"},{"comment":"The lines-of-code comparison would be clearer if it stated whether the LJR LLOC count includes the user-written parser and standard-library stubs, since those are part of the total engineering effort of building a VM with Deegen.","section":"§8.1, Figure 25"},{"comment":"The disassembly figure for GetById is dense and the labels are small; a high-resolution version or an annotated walkthrough would help readers verify the claims about self-modifying code and inline slabs.","section":"§7.1, Figure 18"}],"recommendation":"major_revision","confidential_remarks":"The paper is strong on the systems side and the artifact is a significant asset. The main risk is correctness: the omitted proof in §5.1 and the lack of differential output testing in §8 leave the possibility of silent miscompilation. I would be comfortable with acceptance after the authors add either a soundness argument under explicit API contracts or a fuzz-based differential test, and ideally both. The performance claims themselves are plausible and well-evidenced by disassembly and benchmark measurements."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Deegen is the real thing: the first system I know that statically generates both a state-of-the-art interpreter and a baseline JIT from C++ bytecode semantics, and the LJR evaluation backs it up with real disassembly, 44 benchmarks, and an open artifact. The Copy-and-Patch extension for polymorphic IC, inline slab, and hot-cold splitting is a genuine advance over the original Copy-and-Patch paper. The interpreter numbers (179% vs PUC Lua, 31% vs LuaJIT's interpreter) are plausible from the disassembly; the baseline JIT's 1.62 GiB/s compile throughput makes the negligible startup claim credible.\n\nThe soft spots are real but not fatal. The one that matters is in Section 5.1: Algorithm A, which removes or strength-reduces type checks based on the user-described type hierarchy, is presented without a correctness argument. The paper literally says \"We omit an argument of correctness due to space.\" The transformation is safe only if the user's type checkers are pure functions of the operand type and if the type hierarchy is a faithful abstraction of the boxing scheme. Neither is stated as an API contract. If those assumptions fail, the generated fast/slow paths can silently compute wrong results while still being fast. That is structurally different from the speed claim and is the load-bearing risk. The good news is it's empirically checkable: the artifact is public, LJR is a standard-compliant Lua 5.1 VM, and differential fuzzing against an unoptimized interpreter or PUC Lua would exercise exactly this transformation. A referee should require that test, plus either a proof sketch of the SCCP-based over-approximation or a crisp statement of the assumptions.\n\nMinor concerns: the benchmarks are averages of three runs on one laptop without error bars, and the baseline JIT is compared against LuaJIT's optimizing JIT, which is apples-to-oranges. The authors acknowledge the second point themselves, and the 13/44 JIT wins are driven by inline caching that LuaJIT lacks, which is a fair comparison of high-level design choices if not of JIT tiers.\n\nWho this is for: anyone building dynamic language VMs, and anyone working on compiler generators or meta-compilation. It deserves a serious referee. The omitted correctness argument should be addressed in revision, but the core contribution is real.","headline":"First static generation of a competitive interpreter and baseline JIT from bytecode semantics, with strong empirical evidence; the omitted correctness argument for type-check elimination is the main gap, and it is testable.","tokens_in":53117,"tokens_out":2807,"would_cite":true,"duration_ms":25352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Deegen claims that a two-tier VM with a state-of-the-art interpreter and baseline JIT can be generated automatically from C++ bytecode semantics, and backs the claim with a Lua 5.1 implementation that beats the PUC Lua interpreter by 179%.","keywords":["Deegen","dynamic language VMs","bytecode semantics","meta-compiler","inline caching","baseline JIT","type-based optimization","Copy-and-Patch"],"falsifier":"Take LJR's Add bytecode, which splits on a speculation that both operands are tDoubleNotNaN, and execute it with one operand a NaN boxed as an impure NaN or a table; if the generated fast path produces a result different from PUC Lua's rather than transferring to the slow path, Algorithm A or the user-supplied type description is unsound. A systematic version is to mutate one strength-reduction rule in the type-hierarchy description and rerun the 44-benchmark suite against PUC Lua, since any divergence in outputs would falsify the automatic-optimization claim independently of the speed numbers.","tokens_in":2050,"feed_emoji":"⚙️","tokens_out":2596,"duration_ms":98417,"temperature":0.7,"pith_summary":"Deegen is a meta-compiler: instead of writing a VM by hand, the language implementer writes each bytecode's execution semantics as a C++ function, and Deegen generates, at build time, a complete two-tier execution engine---an optimized interpreter, a baseline JIT, and the profiling, tier-up, and on-stack-replacement logic that switches between them. The paper's central claim is that this automatic path reaches performance previously reserved for hand-assembled VMs: the generated Lua VM's interpreter is 179% faster than PUC Lua and 31% faster than LuaJIT's interpreter, and its baseline JIT, with negligible startup cost, is 360% faster than PUC Lua and only 33% slower on average than LuaJIT's optimizing JIT. Because everything is generated from a single source of truth, the engineering cost stays close to that of writing a simple interpreter: 42 bytecode definitions expand into 255 specialized bytecodes, versus roughly 86 hand-written bytecode handlers in the comparison LuaJIT. The paper explicitly frames a third-tier optimizing JIT as future work, so this is a demonstration of the two lower tiers rather than of a full multi-tier optimizing pipeline.","feed_headline":"Bytecode specs in C++ auto-generate a JIT-capable VM","feed_subtitle":"Generated Lua interpreter beats PUC Lua by 179%; baseline JIT trails LuaJIT's optimizer by just 33%.","key_machinery":"The load-bearing mechanism is the bytecode semantic description framework. Users specify operands, result types, variants, speculative type hints, and slow paths through Deegen APIs, while the execution semantics themselves are ordinary C++ functions compiled to LLVM IR. Algorithm A, the core of the type-based optimization, runs sparse conditional constant propagation once per combination of guessed operand types, recording which type checks are trivially true, trivially false, or reducible by user-supplied strength-reduction rules; this turns type speculation into automatic fast-path and slow-path splitting. Generic inline caching is expressed as an idempotent computation $\\lambda_i$ mapping an IC key to an IC state, followed by a cheap effect $\\lambda_e(\\text{Input}, \\text{state})$; Deegen desugars these lambdas into tier-specific code, monomorphic with quickening in the interpreter and polymorphic self-modifying stubs in the JIT. Baseline JIT code generation is Copy-and-Patch: bytecode contents and IC state are burnt in as constants, and a CallBr (asm-goto) IR trick lets Deegen extract main-logic and IC-stub stencils from ordinary LLVM-generated assembly, including the inline-slab self-modifying stub chain.","core_discovery":"On the paper's own terms, the discovery is that the two hardest components of a dynamic-language VM---a state-of-the-art interpreter and a baseline JIT---can be produced automatically rather than written by assembly experts, and that this generation can happen without surrendering either startup speed or steady-state throughput. Deegen compiles the C++ bytecode semantics to LLVM IR, runs domain-specific passes for type-check removal, strength reduction, inline-cache lowering, hot-cold splitting, and stencil extraction, and emits a self-contained runtime that uses Copy-and-Patch to generate machine code on demand. The resulting VM, LuaJIT Remake, is a standard-compliant Lua 5.1 implementation whose interpreter outperforms PUC Lua by 179% and LuaJIT's interpreter by 31%, and whose baseline JIT compiles 19.1 million bytecodes per second while running 360% faster than PUC Lua and 33% slower than LuaJIT's optimizing JIT, and faster on 13 of 44 benchmarks.","pith_inferences":["Beyond what the paper claims, the architecture suggests that the build-time/runtime split is the real enabler: expensive analyses such as SCCP over type combinations and assembly CFG inspection run once per bytecode definition, not once per user program, which is what makes automatic JIT generation practical.","The same Copy-and-Patch and CallBr machinery could plausibly be reused outside VMs---for DSL runtimes, tree-walking evaluators, or event-driven frameworks---where one semantic description should yield both a fast interpreter and a quick JIT.","A testable extension is to vary the user-supplied type hierarchy and strength-reduction rules, for example adding a small-integer fast path or a tagged-pointer type, and check whether the generated interpreter and JIT track hand-written design expectations; the paper's claim predicts Deegen will exploit any rule expressible in its API.","An implicit production risk the paper does not resolve is that mis-specified IC annotations, such as wrong impossible-key values or wrong range annotations, are documented as undefined behavior, so a production version would need a debug mode that validates these annotations at runtime."],"forward_implications":["A language implementer can obtain both an interpreter and a baseline JIT by writing only bytecode semantics; the tier-up and OSR-entry glue is emitted automatically.","Variant-based specialization means engineering cost scales with semantic definitions rather than with the number of optimized bytecode forms: 42 definitions produce 255 specialized variants.","Because the baseline JIT compiles at gigabytes per second, the interpreter-to-JIT transition can be hotness-driven without observable startup pauses on short-running workloads.","If the described third-tier optimizing JIT is built, the design predicts that the remaining peak-throughput gap to optimizing JITs will be closable without changing the user-facing bytecode description.","Since both tiers come from one source of truth, language-semantics changes propagate consistently to the interpreter and the JIT instead of requiring synchronized hand edits."],"supporting_citations":[{"why":"Supplies the Copy-and-Patch fast code generation technique that the generated baseline JIT uses to compose and configure pre-built stencils.","marker":"[Xu and Kjolstad 2021]"},{"why":"LuaJIT, the state-of-the-art Lua VM whose interpreter and optimizing tracing JIT are the paper's primary performance baselines.","marker":"[Pall 2005]"},{"why":"PUC Lua, the official Lua interpreter used as the normalized baseline for all reported benchmark comparisons.","marker":"[Lua 2012]"},{"why":"Defines the Lua 5.1 language specification that LuaJIT Remake is built to be standard-compliant with.","marker":"[Ierusalimschy et al. 2012]"},{"why":"Supplies the sparse conditional constant propagation algorithm at the core of Algorithm A's type-check elimination and fast-path splitting.","marker":"[Wegman and Zadeck 1991]"},{"why":"Provides the polymorphic inline-cache design that the generated baseline JIT's self-modifying IC stubs implement.","marker":"[Hölzle et al. 1991]"},{"why":"Supplies dynamic quickening, which the interpreter uses to specialize bytecodes to their cached inline-cache state.","marker":"[Brunthaler 2010]"},{"why":"Provides the continuation-passing direct-threading structure that the generated interpreter uses for bytecode dispatch.","marker":"[Steele 1977]"},{"why":"LLVM IR is the intermediate representation Deegen compiles C++ bytecode semantics into and then analyzes and transforms at build time.","marker":"[Lattner and Adve 2004]"},{"why":"Documents the CallBr/asm-goto IR node Deegen repurposes to model polymorphic IC chains and extract IC stubs from assembly.","marker":"[LLVM 2018]"}],"fun_headline_variants":["Meta-compiler auto-builds JIT VMs from C++ semantics","Auto-generate JIT VMs: interpreter beats LuaJIT's","JIT compiler generated automatically for dynamic languages","Deegen: write bytecodes, get a JIT-capable VM","Automatic JIT generation: 179% faster interpreter"],"cache_read_input_tokens":55168,"weakest_assumption_plain":"The load-bearing premise is that the automatically produced type-specialized fast paths are semantically correct whenever the user's type hierarchy and strength-reduction rules are correct, but the paper omits the correctness proof with \"We omit an argument of correctness due to space,\" so a silent miscompile in Algorithm A would invalidate the generated VM while leaving all speed measurements intact.","fun_headline_variants_meta":{"raw":{"variants":["Meta-compiler auto-builds JIT VMs from C++ semantics","Auto-generate JIT VMs: interpreter beats LuaJIT's","JIT compiler generated automatically for dynamic languages","Deegen: write bytecodes, get a JIT-capable VM","Automatic JIT generation: 179% faster interpreter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1714,"prompt_tokens":1143,"completion_tokens":571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":759,"completion_tokens_details":{"reasoning_tokens":498}},"tokens_in":759,"tokens_out":571,"duration_ms":5102,"temperature":1.0,"reasoning_tokens":498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:28:31.410150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take LJR's Add bytecode, which splits on a speculation that both operands are tDoubleNotNaN, and execute it with one operand a NaN boxed as an impure NaN or a table; if the generated fast path produces a result different from PUC Lua's rather than transferring to the slow path, Algorithm A or the user-supplied type description is unsound. A systematic version is to mutate one strength-reduction rule in the type-hierarchy description and rerun the 44-benchmark suite against PUC Lua, since any divergence in outputs would falsify the automatic-optimization claim independently of the speed numbers.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the CallBr/asm-goto IR node Deegen repurposes to model polymorphic IC chains and extract IC stubs from assembly."}],"review_version":1}