{"id":"980407b0-fb47-4254-8c5e-4d0e0435d429","arxiv_id":"2411.16341","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fine-tuned 1.3B-parameter language model translates x86 assembly into ARM and RISC-V assembly with 79-89% functional accuracy on HumanEval, and 1.73x faster execution than Rosetta 2 in a small case study.","lead":"CRT is a fine-tuned language model that takes x86 assembly code as input and generates ARM or RISC-V assembly, reaching 79% to 89% functional accuracy on the HumanEval-derived test suite. The authors report that on an Apple M2, the transpiled ARM code runs 1.73x faster than Apple's Rosetta 2, though the comparison is limited to small -O0 test programs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No decontamination check between the 500k AnghaBench training programs and the 164 HumanEval evaluation problems; if any overlap exists, the reported 79.25%/88.68% accuracies may measure memorization, not generalization.","rationale":"The reader identified the same weakest assumption: the lack of decontamination between AnghaBench training data and the HumanEval evaluation set. I agree this is the most load-bearing concern for the central claim. The paper's strengths—execution-based functional verification under QEMU, thorough hyperparameter and quantization experiments, and comparisons with larger base LLMs—are real, but none of them address train/eval separation. The risk is concrete: with 500k training programs, a single exact or near-exact copy of a HumanEval function in the training set would expose the model to the ground-truth ARM assembly during two epochs of fine-tuning. Since the evaluation has only 164 problems, memorization of a handful of duplicates would disproportionately inflate the reported accuracy. I also considered the incomplete case-study protocol (which programs were included in the speedup measurement) and the assembly-text versus binary framing as additional issues, but these affect secondary claims; the contamination concern directly undermines the headline accuracy and the feasibility argument. If a decontamination audit finds no meaningful overlap, the reported accuracies and the paper's conclusions stand. If overlap is found, the results must be re-evaluated on a cleaned training set. Therefore the verdict remains CONDITIONAL, pending release of a decontamination analysis or cleaned artifacts.","tokens_in":18005,"tokens_out":9876,"duration_ms":85234,"concrete_test":"Perform a decontamination audit: (1) obtain the exact C sources for the 164 HumanEval problems as used in LLM4Decompile; (2) compare them against the 500k AnghaBench training C files using exact hashing and a near-duplicate detector (e.g., SourcererCC or DECKARD) with a high similarity threshold; (3) for any flagged match, compile both to x86 and ARM assembly and measure assembly-level similarity. If matches exist, retrain the model on the decontaminated subset and re-run the HumanEval evaluation; if accuracy drops materially (e.g., more than 5 points), the reported generalization is unsupported. If no meaningful overlap is found, the concern is resolved and the accuracy claim stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline accuracies (79.25% on ARMv5, 88.68% on RISC-V64) are computed on the 164 HumanEval C problems from LLM4Decompile, while the model is fine-tuned on 500k programs randomly sampled from AnghaBench, a GitHub-mined C corpus. HumanEval is one of the most widely mirrored coding benchmarks on GitHub, and AnghaBench scrapes public repositories without any reported deduplication against evaluation sets. The paper reports no decontamination step: no exact-match filtering, no near-duplicate removal, no overlap audit. Because the evaluation uses only 164 problems, even a small number of exact or semantically equivalent duplicates in the 500k training samples would let the model memorize the corresponding x86-to-ARM assembly pairs during two epochs of fine-tuning. This would inflate the measured accuracy without demonstrating generalization. The reported 98.81% line coverage does not mitigate this risk, as coverage measures test-suite thoroughness, not train/eval separation. The central feasibility claim therefore rests on an unverified external data assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRT, a fine-tuned 1.3B-parameter DeepSeek-Coder model that translates x86 assembly to ARMv5 and RISC-V64 assembly. Training pairs are generated by compiling 500k AnghaBench C programs with gcc and cross-compilers at -O0, and evaluation uses the 164 HumanEval C problems from LLM4Decompile, with functional correctness checked by executing the generated assembly under QEMU against unit tests. The authors report 79.25% ARMv5 and 88.68% RISC-V64 test accuracy, and on an Apple M2 Pro report that ARMv8-transpiled code achieves a 1.73x speedup, 2.41x memory efficiency, and 1.47x energy efficiency over Rosetta 2. The paper also documents tokenizer extensions, quantization studies, and an analysis of syntactic versus semantic equivalence classes.","tokens_in":18075,"tokens_out":9418,"duration_ms":88803,"significance":"If the results are clean, this is a solid demonstration that a small fine-tuned LLM can perform CISC-to-RISC assembly translation with functional correctness, and the QEMU-based execution oracle is a considerably stronger signal than syntactic matching alone. The public release of code, models, training data, and benchmarks is also a concrete strength. The ARMv8 case study, although under-specified, addresses a practically relevant deployment setting. The main significance of the paper depends on two fixable points: documented train/eval decontamination and clarity about which programs enter the Rosetta performance comparison. With those provided, the paper would be a useful contribution to LLM-for-code and binary-translation research.","major_comments":[{"comment":"The paper reports the 79.25% ARMv5 and 88.68% RISC-V64 accuracies as evidence that CRT generalizes, but no decontamination step is described between the 500k AnghaBench training programs and the 164 HumanEval evaluation problems. Both corpora originate from public GitHub C code, and HumanEval is widely redistributed, so exact or near-duplicate source functions may appear in AnghaBench. Because the model is fine-tuned for two epochs on compiled x86/ARM pairs, even a few duplicated evaluation programs would allow memorization and inflate the measured accuracy. The 98.81% line coverage reported in §4.3 addresses test-suite thoroughness, not train/eval separation. The authors should report an overlap audit at both the C-source level and the compiled-assembly level, and should report all headline accuracies on the guaranteed-disjoint subset.","section":"§4.1, §4.3, Tables 2–3"},{"comment":"The Apple M2 case study is under-specified in a way that affects the headline deployment claims. Table 4 gives the ARMv8-trained model a functional accuracy of 75.0% (123/164), and Figure 5 shows that 23 programs fail only on the M2, yet §7.2 reports aggregate 1.73x speedup, 2.41x memory efficiency, and 1.47x energy efficiency with no statement of which programs were included in the geometric means. If the performance ratios are computed only over programs that CRT transpiled correctly while Rosetta ran all programs, the comparison is selection-biased. The authors should state the inclusion criterion, report the performance per program or at least for correct and incorrect subsets, and clarify that the abstract's 79.25% refers to the ARMv5 model rather than the ARMv8 model used in deployment.","section":"§7.1–7.2, Table 4, Figure 5"},{"comment":"The functional-correctness metric is the load-bearing measurement, but the evaluation procedure is not fully reproducible from the text. The paper does not specify how the generated ARM assembly is linked with the HumanEval test harness, how non-assembling or non-linking outputs are counted, whether a timeout is used under QEMU, and whether the ground-truth ARM program is compiled with the same flags and linker configuration. These details are necessary to interpret the accuracy numbers and to allow independent replication. I ask the authors to provide the exact evaluation pipeline (compile, link, QEMU invocation, pass/fail criterion) in the main text or an appendix.","section":"§4.2, §4.4"}],"minor_comments":[{"comment":"The abstract reports 88.68% accuracy for RISC-V64 while the contribution list in §1 reports 88.69%; these numbers should be made consistent.","section":"Abstract and §1"},{"comment":"The text says the case study uses a model 'trained on ARMv8', but the methodology in §4.1 only describes ARMv5 cross-compilation; please describe the ARMv8 training set, flags, and epochs explicitly.","section":"§7.1"},{"comment":"The comparison with GPT-4o, DeepSeekCoder2-16B, and Yi-Coder-9B lacks the prompting protocol and decoding settings; without this information, the '9.8x accuracy' claim overstates the significance of the comparison.","section":"Tables 2 and 3"},{"comment":"Calling the 164-problem HumanEval C suite a 'comprehensive test suite' is an overstatement; the authors should either add more diverse benchmarks or soften the wording.","section":"§4.3 and Abstract"},{"comment":"In the manuscript text, Figure 4 appears as a sequence of unicode placeholders rather than readable axis labels; please ensure the actual PDF contains legible figures and captions.","section":"Figure 4"},{"comment":"The statement that 500k samples correspond to 8 billion tokens is not obviously consistent with the reported average of 13k tokens per sample; please clarify whether the count includes both source and target assembly tokens.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the artifact release is valuable, but the missing decontamination audit and the ambiguous selection of programs in the Rosetta comparison are load-bearing. I recommend requiring an explicit overlap analysis and a precise statement of which programs enter the performance statistics before acceptance. The authors should also be encouraged to add confidence intervals for the 164-problem evaluation, since the headline percentages are reported with false precision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is that a fine-tuned 1.3B model produces x86-to-ARM and x86-to-RISC-V assembly translations that execute correctly against unit tests: 79.25% and 88.68% respectively. Prior work (GUESS & SKETCH, LLM4Decompile) did not cover this CISC-to-RISC pair with a functional correctness measure, so the empirical target is new. The paper also does several things well: functional verification under QEMU rather than edit distance alone, a systematic comparison against much larger models, a careful tokenizer extension, and quantization results that are directly useful for deployment. The error analysis in the appendix is honest and specific.\n\nThe soft spots, in proportion. The biggest is the lack of any decontamination check. The training set is 500k programs randomly sampled from AnghaBench, and the evaluation set is the 164 HumanEval problems from LLM4Decompile. AnghaBench is mined from GitHub; HumanEval is mirrored everywhere. The paper reports no exact-match or near-duplicate filtering, so the headline accuracies could partly reflect memorization. That is a real concern, but it is fixable: the authors promise to release code, models, and benchmarks, so a simple overlap audit and a re-run on a disjoint set would settle it. I would not call the paper's central argument broken, just unverified on this specific axis.\n\nSecond, the Apple M2 case study is thinner than the abstract suggests. It does not say which or how many programs were measured, and the 1.73x speedup over Rosetta 2 compares a static translation against a dynamic one, on -O0 functions. The phrase 'diverse real-world applications' is an overstatement relative to what is reported. Third, everything is -O0; optimized assembly is untested, and while the paper acknowledges this, it is easy to miss.\n\nOne small note: the Figure 5 confusion matrix looks unlabeled, but the numbers reconcile with Table 4 if you sum the quadrants, so it's a presentation problem, not an inconsistency. The citation pattern is fine; the prior work is relevant, and the RISC-V extension is a sensible generality check.\n\nOverall, this deserves a serious referee. The likely outcome should be a conditional accept: require the contamination audit, clarify the case-study protocol, and temper the 'real-world applications' framing. I would not want to use the accuracy numbers in my own writing until the overlap question is answered, but the approach and evaluation design are worth citing.\n\nRecommendation: send to peer review, with a clear request for those revisions.","headline":"A credible new empirical result in LLM-based assembly transpilation, with functional verification as the main strength; the headline accuracy needs a contamination audit and the case study needs more detail before the numbers are fully trusted.","tokens_in":18786,"tokens_out":4356,"would_cite":true,"duration_ms":37799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a 1.3B-parameter language model can transpile x86 assembly into executable ARM/RISC-V assembly with 79.25% and 88.68% functional accuracy, and that the transpiled ARM64 code beats Rosetta 2 on an M2 by 1.73x speed, 2.41x…","keywords":["assembly transpilation","CISC to RISC","language model","x86","ARM","RISC-V","binary translation","functional correctness"],"falsifier":"A fuzzy-duplicate search comparing the 164 HumanEval C functions against the training corpus, for example by hashing tokens of the compiled assembly or using a clone detector, would settle the claim. Finding even a few benchmark programs in the training data would make the reported accuracies look like memorization; re-running evaluation on a fresh, never-seen set of C functions with unit tests would test generalization directly.","tokens_in":17666,"feed_emoji":"🔁","tokens_out":9122,"duration_ms":75189,"temperature":0.7,"pith_summary":"This paper tries to establish that a small language model can act as a direct CISC-to-RISC assembly transpiler, converting x86 machine code into ARM and RISC-V assembly without the runtime overhead of an emulator or virtualization layer. Its central evidence is a fine-tuned 1.3B-parameter model that passes 79.25% of the HumanEval-derived test suite when translating x86 to ARMv5 and 88.68% when translating to RISC-V64, with correctness judged by executing the generated assembly against unit tests in an emulated environment. In a deployment case study on an Apple M2, the transpiled ARM64 binaries outperform Apple's Rosetta 2 by 1.73x in speed, 2.41x in memory efficiency, and 1.47x in energy efficiency. If true, this shows that learning-based transpilation can bridge the architectural gap between CISC and RISC instruction sets while preserving semantics and even improving performance, offering an alternative to dynamic binary translation.","feed_headline":"Fine-tuned 1.3B LLM turns x86 assembly into ARM at 79% accuracy","feed_subtitle":"On an M2 it beats Apple's Rosetta 2 by 1.73x speed, 2.41x memory, and 1.47x energy.","key_machinery":"The central machinery is a fine-tuned autoregressive language model that learns the conditional distribution $\\mathrm{P}(Y \\mid X)$ over assembly tokens, where $X$ is x86 assembly and $Y$ is ARM or RISC-V assembly. The model is trained on paired outputs from compiling 500,000 C programs to x86 and ARMv5, which supplies the CISC/RISC correspondence, and an extended tokenizer keeps frequently used opcodes and register names as single tokens so the model can track them across the translation. The load-bearing evaluation is functional rather than syntactic: generated assembly is assembled and executed against unit tests in an emulator, so accuracy is measured by whether the program behaves correctly.","core_discovery":"The paper's central discovery is that direct x86-to-ARM assembly transpilation is a learnable mapping, not a rule-engineering problem. By fine-tuning a 1.3B-parameter code language model on hundreds of thousands of C programs compiled into paired x86 and ARM assembly, and by extending the tokenizer so that opcodes and register names stay as single tokens, the authors obtain a model that generates ARM assembly preserving functional behavior on 79.25% of the benchmark programs and 88.68% for RISC-V64. They further show that correct outputs are not mere copies of ground truth: the generated code often differs syntactically yet still passes the same unit tests, with variations such as different register allocation, reordered commutative operands, and instruction consolidation. In the ARMv8 case study, the transpiled code, compiled natively, runs faster and uses less memory and energy than Rosetta 2's dynamic translation, suggesting one-time transpilation can beat virtualization.","pith_inferences":["Editorial extension: HumanEval functions are short and compile to small assembly functions, so the 79.25% and 88.68% accuracies are likely an upper bound for large real-world binaries; a natural next test is accuracy on multi-thousand-instruction functions.","Editorial extension: Because the training recipe only needs paired compilations, the same method could be retargeted to other ISA pairs, for instance x86 to PowerPC or MIPS, without changing the architecture of the approach.","Editorial extension: The speedup over Rosetta 2 stems from replacing runtime translation with a one-time offline translation, so the performance and energy advantage could widen on long-running static workloads, although the paper does not measure that directly."],"forward_implications":["Legacy x86 binaries whose assembly is available could be converted once into ARM or RISC-V assembly and then natively compiled, eliminating the per-run translation overhead of emulators.","Quantized versions at int8 and int4 precision retain most of the accuracy, so the transpiler can plausibly run on CPUs and resource-constrained devices without a GPU.","On Apple hardware, the transpiled code beats Rosetta 2 on speed, memory, and energy in the tested programs, suggesting that one-time offline transpilation can outperform dynamic translation where a pre-pass is feasible.","The same recipe extends to a second RISC target, RISC-V64, with higher accuracy, indicating the approach transfers across target ISAs rather than being hand-tuned to ARM.","Smaller task-specific models outperform much larger general-purpose code models on this benchmark, implying that fine-tuning and tokenization matter more than raw model size for assembly transpilation."],"supporting_citations":[{"why":"Supplies the 500k C programs that are compiled into paired x86/ARM training data.","marker":"(Da Silva et al., 2021)"},{"why":"Defines the HumanEval problems that become the functional-correctness evaluation benchmark.","marker":"(Chen et al., 2021)"},{"why":"Provides the C-translated HumanEval benchmark and the evaluation set used for testing.","marker":"(Tan et al., 2024)"},{"why":"QEMU emulator executes the transpiled assembly so correctness is checked by running the test cases.","marker":"(Bellard, 2005a)"},{"why":"Base code language model that is fine-tuned into the CRT transpiler.","marker":"(Guo et al., 2024)"},{"why":"Rosetta 2 is the production baseline that the transpiled ARM64 code is compared against on the M2.","marker":"(Apple Inc., 2020)"},{"why":"Prior assembly transpilation work whose alignment approach and edit-distance metric the paper builds on.","marker":"(Lee et al., 2024)"},{"why":"GCC compiles the training programs to x86 and ARM assembly, creating the paired corpus.","marker":"(Compiler, 2009)"}],"fun_headline_variants":["LLM transpiles x86 to ARM at 79% accuracy, beats Rosetta 2","1.3B model converts x86 assembly to ARM, 1.73x faster than Rosetta 2","Learnable x86-to-ARM assembly transpilation beats Rosetta 2 on M2","79% accurate x86-to-ARM transpilation via fine-tuned LLM","From CISC to RISC: LLM-guided assembly transpiler outperforms Rosetta 2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that none of the 164 HumanEval evaluation programs, or close variants, appeared in the 500,000 GitHub-sourced C programs used for training, because the paper reports no decontamination check.","fun_headline_variants_meta":{"raw":{"variants":["LLM transpiles x86 to ARM at 79% accuracy, beats Rosetta 2","1.3B model converts x86 assembly to ARM, 1.73x faster than Rosetta 2","Learnable x86-to-ARM assembly transpilation beats Rosetta 2 on M2","79% accurate x86-to-ARM transpilation via fine-tuned LLM","From CISC to RISC: LLM-guided assembly transpiler outperforms Rosetta 2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000895,"raw_usage":{"total_tokens":3895,"prompt_tokens":1020,"completion_tokens":2875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":2754}},"tokens_in":636,"tokens_out":2875,"duration_ms":18473,"temperature":1.0,"reasoning_tokens":2754,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:14:12.200753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A fuzzy-duplicate search comparing the 164 HumanEval C functions against the training corpus, for example by hashing tokens of the compiled assembly or using a clone detector, would settle the claim. Finding even a few benchmark programs in the training data would make the reported accuracies look like memorization; re-running evaluation on a fresh, never-seen set of C functions with unit tests would test generalization directly.","supporting_citations":[{"cited_title":"F., Kind, B","cited_arxiv_id":null,"evidence_quote":"Supplies the 500k C programs that are compiled into paired x86/ARM training data."}],"review_version":1}