{"id":"5c39f47b-c57d-4137-b86c-59518ccbe1c4","arxiv_id":"2506.14606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A fine-tuned language model with test-driven validation transpiles x86 assembly to ARM/RISC-V, achieving 99% pass rates on HumanEval-C and outperforming Rosetta 2 in a limited case study.","lead":"This paper trains a small language model to translate x86 assembly code into ARM and RISC-V assembly, then runs unit tests to check the translated code actually works. On simple benchmark programs it reaches 99 percent pass rates, and on one small case study it runs faster and uses less memory than Apple's Rosetta 2.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Rosetta 2 efficiency claim in §5.2 is not checkable as written: workload list, compiler flags, per-program variance, and binary-vs-assembly input details are missing, so the 1.73x/1.47x/2.41x numbers may be artifacts of selection or optimization-level mismatch.","rationale":"I read the paper as making two main claims: (1) a fine-tuned LLM with test-based evaluation achieves high functional accuracy on x86-to-ARM and x86-to-RISC-V assembly translation, and (2) GG-transpiled code beats Rosetta 2 on real-world efficiency. Claim (1) is plausible and internally consistent: the training-data construction, tokenizer extension, RoPE extrapolation, beam search, and unit-test evaluation are described in enough detail to reproduce, and the ablation in Table 5 is coherent. The reader's conditional verdict is therefore supported. The decisive problem is claim (2). Section 5.2's real-world case study omits the workload and protocol details necessary to rule out favorable selection or an unfair optimization-level mismatch. Because GG's -O2 accuracy is only 45%, the optimization level is not a minor detail: it can determine whether the comparison is meaningful. The paper's own Limitations section (§7) acknowledges that the test-based guarantee is bounded and that compiler, symbolic, or heuristic baselines are excluded; I weigh that as an honest limitation rather than a hidden flaw. The likely fabricated citation and missing promised artifacts are credibility and reproducibility issues, but they do not by themselves refute the core method. The efficiency comparison is the load-bearing concern because the paper's strongest practical claim depends on it. The proposed concrete test would settle the concern: if the efficiency numbers survive same-flag, per-program comparisons on a disclosed workload, the paper is acceptable conditional on artifact release; if not, the efficiency claims should be withdrawn. Thus I keep the reader's CONDITIONAL verdict.","tokens_in":15570,"tokens_out":9751,"duration_ms":99525,"concrete_test":"Ask the authors to release the exact workload list, build scripts, compiler and optimization flags, and the powermetrics measurement script. Then independently rerun the Figure 3 experiment per program with both x86_64 and arm64 versions compiled at matching -O0 and -O2 levels, executing each program 100 times and reporting per-program means with 95% confidence intervals. If the 1.73x speedup does not survive same-flag comparisons on the majority of programs, the efficiency claim should be removed or downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the Rosetta 2 efficiency comparison in §5.2 and Figure 3. The paper reports 1.73x faster runtime, 1.47x better energy efficiency, and 2.41x better memory usage over Rosetta 2, but it never discloses which programs were measured, how many there were, how the x86 binaries and GG input assembly were produced, or which compiler optimization level was used. This omission matters because GG's own accuracy drops from 99.39% at -O0 to 45.12% at -O2 on ARMv8 HumanEval: if Rosetta's x86 binaries were compiled with -O2 while GG's input assembly came from -O0, the comparison is not apples-to-apples; if both used -O0, Rosetta is handicapped relative to typical real-world use. The motivating scenario is legacy binaries without source, yet the case study appears to compile source code into native ARM and Rosetta binaries and feeds GG compiler-generated assembly, never disassembled or stripped binaries. No confidence intervals or per-program results are reported, so a single outlier workload could drive the aggregate geometric mean. As written, the efficiency claim cannot be independently verified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Guaranteed Guess (GG), an assembly-to-assembly transpilation pipeline that fine-tunes small language models (Qwen2.5-Coder and DeepSeek-Coder variants) to translate x86 assembly into ARMv5, ARMv8, and RISC-V64 assembly. Training data are generated by compiling C/C++ programs from AnghaBench and The Stack v2 at -O0 and -O2, with additional components including an assembly-aware tokenizer extension, RoPE extrapolation, and beam search. Evaluation compiles the predicted assembly, links it with unit tests, and requires all tests to pass; the paper reports 99.39% pass@1 on HumanEval-C for ARMv8 at -O0, 45.12% at -O2, 89.63% on RISC-V64, and 49.23% on BringUpBench. A case study on Apple M2 Pro claims that GG-transpiled code is 1.73x faster, 1.47x more energy-efficient, and 2.41x more memory-efficient than Rosetta 2, with near-native performance. The paper also presents ablations, error analyses, and a similarity analysis across ISAs.","tokens_in":15862,"tokens_out":3775,"duration_ms":38822,"significance":"If the claims hold, this is a meaningful advance in LLM-based ISA transpilation. The work demonstrates that domain-adapted small models can vastly outperform general-purpose LLMs on a difficult low-level code task, and the test-driven evaluation protocol is a practical step beyond token-level metrics. The training and evaluation are not circular: correctness is determined by external unit tests rather than by similarity to training pairs. The paper also includes useful ablations, failure-mode taxonomies, and examples of syntactically different but functionally equivalent translations. However, the headline Rosetta 2 efficiency comparison currently lacks the experimental detail needed for independent verification, and the motivating scenario of translating legacy binaries is not directly evaluated.","major_comments":[{"comment":"The Rosetta 2 comparison is not reproducible as reported. The paper does not state which benchmark programs were measured, how many there were, how the x86 binaries for Rosetta were produced (including the compiler optimization level), whether the GG input was compiler-generated assembly or disassembled machine code, or what linking/runtime environment was used. This matters directly because GG's own accuracy drops from 99.39% at -O0 to 45.12% at -O2 on ARMv8 HumanEval (Table 3); an optimization-level mismatch could explain or even reverse the reported runtime, energy, and memory differences. The geometric means are reported without confidence intervals or per-program results, so a single outlier workload could drive the aggregate. Please disclose the full workload list, compilation flags, measurement protocol, and per-program measurements, and rerun the comparison under matched optimization levels.","section":"§5.2, Figure 3"},{"comment":"The paper frames GG as supporting 'legacy binaries' and 'binary-to-binary translation' without decompilation, but the actual pipeline is trained and evaluated on compiler-generated assembly produced from C/C++ source, not on disassembled or stripped binaries. No experiment demonstrates that the model can consume raw x86 machine code or a disassembly of such a binary as input. This is a load-bearing gap because the motivating application is migration of binaries whose source is unavailable. Please add an evaluation on disassembled or stripped binaries, or substantially revise the claims about legacy-binary support.","section":"§1 and §3.1"},{"comment":"Several key numerical claims are presented without variance or significance information. HumanEval contains only 164 programs, so the differences between some ablations (e.g., +1.04% from the tokenizer extension and +1.21% from beam search in Table 5) correspond to roughly one or two programs. Without multiple training seeds or confidence intervals, the relative contributions of the components in Table 5 should be treated as unstable. Please report confidence intervals, bootstrap estimates, or repeated runs with different random seeds for the main accuracy claims and ablations.","section":"§5.1, Table 3; §5.4, Table 5"}],"minor_comments":[{"comment":"The abstract reports '99%' and '49%' while the body reports 99.39% and 49.23%; please use consistent rounded or exact values.","section":"Abstract"},{"comment":"The figure shows only geometric means without per-program points or error bars; adding a scatter or box plot would make the comparison substantially more informative.","section":"Figure 3"},{"comment":"The caption says 'ARMv8 accuracy' but does not state the benchmark (presumably HumanEval) or optimization level; please specify the exact evaluation setting.","section":"Table 5 caption"},{"comment":"The phrase '36x requests per second' is unclear; please specify the baseline for the '36x' improvement or report an absolute throughput value.","section":"§4.1"},{"comment":"There is a typo: 'as shown inf figure 1' should be 'as shown in Figure 1'.","section":"§4.2"},{"comment":"The table contains the typo 'Groud truth' instead of 'Ground truth'.","section":"Table 8"},{"comment":"The sentence 'This result is 9% lower than ARMv8 which shows how much different RISC-v64 from x86 compared ARMv8' is grammatically unclear and should be rewritten.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The Rosetta 2 efficiency comparison is likely to receive heavy scrutiny from reviewers and readers; I would require full workload and protocol disclosure before publication. The manuscript also cites Guess & Sketch, which shares an author, and while I do not see circularity in the central evaluation, the related-work discussion should more clearly position the incremental contribution relative to that prior system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first LLM CISC-to-RISC transpiler I know of that pairs generation with test-driven validation, and the -O0 numbers on HumanEval (99.39%) are genuinely strong. But the Rosetta 2 efficiency comparison is not checkable as written, and there is a citation that looks fabricated. Both need to be fixed before I'd trust the headline.\n\nWhat's new and good: GG extends Guess & Sketch by tackling the harder x86-to-ARM/RISC-V direction with a fine-tuned 1.5B model. The tokenizer extension and RoPE extrapolation are sensible for assembly, and the ablation study does a clean job of attributing gains to data, context length, tokenizer, and beam search. The evaluation goes beyond token matching: pass@1 with full unit test execution, coverage reporting, and a useful error taxonomy on BringUpBench. The limitations section is honest: accuracy craters at -O2, and the 'guarantee' is only as good as the tests.\n\nWhere it gets soft: the Rosetta 2 comparison in §5.2 is missing the workload list, compiler flags, per-program numbers, and variance. That matters because the paper's own data shows -O0 vs -O2 changes accuracy by ~50 points; if Rosetta's x86 binaries were compiled at -O2 and GG's input at -O0, the 1.73x/1.47x/2.41x numbers would be misleading. Also, the motivating scenario is legacy binaries without source, but the case study appears to compile C to assembly and feed that to GG, not disassembled or stripped binaries. There are also no significance tests anywhere, and the baselines exclude compiler-based or symbolic transpilers. Finally, the reference 'Zhe Wang, John Smith, and Jane Doe' looks like placeholder names; the authors need to verify or remove it. The abstract promises open-source artifacts, but none are available.\n\nBottom line: the accuracy result is interesting and the engineering is thoughtful, so I'd send this to peer review expecting major revision. The authors need to release the artifacts, clean up the citation, and fully specify the Rosetta benchmark before the efficiency claim can be taken at face value.","headline":"First LLM CISC-to-RISC transpiler with test-driven validation shows strong -O0 accuracy, but the Rosetta 2 comparison is not checkable and one citation looks fabricated.","tokens_in":16387,"tokens_out":2946,"would_cite":false,"duration_ms":28293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 1.5B-parameter language model, trained on paired x86/ARM assembly and validated by unit tests, transpiles CISC to RISC with 99.39% test-pass accuracy on HumanEval and beats Rosetta 2 on speed, energy, and memory.","keywords":["CISC-to-RISC transpilation","assembly language model","test-driven validation","x86 to ARM translation","Rosetta 2 comparison","code coverage","tokenizer extension","LLM binary translation"],"falsifier":"Fuzz-test a GG-transpiled program that passes all provided HumanEval or BringUpBench tests with inputs outside the suite; any failing assertion would demonstrate that the 'guarantee' is test coverage, not semantic equivalence. Alternatively, re-run the Rosetta comparison on a published, representative workload list; if the 1.73x speedup does not survive, the efficiency claim is workload-specific.","tokens_in":15410,"feed_emoji":"🔁","tokens_out":7231,"duration_ms":67962,"temperature":0.7,"pith_summary":"The paper claims that a small, custom-trained language model can translate x86 (CISC) assembly directly into ARM and RISC-V (RISC) assembly, and that plugging the output into a software-testing harness converts an unverifiable guess into a checked result. On the 164 HumanEval-C programs compiled at -O0, the flagship 1.5B model's most-likely output passes every unit test in 99.39% of ARMv8 translations and 89.93% of RISC-V64 translations; on the 65 full-program BringUpBench tasks it passes 49.23%. The same transpiled binaries run 1.73x faster, consume 1.47x less energy, and use 2.41x less memory than Apple's Rosetta 2 on an M2 Pro. If true, the result offers a different route across the x86-to-ARM migration problem: a one-time, test-checked translation instead of a persistent runtime translation layer.","feed_headline":"A 1.5B model turns x86 into ARM with 99% test accuracy","feed_subtitle":"GG checks every translation against unit tests and beats Rosetta 2 on speed, energy, and memory.","key_machinery":"The load-bearing object is the GG pipeline itself: a 'guesser' — a custom-trained 1.5B-parameter decoder-only language model — that proposes ARM or RISC-V assembly from x86 assembly, followed by a validator that compiles, links, and runs the candidate against the target program's unit tests with gcov line coverage. The guesser's accuracy comes from four compounding components the paper ablates: training on paired assembly compiled from AnghaBench and The Stack v2, RoPE-based context extrapolation, a tokenizer extended with common opcodes and register names, and 8-beam search. The validator fixes the meaning of 'guaranteed': only a candidate that passes all tests counts as correct, and coverage above 98% gives the test run evidential weight.","core_discovery":"GG (Guaranteed Guess) is presented as the first CISC-to-RISC transpiler built by training a language model on paired assembly: roughly 1.31M C/C++ programs were compiled to x86 and to ARMv5, ARMv8, and RISC-V64 under -O0 and -O2, and the model was fine-tuned with a tokenizer extended for opcodes and registers, RoPE extrapolation to a 32.7k-token context, and 8-beam deterministic decoding. The central claim is that these choices make a 1.5B-parameter model accurate enough at unoptimized x86-to-ARM translation to pass complete unit-test suites in 99.39% of cases, far above general-purpose language models, and that the resulting native binaries are competitive with native compilation and beat the Rosetta 2 translation layer on runtime, energy, and memory. The 'guarantee' the authors claim is explicitly test-driven: a translation is counted correct only if every unit test passes, with line coverage above 98%, and the authors state in their limitations that this is not a proof of full semantic equivalence.","pith_inferences":["The unit-test signal could be turned into an iterative repair loop: when a candidate fails a test, the failure is a concrete reward for resampling or patching, a direction the paper only hints at under 'test-driven transpilation and iterative repair'.","The same data-generation recipe — compile a corpus under two ISAs and train a small model — could produce transpilers for other pairs, such as ARM-to-RISC-V or x86-to-RISC-V with different optimization flags, with effort allocated by ISA similarity.","A natural next artifact would be a public, fixed workload suite for x86-to-ARM migration, so transpilers, emulators, and Rosetta-style layers can be compared on equal footing.","The similarity gradient across target ISAs suggests a cheap a priori predictor: compute an assembly-level similarity score between source and target ISA before investing in training data, and expect accuracy to track it."],"forward_implications":["GG's transpiled binaries run natively, so the per-execution overhead of dynamic translation disappears; the paper measures execution time matching native binaries and faster than Rosetta 2.","The test-driven guarantee means the method can be deployed wherever a program already has a meaningful test suite: passing it is a practical correctness certificate, not a proof of equivalence.","The method transfers to other RISC targets: the same training recipe reaches 89.93% on RISC-V64, showing the approach is not ARM-specific.","The model is small enough and quantizable enough (INT4 with less than 4% accuracy loss) that the authors propose deployment on edge devices as a realistic use case.","Accuracy drops sharply on -O2 binaries (45.12% on ARMv8), so the current guarantee is effectively limited to unoptimized code; the paper points to richer representations as the path to fix this."],"supporting_citations":[{"why":"Supplies the Rosetta 2 baseline whose runtime, energy, and memory usage are compared against GG on Apple Silicon.","marker":"Apple Inc. 2020"},{"why":"Prior neurosymbolic transpilation between ARMv8 and RISC-V that GG extends from RISC-to-RISC to CISC-to-RISC.","marker":"Lee et al. 2024"},{"why":"Provides the HumanEval-C benchmark used for evaluation and motivates decompilation-style baselines from low-level code.","marker":"Tan et al. 2024"},{"why":"AnghaBench supplies the one-million-program compilable C/C++ corpus from which the paired x86/RISC training set is built.","marker":"Da Silva et al. 2021"},{"why":"The Stack v2 adds permissively licensed code to expand the training data beyond AnghaBench.","marker":"Kocetkov et al. 2022"},{"why":"Qwen2.5-Coder is the base model for GG-0.5B and GG-1.5B and supplies the tokenizer that gets extended.","marker":"Hui et al. 2024a"},{"why":"DeepSeek-Coder is the base model for GG-DeepSeek-1.3B and is used in tokenizer fertility comparisons.","marker":"Guo et al. 2024"},{"why":"RoFormer's rotary position embedding is the mechanism behind the RoPE extrapolation that extends the context window to 32.7k tokens.","marker":"Su et al. 2024"},{"why":"BringUpBench supplies the bare-metal, full-project benchmark used to test real-world generalization.","marker":"Austin 2024"}],"fun_headline_variants":["1.5B model transpiles x86 to ARM, passes 99% of tests","LLM transpiler beats Rosetta 2, 99% test pass","Test-guaranteed x86-to-ARM translation: 99% pass, beats Rosetta","1.5B model: CISC-to-RISC with test guarantees, beats Rosetta 2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee is only as strong as the unit tests: if the test suite fails to exercise some input or behavior, a GG translation that passes all tests can still be functionally wrong.","fun_headline_variants_meta":{"raw":{"variants":["1.5B model transpiles x86 to ARM, passes 99% of tests","LLM transpiler beats Rosetta 2, 99% test pass","Test-guaranteed x86-to-ARM translation: 99% pass, beats Rosetta","1.5B model: CISC-to-RISC with test guarantees, beats Rosetta 2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":4129,"prompt_tokens":1074,"completion_tokens":3055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":2960}},"tokens_in":690,"tokens_out":3055,"duration_ms":20603,"temperature":1.0,"reasoning_tokens":2960,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:50:08.380993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fuzz-test a GG-transpiled program that passes all provided HumanEval or BringUpBench tests with inputs outside the suite; any failing assertion would demonstrate that the 'guarantee' is test coverage, not semantic equivalence. Alternatively, re-run the Rosetta comparison on a published, representative workload list; if the 1.73x speedup does not survive, the efficiency claim is workload-specific.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Rosetta 2 baseline whose runtime, energy, and memory usage are compared against GG on Apple Silicon."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior neurosymbolic transpilation between ARMv8 and RISC-V that GG extends from RISC-to-RISC to CISC-to-RISC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AnghaBench supplies the one-million-program compilable C/C++ corpus from which the paired x86/RISC training set is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BringUpBench supplies the bare-metal, full-project benchmark used to test real-world generalization."}],"review_version":2}