{"id":"96470b58-8dda-4b56-ace2-e5c9f8ef26dd","arxiv_id":"2412.16135","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Large language models, particularly GPT-4o-mini and DeepSeekCoder-v2, can generate assembly code obfuscated by dead code, register substitution, and control flow changes.","lead":"Researchers created a dataset of 328,200 assembly code snippets obfuscated with three classic techniques and tested whether large language models like GPT-4o-mini can do the same obfuscation from a simple prompt. They found several models, especially GPT-4o-mini and DeepSeekCoder-v2, produce convincing obfuscated assembly, suggesting LLM-based obfuscation could be a usable tool for malware writers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Success is measured by textual divergence (delta entropy and cosine similarity), not by behavioral equivalence, and the thresholds were set post hoc from the very models being ranked, so the affirmative claim may count broken or trivial outputs as valid obfuscations.","rationale":"I read the paper as a benchmark claim: LLMs can perform assembly obfuscation, supported by a new dataset. The dataset construction and breadth are real contributions, and the multi-model comparison is a useful first map. However, the claim's truth condition is semantic preservation, and the evaluation never directly checks it. The post hoc threshold makes even the relative ranking hard to interpret because the metric is not anchored to a verified ground truth: GPT-4o-mini was used to fix the 'good' range and then reported as the best model. The human evaluation reinforces this gap because it ranks stylistic qualities rather than executing or verifying the outputs. The proposed concrete test would settle the concern on a tractable subset without requiring full-scale equivalence checking; since general equivalence is NP-hard, a sampled validation is the appropriate standard. Because the issue is addressable by adding an executable or symbolic validation subset, I do not recommend rejection: the paper should remain conditional, with the claim either validated on a sample or explicitly downgraded to 'text-level obfuscation.' The reader's conditional verdict is therefore appropriate, and my read does not change it.","tokens_in":13382,"tokens_out":4996,"duration_ms":46934,"concrete_test":"Take a stratified random sample of roughly 300 outputs: 50 per obfuscation technique from GPT-4o-mini and DeepSeekCoder-v2, plus 50 from StarCoder's high-delta control-flow outputs. Assemble the original and generated snippets with a standard x86 assembler (e.g., Keystone or NASM) and execute both under a set of representative initial register, memory, and flag states using Unicorn emulation; compare final register, memory, and flag state. If more than a few percent of sampled outputs fail to assemble or produce different observable behavior, the proxy-based success counts are not evidence of functional obfuscation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The affirmative answer rests entirely on two string-level scores, delta entropy and cosine similarity, applied to generated assembly text. The paper's own definition of obfuscation (Background) requires behavioral equivalence for all inputs, and the Evaluation section concedes that 'Functional correctness is an alternative method but it is impossible at this scale.' The chosen proxies measure textual divergence, not semantics: a model that emits syntactically plausible but semantically broken assembly can score well on delta entropy, while a model that copies the original and appends NOPs can score well on cosine similarity. The acceptable range was not pre-registered; it was fixed after human experts ranked outputs and picked GPT-4o-mini, so the top model partly defines the criterion used to declare success. Table 6's human review ranks insertion, substitution, and rearrangement quality among eight outputs, but it does not assemble or execute any output, so it cannot detect semantic breakage. Under these conditions, the central claim that LLMs 'can generate obfuscated assembly code' is not yet distinguished from the claim that LLMs can produce text that looks different while preserving no verified behavior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MetamorphASM, a benchmark and dataset (MAD) of 328,200 assembly-code samples paired with three obfuscation techniques: dead-code insertion, register substitution, and control-flow change. The authors evaluate eight LLMs (GPT-4o-mini, GPT-3.5, DeepSeekCoder-v2, Codestral, CodeLlama, CodeGemma, Starcoder, LLaMA 3.1) plus a fine-tuned CodeT5 baseline, using character-wise delta entropy and cosine similarity to measure obfuscation quality, with a 200-sample human expert review. They answer the central question in the affirmative, claiming that LLMs, especially GPT-4o-mini, can generate obfuscated assembly code at scale.","tokens_in":13562,"tokens_out":4900,"duration_ms":42627,"significance":"If the main claim were established, the result would be practically significant: it would show that LLMs can act as low-cost, platform-independent metamorphic engines, with direct implications for malware detection and for the design of anti-virus evaluations. The paper's strengths are the scale and release of the dataset, the breadth of models compared, and the inclusion of human review in the evaluation. However, the core evaluation does not verify that generated outputs preserve program behavior, and the success thresholds are calibrated on the very models being ranked; the central affirmative claim is therefore not yet supported at the level the abstract and conclusion assert.","major_comments":[{"comment":"The two evaluation metrics, delta entropy and cosine similarity, are textual similarity measures, not semantic equivalence checks. The Background section defines obfuscation as P'(x) ≡ P(x) for all inputs, but the evaluation never tests this condition. A model that emits syntactically plausible but semantically broken assembly can still score high on delta entropy, and a model that copies the original and appends NOPs can still score high on cosine similarity. The statement that \"Functional correctness is an alternative method but it is impossible at this scale\" justifies why a full equivalence check is not used, but it does not validate the proxies. The paper needs a subsample-based behavioral validation (e.g., assembling outputs and executing them over randomized inputs, or an SMT-based equivalence check on a sample) to calibrate delta entropy and cosine similarity against actual correctness. Without this, the model rankings and the affirmative claim rest on unproxied string-level scores.","section":"Evaluation"},{"comment":"The acceptable ranges—delta entropy of 10–20% and cosine similarity above 0.9—are defined after human experts examined outputs from these same eight models and selected GPT-4o-mini as the closest to human-performed obfuscation. Using these post hoc thresholds to then rank the same models is circular: the top model partially defines the criterion by which all models are judged. The thresholds should be justified independently (for example, by correlation with a held-out correctness check or by pre-registering the criteria) rather than by fitting to the evaluated outputs.","section":"Results and Discussion"},{"comment":"The control-flow-change ground truth in MAD is described as being produced by \"randomly rearrang[ing] parts of the code\" with no stated analysis of data dependencies or instruction independence. Random reordering of assembly instructions does not generally preserve semantics; a snippet in which an instruction is moved before a dependent instruction will break the original behavior. If part of MAD's control-flow ground truth is semantically invalid, then model performance on that obfuscation technique cannot be interpreted as success or failure. An independent equivalence check or at least a dependence-aware construction procedure is needed for the dataset to support the benchmark's claims.","section":"Dataset Metrics"},{"comment":"Several reported values are difficult to reconcile with the stated thresholds and narrative. For example, in Table 3, GPT-4o-mini's dead-code delta entropy is 26.90% at 0-shot and 21.00% at 1-shot, both outside the stated 10–20% expected range, and Starcoder at 1-shot has delta entropy 45.55% with cosine similarity 0.97, a combination that the thresholds do not cleanly classify. The paper should provide an explicit success-rate table based on the thresholds and discuss these borderline or contradictory cases, especially where a model's human ranking in Table 6 does not match its numerical scores.","section":"Tables 3-5"}],"minor_comments":[{"comment":"The formula for delta entropy is malformed: it reads ΔH_AB = 1/N ∑_{x∈n}(|H(A) − H(B)|), but the summation variable x is unused and the set n is undefined; the normalization also needs clarification.","section":"Evaluation"},{"comment":"The abstract mentions \"GPT-3.5/4\" while the experiments use GPT-3.5 and GPT-4o-mini; please align the model naming throughout the paper.","section":"Models and Evaluation"},{"comment":"LLaMA 3.1 is labeled as proprietary, but its weights are openly released; the open/proprietary classification should be corrected.","section":"Models and Evaluation"},{"comment":"The text states that 15,000 samples were selected from \"our extensive repository of 300,000 examples,\" while the dataset is elsewhere described as containing 328,200 samples; these numbers should be reconciled.","section":"Models and Evaluation"},{"comment":"Table 6's criteria are described only as \"ranking the eight outputs\" for insertion, substitution, and rearrangement. Please state explicitly whether the evaluators were permitted to assemble or execute the code or whether the review was purely visual; this affects how much weight the review can carry for semantic preservation.","section":"Human Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful dataset and a clear experimental setup, but the central claim is currently supported only by string-similarity metrics with post hoc thresholds and no behavioral equivalence check. The major comments are all addressable within the scope of a revision: add a small but rigorous behavioral validation subsample, re-derive or de-emphasize the thresholds, and temper the affirmative conclusion until the proxies are calibrated. If the authors instead retain the current evaluation without these fixes, I would not recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing about this paper is that the MetamorphASM dataset is real and useful, while the headline empirical claim is plausible but not fully established. The authors built the first assembly-level obfuscation benchmark, 328,200 (original, obfuscated) pairs across three classic techniques, and they're releasing it. That alone is a solid contribution. The secondary finding, that GPT-4o-mini beats code-specialized models like DeepSeekCoder-v2 and Codestral at this task, is genuinely interesting and not what you'd naively expect.\n\nWhat the paper does well: it's systematic, tests a wide range of models with zero-shot and few-shot prompting, includes a small human study, and the authors are transparent about the limits of their evaluation. They openly say functional correctness checking is infeasible at scale and that they rely on delta entropy and cosine similarity instead. That honesty is appreciated.\n\nThe soft spot is exactly where the stress-test lands. Delta entropy and cosine similarity are string-level metrics. They measure textual divergence, not behavioral equivalence. A model that emits syntactically plausible but semantically broken assembly can score well on delta entropy; a model that copies the original and appends NOPs might still clear cosine similarity. The acceptable ranges (10-20% delta entropy, CS > 0.9) were set after the human experts looked at the same LLM outputs and picked GPT-4o-mini as the best, so the top model partially defines the criterion used to declare success. That's circular, even if not maliciously so. The 200-sample human review is too small to validate the roughly 45,000 scored outputs, and it ranks outputs rather than testing whether they execute correctly. So the central claim, that LLMs can generate obfuscated assembly at scale, is not yet cleanly separated from the claim that LLMs can produce assembly text that looks different.\n\nThis is a real weakness, but it's a fixable one and it doesn't sink the paper. The dataset itself is script-generated and human-vetted, so the benchmark remains valuable regardless of the LLM evaluation. A revision with a validation subset (assemble and run a few hundred outputs, or use SMT on a sample), pre-registered thresholds, and repeated sampling would substantially tighten the main claim. As it stands, the paper deserves serious peer review and a conditional accept rather than a desk reject or a full accept. I'd cite it for the dataset if I worked in malware detection or binary analysis, and I might bring it to reading group.","headline":"The dataset is a genuine contribution, but the LLM evaluation rests on post-hoc text-similarity proxies that don't actually verify the obfuscated code works.","tokens_in":14200,"tokens_out":1775,"would_cite":true,"duration_ms":17315,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models, especially GPT-4o-mini and DeepSeekCoder-v2, can transform short assembly snippets into new obfuscated variants, and the paper's MetamorphASM benchmark quantifies this capability across 328,200 samples.","keywords":["assembly code obfuscation","large language models","MetamorphASM","metamorphic malware","dead code insertion","register substitution","control flow change","obfuscation benchmark"],"falsifier":"Take a random subset of pairs that the scores call successful, assemble and run both the original and obfuscated snippets on the same inputs (or check them with an SMT-based equivalence tool); if a nontrivial fraction fail to assemble or produce different outputs, the claim that LLMs generate valid obfuscated assembly is not supported.","tokens_in":13145,"feed_emoji":"🤖","tokens_out":7667,"duration_ms":61205,"temperature":0.7,"pith_summary":"The paper asks whether LLMs can take short assembly snippets and emit new, obfuscated assembly that a malware author could use without the original source code. It answers yes, building the MetamorphASM benchmark and the 328,200-sample MAD dataset to test the idea across three obfuscation techniques: dead code insertion, register substitution, and control flow change. Across GPT-3.5/4, GPT-4o-mini, and several open-source code models, the authors find that general-purpose GPT models, especially GPT-4o-mini, outperform specialized coder LLMs like DeepSeekCoder-v2, Codestral, CodeLLAMA, CodeGemma, and Starcoder. Success is scored by delta entropy and cosine similarity, backed by a manual expert review, rather than by executing the obfuscated code. If right, the result lowers the bar for polymorphic malware: an attacker can call an API to rewrite code piece by piece instead of shipping a full metamorphic engine.","feed_headline":"LLMs can generate obfuscated assembly code at scale, benchmark shows","feed_subtitle":"A 328,200-sample benchmark finds general GPT models beat specialized code LLMs at rewriting assembly.","key_machinery":"The load-bearing object is the MetamorphASM benchmark with its MAD dataset: 328,200 assembly pairs built by mechanically applying dead code insertion, register substitution, and control flow change to 20-instruction snippets. Evaluation relies on two scores: delta entropy, the character-level entropy difference between original and generated code, and cosine similarity between the two. The good ranges (roughly 10–20% entropy change, similarity above 0.9) were calibrated by three human experts and later spot-checked on 200 samples across eight LLMs.","core_discovery":"The central claim is that LLMs can generate new obfuscated assembly variants from plain assembly snippets, at scale, using only prompts and a few examples. To establish it, the authors construct MAD from disassembled Windows binaries and open-source x64 code, script three standard obfuscations to create 328,200 original-obfuscated pairs, and then ask LLMs to perform similar transformations. They report that GPT-4o-mini and DeepSeekCoder-v2 produce obfuscated code with delta entropy in a declared 10–20% good range and cosine similarity above 0.9 for dead code and control flow change, while CodeGemma and CodeLLAMA largely fail. The paper concludes that the GPT family has outstanding obfuscation performance even against specialized coder models.","pith_inferences":["Because the paper checks only statistical similarity plus 200 human-reviewed samples, the strongest inference to test next is whether the 'successful' outputs actually assemble and run identically; a symbolic-execution or emulation study on a few thousand pairs would settle that.","The same benchmark could be flipped into a deobfuscation task: train a model to recover the original snippet from an obfuscated one, using MAD's paired structure as supervision.","The 20-instruction snippets leave open whether LLM obfuscation scales to full functions or binaries, where control flow and data dependencies are more entangled than in isolated snippets.","The ranking suggests obfuscation skill tracks instruction-following and assembly exposure more than code-model specialization, which would predict that future general models keep improving this capability without security-specific training."],"forward_implications":["If the benchmark reflects real malware rewriting, an attacker can outsource obfuscation to an LLM API, so payloads need not carry a large metamorphic engine.","Static signature-based detection becomes less reliable even for small snippets, since a model can rephrase the same code on demand.","The released MAD gives defenders a shared corpus to train detectors, test deobfuscation, and measure how well anti-virus engines generalize to LLM-obfuscated code.","Among the tested models, control flow change produces the largest entropy shifts and register substitution the smallest, so defenses can prioritize the technique that changes code most.","Specialized coder LLMs are not automatically better at assembly obfuscation; general instruction-tuned models like GPT-4o-mini lead."],"supporting_citations":[{"why":"Supplies the delta entropy metric the paper uses as its large-scale proxy for semantic equivalence in obfuscation.","marker":"Yang et al. (2024)"},{"why":"Grounds the dead code insertion technique used to build one third of the MAD dataset.","marker":"Na, Choi, and Lee 2023"},{"why":"Grounds the register substitution technique used to build another third of the MAD dataset.","marker":"Balakrishnan and Schulze 2005"},{"why":"Grounds the control flow change and instruction permutation technique used to build the final third of the MAD dataset.","marker":"Linn and Debray 2003"},{"why":"Provides the DeepSeekCoder-v2 baseline model that the paper compares against GPT models.","marker":"Zhu et al. 2024"},{"why":"Provides the CodeLLAMA baseline model used in the benchmark comparison.","marker":"Roziere et al. 2023"},{"why":"Provides the CodeT5 model that the paper fine-tunes on MAD as a trainable baseline.","marker":"Wang et al. 2021"},{"why":"Supports the paper's argument that formal equivalence checking is sound but not complete, motivating the statistical evaluation metrics.","marker":"Tristan, Govereau, and Morrisett 2011"}],"fun_headline_variants":["LLMs can obfuscate assembly code at scale, benchmark shows","GPT-4o-mini outshines code-specific LLMs in assembly obfuscation","MetamorphASM: 328k assembly samples test LLM obfuscation skills","LLMs generate obfuscated code, posing new malware risk","Study: General LLMs beat code-tuned models at assembly obfuscation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that delta entropy and cosine similarity, scores saying the output changed the right amount without looking too different, signal that the obfuscated code still does what the original does, and the acceptable range was set after seeing the model results rather than by an independent correctness test.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can obfuscate assembly code at scale, benchmark shows","GPT-4o-mini outshines code-specific LLMs in assembly obfuscation","MetamorphASM: 328k assembly samples test LLM obfuscation skills","LLMs generate obfuscated code, posing new malware risk","Study: General LLMs beat code-tuned models at assembly obfuscation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1824,"prompt_tokens":989,"completion_tokens":835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":739}},"tokens_in":605,"tokens_out":835,"duration_ms":6304,"temperature":1.0,"reasoning_tokens":739,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:45:06.695818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subset of pairs that the scores call successful, assemble and run both the original and obfuscated snippets on the same inputs (or check them with an SMT-based equivalence tool); if a nontrivial fraction fail to assemble or produce different outputs, the claim that LLMs generate valid obfuscated assembly is not supported.","supporting_citations":[],"review_version":1}