JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation

Guoqiang Chen; Xin Jin; Zhiqiang Lin

REVIEW 2 major objections 6 minor 86 references

JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation

T0 review · 2 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read JsDeObsBench demonstrates that, on 36,260 execution-verifiable obfuscated JavaScript programs, LLMs deobfuscate more readably than rule-based tools yet fail to preserve executable semantics in 37.40% of outputs.

desk verdict A solid first benchmark for LLM JS deobfuscation, but the headline simplification advantage is inflated because it includes semantically broken outputs. read the letter →

arxiv 2506.20170 v1 pith:INAEXU7D submitted 2025-06-25 cs.CR

Guoqiang Chen , Xin Jin , Zhiqiang Lin This is my paper

classification cs.CR

keywords JavaScriptDeobfuscationBenchmarkLargeLanguageModelsCodeSimplificationMalwareAnalysisObfuscationTransformationsIn-contextLearningSemanticCorrectness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces JsDeObsBench, a benchmark that measures how well large language models reverse JavaScript obfuscation. It constructs 36,260 obfuscated programs with ground truth, plus 4,515 obfuscated malicious scripts, by applying seven common transformations to filtered coding-challenge solutions, and evaluates six LLMs against two rule-based deobfuscators using four automated checks. The central finding is two-sided: LLM outputs are on average substantially more simplified and closer to the original code than baseline outputs, yet LLM outputs fail syntax checks 2.76% of the time and execution checks 37.40% of the time, while the baselines rarely fail either check. The paper therefore claims LLMs are promising aids for human analysts but are not yet reliable end-to-end deobfuscators.

What carries the argument

The load-bearing machinery is a four-stage evaluation pipeline. Stage one parses each deobfuscation output with a standard ECMAScript parser; stage two runs syntactically valid outputs against the program's input/output test cases inside an isolated container; stage three measures simplification as the relative reduction in a complexity measure derived from counts of operators and operands, from obfuscated to deobfuscated code; stage four scores readability with CodeBLEU, a code-aware similarity metric that compares abstract syntax trees and data-flow graphs between the output and the original program. The dataset is built by taking 1,298 filtered, execution-verified coding-challenge solutions and applying seven transformation types—code compacting, name obfuscation, string obfuscation, dead code injection, control-flow flattening, debug protection, and self-defending—singly and in combination, yielding 36,260 unique obfuscated programs. A one-shot in-context prompt supplies one obfuscated/deobfuscated example to guide each LLM, and for malware an instrumented behavior-tracing sandbox substitutes for test cases when checking semantic preservation.

What would settle it

Take a fresh sample of 1,000 obfuscated scripts collected from live websites whose unobfuscated behavior can be independently recovered, run the same six LLMs and two rule-based deobfuscators through the paper's four evaluators, and check whether the roughly 1.72x simplification advantage and the 37.40% execution-failure rate reproduce; if they do not, the benchmark's representativeness claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that, measured on this benchmark, LLMs are strong at the human-facing half of deobfuscation and weak at the machine-facing half. The best LLM, GPT-4o, achieves 93.42% execution correctness, a 0.3825 simplification score, and a 0.6702 similarity score, while the best open-weights code model, Codestral, reaches 84.16% execution correctness. Rule-based baselines pass syntax 100% of the time and execution about 84% of the time but simplify poorly, with a best simplification score of 0.2214. Averaged across six LLMs, outputs are syntactically valid 97.23% of the time, execute correctly 60.93% of the time, and score 12.13 percentage points higher on simplification and 1.6 points higher on similarity than the baselines. The paper further claims that string obfuscation is the hardest transformation, with 46.63% average execution correctness, code compacting is the easiest at 79.18%, combining more than six transformations defeats all tested methods, and on the 4,515 obfuscated malware samples LLMs keep their simplification lead, averaging 28.85% higher simplification than baselines, while execution correctness drops sharply and GPT-4o refuses many requests for safety reasons.

Load-bearing premise

The benchmark's results transfer to real-world obfuscated JavaScript only if coding-challenge programs obfuscated with one popular obfuscator's seven transformation types are representative of the obfuscated scripts, including malware, that security analysts actually encounter.

Editorial extensions

If this is right

Security analysts can use LLM-based deobfuscation to make obfuscated scripts substantially more readable: the best LLM simplification score is 1.72 times the best baseline score, and LLM outputs are closer to the original code on CodeBLEU.
LLM deobfuscation is not yet safe to run unattended: 37.40% of LLM outputs fail execution checks even after passing syntax checks, so any practical deployment must verify behavior before relying on the result.
The benchmark shows a performance cliff with combined transformations: execution pass rates collapse for every method as the number of transformations grows, and six or more transformations mark near-total failure for most LLMs.
The released dataset, evaluation pipeline, and leaderboard give future LLM releases a reusable, execution-verifiable yardstick for JavaScript deobfuscation, so progress on syntax and semantic correctness can be tracked directly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the simplification advantage survives on live obfuscated web scripts, a practical hybrid could pair an LLM with a rule-based verifier: accept the LLM's output only when syntax and behavioral checks pass, which would filter out much of the 37.40% execution-failure rate before the code reaches an analyst.
The malware results hint that CodeBLEU similarity penalizes LLMs for legitimately rewriting obfuscated originals; the paper's own case example, with low similarity but identical behavior, suggests similarity should not be read as semantic fidelity for malware.
The single-obfuscator, seven-transformation design leaves open how LLMs fare against newer obfuscation ideas, so extending the pipeline to additional transformation types is the most direct test of the paper's generalizability claim.
A natural follow-up measurement is human-analyst readability: if analyst comprehension time tracks the simplification score, the readability claim is strengthened, and if it does not, the score needs recalibration.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know up front. This is the first execution-verifiable benchmark for LLM-based JavaScript deobfuscation, and the artifact is real: dataset, code, and leaderboard are all out there. But the headline simplification claim (1.72x better than baselines) is computed on outputs that pass syntax only, not execution. Since LLMs fail execution 37.4% of the time versus 16.25% for the baselines, a large slice of their simplification score comes from shorter, semantically wrong code. Halstead length rewards deletion, so the main selling point is partly an artifact.

What the paper does well: the dataset construction is careful. They filter 58k CodeNet JS solutions down to 1,298 unique problems with execution-verified ground truth, apply seven transformations from a widely used obfuscator, and build 36k obfuscated programs. The four-evaluator chain (syntax, execution, simplification, similarity) is a sensible way to separate concerns, and the execution check with real test cases gives genuine semantic ground truth. The case studies of failure modes—self-repeating, context-window truncation, and semantic manipulation like mixing up \x01 and line indices—are genuinely informative. The paper also concedes its own limitations in §7.2: the dataset skews toward coding-challenge programs, and the malware set mixes obfuscated and clean samples.

The soft spots, in order of importance. First, the simplification/similarity metrics are computed only after the syntax check, as stated in §4.3, but not after the execution check. Recomputing those scores on execution-passing outputs is the obvious fix, and it should change the central RQ3 conclusion. Second, all LLM runs are single-sample with temperature 0.1; there are no variance estimates, so claims like "Codestral outperforms Mixtral" rest on point estimates. Minor, but worth a line. Third, CodeNet solutions are public, and the LLMs may have memorized the originals; the obfuscation reduces but does not remove that confound. A held-out set of newly collected programs would strengthen the benchmark. Fourth, the malware evaluation relies on behavior traces in Box-js, which is a rough proxy, and the paper says so.

None of this is fatal. The execution-correctness gap is real and well measured: LLMs generate semantically valid deobfuscations less often than the heuristic tools. The simplification advantage is the part that needs reanalysis. This is a useful measurement contribution for the security and LLM evaluation communities, and the authors have done the hard work of releasing everything needed to rerun it.

For peer review: yes, send it. A serious referee can handle the simplification issue in a revision cycle; desk rejection would be wrong. If it were my call, I would ask for the execution-conditioned scores before accepting.

Referee Report

2 major / 6 minor

Summary. The paper introduces JsDeObsBench, a benchmark for LLM-based JavaScript deobfuscation. The authors construct 36,260 obfuscated general JS programs from 1,298 filtered CodeNet solutions by applying seven JavaScript-Obfuscator transformations and their combinations, plus 4,515 obfuscated malware samples. They propose a chain of four evaluators: syntax correctness, execution correctness, simplification (Halstead-length reduction), and similarity (CodeBLEU). The study evaluates six LLMs and two rule-based deobfuscators, reporting that LLMs outperform baselines on simplification and similarity but have higher syntax and execution failure rates (2.76% and 37.40% on average), and that GPT-4o leads overall while Codestral is the best open-weight model.

Significance. If the measurements are valid, JsDeObsBench would fill a genuine gap: it provides an execution-verifiable deobfuscation dataset with ground truth, a multi-faceted automated evaluation pipeline, and a public leaderboard. The use of external CodeNet test cases for semantic verification is a real strength, as is the release of the dataset and code. However, the headline simplification result is computed on a subset that includes semantically broken outputs, and the public CodeNet source raises contamination concerns that are not addressed. The significance of the empirical claims therefore depends on the follow-up analyses requested below; the benchmark artifact itself is a useful contribution.

major comments (2)

[§4.3, Eq. (1), Tables 4 and 5] The simplification and similarity evaluators are applied to every deobfuscated program that passes the syntax check, regardless of whether it passes the execution check. Since the LLMs have an average execution failure rate of 37.40% versus 16.25% for the baselines (RQ2 answer, §5.3), and since Eq. (1) rewards aggressive shortening, the central RQ3 claim of superior simplification (e.g., GPT-4o 0.3825 vs Synchrony 0.2214 in Table 4, quoted as 1.72x in the abstract) is confounded: part of the advantage may come from syntactically valid but semantically altered, truncated, or otherwise broken code. The paper should report simplification and CodeBLEU scores conditioned on execution success, alongside the current unconditional numbers, and should state explicitly whether any outputs that fail the execution check are excluded from the RQ3 headline.
[§3.1, §4.1, and §7.2] The dataset is built from public CodeNet solutions, and the one-shot demonstration is drawn from the same coding-challenge distribution as the test samples, while §3.1 itself notes that LLM memorization prevents the use of publicly accessible obfuscated programs. This creates a contamination risk that is not addressed: the LLMs may, in part, be recalling original solutions from pretraining, which would inflate simplification and similarity scores and undermine the abstract's 'real-world scenarios' claim. The paper should add a contamination analysis (for example, token n-gram overlap between the test programs and public training corpora, or a holdout evaluation on programs from a different source), and disclose whether the one-shot example is disjoint from the test set.

minor comments (6)

[§5.1] The inference settings say 'we set both top_p and top_n to 1'; this appears to be a typo for top_k, and the temperature of 0.1 should be clarified as a greedy or near-greedy decoding choice rather than a deterministic one.
[Table 2] The row labeled 'Mixtral 7B' lists the model ID 'Mistral-7B-Instruct-v0.3', which is not a Mixtral model; either correct the family name or the model ID.
[Abstract and §5.2] The abstract's '1.72x better code simplification score' and the text's '16.11% lead' refer to different quantities (a ratio versus a percentage-point difference); the paper should disambiguate these to avoid confusion.
[Appendix §A.5] The appendix says the example is 'taken from the Mixtral deobfuscation' while Figure 17(c) labels the output as produced by Codestral; this inconsistency should be fixed.
[Figure 10] The caption refers to zero-shot (Left) and one-shot (Right) panels, but the figure itself does not visually label the two groups; adding explicit labels would help the reader.
[References] Reference [28] contains the typo 'Antcoder developers'; it should be 'AtCoder developers'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's correctness labels come from external CodeNet test cases and original programs, and the simplification-metric caveat is a validity issue, not a circular derivation.

full rationale

JsDeObsBench does not fit parameters to the quantities it later reports as results. Ground-truth labels are the original CodeNet programs and their external test cases (Section 4.1), and malware behavior traces are checked with Box-js; all four evaluators are applied to LLM outputs after the fact. The simplification score S = 1 - HLoC_deobf/HLoC_obf is an independent operationalization, not an input to the LLM or the baselines. Self-citations ([41], [42], [43], [78], [79]) appear only as background or as motivation for in-context learning; none carries a uniqueness claim or supplies a load-bearing premise. The one-shot example and the test instances both come from coding-challenge programs, which is a distribution-familiarity and generalizability concern, not circularity. Finally, the fact that simplification and similarity scores are computed on syntax-passing outputs before the execution filter (Section 4.3) is a real measurement-validity caveat for the 'superior simplification' headline, but it does not make the measured scores equal to the benchmark inputs by construction. Hence no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on representativeness and metric-validity assumptions rather than on fitted parameters or invented physical entities. All listed assumptions are stated or implied in the paper; several are conceded as limitations in §7.2.

assumptions (5)

domain assumption JavaScript-Obfuscator and its seven transformation types are representative of real-world JS obfuscation.
Used in §4.1 to construct the benchmark dataset; if real-world obfuscation differs, the benchmark's ranking may not transfer.
domain assumption CodeNet coding-challenge solutions with passing tests are a valid proxy for real-world JS programs.
Source selection in §4.1; the paper acknowledges in §7.2 that this excludes many real-world and malware scripts.
domain assumption Halstead length reduction measures deobfuscation quality and readability.
Equation (1) in §4.3 defines the simplification score; it assumes shorter code is better without validating human comprehension.
domain assumption CodeBLEU similarity to the original code approximates readability.
Similarity evaluator in §4.3; the authors state this is an automated proxy for human assessment.
ad hoc to paper Public CodeNet programs are not memorized by the tested LLMs in a way that inflates deobfuscation scores.
The leakage concern is raised in §3.1, but no contamination test is performed; the source programs are public.

how reviews work

0 comments

Cite this review

Pith. "Pith review of JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation." pith.science (2026). https://pith.science/paper/INAEXU7D

@misc{pith2026250620170,
  author       = {Pith},
  title        = {Pith review of: JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INAEXU7D}},
  note         = {Machine review of arXiv:2506.20170}
}

read the original abstract

Deobfuscating JavaScript (JS) code poses a significant challenge in web security, particularly as obfuscation techniques are frequently used to conceal malicious activities within scripts. While Large Language Models (LLMs) have recently shown promise in automating the deobfuscation process, transforming detection and mitigation strategies against these obfuscated threats, a systematic benchmark to quantify their effectiveness and limitations has been notably absent. To address this gap, we present JsDeObsBench, a dedicated benchmark designed to rigorously evaluate the effectiveness of LLMs in the context of JS deobfuscation. We detail our benchmarking methodology, which includes a wide range of obfuscation techniques ranging from basic variable renaming to sophisticated structure transformations, providing a robust framework for assessing LLM performance in real-world scenarios. Our extensive experimental analysis investigates the proficiency of cutting-edge LLMs, e.g., GPT-4o, Mixtral, Llama, and DeepSeek-Coder, revealing superior performance in code simplification despite challenges in maintaining syntax accuracy and execution reliability compared to baseline methods. We further evaluate the deobfuscation of JS malware to exhibit the potential of LLMs in security scenarios. The findings highlight the utility of LLMs in deobfuscation applications and pinpoint crucial areas for further improvement.

Figures

Figures reproduced from arXiv: 2506.20170 by the authors.

**Figure 1.** JsDeObsBench Overview. Length Repetitive Functionality Execution Solution Data Online Website Data High-quality JavaScript Dataset JavaScript Programs Open-source Repositories Code Snippets Data Sources Sample Filters [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** JS Source Selection and Filtering. Among the four [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Syntax and Execution Correctness of LLMs and Baselines across Obfuscation Transformations. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (15 more)

**Figure 4.** Figure 4: Code Simplification Scores of LLMs and Baselines across Obfuscation Transformations. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Syntax and Execution Correctness of LLMs and [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Code Simplification Scores of LLMs and Baselines [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Code Similarity Scores of LLMs and Baselines across Obfuscation Transformations. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

**Figure 8.** Figure 8: Code Similarity Scores of LLMs and Baselines [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

**Figure 9.** Figure 9: In-context Learning Prompts for LLM-based JS De [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Syntax and Execution Correctness of LLMs with [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

**Figure 11.** Figure 11: Syntax and Execution Correctness Evaluation [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]

**Figure 12.** Figure 12: Syntax and Execution Correctness of LLMs and Our Baselines across Various Obfuscation Transformations and [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

**Figure 13.** Figure 13: An Example of Self-repeating Error in LLM Deobfuscation Output. In subfigure (c), LLM repeats the input prompt [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: An Example of Error Caused by Context Window Size Limitation in LLM Deobfuscation. In subfigure (c), the LLM [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

**Figure 15.** Figure 15: An Example of Semantic Manipulation Error in LLM Deobfuscation. In subfigure (a), the original JS code concatenates [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]

**Figure 16.** Figure 16: Two Examples of LLM Refusing to Perform Deobfuscation. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

**Figure 17.** Figure 17: An Example of Successful JavaScript Deobfuscation by the LLM. For deobfuscation, the Codestral model not only [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

**Figure 18.** Figure 18: An Example of Deobfuscated JS Malware with an LLM. The deobfuscated code has a 0.0946 low CodeBLEU with the [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

86 extracted references · 56 canonical work pages

[1]

Collection of almost 40.000 javascript malware samples

2024. Collection of almost 40.000 javascript malware samples. https://github. com/HynekPetrak/javascript-malware-collection

2024
[2]

DeepSpeed

2024. DeepSpeed. https://github.com/microsoft/DeepSpeed

2024
[3]

A deobfuscator for scripts obfuscated by Obfuscator.io

2024. A deobfuscator for scripts obfuscated by Obfuscator.io. https://github.com/ ben-sb/obfuscator-io-deobfuscator

2024
[4]

Docker Official Image, Node.js is a JavaScript-based platform for server-side and networking applications

2024. Docker Official Image, Node.js is a JavaScript-based platform for server-side and networking applications. https://hub.docker.com/_/node

2024
[5]

EvalPlus Leaderboard

2024. EvalPlus Leaderboard. https://evalplus.github.io/leaderboard.html

2024
[6]

General purpose JavaScript deobfuscator

2024. General purpose JavaScript deobfuscator. https://github.com/ben-sb/ javascript-deobfuscator

2024
[7]

Hugging Face Hub

2024. Hugging Face Hub. https://huggingface.co/docs/hub/index

work page 2024
[8]

A JavaScript checker and optimizer

2024. A JavaScript checker and optimizer. https://github.com/google/closure- compiler

work page 2024

Show all 86 references

[9]

javascript cleaner & deobfuscator

2024. javascript cleaner & deobfuscator. https://github.com/relative/synchrony

2024
[10]

JavaScript Deobfuscator and Unpacker

2024. JavaScript Deobfuscator and Unpacker. https://github.com/lelinhtinh/de4js

2024
[11]

JavaScript Minifier is an easy-to-use tool for minifying JavaScript code

2024. JavaScript Minifier is an easy-to-use tool for minifying JavaScript code. https://www.toptal.com/developers/javascript-minifier

2024
[12]

JavaScript parser / mangler / compressor / beautifier toolkit

2024. JavaScript parser / mangler / compressor / beautifier toolkit. https://github. com/mishoo/UglifyJS

2024
[13]

JS reverse minifier based on statistical machine translation

2024. JS reverse minifier based on statistical machine translation. https://github. com/bvasiles/jsNaughty

2024
[14]

Node.js ® is a free, open-source, cross-platform JavaScript runtime envi- ronment that lets developers create servers, web apps, command line tools and scripts

2024. Node.js ® is a free, open-source, cross-platform JavaScript runtime envi- ronment that lets developers create servers, web apps, command line tools and scripts. https://nodejs.org/

2024
[15]

Obfuscate JavaScript (beyond repair) with Ruby

2024. Obfuscate JavaScript (beyond repair) with Ruby. https://github.com/rapid7/ jsobfu

2024
[16]

Obfuscate string literals in JavaScript code

2024. Obfuscate string literals in JavaScript code. https://github.com/anseki/ gnirts

2024
[17]

A powerful obfuscator for JavaScript and Node.js

2024. A powerful obfuscator for JavaScript and Node.js. https://github.com/ javascript-obfuscator/javascript-obfuscator

2024
[18]

2024. PyTorch. https://pytorch.org/

2024
[19]

STATISTICAL RENAMING, TYPE INFERENCE AND DEOBFUSCATION

2024. STATISTICAL RENAMING, TYPE INFERENCE AND DEOBFUSCATION. http://jsnice.org/

2024
[20]

This repository contains a list of pseudo-sorted malicious JavaScripts collected from time to time

2024. This repository contains a list of pseudo-sorted malicious JavaScripts collected from time to time. https://github.com/geeksonsecurity/js-malicious- dataset

2024
[21]

Transformers

2024. Transformers. https://huggingface.co/

2024
[22]

A utility to analyze malicious JavaScript

2024. A utility to analyze malicious JavaScript. https://github.com/CapacitorSet/ box-js

2024
[23]

Rohan Bavishi, Michael Pradel, and Koushik Sen. 2018. Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts. doi:10.48550/arXiv.1809.05193

2018 doi arXiv
[24]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

2020
[25]

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21)...

2021
[26]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

2021 arXiv
[27]

Charlie Curtsinger, Benjamin Livshits, Benjamin Zorn, and Christian Seifert
[28]

Antcoder developers. [n. d.]. Atcoder. https://atcoder.jp/ Accessed on 11-29-2024

2024
[29]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

2018 arXiv
[30]

Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information...

2024
[31]

ECMA International. 2017. ECMA-262: ECMAScript Language Specifica- tion. https://ecma-international.org/publications-and-standards/standards/ ecma-262/

2017
[32]

Aurore Fass, Michael Backes, and Ben Stock. 2019. Hidenoseek: Camouflaging malicious javascript in benign asts. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security . 1899–1913

2019
[33]

Aurore Fass, Robert P Krawczyk, Michael Backes, and Ben Stock. 2018. Jast: Fully syntactic detection of malicious (obfuscated) javascript. In Detection of Intrusions and Malware, and Vulnerability Assessment: 15th International Conference, DIMV A 2018, Saclay, France, June 28–...

2018
[34]

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guant- ing Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang
[35]

Maurice H Halstead. 1977. Elements of Software Science (Operating and program- ming systems series). Elsevier Science Inc

1977
[36]

Adrian Herrera. 2020. Optimizing Away JavaScript Obfuscation. In Source Code Analysis and Manipulation 2020 (SCAM’20) . arXiv. doi:10.48550/arXiv.2009.09170 arXiv:2009.09170 [cs]

2020 doi arXiv
[37]

Ariya Hidayat. 2024. ECMAScript parsing infrastructure for multipurpose analy- sis. https://esprima.org

2024
[38]

Peiwei Hu, Ruigang Liang, and Kai Chen. 2024. DeGPT: Optimizing Decompiler Output with LLM. In Proceedings 2024 Network and Distributed System Security Symposium (2024). https://api. semanticscholar. org/CorpusID , Vol. 267622140

2024
[39]

Grégoire Jacob, Hervé Debar, and Eric Filiol. 2008. Behavioral detection of malware: from a survey towards an established taxonomy. Journal in computer Virology 4 (2008), 251–266

2008
[40]

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

2024 arXiv
[41]

Linxi Jiang, Xin Jin, and Zhiqiang Lin. [n. d.]. Beyond Classification: Inferring Function Names in Stripped Binaries via Domain Adapted LLMs. In Proceedings 2025 Network and Distributed System Security Symposium . San Diego, CA, USA

2025
[42]

Xin Jin, Jonathan Larson, Weiwei Yang, and Zhiqiang Lin. 2023. Binary code summarization: Benchmarking chatgpt/gpt-4 and other large language models. arXiv preprint arXiv:2312.09601 (2023)

2023 arXiv
[43]

Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. 2022. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1631–1645

2022
[44]

Mehran Jodavi, Mahdi Abadi, and Elham Parhizkar. 2015. JSObfusDetector: A binary PSO-based one-class classifier ensemble to detect obfuscated JavaScript code. In 2015 The International Symposium on Artificial Intelligence and Signal Processing (AISP). IEEE, 322–327

2015
[45]

Song Li, Mingqing Kang, Jianwei Hou, and Yinzhi Cao. 2022. Mining node. js vulnerabilities via object dependence graph and query. In 31st USENIX Security Symposium (USENIX Security 22) . 143–160

2022
[46]

Yi Li, Aashish Yadavally, Jiaxing Zhang, Shaohua Wang, and Tien N Nguyen. 2023. DeMinify: Neural Variable Name Recovery and Type Inference. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . ...

2023
[47]

Han Liu, Chengnian Sun, Zhendong Su, Yu Jiang, Ming Gu, and Jiaguang Sun
[48]

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35

2023
[49]

Meta AI. 2025. Introducing Llama 3.1: Our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/. Accessed: 2025-01-01

2025
[50]

Mistral AI. 2025. Codestral: Hello, World! Empowering developers and democratising coding with Mistral AI. https://mistral.ai/news/codestral/. Ac- cessed: 2025-01-01

2025
[51]

Marvin Moog, Markus Demmel, Michael Backes, and Aurore Fass. 2021. Statically Detecting JavaScript Obfuscation and Minification Techniques in the Wild. In 2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) (DSN 2021) . 569–580. doi:10....

2021
[52]

OpenAI. 2025. Hello GPT-4 Turbo. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-01-08

2025
[53]

OpenAI. 2025. OpenAI API. https://openai.com/api/. Accessed: 2025-01-07

2025
[54]

Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lam- bert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2023. Understanding the effectiveness of large language models in code translation. arXiv preprint arX...

2023 arXiv
[55]

Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. CodeNet: A Large...

2021
[56]

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

2018
[57]

Sampsa Rauti and Ville Leppänen. 2018. A Comparison of Online JavaScript Obfuscators. In 2018 International Conference on Software Security and Assurance (ICSSA) (ICSSA 2018). 7–12. doi:10.1109/ICSSA45270.2018.00012

2018
[58]

big code

Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting program properties from" big code". ACM SIGPLAN Notices 50, 1 (2015), 111–124

2015
[59]

Kunlun Ren, Weizhong Qiang, Yueming Wu, Yi Zhou, Deqing Zou, and Hai Jin. 2023. An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing ...

2023
[60]

Kunlun Ren, Weizhong Qiang, Yueming Wu, Yi Zhou, Deqing Zou, and Hai Jin. 2023. An Empirical Study on the Effects of Obfuscation on Static Machine Learning-Based Malicious JavaScript Detectors. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing ...

2023
[61]

Zhou, Ambrosio Blanco, and Shuai Ma

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. ArXiv abs/2009.10297 (2020). https://arxiv.org/ abs/2009.10297

2020 arXiv
[62]

Alan Romano, Daniel Lehmann, Michael Pradel, and Weihang Wang. 2022. Wob- fuscator: Obfuscating javascript malware via opportunistic translation to we- bassembly. In2022 IEEE Symposium on Security and Privacy (SP). IEEE, 1574–1589

2022
[63]

Alan Romano, Daniel Lehmann, Michael Pradel, and Weihang Wang. 2022. Wob- fuscator: Obfuscating JavaScript Malware via Opportunistic Translation to We- bAssembly. In 2022 IEEE Symposium on Security and Privacy (SP) (SP 2022) . 1574–

2022
[64]

Alan Romano, Yunhui Zheng, and Weihang Wang. 2020. Minerray: Semantics- aware analysis for ever-evolving cryptojacking detection. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering . 1129–1140

2020
[65]

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan X...

2024 arXiv
[66]

Andrea Saracino, Daniele Sgandurra, Gianluca Dini, and Fabio Martinelli. 2016. Madam: Effective and efficient behavior-based android malware detection and prevention. IEEE Transactions on Dependable and Secure Computing 15, 1 (2016), 83–97

2016
[67]

Shaown Sarker, Jordan Jueckstock, and Alexandros Kapravelos. 2020. Hiding in plain site: Detecting javascript obfuscation through concealed browser api usage. In Proceedings of the ACM Internet Measurement Conference . 648–661

2020
[68]

Mikhail Shcherbakov, Musard Balliu, and Cristian-Alexandru Staicu. 2023. Silent spring: Prototype pollution leads to remote code execution in Node. js. In 32nd USENIX Security Symposium (USENIX Security 23) . 5521–5538

2023
[69]

Philippe Skolka, Cristian-Alexandru Staicu, and Michael Pradel. 2019. Anything to hide? studying minified and obfuscated code in the web. In The world wide web conference. 1735–1746

2019
[70]

Philippe Skolka, Cristian-Alexandru Staicu, and Michael Pradel. 2019. Anything to Hide? Studying Minified and Obfuscated Code in the Web. In The World Wide Web Conference (WWW ’19) . Association for Computing Machinery, New York, NY, USA, 1735–1746. doi:10.1145/3308558.3313752

2019
[71]

Kwangwon Sun and Sukyoung Ryu. 2017. Analysis of JavaScript programs: Challenges and research trends. ACM Computing Surveys (CSUR) 50, 4 (2017), 1–34

2017
[72]

Hieu Tran, Ngoc Tran, Son Nguyen, Hoan Nguyen, and Tien N Nguyen. 2019. Re- covering variable names for minified code with usage contexts. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 1165–1175

2019
[73]

Alessio Trivisonno. [n. d.]. Reverse engineering : Code Deobfuscation in the age of AI. https://infosecwriteups.com/the-cybersecurity-revolution-at-the-age-of- ai-openai-and-code-deobfuscation-3f9dd278b900
[74]

Bogdan Vasilescu, Casey Casalnuovo, and Premkumar Devanbu. 2017. Recovering clear, natural identifiers from obfuscated JS names. InProceedings of the 2017 11th joint meeting on foundations of software engineering . 683–693

2017
[75]

Yutaka Watanobe. [n. d.]. Aizu online judge. https://onlinejudge.u-aizu.ac.jp
[76]

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al
[77]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

2022
[78]

Yuqing Yang, Yue Zhang, and Zhiqiang Lin. 2025. Understanding Miniapp Mal- ware: Identification, Dissection, and Characterization. InProceedings 2025 Network and Distributed System Security Symposium . San Diego, CA, USA

2025
[79]

Yue Zhang, Bayan Turkistani, Allen Yuqing Yang, Chaoshun Zuo, and Zhiqiang Lin. 2021. A Measurement Study of Wechat Mini-Apps. Proc. ACM Meas. Anal. Comput. Syst. 5, 2, Article 14 (June 2021), 25 pages. doi:10.1145/3460081

2021 doi
[80]

Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen. 2023. A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372 (2023)

2023 arXiv
[81]

"); 3 w=w-0; 4 h=h-0; 5 var ans=[]; 6 ans.push(

Joe Zimmerman. 2015. How to obfuscate programs directly. In Annual Interna- tional Conference on the Theory and Applications of Cryptographic Techniques . Springer, 439–467. JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation CCS ’25, October 13–17, 2025...

2015
[1589]

doi:10.1109/SP46214.2022.9833626 ISSN: 2375-1207

2022
[2011]

In 20th USENIX Security Symposium (USENIX Security 11)

ZOZZLE: Fast and precise In-Browser JavaScript malware detection. In 20th USENIX Security Symposium (USENIX Security 11)
[2017]

In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

Stochastic optimization of program obfuscation. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) . IEEE, 221–231

2017
[2022]

Emergent abilities of large language models.arXiv preprint arXiv:2206.07682 (2022)

2022 arXiv
[2024]

arXiv:2401.14196 [cs.SE]

DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. arXiv:2401.14196 [cs.SE]

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Collection of almost 40.000 javascript malware samples

2024. Collection of almost 40.000 javascript malware samples. https://github. com/HynekPetrak/javascript-malware-collection

2024

[2] [2]

DeepSpeed

2024. DeepSpeed. https://github.com/microsoft/DeepSpeed

2024

[3] [3]

A deobfuscator for scripts obfuscated by Obfuscator.io

2024. A deobfuscator for scripts obfuscated by Obfuscator.io. https://github.com/ ben-sb/obfuscator-io-deobfuscator

2024

[4] [4]

Docker Official Image, Node.js is a JavaScript-based platform for server-side and networking applications

2024. Docker Official Image, Node.js is a JavaScript-based platform for server-side and networking applications. https://hub.docker.com/_/node

2024

[5] [5]

EvalPlus Leaderboard

2024. EvalPlus Leaderboard. https://evalplus.github.io/leaderboard.html

2024

[6] [6]

General purpose JavaScript deobfuscator

2024. General purpose JavaScript deobfuscator. https://github.com/ben-sb/ javascript-deobfuscator

2024

[7] [7]

Hugging Face Hub

2024. Hugging Face Hub. https://huggingface.co/docs/hub/index

work page 2024

[8] [8]

A JavaScript checker and optimizer

2024. A JavaScript checker and optimizer. https://github.com/google/closure- compiler

work page 2024