REVIEW 5 major objections 6 minor 42 references
A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read RAWG claims that reward-driven reinforcement learning makes an LLM generator of obfuscated webshells more diverse, more executable, and much harder for VirusTotal to detect than prompt-engineered baselines.
desk verdict The paper runs a sensible SFT+PPO webshell pipeline but the reward-model loss in §3.2 is sign-inverted, so the stated mechanism cannot produce the reported escape rates; the claims are also unverifiable without code, data, or a defined diversity metric. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the argument is a preference reward model $R_\theta$ trained with a binary ranking loss on paired tuples $(x, y_{\text{good}}, y_{\text{bad}})$, where $y_{\text{good}}$ is benign code and $y_{\text{bad}}$ is a malicious webshell. The loss maximizes the sigmoid of the reward gap $\Delta R_\theta = R_\theta(x, y_{\text{good}}) - R_\theta(x, y_{\text{bad}})$, and the resulting reward enters the PPO objective $J(\phi) = \mathbb{E}_{y \sim \pi^{\text{RL}}_\phi(\cdot|x)}[R_\theta(x,y)] - \beta D_{\mathrm{KL}}(\pi^{\text{RL}}_\phi \| \pi^{\text{SFT}})$, which fine-tunes the supervised model while penalizing drift. Around this loop sits a corpus of 5,001 webshell and 5,936 benign PHP samples, balanced across seven obfuscation categories, with an LLM extracting and normalizing salient tokens so the model learns family-level obfuscation cues instead of boilerplate. Escape rate is defined as $1 - N_{\text{detected}}/N_{\text{generated}}$ against VirusTotal, survival rate as the fraction of generated samples that still execute, and rejection rate as the fraction of instructions the model refuses.
What would settle it
Retrain RAWG with a reward model that scores generated samples by VirusTotal's per-engine detection outcomes instead of the authors' benign/malicious labels, and test on detectors held out from training. If the detector-aware reward does not push escape rate clearly above the SFT-only baseline of 0.805, the proxy-reward premise fails; if it does, the paper's reported gains are explained by the reward model rather than by the RL loop alone.
Extended reading notes
Core claim
RAWG's central claim is that webshell generation can be improved by reframing it as preference learning: malicious webshell snippets are treated as 'chosen' data, benign code as 'rejected' data, and a reward model trained with a binary ranking loss encodes the difference. The PPO loop then drives the generator toward payloads that score high under this reward, which the authors identify with stealth and obfuscation quality. On Qwen2.5-Coder-14B this produces an escape rate of 0.857, a survival rate of 0.509, and a rejection rate of 0.030, compared with 0.824/0.453/0.131 for the Hybrid Prompt method run on GPT-4o and an escape rate of 0.232 for CWSOGG. The paper reads these numbers as evidence that reward-driven generation generalizes across obfuscation categories and scales with model capacity, rather than memorizing a single family of evasion tricks.
Load-bearing premise
The reward model is trained only on the authors' own pairings of malicious versus benign code, never on feedback from the VirusTotal detectors it is evaluated against, so the whole RL stage depends on the assumption that this proxy reward actually tracks real-world detection evasion.
Editorial extensions
If this is right
- If the reported escape and survival rates hold, red-team exercises can generate large batches of distinct, executable webshell payloads automatically instead of hand-tuning prompts for each test.
- The seven-category corpus becomes a benchmark on which detection engines can be scored per obfuscation family, exposing which defenses fail on which transformation tactics.
- The drop in rejection rate from 0.346 (Hybrid Prompt) to 0.030 (RAWG) on Qwen2.5-Coder-14B implies the trained policy bypasses safety refusals on webshell-generation instructions, so any deployment needs access controls.
- The comparison of SFT-only (0.805 escape) with SFT+RL (0.857) suggests the RL stage contributes a measurable improvement on top of supervised fine-tuning alone.
- The consistent gains across three backbone models indicate the method's benefits are not tied to one particular open-source LLM or decoding configuration.
Reading between the lines
- Editorial inference: because the reward model never sees detector output, the reported VirusTotal escape rate is also a test of whether the benign/malicious preference proxy transfers; a reward trained directly on detection outcomes could push evasion higher, and measuring that gap would isolate where RAWG's gains come from.
- Editorial inference: the same pipeline should be testable on other web languages, such as JSP or ASPX; training on PHP only and measuring escape-rate decay on non-PHP shells would quantify how much of the evasion is language-specific versus obfuscation-general.
- Editorial inference: by lowering the refusal rate and raising payload diversity, systems like RAWG also make evasive webshell generation more accessible to attackers, so the public release of the corpus and code is itself a decision with security consequences.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RAWG, a pipeline that organizes webshell samples into seven obfuscation categories, normalizes their tokens with an LLM, fine-tunes a code LLM with SFT, trains a reward model on HighBias/LowBias (malicious/benign) pairs, and then applies PPO to maximize the reward. The evaluation compares RAWG against Original Prompt, Hybrid Prompt, and CWSOGG on three LLMs and reports a headline Escape Rate of 0.857, Survival Rate of 0.509, and Rejection Rate of 0.030 on Qwen2.5-Coder-14B, together with sampling-parameter ablations. The paper additionally claims public release of the corpus and presents case studies of VirusTotal evasion and local execution.
Significance. If substantiated, RAWG would be a useful red-teaming resource: the seven-category obfuscation taxonomy and the use of VirusTotal as an external evaluation engine are concrete and relevant. The paper deserves credit for attempting reinforcement learning for webshell generation and for reporting a realistic detection metric. However, the core training objective as written is internally contradictory, the reward model is not trained on detector feedback, the diversity claim has no quantitative support, and no code or data are provided. These issues currently prevent the claimed contributions from being evaluated.
major comments (5)
- [§3.2, Eq. (3.2)] The binary ranking loss has the opposite sign from the stated preference. The text defines y_good ∈ LowBias, y_bad ∈ HighBias and states Rθ(x, y_good) < Rθ(x, y_bad), then minimizes L_Ranking = −E log σ(Rθ(x, y_good) − Rθ(x, y_bad)). Minimizing this loss makes the argument of the sigmoid large and positive, i.e., it trains Rθ(y_good) > Rθ(y_bad), the exact opposite of the stated inequality. Because the PPO objective J(ϕ) in the same section maximizes Rθ(x, y), the written method would reward benign code and suppress webshell generation, so the escape-rate improvements in Tables 1 and 2 cannot be derived from the described mechanism. The label semantics are also inconsistent: the abstract and Section 4.1 say malicious code is the 'chosen' sample, whereas Section 3.2 assigns y_good to LowBias (benign). No code or implementation detail is given to resolve whether the sign or the labels are wrong.
- [§3.2, §4.1] The reward model is never trained on or validated against detector output. Escape Rate is defined in Section 4.1 via VirusTotal, but the reward model in Section 3.2 is trained on the authors' HighBias/LowBias labels. The paper reports no correlation between Rθ scores and VirusTotal verdicts, no reward-model accuracy, and no analysis of whether the PPO reward correlates with the external metric. Without such evidence, the causal claim that the PPO stage (rather than SFT or decoding choices) improves VirusTotal evasion is not established.
- [Abstract, §4.1] The headline claim of 'payload diversity' is not measured. The metric list in Section 4.1 contains only Escape Rate, Survival Rate, and Rejection Rate. No token diversity, entropy, novelty, or family-coverage metric is defined or reported anywhere in Tables 1–2 or the ablations in Section 4.4. The discussion of 'moderate token diversity' in Section 4.4 is qualitative. A quantitative diversity metric is required to support the claimed advantage over prompt-engineering baselines.
- [§3.1, §4.1] It is unclear whether the training and evaluation data are disjoint. Section 3.1 describes a corpus built from 5,001 PHP webshell samples [34], while Section 4.1 says the evaluation set is constructed from prior work [21] with about 330 samples per category. The paper never states whether the evaluation categories or samples overlap the SFT/RL training data. If the generator is tested on material used in training, the reported escape and survival rates would be inflated. Please specify the split and release the exact evaluation set.
- [§4.1, Contribution 1] The reproducibility claims are not fulfilled. Section 4.1 states that each experiment is repeated three times and standard deviations are reported, but Tables 1 and 2 contain only point estimates. The contribution list promises public release of the corpus, yet no repository, URL, or data-access mechanism is provided in the manuscript. Given the sign ambiguity in Section 3.2, code and data are essential to verify the reported results.
minor comments (6)
- [§1] Section 1 contains a duplicated sentence fragment: 'Specifically, researchers still lack an attack-type–annotated benchmark that spans [16]' appears twice in consecutive sentences.
- [Table 2] Table 2, CWSOGG row: the third column entry '1 /' is malformed; either report a numeric rejection rate or state N/A.
- [§4.1] Section 4.1 contains typographical errors 'Currenta' and 'experiements'; please correct them.
- [§4.3] Section 4.3 refers to a case study 'on gpt-o3', but no such model is trained or evaluated in Section 4.1; please clarify which model produced Figure 3 and whether this is a transferability test.
- [References] References [23] and [24] are duplicate versions of the same Ouyang et al. paper; please merge them.
- [Figure 2] The caption of Figure 2 does not state the resulting class sizes after the truncation-based balancing described in Section 3.1; please add the exact numbers.
Circularity Check
No significant circularity: escape is measured against the external VirusTotal detector, and the reward model's internal-sign inconsistency is a correctness risk, not a circular reduction.
full rationale
The central claim—that PPO-trained RAWG escapes VirusTotal more often than prompt baselines—is not circular. The reward model in Section 3.2 is trained on the authors' own HighBias/LowBias pairs, but the headline Escape Rate is computed with VirusTotal (Section 4.1), an external detector not used during reward-model training or PPO; this is an independent, falsifiable measurement. Survival is checked by execution in a virtual environment, and Rejection Rate measures model refusals; none of these metrics is defined in terms of the reward model's own outputs. The seven-category corpus and SFT stage are self-contained inputs, and the paper does not define diversity or escape as equivalent to those inputs by construction. The self-citations ([4], [6], [17] in related work; [2,5] for inspiration) are descriptive and do not carry the empirical claim. The abstract's diversity claim is unmeasured, but an unmeasured claim is not circularity. I do flag one non-circularity defect: Eq. (3.2) states y_good belongs to LowBias and y_bad belongs to HighBias with R_theta(x,y_good) < R_theta(x,y_bad), while minimizing L_Ranking = -E log sigma(R_theta(x,y_good) - R_theta(x,y_bad)) trains the opposite preference, so the PPO objective J(phi) as written would suppress webshells rather than promote evasion. That is an internal-consistency and soundness problem, not a case of a prediction reducing to its own inputs, so it does not raise the circularity score.
Assumptions & free parameters
free parameters (3)
- KL coefficient beta in PPO objective =
not reported
- Sampling hyperparameters (temperature, top_p, top_k) =
1.0 / 0.8 / 50
- Learning rates and sequence length =
3e-5 (reward model), 2e-5 (RL), sequence length 2048
assumptions (4)
- domain assumption Seven obfuscation categories form a complete and accurate partition of webshell obfuscation tactics.
- ad hoc to paper The reward model trained on malicious/benign labels captures stealth and evasiveness.
- domain assumption VirusTotal is a valid and stable proxy for webshell detection effectiveness.
- domain assumption Generated samples can be judged as functional by executing them in a virtual environment.
Cite this review
Pith. "Pith review of A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming." pith.science (2026). https://pith.science/paper/IFHFP27U
@misc{pith2026250524252,
author = {Pith},
title = {Pith review of: A Reward-driven Automated Webshell Malicious-code Generator for Red-teaming},
year = {2026},
howpublished = {\url{https://pith.science/paper/IFHFP27U}},
note = {Machine review of arXiv:2505.24252}
}
read the original abstract
Frequent cyber-attacks have elevated WebShell exploitation and defense to a critical research focus within network security. However, there remains a significant shortage of publicly available, well-categorized malicious-code datasets organized by obfuscation method. Existing malicious-code generation methods, which primarily rely on prompt engineering, often suffer from limited diversity and high redundancy in the payloads they produce. To address these limitations, we propose \textbf{RAWG}, a \textbf{R}eward-driven \textbf{A}utomated \textbf{W}ebshell Malicious-code \textbf{G}enerator designed for red-teaming applications. Our approach begins by categorizing webshell samples from common datasets into seven distinct types of obfuscation. We then employ a large language model (LLM) to extract and normalize key tokens from each sample, creating a standardized, high-quality corpus. Using this curated dataset, we perform supervised fine-tuning (SFT) on an open-source large model to enable the generation of diverse, highly obfuscated webshell malicious payloads. To further enhance generation quality, we apply Proximal Policy Optimization (PPO), treating malicious-code samples as "chosen" data and benign code as "rejected" data during reinforcement learning. Extensive experiments demonstrate that RAWG significantly outperforms current state-of-the-art methods in both payload diversity and escape effectiveness.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[34]
Z. Wang, H. Wang, and L. Hao. Poster: Long php webshell files detection based on sliding window attention. arXiv preprint arXiv:2502.19257, 2025
work page Pith review arXiv 2025
-
[21]
M. Ma, L. Han, and C. Zhou. Large language models are few-shot generators: Proposing hybrid prompt algorithm to generate webshell escape samples. arXiv preprint arXiv:2402.07408, 2024
work page Pith review arXiv 2024
- [1]
-
[2]
S. Cao, R. Cheng, and Z. Wang. Agr: Age group fairness reward for bias mitigation in llms. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025
work page 2025
-
[3]
R. Cheng and S. Cao. Srmir: Shadow reward models based on introspective reasoning for llm alignment. arXiv preprint arXiv:2503.18991, 2025
arXiv 2025
-
[4]
R. Cheng, Y . Ding, S. Cao, S. Shao, and Z. Wang. Gibberish is all you need for membership inference detection in contrastive language-audio pretraining. arXiv preprint arXiv:2410.18371, 2024
work page Pith review arXiv 2024
- [5]
- [6]
Show all 42 references
-
[7]
Chiang, Z
W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality, March 2023. URL https://vicuna.lmsys.org
2023
-
[8]
Php-webshell-dataset
Cyc1e183. Php-webshell-dataset. https://github.com/Cyc1e183/PHP-Webshell-Dataset,
-
[9]
Dettmers, A
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023
2023 arXiv
-
[10]
W. Fan, Z. Yang, Y . Liu, L. Qin, and J. Liu. Honeyllm: A large language model-powered medium-interaction honeypot. In International Conference on Information and Communications Security, pages 253–272. Springer, 2024
2024
-
[11]
Z. Fei, S. Zhang, X. Shen, D. Zhu, X. Wang, J. Ge, and V . Ng. Internlm-law: An open-sourced chinese legal large language model. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2024), pages 9376–9392, Abu Dhabi, UAE, January 2025. Assoc...
2024 doi
-
[12]
F. Han, J. Zhang, C. Deng, J. Tang, and Y . Liu. Can llms handle webshell detection? overcoming detection challenges with behavioral function-aware framework. arXiv preprint arXiv:2504.13811, 2025
2025
-
[13]
H.-L. Hsu, W. Wang, M. Pajic, and P. Xu. Randomized exploration in cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2404.10728, 2024
2024 arXiv
-
[14]
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022
2022
-
[15]
Labrak, A
Y . Labrak, A. Bazoge, E. Morin, P.-A. Gourraud, M. Rouvier, and R. Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Findings of the Association for Computational Linguistic...
2024 doi
-
[16]
H. V . Le, T. N. Nguyen, H. N. Nguyen, and L. Le. An efficient hybrid webshell detection method for webserver of marine transportation systems. IEEE Transactions on Intelligent Transportation Systems, 24(2):2630–2642, 2021
2021
-
[17]
S. Li, R. Cheng, and X. Jia. Tuni: A textual unimodal detector for identity inference in clip models. In Proceedings of the Sixth Workshop on Privacy in Natural Language Processing, pages 1–13, 2025
2025
-
[18]
S. Liu. Towards building a scalable and believable hybrid honeypot framework. 2022
2022
-
[19]
Z. Liu, W. Wang, and P. Xu. Upper and lower bounds for distributionally robust off-dynamics reinforcement learning. arXiv preprint arXiv:2409.20521, 2024
2024 arXiv
-
[20]
R. Luo, L. Sun, Y . Xia, T. Qin, S. Zhang, H. Poon, and T.-Y . Liu. Biogpt: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics , 23(6): bbac409, 2022. doi: 10.1093/bib/bbac409
2022 doi
-
[22]
M. Ma, L. Han, and C. Zhou. Research and application of artificial intelligence based webshell detection model: A literature review. arXiv preprint arXiv:2405.00066, 2024. 9
2024 arXiv
-
[23]
Ouyang, J
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions wit...
-
[24]
Ouyang, J
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022
2022
-
[25]
B. Pang, G. Liang, J. Yang, Y . Chen, X. Wang, and W. He. Cwsogg: Catching web shell obfuscation based on genetic algorithm and generative adversarial network. The Computer Journal, 66(5):1295–1309, 2023
2023
-
[26]
A. Pei, Z. Yang, S. Zhu, R. Cheng, and J. Jia. Selfprompt: Autonomously evaluating llm robustness via domain-constrained knowledge guidelines and refined adversarial prompts.arXiv preprint arXiv:2412.00765, 2024
2024 arXiv
-
[27]
P. Peng, L. Yang, L. Song, and G. Wang. Opening the blackbox of virustotal: Analyzing online phishing scan engines. In Proceedings of the Internet Measurement Conference, pages 478–485, 2019
2019
-
[28]
Schulman, F
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017
2017 arXiv
-
[29]
Starov, J
O. Starov, J. Dahse, S. S. Ahmad, T. Holz, and N. Nikiforakis. No honor among thieves: A large- scale analysis of malicious web shells. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 1021–1032, Republic and Canton of Geneva, CHE, 2016. Int...
2016
-
[30]
Stiennon, L
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020
2020
-
[31]
Taori, I
R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca, 2023
2023
-
[32]
A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang. Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding. arXiv preprint arXiv:2305.12031, 2023
2023 arXiv
-
[33]
T. D. Tu, C. Guang, G. Xiaojun, and P. Wubin. Webshell detection techniques in web appli- cations. In Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT), pages 1–7. IEEE, 2014
2014
-
[35]
S. T. Z. Xuan and V . Selvarajah. Web shell attack and mitigation. In2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon), pages 1–5. IEEE, 2022
2022
-
[36]
Yang, X.-Y
H. Yang, X.-Y . Liu, and C. D. Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023
2023
-
[37]
W. Yang, B. Sun, and B. Cui. A webshell detection technology based on http traffic analysis. In Innovative Mobile and Internet Services in Ubiquitous Computing: Proceedings of the 12th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing ...
2018
-
[38]
B. Yong, W. Wei, K.-C. Li, J. Shen, Q. Zhou, M. Wozniak, D. Połap, and R. Damaševi ˇcius. Ensemble machine learning approaches for webshell detection in internet of things environments. Transactions on Emerging Telecommunications Technologies, 33(6):e4085, 2022. 10
2022
-
[39]
S. Yue, W. Chen, S. Wang, B. Li, C. Shen, S. Liu, Y . Zhou, Y . Xiao, S. Yun, W. Lin, X. Huang, and Z. Wei. Disc-lawllm: Fine-tuning large language models for intelligent legal services, 2023
2023
-
[40]
Y . Zhao, S. Lv, W. Long, Y . Fan, J. Yuan, H. Jiang, and F. Zhou. Malicious webshell family dataset for webshell multi-classification research. Visual Informatics, 8(1):47–55, 2024
2024
-
[41]
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 11 6 Appendix 6.1 Dataset Figures 5 and 6 provide JSON-formatted samples from the t...
1909 arXiv
-
[2021]
Accessed: 2025-05-02
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.