{"id":"e52b012a-0915-4e0a-8d9f-c76764fe4b73","arxiv_id":"2504.21045","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning CodeT5 to produce obfuscated XSS payloads and adding them to training data is reported to restore random-forest XSS detection accuracy from 81.9% to 99.5%, though the evaluation setup leaves the improvement unproven.","lead":"This paper fine-tunes a small code-generation model (CodeT5) to create obfuscated XSS attack strings, then adds them to training data for machine-learning XSS detectors. The authors report accuracy on obfuscated payloads rising from about 82% to 99.5%, but the evaluation may be circular because the same generator likely produced both training and test data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control training set from traditional obfuscators: the 99.5% accuracy gain may simply reflect adding obfuscated positives, not any LLM-specific contribution.","rationale":"I read the paper in good faith. The problem it addresses is real: Section 3.2 convincingly shows that a random forest trained only on non-obfuscated XSS data degrades sharply on obfuscated test payloads, with recall dropping to about 0.52. The proposed pipeline is also plausible as an engineering idea: fine-tune a code model on obfuscation transformations and use it to diversify training data. However, the evaluation does not isolate the LLM's contribution. The 99.5% result in Table 2 compares against a model trained with no obfuscated data at all; it is not compared against a model trained with the same quantity of tool-generated obfuscated data. Since the LLM's fine-tuning data was itself produced by those tools, the observed improvement could come entirely from the presence of obfuscated examples in training rather than from the LLM's generation ability. This is a correctness risk, not merely a disagreement with external consensus, because it affects whether the central claim follows from the presented experiments. The reader's verdict identified related concerns about test-set independence and the malicious-only obfuscation confound, which are real, but the missing non-LLM control is the most direct way to test the claimed mechanism. A single additional experiment would settle it, and as submitted the central claim is not supported.","tokens_in":8986,"tokens_out":4083,"duration_ms":42139,"concrete_test":"Re-run Section 3.3 with a control augmentation set of the same size as the LLM-generated set, built by applying the Section 2.3 tools directly to the same original training payloads without any LLM. Evaluate all models on the same obfuscated test set used in Section 3.3. If the control reaches roughly 99.5%, the LLM-specific contribution is unsupported; if it is materially lower, the central claim survives this particular objection. Also report the number of LLM-generated samples and confirm that the original payloads used for fine-tuning and generation were drawn only from the training split.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that LLM-generated obfuscated XSS payloads, when added to training, raise random-forest accuracy on obfuscated test data to 99.5% from 81.9%. The load-bearing gap is the absence of a non-LLM control: Section 3.3 trains on original data plus LLM-generated obfuscated XSS, but the paper never trains on an equal-sized set of obfuscated payloads produced directly by the Section 2.3 tools (JavaScript obfuscator, base64, URI encoding, and string splitting). Because Section 2.4 states the LLM was fine-tuned on obfuscation outputs generated by exactly those tools plus custom scripts, the LLM may only recombine tool-style transformations. A random forest over bag-of-words features could reach high accuracy by learning tool-specific token patterns, with no LLM-specific benefit. Related secondary gaps reinforce this: Section 3.3 does not state whether the test obfuscations are independent of the LLM's generation pool, and Section 3.4 admits that some generated samples were syntactically invalid yet were used as malicious training data. The missing control is the direct test of the central claim: the paper demonstrates that adding obfuscated training data helps, not that LLM-generated data is what makes the difference.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study on using LLM-generated obfuscated XSS payloads to harden ML-based XSS detectors. It first shows that a random forest trained on a 19,359-sample non-obfuscated dataset reaches 99.8% accuracy on a clean test set but drops to 81.9% on a test set in which malicious payloads have been obfuscated. The authors then fine-tune CodeT5-small on pairs of original and obfuscated payloads, use the fine-tuned model to generate additional obfuscated variants, augment the training set with them, and report a random forest accuracy of 99.5% on the obfuscated test set. They also report that LLM-generated samples are 28.1% more complex by Shannon entropy than samples from traditional obfuscation tools. The claimed contribution is that LLM-based data augmentation is a practical way to harden ML-based XSS detectors against obfuscated attacks.","tokens_in":9248,"tokens_out":6692,"duration_ms":60928,"significance":"If the claims were established, the paper would be a useful practical contribution to web security: the observation that standard classifiers degrade sharply under obfuscation is credible and worth documenting, and the proposed augmentation pipeline is simple enough to adopt. The paper is transparent about dataset composition, compares four classifiers, and reports precision, recall, and F1 in addition to accuracy. However, because the experimental design lacks a non-LLM control, confounds obfuscation with malice in the test set, and does not establish independence between the LLM-generated training samples and the obfuscated test set, the headline result is not currently supported. The significance of the work is therefore contingent on the authors closing these gaps with additional experiments.","major_comments":[{"comment":"The central claim that LLM-generated obfuscated data is responsible for the 99.5% accuracy is not supported because there is no control condition in which the same amount of obfuscated positives generated by the Section 2.3 tools is added to the training set. Since Section 2.4 says the LLM was fine-tuned on outputs of exactly those tools, the observed improvement over the 81.9% baseline could be due merely to adding obfuscated positives or to learning tool-specific token artifacts. A control with traditional obfuscators is required before the LLM-specific benefit can be assessed.","section":"Section 3.3"},{"comment":"The test set is constructed by obfuscating only malicious payloads and leaving benign payloads unmodified, so 'obfuscated' is perfectly correlated with 'malicious' in the test set. Under this design, a model that flags obfuscation artifacts as malicious will appear to improve even if it has not learned XSS semantics. The evaluation should include obfuscated benign JavaScript samples in both training and test sets, and should report whether the model can distinguish obfuscated benign from obfuscated malicious code.","section":"Sections 3.2 and 3.3"},{"comment":"The paper does not state how the obfuscated test set was generated or whether it is disjoint from the LLM-generated augmentation set. The same original payload pool and the same obfuscation techniques are used in Sections 2.4 and 3.2, so without an explicit independence statement and overlap check the 99.5% result may reflect memorization of generator artifacts or train/test leakage rather than generalization to unseen obfuscated attacks.","section":"Section 3.3"},{"comment":"The paper admits that higher temperature settings sometimes produced syntactically invalid code, but that these samples 'contributed to training robust Random Forest models.' This is an explicit admission that at least some training labels were not validated as functional malicious payloads; if those samples are not real XSS, the reported accuracy does not measure XSS detection. The authors must either exclude invalid samples and rerun the experiments or provide a concrete justification, backed by data, for why invalid samples are legitimate malicious training data.","section":"Section 3.4"},{"comment":"The 28.1% complexity increase is reported without sample sizes, variance, or a statistical test, and no evidence is given that higher Shannon entropy correlates with evasion difficulty. This claim is a headline result in the abstract and should be either substantiated with statistical support or softened.","section":"Section 3.4"}],"minor_comments":[{"comment":"Describing XSS as 'one of the variations of command injection' is technically imprecise; XSS is a form of injection, but not typically classified as command injection.","section":"Section 1"},{"comment":"The description of deduplication and filtering is brief; the paper should state how duplicates were removed and why some benign sources were filtered, to support reproducibility.","section":"Section 2.1"},{"comment":"The semantic-equivalence testing is described only qualitatively ('many maintained parity'); the paper should report the number of payloads tested and the fraction that preserved behavior, especially given the invalid outputs mentioned in Section 3.4.","section":"Section 3.5"},{"comment":"The row for Logistic Regression wraps in a way that could confuse readers; typeset it on one line.","section":"Table 2"},{"comment":"The phrase 'varying probabilities' for selecting obfuscation techniques is not quantified; the actual probability distribution should be reported for reproducibility.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant workshop topic and the underlying degradation phenomenon is worth reporting, but the current evaluation does not support the causal claim. I would encourage the editors to request a revision that adds a non-LLM control, fixes the test-set confound, and demonstrates train/test disjointness; without those experiments, the contribution is a useful negative result about classifier fragility rather than a validated augmentation method. The reference formatting and placeholder ACM DOI also suggest the manuscript is not yet in camera-ready shape."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper has a genuinely interesting setup and a well-demonstrated problem, but it skips the one experiment that would actually prove the LLM is doing the work. The claim is that fine-tuning CodeT5-small to generate obfuscated XSS payloads and adding them to training lifts random-forest accuracy on obfuscated test data from 81.9% to 99.5%. That number is not established.\n\nWhat the paper does well: the initial observation is credible and clearly shown. Models trained on plain XSS payloads are very accurate on plain test data (RF at 0.998) and then collapse when tested on obfuscated samples (RF recall 0.523). That is a real, reproducible-sounding vulnerability, and it motivates the work. Fine-tuning a code LLM to generate obfuscated variants for data augmentation is a reasonable idea, and the paper is honest about some of its limitations, including that high temperature sometimes produces syntactically invalid code.\n\nNow the soft spots, in rough order of importance. The central gap is the missing control: Section 3.3 trains on original data plus LLM-generated obfuscated payloads, but never trains on original data plus an equal-sized set of obfuscations produced directly by the Section 2.3 tools. Since the LLM was fine-tuned on those same tools' outputs, the LLM may just be recombining tool-style transformations. A bag-of-words random forest could easily hit high accuracy by learning tool-specific token patterns, with no LLM-specific contribution. The 99.5% versus 81.9% comparison therefore does not isolate the LLM as the cause.\n\nSecond, the paper does not state whether the obfuscated test set used in 3.2 is disjoint from the LLM-generated training samples. It is produced by the same tool families that generated the LLM's fine-tuning data, so overlap or style similarity is a real risk. Third, only malicious payloads are obfuscated; benign payloads stay plain, which gives the model a shortcut unrelated to XSS semantics. Fourth, the paper admits some generated samples are syntactically invalid yet uses them as malicious training data, which can inflate accuracy by teaching the model to flag invalid syntax. Minor points: no error bars, no class-balance numbers, and the 28.1% entropy comparison is reported without enough detail about the baseline set.\n\nThese are all fixable. The core problem is real and the method is plausible, but the evidence as submitted does not support the headline. The paper would benefit from a serious referee who asks for the tool-augmented control and a clear train/test separation statement. I would read it as a workshop-level work-in-progress, not as a settled result.","headline":"Plausible idea and a real problem, but the missing control means the headline result is not demonstrated.","tokens_in":9792,"tokens_out":3340,"would_cite":false,"duration_ms":33323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning CodeT5-small to generate obfuscated XSS payloads raises random-forest accuracy on obfuscated tests from 81.9% to 99.5%.","keywords":["cross-site scripting","obfuscation","large language model","fine-tuning","random forest","data augmentation","Shannon entropy","XSS detection"],"falsifier":"Compare the obfuscated test payloads against the LLM-generated training payloads by exact string match and by stylistic fingerprints (e.g., same base64 blocks, same splitting patterns, same random variable names); if a large fraction of test samples or near-duplicates appear in the training set, the reported accuracy overstates generalization, and a fully held-out test obfuscated by a different tool or by hand would settle the claim.","tokens_in":8756,"feed_emoji":"🛡️","tokens_out":6993,"duration_ms":54986,"temperature":0.7,"pith_summary":"This paper tries to establish that a small, code-specialized language model can be fine-tuned to produce obfuscated XSS payloads that, when added to training data, make machine-learning detectors far harder to fool. The authors report that a random forest trained on ordinary XSS payloads scores 99.8% accuracy on clean tests but only 81.9% on obfuscated ones; after adding LLM-generated obfuscated variants to the training set, accuracy on obfuscated tests rises to 99.5%. They also report that the LLM's output is 28.1% more complex, measured by entropy, than what existing obfuscation tools produce. If correct, the finding offers security teams a practical recipe for creating diverse, adversarial training data without waiting for real-world obfuscated attacks.","feed_headline":"LLM-written obfuscated payloads restore XSS detection to 99.5%","feed_subtitle":"Random forest accuracy on obfuscated XSS jumps from 81.9% to 99.5% when training adds LLM-generated payloads.","key_machinery":"The load-bearing mechanism is the fine-tuned CodeT5-small model acting as an obfuscation engine. CodeT5-small is a 60-million-parameter transformer pretrained on code; the paper fine-tunes it on original-and-obfuscated XSS payload pairs so it learns to rewrite a payload's surface form while keeping its functionality, then samples from it at high temperature to get diverse, hard examples. The second piece is the bag-of-words representation with a random-forest classifier, which turns those examples into a measurable jump in recall on obfuscated inputs.","core_discovery":"The central claim is that LLM-based data augmentation can close most of the accuracy gap that obfuscation opens in ML-based XSS detection. Concretely, the paper fine-tunes CodeT5-small, a 60-million-parameter code model, on pairs of original XSS payloads and their obfuscated versions produced by standard tools (Base64, URI encoding, JavaScript obfuscation, string splitting). The fine-tuned model is then prompted at high temperature to generate new obfuscated variants of base payloads. These generated samples are mixed into the training set of a bag-of-words random forest, and the model is tested on a separately obfuscated test set. The paper reports random-forest accuracy rising from 81.9% (training without obfuscated data) to 99.5%, with an F1 score of 0.993, and the LLM-generated samples having 28.1% higher average entropy than tool-generated ones.","pith_inferences":["The claimed gain depends on the obfuscated test set being independent from the LLM-generated training data; if they share a generator, the 99.5% figure may reflect pattern memorization rather than robust detection, a question the paper leaves open.","The method likely transfers to other attack classes such as SQL injection or command injection, since the same fine-tuning-over-obfuscation loop applies to any payload language.","Entropy measures character unpredictability, not semantic difficulty; a stronger test would be evasion rate against deployed web application firewalls or against the original detector under different feature representations.","A controlled comparison using held-out obfuscations from a tool the LLM never saw during fine-tuning would isolate the LLM's contribution from simple data duplication."],"forward_implications":["Random-forest XSS detection on obfuscated payloads rises from 81.9% to 99.5% accuracy when LLM-generated obfuscated samples are folded into training.","The same augmentation lifts other classifiers too: SVM reaches 96.7%, logistic regression 96.3%, and decision tree 95.0% on obfuscated tests.","LLM-generated payloads are 28.1% more complex by entropy than tool-generated ones, so the training set becomes harder and more diverse.","Because the LLM is only 60M parameters, the approach runs on a single GPU and modest RAM, making it accessible for practical security teams."],"supporting_citations":[{"why":"Defines the entropy metric the paper uses to claim LLM obfuscations are 28.1% more complex than tool outputs.","marker":"[15]"},{"why":"Establishes the baseline that public obfuscation tools produce limited-complexity samples, the gap the LLM is meant to fill.","marker":"[16]"},{"why":"Motivates ML-based XSS detection and the obfuscated-XSS weakness that the paper targets.","marker":"[2]"},{"why":"Documents how obfuscation techniques help malicious JavaScript evade detection, the threat the training augmentation addresses.","marker":"[22]"}],"fun_headline_variants":["LLM-generated obfuscated XSS data lifts detection to 99.5%","Training on LLM-made obfuscated payloads restores XSS accuracy","Random forest XSS accuracy rises from 81.9% to 99.5% via LLM data","LLM-generated payloads: 28% more complex, restoring XSS detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result hinges on the obfuscated test set being genuinely separate from the LLM-generated training samples; the paper does not say how the test obfuscations were made, and if they come from the same generator or the same original payload pool, the 99.5% accuracy could be memorization of generator artifacts rather than detection of malicious XSS semantics.","fun_headline_variants_meta":{"raw":{"variants":["LLM-generated obfuscated XSS data lifts detection to 99.5%","Training on LLM-made obfuscated payloads restores XSS accuracy","Random forest XSS accuracy rises from 81.9% to 99.5% via LLM data","LLM-generated payloads: 28% more complex, restoring XSS detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1809,"prompt_tokens":1113,"completion_tokens":696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":601}},"tokens_in":729,"tokens_out":696,"duration_ms":6232,"temperature":1.0,"reasoning_tokens":601,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:40:12.836088+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the obfuscated test payloads against the LLM-generated training payloads by exact string match and by stylistic fingerprints (e.g., same base64 blocks, same splitting patterns, same random variable names); if a large fraction of test samples or near-duplicates appear in the training set, the reported accuracy overstates generalization, and a fully held-out test obfuscated by a different tool or by hand would settle the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the baseline that public obfuscation tools produce limited-complexity samples, the gap the LLM is meant to fill."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates ML-based XSS detection and the obfuscated-XSS weakness that the paper targets."}],"review_version":1}