{"id":"7e64e064-7ac8-49f7-a19c-dbbc1738c66f","arxiv_id":"2412.11109","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SpearBot uses jailbreak prompts and iterative LLM-critic feedback to generate spear-phishing emails that evade machine detectors and appear human-like.","lead":"This paper introduces SpearBot, a system that uses GPT-4 to write personalized phishing emails and iteratively rewrites them until other AI judges no longer flag them as phishing. It tests these emails against machine defenders and human readers, reporting that they often escape detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The defender evaluation lacks a style-matched legitimate control and the planned template baseline (Section 5.6), so low detection rates may reflect distribution shift rather than SpearBot's deceptive quality; the paper's own 'overfitting' remark supports this.","rationale":"The reader's conditional verdict is appropriate. SpearBot is a plausible framework: the jailbreak prompt and critique loop are clearly described, the ablation in Section 5.8 shows that adding critics lowers ICL detection from 70.3% to 45%, and manual checks (Section 5.4) confirm that the emails function as phishing. Those observations give the framework internal support. The gap is external validity and causal attribution: the defender tables conflate 'hard to detect' with 'deceptive.' Since neither a style-matched legitimate control nor the template baseline promised in Section 5.6 is present, the headline evasion numbers could be reproduced by any modern personalized LLM email, regardless of the critique loop. This does not warrant rejection—the data do show low detection rates—but it requires re-scoping the claim from 'SpearBot produces deceptive emails' to 'SpearBot emails are out-of-distribution for legacy-trained defenders and for GPT-4-based detectors.' A single control experiment would settle the magnitude of the confound.","tokens_in":17398,"tokens_out":6765,"duration_ms":59698,"concrete_test":"Add a control experiment to Section 5: generate (a) style-matched legitimate emails by prompting GPT-4 benignly with the same personal-information profiles, and (b) a no-critique/template-based phishing baseline as Section 5.6 originally planned; run the same ML/PLM/LLM defenders from Tables 5-6 on all sets. If the false-positive rate on (a) is comparable to the recall on SpearBot emails (e.g., both near 1-3% for PLMs), or if the no-critique baseline is detected at roughly the same low rate as SpearBot, the evasion numbers are explained by domain shift or base-LLM style rather than by the critique-based deception, and the central claim should be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's load-bearing inference is that low defender accuracy on SpearBot emails (Tables 5-6) measures the emails' deceptive quality. This inference is insecure because the defenders are trained and evaluated on legacy corpora (CEAS_08, Enron, Ling, Nazario, Nigerian, Assian) while SpearBot emails are GPT-4-generated, personalized, modern-style messages. The SpearBot evaluation set contains no style-matched legitimate emails, so a defender that labels all SpearBot emails 'legitimate' (PLM recall 1-3%, Section 5.5) is indistinguishable evidence of deception from evidence of domain shift. The paper itself attributes the PLM collapse to 'a high degree of overfitting to the trained datasets.' The planned control experiment, 'SpearBot vs Template-based method' (Section 5.6), is empty, so there is no baseline from another generation method to show that the critique refinement—rather than the base LLM or style shift—causes the evasion. The ablation (Section 5.8) compares no-critic vs. SpearBot under one GPT-4 ICL defender, but not against template-based phishing or same-style legitimate mail. Without these controls, 'often evade detection to a large extent, underscoring their deceptive quality' is an overclaim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SpearBot, a framework that uses a jailbroken large language model (GPT-4) to generate personalized spear-phishing emails, then iteratively refines them using feedback from multiple LLM critics (GPT-4, Claude-3-Sonnet, ChatGPT) until the critics no longer classify them as phishing. The authors generate 1,000 emails for 100 virtual individuals (50 students, 50 employees) using 10 phishing strategies, and evaluate the emails against machine-learning (SVM, XGBoost, Random Forest), pre-trained language model (BERT, RoBERTa, GPT-2), and LLM-based (ICL, CoT, ChatSpamDetector) defenders. They report very low detection rates on SpearBot emails, high readability and perceived safety in a human study, and modest generation costs.","tokens_in":17560,"tokens_out":5200,"duration_ms":42628,"significance":"If the evaluation were fully controlled, the paper would provide a concrete demonstration that LLMs can cheaply produce personalized spear-phishing emails that evade current detectors and that humans find convincing. The proposed framework, together with the generated dataset and the systematic comparison across three defender families, would be a useful resource for future research on phishing defense. However, the current evaluation design has several load-bearing gaps that prevent the results from supporting the paper's central claims about 'deceptive quality' and 'state-of-the-art bypass rates.'","major_comments":[{"comment":"The defender evaluation lacks a style-matched legitimate control. All defenders are trained on legacy datasets (CEAS_08, Enron, Ling, Nazario, Nigerian, Assian; Section 5.2) and evaluated on GPT-4-generated SpearBot emails. Because SpearBot emails differ from the training distribution in style, personalization, and time period, the near-zero recall values for the PLM defenders could reflect domain shift rather than the emails' deceptive quality. The paper itself attributes the PLM performance drop to 'a high degree of overfitting to the trained datasets' (Section 5.5), which is inconsistent with the conclusion that the emails 'effectively bypass' the defenders. Without a control set of legitimate emails generated by the same LLM with the same personalization and strategies, the experiments cannot distinguish 'evades detection because it is deceptive' from 'evades detection because it is out-of-distribution.' This directly undermines the abstract's claim that the low detection rates 'underscor[e] their deceptive quality.'","section":"Section 5.5, Tables 5–6"},{"comment":"The section titled 'Spearbot vs Template-based method' is empty. The paper claims 'state-of-the-art bypass rates compared to previous phishing attacks' (Contribution 3), but no comparison to any other generation method is reported. The ablation in Section 5.8 compares only 'no critic' versus 'SpearBot' under a single GPT-4 ICL defender; it does not compare against template-based phishing, human-authored phishing, or other LLM-based generators. Consequently, the results cannot show that the critique-based iterative refinement—rather than the base LLM's style of writing or the distribution shift from legacy corpora—is what causes the observed evasion. This is a load-bearing missing baseline for the paper's central claim.","section":"Section 5.6"},{"comment":"The optimization critics and the evaluation defenders are from the same model family, which creates a self-reference problem. The critics used for iterative refinement are GPT-4, Claude-3-Sonnet, and ChatGPT (gpt-3.5-turbo) (Section 4.4), while the LLM defenders used in the final evaluation are all GPT-4 (gpt-4-1106-preview) with ICL, CoT, or ChatSpamDetector prompts (Section 5.3). The emails are optimized until the GPT-4-based critics no longer flag them, then evaluated against GPT-4-based defenders. This is not a mathematical tautology, but the reported evasion rates for LLM defenders partly measure the framework's ability to exploit the specific inductive biases of the GPT-4 model family. To support a general claim about deceptive quality, the evaluation should include defenders from different model families or otherwise prevent the optimizer from directly tuning to the evaluation defenders.","section":"Sections 4.4, 5.3, 5.5"},{"comment":"The human evaluation is based on a small, non-representative sample: 20 participants, all with a cybersecurity background and master's or PhD education, and all with 10 years of English study. The questionnaire measures self-reported readability and perceived danger, but it does not measure actual susceptibility, such as whether participants would click the embedded link or comply with the request. The claim that the emails are 'significantly readable and deceptive' and 'particularly dangerous' (Section 6.2, Findings 5) goes beyond what this study can support, both because of the sample characteristics and because the outcome measure is a perception judgment rather than a behavioral one.","section":"Sections 6.1–6.2"}],"minor_comments":[{"comment":"In the SpearBot rows of Table 5, 'Acc' is identical to 'Rec' while 'Prec' is 100% because all SpearBot emails are phishing. The metric labels should be clarified: what is reported as 'accuracy' is actually the true positive rate (recall), not accuracy in the usual sense.","section":"Section 5.5, Table 5"},{"comment":"The text says 'accuracy of 27%' and 'detection accuracy of 17%' when describing the fraction of emails detected under each strategy. Since the evaluation set for SpearBot contains only phishing emails, these are true positive rates, not accuracy metrics; please use consistent terminology.","section":"Section 5.7"},{"comment":"The sentence is garbled: 'Defenders based on LLMs demonstrate superior performance in SpearBot compared to those based on machine learning, pre-trained language models and large language models.' This likely should read '...compared to those based on machine learning and pre-trained language models,' since LLM defenders are the subject being discussed.","section":"Section 5.5, Findings 2"},{"comment":"The availability statement says 'we will sooner publicly released all source code and generated email datasets.' This should be rephrased to 'we will soon publicly release' and, if true, should make clear that the artifacts are currently not available.","section":"Section 8"},{"comment":"The cost analysis appears to use only GPT-4 pricing, but the critics include Claude-3-Sonnet and ChatGPT. Please clarify whether the costs of all critic models are included in the reported per-email figures.","section":"Section 5.10"}],"recommendation":"major_revision","confidential_remarks":"The empty Section 5.6 suggests that a planned comparison experiment was omitted, possibly due to an incomplete submission. Combined with the availability statement in Section 8 ('will sooner publicly released'), the manuscript currently does not provide the artifacts promised for reproducibility. The claim of 'state-of-the-art bypass rates' (Contribution 3) is not supported by any baseline comparison, and the lack of a style-matched legitimate control is a fundamental evaluation gap. These issues are fixable in principle, so I do not recommend rejection, but the revision will need to add the missing controls and baselines, or substantially temper the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core idea is real — an iterative critique loop that makes GPT-4 rewrite phishing emails until a panel of LLM critics no longer flags them — but the evaluation as written does not establish the headline claim. Worth a serious referee, but it needs a re-scoped abstract and a real control condition.\n\nWhat's new: previous LLM-phishing work (Roy et al., Bethany et al., Heiding et al.) generated emails in one shot or from templates. SpearBot adds the critique-refinement loop, using multiple LLM critics to rewrite until none flags the email. That is a genuine methodological addition, and the ablation suggests it does improve evasion: detection accuracy drops from 70% to 45% under an ICL defender. The cost analysis (~$0.13–0.15 per email) is a useful sanity check for the threat, and the case studies are well presented. The dataset of 1,000 emails and code are promised but not actually released, so I don't credit that yet.\n\nWhere it's soft: the big one is the defender evaluation. The defenders are trained on legacy corpora (CEAS_08, Enron, Ling, etc.) and then tested on GPT-4-generated, personalized, modern-style emails. With no style-matched legitimate emails in the test set, the low recall (1–3% for the PLMs) is exactly what you'd expect from domain shift — the paper's own 'overfitting' remark basically says this. So the inference from 'evades these defenders' to 'deceptive quality' does not hold. The planned template baseline (Section 5.6) is empty, which would have isolated the effect of the critique loop from the base LLM. That is a load-bearing missing experiment.\n\nSecond, the self-reference issue: emails are optimized against GPT-4 critics and then tested against GPT-4 defenders. It's not a tautology, but it limits external validity — you've tuned to one model family and then shown that family misses them. Adding non-GPT defenders or a different-family critic would help.\n\nThird, the human study uses 20 cybersecurity graduate students, reports averages without error bars or significance tests, and compares against Nigerian phishing emails that are decades old. Suggestive, not conclusive.\n\nThe framework is legitimate, the threat is worth documenting, and several components are well executed. But the headline claims — 'state-of-the-art bypass rates' and 'deceptive quality' — are not supported by the current evidence. This deserves peer review, with the expectation of substantial revision: fill the missing section, add controls, re-scope claims, and actually release the artifacts.","headline":"The critic-refinement loop is a real addition to LLM phishing generation, but the defender evaluation lacks the controls needed to support the paper's headline claims about deceptive quality.","tokens_in":18178,"tokens_out":2677,"would_cite":false,"duration_ms":23002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpearBot pairs a jailbroken LLM generator with LLM critics that iteratively refine its output, producing personalized spear-phishing emails that largely evade machine defenders and read as safe to human judges.","keywords":["spear-phishing","large language models","jailbreak prompts","critique-based optimization","phishing email detection","adversarial email generation","GPT-4","social engineering"],"falsifier":"Retrain each of the six defenders on a balanced mixture of SpearBot-generated emails labeled phishing and contemporary legitimate emails labeled legitimate, then measure accuracy on a held-out set of new SpearBot emails. If accuracy returns to the 90%-plus range seen on the legacy datasets, the original evasion figures were largely distribution shift rather than intrinsic deception; if accuracy stays near chance, the claim that SpearBot emails are genuinely hard to detect survives.","tokens_in":17082,"feed_emoji":"🎣","tokens_out":9900,"duration_ms":74912,"temperature":0.7,"pith_summary":"SpearBot is an adversarial framework that gets large language models to write personalized spear-phishing emails and then polishes those emails until automated detectors no longer flag them as phishing. The paper claims that SpearBot emails largely evade three classes of machine defenders — traditional machine learning, fine-tuned pre-trained language models, and LLM-based detectors — with detection accuracy falling to as low as 1% for a fine-tuned GPT-2 model and 45% for the best LLM-based detector, and that human participants rate the emails as readable and as safe-looking as legitimate messages. The stakes are concrete: if true, fully automated, mass-personalized phishing becomes available at roughly fifteen cents per email, and training material for anti-phishing education can be generated on demand. The authors offer SpearBot as both a demonstration of this threat and a generator of fresh content for phishing-awareness simulation.","feed_headline":"Jailbroken LLMs craft phishing emails that slip past detectors","feed_subtitle":"A jailbreak-plus-critic loop refines emails until machine defenders and human readers call them safe.","key_machinery":"The load-bearing object is the generative-critique optimization loop. The jailbreak is a prompt-level social-engineering sequence: GPT-4 is first cast as a research assistant to a cybersecurity master's student, is asked to summarize research motivations and psychological phishing strategies, and only then is asked to produce a personalized email with a realistic-looking link; this framing slips past the model's safety filters. The critique stage then feeds each generated email to K=3 LLM critics — GPT-4, Claude-3-Sonnet, and GPT-3.5-turbo — each asked to answer whether the email is phishing and to give reasons in a structured XML response. Every affirmative verdict with its reasons is appended to a regeneration prompt that instructs the generator to adjust the message, conceal its intent more subtly, and keep the link attractive; the loop iterates until all critics answer no or the iteration limit T=10 is reached. The critics therefore act as a natural-language loss signal that the generator optimizes against, with no gradients and no feature engineering required.","core_discovery":"The paper's central claim is that a jailbreak-plus-critique loop removes both obstacles to LLM-based spear-phishing: safety alignment and detector evasion. A prompt-level jailbreak frames the task as legitimate cybersecurity research, persuading GPT-4 to produce initial phishing emails it would otherwise refuse to write; the jailbreak succeeds directly on 66% of first queries and 91% within five attempts. Three further LLMs — GPT-4, Claude-3-Sonnet, and GPT-3.5-turbo — then act as critics: whenever any critic labels an email phishing, its written reasons are fed back to the generator, which rewrites the email to hide its intent more subtly, and the loop repeats until all critics pass or ten iterations are exhausted. From 100 virtual targets and ten phishing strategies the authors generate 1,000 emails, and they report that defender accuracy drops to 3% or below for fine-tuned BERT, RoBERTa, and GPT-2, to 21.7% for the best machine-learning defender, and to 45% for the best LLM defender, while manual reviewers identify 89–95% of sampled emails as functional phishing. Combined with human questionnaire scores placing SpearBot emails' perceived safety on par with legitimate mail (2.2 versus 1.3 for older phishing corpora on a 0–4 scale), the paper concludes that LLMs can produce highly personalized, readable, and deceptive phishing content at scale.","pith_inferences":["The reported evasion rates most likely conflate deception with distribution shift: defenders trained on 2000–2008 email corpora may wave through SpearBot emails chiefly because the emails' short, personalized, modern style is far outside their training distribution; the paper's own 'overfitting' remark (Section 5.5) supports reading the numbers this way. Retraining defenders on a mix of SpearBot e","The generate-critique-refine loop is a transferable pattern: the same jailbreak-plus-critic construction should apply to other malicious content such as SMS phishing, fake product reviews, or fraudulent web pages, where the critic is a policy checker rather than an email classifier.","The human questionnaire measured perceived safety and readability among 20 cybersecurity graduate students, not click-through behavior; susceptibility among general-population targets, who are the realistic victims, could differ substantially from this expert sample."],"forward_implications":["Security-awareness programs can generate fresh, personalized test phishing emails on demand at roughly $0.13–$0.15 each, removing the cost barrier that has limited simulated-phishing exercises.","Defenders trained only on legacy corpora (CEAS_08, Enron, Ling, Nazario, Nigerian, Assian) are not a reliable yardstick for LLM-generated attacks; periodic re-training on newly generated phishing content would be needed.","Among the tested defenses, LLM-based in-context learning performs best against SpearBot (45% detection) but still misclassifies more than half of the emails as legitimate, indicating content-only detection is the weakest defense pillar.","The end condition of the loop — all three LLM critics must fail to flag the email — gives the framework a built-in quality gate that static template-based generation lacks."],"supporting_citations":[{"why":"Supplies the six public phishing datasets and the precision/recall/F1/accuracy metrics used to train and evaluate every machine-based defender.","marker":"[13]"},{"why":"Provides ChatSpamDetector, the bespoke LLM detection prompt that serves as the strongest LLM defender baseline SpearBot must evade.","marker":"[29]"},{"why":"Empirical basis for the paper's use of highly customized personal information as the lever that raises spear-phishing success rates.","marker":"[51]"},{"why":"Motivates the critique mechanism by showing that LLM critics can catch flaws in LLM outputs.","marker":"[35]"},{"why":"Earlier large-scale LLM spear-phishing study that SpearBot extends; its emails were constrained by fixed events and templates rather than flexible strategies.","marker":"[10]"},{"why":"Prior human-experiment work showing LLMs can assist phishing creation; it lacked the detector-response analysis that SpearBot contributes.","marker":"[24]"},{"why":"Provides the prompt-level versus token-level jailbreak taxonomy that justifies choosing a prompt-based jailbreak over token optimization.","marker":"[14]"},{"why":"Supplies the TF-IDF feature extraction approach on which the machine-learning defenders are built.","marker":"[9]"}],"fun_headline_variants":["Jailbroken LLM loop crafts phishing emails that evade detection","Critique-driven jailbreak yields phishing emails that fool all defenders","LLM critics refine jailbroken phishing until humans call it safe","Automated jailbreak and critique loop create phishing that evades most detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the low detection rates of defenders trained on legacy email datasets measure the emails' genuine deceptive quality; if those low rates mostly reflect a style mismatch between decades-old training emails and the new personalized email format — the paper itself attributes the collapse to 'a high degree of overfitting to the trained datasets' (Section 5.5) — then the claim that SpearBot emails are intrinsically hard to detect is not established.","fun_headline_variants_meta":{"raw":{"variants":["Jailbroken LLM loop crafts phishing emails that evade detection","Critique-driven jailbreak yields phishing emails that fool all defenders","LLM critics refine jailbroken phishing until humans call it safe","Automated jailbreak and critique loop create phishing that evades most detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001114,"raw_usage":{"total_tokens":4685,"prompt_tokens":1040,"completion_tokens":3645,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":3572}},"tokens_in":656,"tokens_out":3645,"duration_ms":24564,"temperature":1.0,"reasoning_tokens":3572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:17:11.064667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain each of the six defenders on a balanced mixture of SpearBot-generated emails labeled phishing and contemporary legitimate emails labeled legitimate, then measure accuracy on a held-out set of new SpearBot emails. If accuracy returns to the 90%-plus range seen on the legacy datasets, the original evasion figures were largely distribution shift rather than intrinsic deception; if accuracy stays near chance, the claim that SpearBot emails are genuinely hard to detect survives.","supporting_citations":[{"cited_title":"I., R ABBI , F., AND ZIBRAN , M","cited_arxiv_id":null,"evidence_quote":"Supplies the six public phishing datasets and the precision/recall/F1/accuracy metrics used to train and evaluate every machine-based defender."},{"cited_title":"Personalized persuasion: Quan- tifying susceptibility to information exploitation in spear-phishing at- tacks","cited_arxiv_id":null,"evidence_quote":"Empirical basis for the paper's use of highly customized personal information as the lever that raises spear-phishing success rates."},{"cited_title":"Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF feature extraction approach on which the machine-learning defenders are built."}],"review_version":1}