{"id":"9e07d5a7-f6b6-419d-b54c-8fdcbe3523ca","arxiv_id":"2504.19855","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Automated LLM red-teaming achieves higher success rates than manual prompting (69.5% vs 47.6%) but manual solves are faster when they succeed, according to 214,271 attack attempts on the Crucible platform.","lead":"A large dataset from an AI red-teaming capture-the-flag platform shows that automated attack attempts succeed more often than manual ones (69.5% vs 47.6% of user/challenge pairs), although only a small fraction of users automate. The paper argues security testing should pair human creativity with automated systematic exploration, and that defenses must expect high-volume programmatic attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The automation advantage may be an artifact of attempt volume and challenge selection, since the automated label is partly defined by high request counts and no analysis conditions on either.","rationale":"I read the paper as an empirical claim that automated red-teaming approaches outperform manual ones in Crucible, based on a large, real dataset. The analysis pipeline is described in detail, and the qualitative direction (automation enables systematic exploration) is plausible. However, the central comparison is not secure as stated. The automated/manual label is partly defined by the very behaviors that inflate success probability: request volume and regularized timing. Since automated sessions average 472.5 attempts versus 8.0 for manual sessions, the 69.5% versus 47.6% gap could largely reflect that users who keep trying eventually solve, not that automation has an independent advantage. The paper's own Section 6.3 acknowledges selection bias on challenge difficulty and quick-solver behavior, but does not quantify or adjust for it. No statistical tests, confidence intervals, or code/data release are provided, so the reader cannot assess whether the gap is robust. The reader's weakest assumption identified the same core issue, and the proposed re-analysis would settle it. Because the concern is addressable and the data may well support a conditional claim after controls, the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":13832,"tokens_out":4781,"duration_ms":47605,"concrete_test":"Using the full Crucible logs, recompute automated versus manual success rates within strata defined by total attempt count per user/challenge pair (e.g., 1–10, 11–50, 51–200, 201–1000, >1000) nested within each challenge, and fit a logistic regression P(solve) ~ automated + log(1 + attempts) + challenge fixed effects. If the automated coefficient becomes small or non-significant, or the stratified gap falls below roughly 5 percentage points, the headline automation advantage is not supported independently of volume and difficulty. If the gap persists across all attempt strata and within challenges, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison in Section 6.1 (69.5% vs 47.6%) is computed at the user/challenge level after labeling sessions as automated partly by request volume (>1,000 requests, >40 queries per minute in Section 5.2) and then counting any pair that used automation as automated. The paper also reports automated sessions average 472.5 attempts versus 8.0 for manual sessions (Section 6.4). Because a user/challenge pair with hundreds of attempts has a mechanically higher chance of an eventual solve than a pair with a handful of attempts, the comparison may be measuring persistence and attempt volume rather than automation itself. The aggregate comparison also pools challenges with very different difficulty and automation rates. Section 6.3 explicitly admits that harder challenges attract automation and that users who can solve quickly never automate, both of which would inflate the observed gap. Table 1 shows several challenges where manual success equals or exceeds automated success (whatistheflag3, squeeze2, librarian), so the aggregate advantage is not uniform. No regression or stratification controls for attempts, challenge difficulty, or user ability, and no confidence intervals are given. The central claim therefore rests on an unquantified assumption that the automation label, rather than the volume and challenge selection with which it is entangled, drives the 21.8 percentage-point gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes a 400-day dataset from Dreadnode's Crucible platform (214,271 attack attempts by 1,674 users across 30 LLM challenges) to compare automated versus manual AI red-teaming. The authors report that automated approaches achieve substantially higher user/challenge solve rates than manual approaches (69.5% vs 47.6%), that purely automated approaches do even better (76.9%), that manual approaches are faster when successful (5.2x median solve-time advantage), and that the optimal strategy is hybrid: human creativity for strategy with programmatic execution for exploration. Sessions are classified as automated or manual through a three-stage pipeline (heuristics, a supervised classifier, and LLM judges), and the analysis is presented at session, user/challenge, and challenge levels.","tokens_in":14050,"tokens_out":4094,"duration_ms":42724,"significance":"If the headline comparison is valid, this is a valuable empirical contribution: it is among the largest public observational datasets of real LLM red-teaming behavior, and the paper is transparent about its environment, classification pipeline, and limitations. The concrete attack examples (Section 7) and the challenge-by-challenge tables are useful reference material for both offensive and defensive practitioners. However, the central causal claim that automation itself drives the success-rate advantage is not established: the automation label is partly defined by request volume and rate, automated sessions average 472.5 attempts versus 8.0 for manual, and Section 6.3 explicitly acknowledges that harder challenges attract automation while users who can solve quickly never automate. Because no analysis conditions on attempt volume, challenge difficulty, or user persistence, and no confidence intervals or significance tests are reported, the 21.8 percentage-point gap may reflect persistence and challenge selection rather than a property of automation. The significance of the paper is therefore conditional on a substantial reanalysis of the data.","major_comments":[{"comment":"The headline comparison (69.5% vs 47.6% success) is computed at the user/challenge level after labeling sessions as automated partly by request volume (>1,000 requests) and request rate (>40 queries per minute) in Section 5.2, and Section 6.4 reports that automated sessions average 472.5 attempts versus 8.0 for manual sessions. A user/challenge pair with hundreds of attempts has a mechanically higher probability of eventually solving the challenge than a pair with a handful of attempts, regardless of whether the attempts are automated. The paper therefore conflates automation with persistence and attempt volume. To support the claim, the authors should control for the number of attempts (for example, by stratifying user/challenge pairs into attempt-count bands, or by fitting a logistic regression with attempt count as a covariate, ideally with user-level clustering). Without such an analysis, the 21.8 percentage-point gap is not attributable to automation per se.","section":"Section 6.1 vs 6.4 and 5.2"},{"comment":"Section 6.3 explicitly acknowledges a selection bias: harder challenges attract more automation, and users who can solve a challenge manually in a short time never invest in automation. Despite this, no stratification by challenge difficulty or by user ability is presented. Table 1 shows that the automation advantage is not uniform: for whatistheflag3, squeeze2, and librarian, manual success is equal to or greater than automated success, and the overall pattern is driven by a minority of high-volume challenges. The aggregate 1.46x advantage could therefore be entirely explained by the correlation between automation use and difficult, high-attempt challenges. The authors should report challenge-stratified comparisons (including per-challenge automated/user counts) and, where sample sizes permit, a multilevel model that includes random intercepts for challenges and users.","section":"Section 6.3 and Table 1"},{"comment":"The paper reports the headline success rates, the 5.2x time-to-solve ratio, and the per-challenge comparisons without any confidence intervals, standard errors, or significance tests. Given the highly skewed user participation (the top user alone accounts for 37,698 of the 214,271 attempts) and the small numbers of automated solves on several challenges (e.g., fragile, miner, extractor2 in Table 1), the aggregate estimates may be sensitive to a handful of users or sessions. The authors should provide cluster-robust confidence intervals (clustering by user and by challenge) or bootstrap intervals, and should indicate the effective sample sizes underlying each claim. This is not a presentation detail; it determines whether the 'significantly higher' claim is supported.","section":"Section 6.1 (no uncertainty quantification)"},{"comment":"In Section 5.2, any user/challenge pair that included an automated session is counted in the 'automated' category for aggregate success-rate analysis, and Section 6.1 reports that purely automated approaches achieve 76.9% while hybrids achieve 63.1%. Aggregating these two groups obscures the mechanism behind the advantage: a pair that used automation for only a small fraction of its sessions may be classified as automated, and the success may come from the manual portion. The classification protocol also has an internal inconsistency: it is described as a 'majority-rule approach' but then defined as labeling a session automated 'if any judge identified automation patterns.' The authors should report the sensitivity of the main result to (i) treating only purely automated pairs as automated, (ii) using different agreement thresholds among the judge LLMs, and (iii) varying the heuristic thresholds (e.g., request count and rate cutoffs). Without such a sensitivity analysis, the reader cannot know how much of the reported advantage is an artifact of the classification choices.","section":"Section 5.2 and 6.1 (definition of automated at user/challenge level)"}],"minor_comments":[{"comment":"There is a citation mismatch: the text in Section 2.1 attributes a three-modality categorization of LLM attacks (jailbreaking, leaking, injection) to 'Zou et al. [5]', but reference [5] is Banerjee et al., 'SoK: A Systems Perspective on Compound AI Threats and Countermeasures'; the cited categorization appears to belong to a different source. Please correct the reference or the attribution.","section":"References"},{"comment":"Table 1 reports solve times by challenge, but it does not give the number of automated and manual solves per cell. Several AutoTime entries (e.g., extractor2: 1817.9 hours, whatistheflag4: 0.6 hours, probe2: 0.1 hours) are based on very small samples, and the text acknowledges this for some challenges but not for all outlier cells. Including per-cell counts or annotating cells with n below a threshold would improve interpretability.","section":"Table 1"},{"comment":"The sentence 'This 1.46x advantage was consistent across most challenge types' is too strong given Table 1, where manual success is higher or equal on whatistheflag3, squeeze2, and librarian. Please either quantify the proportion of challenges with an automation advantage or soften the claim to reflect the heterogeneity.","section":"Section 6.1"},{"comment":"The sentence describing the 'majority-rule approach' is self-contradictory: it first says the classifications were combined through a majority-rule approach, then states a session was labeled automated if any judge identified automation patterns. Please clarify whether the final label is a union (any judge) or an actual majority vote, and report how many sessions fell into each agreement pattern.","section":"Section 5.2"},{"comment":"Figures 4, 5, and 6 are referenced in the text but not included in the submitted manuscript extract. Please ensure the final version includes the figures with axis labels, units, and a clear explanation of bubble sizes and color scales in Figure 5; the current text description is not sufficient for quantitative interpretation.","section":"Figures 4-6"}],"recommendation":"major_revision","confidential_remarks":"This is a well-motivated empirical study using a unique dataset, and the authors are appropriately transparent about many limitations. My concern is strictly about the inferential gap between the descriptive statistics and the causal/effectiveness claim. The core issue—that automation is defined in part by high volume and that no analysis conditions on volume or difficulty—is fixable: the dataset almost certainly supports attempt-stratified and challenge-stratified analyses, and the authors should be asked to provide them. I do not see grounds for rejection, but the paper should not be accepted in its current form. One additional point for the editor: the dataset originates from the authors' own platform (Dreadnode/Crucible); this does not invalidate the work, but it strengthens the case for explicit robustness checks and, where possible, independent replication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe usable nugget here is the dataset: 214,271 real attack attempts, 1,674 users, 30 challenges in one consistent environment. Nobody else has quantified automated versus manual LLM red-teaming at this scale. The observation that automated approaches solve more user/challenge pairs (69.5% vs 47.6%) is directionally plausible and worth knowing. But the headline number is not established, because the automated label is partially defined by request volume and timing regularity, and the paper's own Section 6.3 admits that harder challenges attract automation while fast manual solvers never bother to automate. The measured gap mixes automation with persistence and challenge difficulty.\n\nWhat the paper does well: the environment and logging architecture are described concretely; the challenge category and model-deployment tables are useful; the attack examples are believable; and the time-to-solve analysis openly flags tiny samples (fragile, miner, extractor2) and acknowledges selection bias. They are not hiding the mess.\n\nThe soft spots are real and central. Automated sessions average 472.5 attempts versus 8.0 for manual, and the classification heuristic labels any session over 1,000 requests as automated. So the success-rate comparison is partly a comparison of persistence. There are no controls for challenge difficulty, user ability, or attempt count, and no confidence intervals or significance tests. Table 1 shows several challenges where manual success equals or beats automated success (whatistheflag3, squeeze2, librarian, squeeze3), so the 21.8-point aggregate gap is not uniform. Counting any hybrid pair as automated is defensible for the question \"does touching automation help?\" but needs a robustness check. There is also a mechanical citation error in Section 2.1: the text says Zou et al. [5] categorized attacks, but reference [5] is Banerjee et al.; similarly [3] is labeled Liu et al. but points to Chao et al. These are fixable, but they do not inspire confidence.\n\nWho should read it: AI security practitioners and red-team operators will get value from the real-world baseline, especially the challenge-by-challenge numbers and the time-to-solve comparisons. The paper deserves a serious referee, but the referee should push on the confounding. The authors should release session-level labels and analysis code, report challenge-controlled estimates and uncertainty, and test sensitivity to their classification thresholds. As written, I would treat the main claim as conditional; with those additions, it could be a solid empirical contribution.\n\nMy recommendation: send it to peer review, but require the revision.","headline":"A genuinely useful first dataset on real LLM red-teaming at scale, but the headline automation advantage is confounded by volume, challenge difficulty, and selection effects that the paper itself acknowledges.","tokens_in":14617,"tokens_out":1697,"would_cite":true,"duration_ms":19786,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated red-teaming attacks beat manual ones 69.5% to 47.6% in a 214,271-attempt LLM CTF dataset.","keywords":["LLM security","prompt injection","jailbreaking","red teaming","automated attacks","Capture The Flag","empirical attack analysis","session classification"],"falsifier":"Take the same 30 challenges and run a fresh, instrumented study in which every session records whether the user is automating by direct self-report or a client-side toggle, then recompute the user-challenge solve rates. If the automated advantage falls below, say, 5 percentage points once challenge difficulty is controlled for, or reverses on challenges with high manual solve rates, the paper's central claim would be refuted. The easiest check on existing data: recompute the 69.5% versus 47.6% comparison only for challenges whose manual solve rate is above 50% and see whether the gap survives.","tokens_in":13586,"feed_emoji":"🤖","tokens_out":7962,"duration_ms":77638,"temperature":0.7,"pith_summary":"This paper analyzes 214,271 attack attempts by 1,674 users on 30 LLM Capture-The-Flag red-teaming challenges and argues that automation is the strongest predictor of success: teams that used any automated probing solved 69.5% of the challenges they attempted, compared with 47.6% for manual-only attempts. The paper treats this as evidence that algorithmic testing is transforming LLM red-teaming, since automated sessions systematically explore the solution space (averaging 472.5 attempts versus 8.0 per manual session) and succeed more often, even though only about 5% of users automated. It also reports a sharp tradeoff: successful manual solves are roughly 5.2 times faster, suggesting human intuition still matters for creative reasoning challenges. These findings would matter for defenders because they point to high-volume automated probing, not just clever prompts, as the main attack surface to defend.","feed_headline":"Automated red-team attacks win 69.5% vs 47.6%","feed_subtitle":"A 30-challenge study of 214,271 attempts finds automation solves more; manual solves stay 5x faster.","key_machinery":"The object that carries the argument is the session classifier, a three-stage pipeline that labels each user-challenge session as automated or manual. Heuristic rules flag sessions with more than 1,000 requests, more than 40 queries in a 60-second window, or very regular timing as automated; a supervised classifier extends those labels using request volume, IP diversity, and timing regularity; LLM judges then review candidate sessions to catch subtler automation. The unit of analysis is the user-challenge pair, where any session containing automation counts as automated, and the counts are 868 automated sessions out of 19,823 total. This pipeline is what turns raw logs into the 69.5% versus 47.6% comparison, so the argument stands or falls with it.","core_discovery":"On the paper's own terms, the central discovery is a reliable empirical gap: at the level of a user attempting a challenge, any involvement of automation raises the solve rate from 47.6% to 69.5%, a 21.8 percentage-point advantage that appears across most of the 30 challenges. Purely automated approaches reached 76.9% and hybrid human-plus-automation approaches 63.1%. The paper explains the gap through thoroughness—automated sessions make hundreds of methodical variations and can persist through hundreds of failures—while manual success is rarer but faster, with a median solve time of 1.5 minutes versus 11.6 minutes for automated solves. The paper is careful to note that this comparison is partly affected by selection: harder challenges attract automation, and users who can solve quickly never bother automating, which may amplify the observed advantage.","pith_inferences":["The 5.2x manual speed advantage suggests a testable prediction: automated agents that can decide to stop or pivot after a few failed creative attempts may match or exceed manual intuition on creative-reasoning challenges—the paper does not test this.","The classifier's reliance on timing regularity and request volume implies that an attacker who deliberately paces automation to imitate human rhythm would be classified as manual; an evasion benchmark could be built from exactly these features.","Because the dataset comes from an open CTF platform, the population skews toward adversarial hobbyists; generalizing the 69.5% versus 47.6% gap to enterprise or insider threats would require applying the same session-classification method to logs from deployed assistant systems.","The paper's challenge-type taxonomy points to a concrete design rule: integration-based challenges, where an LLM touches external tools, may be the next automation frontier, since automated solutions there were up to 2x faster than manual ones."],"forward_implications":["Defenders should design rate limits and complexity-based throttling specifically to raise the cost of high-volume automated probing, not just to block known prompt patterns.","Red-team evaluations should report solve rate and time-to-solve separately, because automation's higher solve rate coexists with a 5.2x slower median solve time.","Challenge design can deliberately include elements that disrupt automation, since the data show some creative-reasoning challenges resist automated approaches.","Because only about 5% of users automated despite the higher success rate, the untapped attack surface is large; defenders should not assume attacker effort stays manual.","Hybrid workflows—human picks the attack direction, automation exhausts the variants—are a measurable best practice for both pure and mixed strategies."],"supporting_citations":[{"why":"Showed language models can generate red-team attacks, the intellectual ancestor of automated approaches.","marker":"[11]"},{"why":"The tool described here established programmatic prompt attack generation, the practice whose effectiveness this paper measures.","marker":"[12]"},{"why":"Demonstrated automated generation of jailbreak prompts, evidence that automation can produce successful attacks.","marker":"[13]"},{"why":"Characterized in-the-wild jailbreak prompts, the qualitative baseline this dataset extends with large-scale success rates.","marker":"[15]"},{"why":"Offers a grounded theory of LLM red teaming in the wild, which this paper's quantitative patterns build on.","marker":"[16]"},{"why":"A global prompt-hacking competition whose techniques supply the attack taxonomy used here.","marker":"[17]"}],"fun_headline_variants":["Automation outperforms manual in AI red-teaming: 69.5% vs 47.6%","Automated attacks win more but are slower in AI red-teaming","Manual red-team solves are 5x faster, but automation wins more often","Only 5.2% use automation, yet it wins 69.5% of red-team challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the automated/manual labels are accurate enough, and that the 21.8-point gap is not mostly a selection effect in which harder challenges attract automation while quick manual solvers never automate.","fun_headline_variants_meta":{"raw":{"variants":["Automation outperforms manual in AI red-teaming: 69.5% vs 47.6%","Automated attacks win more but are slower in AI red-teaming","Manual red-team solves are 5x faster, but automation wins more often","Only 5.2% use automation, yet it wins 69.5% of red-team challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001549,"raw_usage":{"total_tokens":6157,"prompt_tokens":874,"completion_tokens":5283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":5189}},"tokens_in":490,"tokens_out":5283,"duration_ms":36688,"temperature":1.0,"reasoning_tokens":5189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:41:19.720418+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 30 challenges and run a fresh, instrumented study in which every session records whether the user is automating by direct self-report or a client-side toggle, then recompute the user-challenge solve rates. If the automated advantage falls below, say, 5 percentage points once challenge difficulty is controlled for, or reverses on challenges with high manual solve rates, the paper's central claim would be refuted. The easiest check on existing data: recompute the 69.5% versus 47.6% comparison only for challenges whose manual solve rate is above 50% and see whether the gap survives.","supporting_citations":[{"cited_title":"PyRIT: The python risk identification tool for llm red teaming","cited_arxiv_id":null,"evidence_quote":"The tool described here established programmatic prompt attack generation, the practice whose effectiveness this paper measures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrated automated generation of jailbreak prompts, evidence that automation can produce successful attacks."},{"cited_title":"Ignore this title and hackaprompt: Exposing systemic vul- nerabilities of llms through a global scale prompt hacking competition","cited_arxiv_id":null,"evidence_quote":"A global prompt-hacking competition whose techniques supply the attack taxonomy used here."}],"review_version":1}