{"id":"8a6b0679-15b3-40eb-ae6c-d0c3b5533c2f","arxiv_id":"2506.05376","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A position paper from Scale AI argues that red teaming research should prioritize product-level safety specifications, realistic attacker models, and system-level monitoring over abstract model-level harm benchmarks.","lead":"This paper argues that AI red teaming should target product safety specifications and realistic attack scenarios instead of abstract social biases, and should move from model-level robustness to system-level safety. It gives concrete threat models for chatbots, audio assistants, video generators, and agents, and proposes monitoring users and trajectories as key mitigations.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The system-level priority rests on an unmeasured claim that most real-world harms are diffuse and multi-turn; the paper's own catastrophic-single-output examples show why that assumption is load-bearing.","rationale":"The reader's weakest assumption focuses on product-spec definability; my concern is adjacent but distinct. I agree with the conditional verdict, but not primarily because specifications are ambiguous. The paper's second priority claim is an empirical ranking of interventions, and I could not find data in the paper supporting the distributional assumption that most severe real-world harms are diffuse and multi-turn. Since this is a position paper, the absence of such data is not a logical contradiction, and the roadmap remains useful as a proposal. However, before adopting it as established guidance, the field should test whether system-level post-hoc interventions actually cover the severe tail of harm. The paper itself highlights catastrophic single-response failures, which is exactly the case where its priority ranking is least secure. A concrete incident audit would settle the concern. Therefore I do not change the reader's conditional verdict.","tokens_in":16946,"tokens_out":5323,"duration_ms":56992,"concrete_test":"Build a counterfactual severity audit using existing misuse records (AI Incident Database plus industry red-team reports): for each incident, label (i) whether the primary harm occurred within one model response or required multi-turn cooperation, and (ii) whether an ideal post-hoc monitor with 1-hour response could have prevented the primary harm. Then compute the share of severe incidents (e.g., top decile by estimated harm) that are single-response and unpreventable by post-hoc monitoring. If that share is substantial, Section 5's 'vast majority of harms' premise fails and the priority ordering would need to be inverted or conditioned on deployment context.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The second half of the central claim—'system-level safety over model-level robustness'—contains the load-bearing empirical step. Section 5 motivates trajectory and user monitoring by stating that 'many realistic harmful uses of an AI system may require its cooperation over multiple turns,' while conceding that single-response safety 'is only critical in cases where even a single harmful response... can cause outsized or catastrophic harm.' The roadmap never supplies evidence about the relative frequency or severity of diffuse multi-turn misuse versus single-output catastrophic misuse. The introduction's motivating examples ('creating dangerous biological agents and other catastrophic risks') are single-output harms. For such harms, asynchronous monitoring, user banning, and rapid patching act after the fact: they cannot undo the first harmful output. So the move from 'system-level measures are necessary for some harms' to 'system-level safety should be prioritized over model-level robustness' requires a quantitative claim about harm distributions and intervention lead times that the paper does not provide. The phrase 'as long as sufficiently effective asynchronous monitoring detects these failures as they happen' is a conditional, not a demonstrated premise, and the condition is hardest exactly for the highest-severity single-output failures.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that LLM red teaming research currently misallocates effort. It recommends three priorities: (1) red teaming should target product safety specifications rather than abstract social biases or ethical principles; (2) threat models should reflect realistic attackers and deployment contexts rather than idealized single-query settings; and (3) red teaming should expand from model-level refusal robustness to system-level safety, including trajectory monitoring, user monitoring, rapid patching, and red teaming of monitors. The paper motivates these claims through a taxonomy of model/product/system, worked threat-model sketches for chatbots, audio assistants, video generators, and agents, and a set of best practices in Section 4. Section 5 develops the system-level argument, and Section 6 considers alternative views. The paper is explicitly a roadmap rather than an empirical study; its central claims are normative and strategic.","tokens_in":17147,"tokens_out":5111,"duration_ms":50648,"significance":"If the roadmap is accepted, it would redirect a substantial portion of red teaming research from universal jailbreak benchmarks toward deployment-aware evaluation and system-level defenses. The paper's contribution is mainly organizational and rhetorical: it clarifies distinctions among model, product, and system; identifies understudied attack surfaces such as multi-turn conversations, audio, video, agents, and tool outputs; and offers a practical best-practices list in Section 4. It is honest about its status as a position paper, explicitly engaging with an alternative view in Section 6. The main weakness is that the two prioritizations—product over abstract, and system over model—are supported by selected examples rather than by systematic evidence, leaving the strongest comparative claims underdetermined. As a position paper, it need not contain new experiments, but its empirical premises should be supported with a structured literature synthesis or measurement study.","major_comments":[{"comment":"The central comparative claim that system-level safety should be prioritized over model-level robustness rests on the empirical premise that 'many realistic harmful uses of an AI system may require its cooperation over multiple turns' and that asynchronous monitoring can catch failures before harm scales. The manuscript provides examples and citations (e.g., [30, 47, 56]) but no evidence about the relative frequency or severity of diffuse multi-turn harms versus single-output catastrophic harms. The introduction's motivating examples—dangerous biological agents and other catastrophic risks—are precisely the kind of single-response harms for which post-hoc user banning and patching cannot undo the first harmful output. The sentence 'as long as sufficiently effective asynchronous monitoring detects these failures as they happen' is a conditional, not a demonstrated premise, and it is hardest for the highest-severity cases. To support the prioritization, the paper should either supply or cite data on harm distributions and intervention lead times in deployed products, or soften the claim from 'should be prioritized over' to 'should complement.'","section":"Section 5, 'Trajectory and User Monitoring' and 'Rapid Response'"},{"comment":"The first pillar assumes that a product safety specification can be defined precisely enough to serve as a red teaming objective and that such specifications are tied to actual societal harm. The paper acknowledges that products can and should have divergent safety considerations and that many products lack well-defined specifications; its fallback is that researchers should 'define or infer a plausible policy context.' This fallback reintroduces the same normative ambiguity that the paper criticizes in abstract harm categories, because a plausible policy may be vague or contested. The manuscript does not present a worked example in which a product specification is translated into a concrete red teaming metric (what counts as a successful breach, what the evaluation metric is) for a specific product. Without at least one such case study, the claim that product-level red teaming is more tractable and more decision-relevant than model-level harm testing remains an assertion. A worked example would make the proposal easier to adopt and would make its central premise falsifiable.","section":"Section 2, 'Safety Specifications Should Focus on Product-Level Risk'"}],"minor_comments":[{"comment":"The sentence 'There would be no difference in the models vulnerabilities' contains a typo; it should read 'the model's vulnerabilities.'","section":"Section 4, 'Assessing the Delta'"},{"comment":"The claim that 'more difficult single-turn attacks seem to be independent, ad-hoc issues with safeguards that can be quickly patched' is supported by a single self-cited reference [63]; a systematic comparison or an independent replication would make the argument more robust.","section":"Section 3.1, first bullet list"},{"comment":"The alternative view is presented fairly but dismissed in a few sentences; readers who disagree with the product-specification premise would benefit from a more detailed rebuttal, particularly concerning the role of regulation and societal consensus.","section":"Section 6, 'Argument for Static Harm Research'"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a coherent and well-written position paper from an industry group. It relies frequently on self-cited work from the same organization (e.g., refs. 39, 41, 47, 63, 90); this is not disqualifying, but the editor may wish to encourage the authors to add independent evidence and to state potential conflicts of interest more explicitly. The two major comments above concern load-bearing empirical premises; if the authors address them with data or by appropriately weakening the comparative claims, the paper would be a valuable contribution to the red teaming literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful synthesis, not a new result. If you work on jailbreaks or safety evals, reading it will reorganize how you think about what to test. The model/product/system distinction is clean, and the shift from abstract harm categories to product safety specifications is argued with concrete examples (chatbots, audio, video, agents). The good-practices section is practical enough to hand to a grad student.\n\nThe strongest part is Section 5. Trajectory monitoring, user monitoring, and rapid response are real, underused levers, and the paper makes the case for red-teaming the monitor itself (sabotage evaluations) without overclaiming. It also honestly lays out the alternative view (static harm research) in Section 6, which is more than most position papers do.\n\nNow the soft spots. The paper's central move—'system-level over model-level'—rests on an empirical claim that most real harm is diffuse and multi-turn. The paper never measures that. The intro's motivating examples (bioweapons instructions) are exactly the single-output catastrophic kind where async monitoring and banning arrive after the fact. Section 5 says 'many realistic harmful uses require cooperation over multiple turns,' but 'many' is doing uncredited work. To prioritize system-level over model-level robustness, you need some distribution over harm severities and intervention lead times, and this paper doesn't have it. The stress-test note about this is right, though the paper itself flags the caveat with 'as long as sufficiently effective asynchronous monitoring detects these failures as they happen'—that conditional is doing load-bearing work.\n\nAlso, Section 3.1's claim that single-turn attacks are 'ad-hoc and quickly patched' is cited to a Scale AI paper (ref 63) and presented as fact. It may be true, but as written it's a company-internal empirical claim without independent replication. The paper should soften that or show data.\n\nMinor: self-citation is heavy but not disqualifying; the cited work is relevant and much of it is public. The lack of systematic evidence that product-spec red teaming reduces real-world harm more than abstract harm testing is a limitation, but for a roadmap paper it's acceptable if acknowledged—they partly do.\n\nWho's this for: anyone setting red-teaming research priorities, LLM product teams, and safety-eval benchmark builders. It deserves a serious referee. Recommend accept with revisions: add explicit boundary conditions or empirical support for the harm-distribution assumption, and temper the quickly-patched claim.","headline":"A clear, well-organized position paper that reprioritizes red-teaming research; its central empirical assumption about where real-world harms live is undefended, but the roadmap is still worth engaging.","tokens_in":17660,"tokens_out":2068,"would_cite":true,"duration_ms":20319,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM red teaming should target product safety specs and system-level defenses, not abstract harms.","keywords":["LLM red teaming","product safety specifications","realistic threat models","system-level safety","jailbreaking","safeguard evaluation","AI deployment monitoring","multi-turn attacks"],"falsifier":"A head-to-head deployment study would settle it: take the same underlying model in the same product, run one arm with only refusal training and one arm whose red teaming targets the product safety specification with system-level monitoring and user banning, and compare measured harm per active user, for example substantiated abuse reports per 100,000 interactions. If the model-only arm shows equal or lower harm, the paper's priority claim is false.","tokens_in":16760,"feed_emoji":"🛡️","tokens_out":6751,"duration_ms":59041,"temperature":0.7,"pith_summary":"This is a position paper about how AI red teaming research should be spent. The authors argue that the field's current emphasis on jailbreaking models in isolation to test whether they refuse abstractly harmful requests is misdirected. They propose two priorities: red teaming should test violations of a specific product's safety specification under realistic threat models, and it should move from model-level robustness to system-level safety, where monitoring, user banning, and rapid patching catch harm that individual refusals miss. If they are right, red teaming research would shift from benchmark jailbreaks toward deployment-aware evaluation and defense, which they argue is the only way to keep up with chatbots, audio assistants, video generators, and autonomous agents.","feed_headline":"Red teaming should target product specs, not abstract harms","feed_subtitle":"Real-world AI risk drops when red teams test deployed products and monitor users, not just refusal rates.","key_machinery":"The load-bearing mechanism is the three-level distinction among model, product, and system, together with the product safety specification as the red teaming objective. A product safety specification states what behavior is prohibited for that specific deployment, including who the users are, what tools are attached, and what regulations apply, and the paper treats it as the target that a successful red team finding must violate. The second mechanism is the system-level mitigation loop: trajectory monitoring classifies harm over whole interaction histories, user monitoring detects and bans repeat malicious users, and rapid response patches newly found failures before they are exploited at scale. Red teaming the monitor, through sabotage experiments where an adversary must complete a harmful task without detection, stress-tests that loop. These mechanisms convert the vague question of whether a model is safe into testable questions about a specific product and its deployed defenses.","core_discovery":"The paper's central claim is that LLM red teaming will reduce real-world harm only if it is re-anchored around two priorities: product safety specifications over abstract social biases or ethical principles, and system-level safety over model-level robustness. It defines a model as the neural network, a product as the deployed application built on it, and a system as the product plus its deployment infrastructure, including monitors, staff, users, and environment, and argues that the attack surface users actually face is the end-to-end product stack, not the bare model. Because safety is contextual, a behavior that violates one product's policy may be acceptable in another, so red teaming objectives should be explicit, actionable, and measured against the target product's stated spec. The paper then argues that realistic threat models differ by product type, with multi-turn conversations for chatbots, prosodic and multilingual channels for audio assistants, frame-spanning harm for video generators, and tool- and environment-driven attacks for agents, and that safeguards should be tested both independently and in realistic sandboxes. Its final step is the claim that system-level measures such as trajectory monitoring, user monitoring, rapid response to newly discovered jailbreaks, and red teaming the monitor itself through sabotage experiments are necessary to make red teaming relevant to deployed risk.","pith_inferences":["Editorial inference: If product-specific specifications become the primary evaluation target, standardized public benchmarks may fragment, because each product's policy is different; a shared harm taxonomy would survive only where regulators impose one.","Editorial inference: The emphasis on system-level monitoring implies that model-level alignment research could be deprioritized, yet the system-level loop still depends on the model not producing a single catastrophic output before a monitor can intervene, so the two are complements rather than substitutes.","Editorial inference: Red teaming the monitor sets up an arms race between detection evasion and detection improvement; sabotage experiments would need to be repeated continuously as monitors update, making red teaming an ongoing operational process rather than a one-time evaluation.","Editorial inference: A testable consequence is that products with user banning and rapid patching should show lower measured abuse per user than products with only refusal-trained models, holding the underlying model fixed."],"forward_implications":["Red teaming benchmarks would be evaluated against a stated product safety specification and deployment context, so a jailbreak that violates no deployed product's policy would count as low priority.","Multi-turn and trajectory-level attacks would receive more research attention than single-turn jailbreaks, because defenses trained for single turns do not generalize to extended conversations.","Safety investments would shift toward asynchronous monitoring, malicious-user detection, and rapid safeguard deployment, since most harm is diffuse over many requests rather than a single catastrophic output.","White-box red teaming would assume the adversary can fine-tune the model, and black-box red teaming would assume limited query access, making attacks match real attacker resources.","Agent red teaming would require investment in realistic sandboxes and simulated environments, because tool outputs and environmental context are part of the attack surface."],"supporting_citations":[{"why":"Supplies the survey evidence that most safety evaluations are model-centric and ignore deployment context.","marker":"[71]"},{"why":"Documents that frontier models and industry policies already differ in content safety specifications, motivating product-specific red teaming objectives.","marker":"[100, 102]"},{"why":"Shows that multi-turn jailbreaks succeed against models that are robust to single-turn attacks, motivating trajectory-level red teaming.","marker":"[30, 47]"},{"why":"Demonstrates that newly found jailbreaks can be patched with a few examples, supporting rapid response as a system-level mitigation.","marker":"[63]"},{"why":"Defines sabotage evaluations, the template the paper uses for red teaming the monitor.","marker":"[8]"},{"why":"Supplies the classic adversarial-example threat model from vision that the paper contrasts with LLM product threat models.","marker":"[24]"},{"why":"Shows that refusal-trained LLMs can be jailbroken as browser agents, evidence that tool integration expands the attack surface.","marker":"[41]"},{"why":"Exemplifies the shift from keyword matching to rubric-based evaluation of policy violations.","marker":"[81]"},{"why":"Provides the chain-of-thought monitoring approach that red teaming the monitor should stress-test.","marker":"[5]"}],"fun_headline_variants":["Red teaming must prioritize product specs and system safety","Stop red teaming models, start red teaming systems","Focus red teaming on product specs, not abstract ethics","System-level red teaming is key to real-world AI safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The roadmap assumes that a product safety specification can be written precisely enough to serve as a red teaming objective, and that red teaming against that spec plus system-level mitigation reduces real-world harm more effectively than model-level testing.","fun_headline_variants_meta":{"raw":{"variants":["Red teaming must prioritize product specs and system safety","Stop red teaming models, start red teaming systems","Focus red teaming on product specs, not abstract ethics","System-level red teaming is key to real-world AI safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1631,"prompt_tokens":982,"completion_tokens":649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":582}},"tokens_in":598,"tokens_out":649,"duration_ms":6263,"temperature":1.0,"reasoning_tokens":582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:07:31.521070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A head-to-head deployment study would settle it: take the same underlying model in the same product, run one arm with only refusal training and one arm whose red teaming targets the product safety specification with system-level monitoring and user banning, and compare measured harm per active user, for example substantiated abuse reports per 100,000 interactions. If the model-only arm shows equal or lower harm, the paper's priority claim is false.","supporting_citations":[],"review_version":1}