{"id":"0d5dd0bb-e61f-4441-8c74-1885c9f186d5","arxiv_id":"2501.17749","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"External testers generated 10,080 unsafe prompts against OpenAI's o3-mini beta, manually confirmed 87 unsafe behaviors, and found most protection came from an API-level policy filter rather than the model itself.","lead":"Researchers used their own tool ASTRAL to fire 10,080 unsafe prompts at OpenAI's pre-release o3-mini model, then manually confirmed 87 cases where the model gave harmful or unsafe answers. The report suggests o3-mini is safer than earlier OpenAI models, but much of that safety came from an external filter that blocked risky prompts before the model saw them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline safety comparison is confounded by the API-level policy firewall: 4,491 of 10,080 test inputs never reached o3-mini, so 'safer than predecessors' is not established for the model itself.","rationale":"The paper is a transparent and useful pre-deployment safety report: the manual verification of 87 unsafe outputs is genuine empirical work, and the replication package and ASTRAL repository strengthen its value. The central comparative claim, however, rests on comparing unsafe counts across models tested under different conditions. The policy firewall blocked 4,491 of 10,080 inputs, roughly 45% of the test suite, and these were counted as safe without ever reaching the model. A simple sensitivity calculation shows why this matters: among the 5,589 unblocked inputs, the confirmed unsafe rate is 87/5,589 ≈ 1.6%; applying that same rate to the blocked inputs would add about 70 more unsafe behaviors, more than doubling the confirmed total and materially changing the comparison to older models. Because the blocked prompts were flagged as policy violations, they are likely to be at least as unsafe as the unblocked ones if the firewall were removed, so the model-level ranking could easily reverse. The authors explicitly concede in Section 3.2 that the assessment is at the system level and in Finding 3 that they are unsure whether the firewall will be deployed in the public release; they even recommend re-executing the policy-violation cases if it is not. That concession is accurate, but the abstract and Finding 1 nevertheless present the safety advantage as a property of o3-mini itself. The secondary limitation that only evaluator-flagged outputs were manually checked reinforces the point that 87 is a lower bound, not a precise model-level estimate. Since the reader's CONDITIONAL verdict already captures this concern and the paper's own caveats support it, no verdict change is needed, but the requested condition should be explicit: the comparative safety claim should be re-evaluated on an endpoint without the external filter, or the claim should be restricted to the full API system rather than the model in isolation.","tokens_in":8926,"tokens_out":3736,"duration_ms":38972,"concrete_test":"Re-run the 4,491 policy-violation test inputs (TS1: 2,034; TS2: 2,457) against the production o3-mini API after release, with the same ASTRAL evaluator and manual verification protocol, recording how many are blocked and how many produce unsafe outputs. If the public endpoint no longer has the pre-LLM firewall and the unsafe rate among these prompts is comparable to the historical rates for GPT-3.5/GPT-4/GPT-4o, the 'safer than predecessors' claim at the model level is not supported. If OpenAI can provide a firewall-disabled sandbox, use that instead to remove the release-date confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 reports that the beta API raised policy-violation exceptions for a large portion of test inputs, and the evaluator was adapted to count those as safe. The totals are 2,034 policy violations in TS1 (707+762+565) and 2,457 in TS2, i.e., 4,491 of 10,080 inputs never demonstrably reached o3-mini. Finding 1 compares the 32 confirmed unsafe TS1 outputs (or 49 evaluator-flagged) to 752/166/215 unsafe behaviors found in GPT-3.5/GPT-4/GPT-4o in previous work, where no such firewall was present. If the firewall is external and disabled in the public release, the blocked prompts could produce unsafe responses, and the model-level ordering could reverse. The authors acknowledge this in Section 3.2 and Finding 3, but the abstract and Finding 1 still attribute safety to o3-mini itself. The 87 confirmed unsafe outputs are a lower bound conditional on the filter, not a model-level safety estimate. A secondary issue is that only evaluator-flagged outputs were manually checked (Section 3.3), so false negatives remain unmeasured; however, the firewall confound is the decisive problem for the comparative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an external pre-deployment safety evaluation of a beta version of OpenAI's o3-mini API. The authors use their ASTRAL tool to generate 10,080 unsafe prompts across 14 safety categories, 6 writing styles, and 5 persuasion techniques (TS1: 3,780; TS2: 6,300), execute them against the beta API, classify responses with a GPT-3.5-based evaluator, and manually verify every evaluator-flagged 'unsafe' or 'unknown' output. They report 87 confirmed unsafe behaviors, with the largest numbers in controversial topics/politics (c3) and terrorism/organized crime (c13). They note that a large share of prompts triggered API policy-violation exceptions and were counted as safe, and they frame the result as system-level rather than model-level in Section 3.2. The paper concludes that o3-mini 'seems to be safer' than older OpenAI models and other ASTRAL-tested LLMs.","tokens_in":9181,"tokens_out":5199,"duration_ms":50891,"significance":"The paper's main concrete contribution—87 manually confirmed unsafe outputs from a pre-deployment system—is useful as an early-access safety report, and the authors provide a replication package and are transparent about the system-level nature of the test. They also distinguish evaluator-flagged from manually confirmed unsafe counts, which is good methodological practice. However, the headline comparative claims (Findings 1 and 2) rely on counts from the authors' previous study [10] that were collected under different API conditions, were not manually verified, and were not subject to the same policy-violation filter. The novelty is an application of an existing tool to a non-public model, so the lasting value depends on whether the safety evidence is correctly attributed to the model rather than to the surrounding API. As presented, the evidence supports a conditional system-level statement, not the stronger model-level comparisons in the abstract and Findings 1 and 2.","major_comments":[{"comment":"The central comparative claim is confounded by the API-level policy-violation filter. Table 2 implies 2,034 policy-violation responses in TS1 (707+762+565) and 2,457 in TS2, so 4,491 of 10,080 inputs were blocked before demonstrably reaching o3-mini and were redefined as safe. Finding 1 compares the 49 evaluator-flagged (or 32 confirmed) TS1 unsafe outputs with 752, 166, and 215 unsafe behaviors found in GPT-3.5, GPT-4, and GPT-4o in prior work [10], where no such filter is reported and where the counts were not manually verified. The abstract and Finding 1 therefore overstate the model-level conclusion; the evidence supports only a system-level statement conditional on OpenAI's beta API. If the firewall-like mechanism is external and disabled in the public release, the blocked prompts might produce unsafe responses, and the ordering across models could reverse. The authors should reframe the headline claim, report blocked-input counts separately, and re-run the blocked inputs on the released model if access permits.","section":"3.2 and 4.2 (Table 2, Finding 1)"},{"comment":"The manual-verification procedure only inspects outputs that the GPT-3.5 evaluator flagged as 'unsafe' or 'unknown'. Consequently, the 87 confirmed cases are a lower bound whose completeness depends on the evaluator's recall, and recall is not measured in this study. The authors state that the evaluator was chosen because it had higher accuracy in [10], but they do not report precision, recall, or inter-annotator agreement for the o3-mini executions, nor do they estimate how many unsafe outputs among the non-flagged, non-blocked responses could have been missed. This does not invalidate the 87 confirmed instances, but the paper should state prominently that the count is not an estimate of the total number of unsafe behaviors in the executed suite.","section":"3.3"},{"comment":"Finding 2's comparison with Llama 2 mixes confirmed and unconfirmed counts without adjustment: the 49 unconfirmed unsafe outputs in TS1 for o3-mini are directly compared with 95 unsafe outputs for Llama 2 from [10], which the authors themselves note were not manually verified. If manual verification were applied to the Llama 2 counts, the ordering could change. The comparison should be restricted to like-for-like counts under comparable conditions, or the discussion should explicitly state that the Llama 2 figures are unverified and may include false positives.","section":"4.2 (Finding 2)"}],"minor_comments":[{"comment":"The model name is typeset inconsistently (e.g., 'O3-mini' in the title versus 'o3-mini' in the text); use a single spelling consistently.","section":"Throughout"},{"comment":"The sentence 'The reults can be obtained...' contains a typo; also consider citing the Zenodo dataset explicitly as a reference.","section":"Replication Package"},{"comment":"The source includes an unreferenced figure file 'frog.jpg'; either cite the figure in the text or remove it from the submission.","section":"arXiv source"},{"comment":"The nature of the 'policy violation' exception is only conjectured ('we conjecture that the API was able to detect unsafe test inputs'); reporting the API exception type or message would help readers distinguish an external filter from a model-level refusal.","section":"3.2"},{"comment":"The caption says 'Column Saf e' and the meaning of 'Safe (policy violation)' should clarify whether these are API exceptions or generated refusal responses; the current wording conflates them.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a timely but narrow experience report. The replication package and manual verification are strengths, but the central cross-model comparison relies on unverified counts from the authors' own previous work and is further weakened by the API-level firewall. The editor may also wish to consider whether a single beta-model evaluation whose API behavior may differ from the public release fits the journal's scope as a full research article. These considerations support my recommendation of major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a transparent and useful pre-deployment safety report on o3-mini. The 87 manually confirmed unsafe behaviors out of 10,080 generated prompts are a genuine empirical contribution, and the authors are honest about the system-level nature of the test. But the headline comparative claim—that o3-mini is safer than its predecessors—does not hold as a model-level statement, because a large fraction of inputs were blocked by an API policy firewall before they reached the model.\n\nWhat's new: the first public external pre-deployment evaluation of o3-mini, using ASTRAL-generated unsafe prompts across 14 categories, with manual verification of everything the evaluator flagged. The replication package and ASTRAL code are available, which is good practice. The category-level findings—politics and terrorism triggering the most unsafe outputs, plausibly linked to recent events—are worth reporting.\n\nSoft spots: the firewall confound is the big one. Of 10,080 inputs, 4,491 raised policy-violation exceptions and never demonstrably reached o3-mini. Comparing 32 confirmed unsafe TS1 outputs to 752/166/215 from prior work on GPT-3.5/GPT-4/GPT-4o, where no such firewall existed, is not apples-to-apples. If the firewall is external and disabled in the public release, the blocked prompts could elicit unsafe responses, and the safety ordering could reverse. The authors acknowledge this in Finding 3 and recommend re-testing, but the abstract and Finding 1 still lean on the comparison without that caveat. Second, only evaluator-flagged outputs were manually checked, so false negatives remain unmeasured; the 87 is a lower bound conditional on the filter. Third, the evaluator is GPT-3.5 with its own biases—a single-oracle design that should be acknowledged more explicitly.\n\nOverall, the descriptive results are solid and honestly reported. The comparative claims need to be reframed to say: the o3-mini API system as tested, including the firewall, blocked most inputs and produced fewer unsafe responses than older models tested without such filtering. That is still valuable information.\n\nRecommendation: this deserves peer review. It is a legitimate empirical study with open data, a clear methodology, and limitations stated in the text. A serious referee should push for the comparative claims to be reworded and for the firewall's role to be made explicit in every count. I would bring it to a reading group as a case study in evaluating pre-deployment model safety, but I would not cite it as evidence of o3-mini's inherent safety without the caveat.","headline":"Useful pre-deployment safety report on o3-mini with a real empirical core, but the comparative 'safer than predecessors' claim is confounded by the API-level firewall and needs reframing.","tokens_in":9692,"tokens_out":2350,"would_cite":true,"duration_ms":21865,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-release testing of OpenAI's o3-mini found 87 confirmed unsafe behaviors in 10,080 adversarial prompts, with an external API firewall blocking many of the rest.","keywords":["LLM safety testing","ASTRAL","automated test generation","red teaming","pre-deployment evaluation","policy violation guard","safety benchmarks","OpenAI o3-mini"],"falsifier":"Re-run the exact 10,080-prompt suite against the publicly deployed o3-mini (or a beta without the policy firewall) and count human-confirmed harmful outputs among the roughly 4,491 inputs that previously triggered policy violations. If a substantial share of those prompts now elicit harmful content, the paper's 'safer than predecessors' conclusion would be an artifact of the external guard rather than the model.","tokens_in":8760,"feed_emoji":"🛡️","tokens_out":7833,"duration_ms":72816,"temperature":0.7,"pith_summary":"The paper reports a safety test of OpenAI's o3-mini before its public release, performed from outside the model developer's own team. The authors ran 10,080 automatically generated unsafe prompts through a beta version of the model, using their ASTRAL tool to vary 14 harm categories, six writing styles, and five persuasion techniques, including prompts tied to live news events. After manually checking every response the automated judge flagged as unsafe or uncertain, they confirmed 87 genuinely unsafe behaviors, most often on controversial political topics and terrorism. They conclude that o3-mini seems safer than earlier OpenAI models and other tested LLMs, but they identify a key complication: roughly 4,491 inputs were rejected by an external policy-violation mechanism before reaching the model, so the measured safety is that of the API system, not necessarily of o3-mini alone.","feed_headline":"Pre-release test: 87 unsafe outputs in 10,080 adversarial prompts","feed_subtitle":"Automated testing on o3-mini's beta found most danger blocked by an external firewall, not the model itself.","key_machinery":"The central mechanism is ASTRAL, an automated safety-testing pipeline whose black-box coverage criterion forces balanced generation across 14 safety categories, six writing styles, and five persuasion techniques. It combines retrieval-augmented generation, few-shot prompting, and live web browsing to produce novel, up-to-date unsafe prompts rather than reusing static benchmarks. A second LLM, GPT-3.5, acts as an automated oracle classifying outputs as safe, unsafe, or unknown; only the flagged and unknown outputs receive manual human review. The pipeline also treats policy-violation rejections as safe outcomes, which is what reframes the measurement as a system-level safety check.","core_discovery":"On its own terms, the paper's discovery is a quantified pre-deployment safety profile: 87 out of 10,080 adversarial prompts produced human-confirmed unsafe outputs in the beta o3-mini, versus much larger unverified counts found by the same tool in earlier models (752 for GPT-3.5, 166 for GPT-4, 215 for GPT-4o, and 95 for Llama 2 on the first test suite). The authors attribute much of this improvement to a firewall-like API guard that rejected roughly 4,491 test inputs with policy-violation errors before they reached the LLM, and they explicitly note that the assessment is therefore at the level of OpenAI's API rather than the o3-mini model in isolation. They recommend re-running the blocked inputs if the guard is not kept in the public release.","pith_inferences":["This suggests the 87 confirmed cases are a lower bound, because only evaluator-flagged and unknown outputs were manually inspected, and some responses the judge called safe could still be harmful in ways the oracle misses.","This also means the cross-model comparison is partly confounded: earlier models executed all prompts, while o3-mini blocked nearly half at the API boundary, so 'safer' may reflect a different measurement pipeline as much as a safer model.","A natural extension would be to run the same second test suite against the public release and compare both rejection rates and the categories of confirmed failures, measuring whether the firewall and the alignment survive deployment.","Because ASTRAL's prompts include current events, safety performance is time-dependent; a model that is safe for January 2025 events may fail on later news topics, so pre-deployment results should be re-evaluated on a rolling basis."],"forward_implications":["If the API-level policy guard persists in the public release, o3-mini's measured safety level, 87 confirmed unsafe responses in 10,080, is the level users will likely experience.","If the guard is removed, OpenAI should re-execute the roughly 4,491 blocked test cases before claiming o3-mini is safe.","Recent controversial topics, especially U.S. politics (category c3) and terrorism (c13), are the areas where the model most often failed; these categories warrant focused alignment work.","Animal abuse (c1) and drug and weapon content (c5) also produced repeated failures, suggesting those categories deserve special attention alongside c3 and c13.","The safety-helpfulness tradeoff was not measured, so the high observed safety may come with reduced usefulness in borderline situations."],"supporting_citations":[{"why":"Supplies the ASTRAL tool and methodology, and the previous GPT-3.5, GPT-4, GPT-4o, and Llama 2 comparison numbers that ground the claim that o3-mini is safer.","marker":"[10]"},{"why":"Defines the balanced safety-benchmark baseline that ASTRAL's black-box coverage criterion extends to keep test inputs novel and up to date.","marker":"[1]"},{"why":"Raises the safety-helpfulness tradeoff, which the paper explicitly leaves unmeasured as a boundary on interpreting its safety results.","marker":"[25]"}],"fun_headline_variants":["87 unsafe replies in 10,080 attacks on o3-mini beta, firewall eased risk","10k adversarial prompts on o3-mini beta: 87 unsafe replies, rest blocked","Firewall did the heavy lifting: only 87 unsafe outputs in 10k o3-mini probes","o3-mini beta: 87 unsafe outputs in 10k tests, API guard saved the day","Pre-deployment probe: 87 real safety misses in 10,080 mock attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result that o3-mini is safer than its predecessors rests on treating every policy-violation rejection as a safe response; if that external API firewall is absent in the public version, the unexecuted prompts could produce unsafe answers and the safety comparison collapses.","fun_headline_variants_meta":{"raw":{"variants":["87 unsafe replies in 10,080 attacks on o3-mini beta, firewall eased risk","10k adversarial prompts on o3-mini beta: 87 unsafe replies, rest blocked","Firewall did the heavy lifting: only 87 unsafe outputs in 10k o3-mini probes","o3-mini beta: 87 unsafe outputs in 10k tests, API guard saved the day","Pre-deployment probe: 87 real safety misses in 10,080 mock attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":3975,"prompt_tokens":943,"completion_tokens":3032,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2910}},"tokens_in":559,"tokens_out":3032,"duration_ms":21519,"temperature":1.0,"reasoning_tokens":2910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:33:16.538106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact 10,080-prompt suite against the publicly deployed o3-mini (or a beta without the policy firewall) and count human-confirmed harmful outputs among the roughly 4,491 inputs that previously triggered policy violations. If a substantial share of those prompts now elicit harmful content, the paper's 'safer than predecessors' conclusion would be an artifact of the external guard rather than the model.","supporting_citations":[{"cited_title":"Astral: Automated safety testing of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the ASTRAL tool and methodology, and the previous GPT-3.5, GPT-4, GPT-4o, and Llama 2 comparison numbers that ground the claim that o3-mini is safer."}],"review_version":1}