REVIEW 3 major objections 4 minor 1 cited by
Current AI biosecurity safeguards refuse legitimate research more often than they catch hidden threats.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 08:34 UTC pith:WXLYO4XH
load-bearing objection A useful paired refusal benchmark whose headline over-refusal finding is plausible, but the API-vs-model decomposition is not yet validated. the 3 major comments →
BioSecBench-Refusal: A paired metric for performance and alignment in agentic biosecurity risk assessment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that current biosecurity refusal systems in agentic AI are miscalibrated in both directions: they refuse legitimate tasks at high rates, up to 74% on Routine evaluations, while missing many concealed Red-Team hazards, with refusal rates as low as 1%. Refusals are driven predominantly by provider API filters that trigger on surface-level biosecurity cues in the prompt, not on task-specific biological analysis; when requests are not blocked upstream, agentic reasoning sometimes identifies deeper threats. The benchmark pairs 61 Routine tasks adapted from published research with 46 fictional Red-Team tasks that hide hazards in attached data artifacts, and it decomposes refus
What carries the argument
The central object is the paired benchmark itself, consisting of 61 Routine and 46 Red-Team tasks, each run under direct and meta framings. The key mechanism is the pairing: by measuring refusal rates on both legitimate and hazardous tasks under identical conditions, the benchmark isolates whether refusal tracks surface text or underlying biological risk. A two-stage grading pipeline separates provider API refusals (detected via provider-specific signatures such as OpenAI's 'limited access...' message, xAI's 'Content violates usage guidelines', and Anthropic's stop_reason) from model-initiated refusals (assigned by a judge model reading raw trajectories). This decomposition is what reveals t
Load-bearing premise
The headline refusal rates depend on an unvalidated judge model that reads raw trajectories and decides whether the model itself refused; if that classifier mislabels even a fraction of runs, the central comparison between Routine and Red-Team refusal rates shifts.
What would settle it
Manually audit a random sample of the 4,627 scored trajectories, labeling each as API refusal, model refusal, or genuine attempt by human review, and recompute the Routine-versus-Red-Team refusal-rate gap using only human labels. If the gap shrinks to near zero or reverses, the paper's miscalibration claim collapses. Alternatively, show that provider API filters trigger on concealed Red-Team hazards (e.g., a toxin hidden in a FASTA file) as often as on surface keywords in Routine tasks.
If this is right
- Model developers can use the paired benchmark to calibrate refusal stringency to a specific deployment context, adjusting thresholds by biosafety level, agent class, and risk category.
- Provider-side filters, not model reasoning, are the main source of refusal; improving biosecurity safeguards requires shifting from keyword screening toward agentic reasoning over biological data.
- The wide variance in refusal rates across 16 configurations (7–74% Routine, 1–62% Red-Team) shows there is currently no industry consensus on the safety–utility trade-off, and this benchmark provides a common metric for comparing configurations.
- Preliminary evidence that meta-evaluation improves Red-Team detection (e.g., GPT-5.5 refusing 14.5–19.6% under meta framing vs 13% direct) suggests that allowing models to reason could improve hazard identification without necessarily increasing over-refusal.
- The high Routine refusal rates represent a measurable cost to legitimate research; reducing over-refusal while maintaining Red-Team detection could improve both safety and research productivity.
Where Pith is reading between the lines
- If refusals track surface language rather than underlying biology, then adversaries can likely evade filters by concealing hazards in attached data—exactly what the Red-Team set demonstrates—so safety claims based solely on prompt filtering are fragile.
- Because the judge-model refusal detection is not validated against human annotation, the headline refusal-rate gap between Routine and Red-Team tasks could change with a more rigorous measurement protocol; a manual audit of the 4,627 scored trajectories would settle this.
- The benchmark's full set is restricted-access, with only two public examples; independent replication of the miscalibration claim would be easier if a larger public subset were released.
- A testable extension suggested by the paper's own limitation: run the same model-harness pairs with provider filters disabled (or loosened) to estimate how often agentic reasoning would catch hazards that are currently blocked upstream as API refusals.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BioSecBench-Refusal, a paired benchmark of 61 Routine tasks (legitimate analyses adapted from published literature) and 46 Red-Team tasks (fictional scenarios with concealed biosecurity hazards), and evaluates 16 model-harness configurations under direct and meta framings, with three runs per task. The headline results are that direct-framing refusal rates range from 7% to 74% on Routine tasks and 1% to 62% on Red-Team tasks; that Routine tasks are refused at comparable or higher rates than concealed Red-Team hazards; that API-side provider filters, rather than model reasoning, account for most refusals; and that when agents are allowed to reason, they sometimes identify additional Red-Team threats. The authors release a two-example public subset and keep the full 107-task set under restricted access.
Significance. If the central claims are substantiated, the benchmark would be a useful diagnostic for calibrating agentic biosecurity safeguards. The paired Routine/Red-Team design directly addresses the over-refusal versus under-detection trade-off, the literature-derived Routine tasks give the 'legitimate use' baseline a concrete, falsifiable grounding, and the two-frame (direct/meta) design is a thoughtful way to separate surface-level screening from agentic reasoning. The paper also reports a large number of runs (5,136) across diverse providers and harnesses. However, the paper's most important mechanism claim—that refusals are predominantly caused by provider API filters rather than model judgment—rests on two unvalidated components: an unnamed, unvalidated judge model for direct-framing model refusals, and an incomplete API-refusal detection gate that lists signatures only for OpenAI, xAI, and Anthropic, with no Google/Gemini signature. The reader's conditional assessment is appropriate: the headline refusal rates may be broadly right, but the decomposition into API versus model refusals is load-bearing and currently not independently established.
major comments (3)
- [Methods, 'Refusal detection'] Direct-framing 'model refusal' labels are assigned by an unspecified judge model reading raw trajectories, with no judge identity, prompt, validation set, inter-rater agreement, or manual audit reported. This is load-bearing because Figures 2 and 4B separate API refusals from model refusals, and the paper's central attribution claim ('refusals were most often triggered by provider API filters') depends on that separation. If the judge over-labels or under-labels refusals, the pink segments and model-refusal rates change, and the routine-vs-red-team gap could shift. Please report the judge model and prompt, provide a human-labeled validation sample (e.g., 100 trajectories per model family), and give confusion statistics or a sensitivity analysis over judge labels.
- [Methods, 'Refusal gating' and 'Aggregation and statistics'] The deterministic API-refusal gate lists signatures for OpenAI, xAI, and Anthropic, but no Google/Gemini signature, even though Gemini 3.1 Pro and 3.5 Flash are evaluated. Under 'Aggregation and statistics,' runs with no usable artifact are counted as not-blocked. If Gemini provider-side blocks produce a different message or an empty/truncated output, those blocks are misclassified as non-refusals, potentially undercounting API refusals for two configurations whose total direct refusal rates are 35–45%. Report how Gemini provider-side blocks are detected, or explicitly state that none occur, and show a sensitivity analysis for Figures 2 and 4A that excludes no-artifact runs.
- [Methods, 'Agent runs and execution' and 'Aggregation and statistics'] Only three runs per model x harness x evaluation are used, and 509 of 5,136 runs yield no usable artifact. The headline ranges (7–74% Routine, 1–62% Red-Team) and the routine-vs-red-team correlation (Pearson r = 0.91) are therefore based on small per-cell denominators and on treating no-artifact runs as non-refusals. Please report per-cell attempt counts, the distribution of no-artifact runs across the 16 configurations, and exact binomial intervals or a sensitivity analysis that drops no-artifact runs. This is necessary to know whether the magnitude of the reported ranges and the 'API dominates' claim are robust to the 10% of runs that produced no usable artifact.
minor comments (4)
- [Data Availability] Only two example evaluations are released; the full 107-task set is restricted. For a benchmark paper, reviewers and users need at least the evaluation harness, scoring code, and structured metadata for all tasks, even if task texts remain access-controlled. Please also specify the exact license and access mechanism.
- [Methods, 'Aggregation and statistics'] The term 'no usable artifact' is not defined precisely. Does it include API timeouts, malformed outputs, harness crashes, or provider-side blocks lacking the listed signatures? A short enumeration would help readers interpret the 509 excluded runs.
- [Figure 1] Panel F labels 'n=61' and 'n=46' inside the bars may be confused with the bar counts; consider placing the n labels in the panel title or legend, and ensure the multimodal counts are clearly distinct from evaluation counts.
- [References] Reference [1] contains an OCR artifact: 'T errorism' should be 'Terrorism'. Please correct.
Circularity Check
No circularity found: the paper's claims are empirical measurements of model/harness behavior under fixed task sets, with no derivation chain that reduces to its inputs.
full rationale
BioSecBench-Refusal is an empirical benchmark paper rather than a formal derivation. The headline claims—refusal-rate ranges, Routine-vs-Red-Team gaps, and API-vs-model refusal decomposition—are operationalized measurements of external model and harness behavior. The task labels ('Routine' as legitimate-but-keyword-rich, 'Red-Team' as hazardous-but-concealed) are designed inputs, not outputs derived from the measured refusal rates; the observed gap could in principle have been absent or reversed if providers did not filter on surface cues. The API-refusal classification is deterministic and based on provider message signatures, while model refusal is judged from raw trajectories; both are measurement procedures, not equations that force the reported conclusions. The paper contains self-citations (e.g., refs [4], [14]) but these are background context and do not carry the central argument. There are genuine validity concerns—the judge model for model refusal is unvalidated, and the API gating list omits Google despite evaluating Gemini configurations—but these are reliability/measurement limitations, not circularity. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported from the authors' prior work, and no result is true by construction. Therefore the appropriate circularity score is 0.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption The 14-author expert labels define ground truth: Routine tasks are legitimate, Red-Team tasks are hazardous, and their threat-category annotations are correct.
- domain assumption Provider-specific API refusal signatures exactly identify all API-layer blocks.
- domain assumption The judge model's trajectory-level decline labels are an accurate measure of model refusal.
- domain assumption Three independent runs per model-harness-evaluation cell are sufficient to estimate refusal rates and standard errors.
read the original abstract
As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse. We present BioSecBench-Refusal, a benchmark for risk identification and refusal behavior for biological research tasks. The benchmark pairs 61 Routine tasks, legitimate analyses adapted from the published literature, with 46 Red-Team tasks, fictional scenarios that resemble real research but conceal a biosecurity hazard. Across 16 model-harness configurations, refusal rates ranged from 7 percent to 74 percent on Routine tasks and 1 percent to 62 percent on Red-Team tasks, with many configurations refusing legitimate Routine work at comparable or higher rates than concealed hazards. Refusals were most often triggered by provider API filters applied prior to agentic reasoning. However, models given room to reason showed the potential to identify more real threats. We release BioSecBench-Refusal as a tool for model developers to calibrate capability and caution for agentic biotech research and development.
Forward citations
Cited by 1 Pith paper
-
BioTIER: A Refusal Benchmark for Targeted Biological Risk Mitigation
BioTIER, a 542-prompt benchmark with three risk tiers, shows frontier AI models differ by 90 percentage points in refusing dangerous biological queries, with top refusers over-refusing benign topics at the boundary.
Reference graph
Works this paper leans on
-
[1]
National Academies Press, Washington, D.C., 2004
National Research Council.Biotechnology Research in an Age of T errorism. National Academies Press, Washington, D.C., 2004
2004
-
[2]
National Academies Press, Washington, D.C., 2017
National Academies of Sciences, Engineering, and Medicine.Dual Use Research of Concern in the Life Sciences: Current Issues and Controversies. National Academies Press, Washington, D.C., 2017
2017
-
[3]
Dual-use capabilities of concern of biological ai models.PLoS computational biology, 21(5):e1012975, 2025
Jaspreet Pannu, Doni Bloomfield, Robert MacKnight, Moritz S Hanke, Alex Zhu, Gabe Gomes, Anita Cicero, and Thomas V Inglesby. Dual-use capabilities of concern of biological ai models.PLoS computational biology, 21(5):e1012975, 2025
2025
-
[4]
Without safeguards, ai-biology integration risks accelerating future pandemics.Frontiers in Microbiology, 16:1734561, 2026
Dianzhuo Wang, Marian Huot, Zechen Zhang, Kaiyi Jiang, Eugene I Shakhnovich, and Kevin M Esvelt. Without safeguards, ai-biology integration risks accelerating future pandemics.Frontiers in Microbiology, 16:1734561, 2026
2026
-
[5]
Amoresystematicapproach tobiologicalrisk.Science, 350(6267):1471– 1473, 2015
MeganJPalmer, Francis Fukuyama, and DavidA Relman. Amoresystematicapproach tobiologicalrisk.Science, 350(6267):1471– 1473, 2015
2015
-
[6]
Ai and biosecurity: The need for governance.Science, 385(6711):831–833, 2024
Doni Bloomfield, Jaspreet Pannu, Alex W Zhu, Madelena Y Ng, Ashley Lewis, Eran Bendavid, Steven M Asch, Tina Hernandez- Boussard, Anita Cicero, and Tom Inglesby. Ai and biosecurity: The need for governance.Science, 385(6711):831–833, 2024
2024
-
[7]
Safe rlhf: Safe reinforcement learning from human feedback
Juntao Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. InInternational Conference on Learning Representations, volume 2024, pages 50750– 50777, 2024
2024
-
[8]
Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022. 8
Pith/arXiv arXiv 2022
-
[9]
Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837, 2025
Pith/arXiv arXiv 2025
-
[10]
Hoagy Cunningham, Jerry Wei, Zihan Wang, Andrew Persic, Alwin Peng, Jordan Abderrachid, Raj Agarwal, Bobby Chen, Austin Cohen, Andy Dau, et al. Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks.arXiv preprint arXiv:2601.04603, 2026
arXiv 2026
-
[11]
Or-bench: An over-refusal benchmark for large language models
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024
Pith/arXiv arXiv 2024
-
[12]
Lukas Weidener, Marko Brkić, Mihailo Jovanović, Emre Ulgac, and Aakaash Meduri. Refusalbench: Why refusal rate misranks frontier llms on biological research prompts.arXiv preprint arXiv:2605.21545, 2026
Pith/arXiv arXiv 2026
-
[13]
Jasper Götting, Pedro Medeiros, Jon G Sanders, Nathaniel Li, Long Phan, Karam Elabd, Lennart Justen, Dan Hendrycks, and Seth Donoughe. Virology capabilities test (vct): A multimodal virology q&a benchmark.arXiv preprint arXiv:2504.16137, 2025
Pith/arXiv arXiv 2025
-
[14]
Andrew Bo Liu, Samira Nedungadi, Bryce Cai, Alex Kleinman, Harmon Bhasin, and Seth Donoughe. Abc-bench: An agentic bio-capabilities benchmark for biosecurity.arXiv preprint arXiv:2606.11150, 2026
Pith/arXiv arXiv 2026
-
[15]
Agentic baim-llm evaluation (able): Benchmarking llm use of protein design tools
Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi, Jake Yukich, and Seth Donoughe. Agentic baim-llm evaluation (able): Benchmarking llm use of protein design tools. InNeurIPS 2025 Workshop on Biosecurity Safeguards for Generative AI, 2025
2025
-
[16]
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024
Pith/arXiv arXiv 2024
-
[17]
Gary Ackerman, Brandon Behlendorf, Zachary Kallenborn, Sheriff Almakki, Doug Clifford, Jenna LaTourette, Hayley Peterson, Noah Sheinbaum, Olivia Shoemaker, and Anna Wetzel. Biothreat benchmark generation framework for evaluating frontier AI models I: The task-query architecture.arXiv preprint arXiv:2512.08130, 2025
arXiv 2025
-
[18]
Peptide inhibitors of theα-cobratoxin–nicotinic acetylcholine receptor interaction.Journal of medicinal chemistry, 63(22):13709–13718, 2020
Timothy Lynagh, Stephan Kiontke, Maria Meyhoff-Madsen, Bengt H Gless, Jónas Johannesen, Sabrina Kattelmann, Anders Christiansen, Martin Dufva, Andreas H Laustsen, Kanchan Devkota, et al. Peptide inhibitors of theα-cobratoxin–nicotinic acetylcholine receptor interaction.Journal of medicinal chemistry, 63(22):13709–13718, 2020. 9
2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.