Pith. sign in

REVIEW 3 major objections 4 minor 39 references

This paper claims that decomposing the harmfulness of scientific LLM responses into executability and net-new risk yields a metric that tracks expert judgments far more closely than existing LLM-as-a-judge methods, and that this metric expo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:41 UTC pith:6W34NZYR

load-bearing objection A well-built benchmark and a genuinely clever decomposed harm metric — but the 90.17% headline needs a cleaner validation design before I'd trust it. the 3 major comments →

arxiv 2607.18665 v1 pith:6W34NZYR submitted 2026-07-21 cs.AI

SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring

Chunxiao Li , Yuan Xiong , Lijun Li , Tianyi Du , Wenlong Zhang , Lei Bai , Jing Shao This is my paper
classification cs.AI
keywords AI safetyscientific dual-use riskLLM evaluation benchmarkharm decompositionexecutabilitynet-new riskdeep research agentsLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper attempts to establish that harmfulness in scientific Q&A can be measured reliably by separating it into two orthogonal dimensions: how executable the instructions are (executability) and how much genuinely new dangerous knowledge the response adds (net-new risk). To do this, the authors build SciHazard, a benchmark of 2,400 real-world-grounded hazardous questions and 600 oversafety questions across 12 disciplines, and DeHarm-Score, a decomposed scoring pipeline that combines query hazard severity, refusal behavior, and those two response-level dimensions. In a validation study with domain experts, DeHarm-Score reaches 0.774 quadratic weighted kappa, a 90.17% relative improvement over the strongest baseline, and passes perturbation tests that trip up holistic judges. Applied to 31 frontier LLMs and deep research agents, the metric finds agents score 32.3% higher on average than standard LLMs, suggesting that retrieval and tool-use amplify rather than mitigate risk.

Core claim

The central discovery is that a single holistic 'harmfulness' score is unstable even among experts (inter-annotator α=0.568), but decomposing it into Executability (E) and Net-New Risk (N) raises agreement to α=0.832. The automated DeHarm-Score operationalizes this decomposition: the E-branch generates an importance-weighted dynamic checklist from the query and uses two-pass fact matching to score how completely the response satisfies it; the N-branch extracts up to ten technical claims from the response, retrieves public evidence for each, and rates coverage and synthesis barrier. The final score is S=H_q·R·(E+N)/8, where H_q is query hazard severity, R is a refusal indicator, and (E+N)/8 m

What carries the argument

The load-bearing mechanism is the multiplicative DeHarm-Score S = H_q · R · (E+N)/8, which encodes a progressive gating logic: query severity sets the theoretical ceiling, refusal is a binary off-switch, and the normalized sum of Executability (E) and Net-New Risk (N) modulates residual harm. Executability is measured by an importance-weighted dynamic checklist with blocker items that cap the score, while Net-New Risk is measured by retrieval-augmented claim extraction that grades public coverage and synthesis barrier. This decomposition is what allows the metric to distinguish 'detailed but trivial' from 'vague but novel' responses, and it is the reason the metric survives perturbations tha

Load-bearing premise

The metric's validity rests on the assumption that the 173 expert-annotated responses, scored with the paper's Executability and Net-New Risk rubrics, are an unbiased ground truth for scientific harmfulness—if the ground truth is rubric-bound and the metric is tuned to that rubric, the reported agreement may not generalize to other notions of harm.

What would settle it

A concrete test: take a set of responses whose harmfulness is independently judged by real-world consequences (e.g., whether a biosecurity or chemical safety incident actually followed from the information), or have a fresh set of experts score the same responses using a generic 'harmfulness' definition without the E/N rubric, and check whether DeHarm-Score still achieves QWK above 0.7 against that independent criterion. If agreement collapses, the metric is measuring its own rubric rather than scientific harm.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If DeHarm-Score is valid, safety evaluations of scientific LLMs and agents can move from subjective holistic judgments to a reproducible, decomposed protocol that separates actionability from knowledge uplift.
  • The finding that deep research agents score 32.3% higher than standard LLMs implies that agentic retrieval and planning erode refusal barriers and amplify content-level harm, so safety benchmarks for frontier systems must include agentic configurations.
  • The perturbation results imply that simple holistic judges are systematically blind to unexecutable parameter tampering and overly sensitive to safety disclaimers, while the E/N decomposition catches both.
  • The high refusal rate of some models (e.g., o3 at 82.2%) does not imply safety, because conditional harmfulness (DeHarm†) remains high; evaluations should report both refusal rate and per-response harm.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the E/N decomposition is a natural fit for any dual-use knowledge domain (e.g., cyber operations, financial fraud), where 'can the reader act on this?' and 'is this more than public knowledge?' are the operative questions; applying it there would test its generality.
  • Editorial extension: because Net-New Risk is measured against live web retrieval, DeHarm-Score is a moving target—scores depend on what is already public at evaluation time, so cross-time comparisons require the caching protocol the paper introduces.
  • Editorial extension: to confirm that DeHarm-Score tracks real-world harm rather than the authors' rubric, a future study could compare scores against an independent harm criterion (e.g., whether the response actually enables a documented attack), which the current expert validation does not isolate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. SciHazard is a scientific-safety benchmark and an associated evaluation metric, DeHarm-Score. The dataset contains 2,400 harmful and 600 oversafety questions across 12 disciplines, generated by a dual-track pipeline grounded in regulated substances and documented misuse scenarios. DeHarm-Score computes S = Hq * R * (E+N)/8, where E is a checklist-based executability score and N is a retrieval-based net-new-risk score. The authors validate the metric on 173 expert-annotated responses, reporting QWK = 0.774, MAE = 0.196, Spearman rho = 0.812, and use it to benchmark 31 models, finding that deep-research agents have a 32.3% higher mean score than standard LLMs.

Significance. The paper addresses an important evaluation gap: LLM-as-a-judge metrics often fail in knowledge-dense scientific domains. The proposed decomposition into E and N is well motivated, and the authors provide substantial open resources: the benchmark, code, and a timestamp-controlled retrieval caching protocol for reproducibility. The ablations (Appendix G) indicate that retrieval and decomposition contribute non-redundantly. If the human-alignment claim survives a stricter meta-evaluation, this would be a valuable contribution to scientific safety measurement. At present, however, the central claim is not fully supported because the ground truth is rubric-bound, threshold fitting is not disclosed, uncertainty is not quantified, and the baseline comparison is asymmetric.

major comments (3)
  1. [§5.3, Appendix F.3.1/F.3.2] The central validity claim rests on agreement with expert annotations, but those annotations were produced using the authors' own Executability and Net-New Risk rubrics (Appendix B, Table 10), and DeHarm-Score outputs E and N on exactly those scales. The aggregation thresholds are reported only as 'specified in the released configuration file' (F.3.1, F.3.2), with no description of how they were set and no train/test split. If thresholds were chosen after inspecting the 173 validation responses, the reported QWK of 0.774 is an in-sample estimate. Please provide a held-out evaluation, a sensitivity analysis over threshold values, or a statement that all thresholds were fixed before expert annotation.
  2. [§5.3, Tables 3–5] The meta-evaluation is based on only 173 expert-annotated responses, yet no confidence intervals or significance tests are reported for QWK, MAE, or Spearman rho. The 0.774 vs. 0.407 QWK margin over DeepReject could be meaningful, but with n=173 and no uncertainty quantification the reader cannot assess whether the margin is reliable or stable across subdisciplines. Please report bootstrap CIs and, where appropriate, paired significance tests for the aggregate metrics and for the subdimensional results in Tables 3 and 4.
  3. [Eq. (1), Appendix C, Table 5] The headline comparison with baselines is asymmetric. DeHarm-Score receives Hq, a query hazard severity label that was precomputed and separately validated (91.38% exact match, Appendix D), while the baseline methods—DeepReject, StrongReject, LLM-as-a-Judge, SafetyScore—must infer severity end-to-end from the query. If Hq is available as benchmark metadata, it should be given to all methods, or DeHarm-Score should be re-run without Hq, to establish that the reported margin is not an artifact of privileged information. This is especially important because Hq enters multiplicatively in Eq. (1).
minor comments (4)
  1. [Table 1] The row for the proposed dataset lists 'Ours 3000' but the paper describes 2400 harmful + 600 oversafety questions. Please make the arithmetic explicit in the table caption or text.
  2. [§5.2, Table 2] The 'Composite' agreement metric should be defined precisely: is it the agreement of the composite score Hq*R*(E+N)/8 computed from each expert's E and N, and does it use a shared Hq? If Hq is shared across raters, that partly mechanical agreement should be acknowledged.
  3. [Appendix D] The Hq validation study uses 360 queries and is separate from the 173-sample meta-evaluation. Please clarify whether the 173 responses are a subset of those 360 queries or an independent sample, since this affects how the Hq validation supports the main results.
  4. [General] There are inconsistent spellings of the second dimension: 'NET-NEWRISK' in the abstract and figure captions, 'Net-New Risk' in Section 4, and 'Net-new risk' in Appendix A. Please standardize. Also, the claim in Appendix G that gains are 'super-additive' is asserted without an interaction test; a more conservative phrasing would be preferable.

Circularity Check

0 steps flagged

No significant circularity: DeHarm's E/N branches are independently operationalized and externally validated; threshold nondisclosure is a reproducibility weakness, not a demonstrated circular reduction.

full rationale

The central derivation is not circular. DeHarm-Score is defined in Eq. 1 as S = Hq * R * (E+N)/8, with E and N produced by a checklist/retrieval pipeline (Appendix F) rather than by fitting to expert labels. The expert labels were collected using the authors' rubrics in Table 10, but the paper reports no equation or fitting procedure that makes the predicted E/N equal to the human annotations; the QWK values in Tables 3-5 are empirical correlations against independently collected human judgments. The Hq component is externally validated against experts (91.38% exact match, QWK 0.96, Appendix D), and the N-branch relies on live-web retrieval, which provides an external anchor. The ablation study (Appendix G) further shows that retrieval and decomposition each contribute non-redundant gains, supporting the claim that the pipeline's components have independent content. The main caveat is that Appendix F.3.1/F.3.2 state that 'All threshold values are specified in the released configuration file,' with no train/test split or threshold-fitting disclosure; this is a reproducibility weakness and a potential tuning concern, but the paper does not exhibit the specific reduction needed to establish circularity. Self-citations (e.g., SALAD-Bench [15], Intern-S1-Pro [33]) are used as baselines or independently released systems, not as load-bearing justification of DeHarm's validity. Overall, no step in the derivation chain reduces by construction to the metric's own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical entities. The free parameters are the hidden aggregation thresholds/weights in the E and N branches, which are load-bearing for the metric's expert-alignment claim and appear in released config files rather than in the text. The axioms are the conceptual decomposition of harm, the query-severity model, the public-knowledge proxy, the refusal classifier, and the realism of the dataset; each is reasonable but not independently established beyond the paper's own validation.

free parameters (2)
  • E-branch aggregation thresholds and blocking rules = not disclosed; in released config file
    Rule-based tier mapping uses thresholds for blocker failure and coverage density; values are not stated in the paper, and no train/test split is reported, so they may be fitted to the 173 expert annotations (Appendix F.3.1).
  • N-branch criticality weights and thresholds = not disclosed; in released config file
    Proportion of hard-coverage claims and high-synthesis-barrier count are weighted by criticality and thresholded to produce N_rule; coefficients not shown (Appendix F.3.2).
axioms (5)
  • domain assumption Executability and Net-New Risk are orthogonal and jointly sufficient to characterize response-level harmfulness
    Section 4.1 posits this decomposition; the validation against experts uses the same two dimensions, so it cannot independently confirm that no other harm dimension matters.
  • domain assumption Hazard severity of a query is an intrinsic property independent of the response, determinable from regulatory tier and operational intent
    Section D defines the R-Tier x O-Level map; validated on 360 queries at 91.38% exact-match, but full-dataset labels are LLM-committee generated and the map itself is a simplification of real-world risk.
  • domain assumption Public web availability via Serper search is a sufficient proxy for the public-knowledge barrier
    N-branch grades coverage and synthesis barrier based on top-5 snippets (Section 4.2.2, F.3.2); search index coverage, paywalls, or censorship may misestimate actual availability.
  • domain assumption The two-stage refusal classifier correctly identifies substantive refusals
    F.2 reports 99.6% accuracy on 500 labeled responses, but misclassification of partial-compliance responses directly flips R and therefore S.
  • domain assumption The 2,400 hazardous questions are realistic, actionable misuse scenarios
    Generated by an LLM pipeline from regulatory primitives and manually inspected (Section 3.1, Appendix E); full real-world feasibility is not independently verified for all items.

pith-pipeline@v1.3.0-alltime-deepseek · 21485 in / 12262 out tokens · 107774 ms · 2026-08-01T14:41:20.742170+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) increasingly support science, but they can also convert hazardous scientific knowledge into actionable misuse guidance. Existing benchmarks often rely on templated queries disconnected from real-world hazards, and employ LLM-as-a-Judge paradigms without domain grounding. To address this, we introduce SciHazard, a real-world-grounded benchmark for scientific risks and a dataset agnostic evaluation framework for measuring harmfulness. SciHazard contains 2400 hazardous questions and 600 oversafety questions across 12 disciplines, with both queries grounded in regulated entities and documented failure scenarios. To compute \textsc{DeHarm-Score} , we develop a decomposed evaluating procedure that combines query hazard severity, refusal behavior, and response-level risk. For non-refused responses, it further decomposes response-level harm into \textsc{Executability}, quantified via dynamic checklists with importance weighting, and \textsc{Net-new risk}, assessed through retrieval-augmented claim extraction and synthesis-barrier verification. An expert-validation study shows that \textsc{DeHarm-Score} improves agreement with expert annotations by 90.17\% over the strongest baseline. We benchmark 31 frontier LLMs and deep research agents in an extensive scientific safety evaluation. Notably, deep research agents yield 32.3\% higher mean \textsc{DeHarm-Score} than standard LLMs, exposing autonomous agents as a critical blind spot in current safety defenses. Code and dataset are available at https://anonymous.4open.science/r/DeharmScore-7B55.

Figures

Figures reproduced from arXiv: 2607.18665 by Chunxiao Li, Jing Shao, Lei Bai, Lijun Li, Tianyi Du, Wenlong Zhang, Yuan Xiong.

Figure 1
Figure 1. Figure 1: Overview of our dataset. (Left) SciHazard Generation Pipeline. (Right) Comparison with [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our E-Branch and N-Branch. 4.2.1 EXECUTABILITY Branch: Importance-Weighted Dynamic Checklist This branch precisely quantifies the factual completeness and operational viability of a model response through a three-phase end-to-end pipeline as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 15 linked inside Pith

  1. [1]

    Frontier ai regulation: Managing emerging risks to public safety.arXiv preprint arXiv:2307.03718, 2023

    Markus Anderljung, Joslyn Barnhart, Anton Korinek, Jade Leung, Cullen O’Keefe, Jess Whit- tlestone, Shahar Avin, Miles Brundage, Justin Bullock, Duncan Cass-Beggs, et al. Frontier ai regulation: Managing emerging risks to public safety.arXiv preprint arXiv:2307.03718, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Agentharm: A benchmark for measuring harmfulness of llm agents.arXiv preprint arXiv:2410.09024, 2024

    Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of llm agents.arXiv preprint arXiv:2410.09024, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Auto-eval judge: Towards a general agentic framework for task completion evaluation.arXiv preprint arXiv:2508.05508, 2025

    Roshita Bhonsle, Rishav Dutta, Sneha Vavilapalli, Harsh Seth, Abubakarr Jaye, Yapei Chang, Mukund Rungta, Emmanuel Aboah Boateng, Sadid Hasan, Ehi Nosakhare, et al. Auto-eval judge: Towards a general agentic framework for task completion evaluation.arXiv preprint arXiv:2508.05508, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Deerflow: An open-source long-horizon superagent harness

    ByteDance. Deerflow: An open-source long-horizon superagent harness. https://github. com/bytedance/deer-flow, 2026. Version 2.0

    2026

  5. [5]

    Deep research brings deeper harm.arXiv preprint arXiv:2510.11851, 2025

    Shuo Chen, Zonggen Li, Zhen Han, Bailan He, Tong Liu, Haokun Chen, Georg Groh, Philip Torr, V olker Tresp, and Jindong Gu. Deep research brings deeper harm.arXiv preprint arXiv:2510.11851, 2025

    arXiv 2025

  6. [6]

    Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological bulletin, 70(4):213, 1968

    Jacob Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological bulletin, 70(4):213, 1968

    1968

  7. [7]

    Mind2web 2: Evaluating agentic search with agent-as-a-judge.arXiv preprint arXiv:2506.21506, 2025

    Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, et al. Mind2web 2: Evaluating agentic search with agent-as-a-judge.arXiv preprint arXiv:2506.21506, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Sosbench: Benchmarking safety alignment on six scientific domains

    Fengqing Jiang, Fengbo Ma, Zhangchen Xu, Yuetai Li, Zixin Rao, Bhaskar Ramasubrama- nian, Luyao Niu, Bo Li, Xianyan Chen, Zhen Xiang, et al. Sosbench: Benchmarking safety alignment on six scientific domains. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  10. [10]

    Fortress: Frontier risk evaluation for national security and public safety.arXiv preprint arXiv:2506.14922, 2025

    Christina Q Knight, Kaustubh Deshpande, Ved Sirdeshmukh, Meher Mankikar, Scale Red Team, SEAL Team, and Julian Michael. Fortress: Frontier risk evaluation for national security and public safety.arXiv preprint arXiv:2506.14922, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Quantifying cbrn risk in frontier models.arXiv preprint arXiv:2510.21133, 2025

    Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, and Prashanth Harshangi. Quantifying cbrn risk in frontier models.arXiv preprint arXiv:2510.21133, 2025

    arXiv 2025

  12. [12]

    Os-harm: A benchmark for measuring safety of computer use agents.arXiv preprint arXiv:2506.14866, 2025

    Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, Zico Kolter, Nicolas Flammarion, and Maksym Andriushchenko. Os-harm: A benchmark for measuring safety of computer use agents.arXiv preprint arXiv:2506.14866, 2025

    arXiv 2025

  13. [13]

    The measurement of observer agreement for categorical data.biometrics, pages 159–174, 1977

    J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data.biometrics, pages 159–174, 1977

    1977

  14. [14]

    Realharm: A collection of real-world language model application failures

    Pierre Le Jeune, Jiaen Liu, Luca Rossi, and Matteo Dora. Realharm: A collection of real-world language model application failures. InProceedings of the The First Workshop on LLM Security (LLMSEC), pages 87–100, 2025

    2025

  15. [15]

    Salad-bench: A hierarchical and comprehensive safety benchmark for large language models

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. InFindings of the Association for Computational Linguistics: ACL 2024, pages 3923–3954, 2024

    2024

  16. [16]

    The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024

    Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning.arXiv preprint arXiv:2403.03218, 2024. 10

    Pith/arXiv arXiv 2024

  17. [17]

    Scisafeeval: a comprehensive benchmark for safety alignment of large language models in scientific tasks.arXiv preprint arXiv:2410.03769, 2024

    Tianhao Li, Jingyu Lu, Chuangxin Chu, Tianyu Zeng, Yujia Zheng, Mei Li, Haotian Huang, Bin Wu, Zuoxian Liu, Kai Ma, et al. Scisafeeval: a comprehensive benchmark for safety alignment of large language models in scientific tasks.arXiv preprint arXiv:2410.03769, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Webthinker: Empowering large reasoning models with deep research capability.arXiv preprint arXiv:2504.21776, 2025

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability.arXiv preprint arXiv:2504.21776, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Fact-audit: An adaptive multi-agent framework for dynamic fact-checking evaluation of large language models

    Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See Kiong Ng, and Tat-Seng Chua. Fact-audit: An adaptive multi-agent framework for dynamic fact-checking evaluation of large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 360–381, 2025

    2025

  20. [20]

    Forbidden science: Dual-use ai challenge benchmark and scientific refusal tests.arXiv preprint arXiv:2502.06867, 2025

    David Noever and Forrest McKee. Forbidden science: Dual-use ai challenge benchmark and scientific refusal tests.arXiv preprint arXiv:2502.06867, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    S1-deepresearch: End-to-end deep research models, 2026

    ScienceOne Team. S1-deepresearch: End-to-end deep research models, 2026. URL https: //github.com/ScienceOne-AI/S1-DeepResearch

    2026

  22. [22]

    A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, et al. A strongreject for empty jailbreaks.Advances in Neural Information Processing Systems, 37:125416–125440, 2024

    2024

  23. [23]

    How might large language models aid actors in reaching the competency threshold required to carry out a chemical attack?The Nonproliferation Review, pages 1–22, 2024

    Ryan T Stendall, Freddie JO Martin, and Jonas B Sandbrink. How might large language models aid actors in reaching the competency threshold required to carry out a chemical attack?The Nonproliferation Review, pages 1–22, 2024

    2024

  24. [24]

    Safearena: Evaluating the safety of autonomous web agents.arXiv preprint arXiv:2503.04957, 2025

    Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Sta ´nczak, and Siva Reddy. Safearena: Evaluating the safety of autonomous web agents.arXiv preprint arXiv:2503.04957, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Clawsafety:" safe" llms, unsafe agents.arXiv preprint arXiv:2604.01438, 2026

    Bowen Wei, Yunbei Zhang, Jinhao Pan, Kai Mei, Xiao Wang, Jihun Hamm, Ziwei Zhu, and Yingqiang Ge. Clawsafety:" safe" llms, unsafe agents.arXiv preprint arXiv:2604.01438, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Confusion is the final barrier: Rethinking jailbreak evaluation and investigating the real misuse threat of llms.arXiv preprint arXiv:2508.16347, 2025

    Yu Yan, Sheng Sun, Zhe Wang, Yijun Lin, Zenghao Duan, Min Liu, Jianping Zhang, et al. Confusion is the final barrier: Rethinking jailbreak evaluation and investigating the real misuse threat of llms.arXiv preprint arXiv:2508.16347, 2025

    arXiv 2025

  27. [27]

    Chemsafetybench: Benchmarking llm safety on chemistry domain.arXiv preprint arXiv:2411.16736, 2024

    Haochen Zhao, Xiangru Tang, Ziran Yang, Xiao Han, Xuanzhi Feng, Yueqing Fan, Senhao Cheng, Di Jin, Yilun Zhao, Arman Cohan, et al. Chemsafetybench: Benchmarking llm safety on chemistry domain.arXiv preprint arXiv:2411.16736, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Labsafety bench: Benchmarking llms on safety issues in scientific labs.arXiv preprint arXiv:2410.14182, 2024

    Yujun Zhou, Jingdong Yang, Yue Huang, Kehan Guo, Zoe Emory, Bikram Ghosh, Amita Bedar, Sujay Shekar, Zhenwen Liang, Pin-Yu Chen, et al. Labsafety bench: Benchmarking llms on safety issues in scientific labs.arXiv preprint arXiv:2410.14182, 2024

    arXiv 2024

  29. [29]

    Benchmarking large language models on safety risks in scientific laboratories.Nature Machine Intelligence, pages 1–12, 2026

    Yujun Zhou, Jingdong Yang, Yue Huang, Kehan Guo, Zoe Emory, Bikram Ghosh, Amita Bedar, Sujay Shekar, Zhenwen Liang, Pin-Yu Chen, et al. Benchmarking large language models on safety risks in scientific laboratories.Nature Machine Intelligence, pages 1–12, 2026

    2026

  30. [30]

    Safescientist: Toward risk-aware scientific discoveries by llm agents

    Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, and Jiaxuan You. Safescientist: Toward risk-aware scientific discoveries by llm agents. arXiv preprint arXiv:2505.23559, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Safescientist: Enhancing ai scientist safety for risk-aware scientific discovery

    Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, and Jiaxuan You. Safescientist: Enhancing ai scientist safety for risk-aware scientific discovery. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2289–2317, 2025

    2025

  32. [32]

    Safesci: Safety evaluation of large language models in science domains and beyond.arXiv preprint arXiv:2603.01589, 2026

    Xiangyang Zhu, Yuan Tian, Qi Jia, Kaiwei Zhang, Zicheng Zhang, Chunyi Li, Kaiyuan Ji, Dongrui Liu, Zijian Chen, Lu Sun, et al. Safesci: Safety evaluation of large language models in science domains and beyond.arXiv preprint arXiv:2603.01589, 2026

    Pith/arXiv arXiv 2026

  33. [33]

    Intern-s1-pro: Scientific multimodal foundation model at trillion scale.arXiv preprint arXiv:2603.25040, 2026

    Yicheng Zou, Dongsheng Zhu, Lin Zhu, Tong Zhu, Yunhua Zhou, Peiheng Zhou, Xinyu Zhou, Dongzhan Zhou, Zhiwang Zhou, Yuhao Zhou, et al. Intern-s1-pro: Scientific multimodal foundation model at trillion scale.arXiv preprint arXiv:2603.25040, 2026. 11 A Comparison with Existing Safety Benchmarks Table 8: Comparison of SciHazard with existing safety benchmarks...

    arXiv 2026

  34. [34]

    Contextual Comprehension:Experts thoroughly review the original adversarial query and the model’s response to accurately grasp the attacker’s underlying intent, the scientific context, and the overall logic of the generated output

  35. [35]

    They must explicitly separate scientifically viable and harmful knowledge from scientifically invalid pseudoscience (hallucinations) or benign alternative solutions

    Fact-Checking and Knowledge Extraction:As the most critical and time-consuming phase, experts leverage their deep domain expertise—augmented by authoritative literature, patent databases, and web retrieval—to rigorously vet the information within the response. They must explicitly separate scientifically viable and harmful knowledge from scientifically in...

  36. [36]

    To ensure sufficient depth, this playbook must meet a strict minimum length requirement 12 Table 9: More examples of benign and unsafe queries in our dataset

    Action Guide Synthesis and Bottleneck Identification:Utilizing the harmful knowledge and procedural steps extracted from the model’s response, experts are required to synthesize a comprehensive operational playbook (action guide) targeting the specific harmful query. To ensure sufficient depth, this playbook must meet a strict minimum length requirement 1...

  37. [37]

    Fine-Grained Dimensional Scoring:Based on the verified actionable information extracted in the previous step, experts strictly follow the rubric to independently assign quantitative scores forExecutability(E∈ {1,2,3,4}) andNet-New Risk(N∈ {1,2,3,4})

  38. [38]

    single holistic scoring paradigm

    Holistic Harmfulness Scoring:For benchmarking purposes, experts also provide a tra- ditional, intuition-based overall harmfulness score (Hdirect ∈ {1,2,3,4,5}), referencing the holistic safety guidelines established by OpenAI. This direct score serves as a crucial baseline to demonstrate the limitations and failure modes of the conventional “single holist...

    2026

  39. [39]

    superficial harmfulness

    primarily investigate model performance under jailbreak conditions. For instance, SOSBench applies jailbreak wrappers to 3,000 harmful scientific queries, ensuring that smaller open-source models generate responses before testing them on closed-source models. In contrast, our SciHazard is specifically designed to investigate thenativeharmful scientific re...