REVIEW 5 major objections 6 minor 1 cited by
Can Large Language Models Design Biological Weapons? Evaluating Moremi Bio
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A guardrail-free LLM biodesign agent generated 1,020 proteins and 5,000 small molecules that computational toxicity predictors flag as toxic, including one claimed to exceed Sarin, challenging claims that LLMs cannot design bioweapons.
desk verdict A red-team demo with an internally contradictory evidence base: the paper claims LLM-designed bioweapons but reports no sequences, no valid similarity data, and no toxicology benchmarks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the pairing of Moremi Bio Agent, an agentic LLM for high-throughput antibody and small-molecule design, with an automated in silico toxicity pipeline. ToxinPred2 and CSM-Toxin score each generated protein across machine-learning, motif (MERCI), and BLAST channel outputs, while ADMET-AI predicts LD50 and toxicity-related properties for the small molecules; databases of known toxins anchor the similarity and clustering comparisons. This pipeline is what converts raw generated sequences into the paper's claim that 1,020 proteins and 5,000 small molecules are toxic, so the entire argument rides on those predictor scores.
What would settle it
Synthesize and express the top-scoring proteins (e.g., MolSeq10) and a few of the 5,000 small molecules, then measure actual cytotoxicity and animal LD50 alongside ricin, diphtheria toxin, and Sarin under identical protocols; if the 'higher than Sarin' protein shows no meaningful toxicity at doses where known toxins kill, the central claim fails. A cheaper discriminating check: run the same predictors on scrambled versions of the generated sequences and on a panel of known-benign proteins; if benign proteins score just as high, the scores are not evidence of designed toxicity.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that an unguarded LLM biodesign agent can be prompted to generate thousands of novel sequences whose predicted toxicity profiles rival known biological weapons agents. The evidence presented is computational: a t-SNE analysis places most generated proteins and small molecules inside or near clusters of known toxins, the top ten designed proteins match ricin, diphtheria toxin, and disintegrin triflavin at high identity scores and zero E-values, and the claimed 'wake-up call' is a novel protein the authors say has higher toxicity than Sarin. None of the sequences was expressed, synthesized, or tested in living systems; the toxicity verdicts come entirely from predictor scores. The paper's conclusion is that these results challenge the position that LLMs are incapable of designing bioweapons.
Load-bearing premise
The whole case rests on treating in silico toxicity predictor scores as if they meant real biological toxicity: no generated protein or molecule was synthesized, expressed, or tested in cells or animals, and no actual LD50 was measured, so if the predictors are noisy or biased the gap between 'predicted toxic' and 'could serve as a bioweapon' breaks the central claim.
Editorial extensions
If this is right
- If the central claim is right, a guardrail-free LLM biodesign agent can act as a high-throughput generator of candidate toxic sequences, so biosecurity controls must operate at generation time, not just at screening time.
- Toxicity thresholds built into the tool's output filter (the paper's proposed mitigation) would need to block a large share of unguarded outputs, since most generated proteins score near the top of the predictors.
- Assessments that dismiss LLM bioweapon capabilities would need to be revised to distinguish 'the model refuses' from 'the model cannot'.
- The same pipeline, run with guardrails on, could serve as a benchmark for measuring how much safety filtering reduces the fraction of toxic outputs.
- High sequence similarity between generated proteins and ricin or diphtheria toxin implies screening against known toxin databases is a necessary but not sufficient layer of defense.
Reading between the lines
- Beyond the paper, the strongest test would be wet-lab validation: synthesizing and expressing a handful of the top-scoring proteins and measuring actual cytotoxicity and LD50 against ricin and diphtheria toxin; the authors report no such experiment, so the 'designed a bioweapon' phrasing is best read as shorthand for 'designed sequences that in silico screens flag as toxic.'
- The Sarin comparison mixes a protein's predicted toxicity score with a small-molecule nerve agent's lethal dose; a fairer benchmark would compare a generated protein with known protein toxins under the same assay, and if the score is simply saturated near the maximum, the comparison may not carry the meaning the paper assigns it.
- A direct check of predictor specificity would run ToxinPred2 and CSM-Toxin on a panel of well-characterized benign proteins and on scrambled controls; if benign proteins score as high as the generated ones, the 'all 1,020 are toxic' result would reflect a property of the predictor rather than of the designs.
- The proposed mitigations assume the main risk is a user turning off guardrails; the paper leaves open whether adversarially reworded prompts could elicit toxic designs even with safeguards on, which would be a natural extension of this work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an evaluation of Moremi Bio Agent, an LLM-based biodesign system, prompted without safety guardrails to generate toxic proteins and small molecules. The authors state that the system generated 1,020 novel toxic proteins and 5,000 toxic small molecules, and that computational toxicity assessment indicates all are likely toxic, with several proteins similar to ricin, diphtheria toxin, and snake venom disintegrins. They further claim in the Abstract and in Section 5.1 that the work challenges the view that LLMs are incapable of designing bioweapons, and that a novel protein with higher toxicity than Sarin was designed. The paper describes a pipeline using ToxinPred2, CSM-Toxin, and ADMET-AI, presents a comparison table (Table 1) for ten generated proteins against known toxins and random ToxinPred sequences, includes t-SNE visualizations, and proposes mitigation strategies and safeguards.
Significance. If the central claim were supported, the paper would be highly significant for biosecurity policy, as it would provide evidence that current LLM-enabled biodesign tools can generate plausible bioweapon candidates with minimal expertise. The authors also contribute a potentially reusable in silico toxicity-assessment pipeline and a large generated dataset, although neither the pipeline details nor the dataset are made publicly available in the manuscript. However, the significance is undermined by the fact that the reported evidence is internally contradictory and does not support the headline claims: the only differentiating metric in Table 1 is a single machine-learning classifier score, while the motif and similarity scores are constant across all entries, and the comparison to Sarin is not substantiated by any presented data. Thus the paper currently serves more as a cautionary illustration of overclaiming from computational predictions than as a validated demonstration of LLM bioweapon design capability.
major comments (5)
- [§5.1 and Table 1] The claim in Section 5.1 that "we successfully designed a novel protein with higher toxicity than Sarin" is not supported by any data in the manuscript. Table 1 compares only proteins to ToxinPred2 scores and never includes Sarin, a small-molecule nerve agent. No CSM-Toxin or ADMET-AI results are reported for the generated proteins, and the toxicity predictors used are not established as comparable across proteins and small molecules. This is a load-bearing claim for the Abstract and Discussion, and it is unsubstantiated as written.
- [§5 and Table 1] The text in Section 5 states that BLAST results showed "high identity score (0.95–0.98) and E-value (0.00)" against ricin and diphtheria toxin, but Table 1 lists a BLAST score of 0.5 for every entry, including the known toxins Ricin and Diphtheria toxin. These two representations of sequence similarity are directly contradictory. The BLAST score of 0.5 appears to be a constant placeholder rather than an informative similarity measure, and the claimed identity scores are absent from the reported results.
- [§4.2, Table 1] The evidence for the toxicity of the generated proteins (MolSeq1–MolSeq10) reduces to the ToxinPred2 ML score, which ranges from 0.93 to 1.00. The MERCI score is 0.0 for every sequence in the table, including the known toxins, and the BLAST score is constant at 0.5. A single classifier output, with no supporting motif hits, no sequence-similarity evidence, no CSM-Toxin scores, no expression or animal data, and no LD50 measurements, is insufficient to conclude that these proteins are toxic or that they constitute bioweapon design. The paper itself acknowledges in Section 2.3 that the substances are "theoretical," yet the Abstract and Section 5.1 make categorical claims of successful bioweapon design.
- [§3.2 and §4.2] The methodology does not provide enough detail to reproduce the toxicity assessment pipeline. Section 3.2 describes a comparison against "a curated database of known toxic substances" but does not state the database version, alignment thresholds, or classifier parameters. Section 4.2 reports only ten MolSeq proteins, not the full set of 1,020, and gives no sequences, no SMILES strings for the small molecules, and no accession numbers for the comparator toxins. Without these, the reader cannot verify the central quantitative results or assess whether the claims generalize beyond the ten displayed examples.
- [§5 (Pearson citation)] The sentence reporting "high identity score (0.95–0.98) and E-value (0.00)" cites "(Pearson, 2013),", but no Pearson 2013 reference appears in the reference list. If a BLAST methodology or database is meant, it needs a proper citation, and the identity and E-value numbers need to be tied to the actual sequences and BLAST parameters used.
minor comments (6)
- [§5.1] The phrase "posing a significant threat to research and development (R& D)" contains an odd spacing in "R& D"; it should read "R&D."
- [§5] The text refers to "the 1,000 generated" proteins while the Abstract and Section 3.1 state 1,020 generated proteins; this numeric inconsistency should be corrected.
- [§2.3] The parenthetical comment "while theoretical, could be toxic if manufactured" is a useful caveat, but it is contradicted by the stronger claims in the Abstract and Section 5.1; the paper should either temper the headline claims or provide direct experimental evidence.
- [§4.1] The t-SNE figures (Figures 4 and 5) are described but not interpreted with quantitative measures of cluster separation or statistical significance, making it difficult to assess whether the visual "clustering" is meaningful beyond random grouping.
- [Abstract] The Abstract states "all the proteins scored high in toxicity" but Table 1 shows that the comparator proteins Ricin and Diphtheria toxin score lower than several MolSeq sequences; the phrase "all" is ambiguous and should be qualified as "all generated proteins in our sample."
- [General] The paper would benefit from an explicit statement about whether generated sequences and small-molecule structures are withheld for biosecurity reasons; if so, that rationale should appear in the Methodology or a data-availability section.
Circularity Check
No significant circularity: toxicity calls come from external predictors and are not fitted to the claimed outputs; self-citations are background, not load-bearing.
full rationale
The paper's derivation chain is: prompt Moremi Bio Agent to design toxic sequences, score them with third-party predictors (ToxinPred2, CSM-Toxin, ADMET-AI), and compare scores with known toxins. No step in this chain reduces to a fitted parameter or to a definition. The ML, MERCI, BLAST, and hybrid scores in Table 1 are outputs of external tools applied uniformly to generated and reference sequences; they are not constructed from the paper's conclusions. The self-citations to prior Moremi work (references [3] and [11]) describe the agent's background capabilities, but they are not used to prove the toxicity of the newly generated sequences, so they are not load-bearing. The headline claim that a 'novel protein with higher toxicity than Sarin' was designed is not supported by the reported data, since no Sarin comparison is shown and ToxinPred2 is a protein-level predictor, but that is an evidentiary gap rather than circularity. Similarly, the inconsistency between the claimed BLAST identity score of 0.95-0.98 and the constant BLAST score of 0.5 in Table 1 is a correctness problem, not a self-referential reduction. Because no prediction is equivalent to its input by construction and no load-bearing argument reduces to a self-citation, no circular step can be exhibited.
Assumptions & free parameters
assumptions (3)
- domain assumption In silico toxicity predictions from ToxinPred2, CSM-Toxin, and ADMET-AI are reliable proxies for actual biological toxicity and bioweapon potential.
- domain assumption The 1,020 protein and 5,000 small molecule sequences are novel designs, not trivial retrievals from training data.
- domain assumption Moremi Bio Agent, when prompted without guardrails, is representative of LLM-enabled biodesign capabilities.
Cite this review
Pith. "Pith review of Can Large Language Models Design Biological Weapons? Evaluating Moremi Bio." pith.science (2026). https://pith.science/paper/2BJ3AWWS
@misc{pith2026250517154,
author = {Pith},
title = {Pith review of: Can Large Language Models Design Biological Weapons? Evaluating Moremi Bio},
year = {2026},
howpublished = {\url{https://pith.science/paper/2BJ3AWWS}},
note = {Machine review of arXiv:2505.17154}
}
read the original abstract
Advances in AI, particularly LLMs, have dramatically shortened drug discovery cycles by up to 40% and improved molecular target identification. However, these innovations also raise dual-use concerns by enabling the design of toxic compounds. Prompting Moremi Bio Agent without the safety guardrails to specifically design novel toxic substances, our study generated 1020 novel toxic proteins and 5,000 toxic small molecules. In-depth computational toxicity assessments revealed that all the proteins scored high in toxicity, with several closely matching known toxins such as ricin, diphtheria toxin, and disintegrin-based snake venom proteins. Some of these novel agents showed similarities with other several known toxic agents including disintegrin eristostatin, metalloproteinase, disintegrin triflavin, snake venom metalloproteinase, corynebacterium ulcerans toxin. Through quantitative risk assessments and scenario analyses, we identify dual-use capabilities in current LLM-enabled biodesign pipelines and propose multi-layered mitigation strategies. The findings from this toxicity assessment challenge claims that large language models (LLMs) are incapable of designing bioweapons. This reinforces concerns about the potential misuse of LLMs in biodesign, posing a significant threat to research and development (R&D). The accessibility of such technology to individuals with limited technical expertise raises serious biosecurity risks. Our findings underscore the critical need for robust governance and technical safeguards to balance rapid biotechnological innovation with biosecurity imperatives.
Figures
Forward citations
Cited by 1 Pith paper
-
Harmonizing AI Safety Thresholds
The authors propose harmonized AI capability floors: non-zero full-chain TLO cyber completion triggers safeguards, and AI progress at 5× trend for 3 months triggers safeguards, with biorisk left as a diagnostic.
Reference graph
Works this paper leans on
-
[1]
Mohamed B. Abou-Donia, Briana Siracuse, Natasha Gupta, Ashly Sobel Sokol, Sarin (gb, o-isopropyl methylphosphonofluoridate) neurotoxicity: critical review,Critical Reviews in Toxicology,46:845 – 875, 2016, available from:https://api.semanticscholar.org/CorpusID:46754779
work page 2016
-
[2]
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, Sebastian W Bodenstein, David A Evans, Chia-Chun Hung, Michael O’Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvil˙ e ˇZemgulyt˙ e, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Al...
work page 2024
-
[3]
Darlington Ahiale Akogo, Jeremiah Ayensu, Nana Sam, Gertrude Hattoh, Prince Nyarko, Solomon Eshun, Mohammed Alhasan, Henrietta E. Mensah- Brown, Peter Quashie, Moremi bio agent: Application of a foundation model and end-to-end automation in the design and validation of monoclonal antibodies targeting plasmodium falciparum invasion complex,bioRxiv, 2025, a...
work page 2025
-
[4]
semanticscholar.org/CorpusID:942185
Yoav Gal, Ohad Mazor, Reut Falach, Anita Sapoznikov, Chanoch Kronman, Tamar Sabo, Treatments for pulmonary ricin intoxication: Current aspects and future prospects,Toxins,9, 2017, available from:https://api. semanticscholar.org/CorpusID:942185
work page 2017
-
[5]
A. Ghafarollahi, M. J. Buehler, Protagents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning, 2024, available from:https://arxiv.org/abs/2402.04268
arXiv 2024
-
[6]
Alexei Grinbaum, Laurynas Adomaitis, Dual use concerns of generative ai and large language models,ArXiv,abs/2305.07882, 2023, available from: https://api.semanticscholar.org/CorpusID:258686553
work page Pith review arXiv 2023
-
[7]
Nicholas Guise, David Pattie, Kenneth B. Yeh, Kemper Talley, Rachel Fennell Fezzie, 2023 cyberbiosecurity summit underscores 12 challenges associated with cybersecurity and the rapidly growing bioe- conomy,Global Security: Health, Science and Policy,9(1):2401164, 2024,https://doi.org/10.1080/23779497.2024.2401164, available from: https://doi.org/10.1080/2...
-
[8]
semanticscholar.org/CorpusID:17181056
Emilia Lim, Allison Pon, Yannick Djoumbou, Craig Knox, Savita Shrivastava, Anchi Guo, Vanessa Neveu, David Scott Wishart, T3db: a comprehensively annotated database of common toxins and their targets,Nucleic Acids Research,38:D781 – D786, 2009, available from:https://api. semanticscholar.org/CorpusID:17181056
work page 2009
Show all 32 references
-
[9]
J´ essica K A Macˆ edo, Jay William Fox, Mariana S. Castro, Disintegrins from snake venoms and their applications in cancer research and therapy,Current Protein & Peptide Science,16:532 – 548, 2015, available from:https:// api.semanticscholar.org/CorpusID:18509074
2015
-
[10]
Anna Madejska, Miros law M Michalski, Jacek Osek, Marine tetrodotoxin as a risk for human health,Journal of Veterinary Research,63:579 – 586, 2019, available from:https://api.semanticscholar.org/CorpusID: 208602951
2019
-
[11]
digitaloceanspaces.com/moremi-article.pdf
MinohealthAILabs, Moremi ai: Towards artificial general intelligence for health (agi4health) and artificial general intelligence for biology (agi4bio), 2025, available from:https://minohealth-storage.fra1.cdn. digitaloceanspaces.com/moremi-article.pdf
2025
-
[12]
Vladimir Morozov, Carlos H. M. Rodrigues, David Benjamin Ascher, Csm-toxin: A web-server for predicting protein toxicity,Pharmaceutics, 15, 2023, available from:https://api.semanticscholar.org/CorpusID: 256436006
2023
-
[13]
Morris, Eric E
Vincent L. Morris, Eric E. Schmidt, Sahadia Koop, Ian C Macdonald, Marsha Grattan, Rama Khokha, Mary Ann McLane, Stefan Niewiarowski, Ann F. Chambers, Alan C. Groom, Effects of the disintegrin eristostatin on individual steps of hematogenous metastasis.,Experimental cell resea...
1995
-
[14]
National Institute of Standards and Technology (NIST), U.S. AI Safety Institute, Managing misuse risk for dual-use foundation models, Initial Public Draft AI 800-1 ipd, National Institute of Standards and Technology (NIST), July 2024, available from:https://doi.org/10.6028/NIS...
2024 doi
-
[15]
Albano Pinto, Inˆ es P. E. Mac´ ario, S´ ergio M Marques, Joana Louren¸ co, Inˆ es Domingues, Maria Jo˜ ao Botelho, Jana Asselman, Patr ´ ıcia Pereira, Joana Lu ´ ısa Pereira, A short-term exposure to saxitoxin triggers a multitude 13 of deleterious effects in daphnia magna at...
2024
-
[16]
Letizia Polito, Massimo Bortolotti, Maria Giulia Battelli, Giulia Calafato, Andrea Bolognesi, Ricin: An ancient story for a timeless plant toxin,Toxins, 11, 2019, available from:https://api.semanticscholar.org/CorpusID: 182947879
2019
-
[17]
Ernestine Powell, Darlington Ahiale Akogo, Lucas Potter, Xavier-Lewis Palmer, Co-leadership and cross-pollination of university and diy bio spaces: An exploration in consideration of biocybersecurity,Proceedings of the Future Technologies Conference (FTC) 2021, Volume 3, 2021,...
2021
-
[18]
Marta Prygiel, Ewa Mosiej, Maciej Polak, Katarzyna Krysztopa-Grzybowska, Karol Wdowiak, Kamila Formi´ nska, Aleksandra Anna Zasada, Challenges of diphtheria toxin detection,Toxins,16, 2024, available from:https: //api.semanticscholar.org/CorpusID:270082468
2024
-
[19]
Jonas B. Sandbrink, Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools,ArXiv, abs/2306.13952, 2023, available from:https://api.semanticscholar. org/CorpusID:259252204
2023 arXiv
-
[20]
Neelam Sharma, Naorem Leimarembi Devi, Shipra Jain, Gajendra P. S. Raghava, Toxinpred2: an improved method for predicting toxicity of proteins,Briefings in bioinformatics, 2022, available from:https://api. semanticscholar.org/CorpusID:248947558
2022
-
[21]
Soice, Rafael Henrique Santos Rocha, Kimberlee Cordova, Michael A
Emily H. Soice, Rafael Henrique Santos Rocha, Kimberlee Cordova, Michael A. Specter, Kevin M. Esvelt, Can large language models democratize access to dual-use biotechnology?,ArXiv,abs/2306.03809, 2023, available from:https://api.semanticscholar.org/CorpusID:259089339
2023 arXiv
-
[22]
Su´ arez-Isla, Saxitoxin and other paralytic toxins: Toxicological profile, inMarine and Freshwater Toxins, 2015, available from:https: //api.semanticscholar.org/CorpusID:83490085
Benjamin A. Su´ arez-Isla, Saxitoxin and other paralytic toxins: Toxicological profile, inMarine and Freshwater Toxins, 2015, available from:https: //api.semanticscholar.org/CorpusID:83490085
2015
-
[23]
Wu, Rabindra V
Kyle Swanson, Parker Walther, Jeremy Leitz, Souhrid Mukherjee, Joseph C. Wu, Rabindra V. Shivnaraine, James Zou, Admet-ai: a machine learning admet platform for evaluation of large-scale chemical libraries, Bioinformatics,40, 2024, available from:https://api.semanticscholar. o...
2024
-
[24]
Marcelo Der Torossian Torres, Yimeng Zeng, Fangping Wan, Natalie Maus, Jacob R. Gardner, C´ esar de la Fuente-Nunez, A generative artificial 14 intelligence approach for antibiotic optimization,bioRxiv, 2024, available from:https://api.semanticscholar.org/CorpusID:274341833
2024
-
[25]
Truelove, Lindsay T
Shaun A. Truelove, Lindsay T. Keegan, William J. Moss, Lelia H. Chaisson, Emilie Macher, Andrew S. Azman, Justin Lessler, Clinical and epidemiological aspects of diphtheria:a systematic review and pooled analysis.,Clinical infectious diseases : an official publication of the I...
2019
-
[26]
Fabio Urbina, Filippa Lentzos, C´ edric Invernizzi, Sean Ekins, Dual use of artificial-intelligence-powered drug discovery,Nature Machine Intelligence, 4:189 – 191, 2022, available from:https://api.semanticscholar.org/ CorpusID:247302391
2022
-
[27]
Shanow Uthman, Shihui Liu, Flaviano Giorgini, Michael J. R. Stark, M. Costanzo, Raffael Schaffrath, Diphtheria disease and genes involved in formation of diphthamide, key effector of the diphtheria toxin, inInsight and Control of Infectious Disease in Global Scenario, 2012, av...
2012
-
[28]
Grant, Yifeng Liu, Seyed Ali Goldansaz, Stephen M
David Scott Wishart, David Arndt, Allison Pon, Tanvir Sajed, Anchi Guo, Yannick Djoumbou, Craig Knox, Michael Wilson, Yongjie Liang, Jason R. Grant, Yifeng Liu, Seyed Ali Goldansaz, Stephen M. Rappaport, T3db: the toxic exposome database,Nucleic Acids Research,43:D928 – D934, ...
2014
-
[29]
World Health Organization, Cyanobacterial toxins: saxitoxins, Technical Report WHO/HEP/ECH/WSH/2020.8, World Health Organization, Geneva,
2020
-
[30]
Lianlian Wu, Bowei Yan, Ju cai Han, Ruijiang Li, Jian Xiao, Song He, Xiaochen Bo, Toxric: a comprehensive database of toxicological data and benchmarks,Nucleic Acids Research,51:D1432 – D1445, 2022, available from:https://api.semanticscholar.org/CorpusID:253671256
2022
-
[31]
Zhuang Xiang, Keyan Ding, Tianwen Lyu, Yinuo Jiang, Xiaotong Li, Zhuoyi Xiang, Zeyuan Wang, Ming Qin, Kehua Feng, Jike Wang, Qiang Zhang, Huajun Chen, Instructbiomol: Advancing biomolecule understanding and design following human instructions,ArXiv,abs/2410.07919, 2024, availa...
-
[2020]
WHOLIS ID: who-338069
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.