Pith. sign in

REVIEW 5 major objections 5 minor 67 references

A frozen, synthetic enterprise environment with gold answers can turn AI posture-agent trust into a measurable quantity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:20 UTC pith:XJK6TZRR

load-bearing objection Well-specified framework for a real gap, but the 'closes the gap' claim outruns what ships; needs artifacts and judge calibration before it is a benchmark rather than a proposal. the 5 major comments →

arxiv 2607.27288 v1 pith:XJK6TZRR submitted 2026-07-29 cs.CR

Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

Gal Engelberg , Michael Arenzon , Leon Goldberg This is my paper
classification cs.CR
keywords autonomous cyber defensesecurity posture managementagentic AIbenchmarkingtext-to-SQLidentity securitysynthetic environmentsLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the field cannot yet answer whether an AI security agent should be trusted, because real enterprise environments are private, cross-vendor, and correlated, and no shared, queryable target exists for evaluating posture-investigation agents end to end. To close this 'environment data gap,' the authors propose Open Security Benchmark (OSB), which freezes synthetic-but-realistically-shaped enterprise environments into immutable relational snapshots with closed-form gold answers, and evaluates agents either by text-to-SQL over the snapshot or by native API calls against a served instance of the same data. Because the ground truth is fixed and scoring is answer-based, different query strategies for the same question score alike, and because both modalities read from the same snapshot, they measure the same work. If OSB works as claimed, practitioners and researchers can compare posture agents on identical, rotating environments, and trusting a posture agent becomes a matter of measurement rather than judgment.

Core claim

The central claim is that a frozen, content-addressed enterprise environment—a synthetic organization whose vendor tables and data patterns mirror real product data models—can serve as a closed-form ground truth for scoring agentic AI on security posture investigation. The authors assert that this environment can be presented through two interrogation surfaces: a relational text-to-SQL snapshot and a natively served API instance, both backed by the same rows so that a finding is defined once and posed either as a query workload or as a sequence of API calls. Around this frozen snapshot sits a five-component framework: a data layer, a task/evaluation-set layer with a ground-truth atom (verdic

What carries the argument

The load-bearing object is the frozen environment snapshot: a read-only relational database of a synthetic enterprise—44 tables spanning eight vendor-style sources at three organizational scales—bundled per task with a 'ground-truth atom' that fixes the verdict, the closed set of matching subjects, the determining configuration facts, and the minimal sufficient tables/joins. This snapshot is what makes scoring plan-independent, because any SQL or API trace that produces the same answer atom is scored alike. Around it, the framework's harness exposes only two tools (get-schema and run-query) and records every action into an evidence archive, while the scoring layer runs deterministic structur

Load-bearing premise

That scores on synthetic enterprises synthesized from real product data models predict an agent's performance on a live tenant's environment with its operational drift and incident telemetry—a transfer the paper does not yet demonstrate.

What would settle it

Have human analysts score the same agents on a live (or realistically drifted) enterprise environment and compare the rank order with the agents' OSB scores; if the rankings diverge substantially, the synthetic-to-real transfer fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Posture-investigation agents can be compared on identical, immutable enterprise environments, so score differences are attributable to the agent rather than the data.
  • The same snapshot and answer criteria make text-to-SQL and native-API results directly comparable, so a model that passes in one modality can be checked in the other.
  • Rotating the environment (regenerating and republishing a new content-addressed revision) keeps answer-level memorization from carrying over to the next revision.
  • The multi-axis scorecard (answer correctness, verdict, reasoning utility, SQL quality, structural table/join recall) lets teams localize whether an agent fails on comprehension, correlation, or planning.
  • Every run's evidence archive is contract-conformant training data, so the benchmark's output can feed a training loop that improves the next agent rather than only scoring it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If the synthetic-to-real transfer holds, OSB could serve as a standardized pre-deployment check for security agents in regulated industries that cannot share live tenant data, giving buyers a defensible basis for procurement decisions.
  • Editorial inference: The native-API modality implies that a vendor could benchmark agents against its own API surface without exposing a consolidated schema, potentially turning the framework into a venue for vendor-specific agent evaluations.
  • Editorial inference: Because scoring is answer-based, any future interrogation surface—graph queries for transitive access paths, CLI command sequences—would plug into the same ground truth, so the framework's coverage could grow without re-anchoring answers.
  • Editorial inference: A testable extension would be to publish the promised expert-calibration agreement statistics; that would let researchers quantify how much judge-panel drift limits score comparability across model configurations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Open Security Benchmark (OSB), a framework for evaluating agentic AI on enterprise security-posture investigation. It defines two investigation modalities — text-to-SQL over a frozen relational snapshot and native-vendor API calls over a served instance of the same environment — and five framework components: data layer, task/evaluation layer, scoring layer, harness, and bring-your-own path. The framework is instantiated with two identity-security packs totaling 127 tasks and synthetic-organization datasets at three scales. The central claim is that OSB 'closes the environment data gap' by surfacing curated, frozen, synthetic enterprise environments with closed-form ground truth, making agent trustworthiness 'a matter of measurement rather than judgment.'

Significance. If fully realized, OSB would be a valuable shared substrate: the two-modality design is thoughtful, the minimal auditable harness with an evidence archive is a genuine strength, and the explicit trust boundary between agent-visible and evaluator-only artifacts is a sound methodological commitment. The paper also productively distinguishes deterministic structural metrics from semantic judged metrics, and the plan to report bootstrap confidence intervals and inter-judge agreement is appropriate. However, the central claims are currently conditional on artifacts that are not shipped, and the primary semantic metrics are rubric-locked LLM judgments with no calibration evidence. The strengths are real but prospective; the paper is a detailed framework proposal rather than a demonstrated benchmark.

major comments (5)
  1. [§5, §1] The load-bearing claim that OSB 'closes the environment data gap' is unsupported as stated. Section 5 says the datasets 'will be released' and 'the initial catalog is in preparation,' and Table 3 describes 'the reference snapshot' rather than a published artifact. No dataset identifier, hash, or download is provided. Contribution (ii), the curated data catalog, is therefore unverifiable; the central evaluation cannot be reproduced or independently checked. Shipping the datasets and code is a prerequisite for the paper's main claim.
  2. [§4.3, Table 1] The primary metrics are not closed-form measurements. AnswerCorrectnessVsGT, AnswerVerdict, ReasoningUtility, and SQLSemanticAppropriateness are all judged by an LLM panel that maps free-form agent prose onto the 'ground-truth atom'; only Tables/Joins recall/precision/F1 are deterministic. The paper states that expert calibration 'is to be validated' and reports no calibration data, no Cohen's kappa, no confidence intervals, and no judge-bias analysis. This directly contradicts the §8 statement that trust becomes 'a matter of measurement rather than judgment.' The scoring method needs a calibration study on expert-labeled outputs before the primary metrics can be treated as measurements.
  3. [§4.3, §6] The evaluation chain is self-referential by the paper's own account. Section 4.3 says the scoring criteria are 'adopted from the cross-vendor ISPM benchmark [30], reusing its Appendix B rubric texts verbatim,' and Section 6 says both packs are 'derived from the two public Sola ISPM benchmarks [29, 30].' Since these are the same authors' prior benchmarks, the framework's validity depends entirely on the quality of those unpublished peer benchmarks. The paper needs to provide independent evidence that the tasks and rubrics are valid for posture investigation, or at least a clear comparison showing what OSB adds beyond repackaging prior work.
  4. [§5, §7] The synthetic-to-real transfer assumption is acknowledged but not supported. Section 5 asserts the environments 'are synthesized from real ones' with production data patterns, but no data distributions, realism metrics, or comparison to production telemetry are given. Section 7 concedes that fabricated organizations 'do not reproduce a live tenant's operational drift and incident telemetry.' Without a validation protocol or any evidence of transfer, the claim that OSB scores predict performance on real deployments remains an unstated assumption rather than a demonstrated property.
  5. [§5.2] The native-API modality is claimed as a full second surface, but the manuscript says only 'a subset of the eight sources is served this way today' without stating which subset, and the structural role of the request trace is described qualitatively rather than defined as a concrete metric. If the native-API modality is part of the central contribution, it needs a definite specification and a working implementation for all claimed sources.
minor comments (5)
  1. [Table 1] The table headers contain apparent typos ('F amily', 'V endor') and should be cleaned before publication.
  2. [§6] The Okta per-application 'mfa_required' simplification is disclosed, which is good, but the paper should explicitly discuss whether this simplification makes the corresponding tasks unrepresentative of real Okta configuration models and whether the benchmark's realism claim depends on such simplifications.
  3. [§4.3] The description of the judge panel is underspecified: 'several independent traces' is not quantified, and the tie-breaking rule 'toward the lower grade' could systematically bias scores downward. A precise protocol is needed for reproducibility.
  4. [§4.2] The three visibility classes (open, gated, private) are described, but only open/private ship today; the gated leaderboard is future work. The paper should label the current status more explicitly in the framework description so readers do not infer a fully implemented gated mechanism.
  5. [§7] The evidence archive caveats are appropriately honest, but 'a stored run can be reviewed again but not yet automatically re-scored' is a significant limitation for the claimed training-loop utility; this should be stated earlier and more prominently.

Circularity Check

1 steps flagged

Core scoring rubrics and both shipped packs are imported from same-authors' prior Sola ISPM benchmarks; the 'measurement rather than judgment' claim therefore rests on a load-bearing self-citation chain, with no calibration data reported.

specific steps
  1. self citation load bearing [§4.3 Evaluation Metrics / Table 1; §6 Use-Case Packs]
    "OSB scores every run using evaluation criteria adopted from the cross-vendor ISPM benchmark [30], reusing its Appendix B rubric texts verbatim rather than re-deriving them. ... The framework ships today with two identity-security packs, derived from the two public Sola ISPM benchmarks [29, 30], over the environment datasets of Section 5."

    The paper's central premise is that freezing an environment and grading answers makes trusting a posture agent 'a matter of measurement rather than judgment' (§8). But the primary metrics (AnswerCorrectnessVsGT, AnswerVerdict, ReasoningUtility, SQLSemanticAppropriateness) are not deterministic: they are rubric-locked LLM-judge scores, and the rubrics are taken verbatim from [30], an arXiv preprint with overlapping authorship (Engelberg, Koutsyi, Goldberg, Baron). The two shipped task packs are likewise derived from [29] and [30], the same authors' Sola ISPM benchmarks, which are also cited as the external grounding for the tasks. No expert calibration data, inter-judge kappa values, or other independent validation is supplied in the present paper; §4.3 says only that 'the framework treats

full rationale

OSB is not a paper that derives a quantitative prediction from inputs, so the classic fitted-input-called-prediction or self-definitional patterns do not apply. The framework components are largely independent: the synthetic-organization environment generator, the frozen-snapshot data layer, the two-tool harness with evidence archives, and the native-API modality are new and described on their own terms. However, the load-bearing evaluation substance is imported from the same authors' earlier Sola ISPM papers: the scoring criteria are 'adopted from the cross-vendor ISPM benchmark [30], reusing its Appendix B rubric texts verbatim,' and both packs are 'derived from the two public Sola ISPM benchmarks [29, 30].' Since the paper's advertised conclusion is that agent trustworthiness becomes 'a matter of measurement rather than judgment,' that conclusion depends on the validity of those self-cited rubrics and gold references. The paper itself flags the missing support: expert calibration is promised as 'the anchor' but no calibration is reported, and the synthetic-to-real transfer limitation is acknowledged in §7. These are not manufactured concerns; they are stated in the text. Still, the central framework has independent content, so the circularity is partial and load-bearing rather than total. Score 4 reflects one significant self-citation chain without a full definitional reduction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The paper's contribution is a framework, so the ledger captures the design choices and unvalidated assumptions it rests on. No parameters are fitted to data, but several are hand-chosen (scales, taxonomy, thresholds), and the benchmark's validity rests on unproven transfer from synthetic to real environments and on author-authored rubrics and gold answers.

free parameters (4)
  • Organizational scale sizes (small/mid/large) = ~75 / ~400 / ~2,000 employees; ~1,600 / ~8,300 / ~36,000 rows
    Hand-chosen to span "more than an order of magnitude" with "roughly a 5x step" (§5.1, Table 3); generalization is reported as coverage across these three chosen scales.
  • Task taxonomy bins = complexity (easy/medium/hard); focus (authentication/authorization/administration)
    Hand-chosen reporting axes that drive every scorecard breakdown (§6); adjacency and thresholds are not derived from data.
  • Judged-metric scale and tie-break = 0/0.5/1; mode with ties toward the lower grade
    Inherited from the authors' own prior benchmark [30] and self-consistency practice [54]; a design choice, not fitted (§4.3).
  • Time thresholds in example tasks = 90 days inactive; 90-day-old access keys
    Domain thresholds chosen by the authors in representative questions (§5.1, §6); no justification is given for the specific value.
axioms (5)
  • domain assumption Text-to-SQL over a frozen relational snapshot faithfully represents posture investigation
    §3 asserts the relational modality "is faithful to the investigative activity"; this is argued, not demonstrated, and is what makes the entire evaluation paradigm valid.
  • domain assumption Synthetic environments mirror production enterprises
    §5: "the environments are synthesized from real ones" and patterns "mirror those seen in production enterprises"; §7 concedes live drift and telemetry are not reproduced — the transfer claim is unvalidated.
  • domain assumption Rubric-locked LLM judges with mode voting produce trustworthy scores
    §4.3 relies on G-Eval [43] and RAGAS [44] but acknowledges judge drift [52, 53]; expert calibration is promised ("the anchor") but no calibration results appear.
  • ad hoc to paper Production Okta MFA can be represented by a per-application mfa_required flag
    §6: the paper states the task uses "a synthetic-schema simplification; production Okta enforces multi-factor requirements through sign-on policies rather than a per-application flag" — an admitted simplification for one of the pack's representative questions.
  • domain assumption Closed-form ground truth (verdict + subjects + configuration facts) is complete and correct
    §4.2: gold atoms are produced by the authors' generator and "cross-checked" internally; no external audit, and the trust boundary keeps the gold reference in a separate visibility class.
invented entities (3)
  • Synthetic-organization environment catalog no independent evidence
    purpose: Close the "environment data gap" by providing shareable, queryable enterprise snapshots with no real identities
    §5: datasets "will be released" and "the initial catalog is in preparation"; no public handle exists, and no external validation that the fabricated patterns match production.
  • Ground-truth atom no independent evidence
    purpose: Plan-independent answer reference (verdict, subject set, configuration facts, minimal-sufficient evidence set) for scoring
    §4.2/§5.1: a new ledger structure for answers; its correctness is asserted via internal generator cross-checks, with no external falsifiable handle.
  • Native-API emulator of the same environment no independent evidence
    purpose: Serve the frozen snapshot over real vendor API surfaces so agents can interrogate without seeing the tables
    §5.2: "a self-contained, runnable world" computed from the same rows; fidelity to real vendor APIs is claimed ("real vendor tooling") but the served subset and fidelity are not demonstrated.

pith-pipeline@v1.3.0-daily-deepseek · 14555 in / 18626 out tokens · 138535 ms · 2026-08-01T10:20:03.916878+00:00 · methodology

0 comments
read the original abstract

Enterprises are moving toward autonomous cyber defense: agentic AI that builds situational awareness of an organization's security state and reasons from it to assessments, decisions, and actions. This rests on a holistic view of the enterprise's security state, the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent can produce an answer but whether it should be trusted to. The field cannot yet answer this question. Real enterprise environments are private, cross-vendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the environment data gap. We present Open Security Benchmark (OSB), a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment - a frozen, holistic view of the security state - and evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor's native API over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 23 linked inside Pith

  1. [1]

    Automated cyber defence: A review

    Sanyam Vyas, John Hannay, Andrew Bolton, and Pete Burnap. Automated cyber defence: A review. arXiv preprint arXiv:2303.04926, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Richer, Junae Kim, and Damian Marriott

    Maxwell Standen, Martin Lucas, David Bowman, Toby J. Richer, Junae Kim, and Damian Marriott. Cyborg: A gym for the development of autonomous cyber agents. InIJCAI-21 1st International Workshop on Adaptive Cyber Defense, 2021. arXiv:2108.09118

    Pith/arXiv arXiv 2021

  3. [3]

    Mica R. Endsley. Toward a theory of situation awareness in dynamic systems.Human Factors, 37(1):32–64, 1995

    1995

  4. [4]

    Cyber situational awareness - a systematic review of the literature

    Ulrik Franke and Joel Brynielsson. Cyber situational awareness - a systematic review of the literature. Computers & Security, 46:18–31, 2014

    2014

  5. [5]

    Defining identity security posture management (ispm)

    RSA Security. Defining identity security posture management (ispm). https://www.rsa.com/ resources/reports/defining-identity-security-posture-management-ispm/, 2025. 10 Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

    2025

  6. [6]

    Cloud controls matrix (ccm) v4

    Cloud Security Alliance. Cloud controls matrix (ccm) v4. https://cloudsecurityalliance.org/ research/cloud-controls-matrix, 2021

    2021

  7. [7]

    Guide for mapping types of information and information systems to security categories

    National Institute of Standards and Technology. Guide for mapping types of information and information systems to security categories. Technical Report NIST Special Publication 800-60 Vol. 1, Rev. 1, National Institute of Standards and Technology, 2008

    2008

  8. [8]

    OWASP top 10:2021 - the ten most critical web application security risks

    OWASP Foundation. OWASP top 10:2021 - the ten most critical web application security risks. https://owasp.org/Top10/, 2021

    2021

  9. [9]

    Cybersecurity supply chain risk management practices for systems and organizations

    National Institute of Standards and Technology. Cybersecurity supply chain risk management practices for systems and organizations. Technical Report NIST Special Publication 800-161 Rev. 1, National Institute of Standards and Technology, 2022

    2022

  10. [10]

    The NIST cybersecurity framework (CSF) 2.0

    National Institute of Standards and Technology. The NIST cybersecurity framework (CSF) 2.0. Technical Report NIST CSWP 29, National Institute of Standards and Technology, 2024

    2024

  11. [11]

    Security and privacy controls for information systems and organizations

    Joint Task Force. Security and privacy controls for information systems and organizations. Technical Report NIST Special Publication 800-53, Revision 5, National Institute of Standards and Technology, 2020

    2020

  12. [12]

    Cis critical security controls, version 8.https://www.cisecurity.org/ controls/v8, 2021

    Center for Internet Security. Cis critical security controls, version 8.https://www.cisecurity.org/ controls/v8, 2021

    2021

  13. [13]

    Iso/iec 27001:2022 - information security, cybersecurity and privacy protection - information security management systems - requirements.https://www.iso

    International Organization for Standardization. Iso/iec 27001:2022 - information security, cybersecurity and privacy protection - information security management systems - requirements.https://www.iso. org/standard/27001, 2022

    2022

  14. [14]

    Scout suite: Multi-cloud security auditing tool

    NCC Group. Scout suite: Multi-cloud security auditing tool. https://github.com/nccgroup/ ScoutSuite, 2024

    2024

  15. [15]

    Prowler: Open source cloud security tool for aws, azure, gcp and kubernetes

    Prowler. Prowler: Open source cloud security tool for aws, azure, gcp and kubernetes. https: //github.com/prowler-cloud/prowler, 2024

    2024

  16. [16]

    Scubagoggles: Google workspace secure configuration baseline assessment tool.https://github.com/cisagov/ScubaGoggles, 2024

    Cybersecurity and Infrastructure Security Agency (CISA). Scubagoggles: Google workspace secure configuration baseline assessment tool.https://github.com/cisagov/ScubaGoggles, 2024

    2024

  17. [17]

    2025 roi of ai in security report

    Google Cloud. 2025 roi of ai in security report. https://cloud.google.com/resources/content/ roi-of-ai-security, 2025

    2025

  18. [18]

    Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, and Anand Mudgerikar

    Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K. Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, and Anand Mudgerikar. Excytin-bench: Evaluating llm agents on cyber threat investigation.arXiv preprint arXiv:2507.14201, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Orgaccess: A benchmark for role based access control in organization scale llms.arXiv preprint arXiv:2505.19165, 2025

    Debdeep Sanyal, Umakanta Maharana, Yash Sinha, Hong Ming Tan, Shirish Karande, Mohan Kankan- halli, and Murari Mandal. Orgaccess: A benchmark for role based access control in organization scale llms.arXiv preprint arXiv:2505.19165, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W

    Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Mike Yang, Teddy Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Polycarpos Yiorkadjis, Kenny Osele, Gautham ...

    2025

  22. [22]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security.arXiv preprint arXiv:2406.05590, 2024

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security.arXiv preprint arXiv:2406.05590, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Semantic parsing on Freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1533–1544, 2013. 11 Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

    2013

  24. [24]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (...

    2018

  25. [25]

    Chang, Fei Huang, Reynold Cheng, and Yongbin Li

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.-C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. InAdvances in Neural Inform...

    2023

  26. [26]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows.arXiv preprint arXiv:2411.07763, 2024

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows.arXiv preprint arXiv:2411.07763, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  28. [28]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  29. [29]

    Sola-visibility-ispm: Benchmarking agentic ai for identity security posture management visibility.arXiv preprint arXiv:2601.07880, 2026

    Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, Reuven Elezra, Idan Pinto, Tal Moalem, Shmuel Cohen, and Yoni Weintrob. Sola-visibility-ispm: Benchmarking agentic ai for identity security posture management visibility.arXiv preprint arXiv:2601.07880, 2026

    arXiv 2026

  30. [30]

    Cross-vendor sola ispm benchmark: Evaluating agentic ai for federated identity security reasoning.arXiv preprint arXiv:2606.02674, 2026

    Eden Yavin, Gal Engelberg, Konstantin Koutsyi, Leon Goldberg, and Gal Baron. Cross-vendor sola ispm benchmark: Evaluating agentic ai for federated identity security reasoning.arXiv preprint arXiv:2606.02674, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    Guide for conducting risk assessments

    National Institute of Standards and Technology. Guide for conducting risk assessments. Technical Report NIST Special Publication 800-30 Rev. 1, National Institute of Standards and Technology, 2012

    2012

  32. [32]

    Guide to enterprise patch management planning: Preventive maintenance for technology

    National Institute of Standards and Technology. Guide to enterprise patch management planning: Preventive maintenance for technology. Technical Report NIST Special Publication 800-40 Rev. 4, National Institute of Standards and Technology, 2022

    2022

  33. [33]

    Information security continuous monitoring (iscm) for federal information systems and organizations

    National Institute of Standards and Technology. Information security continuous monitoring (iscm) for federal information systems and organizations. Technical Report NIST Special Publication 800-137, National Institute of Standards and Technology, 2011

    2011

  34. [34]

    Cybersoceval: Benchmarking llms capabilities for malware analysis and threat intelligence reasoning.arXiv preprint arXiv:2509.20166, 2025

    Lauren Deason, Adam Bali, Ciprian Bejean, Diana Bolocan, James Crnkovich, Ioana Croitoru, Krishna Durai, Chase Midler, Calin Miron, David Molnar, Brad Moon, Bruno Ostarcevic, Alberto Peltea, Matt Rosenberg, Catalin Sandu, Arthur Saputkin, Sagar Shah, Daniel Stan, Ernest Szocs, Shengye Wan, Spencer Whitman, Sven Krasser, and Joshua Saxe. Cybersoceval: Benc...

    arXiv 2025

  35. [35]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599, 2024

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence.arXiv preprint arXiv:2406.07599, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446, 2024

    Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, Changyu Ren, and Zhoujun Li. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence.arXiv preprint arXiv:2405.03446, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks

    Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. Llms cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In2024 IEEE Symposium on Security and Privacy (S&P), 2024

    2024

  38. [38]

    Zefang Liu, Jialei Shi, and John F. Buford. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. InAAAI-24 Workshop on Artificial Intelligence for Cyber Security (AICS), 2024

    2024

  39. [39]

    Cyber- metric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge.arXiv preprint arXiv:2402.07688, 2024

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. Cyber- metric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge.arXiv preprint arXiv:2402.07688, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    vulnerable by design

    Rhino Security Labs. Cloudgoat: Rhino security labs’ “vulnerable by design” aws deployment tool. https://github.com/RhinoSecurityLabs/cloudgoat, 2024. 12 Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

    2024

  41. [41]

    Steampipe: Zero-etl sql for cloud, saas, and infrastructure apis.https://steampipe.io, 2024

    Turbot. Steampipe: Zero-etl sql for cloud, saas, and infrastructure apis.https://steampipe.io, 2024

    2024

  42. [42]

    Cloudquery: Open-source cloud asset inventory.https://www.cloudquery.io, 2024

    CloudQuery. Cloudquery: Open-source cloud asset inventory.https://www.cloudquery.io, 2024

    2024

  43. [43]

    G-eval: Nlg evaluation using gpt-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  44. [44]

    Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis.arXiv preprint arXiv:2305.15334, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Api-bank: A comprehensive benchmark for tool-augmented llms

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

    2023

  47. [47]

    Open cybersecurity schema framework (ocsf).https://ocsf.io, 2024

    OCSF Project. Open cybersecurity schema framework (ocsf).https://ocsf.io, 2024

    2024

  48. [48]

    Amazon security lake.https://aws.amazon.com/security-lake/, 2024

    Amazon Web Services. Amazon security lake.https://aws.amazon.com/security-lake/, 2024

    2024

  49. [49]

    Okta security best practices.https://help.okta.com/, 2024

    Okta. Okta security best practices.https://help.okta.com/, 2024

    2024

  50. [50]

    Security checklist for medium and large businesses (100+ users).https://support.google

    Google. Security checklist for medium and large businesses (100+ users).https://support.google. com/a/answer/7587183, 2024

    arXiv 2024

  51. [51]

    Bloodhound: Active directory and azure attack-path management.https://github.com/ SpecterOps/BloodHound, 2024

    SpecterOps. Bloodhound: Active directory and azure attack-path management.https://github.com/ SpecterOps/BloodHound, 2024

    2024

  52. [52]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  53. [53]

    Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  55. [55]

    A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960

    Jacob Cohen. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1):37–46, 1960

    1960

  56. [56]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.Nature, 645:633–638, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.Nature, 645:633–638, 2025

    2025

  57. [57]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi...

    Pith/arXiv arXiv 2024

  58. [58]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    Pith/arXiv arXiv 2022

  59. [59]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. InAdvances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1706.03741

    Pith/arXiv arXiv 2017

  60. [60]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Pith/arXiv arXiv 2005

  61. [61]

    Gordon, and J

    Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627–635, 2011. arXiv:1011.0686

    Pith/arXiv arXiv 2011

  62. [62]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. arXiv:2203.14465. 13 Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense

    Pith/arXiv arXiv 2022

  63. [63]

    Direct reasoning optimization: Llms can reward and refine their own reasoning for open-ended tasks.arXiv preprint arXiv:2506.13351v1, 2025

    Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, Leonardo Nunes, Emre Kiciman, Songwu Lu, and Ranveer Chandra. Direct reasoning optimization: Llms can reward and refine their own reasoning for open-ended tasks.arXiv preprint arXiv:2506.13351v1, 2025

    Pith/arXiv arXiv 2025

  64. [64]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255, 2009

    2009

  65. [65]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. InAdvances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1905.00537

    Pith/arXiv arXiv 2019

  66. [66]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2310.06770

    Pith/arXiv arXiv 2024

  67. [67]

    Sheridan, and Christopher D

    Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. A model for types and levels of human interaction with automation.IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 30(3):286–297, 2000. 14

    2000