pith. sign in

arxiv: 2604.09998 · v2 · pith:RBXCPYLCnew · submitted 2026-04-11 · 💻 cs.CR · cs.AI

Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit

Souradip Nath , Chih-Yi Huang , Aditi Ganapathi , Kashyap Thimmaraju , Jaron Mink , Gail-Joon Ahn This is my paper

Pith reviewed 2026-05-10 16:40 UTC · model grok-4.3

classification 💻 cs.CR cs.AI
keywords large language modelscybersecurity operationsReddit analysisLLM adoptionsecurity practitionersSOC workflowsperceptions of AI tools
0
0 comments X

The pith

Security practitioners use LLMs for low-risk productivity tasks in cybersecurity but grant them limited autonomy due to reliability and security concerns.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper examines how cybersecurity professionals discuss and adopt large language models based on posts in Reddit forums. It identifies patterns where LLMs are used independently for routine, low-risk activities that boost productivity, while there is interest in more secure enterprise platforms for sensitive work. Practitioners note real improvements in efficiency and effectiveness from these tools. However, doubts about reliability, the need for verification, and potential security risks prevent wider use with greater independence. Understanding these views can help developers create tools that better fit operational needs and reduce risks in security environments.

Core claim

Analysis of 892 posts from cybersecurity Reddit forums between December 2022 and September 2025 shows that security practitioners use LLMs mainly on their own for low-risk, productivity-focused tasks and express interest in enterprise-grade, security-oriented LLM platforms. They describe meaningful gains in workflow efficiency and effectiveness but highlight ongoing problems with reliability, the extra work of verifying outputs, and security risks, which together restrict how much freedom they allow the tools. The study also offers recommendations for creating and implementing these tools in ways that protect organizations and analysts.

What carries the argument

Mixed-methods study combining qualitative coding and statistical analysis of 892 Reddit posts to map stated LLM tools, use cases, perceived pros and cons, and adoption decisions.

If this is right

  • LLMs are adopted independently for low-risk tasks to improve productivity.
  • Interest exists in specialized enterprise LLM platforms focused on security.
  • Reported gains in efficiency and effectiveness from LLM-assisted work.
  • Concerns over reliability, verification needs, and security risks limit tool autonomy.
  • Recommendations exist for safer development and adoption of LLM tools in security contexts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Developers could prioritize building better verification features to increase trust in LLM outputs for security work.
  • Similar adoption patterns might emerge in other high-stakes technical domains beyond cybersecurity.
  • Actual SOC teams could be surveyed directly to validate whether Reddit discussions match in-person practices.
  • Over time, addressing the identified risks might allow greater integration of LLMs into core security operations.

Load-bearing premise

That the posts collected from three public Reddit forums between December 2022 and September 2025 reflect the full range of security practitioner behaviors and views without major self-selection or platform bias.

What would settle it

Finding through interviews or surveys with a broad sample of working SOC analysts that they grant LLMs more autonomy or use them differently than described in the Reddit data.

Figures

Figures reproduced from arXiv: 2604.09998 by Aditi Ganapathi, Chih-Yi Huang, Gail-Joon Ahn, Jaron Mink, Kashyap Thimmaraju, Souradip Nath.

Figure 1
Figure 1. Figure 1: Opinions of LLM Tools by Factor – All factors exhibit statistically significant differences in sentiment (p < .001); LLM Capabilities and Efficiency are discussed more positively, while other factors are discussed more negatively. Improved Interpretability of Signals. In addition to better contextualization of signals, practitioners (n=10) also noted that LLMs made previously hard-to-understand signals rea… view at source ↗
Figure 2
Figure 2. Figure 2: Number of Threads in Our Dataset Over Time. A Data Collection and Curation The following is the list of keywords used to find potential relevant threads (§ 3.1): ‘SOC AI’, ‘SOC AI Agents’, ‘SOC LLM’, ‘AI-powered SOC’, ‘Agentic AI SOC’, ‘autonomous SOC’, ‘AI SOC Analyst’, ‘AI Security Operations’, ‘AI cybersecurity’, ‘LLM in cybersecurity’, ‘AI augmenting cybersecurity’, ‘AI agents cybersecurity’, ‘LLM for … view at source ↗
read the original abstract

Large language models (LLMs) have recently emerged as promising tools for augmenting Security Operations Center (SOC) workflows, with vendors increasingly marketing autonomous AI solutions for SOCs. However, there remains a limited empirical understanding of how such tools are used, perceived, and adopted by real-world security practitioners. To address this gap, we conduct a mixed-methods analysis of discussions in cybersecurity-focused forums to learn how a diverse group of practitioners use and perceive modern LLM tools for security operations. More specifically, we analyzed 892 posts between December 2022 and September 2025 from three cybersecurity-focused forums on Reddit, and, using a combination of qualitative coding and statistical analysis, examined how security practitioners discuss LLM tools across three dimensions: (1) their stated tools and use cases, (2) the perceived pros and cons of each tool across a set of critical factors, and (3) their adoption of such tools and the expected impacts on the cybersecurity industry and individual analysts. Overall, our findings reveal nuanced patterns in LLM tools adoption, highlighting independent use of LLMs for low-risk, productivity-oriented tasks, alongside active interest around enterprise-grade, security-focused LLM platforms. Although practitioners report meaningful gains in efficiency and effectiveness in LLM-assisted workflows, persistent issues with reliability, verification overheads, and security risks sharply constrain the autonomy granted to LLM tools. Based on these results, we also provide recommendations for developing and adopting LLM tools to ensure the security of organizations and the safety of cybersecurity practitioners.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript presents a mixed-methods analysis of 892 Reddit posts from three cybersecurity forums (December 2022–September 2025) to investigate the uses, perceptions, and adoption of large language models (LLMs) by security practitioners. Through qualitative coding and statistical analysis, the authors identify patterns of independent LLM use for low-risk, productivity-oriented tasks, interest in enterprise-grade security-focused platforms, reported gains in efficiency and effectiveness, and constraints on autonomy due to issues with reliability, verification overheads, and security risks. The paper concludes with recommendations for LLM tool development and adoption in cybersecurity.

Significance. If the observed patterns are representative, this work provides important empirical grounding for understanding LLM integration in security operations centers, moving beyond anecdotal or vendor-driven narratives. It highlights practical barriers to full autonomy and suggests pathways for safer adoption, which could influence both research and industry practices in cybersecurity. The use of public forum data offers a scalable method for tracking emerging technology perceptions in the field.

major comments (3)
  1. [Methods] The qualitative coding process lacks details on the coding scheme (e.g., codebook or categories for use cases, pros/cons, and adoption), inter-rater reliability metrics, number of coders, and disagreement resolution. This is load-bearing for the central claims about nuanced patterns, as all findings derive from these codes.
  2. [Data Collection] No information is given on sampling and filtering of the 892 posts, such as search terms, inclusion/exclusion criteria, or per-forum distribution. The manuscript also does not quantify or correct for Reddit self-selection bias, which directly undermines generalizability to 'real-world security practitioners' as stated in the abstract and findings.
  3. [Results] The statistical analysis is referenced but without naming the tests used, reporting p-values, effect sizes, or how they support the 'nuanced patterns' in adoption and perceptions. This weakens evaluation of the strength of the reported efficiency gains and constraints.
minor comments (2)
  1. [Abstract] The date range in the abstract extends to September 2025; clarify whether data collection is retrospective or if this is a typo.
  2. [Results] Ensure all figures or tables summarizing coded categories include sample sizes per category and confidence intervals where appropriate for transparency.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback, which highlights important areas for improving the transparency and rigor of our mixed-methods analysis. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: [Methods] The qualitative coding process lacks details on the coding scheme (e.g., codebook or categories for use cases, pros/cons, and adoption), inter-rater reliability metrics, number of coders, and disagreement resolution. This is load-bearing for the central claims about nuanced patterns, as all findings derive from these codes.

    Authors: We agree that the Methods section requires greater transparency on the qualitative coding process to substantiate the central claims. In the revised manuscript, we will add a dedicated subsection detailing the codebook, including the hierarchical categories developed for use cases (e.g., code generation, threat analysis, documentation), pros/cons (e.g., efficiency, reliability, security risks), and adoption factors. We will report that two authors independently coded an initial 20% sample of posts, achieving a Cohen's kappa of 0.82 before full coding, with all disagreements resolved through iterative discussion and consensus. This information will directly support the nuanced patterns reported in the findings. revision: yes

  2. Referee: [Data Collection] No information is given on sampling and filtering of the 892 posts, such as search terms, inclusion/exclusion criteria, or per-forum distribution. The manuscript also does not quantify or correct for Reddit self-selection bias, which directly undermines generalizability to 'real-world security practitioners' as stated in the abstract and findings.

    Authors: We will revise the Data Collection section to explicitly list the search terms (e.g., combinations of 'LLM', 'ChatGPT', 'large language model' with 'SOC', 'cybersecurity', 'security operations'), inclusion criteria (posts discussing LLM use in security contexts from Dec 2022–Sep 2025), exclusion criteria (off-topic, spam, or non-practitioner perspectives), and the per-forum distribution (e.g., 412 from r/netsec, 305 from r/cybersecurity, 175 from r/InfoSec). For self-selection bias, we will add a dedicated Limitations paragraph acknowledging that Reddit users may skew toward certain demographics and cannot be fully quantified or corrected without supplementary survey data; however, we maintain that the observed patterns in public discussions still offer valuable empirical grounding for practitioner perceptions, as stated in the abstract. revision: partial

  3. Referee: [Results] The statistical analysis is referenced but without naming the tests used, reporting p-values, effect sizes, or how they support the 'nuanced patterns' in adoption and perceptions. This weakens evaluation of the strength of the reported efficiency gains and constraints.

    Authors: We will expand the Results section to specify the statistical methods employed, including chi-square tests for comparing categorical adoption patterns across use cases and perceived pros/cons, with all p-values (e.g., p < 0.01 for efficiency gains in low-risk tasks) and effect sizes (Cramer's V ranging 0.25–0.45) reported. We will explicitly link these results to the nuanced patterns, such as stronger efficiency gains in productivity tasks versus constraints from reliability issues, thereby providing clearer quantitative support for the qualitative findings. revision: yes

Circularity Check

0 steps flagged

No circularity: purely observational analysis of forum posts

full rationale

The paper performs a mixed-methods study of 892 Reddit posts via qualitative coding and statistical counts to report patterns in LLM use, perceptions, and adoption. No equations, derivations, fitted parameters, or predictions are present. The central claims derive directly from the coded data without reduction to self-citations, ansatzes, or input renaming. The analysis is self-contained against external benchmarks as an empirical report of observed discourse.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on two untested assumptions: that Reddit forum posts serve as a valid proxy for diverse practitioner behavior and that qualitative coding faithfully captures stated perceptions without researcher bias. No free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Reddit posts from three cybersecurity forums between December 2022 and September 2025 reflect the views and practices of a diverse group of security practitioners.
    The entire analysis treats forum text as representative data; this is invoked in the abstract when describing the sample and generalizing findings.

pith-pipeline@v0.9.0 · 5601 in / 1494 out tokens · 39996 ms · 2026-05-10T16:40:07.708918+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.