REVIEW 3 cited by
Evaluation of ChatGPT Model for Vulnerability Detection
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this technical report, we evaluated the performance of the ChatGPT and GPT-3 models for the task of vulnerability detection in code. Our evaluation was conducted on our real-world dataset, using binary and multi-label classification tasks on CWE vulnerabilities. We decided to evaluate the model because it has shown good performance on other code-based tasks, such as solving programming challenges and understanding code at a high level. However, we found that the ChatGPT model performed no better than a dummy classifier for both binary and multi-label classification tasks for code vulnerability detection.
Forward citations
Cited by 3 Pith papers
-
LLM Embedding-based Attribution (LEA): Quantifying Source Contributions to Generative Model's Response for Vulnerability Analysis
LEA uses rank-based linear dependence of layer-0 hidden states to attribute each response token to query, retrieved context, or internal knowledge, and distinguishes valid from generic retrieval with over 95% accuracy.
-
An Initial Exploration of Fine-tuning Small Language Models for Smart Contract Reentrancy Vulnerability Detection
Fine-tuning LLaMA 3B with LoRA on synthetic Solidity contracts yields 67% accuracy on the contracts it chooses to classify, but only about 52% when abstentions are counted as errors.
-
Large Language Models for Security Operations Centers: A Comprehensive Survey
A systematic review of 138 papers classifying LLM applications in SOC workflows by phase, model family, datasets, and maturity.
Discussion (0). Sign in to comment.