Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"

Adam Bauer; Alberto Bacchelli; Francesco Sovrano

REVIEW 4 major objections 4 minor 54 references

Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that chat-based large language models systematically detect vulnerabilities less often when the vulnerable code sits near the end of a long file, a 'lost-in-the-end' effect that chunking can partially fix.

desk verdict A real, practically relevant effect that the paper overclaims and whose causal experiment is confounded; still deserves a serious referee. read the letter →

arxiv 2502.06898 v1 pith:FCEGVURF submitted 2025-02-09 cs.SE cs.AI

Francesco Sovrano , Adam Bauer , Alberto Bacchelli This is my paper · ORCID

classification cs.SEcs.AI

keywords lost-in-the-endlargelanguagemodelsvulnerabilitylocalizationin-filedetectioncode-in-the-haystackinputchunkingCWE-79CWE-89

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that chat-based large language models such as GPT-3.5, GPT-4, GPT-4o, Llama 3, and the Mixtral models systematically miss software vulnerabilities that sit near the end of large source files, a pattern it names the 'lost-in-the-end' effect. The claim matters because developers increasingly paste entire files into chatbots for security review, and the effect held across the three most common and dangerous vulnerability classes studied: XSS (CWE-79), SQL injection (CWE-89), and path traversal (CWE-22). The paper reports statistically significant negative effects of file size and vulnerability position on detection probability, and shows in a controlled 'code-in-the-haystack' experiment that the effect is not an artifact of the skewed distribution of real-world bug positions. It then shows that splitting files into smaller chunks raises recall by over 37% on average, with optimal chunk sizes that vary by vulnerability type.

What carries the argument

The load-bearing mechanism is the 'code-in-the-haystack' experiment: a vulnerable block (a function under 500 characters, or a refactored roughly 500-character segment containing the vulnerable line) is relocated to uniform positions in synthetic files of fixed sizes, with padding drawn from the same repository and placed by a relaxed 0/1 knapsack algorithm. This isolates vulnerability position and file size from confounds, and the resulting detection scores across positions are analyzed with logistic regressions. The character-based measurement of file size and bug position is the quantitative backbone, and the error analysis of false predictions shows that when models fail they report bug positions early in the file.

What would settle it

Run the same code-in-the-haystack protocol with a 25,000-character file and the same vulnerable block, but compare detection when the block is at the first 500 characters versus the last 500 characters; if detection probability at the end matches the beginning within sampling error across the six LLMs, the 'lost-in-the-end' effect is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that six off-the-shelf chat-based LLMs detect vulnerabilities in whole files reliably only when the vulnerable code appears early and the file is short; once the vulnerability is placed deeper into a longer file, detection probability falls, and false predictions cluster near the file's beginning. This 'lost-in-the-end' pattern contradicts the 'lost-in-the-middle' behavior often reported for LLMs on long natural-language contexts. The paper establishes it both on 794 real vulnerable files from the CVE catalog, using logistic regressions with p-values below .05 for all models and CWE types, and in a controlled experiment that moves the same vulnerable block to uniformly distributed positions in files of 4,000 to 25,000 characters. It also quantifies a threshold effect, for example ChatGPT 4, 4o, and Llama 3 struggle with CWE-89 once input exceeds about 4,000 characters, and it shows that naive chunking to smaller inputs substantially improves recall.

Load-bearing premise

The controlled 'code-in-the-haystack' result assumes that moving and refactoring a vulnerable block into files padded with unrelated code from the same repository leaves the vulnerability intact and keeps detection difficulty unchanged except for position.

Editorial extensions

If this is right

Practitioners who paste whole files into GPT-4 or similar chatbots will systematically under-detect vulnerabilities located in the latter part of large files, regardless of the vulnerability type.
Chunking files into smaller inputs is an immediate, no-fine-tuning mitigation: the paper measured an average recall gain of over 37%, with gains up to roughly 95% for CWE-79 under a 500-character chunk size.
The optimal chunk size is vulnerability-dependent: around 500 to 1,500 characters for CWE-79 and CWE-89, and up to 6,500 characters for CWE-22.
After chunking, open-source models such as Mixtral and Llama sometimes match or beat commercial models, suggesting the commercial advantage lies partly in handling longer contexts.
The same 'lost-in-the-end' limitation likely affects other LLM tasks that require reasoning over large files, such as code review, general bug localization, and code summarization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the position-heuristic explanation is right, then fine-tuning or continued pretraining on files with uniformly distributed bug positions should shrink the 'lost-in-the-end' effect; this is a directly testable prediction the paper leaves open.
The character-based thresholds suggest a practical 'safe input size' chart could be built for other CWE types and future models using the same chunking search procedure, without retraining.
Because the effect appeared in all six models, it may stem from the pretraining data distribution rather than any single architecture; comparing models trained on code with different position statistics would separate data-driven from architecture-driven causes.
A natural next experiment is to reverse the file content while keeping the vulnerability's character offset, which would disentangle 'end of sequence' effects from content-level cues.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper reports something worth knowing: off-the-shelf chat LLMs, when asked to find a vulnerability in a whole file, get worse as the file grows and as the bug sits closer to the end. That direction is consistent across a large real-world CVE-derived corpus and the paper's synthetic "code-in-the-haystack" experiment, and the chunking mitigation (RQ3) is a practical, cheap intervention practitioners can try. Adapting needle-in-the-haystack to code is a genuinely new protocol, and the replication package is public.

The soft spots are in the claims, not in the effort. The abstract says "regardless of the vulnerability type ... p < .05", but the paper's own Figure 4 shows the bug-position effect is non-significant for CWE-79 with gpt-3.5-turbo (p=0.897) and gpt-4o (p=0.123). That overstatement matters because CWE-79 is the most common type in the dataset. The "regardless" claim should be dropped or qualified.

The bigger problem is the causal experiment. In the code-in-the-haystack construction, padding is drawn from the same repository via a knapsack fill, so as the vulnerable block moves later, the set and order of functions placed before it change. Position is not manipulated independently of content. A model may be distracted by what the padding contains, not by distance from the file start. The paper's Section 8 acknowledges syntactic errors from the construction and waves them away; those errors are exactly the kind of artifact that could produce a position-correlated drop. The real-world regressions support the direction, but they carry the tailed-distribution confound the synthetic experiment was built to eliminate. So the causal "lost-in-the-end" claim is plausible but not established.

RQ3's chunking gain is also selected on the same data used to report the +37% improvement: the best of four chunk sizes per model/CWE is chosen from the full set, so the number is an upper bound, not an estimate for new files.

Even with these issues, the paper deserves a serious referee. The phenomenon is practically relevant, the dataset and package are reusable, and the flaws are fixable with a softened abstract, a control that varies position while holding padding content constant, and held-out chunk-size validation. I'd send it out, with instructions to the authors to substantiate or scale back the universality claim.

Referee Report

4 major / 4 minor

Summary. The paper evaluates six chat-based LLMs (Mixtral 8x7b/8x22b, Llama 3 70B, GPT-3.5-turbo, GPT-4-turbo, GPT-4o) on in-file vulnerability localization for three CWE types: CWE-22 (path traversal), CWE-89 (SQL injection), and CWE-79 (XSS). The dataset consists of 794 vulnerable files extracted from CVE single-file patches and 794 patched counterparts. RQ1 benchmarks the models and reports low accuracy (best accuracy 0.324). RQ2 uses two analyses: observational logistic regressions on real-world files, and a 'code-in-the-haystack' synthetic experiment with uniformly distributed vulnerability positions, leading to the paper's central 'lost-in-the-end' claim that detection probability declines with file size and vulnerability position. RQ3 proposes line-preserving file chunking at sizes 500-6500 characters and reports an average recall improvement of over 37% relative to the RQ1 baseline. The paper includes a replication package.

Significance. If the 'lost-in-the-end' effect were established, the paper would make a timely and practically important contribution: it identifies a failure mode distinct from the commonly reported 'lost-in-the-middle' effect, and it offers concrete chunking guidance that practitioners could adopt immediately. The study's strengths include the use of real CVE data, six diverse models, a publicly available replication package, explicit power analysis, and repeated runs for the synthetic experiment. However, the central claim as stated is overgeneralized relative to the paper's own data, and the causal evidence from the synthetic experiment is weakened by a confound between vulnerability position and padding content. The RQ3 improvement estimate is also based on selecting the best chunk size on the evaluation set. These issues are fixable, but they currently prevent the paper from supporting its headline claims in full.

major comments (4)

[Abstract and Section 6.2, Figure 4] The abstract claims that 'regardless of the vulnerability type' LLMs significantly (p<.05) underperform for vulnerabilities toward the end of larger files. Figure 4 contradicts this universal claim: for CWE-79, the bug-position logistic regression is non-significant for gpt-3.5-turbo (p=0.897) and gpt-4o (p=0.123), and the file-size regression is non-significant for gpt-4o (p=0.086). The sentence in Section 6.2 stating that 'these regressions confirm the previous results' and the Section 9 claim that the finding is 'consistent across all models and types of vulnerabilities studied' are therefore not accurate. The authors should either restrict the claim to the model-CWE combinations where the effect is actually supported, or provide a meta-analytic or mixed-effects analysis that justifies a general conclusion despite these exceptions.
[Section 6.2] The code-in-the-haystack experiment does not manipulate vulnerability position independently of the surrounding code. As the vulnerable block is moved to position n, the 0/1 knapsack construction selects different sets and orderings of repository functions for the before-block and after-block padding segments, so the observed detection drop at later positions could be caused by the specific padding content rather than by the distance from the start of the file. The statement that the design 'holds all potential confounding factors constant' is therefore unsupported. The threat acknowledged in Section 8—that algorithmic construction and manual refactoring could introduce syntactic errors—is relevant precisely because such errors may be correlated with position and can affect detection. The authors should add syntax verification and a control condition, for example by randomizing which functions appear before the block at each position or by holding the padding set fixed while reordering it.
[Section 7, Table 4] The reported '+37% average recall improvement' is an in-sample selection result. For each model and CWE type, Table 4 selects the chunk size k that maximizes recall on the same 794 files used to compute both the baseline and the improvement, choosing among four candidate values. Selecting the best of several values on the evaluation set produces an optimistically biased estimate of the improvement practitioners would obtain. To support the proposed chunking strategy, the authors should validate it on held-out files or with repeated cross-validation, and report the distribution of improvements across folds rather than only the best-k value.
[Section 6.2 methodology] The dependent variable for the logistic regressions is not adequately defined. The methodology assigns +1 for a correct localization and -1 for an incorrect one, then averages across five vulnerability instances and five runs, resulting in a continuous score in [-1, 1]; however, logistic regression requires binary outcomes, and Figure 4 reports 'Detection Probability.' The paper should state how the averaged score was converted into the binary outcome used for the logistic regression (for example, by thresholding at zero) or should instead model the trial-level data with a mixed-effects logistic regression. Without this clarification, the p-values reported in Figure 4 cannot be interpreted.

minor comments (4)

[Figure 4] The figure is difficult to read because the six model panels are not labeled in the caption and the ordering of the file-size and bug-position rows is not described; please add explicit panel labels.
[Section 5] The same in-context example, a CWE-79 bug, is used in the prompt for all vulnerability types; the potential effect of this example on the detection rates for CWE-22 and CWE-89 should be discussed or controlled, since it may introduce a systematic bias.
[Section 6.2] The power-analysis sentence '273·500/30000≈5' is opaque; please show the calculation explicitly, since 273×500/30,000 is approximately 4.55 and is rounded up to 5.
[Table 3] The number of files differs slightly across models for the same CWE (e.g., 210 vs. 208 for CWE-22), but the table does not explain why; a brief note about failed API calls or output parsing would clarify this.

Circularity Check

1 steps flagged · score 4.0 of 10

RQ3's headline +37% recall gain is an in-sample maximum over four chunk sizes fitted on the same files, so that quantitative claim is partly a fitted maximum; the central lost-in-the-end result is external and not derivational.

fitted input called prediction [Section 7 (RQ3: Input Size Identification), 'Methodology' and 'Results' paragraphs, with Table 4 and Finding 3]
"The chunking strategy was implemented using k values of {6,500, 3,000, 1,500, 500}. ... According to the results detailed in Table 4, we have an average recall improvement of over +37% across all models and CWE types due to chunking"

Table 4 is explicitly titled 'Chunk sizes yielding the highest recall for each LLM and CWE type.' For each model-CWE cell, the chunk size is selected as the argmax recall among the four tested sizes on the same 794 vulnerable files used to compute the baseline recall. The reported '+37% average improvement' is therefore the mean of per-cell selection maxima, not an unbiased estimate of the gain from adopting the proposed chunking strategy.

full rationale

No derivational circularity is present in the central lost-in-the-end claim: the RQ1/RQ2 results are empirical regressions on CVE-sourced files and on synthetically relocated vulnerable blocks, and the logistic-regression outcome is not defined in terms of the predictor. The code-in-the-haystack construction may confound bug position with the specific padding content chosen by the knapsack procedure, and Figure 4 shows non-significant position effects for CWE-79 on gpt-3.5-turbo (p=0.897) and gpt-4o (p=0.123); these are validity and consistency concerns, not circularity, so they are not scored as circular steps. The one genuine pattern of concern is RQ3: the optimal chunk size is selected as the best of four values on the same dataset used to report the +37% recall gain, making that headline a fitted maximum. That is a partial reduction of one quantitative claim to its own selection rule, but it does not circularly generate the lost-in-the-end effect. Score 4 reflects this partial, claim-specific circularity with an otherwise self-contained empirical derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's quantitative claims rest on dataset labeling from CVE patches and on synthetic file construction; neither is machine-verified. RQ3's headline improvement is selected post hoc on the same data. There is no derivational circularity, but the optimal-size result is effectively fit to the data, and the causal experiment depends on the validity of the synthetic relocation procedure.

free parameters (3)

chunk size grid k = 500, 1500, 3000, 6500 characters; best per model and CWE chosen post hoc
Hand-picked values in Section 7; Table 4 reports the best of these as the optimal size, so the +37% average recall gain is a selected maximum, not an unbiased estimate.
vulnerable block size threshold = 500 characters
In Section 6.2, functions larger than 500 characters are truncated and refactored around the vulnerable line; this arbitrary threshold could affect how much context the moved bug retains.
knapsack padding tolerance = about 200 characters
The relaxed 0/1 knapsack construction allows padding segments to exceed capacity by about 200 characters, so constructed file sizes and positions are approximate.

assumptions (5)

domain assumption CVE patch ground truth: pre-patch files contain the target CWE vulnerability, post-patch files fix it, and patch diff lines are the correct vulnerable lines.
Dataset construction in Section 4 adopts this from prior work. If patches are incomplete or mislabeled, precision and recall estimates shift; the authors note Rice's theorem limits certainty for the non-vulnerable files.
domain assumption Refactored and relocated vulnerable blocks behave identically to the original vulnerable code.
Section 6.2 builds synthetic files by moving and refactoring code and padding with other repository content; the causal interpretation of bug position depends on this assumption.
domain assumption Logistic regression adequately captures the relationship between detection and file size or bug position.
The authors assume a linear log-odds relationship and acknowledge in Section 8 that more complex patterns might be missed.
domain assumption A priori power analysis parameters (uniform positions, odds ratio greater than 3.47, R-squared of 0.2) justify the sample of five vulnerabilities per CWE.
Section 6.2 relies on these assumptions to claim sufficient statistical power; if true effect sizes are smaller or positions are not uniform, five instances may be insufficient.
domain assumption LLM outputs with temperature 0 are sufficiently deterministic and parseable.
Non-determinism is acknowledged in Section 8; repeated runs were only performed for the controlled experiment, so the other results rest on single or few runs per input.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"." pith.science (2026). https://pith.science/paper/FCEGVURF

@misc{pith2026250206898,
  author       = {Pith},
  title        = {Pith review of: Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End"},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCEGVURF}},
  note         = {Machine review of arXiv:2502.06898}
}

read the original abstract

Recent advancements in artificial intelligence have enabled processing of larger inputs, leading everyday software developers to increasingly rely on chat-based large language models (LLMs) like GPT-3.5 and GPT-4 to detect vulnerabilities across entire files, not just within functions. This new development practice requires researchers to urgently investigate whether commonly used LLMs can effectively analyze large file-sized inputs, in order to provide timely insights for software developers and engineers about the pros and cons of this emerging technological trend. Hence, the goal of this paper is to evaluate the effectiveness of several state-of-the-art chat-based LLMs, including the GPT models, in detecting in-file vulnerabilities. We conducted a costly investigation into how the performance of LLMs varies based on vulnerability type, input size, and vulnerability location within the file. To give enough statistical power to our study, we could only focus on the three most common (as well as dangerous) vulnerabilities: XSS, SQL injection, and path traversal. Our findings indicate that the effectiveness of LLMs in detecting these vulnerabilities is strongly influenced by both the location of the vulnerability and the overall size of the input. Specifically, regardless of the vulnerability type, LLMs tend to significantly (p < .05) underperform when detecting vulnerabilities located toward the end of larger files, a pattern we call the 'lost-in-the-end' effect. Finally, to further support software developers and practitioners, we also explored the optimal input size for these LLMs and presented a simple strategy for identifying it, which can be applied to other models and vulnerability types. Eventually, we show how adjusting the input size can lead to significant improvements in LLM-based vulnerability detection, with an average recall increase of over 37% across all models.

Figures

Figures reproduced from arXiv: 2502.06898 by the authors.

**Figure 2.** Real-World Files: Logistic regressions across LLMs and CWE types; data distributions on the left. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** Code-in-the-Haystack: Heatmap results for the most accurate LLMs. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Code-in-the-Haystack: Comparative logistic regression analysis. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: Code-in-the-Haystack: Distribution of incorrectly predicted bug positions. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 26 canonical work pages

[1]

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make Your LLM Fully Utilize the Context. CoRR abs/2404.16811 (2024). doi:10.48550/ARXIV.2404.16811 arXiv:2404.16811

arXiv doi:10.48550/arxiv.2404.16811 2024
[2]

Adam Bauer and Francesco Sovrano. 2025. Replication Package for In-file Vulnerability Detection. https://doi.org/10. 5281/zenodo.14840519

work page 2025
[3]

Dipkamal Bhusal, Md Tanvirul Alam, Le Nguyen, Ashim Mahara, Zachary Lightcap, Rodney Frazier, Romy Fieblinger, Grace Long Torales, and Nidhi Rastogi. 2024. SECURE: Benchmarking Generative Large Language Models for Cybersecurity Advisory. CoRR abs/2405.20441 (2024). doi:10.48550/ARXIV.2405.20441 arXiv:2405.20441

arXiv doi:10.48550/arxiv.2405.20441 2024
[4]

Lost in the End

Alessio Buscemi. 2023. A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages. CoRR abs/2308.04477 (2023). doi:10.48550/ARXIV.2308.04477 arXiv:2308.04477 Large Language Models for In-File Vulnerability Localization Can Be “Lost in the End” 21

arXiv doi:10.48550/arxiv.2308.04477 2023
[5]

Chong Chen, Jianzhong Su, Jiachi Chen, Yanlin Wang, Tingting Bi, Yanli Wang, Xingwei Lin, Ting Chen, and Zibin Zheng. 2023. When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We? doi:10.48550/ARXIV. 2309.05520 arXiv:2309.05520

arXiv doi:10.48550/arxiv.2309.05520 2023
[6]

Henian Chen, Patricia Cohen, and Sophie Chen. 2010. How Big is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies. Commun. Stat. Simul. Comput. 39, 4 (2010), 860–864. doi:10.1080/03610911003650383

doi:10.1080/03610911003650383 2010
[7]

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending Context Window of Large Language Models via Positional Interpolation. CoRR abs/2306.15595 (2023). doi:10.48550/ARXIV.2306.15595 arXiv:2306.15595

arXiv doi:10.48550/arxiv.2306.15595 2023
[8]

Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023 . ACM, 654–668. doi:10.1145...

work page arXiv 2023

Show all 54 references

[9]

Anton Cheshkov, Pavel Zadorozhny, and Rodion Levichev. 2023. Evaluation of ChatGPT Model for Vulnerability Detection. CoRR abs/2304.07232 (2023). doi:10.48550/ARXIV.2304.07232 arXiv:2304.07232

2023 doi arXiv
[10]

2013.Converting data into evidence: A statistics primer for the medical practitioner

Alfred DeMaris and Steven H Selman. 2013.Converting data into evidence: A statistics primer for the medical practitioner . Springer. doi:10.1007/978-1-4614-7792-1

2013 doi
[11]

Aparna Dhinakaran. 2024. The Needle in a Haystack Test. https://towardsdatascience.com/the-needle-in-a-haystack- test-a94974c1ad38. Accessed: 2024-06-03

2024
[12]

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens. CoRR abs/2402.13753 (2024). doi:10.48550/ARXIV.2402.13753 arXiv:2402.13753

2024 doi arXiv
[13]

Edgar Erdfelder, Franz Faul, and Axel Buchner. 1996. GPOWER: A general power analysis program. Behavior research methods, instruments, & computers 28 (1996), 1–11. doi:10.3758/BF03203630

1996 doi
[14]

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP ...

2020 doi
[15]

Michael Fu and Chakkrit Tantithamthavorn. 2022. LineVul: A Transformer-based Line-Level Vulnerability Prediction. In 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24,

2022
[16]

Michael Fu, Chakkrit Kla Tantithamthavorn, Van Nguyen, and Trung Le. 2023. ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?. In 30th Asia-Pacific Software Engineering Conference, APSEC 2023, Seoul, Republic of Korea, December 4-7, 2023 . IEEE, 6...

2023
[17]

GitLab. 2022. 2022 DevSecOps Report. https://learn.gitlab.com/dev-survey-22/2022-devsecops-report. Accessed: 2024-06-03

2022
[18]

Sivana Hamer, Marcelo d’Amorim, and Laurie Williams. 2024. Just another copy and paste? Comparing the security vul- nerabilities of ChatGPT generated code and StackOverflow answers. doi:10.48550/ARXIV.2403.15600 arXiv:2403.15600

2024 doi arXiv
[19]

Wajih Ul Hassan, Shengjian Guo, Ding Li, Zhengzhang Chen, Kangkook Jee, Zhichun Li, and Adam Bates. 2019. NoDoze: Combatting Threat Alert Fatigue with Automated Provenance Triage. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, Californi...

2019
[20]

Wenpin Hou and Zhicheng Ji. 2024. Comparing large language models and human programmers for generating programming code. Advanced Science (2024), 2412279. doi:10.1002/advs.202412279

2024 doi
[21]

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models. doi:10.48550/ARXIV.2402.14848 arXiv:2402.14848

2024 doi arXiv
[22]

Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV: LLM Knows What You are Looking for Before Generation. doi:10.48550/ARXIV.2404.14469 arXiv:2404.14469

2024 doi arXiv
[23]

Chongyang Liu, Xiang Chen, Xiangwei Li, and Yinxing Xue. 2024. Making vulnerability prediction more practical: Prediction, categorization, and localization. Information and Software Technology 171 (2024), 107458. doi:10.1016/j. infsof.2024.107458

2024
[24]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Trans. Assoc. Comput. Linguistics 12 (2024), 157–173. doi:10.1162/TACL_A_00638

2024 doi
[25]

Yu Liu, Lang Gao, Mingxin Yang, Yu Xie, Ping Chen, Xiaojin Zhang, and Wei Chen. 2024. VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models. CoRR abs/2406.07595 (2024). doi:10. 22 Sovrano et al. 48550/ARXIV.2406.07595 arXiv:2406.07595

2024 arXiv
[26]

Daniel Machlab and Rick Battle. 2024. LLM In-Context Recall is Prompt Dependent. doi:10.48550/ARXIV.2404.08865 arXiv:2404.08865

2024 doi arXiv
[27]

Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. 2023. Prompt Engineering in Large Language Models. In International Conference on Data Intelligence and Cognitive Informatics . Springer, 387–402. doi:10.1007/978-981-99-7962-2_30

2023 doi
[28]

MITRE. 2024. Common Vulnerabilities and Exposures (CVE) Catalog. https://www.cve.org. Accessed: 2024-06-03

2024
[29]

MITRE Corporation. 2024. 2024 CWE Top 25 Most Dangerous Software Weaknesses. https://cwe.mitre.org/top25/ archive/2024/2024_cwe_top25.html. [Online; accessed 5-February-2025]

2024
[30]

Hellendoorn, Bogdan Vasilescu, and Brad A

Daye Nam, Andrew Macvean, Vincent J. Hellendoorn, Bogdan Vasilescu, and Brad A. Myers. 2024. Using an LLM to Help With Code Understanding. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM...

2024
[31]

Kollin Napier, Tanmay Bhowmik, and Shaowei Wang. 2023. An empirical study of text-based machine learning models for vulnerability detection. Empir. Softw. Eng. 28, 2 (2023), 38. doi:10.1007/S10664-022-10276-6

2023 doi
[32]

Thu-Trang Nguyen and Hieu Dinh Vo. 2024. Context-based statement-level vulnerability localization. Inf. Softw. Technol. 169 (2024), 107406. doi:10.1016/J.INFSOF.2024.107406

2024
[33]

Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2024. Arithmetic Without Algorithms: Lan- guage Models Solve Math With a Bag of Heuristics. CoRR abs/2410.21272 (2024). doi:10.48550/ARXIV.2410.21272 arXiv:2410.21272

2024 doi arXiv
[34]

Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023. Giraffe: Adventures in Expanding Context Lengths in LLMs. doi:10.48550/ARXIV.2308.10882 arXiv:2308.10882

2023 doi arXiv
[35]

Chao-Ying Joanne Peng, Kuk Lida Lee, and Gary M Ingersoll. 2002. An introduction to logistic regression analysis and reporting. The journal of educational research 96, 1 (2002), 3–14. doi:10.1080/00220670209598786

2002 doi
[36]

Tao Peng, Shixu Chen, Fei Zhu, Junwei Tang, Junping Liu, and Xinrong Hu. 2023. PTLVD:Program Slicing and Transformer-based Line-level Vulnerability Detection System. In 23rd IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2023, Bogotá, Colo...

2023
[37]

David Pisinger. 1997. A Minimal Algorithm for the 0-1 Knapsack Problem. Oper. Res. 45, 5 (1997), 758–767. doi:10. 1287/OPRE.45.5.758

1997
[38]

Radford, and Bill Chu

Moumita Das Purba, Arpita Ghosh, Benjamin J. Radford, and Bill Chu. 2023. Software Vulnerability Detection using Large Language Models. In 34th IEEE International Symposium on Software Reliability Engineering, ISSRE 2023 - Workshops, Florence, Italy, October 9-12, 2023 . IEEE,...

2023
[39]

Henry Gordon Rice. 1953. Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical society 74, 2 (1953), 358–366. doi:10.2307/1990888

1953 doi
[40]

Sonatype. 2023. State of the Software Supply Chain 9th Edition. https://www.sonatype.com/state-of-the-software- supply-chain/introduction. Accessed: 2024-06-03

2023
[41]

Barr, and Wei Le

Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Earl T. Barr, and Wei Le
[42]

Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, and Surya Nepal. 2022. Transformer-Based Language Models for Software Vulnerability Detection. In Annual Computer Security Applications Conference, ACSAC 2022, Austin, TX, USA, December 5-9, 202...

2022
[43]

Coskun, and Gianluca Stringhini

Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse K. Coskun, and Gianluca Stringhini. 2024. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. In IEEE Symposium on Security and Privac...

2024
[44]

Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald C. Gall. 2018. Context is king: The developer perspective on the usage of static analysis tools. In 25th International Conference on Software Analysis, Evolution and Reengineeri...

2018
[45]

Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. 2024. ReposVul: A Repository- Level High-Quality Vulnerability Dataset. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, ICSE Companio...

2024
[46]

Lost in the End

Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2021...

2021
[47]

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets Long Context Large Language Models. doi:10.48550/ ARXIV.2310.03025 arXiv:2310.03025

2023 doi arXiv
[48]

Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. 2023. Prompt-Enhanced Software Vulnerability Detection Using ChatGPT. doi:10.48550/ARXIV.2308.12697 arXiv:2308.12697

2023 doi arXiv
[49]

Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024. Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon. doi:10.48550/ARXIV.2401.03462 arXiv:2401.03462

2024 doi arXiv
[50]

Xin Zhou, Kisub Kim, Bowen Xu, Jiakun Liu, DongGyun Han, and David Lo. 2023. The Devil is in the Tails: How Long- Tailed Code Distributions Impact Large Language Models. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, Septembe...

2023
[51]

Xin Zhou, Ting Zhang, and David Lo. 2024. Large Language Model for Vulnerability Detection: Emerging Results and Future Directions. doi:10.48550/ARXIV.2401.15468 arXiv:2401.15468

2024 doi arXiv
[52]

Xin Zhou, Ting Zhang, and David Lo. 2024. Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 47–51. doi:10.1145/363...

2024
[2022]

doi:10.1145/3524842.3528452

ACM, 608–620. doi:10.1145/3524842.3528452
[2024]

A Comprehensive Study of the Capabilities of Large Language Models for Vulnerability Detection. (2024). doi:10.48550/ARXIV.2403.17218 arXiv:2403.17218

2024 doi arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make Your LLM Fully Utilize the Context. CoRR abs/2404.16811 (2024). doi:10.48550/ARXIV.2404.16811 arXiv:2404.16811

arXiv doi:10.48550/arxiv.2404.16811 2024

[2] [2]

Adam Bauer and Francesco Sovrano. 2025. Replication Package for In-file Vulnerability Detection. https://doi.org/10. 5281/zenodo.14840519

work page 2025

[3] [3]

Dipkamal Bhusal, Md Tanvirul Alam, Le Nguyen, Ashim Mahara, Zachary Lightcap, Rodney Frazier, Romy Fieblinger, Grace Long Torales, and Nidhi Rastogi. 2024. SECURE: Benchmarking Generative Large Language Models for Cybersecurity Advisory. CoRR abs/2405.20441 (2024). doi:10.48550/ARXIV.2405.20441 arXiv:2405.20441

arXiv doi:10.48550/arxiv.2405.20441 2024

[4] [4]

Lost in the End

Alessio Buscemi. 2023. A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages. CoRR abs/2308.04477 (2023). doi:10.48550/ARXIV.2308.04477 arXiv:2308.04477 Large Language Models for In-File Vulnerability Localization Can Be “Lost in the End” 21

arXiv doi:10.48550/arxiv.2308.04477 2023

[5] [5]

Chong Chen, Jianzhong Su, Jiachi Chen, Yanlin Wang, Tingting Bi, Yanli Wang, Xingwei Lin, Ting Chen, and Zibin Zheng. 2023. When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We? doi:10.48550/ARXIV. 2309.05520 arXiv:2309.05520

arXiv doi:10.48550/arxiv.2309.05520 2023

[6] [6]

Henian Chen, Patricia Cohen, and Sophie Chen. 2010. How Big is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies. Commun. Stat. Simul. Comput. 39, 4 (2010), 860–864. doi:10.1080/03610911003650383

doi:10.1080/03610911003650383 2010

[7] [7]

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending Context Window of Large Language Models via Positional Interpolation. CoRR abs/2306.15595 (2023). doi:10.48550/ARXIV.2306.15595 arXiv:2306.15595

arXiv doi:10.48550/arxiv.2306.15595 2023

[8] [8]

Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. 2023. DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection. In Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 16-18, 2023 . ACM, 654–668. doi:10.1145...

work page arXiv 2023