{"id":"e07c50f1-41ee-413f-bb41-0ad9a661b67f","arxiv_id":"2502.06898","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Large language models detect in-file vulnerabilities best when the vulnerable code appears early in the file, a 'lost-in-the-end' effect, and chunking files into smaller blocks can increase recall.","lead":"Chatbots like ChatGPT get worse at finding security bugs when the bug sits near the end of a long code file. Splitting code into smaller pieces can improve detection, but the best piece size depends on the bug type and model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Code-in-the-haystack experiment confounds vulnerability position with padding content, so the claim that LLMs are 'lost-in-the-end' may be an artifact of how the synthetic files were built.","rationale":"The paper's strongest contribution is the identification of a plausible 'lost-in-the-end' phenomenon, backed by real-world correlations and a replication package. The central claim, however, rests on the code-in-the-haystack experiment being a valid causal control. The reader's weakest-assumption analysis correctly points to the synthetic relocation procedure as the most fragile link: preserving vulnerability semantics and holding detection difficulty constant across positions is non-trivial, and the paper's own Section 8 concedes the construction may produce syntactically erroneous files. I agree that this is the load-bearing concern. My analysis sharpens it: the knapsack-based padding selection means the context preceding the vulnerable block is not constant across positions, so position is confounded with padding content. Even a perfect refactoring would not isolate position unless the surrounding code is fixed. This is a testable threat, and the proposed control (fixed sequence of functions, block repositioned by swapping) directly addresses it. Independent of this threat, the abstract's 'regardless of vulnerability type' overstates the data, since two of the six models show non-significant position effects for CWE-79 in the controlled experiment. The combination of an unvalidated construction and an overbroad generalization justifies a conditional verdict: the paper should be accepted only after the construction-control experiment confirms the effect and the claim is relaxed to the models/CWEs where it is actually observed. If the control experiment refutes the effect, the central claim would collapse; if it confirms, the contribution stands, and the chunking result (Finding 3) remains practically useful regardless.","tokens_in":23372,"tokens_out":9959,"duration_ms":93638,"concrete_test":"Run a control variant of the code-in-the-haystack experiment in which the padding is a fixed, pre-selected sequence of functions (e.g., the original file's functions in their original order) and the vulnerable block is moved by swapping its position with adjacent functions, keeping the file size fixed (e.g., S=16,000). For the same five CWE-22/89/79 instances and the same prompt, compute the logistic regression of detection on block position. If the position slope is no longer significant, the reported lost-in-the-end effect is an artifact of the knapsack-selected padding content rather than a genuine effect of position; if the slope persists, the construction concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 6.2's code-in-the-haystack experiment is the paper's causal evidence for the 'lost-in-the-end' effect. The procedure relocates a vulnerable block into synthetic files where the space before and after the block is filled by functions selected from the same repository via a 0/1 knapsack algorithm (§6.2). As the block's position changes, both the set and the order of the padding functions change, so 'bug position' is not manipulated independently of the surrounding code content. A drop in detection toward the end could thus be caused by the specific padding placed before the block (e.g., functions that are longer, more distracting, or containing code resembling vulnerabilities), not by the distance from the file start. The paper acknowledges in Section 8 that 'algorithmic construction and manual refactoring ... could have introduced syntactical errors' and dismisses this, but syntax errors are exactly the kind of artifact that can make a model miss a bug at the end of a long file. The real-world correlational results (§6.1) are also vulnerable to the tailed-distribution confound the synthetic experiment was meant to eliminate. Moreover, the central claim 'regardless of vulnerability type ... p < .05' is not supported by the synthetic results: in Figure 4, the position effect for CWE-79 is non-significant for gpt-3.5-turbo (p = 0.897) and gpt-4o (p = 0.123). Therefore, without a control that isolates position from padding content, the causal claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates six chat-based LLMs (Mixtral 8x7b/8x22b, Llama 3 70B, GPT-3.5-turbo, GPT-4-turbo, GPT-4o) on in-file vulnerability localization for three CWE types: CWE-22 (path traversal), CWE-89 (SQL injection), and CWE-79 (XSS). The dataset consists of 794 vulnerable files extracted from CVE single-file patches and 794 patched counterparts. RQ1 benchmarks the models and reports low accuracy (best accuracy 0.324). RQ2 uses two analyses: observational logistic regressions on real-world files, and a 'code-in-the-haystack' synthetic experiment with uniformly distributed vulnerability positions, leading to the paper's central 'lost-in-the-end' claim that detection probability declines with file size and vulnerability position. RQ3 proposes line-preserving file chunking at sizes 500-6500 characters and reports an average recall improvement of over 37% relative to the RQ1 baseline. The paper includes a replication package.","tokens_in":23806,"tokens_out":7232,"duration_ms":74325,"significance":"If the 'lost-in-the-end' effect were established, the paper would make a timely and practically important contribution: it identifies a failure mode distinct from the commonly reported 'lost-in-the-middle' effect, and it offers concrete chunking guidance that practitioners could adopt immediately. The study's strengths include the use of real CVE data, six diverse models, a publicly available replication package, explicit power analysis, and repeated runs for the synthetic experiment. However, the central claim as stated is overgeneralized relative to the paper's own data, and the causal evidence from the synthetic experiment is weakened by a confound between vulnerability position and padding content. The RQ3 improvement estimate is also based on selecting the best chunk size on the evaluation set. These issues are fixable, but they currently prevent the paper from supporting its headline claims in full.","major_comments":[{"comment":"The abstract claims that 'regardless of the vulnerability type' LLMs significantly (p<.05) underperform for vulnerabilities toward the end of larger files. Figure 4 contradicts this universal claim: for CWE-79, the bug-position logistic regression is non-significant for gpt-3.5-turbo (p=0.897) and gpt-4o (p=0.123), and the file-size regression is non-significant for gpt-4o (p=0.086). The sentence in Section 6.2 stating that 'these regressions confirm the previous results' and the Section 9 claim that the finding is 'consistent across all models and types of vulnerabilities studied' are therefore not accurate. The authors should either restrict the claim to the model-CWE combinations where the effect is actually supported, or provide a meta-analytic or mixed-effects analysis that justifies a general conclusion despite these exceptions.","section":"Abstract and Section 6.2, Figure 4"},{"comment":"The code-in-the-haystack experiment does not manipulate vulnerability position independently of the surrounding code. As the vulnerable block is moved to position n, the 0/1 knapsack construction selects different sets and orderings of repository functions for the before-block and after-block padding segments, so the observed detection drop at later positions could be caused by the specific padding content rather than by the distance from the start of the file. The statement that the design 'holds all potential confounding factors constant' is therefore unsupported. The threat acknowledged in Section 8—that algorithmic construction and manual refactoring could introduce syntactic errors—is relevant precisely because such errors may be correlated with position and can affect detection. The authors should add syntax verification and a control condition, for example by randomizing which functions appear before the block at each position or by holding the padding set fixed while reordering it.","section":"Section 6.2"},{"comment":"The reported '+37% average recall improvement' is an in-sample selection result. For each model and CWE type, Table 4 selects the chunk size k that maximizes recall on the same 794 files used to compute both the baseline and the improvement, choosing among four candidate values. Selecting the best of several values on the evaluation set produces an optimistically biased estimate of the improvement practitioners would obtain. To support the proposed chunking strategy, the authors should validate it on held-out files or with repeated cross-validation, and report the distribution of improvements across folds rather than only the best-k value.","section":"Section 7, Table 4"},{"comment":"The dependent variable for the logistic regressions is not adequately defined. The methodology assigns +1 for a correct localization and -1 for an incorrect one, then averages across five vulnerability instances and five runs, resulting in a continuous score in [-1, 1]; however, logistic regression requires binary outcomes, and Figure 4 reports 'Detection Probability.' The paper should state how the averaged score was converted into the binary outcome used for the logistic regression (for example, by thresholding at zero) or should instead model the trial-level data with a mixed-effects logistic regression. Without this clarification, the p-values reported in Figure 4 cannot be interpreted.","section":"Section 6.2 methodology"}],"minor_comments":[{"comment":"The figure is difficult to read because the six model panels are not labeled in the caption and the ordering of the file-size and bug-position rows is not described; please add explicit panel labels.","section":"Figure 4"},{"comment":"The same in-context example, a CWE-79 bug, is used in the prompt for all vulnerability types; the potential effect of this example on the detection rates for CWE-22 and CWE-89 should be discussed or controlled, since it may introduce a systematic bias.","section":"Section 5"},{"comment":"The power-analysis sentence '273·500/30000≈5' is opaque; please show the calculation explicitly, since 273×500/30,000 is approximately 4.55 and is rounded up to 5.","section":"Section 6.2"},{"comment":"The number of files differs slightly across models for the same CWE (e.g., 210 vs. 208 for CWE-22), but the table does not explain why; a brief note about failed API calls or output parsing would clarify this.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely question and has a solid real-world dataset, but the headline 'regardless of vulnerability type' claim is directly contradicted by the non-significant CWE-79 coefficients in Figure 4, and the synthetic experiment's confound between position and padding content is a genuine threat to the causal claim. I believe the authors can address these issues with revised claims and additional controls, so I recommend major revision rather than rejection. The replication package is a clear strength and should be emphasized during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports something worth knowing: off-the-shelf chat LLMs, when asked to find a vulnerability in a whole file, get worse as the file grows and as the bug sits closer to the end. That direction is consistent across a large real-world CVE-derived corpus and the paper's synthetic \"code-in-the-haystack\" experiment, and the chunking mitigation (RQ3) is a practical, cheap intervention practitioners can try. Adapting needle-in-the-haystack to code is a genuinely new protocol, and the replication package is public.\n\nThe soft spots are in the claims, not in the effort. The abstract says \"regardless of the vulnerability type ... p < .05\", but the paper's own Figure 4 shows the bug-position effect is non-significant for CWE-79 with gpt-3.5-turbo (p=0.897) and gpt-4o (p=0.123). That overstatement matters because CWE-79 is the most common type in the dataset. The \"regardless\" claim should be dropped or qualified.\n\nThe bigger problem is the causal experiment. In the code-in-the-haystack construction, padding is drawn from the same repository via a knapsack fill, so as the vulnerable block moves later, the set and order of functions placed before it change. Position is not manipulated independently of content. A model may be distracted by what the padding contains, not by distance from the file start. The paper's Section 8 acknowledges syntactic errors from the construction and waves them away; those errors are exactly the kind of artifact that could produce a position-correlated drop. The real-world regressions support the direction, but they carry the tailed-distribution confound the synthetic experiment was built to eliminate. So the causal \"lost-in-the-end\" claim is plausible but not established.\n\nRQ3's chunking gain is also selected on the same data used to report the +37% improvement: the best of four chunk sizes per model/CWE is chosen from the full set, so the number is an upper bound, not an estimate for new files.\n\nEven with these issues, the paper deserves a serious referee. The phenomenon is practically relevant, the dataset and package are reusable, and the flaws are fixable with a softened abstract, a control that varies position while holding padding content constant, and held-out chunk-size validation. I'd send it out, with instructions to the authors to substantiate or scale back the universality claim.","headline":"A real, practically relevant effect that the paper overclaims and whose causal experiment is confounded; still deserves a serious referee.","tokens_in":24225,"tokens_out":2283,"would_cite":false,"duration_ms":22206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that chat-based large language models systematically detect vulnerabilities less often when the vulnerable code sits near the end of a long file, a 'lost-in-the-end' effect that chunking can partially fix.","keywords":["lost-in-the-end","large language models","vulnerability localization","in-file vulnerability detection","code-in-the-haystack","input chunking","CWE-79","CWE-89"],"falsifier":"Run the same code-in-the-haystack protocol with a 25,000-character file and the same vulnerable block, but compare detection when the block is at the first 500 characters versus the last 500 characters; if detection probability at the end matches the beginning within sampling error across the six LLMs, the 'lost-in-the-end' effect is refuted.","tokens_in":23141,"feed_emoji":"🐛","tokens_out":5274,"duration_ms":50710,"temperature":0.7,"pith_summary":"This paper tries to establish that chat-based large language models such as GPT-3.5, GPT-4, GPT-4o, Llama 3, and the Mixtral models systematically miss software vulnerabilities that sit near the end of large source files, a pattern it names the 'lost-in-the-end' effect. The claim matters because developers increasingly paste entire files into chatbots for security review, and the effect held across the three most common and dangerous vulnerability classes studied: XSS (CWE-79), SQL injection (CWE-89), and path traversal (CWE-22). The paper reports statistically significant negative effects of file size and vulnerability position on detection probability, and shows in a controlled 'code-in-the-haystack' experiment that the effect is not an artifact of the skewed distribution of real-world bug positions. It then shows that splitting files into smaller chunks raises recall by over 37% on average, with optimal chunk sizes that vary by vulnerability type.","feed_headline":"AI code assistants miss bugs that sit near the end of long files","feed_subtitle":"Across six models and three top vulnerability types, detection drops in bigger files; chunking input restores recall.","key_machinery":"The load-bearing mechanism is the 'code-in-the-haystack' experiment: a vulnerable block (a function under 500 characters, or a refactored roughly 500-character segment containing the vulnerable line) is relocated to uniform positions in synthetic files of fixed sizes, with padding drawn from the same repository and placed by a relaxed 0/1 knapsack algorithm. This isolates vulnerability position and file size from confounds, and the resulting detection scores across positions are analyzed with logistic regressions. The character-based measurement of file size and bug position is the quantitative backbone, and the error analysis of false predictions shows that when models fail they report bug positions early in the file.","core_discovery":"The central discovery is that six off-the-shelf chat-based LLMs detect vulnerabilities in whole files reliably only when the vulnerable code appears early and the file is short; once the vulnerability is placed deeper into a longer file, detection probability falls, and false predictions cluster near the file's beginning. This 'lost-in-the-end' pattern contradicts the 'lost-in-the-middle' behavior often reported for LLMs on long natural-language contexts. The paper establishes it both on 794 real vulnerable files from the CVE catalog, using logistic regressions with p-values below .05 for all models and CWE types, and in a controlled experiment that moves the same vulnerable block to uniformly distributed positions in files of 4,000 to 25,000 characters. It also quantifies a threshold effect, for example ChatGPT 4, 4o, and Llama 3 struggle with CWE-89 once input exceeds about 4,000 characters, and it shows that naive chunking to smaller inputs substantially improves recall.","pith_inferences":["If the position-heuristic explanation is right, then fine-tuning or continued pretraining on files with uniformly distributed bug positions should shrink the 'lost-in-the-end' effect; this is a directly testable prediction the paper leaves open.","The character-based thresholds suggest a practical 'safe input size' chart could be built for other CWE types and future models using the same chunking search procedure, without retraining.","Because the effect appeared in all six models, it may stem from the pretraining data distribution rather than any single architecture; comparing models trained on code with different position statistics would separate data-driven from architecture-driven causes.","A natural next experiment is to reverse the file content while keeping the vulnerability's character offset, which would disentangle 'end of sequence' effects from content-level cues."],"forward_implications":["Practitioners who paste whole files into GPT-4 or similar chatbots will systematically under-detect vulnerabilities located in the latter part of large files, regardless of the vulnerability type.","Chunking files into smaller inputs is an immediate, no-fine-tuning mitigation: the paper measured an average recall gain of over 37%, with gains up to roughly 95% for CWE-79 under a 500-character chunk size.","The optimal chunk size is vulnerability-dependent: around 500 to 1,500 characters for CWE-79 and CWE-89, and up to 6,500 characters for CWE-22.","After chunking, open-source models such as Mixtral and Llama sometimes match or beat commercial models, suggesting the commercial advantage lies partly in handling longer contexts.","The same 'lost-in-the-end' limitation likely affects other LLM tasks that require reasoning over large files, such as code review, general bug localization, and code summarization."],"supporting_citations":[{"why":"Defines the 'lost-in-the-middle' phenomenon that this paper contrasts with its own 'lost-in-the-end' finding.","marker":"[24]"},{"why":"Shows that large input sizes hinder LLM reasoning, the general effect this paper extends to code and vulnerability localization.","marker":"[21]"},{"why":"Popularized the needle-in-the-haystack test that the code-in-the-haystack experiment adapts for vulnerability localization.","marker":"[11]"},{"why":"Supplies the relaxed 0/1 knapsack algorithm used to build padded files with controlled bug positions.","marker":"[37]"},{"why":"Provides the a priori power analysis used to determine the required sample sizes for the logistic regressions.","marker":"[13]"},{"why":"Supplies the logistic regression method used to estimate the effect of file size and bug position on detection probability.","marker":"[35]"},{"why":"Provides the odds-ratio effect-size benchmark (3.47) used in the power analysis for the controlled experiment.","marker":"[6]"},{"why":"Shows that LLMs are non-robust to small code changes, motivating the controlled manipulation of code position in this study.","marker":"[43]"},{"why":"Documents the long-tailed distribution of CWE types and bug positions that motivates the hypothesis about position-based heuristics in LLMs.","marker":"[50]"}],"fun_headline_variants":["LLMs lose track of bugs near end of long files","Lost in the end: AI misses flaws in file tails","File length hides end-of-file bugs from AI reviewers","AI code review fails on bugs at end of long inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The controlled 'code-in-the-haystack' result assumes that moving and refactoring a vulnerable block into files padded with unrelated code from the same repository leaves the vulnerability intact and keeps detection difficulty unchanged except for position.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lose track of bugs near end of long files","Lost in the end: AI misses flaws in file tails","File length hides end-of-file bugs from AI reviewers","AI code review fails on bugs at end of long inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1563,"prompt_tokens":1064,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":433}},"tokens_in":680,"tokens_out":499,"duration_ms":5696,"temperature":1.0,"reasoning_tokens":433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:22:05.901282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same code-in-the-haystack protocol with a 25,000-character file and the same vulnerable block, but compare detection when the block is at the first 500 characters versus the last 500 characters; if detection probability at the end matches the beginning within sampling error across the six LLMs, the 'lost-in-the-end' effect is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Popularized the needle-in-the-haystack test that the code-in-the-haystack experiment adapts for vulnerability localization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the relaxed 0/1 knapsack algorithm used to build padded files with controlled bug positions."},{"cited_title":"Coskun, and Gianluca Stringhini","cited_arxiv_id":null,"evidence_quote":"Shows that LLMs are non-robust to small code changes, motivating the controlled manipulation of code position in this study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the long-tailed distribution of CWE types and bug positions that motivates the hypothesis about position-based heuristics in LLMs."}],"review_version":1}