{"id":"23620efd-3c81-4207-8c2e-9775adf63493","arxiv_id":"2502.00064","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark study finds mixed sensitivity of LLM vulnerability detection to tokenized input length, with GPT-4 most robust, though chi-square results lack p-values and artifact release.","lead":"This paper tests whether the length of Java code, measured by an external tokenizer, correlates with how well ten large language models detect vulnerabilities. The authors report that GPT-4, Mistral, and Mixtral are unaffected by length, while other models degrade, but the statistical evidence is too thin to support these claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption is that the fixed 30k-vocab BPE tokenizer of Section III.E measures the context length each model actually sees; if per-model tokenizers rank the 140 files differently, the claimed null results for GPT-4, Mistral, and Mixtral are artifacts of the proxy, not findings…","rationale":"The reader correctly identifies the external-tokenizer proxy as the least secure assumption. The paper's own text in Section III.E concedes that inference is performed by each model's internal tokenizer while analysis uses a separate 30k BPE tokenizer. For the central claim to hold, the proxy must be a faithful ranking of the context length each model actually processes; the paper gives no evidence for that. A mismatch could produce exactly the observed pattern: GPT-4, Mistral, and Mixtral might appear robust because the proxy is uncorrelated with their internal length representation, while Gemma and Phi appear sensitive because the proxy happens to correlate with something else, such as file complexity or CWE type. No p-values, effect sizes, confidence intervals, or per-bin counts are reported, so the accepted nulls cannot be checked against power or sample-size limitations. The recommended check, re-running with per-model tokenizers and reporting rank correlations, is direct and inexpensive. If it confirms the pattern, the abstract's recommendation stands on stronger ground; if it flips entries, the paper's central claim is unsubstantiated. For these reasons the REJECT verdict is not changed by this stress-test pass; the paper needs the additional evidence before its headline conclusion can be accepted.","tokens_in":20,"tokens_out":5589,"duration_ms":109180,"concrete_test":"Re-tokenize the 140 Java files with each model's own official tokenizer (e.g., the Mistral-7B-v0.1 tokenizer, Mixtral tokenizer, Gemma/CodeGemma tokenizers, Phi tokenizer, Llama tokenizers, and tiktoken cl100k_base as a GPT-4 proxy) via HuggingFace Transformers or the vendor API, then re-run the chi-square tests from Sections IV-V. If any Table II accept/reject entries flip, especially for Mistral 32768, Mixtral 32768, or GPT-4, the external BPE proxy is the cause. Report Spearman rank correlations between the proxy BPE counts and each model's own token counts; if any correlation is below about 0.9, the proxy and the model's context usage are not measuring the same length construct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that GPT-4, Mistral at 32768 context, and Mixtral at 32768 context show no statistically significant relationship between tokenized Java code length and vulnerability detection accuracy/explicitness, while other models do (Table II and Section V). The independent variable for every chi-square test is the token count produced by a single externally fitted BPE tokenizer with a 30,000-token vocabulary (Section III.E). The paper explicitly states this tokenization is \"separate from the LLMs' detection process.\" That makes the measured construct a proxy, not the input length as represented inside each model. Different LLM families use different internal tokenizers, so ranking files by the external BPE count can diverge from ranking by the count the model actually consumes. If that divergence is large, an accepted null hypothesis says only that response categories are independent of the proxy tokenizer; it does not support the paper's conclusion that a model is robust to context length. The concern is amplified because all 140 files are vulnerable, accuracy is reduced to sensitivity, no p-values or effect sizes are reported, and dozens of chi-square tests are run at alpha=0.05 without correction. The robustness attributed to Mistral, Mixtral, and GPT-4 could therefore be an artifact of proxy-tokenizer mismatch, of low power, or of multiple-testing noise rather than genuine length invariance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether the tokenized length of Java source code affects the accuracy and explicitness of ten LLM families in vulnerability detection. The authors curate 140 vulnerable Java files from Vul4J, obtain responses from 39 model/quantization/context-window configurations, measure code length with a single external BPE tokenizer, and run chi-square tests of independence between binned token counts and two response-quality categories. They report which configurations accept or reject the two null hypotheses, concluding that GPT-4 and the large-context versions of Mistral and Mixtral are robust to input length, while other models are not.","tokens_in":8392,"tokens_out":4681,"duration_ms":46669,"significance":"If the findings were valid, they would be practically useful for selecting and configuring LLMs for vulnerability detection, and the broad multi-model comparison is a reasonable starting point. The paper also makes a concrete falsifiable claim about length-robustness that could inform deployment decisions. However, the current evidence does not support the central claim because the length variable is a proxy that is explicitly decoupled from the tokenization each model actually uses, and because the statistical reporting is too sparse to verify the accept/reject decisions. The comparative breadth and the clear statement of hypotheses are strengths, but the methodology needs substantial rework before the conclusions can be trusted.","major_comments":[{"comment":"The independent variable in every chi-square test is the token count produced by a single externally fitted BPE tokenizer with a 30,000-vocabulary (Section III.E), which the paper states is \"separate from the LLMs' detection process.\" This makes the measured construct a proxy, not the tokenized length that each model actually consumes. Different LLM families use different internal tokenizers, so the ranking of the 140 files by this proxy can diverge from the ranking by a model's own tokenizer. The central conclusion that GPT-4, Mistral at 32768 context, and Mixtral at 32768 context are robust to tokenized length is therefore unsupported unless the authors demonstrate that the proxy strongly correlates with each model's internal token count, or unless they re-run the analysis using each model's own tokenizer. This is the load-bearing issue for the paper's main claim.","section":"Section III.E and Section IV"},{"comment":"The paper reports only accept/reject decisions in Table II; it does not report chi-square statistics, p-values, degrees of freedom, effect sizes, or cell counts. With 140 files and roughly 39 separate tests at alpha=0.05, a reader cannot assess whether accepted nulls reflect genuine independence or low power, nor whether rejected nulls survive multiple-testing correction. The stated effect size (Cohen's w=0.3) and power (0.80) are not connected to the actual tests: no achieved power or minimum detectable effect is given, and the token-length binning scheme is never described, although bin boundaries determine degrees of freedom and expected counts. Please provide the full test statistics and a clear explanation of the binning procedure.","section":"Section IV and Table II"},{"comment":"All 140 dataset files are vulnerable (Section III.A and III.D), so the \"accuracy\" metric is a sensitivity/true-positive rate with no false-positive component. The null hypotheses are framed as being about \"vulnerability detection accuracy,\" but the experimental design cannot detect length effects on false-positive responses because no non-vulnerable files are presented. The robustness claims for GPT-4, Mistral, and Mixtral are therefore only about the ability to recognize known vulnerabilities in vulnerable files, not about overall detection accuracy. This limitation should be stated explicitly, or the dataset should include balanced negative samples.","section":"Section III.D and Section V"},{"comment":"Table I lists 39 configurations rather than 10 models, because several models appear with multiple quantization schemes and context windows. Treating each configuration as an independent test without accounting for the shared underlying model or the multiplicity of tests inflates the risk of false discoveries. The analysis should address this dependency structure, for example by grouping tests by model family or by applying a multiple-testing correction and reporting adjusted conclusions.","section":"Section III.B and Table I"}],"minor_comments":[{"comment":"The title contains spacing errors (\"V ariable\", \"T okenization\"), and the abstract contains typos such as \"damanges,\" \"an d,\" and \"t he\" that should be corrected before resubmission.","section":"Title and Abstract"},{"comment":"The BPE tokenizer's training corpus, vocabulary construction, and any preprocessing of Java code before tokenization are not specified. Please provide these details so that the token-count measure is reproducible.","section":"Section III.E"},{"comment":"The reported context window for CodeLLaMA 70B is 2048, which appears inconsistent with the text stating that CodeLLaMA handles sequences up to 100,000 tokens. Please clarify which context window was used and why the table lists 2048.","section":"Table I"},{"comment":"The statement that \"no studies have addressed factors influencing the quality of LLM responses\" is too broad given that several cited works, including Khare et al. [8] and Dong et al. [10], examine prompt and context-related factors. Please narrow the claim to the specific relationship studied in this paper.","section":"Section II"},{"comment":"The system prompt asks the model to answer \"yes or no,\" but the response categorization includes an \"Irrelevant Response\" category. Please explain how responses that do not follow the yes/no instruction were scored and whether such responses are counted as incorrect, inexplicitness, or both.","section":"Section III.C and Section III.D"},{"comment":"The recommendation that preprocessing should reduce token count while preserving code structure is not directly supported by the experiments, which only test whether length and performance are associated; they do not test whether reducing length improves performance. Please soften this recommendation or add evidence.","section":"Section VI"}],"recommendation":"major_revision","confidential_remarks":"The external-tokenizer issue is fundamental: if the authors cannot show that their proxy token count tracks each model's internal token count, the paper's central claim about which models are length-robust is not supported. I would encourage the editor to require a re-analysis with per-model tokenizers and full statistical reporting. If the re-analysis is not feasible, the paper should be rejected. The all-vulnerable dataset and lack of multiple-comparison control are additional validity concerns that the revision must address."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small empirical study with a genuinely relevant question—does tokenized input length affect LLM vulnerability detection?—but the central measurement leans on a proxy the paper itself admits is separate from each model's own tokenizer, and the statistical reporting is too thin to support the conclusions. I'd want major revision before trusting the robustness claims.\n\nWhat's actually good: the paper looks at Java rather than the usual C/C++, uses the Vul4J dataset, and compares ten model families across multiple context windows. That is a reasonable scope. The observation that some models (Gemma, Phi) show length sensitivity while others (GPT-4, Mistral, Mixtral at long contexts) don't is exactly the kind of practical signal practitioners want. The chi-square approach is straightforward but not wrong.\n\nThe soft spots are severe. The biggest is the unified BPE tokenizer with a 30k vocabulary. Every LLM has its own tokenizer, and the paper gives no evidence that the external token count ranks the 140 files the same way the model's real tokenizer does. Since that external count is the independent variable for every test, the accepted nulls for GPT-4, Mistral, and Mixtral could just mean responses are independent of that particular proxy—not that the models are robust to input length. The stress-test note lands.\n\nSecond, all 140 files are vulnerable. Accuracy is therefore sensitivity only; a model that says \"yes\" to everything scores perfectly. That misses false positives entirely and makes the accuracy-based robustness claims hard to interpret.\n\nThird, Table II reports only accept/reject. No p-values, no effect sizes despite Cohen's w being set, no confidence intervals, and no multiple-testing correction across the dozens of tests. With n=140, power is okay for a medium effect, but the binning scheme and tokenizer choice are free parameters that could flip results. The Mistral and Mixtral robustness conclusions rest on one row each.\n\nMinor: Table I lists CodeLLaMA 70B with a context window of 2048, probably a typo, and the abstract has typos that suggest haste.\n\nWho is this for? Researchers picking LLMs for Java vulnerability detection who want guidance on length sensitivity. The question deserves an answer, but this version doesn't provide it with enough rigor. If I were an editor, I'd send it to peer review because the topic is timely and the flaws are fixable—per-model tokenization or a validated proxy, a dataset with both vulnerable and clean files, full statistics, and ideally a pre-registered analysis. As written, I would not cite the robustness results.","headline":"A useful question undercut by a load-bearing proxy tokenizer and thin statistics; the robustness claims for GPT-4, Mistral, and Mixtral should not be trusted as written.","tokens_in":8890,"tokens_out":2508,"would_cite":false,"duration_ms":24808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models differ sharply in whether input code length affects their Java vulnerability verdicts, with GPT-4, Mistral, and Mixtral at full context showing no measured effect.","keywords":["large language models","vulnerability detection","Java","context window","tokenized code length","chi-square test","robustness evaluation","Byte Pair Encoding"],"falsifier":"Recompute the tokenized length of the 140 files with each model's native tokenizer and rerun the chi-square tests; if GPT-4 or Mistral then shows a significant length-performance relationship, the reported robustness is an artifact of the proxy tokenizer rather than a property of the models.","tokens_in":7881,"feed_emoji":"🛡️","tokens_out":8907,"duration_ms":73006,"temperature":0.7,"pith_summary":"The paper asks whether the length of tokenized Java code changes how accurately ten large language models detect vulnerabilities. Using a uniform Byte Pair Encoding tokenizer to measure each file's length and chi-square tests on 140 known-vulnerable Java files, it finds that GPT-4, and also Mistral and Mixtral when given their full 32,768-token context, show no statistically significant link between code length and either accuracy or explicitness. In contrast, Gemma, CodeGemma, Phi, and most LLaMA configurations show a significant relationship. The authors argue that a robust detector should be indifferent to input length, so models that accept the null hypotheses are preferable for Java vulnerability detection, and they recommend preprocessing that reduces token count for the length-sensitive models.","feed_headline":"GPT-4, Mistral, Mixtral judge Java bugs regardless of code length","feed_subtitle":"Ten-model chi-square study shows length-robustness is the exception, not the rule, in Java flaw detection.","key_machinery":"The measurement machinery is a unified Byte Pair Encoding tokenizer with a 30,000-token vocabulary, applied externally to compute every Java file's tokenized length while each LLM receives the raw code and uses its own internal tokenization. The statistical engine is the chi-square test of independence run under two null hypotheses: no relationship between tokenized length and vulnerability-identification accuracy, and no relationship between tokenized length and response explicitness. Responses are manually categorized as correct (explicitly vulnerable), incorrect (explicitly not vulnerable), or irrelevant, and the test compares these categories across token-length groups at a 0.05 significance level, 0.80 power, and a medium effect size of 0.3.","core_discovery":"The paper reports that model families have sharply different length-robustness profiles in Java vulnerability detection. At a significance level of 0.05 with power 0.8 and a medium effect size, GPT-4 accepts both null hypotheses, and Mistral and Mixtral accept both when the context window is set to 32,768 tokens, meaning no statistical relationship was detected between tokenized code length and accurate or explicit vulnerability verdicts. Most other configurations, including all Gemma variants, reject at least one hypothesis. The paper also reports that increasing parameter count alone does not guarantee robustness, that a larger context window helps only some models, and that higher-precision fp16 inference can improve length-independence for CodeGemma and Phi without resolving explicitness issues.","pith_inferences":["The unified BPE tokenizer is only a proxy for what each model actually consumes; re-measuring length with each model's native tokenizer could change which configurations accept or reject the null hypotheses.","Because the dataset contains only vulnerable files, 'accuracy' in this study is a true-positive rate, and the length-robustness of false-positive decisions (calling safe code vulnerable) is never tested.","The chi-square design compares length bins rather than a continuous dose-response; a regression-based analysis would indicate whether the length effect is monotonic and where a critical threshold lies.","The fact that Mistral and Mixtral become length-robust only at the largest context window suggests that context pressure, not just architecture, drives the effect, and that intermediate context sizes could be scanned to locate the transition."],"forward_implications":["For GPT-4, and for Mistral or Mixtral run at their full 32,768-token context, length-based preprocessing of Java code appears unnecessary for vulnerability detection accuracy and explicitness.","Running Mistral or Mixtral at a restricted 2,048-token context makes them length-sensitive, so context-window allocation is a practical lever for robustness in these open models.","Parameter count alone is not a reliable predictor of length-robustness; architecture and training choices, plus numerical precision for some families, matter more.","For length-sensitive models such as Gemma, CodeGemma, and Phi, reducing token count while preserving code structure is a plausible route to more consistent detection, as the paper explicitly recommends."],"supporting_citations":[{"why":"provides the 140 known-vulnerable Java files and ground-truth labels used for all accuracy and explicitness measurements.","marker":"[24]"},{"why":"defines the Byte Pair Encoding tokenizer used to compute the unified tokenized code-length measure.","marker":"[25]"},{"why":"is the prior LLM vulnerability-detection study whose prompt-based setup and multi-setting evaluation this paper extends.","marker":"[8]"},{"why":"is the earlier transformer-based vulnerability-detection framework that motivates applying LLMs to code.","marker":"[7]"}],"fun_headline_variants":["GPT-4, Mistral, Mixtral unaffected by Java code length","Length-robust LLMs rare in Java flaw detection","Token count sways only weaker LLMs on Java bugs","Context window size matters for most LLM bug checkers","Parameter count doesn't guarantee length-robust bug detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes that a single external BPE tokenizer with a 30,000-token vocabulary produces counts that faithfully track how much of each model's effective context the code occupies, even though every LLM uses its own internal tokenization.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4, Mistral, Mixtral unaffected by Java code length","Length-robust LLMs rare in Java flaw detection","Token count sways only weaker LLMs on Java bugs","Context window size matters for most LLM bug checkers","Parameter count doesn't guarantee length-robust bug detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2855,"prompt_tokens":787,"completion_tokens":2068,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":1983}},"tokens_in":403,"tokens_out":2068,"duration_ms":13541,"temperature":1.0,"reasoning_tokens":1983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:38:18.302861+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the tokenized length of the 140 files with each model's native tokenizer and rerun the chi-square tests; if GPT-4 or Mistral then shows a significant length-performance relationship, the reported robustness is an artifact of the proxy tokenizer rather than a property of the models.","supporting_citations":[{"cited_title":"A new algorithm for data compression,","cited_arxiv_id":null,"evidence_quote":"defines the Byte Pair Encoding tokenizer used to compute the unified tokenized code-length measure."}],"review_version":1}