{"id":"8d56d1c0-cd2d-48ff-8843-b241735d5416","arxiv_id":"2504.18784","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuned open-source LLMs classify regex-extracted secret candidates with 0.985 binary F1 and 0.982 multiclass F1 on the SecretBench benchmark.","lead":"This paper combines regex-based candidate extraction with large language model classification to detect leaked API keys and credentials in source code. Fine-tuned open models reach F1 scores above 0.98 on a GitHub benchmark, suggesting locally deployable secret scanners are feasible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Candidate-level random split leaks same-repository examples into training and test; reported F1 may reflect memorization of file-level patterns rather than generalization to unseen codebases.","rationale":"The paper is a solid empirical study: it provides a replication package, reports per-class and per-model metrics, and the fine-tuning gains are large and consistent. The central claim, however, is generalization of a fine-tuned detector to source code in the wild. That claim rests on the test set being representative of unseen data. The weakest point is the split: Section III-C3 creates train/validation/test from one 45,000-candidate pool by random candidate sampling, with no repository or file-level separation. With only 818 source repositories and 200-character overlapping contexts, the test set almost certainly contains candidates from repositories and files that contributed training examples. This allows the model to exploit file/repo-level regularities, so Table V's F1=0.9852 may not transfer to new codebases. The Section VI contamination paragraph only addresses pretraining exposure, not this split leakage. The fix is straightforward: re-run with repository-stratified folds and report held-out F1. This does not invalidate the paper; it makes the reported number conditional on an evaluation design that matches the deployment scenario. Hence the reader's conditional verdict stands.","tokens_in":15954,"tokens_out":5562,"duration_ms":60168,"concrete_test":"Run a repository-stratified evaluation: split the 818 SecretBench repositories (using the file path or repository metadata) into 5 folds; for each fold, train the LLaMA-3.1-8B model on candidates from the other 4 folds and evaluate on held-out repositories, keeping the same class balance as Section III-C3. Compare the mean and standard deviation of held-out F1/F2 to Table V's 0.9852/0.9855. If held-out F1 drops materially (e.g., >0.02) or varies widely across folds, the candidate-level split inflated the headline result; if it stays within ~0.01, the leakage concern is minor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The binary and multiclass results in Tables V and VI are obtained by randomly partitioning a 45,000-candidate pool (Section III-C3) without grouping by repository or file. SecretBench's 97,479 candidates come from only 818 repositories, so with 24,000 training and 3,000 test examples drawn from the same pool, many test candidates almost certainly share a file or repository with training candidates. Since each input is the candidate plus a 200-character surrounding context window, such shared-origin examples are near-duplicates; the model can memorize file-level and repo-level cues or even repeated candidate strings rather than learning transferable detection. The paper's own threat section (Section VI) discusses LLM pretraining contamination but never addresses this train/test split leakage. Therefore the headline F1=0.9852 as evidence that a fine-tuned 8B model detects secrets in unseen code is not yet established; the claim is conditional on a held-out-repository evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid secret-detection pipeline that first extracts candidate strings with regular expressions and then classifies them with large language models (LLMs). Using the SecretBench dataset (97,479 candidates from 818 GitHub repositories), the authors curate a balanced subset, evaluate pretrained GPT-4o and DeepSeek-V3 under zero-, one-, and few-shot prompting, and fine-tune LLaMA-3.1-8B, Mistral-7B, CodeLLaMA-7B, Gemma-7B, and DeepSeek-7B with QLoRA. They report that fine-tuned LLaMA-3.1-8B reaches an F1-score of 0.9852 in binary classification and that Mistral-7B reaches a weighted F1-score of 0.9820 in multiclass classification, arguing that small open models are accurate enough for local deployment.","tokens_in":16194,"tokens_out":5693,"duration_ms":62162,"significance":"If the reported results generalize beyond the specific benchmark split, the work would be a practically useful demonstration that small, locally deployable LLMs can filter regex-extracted secret candidates with high precision. The paper has several strengths: it provides a replication package, describes hyperparameters and resource usage in detail, evaluates multiple models and prompting strategies, and reports both binary and multiclass results with F1 and F2 metrics. The main caveat is that the headline numbers rest on a candidate-level random split, which does not yet establish performance on unseen codebases.","major_comments":[{"comment":"The random candidate-level split does not provide a valid estimate of generalization to unseen code. SecretBench contains 97,479 candidates from only 818 repositories, and the paper forms 24,000 training, 3,000 validation, and 3,000 test samples by randomly partitioning the curated 45,000-item pool without grouping by repository or file. Since each input includes a 200-character context window around the candidate, test candidates sharing a file or repository with training candidates are near-duplicates, and the model can memorize file-level or repository-level cues rather than learning transferable signals. The Section VI internal-validity discussion addresses LLM pretraining contamination but does not mention this split leakage. Please re-run the evaluation with a repository-level or file-level split and report the resulting cross-repo F1; if a full re-run is infeasible, at minimum quantify how many test candidates share a file or repository with training candidates and report metrics on the non-overlapping subset.","section":"Section III-C3, Tables V and VI"},{"comment":"The claim of outperforming regex-only baselines is supported only by a theoretical 'flag all candidates' model, which by construction has 50% precision on the balanced 3,000-sample test set and was not actually run. Real tools such as Gitleaks and TruffleHog apply their own candidate-filtering rules and would not necessarily flag every candidate, so the stated comparison is not an empirical tool benchmark. Please run Gitleaks, TruffleHog, or the SecretBench regex set on the same test candidates and report their precision, recall, and F1, or alternatively qualify the claim as an improvement over an all-candidates baseline.","section":"Section V-B and abstract"},{"comment":"The data curation description leaves the exact relationship among subsets unclear. The paper lists a balanced train set, an imbalanced train set, a validation set, and a test set drawn from the same 45,000-candidate pool, and later states that the test set is the same as the one used in Section III-B. The reader cannot determine whether the imbalanced and balanced training sets overlap with each other or with the validation/test sets, nor whether multiclass training and test samples are disjoint by repository or file. Please state the exact composition and overlap of all subsets used in each experiment.","section":"Section III-A2"}],"minor_comments":[{"comment":"The abstract says 'Mistral-7B reached 0.982 accuracy,' but Table VI reports a weighted F1-score, not accuracy; please use the correct term for consistency.","section":"Abstract"},{"comment":"There is a garbled sentence: 'we evaluated GPT-4o [28] and Deepseek-V3 [29] on detecting secrets as and experimented' and an earlier phrase 'have shown remarkable performance have demonstrated'; these need copyediting.","section":"Section III-B2"},{"comment":"The per-class counts in Table I sum to 15,075, not the 15,084 true secrets stated in Section III-A1; please reconcile the totals.","section":"Table I"},{"comment":"Table X should state explicitly whether inference time is per candidate, per file, or per batch, and on which hardware it was measured.","section":"Table X"},{"comment":"The paper says multiclass models are evaluated on the same 3,000-sample test set used for the pretrained models, but Section III-A2 indicates that only positive samples are used for multiclass classification; please clarify whether this test set contains only positive samples or also includes negatives.","section":"Section III-C4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that after moving to a repository-level split, the headline F1 values may drop substantially. I recommend requesting that experiment explicitly before acceptance; the paper is otherwise within scope for an empirical software-engineering venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: Fine-tuning open LLMs on SecretBench is a useful empirical exercise, and the resource-efficiency data are genuinely practical, but the headline F1=0.9852 is not a generalization result because the train/test split leaks same-repository examples. The number is real but its meaning is limited.\n\nWhat's new: they fine-tune five 7B/8B models (LLaMA-3.1, Mistral, CodeLLaMA, Gemma, DeepSeek) on SecretBench, add a multiclass secret-type taxonomy, and run a balanced-vs-imbalanced training comparison. The hybrid regex-extract-then-LLM-classify pipeline is a natural extension of Saha et al./Feng et al./Wahab et al., but the scale and the local-deployment focus are new. The replication package is there, and the training/inference VRAM and latency tables are the kind of concrete data teams ask for. Citation pattern is fine—they credit SecretBench and the prior ML detectors they build on.\n\nWhere it hurts: the split is by candidate, not by repository or file. SecretBench has 97,479 candidates from 818 repos, so the 3,000 test samples almost certainly share files/repos with the 24,000 training samples. With a 200-character context window, many test inputs are near-duplicates of training inputs. The paper's own threat section talks about LLM pretraining contamination but never addresses this. So the F1=0.9852 should be read as in-benchmark performance, not evidence the model detects secrets in unseen codebases. Also, the only baseline is a theoretical 'flag all candidates' 50% precision; they never run Gitleaks or TruffleHog to show the improvement over actual tools. No error bars or repeated runs either. The multiclass part trains and tests only on true secrets, so it measures type labeling rather than false-positive rejection.\n\nThe paper is readable and honest in places—the false-negative analysis, the context-window sensitivity, and the imbalanced-training robustness check are worthwhile. The fix is straightforward: redo the split at repo or file level, run real tools as baselines, and report variance across seeds. If the high F1 survives a repo-level split, this is a solid practical contribution.\n\nWho it's for: software engineering and security researchers working on secret scanning, plus anyone choosing between local fine-tuned LLMs and commercial APIs for this task. It deserves a serious referee, but the current version needs major revision before the central claim can be accepted.","headline":"Fine-tuned open LLMs on SecretBench is a solid empirical exercise, but the headline F1 is inflated by candidate-level train/test leakage.","tokens_in":16663,"tokens_out":4274,"would_cite":false,"duration_ms":36291,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned small LLMs detect hardcoded secrets at 0.985 F1, outperforming regex-only scanning.","keywords":["secret detection","large language models","source code security","fine-tuning","regex","false positives","multiclass classification","local deployment"],"falsifier":"Split the dataset by repository instead of by individual candidate: train on a random 80% of repositories and test on the remaining 20%. If the fine-tuned model's F1 drops materially below 0.9852 on repositories it never saw, the claim of generalization to unseen codebases is not supported; an even cleaner check is to run the same model on freshly generated secrets that postdate the benchmark.","tokens_in":15808,"feed_emoji":"🔑","tokens_out":8397,"duration_ms":74892,"temperature":0.7,"pith_summary":"This paper claims that a hybrid pipeline—regex-based candidate extraction followed by large-language-model classification—can find hardcoded secrets in source code with far fewer false alarms than regex or entropy tools alone. On a balanced test set drawn from 818 public code repositories, fine-tuning an 8-billion-parameter open model for binary secret detection reaches an F1-score of 0.9852, and a 7-billion-parameter model reaches a weighted F1 of 0.982 for classifying secret types. The paper also argues that these small open models are accurate enough to replace commercial API calls, running in under 13 GB of inference memory on local hardware. If true, secret scanning could be embedded in development workflows without shipping code to third parties.","feed_headline":"Fine-tuned small LLMs detect hardcoded secrets at 0.985 F1","feed_subtitle":"Near-perfect F1 means fewer false alarms and safe on-premise scanning, with no code sent to commercial APIs.","key_machinery":"The load-bearing mechanism is the context window: each regex-extracted candidate is paired with 200 characters of surrounding source code, and the pair is rendered into a structured prompt asking for 'Secret' or 'Non-sensitive', or for the secret type. This context lets the model treat a string like 'sk_test_...' as sensitive when it appears in a live API call but ignore 'xxxxxxxxxx' as a placeholder. Fine-tuning is done with quantized low-rank adapters (QLoRA-style parameter-efficient tuning) on the small open models, training attention layers on the balanced candidate set; that is what lifts raw zero-shot F1 scores from roughly 0.3–0.5 to above 0.94 across all five models tested. The comparison point is a regex-only tool, which would call every extracted candidate a secret and therefore score at most 50% precision on the balanced test set.","core_discovery":"The central discovery is that contextual language understanding, not better pattern matching, is what removes false positives in secret detection. A regex pass first proposes candidate strings; then a model reads a 200-character window of surrounding code and decides whether the candidate is a real secret. After parameter-efficient fine-tuning, the best small model correctly labels 1,496 of 1,500 non-secrets and 1,463 of 1,500 secrets, yielding the reported F1 of 0.9852, while raw prompted models stay well below that. For multiclass labeling, a fine-tuned 7-billion-parameter model reaches a weighted F1 of 0.982, with near-perfect scores on private keys, API keys, and authentication tokens, and weaker but strong scores on rare categories. The authors interpret this as evidence that fine-tuned open models are a practical middle path: more accurate than regex, cheaper and more private than commercial APIs.","pith_inferences":["Editorial inference: A repository-level train/test split would be the natural stress test; if performance drops sharply when no file from the test repository appears in training, the reported F1 is partly a memorization effect rather than generalization to unseen codebases.","Editorial inference: The same hybrid could be extended beyond code to configuration files, CI logs, and issue trackers, since the model only needs a candidate string plus local context.","Editorial inference: If deployed, a model that rejects low-confidence candidates could cut the paper's 37 false negatives at some false-positive cost, trading alert noise against missed secrets in either direction.","Editorial inference: The near-perfect multiclass scores on rare categories suggest the benchmark's label boundaries, not the model, may be the main source of residual errors; a cleaned relabeling study would show how much headroom remains."],"forward_implications":["Secret scanning can run on a local workstation: training stayed under 17 GB of GPU memory and inference under 13 GB, so CI/CD pipelines and IDEs could scan without sending code to commercial APIs.","Multiclass labels give security teams the secret type (private key, API key, token, password), so a leak can be routed directly to the right revocation action.","Because fine-tuning improves every tested model and context windows of 200 characters already suffice, the approach is practical on modest hardware rather than requiring frontier models.","A balanced training set and a skewed, realistic training set produced nearly the same scores, suggesting the method survives the imbalanced conditions of real codebases."],"supporting_citations":[{"why":"Supplies the labeled candidate-secret dataset from 818 public repositories used for training and evaluation.","marker":"[23]"},{"why":"Represents the regex-based scanner whose candidates the LLM classifier filters, and the baseline for precision.","marker":"[8]"},{"why":"Provides the earlier machine-learning filter (86.7% F1) that the hybrid approach claims to surpass.","marker":"[17]"},{"why":"Documents false-positive rates in existing secret detection tools, motivating the contextual LLM step.","marker":"[14]"},{"why":"Provides quantized low-rank fine-tuning, the method that makes 7B-8B model training fit on one consumer GPU.","marker":"[37]"},{"why":"Establishes the large-scale regex-plus-entropy detection baseline whose precision ceiling the paper aims to exceed.","marker":"[15]"}],"fun_headline_variants":["Fine-tuned LLMs reduce secret-scan false positives to near zero","Hybrid regex-LLM approach hits 0.985 F1 on secret detection","Open-source LLMs offer secure, local secret detection at scale","Context-aware LLMs beat regex for spotting hardcoded secrets","0.985 F1: Fine-tuned LLMs excel at secret breach detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported near-perfect scores assume the test candidates have not been effectively seen during training: the train/test split is random at the candidate level, so strings from the same repository or file can appear on both sides, and the benchmark itself is drawn from public code that the models may have seen during pretraining.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLMs reduce secret-scan false positives to near zero","Hybrid regex-LLM approach hits 0.985 F1 on secret detection","Open-source LLMs offer secure, local secret detection at scale","Context-aware LLMs beat regex for spotting hardcoded secrets","0.985 F1: Fine-tuned LLMs excel at secret breach detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000236,"raw_usage":{"total_tokens":1514,"prompt_tokens":967,"completion_tokens":547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":450}},"tokens_in":583,"tokens_out":547,"duration_ms":4814,"temperature":1.0,"reasoning_tokens":450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:08:51.099693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the dataset by repository instead of by individual candidate: train on a random 80% of repositories and test on the remaining 20%. If the fine-tuned model's F1 drops materially below 0.9852 on repositories it never saw, the claim of generalization to unseen codebases is not supported; an even cleaner check is to run the same model on freshly generated secrets that postdate the benchmark.","supporting_citations":[{"cited_title":"(2016) Trufflehog","cited_arxiv_id":null,"evidence_quote":"Represents the regex-based scanner whose candidates the LLM classifier filters, and the baseline for precision."},{"cited_title":"Secrets in source code: Reducing false positives using machine learning,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier machine-learning filter (86.7% F1) that the hybrid approach claims to surpass."},{"cited_title":"A comparative study of software secrets reporting by secret detection tools,","cited_arxiv_id":null,"evidence_quote":"Documents false-positive rates in existing secret detection tools, motivating the contextual LLM step."},{"cited_title":"How bad can it git? characterizing secret leakage in public github repositories","cited_arxiv_id":null,"evidence_quote":"Establishes the large-scale regex-plus-entropy detection baseline whose precision ceiling the paper aims to exceed."}],"review_version":1}