{"id":"17bb12cb-eac7-43f1-b301-c4908a60461b","arxiv_id":"2507.16685","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"VulGuard provides a unified, installable pipeline for mining commits, labeling vulnerability-inducing changes, and training/evaluating eight JIT vulnerability prediction models.","lead":"VulGuard is an open-source tool that automates building datasets and training/evaluating eight just-in-time vulnerability prediction models from GitHub commit histories. It is a practical contribution to reproducible benchmarking in software security, demonstrated on FFmpeg and the Linux kernel.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The default auto-annotation (Table I regex + V-SZZ, Sec. III-A3) is load-bearing: the paper concedes it 'may introduce noises,' but Tables IV-V rest on it, with no precision or ranking-robustness check. A manual audit of sampled labels and a manual-patch-list rerun would settle it.","rationale":"The central claim is that VulGuard is a reliable, end-to-end, installable pipeline for JIT-VP dataset construction and model evaluation, demonstrated on FFmpeg and Linux. For that claim to hold, the automated labeling - the step that turns raw git history into a labeled dataset - must be trustworthy, because every number in Tables IV and V and every downstream use of the tool inherits those labels. The reader identified this exact step as the weakest assumption, and the manuscript itself flags it: Section III-A3 states the Zhou et al. regex 'may introduce noises' and recommends a manual patch list. That self-report is in-scope evidence and it lands. The specific failure modes are concrete: security-keyword regexes match non-fixing commits (feature additions, hardening discussions, documentation) and miss fixes that do not mention security terms; and V-SZZ, like all SZZ variants, misattributes blame on renames, refactors, and merge-heavy histories, so a fix's blame can land on a benign commit that becomes a spurious positive. With a 0.9% positive rate in Table VI (~9,715 VICs among 1,081,882 commits), even low false-positive rates add hundreds of wrong labels to the training and test splits, and those wrong labels directly drive the reported PR-AUC, MCC, F1, and ROC values. I also note that the printed regex in Table I contains apparent typesetting errors ('bopen.redirect', 'privelege', and an unbalanced 'de)?serializ' that would make the pattern fail to parse). If the artifact contains the same string, the default mining command cannot run at all; if it differs, the paper does not precisely specify the regex used, which independently weakens reproducibility of the labeling step. Either way, the annotation step is where the paper is least secure. Credit where due: the paper is honest about the limitation, the artifact appears to be released with a CLI and documentation, the parallel mining design is sensible, and the tool supports user-supplied patch lists - the authors' own recommended mitigation. This is why the concern is not fatal: VulGuard can still serve as infrastructure if users supply curated patch lists. But the default automated path, the headline feature, is unvalidated, and the demonstration numbers are not shown to be robust to its noise. The reader's CONDITIONAL verdict is therefore the right call; the empirical illustrations should be read as indicative pending label-quality evidence. My proposed check - a sample-level precision audit plus a manual-list rerun - would reveal whether the noise actually changes model rankings. If rankings are stable, the concern evaporates; if they move, the tool's 'standardized benchmarking' claim needs qualification.","tokens_in":8412,"tokens_out":9365,"duration_ms":93115,"concrete_test":"Audit label precision: take 100 auto-labeled VICs per project (stratified across the test split of Table VI); for each, use the tool's own blame output to check whether the fixing commit's hunks actually touch code introduced by that VIC, with two raters and a reported Cohen's kappa. If precision is below ~0.8, the default labels are too noisy for 'standardized benchmarking'. Decisive follow-up: re-run a lightweight model (LR) on a 50k-commit FFmpeg slice labeled (a) by the default regex+V-SZZ and (b) by a manually curated patch list; if the eight-model ranking on PR-AUC or F1 changes by more than one position, the annotation noise propagates to the headline comparisons in Tables IV-V.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the automated commit annotation (Section III-A3): fixing commits are identified with the Zhou et al. regex (Table I), and vulnerability-inducing commits (VICs) are then traced with V-SZZ. The tool's central promise - reliable dataset construction for standardized JIT-VP benchmarking - holds only if these labels are trustworthy, because every downstream number (Tables IV, V) and every dataset published through the tool inherits the labels. Two known failure modes compound. (1) Security-keyword regexes are low-precision: words like 'attack', 'overflow', or 'insecure' appear in feature additions, refactoring, and security-hardening commits that are not fixes, and fixes without such words are missed; the printed regex also contains apparent typos ('bopen.redirect', 'privelege', an unbalanced 'de)?serializ' that would not parse), so even the paper's specification of the filter is unreliable. (2) V-SZZ misattributes blame on renames, refactors, and merge-heavy histories, turning a benign commit into a spurious VIC. The paper itself concedes the regex 'may introduce noises' and recommends a manual patch list - an admission that the default path is unvalidated. Given Table VI's 0.9% positive rate (9,715 VICs among 1,081,882 commits), even modest false-positive rates inject hundreds of wrong labels into splits, directly corrupting the reported PR-AUC/MCC/F1/ROC values. This is not an external disagreement; the manuscript flags the missing support but never quantifies the noise or shows model rankings are robust to it. The tool's manual-list option mitigates the concern for careful users, but the headline automated pipeline - and the demonstration built on it - remains unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VulGuard, an installable Python package and CLI that mines Git repositories, filters commits, labels vulnerability-inducing commits using a keyword regex plus V-SZZ, extracts features, and trains/evaluates eight JIT vulnerability prediction models. The authors demonstrate the pipeline on FFmpeg and the Linux kernel (1,081,882 commits) and report results under 'ideal' and 'realistic' settings. The engineering contribution is concrete and the release is referenced, but the empirical demonstration rests on an automated labeling step that the paper itself acknowledges may introduce noise, and the reported metrics are not accompanied by validation or error bars.","tokens_in":8734,"tokens_out":8293,"duration_ms":86197,"significance":"VulGuard addresses a real gap in JIT-VP research: end-to-end, reproducible dataset construction and model evaluation. If the label-quality concern is resolved, the tool would be a valuable community asset, with parallel extraction, support for eight models, a customizable pipeline, and large-scale demonstration data. The paper's honest reporting of the realistic-setting performance drop is also useful. However, the current manuscript does not establish that the default annotation path produces trustworthy labels, so the benchmark numbers in Tables IV and V cannot yet be taken as reliable.","major_comments":[{"comment":"The default annotation pipeline (Zhou et al. regex for fixing commits, then V-SZZ for vulnerability-inducing commits) is load-bearing for the paper's demonstration. The paper itself states that the regex 'may introduce noises' and recommends complementing it with a manual patch list, but every downstream result in Tables IV and V and the dataset statistics in Table VI are produced without that validation. With a 0.9% positive rate (9,715 VICs out of 1,081,882 commits, Table VI), even modest false-positive or false-negative rates from security-keyword matching or V-SZZ misattribution inject hundreds of wrong labels into the training and test splits, directly biasing the reported PR-AUC, MCC, F1, and ROC-AUC values. The manuscript should provide a manual audit of a random sample of labeled VICs (precision) and, ideally, a rerun with a manually verified patch list or an NVD/CVE-based ground truth to show that model rankings and absolute metrics are stable.","section":"III-A3, Tables IV-VI"},{"comment":"The printed regular expression is not a valid regex specification. In the medium-vulnerability row, the token 'de)?serializ' contains an unmatched closing parenthesis, which would cause a regex compiler error in standard engines such as Python re or grep -E, and the tokens 'bopen.redirect' and 'privelege' appear to be typos for 'open.redirect' and 'privilege'; similar issues include '\\bcross--origin\\b' with a double hyphen. Since Table I is the specification of the tool's default patch-commit filter, the paper must provide the exact, tested regex and confirm that the released code matches it. If the code contains the same typo, the filter silently fails to match intended patterns; if the code differs, the paper's specification is misleading.","section":"Table I"},{"comment":"The evaluation reports single-run metric values without error bars, seeds, or hyperparameter details. For deep learning models (DeepJIT, JITFine, SimCom), and especially for the near-degenerate Linux realistic results (e.g., MCC and F1 of 0.000 for DeepJIT and JITFine in Table V), the reader cannot determine whether the reported values are stable or artifacts of a single run. Since standardized evaluation is a central selling point of the tool, the paper should report multiple runs with variance or provide fixed seeds and full training configurations.","section":"V-B, Tables IV-V"}],"minor_comments":[{"comment":"'An Unified' is grammatically incorrect; change to 'A Unified'.","section":"Abstract and title"},{"comment":"'Vulnerability-neural commits' should be 'vulnerability-neutral commits'; the prose in Section V-A calls them 'security-unrelated changes', and the abbreviation 'VNC' is not defined in the table.","section":"Table VI"},{"comment":"The threshold-dependent and threshold-independent labels appear inconsistent; the caption says 'ED is threshold dependent' while the table rows are marked 'DE' and 'ID'. Please correct the abbreviations.","section":"Table III"},{"comment":"Report the hardware configuration used for the stated extraction times (about 1 hour for FFmpeg, 12 hours for Linux) so the timing claims are reproducible.","section":"V-A"},{"comment":"The example JSON for custom patch commits is not displayed with matched braces; format it consistently with the JSON example in Section IV-C.","section":"IV-B"},{"comment":"The realistic evaluation setting is inherited from the companion ICSME 2025 paper [7], but this manuscript does not define VNC commits or the exact construction of the ideal/realistic splits; either include the definitions or point to a freely accessible version of [7].","section":"III-A3 and V-B"}],"recommendation":"major_revision","confidential_remarks":"The main risk is label validity: the authors' own caveat about regex noise, combined with the invalid regex in Table I, means the paper cannot be accepted as is. The GitHub release should be checked for consistency with a corrected Table I. Also, the empirical results appear to be carried over from the companion ICSME 2025 paper; the tool paper should include enough independent validation to stand alone. This is a scope fit for a software engineering venue, and the engineering contribution is real, but the demonstration claims need strengthening before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe thing to know: VulGuard is a genuinely useful piece of infrastructure, and the authors are refreshingly honest about its weakest link. The automated commit-annotation step is noisy, and because every downstream number in the paper inherits that noise, the reported PR-AUC/MCC values should be treated as a demo, not a benchmark. But the tool's manual-patch-list option means you can still get real work done with it.\n\nWhat's new: no other tool I know packages the whole JIT-VP pipeline -- commit mining, feature extraction, SZZ-based labeling, data splitting, and training/evaluating eight published models -- behind a single CLI and Python API. Using V-SZZ instead of B-SZZ is the right call for vulnerability work. The realistic evaluation setting (including vulnerability-neutral commits) is a real improvement over the idealized settings most prior work uses, and the authors deserve credit for pushing that. The FFmpeg and Linux demonstrations show the tool scales to a million-plus commits.\n\nThe soft spots, in proportion: the default regex-based fixing-commit filter (Table I) is the load-bearing step, and it is exactly as shaky as you'd fear. The regex has apparent typos (e.g., 'bopen.redirect', 'privelege', an unbalanced 'de)?serializ'), and regexes like this are low-precision by nature. V-SZZ misattributes blame on renames and refactors. The paper concedes the noise but never quantifies it -- no precision/recall on labels, no manual audit, no robustness check of model rankings against label noise. With a 0.9% positive rate, even a few percent false positives inject thousands of wrong labels and directly corrupt the reported metrics. The companion ICSME paper [7] is the real empirical study; this submission is the tool description with a summary of those results. That is fine, but the demo numbers should not be read as independent validation of the models.\n\nNone of this is fatal. The tool is real, the code is out, and the manual patch-list path mitigates the labeling problem for careful users. What is missing is a validation of the default automated path, or a clear demotion of it to a convenience feature.\n\nBottom line: worth a serious referee. I would send it out with a request for a label-quality audit and a robustness analysis. Fix the regex, report precision/recall on a sample of mined labels, and show that model rankings hold up under label noise. Then it is a solid tool paper.\n\nBest.","headline":"VulGuard is a real, installable end-to-end JIT-VP pipeline, but its automated commit-labeling step is the load-bearing flaw that the demo numbers inherit; the authors admit the noise but never quantify it.","tokens_in":9312,"tokens_out":2737,"would_cite":true,"duration_ms":30426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VulGuard is an automated tool that carries out an entire just-in-time vulnerability prediction study, from raw git history to trained model scores.","keywords":["just-in-time vulnerability prediction","V-SZZ","commit mining","vulnerability datasets","reproducible benchmarking","deep learning vulnerability detection","continuous integration","FFmpeg"],"falsifier":"Take a repository with a manually verified list of vulnerability-fixing commits, run VulGuard's mining command once with the built-in regular expression and once with the manual list as the patch source, and compare the resulting vulnerability-inducing commit sets and the rankings of the eight models; poor agreement between the two sets, or a different model ranking under the manual list, would show that the automated annotation is the fragile step.","tokens_in":8229,"feed_emoji":"🛡️","tokens_out":7966,"duration_ms":79771,"temperature":0.7,"pith_summary":"This paper presents VulGuard, an installable tool that automates the entire just-in-time vulnerability prediction (JIT-VP) pipeline, from mining raw commit histories on GitHub through extracting features and labeling commits to training and evaluating prediction models. The tool is demonstrated on FFmpeg and the Linux kernel, processing 1,081,882 commits in roughly one hour and twelve hours respectively, using eight models ranging from logistic regression to graph-based and deep-learning approaches. The point of the exercise is reproducibility and scale: JIT-VP research is currently hampered by repository-specific, error-prone data curation, and VulGuard is an attempt to replace that with a standardized, command-line driven workflow that can also plug into CI/CD. The paper also reports that under a realistic evaluation setting that includes security-unrelated commits, all models perform substantially worse than under an idealized vulnerable-only setting.","feed_headline":"One pipeline mines 1M commits and tests 8 vulnerability models","feed_subtitle":"VulGuard automates the full just-in-time vulnerability prediction workflow, from raw git history to trained models.","key_machinery":"The load-bearing object is the commit-annotation chain at the center of the extraction module: a regular expression from prior work selects candidate vulnerability-fixing commits, and the V-SZZ algorithm traces those fixes back through blame information to the vulnerability-inducing commits that become the positive training labels. V-SZZ is an enhanced variant of the classic SZZ algorithm designed for vulnerability-inducing changes. Everything downstream—dataset balance, model training, and the ideal-versus-realistic comparison—depends on this chain producing trustworthy labels; the paper notes that the regular expression may introduce noise and recommends complementing it with a manual patch list.","core_discovery":"The central claim is that one tool can carry a complete JIT-VP study without hand-built data pipelines. VulGuard's extraction module filters merge, whitespace-only, and comment-only commits, keeps source files in the repository's primary language, extracts commit messages, diffs, blame data, expert features, and code property graphs in parallel, and serializes everything as JSONL. Its annotation step flags fixing commits with a regular expression drawn from prior work or with a user-provided patch list, then runs V-SZZ to trace each fix to the commits that introduced the vulnerability; the data are split chronologically to avoid leakage. The evaluation module trains and scores eight models—VCCFinder, CodeJIT, logistic regression, LAPredict, TLEL, DeepJIT, SimCom, and JITFine—and computes classification and effort-aware metrics. Applied to FFmpeg and the Linux kernel, the tool produces over one million labeled commits and reveals a consistent drop in every model's PR-AUC, MCC, F1, and ROC-AUC when moving from the idealized to the realistic setting.","pith_inferences":["If the regular-expression-based annotation is noisy, the size of the reported ideal-to-realistic gap could be partly an artifact of labeling errors rather than true model behavior; rerunning the pipeline with a manually verified patch list would separate the two effects.","The same pipeline could serve as the seed of a standardized JIT-VP benchmark: fixing the commit cutoff, the model implementations, and the chronological splitting rule would make cross-paper comparisons meaningful only when those settings are shared.","Because VulGuard also ships B-SZZ, AG-SZZ, and MA-SZZ, it can be used directly to compare defect-inducing versus vulnerability-inducing labels on the same commit history, which would test whether JIT-defect and JIT-vulnerability prediction are the same underlying problem.","The realistic Linux dataset contains well under one percent vulnerability-inducing positive commits among non-fixing commits, so cost-sensitive and class-imbalance techniques are a natural next testbed for these models."],"forward_implications":["Researchers can construct new JIT-VP datasets from any supported C/C++, Java, JavaScript, or Python repository using two command-line commands, and can substitute their own patch-commit list when the default regular expression is not trustworthy.","Eight published JIT prediction models can be trained, evaluated, and run for inference in a common framework, which makes direct comparison of machine-learning, deep-learning, ensemble, and graph-based model families possible without reimplementing each one.","Adopting the realistic evaluation setting, which includes security-unrelated commits, changes conclusions about model quality: every metric in the reported tables drops substantially relative to the idealized setting, so evaluations that omit unrelated commits are likely over-optimistic.","Because the tool outputs JSONL datasets and integrates a graph builder, the same mined data can be reused for other vulnerability analysis tasks such as fine-grained localization or message-based detection.","Trained models can be embedded in CI/CD workflows, so a developer can score a new commit at push time rather than in a batch research pipeline."],"supporting_citations":[{"why":"The companion empirical study whose idealized and realistic results are reproduced in Tables IV and V; it grounds the tool's demonstration.","marker":"[7]"},{"why":"Source of the expert/process features used by several implemented models.","marker":"[8]"},{"why":"Supplies the V-SZZ tracing algorithm that labels vulnerability-inducing commits.","marker":"[9]"},{"why":"Supplies the regular expression used by default to flag candidate fixing commits.","marker":"[24]"},{"why":"Defines VCCFinder, one of the eight implemented models, and contributes expert-feature practice.","marker":"[3]"},{"why":"Defines CodeJIT and supplies the code-property-graph artifacts the graph builder builds on.","marker":"[4]"},{"why":"Provides the replication codebase from which several model implementations, including DeepJIT, LR-JIT, and LAPredict, are drawn.","marker":"[11]"},{"why":"Defines DeepJIT, one of the eight implemented models.","marker":"[13]"},{"why":"Defines SimCom, one of the eight implemented models.","marker":"[17]"},{"why":"Defines JITFine, one of the eight implemented models.","marker":"[18]"}],"fun_headline_variants":["VulGuard: one tool for JIT vulnerability prediction","Automated JIT-VP: mine commits, test 8 models","Unified pipeline for vulnerability prediction research","From git history to 8 models in one tool","Scalable JIT-VP: 1M+ commits, 8 models, one tool"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire tool's output depends on the automated assignment of vulnerability labels: a regular expression picks fixing commits, and V-SZZ traces them back to the commits that supposedly introduced the vulnerability, so any systematic error in that chain makes the datasets and all model comparisons built on them unreliable.","fun_headline_variants_meta":{"raw":{"variants":["VulGuard: one tool for JIT vulnerability prediction","Automated JIT-VP: mine commits, test 8 models","Unified pipeline for vulnerability prediction research","From git history to 8 models in one tool","Scalable JIT-VP: 1M+ commits, 8 models, one tool"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000737,"raw_usage":{"total_tokens":3286,"prompt_tokens":932,"completion_tokens":2354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2281}},"tokens_in":548,"tokens_out":2354,"duration_ms":17740,"temperature":1.0,"reasoning_tokens":2281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:03:24.868009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a repository with a manually verified list of vulnerability-fixing commits, run VulGuard's mining command once with the built-in regular expression and once with the manual list as the patch source, and compare the resulting vulnerability-inducing commit sets and the rankings of the eight models; poor agreement between the two sets, or a different model ranking under the manual list, would show that the automated annotation is the fragile step.","supporting_citations":[{"cited_title":"A large-scale empirical study of just-in-time quality assurance,","cited_arxiv_id":null,"evidence_quote":"Source of the expert/process features used by several implemented models."},{"cited_title":"Toward realistic evaluations of just-in-time vulnerability pre- diction,","cited_arxiv_id":null,"evidence_quote":"The companion empirical study whose idealized and realistic results are reproduced in Tables IV and V; it grounds the tool's demonstration."},{"cited_title":"V-szz: automatic identifi- cation of version ranges affected by cve vulnerabilities,","cited_arxiv_id":null,"evidence_quote":"Supplies the V-SZZ tracing algorithm that labels vulnerability-inducing commits."},{"cited_title":"Automated identification of security issues from commit messages and bug reports,","cited_arxiv_id":null,"evidence_quote":"Supplies the regular expression used by default to flag candidate fixing commits."},{"cited_title":"Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits,","cited_arxiv_id":null,"evidence_quote":"Defines VCCFinder, one of the eight implemented models, and contributes expert-feature practice."},{"cited_title":"Code-centric learning-based just-in-time vulnerability detection,","cited_arxiv_id":null,"evidence_quote":"Defines CodeJIT and supplies the code-property-graph artifacts the graph builder builds on."},{"cited_title":"Deep just-in-time defect prediction: how far are we?","cited_arxiv_id":null,"evidence_quote":"Provides the replication codebase from which several model implementations, including DeepJIT, LR-JIT, and LAPredict, are drawn."},{"cited_title":"Deepjit: an end-to-end deep learning framework for just-in-time defect prediction,","cited_arxiv_id":null,"evidence_quote":"Defines DeepJIT, one of the eight implemented models."},{"cited_title":"Simple or complex? together for a more accurate just-in-time defect predictor,","cited_arxiv_id":null,"evidence_quote":"Defines SimCom, one of the eight implemented models."},{"cited_title":"The best of both worlds: integrating semantic features with expert features for defect prediction and localization,","cited_arxiv_id":null,"evidence_quote":"Defines JITFine, one of the eight implemented models."}],"review_version":1}