{"id":"74e58f55-4320-4b19-8a3f-03d7db7c1d51","arxiv_id":"2504.16310","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A proposal for using LLMs to generate synthetic vulnerability review comments from security fix commits, with the intended but untested goal of improving automated code review models.","lead":"This preprint presents a research plan to build a synthetic dataset of security-focused code review comments using large language models and to fine-tune automated code review models on it. No experiments or results are included, and the paper reports only initial repository and commit counts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic reviews are generated from the fixing commit's diff, so high apparent quality may reflect answer leakage rather than transferable review skill; the planned test set is too small to establish improvement.","rationale":"The reader's verdict of UNVERDICTED is appropriate because the paper is a plan with no completed experiments or measured outcomes. The reader's weakest_assumption—that a fixing commit's diff and message suffice to reconstruct the prompting review—is real, and I partially agree with it. My concern sharpens it: the problem is not only that the reconstruction may be implausible; the prompt includes the post-fix diff, so the generated comments are conditioned on the solution itself. This is an answer-leakage / input-distribution mismatch that could make even human-plausible synthetic reviews poor training targets for the code-to-comment task, where the model must identify a vulnerability before any fix exists. The proposed concrete test would directly measure whether the apparent quality of the synthetic reviews depends on seeing the fixed diff. Because the paper is a future-work proposal, the correct status remains UNVERDICTED rather than ACCEPT or REJECT; the leakage concern is a reason to revise the generation design, but it does not convert an unverified plan into a verified negative result.","tokens_in":9616,"tokens_out":4789,"duration_ms":51899,"concrete_test":"Take a random sample of the 35,950 filtered vulnerability-fixing commits. For each commit, generate a review with the planned prompt (post-fix diff plus commit message) and with a blinded prompt that provides only the vulnerable pre-fix changed file, without the diff or commit message, using the same LLM. Have two authors score both outputs using the Section III.B.6 suitability criteria (coherent, addresses the vulnerability, could plausibly prompt the commit). If the blinded set's precision is substantially lower than the planned set's precision, the quality of the synthetic reviews is an artifact of leakage from the fixed diff, and fine-tuning on them will not transfer to real code review inputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the synthetic dataset will improve code-to-comment review models—requires that the generated synthetic reviews be usable training targets for the real review task. Section III.B.5 instructs the LLM to generate a review from the fixing commit's diff and commit message, meaning the correct fix is already visible in the prompt. The generated reviews are therefore reverse-engineered from the solution, not written from the reviewer's vantage point before the fix. This creates an input mismatch: fine-tuning inputs always include the post-fix version in the diff, so the model can learn to paraphrase the fix; at inference, a real submitted diff under review does not contain a fixed version, so comments that merely describe the fixed code will not transfer. The planned evaluation in Section III.C.3 compounds the risk: it relies on only 43 and 63 real security-related samples for the two tasks. The Threats to Validity section (Section V) acknowledges keyword-filtering and subjectivity concerns but does not identify this leakage, leaving the most load-bearing assumption unaddressed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a method for constructing a synthetic dataset of vulnerability-focused code review comments. The pipeline selects Java projects and single-file commits, filters commits with an iteratively refined security keyword list, prompts several LLMs (GPT-4o, Claude 3.5 Sonnet, Flan-T5, Qwen 2.5) with zero-shot, chain-of-thought, and self-reflection prompts to generate review comments from commit diffs and messages, and then evaluates the combinations manually. The resulting dataset would be used to fine-tune three code-to-comment models (CodeReviewer, a Lin et al. variant, and GPT-4o prompting) and evaluated on a filtered test subset of an existing real dataset using BLEU-4 and manual metrics. The paper reports only early findings: counts of candidate repositories (5,973), mined commits (3,827,517), keyword-filtered commits (43,131), and commits after test-file exclusion (35,950). It contains no generated synthetic reviews, no precision measurements, no fine-tuning experiments, and no evaluation on real review comments; RQ1 and RQ2 are posed but not answered.","tokens_in":9795,"tokens_out":4986,"duration_ms":52678,"significance":"If the proposed approach worked, it could address a recognized limitation in automated code review research: the scarcity of security-related review comments in existing training datasets. The methodology is thoughtfully structured, and the plan to validate generated reviews through independent manual assessment with Cohen's Kappa is commendable. The authors are transparent that their central claim is an anticipation. However, as a journal submission the manuscript is currently a research proposal rather than a completed empirical study, and no evidence is provided that synthetic reviews of this kind improve model precision. The approach also rests on a load-bearing assumption—that a vulnerability-fixing diff and its commit message contain enough information to reconstruct a plausible pre-fix review comment—which is plausible but untested and potentially problematic. The topic is relevant, but the current contribution is limited by the absence of results and by the unaddressed risk of answer leakage.","major_comments":[{"comment":"The paper presents no completed experiments: Section IV reports only repository and commit counts, and the abstract's statement that 'the synthetic dataset will improve the performance of the original code review models' is explicitly an anticipation. There are no generated synthetic reviews, no manual precision results, no fine-tuning runs, and no comparison on real review comments. Consequently, neither RQ1 nor RQ2 is answered, and the title's promise of 'improving automated secure code reviews' is not supported by any empirical evidence. This is a proposal, not a completed study.","section":"Section IV and overall"},{"comment":"The prompt design introduces a train/test leakage risk. The prompt provides the post-fix diff and commit message and asks for a review comment that 'could have led to making said commit,' meaning the corrected code is fully visible to the LLM. The generated comments can therefore be reverse-engineered paraphrases of the fix rather than diagnoses written from a reviewer's pre-fix perspective. When these comments are used as fine-tuning targets, the model may learn to reproduce or describe the fixed code; at inference, a submitted diff under review does not contain a fixed version, so such comments would not transfer. Section V's threats-to-validity discussion does not acknowledge this leakage. The authors should either condition generation on the pre-fix code, mask the fix, or otherwise demonstrate that the generated comments are not simply echoes of the corrected diff.","section":"Section III.B.5"},{"comment":"The planned real-world test set is too small to support a meaningful claim of improvement: the authors report only 43 and 63 potentially security-related samples for the code-to-comment and code & comment-to-code tasks, respectively. With samples in the tens, BLEU-4 differences and manual evaluation scores will have very wide confidence intervals, and the manual review of all generated outputs could easily be influenced by a handful of cases. The paper should either substantially enlarge the curated test set from additional sources, or provide confidence intervals, effect sizes, and an explicit analysis of how the small sample limits the conclusions.","section":"Section III.C.3"},{"comment":"The same security keyword list is used both to select the training commits (Section III.B.3) and to filter the real-world evaluation set (Section III.C.3). This creates a circularity risk: a model fine-tuned on comments generated from keyword-matched commits may only need to recognize those keywords to appear successful on a test set filtered by the same list, rather than to understand vulnerabilities generally. Additionally, the test set will not represent the full distribution of real security reviews, which often require context beyond a keyword match. The manuscript should use an independently constructed test set, report the degree of keyword overlap, and evaluate robustness to keyword choice.","section":"Section III.B.3 and Section III.C.3"}],"minor_comments":[{"comment":"There are language errors in the opening of the related work section, including 'This sections summarizes' and 'effectiveness of this models'; these should be corrected to 'This section' and 'these models.'","section":"Section II.A"},{"comment":"The related work on artificial dataset generation is thin and does not explicitly discuss prior attempts to generate synthetic code review comments or to reverse-engineer commits into review feedback. The novelty claim in Section II.C would be easier to assess if the authors compared their approach with existing data augmentation work for code review datasets.","section":"Section II.C"},{"comment":"The plan to retain only keywords with precision above 75% and to 'adjust this threshold depending on the number of commits remaining' is a potential source of selection bias. The manuscript should report the final keyword list, the precision of each keyword, and the effect of threshold changes on dataset size.","section":"Section III.B.3"},{"comment":"Since the contribution is a dataset, the paper should include a data availability statement with a repository or DOI for the generated dataset, the exact prompts, the model versions, and the manual evaluation annotations. None of these artifacts are currently provided.","section":"General"}],"recommendation":"reject","confidential_remarks":"This manuscript is a research proposal: it contains a plan, early commit-count findings, and no evaluation results. The central claim is an anticipation, and the one explicitly identified risk (answer leakage in the prompt design) is both plausible and unaddressed in the threats-to-validity section. The topic is suitable for a software engineering venue, and the authors' planned methodology has merit, but the paper as submitted does not meet the bar of a completed empirical study. If the authors carry out the experiments and address the leakage and test-size issues, a substantially revised version could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a proposal, not a research result. The paper describes a plan to generate synthetic vulnerability-focused code reviews by reverse-engineering fixing commits, and gives some early corpus counts. It is well written, honest about being a plan, and the idea is worth pursuing. But there is no dataset, no trained model, no evaluation, so the central claim is untested. For a normal peer-reviewed venue, I would desk reject it as incomplete.\n\nWhat is genuinely new: using vulnerability-fixing commits as the source for synthetic review comments is an angle I haven't seen. The methodology is laid out in reasonable detail, and the early numbers (35,950 candidate commits after filtering) suggest the raw material exists. The authors also acknowledge the subjectivity of their evaluation and the Java-only scope.\n\nThe soft spots are real. The generation prompt gives the LLM the post-fix diff and asks for a review that could have prompted that fix. That means the LLM sees the solution before writing the comment. Generated reviews will naturally describe what the patch changed, not how a reviewer would have spotted the flaw before the fix. Fine-tuning on those comments will teach a model to paraphrase diffs, and that skill will not transfer to real reviews where the fixed version isn't in the input. The stress-test note is right that this is a load-bearing issue, and the threats-to-validity section doesn't mention it.\n\nThe planned evaluation is thin: 43 and 63 real samples, and the test set is filtered by the same keyword list used to build the training data. That is not a reliable basis for claiming improvement. BLEU-4 is also a weak metric here, though the manual evaluation is welcome. None of this is fatal to the idea. A careful data-generation design that withholds the fix (e.g., generating from the pre-fix state or from the vulnerability alone) and a larger real test set would make the study meaningful.\n\nWho this is for: someone working on automated code review or synthetic training data may want to read it as a short position piece. I wouldn't cite it as a result.\n\nRecommendation: this is not ready for peer review as a full paper. It could be a workshop contribution or a \"research plan\" paper if the venue permits that. I would tell the authors to run the experiments and come back with the dataset and evaluation.","headline":"A well-written proposal for a synthetic vulnerability-review dataset, but it has no results yet and a treatable leakage risk in the generation design.","tokens_in":10300,"tokens_out":2581,"would_cite":false,"duration_ms":24258,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a pipeline that reverse-engineers vulnerability-fixing commits into synthetic security review comments and anticipates that fine-tuning code review models on them will improve their precision on real security-critical…","keywords":["code review automation","review comments","large language models","synthetic dataset generation","vulnerability detection","security-focused code review","fine-tuning","code-to-comment"],"falsifier":"The claim would be falsified if a model fine-tuned on the synthetic dataset performs no better than its untuned baseline on the paper's real, manually verified security review test set. A reader could check this by taking the filtered security-related review comments from the test partition of the existing large review dataset and comparing the two models under the paper's metrics, such as BLEU-4 and manual judgments of semantic equivalence and applicability.","tokens_in":9439,"feed_emoji":"🔐","tokens_out":7602,"duration_ms":66476,"temperature":0.7,"pith_summary":"The paper proposes a way to address the scarcity of security-focused code review data: mine vulnerability-fixing commits from mature open-source Java projects, then use large language models to write the review comment that could plausibly have prompted each commit. The central assertion, stated as an anticipation rather than a completed result, is that fine-tuning existing code review models on this synthetic dataset will improve their performance on real security reviews. The authors plan to test multiple LLMs and prompt strategies, filter the best combination by human review, generate a large dataset, and evaluate fine-tuned models on a small set of real security-related review comments. If the claim holds, automated code review tools could learn to flag security flaws despite the near-absence of real vulnerability comments in existing training data.","feed_headline":"Synthetic security reviews could sharpen automated code review","feed_subtitle":"A plan to turn vulnerability-fixing commits into training data for code review models that rarely see security flaws.","key_machinery":"The load-bearing object is the reverse-engineering prompt: a prompt that feeds an LLM the diff and commit message of a post-fix commit and asks it to write the review comment that would have led to that change. This construction matters because it converts one abundant type of data, vulnerability-fixing commits, into the scarce type of data, security-focused review comments, without manual annotation at scale. The prompt pipeline is paired with a security-keyword filter, a human precision screen over 1,200 generated comments (four LLMs across three prompt strategies on 100 sampled commits), and a fine-tuning comparison against baseline models on a real security review test set.","core_discovery":"The paper's central claim is that a vulnerability-fixing commit carries enough information to reverse-engineer the human review that triggered it. Given only the diff hunk and the commit message of a commit that patched a vulnerability, an LLM can produce a coherent review comment that names the flaw and could have prompted the fix. The authors argue this matters because real security-related review comments are a tiny fraction of available review datasets, and they report early evidence that the gap is large: in a dataset of 20,000 review comments, only 614 were security-related. Their proposed pipeline filters candidate commits by security keywords, refines the keyword list through two rounds of human sampling, screens generated comments for coherence and plausibility, and then fine-tunes three existing code review models on the best-generated corpus. The anticipated outcome is that these fine-tuned models achieve higher-precision vulnerability comments on real, unseen security reviews than the original models.","pith_inferences":["The transfer claim could be tested more sharply on the subset of vulnerability-fixing commits that were actually preceded by a review discussion: if the synthetic comment differs systematically from the real review that prompted the fix, the reverse-engineering premise is weaker than the paper assumes.","If the generated comments mostly identify the vulnerability class rather than the vulnerable line or the exact condition, fine-tuning may improve topic-level relevance without improving line-level localization that reviewers need.","The keyword-filtered real test set may share the blind spots of the keyword list, so measured transfer could overstate real-world performance; a stronger test would include vulnerabilities found by independent security reports.","The method's success likely depends on commit-message quality; commits with terse or generic messages may produce plausible-sounding but disconnected reviews, which suggests a cheap quality gate: compare synthetic comments against those generated from randomized message text."],"forward_implications":["If fine-tuning works, automated code review systems could gain a capability they currently lack: reliably commenting on security-critical flaws rather than general code quality issues.","The pipeline would turn one of the most common forms of software data, commit histories, into scarce review labels, reducing dependence on manually collected security review datasets.","The same reverse-engineering approach could be retargeted to other underrepresented review types, such as performance or accessibility concerns, wherever commits encode the fix.","The planned real-world evaluation would provide the first direct estimate of how much LLM-generated security review data transfers to genuine reviewer comments.","The two-stage selection process, human screening of 1,200 generated comments before mass generation, offers a reusable recipe for choosing an LLM and prompt strategy for synthetic dataset construction."],"supporting_citations":[{"why":"Establishes the motivating scarcity: only 614 of 20,000 review comments were security-related.","marker":"[8]"},{"why":"Supplies the main baseline code review model and the large dataset whose test partition is used for real-world evaluation.","marker":"[10]"},{"why":"Provides the pre-training and fine-tuning approach for code review models that the paper adapts.","marker":"[11]"},{"why":"Supplies an improved fine-tuned baseline and the manual evaluation metrics of semantic equivalence and applicability.","marker":"[12]"},{"why":"Provides the criteria for selecting mature Java repositories and the dataset tuple structure used for review tasks.","marker":"[13]"},{"why":"Supplies the initial security keyword list and the iterative refinement methodology for filtering vulnerability-related commits.","marker":"[23]"},{"why":"Supplies the chain-of-thought prompting technique tested as one of the three prompt strategies.","marker":"[25]"},{"why":"Supplies the self-reflection prompting technique tested as one of the three prompt strategies.","marker":"[26]"}],"fun_headline_variants":["Synthetic security comments train code review models to spot flaws","AI-generated security reviews fill a critical gap in code review training","From vulnerability-fixing commits to security review training data","LLMs turn vulnerability fixes into synthetic review comments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that a vulnerability-fixing commit's diff and commit message contain enough information to reconstruct the review comment that would have prompted the commit, so synthetic comments can stand in for real reviewer feedback.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic security comments train code review models to spot flaws","AI-generated security reviews fill a critical gap in code review training","From vulnerability-fixing commits to security review training data","LLMs turn vulnerability fixes into synthetic review comments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2714,"prompt_tokens":922,"completion_tokens":1792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1727}},"tokens_in":538,"tokens_out":1792,"duration_ms":11826,"temperature":1.0,"reasoning_tokens":1727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:05:34.778538+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The claim would be falsified if a model fine-tuned on the synthetic dataset performs no better than its untuned baseline on the paper's real, manually verified security review test set. A reader could check this by taking the filtered security-related review comments from the test partition of the existing large review dataset and comparing the two models under the paper's metrics, such as BLEU-4 and manual judgments of semantic equivalence and applicability.","supporting_citations":[{"cited_title":"Empirical analysis of security-related cod e reviews in npm packages,","cited_arxiv_id":null,"evidence_quote":"Supplies the initial security keyword list and the iterative refinement methodology for filtering vulnerability-related commits."},{"cited_title":"Chain-of-thought prompting elicits r easoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought prompting technique tested as one of the three prompt strategies."}],"review_version":1}