{"id":"d4d4ba39-1ad9-4208-8b41-b8aaa039cf09","arxiv_id":"2606.19814","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"CoRaCommit retrieves similar commit exemplars to augment LLM prompts for generating commit messages in VS Code and outperforms prior extensions on BLEU, CIDEr, METEOR, and ROUGE-L using 945 ApacheCM commits.","lead":"CoRaCommit is a VS Code extension that retrieves similar past commit messages to help large language models generate new ones, while running multiple models in parallel and learning from user feedback. A smart generalist might read it to see how retrieval techniques can make AI coding assistants more practical for everyday developer workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Automatic metrics alone do not establish that retrieval-augmented messages are more useful to developers","rationale":"The reader's weakest_assumption directly identifies the same gap. Because the work is framed as a practical extension rather than a pure metric-improvement paper, the absence of any human-grounded validation leaves the central demonstration under-supported. A conditional verdict is appropriate until such validation is supplied; the low-confidence abstract-only review does not change this assessment.","tokens_in":1710,"tokens_out":354,"duration_ms":14714,"concrete_test":"Run a blinded pairwise preference study with ≥10 experienced developers on a random sample of 50 test commits; present CoRaCommit output vs. the strongest baseline output and ask raters to choose which better describes the change intent (or rate 1-5 on accuracy/usefulness). Compute win rate and statistical significance; if no reliable preference for CoRaCommit, the effectiveness claim does not follow from the metric results.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim is that CoRaCommit outperforms existing VS Code extensions on BLEU/CIDEr/METEOR/ROUGE-L over 945 ApacheCM commits, thereby demonstrating the effectiveness of exemplar retrieval. This rests on the untested premise that gains in these surface-level overlap metrics correspond to commit messages that better capture developer intent or are preferred in practice. The abstract (and therefore the reported experiment) contains no human evaluation, no developer preference study, and no correlation analysis between the automatic scores and actual usefulness. For a tool paper whose value proposition is improved commit messages inside an IDE, this proxy assumption is load-bearing: metric superiority can be achieved by stylistic mimicry without improving semantic accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents CoRaCommit, a VS Code extension for commit message generation. It retrieves similar commit exemplars to provide context in LLM prompts, invokes multiple LLMs in parallel for candidate comparison, and dynamically recommends LLMs based on user feedback. The central claim is that on 945 commits from the ApacheCM dataset, CoRaCommit outperforms existing VS Code extensions on BLEU, CIDEr, METEOR, and ROUGE-L metrics, demonstrating the effectiveness of retrieval-augmented context.","tokens_in":1832,"tokens_out":414,"duration_ms":39645,"significance":"If the results hold with proper validation, the work could contribute a practical IDE tool for improving commit message quality in software development. The deployment as a VS Code extension and the use of exemplar retrieval are strengths for applicability. However, the significance is reduced because the evaluation relies solely on automatic metrics without evidence that improvements correspond to messages developers find more useful or accurate.","major_comments":[{"comment":"Evaluation section: The abstract claims outperformance on BLEU, CIDEr, METEOR, and ROUGE-L over 945 ApacheCM commits but supplies no information on baseline implementations, statistical tests, dataset splits, or human validation. This makes the central empirical result impossible to verify and is load-bearing for the claim of effectiveness.","section":"Evaluation"},{"comment":"Evaluation section: The paper provides no human evaluation, developer preference study, or correlation analysis between the automatic metric scores and actual usefulness or accuracy of the generated commit messages. This proxy assumption is load-bearing for a tool paper whose value proposition is improved commit messages inside an IDE.","section":"Evaluation"}],"minor_comments":[{"comment":"The abstract could more explicitly name the existing VS Code extensions used as baselines.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads primarily as a tool demonstration paper; the journal should assess whether the level of empirical rigor meets cs.SE standards for claims of outperformance."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed feedback on the evaluation aspects of our work. We address the two major comments point-by-point below, agreeing where revisions are needed to improve verifiability while providing the strongest honest defense of the manuscript's current contributions and scope.","responses":[{"response":"We agree that the Evaluation section requires expansion for full verifiability. In the revised manuscript we will add: explicit descriptions of how the compared VS Code extensions were implemented or replicated for the experiments; details on any statistical significance tests (such as paired tests with p-values) applied to the metric scores; and clarification on the selection process and any splits for the 945 ApacheCM commits. The current manuscript already specifies the dataset and metrics used, but these additions will address the gaps. Human validation is absent from the original experiments, so we will add an explicit limitations subsection noting this and situating the work within prior commit message papers that also rely primarily on automatic metrics.","revision_made":"yes","referee_comment":"[Evaluation] Evaluation section: The abstract claims outperformance on BLEU, CIDEr, METEOR, and ROUGE-L over 945 ApacheCM commits but supplies no information on baseline implementations, statistical tests, dataset splits, or human validation. This makes the central empirical result impossible to verify and is load-bearing for the claim of effectiveness."},{"response":"We recognize that linking metric gains to developer-perceived improvements would provide stronger support for the IDE tool's value. The manuscript's evaluation follows standard practice in commit message generation research by using automatic metrics (BLEU, CIDEr, METEOR, ROUGE-L) as proxies. In revision we will expand the discussion to explicitly address the proxy assumption, cite literature on metric-human judgment correlations in related generation tasks, and note the absence of human studies as a limitation. A full developer preference study is outside the scope of the current experiments, but we will strengthen the text to clarify that the reported gains demonstrate the technical benefit of exemplar retrieval.","revision_made":"partial","referee_comment":"[Evaluation] Evaluation section: The paper provides no human evaluation, developer preference study, or correlation analysis between the automatic metric scores and actual usefulness or accuracy of the generated commit messages. This proxy assumption is load-bearing for a tool paper whose value proposition is improved commit messages inside an IDE."}],"tokens_in":1327,"tokens_out":507,"duration_ms":29048,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"CoRaCommit is a VS Code extension that retrieves similar past commits as prompt context, runs several LLMs in parallel, and recommends a model based on user feedback. The headline takeaway is that the reported gains come only from BLEU, CIDEr, METEOR, and ROUGE-L on 945 ApacheCM commits, with no human check on whether the outputs are actually better for developers.\n\nThe concrete contribution is the packaging of these three features into one working extension. The authors take an established retrieval-augmented approach and apply it to an IDE workflow that previously used direct LLM calls. They ship something runnable and compare it against other extensions on a public dataset. That is a practical engineering step, and the implementation choices are described clearly enough to be reproduced.\n\nThe evaluation is the main limitation. The abstract gives no information on baseline re-implementations, prompt details, statistical tests, or any correlation between the metric scores and developer preference or accuracy. The stress-test note is correct here: surface overlap can improve without better intent capture, and nothing in the reported experiment rules that out. For a tool whose value is improved commit messages in practice, this proxy is load-bearing and unexamined.\n\nThe paper is aimed at readers who build or evaluate developer tooling in software engineering. Someone looking for a ready-to-try extension or an example of retrieval in an IDE setting can extract value. A reader seeking new evidence on what makes commit messages effective will find the work thin.\n\nThe design is straightforward and the claims are internally consistent. It is not a deep theoretical result, but the work is honest about its scope. I would bring it to a reading group on LLM tools for developers. I would not cite it unless I needed the specific extension or dataset numbers. It deserves peer review because the tool exists and the experiment is specified, even if referees will likely ask for human validation or stronger justification of the metrics.","headline":"CoRaCommit adds exemplar retrieval and parallel LLM calls to a VS Code commit-message tool, but its results rest on automatic metrics whose link to actual usefulness is untested.","tokens_in":2345,"tokens_out":467,"would_cite":false,"duration_ms":19752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"CoRaCommit retrieves similar past commits as context to improve LLM-generated commit messages in VS Code.","keywords":["commit message generation","VS Code extension","retrieval-augmented generation","large language models","exemplar retrieval","software engineering","ApacheCM dataset"],"falsifier":"A controlled developer study that asks participants to rate the accuracy, clarity, and helpfulness of messages generated by CoRaCommit versus baseline extensions on the same code changes.","tokens_in":2590,"feed_emoji":"💻","tokens_out":638,"duration_ms":17213,"temperature":0.7,"pith_summary":"Commit messages explain code changes but developers often write them manually, leading to inconsistent quality. Current VS Code extensions feed code diffs straight into large language models without pulling in similar past commits as references. CoRaCommit adds retrieval of matching commit exemplars to the prompt, runs several LLMs in parallel to compare outputs, and uses user feedback to recommend which model to favor. On 945 commits from the ApacheCM dataset, the extension records higher scores than prior tools across BLEU, CIDEr, METEOR, and ROUGE-L. The results support the claim that exemplar retrieval supplies useful context for automated commit message generation.","feed_headline":"Retrieved exemplars raise commit message quality in VS Code","feed_subtitle":"The extension pulls similar past commits into LLM prompts and compares outputs from multiple models, beating prior tools on BLEU, CIDEr, MET","key_machinery":"Retrieval of similar commit exemplars to augment LLM prompts, combined with parallel multi-LLM generation and feedback-driven model recommendation.","core_discovery":"CoRaCommit is a VS Code extension that augments commit message generation by retrieving similar commit exemplars to serve as prompt context for large language models, invoking multiple LLMs in parallel to produce and compare candidate messages, and dynamically recommending an LLM based on collected user feedback, with experiments on the ApacheCM dataset showing superior performance on standard automatic metrics.","pith_inferences":["The same retrieval step could be tested on related tasks such as generating pull-request descriptions or code-review comments.","Developers working in other IDEs might see comparable gains if the retrieval mechanism were ported beyond VS Code.","Over time, the growing set of user-accepted messages could serve as additional high-quality exemplars for future retrievals."],"forward_implications":["Adding retrieved commit exemplars to prompts produces higher automatic metric scores than using the diff alone.","Running multiple LLMs in parallel and selecting among their outputs improves the final message quality.","Incorporating user feedback allows the system to recommend more suitable LLMs over repeated use.","The retrieval-augmented approach addresses the limitation of existing extensions that ignore similar historical examples."],"fun_headline_variants":["CoRaCommit retrieves commit exemplars for VS Code messages","Retrieved exemplars improve LLM commit generation in VS Code","VS Code tool compares multiple LLMs with commit retrieval","CoRaCommit uses past commits to refine message candidates"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Higher scores on automatic metrics such as BLEU correspond to commit messages that developers actually find more useful or accurate.","fun_headline_variants_meta":{"raw":{"variants":["CoRaCommit retrieves commit exemplars for VS Code messages","Retrieved exemplars improve LLM commit generation in VS Code","VS Code tool compares multiple LLMs with commit retrieval","CoRaCommit uses past commits to refine message candidates"]},"model":"grok-4.3","cost_usd":0.003932,"raw_usage":{"total_tokens":1993,"prompt_tokens":625,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":39324500,"prompt_tokens_details":{"text_tokens":625,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1306,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":625,"tokens_out":62,"duration_ms":11423,"temperature":1.0,"reasoning_tokens":1306,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T16:54:47.564061+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled developer study that asks participants to rate the accuracy, clarity, and helpfulness of messages generated by CoRaCommit versus baseline extensions on the same code changes.","supporting_citations":[],"review_version":1}