{"id":"c424d00b-eba7-488b-addd-e7ab36cc6101","arxiv_id":"2608.02407","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Antares-3B, a 3B model trained with SFT plus GRPO, matches GPT-5.5 on repository-scale vulnerability localization at roughly 1/100th the inference cost.","lead":"Antares builds 350M-to-3B language models that search a codebase through a terminal and pick out files likely to contain a vulnerability, given only the vulnerability category. On the authors' VLoc benchmark, Antares-3B lands close to GPT-5.5 while running at a small fraction of the cost and time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gap rests on a self-reported benchmark: the only disjointness evidence is an assertion in §4.3, and VLoc Bench is authored by the same organization that built the RL data.","rationale":"The reader's weakest assumption is exactly the load-bearing point: VLoc Bench is the sole yardstick for the headline claim, and the paper's only protection against training/evaluation contamination is a self-reported disjointness assertion. The magnitude of the claimed effect (3B approaching GPT-5.5, outperforming GLM-5.2) is large, which makes benchmark validity critical. If the benchmark labels are wrong or the RL corpus overlaps VLoc Bench, the central contribution reduces to fitting a proprietary benchmark rather than a general capability. The paper does have genuine independent support: the SWE-Bench transfer results (Appendix B.1) show Antares-3B at 66.54 File F1, close to CodeScout-4B, without SWE-Bench training; this suggests real repository-navigation transfer and partially mitigates contamination risk. It does not, however, validate the VLoc Bench vulnerability labels or prove disjointness from the RL corpus. The internal controls—standardized harness, three-run averages, variance reduction from GRPO—are honest and well documented, but they cannot cure a benchmark-level validity failure. Therefore the appropriate verdict is the same CONDITIONAL: accept only after an independent overlap audit and label validation. No verdict change is needed from my read.","tokens_in":38430,"tokens_out":4685,"duration_ms":50751,"concrete_test":"Obtain the VLoc Bench repository snapshot list and the hashed/versioned RL training repository snapshots. For each of the 290 VLoc Bench repositories, compute git tree hashes and normalized file-path/content hashes, then compare against every RL training snapshot to detect exact or near-duplicate overlap. If any overlap is found, re-run Tables 4 and 5 excluding the overlapping tasks; if Antares-3B's File F1 drops by more than ~0.01 or the ordering against GLM-5.2 changes, the headline comparison is invalid. If no overlap is found and an independent re-labeling of a random 50-task sample from raw CVE-fix commits shows high agreement, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that task-specific post-training at 3B can substitute for scale—depends entirely on VLoc Bench [7] being a fair, correctly labeled, externally meaningful benchmark. Two conditions must hold: (i) the 500 tasks across 290 repositories have correct ground-truth vulnerable-file labels, and (ii) the RL training corpus (§4.3) is disjoint from the evaluation set. The paper provides only assertions: 'Training and evaluation datasets are strictly disjoint' (§4.3) and 'No repositories from the evaluation set appear in the Antares training corpus' (Appendix A.2). No overlap-audit method, repository hashes, dataset versioning, or third-party label validation is provided. VLoc Bench is cited as a 'Manuscript, 2026' by Cisco Foundation AI—the same organization that built the proprietary RL pipelines and the models—so the benchmark's construction is not independent of the training data. If either condition fails, the 0.223 vs 0.229 comparison against GPT-5.5 and the >200× size comparison against GLM-5.2 become self-referential: the model could be exploiting memorized repository layouts or label artifacts rather than performing generalizable localization. The paper's own sensitivity results (C.2, C.3.1) show prompt/harness changes move Antares by 0.008–0.012 File F1, nearly as large as the headline GPT-5.5 gap, which makes external-benchmark validity even more load-bearing, not less.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Antares, a family of 350M/1B/3B language models initialized from IBM Granite and post-trained via SFT on cybersecurity reasoning, repository exploration, and terminal trajectories, followed by GRPO with verifiable file-level rewards. The models act as terminal agents in a read-only Docker sandbox with a 15-command budget, receiving only a CWE description. On VLoc Bench (500 tasks, 290 repositories), Antares-3B reaches 0.223 File F1, close to GPT-5.5 (0.229) and above larger open-weight models such as GLM-5.2 (0.186). Appendix results report transfer to SWE-Bench localization and BFCL multi-turn tool use, along with cost and latency advantages. The central claim is that task-specific agentic post-training can substitute for model scale.","tokens_in":38743,"tokens_out":5188,"duration_ms":50093,"significance":"If the VLoc Bench results are valid, this is a practically important result: compact local models could perform repository-scale vulnerability localization at a fraction of the cost of frontier APIs, with implications for air-gapped security workflows. The paper has real strengths: a clearly specified agent protocol, detailed appendix ablations (semantic conditioning, prompt and harness sensitivity, static-analysis comparison), transfer experiments, and release of the 350M and 1B variants. However, the central evidence rests on a self-authored, unpublished benchmark with no demonstrated external label validation and only an asserted train/eval disjointness. Moreover, the headline gap to GPT-5.5 is smaller than the measured effect of prompt/harness changes. The contribution is therefore conditional on independent benchmark validation and on a more robust comparison protocol.","major_comments":[{"comment":"The central comparison is measured on VLoc Bench [7], a benchmark authored by the same organization and cited as 'Manuscript, 2026.' The only evidence of train/eval separation is the assertion 'strictly disjoint' in §4.3 and a sentence in A.2. Provide a concrete overlap audit (e.g., repository hashes, task IDs, versioned snapshots, commit ranges) and an external label-validation protocol, or evaluate on an independent vulnerability-localization benchmark. Without this, the 0.223 vs 0.229 comparison and the scale-substitution claim are self-referential.","section":"§4.3, §7.1, Appendix A.2"},{"comment":"The GRPO reward's primary component is file-level agreement between submitted paths and ground-truth files, and the evaluation metric is File F1. Both the RL corpus and VLoc Bench are built by the same proprietary pipelines. This creates an incentive to optimize for benchmark-specific label artifacts rather than general localization. Show that VLoc Bench labels are independently derived from the RL labels (e.g., different labelers, different pipelines, or audited samples), or the reward and metric circularity remains a load-bearing concern.","section":"§5.2.4, §7.2"},{"comment":"The headline gap between Antares-3B (0.223) and GPT-5.5 xhigh (0.229) is 0.006. A prompt-only change raises Antares-3B to 0.2313 (C.2), and FAPO harness optimization raises it to 0.235 (C.3.1). The claimed 'approaching GPT-5.5' ranking is therefore within the sensitivity of evaluation configuration. Report confidence intervals or variance for all models in Tables 4 and 5 (only Antares variance is in Table 6) and specify a canonical protocol for headline comparisons.","section":"§8.1, Appendix C.2, C.3.1"},{"comment":"Under a native agent harness with an unconstrained budget, Claude Opus 4.6 reaches 0.284 File F1, substantially above all standardized-harness scores. This shows the benchmark is not saturated and that harness choice strongly affects rankings. The main text should report this result and qualify the 'capability cliff' and scale-substitution conclusions, since the standardized harness may understate frontier-model performance.","section":"§C.3.2, §8.1"},{"comment":"Antares-3B, the model on which the headline result depends, is not released; only the 350M and 1B variants are public. Independent verification of the central claim is therefore impossible without the weights. To make the result reproducible, either release Antares-3B under the stated acceptable-use license or provide a hosted evaluation endpoint with detailed logs and exact model versions.","section":"§10"}],"minor_comments":[{"comment":"Axis labels contain letter-spacing artifacts ('R u n t i m e'); please clean up the figure for publication.","section":"Figure 10"},{"comment":"The Antares-350M row lists 'Granite 4.0 350M 32K' in a way that is ambiguous; clarify which base checkpoint and context length apply.","section":"Table 1"},{"comment":"Values 0.2313 and 0.2292 are reported with more decimal places than Tables 4 and 5; standardize precision across the paper.","section":"Appendix C.2"},{"comment":"The text says code search trajectories constitute '15%' of the SFT corpus, while Table 2 lists 15.4%; reconcile the numbers.","section":"§4.2, Table 2"},{"comment":"Cost estimates would benefit from a footnote stating the exact H100 rental rate, API prices, and date of price lookup, since these are used to derive the headline cost claims.","section":"§3.3, Figure 10"}],"recommendation":"major_revision","confidential_remarks":"The self-authored benchmark issue is not a presentation concern; it is the load-bearing support for the paper's main claim. The revision should make the benchmark and its label derivation auditable, report baseline variance, and address the prompt/harness sensitivity of the headline ranking. If the authors cannot provide such evidence, the paper should be substantially reframed as a systems contribution rather than as evidence that task-specific training substitutes for scale."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading, worth engaging, but the central comparison should be treated as provisional. The paper is well built: a two-stage SFT+GRPO pipeline, a clean agent harness, three scales, careful ablations (semantic conditioning beats ECHO and no-auxiliary at all sizes), honest sensitivity analyses, and a useful behavior analysis showing RL induces a search-verify-refine policy with reduced variance. The transfer results to SWE-Bench Verified and BFCL-v3 are genuine evidence that the training produces a general navigation policy, not just a VLoc Bench overfit; Antares-3B at 66.5 File F1 is within 2 points of CodeScout-4B/14B despite never seeing SWE-Bench data. That is the most credible part of the paper.\n\nThe soft spot is exactly where the reader and stress-test put it: VLoc Bench is a manuscript citation from the same Foundation AI group, labels are not independently audited, and disjointness between the proprietary RL corpus and the benchmark is asserted but not demonstrated. There is no repository hash list, no overlap audit, no benchmark version pinning. On top of that, the baseline tables show no variance despite a claimed three runs per model, and the flagship 3B weights are withheld. The prompt-sensitivity result (C.2) is almost uncomfortable: adding an 'explore first' instruction moves Antares-3B from 0.223 to 0.231, more than the gap to GPT-5.5 at default reasoning (0.221) and comparable to the xhigh number. That makes the benchmark validity load-bearing, not just a standard 'more external validation would be nice' caveat.\n\nI do not think this is a fraud or a pathology. The paper acknowledges its own limitations, ships the 350M and 1B models, and the transfer experiments give me some confidence the policy is not purely memorizing the eval set. But as submitted, the headline claim is a self-referential comparison on an unaudited benchmark with margins that are inside the prompt sensitivity. A reader should not treat 'approaches GPT-5.5' as established.\n\nRecommendation: accept for peer review, but with teeth. Require benchmark release with versioning, a documented overlap audit (hashes, dates, filtering), variance for all baselines, and either the 3B weights or an independent third-party evaluation. If those conditions are met, this becomes a solid empirical contribution. As is, it is a strong technical report that should be labeled conditional.","headline":"A serious paper with a real credibility problem: the headline gap to GPT-5.5 rests on a homegrown benchmark no one outside the author group has audited, and the margins are about the same size as their own prompt-sensitivity effects.","tokens_in":39338,"tokens_out":2839,"would_cite":false,"duration_ms":31683,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3-billion-parameter model trained to search codebases through a terminal localizes vulnerabilities nearly as well as GPT-5.5 and beats open-weight models 200× larger—showing that task-specific agentic training can substitute for raw scale","keywords":["vulnerability localization","agentic security agents","small language models","GRPO reinforcement learning","terminal-based code search","verifiable rewards","CWE-conditioned analysis","repository-scale benchmark"],"falsifier":"Run the released Antares models on a freshly built vulnerability-localization suite assembled by an independent team from newly disclosed CVEs, with ground-truth file labels verified by a second curator and a published audit showing no overlap with the Antares training repositories. If small-model File F1 collapses toward the general-purpose open-weight baselines (below GLM-5.2's 0.186, or below the SFT-only checkpoint's 0.188), the claim that GRPO-trained compact agents approach frontier localization is an artifact of benchmark construction. A cheaper first check: scan the released Antares-1B","tokens_in":38266,"feed_emoji":"🐛","tokens_out":8306,"duration_ms":70665,"temperature":0.7,"pith_summary":"This paper claims that task-specific agentic post-training can substitute for model scale in repository-scale vulnerability localization. Its Antares family—350M, 1B, and 3B parameter models initialized from small Granite checkpoints—is trained in two stages: supervised fine-tuning on cybersecurity reasoning and terminal exploration traces, followed by reinforcement learning (GRPO) with verifiable file-level rewards. Evaluated on VLoc Bench, a 500-task benchmark built from 290 real repositories where the model receives only a CWE description and read-only terminal access, Antares-3B reaches 0.223 File F1, approaching GPT-5.5 (0.229) and clearly beating GLM-5.2, a 753B model. The paper further shows that GRPO teaches a search–verify–refine policy and cuts run-to-run variance, and that the full sweep runs on a single H100 GPU in about 15 minutes for under a dollar, making repeated local security scans practical. A sympathetic reader would care because, if true, defenders could run frontier-competitive localization on commodity hardware without sending proprietary source code to external APIs.","feed_headline":"3B security model rivals GPT-5.5 at finding vulnerable files","feed_subtitle":"Task-specific training, not raw scale, earns the win—and a 500-repo sweep runs on one GPU for under $1.","key_machinery":"The load-bearing mechanism is the two-stage post-training pipeline acting through a constrained terminal-agent loop. Stage one is supervised fine-tuning on a corpus mixing cybersecurity reasoning (71.5%), code-search trajectories (15.4%), and deep-research traces, with a semantic-conditioning auxiliary objective that aligns internal representations of functionally similar terminal observations rather than predicting their exact tokens. Stage two applies Group Relative Policy Optimization (GRPO) with programmatically verified rewards—localization quality, submission behavior, tool-use compliance, exploration, and malformed-output penalties—computed over complete multi-turn rollouts in Docker","core_discovery":"Antares establishes that compact models trained end-to-end for the agentic task of vulnerability localization can match or approach frontier-scale generalist models. Given only a CWE category description, the model explores a read-only repository under a 15-command terminal budget, gathers evidence, and submits ranked vulnerable file paths. The central measured result is a capability cliff rather than smooth scaling: Antares-3B scores 0.223 File F1 on VLoc Bench versus 0.229 for GPT-5.5 (xhigh) and 0.186 for GLM-5.2 (753B), while Antares-1B posts the highest recall of any evaluated system. The paper attributes the result to a two-stage recipe—SFT with semantic conditioning on terminal observ","pith_inferences":["If the central claim holds, the implication extends beyond security: any high-value, tool-mediated task with a verifiable outcome—code review triage, incident response, license compliance—may be a candidate for the same SFT + GRPO-at-small-scale recipe, not just vulnerability search.","The benchmark-difficulty analysis (structure beats severity; difficulty tracks ecosystem and repository scale) suggests a testable design rule for agent training curricula: rewards should be densified on distributed, multi-file vulnerabilities, where all models currently collapse, rather than on small single-file cases where grep-based elimination already suffices.","The 350M model's 25% GRPO gain—the largest of the three scales—hints that reinforcement learning from verifiable rewards may be the highest-leverage ingredient precisely where capacity is scarcest, a hypothesis the paper does not itself test explicitly.","The paper releases only the 350M and 1B variants while retaining Antares-3B internally, so the publicly reproducible claim is weaker than the headline result; an external re-run of the full family on an independently built vulnerability-localization suite would settle how much of the cliff is real."],"forward_implications":["If Antares-3B's VLoc Bench score is taken at face value, local single-GPU inference can substitute for frontier APIs on repository-scale vulnerability localization: the 500-task sweep costs under $1 and about 15 minutes, versus roughly $141 and 5 hours for GPT-5.5 under the same harness.","Because GRPO reduces run-to-run variance by 42–65%, a single evaluation run of the trained models gives a reliable behavior estimate, making repeated scans across branches and CI/CD events practical.","The policy transfers beyond security: on SWE-Bench Verified, Antares-3B reaches 66.54 File F1, within two points of CodeScout-14B, a model trained specifically on SWE-Bench, even though Antares never saw SWE-Bench repositories.","The learned search behavior is instruction-sensitive: adding an explore-first prompt lifts Antares-3B from 0.223 to 0.231 File F1, slightly above GPT-5.5 (xhigh), without retraining.","Performance degrades sharply on large repositories and multi-file vulnerabilities, and even a cost-unconstrained native frontier agent reaches only 0.284 File F1 on the same benchmark, so the task is far from saturated."],"fun_headline_variants":["3B agentic model approaches GPT-5.5 at vulnerability hunting","Compact 3B model rivals GPT-5.5 on vulnerable code search","Task-specific training lets 3B model beat 200x larger open models","Agentic security: 3B nears GPT-5.5, sweeps 500 repos in 15 min","Vulnerability localization: 3B model rivals GPT-5.5 at low cost"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything hinges on VLoc Bench being a fair, correctly labeled, externally meaningful benchmark that is truly disjoint from Antares' training data—yet the paper asserts strict disjointness (Section 4.3) while the benchmark is a same-organization manuscript built from the same proprietary pipelines as the RL corpus, so its independence is claimed rather than independently audited.","fun_headline_variants_meta":{"raw":{"variants":["3B agentic model approaches GPT-5.5 at vulnerability hunting","Compact 3B model rivals GPT-5.5 on vulnerable code search","Task-specific training lets 3B model beat 200x larger open models","Agentic security: 3B nears GPT-5.5, sweeps 500 repos in 15 min","Vulnerability localization: 3B model rivals GPT-5.5 at low cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000796,"raw_usage":{"total_tokens":3322,"prompt_tokens":704,"completion_tokens":2618,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":2520}},"tokens_in":448,"tokens_out":2618,"duration_ms":22020,"temperature":1.0,"reasoning_tokens":2520,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:46:17.251772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the released Antares models on a freshly built vulnerability-localization suite assembled by an independent team from newly disclosed CVEs, with ground-truth file labels verified by a second curator and a published audit showing no overlap with the Antares training repositories. If small-model File F1 collapses toward the general-purpose open-weight baselines (below GLM-5.2's 0.186, or below the SFT-only checkpoint's 0.188), the claim that GRPO-trained compact agents approach frontier localization is an artifact of benchmark construction. A cheaper first check: scan the released Antares-1B","supporting_citations":[],"review_version":1}