Pith. sign in

REVIEW 5 major objections 5 minor 78 references

A 3-billion-parameter model trained to search codebases through a terminal localizes vulnerabilities nearly as well as GPT-5.5 and beats open-weight models 200× larger—showing that task-specific agentic training can substitute for raw scale

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:46 UTC pith:G57Q6GTY

load-bearing objection A serious paper with a real credibility problem: the headline gap to GPT-5.5 rests on a homegrown benchmark no one outside the author group has audited, and the margins are about the same size as their own prompt-sensitivity effects. the 5 major comments →

arxiv 2608.02407 v1 pith:G57Q6GTY submitted 2026-08-03 cs.CR cs.AI

Antares: Foundation Models for Agentic Vulnerability Localization

Supriti Vijay , Aman Priyanshu , Didier Chapoteau , Arthur Goldblatt , Jianliang He , Kimia Majd , Fraser Burch , Baturay Saglam
show 3 more authors
Takahiro Matsumoto Zhuoran Yang Amin Karbasi
This is my paper
classification cs.CR cs.AI
keywords vulnerability localizationagentic security agentssmall language modelsGRPO reinforcement learningterminal-based code searchverifiable rewardsCWE-conditioned analysisrepository-scale benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that task-specific agentic post-training can substitute for model scale in repository-scale vulnerability localization. Its Antares family—350M, 1B, and 3B parameter models initialized from small Granite checkpoints—is trained in two stages: supervised fine-tuning on cybersecurity reasoning and terminal exploration traces, followed by reinforcement learning (GRPO) with verifiable file-level rewards. Evaluated on VLoc Bench, a 500-task benchmark built from 290 real repositories where the model receives only a CWE description and read-only terminal access, Antares-3B reaches 0.223 File F1, approaching GPT-5.5 (0.229) and clearly beating GLM-5.2, a 753B model. The paper further shows that GRPO teaches a search–verify–refine policy and cuts run-to-run variance, and that the full sweep runs on a single H100 GPU in about 15 minutes for under a dollar, making repeated local security scans practical. A sympathetic reader would care because, if true, defenders could run frontier-competitive localization on commodity hardware without sending proprietary source code to external APIs.

Core claim

Antares establishes that compact models trained end-to-end for the agentic task of vulnerability localization can match or approach frontier-scale generalist models. Given only a CWE category description, the model explores a read-only repository under a 15-command terminal budget, gathers evidence, and submits ranked vulnerable file paths. The central measured result is a capability cliff rather than smooth scaling: Antares-3B scores 0.223 File F1 on VLoc Bench versus 0.229 for GPT-5.5 (xhigh) and 0.186 for GLM-5.2 (753B), while Antares-1B posts the highest recall of any evaluated system. The paper attributes the result to a two-stage recipe—SFT with semantic conditioning on terminal observ

What carries the argument

The load-bearing mechanism is the two-stage post-training pipeline acting through a constrained terminal-agent loop. Stage one is supervised fine-tuning on a corpus mixing cybersecurity reasoning (71.5%), code-search trajectories (15.4%), and deep-research traces, with a semantic-conditioning auxiliary objective that aligns internal representations of functionally similar terminal observations rather than predicting their exact tokens. Stage two applies Group Relative Policy Optimization (GRPO) with programmatically verified rewards—localization quality, submission behavior, tool-use compliance, exploration, and malformed-output penalties—computed over complete multi-turn rollouts in Docker

Load-bearing premise

Everything hinges on VLoc Bench being a fair, correctly labeled, externally meaningful benchmark that is truly disjoint from Antares' training data—yet the paper asserts strict disjointness (Section 4.3) while the benchmark is a same-organization manuscript built from the same proprietary pipelines as the RL corpus, so its independence is claimed rather than independently audited.

What would settle it

Run the released Antares models on a freshly built vulnerability-localization suite assembled by an independent team from newly disclosed CVEs, with ground-truth file labels verified by a second curator and a published audit showing no overlap with the Antares training repositories. If small-model File F1 collapses toward the general-purpose open-weight baselines (below GLM-5.2's 0.186, or below the SFT-only checkpoint's 0.188), the claim that GRPO-trained compact agents approach frontier localization is an artifact of benchmark construction. A cheaper first check: scan the released Antares-1B

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If Antares-3B's VLoc Bench score is taken at face value, local single-GPU inference can substitute for frontier APIs on repository-scale vulnerability localization: the 500-task sweep costs under $1 and about 15 minutes, versus roughly $141 and 5 hours for GPT-5.5 under the same harness.
  • Because GRPO reduces run-to-run variance by 42–65%, a single evaluation run of the trained models gives a reliable behavior estimate, making repeated scans across branches and CI/CD events practical.
  • The policy transfers beyond security: on SWE-Bench Verified, Antares-3B reaches 66.54 File F1, within two points of CodeScout-14B, a model trained specifically on SWE-Bench, even though Antares never saw SWE-Bench repositories.
  • The learned search behavior is instruction-sensitive: adding an explore-first prompt lifts Antares-3B from 0.223 to 0.231 File F1, slightly above GPT-5.5 (xhigh), without retraining.
  • Performance degrades sharply on large repositories and multi-file vulnerabilities, and even a cost-unconstrained native frontier agent reaches only 0.284 File F1 on the same benchmark, so the task is far from saturated.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the implication extends beyond security: any high-value, tool-mediated task with a verifiable outcome—code review triage, incident response, license compliance—may be a candidate for the same SFT + GRPO-at-small-scale recipe, not just vulnerability search.
  • The benchmark-difficulty analysis (structure beats severity; difficulty tracks ecosystem and repository scale) suggests a testable design rule for agent training curricula: rewards should be densified on distributed, multi-file vulnerabilities, where all models currently collapse, rather than on small single-file cases where grep-based elimination already suffices.
  • The 350M model's 25% GRPO gain—the largest of the three scales—hints that reinforcement learning from verifiable rewards may be the highest-leverage ingredient precisely where capacity is scarcest, a hypothesis the paper does not itself test explicitly.
  • The paper releases only the 350M and 1B variants while retaining Antares-3B internally, so the publicly reproducible claim is weaker than the headline result; an external re-run of the full family on an independently built vulnerability-localization suite would settle how much of the cliff is real.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper introduces Antares, a family of 350M/1B/3B language models initialized from IBM Granite and post-trained via SFT on cybersecurity reasoning, repository exploration, and terminal trajectories, followed by GRPO with verifiable file-level rewards. The models act as terminal agents in a read-only Docker sandbox with a 15-command budget, receiving only a CWE description. On VLoc Bench (500 tasks, 290 repositories), Antares-3B reaches 0.223 File F1, close to GPT-5.5 (0.229) and above larger open-weight models such as GLM-5.2 (0.186). Appendix results report transfer to SWE-Bench localization and BFCL multi-turn tool use, along with cost and latency advantages. The central claim is that task-specific agentic post-training can substitute for model scale.

Significance. If the VLoc Bench results are valid, this is a practically important result: compact local models could perform repository-scale vulnerability localization at a fraction of the cost of frontier APIs, with implications for air-gapped security workflows. The paper has real strengths: a clearly specified agent protocol, detailed appendix ablations (semantic conditioning, prompt and harness sensitivity, static-analysis comparison), transfer experiments, and release of the 350M and 1B variants. However, the central evidence rests on a self-authored, unpublished benchmark with no demonstrated external label validation and only an asserted train/eval disjointness. Moreover, the headline gap to GPT-5.5 is smaller than the measured effect of prompt/harness changes. The contribution is therefore conditional on independent benchmark validation and on a more robust comparison protocol.

major comments (5)
  1. [§4.3, §7.1, Appendix A.2] The central comparison is measured on VLoc Bench [7], a benchmark authored by the same organization and cited as 'Manuscript, 2026.' The only evidence of train/eval separation is the assertion 'strictly disjoint' in §4.3 and a sentence in A.2. Provide a concrete overlap audit (e.g., repository hashes, task IDs, versioned snapshots, commit ranges) and an external label-validation protocol, or evaluate on an independent vulnerability-localization benchmark. Without this, the 0.223 vs 0.229 comparison and the scale-substitution claim are self-referential.
  2. [§5.2.4, §7.2] The GRPO reward's primary component is file-level agreement between submitted paths and ground-truth files, and the evaluation metric is File F1. Both the RL corpus and VLoc Bench are built by the same proprietary pipelines. This creates an incentive to optimize for benchmark-specific label artifacts rather than general localization. Show that VLoc Bench labels are independently derived from the RL labels (e.g., different labelers, different pipelines, or audited samples), or the reward and metric circularity remains a load-bearing concern.
  3. [§8.1, Appendix C.2, C.3.1] The headline gap between Antares-3B (0.223) and GPT-5.5 xhigh (0.229) is 0.006. A prompt-only change raises Antares-3B to 0.2313 (C.2), and FAPO harness optimization raises it to 0.235 (C.3.1). The claimed 'approaching GPT-5.5' ranking is therefore within the sensitivity of evaluation configuration. Report confidence intervals or variance for all models in Tables 4 and 5 (only Antares variance is in Table 6) and specify a canonical protocol for headline comparisons.
  4. [§C.3.2, §8.1] Under a native agent harness with an unconstrained budget, Claude Opus 4.6 reaches 0.284 File F1, substantially above all standardized-harness scores. This shows the benchmark is not saturated and that harness choice strongly affects rankings. The main text should report this result and qualify the 'capability cliff' and scale-substitution conclusions, since the standardized harness may understate frontier-model performance.
  5. [§10] Antares-3B, the model on which the headline result depends, is not released; only the 350M and 1B variants are public. Independent verification of the central claim is therefore impossible without the weights. To make the result reproducible, either release Antares-3B under the stated acceptable-use license or provide a hosted evaluation endpoint with detailed logs and exact model versions.
minor comments (5)
  1. [Figure 10] Axis labels contain letter-spacing artifacts ('R u n t i m e'); please clean up the figure for publication.
  2. [Table 1] The Antares-350M row lists 'Granite 4.0 350M 32K' in a way that is ambiguous; clarify which base checkpoint and context length apply.
  3. [Appendix C.2] Values 0.2313 and 0.2292 are reported with more decimal places than Tables 4 and 5; standardize precision across the paper.
  4. [§4.2, Table 2] The text says code search trajectories constitute '15%' of the SFT corpus, while Table 2 lists 15.4%; reconcile the numbers.
  5. [§3.3, Figure 10] Cost estimates would benefit from a footnote stating the exact H100 rental rate, API prices, and date of price lookup, since these are used to derive the headline cost claims.

Circularity Check

0 steps flagged

No demonstrable circularity; main concern is unverified benchmark disjointness, which is a data-validity risk rather than an in-manuscript reduction.

full rationale

Walking the claimed derivation chain (Granite base + SFT + GRPO with verifiable file-level rewards, then evaluation on VLoc Bench plus SWE-Bench and BFCL-v3), I find no step where a reported 'prediction' is equal to its inputs by construction. The RL reward component 'Localization quality' (Table 3) measures agreement between submitted and ground-truth files, which matches the evaluation metric 'File F1'; but optimizing a policy on a reward that is aligned with the downstream metric is standard practice and is not circular unless the same labeled tasks appear in training and evaluation. The paper asserts the opposite: 'Training and evaluation datasets are strictly disjoint' (Section 4.3) and 'No repositories from the evaluation set appear in the Antares training corpus' (Appendix A.2). Whether that assertion is true is not verifiable from the manuscript, and the benchmark [7] is a self-published manuscript from the same organization (Cisco Foundation AI) with no disclosed overlap audit. That is a real correctness/validity risk, especially because the headline gap between Antares-3B and GPT-5.5 (0.223 vs 0.229) is within the prompt/harness sensitivity shown in C.2 and C.3.1 (0.008-0.012 changes). However, none of this constitutes an in-paper reduction: no equation or fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem or ansatz is imported from the authors' prior work. Moreover, the transfer results on SWE-Bench Verified/Lite and BFCL-v3 provide independent, externally anchored evidence that a nontrivial, generalizable repository-navigation policy was learned, which would be hard to explain if the VLoc Bench numbers were purely circular. I therefore score circularity at 1, while noting that the benchmark-validity and disjointness concerns belong in the correctness/reproducibility review rather than as evidence of circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim depends on the VLoc Bench benchmark, which is self-authored and not independently published; the RL reward is file-level F1, the same metric the benchmark reports, and the training corpus is proprietary with only an assertion of disjointness. Many hand-chosen constraints — terminal budget, observation truncation, reward-component weights, data mix — materially affect outcomes. Transfer results on SWE-Bench and BFCL-v3 give some independent support for transferable search skills, but they do not ground the headline comparison against GPT-5.5.

free parameters (5)
  • SFT corpus composition = 71.5% cybersecurity reasoning / 13.1% deep research / 15.4% code search
    Chosen by hand; no ablation shows robustness of this mix, yet it directly determines the balance between security knowledge and terminal fluency.
  • GRPO reward component weights
    Weights for localization quality, submission behavior, tool-use compliance, exploration, and malformed-output penalties are not disclosed. They determine the learned policy and final F1.
  • GRPO rollout group size, KL coefficient, learning rate, and number of rollouts
    Not reported. These standard RL hyperparameters materially affect stability, variance reduction, and final performance.
  • Terminal budget and observation truncation = 15 commands; 2000-character observations
    Hand-set constraints. The paper's own experiments show GPT-5.5 improves at budget 30 and Antares improves with budget 25 and a prompt change, so results are sensitive to these choices.
  • FAPO-optimized prompt, budget, and frequency penalty = budget 25, frequency penalty 0.3, four-phase prompt
    Optimized directly against VLoc Bench target scores in Appendix C.3.1, raising File F1 from 0.223 to 0.235. This is tuning on the evaluation set and is not the main reported result, but it demonstrates instability.
axioms (5)
  • domain assumption VLoc Bench is a valid, correctly labeled benchmark of real vulnerabilities.
    All headline comparisons depend on it; it is self-authored, cited as a 'Manuscript, 2026,' and not independently audited (Ref [7], Sections 1 and 7.1).
  • domain assumption RL training corpus and VLoc Bench are strictly disjoint.
    Asserted in Section 4.3, but no overlap audit, hash comparison, or external verification is provided; both derive from the same proprietary pipelines.
  • domain assumption The standardized agent harness gives every model an equivalent, fair task view.
    Section 7.4 and A.1 standardize the harness, but C.3.2 shows native frontier orchestration reaches 0.284 File F1, so the harness may disadvantage some models.
  • domain assumption GRPO's multi-component programmatic rewards correctly measure localization ability.
    Reward components (Table 3) are designed by the authors; exploration-behavior rewards may reward search volume rather than true localization quality.
  • domain assumption Granite 4.0 base models have near-zero File F1 and are a neutral initialization.
    Reported in Section 5.1 and Figure 8; no external confirmation that base-model behavior is as described.

pith-pipeline@v1.3.0-daily-deepseek · 38031 in / 15203 out tokens · 145509 ms · 2026-08-04T07:46:17.251772+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Antares: Foundation Models for Agentic Vulnerability Localization." pith.science (2026). https://pith.science/paper/G57Q6GTY

@misc{pith2026260802407,
  author       = {Pith},
  title        = {Pith review of: Antares: Foundation Models for Agentic Vulnerability Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G57Q6GTY}},
  note         = {Machine review of arXiv:2608.02407}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vulnerability localization is a fundamental step in software security, requiring models to reason over large codebases and iteratively identify vulnerable implementations. We present Antares, a family of compact language models (350M, 1B, and 3B parameters) for agentic vulnerability localization. Based on IBM Granite base models, Antares is trained through a two-stage pipeline that combines supervised fine-tuning on cybersecurity reasoning and repository exploration data with reinforcement learning from verifiable rewards over vulnerable repositories. Across extensive evaluations, Antares-3B approaches GPT-5.5 while outperforming open-weight models over 200x larger in size. The Antares family further enables fast, low-cost local inference, completing a full 500-task evaluation sweep in approximately 15 minutes on a single H100 GPU, corresponding to an amortized evaluation time of under 2 seconds and less than $0.002 per task.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 1 canonical work pages

  1. [1]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence, 2024

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence, 2024. URLhttps://arxiv.org/abs/2406.07599

    Pith/arXiv arXiv 2024

  2. [2]

    False negative - that one is going to kill you

    Amit Seal Ami, Kevin Moran, Denys Poshyvanyk, and Adwait Nadkarni. "False negative - that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing . In 2024 IEEE Symposium on Security and Privacy (SP), pages 3979–3997, Los Alamitos, CA, USA, May 2024. IEEE Computer Society. doi: 10.1109/SP54263.2024.00019. U...

    arXiv 2024

  3. [3]

    Claude code

    Anthropic. Claude code. https://www.anthropic.com/product/claude-code, 2026. Anthropic product page. Accessed: 2026-06-29

    2026

  4. [4]

    Cursor: The ai code editor

    Anysphere. Cursor: The ai code editor. https://cursor.com/ , 2026. Cursor product page. Accessed: 2026-06-29. 26

    2026

  5. [5]

    URL https://arxiv.org/abs/2506 .07982

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.𝜏2-bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506 .07982

    2025

  6. [6]

    Agenticscr: An autonomous agentic secure code review for immature vulnerabilities detection, 2026

    Wachiraphan Charoenwet, Kla Tantithamthavorn, Patanamon Thongtanunam, Hong Yi Lin, Min- woo Jeong, and Ming Wu. Agenticscr: An autonomous agentic secure code review for immature vulnerabilities detection, 2026. URLhttps://arxiv.org/abs/2601.19138

    arXiv 2026

  7. [7]

    Vulnerability localization benchmark

    Cisco Foundation AI. Vulnerability localization benchmark. Manuscript, 2026. Available at https://github.com/cisco-foundation-ai/vulnerability-localization-benchmark

    2026

  8. [8]

    Composer: Building a fast frontier model with rl

    Cursor Team. Composer: Building a fast frontier model with rl. https://cursor.com/blog/co mposer, 2025. Cursor research blog. Accessed: 2026-06-29

    2025

  9. [9]

    Rader: Reasoning-aware dense retrieval models,

    Debrup Das, Sam O’ Nuallain, and Razieh Rahimi. Rader: Reasoning-aware dense retrieval models,

  10. [10]

    Deephat-v1-7b

    DeepHat. Deephat-v1-7b. https://huggingface.co/DeepHat/DeepHat-V1-7B, 2025. Hugging Face model repository. Finetuned from Qwen2.5-Coder-7B. Accessed: 2026-06-29

    2025

  11. [11]

    Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. Swe-bench pro: Can ai agents solve long-ho...

    Pith/arXiv arXiv 2025

  12. [12]

    Codeql: Semantic code analysis

    GitHub. Codeql: Semantic code analysis. https://codeql.github.com, 2025. GitHub documen- tation. Accessed: 2026-06-29

    2025

  13. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, ...

    2025

  14. [14]

    Repoaudit: An autonomous llm-agent for repository-level code auditing, 2025

    Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. Repoaudit: An autonomous llm-agent for repository-level code auditing, 2025. URL https://arxiv.org/abs/ 2501.18160

    Pith/arXiv arXiv 2025

  15. [15]

    Granite-4.0-1b

    IBM Granite Team. Granite-4.0-1b. https://huggingface.co/ibm-granite/granite-4.0-1b ,

  16. [16]

    Granite-4.0-350m

    IBM Granite Team. Granite-4.0-350m. https://huggingface.co/ibm-granite/granite-4.0 -350m, 2025. Hugging Face model repository. Accessed: 2026-06-29

    2025

  17. [17]

    Accessed: 2026-06-29

    Hugging Face model repository. Accessed: 2026-06-29

    2026

  18. [18]

    Kakade, and Carles Domingo-Enrich

    Samy Jelassi, Mujin Kwun, Rosie Zhao, Yuanzhi Li, Nicolo Fusi, Yilun Du, Sham M. Kakade, and Carles Domingo-Enrich. Matching features, not tokens: Energy-based fine-tuning of language models, 2026. URLhttps://arxiv.org/abs/2603.12248

    arXiv 2026

  19. [19]

    Granite-4.0-micro

    IBM Granite Team. Granite-4.0-micro. https://huggingface.co/ibm-granite/granite-4.0 -micro, 2025. Hugging Face model repository. Accessed: 2026-06-29

    2025

  20. [20]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  21. [21]

    Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning, 2025

    Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning, 2025. URLhttps://arxiv.org/abs/2503.00223

    Pith/arXiv arXiv 2025

  22. [22]

    Dense passage retrieval for open-domain question answering, 2020

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020. URL https://arxiv.org/abs/2004.04906

    Pith/arXiv arXiv 2020

  23. [23]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URLhttps://arxiv.org/abs/2503.09516

    Pith/arXiv arXiv 2025

  24. [24]

    Fapo: Fully automated prompt optimization of multi-step llm pipelines, 2026

    Paul Kassianik, Baturay Saglam, Huaibo Zhao, Blaine Nelson, Supriti Vijay, Aman Priyanshu, and Amin Karbasi. Fapo: Fully automated prompt optimization of multi-step llm pipelines, 2026. URL https://arxiv.org/abs/2606.19605

    Pith/arXiv arXiv 2026

  25. [25]

    Llama-3.1-foundationai-securityllm-base-8b technical report, 2025

    Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, Aman Priyanshu, Adam Swanda, Amy Chang, Hyrum Anderson, Kojin Oshiba, Omar Santos, Yaron Singer, and Amin Karbasi. Llama-3.1-foundationai-securityllm-base-8b technical report, 2025. URL https://arxiv.org...

    Pith/arXiv arXiv 2025

  26. [26]

    Openresearcher: A fully open pipeline for long-horizon deep research trajectory synthesis, 2026

    Zhuofeng Li, Dongfu Jiang, Xueguang Ma, Haoxiang Zhang, Ping Nie, Yuyu Zhang, Kai Zou, Jianwen Xie, Yu Zhang, and Wenhu Chen. Openresearcher: A fully open pipeline for long-horizon deep research trajectory synthesis, 2026. URLhttps://arxiv.org/abs/2603.20278

    arXiv 2026

  27. [27]

    Webthinker: Empowering large reasoning models with deep research capability,

    Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yongkang Wu, Ji-Rong Wen, Yutao Zhu, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability,

  28. [28]

    URLhttps://arxiv.org/abs/2504.21776. 28

    Pith/arXiv arXiv

  29. [29]

    Vulnllm-r: Specialized reasoning llm with agent scaffold for vulnerability detection,

    Yuzhou Nie, Hongwei Li, Chengquan Guo, Ruizhe Jiang, Zhun Wang, Bo Li, Dawn Song, and Wenbo Guo. Vulnllm-r: Specialized reasoning llm with agent scaffold for vulnerability detection,

  30. [30]

    Synthesizing multi-agent harnesses for vulnerability discovery, 2026

    Hanzhi Liu, Chaofan Shou, Xiaonan Liu, Hongbo Wen, Yanju Chen, Ryan Jingyang Fang, and Yu Feng. Synthesizing multi-agent harnesses for vulnerability discovery, 2026. URL https: //arxiv.org/abs/2604.20801

    Pith/arXiv arXiv 2026

  31. [31]

    The minimax-m2 series: Mini activations unleashing max real-world intelligence, 2026

    MiniMax, :, Aili Chen, Aonian Li, Baichuan Zhou, Bangwei Gong, Binyang Jiang, Boji Dan, Changqing Yu, Chao Wang, Cheng Ma, Cheng Zhong, Cheng Zhu, Chengjun Xiao, Chengyi Yang, Chengyu Du, Chenyang Zhang, Chi Zhang, Chuangyi Huang, Chunhao Zhang, Chunhui Du, Chunyu Zhao, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dongyu Zhang, Enhui Yang, Fei Yu, Gua...

    Pith/arXiv arXiv 2026

  32. [32]

    OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, V...

    Pith/arXiv arXiv 2025

  33. [33]

    URLhttps://arxiv.org/abs/2512.07533

    arXiv

  34. [34]

    Introducing swe-bench verified

    OpenAI. Introducing swe-bench verified. openai.com, August 2024

    2024

  35. [35]

    OpenAI. Codex. https://chatgpt.com/codex/, 2026. OpenAI product page. Accessed: 2026-06- 29

    2026

  36. [36]

    A practical approach to the automatic classification of security-relevant commits

    Antonino Sabetta and Michele Bezzi. A practical approach to the automatic classification of security-relevant commits. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), page 579–582. IEEE, 2018. doi: 10.1109/icsme.2018.00058. URL http: //dx.doi.org/10.1109/ICSME.2018.00058

    arXiv 2018

  37. [37]

    Introducing swe-grep and swe-grep-mini: Rl for multi-turn, fast context retrieval

    Ben Pan, Carlo Baronio, Albert Tam, Pietro Marsella, Mokshit Jain, Daniel Chiu, Swyx, and Silas Alberti. Introducing swe-grep and swe-grep-mini: Rl for multi-turn, fast context retrieval. https://cognition.com/blog/swe-grep, October 2025. Cognition AI blog. Accessed: 2026-06- 29

    2025

  38. [38]

    Gonzalez

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste- Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, ed...

    2025

  39. [39]

    Tongsearch-qr: Reinforced query reasoning for retrieval, 2025

    Xubo Qin, Jun Bai, Jiaqi Li, Zixia Jia, and Zilong Zheng. Tongsearch-qr: Reinforced query reasoning for retrieval, 2025. URLhttps://arxiv.org/abs/2506.11603

    Pith/arXiv arXiv 2025

  40. [40]

    Reasonir: Training retrievers for reasoning tasks, 2025

    Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. Reasonir: Training retrievers for reasoning tasks, 2025. URLhttps://arxiv.org/abs/2504.20595. 30

    Pith/arXiv arXiv 2025

  41. [41]

    Lily-Cybersecurity-7B-v0.2

    Sego Lily Labs. Lily-Cybersecurity-7B-v0.2. https://huggingface.co/segolilylabs/Lily-C ybersecurity-7B-v0.2, 2024. Hugging Face model repository. Accessed: 2026-06-29

    2024

  42. [42]

    Semgrep, Inc. Semgrep. https://semgrep.dev, 2025. Semgrep product page. Accessed: 2026-06- 29

    2025

  43. [43]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security, 2025

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security, 2025. URLhttps://arxiv.org/abs/2406.05590

    Pith/arXiv arXiv 2025

  44. [44]

    Sonarqube

    SonarSource. Sonarqube. https://www.sonarsource.com/products/sonarqube/ , 2025. SonarSource product page. Accessed: 2026-06-29

    2025

  45. [45]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  46. [46]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Echo: Terminal agents learn world models for free, 2026

    Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, and Dimitris Papailiopoulos. Echo: Terminal agents learn world models for free, 2026. URLhttps://arxiv.org/abs/2605.24517

    Pith/arXiv arXiv 2026

  48. [48]

    Falcon-h1r: Pushing the reasoning frontiers with a hybrid model for efficient test-time scaling,

    Falcon LLM Team, Iheb Chaabane, Puneesh Khanna, Suhail Mohmad, Slim Frikha, Shi Hu, Abdal- gader Abubaker, Reda Alami, Mikhail Lubinets, Mohamed El Amine Seddik, and Hakim Hacid. Falcon-h1r: Pushing the reasoning frontiers with a hybrid model for efficient test-time scaling,

  49. [49]

    Granite 4.0 nano: Just how small can you go? https://huggin gface.co/blog/ibm-granite/granite-4-nano , October 2025

    Kate Soule and Rameswar Panda. Granite 4.0 nano: Just how small can you go? https://huggin gface.co/blog/ibm-granite/granite-4-nano , October 2025. Hugging Face Blog. Published October 28, 2025. Accessed: 2026-06-29

    2025

  50. [50]

    Codescout: An effective recipe for reinforcement learning of code search agents, 2026

    Lintang Sutawika, Aditya Bharat Soni, Bharath Sriraam R R, Apurva Gandhi, Taha Yassine, Sanidhya Vijayvargiya, Yuchen Li, Xuhui Zhou, Yilin Zhang, Leander Melroy Maben, and Graham Neubig. Codescout: An effective recipe for reinforcement learning of code search agents, 2026. URL https://arxiv.org/abs/2603.17829

    arXiv 2026

  51. [51]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models,

    5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowe...

  52. [52]

    URLhttps://arxiv.org/abs/2508.06471

    Pith/arXiv arXiv

  53. [53]

    Think before you retrieve: Learning test-time adaptive search with small language models, 2025

    Supriti Vijay, Aman Priyanshu, Anu Vellore, Baturay Saglam, and Amin Karbasi. Think before you retrieve: Learning test-time adaptive search with small language models, 2025. URL https: //arxiv.org/abs/2511.07581

    arXiv 2025

  54. [54]

    Text embeddings by weakly-supervised contrastive pre-training, 2024

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024. URL https://arxiv.org/abs/2212.03533

    Pith/arXiv arXiv 2024

  55. [55]

    Kimi Team, Yifan Bai, Yiping Bao, Y. Charles, Cheng Chen, Guanduo Chen, Haiting Chen, Huarong Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Chenxiao Gao, Hongcheng Gao, Peizhong Ga...

    Pith/arXiv arXiv 2026

  56. [56]

    Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, and John Langford

    Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, and John Langford. Next-latent prediction transformers learn compact world models, 2026. URLhttps://arxiv.org/abs/2511.05963

    Pith/arXiv arXiv 2026

  57. [57]

    Dubniczky, Ridhi Jain, and Lucas C

    Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Bilel Cherif, Richard A. Dubniczky, Ridhi Jain, and Lucas C. Cordeiro. Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview, pages 33–47. Springer Nature Switzerland, Cham, 2026. ISBN 978-3-031-99447-0. doi: 10.1007/978-3-031-99447-...

    work page doi:10.1007/978-3-031-99447-0_3 2026

  58. [58]

    Qrs: A rule-synthesizing neuro-symbolic triad for autonomous vulnerability discovery, 2026

    George Tsigkourakos and Constantinos Patsakis. Qrs: A rule-synthesizing neuro-symbolic triad for autonomous vulnerability discovery, 2026. URLhttps://arxiv.org/abs/2602.09774

    arXiv 2026

  59. [59]

    Llama-3.1-foundationai- securityllm-8b-instruct technical report, 2025

    Sajana Weerawardhena, Paul Kassianik, Blaine Nelson, Baturay Saglam, Anu Vellore, Aman Priyanshu, Supriti Vijay, Massimo Aufiero, Arthur Goldblatt, Fraser Burch, Ed Li, Jianliang He, Dhruv Kedia, Kojin Oshiba, Zhouran Yang, Yaron Singer, and Amin Karbasi. Llama-3.1-foundationai- securityllm-8b-instruct technical report, 2025. URLhttps://arxiv.org/abs/2508.01059

    Pith/arXiv arXiv 2025

  60. [60]

    From trace to line: Llm agent for real-world oss vulnerability localization, 2026

    Haoran Xi, Minghao Shao, Brendan Dolan-Gavitt, Muhammad Shafique, and Ramesh Karri. From trace to line: Llm agent for real-world oss vulnerability localization, 2026. URL https://arxiv. org/abs/2510.02389

    arXiv 2026

  61. [61]

    Reposvul: A 32 repository-level high-quality vulnerability dataset, 2024

    Xinchen Wang, Ruida Hu, Cuiyun Gao, Xin-Cheng Wen, Yujia Chen, and Qing Liao. Reposvul: A 32 repository-level high-quality vulnerability dataset, 2024. URL https://arxiv.org/abs/2401.1 3169

    2024

  62. [62]

    Exploitgym: Can ai agents turn security vulnerabilities into real attacks?, 2026

    Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, Kurt Thomas, Yan Shoshitaishvili, Wenbo Guo, Jingxuan He, Thorsten Holz, and Dawn Song. Exploitgym: Can ai agents turn security vulnerabilities into real attacks?, 2026. URLhttps://arxiv.org/abs/2605.11086

    Pith/arXiv arXiv 2026

  63. [63]

    Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale, 2026

    Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale, 2026. URL https://arxiv.or g/abs/2506.02548

    arXiv 2026

  64. [64]

    Vulagent: Hypothesis-validation based multi- agent vulnerability detection, 2025

    Ziliang Wang, Ge Li, Jia Li, Hao Zhu, and Zhi Jin. Vulagent: Hypothesis-validation based multi- agent vulnerability detection, 2025. URLhttps://arxiv.org/abs/2509.11523

    arXiv 2025

  65. [65]

    Fastcontext: Training efficient repository explorer for coding agents,

    Shaoqiu Zhang, Maoquan Wang, Yuling Shi, Yuhang Wang, Xiaodong Gu, Yongqiang Yao, Tori Gong, Sheng Chen, Rao Fu, Anisha Agarwal, Spandan Grag, Gabriel Ryan, Colin Merkel, Yufan Huang, and Shengyu Fu. Fastcontext: Training efficient repository explorer for coding agents,

  66. [66]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments,

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments,

  67. [67]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  68. [68]

    Llama-3.1-foundationai-securityllm-reasoning-8b technical report,

    Zhuoran Yang, Ed Li, Jianliang He, Aman Priyanshu, Baturay Saglam, Paul Kassianik, Sajana Weerawardhena, Anu Vellore, Blaine Nelson, Neusha Javidnia, Arthur Goldblatt, Fraser Burch, Avi Zohary, Assaf Eisenman, Mahdi Sabbaghi, Supriti Vijay, Rahim Dharssi, Dhruv Kedia, Kojin Oshiba, Yaron Singer, and Amin Karbasi. Llama-3.1-foundationai-securityllm-reasoni...

  69. [69]

    URLhttps://arxiv.org/abs/2601.21051

    arXiv

  70. [70]

    URLhttps://arxiv.org/abs/2406.12045

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.𝜏-bench: A benchmark for tool- agent-user interaction in real-world domains, 2024. URLhttps://arxiv.org/abs/2406.12045

    Pith/arXiv arXiv 2024

  71. [71]

    Primus: A pioneering collection of open-source datasets for cybersecurity llm training, 2025

    Yao-Ching Yu, Tsun-Han Chiang, Cheng-Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao. Primus: A pioneering collection of open-source datasets for cybersecurity llm training, 2025. URL https://arxiv.org/abs/2502.11191

    arXiv 2025

  72. [73]

    URLhttps://arxiv.org/abs/2606.14066. 33

    Pith/arXiv arXiv

  73. [75]

    type": "function

    URLhttps://arxiv.org/abs/2504.03160. A. Model and Evaluation Details This section provides the implementation and behavioral details needed to interpret the main evaluation. We first document the prompt and tool interface used to ensure a controlled comparison across models. We then compare complete Antares-3B and GPT-5.5 trajectories on the same reposito...

    Pith/arXiv arXiv 2000

  74. [76]

    Explore first(3–4 calls): map the repository structure before searching and form an initial model of the codebase organization

  75. [77]

    Targeted search(4–6 calls): use the structural overview to search for vulnerability-relevant patterns in likely directories

  76. [78]

    The model checkpoint, inference parameters, tool definitions, sandbox, scoring function, and evalu- ation entries remain unchanged

    Verify and read(3–5 calls): inspect candidate files and confirm that the vulnerability is imple- mented in the submitted paths. The model checkpoint, inference parameters, tool definitions, sandbox, scoring function, and evalu- ation entries remain unchanged. No retraining or gradient updates are performed. Table 13 | Command-distribution comparison acros...

    2026

  77. [2025]

    URLhttps://arxiv.org/abs/2505.18405

    Pith/arXiv arXiv

  78. [2026]

    URLhttps://arxiv.org/abs/2601.02346. 31

    arXiv