Pith. sign in

REVIEW 5 major objections 5 minor 53 references

A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Similarity-based retrieval-augmented generation improves code completion on closed-source codebases, and combining BM25 lexical retrieval with GTE-Qwen semantic retrieval gives the best results for most open-source LLMs.

desk verdict Industrial-scale RAG-for-code-completion study with a useful hybrid-retrieval takeaway, but the evidence base is too small and internally inconsistent to trust the quantitative rankings. read the letter →

arxiv 2507.18515 v1 pith:4UQ3VP7X submitted 2025-07-24 cs.SE

Zezhou Yang , Ting Peng , Cuiyun Gao , Chaozheng Wang , Hailiang Huang , Yuetang Deng This is my paper
classification cs.SE
keywords retrieval-augmentedgenerationcodecompletionlargelanguagemodelsclosed-sourcerepositoriessemanticretrievalBM25C++codebaseempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation is usually tested on public repositories; this paper asks whether it also works on a large proprietary codebase with its own idioms, frameworks, and conventions. The authors built a hand-annotated benchmark of 100 functions and a retrieval corpus from 1,669 internal C++ repositories, then ran 26 open-source LLMs from 0.5B to 671B parameters with two families of retrieval: identifier-based RAG, which looks up definitions of names in the code, and similarity-based RAG, which retrieves whole analogous functions. They report that both families improve completion quality over the base models, that similarity-based RAG improves considerably more, and that the best single retrievers are BM25 and GTE-Qwen, with their combination performing best on models of 7B parameters and above. If the results hold, engineers working on closed-source systems can raise code completion quality by adding a retrieval layer to an off-the-shelf open-source model, with no retraining and no exposure of proprietary code to a third-party service.

What carries the argument

The load-bearing mechanism is a fine-grained retrieval corpus built from 1,669 internal C++ repositories. A preprocessing algorithm extracts function definitions, function declarations, class definitions, and protobuf message definitions as separate retrieval units, recursively processes header dependencies, strips auto-generated protobuf code, and converts macros into function-like forms, so each indexed unit is a coherent object rather than an arbitrarily sliced file. On top of this corpus, identifier-based RAG uses a lookup service keyed by identifier name and type, with an LLM deciding which names need definitions, while similarity-based RAG indexes all function definitions and retrieves with either BM25 lexical scoring or cosine similarity over code embeddings; the retrieved snippet is then concatenated into the prompt. The paper also treats query completeness as part of the machinery, finding that most retrievers score better when the query is the complete code snippet, whereas GTE-Qwen is stronger with the incomplete code context that actually occurs during completion.

What would settle it

Take a random sample of real completion events from the production IDE, run the same RAG pipelines with the same 26 models, and compare CodeBLEU and Edit Similarity against the no-RAG baseline: if similarity-based RAG does not beat identifier-based RAG and the base model across most scales on this sample, the paper's consistency finding collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that in closed-source repositories, both identifier-based and similarity-based retrieval-augmented generation consistently improve code completion across models and scales, and that similarity-based RAG is the stronger family. The authors support this with CodeBLEU and Edit Similarity measurements on 26 open-source LLMs: for example, DeepSeek-V3 rises from 35.23/54.85 to 60.28/73.11 with GTE-Qwen retrieval, and Qwen2.5-Coder-32B rises to 63.73/72.25 when BM25 and GTE-Qwen are combined. They further claim that lexical and semantic retrieval are complementary because they return largely disjoint candidates, so the BM25+GTE-Qwen hybrid is optimal for most models at 7B and above, while small models below 7B may not benefit from the hybrid. A developer survey of three engineers on a random 52-example subset is presented as confirming that the hybrid completions look better to humans, with missing or incorrect logic as the dominant failure mode.

Load-bearing premise

The central claim depends on the 100 benchmark functions chosen and annotated by three developers from the authors' group being representative of real code completion requests in the WeChat codebase; if they over-represent central, well-documented code paths, the measured gains and the ranking of retrievers may not hold on ordinary development work.

Editorial extensions

If this is right

  • Teams with proprietary codebases can treat similarity-based RAG as a no-retraining upgrade path: attach a BM25 or GTE-Qwen index to an existing open-source LLM and expect higher completion scores across model sizes.
  • Identifier-based retrieval of function definitions is a smaller but still consistent win, useful when the completion depends on an internal API the model has never seen.
  • For models of 7B parameters and above, combining BM25 with GTE-Qwen should be the default configuration, because their candidates overlap little and the model can use both.
  • Retrieval queries should be chosen per technique: use the incomplete code context for GTE-Qwen and the complete function for BM25, UniXcoder, and CoCoSoDa.
  • Very small models below 7B should not automatically be given hybrid retrieval; the paper's numbers show the combination can hurt them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The near-zero overlap between lexical and semantic retrievers suggests that fusing any two structurally different retrievers, not just BM25 and GTE-Qwen, is a promising recipe for RAG-based code completion, if the generator is large enough to reconcile conflicting snippets.
  • Because the 100 benchmark functions were hand-picked by the authors' group, the absolute quality scores likely overstate what developers would see on arbitrary completion events; a random sample from production completion logs would convert the ranking into a deployment-ready estimate.
  • The small-model failure of the hybrid implies a capacity threshold below which extra retrieved context is noise; a testable extension would set retrieval budgets or hybrid ratios by model size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper reports an empirical study of retrieval-augmented generation (RAG) for code completion in the proprietary WeChat C++ codebase. The authors construct a 100-example manually annotated benchmark and a retrieval corpus from 1,669 internal repositories, then compare identifier-based RAG with similarity-based RAG using BM25, CodeBERT, UniXcoder, CoCoSoDa, and GTE-Qwen across 26 open-source LLMs ranging from 0.5B to 671B parameters. They report CodeBLEU and edit similarity scores and claim that both RAG families improve completion performance, that similarity-based RAG substantially outperforms identifier-based RAG, and that combining BM25 with GTE-Qwen yields the best results for most models. A developer survey with three developers is also reported.

Significance. If the empirical claims hold, the paper would provide useful practical guidance for teams working on proprietary code: adding similarity-based retrieval to open-source LLMs could improve code completion without retraining. The study's strengths include the unusually broad model coverage (26 LLMs, including very large models), the industrial-scale proprietary corpus, a fine-grained preprocessing algorithm for C++ header/proto/macro handling, and a human survey that complements automatic metrics. The main weakness is that the central quantitative comparisons rest on a small, manually selected benchmark with no confidence intervals, significance tests, or representativeness analysis, and some table values are internally inconsistent. These issues directly affect the strength of the headline conclusions, so the paper needs substantive revision before the claims can be accepted.

major comments (5)
  1. [Table III, first row] The BM25 baseline for Qwen2.5-Coder-0.5B-Instruct is reported as 27.63/32.45 in Table III, but Tables I and II report the same model's BM25 (incomplete) result as 31.43/41.25. The Table III value is identical to OpenCoder-1B's BM25 value in Table I, suggesting a copy/paste or alignment error. Since RQ3's conclusion that BM25+GTE-Qwen is optimal depends on comparing combination results against these baselines, this inconsistency must be resolved and the affected rows of Table III re-verified.
  2. [Section IV-A, Finding 1 and Table I] The claim that both RAG methods 'consistently improve code completion performance across different models and scales' is not supported by Table I. For example, CodeLlama-70B-Instruct has a base CB/ES of 22.50/33.10, yet every listed RAG variant is lower, including GTE-Qwen at 16.26/20.70 and function-definition retrieval at 11.60/17.39. Several other rows, such as CodeLlama-34B-Instruct, also contain many below-base values. The finding and the abstract's wording should be qualified to specify which models benefit, and the table caption's rule about gray highlighting for improvements over base should be checked for consistent application.
  3. [Section IV, Tables I-III] No statistical tests, confidence intervals, or per-example variance are reported anywhere in the experimental section. With only 100 benchmark examples, differences of 1-3 CodeBLEU points are likely within noise; for instance, DeepSeek-V2.5 achieves 55.67/69.18 with BM25 and 55.29/68.21 with GTE-Qwen, a difference of 0.38 CB points that cannot support the qualitative claim that one technique 'substantially' outperforms another. The authors should report paired significance tests (e.g., bootstrap or McNemar) and effect sizes for the headline comparisons, including the BM25+GTE-Qwen combination results in Table III.
  4. [Sections II-C and II-D1] The comparison between identifier-based and similarity-based RAG is confounded by corpus composition. Similarity-based RAG is explicitly restricted to function definitions ('we only utilize function definitions as the retrieval source for the similarity-based RAG method'), while identifier-based RAG retrieves protobuf messages, class definitions, and function declarations in addition to function definitions. The reported superiority of similarity-based RAG in RQ1 may therefore reflect the type of retrieval units rather than the retrieval strategy. The authors should either use the same underlying retrieval units for both families or add a controlled comparison, such as similarity-based retrieval over the same multi-type corpus.
  5. [Sections II-A and V-C] The evaluation benchmark consists of 100 manually selected functions annotated by three developers from the authors' group, with a stated goal of representing 'real-world code completion scenarios' but no quantitative evidence of representativeness. The paper acknowledges external validity as a threat in Section V-C but does not quantify how the selected functions compare with the distribution of real completion requests in WeChat (e.g., by function length, domain, call frequency, or difficulty). Without such an analysis, or a larger randomly sampled evaluation set, the measured gains and rankings may not generalize to actual development workloads. This is a load-bearing limitation for the central claims and should be addressed explicitly.
minor comments (5)
  1. [Section VI heading] The heading 'RELATED WROK' contains a typo and should read 'RELATED WORK'; Section IV-C's heading 'Exploration on Retreived Results' should be 'Retrieved Results'.
  2. [Section III-A] The BM25 parameters k and b are not reported; the authors should state the values used (or state that library defaults were used) because these parameters affect retrieval quality and the subsequent comparison.
  3. [Section II-C2] The identifier extraction step relies on Qwen2.5-72B-Instruct, which is also one of the evaluated generators, but no extraction accuracy or manual inspection results are reported; a brief evaluation of extraction quality would strengthen confidence in the identifier-based RAG results.
  4. [Section II-C3] The four prompt templates for identifier-based RAG are described but not shown; including them in an appendix would improve reproducibility.
  5. [General] The paper does not state what artifacts, if any, will be released; given the proprietary codebase this is understandable, but the authors should explicitly specify what can be shared (e.g., anonymized benchmark examples, prompts, and retrieval settings).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the reported findings are direct measurements, with only a non-load-bearing self-citation in the motivation.

full rationale

The paper is an empirical study whose central claims (both RAG families improve completion, similarity-based RAG outperforms identifier-based RAG, and BM25+GTE-Qwen is the best combination) are supported by direct measurements reported in Tables I-III on a 100-function benchmark. There is no fitted parameter that is later renamed as a prediction: BM25 uses standard TF-IDF/BM25 equations, semantic retrieval uses off-the-shelf encoders, all LLMs are frozen with temperature 0, and CodeBLEU/Edit Similarity are evaluation metrics rather than quantities solved for by the claimed effects. The identifier-extraction step for identifier-based RAG uses Qwen2.5-72B-Instruct, which is itself one of the evaluated LLMs; this is a possible measurement confound for that row and a comparison asymmetry with similarity-based RAG, but it is not a circular derivation because the retrieved background knowledge is real codebase content and the completion model still generates the output. Similarly, the benchmark is manually constructed by the authors and the validation survey comes from the same organization, but these are evidence-quality and external-validity concerns, not equation-level or definition-level circularity. The only self-citation, [5], supports the background statement that 87% of developers report efficiency improvements from code completion tools; it is not load-bearing for any of the paper's RAG findings. No uniqueness theorem, ansatz, or prior result from the authors is imported to force the paper's conclusions. Under the requirement that circularity must be exhibited as a specific reduction of an output to its input, no such reduction is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The evaluation depends on several unstated background assumptions about metric validity, benchmark representativeness, corpus sufficiency, and tool reliability. These are ordinary empirical assumptions, not hidden target results, but they are load-bearing because the paper draws general conclusions about RAG from a 100-example private benchmark.

free parameters (4)
  • BM25 k = not reported (BM25 default k approximately 1.5 assumed)
    Controls term frequency saturation in Eq. (11); lexical retrieval results depend on it.
  • BM25 b = not reported (BM25 default b approximately 0.75 assumed)
    Controls length normalization in Eq. (11); affects which functions are retrieved.
  • Number of retrieved snippets = 4
    Chosen in Section III-D.2 to keep prompts under 2k tokens; top-k directly affects RAG gains.
  • CodeBLEU weights = 0.25 each
    Set to alpha=beta=gamma=delta=0.25 in Eq. (12); all reported CodeBLEU scores depend on these weights.
assumptions (4)
  • domain assumption CodeBLEU and Edit Similarity adequately approximate code completion quality.
    These metrics are the primary evaluation signal, but the paper itself notes in Section V-C that they may not capture semantic correctness and functionality.
  • domain assumption The 100-example manual benchmark represents real code completion scenarios in WeChat.
    All RQ findings are measured on this benchmark; the authors acknowledge external validity limits in Section V-C but do not quantify representativeness.
  • domain assumption The retrieval corpus contains relevant code for the benchmark completion tasks.
    If a benchmark function has no similar code in the 1,669 repositories, RAG cannot help; corpus coverage is not independently measured.
  • ad hoc to paper Qwen2.5-72B-Instruct reliably extracts identifiers for identifier-based RAG.
    The Need_To_Lookup extraction step (Eq. 2) uses this model; extraction errors would systematically penalize identifier-based RAG.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat." pith.science (2026). https://pith.science/paper/4UQ3VP7X

@misc{pith2026250718515,
  author       = {Pith},
  title        = {Pith review of: A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4UQ3VP7X}},
  note         = {Machine review of arXiv:2507.18515}
}
read the original abstract

Code completion, a crucial task in software engineering that enhances developer productivity, has seen substantial improvements with the rapid advancement of large language models (LLMs). In recent years, retrieval-augmented generation (RAG) has emerged as a promising method to enhance the code completion capabilities of LLMs, which leverages relevant context from codebases without requiring model retraining. While existing studies have demonstrated the effectiveness of RAG on public repositories and benchmarks, the potential distribution shift between open-source and closed-source codebases presents unique challenges that remain unexplored. To mitigate the gap, we conduct an empirical study to investigate the performance of widely-used RAG methods for code completion in the industrial-scale codebase of WeChat, one of the largest proprietary software systems. Specifically, we extensively explore two main types of RAG methods, namely identifier-based RAG and similarity-based RAG, across 26 open-source LLMs ranging from 0.5B to 671B parameters. For a more comprehensive analysis, we employ different retrieval techniques for similarity-based RAG, including lexical and semantic retrieval. Based on 1,669 internal repositories, we achieve several key findings: (1) both RAG methods demonstrate effectiveness in closed-source repositories, with similarity-based RAG showing superior performance, (2) the effectiveness of similarity-based RAG improves with more advanced retrieval techniques, where BM25 (lexical retrieval) and GTE-Qwen (semantic retrieval) achieve superior performance, and (3) the combination of lexical and semantic retrieval techniques yields optimal results, demonstrating complementary strengths. Furthermore, we conduct a developer survey to validate the practical utility of RAG methods in real-world development environments.

Figures

Figures reproduced from arXiv: 2507.18515 by the authors.

Figure 1
Figure 1. Statistics of our benchmark. codebases, addressing the challenge of context extraction in RAG for code completion. 3) Our experiment results reveal the complementary nature of lexical and semantic retrieval techniques, demonstrat￾ing that their combination can further enhance RAG￾based code completion performance. 4) We validate our empirical findings through a developer survey, confirming that the observed performa… view at source ↗
Figure 2
Figure 2. The results analysis of developer survey. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 32 canonical work pages

  1. [1]

    GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model,

    W. Liu, A. Yu, D. Zan, B. Shen, W. Zhang, H. Zhao, Z. Jin, and Q. Wang, “GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model,” arXiv preprint arXiv:2406.07003, 2024

    arXiv 2024

  2. [2]

    Teaching Code LLMs to Use Autocompletion Tools in Repository- Level Code Generation,

    C. Wang, J. Zhang, Y . Feng, T. Li, W. Sun, Y . Liu, and X. Peng, “Teaching Code LLMs to Use Autocompletion Tools in Repository- Level Code Generation,” arXiv preprint arXiv:2401.06391 , 2024

    arXiv 2024

  3. [3]

    Language Models for Code Completion: A Practical Evalua- tion,

    M. Izadi, J. Katzy, T. van Dam, M. Otten, R. M. Popescu, and A. van Deursen, “Language Models for Code Completion: A Practical Evalua- tion,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20,

    work page 2024

  4. [4]

    RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems,

    T. Liu, C. Xu, and J. J. McAuley, “RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems,” inThe Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    work page 2024

  5. [5]

    How Practitioners Expect Code Completion?

    C. Wang, J. Hu, C. Gao, Y . Jin, T. Xie, H. Huang, Z. Lei, and Y . Deng, “How Practitioners Expect Code Completion?” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023 , S. Chandra, K. Blincoe, and P. Tonella, Eds....

    work page 2023

  6. [6]

    What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?

    S. Gao, X. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?” in 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 2023, pp. 761–773

    work page 2023

  7. [7]

    Generative Type Inference for Python,

    Y . Peng, C. Wang, W. Wang, C. Gao, and M. R. Lyu, “Generative Type Inference for Python,” in 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023. IEEE, 2023, pp. 988–999

    work page 2023

  8. [8]

    Evaluating Large Language Models in Class-Level Code Generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Evaluating Large Language Models in Class-Level Code Generation,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 . ACM, 2024, pp. 81:1–81:13

    work page 2024

Show all 53 references
  1. [9]

    Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning,

    X. Du, M. Wen, J. Zhu, Z. Xie, B. Ji, H. Liu, X. Shi, and H. Jin, “Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning,” in Findings of the Association for Computa- tional Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, Aug...

    2024

  2. [10]

    Exploring RAG-based Vulnerability Augmentation with LLMs,

    S. S. Daneshvar, Y . Nong, X. Yang, S. Wang, and H. Cai, “Exploring RAG-based Vulnerability Augmentation with LLMs,” arXiv preprint arXiv:2408.04125, 2024

    2024 arXiv

  3. [11]

    REPOFUSE: Repository-Level Code Completion with Fused Dual Context,

    M. Liang, X. Xie, G. Zhang, X. Zheng, P. Di, W. Jiang, H. Chen, C. Wang, and G. Fan, “REPOFUSE: Repository-Level Code Completion with Fused Dual Context,” CoRR, vol. abs/2402.14323, 2024

    2024 arXiv

  4. [12]

    Repo- former: Selective Retrieval for Repository-Level Code Completion,

    D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repo- former: Selective Retrieval for Repository-Level Code Completion,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

    2024

  5. [13]

    Dataflow-Guided Retrieval Augmen- tation for Repository-Level Code Completion,

    W. Cheng, Y . Wu, and W. Hu, “Dataflow-Guided Retrieval Augmen- tation for Repository-Level Code Completion,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11- 16, 2024 , ...

    2024

  6. [14]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , 2023

    2023 arXiv

  7. [15]

    RAG-Enhanced Commit Message Generation,

    L. Zhang, H. Zhang, C. Wang, and P. Liang, “RAG-Enhanced Commit Message Generation,” arXiv preprint arXiv:2406.05514 , 2024

    2024 arXiv

  8. [16]

    Studying LLM Performance on Closed-and Open-source Data,

    T. Ahmed, C. Bird, P. Devanbu, and S. Chakraborty, “Studying LLM Performance on Closed-and Open-source Data,” arXiv preprint arXiv:2402.15100, 2024

    2024 arXiv

  9. [17]

    ReACC: A Retrieval-Augmented Code Completion Framework,

    S. Lu, N. Duan, H. Han, D. Guo, S. Hwang, and A. Svyatkovskiy, “ReACC: A Retrieval-Augmented Code Completion Framework,” in Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, ...

    2022

  10. [18]

    FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval- Augmented Code Completion,

    Q. Guo, X. Li, X. Xie, S. Liu, Z. Tang, R. Feng, J. Wang, J. Ge, and L. Bu, “FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval- Augmented Code Completion,” in Proceedings of the 33rd ACM SIG- SOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, ...

    2024

  11. [19]

    Why do commercial companies contribute to open source software?

    M. Andersen-Gott, G. Ghinea, and B. Bygstad, “Why do commercial companies contribute to open source software?” International journal of information management , vol. 32, no. 2, pp. 106–117, 2012

    2012

  12. [20]

    Tencent Announces 2024 Third Quarter Results,

    Tencent, “Tencent Announces 2024 Third Quarter Results,” https://www. tencent.com/en-us/investors/financial-news.html, 2024.11.13

    2024

  13. [21]

    A comparison of open-source static analysis tools for vulnerability detection in c/c++ code,

    A. Arusoaie, S. Ciob ˆaca, V . Craciun, D. Gavrilut, and D. Lucanu, “A comparison of open-source static analysis tools for vulnerability detection in c/c++ code,” in 2017 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) . IEEE, ...

    2017

  14. [22]

    A language and platform independent co-simulation framework based on the functional mock-up interface,

    L. I. Hatledal, A. Styve, G. Hovland, and H. Zhang, “A language and platform independent co-simulation framework based on the functional mock-up interface,” IEEE Access, vol. 7, pp. 109 328–109 339, 2019

    2019

  15. [23]

    Identify- ing and understanding header file hotspots in c/c++ build processes,

    S. McIntosh, B. Adams, M. Nagappan, and A. E. Hassan, “Identify- ing and understanding header file hotspots in c/c++ build processes,” Automated Software Engineering , vol. 23, pp. 619–647, 2016

    2016

  16. [24]

    Retrieval meets long context large language models,

    P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro, “Retrieval meets long context large language models,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ...

    2024

  17. [25]

    Improvements to BM25 and Language Models Examined,

    A. Trotman, A. Puurula, and B. Burgess, “Improvements to BM25 and Language Models Examined,” in Proceedings of the 2014 Australasian Document Computing Symposium, ADCS 2014, Melbourne, VIC, Aus- tralia, November 27-28, 2014 , J. S. Culpepper, L. A. F. Park, and G. Zuccon, Eds....

    2014

  18. [26]

    CodeBERT: A Pre-Trained Model for Programming and Natural Languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A Pre-Trained Model for Programming and Natural Languages,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNL...

    2020

  19. [27]

    UniXcoder: Unified Cross-Modal Pre-training for Code Representation,

    D. Guo, S. Lu, N. Duan, Y . Wang, M. Zhou, and J. Yin, “UniXcoder: Unified Cross-Modal Pre-training for Code Representation,” in Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22...

    2022

  20. [28]

    CoCoSoDa: Effective Contrastive Learning for Code Search,

    E. Shi, Y . Wang, W. Gu, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun, “CoCoSoDa: Effective Contrastive Learning for Code Search,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023, pp. 2198–2210

    2023

  21. [29]

    To- wards General Text Embeddings with Multi-stage Contrastive Learning,

    Z. Li, X. Zhang, Y . Zhang, D. Long, P. Xie, and M. Zhang, “To- wards General Text Embeddings with Multi-stage Contrastive Learning,” CoRR, vol. abs/2308.03281, 2023

    2023 arXiv

  22. [30]

    Qwen2 Technical Report,

    Q. Team, “Qwen2 Technical Report,” CoRR, vol. abs/2407.10671, 2024

    2024 arXiv

  23. [31]

    Qwen2.5-Coder Technical Report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, A. Yang, R. Men, F. Huang, X. Ren, X. Ren, J. Zhou, and J. Lin, “Qwen2.5-Coder Technical Report,” CoRR, vol. abs/2409.12186, 2024

    2024 arXiv

  24. [32]

    DeepSeek LLM: Scaling Open-Source Language Models with Longtermism,

    DeepSeek-AI, “DeepSeek LLM: Scaling Open-Source Language Models with Longtermism,” CoRR, vol. abs/2401.02954, 2024

    2024 arXiv

  25. [33]

    DeepSeek-Coder-V2: Breaking the Barrier of Closed- Source Models in Code Intelligence,

    DeepSeek-AI, “DeepSeek-Coder-V2: Breaking the Barrier of Closed- Source Models in Code Intelligence,” CoRR, vol. abs/2406.11931, 2024

    2024 arXiv

  26. [34]

    Code Llama: Open Foundation Models for Code,

    B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scial...

    2023 arXiv

  27. [35]

    Yi: Open Foundation Models by 01.AI,

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y . Xu, Y . Liu, Y . Wang, Y . Cai, Z. Gu, Z. Liu, and Z. Dai, “Yi: Open Foundatio...

    2024 arXiv

  28. [36]

    OpenCoder: The Open Cookbook for Top- Tier Code Large Language Models,

    S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y . Xu, J. Yang, J. H. Liu, C. Zhang, L. Chai, R. Yuan, Z. Zhang, J. Fu, Q. Liu, G. Zhang, Z. Wang, Y . Qi, Y . Xu, and W. Chu, “OpenCoder: The Open Cookbook for Top- Tier Code Large Language Models,” CoRR, vol. abs/2411.04905, 2024

    2024 arXiv

  29. [37]

    Codestral,

    M. AI, “Codestral,” https://mistral.ai/news/codestral/, 2024.05.29

    2024

  30. [38]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models,

    Meta, “Llama 3.2: Revolutionizing edge AI and vision with open, customizable models,” https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/, 2024.09.25

    2024

  31. [39]

    Introducing Llama 3.1: Our most capable models to date,

    Meta, “Introducing Llama 3.1: Our most capable models to date,” https: //ai.meta.com/blog/meta-llama-3-1/, 2024.07.23

    2024

  32. [40]

    Llama-3.3-70B-Instruct,

    Meta, “Llama-3.3-70B-Instruct,” https://huggingface.co/meta-llama/ Llama-3.3-70B-Instruct, 2024.12.06

    2024

  33. [41]

    Qwen2.5: A Party of Foundation Models,

    Q. Team, “Qwen2.5: A Party of Foundation Models,” https://qwenlm. github.io/blog/qwen2.5/, 2024.09

    2024

  34. [42]

    DeepSeek-V3 Technical Report,

    DeepSeek-AI, “DeepSeek-V3 Technical Report,” CoRR, vol. abs/2412.19437, 2024

    2024 arXiv

  35. [43]

    CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,

    S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,” CoRR, vol. abs/2009.10297, 2020

    2009 arXiv

  36. [44]

    Bleu: a Method for Automatic Evaluation of Machine Translation,

    K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA . ACL, 2002, pp. 311–318

    2002

  37. [45]

    BM25S: Orders of magnitude faster lexical search via eager sparse scoring,

    X. H. L `u, “BM25S: Orders of magnitude faster lexical search via eager sparse scoring,” CoRR, vol. abs/2407.03618, 2024

    2024 arXiv

  38. [46]

    Re2G: Retrieve, Rerank, Generate,

    M. R. Glass, G. Rossiello, M. F. M. Chowdhury, A. Naik, P. Cai, and A. Gliozzo, “Re2G: Retrieve, Rerank, Generate,” in Proceedings of the 2022 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, NAACL 2022, ...

    2022

  39. [47]

    Don’t Forget to Connect! Improving RAG with Graph-based Reranking,

    J. Dong, B. Fatemi, B. Perozzi, L. F. Yang, and A. Tsitsulin, “Don’t Forget to Connect! Improving RAG with Graph-based Reranking,”arXiv preprint arXiv:2405.18414, 2024

    2024 arXiv

  40. [48]

    The Chronicles of RAG: The Retriever, the Chunk and the Generator,

    P. Finardi, L. Avila, R. Castaldoni, P. Gengo, C. Larcher, M. Piau, P. Costa, and V . Carid ´a, “The Chronicles of RAG: The Retriever, the Chunk and the Generator,” arXiv preprint arXiv:2401.07883 , 2024

    2024 arXiv

  41. [49]

    Code completion with statis- tical language models,

    V . Raychev, M. T. Vechev, and E. Yahav, “Code completion with statis- tical language models,” in ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’14, Edinburgh, United Kingdom - June 09 - 11, 2014 , M. F. P. O’Boyle and K. Pingali, Eds. ACM, 201...

    2014

  42. [50]

    When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference,

    Z. Sun, X. Du, F. Song, S. Wang, and L. Li, “When Neural Code Completion Models Size up the Situation: Attaining Cheaper and Faster Completion through Dynamic Model Inference,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–12

    2024

  43. [51]

    Exploring and Improving Code Completion for Test Code,

    T. Zhu, Z. Liu, T. Xu, Z. Tang, T. Zhang, M. Pan, and X. Xia, “Exploring and Improving Code Completion for Test Code,” in Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 2024, pp. 137–148

    2024

  44. [52]

    STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysis,

    J. Liu, Y . Chen, M. Liu, X. Peng, and Y . Lou, “STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysis,” arXiv preprint arXiv:2406.10018 , 2024

    2024 arXiv

  45. [2024]

    79:1–79:13

    ACM, 2024, pp. 79:1–79:13

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.