Pith. sign in

REVIEW 4 major objections 8 minor 4 cited by

CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repository-level code QA is still a hard, unsolved problem for large language models.

desk verdict The benchmark fills a real gap, but the missing grounding check means the central finding is not yet supported. read the letter →

arxiv 2501.03447 v1 pith:WIAV6IZF submitted 2025-01-07 cs.SE

Jialiang Chen , Kaifa Zhao , Jie Liu , Chao Peng , Jierui Liu , Hang Zhu , Pengfei Gao , Ping Yang
show 1 more author
Shuiguang Deng
This is my paper · ORCID
classification cs.SE
keywords coderepositoryquestionansweringLLM-as-a-judgebenchmarkconstructionGitHubissuesretrieval-augmentedgenerationlong-contextmodelscomprehensionCoReQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces CoReQA, a benchmark of 1,563 question-answer pairs built from real GitHub issues and comments across 176 popular repositories in four programming languages. Its aim is to test whether large language models can answer repository-level questions that require understanding code spread across many files, not just single functions or files. Using an LLM-as-a-judge that scores accuracy, completeness, relevance, and clarity, the authors find that GPT-4o, DeepSeek-V2, and Gemini-1.5 all score in the 5–8 range out of 10, with completeness particularly low. Providing BM25-retrieved context barely helps, and giving Gemini-1.5 the entire repository text improves scores only slightly. The paper concludes that repository-level QA remains an open challenge and argues for better retrieval and context-selection methods.

What carries the argument

The central object is the CoReQA benchmark itself, built by an automated pipeline that rewrites GitHub issue titles and bodies into questions, generates reference answers from issue comments with a chain-of-thought prompt, retrieves ten BM25 chunks as reference context, and scores model outputs with an LLM-as-a-judge on four dimensions plus pairwise comparisons. The pipeline carries the argument because every reported number is a product of this construction and evaluation process.

What would settle it

Have human annotators with software engineering expertise score the same 200 model answers that the LLM judge scored, then measure inter-rater agreement; if agreement is near chance, the benchmark's conclusion that models struggle is not established. Alternatively, run one model on a subset where the reference answer is the actual accepted fix from the issue thread, and see whether scores track the correctness of the accepted fix.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that state-of-the-art language models cannot yet answer repository-level code questions effectively. In the no-context setting, average scores across models are 6.37 accuracy, 5.70 completeness, 7.33 relevance, and 8.09 clarity out of 10; BM25 retrieval lifts these to 6.40, 5.74, 7.36, and 8.11. For the 44 long-context pairs tested, Gemini-1.5 with the entire repository as context reaches only 5.94 accuracy and 5.45 completeness. The authors interpret these results as evidence that neither generic lexical retrieval nor raw long-context access supplies the precise, semantically relevant information needed, and that the bottleneck is the combination of retrieval quality and the models' own repository-level reasoning.

Load-bearing premise

The load-bearing premise is that the LLM-generated reference answers and the LLM judge's scores truly measure answer quality; if the machine-written references are wrong or the judge is biased relative to human experts, the reported low scores would not show that models fail at repository-level QA.

Editorial extensions

If this is right

  • If repository-level QA is as hard as the benchmark suggests, then current AI coding assistants that answer questions in IDEs will tend to give incomplete or inaccurate answers on cross-file questions, since they usually rely on similar retrieval approaches.
  • Benchmarks that measure only code generation or completion cannot reveal this specific weakness; evaluations of code comprehension need repository-level QA pairs.
  • The small gain from BM25 indicates that improving retrieval, for example with semantic or structure-aware methods, is a necessary direction, and the paper's own results motivate that.
  • Long-context models benefit from more context on the tested subset but remain below the level of useful reliability, implying that context length alone is not a sufficient fix.
  • CoReQA can serve as a reusable testbed for future retrieval-augmented and long-context methods, allowing direct comparison of new approaches against the reported baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the reference answers and the judge are both LLM-generated, the absolute scores may partly reflect the judge's own blind spots; a human-evaluation subset would clarify how much of the failure is real.
  • Editorial inference: a testable extension the authors do not pursue is replacing BM25 with a code-aware retriever that respects function boundaries and call graphs, then re-running the same three models to isolate how much of the low ceiling is retrieval versus model reasoning.
  • Editorial inference: the same construction pipeline, issues plus positive comments, could be applied to languages beyond the four studied, but only if the LLM-generated references are validated in those languages first.
  • Editorial inference: the observed decline in scores for issues created closer to the models' training cutoff suggests the benchmark may partly measure memorization rather than comprehension; filtering out issues that appear in training data would give a cleaner signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. CoReQA is a repository-level question-answering benchmark assembled from GitHub issues and comments of popular repositories in Python, Java, Go, and TypeScript. The construction pipeline filters issues by tags, code content, and positive comments; rewrites issues into questions via LLM prompting; and generates reference answers by asking an LLM to summarize issue comments. Evaluation uses an LLM-as-a-judge with absolute scores (accuracy, completeness, relevance, clarity) and pairwise comparisons, plus a small long-context setting where Gemini-1.5 receives the whole repository. The main empirical claim is that state-of-the-art LLMs achieve only modest scores (e.g., average accuracy ~6.37/10 without context) and that neither BM25 retrieval nor full-repository context substantially improves performance, demonstrating that repository-level QA remains challenging. The paper also reports analyses by language, issue time period, and question length.

Significance. If validated, CoReQA fills a genuine gap: existing code QA benchmarks are mostly method- or file-level, template-based, or domain-specific, whereas CoReQA targets cross-file, real-world developer questions. The multi-language coverage (Python, Java, Go, TypeScript), the detailed filtering pipeline, and the temporal analysis of performance decay are useful contributions. The explicit discussion of limitations and threats to validity is commendable. However, the benchmark's construct validity is not yet established: the reference answers are machine-generated summaries of comments with only sample-level human checking, and the evaluator is an unnamed LLM judge validated only for self-consistency. These issues directly affect the credibility of the quantitative conclusions, so the significance is conditional on additional human grounding and judge validation.

major comments (4)
  1. [§III-A2, §III-C, Table VI] The reference answers in CoReQA are produced by prompting an LLM to summarize issue comments (Fig. 3), and the scores are produced by an LLM-as-a-judge whose only reported validation is the standard deviation of repeated scores (Table VI). No human-annotated gold set, no inter-annotator agreement, and no agreement between the judge and human raters are reported. Because both the reference and the judge are LLM outputs, the absolute scores in Tables III–V may measure how closely a model reproduces LLM-style summaries rather than true repository QA ability. This is load-bearing for the paper's central claim that state-of-the-art models 'struggle' at repository-level QA. Please add a human-validated subset (e.g., 100–200 pairs) with human-written or human-verified gold answers, report judge–human agreement, and name the judge model.
  2. [§IV-A2, Table V] The long-context experiment supplies only the repository text (Markdown-formatted) and not the issue comments from which the reference answers are generated. The paper never checks that the reference answer is inferable from the repository at the evaluated commit. Issue comments often contain workarounds, design rationale, or code that never enters the codebase; the filtering only excludes comments that point to other issues/commits, not comments whose content is absent from the repository. If a substantial fraction of the 44 long-context pairs have references not grounded in the repository, the low scores (e.g., 5.94 accuracy, 5.45 completeness) reflect missing information rather than model limitations. Please add a grounding analysis: for each long-context pair, annotate whether the reference answer is supported by the repository content at the evaluated commit, and report the fraction of ungrounded pairs. Also state how the 44 pairs were selected from the 1,563 total pairs.
  3. [§IV-D, Table VI, Table VII] RQ3 is titled 'Does the LLM-as-a-judge based evaluator effective in measuring question answering performance?' but the experiment only measures the judge's stability (standard deviation ~0.55 across five runs) and positional bias. Stability is not validity: a consistent judge can be consistently wrong. The 'margin of error of 5.5%' is the relative standard deviation of repeated scores, not an accuracy measure against human judgments. Table VII also shows nontrivial order effects, and the claim that positional bias 'does not affect overall judgments' is supported only by coarse aggregate direction, not by per-pair consistency. Please add a human agreement study (e.g., Cohen's kappa between the judge and human raters, or judge–human correlation) and provide the judge model name and full prompt.
  4. [§III-A2, §III-C] The paper does not identify the LLMs used for question rewriting, reference answer generation, or judging, and it does not include the evaluation prompts (the judge prompt is only mentioned in a footnote). For a benchmark paper, this makes the results non-reproducible and the benchmark difficult to reuse. Please specify all models with versions and access dates, include the full judge prompts in an appendix or supplementary material, and provide a dataset and code release link.
minor comments (8)
  1. [§I, §III-A2, Table I] The repository count is inconsistent: the abstract and Section I say 190 repositories, while Table I reports 176 total (Python 46 + Java 33 + Go 48 + TypeScript 49 = 176). Please align these numbers.
  2. [§IV-B] The text says 'Table V presents the overall results of our evaluation' for the no-context and BM25 settings, but Table V is actually the long-context results; the overall results are in Table III. Please fix the cross-reference.
  3. [§III-A2] There is a typo 'Tthe' in the Related Content Retrieval paragraph. Also, the paragraph does not clarify how the 44 long-context pairs were selected from the 1,563 QA pairs.
  4. [§VI-A] The ethics statement mentions 'manual sampling and verification on a subset of the annotated questions and answers,' but no numbers or criteria are given. Please quantify the subset and describe the verification procedure, since this is the only human check mentioned in the paper.
  5. [§III-C, Table VI] The BLEU scores in Table VI are said to be computed on 200 randomly selected QA pairs, but the caption does not state this; please add it to the caption.
  6. [§IV-D, Fig. 7] The text says 'in 5 out of 10 cases, the judger tends to assess the answer presented first as the better one,' but it is unclear whether '10 cases' refers to 10 pairs, 10 comparisons, or 10 model-pair settings. Please clarify the base of this fraction.
  7. [Fig. 3] The reference answer generation prompt has two numbered step 5 entries and a numbering inconsistency (step 6 appears after two step 5s). Please clean up the prompt formatting.
  8. [Table VII] The table description refers to 'muted mauve' and 'light pinkish' backgrounds, but the table as rendered appears to have no colors or has colors that are not described in a legend. Please add a legend or describe the shading in the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CoReQA's results are empirical measurements, not derivations; the LLM-judge dependence is an acknowledged validity limitation, not a tautological step.

full rationale

CoReQA is an empirical benchmark paper, not a derivation. The central claim—that LLMs score low on repository-level QA—is a measurement outcome, not a consequence of the benchmark's construction by definition. The reference answers are LLM-generated summaries of issue comments (Section III-A2, Fig. 3) and the evaluator is an LLM-as-a-judge (Section III-C), which creates a legitimate construct-validity concern: scores partly reflect agreement with LLM-generated references and an LLM judge's preferences rather than an independent ground truth. However, this is not circular in the required sense: there is no fitted parameter renamed as a prediction, no equation in which an output equals an input by construction, and no load-bearing self-citation or imported uniqueness theorem. The paper itself flags the limitation in Section VI-C ('the evaluation method relies on the inherent capabilities of the LLM'), so the concern is acknowledged rather than hidden. The additional risk that some reference answers may not be inferable from the repository alone is a validity threat to the long-context comparison, but it does not make the reported scores equivalent to the benchmark inputs. Accordingly, no circular step is exhibited, and the score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central measurement rests on hand-chosen filtering thresholds and on the assumption that LLM-generated references and LLM judge scores are valid without full human validation. There are no new physical or formal entities.

free parameters (7)
  • minimum positive comments = 3
    Issues with fewer than three positive emoji reactions are excluded. This hand-chosen threshold shapes benchmark difficulty and composition.
  • maximum issues per repository = 300
    Caps per-repo contribution to broaden coverage, affecting language distribution and question diversity.
  • repository selection thresholds = top 500 stars, 1,000 to 50,000 closed issues, >200K estimated tokens for long-context subset
    Hand-chosen popularity and scale filters determine which repositories are included.
  • token estimate divisor = 4 (characters divided by 4)
    Used to approximate repository token length for long-context selection; a coarse approximation.
  • BM25 top-k chunks = 5 per source, 10 total
    Reference context size for the retrieval setting, directly affecting BM25-condition results.
  • reference answer generation temperature = 0.8
    Mentioned in Section VI-A as a randomness source; no analysis justifies this value.
  • evaluation temperature = 0.2
    Set for evaluated models to make outputs more deterministic; the choice affects comparability across models.
assumptions (4)
  • domain assumption Positive comment reactions (+1, laugh, hooray, heart, rocket) indicate that comments correctly address the issue.
    Used to filter for quality in Section III-A1; emoji reactions signal agreement, not verified correctness.
  • domain assumption LLM-generated reference answers are correct and complete enough to serve as ground truth.
    Answers are produced by prompting an LLM to summarize issue comments (Figure 3); only sample-level human inspection is reported.
  • domain assumption LLM-as-a-judge scores reflect true answer quality across accuracy, completeness, relevance, and clarity.
    RQ3 checks stability (low variance) but not validity against human judgments; the judge model is not identified.
  • domain assumption BM25-retrieved chunks contain sufficient information to answer the question.
    Reference context is constructed with BM25 over LangChain chunks; the paper's own results suggest retrieval is weak, so this may not hold.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering." pith.science (2026). https://pith.science/paper/WIAV6IZF

@misc{pith2026250103447,
  author       = {Pith},
  title        = {Pith review of: CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WIAV6IZF}},
  note         = {Machine review of arXiv:2501.03447}
}
read the original abstract

Large language models that enhance software development tasks, such as code generation, code completion, and code question answering (QA), have been extensively studied in both academia and the industry. The models are integrated into popular intelligent IDEs like JetBrains and Cursor. Current benchmarks for evaluating models' code comprehension capabilities primarily focus on code generation or completion, often neglecting QA, which is a crucial aspect of understanding code. Existing code QA benchmarks are derived from code comments with predefined patterns (e.g., CodeQA) or focus on specific domains, such as education (e.g., CS1QA). These benchmarks fail to capture the real-world complexity of software engineering and user requirements for understanding code repositories. To address this gap, we introduce CoReQA, a benchmark for Code Repository-level question answering, constructed from GitHub issues and comments from 176 popular repositories across four programming languages. Since questions and answers may include both natural language and code snippets, traditional evaluation metrics such as BLEU are inadequate for assessing repository-level QA performance. Thus, we provide an LLM-as-a-judge framework to evaluate QA performance from five aspects. Based on CoReQA, we evaluate the performance of three baselines, including two short-context models using generic retrieval strategies and one long-context model that utilizes the entire repository context. Evaluation results show that state-of-the-art proprietary and long-context models struggle to address repository-level questions effectively. Our analysis highlights the limitations of language models in assisting developers in understanding repositories and suggests future directions for improving repository comprehension systems through effective context retrieval methodologies.

Figures

Figures reproduced from arXiv: 2501.03447 by the authors.

Figure 1
Figure 1. A question-answer pair from a sample GitHub issue and comments. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall pipeline of CoReQA. of requirements, alongside the aggregation of pertinent data, logs, and error messages. Despite the advancements in large language models [1]– [4], effectively tackling cross-file code comprehension remains a substantial hurdle. Interpreting the intent behind cross￾file requirements is often inconsistent due to inherent am￾biguity [34], [35]. Additionally, retrieving relevant contents… view at source ↗
Figure 4
Figure 4. Filtering and selection process for CoReQA. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Reference answer generation prompt. To ensure that CoReQA is capable of evaluating long￾context models, token lengths for each repository are estimated using OpenAI’s tokenizer calculation method [43]: the token length is approximated by dividing the number of characte…
Figure 5
Figure 5. Figure 5: Performance of models on QA pairs over various time periods. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Performance of GPT-4o over various question token [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pairwise evaluation comparison of evaluation results [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

    cs.SE 2026-07 conditional novelty 7.0 of 10

    RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.

  2. RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

    cs.SE 2026-08 conditional novelty 6.0 of 10

    RepoProbe uses GitHub Discussions and checklist-based verification to measure repository-level code understanding in LLMs, finding a persistent gap between fluent explanations and evidence-grounded correctness plus a ...

  3. Know Before Fix: QA-Driven Repository Knowledge Acquisition for Software Issue Resolution

    cs.SE 2026-07 conditional novelty 6.0 of 10

    QA-driven pre-repair knowledge acquisition (Questioner + Answerer, then Resolver) raises SWE-bench Verified Pass@1 by up to 4.4 points over Mini-SWE-Agent and other pre-repair methods.

  4. SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering

    cs.SE 2025-02 conditional novelty 6.0 of 10

    SyncBench quantifies agent out-of-sync recovery from 21 real repositories and finds state-of-the-art LLM agents succeed in under 34% of tasks and ask for help in under 5% of turns.

Reference graph

Works this paper leans on

54 extracted references · 32 canonical work pages · cited by 4 Pith papers

  1. [1]

    Deepseek: Advanced ai capabilities,

    D. AI, “Deepseek: Advanced ai capabilities,” https://www.deepseek. com/en, 2024, accessed: 2024-06-18

    work page 2024

  2. [2]

    Chatgpt: A large language model,

    OpenAI, “Chatgpt: A large language model,” https://www.openai.com/ chatgpt, 2024, accessed: 2024-06-18

    work page 2024

  3. [3]

    Claude: An ai assistant,

    Anthropic, “Claude: An ai assistant,” https://claude.ai/, 2024, accessed: 2024-06-18

    work page 2024

  4. [4]

    Gemini: A multimodal ai,

    G. DeepMind, “Gemini: A multimodal ai,” https://gemini.google.com/, 2024, accessed: 2024-06-18

    work page 2024

  5. [5]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

  6. [6]

    Cursor: Ai-powered coding assistant,

    Cursor, “Cursor: Ai-powered coding assistant,” https://www.cursor.com/, 2024, accessed: 2024-06-18

    work page 2024

  7. [7]

    Using an llm to help with code understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  8. [8]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

Show all 54 references
  1. [9]

    Repocoder: Repository-level code completion through iterative retrieval and generation,

    F. Zhang, B. Chen, Y . Zhang, J. Keung, J. Liu, D. Zan, Y . Mao, J.- G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023,...

    2023

  2. [10]

    Cctest: Testing and repairing code completion systems,

    Z. Li, C. Wang, Z. Liu, H. Wang, D. Chen, S. Wang, and C. Gao, “Cctest: Testing and repairing code completion systems,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 1238–1250

    2023

  3. [11]

    Learning deep semantics for test completion,

    P. Nie, R. Banerjee, J. J. Li, R. J. Mooney, and M. Gligoric, “Learning deep semantics for test completion,” in 2023 IEEE/ACM 45th Interna- tional Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2111–2123

    2023

  4. [12]

    Few-shot training llms for project-specific code-summarization,

    T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering , 2022, pp. 1–5

    2022

  5. [13]

    An analysis of the automatic bug fixing performance of chatgpt,

    D. Sobania, M. Briesch, C. Hanna, and J. Petke, “An analysis of the automatic bug fixing performance of chatgpt,” in 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) . IEEE, 2023, pp. 23–30

    2023

  6. [14]

    Analyzing bug fix for automatic bug cause classification,

    Z. Ni, B. Li, X. Sun, T. Chen, B. Tang, and X. Shi, “Analyzing bug fix for automatic bug cause classification,” Journal of Systems and Software, vol. 163, p. 110538, 2020

    2020

  7. [15]

    Generating bug- fixes using pretrained transformers,

    D. Drain, C. Wu, A. Svyatkovskiy, and N. Sundaresan, “Generating bug- fixes using pretrained transformers,” in Proceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming , 2021, pp. 1–8

    2021

  8. [16]

    Github copilot,

    GitHub, “Github copilot,” https://github.com/features/copilot, 2023, ac- cessed: 2024-06-19

    2023

  9. [17]

    Jetbrains: Essential tools for software developers and teams,

    JetBrains, “Jetbrains: Essential tools for software developers and teams,” https://www.jetbrains.com/, 2024, accessed: 2024-06-19

    2024

  10. [18]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

    2024

  11. [19]

    Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,

    K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,” arXiv preprint arXiv:2401.07339 , 2024

    2024 arXiv

  12. [20]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  13. [21]

    Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,

    Y . Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang, “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” in Thirty-seventh Conference on Neural Information Processing Syste...

    2023

  14. [22]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in The Twelfth International Conference on Learning Representations, 2023

    2023

  15. [23]

    Codequeries: A dataset of semantic queries over code,

    S. P. Sahu, M. Mandal, S. Bharadwaj, A. Kanade, P. Maniatis, and S. Shevade, “Codequeries: A dataset of semantic queries over code,” in Proceedings of the 17th Innovations in Software Engineering Confer- ence, 2024, pp. 1–11

    2024

  16. [24]

    Codeqa: A question answering dataset for source code comprehension,

    C. Liu and X. Wan, “Codeqa: A question answering dataset for source code comprehension,” in Findings of the Association for Computational Linguistics: EMNLP 2021 , 2021, pp. 2618–2632

    2021

  17. [25]

    Cs1qa: A dataset for assisting code- based question answering in an introductory programming course,

    C. Lee, Y . Seonwoo, and A. Oh, “Cs1qa: A dataset for assisting code- based question answering in an introductory programming course,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2022

  18. [26]

    A neural question answering system for basic questions about subroutines,

    A. Bansal, Z. Eberhart, L. Wu, and C. McMillan, “A neural question answering system for basic questions about subroutines,” in 2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2021, pp. 60–71

    2021

  19. [27]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globe...

    2023

  20. [28]

    Ragas: Au- tomated evaluation of retrieval augmented generation,

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “Ragas: Au- tomated evaluation of retrieval augmented generation,” arXiv preprint arXiv:2309.15217, 2023

    2023 arXiv

  21. [29]

    Simple bm25 extension to multiple weighted fields,

    S. Robertson, H. Zaragoza, and M. Taylor, “Simple bm25 extension to multiple weighted fields,” in Proceedings of the thirteenth ACM international conference on Information and knowledge management , 2004, pp. 42–49

    2004

  22. [30]

    Neural code comprehen- sion: A learnable representation of code semantics,

    T. Ben-Nun, A. S. Jakobovits, and T. Hoefler, “Neural code comprehen- sion: A learnable representation of code semantics,” Advances in neural information processing systems , vol. 31, 2018

    2018

  23. [31]

    Codecompass: an open software comprehension framework for industrial usage,

    Z. Porkol ´ab, T. Brunner, D. Krupp, and M. Csord ´as, “Codecompass: an open software comprehension framework for industrial usage,” in Proceedings of the 26th Conference on Program Comprehension , 2018, pp. 361–369

    2018

  24. [32]

    Fold2vec: Towards a statement-based representation of code for code comprehension,

    F. Bertolotti and W. Cazzola, “Fold2vec: Towards a statement-based representation of code for code comprehension,” ACM Transactions on Software Engineering and Methodology , vol. 32, no. 1, pp. 1–31, 2023

    2023

  25. [33]

    Repoagent: An llm-powered open-source framework for repository-level code documentation generation,

    Q. Luo, Y . Ye, S. Liang, Z. Zhang, Y . Qin, Y . Lu, Y . Wu, X. Cong, Y . Lin, Y . Zhang et al. , “Repoagent: An llm-powered open-source framework for repository-level code documentation generation,” arXiv preprint arXiv:2402.16667, 2024

    2024 arXiv

  26. [34]

    Intent detection for code-mix utterances in task oriented dialogue systems,

    P. Jayarao and A. Srivastava, “Intent detection for code-mix utterances in task oriented dialogue systems,” in 2018 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT), 2018, pp. 583–587

    2018

  27. [35]

    Build-a-bot: teaching conversational ai using a transformer-based intent recognition and ques- tion answering architecture,

    K. Pearce, S. Alghowinem, and C. Breazeal, “Build-a-bot: teaching conversational ai using a transformer-based intent recognition and ques- tion answering architecture,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 13, 2023, pp. 16 025–16 032

    2023

  28. [36]

    A survey on in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey on in-context learning,” arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  29. [37]

    Retrieve-and- sample: Document-level event argument extraction via hybrid retrieval augmentation,

    Y . Ren, Y . Cao, P. Guo, F. Fang, W. Ma, and Z. Lin, “Retrieve-and- sample: Document-level event argument extraction via hybrid retrieval augmentation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023, ...

    2023

  30. [38]

    From matching to generation: A survey on generative information retrieval,

    X. Li, J. Jin, Y . Zhou, Y . Zhang, P. Zhang, Y . Zhu, and Z. Dou, “From matching to generation: A survey on generative information retrieval,” arXiv preprint arXiv:2404.14851 , 2024

    2024 arXiv

  31. [39]

    Hyena hierarchy: Towards larger con- volutional language models,

    M. Poli, S. Massaroli, E. Nguyen, D. Y . Fu, T. Dao, S. Baccus, Y . Bengio, S. Ermon, and C. R´e, “Hyena hierarchy: Towards larger con- volutional language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 043–28 078

    2023

  32. [40]

    Transformer-xl: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. Carbonell, Q. V . Le, and R. Salakhutdi- nov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint arXiv:1901.02860 , 2019

    1901 arXiv

  33. [41]

    Soaring from 4k to 400k: Extending llm’s context with activation beacon,

    P. Zhang, Z. Liu, S. Xiao, N. Shao, Q. Ye, and Z. Dou, “Soaring from 4k to 400k: Extending llm’s context with activation beacon,” arXiv preprint arXiv:2401.03462, 2024

    2024 arXiv

  34. [42]

    The Stack: A Large-scale Context-Aware Code Dataset,

    Hugging Face, “The Stack: A Large-scale Context-Aware Code Dataset,” https://huggingface.co/datasets/bigcode/the-stack, Jan. 2023

    2023

  35. [43]

    Openai tokenizer,

    OpenAI, “Openai tokenizer,” https://platform.openai.com/tokenizer, 2024, accessed: 2024-06-18

    2024

  36. [44]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdh- ery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  37. [45]

    Fredriksson, D

    T. Fredriksson, D. Issa Mattos, J. Bosch, and H. Olsson, Data Labeling: An Empirical Investigation into Industrial Challenges and Mitigation Strategies, 11 2020, pp. 202–216

    2020

  38. [46]

    LangChain,

    H. Chase, “LangChain,” Oct. 2022. [Online]. Available: https: //github.com/langchain-ai/langchain

    2022

  39. [47]

    Rank-BM25: A Collection of BM25 Algorithms in Python,

    D. Brown, “Rank-BM25: A Collection of BM25 Algorithms in Python,”

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  41. [49]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline,

    T. Li, W.-L. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica, “From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline,” arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  42. [50]

    Chatbot arena: An open platform for evaluating llms by human preference,

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating llms by human preference,” 2024

    2024

  43. [51]

    Repobench: Benchmarking repository- level code auto-completion systems,

    T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repository- level code auto-completion systems,” arXiv:2306.03091, Jun 2023

    2023 arXiv

  44. [52]

    Six strategies for getting better results,

    OpenAI, “Six strategies for getting better results,” https://platform.openai.com/docs/guides/prompt-engineering/ six-strategies-for-getting-better-results, 2024

    2024

  45. [53]

    The stack dataset,

    BigCode, “The stack dataset,” 2024, collection of source code in over 300 programming languages. [Online]. Available: https://huggingface. co/datasets/bigcode/the-stack-dedup/blob/main/licenses.json 12

    2024

  46. [2020]

    Available: https://doi.org/10.5281/zenodo.4520057

    [Online]. Available: https://doi.org/10.5281/zenodo.4520057

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.