{"id":"012dd298-8d4c-4d75-a5e3-1ad96d5f79ef","arxiv_id":"2501.03447","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CoReQA is a new repository-level code QA benchmark built from GitHub issues; evaluation shows current large language models answer such questions poorly even with retrieved or full-repository context.","lead":"This paper introduces CoReQA, a benchmark of 1,563 question-answer pairs built from real issues in 176 popular GitHub repositories across Python, Java, Go, and TypeScript. It tests whether large language models can answer repository-level coding questions, and finds that even top models with full repository context score only mediocre marks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified grounding of reference answers in repository content: low long-context scores may reflect unanswerable questions, not model limitations.","rationale":"I found the reader's weakest assumption—LLM-generated references and judge without human validation—valid but not the most fundamental issue. Even a perfect human judge cannot rescue a benchmark whose reference answers are not entailed by the provided context. The paper's pipeline derives questions from issue bodies and answers from issue comments, while evaluation settings supply either no context, BM25-retrieved chunks, or the full repository, never the comments. The BM25 'reference context' is meant to be the repository-side evidence, but the reference answer is not checked against it. The paper's temporal analysis (Fig. 5) shows performance drops for recent issues, which the authors attribute to training-data contamination; it could also indicate repository drift, where older issues' resolutions have since been removed or changed, making them unanswerable from the current repo. The 44 long-context subset is small enough that a handful of ungrounded questions could skew the conclusion. My proposed human-grounded test would settle whether the reference answers are recoverable from the repository. If they are, the central claim survives this attack; if not, the benchmark needs substantial revision (e.g., pinning repositories to the issue-close commit and verifying answerability). Thus I keep the CONDITIONAL verdict, adding this as an explicit condition alongside the reader's.","tokens_in":17032,"tokens_out":7182,"duration_ms":65530,"concrete_test":"Select the 44 long-context QA pairs and an additional random 50 from the full set. For each, give a human developer the question and the complete repository text (with no issue comments), and ask them to answer from the repository alone. Then compare human answers to the reference answers on the key claims. If a human cannot locate the reference answer in a substantial fraction (e.g., >30%), the benchmark contains unanswerable questions and the 'struggle' conclusion is invalid. A cheaper automated version: check whether code snippets and distinctive phrases in each reference answer appear in the repository text at the evaluated commit; near-zero overlap flags grounding failures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that state-of-the-art LLMs struggle even when given the entire repository—implicitly assumes the reference answers are inferable from the repository text. But reference answers are generated by summarizing issue comments (Section III-A2, Fig. 3), and the long-context setting provides only the repository, not the comments (Section IV-A2). The paper never checks that the answer is actually grounded in the repository at the evaluated commit. Issue comments often contain workarounds, design rationale, or code that never enters the repository; the paper only excludes comments that point to other issues/commits, not comments whose content is absent from the codebase. If a substantial fraction of the 1,563 or the 44 long-context QA pairs have answers not derivable from the repository, the low scores (e.g., 5.94 accuracy with full repo) would reflect missing information rather than an LLM's inability to exploit context. This would break the benchmark's construct validity as a repository-QA test and directly undermine the paper's main conclusion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CoReQA is a repository-level question-answering benchmark assembled from GitHub issues and comments of popular repositories in Python, Java, Go, and TypeScript. The construction pipeline filters issues by tags, code content, and positive comments; rewrites issues into questions via LLM prompting; and generates reference answers by asking an LLM to summarize issue comments. Evaluation uses an LLM-as-a-judge with absolute scores (accuracy, completeness, relevance, clarity) and pairwise comparisons, plus a small long-context setting where Gemini-1.5 receives the whole repository. The main empirical claim is that state-of-the-art LLMs achieve only modest scores (e.g., average accuracy ~6.37/10 without context) and that neither BM25 retrieval nor full-repository context substantially improves performance, demonstrating that repository-level QA remains challenging. The paper also reports analyses by language, issue time period, and question length.","tokens_in":17195,"tokens_out":5569,"duration_ms":49946,"significance":"If validated, CoReQA fills a genuine gap: existing code QA benchmarks are mostly method- or file-level, template-based, or domain-specific, whereas CoReQA targets cross-file, real-world developer questions. The multi-language coverage (Python, Java, Go, TypeScript), the detailed filtering pipeline, and the temporal analysis of performance decay are useful contributions. The explicit discussion of limitations and threats to validity is commendable. However, the benchmark's construct validity is not yet established: the reference answers are machine-generated summaries of comments with only sample-level human checking, and the evaluator is an unnamed LLM judge validated only for self-consistency. These issues directly affect the credibility of the quantitative conclusions, so the significance is conditional on additional human grounding and judge validation.","major_comments":[{"comment":"The reference answers in CoReQA are produced by prompting an LLM to summarize issue comments (Fig. 3), and the scores are produced by an LLM-as-a-judge whose only reported validation is the standard deviation of repeated scores (Table VI). No human-annotated gold set, no inter-annotator agreement, and no agreement between the judge and human raters are reported. Because both the reference and the judge are LLM outputs, the absolute scores in Tables III–V may measure how closely a model reproduces LLM-style summaries rather than true repository QA ability. This is load-bearing for the paper's central claim that state-of-the-art models 'struggle' at repository-level QA. Please add a human-validated subset (e.g., 100–200 pairs) with human-written or human-verified gold answers, report judge–human agreement, and name the judge model.","section":"§III-A2, §III-C, Table VI"},{"comment":"The long-context experiment supplies only the repository text (Markdown-formatted) and not the issue comments from which the reference answers are generated. The paper never checks that the reference answer is inferable from the repository at the evaluated commit. Issue comments often contain workarounds, design rationale, or code that never enters the codebase; the filtering only excludes comments that point to other issues/commits, not comments whose content is absent from the repository. If a substantial fraction of the 44 long-context pairs have references not grounded in the repository, the low scores (e.g., 5.94 accuracy, 5.45 completeness) reflect missing information rather than model limitations. Please add a grounding analysis: for each long-context pair, annotate whether the reference answer is supported by the repository content at the evaluated commit, and report the fraction of ungrounded pairs. Also state how the 44 pairs were selected from the 1,563 total pairs.","section":"§IV-A2, Table V"},{"comment":"RQ3 is titled 'Does the LLM-as-a-judge based evaluator effective in measuring question answering performance?' but the experiment only measures the judge's stability (standard deviation ~0.55 across five runs) and positional bias. Stability is not validity: a consistent judge can be consistently wrong. The 'margin of error of 5.5%' is the relative standard deviation of repeated scores, not an accuracy measure against human judgments. Table VII also shows nontrivial order effects, and the claim that positional bias 'does not affect overall judgments' is supported only by coarse aggregate direction, not by per-pair consistency. Please add a human agreement study (e.g., Cohen's kappa between the judge and human raters, or judge–human correlation) and provide the judge model name and full prompt.","section":"§IV-D, Table VI, Table VII"},{"comment":"The paper does not identify the LLMs used for question rewriting, reference answer generation, or judging, and it does not include the evaluation prompts (the judge prompt is only mentioned in a footnote). For a benchmark paper, this makes the results non-reproducible and the benchmark difficult to reuse. Please specify all models with versions and access dates, include the full judge prompts in an appendix or supplementary material, and provide a dataset and code release link.","section":"§III-A2, §III-C"}],"minor_comments":[{"comment":"The repository count is inconsistent: the abstract and Section I say 190 repositories, while Table I reports 176 total (Python 46 + Java 33 + Go 48 + TypeScript 49 = 176). Please align these numbers.","section":"§I, §III-A2, Table I"},{"comment":"The text says 'Table V presents the overall results of our evaluation' for the no-context and BM25 settings, but Table V is actually the long-context results; the overall results are in Table III. Please fix the cross-reference.","section":"§IV-B"},{"comment":"There is a typo 'Tthe' in the Related Content Retrieval paragraph. Also, the paragraph does not clarify how the 44 long-context pairs were selected from the 1,563 QA pairs.","section":"§III-A2"},{"comment":"The ethics statement mentions 'manual sampling and verification on a subset of the annotated questions and answers,' but no numbers or criteria are given. Please quantify the subset and describe the verification procedure, since this is the only human check mentioned in the paper.","section":"§VI-A"},{"comment":"The BLEU scores in Table VI are said to be computed on 200 randomly selected QA pairs, but the caption does not state this; please add it to the caption.","section":"§III-C, Table VI"},{"comment":"The text says 'in 5 out of 10 cases, the judger tends to assess the answer presented first as the better one,' but it is unclear whether '10 cases' refers to 10 pairs, 10 comparisons, or 10 model-pair settings. Please clarify the base of this fraction.","section":"§IV-D, Fig. 7"},{"comment":"The reference answer generation prompt has two numbered step 5 entries and a numbering inconsistency (step 6 appears after two step 5s). Please clean up the prompt formatting.","section":"Fig. 3"},{"comment":"The table description refers to 'muted mauve' and 'light pinkish' backgrounds, but the table as rendered appears to have no colors or has colors that are not described in a legend. Please add a legend or describe the shading in the caption.","section":"Table VII"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful benchmark resource, but the current validation is insufficient for the strength of the claims. The central issues are: (1) both reference answers and scoring are LLM-generated with no human-grounded validation; (2) the long-context comparison does not establish that the reference answers are answerable from the repository text alone. These are fixable within the scope of a revision (e.g., adding a human-validated subset, grounding analysis, and a named judge model), so I do not recommend rejection. However, if the authors cannot provide meaningful human validation, the conclusions should be substantially softened. Also, the inconsistencies in repository counts and table cross-references suggest the manuscript needs careful proofreading before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoReQA targets a real gap: there is no repository-level code QA benchmark, and the existing ones are method- or file-level. The construction pipeline is described in detail, the temporal analysis showing performance drops for recent issues is a nice touch, and the authors are transparent about the LLM-as-judge limitations. So there is real work here.\n\nThe problem is that the reference answers are generated by summarizing issue comments, but the evaluation setting gives models only the repository text. The paper never checks that the answer is actually inferable from the code at the committed state. Issue comments often contain workarounds or design rationale that never enters the repo. If a substantial fraction of the 1,563 pairs (or the 44 long-context pairs) have answers not derivable from the repository, then the low scores—including the 5.94 accuracy with the full repository—reflect missing information, not model failure. That directly undermines the paper's main conclusion. The stress-test note lands.\n\nOther soft spots: the judge is an unnamed LLM validated only for self-consistency, not against human ratings. No dataset or code is released, which is a problem for a benchmark paper. And there are internal inconsistencies (176 vs 190 repos, Table V vs Table III) that suggest the manuscript was assembled in a hurry.\n\nThat said, the core idea is sound, and the benchmark could become a useful resource if the authors add a grounding analysis (e.g., check that the answer appears in the repo or in retrieved chunks), validate the judge against humans, and release the artifacts. The current version is not ready for acceptance, but it deserves a serious referee who will demand those fixes. I would engage with it, but I would not cite it until the validity issues are resolved.","headline":"The benchmark fills a real gap, but the missing grounding check means the central finding is not yet supported.","tokens_in":17782,"tokens_out":3084,"would_cite":false,"duration_ms":27848,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repository-level code QA is still a hard, unsolved problem for large language models.","keywords":["code repository question answering","LLM-as-a-judge","benchmark construction","GitHub issues","retrieval-augmented generation","long-context models","code comprehension","CoReQA"],"falsifier":"Have human annotators with software engineering expertise score the same 200 model answers that the LLM judge scored, then measure inter-rater agreement; if agreement is near chance, the benchmark's conclusion that models struggle is not established. Alternatively, run one model on a subset where the reference answer is the actual accepted fix from the issue thread, and see whether scores track the correctness of the accepted fix.","tokens_in":16813,"feed_emoji":"🤖","tokens_out":5151,"duration_ms":41501,"temperature":0.7,"pith_summary":"This paper introduces CoReQA, a benchmark of 1,563 question-answer pairs built from real GitHub issues and comments across 176 popular repositories in four programming languages. Its aim is to test whether large language models can answer repository-level questions that require understanding code spread across many files, not just single functions or files. Using an LLM-as-a-judge that scores accuracy, completeness, relevance, and clarity, the authors find that GPT-4o, DeepSeek-V2, and Gemini-1.5 all score in the 5–8 range out of 10, with completeness particularly low. Providing BM25-retrieved context barely helps, and giving Gemini-1.5 the entire repository text improves scores only slightly. The paper concludes that repository-level QA remains an open challenge and argues for better retrieval and context-selection methods.","feed_headline":"Full repository, weak answers: LLMs score ~6/10 on code QA","feed_subtitle":"CoReQA benchmark: even with all repo text, Gemini-1.5 gets 5.94/10 accuracy on real GitHub issues.","key_machinery":"The central object is the CoReQA benchmark itself, built by an automated pipeline that rewrites GitHub issue titles and bodies into questions, generates reference answers from issue comments with a chain-of-thought prompt, retrieves ten BM25 chunks as reference context, and scores model outputs with an LLM-as-a-judge on four dimensions plus pairwise comparisons. The pipeline carries the argument because every reported number is a product of this construction and evaluation process.","core_discovery":"On the paper's own terms, the central discovery is that state-of-the-art language models cannot yet answer repository-level code questions effectively. In the no-context setting, average scores across models are 6.37 accuracy, 5.70 completeness, 7.33 relevance, and 8.09 clarity out of 10; BM25 retrieval lifts these to 6.40, 5.74, 7.36, and 8.11. For the 44 long-context pairs tested, Gemini-1.5 with the entire repository as context reaches only 5.94 accuracy and 5.45 completeness. The authors interpret these results as evidence that neither generic lexical retrieval nor raw long-context access supplies the precise, semantically relevant information needed, and that the bottleneck is the combination of retrieval quality and the models' own repository-level reasoning.","pith_inferences":["Editorial inference: because the reference answers and the judge are both LLM-generated, the absolute scores may partly reflect the judge's own blind spots; a human-evaluation subset would clarify how much of the failure is real.","Editorial inference: a testable extension the authors do not pursue is replacing BM25 with a code-aware retriever that respects function boundaries and call graphs, then re-running the same three models to isolate how much of the low ceiling is retrieval versus model reasoning.","Editorial inference: the same construction pipeline, issues plus positive comments, could be applied to languages beyond the four studied, but only if the LLM-generated references are validated in those languages first.","Editorial inference: the observed decline in scores for issues created closer to the models' training cutoff suggests the benchmark may partly measure memorization rather than comprehension; filtering out issues that appear in training data would give a cleaner signal."],"forward_implications":["If repository-level QA is as hard as the benchmark suggests, then current AI coding assistants that answer questions in IDEs will tend to give incomplete or inaccurate answers on cross-file questions, since they usually rely on similar retrieval approaches.","Benchmarks that measure only code generation or completion cannot reveal this specific weakness; evaluations of code comprehension need repository-level QA pairs.","The small gain from BM25 indicates that improving retrieval, for example with semantic or structure-aware methods, is a necessary direction, and the paper's own results motivate that.","Long-context models benefit from more context on the tested subset but remain below the level of useful reliability, implying that context length alone is not a sufficient fix.","CoReQA can serve as a reusable testbed for future retrieval-augmented and long-context methods, allowing direct comparison of new approaches against the reported baselines."],"supporting_citations":[{"why":"Provides the existing single-file code QA benchmark that CoReQA contrasts with to argue for repository-level evaluation.","marker":"[24]"},{"why":"Supplies the education-focused code QA dataset that CoReQA positions as too narrow for real-world repository understanding.","marker":"[25]"},{"why":"Establishes the real-world GitHub-issue evaluation style and also appears in the BM25 retrieval citation chain, anchoring the task setting.","marker":"[22]"},{"why":"Offers a cross-file QA dataset that CoReQA extends by using whole repositories and natural user questions.","marker":"[23]"},{"why":"Supplies the LLM-as-a-judge method that the CoReQA evaluator is built on, including the stability rationale.","marker":"[27]"},{"why":"Defines the BM25 retrieval algorithm used to create the reference context for short-context model evaluations.","marker":"[29]"},{"why":"Provides the self-consistency technique used to format and stabilize question generation from issues.","marker":"[44]"},{"why":"Supplies the chain-of-thought prompting used both for reference answer generation and for the judge's scoring prompts.","marker":"[48]"}],"fun_headline_variants":["Even full repo context: LLMs score only 5.94/10 on QA","CoReQA: Full repo context fails, LLMs get ~6/10","Repo QA: full context isn't enough, best LLM gets 5.94/10","Long-context models choke on repository QA: 5.94 accuracy","LLMs underperform on repo QA even with all code: 5.94/10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated reference answers and the LLM judge's scores truly measure answer quality; if the machine-written references are wrong or the judge is biased relative to human experts, the reported low scores would not show that models fail at repository-level QA.","fun_headline_variants_meta":{"raw":{"variants":["Even full repo context: LLMs score only 5.94/10 on QA","CoReQA: Full repo context fails, LLMs get ~6/10","Repo QA: full context isn't enough, best LLM gets 5.94/10","Long-context models choke on repository QA: 5.94 accuracy","LLMs underperform on repo QA even with all code: 5.94/10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001779,"raw_usage":{"total_tokens":7048,"prompt_tokens":1015,"completion_tokens":6033,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":5923}},"tokens_in":631,"tokens_out":6033,"duration_ms":39202,"temperature":1.0,"reasoning_tokens":5923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:32.523228+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators with software engineering expertise score the same 200 model answers that the LLM judge scored, then measure inter-rater agreement; if agreement is near chance, the benchmark's conclusion that models struggle is not established. Alternatively, run one model on a subset where the reference answer is the actual accepted fix from the issue thread, and see whether scores track the correctness of the accepted fix.","supporting_citations":[{"cited_title":"Codeqa: A question answering dataset for source code comprehension,","cited_arxiv_id":null,"evidence_quote":"Provides the existing single-file code QA benchmark that CoReQA contrasts with to argue for repository-level evaluation."},{"cited_title":"Cs1qa: A dataset for assisting code- based question answering in an introductory programming course,","cited_arxiv_id":null,"evidence_quote":"Supplies the education-focused code QA dataset that CoReQA positions as too narrow for real-world repository understanding."},{"cited_title":"Swe-bench: Can language models resolve real-world github issues?","cited_arxiv_id":null,"evidence_quote":"Establishes the real-world GitHub-issue evaluation style and also appears in the BM25 retrieval citation chain, anchoring the task setting."},{"cited_title":"Codequeries: A dataset of semantic queries over code,","cited_arxiv_id":null,"evidence_quote":"Offers a cross-file QA dataset that CoReQA extends by using whole repositories and natural user questions."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-as-a-judge method that the CoReQA evaluator is built on, including the stability rationale."},{"cited_title":"Simple bm25 extension to multiple weighted fields,","cited_arxiv_id":null,"evidence_quote":"Defines the BM25 retrieval algorithm used to create the reference context for short-context model evaluations."}],"review_version":1}