{"id":"29637a10-def3-4efe-92cd-743c3a83f662","arxiv_id":"2508.02997","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MRG-Bench is a multi-language, runnable-test benchmark showing that LLMs' main failure in repository-level code generation is understanding user requirements.","lead":"This paper presents MRG-Bench, a new benchmark for repository-level code generation across Python, Java, and Go with runnable test cases. It reports that today's best models pass under 33% of tasks and that most failures stem from misunderstanding user requirements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 22-project, 383-sample selection pipeline can shift task difficulty and code distribution, so the headline Pass@1 and failure-cause proportions may not generalize beyond the retained repositories.","rationale":"The paper is internally coherent as MRG-Bench: it releases a runnable multilingual benchmark, reports Pass@1 on current models, and uses LLM consensus annotation to attribute failures. The strongest evidence is the released code and Docker images, runnable tests, and the CDD leakage check in Table 8. The load-bearing condition for the headline numbers is that the 22 retained projects are not a skewed subset. The filtering in Sections 3.2-3.4 is intense, and RQ1.1's embedding-based representativeness test is too weak to rule out selection effects: it checks docstring semantics, not task difficulty, dependency structure, or test strictness. A representative-docstring distribution does not imply representative code-generation tasks. If the pipeline were re-run on fresh repositories, the Pass@1 aggregate and the What/How split could both move materially. This warrants keeping the reader's conditional verdict until the benchmark is validated on an independent sample or until error bars make the uncertainty visible. I do not see an internal contradiction that would justify rejection; the concern is about generalization, not soundness.","tokens_in":18798,"tokens_out":6236,"duration_ms":71328,"concrete_test":"Run the full MRG-Bench pipeline on 20-30 additional repositories drawn from the same language/star/date filter using the released framework, then recompute per-language Pass@1 and the What/How failure proportions on the added samples. If average Pass@1 shifts by more than about 5 points or the 'What information' share drops below 50%, the 22-project subset is not stable enough to support the headline conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims—that current models plateau at 32.5% Pass@1 and that over 68% of failures stem from missing 'What information'—both inherit a common condition: MRG-Bench's 383 samples must be representative of repository-level code generation tasks. That condition is weakest where the dataset is filtered hardest. Sections 3.2-3.4 reduce 1,000 candidate repositories to 22, using criteria that exclude projects without runnable Linux test suites, functions without developer comments, functions whose test links cannot be inferred from the call-graph heuristic, and any function not reaching 100% line coverage. Each criterion is reasonable operationally, but together they define a narrow task distribution: well-documented, fully test-covered, call-graph-resolvable functions in recent popular repositories. The RQ1.1 representativeness check does not resolve this: Figure 3b's reconstruction error compares only docstring embeddings of MRG-Bench against a 10,000-function random sample, and a small real-repo sample will look close to such a large reference set regardless of whether the selection filters bias task difficulty. If the retained projects are atypical, the measured Pass@1 and the failure-cause proportions in Figure 5 need not transfer to typical development.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MRG-Bench, a multi-language repository-level code generation benchmark containing 383 function-level samples from 22 real-world open-source projects (Python, Java, and Go), together with a call-graph analysis framework and runnable test environments. It evaluates a range of LLMs, long-context models, reasoning models, and RAG methods, reporting that the best model (Claude-3.5-Sonnet) achieves only 32.5% Pass@1 with in-file context, that RAG methods underperform simple in-file context, and that over 68% of failures are attributed to missing 'What information' (understanding user requirements), based on an LLM-based failure annotation protocol. The benchmark, data, and code are publicly released.","tokens_in":19034,"tokens_out":7496,"duration_ms":78266,"significance":"If the results hold, MRG-Bench would be a valuable resource for repository-level code generation evaluation, particularly for its multi-language coverage and runnable test cases. The central finding that models plateau at 32.5-40% Pass@1 and that requirement understanding is the dominant failure cause is practically important and could guide future research on context augmentation. The paper also provides a reusable call-graph analysis tool. However, the heavy filtering in dataset construction, the small sample size, and the unvalidated LLM-based failure annotation mean that the quantitative claims currently have limited generalizability. With appropriate additional analysis, this could become a solid benchmark contribution.","major_comments":[{"comment":"The selection pipeline retains only 22 of 152 repositories (383 of 580 candidate functions) using criteria that include developer-written comments and 100% line coverage, which defines a narrow task distribution. Consequently, the headline Pass@1 of 32.5% (Table 4) and the failure-cause proportions in Figure 5 may not generalize to typical repository-level code generation. The RQ1.1 representativeness check is insufficient to address this: the reconstruction-error metric compares docstring embeddings of the 383 queries against a 10,000-function reference set, and a small query set will mechanically have low nearest-neighbor error even if the selected functions are atypical in difficulty or complexity. Please provide evidence that Pass@1 and the 'What vs How' proportions are stable across subsets (e.g., per language, per repository, or by coverage level), and discuss the likely effect of the 100% line-coverage filter on task difficulty.","section":"Sections 3.3–3.4 and 5.1"},{"comment":"The central claim that over 68% of failures stem from missing 'What information' rests entirely on LLM majority-vote annotation (five models, retaining only 5:0 and 4:1 agreements and discarding 3:2 cases) with no human validation. There is no evidence that the LLM vote corresponds to ground-truth failure causes, and the 13.7% of discarded cases may be systematically different from the retained ones. To make RQ3's conclusion load-bearing, please provide a human-annotated subset (e.g., 50–100 cases), report inter-annotator agreement between the LLM voting and human labels, and show the sensitivity of the reported proportions to the agreement threshold.","section":"Section 5.3 and Figure 5"},{"comment":"The reconstruction-error definition (average distance from each query to its nearest neighbor in the reference set) is sensitive to the sizes of Q and P. Because MRG-Bench has only 383 queries while EvoCodeBench and CoderEval-Java likely have many more, the lower reconstruction error for MRG-Bench could reflect sample size rather than superior representativeness. The paper should report the sizes of the compared query sets, use a sample-size-controlled baseline (e.g., repeatedly subsample the larger datasets to 383 and recompute), or adopt a distributional distance such as MMD so that the RQ1.1 comparison is fair and meaningful.","section":"Section 5.1, Eq. (1)"},{"comment":"All Pass@1 and Pass@3 results are reported as point estimates without confidence intervals or significance tests, despite small per-language sample sizes (e.g., 96 Java and 124 Python functions). Several comparative conclusions, such as 'RAG-related methods are inferior to in-file context' (Table 7 vs. Table 4) and 'providing callee functions yields less improvement than in-file context' (Table 5), rely on differences of a few percentage points that may be within sampling noise. Please report bootstrap confidence intervals or paired permutation tests for the key model and context comparisons, and clarify how Pass@k is computed (e.g., whether it is the unbiased estimator used in prior work or a simple sample mean over fixed samples).","section":"Section 4.2 and Tables 3–7"}],"minor_comments":[{"comment":"The text refers to the open-source long-context model as DeepSeek-V3, while Table 6 lists 'DeepSeek-V2.5'; please correct this inconsistency.","section":"Section 5.2 and Table 6"},{"comment":"The dataset name is inconsistently spelled as 'EvolCodeBench' and 'EvoCodeBench'; standardize the spelling throughout.","section":"Sections 1, 3.2, and Table 1"},{"comment":"The names of the coverage measurement tools are displayed as placeholder symbols; please list the actual tools (e.g., coverage.py, JaCoCo, go test -cover) so the procedure is reproducible.","section":"Section 3.4"},{"comment":"The prompt used for failure annotation is not visible in the text; please include the full prompt text in the caption or an appendix so that the annotation instruction is reproducible and auditable.","section":"Figure 4"},{"comment":"The reference list appears twice at the end of the manuscript; remove the duplicate set.","section":"References"},{"comment":"The headline 32.5% Pass@1 for Claude-3.5-Sonnet is measured under the in-file-context setting (Table 4), but neither the abstract nor the first section states this context, which could mislead readers about the model's baseline performance; please specify the setting explicitly.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript body is titled 'MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation', but the reviewer-face title in the prompt is 'CoCoTen: Detecting Adversarial Inputs...'; the editorial office should verify that the correct paper has been assigned, as the abstract and full text are entirely about MRG-Bench. The benchmark contribution is sound in conception and releases useful artifacts, but the generalizability and annotation-validation concerns above require substantial additional analysis before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: arXiv 2508.02997 is listed as CoCoTen, about adversarial-input detection via co-occurrence tensors, but the manuscript you actually read is MRG-Bench, a repository-level code generation benchmark. The arXiv metadata credits Kadali and Papalexakis; the paper text is authored by Li, Gao, and Zhang. That is a metadata integrity problem, and it has to be fixed before this goes anywhere. I am reviewing the content as it appears, not the title.\n\nThe MRG-Bench part is a decent, incremental contribution. It adds multi-language (Python, Java, Go) coverage with runnable project-level tests, and the authors release the data, code, and test logs. The function-call-graph extraction framework is a concrete tool others can reuse. Empirically, they test a genuinely wide spread of models (open, closed, reasoning, RAG) and show that the best get only about 32.5% Pass@1 with in-file context, and that RAG methods underperform simply providing the file. The CDD leakage check is good practice. The What/How failure annotation is a nice idea, decomposing failures into requirement understanding versus implementation.\n\nThe soft spots are real but expected for a benchmark this size. The dataset is 383 samples from 22 repositories after heavy filtering: developer comments, test-link inference, and 100% line coverage. The authors acknowledge the low coverage in Section 6, which I respect, but the acknowledgment does not fix the generalizability question. The RQ1.1 representativeness check compares docstring embeddings of MRG-Bench against a random sample of 10k functions; that shows the comments look similar, but it does not tell you whether the filtered tasks have similar difficulty or code distribution to typical development. The headline Pass@1 and the claim that over 68% of failures are 'What' problems both inherit that condition. Also, Pass@k has no error bars, and the failure annotation uses five LLMs voting (5:0 or 4:1 kept) without human validation. It is a reasonable proxy, but still a proxy.\n\nDespite those caveats, this paper deserves a serious referee. The dataset and the call-graph framework are shipped, reproducible artifacts, and the multi-language runnable-test angle fills a clear gap in the benchmark literature. The right fix is to correct the title, abstract, and author list first, then send it out with reviewers asked to scrutinize the filtering pipeline and the annotation protocol. I would not desk-reject the MRG-Bench content; I would reject the current metadata. Fix that and it is a legitimate submission. I would probably bring it to a reading group for the benchmark design discussion.","headline":"Despite the CoCoTen title/abstract and mismatched author list, the actual manuscript is MRG-Bench, a solid and reusable multi-language repo-level code-gen benchmark that deserves peer review after the metadata is fixed.","tokens_in":19574,"tokens_out":3255,"would_cite":true,"duration_ms":39325,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repository-level code generation fails mainly because models cannot understand what the user wants, not because they lack implementation knowledge.","keywords":["repository-level code generation","LLM evaluation benchmark","multilingual code generation","Pass@1","failure cause analysis","retrieval-augmented generation","contextual information","function call graph"],"falsifier":"Run the same failure annotation on a new sample of repository functions drawn without the 100% line-coverage filter; if the share of failures attributed to missing 'What information' falls well below 68% and context rescues Python failures as much as Java and Go, the paper's central claim would be a filtering artifact rather than a general property of current models.","tokens_in":18581,"feed_emoji":"🧩","tokens_out":8939,"duration_ms":92561,"temperature":0.7,"pith_summary":"This paper introduces MRG-Bench, a multilingual, repository-level code generation benchmark built from 383 real-world functions across Python, Java, and Go, with runnable test cases and 100% line coverage. Evaluated on this benchmark, current LLMs perform poorly: the best model, Claude-3.5-Sonnet, reaches only 32.5% average Pass@1, and RAG methods do worse than simply providing the file containing the target function. By annotating failure cases with five-model voting, the paper finds that over 68% of failures come from missing 'What information'—the model cannot map the user requirement to the functionality it calls for—rather than missing implementation knowledge. The paper also shows that context needs differ by language: in-file content helps Java and Go but barely helps Python. The central conclusion is that current models are weakest at requirement understanding, so improving that stage is the most promising route to better repository-level code generation.","feed_headline":"Best coding model passes just 32.5% of real repo tasks","feed_subtitle":"A new multilingual benchmark shows most failures come from not understanding the ask, not missing code.","key_machinery":"The central object is MRG-Bench, a 383-sample benchmark of function-level code generation tasks drawn from 22 real-world repositories, each sample containing a developer-written comment, function signature, referenced function bodies, test cases, called private functions, and repository information. Its construction pipeline uses per-language function call graph analyzers, links test functions to target functions, and filters to functions with 100% line coverage. The argument is carried by a two-axis failure taxonomy, 'What to do' versus 'How to do it', applied through a five-model voting annotation prompt that retains only 5:0 and 4:1 agreement cases, covering 86.3% of failures.","core_discovery":"The central claim is that state-of-the-art LLMs fail at repository-level code generation primarily because they cannot comprehend what functionality the user requirement corresponds to, not because they lack implementation details. The paper supports this by decomposing generation into a 'What to do' stage and a 'How to do it' stage, then using a five-model voting prompt to classify Claude-3.5-Sonnet's failed cases; over 68% of failures are attributed to missing 'What information'. Providing callable functions, the 'How information', yields only modest gains, while providing in-file context gives larger but language-dependent improvements. Python is the outlier: it has weaker semantic associations between functions, which explains why context rarely rescues its failures. The paper positions MRG-Bench as a more realistic evaluation than single-language, standalone benchmarks and argues that future methods should mine 'What information' for each language separately.","pith_inferences":["A direct test of the paper's main claim would be to build a retriever that fetches repository documentation and sibling implementations for each requirement and check whether Python failure rates drop; the paper's analysis predicts they will not.","The heavy filtering—developer comments, matched tests, 100% coverage, and 22 of 152 surviving repositories—may select for well-specified functions, so a broader sample could shift the reported 'What information' share.","The five-model voting labels inherit the annotators' own blind spots, so the taxonomy would be stronger if a subset were checked against human annotations.","The language difference implies a directly measurable property: the alignment between function comments, names, and call graphs is weaker in Python than in Java and Go."],"forward_implications":["Improving requirement understanding should raise Pass@1 more than adding code context, because most failures occur before implementation begins.","RAG pipelines should target 'What information'—READMEs, feature descriptions, and scenarios—rather than code snippets, since current retrieval methods underperform simply giving the target file.","Benchmark results from Python-only datasets overstate model ability for Java and Go, because every model performs best in Python.","Context engineering should be language-specific: in-file and RAG context for Java and Go, but finer-grained target-function information for Python."],"supporting_citations":[{"why":"It supplies the RepoCoder baseline and the repository-age selection criterion used in construction.","marker":"[29]"},{"why":"It provides the comparison dataset for semantic-distribution and reconstruction-error analysis.","marker":"[17]"},{"why":"It provides the Java comparison subset and represents pragmatic code generation benchmarks.","marker":"[26]"},{"why":"It is the contrasting benchmark that delimits MRG-Bench as code generation rather than bug fixing.","marker":"[13]"},{"why":"It is the reference dataset for model selection and for the data-leakage comparison.","marker":"[2]"},{"why":"It provides the main open-source model, DeepSeek-Coder-33B, used in the context and RAG experiments.","marker":"[11]"},{"why":"It supplies the embeddings used for semantic-distribution and reconstruction-error analysis.","marker":"[16]"},{"why":"It supplies the CDD method used to estimate and argue for low data leakage in MRG-Bench.","marker":"[6]"}],"fun_headline_variants":["Jailbreak detection with 96.6% better F1 from just 0.5% labels","CoCoTen spots adversarial prompts with 0.5% labels, 2.3-128x speedup","Detect LLM jailbreaks with sparse labels: F1 0.83 and 2.3-128x speedups","CoCoTen detects adversarial prompts using 0.5% labels, 96.6% F1 lift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 383-sample benchmark is assumed to remain representative after heavy filtering—functions needed developer-written comments, matched test cases, and 100% line coverage, and only 22 of 152 repositories survived—so if that selection shifts task difficulty or code distribution, the measured Pass@1 numbers and failure-cause proportions will not generalize to typical development.","fun_headline_variants_meta":{"raw":{"variants":["Jailbreak detection with 96.6% better F1 from just 0.5% labels","CoCoTen spots adversarial prompts with 0.5% labels, 2.3-128x speedup","Detect LLM jailbreaks with sparse labels: F1 0.83 and 2.3-128x speedups","CoCoTen detects adversarial prompts using 0.5% labels, 96.6% F1 lift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3632,"prompt_tokens":920,"completion_tokens":2712,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2597}},"tokens_in":536,"tokens_out":2712,"duration_ms":21867,"temperature":1.0,"reasoning_tokens":2597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:44:08.523471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same failure annotation on a new sample of repository functions drawn without the 100% line-coverage filter; if the share of failures attributed to missing 'What information' falls well below 68% and context rescues Python failures as much as Java and Go, the paper's central claim would be a filtering artifact rather than a general property of current models.","supporting_citations":[],"review_version":1}