Pith. sign in

REVIEW 4 major objections 6 minor 31 references

CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Repository-level code generation fails mainly because models cannot understand what the user wants, not because they lack implementation knowledge.

desk verdict Despite the CoCoTen title/abstract and mismatched author list, the actual manuscript is MRG-Bench, a solid and reusable multi-language repo-level code-gen benchmark that deserves peer review after the metadata is fixed. read the letter →

arxiv 2508.02997 v3 pith:6NRRKH6K submitted 2025-08-05 cs.CL

Sri Durga Sai Sowmya Kadali , Evangelos E. Papalexakis This is my paper
classification cs.CL
keywords repository-levelcodegenerationLLMevaluationbenchmarkmultilingualPass@1failurecauseanalysisretrieval-augmentedcontextualinformationfunctioncallgraph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MRG-Bench, a multilingual, repository-level code generation benchmark built from 383 real-world functions across Python, Java, and Go, with runnable test cases and 100% line coverage. Evaluated on this benchmark, current LLMs perform poorly: the best model, Claude-3.5-Sonnet, reaches only 32.5% average Pass@1, and RAG methods do worse than simply providing the file containing the target function. By annotating failure cases with five-model voting, the paper finds that over 68% of failures come from missing 'What information'—the model cannot map the user requirement to the functionality it calls for—rather than missing implementation knowledge. The paper also shows that context needs differ by language: in-file content helps Java and Go but barely helps Python. The central conclusion is that current models are weakest at requirement understanding, so improving that stage is the most promising route to better repository-level code generation.

What carries the argument

The central object is MRG-Bench, a 383-sample benchmark of function-level code generation tasks drawn from 22 real-world repositories, each sample containing a developer-written comment, function signature, referenced function bodies, test cases, called private functions, and repository information. Its construction pipeline uses per-language function call graph analyzers, links test functions to target functions, and filters to functions with 100% line coverage. The argument is carried by a two-axis failure taxonomy, 'What to do' versus 'How to do it', applied through a five-model voting annotation prompt that retains only 5:0 and 4:1 agreement cases, covering 86.3% of failures.

What would settle it

Run the same failure annotation on a new sample of repository functions drawn without the 100% line-coverage filter; if the share of failures attributed to missing 'What information' falls well below 68% and context rescues Python failures as much as Java and Go, the paper's central claim would be a filtering artifact rather than a general property of current models.

Watch

Extended reading notes

Core claim

The central claim is that state-of-the-art LLMs fail at repository-level code generation primarily because they cannot comprehend what functionality the user requirement corresponds to, not because they lack implementation details. The paper supports this by decomposing generation into a 'What to do' stage and a 'How to do it' stage, then using a five-model voting prompt to classify Claude-3.5-Sonnet's failed cases; over 68% of failures are attributed to missing 'What information'. Providing callable functions, the 'How information', yields only modest gains, while providing in-file context gives larger but language-dependent improvements. Python is the outlier: it has weaker semantic associations between functions, which explains why context rarely rescues its failures. The paper positions MRG-Bench as a more realistic evaluation than single-language, standalone benchmarks and argues that future methods should mine 'What information' for each language separately.

Load-bearing premise

The 383-sample benchmark is assumed to remain representative after heavy filtering—functions needed developer-written comments, matched test cases, and 100% line coverage, and only 22 of 152 repositories survived—so if that selection shifts task difficulty or code distribution, the measured Pass@1 numbers and failure-cause proportions will not generalize to typical development.

Editorial extensions

If this is right

  • Improving requirement understanding should raise Pass@1 more than adding code context, because most failures occur before implementation begins.
  • RAG pipelines should target 'What information'—READMEs, feature descriptions, and scenarios—rather than code snippets, since current retrieval methods underperform simply giving the target file.
  • Benchmark results from Python-only datasets overstate model ability for Java and Go, because every model performs best in Python.
  • Context engineering should be language-specific: in-file and RAG context for Java and Go, but finer-grained target-function information for Python.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's main claim would be to build a retriever that fetches repository documentation and sibling implementations for each requirement and check whether Python failure rates drop; the paper's analysis predicts they will not.
  • The heavy filtering—developer comments, matched tests, 100% coverage, and 22 of 152 surviving repositories—may select for well-specified functions, so a broader sample could shift the reported 'What information' share.
  • The five-model voting labels inherit the annotators' own blind spots, so the taxonomy would be stronger if a subset were checked against human annotations.
  • The language difference implies a directly measurable property: the alignment between function comments, names, and call graphs is weaker in Python than in Java and Go.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces MRG-Bench, a multi-language repository-level code generation benchmark containing 383 function-level samples from 22 real-world open-source projects (Python, Java, and Go), together with a call-graph analysis framework and runnable test environments. It evaluates a range of LLMs, long-context models, reasoning models, and RAG methods, reporting that the best model (Claude-3.5-Sonnet) achieves only 32.5% Pass@1 with in-file context, that RAG methods underperform simple in-file context, and that over 68% of failures are attributed to missing 'What information' (understanding user requirements), based on an LLM-based failure annotation protocol. The benchmark, data, and code are publicly released.

Significance. If the results hold, MRG-Bench would be a valuable resource for repository-level code generation evaluation, particularly for its multi-language coverage and runnable test cases. The central finding that models plateau at 32.5-40% Pass@1 and that requirement understanding is the dominant failure cause is practically important and could guide future research on context augmentation. The paper also provides a reusable call-graph analysis tool. However, the heavy filtering in dataset construction, the small sample size, and the unvalidated LLM-based failure annotation mean that the quantitative claims currently have limited generalizability. With appropriate additional analysis, this could become a solid benchmark contribution.

major comments (4)
  1. [Sections 3.3–3.4 and 5.1] The selection pipeline retains only 22 of 152 repositories (383 of 580 candidate functions) using criteria that include developer-written comments and 100% line coverage, which defines a narrow task distribution. Consequently, the headline Pass@1 of 32.5% (Table 4) and the failure-cause proportions in Figure 5 may not generalize to typical repository-level code generation. The RQ1.1 representativeness check is insufficient to address this: the reconstruction-error metric compares docstring embeddings of the 383 queries against a 10,000-function reference set, and a small query set will mechanically have low nearest-neighbor error even if the selected functions are atypical in difficulty or complexity. Please provide evidence that Pass@1 and the 'What vs How' proportions are stable across subsets (e.g., per language, per repository, or by coverage level), and discuss the likely effect of the 100% line-coverage filter on task difficulty.
  2. [Section 5.3 and Figure 5] The central claim that over 68% of failures stem from missing 'What information' rests entirely on LLM majority-vote annotation (five models, retaining only 5:0 and 4:1 agreements and discarding 3:2 cases) with no human validation. There is no evidence that the LLM vote corresponds to ground-truth failure causes, and the 13.7% of discarded cases may be systematically different from the retained ones. To make RQ3's conclusion load-bearing, please provide a human-annotated subset (e.g., 50–100 cases), report inter-annotator agreement between the LLM voting and human labels, and show the sensitivity of the reported proportions to the agreement threshold.
  3. [Section 5.1, Eq. (1)] The reconstruction-error definition (average distance from each query to its nearest neighbor in the reference set) is sensitive to the sizes of Q and P. Because MRG-Bench has only 383 queries while EvoCodeBench and CoderEval-Java likely have many more, the lower reconstruction error for MRG-Bench could reflect sample size rather than superior representativeness. The paper should report the sizes of the compared query sets, use a sample-size-controlled baseline (e.g., repeatedly subsample the larger datasets to 383 and recompute), or adopt a distributional distance such as MMD so that the RQ1.1 comparison is fair and meaningful.
  4. [Section 4.2 and Tables 3–7] All Pass@1 and Pass@3 results are reported as point estimates without confidence intervals or significance tests, despite small per-language sample sizes (e.g., 96 Java and 124 Python functions). Several comparative conclusions, such as 'RAG-related methods are inferior to in-file context' (Table 7 vs. Table 4) and 'providing callee functions yields less improvement than in-file context' (Table 5), rely on differences of a few percentage points that may be within sampling noise. Please report bootstrap confidence intervals or paired permutation tests for the key model and context comparisons, and clarify how Pass@k is computed (e.g., whether it is the unbiased estimator used in prior work or a simple sample mean over fixed samples).
minor comments (6)
  1. [Section 5.2 and Table 6] The text refers to the open-source long-context model as DeepSeek-V3, while Table 6 lists 'DeepSeek-V2.5'; please correct this inconsistency.
  2. [Sections 1, 3.2, and Table 1] The dataset name is inconsistently spelled as 'EvolCodeBench' and 'EvoCodeBench'; standardize the spelling throughout.
  3. [Section 3.4] The names of the coverage measurement tools are displayed as placeholder symbols; please list the actual tools (e.g., coverage.py, JaCoCo, go test -cover) so the procedure is reproducible.
  4. [Figure 4] The prompt used for failure annotation is not visible in the text; please include the full prompt text in the caption or an appendix so that the annotation instruction is reproducible and auditable.
  5. [References] The reference list appears twice at the end of the manuscript; remove the duplicate set.
  6. [Abstract and Section 1] The headline 32.5% Pass@1 for Claude-3.5-Sonnet is measured under the in-file-context setting (Table 4), but neither the abstract nor the first section states this context, which could mislead readers about the model's baseline performance; please specify the setting explicitly.

Circularity Check

0 steps flagged · score 2.0 of 10

No substantive circularity: the Pass@1 numbers and failure-cause proportions are empirical measurements, and the lone self-citation is in related work and not load-bearing.

full rationale

The paper's central claims are measurements rather than derivations. MRG-Bench is constructed from real GitHub repositories with executable tests; the headline Pass@1 of 32.5% for Claude-3.5-Sonnet is obtained by running generated code against project test cases, so it cannot reduce to an input parameter. The representativeness claim in RQ1.1 is checked by comparing docstring embedding reconstruction errors of MRG-Bench, EvoCodeBench, and CoderEval-Java against a 10,000-function random sample; MRG-Bench is not defined in terms of that reference distribution, so the comparison is an empirical evaluation of the benchmark's stated design goal. The RQ3 failure annotation uses five LLMs to vote on a two-category 'What/How' prompt; this is a proxy measurement and the 68% figure is the raw voting result, not a fitted constant or an equation that equals an input. The 22-project, 383-sample filtering pipeline (Sections 3.2-3.4) is an external-validity threat that could limit generalizability, but a selection-bias concern is not a circularity concern. The only overlap with the authors is the citation of CoderUJB [28], which shares coauthor Shikun Zhang, but that citation appears only in Related Work and is not used to justify MRG-Bench's construction or conclusions; it is therefore not load-bearing. No equation in the paper reduces a prediction to its own input, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard assumptions about embedding quality, static-analysis reliability, and leakage detection, plus one paper-specific assumption that LLM majority voting labels failure causes accurately. Experimental settings (top-k, truncation, temperature) are choices that influence the quantitative findings.

free parameters (5)
  • Line-coverage filter threshold = 100%
    Functions not achieving 100% line coverage were removed during dataset construction (Section 3.4), shaping the final 383-sample benchmark.
  • RAG retrieval top-k = 5
    RAG experiments retrieve the top 5 code blocks (Section 5.2.4); this choice affects RAG performance comparisons.
  • Context truncation limits = 6000 tokens (static), 100K tokens (long context)
    File context truncated to 6000 tokens and long-context runs to 100K tokens (Sections 5.2.1, 5.2.3).
  • Failure-annotation vote retention rule = 5:0 and 4:1
    Only samples with unanimous or near-unanimous LLM votes (5:0, 4:1) were retained, discarding 3:2 cases (Section 5.3).
  • Sampling temperature = 0.6
    Generation temperature set to 0.6 for all model evaluations (Section 4.2), affecting Pass@k sampling.
assumptions (5)
  • domain assumption NV-Embedding-2 embeddings capture semantic similarity of code docstrings
    Reconstruction error and t-SNE distribution comparisons (Section 5.1) assume embedding distance correlates with semantic closeness.
  • domain assumption Tree-sitter-based function call graph analysis correctly identifies dependencies in Python, Java, and Go
    Dataset construction matches test functions to functions via call graphs (Section 3.3); errors here would mislabel samples.
  • ad hoc to paper LLM majority-vote annotation reliably identifies failure causes without human validation
    Section 5.3 uses five LLMs with temperature 0 and retains 5:0/4:1 votes; no human annotation is reported, so reliability depends on LLM agreement.
  • domain assumption CDD data-leakage detection with default parameters provides trustworthy contamination estimates
    Section 6 interprets low CDD ratios as low leakage without sensitivity analysis.
  • domain assumption Repositories created after January 2023 have limited presence in LLM training data
    Section 3.2 justifies the recency filter as a leakage mitigation, an assumption about training corpora.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors." pith.science (2026). https://pith.science/paper/6NRRKH6K

@misc{pith2026250802997,
  author       = {Pith},
  title        = {Pith review of: CoCoTen: Detecting Adversarial Inputs to Large Language Models through Latent Space Features of Contextual Co-occurrence Tensors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6NRRKH6K}},
  note         = {Machine review of arXiv:2508.02997}
}
read the original abstract

The widespread use of Large Language Models (LLMs) in many applications marks a significant advance in research and practice. However, their complexity and hard-to-understand nature make them vulnerable to attacks, especially jailbreaks designed to produce harmful responses. To counter these threats, developing strong detection methods is essential for the safe and reliable use of LLMs. This paper studies this detection problem using the Contextual Co-occurrence Matrix, a structure recognized for its efficacy in data-scarce environments. We propose a novel method leveraging the latent space characteristics of Contextual Co-occurrence Matrices and Tensors for the effective identification of adversarial and jailbreak prompts. Our evaluations show that this approach achieves a notable F1 score of 0.83 using only 0.5% of labeled prompts, which is a 96.6% improvement over baselines. This result highlights the strength of our learned patterns, especially when labeled data is scarce. Our method is also significantly faster, speedup ranging from 2.3 to 128.4 times compared to the baseline models.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 21 linked inside Pith

  1. [1]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

  2. [2]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug.arXiv preprint arXiv:2304.05128(2023)

    arXiv 2023

  3. [3]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501. 12948

    arXiv 2025

  4. [4]

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. 2024. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.Advances in Neural Information Processing Systems36 (2024)

    2024

  5. [5]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin

  6. [6]

    Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Memorization: Data Contamination and Trustwor- thy Evaluation for Large Language Models. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics...

    doi:10.18653/v1/2024.findings-acl.716 2024

  7. [7]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually- crafted benchmark for evaluating llms on class-level code generation.arXiv preprint arXiv:2308.01861(2023)

    arXiv 2023

  8. [8]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999 (April 2023). doi:10.48550/arXiv.2204.05999 arXiv:2204.05999 [cs]

    arXiv doi:10.48550/arxiv.2204.05999 2023

Show all 31 references
  1. [9]

    2024.Copilot

    GitHub. 2024.Copilot. https://github.com/features/copilot

    2024

  2. [10]

    Wenhan Xiong Grattafiori, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Syn- naeve. 2023. Code Llama: Open Foundation Models for Code.arXiv preprint arXiv:2308.12950(2023)

    2023 arXiv

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  4. [12]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Mea- suring coding challenge competence with apps.arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  5. [13]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    2023 arXiv

  6. [14]

    Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. 2023. xcodeeval: A large scale multilin- gual multitask benchmark for code understanding, generation, translation and retrieval.arXiv preprint arXiv:2303.03004(2023)

    2023 arXiv

  7. [15]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Mem- ory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating...

    2023

  8. [16]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models.arXiv preprint arXiv:2405.17428 (2024)

    2024 arXiv

  9. [17]

    Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories. arXiv:2404.00599 (March 2024). http://arxiv.org/abs/2404.00599 arXiv:2404.00599 [cs]

    2024 arXiv

  10. [18]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems.arXiv preprint arXiv:2306.03091 (2023)

    2023 arXiv

  11. [19]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation.arXiv preprint arXiv:2402.19173(2024)

    2024 arXiv

  12. [20]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al . 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    2021 arXiv

  13. [21]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis.arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  14. [22]

    2024.ChatGPT

    OpenAI. 2024.ChatGPT. https://chatgpt.com/

    2024

  15. [23]

    Luca Pasquini, Stefano Cristiani, Ramón García López, Martin Haehnelt, Michel Mayor, Jochen Liske, Antonio Manescau, Gerardo Avila, Hans Dekker, Olaf Iwert, et al. 2010. Codex. InGround-based and Airborne Instrumentation for Astronomy III, Vol. 7735. SPIE, 957–968

    2010

  16. [24]

    Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, and Ge Li. 2022. Incorporating domain knowledge through task augmentation for front-end javascript code generation. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Fou...

    2022

  17. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  18. [26]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of prag- matic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on So...

    2024

  19. [27]

    Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. CERT: Continual Pre-training on Sketches for Library-oriented Code Generation. InProceedings of the Thirty-First International Joint Conference on Artificial...

    2022 doi

  20. [28]

    Zhengran Zeng, Yidong Wang, Rui Xie, Wei Ye, and Shikun Zhang. 2024. CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios.arXiv preprint arXiv:2403.19287(2024)

    2024 arXiv

  21. [29]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code com- pletion through iterative retrieval and generation.arXiv preprint arXiv:2303.12570 (2023)

    2023 arXiv

  22. [30]

    Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. arXiv:2401.07339 (Jan. 2024). http://arxiv.org/abs/2401.07339 arXiv:2401.07339 [cs]. Received 20 Feb...

    2024 arXiv

  23. [2023]

    arXiv:2301.09043 (Jan

    CodeScore: Evaluating Code Generation by Learning Code Execution. arXiv:2301.09043 (Jan. 2023). doi:10.48550/arXiv.2301.09043 arXiv:2301.09043 [cs]. ���������� �������� ����� ����� ������� ��� ���� ���� ��� ������ �����

    2023 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.