Pith. sign in

REVIEW 5 major objections 5 minor 16 cited by

FullStack Bench: Evaluating LLMs as Full Stack Coders

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new benchmark tests code LLMs across 11 real-world domains and 16 languages, and its 27-model evaluation shows high HumanEval scores do not carry over to full-stack tasks.

desk verdict Useful benchmark resource with released artifacts, but the headline Pass@1 numbers rest on unverified test quality and a circular difficulty split. read the letter →

arxiv 2412.00535 v6 pith:4YORXJVV submitted 2024-11-30 cs.AI cs.SE

Bytedance-Seed-Foundation-Code-Team: Yao Cheng , Jianfeng Chen , Jie Chen , Li Chen , Liyu Chen , Wentao Chen , Zhengyu Chen , Shijie Geng
show 46 more authors
Aoyan Li Bo Li Bowen Li Linyi Li Boyi Liu Jiaheng Liu Kaibo Liu Qi Liu Shukai Liu Siyao Liu Tianyi Liu Tingkai Liu Yongfei Liu Rui Long Jing Mai Guanghan Ning Z.Y. Peng Kai Shen Jiahao Su Jing Su Tao Sun Yifan Sun Yunzhe Tao Guoyin Wang Siwei Wang Xuwu Wang Yite Wang Zihan Wang Jinxiang Xia Liang Xiang Xia Xiao Yongsheng Xiao Chenguang Xi Shulin Xin Jingjing Xu Shikun Xu Hongxia Yang Jack Yang Yingxiang Yang Jianbo Yuan Jun Zhang Yufeng Zhang Yuyu Zhang Shen Zheng He Zhu Ming Zhu
This is my paper · ORCID
classification cs.AIcs.SE
keywords codegenerationbenchmarkfull-stackprogrammingmultilingualevaluationsandboxexecutionPass@1LLMunittestingsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing code benchmarks measure too narrow a slice of what programmers actually do, and it backs that claim with a new dataset, FullStack Bench, designed to reflect real-world full-stack development. The dataset contains 3,374 human-annotated problems across 11 application domains and 16 programming languages, with 15,168 unit tests for automated scoring. To run such a broad evaluation, the paper also presents SandboxFusion, a sandbox execution tool supporting 23 languages and many packages. Evaluating 27 code and general LLMs, the paper finds that models with high HumanEval scores often score much lower on FullStack Bench, suggesting that current benchmarks overstate real-world coding ability. If correct, FullStack Bench and SandboxFusion give the field a more representative way to measure code LLMs and a practical tool for running such evaluations.

What carries the argument

The load-bearing object is the FullStack Bench dataset itself, with each problem consisting of a question, unit test cases, a reference solution, and domain labels, plus its companion execution engine SandboxFusion. SandboxFusion carries the argument by providing a unified pipeline—prompt generation, code extraction, test synthesis, isolated execution, judgment, and metric calculation—that makes automated evaluation feasible across languages and domains, including resource isolation via namespaces and cgroups. The paper's domain distribution, derived from a 500k-question StackOverflow analysis, is what grounds the claim that the benchmark mirrors real-world programming scenes.

What would settle it

One concrete check is to independently re-annotate a random sample of FullStack Bench problems with fresh annotators and compare their unit tests and reference solutions to the released ones; if a substantial fraction (say, more than 10%) of problems yield different expected outputs or additional failing tests for the reference solution, the Pass@1 numbers lose their meaning. Another check is to compute the score of a trivial baseline, such as a model that always returns the empty program or a constant, to see whether the test harness's extraction and judgment pipeline correctly fails such outputs.

Watch

Extended reading notes

Core claim

The central claim is that FullStack Bench offers a more representative measure of code LLM capability by spanning 11 application domains—basic programming, advanced programming, software engineering, data analysis, mathematics, desktop and web development, machine learning, scientific computing, database, multimedia, and operating systems—rather than the narrow domains of existing benchmarks. The paper further claims that its 15,168 unit tests and bilingual (Chinese/English) construction provide a reliable Pass@1 evaluation across 16 languages, and that SandboxFusion makes this evaluation practical by supporting 23 languages, including front-end browsers and deep-learning packages. The empirical finding that models cluster in the upper triangle when comparing HumanEval to FullStack Bench scores is presented as evidence that current benchmarks overestimate real-world coding performance.

Load-bearing premise

The 15,168 unit tests and their expected outputs are correct and complete enough to distinguish correct from incorrect programs, and the paper reports no inter-annotator agreement, test-case validation, or audit statistics to verify this.

Editorial extensions

If this is right

  • Code LLM rankings will shift: models that excel on HumanEval will not necessarily lead on FullStack Bench, so leaderboards that rely on narrow benchmarks give a misleading picture of practical coding skill.
  • Domain-specific weaknesses become visible, such as low performance in mathematics and scientific computing for code-focused models, informing where training data and reasoning ability need improvement.
  • SandboxFusion enables execution-based evaluation and training loops, including self-refinement with compiler feedback, which the paper shows improves accuracy over best-of-N sampling.
  • The bilingual construction (1,687 problems in each of Chinese and English) allows measurement of how prompt language affects code generation performance.
  • Scaling laws still hold on FullStack Bench, but with diminishing returns as model size grows, guiding expectations for further scaling.
  • The 15,168 unit tests support fine-grained Pass@1 measurement that can be reused for reinforcement learning or iterative refinement beyond simple benchmarking.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is to measure contamination: since FullStack Bench draws from GitHub and XLCoST and uses LLM-assisted generation, its difficulty estimates could be inflated or deflated if parts of it appear in training corpora; a contamination audit would make the benchmark's reported scores more interpretable.
  • The paper's claim of real-world representativeness rests on a StackOverflow-derived domain distribution; one could test this by comparing FullStack Bench scores against performance on fresh, naturally occurring developer tasks, which would clarify whether the benchmark predicts real-world usefulness or only its own distribution.
  • The finding that some small models outperform larger ones on certain domains suggests FullStack Bench could be used as a diagnostic to isolate which skills emerge with scale, a question the paper raises but does not fully resolve.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces FullStack Bench, a code-generation benchmark of 3,374 problems spanning 11 application domains and 16 programming languages, with 15,168 human-annotated unit tests, reference solutions, and bilingual (Chinese/English) prompts. It also releases SandboxFusion, a sandbox execution framework that supports 23 languages and multiple existing code benchmarks, and reports Pass@1 results for 27 open- and closed-source models across domains, languages, difficulty levels, and a scaling-law analysis. The authors argue that FullStack Bench measures real-world full-stack coding ability more broadly than existing benchmarks such as HumanEval, MBPP, DS-1000, and xCodeEval.

Significance. If the benchmark is reliable, this is a useful community resource: it is larger in domain and language coverage than most existing code benchmarks, is execution-based rather than model-judged, and includes released data, reference solutions, and a sandbox. The evaluation of 27 models with per-domain and per-language breakdowns is a substantial empirical effort, and the finding that many models score high on HumanEval but lower on FullStack Bench is a potentially valuable observation about benchmark saturation. The main strengths are the released assets, the breadth of the task design, and the execution-based correctness check, which is not circular. However, the paper currently lacks several pieces of evidence that are needed to trust the benchmark's measurements: test-suite validation, inter-annotator agreement, contamination analysis, validation of the LLM-generated domain labels, and confidence intervals for the reported Pass@1 values. These gaps are fixable and do not invalidate the core idea, but they are load-bearing for the central claim that FullStack Bench provides a broader and more representative evaluation.

major comments (5)
  1. [Section 2.2 and Section 4.5] The difficulty labels are computed from the pass/fail votes of six named models (DeepSeek-Coder-6.7B, DeepSeek-Coder-33B, Qwen2.5-Coder-7B, LLaMA3.1-70B, Claude-3.5-Sonnet, GPT-4o), and the same model families are then evaluated in Table 2 (e.g., DeepSeek-Coder-6.7B-Instruct, DeepSeek-Coder-33B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama3.1-70B-Instruct, Claude-35-Sonnet, GPT-4o). Because "easy" and "hard" are defined by whether these voters solved the problem, the difficulty-stratified curves in Figure 9 partly measure models against labels that were constructed from the same models; the analysis is therefore not an independent validation of difficulty. I recommend re-labeling difficulties with held-out annotators or models, or explicitly reframing the figure as a calibration check of the labeling procedure rather than as an evaluation finding.
  2. [Section 2.2 and Figure 11] The Pass@1 metric of Section 2.4 treats a model output as correct only if it passes all unit tests, so the reliability of every number in Tables 2 and 3 depends on the correctness and completeness of the 15,168 assertions. The paper reports a human annotation and cross-refinement process but gives no inter-annotator agreement, no audit or re-check statistics, no mutation testing, and no held-out validation of the test suites. Several examples in Figure 11 are concerning: the TypeScript maxInArray problem checks only one array, and the palindrome problem checks only three small cases, so plausible incorrect implementations would pass. The Section 2.1 statement that the authors "strive to cover all error types" needs measurable support; please add test-suite validation evidence such as per-problem assertion counts, mutation scores, or a sample audit, and report inter-annotator agreement on a subset of problems.
  3. [Section 1 and Section 2.2] The 11-domain taxonomy and the claimed match to StackOverflow (94.3% vs. 88.1%) are derived from LLM-generated tags on both the StackOverflow sample and the FullStack Bench items, with no human validation or agreement statistics reported. If the tagger is biased, the domain coverage and the "simulates StackOverflow" claim can be consistent by construction rather than by genuine representativeness. Please provide a human-labeled validation sample with precision/recall or agreement numbers for the domain labels, and state which LLM version and prompt were used.
  4. [Section 4.1 and Tables 2-3] All results are reported as single Pass@1 values with no confidence intervals or significance tests, even though many per-domain and per-language cells are small (for example, the D column in Table 3 appears to correspond to 46 problems and the MM column in Table 2 to 60 problems). Consequently, statements such as "the performance gaps are relatively larger in C#, D, PHP, Ruby, Rust, and Scala" (Section 4.3) or the ranking of closely spaced closed-source models in Table 2 may not be robust. Please add confidence intervals or exact binomial intervals and, where rankings are claimed, a significance or effect-size analysis.
  5. [Section 4.1 and Figure 2] No contamination analysis is reported, despite the dataset being built partly from GitHub code and XLCoST and the evaluated models being trained on large web and GitHub corpora. Without measuring n-gram overlap or memorization of the 3,374 problems, the claim that FullStack Bench is a more realistic measure of coding ability, and the HumanEval-versus-FullStack gap in Figure 2, could be partly explained by data leakage. Please add a contamination check such as overlap with training corpora or time-based splits, and discuss the impact on the reported leaderboard.
minor comments (5)
  1. [Section 4.6 and Figure 10] The Reflection-versus-BoN experiment is under-specified: the paper does not state which model was used, how many problems were evaluated, how feedback was generated, or how many independent trials were run. Please provide these details so the claim that SandboxFusion feedback improves accuracy is reproducible.
  2. [Section 3 and Table 7] The paper describes SandboxFusion as efficient and easy to deploy, but no quantitative measurements of throughput, latency, resource usage, or end-to-end evaluation cost are reported. Adding such numbers would substantiate the "effective" claim in the abstract.
  3. [Section 4.3] The observation that "some native English speakers, such as StarCoder2-15B-Instruct-v0.1, OpenAI-o1-preview, and OpenAI-o1-mini, perform better on Chinese questions" is surprising and is not accompanied by any analysis of prompt translation, sample sizes, or statistical significance; please clarify or temper this claim.
  4. [Section 2.2] The domain list labels "Data Analysis" as "DP" in the text but "DA" in Table 2; please make the abbreviations consistent.
  5. [Throughout] There are several typos and formatting issues, including "To access the code-related abilities" (should be "assess"), "Humaneval" for "HumanEval", and inconsistent hyphenation of "Full-Stack"/"FullStack". A careful proofreading pass is recommended.

Circularity Check

1 steps flagged · score 2.0 of 10

Difficulty labels are set by the same six models whose difficulty-stratified pass rates are reported in Section 4.5; the main unit-test benchmark is otherwise self-contained.

  1. self definitional [Section 2.2 (Data Construction and Quality Control); analyzed in Section 4.5 (difficulty performance)]
    "Specifically, for each question, if only one model obtains the correct answer, this question is classified as a hard sample, and if five or six models obtain the correct answer, this question is classified as an easy sample."

    This classification is the output of a vote by six named models (DeepSeek-Coder-6.7B, DeepSeek-Coder-33B, Qwen2.5-Coder-7B, LLaMA3.1-70B, Claude-3.5-Sonnet, GPT-4o), and Table 2 reports Pass@1 for those same six models (as DeepSeek-Coder-6.7B-Instruct, DeepSeek-Coder-33B-Instruct, Qwen2.5-Coder-7B-Instruct, Llama3.1-70B-Instruct, Claude-35-Sonnet, GPT 4o-0806). Section 4.5 then plots model performance on these self-defined difficulty buckets. For the voting models, a hard problem is by construction one that almost none of them solved, and an easy problem is one that almost all of them solved, so the difficulty-stratified pass rates in Section 4.5 are partly a restatement of the labeling rule rather than an independent finding about model ability.

full rationale

The central derivation chain is self-contained: FullStack Bench problems are collected from GitHub, documents, XLCoST, and expert-written questions; instructions, unit tests, and reference solutions are produced and human-verified; and model quality is scored by Pass@1 against those external unit tests. That correctness judgment is not fitted to any model output, and the domain and language comparisons in Tables 2 and 3 rest on those independent tests, so the benchmark's main ranking claim does not reduce to its inputs. The one genuine self-reference is the difficulty labeling: Section 2.2 defines easy/medium/hard by a vote of six models, and Section 4.5 reports difficulty-stratified pass rates for the same models, making that particular analysis partly tautological for the voting models. No load-bearing self-citation chain is present; LIME is cited only for the voting methodology. Concerns about unverified unit-test quality or missing inter-annotator agreement are correctness risks, not circularity, and therefore do not raise the circularity score beyond this mild localized self-reference.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim of benchmark representativeness rests on design choices (domain taxonomy, difficulty thresholds) and process assumptions (annotation quality, LLM labeling accuracy) that are not independently validated in the paper. No new physical or conceptual entities are introduced; SandboxFusion is a released software artifact.

free parameters (3)
  • Domain taxonomy thresholds = 11 domains covering 88.1% of StackOverflow
    The taxonomy and the cutoff at 88.1% are chosen by the authors based on LLM labels; they determine the benchmark's domain composition and the 'simulates StackOverflow' claim.
  • Difficulty voting thresholds = easy: 5-6 of 6 models correct; hard: 1 model correct; else medium
    Chosen by hand in Section 2.2; defines the difficulty split used in Section 4.5.
  • Number of voting models = 6
    Six models are selected for difficulty filtering (DeepSeek-Coder-6.7B/33B, Qwen2.5-Coder-7B, LLaMA3.1-70B, Claude-3.5-Sonnet, GPT-4o). The count and membership are design choices affecting which problems are labeled easy/medium/hard.
assumptions (4)
  • domain assumption LLM-generated domain labels for StackOverflow questions are sufficiently accurate to derive the application-domain taxonomy.
    Used in Section 2.2 and Figure 1 to justify the 11 domain categories; no human validation or accuracy metric is reported.
  • domain assumption The 500k sampled StackOverflow questions represent real-world code development scenarios.
    The paper claims FullStack Bench simulates StackOverflow well based on this sample; the sample's representativeness is assumed.
  • domain assumption Human annotation with cross-refinement ensures problem quality and unit test correctness.
    Section 2.2 describes the process but reports no inter-annotator agreement or post-release audit; the central claim of benchmark reliability rests on this.
  • ad hoc to paper Difficulty defined by six LLMs' correctness corresponds to a meaningful notion of difficulty.
    Section 2.2 defines easy/medium/hard by votes; Section 4.5 uses this ordering to draw conclusions about model scaling, making the difficulty axis dependent on the same models measured.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FullStack Bench: Evaluating LLMs as Full Stack Coders." pith.science (2026). https://pith.science/paper/4YORXJVV

@misc{pith2026241200535,
  author       = {Pith},
  title        = {Pith review of: FullStack Bench: Evaluating LLMs as Full Stack Coders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4YORXJVV}},
  note         = {Machine review of arXiv:2412.00535}
}
read the original abstract

As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.

Figures

Figures reproduced from arXiv: 2412.00535 by the authors.

Figure 1
Figure 1. Application domain distributions of different code evaluation datasets. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Performance plot of tested LLMs on HumanEval and FullStack Bench. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of data collection process of FullStack Bench. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: SandboxFusion Architecture. computer hardware and software resources. • Others: Apart from the above 11 mainstream application domains, other domains are categorized as “Others”. 2.3. Bilingual Benchmark Construction The collected questions are in Chinese or English. F…
Figure 5
Figure 5. Figure 5: Visualization on domain performance. sourced models has been challenged, with some closed-sourced models being matched or even surpassed by pioneers in open-sourced ones. DeepSeekCoder-v2-Instruct, a 236B-MoE model, is the best behavior of open-sourced models, which pu…
Figure 8
Figure 8. Figure 8: Performance of different sizes. We categorized the model into 5 series based on the criteria in [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 6
Figure 6. Figure 6: General performance on different programming languages. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Visualization of performance on different languages. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Visualization of performance on different difficulties. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Comparison between BoN and Reflection. Coder [Hui et al., 2024] gains remarkable progress in multilingual programming code generation and debugging tasks, such as MultiPL-E [Cassano et al., 2022], McEval [Chai et al., 2024], and MdEval [Liu et al., 2024b]. Code Benchm…
Figure 11
Figure 11. Figure 11: Visualization on some cases of our FullStack Bench. [PITH_FULL_IMAGE:figures/full_fig_p027_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CodeContests+: High-Quality Test Case Generation for Competitive Programming

    cs.SE 2025-06 conditional novelty 7.0 of 10

    A new dataset, CodeContests+, uses an LLM agent system to generate and validate competitive programming test cases, improving evaluation accuracy and reinforcement learning training.

  2. Parameter Exploration for RLVR via Variational Learning

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Generating rollouts from multiple weight-perturbed versions of the policy, pooled inside a GRPO group, improves RLVR performance over standard GRPO on math and code benchmarks.

  3. Towards Reliable C-to-Rust Translation with Rule-Guided Reasoning and Reinforcement Learning

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A rule-guided MCTS plus dual-reward reinforcement learning pipeline improves LLM-based C-to-Rust translation accuracy and cuts unsafe Rust output across three benchmarks.

  4. Vibration-Based Energy Metric for Restoring Needle Alignment in Autonomous Robotic Ultrasound

    cs.RO 2025-08 unverdicted novelty 6.0 of 10

    A vibration-based energy metric, obtained by periodically vibrating the needle, is used as an out-of-plane feedback signal to re-align the ultrasound probe with the needle plane, reporting 0.41 mm translation and 0.51...

  5. IFEvalCode: Controlled Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.

  6. Turning the Tide: Repository-based Code Reflection

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.

  7. AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length

    cs.SE 2025-06 conditional novelty 6.0 of 10

    AdaptiveLLM routes coding problems to cost-effective LLMs by clustering chain-of-thought lengths into three difficulty levels, improving pass@1 by 7.86% over ComplexityNet while cutting inference cost by 88.9%.

  8. Seed-Coder: Let the Code Model Curate Data for Itself

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.

  9. SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

    cs.CL 2026-07 conditional novelty 5.0 of 10

    An Ascend-NPU training stack reaches 34.22% MFU on DeepSeek-V4-Pro, and a solver-verified CPT+SFT recipe raises OR benchmark averages to 71.81% (Flash) and 77.33% (Pro).

  10. UniCode: Augmenting Evaluation for Code Reasoning

    cs.SE 2025-10 conditional novelty 5.0 of 10

    UniCode's LLM-generated coding benchmark drops top-model pass@1 to 70.3% and indicates current LLMs rely on memorized seed logic instead of generalizing to new algorithmic problems.

  11. Dream-Coder 7B: An Open Diffusion Language Model for Code

    cs.CL 2025-09 conditional novelty 5.0 of 10

    An open 7B diffusion language model for code, adapted from Qwen2.5-Coder, reaches 21.4% pass@1 on LiveCodeBench and shows adaptive, non-left-to-right generation patterns.

  12. RLPR: Extrapolating RLVR to General Domains without Verifiers

    cs.LG 2025-06 conditional novelty 5.0 of 10

    RLPR uses an LLM's token probabilities for reference answers, debiased and filtered by reward variance, as a verifier-free reinforcement learning reward and reports gains across seven reasoning benchmarks.

  13. Multi-Agent Collaboration for Multilingual Code Instruction Tuning

    cs.CL 2025-02 reject novelty 5.0 of 10

    A multi-agent collaboration framework for generating multilingual code instruction data is introduced, but the reported evaluations show the fine-tuned model underperforming its own base instruct model.

  14. BitsAI-CR: Automated Code Review via LLM in Practice

    cs.SE 2025-01 conditional novelty 5.0 of 10

    An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.

  15. ExecRepoBench: Multi-level Executable Code Completion Evaluation

    cs.CL 2024-12 reject novelty 5.0 of 10

    ExecRepoBench is an executable, repository-level code completion benchmark with unit tests, and Qwen2.5-Coder-Instruct-C is a 7B model fine-tuned on the accompanying Repo-Instruct corpus.

  16. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

Reference graph

Works this paper leans on

76 extracted references · 23 canonical work pages · cited by 16 Pith papers

  1. [1]

    Abadi, A

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man\' e , R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanho...

    work page 2015

  2. [2]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    L. A. Agrawal, A. Kanade, N. Goyal, S. K. Lahiri, and S. K. Rajamani. Guiding language models of code with global context using monitors. arXiv preprint arXiv:2306.10763, 2023

    arXiv 2023

  4. [4]

    L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. URL https://arxiv.org/abs/2301.03988

    arXiv 2023

  5. [5]

    Athiwaratkun, S

    B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang. Multi-lingual evaluation of code generation models. In The Eleventh Intern...

    2023

  6. [7]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021 b . URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  7. [8]

    J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  8. [9]

    Bairi, A

    R. Bairi, A. Sonwane, A. Kanade, A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, S. Shet, et al. Codeplan: Repository-level coding using llms and planning. arXiv preprint arXiv:2309.12499, 2023

    arXiv 2023

Show all 76 references
  1. [10]

    Black, L

    S. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , 2021. URL https://doi.org/10.5281/zenodo.5297715

    2021 doi

  2. [11]

    Black, S

    S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. GPT - N eo X -20 B : An open-source autoregressive language model. In Proceedings ...

    2022 doi

  3. [12]

    Cassano, J

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022

    2022 arXiv

  4. [13]

    Cassano, J

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software ...

    2023

  5. [14]

    L. Chai, S. Liu, J. Yang, Y. Yin, K. Jin, J. Liu, T. Sun, G. Zhang, C. Ren, H. Guo, et al. Mceval: Massively multilingual code evaluation. arXiv preprint arXiv:2406.07436, 2024

    2024 arXiv

  6. [16]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.03374, 2021 b . URL https://arxiv.org/abs/2107.03374

    2021 arXiv

  7. [17]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24 0 (240): 0 1--113, 2023

    2023

  8. [18]

    J. Dai, J. Lu, Y. Feng, D. Huang, G. Zeng, R. Ruan, M. Cheng, H. Tan, and Z. Guo. Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation, 2024. URL https://arxiv.org/abs/2405.11430

    2024 arXiv

  9. [19]

    K. Deng, J. Liu, H. Zhu, C. Liu, J. Li, J. Wang, P. Zhao, C. Zhang, Y. Wu, X. Yin, Y. Zhang, W. Su, B. Xiang, T. Ge, and B. Zheng. R2c2-coder: Enhancing and benchmarking real-world repository-level code completion abilities of code large language models. ArXiv, abs/2406.01359, 2024

    2024 arXiv

  10. [20]

    Y. Ding, Z. Wang, W. U. Ahmad, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007, 2022. URL https://arxiv.org/abs/2212.10007

    2022 arXiv

  11. [21]

    S. Dou, J. Zhang, J. Zang, Y. Tao, W. Zhou, H. Jia, S. Liu, Y. Yang, Z. Xi, S. Wu, S. Zhang, M. Wu, C. Lv, L. Xiong, W. Zhan, L. Zhang, R. Weng, J. Wang, X. Cai, Y. Wu, M. Wen, R. Zheng, T. Ji, Y. Cao, T. Gui, X. Qiu, Q. Zhang, and X. Huang. Multi-programming language sandbox ...

    2024 arXiv

  12. [22]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/fo...

    2023

  13. [23]

    L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764--10799. PMLR, 2023

    2023

  14. [24]

    A. Gu, B. Rozi \`e re, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  15. [26]

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024 b

    2024 arXiv

  16. [27]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with APPS . In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Rou...

    2021

  17. [28]

    Huang, D

    J. Huang, D. Tang, L. Shou, M. Gong, K. Xu, D. Jiang, M. Zhou, and N. Duan. Cosqa: 20,000+ web queries for code search and question answering. arXiv preprint arXiv:2105.13239, 2021

    2021 arXiv

  18. [29]

    Huang, T

    S. Huang, T. Cheng, J. Klein Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. Liu, C. Zhang, L. Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv e-prints, pages arXiv--2411, 2024 a

    2024

  19. [30]

    Huang, T

    S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. H. Liu, C. Zhang, L. Chai, R. Yuan, Z. Zhang, J. Fu, Q. Liu, G. Zhang, Z. Wang, Y. Qi, Y. Xu, and W. Chu. Opencoder: The open cookbook for top-tier code large language models. In arXiv preprint, 2024 b

    2024

  20. [31]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  21. [32]

    F. Huq, M. Hasan, M. M. A. Haque, S. Mahbub, A. Iqbal, and T. Ahmed. Review4repair: Code review aided automatic program repairing. Information and Software Technology, 143: 0 106765, 2022

    2022

  22. [33]

    Husain, H.-H

    H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019. URL https://arxiv.org/abs/1909.09436

    1909 arXiv

  23. [34]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  24. [35]

    M. Jiao, T. Yu, X. Li, G. Qiu, X. Gu, and B. Shen. On the evaluation of neural code translation: Taxonomy and benchmark. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1529--1541. IEEE, 2023

    2023

  25. [36]

    M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, and S. Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval, 2023

    2023

  26. [37]

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626, 2023

    2023

  27. [38]

    Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-T. Yih, D. Fried, S. Wang, and T. Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022

    2022 arXiv

  28. [39]

    Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. I. Wang, and T. Yu. DS-1000: A natural and reliable benchmark for data science code generation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Internationa...

    2023

  29. [40]

    Difysandbox

    LangGenius. Difysandbox. https://github.com/langgenius/dify-sandbox, 2024

    2024

  30. [41]

    H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. ArXiv, abs/2207.01780, 2022. URL https://api.semanticscholar.org/CorpusID:250280117

    2022 arXiv

  31. [42]

    R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. URL https://arxiv.org/abs/2305.06161

    2023 arXiv

  32. [43]

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, et al. Competition-level code generation with alphacode. ArXiv preprint, abs/2203.07814, 2022. URL https://arxiv.org/abs/2203.07814

    2022 arXiv

  33. [44]

    Z. Li, J. Zhang, C. Yin, Y. Ouyang, and W. Rong. Procqa: A large-scale community-based programming question answering dataset for code search. arXiv preprint arXiv:2403.16702, 2024

    2024 arXiv

  34. [45]

    J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Process...

    2023

  35. [46]

    J. Liu, K. Deng, C. Liu, J. Yang, S. Liu, H. Zhu, P. Zhao, L. Chai, Y. Wu, K. Jin, G. Zhang, Z. M. Wang, G. Zhang, B. Xiang, W. Su, and B. Zheng. M2rc-eval: Massively multilingual repository-level code completion evaluation. In arXiv preprint, 2024 a

    2024

  36. [47]

    M. Liu, N. Pinckney, B. Khailany, and H. Ren. Verilogeval: Evaluating large language models for verilog code generation, 2023 b . URL https://arxiv.org/abs/2309.07544

    2023 arXiv

  37. [48]

    S. Liu, L. Chai, J. Yang, J. Shi, H. Zhu, L. Wang, K. Jin, W. Zhang, H. Zhu, S. Guo, et al. Mdeval: Massively multilingual code debugging. arXiv preprint arXiv:2411.02310, 2024 b

    2024 arXiv

  38. [49]

    T. Liu, C. Xu, and J. J. McAuley. Repobench: Benchmarking repository-level code auto-completion systems. CoRR, abs/2306.03091, 2023 c . doi:10.48550/ARXIV.2306.03091. URL https://doi.org/10.48550/arXiv.2306.03091

    2023 doi arXiv

  39. [50]

    Lozhkov, R

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  40. [51]

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU. Code XGLUE : A machine learning benchmark dataset for cod...

    2021

  41. [52]

    S. Lu, N. Duan, H. Han, D. Guo, S.-w. Hwang, and A. Svyatkovskiy. R e ACC : A retrieval-augmented code completion framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6227--6240, Dublin, Ireland, M...

    2022 doi

  42. [53]

    Madaan, N

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024

    2024

  43. [54]

    Nijkamp, B

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=iaYcJKpY2B_

    2023

  44. [55]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K \"o pf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high...

    1912 arXiv

  45. [56]

    H. Pei, J. Zhao, L. Lausen, S. Zha, and G. Karypis. Better context makes better code language models: A case study on function call argument completion. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Ap...

    2023 doi

  46. [57]

    J. A. Prenner and R. Robbes. Runbugrun--an executable dataset for automated program repair. arXiv preprint arXiv:2304.01102, 2023

    2023 arXiv

  47. [58]

    Richter and H

    C. Richter and H. Wehrheim. Tssb-3m: Mining single statement bugs at massive scale. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 418--422, 2022

    2022

  48. [59]

    Roziere, J

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  49. [60]

    Shrivastava, D

    D. Shrivastava, D. Kocetkov, H. de Vries, D. Bahdanau, and T. Scholak. Repofusion: Training code models to understand your repository. arXiv preprint arXiv:2306.10998, 2023 a

    2023 arXiv

  50. [61]

    Shrivastava, H

    D. Shrivastava, H. Larochelle, and D. Tarlow. Repository-level prompt generation for large language models of code. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, vol...

    2023

  51. [62]

    A. Su, A. Wang, C. Ye, C. Zhou, G. Zhang, G. Zhu, H. Wang, H. Xu, H. Chen, H. Li, et al. Tablegpt2: A large multimodal model with tabular data integration. arXiv preprint arXiv:2411.02059, 2024

    2024 arXiv

  52. [63]

    T. Sun, L. Chai, Y. Y. Jian Yang, H. Guo, J. Liu, B. Wang, L. Yang, and Z. Li. Unicoder: Scaling code large language model via universal code. ACL, 2024

    2024

  53. [64]

    L. Team. The llama 3 herd of models. arXiv preprint arXiv: 2407.21783, 2024

    2024 arXiv

  54. [65]

    R. Tian, Y. Ye, Y. Qin, X. Cong, Y. Lin, Z. Liu, and M. Sun. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621, 2024

    2024 arXiv

  55. [66]

    T. Wei, L. Zhao, L. Zhang, B. Zhu, L. Wang, H. Yang, B. Li, C. Cheng, W. Lü, R. Hu, C. Li, L. Yang, X. Luo, X. Wu, L. Liu, W. Cheng, P. Cheng, J. Zhang, X. Zhang, L. Lin, X. Wang, Y. Ma, C. Dong, Y. Sun, Y. Chen, Y. Peng, X. Liang, S. Yan, H. Fang, and Y. Zhou. Skywork: A more...

    2023 arXiv

  56. [67]

    X. Wu, J. Yang, L. Chai, G. Zhang, J. Liu, X. Du, D. Liang, D. Shu, X. Cheng, T. Sun, et al. Tablebench: A comprehensive and complex benchmark for table question answering. arXiv preprint arXiv:2408.09174, 2024

    2024 arXiv

  57. [68]

    F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1--10, 2022

    2022

  58. [69]

    W. Yan, Y. Tian, Y. Li, Q. Chen, and W. Wang. Codetransocean: A comprehensive multilingual benchmark for code translation. arXiv preprint arXiv:2310.04951, 2023

    2023 arXiv

  59. [70]

    Young, B

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  60. [71]

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1--12, 2024

    2024

  61. [72]

    Zhang, B

    F. Zhang, B. Chen, Y. Zhang, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023. URL https://arxiv.org/abs/2303.12570

    2023 arXiv

  62. [73]

    Zhang, H

    S. Zhang, H. Zhao, X. Liu, Q. Zheng, Z. Qi, X. Gu, X. Zhang, Y. Dong, and J. Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts, 2024. URL https://arxiv.org/abs/2405.04520

    2024 arXiv

  63. [74]

    C. T. H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. tij Bansal, L. Vilnis, M. Wirth, P. Michel, P. Choy, P. Joshi, R. Kumar, S. Hashmi, S. Agrawal, Z. Gong, J. Fine, T. B. Warkentin, A. J. Hartman, B. Ni, K. Korevec, K. Sch...

    2024 arXiv

  64. [75]

    Zheng, J

    K. Zheng, J. M. Han, and S. Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics, 2022. URL https://arxiv.org/abs/2109.00110

    2022 arXiv

  65. [76]

    Zheng, X

    Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, T. Su, Z. Yang, and J. Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, abs/2303.17568, 2023. doi:10.4855...

    2023 doi arXiv

  66. [77]

    K. Zhu, Q. Zang, S. Jia, S. Wu, F. Fang, Y. Li, S. Guo, T. Zheng, B. Li, H. Wu, et al. Lime-m: Less is more for evaluation of mllms. arXiv preprint arXiv:2409.06851, 2024 a

    2024 arXiv

  67. [78]

    M. Zhu, A. Jain, K. Suresh, R. Ravindran, S. Tipirneni, and C. K. Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.08474, 2022

    2022 arXiv

  68. [79]

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024 b

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.