REVIEW 20 cited by
McEval: Massively Multilingual Code Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Code large language models (LLMs) have shown remarkable advances in code understanding, completion, and generation tasks. Programming benchmarks, comprised of a selection of code challenges and corresponding test cases, serve as a standard to evaluate the capability of different LLMs in such tasks. However, most existing benchmarks primarily focus on Python and are still restricted to a limited number of languages, where other languages are translated from the Python samples (e.g. MultiPL-E) degrading the data diversity. To further facilitate the research of code LLMs, we propose a massively multilingual code benchmark covering 40 programming languages (McEval) with 16K test samples, which substantially pushes the limits of code LLMs in multilingual scenarios. The benchmark contains challenging code completion, understanding, and generation evaluation tasks with finely curated massively multilingual instruction corpora McEval-Instruct. In addition, we introduce an effective multilingual coder mCoder trained on McEval-Instruct to support multilingual programming language generation. Extensive experimental results on McEval show that there is still a difficult journey between open-source models and closed-source LLMs (e.g. GPT-series models) in numerous languages. The instruction corpora, evaluation benchmark, and leaderboard are available at \url{https://mceval.github.io/}.
Forward citations
Cited by 20 Pith papers
-
SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner
SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.
-
MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models
MDEval scores Markdown Awareness as the normalized edit distance between a model's HTML-tagged output and a GPT-4o rewrite, and reports human-alignment accuracy of 84.1% when ties are excluded.
-
IFEvalCode: Controlled Code Generation
A 1,620-sample, 8-language, Chinese/English benchmark separates code correctness from instruction-following and shows instruction compliance is far lower than correctness across 40+ LLMs.
-
Turning the Tide: Repository-based Code Reflection
A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.
-
Is Quantization a Deal-breaker? Empirical Insights from Large Code Models
4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.
-
Multilingual Multimodal Software Developer for Code Generation
A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.
-
FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation
A new benchmark adds 148 interactive front-end development tasks with automated sandbox tests, reporting a 90.54% agreement rate with human evaluation across four LLMs.
-
Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability
Flow2Code, a new 15-language, 16,866-flowchart benchmark, shows current multimodal LLMs generate code from plain code flowcharts only about half the time on average and substantially worse from pseudocode flowcharts.
-
Evaluating and Aligning CodeLLMs on Human Preference
A new 397-sample, 44-language code Q&A benchmark with LLM-judged human preference reveals gaps between open and proprietary code models and a 20B-token synthetic instruction corpus.
-
FullStack Bench: Evaluating LLMs as Full Stack Coders
FullStack Bench and SandboxFusion offer a broader, execution-based benchmark for code LLMs with 3,374 problems in 16 languages and 11 domains.
-
Quantize with Confidence? An Empirical Study of Quantization for Code Generation
4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.
-
Context-Aware CodeLLM Eviction for AI-assisted Coding
CACE, a context-aware eviction policy, cuts code-model reloads and response latency in self-hosted AI-assistant serving compared with LRU, though its future-demand factor reads the actual test workload.
-
TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models
TReB evaluates 26 large language models on 26 table reasoning subtasks using textual, programmatic, and interleaved reasoning modes, finding that the best model reaches only about 70 on a 0-100 judging scale.
-
dots.llm1 Technical Report
A 14B-active MoE model roughly matches Qwen2.5-72B on a broad benchmark suite while reporting about a 4x reduction in training GPU-hours.
-
Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective
A new four-task benchmark and a fine-tuned judge model show that leading AI coding assistants still generate vulnerable code, even though many can spot vulnerabilities when prompted.
-
Multi-Agent Collaboration for Multilingual Code Instruction Tuning
A multi-agent collaboration framework for generating multilingual code instruction data is introduced, but the reported evaluations show the fine-tuned model underperforming its own base instruct model.
-
BitsAI-CR: Automated Code Review via LLM in Practice
An industrial LLM-based code review system with a two-stage generate-and-filter pipeline and a data flywheel reached 75% precision and a 26.7% developer-action rate on Go code at ByteDance.
-
ExecRepoBench: Multi-level Executable Code Completion Evaluation
ExecRepoBench is an executable, repository-level code completion benchmark with unit tests, and Qwen2.5-Coder-Instruct-C is a 7B model fine-tuned on the accompanying Repo-Instruct corpus.
-
Generating Automotive Code: Large Language Models for Software Development and Verification in Safety-Critical Systems
An LLM-based, feedback-driven code generation pipeline produced an ISO-inspired ACC implementation that passed static and CARLA simulation checks in all three test runs.
-
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
A survey that frames the core problem of LLM evaluation as 'evaluation generalization': finite test sets cannot scale with unbounded model capabilities, and proposes two transitions in evaluation design.
Discussion (0). Continue with ORCID to comment.