Pith. sign in

REVIEW 4 cited by

MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.08388 v2 pith:4DXV3B7K submitted 2022-03-16 cs.CL

Zhiruo Wang , Grace Cuenca , Shuyan Zhou , Frank F. Xu , Graham Neubig This is my paper · ORCID
classification cs.CL
keywords codelanguagesgenerationenglishnaturaldatasetmconalaacross
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric. This creates a barrier for program developers who are not proficient in English. To mitigate this gap in technology development across languages, we propose a multilingual dataset, MCoNaLa, to benchmark code generation from natural language commands extending beyond English. Modeled off of the methodology from the English Code/Natural Language Challenge (CoNaLa) dataset, we annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. We present a quantitative evaluation of performance on the MCoNaLa dataset by testing with state-of-the-art code generation systems. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts, revealing the challenges in adapting code generation to new languages.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

    cs.SE 2026-07 conditional novelty 8.0 of 10

    RuBench, the first repository-level coding benchmark with natively authored (non-translated) Russian task specifications, measures deployed coding agents on 25 contamination-gated fix tasks and documents model substit...

  2. ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new open-source benchmark evaluates LLM-generated end-to-end ML pipelines from Kaggle competition descriptions translated into 13 languages, with 6 private tasks to limit data leakage.

  3. Fortran2CPP: Automating Fortran-to-C++ Translation using LLMs via Multi-Turn Dialogue and Dual-Agent Integration

    cs.LG 2024-12 conditional novelty 6.0 of 10

    The paper introduces a dialogue-based fine-tuning dataset for Fortran-to-C++ translation and reports up to 3.31x CodeBLEU improvement and up to 92 percent compilation success on fine-tuned LLMs.

  4. Toward Inclusive AI-Driven Development: Exploring Gender Differences in Code Generation Tool Interactions

    cs.SE 2025-07 unverdicted novelty 4.0 of 10

    A registered-report style proposal for testing gender differences in how developers interact with AI code generation tools, with no results reported yet.

Pith tools