Pith. sign in

REVIEW 5 major objections 6 minor 45 references

ML2B is the first benchmark to test LLMs on building complete ML pipelines from task descriptions in 13 languages — and it finds the language penalty is task-dependent.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:49 UTC pith:ST5S5AOF

load-bearing objection Useful benchmark, but the percentile normalization is uncalibrated and the headline numbers overreach. the 5 major comments →

arxiv 2509.22768 v3 pith:ST5S5AOF submitted 2025-09-26 cs.CL

ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation

Ekaterina Trofimova , Zosia Shamina , Maria Selifanova , Artem Zaitsev , Remi Savchuk , Maxim Minets , Daria Ozerova , Emil Sataev
show 2 more authors
Denis Zuenko Andrey E. Ustyuzhanin
This is my paper
classification cs.CL
keywords multilingual benchmarkLLM code generationmachine learning pipelinescross-lingual evaluationKaggleAutoML agentsdata leakagetask-dependent language gap
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ML²B, which it describes as the first benchmark for evaluating large language models on generating complete, runnable machine-learning pipelines from task descriptions written in languages other than English. It takes Kaggle competitions across tabular, text, and image domains, translates their descriptions into a set of languages (13 in the body, 14 in the front-matter abstract), and grades each model's generated code in a network-isolated container that requires explicit train and predict functions. Scores are reported as percentiles on the original Kaggle leaderboards, so results across competitions are meant to be comparable. The central finding is that performance loss on non-English prompts is not uniform: it ranges from near-parity to severe degradation depending on the task, modality, and language, with English usually but not always the strongest. A sympathetic reader would care because prior ML code benchmarks are English-only, and this paper is trying to measure a capability that matters for global use of LLMs in data science.

Core claim

On the paper's own terms, the discovery is that cross-lingual ML pipeline generation cannot be summarized by a single 'English-versus-other' gap. When the same competitions are presented in 13 languages under identical evaluation conditions, the drop in solution quality (measured as Kaggle-leaderboard percentile) varies by task type: image categorization and text classification stay relatively stable across languages, while tabular classification and regression show much larger and more inconsistent degradation, especially in low-resource languages such as Kazakh, Belarusian, and Romanian. The paper reports 15–45% degradation on non-English tasks overall, with some languages occasionally out

What carries the argument

The load-bearing instrument is the ML²B benchmark itself: a set of real Kaggle competitions with manually reviewed task descriptions and data cards, translated by native speakers with ML/CS backgrounds, plus a network-isolated grader that compiles submitted code, requires fixed train/predict (or train/prepare_val/predict) function signatures, and returns the competition metric. Results are normalized to percentile ranks on the official Kaggle public leaderboard, which is what lets the paper compare performance across heterogeneous metrics and tasks. The modular predict-format interface, which keeps prediction data out of the training function, is the mechanism the paper uses to curb preproce

Load-bearing premise

All cross-lingual scores are reported as percentiles on official Kaggle leaderboards, but no evidence is given that a solution's relative rank under ML²B's re-split, function-signature-constrained, network-isolated grading matches its rank under the original competition's submission conditions; if that monotonic mapping fails, the language comparisons shift and the headline gap numbers are not trustworthy.

What would settle it

Take a sample of the public Kaggle competitions in ML²B, run a range of submitted public solutions through the benchmark's grader under the same train/predict constraints, and compute the rank correlation between ML²B scores and official leaderboard positions; a weak or non-monotonic correlation would falsify the percentile-based cross-lingual comparisons, since the scores would no longer be interpretable as Kaggle percentiles.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the benchmark measures what it claims, English-only ML code-generation results overstate how ready LLMs are for global data-science users; non-English prompts can cost 15–45% in pipeline quality.
  • Low-resource languages are the weak tail: Kazakh, Belarusian, and Romanian descriptions produce higher failure rates (non-functional code) and worse median percentiles, so deployment in those languages needs explicit mitigation.
  • Task type, not language family alone, predicts the gap: tabular classification and regression are the fragile cases, while image and text tasks transfer better across languages.
  • Combining a reasoning model with a coding model in a hybrid agent reduces the cross-lingual gap in difficult domains but does not remove it, suggesting architecture-level fixes are possible but incomplete.
  • The static-leakage analysis (11% flagged, about 6% considered potential after false-positive review) implies that even structured interfaces do not fully prevent preprocessing leakage, and leakage auditing should be part of ML-agent evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the percentile normalization assumes ML²B's re-split, function-constrained, no-internet grading preserves the same ranking of solutions as the official Kaggle leaderboard. That mapping is uncalibrated, so a validation study comparing ML²B scores with official ranks on a sample of public competitions would either confirm or undermine the headline comparisons.
  • Editorial inference: because translation validation used a single annotator per text and only about two-thirds of translations were rated both natural and meaning-equivalent, some of the low-resource-language gap may be an artifact of translation quality rather than a limitation of the model's multilingual reasoning; re-running with corrected translations or with the English original appended as c
  • Note on the manuscript itself: front-matter and body disagree on benchmark scale (35 competitions, 14 languages, 490 instances in the arXiv abstract versus 30, 13, 390 in Section 3), so the exact coverage behind the aggregate percentages should be confirmed before use.
  • The benchmark design points to a concrete testable extension: feed models the English original plus a machine translation, or a language-invariant structured task specification, and measure whether the cross-lingual gap closes; if it does, the bottleneck is comprehension of the prompt rather than pipeline generation skill.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces ML²B, a benchmark for evaluating LLMs on end-to-end ML pipeline generation from multilingual natural-language descriptions. It curates 30 (abstract: 35) Kaggle competitions, translates their metadata into 13 (abstract: 14) languages, and evaluates three frontier models and two hybrid agents under two execution frameworks. The evaluation uses a Docker-isolated grader that re-splits each competition's data and requires fixed train/predict function signatures; results are then mapped to percentiles on the Kaggle public leaderboard. The headline findings are that cross-lingual degradation is task-dependent, that English generally remains the strongest language, and that low-resource languages show higher failure rates and worse percentile ranks.

Significance. If the measurement pipeline is valid, ML²B would be a valuable resource: it is the first multilingual benchmark for complete ML pipeline generation, it is open-sourced, it includes private competitions to reduce benchmark-data leakage, and it reports both containerized grading and static leakage analysis. These are real strengths. However, the paper's central comparative claims depend on an uncalibrated mapping from locally computed scores to Kaggle leaderboard percentiles, and several primary descriptive numbers disagree between the abstract and the body. The translation-validation protocol also has a documented quality ceiling that is itself language-dependent. The benchmark infrastructure is promising, but the empirical conclusions as currently presented are not yet supported by the evidence in the manuscript.

major comments (5)
  1. [§3.5, §3.2.1, §3.2.3, Tables 1/4] The percentile normalization is uncalibrated. The paper states that "Each model's result is expressed as its percentile rank on the Kaggle public leaderboard," but the scores are computed locally: competitionDataSplitter re-splits the data, the agent is constrained to train/predict function signatures, the grader runs in a network-isolated Docker container, and the submission format differs from Kaggle's. No procedure or calibration data is given for converting these local scores into public-leaderboard percentiles. If the re-split or the constrained format changes difficulty or score distributions, the median percentiles in Tables 1 and 4 are not comparable across tasks or languages, and the 15–45% degradation and English-anchor claims may be artifacts of the normalization. Please provide calibration evidence (e.g., the distribution of public solutions evaluated under the same local spl
  2. [Abstract vs. §1, §3.1, §3.4] The benchmark scope is inconsistent between the abstract and the body. The abstract reports 35 Kaggle competitions, 14 languages, 490 task-language pairs, and 10 private competitions; the introduction and benchmark description report 30 competitions, 13 languages, 390 instances, and 6 private competitions. The language list in §3.4 also contains 13 languages, not 14. Since the claimed contribution is the benchmark itself, these numbers must agree or the arxiv version must be clearly explained.
  3. [§3.4, Appendix G] Translation quality is a confound for every cross-lingual comparison. The validation used a single unpaid annotator per text, and Appendix G reports that only about two-thirds of translations were rated both natural and meaning-equivalent, with 4% rated natural but semantically different and 1.3% neither natural nor equivalent. Low-resource languages (Romanian, Kazakh) have the lowest approval rates. Because the benchmark's purpose is to measure language effects on ML pipeline generation, prompts that are not meaning-equivalent or natural in the target language cannot be interpreted as clean language variants. Please report per-language and per-field exclusion rates, state whether low-quality translations were retained in the benchmark, and show that the main results are robust to restricting the analysis to translations that passed validation.
  4. [Tables 1/4, §4, §3.5] The handling of non-functional code in percentile medians is unspecified. Table 4 uses '–' for failed tasks, and Section 4 states that low-resource languages have "frequent generation of non-functional code." Table 1 reports median percentiles per language-model pair, but the text does not state whether failed runs are assigned the 100th percentile, excluded from the median, or imputed. These choices materially change the medians for low-resource languages. Please report per-language success rates and state the exact rule used for non-functional code when computing all medians.
  5. [§3.1, §3.4, Appendix B] There is a potential circularity/contamination concern in prompt construction. Task descriptions for public competitions are reverse-inferred from solution code by GPT-4o and Claude 3.5 (Appendix B), and translations are produced by GPT-4o; among the evaluated systems are GPT-4.1-mini and hybrids built on GPT models. If the evaluated models have prior exposure to the source competitions, or if the LLM-generated descriptions are biased toward the style of the reference solutions, the benchmark may overstate or understate language effects. I am not treating this as disqualifying, but the paper should discuss contamination risk and, if feasible, report results separately for the private competitions, whose descriptions were generated manually.
minor comments (6)
  1. [Table 1] Table 1 is typeset incorrectly: several numeric fields are run together (e.g., "564564 68 59 7832 44"), and the table would benefit from explicit language column headers and a legend for the '–' symbol.
  2. [§3.4 / Appendix F] The number of form questions is inconsistent: §3.4 says "31 questions each," while Appendix F says "30 questions each." Please align.
  3. [Figure 6 caption] The caption of Figure 6 reads "Example of question block in Google Form for Romanian language," which appears to be a copy-paste error; Figure 6 is the code-based task-description generation framework.
  4. [§3.4, Table 1] Language names are abbreviated inconsistently: "Arab," "Belarus," "Russian" etc.; spell out "Arabic," "Belarusian" in running text and table headers.
  5. [Appendix D] The table title says "Selection criteria for GenML²Bench," but the benchmark is called ML²B. Please correct the name.
  6. [Table 7 / Figure 4] The scores in Table 7 and Figure 4 are labeled "normalized," but the normalization procedure is not defined in §3.5, which only describes percentile normalization. Please specify how these normalized metric scores are computed.

Circularity Check

0 steps flagged

No significant circularity: the benchmark results are empirical outputs, not restatements of construction choices.

full rationale

The paper's central claims — cross-lingual degradation, task-dependent gaps, English-anchor findings — are empirical results from running LLM agents on translated Kaggle-style tasks and scoring the generated code with external competition metrics. The construction chain does not contain a step where an output is defined as its input. (1) Section 3.5's percentile normalization states "Each model's result is expressed as its percentile rank on the Kaggle public leaderboard," but the paper does not show that local re-split scores were calibrated to official leaderboard distributions. This is a validity/evidence gap, not circularity: the local score is not defined in terms of the percentile, and the mapping is an empirical claim that could be tested or falsified. (2) Appendix B's reverse inference of task descriptions from ML code is a dataset-construction choice; the benchmark still scores models on held-out prediction quality, not on reproducing the code used to write the prompt. No fitted parameter is renamed as a prediction. (3) Using GPT-4o for translations while evaluating GPT-4.1-mini creates possible in-distribution bias, but the benchmark's headline comparisons are not derived from the translation model by definition, and translations were human-validated. The paper itself notes that only about two-thirds of translations were rated both natural and meaning-equivalent (Appendix G), which is a limitation, not a circular reduction. (4) Code4ML (Drozdova et al., 2023) shares authors with the present paper, but it is used as an external data source rather than as a load-bearing authority for the paper's conclusions; the core evaluation is self-contained against Kaggle competition metrics. Internal inconsistencies (abstract vs body counts, Table 1 typesetting, '–' handling) are correctness concerns, not circular reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters: this is a benchmark-introduction paper, not a fitted derivation; the measurements are percentile ranks from external Kaggle leaderboards, which are inputs rather than fitted constants. No new theoretical entities are posited; the MONO PREDICT and MODULAR PREDICT submission formats are engineering artifacts, not postulated entities. The load-bearing assumptions are the evaluation-mapping validity, leakage controls, and translation adequacy listed above.

axioms (5)
  • domain assumption Percentile rank on the Kaggle public leaderboard is a valid cross-competition normalization of local container scores.
    Section 3.5 maps ML2B's locally computed scores onto official leaderboard distributions without calibration or monotonicity checks; the entire Table 1 depends on this mapping.
  • domain assumption Re-splitting Kaggle training data and enforcing fixed train/predict function signatures preserves the original competition difficulty.
    Sections 3.2.3 and 3.5 assume the constrained submission format and local split do not change task difficulty, so percentile ranks remain comparable to real Kaggle submissions.
  • domain assumption LLM-reverse-inferred task descriptions do not leak solution content after human review.
    Appendix B generates descriptions from solution code with a 'do not include solution information' prompt; residual solution hints would distort task difficulty, though not necessarily language ordering.
  • domain assumption Single-annotator, unpaid validation of GPT-4o translations yields texts that are natural and meaning-preserving in all 13 languages.
    Section 3.4 and Appendix G: about two-thirds of texts rated natural and equivalent, with Romanian and Kazakh lowest; one annotator per text, no inter-annotator agreement.
  • domain assumption The 6 private competitions have no public solutions and are absent from LLM training data.
    Section 2.6 asserts this without an audit; Kaggle leaderboards and datasets remain public, and the listed abstract says 10 private competitions rather than 6.

pith-pipeline@v1.3.0-alltime-deepseek · 14830 in / 21904 out tokens · 200423 ms · 2026-08-04T14:49:18.782405+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation." pith.science (2026). https://pith.science/paper/ST5S5AOF

@misc{pith2026250922768,
  author       = {Pith},
  title        = {Pith review of: ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ST5S5AOF}},
  note         = {Machine review of arXiv:2509.22768}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce ML2B, the first benchmark for evaluating cross-lingual task comprehension in end-to-end ML pipeline generation by large language models. Despite growing global AI adoption, no systematic evaluation exists for ML pipeline generation beyond English task descriptions. ML2B addresses this gap with 35 Kaggle competitions spanning tabular, text, and image domains, translated into 14 languages by native-speaker researchers with ML expertise, yielding 490 task-language pairs. To ensure evaluation integrity, the benchmark incorporates 10 private competitions without publicly available solutions and employs network-isolated evaluation infrastructure restricting runtime access to essential ML resources. We provide standardized evaluation protocols, an AutoGluon algorithmic baseline, and comprehensive failure mode analysis. Experiments with frontier models (GPT-4.1-mini, GPT-OSS-120b, Gemini-2.5-Flash) reveal that cross-lingual performance degradation is highly task-dependent rather than following traditional resource-availability hierarchies, with gaps ranging from language advantages to severe degradation depending on competition characteristics. These findings challenge conventional assumptions about multilingual model capabilities and underscore the necessity of systematic cross-lingual evaluation for ML pipeline generation. We open-source the benchmark, baselines, and evaluation infrastructure at https://github.com/enaix/ml2b.

Figures

Figures reproduced from arXiv: 2509.22768 by Andrey E. Ustyuzhanin, Artem Zaitsev, Daria Ozerova, Denis Zuenko, Ekaterina Trofimova, Emil Sataev, Maria Selifanova, Maxim Minets, Remi Savchuk, Zosia Shamina.

Figure 1
Figure 1. Figure 1: Structure of the ML²B benchmark inference pipeline to generate draft task descriptions for a large set of ML competitions, filtering out student assignments and non-English materials (see Appendix B). All LLM-generated task descrip￾tions undergo a manual review to ensure clarity and prevent inadvertent leakage of information that can give models an unfair advantage, such as dataset sizes or model parameter… view at source ↗
Figure 2
Figure 2. Figure 2: Structure of the code grader [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Code flow diagram of benchmark submission formats [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall comparison by metrics [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overall comparison by domains Together, the charts highlight that no single LLM is universally optimal: performance varies by both metric type and application domain, underscoring the need for task-specific model selection in multilingual ML code generation [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Code-Based problem statement generation framework. The scheme incorporates three [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Task description evaluation prompt: (A) Scoring strategy component; (B) Assessment [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example of the prompt used in translation experiments. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example of question block in Google Form for Romanian language [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Distribution of competitions over domains [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Distribution of response types within each language [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Distribution of translation evaluation outcomes. (A) Overall distribution across all lan [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Example of question block in Google Form for Romanian language [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Code flow diagram of a false-positive data leakage [PITH_FULL_IMAGE:figures/full_fig_p019_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 1 canonical work pages

  1. [1]

    MEGA : Multilingual evaluation of generative AI

    Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA : Multilingual evaluation of generative AI . In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id=jmopGajkFY

    2023

  2. [2]

    Don’t push the button! exploring data leakage risks in machine learning and transfer learning

    Andrea Apicella, Francesco Isgrò, and Roberto Prevete. Don’t push the button! exploring data leakage risks in machine learning and transfer learning. Artificial Intelligence Review, 58 0 (11), August 2025. ISSN 1573-7462. doi:10.1007/s10462-025-11326-3. URL http://dx.doi.org/10.1007/s10462-025-11326-3

    doi:10.1007/s10462-025-11326-3 2025

  3. [3]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of C hat GPT on reasoning, hallucination, and interactivity. In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti,...

    2023

  4. [4]

    MLE -bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLE -bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview...

    2025

  5. [5]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    R o C ode: A dataset for measuring code intelligence from problem definitions in R omanian

    Adrian Cosma, Ioan-Bogdan Iordache, and Paolo Rosso. R o C ode: A dataset for measuring code intelligence from problem definitions in R omanian. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resour...

    2024

  7. [7]

    Abstract interpretation-based data leakage static analysis, 2024

    Filip Drobnjaković, Pavle Subotić, and Caterina Urban. Abstract interpretation-based data leakage static analysis, 2024. URL https://arxiv.org/abs/2211.16073

    Pith/arXiv arXiv 2024

  8. [8]

    Code4ml: a large-scale dataset of annotated machine learning code

    Anastasia Drozdova, Ekaterina Trofimova, Polina Guseva, Anna Scherbakova, and Andrey Ustyuzhanin. Code4ml: a large-scale dataset of annotated machine learning code. PeerJ Computer Science, 9: 0 e1230, 2023

    2023

  9. [9]

    Neural architecture search: A survey

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20 0 (55): 0 1--21, 2019

    2019

  10. [10]

    Efficient and robust automated machine learning

    Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Advances in neural information processing systems, 28, 2015

    2015

  11. [11]

    How good are gpt models at machine translation? a comprehensive evaluation, 2023

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation, 2023. URL https://arxiv.org/abs/2302.09210

    Pith/arXiv arXiv 2023

  12. [12]

    DA -code: Agent data science code generation benchmark for large language models

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. DA -code: Agent data science code generation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processin...

    doi:10.18653/v1/2024.emnlp-main.748 2024

  13. [13]

    Codesearchnet challenge: Evaluating the state of semantic code search

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019

    Pith/arXiv arXiv 1909

  14. [14]

    Aide: Ai-driven exploration in the space of code

    Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138, 2025

    Pith/arXiv arXiv 2025

  15. [15]

    Is chatgpt a good translator? yes with gpt-4 as the engine, 2023

    Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine, 2023. URL https://arxiv.org/abs/2301.08745

    Pith/arXiv arXiv 2023

  16. [16]

    Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries

    Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In Proceedings of the ACM Web Conference 2024, WWW '24, pp.\ 2627–2638, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400701719. doi:10.1145/35...

    arXiv 2024

  17. [17]

    Leakage and the reproducibility crisis in machine-learning-based science

    Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4 0 (9): 0 100804, 2023. ISSN 2666-3899. doi:https://doi.org/10.1016/j.patter.2023.100804. URL https://www.sciencedirect.com/science/article/pii/S2666389923001599

    arXiv 2023

  18. [18]

    Ds-1000: a natural and reliable benchmark for data science code generation

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: a natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  19. [19]

    H2o automl: Scalable automatic machine learning

    Erin LeDell and Sebastien Poirier. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020. ICML, 2020

    2020

  20. [20]

    Bridging the language gap: Enhancing multilingual prompt-based code generation in llms via zero-shot cross-lingual transfer

    Mingda Li, Abhijit Mishra, and Utkarsh Mujumdar. Bridging the language gap: Enhancing multilingual prompt-based code generation in llms via zero-shot cross-lingual transfer. arXiv preprint arXiv:2408.09701, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    DARTS : Differentiable architecture search

    Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS : Differentiable architecture search. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1eYHoC5FX

    2019

  23. [23]

    On leakage of code generation evaluation datasets, 2024

    Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gallé. On leakage of code generation evaluation datasets, 2024. URL https://arxiv.org/abs/2407.07565

    Pith/arXiv arXiv 2024

  24. [24]

    Evaluating programming language confusion

    Micheline B \'e n \'e dicte Moumoula, Abdoul Kader Kabore, Jacques Klein, and Tegawend \'e F Bissyande. Evaluating programming language confusion. arXiv preprint arXiv:2503.13620, 2025

    arXiv 2025

  25. [25]

    Crosslingual generalization through multitask finetuning

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning...

    2023

  26. [26]

    Olson, Nathan Bartley, Ryan J

    Randal S. Olson, Nathan Bartley, Ryan J. Urbanowicz, and Jason H. Moore. Evaluation of a tree-based pipeline optimization tool for automating data science. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO '16, pp.\ 485–492, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342063. doi:10.1145/290881...

    arXiv 2016

  27. [27]

    Dscodebench: A realistic benchmark for data science code generation

    Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M Zhang. Dscodebench: A realistic benchmark for data science code generation. arXiv preprint arXiv:2505.15621, 2025

    arXiv 2025

  28. [28]

    Efficient neural architecture search via parameters sharing

    Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp.\ 4095--4104. PMLR, 2018

    2018

  29. [29]

    Meta kaggle code, 2023

    Jim Plotts and Megan Risdal. Meta kaggle code, 2023. URL https://www.kaggle.com/ds/3240808

    arXiv 2023

  30. [30]

    m H uman E val - a multilingual benchmark to evaluate large language models for code generation

    Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. m H uman E val - a multilingual benchmark to evaluate large language models for code generation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2025

  31. [31]

    Do gpts produce less literal translations?, 2023

    Vikas Raunak, Arul Menezes, Matt Post, and Hany Hassan Awadalla. Do gpts produce less literal translations?, 2023. URL https://arxiv.org/abs/2305.16806

    Pith/arXiv arXiv 2023

  32. [32]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Sasse, E

    L. Sasse, E. Nicolaisen-Sobesky, J. Dukart, S. B. Eickhoff, M. Götz, S. Hamdan, V. Komeyer, A. Kulkarni, J. M. Lahnakoski, B. C. Love, F. Raimondo, and Kaustubh R. Patil. Overview of leakage scenarios in supervised machine learning. Journal of Big Data, 12 0 (1), May 2025. ISSN 2196-1115. doi:10.1186/s40537-025-01193-8. URL http://dx.doi.org/10.1186/s4053...

    doi:10.1186/s40537-025-01193-8 2025

  34. [34]

    Biocoder: a benchmark for bioinformatics code generation with large language models

    Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen, and Mark B Gerstein. Biocoder: a benchmark for bioinformatics code generation with large language models. Bioinformatics, 40 0 (Supplement\_1): 0 i266--i276, 2024

    2024

  35. [35]

    Scicode: A research coding benchmark curated by scientists

    Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huert...

    Pith/arXiv arXiv 2024

  36. [36]

    Lightautoml: Automl solution for a large financial services ecosystem, 2022

    Anton Vakhrushev, Alexander Ryzhkov, Maxim Savchenko, Dmitry Simakov, Rinchin Damdinov, and Alexander Tuzhilin. Lightautoml: Automl solution for a large financial services ecosystem, 2022. URL https://arxiv.org/abs/2109.01528

    Pith/arXiv arXiv 2022

  37. [37]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai soft...

    Pith/arXiv arXiv 2025

  38. [38]

    Mconala: A benchmark for code generation from multiple natural languages

    Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F Xu, and Graham Neubig. Mconala: A benchmark for code generation from multiple natural languages. arXiv preprint arXiv:2203.08388, 2022

    Pith/arXiv arXiv 2022

  39. [39]

    Lewis, and Christian Kästner

    Chenyang Yang, Rachel A Brower-Sinning, Grace A. Lewis, and Christian Kästner. Data leakage in notebooks: Static detection and better processes, 2022. URL https://arxiv.org/abs/2209.03345

    Pith/arXiv arXiv 2022

  40. [40]

    Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, 2025

    Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. Lessleak-bench: A first investigation of data leakage in llms across 83 software engineering benchmarks, 2025. URL https://arxiv.org/abs/2502.06215

    Pith/arXiv arXiv 2025

  41. [41]

    Marc-Andr\' e Z\" o ller and Marco F. Huber. Benchmark and survey of automated machine learning frameworks. J. Artif. Int. Res., 70: 0 409–472, May 2021. ISSN 1076-9757. doi:10.1613/jair.1.11854. URL https://doi.org/10.1613/jair.1.11854

    work page doi:10.1613/jair.1.11854 2021

  42. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  43. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  44. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  45. [45]

    9rD= <8rrr, 5jTXyyy5V ꬦM`0HΞ=[X5Z ꬀[t ߿XܹSԮ];i qsv Ÿu9 >|Xo =z(00PΝ n: UVzǝ]2Z 4 ڸqV 6c4j(] rR g ǎiJHH=--M + P u5k, iӦ . egg+::Z /VLL ] 3Fwy P uӧ-` (WWW+

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    1920