Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SimdBench, a 136-task benchmark across SSE, AVX, Neon, SVE, and RVV, reports that all 18 evaluated LLMs show lower pass@k on SIMD-intrinsic code than on scalar code for the same tasks.

desk verdict Solid first benchmark for LLM SIMD intrinsic code generation; universal-decrease finding is credible but needs a HumanEval-contamination sensitivity check. read the letter →

arxiv 2507.15224 v1 pith:WD62EOEI submitted 2025-07-21 cs.SE cs.AI

Yibo He , Shuoran Zhao , Jiaming Huang , Yingjie Fu , Hao Yu , Cunjian Huang , Tao Xie This is my paper
classification cs.SEcs.AI
keywords SIMDintrinsicslargelanguagemodelscodegenerationbenchmarkpass@kvectorizationdifferentialtestingRVV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SimdBench asks a question no previous code-generation benchmark has measured: whether large language models can write correct, fast code using SIMD intrinsics, the architecture-specific built-in functions that vectorize loops on x86, ARM, and RISC-V. The paper builds 136 tasks across five intrinsic families (SSE, AVX, Neon, SVE, RVV), each with a canonical scalar solution, a randomized differential-testing harness, and a Google-Benchmark-based performance harness, and evaluates 18 LLMs on them. Its central finding is that every evaluated model generates semantically correct SIMD-intrinsic code less reliably than scalar code on the same task descriptions, a universal decrease in pass@k; the strongest model, DeepSeek-R1, averages 75.44 percent pass@5 across the five intrinsics versus 92.65 percent on scalar code. A complementary finding is that a meaningful share of the SIMD samples that do compile and pass run faster than the scalar solution even at -O3, so the deficit is in generating correct intrinsics, not in their performance potential. If the findings hold, SIMD-intrinsic generation is a distinct, currently weak LLM capability with identifiable bottlenecks that fresher training data, retrieval over intrinsic documentation, or two-step scalar-to-vector generation could target.

What carries the argument

The carrying mechanism is SimdBench's per-task evaluation triple: a canonical scalar solution, a correctness harness, and a performance harness. Correctness is decided by differential testing: the generated code must compile with the target intrinsic headers, contain actual intrinsics, execute, and match the scalar reference over 1,000 seeded-then-randomized input batches per task, with Neon, SVE, and RVV runs executed under QEMU 10.0.0 emulation. Performance is measured with the Google Benchmark library at both -O0 and -O3 against the scalar baseline on native hardware, with each sample's speedup trimmed of its slowest and fastest 20 percent of runs and aggregated as efficient@k. The task set is engineered for vectorizability: 62 hand-crafted tasks covering six operation types distilled from the intrinsics documentation (data movement, integer, floating-point, comparison, logical, and reinterpret-cast operations) and 74 tasks filtered from HumanEval and HumanEval-CPP for batch-processing suitability, with prompts specifying element types, element widths, and unsafe-input behaviors.

What would settle it

Re-run the correctness harness with changed random seeds and with Neon, SVE, and RVV executed on native ARM and RISC-V hardware instead of QEMU, and audit the 74 HumanEval-derived tasks for training-data contamination. If any intrinsic family's pass@k gap versus scalar shrinks to near zero under those conditions, the universal-decrease claim would be falsified for that family; if the gap persists with five independently drawn samples per task, the claim holds.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that large language models show a universal drop in correctness when the coding task shifts from scalar code to explicit SIMD-intrinsic vectorization. With SimdBench's 136 tasks and its differential-testing protocol, the authors report that all 18 evaluated models decrease in pass@k on every one of the five intrinsic types relative to their scalar-code pass@k on the same descriptions. The best performer, DeepSeek-R1, reaches an average pass@5 of 75.44 percent across the five intrinsics while scoring 92.65 percent scalar pass@5, and seven models stay at or under 20 percent pass@5 on all five intrinsic types. On the performance side, among samples that pass correctness checks, a substantial number are faster than the canonical scalar solution compiled with -O3, which the paper reads as evidence that LLM-written intrinsics can capture headroom that compiler auto-vectorization misses. The paper attributes most failures to two error classes: compilation errors from undeclared or outdated intrinsic identifiers, dominant for SVE and RVV, and logical bugs, dominant for SSE, AVX, and Neon.

Load-bearing premise

The correctness numbers rest on the benchmark's oracle: a sample counts as correct only if it passes differential testing against the authors' canonical scalar solution over 1,000 randomized input batches per task, with ARM and RISC-V code executed under QEMU emulation, so if the test generators miss corner cases, the canonical solutions encode hidden assumptions, or the emulator's SIMD semantics diverge from real hardware, the reported pass@k values and the universal-decrease finding would be distorted.

Editorial extensions

If this is right

  • Any of the 18 evaluated models is expected to produce fewer correct solutions when asked for SIMD intrinsics than for scalar code on the same description, with the largest drops on the length-agnostic SVE and RVV families.
  • When LLM-generated SIMD code does pass correctness checks, a meaningful share runs faster than the canonical scalar solution at -O3, implying LLM vectorization can capture performance headroom that compiler auto-vectorization leaves unused.
  • The dominant failure modes separate by instruction family, with undeclared-identifier compilation errors for SVE and RVV and logical bugs for SSE, AVX, and Neon, giving fresher training data, retrieval over intrinsic documentation, and two-step scalar-to-vector generation concrete and measurable targets.
  • The only reasoning model in the study, DeepSeek-R1, leads on every intrinsic type, supporting reasoning-style inference as a promising direction for low-level code generation.
  • SimdBench's open harness provides a reusable differential-testing and performance-measurement standard across five instruction families, including reproducible seeds, compiler flags, and QEMU-based cross-execution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that per-intrinsic pass@k tracks how recent the instruction-set documentation is in the models' training data: the near-total undeclared-identifier failures on RVV and the similar profile on SVE fit a corpus lagging the 2021 RVV 1.0 naming update, which would make SimdBench a usable proxy for documentation recency.
  • The paper does not report a contamination audit for its 74 HumanEval-derived tasks, so in my reading a rebuilt version of SimdBench with fresh, non-public tasks would be the clean test of whether the universal-decrease gap is intrinsic to SIMD generation or partly inflated by memorized scalar baselines.
  • Because RVV was excluded from performance evaluation for lack of correct samples, I infer that closing the identifier-error gap via retrieval or updated training data is the binding constraint that would reveal whether RVV's length-agnostic model also translates into speedups.
  • I also infer the harness can be run in reverse: valid LLM-written SIMD samples that beat -O3 scalar code could serve as reference candidates for compiler auto-vectorization research or as a differential oracle for fuzzing SIMD-intrinsic compiler backends.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SimdBench, a benchmark of 136 C/C++ tasks for SIMD-intrinsic code generation across five intrinsic families (SSE, AVX, Neon, SVE, RVV), mixing 62 hand-crafted tasks with 74 tasks adapted from HumanEval and HumanEval-CPP. The authors evaluate 18 LLMs on correctness (pass@1, pass@5) and performance (speedup, efficient@k), using differential testing against canonical scalar solutions and Google Benchmark for timing. The headline finding is that every evaluated model shows lower pass@k on every SIMD-intrinsic family than on scalar code, with DeepSeek-R1 achieving the best SIMD results; secondary findings concern the speedups achieved by valid generated SIMD code and the dominant error types ('use of undeclared identifier' and logical bugs).

Significance. SimdBench addresses a real and underexplored capability gap: existing code-generation benchmarks do not test explicit SIMD vectorization, which is important for performance-critical software. The evaluation is broad (18 models, five intrinsic families) and the methodology has concrete strengths: identical compilers and flags for scalar/SIMD comparisons, trimmed speedup means, native hardware for performance runs, QEMU confined to correctness checks, and a public repository. If the headline claim survives a task-source decomposition and an oracle-quality validation, the paper would be a useful reference for the community. The error-type analysis (Table 5, Figure 5) is also informative for future work on training data and retrieval-augmented generation.

major comments (3)
  1. [Section 3.2, Table 4] The universal-decrease claim is computed over all 136 tasks, of which 74 are adapted from HumanEval and HumanEval-CPP. All evaluated models were trained on HumanEval, and the scalar baseline is obtained by removing the SIMD-enabling sentence from the same prompts, so for these 74 tasks the scalar pass@k can be inflated by memorization while the SIMD variant is novel. The paper reports no contamination analysis and Table 4 aggregates both task sources. Please report pass@1/pass@5 separately for the 62 hand-crafted and 74 HumanEval-derived tasks, by intrinsic family, and discuss whether the universal-decrease conclusion is affected.
  2. [Section 3.2 (prompt processing)] A related confound is that the scalar and SIMD prompts differ not only in the requirement to use intrinsics but also in length and low-level detail: Table 2 shows SimdBench prompts average 1813.8 characters versus 503.3 for HumanEval-CPP, and the SIMD prompts contain additional implementation details. The measured drop in pass@k could therefore partly reflect prompt complexity or the difficulty of following an extra constraint, rather than SIMD semantics per se. Please isolate this effect, for example by including control prompts that add a low-level but non-SIMD instruction requirement, or by comparing against a scalar prompt of matched length and detail.
  3. [Section 3.3, Section 4 (correctness oracle)] Correctness for Neon, SVE, and RVV is established only by differential testing under QEMU 10.0.0 with 1,000 randomized input batches per task against canonical scalar solutions. The paper does not validate that QEMU's SIMD semantics agree with native hardware for the tested intrinsic subset, nor that the input generators exercise corner cases such as alignment, overflow, and tail lanes. Because the reported pass@k values and the universal-decrease result depend on this oracle, please provide evidence: run a sample of generated programs on native ARM/RISC-V hardware and report any divergences, and document generator coverage (or add targeted edge-case tests). If divergences are found, the affected models should be re-evaluated.
minor comments (5)
  1. [Section 3.4] The pass@k, speedup, and efficient@k formulas are not numbered, and the speedup formula reuses n and k with different meanings from pass@k; please add equation numbers and disambiguate the notation.
  2. [Table 4] The numeric formatting in Table 4 is inconsistent (e.g., '01.76' vs '1.76' and '00.00' vs '0.00'); please use a uniform two-decimal format.
  3. [Section 3.3] The paper does not state how many repetitions Google Benchmark performs for each performance test case, nor how the trimmed mean interacts with the preset arguments; please specify these parameters so the performance results are reproducible.
  4. [Figure 3] Figure 3 uses a log scale for speedup, but the paper does not say how cases with speedup exactly equal to 1.0 are handled in the efficient@k computation; please clarify the treatment of the equality boundary.
  5. [References / front matter] The manuscript still contains ACM placeholder fields (Conference'17, DOI) and an anonymous repository URL; these should be updated before publication. Reference [9] should be de-anonymized in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SimdBench reports externally measured pass@k, speedup, and efficient@k values against canonical scalar references, with no fitted parameter being relabeled as a prediction.

full rationale

This is a benchmark paper whose central claims (universal decrease in SIMD-intrinsic pass@k, speedup of valid vectorized samples, dominant error types) are empirical measurements obtained by running 18 LLMs on 136 tasks and comparing outputs with canonical scalar solutions through differential testing and Google Benchmark. No model parameter is fitted to the benchmark data and then reported as a prediction: the sampling temperature (T=0.2), number of samples (N=5), correctness iterations (1,000), and speedup trim ratio (20%) are disclosed experimental choices, not fitted quantities. The correctness oracle is an external reference (the authors' canonical scalar implementation), and the performance baseline is the same canonical scalar code compiled with the same compiler and flags; this is a standard evaluation methodology rather than a self-referential definition. The only self-citations—RVISmith [23], CoderEval [55], and the anonymous SimdBench repository [9]—are contextual or directional and are not load-bearing premises for the measured results. A residual concern is that 74 of the 136 tasks are HumanEval-derived and no contamination analysis is reported, so the scalar baseline could be inflated by memorization; however, that is a validity threat to the experimental design, not a circular derivation in which the paper's output is equivalent to its input by construction. None of the paper's equations or definitions reduce pass@k, speedup, or efficient@k to a fitted parameter or to a self-cited theorem, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The ledger captures the six premises the headline numbers depend on: test-oracle reliability, emulator fidelity, task representativeness, sampling behavior, absence of contamination, and the standard pass@k estimator. All are measurement-side assumptions; none involve fitted constants or invented entities. The central findings are measurements, not fits. The 20 percent speedup trim, N=5 sample count, and 1,000 test iterations are disclosed methodological constants, not parameters fitted to make results hold.

assumptions (6)
  • domain assumption Differential testing against the authors' canonical scalar solution is a valid correctness oracle.
    Invoked in Section 3.3; every pass@k value depends on this oracle. Weak or buggy test generators would inflate measured correctness.
  • domain assumption QEMU 10.0.0 correctly emulates Neon, SVE, and RVV SIMD semantics for correctness evaluation.
    Used in Section 4 for all ARM and RISC-V correctness runs; an emulator or compiler mismatch could misclassify generated code.
  • ad hoc to paper The six operation types and the 136 selected tasks are representative of SIMD-intrinsic programming workloads.
    Defined in Section 3.2 from the authors' reading of intrinsic documentation plus a HumanEval filtering; the generality of the findings depends on this sampling.
  • domain assumption N=5 samples at temperature 0.2 from each official API are representative of each model's capability.
    Stated in Section 4; a single run with five samples per task bounds the precision of pass@k and efficient@k.
  • domain assumption LLM training corpora have not memorized SimdBench tasks, especially the 74 HumanEval-derived ones, in a way that inflates pass@k.
    No contamination analysis is reported in Section 3.2 or Section 4; HumanEval prompts are widely distributed.
  • standard math The unbiased pass@k estimator of Chen et al. (2021) is the correct statistic for k=5 samples.
    Used in Section 3.4; standard estimator, appropriate here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation." pith.science (2026). https://pith.science/paper/WD62EOEI

@misc{pith2026250715224,
  author       = {Pith},
  title        = {Pith review of: SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WD62EOEI}},
  note         = {Machine review of arXiv:2507.15224}
}
read the original abstract

SIMD (Single Instruction Multiple Data) instructions and their compiler intrinsics are widely supported by modern processors to accelerate performance-critical tasks. SIMD intrinsic programming, a trade-off between coding productivity and high performance, is widely used in the development of mainstream performance-critical libraries and daily computing tasks. Large Language Models (LLMs), which have demonstrated strong and comprehensive capabilities in code generation, show promise in assisting programmers with the challenges of SIMD intrinsic programming. However, existing code-generation benchmarks focus on only scalar code, and it is unclear how LLMs perform in generating vectorized code using SIMD intrinsics. To fill this gap, we propose SimdBench, the first code benchmark specifically designed for SIMD-intrinsic code generation, comprising 136 carefully crafted tasks and targeting five representative SIMD intrinsics: SSE (x86 Streaming SIMD Extension), AVX (x86 Advanced Vector Extension), Neon (ARM Advanced SIMD Extension), SVE (ARM Scalable Vector Extension), and RVV (RISC-V Vector Extension). We conduct a systematic evaluation (measuring both correctness and performance) of 18 representative LLMs on SimdBench, resulting in a series of novel and insightful findings. Our evaluation results demonstrate that LLMs exhibit a universal decrease in pass@k during SIMD-intrinsic code generation compared to scalar-code generation. Our in-depth analysis highlights promising directions for the further advancement of LLMs in the challenging domain of SIMD-intrinsic code generation. SimdBench is fully open source at https://anonymous.4open.science/r/SimdBench-1B3F/ to benefit the broader research community.

Figures

Figures reproduced from arXiv: 2507.15224 by the authors.

Figure 1
Figure 1. Examples of Matrix Multiplication Implementa [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of a Task in SimdBench. 3.1 Overview of SimdBench [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of Speedup Results for Valid Samples. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Top-20 Efficient@5 and Pass@5 Results at [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Distribution of Reasons for Invalid Cases (%). [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    AutoPass uses evidence from compiler states and runtime feedback to guide LLM agents in tuning LLVM optimizations, delivering 1.043x and 1.117x geometric-mean speedups over -O3 on x86-64 and ARM64.

  2. AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    AutoVecCoder combines VecPrompt for automated intrinsic knowledge synthesis and VecRL for efficiency-aligned RL to train an 8B LLM that achieves SOTA on SimdBench SSE/AVX subsets and sometimes exceeds -O3 compiler results.

Reference graph

Works this paper leans on

60 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mistral AI. 2024. Mistral-Large-Instruct-2411. https://huggingface.co/mistralai/ Mistral-Large-Instruct-2411

    work page 2024

  2. [2]

    Mistral AI. 2025. Codestral 25.01. https://mistral.ai/news/codestral-2501

    work page 2025

  3. [3]

    Algorithmica. 2025. Masking and Blending. https://en.algorithmica.org/hpc/ simd/masking/

    work page 2025

  4. [4]

    Anthropic. 2025. Claude. https://www.anthropic.com/claude/

    work page 2025

  5. [5]

    ARM. 2023. Neon. https://developer.arm.com/Architectures/Neon

    work page 2023

  6. [6]

    ARM. 2025. SVE Optimization Guide. https://developer.arm.com/documentation/ 102699/0100

    work page 2025

  7. [7]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Su...

    arXiv 2023

  8. [8]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv:2108.07732 [cs.PL] https://arxiv.org/abs/2108.07732

    arXiv 2021

Show all 60 references
  1. [9]

    Anonymous Author(s). 2025. SimdBench. https://anonymous.4open.science/r/ SimdBench-1B3F/

    2025

  2. [10]

    Baghsorkhi, Nalini Vasudevan, and Youfeng Wu

    Sara S. Baghsorkhi, Nalini Vasudevan, and Youfeng Wu. 2016. FlexVec: Auto- Vectorization for Irregular Loops. InProceedings of the 37th ACM SIGPLAN Confer- ence on Programming Language Design and Implementation . doi:10.1145/2980983. 2908111

    2016 doi

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] https://arxiv.org/abs/2107.03374

    2021 arXiv

  4. [12]

    Yishen Chen, Charith Mendis, Michael Carbin, and Saman Amarasinghe. 2021. VeGen: A Vectorizer Generator for SIMD and Beyond. In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems . doi:10.1145/3445814.3446692

    2021

  5. [13]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437

    2024 arXiv

  6. [14]

    DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 [cs.CL] https://arxiv.org/abs/2501. 12948

    2025 arXiv

  7. [15]

    Mingzhe Du, Luu Anh Tuan, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A Code Efficiency Benchmark for Code Large Language Models. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems . https: //arxiv.org/abs/2402.07844

    2024 arXiv

  8. [16]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating Large Language Models in Class-Level Code Generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engin...

    2024

  9. [17]

    Jing Ge Feng, Ye Ping He, Qiu Ming Tao, and Fazli Wahid. 2021. Evaluation of Compilers’ Capability of Automatic Vectorization Based on Source Code Analysis. Scientific Programming (2021). doi:10.1155/2021/3264624

    2021 doi

  10. [18]

    Sheng-Yu Fu, Jan-Jan Wu, and Wei-Chung Hsu. 2015. Improving SIMD Code Generation in QEMU. In Proceedings of the 2015 Design, Automation & Test in Europe Conference & Exhibition . 1233–1236

    2015

  11. [19]

    Google. 2025. Gemini Developer API. https://ai.google.dev/gemini-api/docs

    2025

  12. [20]

    Google. 2025. A microbenchmark support library. https://github.com/google/ benchmark

    2025

  13. [21]

    Google. 2025. XNNPACK. https://github.com/google/XNNPACK

    2025

  14. [22]

    Gemini Team Google. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https://arxiv.org/abs/2312.11805

    2025 arXiv

  15. [23]

    Yibo He, Cunjian Huang, Xianmiao Qu, Hongdeng Chen, Wei Yang, and Tao Xie

  16. [24]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Ja- cob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv:2105.09938 [cs.SE] https://arxiv.org/abs/2105.09938

    2021 arXiv

  17. [25]

    hqztrue. 2025. GitHub Repository: LeetCodeSolutions. https://github.com/ hqztrue/LeetCodeSolutions/blob/master/1801-1900/

    2025

  18. [26]

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie M. Zhang. 2024. EffiBench: Benchmarking the Efficiency of Automatically Generated Code. InPro- ceedings of the 38th Annual Conference on Neural Information Processing Systems . https://arxiv.org/abs/2402.02037

    2024 arXiv

  19. [27]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024 arXiv

  20. [28]

    Intel. 2024. Intel ® Implicit SPMD Program Compiler. https://ispc.github.io/

    2024

  21. [29]

    Intel. 2024. Intel ® Intrinsics Guide. https://www.intel.com/content/www/us/en/ docs/intrinsics-guide/index.html

    2024

  22. [30]

    RISC-V International. 2025. RISC-V Vector Intrinsic Document. https://github. com/riscv-non-isa/rvv-intrinsic-doc

    2025

  23. [31]

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2025. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. arXiv:2410.00752 [cs.SE] https://arxiv.org/abs/2410.00752 Conference’17, July 2017, Washington, DC, USA Yibo He et al

    2025 arXiv

  24. [32]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In Proceedings of the 12th International Conference on Learning Representations. https://openrevi...

    2024

  25. [33]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle- moyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. In Pro- ceedings of the 40th International Conference on Mach...

    2023 arXiv

  26. [34]

    Jianling Li, Shangzhan Li, Zhenye Gao, Qi Shi, Yuxuan Li, Zefan Wang, Jiacheng Huang, Haojie Wang, Jianrong Wang, Xu Han, Zhiyuan Liu, and Maosong Sun

  27. [35]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    2022

  28. [36]

    arXiv:2502.14752 [cs.CL] https://arxiv.org/abs/2502.14752

    TritonBench: Benchmarking Large Language Model Capabilities for Gener- ating Triton Operators. arXiv:2502.14752 [cs.CL] https://arxiv.org/abs/2502.14752

    arXiv

  29. [37]

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. EvalPlus Leaderboard. https://evalplus.github.io/leaderboard

    2024

  30. [38]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In Proceedings of the 37th International Conference on Neural Information Processing System...

    2023

  31. [39]

    Charith Mendis, Cambridge Yang, Yewen Pu, Dr.Saman Amarasinghe, and Michael Carbin. 2019. Compiler Auto-Vectorization with Imitation Learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. https://dl.acm.org/doi/10.5555/3454287.3455597

    2019

  32. [40]

    Jiawei Liu, Songrun Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and Lingming Zhang. 2024. Evaluating Language Models for Efficient Code Generation. In Proceedings of the 1st Conference on Language Modeling . https://openreview.net/ forum?id=IBCBMeAhmC

    2024

  33. [41]

    Dorit Nuzman and Richard Henderson. 2006. Multi-Platform Auto-Vectorization. In Proceedings of the 4th International Symposium on Code Generation and Opti- mization. doi:10.1109/CGO.2006.25

    2006 doi

  34. [42]

    Microsoft. 2025. ONNX Runtime. https://github.com/microsoft/onnxruntime

    2025

  35. [43]

    OpenAI. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv. org/abs/2303.08774

    2024 arXiv

  36. [44]

    Dorit Nuzman, Ira Rosen, and Ayal Zaks. 2006. Auto-Vectorization of Inter- leaved Data for SIMD. In Proceedings of the 27th ACM SIGPLAN Conference on Programming Language Design and Implementation . doi:10.1145/1133981.1133997

    2006

  37. [45]

    OpenCV.AI. 2025. OpenCV. https://opencv.org/

    2025

  38. [46]

    OpenAI. 2025. OpenAI Platform. https://platform.openai.com/docs/models

    2025

  39. [47]

    Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. 2024. Learning Performance-Improving Code Edits. arXiv:2302.07867 [cs.SE] https://arxiv.org/abs/2302.07867

    2024 arXiv

  40. [48]

    Yun Peng, Jun Wan, Yichen Li, and Xiaoxue Ren. 2025. COFFE: A Code Efficiency Benchmark for Code Generation. arXiv:2502.02827 [cs.SE] https://arxiv.org/abs/ 2502.02827

    2025 arXiv

  41. [49]

    Sergi Siso, Wes Armour, and Jeyarajan Thiyagalingam. 2019. Evaluating Auto- Vectorizing Compilers through Objective Withdrawal of Useful Information.ACM Transactions on Architecture and Code Optimization (2019). doi:10.1145/3356842

    2019 doi

  42. [50]

    simdjson. 2025. simdjson. https://github.com/simdjson/simdjson

    2025

  43. [51]

    Qwen Team. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] https: //arxiv.org/abs/2412.15115

    2025 arXiv

  44. [52]

    Qwen Team. 2025. Qwen2.5-1M Technical Report. arXiv:2501.15383 [cs.CL] https://arxiv.org/abs/2501.15383

    2025 arXiv

  45. [53]

    xAI. 2025. Grok. https://grok.com/

    2025

  46. [54]

    Theodoros Theodoridis and Zhendong Su. 2024. Refined Input, Degraded Output: The Counterintuitive World of Compiler Behavior. In Proceedings of the 45th ACM SIGPLAN Conference on Programming Language Design and Implementation . doi:10.1145/3656404

    2024 doi

  47. [55]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceed- ings of the IEEE/ACM 46th International Conference on S...

    2024

  48. [56]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. 2025. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?. ...

    2025

  49. [57]

    Zhongchun Zheng, Kan Wu, Long Cheng, Lu Li, Rodrigo C. O. Rocha, Tianyi Liu, Wei Wei, Jianjiang Zeng, Xianwei Zhang, and Yaoqing Gao. 2025. VecTrans: Enhancing Compiler Auto-Vectorization through LLM-Assisted Code Transfor- mations. arXiv:2503.19449 [cs.SE] https://arxiv.org/a...

    2025 arXiv

  50. [58]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2024. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Bench- marking on HumanEval-X. arXiv:2303.17568 [cs.LG] ht...

    2024 arXiv

  51. [60]

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Ya...

    2025 arXiv

  52. [2025]

    arXiv:2507.03773 [cs.CR] https://arxiv.org/abs/2507.03773

    RVISmith: Fuzzing Compilers for RVV Intrinsics. arXiv:2507.03773 [cs.CR] https://arxiv.org/abs/2507.03773

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.