Pith. sign in

REVIEW 5 major objections 6 minor 27 references

C2RUST-BENCH: A Minimized, Representative Dataset for C-to-Rust Transpilation Evaluation

T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 2,905-function benchmark claims to capture the full range of C-to-Rust transpilation difficulty across 15,503 real-world functions.

desk verdict Useful dataset and selection pipeline, but the representativeness claim rests on a circular validation on the same microbenchmark with no random baseline. read the letter →

arxiv 2504.15144 v1 pith:DHM6DERP submitted 2025-04-21 cs.CR cs.AIcs.PL

Melih Sirlanci , Carter Yagemann , Zhiqiang Lin This is my paper · ORCID
classification cs.CRcs.AIcs.PL
keywords C-to-RusttranspilationbenchmarkdatasetrepresentativesubsetselectionMaintainabilityIndexLLMevaluationmemorysafetymigrationfunctionprogramcomplexitymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that evaluating C-to-Rust transpilation does not require running every function of a large program pool, because a carefully chosen subset can stand in for the whole. It builds C2Rust-Bench, a set of 2,905 functions selected from 15,503 real-world functions, and claims this subset reproduces the distribution of transpilation difficulty found in the full pool. The motivation is practical: memory-safety migration from C to Rust is a priority, but testing transpilers on very large datasets is slow, and no shared evaluation set existed. If the representativeness claim holds, future work can compare transpilers on the small set and trust that the ranking transfers to the larger set, cutting evaluation time by roughly 79 percent.

What carries the argument

The mechanism is the four-metric binning and systematic sampling pipeline. Maintainability Index, a composite of cyclomatic complexity, Halstead volume, and source lines of code, measures code complexity; the two Rust-specific metrics add memory-operation and type-system difficulty that generic code metrics miss. Partitioning each metric into equal-width intervals forms multidimensional bins, PCA reduces the four metrics to one ordering score per bin, and systematic sampling spreads the choice across each bin. This combination is what lets a small set claim to cover the variety of transpilation challenges in the large set.

What would settle it

Take a transpiler or LLM not among the nine studied, transpile all 15,503 functions and the 2,905-function benchmark, then compare the two sets on a difficulty measure that is not compilation-error fixing attempts, such as semantic equivalence test pass rate or idiomaticity score. If the benchmark's distribution diverges from the full pool's on that measure, or if transpiler rankings on the benchmark disagree with rankings on the full pool, the representativeness claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is a selection procedure that reduces a 15,503-function C program pool to 2,905 functions while preserving the shape of transpilation difficulty, operationalized as the number of compilation-error fixing attempts an LLM needs. Each function is scored on four complexity metrics: the Maintainability Index of the original C code, the Maintainability Index of the LLM-transpiled Rust code, the average number of statements inside Rust unsafe blocks, and the number of unique data types in the Rust output. Functions are binned by partitioning each metric's range, ordered within each bin by a PCA complexity score, and sampled systematically so every bin contributes. The paper validates the result by comparing the compilation-error fixing attempt distribution of the selected set with that of the full pool and reports that the selected functions remain representative across nine different LLMs, not just the one used during selection.

Load-bearing premise

The whole representativeness claim rests on assuming that the distribution of compilation-error fixing attempts is a valid and sufficient proxy for C-to-Rust transpilation difficulty, and that matching that distribution on the selected set implies the set is representative for the full pool and for other LLMs.

Editorial extensions

If this is right

  • Evaluating a C-to-Rust transpiler on C2Rust-Bench should produce results that transfer to the 15,503-function pool, so researchers can skip the full run.
  • Transpilation evaluation time drops by roughly 79 percent, from 246 hours to 52 hours on the authors' setup.
  • The benchmark gives competing transpilation tools a common data set, making their reported results directly comparable.
  • The selection procedure is reusable: given a function pool and a transpiler, the same four-metric binning and sampling steps can produce a custom minimized evaluation set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same selection pipeline could plausibly be applied to other migration tasks, such as C++ to Rust or C to memory-safe variants, but the proxy metric would need to be revalidated for each target language.
  • The compilation-error proxy says nothing directly about whether transpiled code is semantically correct, idiomatic, or memory-safe; a benchmark built on it may need to be paired with execution-based or verification-based tests to capture those dimensions.
  • A natural stress test is to rebuild the benchmark from a different large program pool or with a different base LLM; if the selected functions change substantially, generalizing the current 2,905 functions beyond this pool would be risky.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces C2Rust-Bench, a dataset of 2,905 C functions selected from a pool of 15,503 functions drawn from 65 real-world programs, intended to be a minimized yet representative benchmark for evaluating C-to-Rust transpilation. The selection method defines four complexity metrics: Maintainability Index of the C code, Maintainability Index of the corresponding Rust code, an unsafe-code complexity metric, and a data-type complexity metric. Functions are placed into multidimensional bins by partitioning each metric axis, and systematic sampling is applied within bins after ordering by a PCA-based complexity score. A transpilation tool with a compilation-error-fixing loop and a local LLM (qwen2.5-coder:32b) is used to obtain Rust code and to produce a distribution of compilation-error fixing attempts. The paper tunes two hyperparameters (number of partitions and sampling ratio) on a 1,573-function microbenchmark set, then evaluates the resulting selections on the same set across nine LLMs by comparing normalized compilation-error fixing attempt distributions using a relative-difference score. The central claim is that the final 2,905-function set is representative of C-to-Rust transpilation difficulty for the full 15,503-function pool.

Significance. If the representativeness claim were established, C2Rust-Bench would fill a genuine gap in the C-to-Rust transpilation literature: a standardized, smaller evaluation set that saves significant LLM inference time while still covering the range of transpilation difficulty. The paper contributes a concrete dataset artifact and open-source selection code, and the cross-LLM evaluation in §5.4 is a useful step toward showing that the selection is not purely idiosyncratic to one model. The reduction numbers are also meaningful: 81.3% fewer functions and 78.9% less transpilation time. However, the current validation is self-referential: the same microbenchmark set and the same relative-difference metric are used for hyperparameter tuning and for the representativeness claim, and no random-sampling or alternative-method baseline is provided. The proxy itself (compilation-error fixing attempts) covers only compilability, not semantic correctness, idiomaticity, or memory safety, which are central to the stated use case. The final 2,905-function selection is never validated against the full pool.

major comments (5)
  1. [§5.3–§5.4] The representativeness validation is in-sample and self-referential. In §5.3, the hyperparameters (number_of_partition=9, ratio_of_sampling=0.166) are chosen by minimizing the relative-difference score of Eq. (3) on the 1,573-function microbenchmark set. In §5.4, the same score on the same set is then used to claim that the selected functions are representative. This means the evaluation metric is exactly the objective that was optimized, and the data are exactly the data used for optimization. The cross-LLM results in Figure 2 show that the proxy ranking is not unique to qwen2.5-coder:32b, but they do not break the tuning-evaluation coupling. Please add an out-of-sample evaluation, for example by holding out a portion of the microbenchmark set during tuning, or by validating the final selection from the full pool against the full pool.
  2. [§5.4–§5.5] The final C2Rust-Bench set is never validated. Section 5.4 performs selection only on the microbenchmark set, not on the full 15,503-function pool, and reports relative-difference scores between the selected functions and the microbenchmark set. Section 5.5 then presents the final 2,905-function selection from the full pool without any representativeness score against that pool. Since the abstract and introduction claim representativeness for the final 2,905 functions selected from 15,503, the paper must either report the relative-difference (or an equivalent metric) for the final selection against the full pool, or explicitly justify why the microbenchmark-based validation transfers to the large-set selection.
  3. [§5.3] No random-sampling baseline or alternative selection method is compared. The paper reports that the tuned selection achieves a relative-difference score of 11.2% and that other LLMs yield scores between roughly 15% and 35%, but without a baseline it is not established that the partitioning-based method outperforms a random subset of the same size. Subset size, binning, and the normalization in Eq. (3) could by themselves produce low scores. Please add a random-sampling baseline (with multiple seeds) and, ideally, a stratified-sampling baseline, and report the distribution of relative-difference scores under those baselines.
  4. [§3.2, §5.3] The representativeness claim rests entirely on the assumption that the distribution of compilation-error fixing attempts is a valid and sufficient proxy for C-to-Rust transpilation difficulty. This proxy measures only compilability of the transpiled Rust code; it does not capture semantic correctness, idiomaticity, or memory safety, all of which are central to the transpilation evaluation use case described in §1 and §2.2. The paper should either provide evidence that this proxy correlates with those properties (e.g., by comparing the proxy against semantic-equivalence or memory-safety metrics on a sample) or qualify the representativeness claim to compilability-based difficulty and adjust the wording in the abstract accordingly.
  5. [§5.1] The construction of the microbenchmark set is underspecified. The text states only that "we sample ~10% of the functions in the large dataset" and obtains 1,573 functions. The sampling method is not described (uniform random, stratified, seeded), and no comparison is given between the metric distributions of the microbenchmark set and the large set. Since the microbenchmark set is used both for hyperparameter tuning and for the main validation, its representativeness is load-bearing. Please specify the sampling procedure and demonstrate that the microbenchmark set reflects the large set's distribution on the four complexity metrics.
minor comments (6)
  1. [§5.3, Eq. (3)] Equation (3) divides by expected_value_i for each i from 0 to 20; if any expected count is zero, the score is undefined. The paper does not state how zero-count bins are handled; please clarify.
  2. [§4.1] There is a typo in §4.1: "preprare" should be "prepare".
  3. [Figure 3] The y-axis label in Figure 3 reads "Fuctions," which should be "Functions." There is also a typo in §5.4: "microbencmark" should be "microbenchmark."
  4. [§1] The introduction refers to the "US White House Office of National Cyber Directory" and cites ONCD; the correct name is the "Office of the National Cyber Director."
  5. [References] Reference [1] is dated 2025 in the bibliography, but the cited CISA article "The Urgent Need for Memory Safety in Software Products" was published in December 2023. Please correct the date.
  6. [Table 1] The program name "json.h" in Table 1 appears to be a file name rather than a program name; please clarify what this entry represents.

Circularity Check

1 steps flagged · score 4.0 of 10

Representativeness is tuned and evaluated with the same compilation-attempt distribution on the same microbenchmark, making the chosen-LLM validation in-sample; the final 2,905-function set is never scored against the full pool.

  1. fitted input called prediction [§5.3 (Eq. 3) and §5.4 'Cross-LLM Evaluation' Methodology]
    "To select best hyperparameters, we look for minimizing the relative difference score. The combination (9, 0.166), placed at the index of 883, which is marked by the red dashed line in Figure 1, has the lowest relative difference score of 11.2%. ... To evaluate the selected functions, we use the compilation error fixing attempt from the transpilation process as in §5.3."

    The representativeness score in §5.4 is Equation 3, the relative difference of compilation-error fixing attempt distributions. This is exactly the objective minimized during hyperparameter tuning in §5.3, and it is computed on the same 1,573-function microbenchmark with the same qwen2.5-coder:32b feedback. Therefore the low relative-difference score reported for qwen2.5-coder:32b is the tuned value or a re-computation of it, not an independent confirmation: the fitting metric and the validation metric coincide on the same data. The final 2,905-function C2Rust-Bench is never scored against the full 15,503-function pool, so the only direct evidence for representativeness with the chosen LLM is in-sample.

full rationale

The dataset construction itself is largely independent: the selection uses complexity metrics (Maintainability Index, unsafe-code complexity, data-type complexity) and PCA-based systematic sampling, none of which are derived from the compilation-attempt feedback used for validation. However, the paper's validation of representativeness is partially self-referential. In §5.3 the two selection hyperparameters are chosen by minimizing the relative-difference score on the microbenchmark set, and in §5.4 the same score on the same microbenchmark set is presented as evidence that the selected functions are representative. For qwen2.5-coder:32b, the model used for selection and tuning, this is an in-sample evaluation of the optimized objective. The cross-LLM results in §5.4 are genuinely informative for other models, since those scores were not minimized during tuning. Still, the paper provides no random-sampling baseline and never measures the representativeness of the final 2,905-function benchmark against the full 15,503-function pool. These gaps weaken the support for the central claim but do not make the construction itself circular, so a moderate score is appropriate.

Assumptions & free parameters 2 free parameters · 7 assumptions · 0 invented entities

The central claim rests on two tuned hyperparameters, several domain-specific metric definitions, and an internal validation proxy. The two grid-searched hyperparameters are explicit free parameters. The axioms list the unproven assumptions about what makes code hard to transpile and about the validity of the error-fixing proxy.

free parameters (2)
  • number_of_partition = 9
    Tuned in §5.3 by grid search over 1..20 to minimize relative difference in compilation-error fixing attempt distribution across bins; controls bin granularity.
  • ratio_of_sampling = 0.166
    Tuned in §5.3 by grid search over 0.002..0.2 with step 0.002 to minimize the same relative difference metric; controls fraction sampled per bin.
assumptions (7)
  • domain assumption Maintainability Index of C and Rust code is a valid complexity proxy for C-to-Rust transpilation difficulty.
    Introduced in §3.2; the selection and all reported metrics depend on MI summarizing cyclomatic complexity, Halstead volume, and SLoC. No evidence links MI to actual transpilation success beyond the paper's internal proxy.
  • domain assumption Unsafe-code density in transpiled Rust reflects transpilation complexity.
    Stated in §3.2; the average unsafe statements per block is used as one of the four binning metrics. It assumes more unsafe code indicates harder transpilation, which is plausible but not independently calibrated.
  • domain assumption Number of unique data types in transpiled Rust reflects transpilation complexity.
    Defined in §3.2 and §4.3; used as the fourth metric. Assumes type diversity is a meaningful axis of transpilation difficulty without external evidence.
  • ad hoc to paper The distribution of compilation-error fixing attempts is a valid and sufficient proxy for representativeness of transpilation difficulty.
    Introduced in §5.3 and used in §5.4 as the only quantitative validation of representativeness. This proxy is the load-bearing validation instrument; the paper provides no evidence that it correlates with semantic correctness or quality of the transpiled code.
  • domain assumption The ~10% microbenchmark sample is representative of the large dataset.
    Stated in §5.1; the microbenchmark is used for LLM selection, hyperparameter tuning, and cross-LLM validation. If the random sample is biased, all tuned parameters and validation results are biased.
  • domain assumption Segmenting C files into individual functions preserves the features relevant to transpilation.
    The method in §4.1 splits preprocessed C files into functions, treating each function as an independent unit. The paper assumes function-level transpilation is the right granularity, following [8], but does not evaluate whether cross-function dependencies affect representativeness.
  • standard math PCA score ordering and systematic sampling yield a diverse, representative sample within each bin.
    The selection procedure in §3.3 rests on standard statistical assumptions that ordering by the first principal component and taking evenly spaced samples covers the bin's spread. This is a reasonable heuristic, not a proven guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of C2RUST-BENCH: A Minimized, Representative Dataset for C-to-Rust Transpilation Evaluation." pith.science (2026). https://pith.science/paper/DHM6DERP

@misc{pith2026250415144,
  author       = {Pith},
  title        = {Pith review of: C2RUST-BENCH: A Minimized, Representative Dataset for C-to-Rust Transpilation Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHM6DERP}},
  note         = {Machine review of arXiv:2504.15144}
}
read the original abstract

Despite the effort in vulnerability detection over the last two decades, memory safety vulnerabilities continue to be a critical problem. Recent reports suggest that the key solution is to migrate to memory-safe languages. To this end, C-to-Rust transpilation becomes popular to resolve memory-safety issues in C programs. Recent works propose C-to-Rust transpilation frameworks; however, a comprehensive evaluation dataset is missing. Although one solution is to put together a large enough dataset, this increases the analysis time in automated frameworks as well as in manual efforts for some cases. In this work, we build a method to select functions from a large set to construct a minimized yet representative dataset to evaluate the C-to-Rust transpilation. We propose C2RUST-BENCH that contains 2,905 functions, which are representative of C-to-Rust transpilation, selected from 15,503 functions of real-world programs.

Figures

Figures reproduced from arXiv: 2504.15144 by the authors.

Figure 1
Figure 1. The change of relative difference over combinations [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. The relative difference for 9 LLMs. Results. In [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Compilation error fixing attempt distribution of selected and microbenchmark sets for 9 LLMs. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The instructions given to LLM for initial transpila [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The instructions given to LLM for fixing compila [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 17 canonical work pages

  1. [1]

    The Urgent Need for Memory Safety in Software Products

    2025. The Urgent Need for Memory Safety in Software Products. https://www. cisa.gov/news-events/news/urgent-need-memory-safety-software-products

    work page 2025

  2. [2]

    Fact Sheet: ONCD Report Calls for Adoption of Memory Safe Programming Languages and Addressing the Hard Research Problem of Software Measura- bility

    2025. Fact Sheet: ONCD Report Calls for Adoption of Memory Safe Programming Languages and Addressing the Hard Research Problem of Software Measura- bility. https://bidenwhitehouse.archives.gov/oncd/briefing-room/2024/02/26/ memory-safety-fact-sheet/

    work page 2025

  3. [3]

    TRACTOR: Translating All C to Rust

    2025. TRACTOR: Translating All C to Rust. https://www.darpa.mil/research/ programs/translating-all-c-to-rust

    work page 2025

  4. [4]

    Mehmet Emre, Ryan Schroeder, Kyle Dewey, and Ben Hardekopf. 2021. Trans- lating C to safer Rust. Proceedings of the ACM on Programming Languages 5, OOPSLA (2021), 1–29

    work page 2021

  5. [5]

    Hanliang Zhang, Cristina David, Yijun Yu, and Meng Wang. 2023. Ownership guided C to Rust translation. In International Conference on Computer Aided Verification. Springer, 459–482

    2023

  6. [6]

    Aidan ZH Yang, Yoshiki Takashima, Brandon Paulsen, Josiah Dodds, and Daniel Kroening. 2024. VERT: Verified equivalent rust transpilation with large language models as few-shot learners. arXiv preprint arXiv:2404.18852 (2024)

    arXiv 2024

  7. [7]

    Jaemin Hong and Sukyoung Ryu. 2024. Don’t Write, but Return: Replacing Output Parameters with Algebraic Data Types in C-to-Rust Translation. Proceedings of the ACM on Programming Languages 8, PLDI (2024), 716–740

    2024

  8. [8]

    Momoko Shiraishi and Takahiro Shinagawa. 2024. Context-aware Code Segmen- tation for C-to-Rust Translation using Large Language Models. arXiv preprint arXiv:2409.10506 (2024)

    arXiv 2024

Show all 27 references
  1. [9]

    Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, Wil Robertson, Frederick Ulrich, and Ryan Whelan. 2016. Lava: Large-scale automated vulnerability addition. In 2016 IEEE symposium on security and privacy (SP). IEEE, 110–121

    2016

  2. [10]

    Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. 2020. Magma: A ground- truth fuzzing benchmark. Proceedings of the ACM on Measurement and Analysis of Computing Systems 4, 3 (2020), 1–29

    2020

  3. [11]

    Olivier Bachem, Mario Lucic, and Andreas Krause. 2017. Practical coreset con- structions for machine learning. arXiv preprint arXiv:1703.06476 (2017)

    2017 arXiv

  4. [12]

    Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017)

    2017 arXiv

  5. [13]

    Georgii Novikov, Maxim Panov, and Ivan Oseledets. 2021. Dataset Reduction via Bias-Variance Minimization. In 2021 5th Scientific School Dynamics of Complex Networks and their Applications (DCNA) . IEEE, 143–146

    2021

  6. [14]

    Hojun Lee, Suyoung Kim, Junhoo Lee, Jaeyoung Yoo, and Nojun Kwak. 2024. Coreset selection for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7682–7691

    2024

  7. [15]

    Haohao Song, Qiao Xiang, and Jiwu Shu. 2025. Leave No Stone Unturned: Optimizing Subpattern Information Entropy for Coreset Selection. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2025

  8. [16]

    Don Coleman, Dan Ash, Bruce Lowther, and Paul Oman. 1994. Using metrics to evaluate software system maintainability. Computer 27, 8 (1994), 44–49

    1994

  9. [17]

    Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308–320

    1976

  10. [18]

    Maurice H Halstead. 1977. Elements of Software Science (Operating and program- ming systems series). Elsevier Science Inc

    1977

  11. [19]

    immunant. 2025. c2rust. https://github.com/immunant/c2rust

    2025

  12. [20]

    Dokyung Song, Julian Lettner, Prabhu Rajasekaran, Yeoul Na, Stijn Volckaert, Per Larsen, and Michael Franz. 2019. SoK: Sanitizing for security. In 2019 IEEE Symposium on Security and Privacy (SP) . IEEE, 1275–1295

    2019

  13. [21]

    Paul Oman and Jack Hagemeister. 1992. Metrics for assessing a software system’s maintainability. In Proceedings Conference on Software Maintenance 1992 . IEEE Computer Society, 337–338

    1992

  14. [22]

    Jarod42. 2025. ccccc. https://github.com/Jarod42/ccccc

    2025

  15. [23]

    Luca Ardito, Luca Barbato, Marco Castelluccio, Riccardo Coppola, Calixte Denizet, Sylvestre Ledru, and Michele Valsesia. 2020. rust-code-analysis: A Rust library to analyze and extract maintainability information from source codes. SoftwareX 12 (2020), 100635

    2020

  16. [24]

    immunant. 2025. c2rust. https://github.com/immunant/c2rust/tree/master/ examples

    2025

  17. [25]

    J Arturo Olvera-López, J Ariel Carrasco-Ochoa, J Francisco Martínez-Trinidad, and Josef Kittler. 2010. A review of instance selection methods. Artificial Intelli- gence Review 34 (2010), 133–143

    2010

  18. [26]

    Sariel Har-Peled and Akash Kushal. 2005. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-first annual symposium on Com- putational geometry. 126–134

    2005

  19. [27]

    Ivor W Tsang, James T Kwok, Nello Cristianini, et al. 2005. Core vector machines: Fast SVM training on very large data sets. Journal of machine Learning research 6, 4 (2005). A IMPLEMENTATION DETAILS A.1 Transpilation of C Functions into Rust Behave like you are an expert of C...

    2005

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.