{"id":"33ecf66a-4833-4881-b5cb-5bc0d61f73c1","arxiv_id":"2504.15144","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"C2Rust-Bench is a reduced, purportedly representative set of 2,905 C functions for evaluating C-to-Rust transpilation tools, selected from 15,503 functions via complexity-metric binning with PCA and systematic sampling.","lead":"A team from Ohio State built a smaller benchmark for testing tools that translate C code into the memory-safe language Rust. They chose 2,905 functions from more than 15,000 in real-world programs, aiming to keep the hard cases while cutting evaluation time by about 79%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Representativeness is tuned and evaluated on the same microbenchmark using a compilation-attempt proxy; overfitting to that proxy and the absence of a random baseline leave the central claim unproven.","rationale":"The reader's weakest_assumption correctly identifies the compilation-error-fixing-attempt proxy and the lack of a random baseline as the central weakness. My stress-test agrees with that diagnosis but identifies a distinct mechanism: the selection hyperparameters are tuned by minimizing exactly the same relative-difference metric (Eq. 3) on the same microbenchmark set that is later used to evaluate representativeness. This tuning-evaluation coupling means the reported scores are best-case numbers for the method, and with no random baseline the paper cannot distinguish genuine representativeness from overfitting to the proxy distribution. I do not escalate to reject because the dataset and artifacts are released, the method is deterministic and parameter-light, and the cross-LLM generalization check in §5.4 provides genuine evidence that the proxy-based ranking is not an artifact of one LLM. The gap is fillable with a concrete random-baseline experiment and/or evaluation against the full pool, so a conditional verdict is appropriate. I mark agreement as partial because the reader's weakest_assumption captures the proxy risk but not the specific tuning-evaluation coupling that makes the current evidence inconclusive.","tokens_in":15580,"tokens_out":1652,"duration_ms":16234,"concrete_test":"Run the following check: (1) take the microbenchmark set of 1,573 functions, (2) generate 100 random subsets of size 290 (the same 18.4% sampling ratio used to select from the microbenchmark), (3) compute the Eq. 3 normalized relative-difference score for each random subset using qwen2.5-coder:32b's compilation-attempt distribution, and (4) compare the distribution of these 100 scores with the score of the selected subset. If a large fraction of random subsets match or beat the selected subset's score, the selection method has not been shown to outperform random sampling. A complementary check is to compute the same score on the full 15,503-function pool for the final 2,905-function benchmark, since the current validation never evaluates against the full pool.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that C2Rust-Bench's 2,905 functions are representative of C-to-Rust transpilation difficulty across the 15,503-function pool. Validation rests on matching the distribution of compilation-error fixing attempts: §5.3 tunes the two selection hyperparameters to minimize the normalized relative-difference score (Eq. 3) on the 1,573-function microbenchmark set, and §5.4 measures representativeness using that same score on that same set. This creates a selection loop in which the metric used for evaluation is exactly the metric optimized during tuning, on exactly the same data. The paper provides no evaluation of the final 2,905-function set against the full pool and no random-sampling baseline. Without a baseline, the observed relative-difference scores do not show that the selection method adds representativeness beyond what a random subset of the same size would achieve; subset size, binning, and the proxy itself could account for the match. The cross-LLM generalization in §5.4 is real evidence that the proxy ranking is not idiosyncratic to qwen2.5-coder:32b, but it does not break the tuning-evaluation coupling. The reader's weakest_assumption correctly flags the proxy assumption; the sharper concern is that the paper's own validation cannot distinguish selection-by-method from selection-by-chance on the metric being optimized.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces C2Rust-Bench, a dataset of 2,905 C functions selected from a pool of 15,503 functions drawn from 65 real-world programs, intended to be a minimized yet representative benchmark for evaluating C-to-Rust transpilation. The selection method defines four complexity metrics: Maintainability Index of the C code, Maintainability Index of the corresponding Rust code, an unsafe-code complexity metric, and a data-type complexity metric. Functions are placed into multidimensional bins by partitioning each metric axis, and systematic sampling is applied within bins after ordering by a PCA-based complexity score. A transpilation tool with a compilation-error-fixing loop and a local LLM (qwen2.5-coder:32b) is used to obtain Rust code and to produce a distribution of compilation-error fixing attempts. The paper tunes two hyperparameters (number of partitions and sampling ratio) on a 1,573-function microbenchmark set, then evaluates the resulting selections on the same set across nine LLMs by comparing normalized compilation-error fixing attempt distributions using a relative-difference score. The central claim is that the final 2,905-function set is representative of C-to-Rust transpilation difficulty for the full 15,503-function pool.","tokens_in":15879,"tokens_out":4366,"duration_ms":39023,"significance":"If the representativeness claim were established, C2Rust-Bench would fill a genuine gap in the C-to-Rust transpilation literature: a standardized, smaller evaluation set that saves significant LLM inference time while still covering the range of transpilation difficulty. The paper contributes a concrete dataset artifact and open-source selection code, and the cross-LLM evaluation in §5.4 is a useful step toward showing that the selection is not purely idiosyncratic to one model. The reduction numbers are also meaningful: 81.3% fewer functions and 78.9% less transpilation time. However, the current validation is self-referential: the same microbenchmark set and the same relative-difference metric are used for hyperparameter tuning and for the representativeness claim, and no random-sampling or alternative-method baseline is provided. The proxy itself (compilation-error fixing attempts) covers only compilability, not semantic correctness, idiomaticity, or memory safety, which are central to the stated use case. The final 2,905-function selection is never validated against the full pool.","major_comments":[{"comment":"The representativeness validation is in-sample and self-referential. In §5.3, the hyperparameters (number_of_partition=9, ratio_of_sampling=0.166) are chosen by minimizing the relative-difference score of Eq. (3) on the 1,573-function microbenchmark set. In §5.4, the same score on the same set is then used to claim that the selected functions are representative. This means the evaluation metric is exactly the objective that was optimized, and the data are exactly the data used for optimization. The cross-LLM results in Figure 2 show that the proxy ranking is not unique to qwen2.5-coder:32b, but they do not break the tuning-evaluation coupling. Please add an out-of-sample evaluation, for example by holding out a portion of the microbenchmark set during tuning, or by validating the final selection from the full pool against the full pool.","section":"§5.3–§5.4"},{"comment":"The final C2Rust-Bench set is never validated. Section 5.4 performs selection only on the microbenchmark set, not on the full 15,503-function pool, and reports relative-difference scores between the selected functions and the microbenchmark set. Section 5.5 then presents the final 2,905-function selection from the full pool without any representativeness score against that pool. Since the abstract and introduction claim representativeness for the final 2,905 functions selected from 15,503, the paper must either report the relative-difference (or an equivalent metric) for the final selection against the full pool, or explicitly justify why the microbenchmark-based validation transfers to the large-set selection.","section":"§5.4–§5.5"},{"comment":"No random-sampling baseline or alternative selection method is compared. The paper reports that the tuned selection achieves a relative-difference score of 11.2% and that other LLMs yield scores between roughly 15% and 35%, but without a baseline it is not established that the partitioning-based method outperforms a random subset of the same size. Subset size, binning, and the normalization in Eq. (3) could by themselves produce low scores. Please add a random-sampling baseline (with multiple seeds) and, ideally, a stratified-sampling baseline, and report the distribution of relative-difference scores under those baselines.","section":"§5.3"},{"comment":"The representativeness claim rests entirely on the assumption that the distribution of compilation-error fixing attempts is a valid and sufficient proxy for C-to-Rust transpilation difficulty. This proxy measures only compilability of the transpiled Rust code; it does not capture semantic correctness, idiomaticity, or memory safety, all of which are central to the transpilation evaluation use case described in §1 and §2.2. The paper should either provide evidence that this proxy correlates with those properties (e.g., by comparing the proxy against semantic-equivalence or memory-safety metrics on a sample) or qualify the representativeness claim to compilability-based difficulty and adjust the wording in the abstract accordingly.","section":"§3.2, §5.3"},{"comment":"The construction of the microbenchmark set is underspecified. The text states only that \"we sample ~10% of the functions in the large dataset\" and obtains 1,573 functions. The sampling method is not described (uniform random, stratified, seeded), and no comparison is given between the metric distributions of the microbenchmark set and the large set. Since the microbenchmark set is used both for hyperparameter tuning and for the main validation, its representativeness is load-bearing. Please specify the sampling procedure and demonstrate that the microbenchmark set reflects the large set's distribution on the four complexity metrics.","section":"§5.1"}],"minor_comments":[{"comment":"Equation (3) divides by expected_value_i for each i from 0 to 20; if any expected count is zero, the score is undefined. The paper does not state how zero-count bins are handled; please clarify.","section":"§5.3, Eq. (3)"},{"comment":"There is a typo in §4.1: \"preprare\" should be \"prepare\".","section":"§4.1"},{"comment":"The y-axis label in Figure 3 reads \"Fuctions,\" which should be \"Functions.\" There is also a typo in §5.4: \"microbencmark\" should be \"microbenchmark.\"","section":"Figure 3"},{"comment":"The introduction refers to the \"US White House Office of National Cyber Directory\" and cites ONCD; the correct name is the \"Office of the National Cyber Director.\"","section":"§1"},{"comment":"Reference [1] is dated 2025 in the bibliography, but the cited CISA article \"The Urgent Need for Memory Safety in Software Products\" was published in December 2023. Please correct the date.","section":"References"},{"comment":"The program name \"json.h\" in Table 1 appears to be a file name rather than a program name; please clarify what this entry represents.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real need and ships artifacts, but the current validation is circular and lacks a baseline. The recommended revision path—adding a random baseline, an out-of-sample or full-pool validation, and a justification of the proxy—is well within scope of a major revision and does not require changing the core approach. I would not recommend rejection, but the central representativeness claim must be supported before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Karim,\n\nThe paper builds a plausible selection pipeline for a C-to-Rust transpilation benchmark and ships a dataset plus code, which is more than most work in this space. But the central claim—that the selected 2,905 functions are representative of the 15,503-function pool—is not actually demonstrated. The validation is circular: the two hyperparameters are tuned to minimize the relative difference in compilation-error fixing attempt distributions on the microbenchmark, and the same metric on the same microbenchmark is then used to claim representativeness. There is no random-sampling baseline, and the final dataset is never evaluated against the full pool.\n\nWhat is genuinely new: the use of domain-specific complexity metrics (MI for C and Rust, unsafe-code density, data-type variety) to drive selection, and the combination of partitioning with PCA and systematic sampling. The cross-LLM evaluation in §5.4 is real evidence that the proxy is not idiosyncratic to qwen2.5-coder:32b—the relative difference scores are similar across models—though this does not break the tuning-evaluation coupling. The reduction in size and time is substantial, and the artifacts are properly released.\n\nThe main soft spot is the missing external validation. A simple random-sampling baseline would have told the reader whether the method adds anything beyond subset size and binning. The proxy itself—compilation-error fixing attempts—only measures compilability, not semantic correctness, idiomaticity, or memory safety of the transpiled code, so the representativeness claim is only about one narrow aspect of transpilation difficulty. The paper should say this explicitly. Also, the large pool is itself only the union of datasets from four prior works; \"representative\" means representative of that pool, not of all real-world C programs.\n\nThis is a decent dataset paper with a real methodological gap. It deserves a serious referee, but the authors need to add a random baseline and an evaluation of the final set against the full pool before the representativeness claim is sound. If that can't be done, the claim should be softened. I'd engage with it and would bring it to a reading group to discuss the validation issue, but I probably wouldn't cite the benchmark as a ground truth until the baseline exists.","headline":"Useful dataset and selection pipeline, but the representativeness claim rests on a circular validation on the same microbenchmark with no random baseline.","tokens_in":16351,"tokens_out":2362,"would_cite":true,"duration_ms":20266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2,905-function benchmark claims to capture the full range of C-to-Rust transpilation difficulty across 15,503 real-world functions.","keywords":["C-to-Rust transpilation","benchmark dataset","representative subset selection","Maintainability Index","LLM evaluation","memory safety migration","function selection","program complexity metrics"],"falsifier":"Take a transpiler or LLM not among the nine studied, transpile all 15,503 functions and the 2,905-function benchmark, then compare the two sets on a difficulty measure that is not compilation-error fixing attempts, such as semantic equivalence test pass rate or idiomaticity score. If the benchmark's distribution diverges from the full pool's on that measure, or if transpiler rankings on the benchmark disagree with rankings on the full pool, the representativeness claim is refuted.","tokens_in":15414,"feed_emoji":"🦀","tokens_out":5126,"duration_ms":41880,"temperature":0.7,"pith_summary":"The paper argues that evaluating C-to-Rust transpilation does not require running every function of a large program pool, because a carefully chosen subset can stand in for the whole. It builds C2Rust-Bench, a set of 2,905 functions selected from 15,503 real-world functions, and claims this subset reproduces the distribution of transpilation difficulty found in the full pool. The motivation is practical: memory-safety migration from C to Rust is a priority, but testing transpilers on very large datasets is slow, and no shared evaluation set existed. If the representativeness claim holds, future work can compare transpilers on the small set and trust that the ranking transfers to the larger set, cutting evaluation time by roughly 79 percent.","feed_headline":"One benchmark shrinks C-to-Rust testing from 15,503 to 2,905 functions","feed_subtitle":"If representative, the 81% smaller set reproduces full-pool transpilation difficulty for LLM evaluation.","key_machinery":"The mechanism is the four-metric binning and systematic sampling pipeline. Maintainability Index, a composite of cyclomatic complexity, Halstead volume, and source lines of code, measures code complexity; the two Rust-specific metrics add memory-operation and type-system difficulty that generic code metrics miss. Partitioning each metric into equal-width intervals forms multidimensional bins, PCA reduces the four metrics to one ordering score per bin, and systematic sampling spreads the choice across each bin. This combination is what lets a small set claim to cover the variety of transpilation challenges in the large set.","core_discovery":"The central discovery is a selection procedure that reduces a 15,503-function C program pool to 2,905 functions while preserving the shape of transpilation difficulty, operationalized as the number of compilation-error fixing attempts an LLM needs. Each function is scored on four complexity metrics: the Maintainability Index of the original C code, the Maintainability Index of the LLM-transpiled Rust code, the average number of statements inside Rust unsafe blocks, and the number of unique data types in the Rust output. Functions are binned by partitioning each metric's range, ordered within each bin by a PCA complexity score, and sampled systematically so every bin contributes. The paper validates the result by comparing the compilation-error fixing attempt distribution of the selected set with that of the full pool and reports that the selected functions remain representative across nine different LLMs, not just the one used during selection.","pith_inferences":["The same selection pipeline could plausibly be applied to other migration tasks, such as C++ to Rust or C to memory-safe variants, but the proxy metric would need to be revalidated for each target language.","The compilation-error proxy says nothing directly about whether transpiled code is semantically correct, idiomatic, or memory-safe; a benchmark built on it may need to be paired with execution-based or verification-based tests to capture those dimensions.","A natural stress test is to rebuild the benchmark from a different large program pool or with a different base LLM; if the selected functions change substantially, generalizing the current 2,905 functions beyond this pool would be risky."],"forward_implications":["Evaluating a C-to-Rust transpiler on C2Rust-Bench should produce results that transfer to the 15,503-function pool, so researchers can skip the full run.","Transpilation evaluation time drops by roughly 79 percent, from 246 hours to 52 hours on the authors' setup.","The benchmark gives competing transpilation tools a common data set, making their reported results directly comparable.","The selection procedure is reusable: given a function pool and a transpiler, the same four-metric binning and sampling steps can produce a custom minimized evaluation set."],"supporting_citations":[{"why":"Supplies real-world C programs that form part of the large dataset used for selection.","marker":"[4]"},{"why":"Supplies another set of C programs folded into the 15,503-function pool.","marker":"[5]"},{"why":"Supplies the LLM-based transpilation work whose evaluation programs are included in the pool.","marker":"[6]"},{"why":"Supplies further C programs and the output-parameter translation technique that motivates Rust-side metrics.","marker":"[7]"},{"why":"Provides the c2rust example programs included in the large dataset.","marker":"[24]"},{"why":"Defines the Maintainability Index used as a complexity metric for C and Rust functions.","marker":"[16]"},{"why":"Defines cyclomatic complexity, one of the three components of the Maintainability Index.","marker":"[17]"},{"why":"Defines Halstead volume, another component of the Maintainability Index.","marker":"[18]"},{"why":"Provides the Rust code analysis tool used to compute Maintainability Index for transpiled Rust code.","marker":"[23]"}],"fun_headline_variants":["C-to-Rust eval dataset shrinks 81% without losing difficulty","Representative C-to-Rust benchmark: 2,905 functions from 15,503","New method picks 2,905 C functions that mimic full transpilation test","C2RUST-BENCH: 81% smaller, still representative for transpilation","From 15k to 2.9k: a representative C-to-Rust test set"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole representativeness claim rests on assuming that the distribution of compilation-error fixing attempts is a valid and sufficient proxy for C-to-Rust transpilation difficulty, and that matching that distribution on the selected set implies the set is representative for the full pool and for other LLMs.","fun_headline_variants_meta":{"raw":{"variants":["C-to-Rust eval dataset shrinks 81% without losing difficulty","Representative C-to-Rust benchmark: 2,905 functions from 15,503","New method picks 2,905 C functions that mimic full transpilation test","C2RUST-BENCH: 81% smaller, still representative for transpilation","From 15k to 2.9k: a representative C-to-Rust test set"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001001,"raw_usage":{"total_tokens":4207,"prompt_tokens":887,"completion_tokens":3320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":3210}},"tokens_in":503,"tokens_out":3320,"duration_ms":19692,"temperature":1.0,"reasoning_tokens":3210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:31:48.798237+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a transpiler or LLM not among the nine studied, transpile all 15,503 functions and the 2,905-function benchmark, then compare the two sets on a difficulty measure that is not compilation-error fixing attempts, such as semantic equivalence test pass rate or idiomaticity score. If the benchmark's distribution diverges from the full pool's on that measure, or if transpiler rankings on the benchmark disagree with rankings on the full pool, the representativeness claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies real-world C programs that form part of the large dataset used for selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the c2rust example programs included in the large dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Maintainability Index used as a complexity metric for C and Rust functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Rust code analysis tool used to compute Maintainability Index for transpiled Rust code."}],"review_version":1}