{"id":"06bd722c-840e-4e28-b075-dd247edd2d21","arxiv_id":"2501.01054","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"More LLM-generated unit tests improve best-of-N code selection, and the lightweight CodeRM-8B generator plus dynamic scaling delivers these gains at lower cost.","lead":"This paper finds that generating more unit tests and voting over them helps pick correct code from a pool of LLM-generated candidates, and it trains a small 8B model, CodeRM-8B, to generate those tests efficiently. The approach improves code selection accuracy on benchmarks like HumanEval Plus by up to 18% for small models and about 3% for GPT-4o-mini.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CodeRM-8B's large gains on HumanEval+/MBPP+ may be inflated by train/eval contamination; the contamination-controlled LiveCodeBench results are much smaller and show mixed parity with Llama3.1-70B.","rationale":"The dynamic scaling issue raised by the reader is real but secondary: the authors acknowledge in Section 7 that adapting Damani et al.'s budget allocation to test budgets \"may not be entirely appropriate,\" and Figure 6's gains are modest; this does not threaten the main scaling and parity result. The strongest claim is about the effectiveness of unit-test scaling and CodeRM-8B's parity with Llama3.1-70B, and that claim is most vulnerable to train/evaluation leakage. The paper's own LiveCodeBench results show that the effect shrinks substantially when contamination is controlled, so a contamination check is the single most informative experiment. I therefore keep the reader's conditional verdict: the paper should release artifacts and add a deduplication and retraining analysis before the quantitative claims can be taken at face value.","tokens_in":17211,"tokens_out":17001,"duration_ms":163837,"concrete_test":"Retrain CodeRM-8B with the same synthetic-data pipeline after removing from CodeFeedback-Filtered-Instruction and TACO all problems that are exact or near-duplicate matches (e.g., 8-gram overlap after normalization) to HumanEval Plus, MBPP Plus, and the 168 LiveCodeBench queries, then recompute Table 1. If the deduplicated model's HumanEval+/MBPP+ gains drop to the few-point range seen on LiveCodeBench, and CodeRM-8B no longer matches or beats Llama3.1-70B on most cells, contamination is the primary driver. If the large gains persist after deduplication, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CodeRM-8B is trained by SFT on synthetic unit tests generated from CodeFeedback-Filtered-Instruction and TACO (Section 3.1). HumanEval and MBPP are common components of open code instruction collections, and the paper does not report deduplicating these benchmarks from the training mixture. The size of the effect is telling: on HumanEval+ the Llama3-8B gain is 18.43 points and on MBPP+ it is 17.51 points, whereas on LiveCodeBench, which the authors deliberately filtered to be contamination-free, the same setup yields only 3.23 points for Llama3-8B and 4.37 points for GPT-4o-mini. The parity claim is also weaker on uncontaminated data: CodeRM-8B trails Llama3.1-70B on LiveCodeBench for Llama3-70B (27.73 vs 28.46) and GPT-3.5 (21.76 vs 22.80), while on HumanEval+ and MBPP+ it usually leads. If the large benchmark-specific gains come from memorized test patterns, the central claim that scaling unit tests improves reward signal quality survives only in weakened form, and the \"on par with an 8x larger model\" headline becomes a statement about test-set leakage rather than about general unit-test generation ability. This is a standard contamination risk, not an accusation of misconduct.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether scaling the number of LLM-generated unit tests improves the quality of execution-based reward signals for code generation. A pioneer experiment on HumanEval Plus shows a positive correlation between unit-test count and best-of-N selection accuracy, with larger gains on harder problems. Based on this, the authors train CodeRM-8B, an 8B-parameter unit-test generator, via SFT on synthetic tests produced from CodeFeedback-Filtered-Instruction and TACO, and they implement a dynamic test-allocation scheme that assigns more tests to harder problems using a learned difficulty classifier. Experiments on HumanEval Plus, MBPP Plus, and LiveCodeBench report substantial gains over vanilla sampling and compare favorably with a Llama3.1-70B test generator.","tokens_in":17512,"tokens_out":2967,"duration_ms":27143,"significance":"If the results hold, the paper makes a useful empirical contribution: it documents a scaling behavior for test-based verifiers, provides a lightweight generator that reduces the inference cost of test scaling, and proposes an input-adaptive allocation strategy. The pioneer experiment is thoughtfully designed with bootstrap confidence intervals, and the synthetic data pipeline with execution-based filtering is a practical contribution that the authors plan to release. The main risk is benchmark contamination: the large gains on HumanEval Plus and MBPP Plus may be inflated if those benchmarks overlap the training mixture, and the contamination-controlled LiveCodeBench results are substantially weaker and show mixed parity with the much larger Llama3.1-70B generator. Because the headline claims rely on the two potentially contaminated benchmarks, this issue must be resolved before the central claims can be accepted.","major_comments":[{"comment":"The training data for CodeRM-8B is derived from CodeFeedback-Filtered-Instruction and TACO (Section 3.1). CodeFeedback-Filtered-Instruction is a collection built from open instruction-tuning datasets that are known to contain HumanEval and MBPP style problems, and the paper does not report any deduplication of benchmark problems from the training mixture. The magnitude of the reported gains is consistent with this concern: on HumanEval Plus and MBPP Plus the gains over vanilla are as large as 18.43 and 17.51 points, while on LiveCodeBench, which the authors deliberately restrict to 2024-01 through 2024-09 to avoid contamination, the same setup yields at most 4.37 points. On LiveCodeBench, CodeRM-8B also trails Llama3.1-70B for the Llama3-70B policy (27.73 vs 28.46) and for GPT-3.5 (21.76 vs 22.80), whereas on HumanEval Plus and MBPP Plus it usually leads. The authors should report whether HumanEval and MBPP problems were removed from the training data, and they should either present the contamination-controlled results as the primary evidence or substantially temper the claim that CodeRM-8B is 'on par' with a model nearly 8x larger.","section":"Section 4.1, Table 1"},{"comment":"The main results in Table 1 are reported as point estimates without confidence intervals or significance tests, despite the paper's own pioneer experiment using bootstrap resampling to report means and confidence intervals (Section 2.1). Given that the abstract and Section 4.2 use the word 'significantly' to describe the improvements, the authors should provide bootstrap intervals or paired significance tests for the main benchmark results, at least for the comparison between CodeRM-8B and Llama3.1-70B. Without such intervals, the parity claim and the differences between methods on individual benchmarks are not statistically grounded.","section":"Table 1"},{"comment":"The dynamic scaling mechanism uses the reward function q(x,b) = 1 - (1 - lambda)^b, which is imported from Damani et al. (2024), where b is a policy sampling budget. The paper itself acknowledges in Section 7 that this adaptation 'may not be entirely appropriate' because Damani et al. optimize sampling budgets for the policy model, not unit-test budgets for a reward model. The function assumes that each unit test has an independent and constant probability lambda of being passed by a correct solution, which is unlikely to hold for noisy, generated unit tests. Figure 6 shows only modest gains from dynamic allocation, and the gold-pass-rate oracle on HumanEval Plus yields little improvement. The authors should either derive a task-appropriate allocation objective or provide evidence that the chosen parametric form is a reasonable approximation for test budget allocation; otherwise the dynamic scaling component remains an unvalidated adaptation.","section":"Section 3.2, Section 7"}],"minor_comments":[{"comment":"The caption says 'Quintile 2 (hardest) has the lowest pass rate,' but the text and figure label indicate Quintile 1 is easiest and Quintile 5 is hardest; this appears to be a typo.","section":"Figure 3"},{"comment":"The sentence 'Quintile 1 (easiest) has the highest pass rate, while Quintile 2 (hardest) has the lowest pass rate' is internally inconsistent and should read 'Quintile 5 (hardest)'.","section":"Section 2.2"},{"comment":"The conclusion contains the phrase 'with more greater benefits observed in challenging problems'; this should be corrected to 'even greater benefits'.","section":"Section 6"},{"comment":"Several prompt headings contain typos: 'Repairation' should be 'Repair', 'Genearting' should be 'Generating', and 'entance' should be 'entrance'.","section":"Appendix E"},{"comment":"The introduction uses 'verifers' (should be 'verifiers') and 'compution' (should be 'computation'); these should be corrected.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The contamination concern is the main substantive issue. The authors already demonstrate awareness of contamination by filtering LiveCodeBench to a recent time window, so they should be able to provide benchmark-dedup statistics or rerun the main comparisons with a contamination-controlled split. If the contamination cannot be resolved, the paper's headline claims would need to be scaled back substantially. I would not reject outright because the pioneer experiment and the unit-test scaling idea are valuable and likely to be reproducible, but the current manuscript does not support the strength of the stated conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: the paper has a real, useful empirical core, but the headline numbers are bigger than the actual effect. The contamination-controlled results tell the truer story: scaling unit tests does help, but the large gains on HumanEval+/MBPP+ are suspect. CodeRM-8B's claim of parity with Llama3.1-70B does not survive on LiveCodeBench.\n\nWhat is genuinely new: a trained 8B unit-test generator with a synthetic-data pipeline that uses execution feedback to repair tests and filters false positives, plus majority-voting selection that improves best-of-N across several policy models. The pioneer experiment on unit-test scaling is well-designed, uses bootstrap confidence intervals, and the difficulty-dependent observation is interesting. The paper is honest in its limitations section, explicitly noting that the dynamic-scaling adaptation from Damani et al. may not be appropriate.\n\nThe soft spots are real. The training mixture (CodeFeedback-Filtered-Instruction, TACO) is not deduplicated against HumanEval/MBPP, and the effect-size gap between those benchmarks and LiveCodeBench is exactly the signature of memorized test patterns (18.43 vs 3.23 for Llama3-8B; 17.51 vs 4.37 on MBPP+ vs LiveCodeBench). Parity with Llama3.1-70B erodes on LiveCodeBench: CodeRM-8B trails on the Llama3-70B and GPT-3.5 policies. This is a standard contamination risk, not misconduct, but the magnitude claims need to be restated. There is also no code or data released yet, Table 1 has no error bars or significance tests, and the dynamic-scaling gains are modest (about 0.5% on MBPP+).\n\nBottom line: the central claim—more unit tests improve reward-signal reliability—holds, but in weakened form. The paper deserves a serious referee, but it needs a contamination analysis, artifact release, and more careful claims before I'd trust the headline numbers.","headline":"A useful empirical core on unit-test scaling, but the headline gains on HumanEval+/MBPP+ are inflated by a likely contamination effect that the LiveCodeBench numbers expose.","tokens_in":18010,"tokens_out":2201,"would_cite":true,"duration_ms":20624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generating more unit tests improves the reward signal for selecting correct code, and an 8B-parameter generator can match a 70B model at this task.","keywords":["code generation","unit test generation","reward signal","majority voting","test-time scaling","dynamic compute allocation","LLM verifier","execution feedback"],"falsifier":"Compare, on a held-out set with known ground-truth pass rates, the best-of-N accuracy achieved by dynamic allocation guided by a difficulty probe against equal allocation across several total budgets; if the probe-guided curve does not stay at or above the equal-allocation curve, the reward function $q(x,b)=1-(1-\\lambda)^b$ is not correctly capturing the value of test budget, and a calibration plot of predicted versus true $\\lambda$ would show whether the probe or the functional form is at fault.","tokens_in":16995,"feed_emoji":"🧪","tokens_out":6836,"duration_ms":55459,"temperature":0.7,"pith_summary":"The paper asks whether generating more unit tests improves the quality of the reward signal used to pick the best code solution from many LLM samples. The authors report that scaling unit tests consistently improves best-of-N selection accuracy, with larger gains on harder problems. To make scaling practical, they train CodeRM-8B, an 8B-parameter unit-test generator, on a synthetic pipeline of filtered and repaired tests, and show it reaches the selection quality of Llama3.1-70B at a fraction of the cost. They then add a dynamic allocation mechanism that spends more test-generation budget on problems predicted to be harder, squeezing extra accuracy out of a fixed compute budget.","feed_headline":"More unit tests pick better code from LLM samples","feed_subtitle":"An 8B test generator matches a 70B model and spends more tests on harder problems.","key_machinery":"Unit-test-based majority voting: for a question, the policy model samples N candidate solutions and a reward model generates M unit tests; each solution is executed on each test to produce a binary pass/fail vector, and the selected solution is the one passing the most tests. The paper's second load-bearing object is the synthetic data pipeline that trains CodeRM-8B, which uses execution feedback to repair failing tests and filters out tests that accept incorrect solutions. The third is the dynamic allocation scheme built on the reward function $q(x,b)=1-(1-\\lambda)^b$, where $\\lambda$ is the predicted pass rate of the policy model on problem $x$, and $b$ is the number of unit tests allocated; a greedy algorithm spends more tests on problems with smaller $\\lambda$.","core_discovery":"The paper's central claim is that unit-test scaling is a usable axis of test-time compute for code reward modeling: for a fixed set of candidate solutions, majority voting over more LLM-generated unit tests identifies the correct solution more often, and this benefit concentrates on difficult problems. On HumanEval Plus, MBPP Plus, and LiveCodeBench, CodeRM-8B improves selection accuracy for policy models from Llama3-8B to GPT-4o-mini (e.g., +18.43 percentage points on HumanEval Plus for Llama3-8B, +3.42 for GPT-4o-mini), and its best-of-N performance is on par with Llama3.1-70B, a model with nearly 8x more parameters. The authors also show that a difficulty probe trained on the policy model's hidden states can predict pass rates well enough to guide a greedy, budget-aware allocation of unit tests, yielding additional accuracy gains at fixed compute.","pith_inferences":["Unit-test diversity may matter at least as much as individual test accuracy: the paper's own data show a weaker generator catching up at scale, which suggests training objectives that explicitly maximize coverage across tests.","The dynamic allocation formula could be replaced by an end-to-end learned allocator that takes the policy's hidden-state difficulty probe as input and directly optimizes final selection accuracy, rather than assuming a geometric survival curve.","The same scaling argument should transfer to any execution-grounded verifier, such as SQL query validation or proof-checking, where binary pass/fail signals can be aggregated by majority vote.","A testable extension is to jointly allocate compute between sampling more candidate solutions and sampling more unit tests, since both axes appear to have diminishing returns."],"forward_implications":["For a fixed set of sampled solutions, increasing the number of unit tests improves the accuracy of majority-vote selection across model sizes and reward models.","CodeRM-8B matches the selection performance of Llama3.1-70B with roughly one-eighth the parameters, so unit-test scaling can be done with a small local generator.","Harder problems benefit more from additional unit tests, so routing extra test budget toward them improves accuracy at fixed compute.","Filtering false-positive unit tests during synthetic data construction is a major driver of the trained generator's quality."],"supporting_citations":[{"why":"Supplies the HumanEval Plus and MBPP Plus benchmarks with comprehensive ground-truth tests used for evaluation.","marker":"Liu et al., 2023"},{"why":"Provides the CodeT baseline that also uses generated tests with dual execution agreement, a main execution-based method to beat.","marker":"Chen et al., 2023"},{"why":"Provides the MBR-Exec baseline, an execution-result-based minimum Bayes risk decoding verifier.","marker":"Shi et al., 2022"},{"why":"Introduces best-of-N verification and the warning that excessive test-time compute can mislead verifiers, which the paper cites for performance drops.","marker":"Cobbe et al., 2021"},{"why":"Supplies the self-consistency majority-voting rule used to aggregate binary pass/fail results.","marker":"Wang et al., 2023"},{"why":"Source of the dynamic allocation method and the reward function q(x,b) that the paper adapts from policy-sampling budget to unit-test budget.","marker":"Damani et al., 2024"},{"why":"Introduces the probing method used to train the lightweight difficulty classifier from hidden states.","marker":"Alain and Bengio, 2017"},{"why":"Provides the bootstrap resampling used to compute the scaling curves and confidence intervals in the pioneer experiment.","marker":"Efron, 1979"}],"fun_headline_variants":["Scaling unit tests improves code reward quality","More unit tests, better code reward signals","Dynamic test scaling yields gains on code benchmarks","CodeRM-8B matches 70B with scalable unit tests","Harder problems gain more from unit test scaling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole dynamic-scaling gain rests on the assumption that the probability of at least one correct solution, $1-(1-\\lambda)^b$, is the right measure of what extra unit tests achieve—an assumption the authors borrow from a different setting and flag as imperfectly suited.","fun_headline_variants_meta":{"raw":{"variants":["Scaling unit tests improves code reward quality","More unit tests, better code reward signals","Dynamic test scaling yields gains on code benchmarks","CodeRM-8B matches 70B with scalable unit tests","Harder problems gain more from unit test scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2371,"prompt_tokens":953,"completion_tokens":1418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1346}},"tokens_in":569,"tokens_out":1418,"duration_ms":11601,"temperature":1.0,"reasoning_tokens":1346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:35:48.969938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare, on a held-out set with known ground-truth pass rates, the best-of-N accuracy achieved by dynamic allocation guided by a difficulty probe against equal allocation across several total budgets; if the probe-guided curve does not stay at or above the equal-allocation curve, the reward function $q(x,b)=1-(1-\\lambda)^b$ is not correctly capturing the value of test budget, and a calibration plot of predicted versus true $\\lambda$ would show whether the probe or the functional form is at fault.","supporting_citations":[],"review_version":1}