{"id":"ba07240c-6374-4efe-848b-157bcaeca55a","arxiv_id":"2608.04439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ExeCRE estimates reference-code reliability from execution-consistency patterns using Dawid-Skene, filters unreliable code before it generates self-correction tests, and improves Pass@1 on LiveCodeBench across four LLMs.","lead":"This paper presents ExeCRE, a method that checks how trustworthy a program is by running several candidate programs on many random inputs and seeing whether they agree. It uses those agreement patterns to stop unreliable test signals from entering LLM self-correction for code generation, improving correctness on hard problems and cutting wasted revision steps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Majority-defined latent state can equate buggy consensus with reliability; Section 7 concedes this, and Table 8 does not isolate majority-wrong inputs.","rationale":"The paper has genuine strengths: a public artifact, cutoff-clean results on Qwen2.5-Coder-32B-Instruct and LLaMA-3.1-8B-Instruct, and a large reduction in misleading feedback. The concern is not that the method is useless, but that its central reliability estimate is only as valid as the assumption that majority consistency tracks correctness. The reader's weakest assumption identifies exactly this point, and the paper itself concedes it in Section 7. Table 8 provides some empirical reassurance, but it does not condition on the dangerous regime where the wrong candidates form the majority. The proposed test would settle whether this is a rare edge case or a systemic confound. Because the empirical comparisons still show overall gains and the limitation is acknowledged, the existing CONDITIONAL verdict remains appropriate; no adjustment is needed.","tokens_in":21098,"tokens_out":4530,"duration_ms":43637,"concrete_test":"Use the released artifact to stratify existing results by majority-correctness. For each LiveCodeBench problem, using hidden-test labels, compute m_i = fraction of generated inputs on which the argmax output (the U_ij = 1 side) is semantically correct. Report ExeCRE's incorrect-adoption rate separately for problems with high m_i versus low m_i, and compute the correlation between m_i and incorrect adoption. If incorrect adoption concentrates in low-m_i problems, the latent state is tracking majority output rather than correctness; if it does not, the Section 7 caveat is not a first-order limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the binary projection in Section 3.2.2: U_ij = 1 iff O_ij equals argmax_v |{k: O_ik = v}|. This defines the latent 'consistency state' solely from the majority side of each input. Dawid–Skene is then fit to U, and the reliability score in Section 3.2.4, alpha_j = p_1 * pi^(j)_{1,1} + p_0 * pi^(j)_{0,0}, rewards agreement with that inferred state. Nothing in the model ties the latent state to semantic correctness. If 6 or more of the 10 sampled reference codes share a bug, the majority side is the buggy output, so a buggy code can obtain high alpha and be selected as C*. The paper's own Section 7 states this failure mode: 'ExeCRE can adopt an incorrect reference code when many wrong candidates share a bug that generated inputs do not expose.' The RQ4 robustness analysis (Table 8) does not settle the point: it buckets by number of wrong output groups, but not by whether wrong candidates form the majority, and it shows 10.2% incorrect adoption even when all wrong codes form a single group. Since the entire downstream self-correction trusts C*, this is not a cosmetic caveat; it is the condition under which 'reliability' reduces to majority agreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ExeCRE, a framework that estimates the reliability of LLM-generated reference code without ground-truth tests. ExeCRE constructs many inputs from an LLM-extracted schema, executes up to 10 sampled candidate codes on these inputs, projects the raw outputs to binary majority-consistency labels, and fits a Dawid–Skene model to infer latent consistency states and per-code error rates. The resulting reliability score alpha_j is used to either select a trusted reference code for building oracle tests in a self-correction pipeline or to fall back to public tests when no candidate exceeds a threshold. The authors evaluate on 182 LiveCodeBench problems with GPT-5.2, DeepSeek-V3.2, Qwen2.5-Coder-32B-Instruct, and LLaMA-3.1-8B-Instruct, plus a smaller GSM8K program-of-thought study. They report improved Pass@1 relative to several baselines, substantially reduced misleading feedback, and competitive F1 for semantic reliability identification.","tokens_in":21403,"tokens_out":7125,"duration_ms":63208,"significance":"If the empirical findings hold, the paper makes a useful contribution: it provides a practical, execution-only method for estimating code reliability in self-correction pipelines and demonstrates, with hidden-test grounding, that consistency-based aggregation can reduce harmful feedback. The strongest evidence is the large reduction in misleading feedback on already-correct code (Table 6) and the RQ2 hidden-test evaluation showing that the reliability score has meaningful semantic discrimination. The paper is also transparent about its main limitation: a majority of wrong candidates that share a bug can define the 'reliable' side. Because the central mechanism is empirical rather than guaranteed, the value of the paper depends on how well the evaluation isolates the majority-wrong regime. The manuscript includes reproducible artifacts, a threshold-sensitivity analysis, an input-generator ablation, and careful contamination caveats, all of which strengthen the presentation.","major_comments":[{"comment":"The binary projection U_ij = 1 iff O_ij equals the modal output of the sampled codes for input i, and the reliability score alpha_j = p1*pi^(j)_{1,1} + p0*pi^(j)_{0,0} rewards agreement with the inferred latent consistency state. This latent state is derived from the majority side of each input, so a buggy consensus can receive high reliability. Section 7 concedes this failure mode. Table 8 does not resolve it: it buckets instances by the number of wrong output groups, but it does not report whether the wrong candidates form the majority, nor does it report incorrect-adoption rates conditional on the modal output being semantically wrong. The one-group row (10.2% incorrect adoption) is precisely the regime where the majority is likely to be wrong, yet the table does not separate that case. Please add an analysis that isolates inputs or problems where the modal output is incorrect and report incorrect-adoption and selected-code accuracy in those cases. If the error is high in that regime, the claim that alpha tracks semantic reliability should be revised to an explicit diversity assumption; if the error is low, the analysis would substantially strengthen the paper.","section":"Sec. 3.2.2, Sec. 3.2.4, Sec. 7, Table 8"},{"comment":"The abstract and RQ1 claim that ExeCRE 'consistently improves' effectiveness, but the Qwen2.5-Coder-32B result does not clearly support this: ExeCRE is 32.5±1.4 versus TextGrad's 31.5±1.0, a difference within one standard deviation, and no significance tests or confidence intervals are reported anywhere in Table 2. The claim should be supported with paired significance tests across the five runs or softened to say that ExeCRE improves on the models where the difference is significant. Without this, 'consistently' is not established for the smallest two models in the main comparison.","section":"Sec. 5.1, Table 2"},{"comment":"The paragraph after the projection states that 'reliable behavior may correspond to the minority side when several codes share the same systematic mistake.' This is not operationalized in the DS fit: because U_ij labels the modal output as 1 for every input, a code that is always in the majority will have a high expected agreement with the latent state even if that majority is systematically wrong. The RQ2 evaluation provides empirical grounding, but it does not directly measure how often the inferred latent state is the incorrect side. Please add a diagnostic that reports, over the evaluation problems, the fraction of inputs where the modal output is incorrect and how often the selected code is correct on those inputs.","section":"Sec. 3.2.2, Sec. 5.2"}],"minor_comments":[{"comment":"In the manuscript text, the entries in Table 2 are concatenated without clear column separation (e.g., '62.7±1.899.6±1.874.2±3.534.9±3.1'); the final PDF must render this as a proper table with separate All/Easy/Medium/Hard columns.","section":"Table 2"},{"comment":"The RQ2 reliability-identification results are reported without standard deviations or confidence intervals; since the paper elsewhere emphasizes mean±std over five runs, the same reporting should be applied to the F1/Precision/Recall tables.","section":"Sec. 5.2, Tables 4 and 5"},{"comment":"The tie-breaking rule for the majority projection is described as 'arbitrarily selected'; this can affect U and therefore alpha, and the tie-breaking rule should be specified and preferably evaluated for sensitivity.","section":"Sec. 3.2.2"},{"comment":"The illustrative example reports scores 13/21 and 11/21 for voting, but does not show how these numbers are derived from the three inputs and seven codes; please include the computation.","section":"Sec. 5.2.1"},{"comment":"The input-generator ablation is limited to 65 function-call problems; please state whether this subset is representative of the full 182-problem set and whether the conclusion transfers to stdin-style problems.","section":"Sec. 5.4.2"},{"comment":"The statement that invalid schemas 'usually lead to low reliability scores rather than confident adoption' is not quantified; reporting the distribution of alpha for schema-failure cases would make the robustness argument more concrete.","section":"Sec. 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its key limitation, and the hidden-test evaluation in RQ2 is the right kind of grounding. My main concern is that Table 8 does not directly test the majority-wrong regime; if the authors can add that analysis and the results are favorable, I would view the central claim as adequately supported. I also recommend de-emphasizing the GPT-5.2 headline numbers in the abstract, since the paper itself notes the contamination caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my take on ExeCRE. The paper does something real: it treats the correctness of reference code as a latent variable and estimates it with Dawid-Skene EM on a binary projection of execution outputs. That's a new angle—CodeT and ALGO use agreement, but they don't try to infer per-code reliability from the pattern of agreement across many generated inputs. The authors integrate this into self-correction by selecting only a sufficiently reliable reference code, else falling back to public tests. The result is a consistent Pass@1 improvement across four models—modest on Qwen (32.5 vs 31.5) but larger on DeepSeek and LLaMA—and a dramatic drop in misleading feedback: on GPT-5.2, from 113.2 cases to 14.0. That's a practically meaningful reduction in unnecessary corrections. The RQ2 reliability identification results are also decent: ExeCRE beats LLM judges and simple voting in F1 for picking a correct reference code. The paper ships a public artifact and the experiments are reasonably controlled.\n\nThe soft spot is the one the stress-test flagged. The binary projection U_ij defines the 'consistent' side as the majority output for that input. Dawid-Skene then learns which code tends to agree with that side and calls it reliable. If six of ten sampled codes share a bug, the majority side is wrong, and a buggy code can get a high alpha and be selected as the reference. Section 7 admits this, but the RQ4 table doesn't directly measure how often majority-wrong happens—it buckets by number of wrong output groups, not by whether the wrong codes form the majority. Table 8 shows 10.2% incorrect adoption even when all wrong codes form one group, which is not reassuring. However, this does not sink the paper. The empirical hidden-test validation shows that in practice the estimator tracks correctness well enough to beat baselines, and the authors are candid about the failure mode. It's a limitation, not a fatal flaw.\n\nOther minor issues: the paper overuses 'significantly' without significance tests; some reliability tables lack error bars; the 'semantic-only' setting treats TLE as correct, which is arguable for brute-force reference code but should be discussed. The GPT-5.2 contamination caveat is disclosed, and the cutoff-clean results on Qwen/LLaMA still show the qualitative pattern.\n\nWho is this for? Anyone working on execution-guided self-correction, test generation, or using consensus to validate LLM-generated code. It's a within-subfield contribution, useful and honest. I'd send it to peer review. A serious referee should ask for the majority-wrong analysis and proper statistical tests, but the core idea and experiments are worth engaging with.","headline":"Solid, honest paper: ExeCRE's Dawid–Skene on binary execution consistency is a genuine new technique for gating self-correction, and the empirical reduction in misleading feedback is large; the main caveat—majority consensus can masquerade as reliability—is acknowledged but not solved.","tokens_in":21896,"tokens_out":3563,"would_cite":true,"duration_ms":30910,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ExeCRE claims execution-consistency patterns can stand in for ground-truth tests when estimating the reliability of LLM-generated reference code, and that gating self-correction on that estimate improves final correctness.","keywords":["large language models","code generation","self-correction","reliability estimation","execution consistency","Dawid-Skene","reference code","program-of-thought reasoning"],"falsifier":"Take a problem where seven of ten sampled reference codes implement the same wrong threshold (say, 'subtotal > 25' instead of 'subtotal >= 25'), generate inputs that rarely hit the boundary value, and run ExeCRE; if the selected code clears the 0.95 threshold yet fails the hidden tests that do hit the boundary, the reliability estimate has adopted an incorrect reference, contradicting the paper's robustness claim.","tokens_in":20910,"feed_emoji":"🧩","tokens_out":7967,"duration_ms":66296,"temperature":0.7,"pith_summary":"The paper tries to establish that the reliability of LLM-generated reference code can be estimated without hidden tests or LLM judgment, by executing several candidate codes on many generated inputs and analyzing who agrees with whom. Concretely, ExeCRE samples ten brute-force reference codes, runs them on roughly three hundred schema-constrained inputs per problem, and turns each output into a binary signal: does this code match the plurality output on this input? A Dawid–Skene model then treats inputs as items and codes as noisy annotators, inferring a latent 'reliable behavior' state per input and a per-code error rate. Only a code whose inferred reliability clears a threshold is allowed to generate verification tests; otherwise the pipeline falls back to public tests. On 182 LiveCodeBench problems across four models, this raises Pass@1 (the fraction of problems solved on the first attempt) relative to the TextGrad-style baseline it is built on and cuts the number of misleading feedback cases on already-correct code roughly eightfold on GPT-5.2. The practical upshot is that generated code can police itself through execution agreement before it is trusted as an oracle.","feed_headline":"Execution-agreement scores cut misleading code-fix feedback 8x","feed_subtitle":"A Dawid–Skene reliability gate over LLM code outputs lifts self-correction accuracy across four models.","key_machinery":"The load-bearing object is a binary execution-consistency matrix $U$, where $U_{ij}=1$ if candidate code $C_j$'s output on input $I_i$ equals the plurality output across all candidates on that input, and $U_{ij}=0$ otherwise; failed executions are NaN and excluded. On this matrix the paper runs the Dawid–Skene EM algorithm, originally an annotator-agreement model: inputs are items, codes are annotators, and each input has a latent binary state indicating which side of the partition is the reliable behavior. The inferred parameters $\\hat{p}_q$ (class priors) and $\\hat{\\pi}^{(j)}_{qa}$ (per-code error rates) combine into a reliability score $\\alpha_j=\\hat{p}_1\\hat{\\pi}^{(j)}_{11}+\\hat{p}_0\\hat{\\pi}^{(j)}_{00}$, the expected probability that code $C_j$ matches the latent consistency state. The machinery does the work of distinguishing a code that is correct but sometimes disagrees with a buggy majority from a code that is systematically on the wrong side, which plain voting cannot do.","core_discovery":"The central discovery is that the latent variable separating reliable from unreliable behavior on each input can be recovered from a binary projection of execution outputs, and that the resulting per-code error rates track semantic correctness better than simple majority voting, LLM self-judgment, or fine-tuned judge models. In the paper's own comparison on DeepSeek-V3.2, ExeCRE identifies correct reference code with an F1 of 76.62, ahead of ExeCRE-Voting (70.83), the best judge-model baseline (72.64), and LLM-judgment baselines (below 63). The same filtering reduces misleading feedback on already-correct code from 113.2 to 14.0 problems on GPT-5.2 and from 30.0 to 1.2 on LLaMA-3.1-8B-Instruct over the 182-problem set. The paper also reports that replacing the schema-driven input generator with fuzzing or LLM-generated inputs keeps reliability identification F1 in the same band, and that applying the same reliability gate to code-based math reasoning on GSM8K lifts accuracy on Qwen2.5-Coder-32B from 90.28% to 92.96% and on LLaMA-3.1-8B from 75.16% to 78.85%.","pith_inferences":["Editorial extension: because the projection measures agreement with the panel's plurality rather than with ground truth, a single correct code surrounded by many wrong codes that share one bug will be scored low; weighting codes by an external prior (e.g., model confidence or prior pass rates) could make the estimator robust to that asymmetry.","Editorial extension: the same consistency matrix could be reused to estimate the reliability of generated test inputs themselves, not just reference codes, turning the method into a generic quality gate inside longer agentic coding loops.","Editorial extension: the paper's binary projection discards the identity of the outputs, so two codes can appear consistent yet differ on edge cases; coupling ExeCRE with an oracle-free disagreement measure that bounds pointwise error could give a certificate-like lower bound on correctness.","Editorial extension: since schema extraction fails on 3–19% of problems depending on the model, a perturbation-based or learned input constructor might extend the reliability gate to repository-level tasks where schemas are harder to extract."],"forward_implications":["Reference codes can be screened before they generate tests, so self-correction iterates only on failures that are likely real; the misleading-feedback counts drop by roughly an order of magnitude across all four evaluated models.","The reliability gate transfers beyond LeetCode-style problems: on GSM8K program-of-thought reasoning, the same estimator decides when generated code should be trusted versus when the model should answer directly.","The estimator is not tied to a particular input generator; replacing schema-driven inputs with fuzzing or LLM-generated tests keeps reliability identification F1 between 0.79 and 0.81 on function-call problems.","The 0.95 adoption threshold is not a delicate knob: precision and recall trade off smoothly while F1 stays stable over a broad range, which makes the method practical without per-benchmark tuning."],"supporting_citations":[{"why":"Supplies the Dawid–Skene EM model that ExeCRE adapts to estimate code-specific error rates from binary consistency labels.","marker":"[8]"},{"why":"Establishes that brute-force reference codes are often accurate enough to construct oracle tests, the premise ExeCRE builds on and evaluates.","marker":"[54]"},{"why":"Provides the TextGrad self-correction pipeline that ExeCRE integrates with and improves over in the main comparison.","marker":"[53]"},{"why":"ConTested is a consistency-based test-driven self-correction baseline; the paper compares misleading feedback against it.","marker":"[10]"},{"why":"CodeT supplies the LLM-generated-test approach used as a comparison and as one alternative input generator in the ablation.","marker":"[2]"},{"why":"Oracle-Guided selection is a baseline and its fuzzing method is one of the input generators tested in the ablation.","marker":"[16]"},{"why":"B4 is the Bayesian noisy-test selection baseline that ExeCRE must beat on Pass@1.","marker":"[3]"},{"why":"LiveCodeBench is the benchmark and contamination-window definition used for the main 182-problem evaluation.","marker":"[27]"},{"why":"Supplies the implementation of the Dawid–Skene estimator used in the experiments.","marker":"[48]"},{"why":"GSM8K provides the test set for the additional code-based mathematical reasoning study.","marker":"[6]"}],"fun_headline_variants":["Execution-consistency gate cuts misleading code fixes 8x","Dawid-Skene over code outputs slashes bad feedback 8x","Reliability stats on code execution trim false fixes 113→14","Consistency-based reliability boosts self-correction accuracy","Execution-consistency model lifts code self-correction scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The estimate only works if the plurality of sampled candidate codes on a given input is usually the semantically correct behavior; if most candidates share the same bug that the generated inputs do not expose, the latent state can be defined by the wrong side and an incorrect code can receive a high reliability score.","fun_headline_variants_meta":{"raw":{"variants":["Execution-consistency gate cuts misleading code fixes 8x","Dawid-Skene over code outputs slashes bad feedback 8x","Reliability stats on code execution trim false fixes 113→14","Consistency-based reliability boosts self-correction accuracy","Execution-consistency model lifts code self-correction scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1725,"prompt_tokens":1081,"completion_tokens":644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":697,"tokens_out":644,"duration_ms":5778,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:40:17.129084+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a problem where seven of ten sampled reference codes implement the same wrong threshold (say, 'subtotal > 25' instead of 'subtotal >= 25'), generate inputs that rarely hit the boundary value, and run ExeCRE; if the selected code clears the 0.95 threshold yet fails the hidden tests that do hit the boundary, the reliability estimate has adopted an incorrect reference, contradicting the paper's robustness claim.","supporting_citations":[],"review_version":2}