{"id":"0f975d47-7b17-4ec4-9cb6-ae5edc38ad54","arxiv_id":"2604.03922","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Expected leave-one-out AUC of a test equals a positive coefficient times its latent discriminative power, enabling closed-form and optimized test weights that approximate the oracle ranking of code candidates.","lead":"ACES ranks LLM-generated code by weighting each generated test according to how well its pass/fail pattern agrees with the ranking induced by all other tests. The method breaks the circular dependency between code quality and test quality using only the binary pass matrix and improves Pass@k on standard code-generation benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The reader correctly isolates Theorem 3 as the strongest claim and Assumption 4 plus the conditional-independence model as the weakest points. Both are already stated, proved, and stress-tested in the manuscript; the empirical sign-recovery plots and the complementary ACES-O variant further blunt the practical impact of the assumption. No additional load-bearing gap appears that would move the verdict away from ACCEPT.","tokens_in":37645,"tokens_out":456,"duration_ms":23605,"concrete_test":"Generate a synthetic pass matrix under the paper’s model but with heterogeneous incorrect codes (each incorrect code draws its own β-vector from a Dirichlet, inducing within-class score non-exchangeability). Recompute the empirical correlation between (LOO-AUC_j − ½)·p_j(1−p_j) and true δ_j; if the sign-recovery rate falls below 85 % while the homogeneous baseline stays above 94 %, the identity’s practical robustness is weaker than claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central LOO-AUC identity (Theorem 3) and the consequent recovery of sign(δ_j) by ACES-C (Theorem 6) are mathematically sound under the paper’s explicit generative model: codes drawn i.i.d., class-conditional pass rates α_j, β_j homogeneous within each correctness class, and mutual independence of test outcomes given the labels. These are the same assumptions used by classical item-response theory and by the Hoeffding argument in Theorem 2; they are stated clearly, and the finite-sample empirical check in Figure 4 shows that the observable quality score still recovers the sign of ground-truth δ_j on real LLM pass matrices. Assumption 4 is the only additional fragility, but it is quantified (71–83 % of non-trivial tasks), its failure regime is isolated, and ACES-O is supplied precisely for that regime. No hidden inconsistency or circularity remains in the argument that converts leave-one-out consistency into a ranking-relevant weight.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper addresses the circular dependency in selecting LLM-generated code with LLM-generated tests: reliable tests are needed to judge code, and reliable code to judge tests. The key claim is that test value for ranking is discriminative power δ_j = α_j − β_j, not correctness per se. They define leave-one-out AUC (LOO-AUC) from the binary pass matrix alone, prove E[LOO-AUC_j(w)] − 1/2 = c_j(w)·δ_j (Theorem 3), and derive two methods: ACES-C (closed-form pass-rate-corrected weights under Assumption 4, with oracle-approximation guarantees) and ACES-O (iterative logistic-surrogate optimization of a LOO-AUC objective without that assumption). Both use only the pass matrix. Empirically they report state-of-the-art Pass@k among execution-only methods on HumanEval, HumanEval+, and MBPP, with further gains when combined with DS3 static analysis.","tokens_in":37907,"tokens_out":744,"duration_ms":16487,"significance":"If the results hold, this is a genuine advance: a first provable, pass-matrix-only criterion for separating informative from misleading tests, with closed-form and optimization variants that are cheap and complementary. The LOO-AUC identity is clean and useful beyond code generation (noisy evaluators, LLM-as-judge ensembles). Strengths include complete proofs (Appendix A), quantitative oracle approximation (Corollary 8), explicit isolation of Assumption 4 and its failure regime, thorough ablations (selection vs weighting, sensitivity to m′/n′/K, convergence, hyperparameters), and reproducible comparison on shared GPT-3.5-Turbo candidates/tests. The work is both theoretically grounded and practically deployable.","major_comments":[{"comment":"Section 2.2 and Theorem 3 rest on conditional independence of test outcomes given correctness labels. That is standard IRT-style modeling and is stated clearly, but LLM-generated tests are often template-correlated and share failure modes. The identity remains valid under the model, yet the practical reliability of LOO-AUC as a proxy for δ_j depends on how much residual dependence remains in real pass matrices. Please quantify pairwise column correlations (or conditional mutual information) on the experimental pass matrices and, if non-negligible, discuss how dependence would bias LOO-AUC or the recovered weights. The conclusion flags this as future work; a short empirical check in §4.3 or Appendix C would make the central claim more robust for practitioners.","section":null},{"comment":"Assumption 4 (δ̄ > 2√(ln 2/m)) is the only extra fragility for ACES-C (Proposition 5, Theorem 6). Table 7 and Figures 3/6–8 show it holds on 71–83% of non-trivial tasks and that the Hard region (low δ̄) is essentially unsolved by any pass-matrix method. That isolation is honest, but the main text still presents ACES as generally SOTA among execution-only methods. Please state more explicitly in §4.2–4.3 (and perhaps the abstract) the practical decision rule: when estimated average quality is near or below the threshold, prefer ACES-O or complementary static signals (DS3) rather than ACES-C alone. This is load-bearing for correct use of the closed-form guarantee.","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real novelty is Theorem 3: expected leave-one-out AUC excess is proportional to a test’s latent discriminative power δ_j, with a coefficient that depends only on pass-rate variance and leave-one-out ranking quality. That identity is what finally lets you weight tests for ranking without ever knowing which codes or tests are correct. Everything else—ACES-C’s closed-form pass-rate correction, ACES-O’s logistic surrogate, the Pass@k Hoeffding bound—follows from it.\n\nThe math is clean. The case decomposition of AUC under the usual conditional-independence model is standard and correctly applied; Corollary 8 gives a quantitative near-oracle SNR guarantee once Assumption 4 holds. Empirically they beat majority vote, CodeT, and MBR-exec on HumanEval / HumanEval+ / MBPP using only the pass matrix, and the combination with DS3’s static signals is the overall best. Figure 4 shows the observable quality score recovers the sign of ground-truth δ_j on real LLM matrices; the hard-task regime where Assumption 4 fails is isolated and ACES-O is supplied for it. Runtime is negligible.\n\nSoft spots are real but secondary. Assumption 4 (average δ̄ above a 1/√m threshold) is the load-bearing extra condition; it holds on 71–83 % of non-trivial tasks and the paper is honest about the rest. Conditional independence of test outcomes given labels is classical but still an idealization for LLM-generated suites. ACES-O has a few free knobs (γ, η, K), though the sensitivity plots look flat. None of these break the central claim.\n\nThis is for anyone running generate-many-then-select code pipelines or thinking about noisy-evaluator ensembles more generally. The theory is tight enough and the gains concrete enough that a serious editor should send it to referees. I would bring it to reading group and I expect to cite the LOO-AUC identity.","headline":"Clean LOO-AUC identity that turns the binary pass matrix into principled test weights; theory and Pass@k gains both look solid.","tokens_in":38508,"tokens_out":502,"would_cite":true,"duration_ms":7001,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Test votes should rank, not merely count: leave-one-out AUC identifies which generated tests distinguish correct code from incorrect code without knowing either label.","keywords":["code generation","LLM","test selection","leave-one-out AUC","Pass@k","weighted voting","discriminative power","binary pass matrix"],"falsifier":"On problems with known correct codes, compute each test’s true discriminative power; if the sign of LOO-AUC excess systematically disagrees with that sign whenever average discriminative power is positive, the central identity is false.","tokens_in":38564,"feed_emoji":"🧪","tokens_out":847,"duration_ms":17125,"temperature":0.7,"pith_summary":"Choosing among many LLM-generated programs by running them on LLM-generated tests is circular: you need good tests to judge code and good code to judge tests. This paper claims you never need to decide which tests are correct. A test is valuable for selection only if it separates correct programs from incorrect ones. By holding out each test, ranking candidates from the remaining tests, and measuring agreement with the held-out pass/fail column (leave-one-out AUC), the authors prove that this observable agreement is proportional to the test’s latent discriminative power. From that identity they build two lightweight weighting schemes that use only the binary pass matrix and raise Pass@k on standard code-generation benchmarks.","feed_headline":"Leave-one-out AUC scores which tests really rank code","feed_subtitle":"A consistency identity turns noisy generated tests into reliable votes without knowing which codes are correct.","key_machinery":"The LOO-AUC Identity (Theorem 3): expected leave-one-out AUC excess equals a positive coefficient times the test’s latent discriminative power. The coefficient depends only on pass-rate variance and leave-one-out ranking quality, so the identity converts an unobservable quality into a measurable consistency score that drives both closed-form and optimized weights.","core_discovery":"The expected leave-one-out AUC of a test, measured against the ranking induced by all other tests, is proportional to that test’s discriminative power—the difference in pass rates between correct and incorrect codes. Therefore tests can be weighted by an observable consistency score that recovers the sign of their true ranking value in expectation, breaking the circular dependency without external labels.","pith_inferences":["The same leave-one-out consistency score could reweight noisy generated verifiers in math or multi-step reasoning, not only code.","When LLM-generated tests are highly correlated, the independence assumptions behind the expectation identity loosen, suggesting a direct experiment on de-correlated test suites.","Pre-filtering candidates before optimization acts as a curriculum for the ranking objective; similar staged weighting may help other multi-annotator ranking problems."],"forward_implications":["Closed-form ACES-C weights recover the sign of each test’s discriminative power in expectation and approach oracle signal-to-noise under the average-quality condition.","Iterative ACES-O can still improve rankings when average test quality is low by co-evolving weights and the leave-one-out ranking.","Uniform majority voting is improved by reweighting tests with LOO-AUC consistency using nothing beyond the binary pass matrix.","Combining the same scores with complementary static-analysis signals further raises Pass@k beyond either signal alone.","The authors argue the same internal-consistency principle extends to other noisy-evaluator settings such as LLM-as-judge ensembles and process reward models."],"fun_headline_variants":["Leave-one-out AUC weights tests by true ranking power","ACES scores tests via LOO consistency without labels","Tests ranked by LOO-AUC separate correct from wrong code","Observable LOO-AUC recovers each test's discriminative value","Consistency via held-out AUC breaks circular test dependency"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"The closed-form guarantees need the average test to be slightly better than random at separating correct from incorrect code; if the whole pool is worse than chance on average, the simple correction no longer works.","fun_headline_variants_meta":{"raw":{"variants":["Leave-one-out AUC weights tests by true ranking power","ACES scores tests via LOO consistency without labels","Tests ranked by LOO-AUC separate correct from wrong code","Observable LOO-AUC recovers each test's discriminative value","Consistency via held-out AUC breaks circular test dependency"]},"model":"grok-4.5","effort":"low","cost_usd":0.00384,"raw_usage":{"total_tokens":1264,"prompt_tokens":840,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":38400000,"prompt_tokens_details":{"text_tokens":840,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":361,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":840,"tokens_out":63,"duration_ms":3143,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T19:52:48.471868+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On problems with known correct codes, compute each test’s true discriminative power; if the sign of LOO-AUC excess systematically disagrees with that sign whenever average discriminative power is positive, the central identity is false.","supporting_citations":[],"review_version":2}