{"id":"ceb1034d-d181-4646-8b3e-3cb930635193","arxiv_id":"2506.11021","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Functional clustering uses self-generated tests to group LLM code by exact output behavior, and abstains when the largest group is too small, trading answer coverage for lower error rates.","lead":"Functional clustering wraps a code LLM in a filter: sample many candidate programs, run them on test inputs the model writes itself, and only answer when a large group of candidates produces identical outputs. This trades coverage for reliability, and the paper reports a 65% to 2% error reduction on LiveCodeBench, but the threshold that achieves this is tuned on the benchmark itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core guarantee is about sample consensus under a self-generated test oracle, not correctness on hidden tests; the strongest error-rate claims are in-sample and not covered by the stated bound.","rationale":"Functional clustering is a reasonable and clearly described wrapper idea, and the paper is honest about several limitations (multi-output tasks, time limits, sandboxing). The Chernoff derivation in Appendix A is internally correct as a statement about Bernoulli tails. But the central claim of the abstract—'exact confidence estimate' and 'exponential guarantees'—requires the test-based equivalence relation to be a faithful proxy for correctness, and that is precisely the assumption the paper flags as a limitation. The concern is not disagreement with current consensus; it is a mismatch between the mathematical object being bounded (mass of an equivalence class under the LLM's own sampling distribution) and the object the user cares about (correctness on the benchmark's hidden test distribution). The empirical results may still be useful as a heuristic selective-coder baseline, and the manual audits add qualitative evidence, but they do not establish the claimed guarantee. The reader's reject verdict is therefore appropriate; no change to the verdict is needed.","tokens_in":16163,"tokens_out":5097,"duration_ms":56684,"concrete_test":"Re-run the LiveCodeBench pipeline with test inputs generated by an independent source (e.g., a random input sampler or a different model family such as Claude) instead of GPT-4o, and with the acceptance thresholds fitted on a random half of the tasks and evaluated on the held-out half. Compare dominant-cluster accuracy and the error rates at the fitted tau0% and tau2% thresholds against the paper's numbers. If error rates at fixed thresholds increase materially or the zero-error operating point disappears, the confidence estimate and the 'exponential guarantee' are artifacts of shared generator-candidate misinterpretation and in-sample threshold tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the empirical mass of the largest cluster is an exact confidence estimate with exponential reliability guarantees (Abstract; Eq. 5). The load-bearing step is substituting test-based equivalence (Eq. 6) for universal equivalence (Eq. 1) while keeping the language of Eq. 2 and Eq. 5. Eq. 5 bounds the probability that the empirical mass of a fixed equivalence class exceeds tau when its true mass C is below tau. However, Algorithm 1 selects the largest empirical class after seeing the data, and C is the mass of a class under the LLM sampling distribution, not the probability that a member of that class is correct on the benchmark's hidden tests. The test inputs in Eq. 7 are drawn from a task-dependent distribution D that in Section 4 is generated by GPT-4o from the same prompt that produced the candidate programs; a systematic misreading of the spec contaminates both tests and programs, so the largest cluster can be confidently wrong. The paper's Limitations section concedes that behavioral agreement on a finite, auto-generated test set is assumed to be a faithful proxy for universal correctness, and the residual-error categories HC and HA show this failure mode is real. Therefore the headline 'error rate from ~65% to 2%, and 0% at a conservative threshold' is an in-sample description of 219 tasks with thresholds chosen after seeing the results; the stated exponential guarantee does not attach to the measured object.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a black-box post-processing wrapper for LLM code generation. For each prompt, it samples n candidate programs, generates m test inputs using the same LLM, executes all candidate programs on all inputs, and clusters programs by identical output vectors. The empirical fraction of samples in the largest cluster, denoted rho, is used as a confidence estimate; the wrapper returns a representative of the largest cluster if rho >= tau and abstains otherwise. The authors derive a Chernoff bound (Eq. 5) for the probability that the empirical mass of a fixed equivalence class exceeds tau when its true mass is below tau, and report experiments on HumanEval and LiveCodeBench claiming error rates reduce from roughly 65% to 2% and to 0% at a conservative threshold.","tokens_in":16620,"tokens_out":5469,"duration_ms":49405,"significance":"The problem addressed is important and the method is simple, black-box, and potentially applicable to closed APIs. The paper ships code and reproduces its prompts in an appendix, which is good practice. However, the central theoretical guarantee does not apply to the decision rule actually used (selection of the largest class), and the test oracle is generated by the same model family as the candidate programs, so a systematic misinterpretation of the specification can produce a large, confidently wrong cluster. The headline zero-error result is obtained by tuning the threshold on the 219-task test set, so it is an in-sample description rather than a certified or predicted error rate. The method may have heuristic value as a consensus filter, but the paper does not substantiate the claimed exponential reliability guarantees for the measured object.","major_comments":[{"comment":"The Chernoff bound in Eq. (5) bounds the tail probability for a fixed equivalence class with true probability mass C under the sampling distribution P. Algorithm 1, however, selects the largest empirical class after seeing the data. Because the number of classes can be large and the class identity is data-dependent, the probability that the selected class has empirical mass at least tau while its true mass is below tau is not controlled by Eq. (5). The paper does not provide a union bound or any argument that the selection step preserves the exponential guarantee, so the claimed reliability guarantee does not attach to the rule that is evaluated.","section":"Section 3.3, Eq. (5); Algorithm 1"},{"comment":"The test inputs in Eq. (7) are drawn from a task-dependent distribution D, and Section 4 states that the self-generated test suites are produced by GPT-4o alone, the same model family that generates the candidate programs. If the model systematically misinterprets the prompt, its test inputs share that misinterpretation, so programs in the largest equivalence class can all implement the same wrong specification. The paper's own Limitations section concedes that behavioral agreement on a finite, auto-generated test set is assumed to be a faithful proxy for universal correctness, and the HC and HA error categories in Table 2 are direct instances of this failure mode. Eq. (7) requires D(B) = delta > 0 for the bug set B that matters, but no evidence is given that the self-generated distribution assigns positive measure to the hidden-test bug set.","section":"Section 3.4, Eq. (6)-(7); Section 4"},{"comment":"The thresholds tau0%, tau1%, and tau2% are defined in Table 2 as thresholds tuned so that the returned-answer error rate does not exceed 0%, 1%, or 2% on the same 219 LiveCodeBench tasks. Consequently, the abstract's claim that the error rate drives it to 0% at a conservative threshold while still answering 15.6% of prompts is an in-sample statement obtained by fitting tau to the test set. No held-out split, cross-validation, or separate calibration set is reported, so the result does not support a general prediction of zero error at a fixed threshold.","section":"Section 4.2, Table 2"},{"comment":"The claim that the accuracy-coverage trade-off follows the exponential tail predicted by Eq. 5 is not quantitatively supported. Eq. (5) is an upper bound on a tail probability as a function of n and the gap tau - C, whereas Figure 1b reports observed correctness rates on 219 tasks with a single n = 100 and no confidence intervals. A quantitative comparison, for example fitting the predicted curve or reporting error bars, would be needed to substantiate this statement.","section":"Section 4.2, Figure 1b"}],"minor_comments":[{"comment":"The formalism assumes programs are deterministic functions f_phi: X -> Y, but many LLM-generated programs use randomness (for example, random.choice or hash-based iteration); the paper does not explain how output vectors are defined for such programs.","section":"Section 3.1"},{"comment":"The abstract claims that the verifier preserves baseline pass@1 on solvable tasks, but Table 2 reports 'Expected' accuracy rather than pass@1, so the reader cannot verify that pass@1 is preserved; please clarify the metric and provide the corresponding comparison.","section":"Abstract and Section 4"},{"comment":"The figure caption describes cumulative wrong/correct step curves but does not define how they aggregate the 100 samples per task or how the y-axis is normalized; adding this detail would improve interpretability.","section":"Figure 1"},{"comment":"The prompt for test-case generation asks GPT-4o for a comprehensive list of valid input test cases, but there is no check on the diversity or validity of these inputs; a mis-parsed or degenerate test input could make all programs fall into one cluster trivially, and this risk is not discussed.","section":"Appendix B"}],"recommendation":"reject","confidential_remarks":"The paper addresses a timely problem and the experimental wrapper is easy to implement, but the central claims of exact confidence estimation and exponential reliability are not supported by the analysis or the evaluation protocol. The issues are structural: the guarantee is derived for a different decision rule than the one evaluated, and the test oracle is not independent of the generator. A revision that reframes the method as an empirical consensus heuristic and adds proper out-of-sample threshold selection would be a different, more modest paper; in the current form, the claims overstate what is established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes an attractive black-box wrapper: sample many candidate programs and test inputs, run them in a sandbox, cluster by exact I/O behavior, and treat the size of the dominant cluster as a confidence score, abstaining below a threshold. That idea is clean, the experiments are honestly reported, and the manual failure analysis is useful. The authors have found a real practical lever.\n\nThe problem is the theory. Eq. 5 is a correct Chernoff bound, but it bounds the probability that the empirical mass of a fixed true functional-equivalence class exceeds tau when the true class probability C is below tau. The measured rho is the mass of a cluster defined by self-generated tests, not by universal equivalence. Eq. 7 connects test-based agreement to a distribution D that is never shown to match the benchmark's hidden tests. So the 'exact confidence' and 'exponential guarantees' in the abstract do not attach to the empirical object. On top of that, Algorithm 1 selects the largest cluster after seeing the data; the bound for a fixed class does not directly cover that selection.\n\nThe circularity is real. The test inputs come from GPT-4o, the same model family that wrote the candidates. When the model misreads the prompt, its tests share the misreading, so a large cluster can be confidently wrong. The paper's own residual categories HC and HA demonstrate exactly this. The limitations section acknowledges the proxy assumption, but the abstract overstates what is delivered.\n\nThe empirical error rates are in-sample. The tau2% and tau0% thresholds are fit on LiveCodeBench, so the reported 2% and 0% errors describe the benchmark that was used to pick the threshold, not a prediction. \"Guaranteeing zero observed errors\" is not a guarantee.\n\nOne more omission: the paper never cites CodeT, which is essentially the same recipe—sample code, generate tests, select by execution consensus. That needs to be addressed.\n\nDespite these flaws, the wrapper is a legitimate contribution. The paper deserves peer review because the core idea has value and the problems are correctable: reframe the claims as empirical, compare directly against CodeT, and validate thresholds on held-out tasks. I would not cite it in its current form, but I would encourage the authors to revise. If you work on code generation or uncertainty quantification, it is worth a skim to see the failure mode; it could also serve in a reading group as an instructive example of overclaiming.","headline":"A clean empirical wrapper for filtering LLM code output, but the headline 'exact confidence' and exponential guarantees do not survive contact with the actual test-based oracle and in-sample threshold tuning.","tokens_in":16956,"tokens_out":3947,"would_cite":false,"duration_ms":40422,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Functional clustering turns LLM sampling noise into a quantified confidence score and cuts returned-answer error from ~65% to 2% (and to 0% at a conservative threshold).","keywords":["functional clustering","LLM code generation","hallucination detection","confidence estimation","selective abstention","behavioral equivalence","test-based oracle","Chernoff bound"],"falsifier":"Construct a task whose failure mode lies outside the input generator's reach—for example, a function with a subtle error on large integer arguments while the generated tests use only small integers. If the dominant cluster passes the self-generated suite, exceeds the acceptance threshold, and still fails hidden tests, the faithfulness premise is violated. A reader could also scan LiveCodeBench output for any dominant-cluster failure that is not a prompt misinterpretation but a genuine algorithmic bug shared by all samples.","tokens_in":15990,"feed_emoji":"⚙️","tokens_out":5487,"duration_ms":52266,"temperature":0.7,"pith_summary":"Coding LLMs still produce plausible but subtly wrong programs, so autonomous deployment is unsafe. This paper proposes a black-box wrapper: sample many candidate programs, execute each on automatically generated test inputs, and group those with identical outputs. The size of the largest group becomes a confidence score, and a single threshold on that score lets the wrapper answer only when confident and abstain otherwise. On LiveCodeBench the wrapper reportedly preserves baseline pass@1 while cutting the error rate of returned answers from roughly 65% to 2%, and reaches 0% at a more conservative threshold while still answering 15.6% of prompts. Manual audits attribute the few residual failures to misreading the prompt rather than to random generation noise.","feed_headline":"Grouping identical outputs cuts LLM code errors from 65% to 2%","feed_subtitle":"A black-box verifier clusters programs by identical test behavior, then abstains unless a cluster is large enough.","key_machinery":"The central object is the functional equivalence class: two programs are equivalent when their output vectors over the generated test inputs match exactly. The machinery is the routine that samples $n$ programs, runs each on $m$ generated inputs, and clusters them by these vectors, computing $\\rho = s_{\\max}/n$, the empirical mass of the largest class. This statistic is the confidence score, and the abstention rule is $\\rho \\ge \\tau$. What carries the argument is the concentration inequality in Eq. 5, which bounds the probability of accepting a truly low-mass class by an exponentially decaying tail in $n$, and Eq. 7, which says a real behavioral difference is missed only if none of the $m$ random test inputs happen to hit the differing region.","core_discovery":"The central claim is that functional equivalence classes—programs returning identical outputs on an auto-generated test suite—concentrate the probability mass of correct solutions, while hallucinated bugs scatter mass across many small classes. The paper uses the empirical mass $\\hat{C}_n$ of the largest cluster as the confidence estimate and returns a representative program only when $\\hat{C}_n \\ge \\tau$. It proves a Chernoff-style bound, $\\Pr[\\hat{C}_n \\ge \\tau \\mid C] \\le \\exp(-n D_{\\mathrm{KL}}(\\tau\\|C))$, on the chance of accepting a cluster whose true mass is below threshold, giving an exponential reliability guarantee. The authors claim this turns any black-box LLM into a selective coder: on LiveCodeBench, error on returned answers drops from ~65% to 2%, and to 0% at a conservative threshold while still solving 15.6% of benchmark tasks, with HumanEval pass@1 preserved.","pith_inferences":["If this result holds, token-level likelihood or embedding-distance confidence scores become less necessary for code: behavioral clustering captures probability mass that syntactic variants split, which could improve any downstream abstention or re-ranking system.","The faithfulness of the confidence score depends on the input generator's coverage; pairing the method with coverage-guided or mutation-based test generation could reveal blind spots that the current self-generated suites miss.","For tasks that admit several valid outputs—permutations, tie-breaking, regex matches—the single-class assumption excludes them by design; an extension that clusters under an equivalence relation appropriate to the task could recover them.","Across models, one model's dominant cluster can be wrong while another model's cluster disagrees; using cross-model disagreement as an additional abstention signal might reduce the residual prompt-misinterpretation errors the paper observes."],"forward_implications":["Because the wrapper needs only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models, without logits, fine-tuning, or auxiliary classifiers.","A user can trade coverage for reliability with one scalar threshold; doubling the sample size roughly squares the false-acceptance bound from Eq. 5.","On LiveCodeBench, thresholding at 2% residual error answers 36.8% of tasks while preserving baseline pass@1 on solvable tasks, and a conservative threshold answers 15.6% with zero observed errors.","Manual audits attribute residual failures to prompt misinterpretation; rewriting four ambiguous HumanEval specifications turned all four into high-confidence correct solutions.","The same test-based oracle can validate optimized rewrites or ensemble outputs, because equivalence is checked behaviorally rather than syntactically."],"supporting_citations":[{"why":"LiveCodeBench dataset: supplies the production-style benchmark and hidden-test evaluation behind the paper's main reliability numbers.","marker":"[6]"},{"why":"HumanEval dataset: the micro-level benchmark whose pass@1 preservation is reported.","marker":"[2]"},{"why":"GPT-4o: the primary model whose sampling distribution defines the candidate programs and equivalence classes; the system card is cited as the model source.","marker":"[14]"},{"why":"Claude 3 Haiku: the second model used to show that the thresholding behavior generalizes across heterogeneous sampling distributions.","marker":"[1]"},{"why":"Semantic entropy: the embedding-based uncertainty baseline the paper contrasts with exact behavioral clustering.","marker":"[4]"},{"why":"Symbolic-trace clustering: the prior trace-based method whose sensitivity to control-flow changes motivates the coarser output-vector equivalence.","marker":"[17]"},{"why":"Conformal abstention: the selective-answer framework into which functional clustering plugs as the underlying score.","marker":"[24]"}],"fun_headline_variants":["Cluster by test behavior to cut LLM code errors from 65% to 2%","Black-box clustering eliminates 97% of LLM code hallucinations","Cut LLM coding errors to 2% with functional clustering","Behavioral clustering cuts LLM errors to 2%","Functional clustering: from 65% code error to 2% reliably"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that agreement on the finite, automatically generated test-input set is a faithful stand-in for universal correctness: if the LLM's own input generator systematically misses the inputs that expose a bug, all sampled programs can agree, the largest cluster can be confidently wrong, and the confidence score will not reveal it.","fun_headline_variants_meta":{"raw":{"variants":["Cluster by test behavior to cut LLM code errors from 65% to 2%","Black-box clustering eliminates 97% of LLM code hallucinations","Cut LLM coding errors to 2% with functional clustering","Behavioral clustering cuts LLM errors to 2%","Functional clustering: from 65% code error to 2% reliably"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000809,"raw_usage":{"total_tokens":3564,"prompt_tokens":971,"completion_tokens":2593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2514}},"tokens_in":587,"tokens_out":2593,"duration_ms":15705,"temperature":1.0,"reasoning_tokens":2514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:51:48.296872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a task whose failure mode lies outside the input generator's reach—for example, a function with a subtle error on large integer arguments while the generated tests use only small integers. If the dominant cluster passes the self-generated suite, exceeds the acceptance threshold, and still fails hidden tests, the faithfulness premise is violated. A reader could also scan LiveCodeBench output for any dominant-cluster failure that is not a prompt misinterpretation but a genuine algorithmic bug shared by all samples.","supporting_citations":[{"cited_title":"The claude 3 model family: Opus, sonnet, haiku, March 2024","cited_arxiv_id":null,"evidence_quote":"Claude 3 Haiku: the second model used to show that the thresholding behavior generalizes across heterogeneous sampling distributions."}],"review_version":1}