{"id":"6653c5ab-9e9c-473d-a1d1-8ea68d009294","arxiv_id":"2411.15587","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ConAIR iteratively corrects the most inconsistent LLM-generated test with user feedback, then repairs code, achieving large Pass@1 gains on HumanEval, HumanEval+, and MBPP in simulation.","lead":"ConAIR is a framework that asks a human to check and correct the one LLM-generated test most code candidates fail, then uses the corrected test to repair the code, iterating a few times. The authors report large Pass@1 gains over GPT-3.5, GPT-4o, and prior consistency methods, but the gains come mostly from simulated oracles rather than real users.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'fewest-passing-codes' test-selection heuristic is unvalidated; without an ablation against random selection, the 4-round user-effort claim is not established.","rationale":"The paper proposes ConAIR, an iterative interaction framework that asks users to correct selected LLM-generated tests and then fixes code accordingly. The reported gains are substantial and consistently positive across datasets and base models, and the user study provides some evidence that the interaction is lightweight. However, the central claim of 'minimal human effort' (about 4 rounds) depends critically on the test-ranking heuristic in Section 3.2 and Algorithm 1: selecting the test that the fewest code candidates pass, on the assumption that this test is the most likely to be erroneous. The paper does not provide any ablation or analysis validating this heuristic, such as comparing it with random test selection or with selecting the most-passed test. If the heuristic is no better than random, the co-evolution might still improve performance merely because any user-corrected test adds reliable information, and the number of rounds could be similar or higher. The internal inconsistency in Algorithm 1 line 3 (argmax vs argmin) further obscures the intended behavior. The paper's own Limitation section acknowledges that incorrect test corrections can mislead the model, and the o1 simulation demonstrates measurable sensitivity to oracle errors on MBPP, yet no analysis isolates the contribution of the ranking heuristic. Therefore, the reader's CONDITIONAL verdict is appropriate: the mechanism is plausible, but the user-effort savings are not yet established. Our proposed random-selection ablation would directly settle whether the heuristic is doing the claimed work, and would be a concrete step toward validating or qualifying the headline improvement figures.","tokens_in":19773,"tokens_out":6371,"duration_ms":55192,"concrete_test":"Run ConAIR_GT on HumanEval and MBPP with Algorithm 1 line 3 replaced by uniform random selection from T_unk (same seed, same ground-truth oracle, same code-fixing LLM and termination criteria). Compare average interaction rounds and final Pass@1 against the reported ConAIR_GT. If random selection achieves comparable Pass@1 with only marginally more rounds, the Con_{c->t} ranking heuristic is not load-bearing for the user-effort claim; if the heuristic produces a large reduction in rounds (e.g., >40%), the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5.1.2, Table 1) is that ConAIR improves GPT-3.5 by 32.9% with only about 4 user interactions. The mechanism that is supposed to keep user effort low is the consistency-voting ranking in Eq. (2)/Algorithm 1 line 3: the test passed by the fewest codes is assumed to be the most likely to be erroneous, and therefore the best one to ask the user to correct. This assumption is load-bearing but not validated anywhere in the paper. No experiment compares the chosen test with a random test, with the test passed by the most codes, or with asking the user to check all tests. If the heuristic is no better than random, the same final performance might be achievable with the same number of rounds by selecting tests arbitrarily, and the claimed advantage of 'consistency-augmented' interaction disappears. In fact, Algorithm 1 line 3 states 'argmax_t Con_{c->t}', which is the opposite of the text's 'select the most inconsistent test' (the test with the lowest Con_{c->t}); if implemented literally, the algorithm would ask users about the most-passed test, likely correct, wasting iterations. The paper's own Limitation states that 'performance depends on the quality of the corrected tests'; the o1 simulation shows that when the oracle makes mistakes (48.3% error rate on MBPP), the gains shrink (ConAIR_o1 on MBPP Pass@1 74.30 vs ConAIR_GT 83.90). This sensitivity makes it essential to know whether the ranking heuristic is actually identifying erroneous tests. Without an ablation, the '4-round minimal effort' claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ConAIR, an interactive code-generation framework that iteratively ranks LLM-generated tests by how many generated code candidates pass them, asks the user (or a simulator) to correct the most inconsistent test, and then uses the corrected test to repair code candidates. It reports experiments on HumanEval, HumanEval+, and MBPP with two simulated feedback variants (ground-truth-based and OpenAI o1-based), showing large Pass@1 improvements over GPT-3.5, GPT-4o, and prior consistency post-processors such as MPSC, with an average of about 4.5 interaction rounds. A small user study and a time/cost analysis are also included.","tokens_in":20099,"tokens_out":6015,"duration_ms":51616,"significance":"The central idea—improving consistency-based code generation by having the user validate only the most suspicious generated tests—is practical and potentially valuable. The method has no fitted parameters, the evaluation uses external ground truth, and the reported gains over GPT-3.5 and MPSC are large. If the results are reproducible, ConAIR would be a useful lightweight post-processing technique. However, the paper's main efficiency claim depends on an unvalidated test-selection heuristic, and the manuscript contains several internal inconsistencies in the algorithm description and in the reported numbers. These issues currently prevent the results from being fully trusted.","major_comments":[{"comment":"The ranking rule is specified inconsistently. Eq. (2) defines Con_c->t as the count of codes that pass a test, and Section 3.2 states that the lower this value, the more likely the test is incorrect and therefore the test should be selected for correction. Algorithm 1 line 3, however, sets t_w <- argmax_{t in T_unk} Con_c->t(t, C), which selects the test passed by the most codes—the opposite rule. Since this ranking is the mechanism claimed to keep user effort to about four rounds, the paper must state which rule was actually implemented and correct the algorithm/text mismatch.","section":"Section 3.2, Eq. (2); Section 3.3, Algorithm 1 line 3"},{"comment":"The central assumption that the test passed by the fewest codes is the most likely to be erroneous—and hence the best target for user correction—is never validated. No ablation compares the proposed selection with random test selection, with the most-passed test, or with asking the user to correct all tests. Without such a comparison, the reported four-round efficiency cannot be attributed to consistency-based ranking; random selection might achieve similar results. This ablation is needed to support the paper's key practical claim.","section":"Section 3.2; Section 5.1.2, Table 1"},{"comment":"The headline '32.9% improvement' is a relative improvement over the GPT-3.5 baseline, not an absolute gain. The absolute Pass@1 gains in Table 1 are +25.33, +21.25, and +17.10 on HumanEval, HumanEval+, and MBPP, with a mean of 21.2 percentage points; 32.9% only holds as a relative percentage. The abstract and introduction state '33% improvement' without this qualification. The paper should report both absolute and relative improvements to avoid ambiguity.","section":"Section 5.1.2, Table 1; Abstract; Introduction"},{"comment":"Some reported numbers are internally inconsistent. Table 5 lists ConAIR_o1 on MBPP with Pass@1 = 73.68, while Table 1 reports 74.30 for the same configuration. In addition, Section 5.1.2 credits ConAIR_o1 (GPT-4o-based) with raising HumanEval from 84.67 to 97.59, but Table 2 shows 97.59 is the ConAIR_GT result, while ConAIR_o1 reaches 94.96. These discrepancies must be reconciled, and the paper should clarify which subset or run each table reports.","section":"Section 5.3, Table 5; Section 5.1.2, Table 2"},{"comment":"No variance, error bars, or multiple independent runs are reported for any experiment. Because all numbers come from stochastic LLM sampling and iterative fixing, the differences of a few percentage points between ConAIR and baselines may fall within run-to-run noise. The authors should report repeated-run statistics or explicitly justify and clearly state the single-run nature of the evaluation.","section":"Section 5.1; Tables 1 and 2"}],"minor_comments":[{"comment":"The algorithm header swaps the variable names: it says 'Input: test case set C; code set T', while the text and the rest of the algorithm use T for tests and C for codes.","section":"Section 3.3, Algorithm 1 header"},{"comment":"The MBPP column header reads 'Pass@1 Pass@1 Pass@5'; it should presumably be 'Pass@1 Pass@2 Pass@5'.","section":"Section 5.1.2, Table 2"},{"comment":"The user study uses only 6 participants and 20 problems, and the reported Pass@1 for ConAIR and Writing Code are identical (90 vs 90). The Wilcoxon tests are applied only to time, not to correctness; the conclusion that ConAIR is more efficient than writing code should be framed accordingly.","section":"Section 5.2.1 and 5.2.2"},{"comment":"The text says that 'the results of ConAIR_o1 on HumanEval and HumanEval+ are similar,' but Table 1 shows Pass@1 values of 92.45 and 78.03 for these two datasets, which are not similar; this appears to be a misstatement.","section":"Section 5.1.2"},{"comment":"The abstract and introduction say the average number of interaction rounds is 4, while Table 3 reports an average of 4.53 across the three benchmarks; please reconcile the wording.","section":"Abstract; Section 5.2.2, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real limitation of consistency-based code generation and reports promising empirical results, but the unvalidated test-selection heuristic and the algorithm-text contradiction are load-bearing issues. The ablation of the selection rule and reconciliation of the reported numbers are essential before the main efficiency claim can be accepted. The paper is within scope for the venue, and the limitations section is candid; the remaining concerns are fixable with additional experiments and clarifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this paper is worth your attention, but with the brakes on. The genuinely new piece is the rank-correct-fix loop: ConAIR iteratively picks a test, asks the user to confirm or correct it, then repairs code against that corrected test, making both tests and code more reliable. That combination doesn't appear in CodeT or MPSC, the closest baselines. The paper also documents that LLM-generated tests are wrong 37.7% of the time, and shows large empirical gains — e.g., GPT-3.5-based ConAIR beats GPT-4o on HumanEval. The simulated experiments are extensive and honestly reported.\n\nThe soft spots are real, and one is structural. The claim that only ~4 user rounds are needed rests on the consistency heuristic that the test passed by the fewest code candidates is the most likely to be erroneous. No ablation compares this with random selection, with the most-passed test, or with asking the user to check more tests. If the heuristic is no better than random, the 'minimal human effort' selling point collapses. The paper's own limitation section admits performance depends on corrected test quality, and the o1 simulation on MBPP (48.3% error rate) shows gains shrink from +17.10 to +7.50 points. So the heuristic needs validation.\n\nAlso, Algorithm 1 line 3 says 'argmax Con_{c->t}', which is the opposite of the prose's 'select the most inconsistent test' (that would be argmin). If implemented literally, the algorithm selects the most-passed test, so the pseudocode contradicts the method. That will confuse any implementer and must be fixed.\n\nMinor issues: the headline 32.9% improvement is relative, not absolute; Table 1's deltas are percentage points, and the abstract doesn't say which. The user study has 6 participants and 20 problems, so it's suggestive, not strong. No artifacts are released, which hurts reproducibility.\n\nThat said, the central idea is sound and well-motivated, and the evidence is considerably better than average for an arxiv tech report. The paper deserves serious peer review, but it should not be accepted as-is. The referee should ask for: (1) a correction of the pseudocode, (2) an ablation of the ranking heuristic against random and worst-case selection, (3) explicit statement of relative vs absolute gains, and (4) ideally a release of code and prompts. With those, this could be a solid contribution to the human-in-the-loop code generation literature.","headline":"A genuinely new human-in-the-loop code-repair loop, but the load-bearing test-ranking heuristic goes unablated and the pseudocode has an argmax/argmin bug.","tokens_in":20625,"tokens_out":3401,"would_cite":false,"duration_ms":28352,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Consistency voting over LLM-generated tests is unreliable, so ConAIR has users correct the most suspicious test and iteratively co-evolves tests and code, lifting GPT-3.5's code-generation accuracy by 32.9% in about four interaction rounds.","keywords":["code generation","large language models","consistency voting","LLM-generated tests","human-in-the-loop","iterative refinement","co-evolution of code and tests"],"falsifier":"Re-run ConAIR on HumanEval with the ranking rule reversed — have the user correct the test that the most programs pass first instead of the least-passed one. If Pass@1 does not drop, the least-passed-test heuristic is not what drives the gain; similarly, if measuring the error rate of the tests selected by $Con_{c\\to t}$ shows they are no more likely to be wrong than the average generated test, the ranking mechanism is not doing the claimed localization.","tokens_in":19588,"feed_emoji":"🔧","tokens_out":8748,"duration_ms":71776,"temperature":0.7,"pith_summary":"Code-generation models are usually judged by consistency: generate many candidate programs and many tests, then pick the program that passes the most tests and shares behavior with the most peers. The paper's central observation is that the tests are produced by the same unreliable model, so this vote is built on a flawed indicator; the authors measure an average 37.7% error rate in LLM-generated tests on HumanEval, HumanEval+, and MBPP. ConAIR therefore inserts a cheap human step into the voting loop: in each round it uses code-to-test consistency voting to pick the test that the fewest candidates pass, asks the user to correct that test, repairs the candidates that fail it, and repeats. With an average of about four correction rounds, the framework lifts GPT-3.5's Pass@1 from 68.38% to 93.71% on HumanEval, outperforming both the previous consistency-based post-processor MPSC and the stronger general model GPT-4o. The claim is that consistency is valuable only when the indicator is trustworthy, and that a small amount of targeted user feedback can make it trustworthy enough to drive large reliability gains.","feed_headline":"Four human fixes lift LLM code accuracy by 33 percent","feed_subtitle":"The trick: use consistency voting to find the buggiest LLM-written test, have a human correct it, then repair the code around it.","key_machinery":"The load-bearing object is a pair of opposite consistency votes. $Con_{c\\to t}(t, C) = \\sum_{c} Con(c,t)$ counts how many candidate programs satisfy a test, and the test with the lowest count is treated as the most likely wrong test and sent to the user for correction; $Con_{t\\to c}$ checks which programs satisfy all corrected tests and selects the final output. The iterative rank-correct-fix algorithm (Algorithm 1) alternates these votes: correct one test per round, discard or repair programs that fail it, re-run the surviving programs against the remaining unknown tests, and stop when a program passes every test. The mechanism works by making both indicators more reliable over time, since better tests make code-to-test voting trustworthy and better programs make test-to-code voting trustworthy.","core_discovery":"ConAIR establishes that the failure mode of consistency-based code generation is not the voting mechanism but the consistency indicator. Because tests generated by the same LLM carry a high error rate (37.7% on average across HumanEval, HumanEval+, and MBPP), a buggy program can pass more generated tests than a correct one and therefore win the majority vote. The proposed solution is a rank-correct-fix co-evolution loop: rank tests by how many candidate programs pass them (code-to-test consistency, $Con_{c\\to t}$), select the least-passed test as the most likely erroneous, have a user correct that test's expected output, then use the same LLM to repair every candidate that fails the corrected test. The loop terminates when a program passes all corrected tests or all tests have been corrected. The paper reports that this converges in an average of 4.53 interaction rounds and, built on GPT-3.5, improves Pass@1 by an average of 32.9% over the base model, 11.1% over MPSC, and 12.32% over GPT-4o, with further gains when the base model is GPT-4o or o1.","pith_inferences":["The rank-correct-fix loop is a generic recipe for any domain where an LLM generates both candidate answers and its own verifiers (assertions, unit tests, formal specifications), not just code; the same least-trusted-verifier heuristic could select what a human should check.","Because the paper attributes the gain to correcting tests rather than to the initial code candidates, a testable corollary is that improving the test-generation step alone, without the code-repair step, would capture a large share of the improvement; the authors do not run this ablation.","The 37.7% test error rate suggests that pre-filtering generated tests by cross-model agreement before any human round could reduce the number of interaction rounds below four, which the paper does not explore.","ConAIR-o1's near-parity with ConAIR-GT on HumanEval indicates that a strong reasoning model can stand in for the human on well-specified problems, but its 48.3% error rate on MBPP warns that on under-specified problems the surrogate oracle itself needs the same kind of scrutiny."],"forward_implications":["A weak base generator (GPT-3.5) plus a few corrected tests can beat a much stronger generator (GPT-4o) on HumanEval, HumanEval+, and MBPP, so test-quality correction can substitute for raw model scale.","The same loop generalizes to stronger base models: building ConAIR on GPT-4o lifts HumanEval Pass@1 from 84.67 to 97.59 with ground-truth-simulated user feedback, and building it on o1 reaches 100% Pass@1 on HumanEval.","User effort stays small in practice: an average of 4.53 interaction rounds across the three benchmarks, and a user study in which participants spent less time and rated the test-fixing setting easier than writing or fixing code directly.","The framework is cheap enough to be practical: ConAIR-o1 on HumanEval costs about $0.56 and 2.26 minutes per problem, roughly one-tenth the time and cost of using o1 alone."],"supporting_citations":[{"why":"Defines the CodeT consistency approach that selects code by inter- and intra-consistency over LLM-generated tests; this is the main method ConAIR extends and improves.","marker":"[4]"},{"why":"MPSC is the state-of-the-art post-processing baseline with multi-perspective self-consistency; supplies the comparison setup and the 11.1% improvement target.","marker":"[12]"},{"why":"HumanEval is one of the three evaluation benchmarks and the source of the motivating examples.","marker":"[5]"},{"why":"HumanEval+ supplies stricter additional tests for HumanEval and is used to measure robustness.","marker":"[18]"},{"why":"MBPP is the third evaluation benchmark and drives the higher interaction-round counts.","marker":"[3]"},{"why":"Self-Consistency is the foundational majority-voting technique that motivates the consistency framework.","marker":"[36]"},{"why":"Supplies the theoretical premise that multiple reasoning paths converge on a correct answer, which underlies consistency voting.","marker":"[30]"},{"why":"The o1-preview model simulates user feedback in the automated variant ConAIR-o1.","marker":"[25]"}],"fun_headline_variants":["Fix the LLM's faulty tests to lift code accuracy 33%","Human-corrected tests: 33% more reliable LLM code output","Four user check-ins on LLM tests yield 33% code gain","Rank, correct, repair: ConAIR lifts code accuracy 33%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop depends on the heuristic that the test passed by the fewest candidate programs is the one most likely to be erroneous, and on the assumption that a human (or surrogate) can correct that test accurately and cheaply; if the suspicious test is actually correct or the correction is wrong, the loop can discard good programs or propagate bad outputs.","fun_headline_variants_meta":{"raw":{"variants":["Fix the LLM's faulty tests to lift code accuracy 33%","Human-corrected tests: 33% more reliable LLM code output","Four user check-ins on LLM tests yield 33% code gain","Rank, correct, repair: ConAIR lifts code accuracy 33%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001098,"raw_usage":{"total_tokens":4666,"prompt_tokens":1111,"completion_tokens":3555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":727,"completion_tokens_details":{"reasoning_tokens":3475}},"tokens_in":727,"tokens_out":3555,"duration_ms":22719,"temperature":1.0,"reasoning_tokens":3475,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:07:26.553229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run ConAIR on HumanEval with the ranking rule reversed — have the user correct the test that the most programs pass first instead of the least-passed one. If Pass@1 does not drop, the least-passed-test heuristic is not what drives the gain; similarly, if measuring the error rate of the tests selected by $Con_{c\\to t}$ shows they are no more likely to be wrong than the average generated test, the ranking mechanism is not doing the claimed localization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CodeT consistency approach that selects code by inter- and intra-consistency over LLM-generated tests; this is the main method ConAIR extends and improves."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HumanEval+ supplies stricter additional tests for HumanEval and is used to measure robustness."},{"cited_title":"Le, Ed H","cited_arxiv_id":null,"evidence_quote":"Self-Consistency is the foundational majority-voting technique that motivates the consistency framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical premise that multiple reasoning paths converge on a correct answer, which underlies consistency voting."},{"cited_title":"GIntroducing OpenAI o1-preview","cited_arxiv_id":null,"evidence_quote":"The o1-preview model simulates user feedback in the automated variant ConAIR-o1."}],"review_version":1}