{"id":"2d309a31-f3be-4e65-bde6-8d1750245e51","arxiv_id":"2607.02808","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Across 224 problems and twelve models, LLM-generated implementations show strongly correlated failures, so majority-vote N-version ensembles realize only about 0.43–0.44 of the reliability gain expected under independence.","lead":"LLM-generated code versions fail on the same tests far more often than chance, so N-version ensembles capture under half the reliability gain independence would allow. The paper supplies a reusable methodology and a large multi-model study that practitioners can use before betting on LLM redundancy.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Hypergeometric Z and redundancy-effectiveness both rest on test exchangeability; residual difficulty variation can inflate Z and deflate eff, overstating the measured independence deficit.","rationale":"The reader correctly flags the contest-problem proxy as the principal external-validity threat and therefore issues CONDITIONAL. That concern is real, yet the single most load-bearing threat to the numerical strongest claim itself is the exchangeability assumption that underpins both Z and redundancy effectiveness. Because the paper already supplies convergent qualitative evidence (shared root causes, high within-model structural similarity, absolute reliability gains of only +0.03), a moderate bias in the two statistics would not overturn the qualitative conclusion that independence fails. Consequently I leave the verdict and confidence unchanged while noting that the proposed hardness-weighted recomputation on the released artefacts would settle whether the quantitative deficit is overstated.","tokens_in":17936,"tokens_out":545,"duration_ms":51569,"concrete_test":"On the public dataset, for every problem recompute Eij under a hardness-weighted null: let h_t = fraction of retained solutions failing test t; obtain the expected overlap of two size-Fi and size-Fj subsets drawn without replacement with probability proportional to h_t (Monte-Carlo or weighted hypergeometric). Form new Z'ij. If mean Z' drops below 0.5 or the fraction of significant positive pairs falls below 30 %, the behavioural-diversity claim is materially weakened; otherwise the bias is negligible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest quantitative claims (mean Z ≈ 2.04 with 65.9 % of pairs significant; eff = 0.43/0.44) are produced by Eqs. (1)–(4). Both the hypergeometric expectation Eij = Fi Fj / T and the single average-Pmaj raised to the power T treat every test as exchangeable. Even after the non-edge/edge split, residual difficulty variation remains. Harder tests raise observed co-failure counts relative to the uniform null, generating positive Z for processes that are independent conditional on difficulty. Likewise [Pmaj_avg]^T overestimates the product of heterogeneous per-test majority probabilities, inflating headroom and thereby lowering reported eff. Manual root-cause inspection and within-model CodeBLEU similarity supply independent qualitative support, yet the precise numerical statements “far more often than expected” and “only 0.43 of the attainable gain” are sensitive to this modelling choice. The paper notes the issue in §V but never quantifies its magnitude.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes the first systematic methodology for assessing failure independence of LLM-generated code for N-Version Programming, combining structural diversity (CodeBLEU), behavioral diversity (pairwise co-failure Z-scores against a hypergeometric independence baseline), N-version reliability under majority voting with a redundancy-effectiveness metric, and manual root-cause inspection. Instantiated on 224 curated contest problems, twelve models, five languages, and three prompting strategies, the study finds that same-model implementations are highly similar and that failures co-occur far more often than independence predicts (mean Z ≈ 2.04 on non-edge tests; 65.9% of pairs significant). Three- and five-version ensembles realize only 0.43 and 0.44 of the reliability gain available under independence (below 0.3 for same-model ensembles). Manual analysis indicates that even distinct failure patterns often share root causes. The authors conclude that current LLM-generated solutions do not satisfy NVP’s independence assumption, though heterogeneous models help partially, and that the methodology can track this property as models evolve.","tokens_in":18238,"tokens_out":1688,"duration_ms":23651,"significance":"The work addresses a timely and practically important question: whether cheap multi-version generation with LLMs can revive NVP. Its main strengths are (i) a multi-dimensional methodology that goes beyond correctness and surface diversity to failure correlation and reliability gains, (ii) a large, carefully filtered empirical study with sensitivity checks on filtering thresholds and a public dataset of prompts, code, tests, and metrics, and (iii) triangulation via structural metrics, behavioral Z-scores, voting simulations, and manual fault inspection. If the independence deficit holds, the paper usefully constrains expectations for LLM-based redundancy and supplies a reusable evaluation tool. The redundancy-effectiveness metric is a clear conceptual contribution relative to reporting absolute reliability alone.","major_comments":[{"comment":"§III-B2, Eqs. (1)–(2) and §V Construct Validity: The headline behavioral claim (mean Z ≈ 2.04; 65.9% of pairs significant) rests on a hypergeometric null that assumes test exchangeability. Residual difficulty variation within the non-edge and edge strata can inflate co-failure counts relative to the uniform null and produce positive Z even under conditional independence. The paper acknowledges the issue but does not quantify its magnitude. A load-bearing revision is needed: e.g., difficulty-stratified or per-test nulls, a simulation under known difficulty heterogeneity, or bounds showing how large residual variation would have to be to erase the reported deviation. Without this, the precise numerical claim “far more often than expected under independence” is not fully secured.","section":"§III-B2, Eqs. (1)–(2); §V"},{"comment":"§III-B3, Eqs. (3)–(4) and Table II: Redundancy effectiveness uses a single average majority-pass probability raised to the power T under the same exchangeability assumption. Heterogeneous per-test pass rates make [Pmaj_avg]^T an overestimate of the product of true per-test majority probabilities, which inflates headroom and systematically lowers reported eff (0.43/0.44). Because these fractions are the abstract’s central quantitative result, the paper should either recompute headroom with a product over heterogeneous per-test majority probabilities (or a difficulty-aware model) or demonstrate that the bias is small relative to the gap from 1.0. The qualitative conclusion that gains are modest may survive; the specific 0.43/0.44 figures currently do not.","section":"§III-B3, Eqs. (3)–(4); Table II"},{"comment":"§III-C and §V External Validity: The transfer argument—that unambiguous contest problems with single correct behavior are a best-case proxy for discrete NVP components—is stated clearly but remains an untested axiom. Shared libraries, multi-correct behaviors, and specification ambiguity in real components could change correlation structure in either direction. At minimum, the paper should either (a) add a small non-contest case study (e.g., library-style or systems components with executable oracles) or (b) more sharply bound the claim to “well-specified single-behavior components” and avoid language that generalizes to NVP deployment without qualification. This is load-bearing for the practical implication that LLM-NVP is limited in the settings where NVP is used.","section":"§III-C; §V External Validity"}],"minor_comments":[{"comment":"§III-E Procedure step 4: typo “remaning solutions” → “remaining solutions”.","section":"§III-E"},{"comment":"Figure 3 and Figure 4 captions and body text would benefit from explicit statement of the CodeBLEU component weights (0.1, 0.1, 0.4, 0.4) near the figures, not only in §III-B1, so readers can interpret n-gram vs AST/dataflow contributions without flipping back.","section":"Fig. 3; Fig. 4; §III-B1"},{"comment":"Table II: clarify in the caption that absolute reliability for homogeneous model ensembles is influenced by the filtering step (strong models contribute more fully correct solutions), so hom-vs-het comparison should be read primarily through redundancy effectiveness—as noted in the text but easy to miss when reading the table alone.","section":"Table II"},{"comment":"§IV-D: the manual analysis covers ~650 implementations from 15 problems selected for extreme/intermediate behavioral similarity. State more explicitly how many distinct root-cause categories were coded and whether coding was dual-reviewed, to strengthen reproducibility of the fault-level claims.","section":"§IV-D"},{"comment":"Related work (§II): briefly contrast the proposed Z-score and redundancy-effectiveness metrics with Zheng et al.’s AST-similarity N-version performance study and with Galápagos, so the novelty claim is sharper for readers familiar with those lines.","section":"§II"},{"comment":"Dataset URL is given as a tinyurl; prefer a stable archival link (Zenodo/DOI or institutional repo) in the camera-ready version for long-term reproducibility.","section":"Abstract; §I"}],"recommendation":"major_revision","confidential_remarks":"The empirical design is careful and the multi-metric methodology is a genuine contribution; I would not reject. The exchangeability issue is the main reason I chose major_revision rather than minor: the abstract’s punchline numbers (Z and 0.43/0.44) are exactly the quantities most sensitive to that assumption, and the paper currently only notes the threat without bounding it. If the authors add a difficulty-aware sensitivity analysis and slightly tighten external-validity language, this should be a clear accept. Fit for a software engineering / dependability venue is strong."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is simple: this is the first systematic way to check whether LLM-generated multi-version code actually fails independently, and on a large multi-factor study the answer is no. Three- and five-version majority voting only capture about 0.43–0.44 of the reliability gain you would get under independence; same-model ensembles do worse. Heterogeneous models help a bit; language and prompting barely move the needle.\n\nWhat is new is the package, not any single metric. They combine CodeBLEU structural similarity, pairwise co-failure Z against a hypergeometric null, redundancy effectiveness under majority voting, and manual root-cause inspection. The design is large (224 filtered problems, twelve models, five languages, three prompts), they ship a public dataset, and they run sensitivity checks on the filtering thresholds. That is real work. The qualitative story is consistent: same-model outputs look alike, fail alike, and share root causes even when the surface failure patterns differ.\n\nSoft spots, in proportion. Contest problems with clean oracles are a best-case proxy for the discrete components where NVP is used; if correlation is already high there, it is unlikely to vanish in messier code, but transfer is still an assumption. The stress-test point on test exchangeability is fair: residual difficulty variation can inflate Z and deflate effectiveness, and they never quantify the bias. They do split edge vs non-edge and flag the issue in threats, and the manual fault analysis plus within-model structural similarity give independent support for the direction of the claim. So the precise “0.43 of attainable gain” is softer than the abstract suggests; the broader conclusion that current LLMs do not deliver classical NVP independence is not.\n\nThis is for people working on LLM reliability, multi-version ensembles, or evaluation methodology. It is not a theory paper and not a systems paper. I would send it to peer review; a serious referee can push on external validity and the exchangeability modeling without the paper collapsing. Worth engaging if you care about whether multi-LLM redundancy is more than theater.","headline":"First solid methodology for LLM failure independence; the correlation result holds directionally even if the exact Z/eff numbers are a bit soft on exchangeability.","tokens_in":18796,"tokens_out":519,"would_cite":true,"duration_ms":9871,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLM-generated code fails on the same tests far more often than independence would allow, so majority-vote ensembles capture under half the reliability gain N-version programming promises.","keywords":["N-Version Programming","Large Language Models","Code Generation","Failure Independence","Behavioral Diversity","Software Reliability","Majority Voting","Fault Correlation"],"falsifier":"Repeat the same protocol on a new set of models or on real library/system components whose specifications admit multiple correct behaviors; if three- and five-version ensembles then realize redundancy effectiveness near 1.0 (or at least well above 0.5) while the hypergeometric Z-scores drop to near zero, the independence deficit disappears and the central claim fails.","tokens_in":18857,"feed_emoji":"🔀","tokens_out":754,"duration_ms":7928,"temperature":0.7,"pith_summary":"N-version programming tries to raise reliability by running several independently written versions of the same program and taking a majority vote. Its classical bottleneck was cost; large language models now make generating many versions cheap, so the open question is whether those versions actually fail independently. This paper supplies the first systematic way to measure that independence for LLM-generated code and then applies it at scale: twelve models, five languages, three prompting styles, and 224 programming problems with carefully built test suites. The measurements show that solutions from the same model are nearly structural clones, that even solutions from different models still fail the same tests far more often than a hypergeometric independence baseline predicts, and that three- and five-version majority ensembles realize only about 0.43–0.44 of the reliability improvement independence would have delivered. Manual inspection further reveals that many seemingly different failure patterns share the same root cause. Heterogeneous models help a little; language and prompting barely matter. The result is both a concrete negative finding for today’s models and a reusable yardstick for checking whether future models ever become diverse enough for true N-version fault tolerance.","feed_headline":"LLM code fails together, so majority voting recovers little","feed_subtitle":"Three- and five-version ensembles capture under half the reliability gain independence would give.","key_machinery":"A three-layer measurement stack: (1) CodeBLEU structural similarity, (2) a hypergeometric Z-score that compares observed co-failures against the overlap expected under independence given each version’s failure count, and (3) redundancy effectiveness—the fraction of the theoretical majority-vote reliability gain actually realized. Manual root-cause inspection closes the loop by checking whether distinct failure patterns still share the same fault.","core_discovery":"Across 224 problems, LLM-generated implementations fail on the same test cases far more often than expected under statistical independence (mean Z ≈ 2.04; 65.9 % of pairs significantly correlated). Consequently, three- and five-version majority-vote ensembles realize only 0.43 and 0.44 of the reliability gain that would be available if failures were independent, and the fraction falls below 0.3 when all versions come from the same model. Even when failure signatures look different, manual fault analysis shows they frequently share the same underlying root cause.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM code fails on same tests far more than independence predicts","Majority voting recovers under half the gain if failures were independent","Same-model LLM ensembles realize under 0.3 of ideal reliability gain","Even diverse failure signatures often share root causes in LLM code","Heterogeneous models raise diversity yet still violate NVP independence"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That short, unambiguous contest-style programming problems with high-coverage unit tests are a fair proxy for the well-specified software components where N-version programming is actually used, so the measured failure correlation will transfer beyond competitive programming.","fun_headline_variants_meta":{"raw":{"variants":["LLM code fails on same tests far more than independence predicts","Majority voting recovers under half the gain if failures were independent","Same-model LLM ensembles realize under 0.3 of ideal reliability gain","Even diverse failure signatures often share root causes in LLM code","Heterogeneous models raise diversity yet still violate NVP independence"]},"model":"grok-4.5","effort":"low","cost_usd":0.003536,"raw_usage":{"total_tokens":1192,"prompt_tokens":856,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":35360000,"prompt_tokens_details":{"text_tokens":856,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":267,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":856,"tokens_out":69,"duration_ms":3342,"temperature":1.0,"reasoning_tokens":267,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T06:54:22.936695+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Repeat the same protocol on a new set of models or on real library/system components whose specifications admit multiple correct behaviors; if three- and five-version ensembles then realize redundancy effectiveness near 1.0 (or at least well above 0.5) while the hypergeometric Z-scores drop to near zero, the independence deficit disappears and the central claim fails.","supporting_citations":[],"review_version":1}