{"id":"f32825e5-9241-4e9f-9635-e5fd88fa65b0","arxiv_id":"2502.18473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLM-generated probes with execution feedback find counterexamples that unit tests miss, and semantic clustering based on these probes improves code-generation evaluation.","lead":"This paper introduces ProbeGen, a method that uses an LLM to generate small input examples (\"probes\") that make two code implementations return different outputs, thereby disproving their equivalence. It reports that about 18% of code samples that pass benchmark unit tests are actually not equivalent to the ground truth solution, and it shows the approach can improve pass@1 by clustering semantically similar solutions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 18.7% headline depends on a non-random, unquantified human-validated filter-precision estimate; if precision is optimistic, the central 'unit tests miss significant differences' claim is materially weakened.","rationale":"I agree with the reader that the spurious-counterexample filter is the weakest load-bearing assumption. The central quantitative claim is the 18.7%/18% figure: it is the headline result and the basis for the abstract's statement that unit tests miss a large fraction of significant semantic differences. All quantitative claims about 'significant' differences pass through the LLM filter, whose precision is measured on a small, non-random, author-annotated subset with no reported N or confidence interval, and whose recall is never measured. The calculation 24.3% x 0.771 also assumes the filter-accepted set is exactly the set of counterexamples, while filtered-out examples could include true positives, so the estimate is not a bound. The paper's own limitation section is honest about many issues but does not flag this uncertainty. Because the strict functional counterexample rate (43.1%, before significance filtering) is deterministic and execution-backed, the core insight of ProbeGen survives even if the filter is imperfect; however, the specific '18% of samples considered equivalent by unit tests are actually different' headline is not supported with adequate rigor. This justifies the reader's CONDITIONAL verdict: the method is sound and interesting, but the headline number needs either artifact release, a proper statistical treatment, or a reframing around strict counterexamples. No fatal flaw is present.","tokens_in":16652,"tokens_out":13011,"duration_ms":120969,"concrete_test":"Re-run the human annotation on a random sample (e.g., n=100) of ProbeGen counterexamples from unit-test-passing samples, including both filter-accepted and filter-rejected examples, with the same annotation protocol but at least two independent annotators and a pre-registered definition of 'significant'. Compute filter precision and recall and estimate the true significant-difference rate as TP/(TP+FN) over all unit-test-passing runnable samples, with a 95% confidence interval. If the corrected estimate is not within 5 percentage points of 18.7%, or if the confidence interval is wider than +/-10 points, the abstract's headline number should be replaced by the strict counterexample rate (43.1%) or reported with explicit uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 4.2 reports that 24.3% of unit-test-passing samples are flagged by ProbeGen's spurious filter as significantly different, then multiplies by a filter precision of 77.1% to get the abstract's 18.7% (18% in the abstract). This calculation is the quantitative core of the paper's central claim. The precision estimate rests on a convenience sample of '2 samples per LBPP problem' of counterexamples; the number of annotated examples is not reported, the sample is not random (problems with many counterexamples are capped, problems with one are over-represented), and the inter-annotator agreement is only kappa=0.71, so the labels themselves are unstable. Furthermore, the correction uses precision only and ignores recall: the filter may discard true positive counterexamples, so 18.7% is neither a lower bound nor an upper bound, just an unqualified point estimate. No confidence interval is given for any headline percentage. If the true filter precision on a representative sample is, say, 50% instead of 77.1%, the headline becomes 12.2%; if the sample is further biased toward easy problems, it could be lower. The strict counterexample rate of 43.1% (24.3% plus the 18.8% filtered out) is deterministic and does not depend on the filter, so the underlying observation that unit tests miss strict functional differences is robust; the vulnerability is specifically the 'significant difference' headline that depends on the filter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ProbeGen, a white-box method that uses LLMs with execution feedback to generate probes (input usages) that expose behavioral differences between code implementations, thereby disproving strict functional equivalence. On the LBPP Python benchmark, the authors report that ProbeGen disproves 24.3% of samples that pass the provided unit tests after applying an LLM-based spurious-counterexample filter, and estimate 18.7% (abstract: 18%) after correcting for the filter's measured precision of 77.1%. The paper also uses ProbeGen for semantic clustering and semantic self-consistency, reporting a pass@1 improvement from 40.6 to 44.6. The core mechanism is logically sound: a probe that produces different outputs on two implementations deterministically disproves equivalence. However, the headline quantitative claims rest on a small, non-random human-annotated sample for filter precision, with no confidence intervals and no correction for filter recall.","tokens_in":16940,"tokens_out":6203,"duration_ms":56520,"significance":"If the empirical claims are reliable, the paper makes a useful contribution: it provides a practical method to complement weak unit tests, a new task for LLM code understanding, and a search-strategy analysis with a generalized pass@k estimator. The strict counterexample detection is deterministic and falsifiable, and the appendix gives concrete trajectories illustrating the method. The significance is moderated, however, by the reliance on a single probe model (Gemini Flash 2.0), the subjectivity of the 'significant difference' notion (human κ=0.71), and the lack of statistical rigor in the headline estimates. The paper's central insight—that unit-test equivalence can hide detectable semantic differences—appears robust, but the magnitude of the effect is not yet established to the precision claimed.","major_comments":[{"comment":"The headline 18.7% (abstract: 18%) is computed as 24.3% × 0.771, where 0.771 is the precision of the spurious-counterexample filter estimated from a human-annotated sample described only as '2 samples per LBPP problem (if more than 2 exist)' among counterexample-positive, unit-test-passing samples. The annotation sample size is not reported, the sample is not random (problems with many counterexamples are capped and problems with a single counterexample are over-represented), no confidence interval is given, and the correction uses precision only while ignoring filter recall. Because the filter is acknowledged in Sec. 3.1 to be able to introduce false negatives, 18.7% is neither an unbiased estimate nor a bound. This is load-bearing for the paper's central quantitative claim. Please report the exact annotation sample size and selection procedure, compute a confidence interval for precision (e.g., Wilson interval), perform a sensitivity analysis of the headline rate under plausible precision and recall values, and state explicitly the recall assumption.","section":"Sec. 4.2, Table 1, Abstract"},{"comment":"The semantic self-consistency improvement (pass@1 40.6 → 44.6, a 10% relative increase) is reported without any measure of uncertainty, and the effect disappears when the spurious-counterexample filter is removed (PROBE GEN w/o Filtering gives 40.6). Since this result depends on the same filter whose precision estimate is uncertain, the improvement is not yet established. Please provide confidence intervals or a significance test, and report the number of samples and a per-problem breakdown of the improvement.","section":"Sec. 5.1, Table 2"},{"comment":"The inter-annotator agreement of κ=0.71, described as 'substantial', means that the ground-truth label of 'significant difference' is unstable for a nontrivial fraction of cases. The paper should quantify how this label noise affects the precision estimate—for example, by reporting per-annotator precision values or a confidence interval that accounts for annotator disagreement—and discuss how the headline rate changes under alternative annotation judgments. This is important because the filter's precision is the only quantity used to correct the central estimate.","section":"Sec. 4.2, Human Evaluation"}],"minor_comments":[{"comment":"The abstract's '18%' does not mention that this is an estimate after filtering and correcting for filter precision; the underlying strict counterexample rate (without filtering) is substantially higher (24.3% + 18.8% = 43.1% of unit-test-passing samples). Please state both the strict and the filtered rates explicitly to avoid conflating 'disproved equivalence' with 'significant semantic difference'.","section":"Abstract / Sec. 4.2"},{"comment":"The captions in Figure 6 are placeholder text ('Lorem ipsum') and do not describe the actual search-strategy contour plots; these must be replaced with informative captions before publication.","section":"Appendix, Fig. 6"},{"comment":"The sample trajectories contain several typos and formatting errors ('differneces', 'For For input', 'count_string(ßßßß)' with mismatched quotes, and an unescaped Greek sigma in the text). These should be corrected for readability.","section":"Appendix D"},{"comment":"The relationship between the mutually-runnable percentages in Table 1 and the statement that unit-test-passing samples 'consist of 44.4% of all samples' is not fully explained; please clarify whether the 24.3% figure is computed over all unit-test-passing samples or only those that are also runnable by ProbeGen.","section":"Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a sound and useful mechanism for counterexample search, but the central empirical claims need stronger statistical support. The 'Lorem ipsum' captions in the appendix suggest the manuscript was not fully polished, and I would encourage the editor to require the authors to address the statistical concerns before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe strict counterexample finding is the real result. On LBPP, ProbeGen's LLM probes with execution feedback expose strict functional divergence between ground truth and unit-test-passing implementations in 43.1% of samples. That number is deterministic and does not depend on the spurious filter. The more-publicized 18.7% does depend on the filter, and that figure is fragile. It comes from multiplying the post-filter 24.3% by a precision estimate of 77.1% that was measured on a small, non-random human-annotated sample (two samples per LBPP problem, capped at two, with inter-annotator agreement of kappa=0.71). No confidence intervals, no recall estimate. If precision is optimistic, the headline drops; if recall is poor, it could be higher. It is an unqualified point estimate.\n\nWhat is actually new: assertion-free probes (inputs only, no expected outputs), iterative LLM probe generation with execution feedback as a tree search, the search-strategy cost/accuracy Pareto analysis, and the semantic-clustering/self-consistency application. The logical core is sound; a genuine counterexample disproves strict equivalence. The paper also states its own limitations clearly (cost, deterministic executable code, sandboxing).\n\nSoft spots in order of severity. First, the filter precision problem above; the paper's new quantitative claims rest on it. Second, no code or data artifacts are released, which hampers reproduction of the headline. Third, only one probe model (Gemini Flash 2.0) and one benchmark are used, so generalization is untested. Fourth, the semantic self-consistency gain is a 10% relative pass@1 improvement, small, and it disappears without the filter.\n\nNone of this is fatal. The method is a real contribution, and the strict counterexample observation is solid. But the abstract's 'disproves 18%' overstates what is measured. A serious referee should ask for artifacts, a larger random annotation sample, or at least interval estimates on the precision. I would send it to review.\n\nFor your own work: worth reading for the search-strategy analysis and the semantic clustering idea. I would probably cite it if I were building on code evaluation methods, but I would wait for the tightened version.","headline":"The strict counterexample rate is robust; the filtered 18.7% headline rests on a fragile LLM filter precision estimate.","tokens_in":17496,"tokens_out":3478,"would_cite":true,"duration_ms":32486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProbeGen, a white-box, execution-guided search that asks an LLM to generate probe programs until two implementations of the same interface produce observably different results, and on the LBPP benchmark it flags an estimated 18.7% of…","keywords":["program equivalence","probe generation","counterexample search","differential testing","LLM execution feedback","code synthesis evaluation","semantic self-consistency","unit test adequacy"],"falsifier":"Adjudicate every unit-test-passing counterexample ProbeGen produces on a fresh sample of, say, 1,000 LBPP outputs, using three independent annotators per example instead of two authors on a subset. If the agreement rate on significant differences falls well below 77.1%, or if the implied corrected hidden-divergence rate approaches zero, the claim that ProbeGen finds substantial semantic differences that unit tests miss would be falsified.","tokens_in":16393,"feed_emoji":"🐛","tokens_out":12030,"duration_ms":93666,"temperature":0.7,"pith_summary":"The paper claims that the standard way of judging whether an LLM-generated program is correct — running a small set of hand-written unit tests — is too weak, and that the missing behavioral differences can be found by asking an LLM to write probes. A probe is a program that feeds inputs to an interface and returns a value, so that if two implementations produce different probe values they are functionally different, with the difference proven by execution. ProbeGen iterates this process: the LLM sees both implementations and the execution outputs of all previous probes, then generates new probes until the outputs diverge, after which a second LLM filter judges whether the divergence is significant or an artifact of unspecified behavior. On the LBPP code-synthesis benchmark, ProbeGen disproves equivalence to the ground truth for 24.3% of the samples whose unit tests pass, a figure that drops to an estimated 18.7% after correcting for the filter's measured 77.1% precision. If the paper is right, a sizable fraction of code currently marked correct carries hidden semantic differences, and the same machinery, used to cluster candidate solutions by observable behavior, improves pass@1 by about 10%.","feed_headline":"LLM-written probes flag 18.7% of unit-test-passing code as different","feed_subtitle":"Execution-guided LLM probes disprove functional equivalence where benchmark test suites see none.","key_machinery":"The load-bearing object is the probe: a function $p$ that takes an implementation of an interface and returns a value, so that $p(f_i) \\neq p(f_j)$ for two implementations is an execution-proven disproof of their functional equivalence. Unit tests are the special case where the probe returns True or False by comparing against an expected output; generating probes is easier because the LLM only has to produce valid inputs, not predict outputs. ProbeGen is an iterative tree search: at each turn the LLM receives the source of both implementations plus the string-represented outputs of every prior probe, and samples new probes as Python generators, with type checking enforcing explicit preconditions; search strategies trade branching factor against depth, and the paper's Decreasing strategy reaches a 90% probability of finding an existing counterexample at roughly a quarter of the cost of a full uniform search. The final component is the spurious-counterexample filter, another LLM prompt that decides whether a divergence violates implicit preconditions or concerns output details the task leaves unspecified; its measured 77.1% precision is what converts the raw counterexample rate of 24.3% into the headline estimate of 18.7%.","core_discovery":"The paper's central claim is that functional-equivalence checking between two implementations of an interface can be carried out by an LLM-driven, execution-guided search for probe programs, and that this search frequently finds counterexamples in code that passes the official unit tests. In the LBPP code-synthesis benchmark, ProbeGen finds counterexamples in 24.3% of the samples whose unit tests pass; applying the paper's LLM-based filter for spurious counterexamples, and correcting for the filter's measured 77.1% precision, brings that estimate down to 18.7%. The paper further claims that the same probe machinery, used to partition candidate implementations by observable behavior, supports a semantic form of self-consistency: choosing the largest semantic cluster raises pass@1 from 40.6% to 44.6%, a roughly 10% relative gain, while the same procedure without spurious-counterexample filtering yields no gain at all.","pith_inferences":["Because the only requirements are two executable implementations, a value-equality operator, and bounded execution time, the same probe search could power change-validation tools that compare code before and after an edit — an application the paper names but does not develop.","The headline 18.7% is as much a measurement of the LBPP test suites as of the models; rerunning ProbeGen on other benchmarks would yield a directly comparable hidden-divergence rate that quantifies how much semantic slack different test suites leave.","The method's practical bottleneck is not finding counterexamples but deciding intent: attaching formally generated preconditions to the significance filter, which the paper cites as adjacent work, would move the pipeline toward verifiable significance instead of relying on the 77.1%-precision judge."],"forward_implications":["Running ProbeGen on unit-test-passing samples before declaring a code-generation result correct would reclassify a reported 18.7% of supposedly correct samples as behaviorally different from the ground truth.","Where unit tests do not exist, ProbeGen can act as the equivalence oracle itself, since it needs only the implementations and a value-equality operator; it agrees with unit tests on 83.3% of the outcomes for mutually runnable samples.","Semantic self-consistency becomes practical: choosing the largest semantic cluster of candidate implementations improves pass@1 by about 10% relative to the baseline, and this gain disappears when spurious counterexamples are not filtered.","The recursive success-probability estimator gives a budget-planning tool: for a given LLM, the Decreasing search strategy reaches 90% probability of finding an existing counterexample at roughly a quarter of the cost of a full uniform search."],"supporting_citations":[{"why":"Supplies the pass@k estimator that the paper generalizes into the recursive success-probability sigma used to compare search strategies.","marker":"(Chen et al., 2021)"},{"why":"Defines the equivalence-modulo-inputs paradigm that ProbeGen instantiates when its probes compare implementations on explicit inputs.","marker":"(Le et al., 2014)"},{"why":"Establishes differential testing, the testing paradigm the probe search operates within.","marker":"(McKeeman, 1998)"},{"why":"Provides the LBPP benchmark whose unit tests and ground-truth implementations generate all of the paper's headline numbers.","marker":"(Matton et al., 2024)"},{"why":"Supports the assumption that LLMs can take execution feedback into account, which ProbeGen's multi-turn loop depends on.","marker":"(Chen et al., 2023b)"},{"why":"Defines self-consistency, the baseline that semantic self-consistency modifies and is compared against.","marker":"(Wang et al., 2022)"},{"why":"Defines property-based testing, the random-input baseline that ProbeGen is measured against on runnable samples.","marker":"(Fink & Bishop, 1997)"}],"fun_headline_variants":["LLM probes find 18.7% of 'passing' code is not equivalent","ProbeGen: LLM probes expose 18.7% false equivalence in tests","LLM probes disprove equivalence in 18.7% of test-passing code","Semantic clustering with LLM probes boosts pass@1 by 10%","LLM-driven probes catch what unit tests miss: 18.7% differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers stand or fall with the LLM filter that decides which counterexamples count as real differences: its precision is estimated at 77.1% from human annotations of a two-per-problem subset, so if that estimate is optimistic, the 18.7% figure and the 10% pass@1 improvement both shrink.","fun_headline_variants_meta":{"raw":{"variants":["LLM probes find 18.7% of 'passing' code is not equivalent","ProbeGen: LLM probes expose 18.7% false equivalence in tests","LLM probes disprove equivalence in 18.7% of test-passing code","Semantic clustering with LLM probes boosts pass@1 by 10%","LLM-driven probes catch what unit tests miss: 18.7% differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000942,"raw_usage":{"total_tokens":3982,"prompt_tokens":859,"completion_tokens":3123,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":3014}},"tokens_in":475,"tokens_out":3123,"duration_ms":19417,"temperature":1.0,"reasoning_tokens":3014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:48:15.459382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Adjudicate every unit-test-passing counterexample ProbeGen produces on a fresh sample of, say, 1,000 LBPP outputs, using three independent annotators per example instead of two authors on a subset. If the agreement rate on significant differences falls well below 77.1%, or if the implied corrected hidden-divergence rate approaches zero, the claim that ProbeGen finds substantial semantic differences that unit tests miss would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes differential testing, the testing paradigm the probe search operates within."},{"cited_title":"and Bishop, M","cited_arxiv_id":null,"evidence_quote":"Defines property-based testing, the random-input baseline that ProbeGen is measured against on runnable samples."}],"review_version":1}