{"id":"7469a5cf-5afd-41ee-8602-e5b094e2526c","arxiv_id":"2602.03061","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A one-step semiparametric estimator using pairwise comparison signals as control variates achieves the efficiency bound for estimating LLM accuracy on math benchmarks.","lead":"An LLM that cannot solve a math problem can still often say which of two candidate solutions is better. This paper uses that pairwise signal to build a lower-variance statistical estimate of the model's accuracy on small benchmarks like AIME.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real data-collection protocol may violate Assumption 3.1's dependence between target answer and auxiliary signal, so τ(X,Z)=m(X), and the strict variance-reduction guarantee does not apply.","rationale":"The reader identified Assumption 3.1's known p(z|x) as the weakest point. But even when p(z|x) is known and M is large, the estimator's entire payoff rests on τ(X,Z)≠m(X), i.e., on Z being informative about Y beyond X. The actual data collection samples Y and Z in separate, independent model calls, so under the standard i.i.d. sampling assumption for LLMs, p(y,g,z|x)=p(y,g|x)p(z|x) and τ=m. This is not an internal inconsistency in the theorems—they are correct under Assumption 3.1—but it is a mismatch between the model and the empirical protocol. The simulation validates the method under a carefully constructed shared-latent dependence that mimics this assumption, but the real experiments never verify that such dependence exists. The central contribution is an evaluation framework with a provable efficiency gain; if the gain condition is not met in practice, the headline empirical claims ('consistent variance reduction', 'more reliable rankings') are not established. The paper can be salvaged by measuring the conditional dependence, or by changing the protocol to make V depend on Y (e.g., comparing Y against a candidate), so CONDITIONAL is appropriate rather than outright REJECT. A direct empirical test would settle whether the conditional dependence actually exists; until then, the strongest claim should be viewed as conditional on an unverified property of LLM sampling.","tokens_in":13242,"tokens_out":21099,"duration_ms":233103,"concrete_test":"On AIME 2025 (or any benchmark), fix the auxiliary chains (W1,W2); for each problem X, draw R=20 independent target-model answers Y^(r) and R independent comparison signals V^(r). Estimate the within-problem regression of φ(Y^(r),G) on bτ(X,Z^(r)); if the average within-problem coefficient is statistically zero, then τ=m and the efficiency condition fails. Alternatively, bootstrap the one-step estimator over repeated subsamples and compare its empirical variance to that of the naive estimator; no variance reduction confirms the concern.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The efficiency claim depends on τ(X,Z)=E[φ(Y,G)|X,Z] differing from m(X)=E[φ(Y,G)|X] (Corollary 4.1). Under the protocol in §5.2.1, Y is generated by one target-model call and V by a separate comparison call, while W1,W2 are generated by auxiliary models. In the standard LLM sampling model, separate API calls are conditionally independent given X, so p(y,g,z|x)=p(y,g|x)p(z|x), which implies τ=m. The simulation (§5.1.1) creates dependence artificially by sharing a latent ε across Y and W; the real experiments do not induce such a shared latent. If τ=m, the EIF in Eq. (3.1) reduces to φ(Y,G)−θ, the one-step estimator degenerates to the naive average (up to MC noise), the semiparametric efficiency bound offers no improvement, and the 'guaranteed strict variance reduction' has no force. The reported 'Improv.' tables measure closeness to a single ground-truth accuracy, not variance reduction, and the paper never tests whether τ≠m on real data. Thus the central empirical claim that pairwise comparison signals improve evaluation efficiency is unsupported unless conditional dependence is demonstrated or explicitly engineered.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a semiparametric method for estimating the accuracy of an LLM on a benchmark by combining the usual labeled outcomes with auxiliary pairwise-comparison signals. It derives an efficient influence function (EIF) under the assumption that the conditional distribution of the auxiliary signals given the input is known, constructs a cross-fitted one-step estimator, and proves asymptotic normality plus strict variance reduction over the naive sample mean whenever the auxiliary signal is not conditionally independent of the outcome. Simulations with an artificially shared latent component support the theory. Experiments on GPQA Diamond, AIME 2025, and GSM8K report one-shot improvements of the one-step estimator over the naive estimator on small subsets.","tokens_in":13583,"tokens_out":12965,"duration_ms":139029,"significance":"If the theoretical claims are correct, the paper offers a clean control-variate framework for LLM evaluation with a clear efficiency bound, and the estimator is unbiased for fixed nuisance functions—a useful property. The simulation design illustrates the variance-reduction mechanism. However, the practical value depends on whether the auxiliary signals actually carry conditional information about the target outcome under the real-world data-collection protocol, which the paper does not establish. The theoretical contribution is solid, but the empirical evidence is not yet convincing.","major_comments":[{"comment":"The strict variance reduction guarantee requires τ(X,Z) ≠ m(X). In the real protocol, Y is generated by one target-model call and V by a separate comparison call, while W1,W2 come from auxiliary models. Standard LLM sampling makes these draws conditionally independent given X, so E[ϕ(Y,G)|X,Z] = E[ϕ(Y,G)|X] = m(X). The statement in Remark 4.3 that 'since Z is also partially obtained by the target model, this independence is naturally violated' is not justified; being generated by the same model does not create dependence across independent samples. Under τ=m, the one-step estimator reduces to the naive average up to Monte Carlo noise, and Corollary 4.1 has no force. To support the central empirical claim, the paper must either engineer a shared latent dependence between Y and V (as in the simulation) or empirically demonstrate τ≠m on real data (e.g., by testing whether Z improves predict","section":"§5.2.1, Corollary 4.1, Remark 4.3"},{"comment":"The reported 'Improv.' is the absolute-error difference to the full-data GT% for a single random subset. This does not measure variance reduction; it conflates sampling noise with estimation error. Since only one split is used, positive values can occur by chance, and indeed Table 2 contains negative improvements (Qwen3-Next-80B: -1.06% Config 1, -5.33% Config 2; Llama-3.3-70B: -1.66% Config 1, -0.43% Config 2). The abstract's claim of 'consistent variance reduction' is unsupported. The paper should report repeated-subsample or bootstrap estimates of variance (or MSE) for both estimators, with standard errors, across several subset sizes.","section":"§5.2.2, Tables 1–3"},{"comment":"Theorem 4.1 requires M→∞ for the Monte Carlo approximation of m(x) to be negligible, but the real experiments use M=10 auxiliary samples per instance. With M=10, the MC error contributes non-negligibly to the estimator's variance, so the semiparametric efficiency bound is not attained and the variance formula σ²_eff = Var(ψ) is not exact. The paper should either use a much larger M (it claims sampling is cheap) or provide a finite-M variance expression and demonstrate that M=10 is adequate.","section":"§4, Assumption 4.3 vs §5.2.2"},{"comment":"The assumption that p(z|x) is 'known' is not literally satisfied for LLMs: the distribution of the auxiliary signals is not available in closed form; it is only sampleable via the stochastic generation process, which depends on decoding settings, temperature, prompts, and model versions. If the generator changes or is misspecified, the Monte Carlo samples are not from the true p(z|x), and the unbiasedness of the one-step estimator fails. The paper should specify precisely what 'known' means and how practitioners should fix the generator to guarantee correct sampling.","section":"Assumption 3.1"}],"minor_comments":[{"comment":"'Guarantees strict variance reduction' is unconditional, but Corollary 4.1 requires τ≠m. Qualify the claim in the abstract.","section":"Abstract"},{"comment":"The semantic regressor is Gemini-3-Flash-Preview, which is also a target model in Tables 1 and 2. Clarify why using the same model as both target and nuisance function does not create a self-referential bias.","section":"Remark 3.3 and §5.2.2"},{"comment":"The row for Claude-Sonnet-4.5, Config 2 reports Improv. +12.00%, which is a large outlier relative to other entries (most are within ±6%). Explain or verify this value.","section":"Table 2"},{"comment":"The argument order in bτ(w1j,w2j,vj,x) is inconsistent with the earlier definition bτ(x,z); use a consistent convention.","section":"Eq. (3.4)"},{"comment":"There is a typo: 'performs remably well' should be 'remarkably well'.","section":"Remark 3.3"},{"comment":"The one-step estimates can exceed the [0,1] range for accuracy (e.g., 99.93% in Table 2). State that the estimator is not constrained to the parameter space and that this is a known property of one-step estimators.","section":"§5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The theoretical machinery is sound but the paper's central empirical claim rests on the existence of conditional dependence between the auxiliary signals and the target outcome. Under the current data-collection protocol, independent API calls imply τ=m, so the one-step estimator cannot outperform the naive estimator in the way claimed. This is fixable either by redesigning the protocol to create shared latent dependence (as in the simulation) or by providing direct empirical evidence of τ≠m. If the authors cannot establish such dependence, the paper's main contribution would be reduced to a conditional theoretical result with limited real-world application. The single-split 'Improv.' tables are also insufficient; variance and confidence intervals are needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's new: deriving the EIF for LLM accuracy under a known auxiliary-signal distribution and using pairwise comparisons as control variates is a real contribution. The theorem is correct: with tau != m, the one-step estimator achieves the bound and strictly beats the naive average. The simulation is honest—the shared latent eps builds real dependence—and the estimator tracks the oracle as noise grows. I believe the math.\n\nThe soft spot is the bridge to practice. Under the §5.2.1 protocol, Y, W1, W2, and V come from separate sampling calls. For a fixed target model, those are conditionally independent given X under the standard sampling model, so tau(X,Z)=m(X) and the EIF degenerates. Remark 4.3 says dependence is 'naturally violated' because Z is partly produced by the target model, but that's only true if the model's own answer is shared; independent draws don't create that. The paper never tests whether tau != m on real data. The 'Improv.' columns measure error relative to one ground-truth estimate, not repeated-sampling variance, and Table 2 has three negative entries, contradicting 'consistent variance reduction.' No error bars, no repeated subsets, no code or data.\n\nThis is fixable: engineer dependence (condition the comparison on the target model's sampled answer, or sample auxiliary chains from the same trajectory), or show empirically that tau varies with Z. Absent that, the real-world results are not evidence of efficiency.\n\nWho is this for? Methodologists working on LLM evaluation will find the EIF derivation useful, and it deserves a serious referee. But the empirical section needs redoing before publication. I would accept it for peer review because the theory is solid and the question matters, but I would insist the real-data claims be scaled back or supported with actual variance estimates.\n\nTake it to reading group as a case study in how control-variate gains depend on the generative model.","headline":"The semiparametric theory is sound, but the real-data protocol violates the paper's own dependence condition, so the headline variance-reduction claim holds only in simulation.","tokens_in":594,"tokens_out":1520,"would_cite":true,"duration_ms":74235,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F12","62G05","62G20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise comparison signals from an LLM can be folded into accuracy estimation as control variates, yielding the semiparametrically efficient estimator that strictly beats naive averaging.","keywords":["semiparametric efficiency","efficient influence function","control variates","LLM evaluation","mathematical reasoning","pairwise comparisons","one-step estimator","variance reduction"],"falsifier":"Generate a synthetic dataset where the target model's pairwise judgments are conditionally independent of correctness given X (so τ(X,Z)=m(X) a.s.); the one-step estimator's asymptotic variance should equal the naive estimator's, and any observed improvement beyond sampling noise would contradict Corollary 4.1. More directly, a misspecified p(z|x) (e.g., assuming a different temperature or auxiliary model than actually generating Z) should produce biased θ̂₁₋ₛₜₑₚ estimates that deviate from θ in expectation, testable via Monte Carlo with a controlled data-generating process.","tokens_in":13184,"feed_emoji":"📉","tokens_out":4645,"duration_ms":43986,"temperature":0.7,"pith_summary":"Evaluating a language model on a small math benchmark yields noisy accuracy estimates; a single correct or wrong answer can move the reported number by several points, making model rankings unstable. The paper's central move is to exploit the 'generation–verification gap': on hard problems a model often can't produce the right answer, yet its pairwise judgment of which of two candidate solutions is better still carries reliable signal. The authors build a semiparametric estimator that treats these pairwise comparison signals as control variates, deriving the efficient influence function for the accuracy parameter and constructing a one-step, cross-fitted estimator. They prove it attains the semiparametric efficiency bound—so no regular estimator using the same data has lower asymptotic variance—and that it strictly beats naive sample averaging whenever the comparison signals are not conditionally independent of correctness. Simulations and experiments on GPQA Diamond, AIME 2025, and GSM8K show the estimator gives more precise accuracy estimates and more reliable rankings, especially in the small-sample regime where conventional evaluation is most unstable.","feed_headline":"Pairwise signals make LLM accuracy estimates provably tighter","feed_subtitle":"Even when final answers are wrong, a model's pairwise judgments cut estimation variance to the theoretical optimum.","key_machinery":"The efficient influence function (EIF) for the target parameter under the semiparametric model with known p(z|x): ψ(X,Y,G,Z) = (m(X)−θ) − (τ(X,Z)−φ(Y,G)). This decomposition is the engine: the term (m(X)−θ) is a variance-reduction projection onto the known auxiliary distribution, and (τ(X,Z)−φ(Y,G)) is an orthogonal correction that is zero in expectation, so the estimator stays √N-consistent while its variance drops to Var(ψ). The one-step algorithm computes m via Monte Carlo sampling from p(z|x) and fits τ by cross-fitting (or, in small samples, by treating an off-the-shelf LLM as a fixed 'semantic regressor').","core_discovery":"The paper establishes that LLM accuracy θ = E[φ(Y,G)] can be estimated at the semiparametric efficiency bound by augmenting direct correctness observations with pairwise comparison signals Z=(W1,W2,V) that the target model produces when judging auxiliary reasoning chains. Under the assumption that the conditional distribution p(z|x) is known (since it is generated by a known prompting mechanism), the efficient influence function is ψ(X,Y,G,Z) = (m(X)−θ) − (τ(X,Z)−φ(Y,G)), where τ is the outcome regression and m is its marginalization over the auxiliary distribution. Theorem 4.1 shows the one-step estimator satisfies √N(θ̂₁₋ₛₜₑₚ − θ) → N(0, σ²_eff) with σ²_eff = Var(ψ), and Corollary 4.1 prov","pith_inferences":["An immediate consequence the authors leave implicit is that the method's variance reduction scales with the strength of the generation–verification gap; on benchmarks where the model's preferences carry little signal relative to its own answers, gains will be small, so users should empirically measure τ−m before adopting the estimator.","The framework suggests a new criterion for selecting auxiliary models: instead of maximizing judge accuracy alone, one can rank candidate auxiliary configurations by the reduction they induce in σ²_eff, making auxiliary-model selection an optimization problem in its own right.","A natural extension is to apply the same control-variate construction to human evaluation or non-exact metrics (e.g., rubrics, partial credit), where the 'ground truth' is itself noisy; the efficiency-bound argument remains valid as long as the metric is a bounded functional and p(z|x) is known.","The reliance on known p(z|x) is a strong idealization; a robustness check that perturbs p(z|x) (e.g., simulating decoder-temperature shifts) and measures how much variance reduction survives would tell practitioners how much of the guarantee holds outside the idealized setting."],"forward_implications":["Accuracy estimates on small benchmarks like AIME 2025 (30 problems) can be reported with valid confidence intervals rather than single noisy percentages.","Model rankings become more stable under repeated evaluation, since the variance driving rank flips is provably reduced.","The framework generalizes beyond pairwise comparisons to any auxiliary signal (e.g., Likert scores, multi-way rankings) that shares mutual information with correctness.","The same EIF-based construction can be derived for other statistical functionals such as pass@k accuracy, extending the efficiency gain to a broader class of evaluation metrics.","Practitioners can achieve the same precision as naive estimation with fewer samples, reducing compute and query costs in small-data regimes."],"fun_headline_variants":["Wrong answers still yield reliable LLM rankings via pairwise signals","Pairwise comparisons cut LLM evaluation variance to the theoretical limit","When LLMs get it wrong, their pairwise picks still sharpen scores","Pairwise judgments halve the noise in LLM math rankings","Optimal precision: pairwise signals make LLM scores provably tight"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole construction assumes the distribution of auxiliary comparison signals given the input prompt is exactly known and can be sampled from at will; if this conditional distribution is misspecified or shifts (e.g., with decoding settings or model version), the estimator can be biased and the guaranteed variance reduction is not assured.","fun_headline_variants_meta":{"raw":{"variants":["Wrong answers still yield reliable LLM rankings via pairwise signals","Pairwise comparisons cut LLM evaluation variance to the theoretical limit","When LLMs get it wrong, their pairwise picks still sharpen scores","Pairwise judgments halve the noise in LLM math rankings","Optimal precision: pairwise signals make LLM scores provably tight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3840,"prompt_tokens":770,"completion_tokens":3070,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2993}},"tokens_in":514,"tokens_out":3070,"duration_ms":21266,"temperature":1.0,"reasoning_tokens":2993,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:06:40.526918+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a synthetic dataset where the target model's pairwise judgments are conditionally independent of correctness given X (so τ(X,Z)=m(X) a.s.); the one-step estimator's asymptotic variance should equal the naive estimator's, and any observed improvement beyond sampling noise would contradict Corollary 4.1. More directly, a misspecified p(z|x) (e.g., assuming a different temperature or auxiliary model than actually generating Z) should produce biased θ̂₁₋ₛₜₑₚ estimates that deviate from θ in expectation, testable via Monte Carlo with a controlled data-generating process.","supporting_citations":[],"review_version":1}