REVIEW 4 major objections 6 minor 3 cited by
Pairwise comparison signals from an LLM can be folded into accuracy estimation as control variates, yielding the semiparametrically efficient estimator that strictly beats naive averaging.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 05:06 UTC pith:X2ICAZ3N
load-bearing objection The semiparametric theory is sound, but the real-data protocol violates the paper's own dependence condition, so the headline variance-reduction claim holds only in simulation. the 4 major comments →
Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that LLM accuracy θ = E[φ(Y,G)] can be estimated at the semiparametric efficiency bound by augmenting direct correctness observations with pairwise comparison signals Z=(W1,W2,V) that the target model produces when judging auxiliary reasoning chains. Under the assumption that the conditional distribution p(z|x) is known (since it is generated by a known prompting mechanism), the efficient influence function is ψ(X,Y,G,Z) = (m(X)−θ) − (τ(X,Z)−φ(Y,G)), where τ is the outcome regression and m is its marginalization over the auxiliary distribution. Theorem 4.1 shows the one-step estimator satisfies √N(θ̂₁₋ₛₜₑₚ − θ) → N(0, σ²_eff) with σ²_eff = Var(ψ), and Corollary 4.1 prov
What carries the argument
The efficient influence function (EIF) for the target parameter under the semiparametric model with known p(z|x): ψ(X,Y,G,Z) = (m(X)−θ) − (τ(X,Z)−φ(Y,G)). This decomposition is the engine: the term (m(X)−θ) is a variance-reduction projection onto the known auxiliary distribution, and (τ(X,Z)−φ(Y,G)) is an orthogonal correction that is zero in expectation, so the estimator stays √N-consistent while its variance drops to Var(ψ). The one-step algorithm computes m via Monte Carlo sampling from p(z|x) and fits τ by cross-fitting (or, in small samples, by treating an off-the-shelf LLM as a fixed 'semantic regressor').
Load-bearing premise
The whole construction assumes the distribution of auxiliary comparison signals given the input prompt is exactly known and can be sampled from at will; if this conditional distribution is misspecified or shifts (e.g., with decoding settings or model version), the estimator can be biased and the guaranteed variance reduction is not assured.
What would settle it
Generate a synthetic dataset where the target model's pairwise judgments are conditionally independent of correctness given X (so τ(X,Z)=m(X) a.s.); the one-step estimator's asymptotic variance should equal the naive estimator's, and any observed improvement beyond sampling noise would contradict Corollary 4.1. More directly, a misspecified p(z|x) (e.g., assuming a different temperature or auxiliary model than actually generating Z) should produce biased θ̂₁₋ₛₜₑₚ estimates that deviate from θ in expectation, testable via Monte Carlo with a controlled data-generating process.
If this is right
- Accuracy estimates on small benchmarks like AIME 2025 (30 problems) can be reported with valid confidence intervals rather than single noisy percentages.
- Model rankings become more stable under repeated evaluation, since the variance driving rank flips is provably reduced.
- The framework generalizes beyond pairwise comparisons to any auxiliary signal (e.g., Likert scores, multi-way rankings) that shares mutual information with correctness.
- The same EIF-based construction can be derived for other statistical functionals such as pass@k accuracy, extending the efficiency gain to a broader class of evaluation metrics.
- Practitioners can achieve the same precision as naive estimation with fewer samples, reducing compute and query costs in small-data regimes.
Where Pith is reading between the lines
- An immediate consequence the authors leave implicit is that the method's variance reduction scales with the strength of the generation–verification gap; on benchmarks where the model's preferences carry little signal relative to its own answers, gains will be small, so users should empirically measure τ−m before adopting the estimator.
- The framework suggests a new criterion for selecting auxiliary models: instead of maximizing judge accuracy alone, one can rank candidate auxiliary configurations by the reduction they induce in σ²_eff, making auxiliary-model selection an optimization problem in its own right.
- A natural extension is to apply the same control-variate construction to human evaluation or non-exact metrics (e.g., rubrics, partial credit), where the 'ground truth' is itself noisy; the efficiency-bound argument remains valid as long as the metric is a bounded functional and p(z|x) is known.
- The reliance on known p(z|x) is a strong idealization; a robustness check that perturbs p(z|x) (e.g., simulating decoder-temperature shifts) and measures how much variance reduction survives would tell practitioners how much of the guarantee holds outside the idealized setting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a semiparametric method for estimating the accuracy of an LLM on a benchmark by combining the usual labeled outcomes with auxiliary pairwise-comparison signals. It derives an efficient influence function (EIF) under the assumption that the conditional distribution of the auxiliary signals given the input is known, constructs a cross-fitted one-step estimator, and proves asymptotic normality plus strict variance reduction over the naive sample mean whenever the auxiliary signal is not conditionally independent of the outcome. Simulations with an artificially shared latent component support the theory. Experiments on GPQA Diamond, AIME 2025, and GSM8K report one-shot improvements of the one-step estimator over the naive estimator on small subsets.
Significance. If the theoretical claims are correct, the paper offers a clean control-variate framework for LLM evaluation with a clear efficiency bound, and the estimator is unbiased for fixed nuisance functions—a useful property. The simulation design illustrates the variance-reduction mechanism. However, the practical value depends on whether the auxiliary signals actually carry conditional information about the target outcome under the real-world data-collection protocol, which the paper does not establish. The theoretical contribution is solid, but the empirical evidence is not yet convincing.
major comments (4)
- [§5.2.1, Corollary 4.1, Remark 4.3] The strict variance reduction guarantee requires τ(X,Z) ≠ m(X). In the real protocol, Y is generated by one target-model call and V by a separate comparison call, while W1,W2 come from auxiliary models. Standard LLM sampling makes these draws conditionally independent given X, so E[ϕ(Y,G)|X,Z] = E[ϕ(Y,G)|X] = m(X). The statement in Remark 4.3 that 'since Z is also partially obtained by the target model, this independence is naturally violated' is not justified; being generated by the same model does not create dependence across independent samples. Under τ=m, the one-step estimator reduces to the naive average up to Monte Carlo noise, and Corollary 4.1 has no force. To support the central empirical claim, the paper must either engineer a shared latent dependence between Y and V (as in the simulation) or empirically demonstrate τ≠m on real data (e.g., by testing whether Z improves predict
- [§5.2.2, Tables 1–3] The reported 'Improv.' is the absolute-error difference to the full-data GT% for a single random subset. This does not measure variance reduction; it conflates sampling noise with estimation error. Since only one split is used, positive values can occur by chance, and indeed Table 2 contains negative improvements (Qwen3-Next-80B: -1.06% Config 1, -5.33% Config 2; Llama-3.3-70B: -1.66% Config 1, -0.43% Config 2). The abstract's claim of 'consistent variance reduction' is unsupported. The paper should report repeated-subsample or bootstrap estimates of variance (or MSE) for both estimators, with standard errors, across several subset sizes.
- [§4, Assumption 4.3 vs §5.2.2] Theorem 4.1 requires M→∞ for the Monte Carlo approximation of m(x) to be negligible, but the real experiments use M=10 auxiliary samples per instance. With M=10, the MC error contributes non-negligibly to the estimator's variance, so the semiparametric efficiency bound is not attained and the variance formula σ²_eff = Var(ψ) is not exact. The paper should either use a much larger M (it claims sampling is cheap) or provide a finite-M variance expression and demonstrate that M=10 is adequate.
- [Assumption 3.1] The assumption that p(z|x) is 'known' is not literally satisfied for LLMs: the distribution of the auxiliary signals is not available in closed form; it is only sampleable via the stochastic generation process, which depends on decoding settings, temperature, prompts, and model versions. If the generator changes or is misspecified, the Monte Carlo samples are not from the true p(z|x), and the unbiasedness of the one-step estimator fails. The paper should specify precisely what 'known' means and how practitioners should fix the generator to guarantee correct sampling.
minor comments (6)
- [Abstract] 'Guarantees strict variance reduction' is unconditional, but Corollary 4.1 requires τ≠m. Qualify the claim in the abstract.
- [Remark 3.3 and §5.2.2] The semantic regressor is Gemini-3-Flash-Preview, which is also a target model in Tables 1 and 2. Clarify why using the same model as both target and nuisance function does not create a self-referential bias.
- [Table 2] The row for Claude-Sonnet-4.5, Config 2 reports Improv. +12.00%, which is a large outlier relative to other entries (most are within ±6%). Explain or verify this value.
- [Eq. (3.4)] The argument order in bτ(w1j,w2j,vj,x) is inconsistent with the earlier definition bτ(x,z); use a consistent convention.
- [Remark 3.3] There is a typo: 'performs remably well' should be 'remarkably well'.
- [§5.2.2] The one-step estimates can exceed the [0,1] range for accuracy (e.g., 99.93% in Table 2). State that the estimator is not constrained to the parameter space and that this is a known property of one-step estimators.
Circularity Check
No significant circularity: the EIF derivation is self-contained; efficiency gains are conditional on an empirical dependence assumption, not on a definitional tautology.
full rationale
Walking the derivation chain, the EIF in Eq. (3.1) is derived from the stated semiparametric model in Assumption 3.1, and the variance-reduction identity Var(ψ) = Var(ϕ(Y,G)) − E[(τ(X,Z)−m(X))^2] follows algebraically from τ(X,Z)=E[ϕ|X,Z] and m(X)=E[τ|X]. Theorem 4.1 and Corollary 4.1 are standard semiparametric results applied to that model; no parameter is fitted to θ itself, and the one-step estimator is an unbiased control-variate construction. The condition τ≠m is explicitly identified in Corollary 4.1 and Remark 4.3 as the requirement for strict variance reduction; whether the real-world protocol in §5.2.1 induces this condition (rather than conditional independence of Y and Z given X, which would make τ=m) is an empirical/assumption question, not a circularity in the derivation. Assumption 3.1/Remark 3.1 and the M=10 MC approximation (contrary to Assumption 4.3) are correctness risks, not circular steps. The self-citations (Dong et al. 2026; Xu et al. 2025; Jin et al. 2025a/b) appear in related-work and motivation contexts and are not load-bearing for the efficiency proof. The use of Gemini-3-Flash-Preview as a fixed semantic regressor, including when it is also a target model, is a practical confound but does not make the estimator's derivation circular. Overall, the central derivation is self-contained; score reflects only the mild self-referential elements in the experimental design and auxiliary-signal assumptions.
Axiom & Free-Parameter Ledger
free parameters (2)
- Monte Carlo sample count M =
500 (simulation); 10 (real-world)
- Cross-fitting fold count K =
5
axioms (5)
- domain assumption Assumption 3.1: p(y,g,z,x)=p(y,g|z,x)·p(z|x)·p(x) with p(z|x) known and fixed
- standard math Assumption 4.1: E[φ(Y,G)^2] < ∞
- domain assumption Assumption 4.2: bτ converges to τ in L2
- domain assumption Assumption 4.3: M→∞ as N→∞
- domain assumption Corollary 4.1 condition: τ(X,Z) ≠ m(X) with positive probability
Cite this review
Pith. "Pith review of Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals." pith.science (2026). https://pith.science/paper/X2ICAZ3N
@misc{pith2026260203061,
author = {Pith},
title = {Pith review of: Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals},
year = {2026},
howpublished = {\url{https://pith.science/paper/X2ICAZ3N}},
note = {Machine review of arXiv:2602.03061}
}
read the original abstract
Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.
Figures
Forward citations
Cited by 3 Pith papers
-
LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency
LLM pairwise evaluation is recast as low-rank tensor completion, yielding semiparametric efficient estimators and asymptotic normality for ability functionals via a score-whitening correction for anisotropic operators.
-
Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons
Low-rank matrix modeling with cross-fitted debiased estimators and multiplier bootstrap yields stable task-specific LLM rankings and asymptotically valid simultaneous confidence sets from sparse pairwise data.
-
Reinforcement Learning from Human Feedback: A Statistical Perspective
A statistical survey of RLHF for LLM alignment that connects preference learning and policy optimization to models like Bradley-Terry-Luce while reviewing methods, extensions, and open challenges.
Reference graph
Works this paper leans on
-
[1]
Ahn, J. et al. (2024). Large language models for mathematical reasoning: Progresses and challenges.arXiv preprint arXiv:2402.00157. 13 Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., and Jordan, M. I. (2024). Autoeval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008. Bradley, R. A. and Terry, M. E. (1952). Rank ...
Pith/arXiv arXiv 2024
-
[30]
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168. Deng, A., Xu, Y., Kohavi, R., and Walker, T. (2013). Improving the sensitivity of online controlledexperimentsbyutilizing...
Pith/arXiv arXiv 2021
-
[36]
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. (2024). Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling. Robins, J. M. and Rotnitzky, A. (1995). Semiparametric efficiency in multivariate missing data problems.Journal of the American Statistical Associatio...
Pith/arXiv arXiv 2024
-
[2025]
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623. Zhou, Z., Song, Y., and Zanette, A. (2025). Accelerating unbiased LLM evaluation via synthetic feedback. InarXiv pr...
Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.