Pith. sign in

REVIEW 3 major objections 5 minor 7 references

Resurrecting saturated LLM benchmarks with adversarial encoding

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pairing multiple-choice questions and padding the answer list with obviously wrong options reliably lowers LLM accuracy, and that this effect can be packaged as Re-MMLU, a cheap benchmark that on large models matches…

desk verdict The pairing result is real but the distractor experiment is confounded by the falling random-chance baseline; the 'resurrection' claim needs more careful analysis. read the letter →

arxiv 2502.06738 v1 pith:4I63GVLR submitted 2025-02-10 cs.LG

Igor Ivanov , Dmitrii Volkov This is my paper · ORCID
classification cs.LG
keywords adversarialencodingbenchmarksaturationquestionpairingdistractoroptionsLLMrobustnessmultiple-choicebenchmarksRe-MMLUfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that LLM performance on standard multiple-choice benchmarks is brittle to two tiny format changes: putting two questions together and adding extra wrong answers. Across WMDP-bio, GPQA, and MMLU variants, these changes consistently reduce accuracy, with the largest relative drops on smaller models. Because current frontier models have nearly saturated old benchmarks, the authors argue the same trick can be used to 'unsaturate' them at almost no cost. They package the idea into Re-MMLU, a paired and expanded version of MMLU, on which large models score close to the far more expensive MMLU-Pro. If the claim holds, benchmark developers have a cheap way to keep old tests useful.

What carries the argument

The two mechanisms are the paired-question encoding and distractor expansion. In the paired-question encoding, two multiple-choice questions are presented at once and the model must emit one answer covering both (either as a two-digit label or as a combined answer choice from the Cartesian product); this forces joint decoding and produces large, uneven drops across models. In distractor expansion, the original answer options are kept and obviously wrong city names are appended, increasing the option count from 4 to 26; this produces uniform, gradual drops. Both mechanisms lower model scores, and combining them is what defines the Re-MMLU benchmark; the performance gap they create is the headroom the paper uses to unsaturate old tests.

What would settle it

Run a matched control where the original three wrong options are replaced by city names instead of adding city names, keeping four options in both versions; if scores do not fall, the distractor effect is about option count, not confusion. Separately, compute the drop expected solely from the chance baseline falling from 25% to about 4% as options grow from 4 to 26; if that accounts for the whole observed drop, the unsaturation claim would be a scoring artifact.

Watch

Extended reading notes

Core claim

The paper's central claim is that two simple, general-purpose question encodings—pairing two questions into one response and adding obviously wrong distractor options—reliably reduce LLM accuracy, and do so in a way that raises the effective performance ceiling of a benchmark. On WMDP-bio, GPQA, and MMLU-Pro, paired questions produced relative drops ranging from roughly 1% to 66%, with smaller models losing more; adding city-name distractors to WMDP-bio lowered scores by 9–17% as the option count grew from 4 to 26. The authors interpret this as evidence that recall is brittle to encoding rather than evidence of changed knowledge. They combine both effects to build Re-MMLU from MMLU, and report that large models perform within a mean absolute error of under 5% compared with MMLU-Pro, suggesting the cheap encoding reproduces the difficulty of an expensively constructed benchmark.

Load-bearing premise

The paper's interpretation depends on the drop being caused by the encoding confusing the model, rather than by the simple fact that more answer options make random guessing less likely to succeed.

Editorial extensions

If this is right

  • Saturated benchmarks can be given a second life by re-releasing them with paired questions and extra distractors, at near-zero design cost.
  • On frontier models, cheap adversarial encodings reproduce the difficulty of expert-built benchmarks: tinyRe-MMLU tracks MMLU-Pro within a mean absolute error under 5%.
  • Adding distractor options is a predictable difficulty knob, producing similar degradation across all tested models, whereas question pairing is more model-dependent and partially recoverable by fine-tuning.
  • Because the drop persists after fine-tuning and few-shot prompting, format brittleness appears to be a separate factor from knowledge, so benchmark gains may overstate real capability improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The lowered chance baseline from 25% to about 4% when moving from 4 to 26 options is not controlled for; at least part of the reported distractor drop could be a scoring artifact, so the 'confusion' mechanism should be treated as an open question.
  • If encoding difficulty is an independent axis, benchmark designers could report two numbers—knowledge accuracy and format robustness—and red-teamers could use the same encodings to stress-test models for deployment.
  • The similarity between tinyRe-MMLU and MMLU-Pro scores on large models may indicate a shared bottleneck, but the paper only observes the correlation; checking per-question error overlap would test that.
  • The authors predict that reasoning models will resist these encodings, but they do not test them; running the same experiments on such models is a natural test of the brittleness explanation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies two simple perturbations of multiple-choice LLM benchmarks: combining two questions into a single composite query and adding obviously wrong distractor options. On WMDP-bio, GPQA, and MMLU variants, it reports relative accuracy drops under both interventions, proposes H1 (question pairing confuses LLMs) and H2 (more answer options lower performance), and uses these effects to construct Re-MMLU, a modified MMLU with paired questions and 10 options. On the tinyRe-MMLU subset, large models score close to MMLU-Pro, and the paper suggests that this approach can resurrect saturated benchmarks.

Significance. If the effects are robust and correctly separated from the trivial change in random-guessing probability, the paper would provide a cheap, practical way to re-differentiate frontier models on existing question banks and to extend the useful life of saturated benchmarks. The paper is transparent about its released datasets, tests multiple model families, includes a fine-tuning control for the pairing intervention, and formulates specific falsifiable claims about option counts, pairing, and reasoning models. The main weakness is that the quantitative interpretation of the option-count experiment is confounded by the falling chance baseline, and the Re-MMLU comparison is based on a tiny subset rather than the full released benchmark.

major comments (3)
  1. [Section 2.4] The report attributes the 9-13% relative drop at 10 options and 11-17% at 26 options to model confusion, but increasing the option count from 4 to 10 or 26 lowers the random-guessing baseline from 25% to 10% or 3.8%. For a model that knows a fraction k of items and guesses uniformly on the rest, expected accuracy is k+(1-k)/N. For k=0.8, the baseline shift alone predicts relative drops of about 3.5% at N=10 and 5.0% at N=26 relative to N=4; for k=0.6 it predicts 8.6% and 12%. The observed drops are therefore compatible with a large contribution from the lowered chance baseline, and the residual confusion effect is not quantified. Please report chance-normalized accuracy, for example (acc - 1/N)/(1 - 1/N), or an equivalent model-based correction for every option-count condition, and state the per-condition raw scores. The cross-benchmark comparison in Figure 2 does not resolve this, because MMLU-Pro differs from WMDP-bio and GPQA in content difficulty as well as option count. Since this experiment is the direct evidence for H2 and Re-MMLU inherits the 10-option design, the issue is load-bearing for the central claim.
  2. [Section 3.3 and Section 5] The paper claims that current models perform similarly on Re-MMLU and MMLU-Pro with MAE below 5%, but all reported evaluations are on tinyRe-MMLU, a subset of MMLU, and not on the released full Re-MMLU dataset. Figure 7 only plots tinyRe-MMLU, and Section 5 states that current models perform similarly on Re-MMLU and MMLU-Pro without the 'tiny' qualifier. The extrapolation from tinyRe-MMLU to Re-MMLU requires evidence that the subset is representative, for example a correlation analysis in the style of tinyBenchmarks, or an evaluation on the full Re-MMLU set. As written, the Section 5 sentence overstates what the data show.
  3. [Section 2.1 and Figure 4] The scoring rule for paired questions is never defined. The text says the model must select the correct combination for each pair, which implies a joint score with chance success 1/16 for two four-option questions, while the Figure 4 caption says drops are calculated per individual question, which would keep the chance baseline at 1/4 per question. These two readings lead to different interpretations of every reported drop. If joint scoring is used, the pairing manipulation has the same chance-baseline confound as the distractor experiment; if per-question scoring is used, the drop is not separated from the overall accuracy level unless chance-corrected. Please state the scoring unit, report both pair-level and per-question accuracies, and supply confidence intervals or per-run variation.
minor comments (5)
  1. [Figure 2 and Figure 6] Figure 6 reports averages over three runs but does not show error bars or per-run spread, and Figure 2 has no confidence intervals at all; the number of items per benchmark is not stated, so claims about small differences between models (for example Gemini 1.5 Pro versus Mistral Large on WMDP-bio) may be within noise.
  2. [Section 2.3.2] The sentence 'We didn't observe any statistically significant improvement' is not accompanied by the test used, effect sizes, or confidence intervals; please report the actual numbers.
  3. [Section 3.1] There is a typo: 'MWDP-bio' should be 'WMDP-bio'.
  4. [Appendix A] The sample questions contain typos: 'Mycobaterium' should be 'Mycobacterium', and 'Hemaggultinin' should be 'Hemagglutinin'.
  5. [Section 3.3] The phrase 'capability barrier' is vague; please define what barrier means operationally or rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper's claims rest on empirical interventions and independent evaluations, not on fitted inputs or self-citations.

full rationale

The paper is an empirical study, not a derivational one. Its two hypotheses (H1: question pairing confuses LLMs; H2: more answer options lower performance) are formulated from observations on MMLU-Pro and then tested on separate benchmarks and separate model sets. The distractor experiment in Section 2.4 does suffer from a statistical confound: as the number of answer options increases from 4 to 26, the random-guess baseline falls from 1/4 to 1/26, which alone can produce part of the observed performance drop. This is a validity threat to the interpretation that the drop is caused by model confusion, but it is not circularity: the paper does not fit a parameter to the outcome and then rename that fit as a prediction, nor does it define the challenge metric in terms of the result. The Re-MMLU comparison in Section 3.3 is an independent evaluation on tinyMMLU against the external MMLU-Pro benchmark, with no fitted parameters feeding the comparison. There are no self-citations that carry the argument, and no uniqueness theorems or prior-work assumptions are imported to force the conclusions. The concern about the uncontrolled guessing baseline belongs in a correctness review, not a circularity score, because the paper's predictions would be equally falsifiable even if the baseline effect were large. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters and no new theoretical entities. It relies on domain assumptions about the validity of benchmark scores, the fairness of the paired-question scoring, and the neutrality of the chosen distractors. The new datasets Re-MMLU and tinyRe-MMLU are real artifacts, not postulated entities.

assumptions (3)
  • domain assumption The observed performance drops reflect genuine model confusion rather than artifacts of the response format or the lowered random-guess baseline.
    Section 2.4 adds city-name distractors and attributes the drop to model confusion without controlling for the reduced probability of guessing correctly among more options.
  • domain assumption The two-digit answer format for paired questions is scored fairly and does not introduce a significant output-format penalty.
    Section 2.1 paired questions require the model to output 'ANSWER: $NUMBER' with two digits; no analysis separates format errors from knowledge errors.
  • domain assumption The city names used as distractors are semantically neutral with respect to the biology questions and would not confuse human experts.
    Section 2.4 states they 'would not confuse human test-takers familiar with biology' but this is asserted, not tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Resurrecting saturated LLM benchmarks with adversarial encoding." pith.science (2026). https://pith.science/paper/4I63GVLR

@misc{pith2026250206738,
  author       = {Pith},
  title        = {Pith review of: Resurrecting saturated LLM benchmarks with adversarial encoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4I63GVLR}},
  note         = {Machine review of arXiv:2502.06738}
}
read the original abstract

Recent work showed that small changes in benchmark questions can reduce LLMs' reasoning and recall. We explore two such changes: pairing questions and adding more answer options, on three benchmarks: WMDP-bio, GPQA, and MMLU variants. We find that for more capable models, these predictably reduce performance, essentially heightening the performance ceiling of a benchmark and unsaturating it again. We suggest this approach can resurrect old benchmarks.

Figures

Figures reproduced from arXiv: 2502.06738 by the authors.

Figure 1
Figure 1. Example of converting two single questions into a paired multiple-choice question. See [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Relative performance drop on paired questions for WMDP-bio, GPQA and MMLU [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Alternative method for pairing questions. The resulting benchmark has the square of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Relative performance drops on two versions of paired-question WMDP-bio, calculated [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Relative performance drops on paired-question benchmarks: original vs. WMDP [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Relative performance drop on modified WMDP-bio with varying answer options. Each [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Scores of LLMs on MMLU-Pro, the original tinyMMLU, and tinyRe-MMLU. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages

  1. [1]

    (2024, July)

    Balestriero, R., Cosentino, R., & Shekkizhar, S. (2024, July). Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation(No. arXiv:2312.01648). arXiv. doi: 10.48550/arXiv.2312.01648

    arXiv doi:10.48550/arxiv.2312.01648 2024

  2. [2]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arxiv:2009.03300

    arXiv 2020

  3. [3]

    Kiela, D. (2023). Plotting progress in ai. Retrieved from https://contextual.ai/blog/ plotting-progress-in-ai/

    work page 2023

  4. [4]

    Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024). Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arxiv:2410.05229

    arXiv 2024

  5. [5]

    M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M

    Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024). tinybenchmarks: evaluating llms with fewer examples. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:34303-34326, 2024

    work page 2024

  6. [6]

    Wang, Z., Kodner, J., & Rambow, O. (2024). Evaluating llms with multiple problems at once: A new paradigm for probing llm capabilities. arXiv preprint arxiv:2406.10786v1

    work page Pith review arXiv 2024

  7. [7]

    Zheng, C., Zhou, H., Meng, F., Zhou, J., & Huang, M. (2023). Large language models are not robust multiple choice selectors. arXiv preprint arxiv:2309.03882. A Sample data A.1 Original pairing method, WMDP-bio First question Mutations in what gene make the influenza virus resistant to antivirals of the adamantane class? 1 - Neuraminidase (NA) 2 - Polymera...

    arXiv 2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.