Pith. sign in

REVIEW 4 major objections 6 minor 79 references

Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Debate-driven evaluation transforms QA benchmarks into contamination-resistant reasoning tests.

desk verdict A plausible debate-based evaluation protocol with a useful released benchmark, but the central contamination-resistance claim rests on a small, likely confounded comparison. read the letter →

arxiv 2507.17747 v2 pith:SN7XSC6R submitted 2025-07-23 cs.CL cs.AI

Linbo Cao , Jinman Zhao This is my paper
classification cs.CLcs.AI
keywords debate-drivenevaluationdatacontaminationLLM-as-judgemulti-agentdebatebenchmarkrecyclingTrueSkillrankingQAmemorizationvsreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard QA benchmarks can be recycled into harder, contamination-resistant evaluations by turning each question into a structured adversarial debate. One model defends the official answer while a second model must propose and defend an alternative, and an answer-blind judge picks the stronger case after several rounds. The central evidence is a fine-tuning experiment: a Llama 3.1 8B model trained directly on the 50 test questions jumped from 50% to 82% accuracy on ordinary QA, yet its debate win rate fell against both its original self and a stronger opponent. If the approach holds, model rankings can be produced without writing new benchmark questions, and memorized test-set knowledge stops inflating scores.

What carries the argument

The load-bearing mechanism is the structured adversarial debate: a Pro model receives the official answer and must defend it, a Con model is told the official answer is wrong and must construct and defend an alternative, and a judge that never sees the correct answer decides which side argued better, with 'continue' extending the debate up to five rounds and Pro winning by default if no verdict emerges. Double round-robin pairings with role reversals cancel positional bias, and TrueSkill ratings are used to keep reference-model scores stable as new models join. This mechanism converts a memorization-friendly multiple-choice item into a multi-round reasoning contest in which recalling the answer is necessary but not sufficient.

What would settle it

Run the same double round-robin tournament but with a judge that always outputs 'positive' after every debate: if the resulting ranking matches the paper's main ranking, the results are driven by role and format instead of argument quality. A second concrete check is to swap the official answer for a plausible but wrong answer in the Pro prompt; if Pro's win rate does not drop, the judge is rewarding confidence or role, not correctness.

Watch

Extended reading notes

Core claim

The central claim is that debate-style evaluation exposes shallow memorization that standard accuracy hides: a model fine-tuned on the test set gains 32 points of QA accuracy but does not improve, and even declines, in debate settings. The paper reports that the fine-tuned model's overall debate win rate dropped from 0.50 to 0.46 against the original model and from 0.17 to 0.16 against DeepSeek V3, with questioning ability falling sharply from 0.12 to 0.06. Across eleven models, debate outcomes formed a nearly transitive ranking (98%+ consistency), and six of seven judge models, including a contaminated one, produced identical debater rankings. The paper concludes that debate-driven evaluation distinguishes genuine reasoning from memorized answers without requiring new datasets or contamination filtering.

Load-bearing premise

An LLM judge that never sees the correct answer can reliably rank debaters by argument quality, so debate win rates reflect reasoning ability rather than persuasiveness, judge bias, or the Pro-favoring fallback rules.

Editorial extensions

If this is right

  • Existing QA benchmarks such as MMLU-Pro and GPQA can be repurposed as harder debate benchmarks, extending their useful life and avoiding costly new dataset curation.
  • A model trained on test data will not gain, and may lose, standing in debate rankings, so debate scores provide a contamination-resistant complement to raw accuracy.
  • The near-perfect transitivity of debate win rates lets a new model be placed between two reference models through a logarithmic number of comparisons rather than a full round robin.
  • Since even weak judges reproduce the same debater ordering, the evaluation can in principle remain usable as models grow more capable than the judges.
  • TrueSkill ratings let new models enter the benchmark without reshuffling the scores of existing reference models, unlike Elo or Bradley–Terry.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same debate protocol could be applied to tasks without a single official answer, such as code correctness or proof verification, where the 'Con' side constructs counterexamples; the paper does not test this.
  • Beyond the paper: because the judge is blind to the answer and the Pro side wins after five undeclared rounds, a hyper-persuasive but wrong Con could exploit the format; a useful stress test would swap the official answer for a plausible wrong one and check whether Pro still wins at the same rate.
  • Beyond the paper: the 50-question MMLU-Pro sample in the main tournament leaves open whether the transitivity and judge-robustness findings persist on larger, more diverse question sets; the paper's GPQA appendix is a step in that direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a debate-driven evaluation paradigm that converts QA benchmarks into structured adversarial debates: a Pro model is given the official answer to defend, a Con model is instructed that the official answer is incorrect and must argue for an alternative, and a judge model blind to the correct answer decides the winner. The authors evaluate 11 models on 50 MMLU-Pro questions using a double round-robin protocol with GPT-4o as judge, report debate win counts and rankings, and present a fine-tuning experiment in which LoRA fine-tuning Llama 3.1 8B on the test set improves QA accuracy from 50% to 82% but does not improve (and may slightly harm) debate performance. They also run a judge-variation study with seven judges, a confirmatory GPQA experiment, and a TrueSkill-based ranking analysis. The central claim is that this paradigm 'directly addresses data contamination,' recycling existing QA data into a harder, contamination-resistant evaluation.

Significance. If the central claim were established, the framework would be a valuable and low-cost complement to existing dynamic and contamination-resistant benchmarks. The paper has concrete strengths: it releases code and a large public corpus of 5,500 debate transcripts, provides detailed prompts (Appendix E), uses a double round-robin design to reduce positional bias, and complements the main results with a full GPQA evaluation. The TrueSkill stability analysis (Section 5.6, Appendix D) is a useful practical contribution. However, the headline contamination-resistance result is currently supported by small, statistically unquantified differences on 50 questions and is confounded by the role-assignment protocol, so the significance of the empirical claims is not yet at the level the abstract and Section 6 assert.

major comments (4)
  1. [§5.4, Table 2] The central claim that debate evaluation 'directly addresses data contamination' rests on win-rate differences that are not statistically supported. The key comparisons are 0.50→0.46 (overall, vs. Llama 3.1 8B), 0.17→0.16 (overall, vs. DeepSeek V3), and 0.12→0.06 (questioning, vs. DeepSeek V3). With 50 questions, the standard error of a binomial win rate near 0.5 is about 0.07, so 0.46 is within one standard error of 0.50; the 0.17→0.16 difference is negligible; and 0.12→0.06 corresponds to roughly three debates, which is not significant. No confidence intervals, p-values, or effect-size measures are reported. The conclusion that fine-tuning 'failed to improve, and can even harm' debate performance is therefore not established by the data as presented. The authors should report exact counts, confidence intervals, and a paired significance test (e.g., McNemar's test if the same 50 questions are used in both conditions).
  2. [§3, Appendix E.1/E.2] The debate protocol confounds the contamination-resistance interpretation. Pro is handed the official answer (prompt E.1), and Con is explicitly told 'the system has rejected the answer as incorrect' (prompt E.2). A model that has memorized the official answer is therefore forced to argue against knowledge it holds, and its Con win rate can drop simply because of knowledge conflict rather than because it 'lacks deeper comprehension.' The fine-tuned Llama 3.1 8B knows the official answer on roughly 82% of the 50 questions, and indeed its Con win rate against DeepSeek V3 drops from 0.12 to 0.06 while its defending rate rises from 0.22 to 0.26—exactly the pattern predicted by knowledge conflict. The paper does not provide a control in which the Pro side is not given the answer, or in which models can choose which side to argue, so the 'shallow memorization vs. genuine understanding' conclusion in Section 6 is not uniquely supported. A control condition, or at least an analysis separating questions where the fine-tuned model is correct from those where it is wrong, is needed.
  3. [§5.5, Table 3 and Appendix B] The claim that 'even weaker judges can reliably differentiate stronger debaters' relies on a post-hoc exclusion of the weakest judge. Mistral 7B is excluded because it failed to follow the output format, and Table 4 shows that all of its pairwise win rates are approximately 0.50—the exact outcome expected from the fallback rule that defaults malformed judge outputs to 'positive.' The paper should pre-specify a judge-eligibility criterion (e.g., format compliance rate above a threshold) and report results both with and without the excluded judge. In addition, the main experiment uses GPT-4o as the judge while GPT-4o is also one of the eleven debaters; no analysis tests for judge self-preference or for contamination of the judge on MMLU-Pro. Since the judge is 'blind' only to the official answer, a judge that has memorized the test set could still favor arguments aligned with the memorized answer, which would systematically favor the Pro side. The authors should test judge contamination (e.g., by comparing outcomes on questions the judge is likely to know vs. not) and either exclude the judge from the debater pool or report a sensitivity analysis.
  4. [§5.1, §5.3] The headline rankings in Table 1 and the reported transitivity of the combined heatmap are based on only 50 questions per pairwise matchup. While the aggregate win counts in Table 1 come from 5,500 debates, each individual pairwise cell in Figure 5 is a win rate over 50 questions, giving a standard error of about 0.07 per cell. The 'near-perfect transitivity' claim (98%+) is computed over 55 pairings, but with this per-cell noise the number of expected transitivity violations under random outcomes should be reported; the single observed violation is not strong evidence of a stable hierarchy. The paper should provide confidence intervals for the pairwise win rates and a statistical test for transitivity, or explicit bounds on the number of violations expected by chance.
minor comments (6)
  1. [§6] There is a typo in the second paragraph: 'remedy forbenchmark saturation' should be 'remedy for benchmark saturation.'
  2. [Abstract and §1] The phrase 'even weaker judges can reliably differentiate stronger debaters' is too strong given the post-hoc exclusion of Mistral 7B and the lack of a statistical test; consider softening to 'six of seven judges, after excluding one that failed format compliance, produced identical rankings.'
  3. [Table 2] The 'theoretical win rate for a self-match' notation is unclear: the self-match row reports 0.50 overall with em-dashes for role-specific rates, but no self-match was actually run. Please clarify how the 0.50 value is derived and whether the comparison 'vs. Llama 3.1 8B' uses the original model as the sole opponent across all 50 questions.
  4. [§4] The term '0-shot CoT' is used without a definition or prompt template; please provide the exact chain-of-thought prompt used for the QA accuracy measurement, or cite it.
  5. [Appendix D] Tables 6 and 7 use angle brackets around the new model's score (e.g., '<315.20>'), which is nonstandard and could be confused with a confidence interval; please use a footnote or a different notation.
  6. [§5.5, Table 3] The judge models are ordered 'strongest to weakest' in Table 3, but the ordering criterion is not specified. Please state how judge capability was determined (e.g., by QA accuracy on the same 50 questions) so the reader can interpret the claim about weaker judges.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: debate win rates are measured outcomes, and the fine-tuning comparison is an experiment, not a constructional equivalence.

full rationale

The paper's central quantities—QA accuracy, debate win counts, pairwise win rates, and judge-tournament rankings—are all measured outcomes obtained from a defined protocol, not quantities defined in terms of the claims they support. The contamination experiment (Section 5.4) compares measured debate win rates of a LoRA-fine-tuned Llama 3.1 8B against its own baseline and against DeepSeek V3; the reported decline is an empirical observation, not a fitted parameter renamed as a prediction. The ranking system in Section 3 and 5.6 (TrueSkill on pairwise debate outcomes) is a standard rating procedure applied to measured matches; anchoring new models to stored reference debates is incremental-rating design, not self-definition. The weak-to-strong judge claim rests on new tournaments (Section 5.5) with seven judge models and replicated rankings, and the external citation to Khan et al. (2024) is corroborative rather than load-bearing. There are no self-citations by the present authors, no imported uniqueness theorem, and no ansatz smuggled in via citation. The main threats to the contamination claim—the forced-negation protocol (Con is told the official answer is incorrect, so a model that has memorized the correct answer faces knowledge conflict rather than pure shallow-memorization failure), the 50-question sample size (win-rate standard errors around 0.07), and untested judge contamination—are methodological and statistical correctness risks, not cases where an output is equivalent to its input by construction. The Limitations section itself concedes judge-bias investigation is needed, which further supports classifying these issues as correctness risks. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The protocol has no fitted parameters in the physics sense, but several hand-chosen design constants and modeling assumptions carry the results. The TrueSkill priors and round limits affect the reported rankings, and the contamination experiment relies on an unvalidated proxy.

free parameters (3)
  • TrueSkill priors (mu, sigma, beta, tau) = 25, 8.333, 4.5, 0.01
    Hand-chosen in Section 5.6 to match observed win counts; not fitted by optimization, but the choice affects all benchmark scores.
  • Debate round count = 2 to 5
    Design choice in Section 3 based on prior work; the cap of 5 and the Pro-victory default favor one side.
  • MMLU-Pro subset size = 50 random questions
    Section 4. Sample is small; seed not reported. All headline claims depend on n=50.
assumptions (5)
  • domain assumption LLM judges can assess argument quality independently of correctness.
    Core premise of the evaluation protocol; stated in Section 3 and the judge prompt in Appendix E.3. Not validated in the paper beyond comparing judge rankings.
  • domain assumption Fine-tuning on the test set via LoRA is a valid proxy for real-world data contamination.
    Section 5.4 equates deliberate training on the 50 evaluation questions with contamination. Real contamination is partial and unlabeled; the proxy is extreme.
  • domain assumption The official MMLU-Pro answers are correct ground truth.
    Used to assign Pro and Con roles; if an official answer is wrong, the debate setup is misleading. Section 3.
  • domain assumption Transitivity observed on 50 questions generalizes to other datasets and larger model sets.
    Section 5.3 uses near-perfect transitivity to justify O(log n) partial tournaments and benchmark stability; based on a single 50-question sample.
  • ad hoc to paper After five rounds without a winner, Pro is awarded victory; malformed judge outputs default to positive.
    Section 3 and Section 5.5. These rules bias outcomes toward the Pro side and toward the official answer, and are not justified by data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks." pith.science (2026). https://pith.science/paper/SN7XSC6R

@misc{pith2026250717747,
  author       = {Pith},
  title        = {Pith review of: Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SN7XSC6R}},
  note         = {Machine review of arXiv:2507.17747}
}
read the original abstract

As frontier language models increasingly saturate standard QA benchmarks, concerns about data contamination, memorization, and escalating dataset creation costs persist. We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversarial debates--where one model is given the official answer to defend, and another constructs and defends an alternative answer--adjudicated by a judge model blind to the correct solution. By forcing multi-round argumentation, this approach substantially increases difficulty while penalizing shallow memorization, yet reuses QA items to reduce curation overhead. We make two main contributions: (1) an evaluation pipeline to systematically convert QA tasks into debate-based assessments, and (2) a public benchmark that demonstrates our paradigm's effectiveness on a subset of MMLU-Pro questions, complete with standardized protocols and reference models. Empirical results validate the robustness of the method and its effectiveness against data contamination--a Llama 3.1 model fine-tuned on test questions showed dramatic accuracy improvements (50% -> 82%) but performed worse in debates. Results also show that even weaker judges can reliably differentiate stronger debaters, highlighting how debate-based evaluation can scale to future, more capable systems while maintaining a fraction of the cost of creating new benchmarks. Overall, our framework underscores that "pretraining on the test set is no longer all you need," offering a sustainable path for measuring the genuine reasoning ability of advanced language models.

Figures

Figures reproduced from arXiv: 2507.17747 by the authors.

Figure 1
Figure 1. Illustration of how debates might encourage deeper reasoning, distinguishing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Single debate pipeline showing question transformation, role assignment, multi [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Double round-robin evaluation process showing model pairings, role alternation, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of Total Debate Wins vs. Defending-Only and Questioning-Only [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Pairwise head-to-head win rate heatmaps across all 11 models. Each cell [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Head-to-head win rate heatmap (Questioning-Only). Each entry reflects win rates [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Pairwise win rate heatmaps for each judge model. Each subfigure (a)–(f) shows [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 37 canonical work pages

  1. [1]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. Online at https://www.anthropic.com/news/claude-3-5-sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  2. [2]

    Claude 3.5 haiku

    Anthropic. Claude 3.5 haiku. https://www.anthropic.com/claude/haiku, 2025

    work page 2025

  3. [3]

    ARC-AGI-2 + ARC Prize 2025 is Live! Blog Post, https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025, March 24 2025

    ARC Prize Foundation and Greg Kamradt. ARC-AGI-2 + ARC Prize 2025 is Live! Blog Post, https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025, March 24 2025

    work page 2025

  4. [4]

    Benchmarking foundation models with language-model-as-an-examiner

    Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with language-model-as-an-examiner. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, N...

    arXiv 2023

  5. [5]

    Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms

    Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 67--93. Association for Computational Linguistics, 2024. URL https://aclanthol...

    work page 2024

  6. [6]

    Adversarial multi-agent evaluation of large language models through iterative debates

    Chaithanya Bandi and Abir Harrasse. Adversarial multi-agent evaluation of large language models through iterative debates. arXiv preprint arXiv:2410.04663, 2024. URL https://arxiv.org/abs/2410.04663

    arXiv 2024

  7. [7]

    Flageval

    Beijing Academy of Artificial Intelligence . Flageval. Website, 2025. URL https://flageval.baai.ac.cn/

    work page 2025

  8. [8]

    Cot-kinetics: A theoretical modeling assessing lrm reasoning process, 2025

    Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. Cot-kinetics: A theoretical modeling assessing lrm reasoning process, 2025. URL https://arxiv.org/abs/2505.13408

    arXiv 2025

Show all 79 references
  1. [9]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952. doi:10.2307/2334029. URL https://www.jstor.org/stable/2334029

    1952

  2. [10]

    Sparks of artificial general intelligence: Early experiments with gpt-4, 2023

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with g...

    2023 arXiv

  3. [11]

    The role of deductive and inductive reasoning in large language models, 2025

    Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. The role of deductive and inductive reasoning in large language models, 2025. URL https://arxiv.org/abs/2410.02892

    2025 arXiv

  4. [12]

    Are we on the right way for evaluating large vision-language models? In A

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and ...

    2024

  5. [13]

    Jordan, Joseph E

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the Fort...

    2024

  6. [14]

    Arc prize 2024: Technical report, 2024

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report, 2024. URL https://arxiv.org/abs/2412.04604

    2024 arXiv

  7. [15]

    Abstraction and reasoning corpus for artificial general intelligence (arc-agi), 2019

    François Chollet. Abstraction and reasoning corpus for artificial general intelligence (arc-agi), 2019. URL https://github.com/fchollet/ARC-AGI

    2019

  8. [16]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. U...

    arXiv

  9. [17]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024. URL https://arxiv.org/abs/2412.19437

    2024 arXiv

  10. [18]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  11. [19]

    Investigating data contamination in modern benchmarks for large language models

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of ...

    2024 doi

  12. [20]

    Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms

    Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp.\ 236--247, 2025 a

    2025

  13. [21]

    Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices

    Junhao Dong, Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. Improving zero-shot adversarial robustness in vision-language models by closed-form alignment of adversarial path simplices. In Forty-second International Conference on Machine Learnin...

    2025

  14. [22]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2305.14325

    2024 arXiv

  15. [23]

    Arpad E. Elo. The rating of chessplayers : past and present. Ishi Press International, Bronx, NY, 2nd printing, 2008 edition, 1978. ISBN 9780923891275. URL https://lccn.loc.gov/2010549499

    2008

  16. [24]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, L...

    2024 arXiv

  17. [25]

    Time travel in llms: Tracing data contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=2Rwq6c3tvr

    2024

  18. [26]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  19. [27]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge. CoRR, abs/2411.15594, 2024. URL https://arxiv.org/a...

    2024 arXiv

  20. [28]

    Passonneau, Megan Ung, and Adina Williams

    Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using smart filtering of benchmark datasets, 2024. URL https://arxiv.org/abs/2410.20245

    2024 arXiv

  21. [29]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL h...

    2021

  22. [30]

    Trueskill : A bayesian skill rating system

    Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill : A bayesian skill rating system. In B. Sch\" o lkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper_files/paper/20...

    2006

  23. [31]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  24. [32]

    Ai safety via debate

    Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. CoRR, abs/1805.00899, 2018. URL https://arxiv.org/abs/1805.00899

    2018 arXiv

  25. [33]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. URL https://arxiv.org/abs/2310.06825

    2023 arXiv

  26. [34]

    Jiang, Alexandre Sablayrolles, Antoine Roux, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. URL https://arxiv.org/abs/2401.04088

    2024 arXiv

  27. [35]

    Bowman, Tim Rockt \"a schel, and Ethan Perez

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rockt \"a schel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. In Forty-first International Conference on Ma...

    2024

  28. [36]

    Debate helps weak-to-strong generalization

    Hao Lang, Fei Huang, and Yongbin Li. Debate helps weak-to-strong generalization. In Proceedings of the AAAI 2025 Special Track on AI Alignment (Oral presentation), 2025. URL https://arxiv.org/abs/2501.13124

    2025 arXiv

  29. [37]

    Llms-as-judges: A comprehensive survey on llm-based evaluation methods

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: A comprehensive survey on llm-based evaluation methods. CoRR, abs/2412.05579, 2024 a . URL https://arxiv.org/abs/2412.05579

    2024 arXiv

  30. [38]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual mee...

    2024 arXiv

  31. [39]

    A debate-driven experiment on llm hallucinations and accuracy

    Ray Li, Tanishka Bagade, Kevin Martinez, Flora Yasmin, Grant Ayala, Michael Lam, and Kevin Zhu. A debate-driven experiment on llm hallucinations and accuracy. arXiv preprint arXiv:2410.19485, 2024 c . URL https://arxiv.org/abs/2410.19485

    2024 arXiv

  32. [40]

    Manning, Christopher R \' e , Diana Acosta - Navas, Drew A

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R \' e , Diana Acosta ...

    2023

  33. [41]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al - Onaizan, Mohit Bansal, and Yun - Nung Chen (eds.), Proceedings of th...

    2024

  34. [42]

    An empirical analysis on large language models in debate evaluation

    Xinyi Liu, Pinxin Liu, and Hangfeng He. An empirical analysis on large language models in debate evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL) -- Short Papers, pp.\ 470--487, Bangkok, Thailand, 2024. Association fo...

    2024

  35. [43]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

    Meta AI . The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation . https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025

    2025

  36. [44]

    Discursive socratic questioning: Evaluating the faithfulness of language models' understanding of discourse relations

    Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, and Min-Yen Kan. Discursive socratic questioning: Evaluating the faithfulness of language models' understanding of discourse relations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Me...

    2024 doi

  37. [45]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs , 2025

    Microsoft . Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs , 2025. URL https://arxiv.org/abs/2503.01743

    2025 arXiv

  38. [46]

    Cheaper, better, faster, stronger

    MistralAI. Cheaper, better, faster, stronger. https://mistral.ai/news/mixtral-8x22B, 2024

    2024

  39. [47]

    Mistral large

    MistralAI. Mistral large. https://mistral.ai/news/mistral-large, 2025

    2025

  40. [48]

    Evaluating the performance of large language models via debates

    Behrad Moniri, Hamed Hassani, and Edgar Dobriban. Evaluating the performance of large language models via debates. arXiv preprint arXiv:2406.11044, 2024. URL https://arxiv.org/abs/2406.11044

    2024 arXiv

  41. [49]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774

    2023 arXiv

  42. [50]

    GPT-4o System Card

    OpenAI. GPT-4o System Card . arXiv preprint arXiv:2410.21276, 2024 a . URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  43. [51]

    GPT-4o mini: advancing cost-efficient intelligence

    OpenAI. GPT-4o mini: advancing cost-efficient intelligence . https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024 b

    2024

  44. [52]

    Openai o1 system card, 2024 c

    OpenAI. Openai o1 system card, 2024 c . URL https://arxiv.org/abs/2412.16720

    2024 arXiv

  45. [53]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=KS8mIvetg2

    2024

  46. [54]

    Mapping global dynamics of benchmark creation and saturation in artificial intelligence

    Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13 0 (1): 0 6793, 2022. doi:10.1038/s41467-022-34591-0. URL https://www.nature.c...

    2022 doi

  47. [55]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, et al. Humanity's last exam. CoRR, abs/2501.14249, 2025. URL https://doi.org/10.48550/arXiv.2501.14249

    2025 doi arXiv

  48. [56]

    Introducing gemini 2.0: our new ai model for the agentic era

    Sundar Pichai, Demis Hassabis, Koray Kavukcuoglu, and Google DeepMind. Introducing gemini 2.0: our new ai model for the agentic era. Blog post on The Keyword, 2024. URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/#gemini-2-0

    2024

  49. [57]

    Multi-layered evaluation using a fusion of metrics and LLMs as judges in open-domain question answering

    Rashin Rahnamoun and Mehrnoush Shamsfard. Multi-layered evaluation using a fusion of metrics and LLMs as judges in open-domain question answering. In Proceedings of the 31st International Conference on Computational Linguistics, pp.\ 6088--6104, Abu Dhabi, UAE, 2025. Associati...

    2025

  50. [58]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  51. [59]

    Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark

    Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 107...

    2023 doi

  52. [60]

    Pretraining on the test set is all you need, 2023

    Rylan Schaeffer. Pretraining on the test set is all you need, 2023. URL https://doi.org/10.48550/arXiv.2309.08632. arXiv preprint arXiv:2309.08632

    2023 doi arXiv

  53. [61]

    Detecting pretraining data from large language models

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/fo...

    2024

  54. [62]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. URL https://openreview.net/forum?id=uyTL5Bvosj

    2023

  55. [63]

    MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs

    Saeid Asgari Taghanaki, Aliasgahr Khani, and Amir Khasahmadi. MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs . In Proceedings of the 38th Conference on Neural Information Processing Systems, NeurIPS 2024, Safe Generative AI Track, 2024. URL https://...

    2024 arXiv

  56. [64]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Confe...

    2019

  57. [65]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9...

    2019

  58. [66]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://o...

    2023

  59. [67]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024

  60. [68]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu...

    2022

  61. [69]

    Livebench: A challenging, contamination-free LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and M...

    2025

  62. [70]

    QUD eval: The evaluation of questions under discussion discourse parsing

    Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. QUD eval: The evaluation of questions under discussion discourse parsing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp...

    2023 doi

  63. [71]

    Benchmark data contamination of large language models: A survey, 2024

    Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey, 2024. URL https://arxiv.org/abs/2406.04244

    2024 arXiv

  64. [72]

    Kieval: A knowledge-grounded interactive evaluation framework for large language models

    Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-grounded interactive evaluation framework for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024 arXiv

  65. [73]

    Auto-arena: Automating llm evaluations with agent peer battles and committee discussions, 2024

    Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, and Lidong Bing. Auto-arena: Automating llm evaluations with agent peer battles and committee discussions, 2024. URL https://arxiv.org/abs/2405.20267. arXiv:2405.20267

    2024 arXiv

  66. [74]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Process...

    2023

  67. [75]

    Dyval: Dynamic evaluation of large language models for reasoning tasks

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=gjfOL9z5Xr

    2024

  68. [76]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  69. [77]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  70. [78]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  71. [79]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.