Pith. sign in

REVIEW 4 major objections 4 minor 69 references

Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DEEVO evolves prompts by debate and Elo scoring, outperforming manual and automated baselines with no ground-truth labels.

desk verdict A plausible prompt-optimization pipeline with one genuinely encouraging close-ended result, but the open-ended superiority claim rests on a likely circular LLM-judge setup. read the letter →

arxiv 2506.00178 v2 pith:XI75II53 submitted 2025-05-30 cs.AI cs.NE

Anirudh Nair , Adi Banerjee , Laurent Mombaerts , Matthew Hagen , Tarik Borogovac This is my paper
classification cs.AIcs.NE
keywords promptoptimizationmulti-agentdebateEloratingevolutionaryalgorithmlabel-freesubjectivequalityLLMasjudgegeneticoperators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt engineering is a bottleneck for LLM applications, especially when quality is subjective and no numerical score exists. This paper argues that an evolutionary loop can remove that bottleneck: DEEVO evolves a population of prompts by having LLMs debate the outputs of prompt pairs, converting the debate verdict into Elo-rating updates, and then combining winning prompt elements through debate-informed crossover and mutation. The authors claim that this label-free loop outperforms manual prompt engineering and existing optimizers on both closed-ended tasks with ground truth and open-ended tasks judged purely by an LLM. If true, DEEVO would let practitioners continuously improve prompts for subjective tasks without designing a metric, collecting labels, or exposing model internals.

What carries the argument

The load-bearing mechanism is the debate-to-Elo loop. In each generation, prompt pairs generate responses to a sampled task input; a fixed LLM runs a three-round structured debate over the two responses (opening statements, rebuttals, final verdict), producing both a winner and a transcript. The verdict updates each prompt's Elo rating, a tournament score updated by pairwise comparison outcomes, through the standard formula $r'_i = r_i + K(s_i - e_i)$, where $e_i$ is the expected score from the logistic comparison, while a crossover agent uses the transcript to build an offspring that combines the winner's strong elements with salvageable parts of the loser; a mutation agent then optionally adds, edits, removes, or restructures instructions. Two quotas—a newcomer quota and veteran selection by Elo—keep the population from freezing out fresh candidates or letting stale ratings dominate. The debate transcript also feeds the crossover step, making the genetic operators informed rather than arbitrary splicing.

What would settle it

Run DEEVO on a task where human preference ratings are available, then compare the Elo-selected prompts' outputs against human judgments: if the debate winners are not preferred by humans significantly more often than chance, the loop is optimizing the judge's bias rather than task quality.

Watch

Extended reading notes

Core claim

DEEVO's central claim is that Elo ratings derived from multi-agent debate verdicts are a sufficient fitness signal to drive prompt evolution in the absence of ground truth. On the closed-ended ABCD dialogue benchmark the best DEEVO prompt reaches 83.7% accuracy versus 77.3% for the SPO baseline and 74.5% for chain-of-thought, and on BBH-Navigate it reaches an F1 of 97.0, nearly identical to SPO's 97.2; on the open-ended MT-Bench writing, roleplay, and humanities categories DEEVO beats SPO on every tested LLM, with win rates from roughly 66% to 85%. The paper also reports positive point-biserial correlations between Elo and held-out accuracy, and an ablation showing that replacing multi-agent debate with a single-pass judge lowers ABCD accuracy from 83.7% to 74.1%, which the authors attribute to the debate's resistance to evaluation bias.

Load-bearing premise

The load-bearing premise is that the large language model's debate verdicts and the judge used for open-ended evaluation reflect genuine output quality rather than the judge's own stylistic biases, so that winning the tournament means actually being better.

Editorial extensions

If this is right

  • Subjective-quality tasks that currently need human metric design can be optimized using only LLM debate verdicts as the fitness signal.
  • Elo ratings rise with held-out task performance across generations, so the same ratings used for selection can double as a performance monitor.
  • Debate-informed crossover yields offspring that keep coherent prompt structure, unlike random string splicing in earlier evolutionary prompt optimizers.
  • The ablation against a single-pass judge shows evaluation quality is a first-order determinant of prompt evolution success.
  • Stronger LLMs used for debate, crossover, and mutation translate into better final prompts, implying DEEVO improves automatically as foundation models improve.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same debate-plus-Elo loop could plausibly evolve an entire multi-agent team—number of agents, roles, and inter-agent prompts—rather than a single system prompt, since the evaluation mechanism does not depend on task-specific metrics.
  • If the judge LLM and the generator share blind spots, the loop may converge to outputs that please the judge but not human users; inserting occasional human preference checks into selection would test whether this drift occurs.
  • Because the debate transcript is the only channel carrying quality information, changing debate parameters (rounds, temperature, or adding a devil's advocate) should measurably change the final prompt, offering a cheap sensitivity experiment.
  • Transfers between similar tasks may let prompts optimized on one domain bootstrap a new domain faster than starting from random populations, but DEEVO's current design does not reuse old populations across tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DEEVO, an evolutionary prompt optimization method in which a population of prompts is evaluated through multi-agent debates between LLMs, with Elo ratings used as a fitness proxy to select parents for crossover and mutation. The authors claim that DEEVO significantly outperforms manual prompt engineering and existing prompt optimizers on both closed-ended tasks (ABCD, BBH-Nav) and open-ended tasks (MT-Bench) while requiring no ground-truth labels. The paper includes ablations for model choice and for the debate evaluation, and it reports winrates on MT-Bench.

Significance. If the central claims are correct, DEEVO would be a valuable label-free prompt optimizer for subjective tasks, a genuinely useful contribution to the prompt-engineering literature. The method is clearly described, and the inclusion of two closed-ended benchmarks with ground-truth metrics is a strength that partially validates the Elo fitness proxy. However, the evidence in the paper is mixed. On BBH-Nav, DEEVO does not outperform SPO; the open-ended evaluation is limited to a single comparison and is vulnerable to self-preference circularity because the judge model is not named; and the reported Elo-accuracy correlations are weak. These issues bear directly on the paper's headline claim.

major comments (4)
  1. [Section 4.2, Table 1] The abstract and Section 1 claim that DEEVO 'significantly outperforms' state-of-the-art optimization approaches on close-ended tasks, but the paper's own data contradict this on BBH-Nav: SPO achieves an F1 of 97.2 while DEEVO achieves 97.0, and the text explicitly concedes that 'DEEVO and SPO perform nearly identically.' The subsequent statement that 'we see statistical significance when comparing the performance difference of DEEVO to all other methods' is therefore not supported for the SPO comparison on BBH-Nav. The authors should revise the claim to acknowledge that DEEVO is comparable to SPO on BBH-Nav and significantly better on ABCD, and they should temper the abstract accordingly.
  2. [Section 4.1 and Section 4.2, Table 3] The open-ended evaluation on MT-Bench is incomplete and potentially circular. The judge model is never specified; the text says only that 'the same LLM-judge prompt from the original MT-Bench paper' is used. Because DEEVO's fitness signal is generated by Claude-3.5-Sonnet-V2 multi-agent debates and the evaluated models include Claude-3.5-Sonnet-V2, an unspecified Claude-family judge would make the reported winrates reflect self-preference rather than independent quality. The paper's own Section 5 warns that 'models develop their own implicit evaluation criteria without any human intervention.' To support the open-ended claim, the authors must state the exact judge model, add at least one cross-model judge or human evaluation, and include comparisons to Direct, CoT, PromptBreeder, or BRIDGE on MT-Bench.
  3. [Section 4.2] The reported point-biserial correlations between Elo and accuracy (r = 0.137 for average Elo and r = 0.156 for maximum Elo) are weak, even if statistically significant. A correlation of this size explains only about 2% of the variance, which does not strongly support the assertion that 'Elo can serve as a reliable proxy' for ground-truth fitness. The paper should report the number of observations behind these correlations, provide confidence intervals, and discuss whether the relationship is monotonic across generations; otherwise the validation claim is materially overstated.
  4. [Section 4.1] The setup says 'we adopt MT-Bench for comparison with SPO and Direct,' but the results reported in Table 3 and the surrounding text contain only DEEVO versus SPO winrates; no Direct results are shown for MT-Bench. This missing baseline makes it impossible to assess whether DEEVO improves over simple manual prompting on open-ended tasks, which is part of the abstract's core claim. The authors should add the missing comparison or clearly state why it was omitted.
minor comments (4)
  1. [Figure 1] The caption contains a duplicated phrase: 'Each output is then is then paired' should read 'Each output is then paired.'
  2. [Section 4.2] The p-values in Table 1 are not tied to a stated statistical test, nor is the number of runs (three) and any multiple-comparison correction described; with only three runs, the reliability of these p-values should be discussed.
  3. [Section 4.1] The hyperparameters (mutation rate 0.4, five generations, three debate rounds) are presented without sensitivity analysis; the authors acknowledge this in the text, but even a small sensitivity table or a sentence of justification would strengthen the practical guidance.
  4. [Section 2.3 / Reference [26]] The text uses the acronym 'ERRL' for 'Elo-Rating Based Reinforcement Learning,' but the cited work is titled 'ELO-Rated Sequence Rewards'; please reconcile the acronym and title for accuracy.

Circularity Check

1 steps flagged · score 5.0 of 10

Open-ended MT-Bench winrates are measured by the same Claude model family that supplies DEEVO's debate fitness signal, making the open-ended superiority claim partly self-referential.

  1. fitted input called prediction [Section 3.1 Step 2 and Section 4.1/4.2 (Open-Ended Tasks)]
    "Step 2: Evaluation ... These responses ... are then evaluated through a structured debate process: ... After d rounds, the LLM renders a final verdict declaring either response ri or rj as superior. ... We use the Claude-3.5-Sonnet-V2-20241022 model ... for the two debating agents in the multi-agent debate module, but a temperature of 0 for the LLM-judge that makes the final judgment as per prior work [27]. ... For the open-ended MT-Bench, we use winrates as the metric for evaluation based on the LLM-judge prompt from the original paper for pairwise comparison."

    DEEVO's selection pressure is supplied by a multi-agent debate whose final verdict is rendered by Claude-3.5-Sonnet-V2, and the paper's implementation section says Claude-3.5-Sonnet-V2 is used for 'all the other methods' with no other judge model specified for the MT-Bench comparison. The open-ended winrates are therefore produced by the same model family's comparative preferences that the evolutionary loop was optimizing. The reported winrates largely measure DEEVO's ability to satisfy Claude's own implicit evaluation criteria rather than any independent standard of prompt quality. This is a validation-set echo of the fitness signal, not an independent prediction. The close-ended ABCD and BBH-Nav accuracy results are externally grounded, so the circularity is partial.

full rationale

The framework is not definitionally circular in its close-ended evaluation: Elo is updated from debate winners via Algorithm 2, and ABCD and BBH-Nav are scored against ground-truth accuracy and F1, which are independent of the LLM judge. The open-ended claim is where the derivation chain tightens: the fitness function is a Claude-3.5-Sonnet-V2 multi-agent debate, and the MT-Bench evaluation uses the same LLM-judge prompt and, per the implementation, the same Claude model family as judge. The paper never identifies a different judge model, and its own Limitations section concedes that 'models develop their own implicit evaluation criteria without any human intervention.' Thus the open-ended superiority claim reduces largely to showing that DEEVO optimizes for Claude's preferences and then wins under Claude's preferences. Because the close-ended results and ablations provide some external grounding, the paper is only partially circular rather than wholly so.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of LLM-generated debate judgments as a fitness proxy and on Elo as a selection signal; neither is validated against human preferences or an independent judge. Several governing hyperparameters, including the newcomer quota and Elo K, are never reported, and the reported hyperparameters were chosen without sensitivity analysis.

free parameters (7)
  • mutation_rate = 0.4
    Chosen to balance diversity and stability; no sensitivity analysis was performed, and it gates whether offspring undergo strategic mutation.
  • population_size = 10 initial prompts
    Initial population size set to 10; no sensitivity analysis is reported.
  • generations = 5
    Chosen as a performance and speed tradeoff; all final results are reported after 5 generations.
  • debate_rounds = 3
    Chosen following high-school level debate standards; controls debate depth and computational cost.
  • newcomer_quota = not reported
    Algorithm 1 requires a newcomer quota and the paper claims it balances newcomers and veterans, but the value used in experiments is never stated.
  • Elo_K = not reported
    The Elo update constant K governs rating volatility and is never specified in the paper.
  • LLM_choice_and_temperatures = Claude-3.5-Sonnet-V2; 0.8 for debate, crossover and mutation, 0 for judge
    All optimization and evaluation components use Claude models; the ablation shows 5.1 to 9.6 percent performance swings with model choice, so this is a consequential free choice.
assumptions (4)
  • domain assumption Multi-agent debate verdicts are a valid fitness proxy for subjective prompt quality without ground truth.
    Section 3 Step 2 treats debate winners as the fitness signal; no human validation or independent judge is used to establish that debate verdicts track quality.
  • domain assumption Elo ratings after a small number of pairwise debates and 5 generations are a reliable enough proxy for prompt performance to drive selection.
    Section 3 Step 4 and Section 4.2; the paper's own point-biserial correlations are only 0.137 and 0.156, so this assumption is weak.
  • domain assumption The MT-Bench LLM-judge from the same model family is an unbiased measure of open-ended output quality.
    Section 4.1 and 4.2; DEEVO is optimized with Claude debate judges and evaluated with a Claude MT-Bench judge, with no human or cross-family judge.
  • domain assumption Randomly generated initial prompts provide a sufficiently diverse starting population.
    Section 3.1 Step 1; if initial prompts are poor or homogeneous, evolution cannot explore effectively.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings." pith.science (2026). https://pith.science/paper/XI75II53

@misc{pith2026250600178,
  author       = {Pith},
  title        = {Pith review of: Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XI75II53}},
  note         = {Machine review of arXiv:2506.00178}
}
read the original abstract

Prompt engineering represents a critical bottleneck to harness the full potential of Large Language Models (LLMs) for solving complex tasks, as it requires specialized expertise, significant trial-and-error, and manual intervention. This challenge is particularly pronounced for tasks involving subjective quality assessment, where defining explicit optimization objectives becomes fundamentally problematic. Existing automated prompt optimization methods falter in these scenarios, as they typically require well-defined task-specific numerical fitness functions or rely on generic templates that cannot capture the nuanced requirements of complex use cases. We introduce DEEVO (DEbate-driven EVOlutionary prompt optimization), a novel framework that guides prompt evolution through a debate-driven evaluation with an Elo-based selection. Contrary to prior work, DEEVOs approach enables exploration of the discrete prompt space while preserving semantic coherence through intelligent crossover and strategic mutation operations that incorporate debate-based feedback, combining elements from both successful and unsuccessful prompts based on identified strengths rather than arbitrary splicing. Using Elo ratings as a fitness proxy, DEEVO simultaneously drives improvement and preserves valuable diversity in the prompt population. Experimental results demonstrate that DEEVO significantly outperforms both manual prompt engineering and alternative state-of-the-art optimization approaches on open-ended tasks and close-ended tasks despite using no ground truth feedback. By connecting LLMs reasoning capabilities with adaptive optimization, DEEVO represents a significant advancement in prompt optimization research by eliminating the need of predetermined metrics to continuously improve AI systems.

Figures

Figures reproduced from arXiv: 2506.00178 by the authors.

Figure 1
Figure 1. DEEVO. (1) First an initial set of prompts are either provided or generated. (2) Next, each prompt is executed. (3) Each [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Graph of Elo vs F1-Score for BBH-Nav across 5 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Average Elo for DEEVO updates over 5 generations [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The figure illustrates how the maximum Elo for [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 29 canonical work pages

  1. [1]

    ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition

    Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay, M Saiful Bari, and Bülent Yener. 2025. ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition. arXiv:2503.10673 [cs.CL] https://arxiv.org/abs/ 2503.10673

    work page Pith review arXiv 2025

  2. [2]

    Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. 2024. A Sta- tistical Framework for Ranking LLM-Based Chatbots. arXiv:2412.18407 [stat.ML] https://arxiv.org/abs/2412.18407

    arXiv 2024

  3. [3]

    David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel. 2018. Re-evaluating evaluation. Advances in Neural Information Processing Systems 31 (2018)

    work page 2018

  4. [4]

    Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, and Marzieh Fadaee

    work page

  5. [5]

    Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons.Biometrika 39, 3/4 (1952), 324–345

    1952

  6. [6]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evalu- ators through multi-agent debate. arXiv preprint arXiv:2308.07201 (2023)

    arXiv 2023

  7. [7]

    Derek Chen, Howard Chen, Yi Yang, Alex Lin, and Zhou Yu. 2021. Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2021. Association for Computati...

    work page 2021

  8. [8]

    Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2023. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082 (2023)

    arXiv 2023

Show all 69 references
  1. [9]

    Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155 (2023)

    2023 arXiv

  2. [10]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI...

    2024 arXiv

  3. [11]

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In EMNLP

    2022

  4. [12]

    Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. 2023. Pace: Improv- ing prompt with actor-critic editing for large language model. arXiv preprint arXiv:2308.10088 (2023)

    2023 arXiv

  5. [13]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch

  6. [14]

    Arpad E Elo and Sam Sloan. 1978. The rating of chessplayers: Past and present. (No Title) (1978)

    1978

  7. [15]

    In Forty-first International Conference on Machine Learning

    Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning

  8. [16]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 (2023)

    2023 arXiv

  9. [17]

    Andrew Estornell, Jean-François Ton, Yuanshun Yao, and Yang Liu. 2025. ACC- collab: An actor-critic approach to multi-agent LLM collaboration. In The Thir- teenth International Conference on Learning Representations

    2025

  10. [18]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2025. EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers. arXiv:2309.08532 [cs.CL] https: //arxiv.org/abs/2309.08532

    2025 arXiv

  11. [19]

    Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. 2025. Reward Reasoning Model.arXiv preprint arXiv:2505.14674 (2025)

    2025 arXiv

  12. [20]

    Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuid- berg Dos Martires. 2024. REvolve: Reward Evolution with Large Language Models using Human Feedback. arXiv preprint arXiv:2406.01309 (2024)

    2024 arXiv

  13. [21]

    Inman Harvey. 2009. The microbial genetic algorithm. In European conference on artificial life. Springer, 126–133

    2009

  14. [23]

    Eshaan He et al. 2025. PromptWizard: Feedback-Driven Self-Evolving Prompt Optimization. Microsoft Research (2025)

    2025

  15. [24]

    Abhinav Jain, Swarat Chaudhuri, Thomas Reps, and Chris Jermaine. 2024. Prompt tuning strikes back: Customizing foundation models with low-rank prompt adaptation. arXiv preprint arXiv:2405.15282 (2024)

    2024 arXiv

  16. [25]

    Wenyang Hu, Yao Shu, Zongmin Yu, Zhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, See-Kiong Ng, and Bryan Kian Hsiang Low. 2024. Localized zeroth-order prompt optimization. Advances in Neural Information Processing Systems 37 (2024), 86309–86345

    2024

  17. [26]

    Qi Ju, Falin Hei, Zhemei Fang, and Yunfeng Luo. 2024. ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models. In 2024 IEEE 13th Data Driven Control and Learning Systems Conference (DDCLS) . IEEE, 2062–2069

    2024

  18. [27]

    Yucheng Jiang, Yijia Shao, Dekun Ma, Sina J Semnani, and Monica S Lam. 2024. Into the unknown unknowns: Engaged human learning through participation in language model agent conversations. arXiv preprint arXiv:2408.15232 (2024)

    2024 arXiv

  19. [28]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth Internation...

    2024

  20. [29]

    Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. arXiv preprint arXiv:2402.06782 (2024)

    2024 arXiv

  21. [30]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)

    2021 arXiv

  22. [31]

    Alex Kim, Keonwoo Kim, and Sangwon Yoon. 2024. DEBATE: Devil’s Advocate- Based Assessment and Text Evaluation. arXiv preprint arXiv:2405.09935 (2024)

    2024 arXiv

  23. [32]

    Yan Li et al. 2024. Debatrix: Multi-agent LLM Debate for Scalable Evaluation. arXiv preprint arXiv:2402.13543 (2024)

    2024 arXiv

  24. [33]

    Chengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Yichen Wang, Chen Liu, Yu Lan, and Chao Shen. 2024. Concentrate Attention: Towards Domain- Generalizable Prompt Optimization for Language Models. In Advances in Neu- ral Information Processing Systems , A. Globerson, L. Mackey, D. ...

    2024

  25. [34]

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi

  26. [35]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118 (2023)

    2023 arXiv

  27. [36]

    Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. 2019. Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863 (2019)

    2019 arXiv

  28. [37]

    Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2022. GrIPS: Gradient- free, Edit-based Instruction Search for Prompting Large Language Models. arXiv preprint arXiv:2203.07281 (2022)

    2022 arXiv

  29. [38]

    Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R Bowman. 2023. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702 (2023)

    2023 arXiv

  30. [39]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng

  31. [40]

    Ansh Radhakrishnan. 2023. Anthropic fall 2023 debate progress update. In AI Alignment Forum, Vol. 80. 82–84

    2023

  32. [43]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh

  33. [44]

    gradient descent

    Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495 (2023)

    2023 arXiv

  34. [45]

    Rainer Storn and Kenneth Price. 1997. Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization 11 (1997), 341–359

    1997

  35. [46]

    Pejaver V Rao and Lawrence L Kupper. 1967. Ties in paired-comparison experi- ments: A generalization of the Bradley-Terry model. J. Amer. Statist. Assoc. 62, 317 (1967), 194–204

    1967

  36. [47]

    Archit Sharma, Sedrick Scott Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. 2024. A critical evaluation of ai feedback for aligning large language models. Advances in Neural Information Processing Systems 37 (2024), 29166–29190

    2024

  37. [48]

    Sun et al

    H. Sun et al. 2024. Query-Dependent Prompt Evaluation and Optimization with Offline Inverse Reinforcement Learning. arXiv preprint arXiv:2309.06553 (2024)

    2024 arXiv

  38. [49]

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for language-model-as-a-service. In International Conference on Machine Learning. PMLR, 20841–20855

    2022

  39. [50]

    Ankita Sinha, Wendi Cui, Kamalika Das, and Jiaxin Zhang. 2024. Survival of the Safest: Towards Secure Prompt Optimization through Interleaved Multi-Objective Evolution. arXiv preprint arXiv:2410.09652 (2024)

    2024 arXiv

  40. [51]

    Shange Tang, Yuanhao Wang, and Chi Jin. 2025. Is Elo Rating Reliable? A Study Under Model Misspecification. arXiv preprint arXiv:2502.10985 (2025)

    2025 arXiv

  41. [52]

    Jinwei Su, Yinghui Xia, Ronghua Shi, Jianhui Wang, Jianuo Huang, Yijin Wang, Tianyu Shi, Yang Jingsong, and Lewei He. 2025. DebFlow: Automating Agent Creation via Agent Debate. arXiv preprint arXiv:2503.23781 (2025)

    2025

  42. [53]

    Vighnesh Subramaniam, Antonio Torralba, and Shuang Li. 2024. Debategpt: Fine-tuning large language models with multi-agent debate supervision. (2024)

    2024

  43. [54]

    Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. 2025. Self-Supervised Prompt Opti- mization. arXiv preprint arXiv:2502.06855 (2025)

    2025 arXiv

  44. [55]

    Cilin Yan, Jingyun Wang, Lin Zhang, Ruihui Zhao, Xiaopu Wu, Kai Xiong, Qing- song Liu, Guoliang Kang, and Yangyang Kang. 2024. Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflection. arXiv preprint arXiv:2411.07446 (2024)

    2024 arXiv

  45. [56]

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, Prompt Optimization KDD 2025, Toronto, Canada, Nair and Banerjee et al. et al. 2022. Challenging big-bench tasks and whether chain...

    2022 arXiv

  46. [57]

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang

  47. [58]

    Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky

    Rose E. Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024. Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes. InProceedings of the 2024 Conference of the North American Chapter of the Associatio...

    2024

  48. [59]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  49. [60]

    Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161 (2021)

    2021 arXiv

  50. [61]

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gon- zalez. 2023. TEMPERA: Test-Time Prompt Editing via Reinforcement Learning. In ICLR

    2023

  51. [62]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al . 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736 (2024)

    2024 arXiv

  52. [63]

    Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, and Anna Korhonen. 2024. Fairer preferences elicit improved human-aligned large language model judgments. arXiv preprint arXiv:2406.11370 (2024)

    2024 arXiv

  53. [64]

    arXiv preprint arXiv:2406.14228 (2024)

    Evoagent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint arXiv:2406.14228 (2024)

    2024 arXiv

  54. [66]

    Jiayi Zhang et al. 2024. PREFER: Prompt Optimization with Feedback and Refine- ment. arXiv preprint arXiv:2406.07496 (2024)

    2024 arXiv

  55. [69]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  56. [71]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations

    2022

  57. [72]

    Ziyi Zhu et al . 2024. Bayesian Dynamic Prompt Learning. arXiv preprint arXiv:2402.11344 (2024)

    2024 arXiv

  58. [73]

    \nOpening Statements

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning . A Appendix A.1 Debate Defender System Prompt 1 You a r e...

    2024

  59. [2020]

    arXiv preprint arXiv:2010.15980 (2020)

    Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020)

    2020 arXiv

  60. [2023]

    arXiv:2311.17295 [cs.CL] https://arxiv.org/abs/2311.17295

    Elo Uncovered: Robustness and Best Practices in Language Model Evalua- tion. arXiv:2311.17295 [cs.CL] https://arxiv.org/abs/2311.17295

    arXiv

  61. [2024]

    arXiv preprint arXiv:2406.04770 (2024)

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770 (2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.