Pith. sign in

REVIEW 5 major objections 6 minor 26 references

Multi-Response Preference Optimization with Augmented Ranking Dataset

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a weighted ranking loss over n responses, with weights derived from a cancellation rule, lets a language model learn richer preferences in a single step, and that with a model-generated ranked dataset it generally…

desk verdict The augmentation pipeline has real legs, but the multi-response loss is a heuristic in search of a derivation, and the experiments don't support the headline claim. read the letter →

arxiv 2412.07812 v1 pith:V3JB4TFI submitted 2024-12-10 cs.CL cs.LG

Hansle Gwon , Imjin Ahn , Young-Hak Kim , Sanghyun Park , Tae Joon Jun This is my paper · ORCID
classification cs.CLcs.LG
keywords preferenceoptimizationdirectmulti-responselearningrankinglossdatasetaugmentationrewardmodelLLMalignmentinstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish two things: that preference-optimization datasets can be enlarged by a fully model-driven pipeline, and that a single weighted loss over the full ranked list, called Multi-DPO, lets a language model learn from all n responses at once instead of one chosen/rejected pair. The training objective assigns weight (n-2i+1)/(n-1) to each rank's log-probability ratio, so top responses are pushed up and bottom responses pushed down in one step; with n=2 it reduces exactly to DPO. The paper reports that Multi-DPO generally outperforms DPO on the same data, with the largest gains on small seed datasets, and that the augmented dataset alone contributes more to performance than the training method. A sympathetic reader would take the contribution as a practical recipe: more preference information per training step, plus a scalable way to build the ranked data that recipe needs.

What carries the argument

The central object is the weighted sum of rank-wise log-probability ratios inside a single logistic sigmoid, Equation (16). The cancellation rule produces the weights: in the expansion of all pairwise comparisons, response $y_i$ is preferred in $(n-i)$ terms and dispreferred in $(i-1)$ terms, leaving a net count of $(n-i)-(i-1)=n-2i+1$, normalized by $(n-1)$. The derivation also relies on approximating a product of sigmoid factors by the product of their exponential arguments (Lemma 1), which reduces the listwise objective to one log-sigmoid term.

What would settle it

Train the same policy with the exact all-pairs DPO loss and with Multi-DPO on the same ranked dataset; if Multi-DPO's advantage over DPO disappears or reverses when compared against the exact listwise objective, then the approximation step, not the rank information, is the source of the reported gains. A simpler check is to use synthetic data with known rewards and compare the learned reward ordering to the true ranking.

Watch

Extended reading notes

Core claim

Multi-DPO is the paper's central claim. Starting from the Bradley-Terry pairwise preference probability and writing the probability of a full ranking as a product of adjacent pairwise factors, the authors approximate the product of sigmoid terms by a product of exponentials and collect each response's net exponent after all pairwise comparisons are considered. The rule is that response $y_i$ appears as preferred $(n-i)$ times and as dispreferred $(i-1)$ times, so its weight is $(n-2i+1)$; normalizing by $(n-1)$ gives the objective $$\max_{\pi_\$\theta$} \mathbb{E}_{(x,y_1,\ldots,y_n)\sim D_{\text{rank}}}\left[\log\$\sigma$\left(\$\beta$ \sum_{i=1}^n \frac{n-2i+1}{n-1} \log\frac{\pi_\$\theta$(y_i|x)}{\pi_{\mathrm{ref}}(y_i|x)}\right)\right],$$ which equals DPO when $n=2$ and, for the four-response setting used in experiments, becomes the weights $+1, +1/3, -1/3, -1$. The paper argues that this objective enables the simultaneous learning of multiple responses and reports that it generally outperforms DPO under identical dataset conditions, especially when the seed dataset is small.

Load-bearing premise

The central derivation assumes that the probability of a full ranking factors as the product of adjacent pairwise Bradley-Terry probabilities, and that a product of sigmoid factors can be minimized by minimizing the sum of their exponent arguments; both steps are asserted rather than proved, and the factorization is not valid in general for ranking models, so the weighted loss is at best a heuristic if that assumption fails.

Editorial extensions

If this is right

  • For n=4, Multi-DPO's loss supervises four ranked responses in a single step with weights $+1, +1/3, -1/3, -1$, so top responses are reinforced and bottom responses suppressed together.
  • Because a ranked list is consumed in one forward/backward pass rather than expanded into pairwise examples, Multi-DPO is roughly 1.6-2 times more expensive than DPO on the same data, versus the larger expansion implied by training on all pairwise comparisons.
  • The proposed augmentation pipeline turns a 13,000-pair seed dataset into 37,000-50,000 ranked examples without human labeling, and models trained on augmented data outperform those trained on the seed data.
  • The largest relative gains from Multi-DPO appear on the small seed dataset; once the augmented dataset is large, the dataset itself matters more than the training objective.
  • When n=2, the Multi-DPO objective reduces exactly to the standard DPO objective, making DPO a special case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ranking-factorization assumption fails, the alternating weights act as a deliberate emphasis on extreme ranks rather than a faithful ranking likelihood; a testable consequence is that the loss should be most sensitive to the top and bottom responses, which the weights already favor.
  • The reported gains on augmented data could be partly inflated by self-enhancement bias, since the same family of commercial models generated and judged the responses; a human-rated held-out preference set would separate dataset-quality gains from evaluator bias.
  • The same weighted-sum trick could extend to arbitrary n or to non-uniform rank weights, and switching the underlying ranking model from Bradley-Terry to Plackett-Luce would test whether the mechanism is the weighting scheme or the factorization assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes two contributions for LLM preference optimization: (i) an automated pipeline that augments a seed preference dataset by generating new prompts with GPT-3.5, sampling responses from the policy and generator, and ranking them with a reward model; and (ii) a new objective called Multi-DPO, which is claimed to extend DPO to simultaneously learn from multiple ranked responses. The authors derive a weighted softmax-style loss in Eq. (16) and report experiments on AlpacaEval and MT-bench comparing DPO and Multi-DPO trained on seed, augmented, and combined datasets.

Significance. If the derivation of Multi-DPO were correct, the idea of collapsing multiple pairwise preference comparisons into a single weighted loss would be a valuable efficiency contribution to preference optimization, especially given the computational cost of converting ranked data into pairwise DPO examples. The data augmentation pipeline is also practically relevant and clearly described. However, the central theoretical claim is unsound: the factorization of the full ranking probability and the subsequent approximation do not hold, and the n=3 case directly contradicts the claim of simultaneous multi-response learning. The experimental evaluation is too weak to compensate, using a single seed, one base model, no variance estimates, and results that partly contradict the paper's own conclusions. The paper therefore does not establish its main claims.

major comments (5)
  1. [§4.2, Eq. (18) and Appendix A] The derivation begins by assuming p*(y1>y2>...>yn|x) = ∏_{i=1}^{n-1} p*(yi>yi+1|x). This factorization is not a consequence of the Bradley-Terry model; the probability of a full ranking under a Plackett-Luce or Bradley-Terry ranking model has a different normalization. Since this factorization is the first step leading to Eq. (16), the main objective is not actually derived from the stated ranking assumptions.
  2. [Appendix A, Lemma 1] Lemma 1 states that minimizing (1+B/A)(1+C/B) is approximately equivalent to minimizing (B/A)(C/B). This is incorrect: for positive u and v, (1+u)(1+v) = 1+u+v+uv, and when u and v are small (as they can be for exponential rewards), minimizing the product is not equivalent to minimizing uv. The inequality in Eq. (24) bounds the sum B/A + C/B, not the product, and the inference from bounds on the sum to the claimed approximation in Eq. (25) is a non sequitur. The approximation is therefore uncontrolled, contrary to what the paper asserts.
  3. [§4.2, Eq. (15)-(16); §5.3, Eq. (17)] The weighting scheme in Eq. (15) gives coefficients (n-2i+1). For n=3, the weights are (1,0,-1), so the middle response y2 receives zero gradient from the Multi-DPO loss. This directly contradicts the core claim that Multi-DPO 'enables the simultaneous learning of multiple responses.' The same issue persists in the n=4 loss in Eq. (17), where the two middle responses receive only ±0.33 weights and the objective reduces to a contrast between y1 and y4 plus small corrections, rather than a principled rank-informed objective.
  4. [§6.2-6.3, Tables 2-3] The empirical claim that Multi-DPO 'generally outperformed traditional DPO under the same dataset conditions' is not supported by the reported results. In MT-bench (Table 3), DPO_aug scores 7.04 while MDPO_aug scores 6.96, and DPO_seed+aug scores 6.83 while MDPO_seed+aug scores 7.00; the direction is inconsistent. Moreover, all experiments use a single training run with no error bars or multiple seeds, so even the AlpacaEval differences could reflect noise. The paper's own narrative acknowledges an exception for DPO_aug, which weakens the universality of the claimed improvement.
  5. [§4.1 and §6] The augmentation pipeline depends on a reward model (RM-Gemma-7B) to label synthetic data, and the final evaluation uses GPT-4-turbo as judge in AlpacaEval and MT-bench. This creates a closed loop in which the policy is trained toward the reward model's preferences and then evaluated by a possibly correlated LLM judge. No validation of the reward model against human judgments is provided, so the reported performance gains may reflect alignment with the reward model rather than with human preferences.
minor comments (6)
  1. [Eq. (7)] The DPO objective as printed is missing the logarithm inside the sigmoid: it should be log σ(β log(πθ(yw|x)/πref(yw|x)) − β log(πθ(yl|x)/πref(yl|x))). As written, the expression is dimensionally inconsistent.
  2. [§4.2, Eq. (10)] The product in Eq. (10) uses the index 'yn+1', which should be 'yi+1'. This typo appears multiple times and makes the derivation harder to follow.
  3. [§4.2, text around Eq. (12)-(14)] The paper states that all pairwise comparisons total '(n-1)!' comparisons, but the actual number of pairwise comparisons among n responses is n(n-1)/2. The factorial count is incorrect and does not match the product in Eq. (14).
  4. [§5.2] The number of augmented prompts is described as 39,000 (three per seed prompt), then 2,000 short prompts are excluded, leaving 37,000. The paper should clarify why the fraction of excluded prompts is reported as a round number and whether the exclusion criteria are reproducible.
  5. [§6.1 and Table 1] The dataset format for MDPO_seed is listed as (x, yw, yl, y1, y2), which is inconsistent with the notation used elsewhere for ranked data; the two extra responses should presumably be the policy-generated and generator-generated responses, but this is not made explicit.
  6. [§7] The conclusion acknowledges the objective 'converges to an approximate solution rather than an optimal one,' but this concession is made without quantifying or bounding the approximation, which is the central issue with the derivation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Multi-DPO's weighting is a stated construction from a pairwise-comparison product, with no fitted parameters or load-bearing self-citations.

full rationale

The paper's main claim—that Multi-DPO, Eq. 16, learns from multiple ranked responses—rests on an explicit derivation chain, not on fitting or on self-citation. Eq. 15 follows algebraically from the log of the product in Eq. 14, and the monotonic sigmoid in Eq. 16 preserves the argmax of Eq. 15; the weights (n-2i+1)/(n-1) are fixed combinatorial counts, not parameters estimated from data. No load-bearing reference is to the authors' own prior work; Refs. [10–13] and [7] are standard DPO/RLHF background. The augmented dataset uses an external reward model and LLM judges, which creates a closed AI-judge loop, but this is not a mathematical circularity in the derivation. The paper itself concedes in §7 that the objective 'converges to an approximate solution rather than an optimal one,' and Appendix A's adjacent-product factorization and Lemma 1 are mathematically unsupported—for n=3 the middle response has zero weight, undermining the 'simultaneous learning' claim. These are correctness and validity concerns, not circular reductions of the output to the input, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The paper's central claim rests on three untested modeling assumptions: the full-ranking factorization, the sigmoid-product approximation, and the reliability of reward-model labels for synthetic data. No free parameters are fitted to force the empirical results, but the method is not parameter-free in its design choices.

free parameters (5)
  • n (number of ranked responses per prompt) = 4
    Chosen by the authors; the loss weights depend on n, and no sensitivity analysis is provided.
  • Number of augmented prompts per seed prompt = 3
    Hand-picked; no justification or ablation for this multiplier.
  • Short-prompt exclusion count = 2000 prompts removed
    Prompts deemed 'excessively short' are filtered out with no objective criterion.
  • LoRA rank and alpha = 64, 64
    Training hyperparameters chosen without reported tuning.
  • Learning rate, batch size, epochs = 1e-6, 2, 3
    Training hyperparameters chosen without reported tuning or sensitivity analysis.
assumptions (6)
  • standard math Bradley-Terry model for pairwise preferences
    Used as the starting point in Eq. 1; standard in preference optimization.
  • ad hoc to paper Full ranking probability factorizes into product of adjacent pairwise probabilities
    Assumed in Appendix A, Eq. 18; not valid for the Bradley-Terry model of full rankings.
  • ad hoc to paper Lemma 1 approximation: minimizing the product of sigmoids is approximately minimizing the sum of their exponent arguments
    Unproved bound argument in Appendix A, Eq. 19-25; used to derive the weighted objective.
  • domain assumption Generated prompts approximate the seed prompt distribution
    Few-shot prompt generation with GPT-3.5-turbo assumes distributional similarity without checking.
  • domain assumption Reward model (RM-Gemma-7B) ratings approximate human preferences
    Used to label all synthetic data; no validation of reward-model agreement with humans.
  • domain assumption LLM judges (GPT-4-turbo) approximate human preferences in evaluation
    AlpacaEval and MT-bench rely on LLM-as-a-judge, which has known biases (e.g., self-enhancement).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Response Preference Optimization with Augmented Ranking Dataset." pith.science (2026). https://pith.science/paper/V3JB4TFI

@misc{pith2026241207812,
  author       = {Pith},
  title        = {Pith review of: Multi-Response Preference Optimization with Augmented Ranking Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V3JB4TFI}},
  note         = {Machine review of arXiv:2412.07812}
}
read the original abstract

Recent advancements in Large Language Models (LLMs) have been remarkable, with new models consistently surpassing their predecessors. These advancements are underpinned by extensive research on various training mechanisms. Among these, Preference Optimization has played a significant role in improving the performance of LLMs by incorporating human preferences into the training process. However, constructing preference optimization datasets is challenging and the optimization process is highly sensitive to the dataset quality. In this study, we propose a novel approach to augment Preference Optimization datasets. Additionally, we introduce a Multi-response-based Preference Optimization training method that enables the simultaneous learning of multiple responses.

Figures

Figures reproduced from arXiv: 2412.07812 by the authors.

Figure 1
Figure 1. The preference dataset augmentation is conducted through four stages. In each stage, all data is generated, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Results of MT-bench test. Applying equation (25) to compare all adjacent responses from y1 to yn yields the following. minimize( nY−1 i=i (1 + exp(r(x, yi+1) − r(x, yi))) ≈ minimize( nY−1 i=i exp(r(x, yn+1) − r(x, yn))) = minimize((exp(r(x, yn) − r(x, y1)))) (26) Applying equation (26) to compare all comparable responses from y1 to yn can be expressed as follows. minimize( nY−1 i=1 Yn k=i (exp(r(x, yi) − r(x, yk))))… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 8 canonical work pages

  1. [1]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  2. [2]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  3. [3]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022

    2022

  5. [5]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  6. [6]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  7. [7]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

    work page 2024

Show all 26 references
  1. [9]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  2. [10]

    Aligning language models with preferences through f-divergence minimization

    Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.08215, 2023

    2023 arXiv

  3. [11]

    Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Marc Dymetman. On reinforcement learning and distri- bution matching for fine-tuning language models with no catastrophic forgetting.Advances in Neural Information Processing Systems, 35:16203–16220, 2022

    2022

  4. [12]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  5. [13]

    Reinforcement learning by reward-weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  6. [14]

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  7. [15]

    Self- rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self- rewarding language models. arXiv preprint arXiv:2401.10020, 2024

    2024 arXiv

  8. [16]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38-17, pages 18990–18998, 2024

    2024

  9. [17]

    Aligning large language model with direct multi-preference optimization for recommendation

    Zhuoxi Bai, Ning Wu, Fengyu Cai, Xinyi Zhu, and Yun Xiong. Aligning large language model with direct multi-preference optimization for recommendation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 76–86, 2024

    2024

  10. [18]

    orca_dpo_pairs

    Intel. orca_dpo_pairs. https://huggingface.co/datasets/Intel/orca_dpo_pairs, 2023

    2023

  11. [19]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    2023

  12. [20]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023

    2023 arXiv

  13. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  14. [22]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015

    2015 arXiv

  15. [23]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  16. [24]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https: //github.com/tatsu-lab/alpaca_eval, 2023

    2023

  17. [25]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024

    2024

  18. [26]

    sample instruction,

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. 10 Multi-Re...

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.