Pith. sign in

REVIEW 4 major objections 6 minor 31 references

Tuning LLM Judge Design Decisions for 1/1000 of the Cost

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Systematic hyperparameter search produces open-weight LLM judges that match or beat proprietary evaluators at a fraction of the cost.

desk verdict A well-executed, cost-effective judge-tuning pipeline with real practical value; the accuracy claims rest more on PandaLM than on Arena-Hard, and the proxy-validity gap is real but not disqualifying. read the letter →

arxiv 2501.17178 v4 pith:VMTSQDP6 submitted 2025-01-24 cs.CL cs.AIcs.LG

David Salinas , Omar Swelam , Frank Hutter This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords LLM-as-a-judgehyperparametertuningmulti-fidelityoptimizationmulti-objectivehumanagreementopen-weightmodelsautomaticevaluationpromptsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-as-a-judge systems are usually compared with many design decisions changed at once, so it is hard to know what makes one judge better than another. The paper treats judge design as a search problem: vary the open-weight base model, temperature, prompt structure, output format, and whether both answer orders are averaged, and optimize for agreement with human annotations while tracking cost per annotation. The claim is that this search, made affordable by multi-fidelity pruning and a cheap but discriminating human-agreement metric, finds judges that match or beat existing closed-model and fine-tuned judges on standard test sets while costing roughly $1/100$ to $1/250$ as much per 1,000 annotations, with the entire search itself costing about 1,000 times less than a naive grid. If true, high-quality automatic evaluation no longer requires proprietary APIs or expensive human annotation, and judge performance becomes something the community can cheaply tune and reproduce.

What carries the argument

The machinery is a parametrized judge configuration: an open-weight base model, a temperature, a boolean for averaging both answer orders, and a prompt drawn from 80 combinations of five output formats (best-model letter, Likert scale, pairwise scores, preference score, multi-criteria average), three optional pre-output fields (confidence, answer, explanation), JSON versus raw text, and the presence of an example. Each configuration is scored by human agreement, the fraction of pairwise battles where the judge's preference equals the human label, with cost per annotation as the second objective. The search runs all 4,480 configurations on 400 validation battles, keeps the top 1,200 by non-dominated sort, runs those on 1,200 battles, then runs the top 400 on 3,548 battles. Non-dominated sort ranks configurations in Pareto layers so that one survives only if no other configuration is both more accurate and cheaper, which is what lets the search prune early and find cheap-accurate judges without evaluating every configuration on the full set.

What would settle it

If a fresh batch of human-annotated pairwise battles were collected without the specificity and quality filter, and the paper's selected judges no longer matched human preference or human-ELO rankings while a GPT-4-based judge still did, the claim that agreement on the filtered subset is the right selection signal would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that scaling a judge, using a larger base model or more instructions, is necessary but not sufficient, and that the remaining performance gap is closed by tuning a structured set of design decisions that previous work changed all at once. The paper shows that human agreement, the fraction of pairwise battles on which judge and human agree, separates judge configurations more reliably than Spearman correlation against human-ELO ratings given the same annotation budget, so it is used as the selection metric. Running a three-stage multi-fidelity search over 4,480 configurations produces a Pareto front of open-weight judges, and the selected small, medium, and large judges match or beat GPT-4- and Claude-based Arena-Hard judges on Spearman correlation and outperform PandaLM judges on human agreement, at roughly $1/100$ to $1/250$ of the cost per 1,000 annotations. The result is not a single universal prompt but a method for finding cost-accuracy tradeoffs.

Load-bearing premise

The load-bearing premise is that human agreement on a filtered subset of pairwise battles, where instructions are kept only if a small helper model rates them specific and high-quality, is a faithful proxy for judge quality in general, so judges chosen to maximize agreement on that subset will also rank models correctly on other test sets.

Editorial extensions

If this is right

  • Judge selection can be treated as a cheap tuning problem: the multi-fidelity search over 4,480 configurations costs roughly $2,000, versus an estimated $2 million for evaluating the same configurations with standard benchmark protocols.
  • Practitioners get a Pareto front of open-weight judges, so they can choose a small, medium, or large judge to fit a memory or latency budget without sacrificing the search's accuracy gains.
  • The tuned open-weight judges match or beat GPT-4- and Claude-based Arena-Hard judges on Spearman correlation with human-ELO rankings, at roughly $1/100$ to $1/250$ of the per-annotation cost.
  • Design choices that consistently help are low temperature, averaging both answer orders, and the pairwise-score output format, while asking for explanations or answers before the preference tends to hurt.
  • Fine-tuned judge baselines such as PandaLM and JudgeLM are outperformed on the test sets even though the tuned judges are zero-shot, not fine-tuned, models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the search already optimizes two objectives, the same pipeline could be rerun with additional objectives such as verbosity bias, self-preference bias, or position-bias flip rate, turning known judge weaknesses into optimization targets instead of post-hoc corrections.
  • Editorial inference: the paper's prompt-performance transfer across model sizes suggests that the winning prompt choices are likely to transfer to the next generation of open-weight models, although the paper notes that new reasoning capabilities could break this transfer.
  • Editorial inference: the cost ratio is computed on self-hosted GPU prices; on commercial API providers absolute costs will be higher, but the gap to proprietary closed-model judges should persist because those judges are billed through APIs with their own margins.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a systematic, low-cost procedure for tuning the design decisions of zero-shot LLM judges (base model, temperature, prompt template, output format, response averaging). It defines a search space of 4,480 judge configurations, evaluates them on human agreement with the LMSys dataset using a three-stage multi-fidelity selection driven by non-dominated sorting on accuracy and cost, and reports the resulting judges on held-out LMSys test instructions, the PandaLM test set, and Arena-Hard Spearman correlation with Chatbot Arena ELO. The authors claim that the tuned open-weight judges match or outperform GPT-4-based and Arena-Hard judges while costing one to two orders of magnitude less per annotation, and that the whole search costs roughly 1/1000 of a comparable naive grid evaluation. The paper also provides hyperparameter analyses (model scaling, temperature, output format, prompt components, order averaging) and releases code and annotations.

Significance. If the central claims hold, the paper makes a useful contribution: it is one of the first systematic studies of judge design decisions under a controlled search space, and it demonstrates that the search can be made affordable via multi-fidelity multi-objective optimization. The clean validation/test split, the use of bootstrap errors in most tables, the inclusion of external benchmarks (PandaLM, Arena-Hard, Chatbot Arena ELO), and the release of code and a full annotation dataset are concrete strengths. The open-weight nature of the resulting judges is a practical benefit for reproducibility and community leaderboards. The paper also surfaces practically relevant findings (e.g., pair scoring, order averaging, temperature effects) that could guide future judge construction. However, the validity of human agreement on a filtered LMSys set as a proxy for the reported target metrics is not demonstrated at the configuration level, and several accuracy comparisons rest on point estimates without uncertainty quantification; these issues need to be addressed before the headline claims are fully supported.

major comments (4)
  1. [Section 4 and Section 5.3] The optimization target is human agreement on the filtered LMSys validation set, but the paper's headline claims are about Spearman correlation with Chatbot Arena ELO (Table 4) and PandaLM test accuracy (Table 3). The justification in Section 4 for switching from Spearman correlation to human agreement is aggregate: Table 1 and Figs. 1-2 show that human agreement separates model sizes with lower variance. These facts do not establish that human agreement ranks individual judge configurations in the same order as the target metrics. The only configuration-level evidence in the paper, Fig. 13, is split-half reliability of human agreement itself, which says nothing about proxy-target validity. Because the multi-fidelity loop in Section 5.3 prunes 4480 configurations down to 400 using only human agreement, a weak configuration-level correlation between the proxy and the target would make the final judges not necessarily near-optimal for the reported metrics. I request an additional analysis: compute Spearman correlation (or PandaLM agreement) for all (or a random sample of) configurations at the lowest fidelity and report the rank correlation with human agreement on the shared configurations.
  2. [Appendix A.2] The LMSys instruction filter (quality score >= 5 plus the 'Specificity' criterion, both assigned by Llama3-8B-instruct) determines the entire distribution over which human agreement is measured and all selection decisions are made. The thresholds are introduced without any sensitivity analysis or validation that they preserve the target property of judge quality. As a result, the selected judges may be specialized to high-specificity, high-quality instructions rather than being generally better judges. The authors should show that the ranking of configurations is stable under reasonable variations of these thresholds, or at least report the results on unfiltered or differently filtered LMSys test splits.
  3. [Tables 3 and 4, Abstract] The abstract and conclusion state that the found judges 'outperform existing benchmarks in accuracy,' but the support is weaker than the text suggests. In Table 4, the best result (Ours-medium, 0.93 ± 0.05) overlaps within bootstrap error with GPT-4 (0.90 ± 0.06); the paper's own text acknowledges 'matches or outperforms,' which is the defensible claim. Table 3 lists only point estimates for PandaLM agreement, with no bootstrap errors or significance tests, so the strong claims about outperforming GPT-4 and PandaLM-70B on that dataset are not yet quantified. Please add uncertainty estimates to Table 3 and adjust the abstract/conclusion to reflect the actual significance level of the accuracy gains.
  4. [Section 5.6 and Section 5.3] The paper does not provide a baseline that isolates the contribution of the tuning procedure itself. The final judges are compared against existing judges (Arena-Hard, PandaLM, JudgeLM, GPT-4), but not against, say, a random configuration from the search space or a default prompt with the same base model. Without such a baseline, the reader cannot tell whether the reported test-set gains come from the multi-fidelity search, from the choice of model family, or from the prompt template used in the final configurations. I suggest reporting the test-set performance of several random configurations and of a default (e.g., 'best-model-identifier' with temperature 0 and no added components) configuration on the same three test sets.
minor comments (6)
  1. [Section 5.6 (LMSys paragraph)] The sentence 'The judges we found outperforms all baselines and slightly underperforms or matches Arena-Hard' is internally contradictory; Table 2 shows Ours-large (0.49) slightly below Arena-Hard (0.50), so the phrasing should be revised to say the judges outperform some baselines and match or slightly underperform Arena-Hard.
  2. [Section 5.3 and Appendix B] The method is referred to as both 'successful-halving' and 'sucessful-halving'; the standard name is 'successive halving', and the typo should be fixed.
  3. [Appendix B.1] The cost model is stated transparently, but the central cost-efficiency comparisons rely on self-estimated token prices and a fixed 0.6s/annotation average. A brief sensitivity analysis (e.g., 2x runtime or public API prices) would strengthen the robustness of the '1/100' and '1/1000' cost claims, which are a main selling point of the paper.
  4. [Section 5.4 (Fig. 7 caption)] The caption 'two group of prompts works well' contains a grammar error; it should be 'two groups of prompts work well.'
  5. [References] The reference to the LMSys Kaggle competition is listed as 'lin Chiang, W.'; this should be 'Chiang, W.-L.' for consistency with the main citation format.
  6. [Section 6] The limitation discussion is honest and appropriately notes that only flip rate is checked; it would be helpful to also mention that the search itself does not optimize for any bias metric, although this does not affect the validity of the main results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimization target is a validation-set proxy and the reported claims are evaluated on held-out and external test sets.

full rationale

The paper's derivation chain is self-contained. The search optimizes human agreement on 3,548 filtered LMSys validation instructions (Section 5.3, Fig. 4) with cost as a second objective, using successful halving and non-dominated sort. The final three judges are selected by best validation human agreement within size brackets (Section 5.6). The central accuracy claims are then measured on (i) 3,000 held-out LMSys test instructions disjoint from validation (Appendix A.2, Table 2), (ii) the external PandaLM test set (Table 3), and (iii) the Arena-Hard protocol with Spearman correlation against Chatbot Arena ELO (Table 4). None of these evaluation targets enters the optimization objective or any fitted parameter, so the results are not forced by construction. The only self-citations (Salinas et al. 2021; Schmucker et al. 2021; Izquierdo et al. 2021) justify a design choice — non-dominated sort with epsilon-net tie-breaking in a multi-fidelity loop — but the headline contribution is independently validated on held-out and external data; these citations are not load-bearing. The assumption that human agreement on filtered LMSys is a valid proxy for Spearman/PandaLM accuracy across configurations is a genuine correctness and validity risk (the paper justifies it only with aggregate scaling trends and split-half reliability, Figs. 1-2 and 13), but a proxy-validity gap is not circularity under the definitions used here.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on four measured or hand-chosen parameters: the LMSys instruction filtering thresholds, the average annotation time, the H100 hourly cost, and per-model token costs. These affect the reported cost savings and the composition of the Pareto front. The main domain assumptions are the validity of the filtered LMSys distribution as a tuning target and the grounding of human preferences as truth. No new physical or conceptual entities are introduced.

free parameters (4)
  • LMSys instruction filtering thresholds = score >= 5 and Specificity flag
    Instructions kept for validation and test are filtered by a Llama3-8B-based quality score; the threshold is hand-chosen in Appendix A.2 and affects the distribution on which judges are optimized and evaluated.
  • Average annotation time = 0.6 s per annotation on H100
    Used in Appendix B.3 to convert the number of annotations into a dollar cost for the 1/1000 cost claim; measured from their runs, not independently verified.
  • Cost per H100 hour = 2.79 USD/hour (RunPod)
    Used in the tuning-cost estimate; chosen from a third-party cloud price, driving the headline 2.1K$ estimate.
  • Model token costs = e.g., qwen2.5-72b: 0.58 $/1K tokens, llama-3.1-8b: 0.11 $/1K tokens
    Estimated from runtime measurements and hardware prices in Table 6; these costs determine the Pareto front ordering in the multi-objective search.
assumptions (4)
  • domain assumption Human battle outcomes in LMSys and Chatbot Arena are treated as ground truth for judge quality.
    The paper uses human agreement with LMSys labels as the optimization metric and Chatbot Arena ELO as the reference for Spearman correlation (Sections 3.2 and 5.6).
  • domain assumption The instructions filtered by Llama3-8B-instruct (score >= 5 and Specificity) form a representative evaluation set for comparing LLMs.
    Appendix A.2 uses this model-based filtering to define the validation and test sets; if the filter is biased, the reported improvements may not transfer.
  • standard math Non-dominated sort and successive halving are valid tools for multi-objective multi-fidelity hyperparameter optimization.
    The paper adopts these from existing literature (Karnin et al., 2013; Emmerich and Deutz, 2018; Salinas et al., 2021; Schmucker et al., 2021), relying on their properties without re-deriving them.
  • domain assumption Cost estimates from local token prices and cloud hourly rates approximate the real cost of deploying a judge.
    The Pareto front includes cost as an objective, and the headline 1/1000 claim uses these estimates; the paper acknowledges the estimates are approximate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tuning LLM Judge Design Decisions for 1/1000 of the Cost." pith.science (2026). https://pith.science/paper/VMTSQDP6

@misc{pith2026250117178,
  author       = {Pith},
  title        = {Pith review of: Tuning LLM Judge Design Decisions for 1/1000 of the Cost},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VMTSQDP6}},
  note         = {Machine review of arXiv:2501.17178}
}
read the original abstract

Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention. While several approaches have been proposed, many confounding factors are present between different papers. For instance the model, the prompt and other hyperparameters are typically changed at the same time making apple-to-apple comparisons challenging. In this paper, we propose to systematically analyze and tune the hyperparameters of LLM judges. To alleviate the high cost of evaluating a judge, we propose to leverage multi-objective multi-fidelity which allows to find judges that trade accuracy for cost and also significantly reduce the cost of the search. Our method identifies judges that not only outperform existing benchmarks in accuracy and cost-efficiency but also utilize open-weight models, ensuring greater accessibility and reproducibility. The code to reproduce our experiments is available at this repository https://github.com/geoalgo/judgetuning .

Figures

Figures reproduced from arXiv: 2501.17178 by the authors.

Figure 1
Figure 1. Effect of scaling the LLM judge and increasing the number of instructions on Spearman correlation. In contrast to human agreement, neither Alpaca-Eval, Arena-Hard, nor their union distinguishes the quality difference between 32B and 72B models. 10 1 10 2 10 3 #instructions 2 × 10 1 3 × 10 1 4 × 10 1 6 × 10 1 Human agreement 0.5B 1.5B 3B 7B 32B 72B Random Length [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Effect on scaling the LLM judge and the number of instructions on human-agreement. Human agreement. In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the prompt templating approach. We parametrize the prompt with the following hyperparameters: Pro￾vide answer, Provide explanation, Provide example, use JSON, output preference format. Given each of the 2 4 × 5 = 80 prompt hyperparameter, we generate a prompt like this one. battles. This reduces the cost of the full search from N × P to N × P/3. One could also use a more aggressive cutoff and save fu… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Illustration of the selection process. All 4480 configurations are first evaluated on 400 instructions (left), the top 1200 configurations are then evaluated on 1200 instructions (center) and finally the top 400 configurations are evaluated on 3548 instructions (right)…
Figure 5
Figure 5. Figure 5: We plot the cost per annotation and human agreement of all 4480 judges when using 400 instructions. The model family and the number of parameters are represented with color and size respectively. of scores for all models and prompts. Interestingly, smaller models and l…
Figure 6
Figure 6. Figure 6: Fraction of time each hyperparameter appears in the top 100 configurations for small (<10B) and large models (>10B). qwen2.5-32b llama-3.1-70b qwen2.5-72b gemma-2-27b gemma-2-9b llama-3.1-8b qwen2.5-7b qwen2.5-32b llama-3.1-70b qwen2.5-72b gemma-2-27b gemma-2-9b llama-…
Figure 7
Figure 7. Figure 7: Prompt performance stability across different models. We show the correlation matrix between models when looking at their performance on all the 80 different prompts. LMSys. In [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Prompt to evaluate instruction quality. A. Datasets A.1. Alpaca-Eval and Arena-Hard datasets We consider two datasets that contains prompts, model completions and judge annotations for a grid of prompts and model pairs. The first one is Alpaca-Eval which contains 47 mo…
Figure 9
Figure 9. Figure 9: Example of a prompt for the user prompt "Who is Barack Obama?". In this case, the judge is asked to provide its answer, an explanation, and is provided an example. It is asked to use the Pair format and provide its answer in JSON. . 14 [PITH_FULL_IMAGE:figures/full_fi…
Figure 10
Figure 10. Figure 10: Example of a prompt for the user prompt "Who is Barack Obama?". In this case, the judge is asked to provide its answer. It is asked to use the Likert format and provide its answer in raw text. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Illustration of the non-dominated sorting approach. The process first computes the Pareto front, assigning top ranks to the points in this layer. Next, the Pareto front is determined for the remaining points, which are then assigned the next set of rankings. This proc…
Figure 12
Figure 12. Figure 12: Scatter plot of cost and human agreement on the 400 validation instructions for all judges. We color-code each hyperparameter differently to illustrate the performance of all judges. Even using the same LLM model, there is a large spread of performance when varying ot…
Figure 13
Figure 13. Figure 13: Correlation for different fidelity sizes. For each fidelity, we randomly split the instructions into two buckets and plot the human-agreement on the first bucket versus the same metric computed on the second bucket of instructions for all the available judges, we also…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J

    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL https://arxiv.org/abs/2403.04132

    arXiv 2024

  3. [3]

    Ultrafeedback: Boosting language models with scaled ai feedback, 2024

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    arXiv 2024

  4. [4]

    Doddapaneni, S., Khan, M. S. U. R., Verma, S., and Khapra, M. M. Finding blind spots in evaluator llms with interpretable checklists. arXiv preprint arXiv:2406.13439, 2024

    arXiv 2024

  5. [5]

    Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    arXiv 2024

  6. [6]

    From general LLM to translation: How we dramatically improve translation quality using human evaluation data for LLM finetuning

    Elshin, D., Karpachev, N., Gruzdev, B., Golovanov, I., Ivanov, G., Antonov, A., Skachkov, N., Latypova, E., Layner, V., Enikeeva, E., Popov, D., Chekashev, A., Negodin, V., Frantsuzova, V., Chernyshev, A., and Denisov, K. From general LLM to translation: How we dramatically improve translation quality using human evaluation data for LLM finetuning. In Had...

    2024

  7. [7]

    Emmerich, M. T. and Deutz, A. H. A tutorial on multiobjective optimization: fundamentals and evolutionary methods. Natural computing, 17: 0 585--609, 2018

    work page 2018

  8. [8]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Fernando, C., Banarse, D., Michalewski, H., Osindero, S., and Rockt \"a schel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023

    arXiv 2023

Show all 31 references
  1. [9]

    Grattafiori, A., Dubey, A., Jauhri, A., and Abhinav Pandey, e. a. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  2. [10]

    X., and Hasan, S

    He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., and Hasan, S. Does prompt formatting have any impact on llm performance?, 2024. URL https://arxiv.org/abs/2411.10541

    2024 arXiv

  3. [11]

    On the limitations of fine-tuned judge models for llm evaluation, 2024

    Huang, H., Qu, Y., Zhou, H., Liu, J., Yang, M., Xu, B., and Zhao, T. On the limitations of fine-tuned judge models for llm evaluation, 2024. URL https://arxiv.org/abs/2403.02839

    2024 arXiv

  4. [12]

    Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization

    Izquierdo, S., Guerrero-Viu, J., Hauns, S., Miotto, G., Schrodi, S., Biedenkapp, A., Elsken, T., Deng, D., Lindauer, M., and Hutter, F. Bag of baselines for multi-objective joint neural architecture search and hyperparameter optimization. In 8th ICML Workshop on Automated Mach...

    2021

  5. [13]

    Almost optimal exploration in multi-armed bandits

    Karnin, Z., Koren, T., and Somekh, O. Almost optimal exploration in multi-armed bandits. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp.\ 1238--1246, Atl...

    2013

  6. [14]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  7. [15]

    E., and Stoica, I

    Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

    2024 arXiv

  8. [16]

    Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  9. [17]

    E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keating, N

    lin Chiang, W., Zheng, L., Dunlap, L., Gonzalez, J. E., Stoica, I., Mooney, P., Dane, S., Howard, A., and Keating, N. Lmsys - chatbot arena human preference predictions. https://kaggle.com/competitions/lmsys-chatbot-arena, 2024. Kaggle

    2024

  10. [18]

    Aligning with human judgement: The role of pairwise preference in large language model evaluators

    Liu, Y., Zhou, H., Guo, Z., Shareghi, E., Vuli \'c , I., Korhonen, A., and Collier, N. Aligning with human judgement: The role of pairwise preference in large language model evaluators. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=9gdZI7c6yr

    2024

  11. [19]

    Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-T., Wang, S., and Lin, X. V. LEVER : Learning to verify language-to-code generation with execution. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th Internatio...

    2023

  12. [20]

    Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures, 2024

    Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neubig, G., and You, Y. Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures, 2024. URL https://arxiv.org/abs/2406.06565

    2024 arXiv

  13. [21]

    R., and Feng, S

    Panickssery, A., Bowman, S. R., and Feng, S. Llm evaluators recognize and favor their own generations, 2024. URL https://arxiv.org/abs/2404.13076

    2024 arXiv

  14. [22]

    A multi-objective perspective on jointly tuning hardware and hyperparameters, 2021

    Salinas, D., Perrone, V., Cruchant, O., and Archambeau, C. A multi-objective perspective on jointly tuning hardware and hyperparameters, 2021. URL https://arxiv.org/abs/2106.05680

    2021 arXiv

  15. [23]

    B., Salinas, D., and Archambeau, C

    Schmucker, R., Donini, M., Zafar, M. B., Salinas, D., and Archambeau, C. Multi-objective asynchronous successive halving, 2021. URL https://arxiv.org/abs/2106.12639

    2021 arXiv

  16. [24]

    Efficient prompt optimization through the lens of best arm identification

    Shi, C., Yang, K., Chen, Z., Li, J., Yang, J., and Shen, C. Efficient prompt optimization through the lens of best arm identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  17. [25]

    Fine-tuning and prompt optimization: Two great steps that work better together

    Soylu, D., Potts, C., and Khattab, O. Fine-tuning and prompt optimization: Two great steps that work better together. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, ...

    2024

  18. [26]

    Panda LM : An automatic evaluation benchmark for LLM instruction tuning optimization

    Wang, Y., Yu, Z., Yao, W., Zeng, Z., Yang, L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., Ye, W., Zhang, S., and Zhang, Y. Panda LM : An automatic evaluation benchmark for LLM instruction tuning optimization. In The Twelfth International Conference on Learning ...

    2024

  19. [27]

    H., Le, Q., and Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022. URL https://arxiv.org/abs/2201.11903

    2022 arXiv

  20. [28]

    Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., and Hashimoto, T. B. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12: 0 39--57, 2024. doi:10.1162/tacl_a_00632. URL https://aclanthology.org/...

    2024 doi

  21. [29]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

    2023

  22. [30]

    Fairer preferences elicit improved human-aligned large language model judgments

    Zhou, H., Wan, X., Liu, Y., Collier, N., Vulic, I., and Korhonen, A. Fairer preferences elicit improved human-aligned large language model judgments. CoRR, abs/2406.11370, 2024

    2024 arXiv

  23. [31]

    Judgelm: Fine-tuned large language models are scalable judges

    Zhu, L., Wang, X., and Wang, X. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.