HREF: Human Response-Guided Evaluation of Instruction Following in Language Models

Hannaneh Hajishirzi; Pradeep Dasigi; Xinxi Lyu; Yizhong Wang

REVIEW 3 major objections 5 minor 38 references

HREF: Human Response-Guided Evaluation of Instruction Following in Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a human-written reference to an LLM judge's prompt improves agreement with human preference judgments, and uses that finding to build the HREF benchmark with a private test set and task-level scores.

desk verdict Good benchmark, clever idea; headline agreement gains are selected in-sample and need a held-out check. read the letter →

arxiv 2412.15524 v2 pith:2RONHCQL submitted 2024-12-20 cs.CL cs.AI

Xinxi Lyu , Yizhong Wang , Hannaneh Hajishirzi , Pradeep Dasigi This is my paper · ORCID

classification cs.CLcs.AI

keywords instructionfollowingLLM-as-a-judgehumanreferenceevaluationbenchmarkpairwisepreferencetask-centrictest-setcontaminationagreement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that automatic evaluation of how well language models follow instructions becomes noticeably more trustworthy when the judge model is given a human-written reference response as additional context. On a set of 438 instructions spanning 11 task categories, the authors find that this added reference raises agreement with human preference judgments by up to 3.2% across a wide range of tasks. They turn this observation into HREF, a benchmark of 4,258 instructions with a private test split, a public development split, a composite per-category evaluation setup, and a live leaderboard. If the finding holds, model developers get a cheaper, faster, and contamination-resistant proxy for human preference judgments, with per-task scores that show which skills to improve.

What carries the argument

The load-bearing device is the composite evaluation setup. For each of the 11 task categories, the authors compare LLM-as-a-Judge, LLM-as-a-Judge-with-human-reference, and embedding-based similarity to the human reference, and select the method with the highest leave-one-out agreement against four human annotations per instance. The human reference enters the judge prompt as extra context, not as the output to reproduce, and tie judgments are allowed; RoBERTa-Large provides the embedding similarity when that method wins. This per-category selection is what lets HREF report task-level win rates while keeping a single headline reliability number.

What would settle it

Split the 438-instruction human agreement set by category into method-selection and validation halves, or collect a fresh human-annotated set of equal size, and test whether the per-category winners and the 69.4% composite agreement replicate; if a plain Llama-3.1-70B judge without human reference matches or beats the composite on the held-out half, the central reliability claim fails. Alternatively, once human labels for a sample of the private 4,258-instruction set become available, recompute composite agreement on those held-out instances.

Watch

Extended reading notes

Core claim

Using human-written responses as reference context, rather than as a gold answer to be matched, improves the reliability of LLM-based judges of instruction following. Across the 438-instruction human agreement set, LLM-as-a-Judge with a human reference outperforms the same judge without it for capable judges, and the composite method that selects the best evaluator per category—Llama-3.1-70B-Instruct as judge, human reference in the prompt, or RoBERTa embedding similarity in open-ended categories—reaches a leave-one-out agreement rate of 69.4% with human annotations. That is 2.4 points higher than the human annotators' own inner-agreement rate of 67.0%. The authors also report that judge models, when directly asked to compare a model response against a human response, strongly prefer the model response, so the human reference should be used as a complementary context rather than as the sole comparison target.

Load-bearing premise

The composite method's winning per-category choices and the headline 69.4% agreement are both computed on the same 438-instruction human agreement set, where each category has only about 40 instances, so the claim assumes that these selections, and the agreement rate itself, hold up on the separate 4,258-instruction private evaluation set.

Editorial extensions

If this is right

Adding a human-written reference response to LLM judge prompts raises agreement with human preference judgments by up to 3.2% across the evaluated instruction-following tasks.
The per-category composite evaluator using Llama-3.1-70B-Instruct reaches 69.4% leave-one-out agreement with human annotations, 2.4 points above the human annotators' inner agreement rate.
Because HREF uses open-weight models for both baseline and judge and keeps its 4,258-instruction evaluation set private, it avoids both direct test-set leakage and the indirect contamination that comes from sending test prompts to closed API models.
Task-level scores reveal uneven strengths: Mistral-Large-Instruct-2407 has the best overall expected win rate on the initial 37-model run yet performs poorly on Open QA, illustrating the benchmark's stated purpose of surfacing skill-specific gaps.
Model rankings on the public 430-instruction development set correlate strongly (0.98 Spearman) with rankings on the private evaluation set, supporting the use of the development set for fast iteration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial: If the orthogonal-perspective explanation is right, adding more than one human-written reference per instruction, or references written by different human annotators, should push agreement higher still—an extension the paper does not test.
Editorial: The per-category selection is made on the same 438-instruction set whose agreement rates are then reported as the headline result; a holdout-based replication with a fresh human-annotated set would be the natural check of whether the 69.4% figure transfers.
Editorial: Because human annotators themselves agree only 67.0% under leave-one-out, an automatic evaluator has a low ceiling against today's annotation protocol; disagreement-aware aggregation, such as weighting by annotator confidence, might be a more informative target than the mode-based agreement used here.
Editorial: The finding that judges prefer model-written responses when directly compared against human responses suggests reference-guided evaluation could be misused if someone treats the human response as a gold answer; the paper's setup carefully avoids that, and future benchmarks should keep the same distinction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know about HREF is that the core idea is good and the benchmark will likely be useful, but the headline agreement numbers are not yet trustworthy. The composite method — per-category choice among LLM-as-a-judge with and without human reference, plus RoBERTa embedding — is selected and evaluated on the same 438-instance human agreement set. With roughly 40 instances per category, the differences in Table 2 are within sampling noise, and picking the best method per category on that set inflates the reported 69.4% and the 3.2% improvement. The paper never measures out-of-sample agreement: the development and private evaluation sets have no human annotations, so we only get the in-sample max.

That said, there is real value here. Using human-written responses as additional context in the judge prompt is genuinely new relative to AlpacaEval's model-generated references, and the finding that human references beat model-generated ones across judge models is credible and useful. The task-centric curation — 11 categories from the InstructGPT taxonomy, 4,258 private instructions — directly addresses the task-skew problem in prior benchmarks, and using open-weight Llama models for both judge and baseline is a sensible step for reproducibility and contamination control. The high correlation between development and private set rankings (0.98 Spearman) is good evidence the benchmark orders models consistently.

The soft spots are real but not disqualifying. The selection-on-the-same-set problem is the main one; a nested cross-validation or a separately collected human agreement set for method choice would fix it. There are also no error bars on any of the agreement differences, so we can't tell if the 2.4% edge over human inner-agreement is meaningful. I also found an inconsistency: Table 4 says the development set has no human annotations, yet Table 6 reports human agreement rates on the development set. That needs a correction. Finally, the handling of parse failures and ties is under-specified — parse failures are dropped without reporting their rate, and ties count as wins for the target model.

For whom: anyone building or using instruction-following benchmarks. It deserves a serious referee; the authors have done the work and released the data. But the revision should include a proper holdout validation of the composite selection before the headline claims are taken at face value.

My recommendation: engage with it, but treat 69.4% and 3.2% as upper-bound estimates until the selection bias is addressed.

Referee Report

3 major / 5 minor

Summary. The paper proposes HREF, a new instruction-following benchmark with 4,258 private instructions across 11 task categories, human-written reference responses, and a composite automatic evaluation that selects the most reliable pairwise evaluation method per category. The authors construct a 438-instruction human agreement set with four preference annotations per instance from a separate annotator group, and use it to compare LLM-as-a-judge, embedding-based, perplexity, and heuristic methods. Their central empirical claims are that including human-written responses as additional context improves agreement with human judges by up to 3.2%, and that the composite method with Llama-3.1-70B-Instruct reaches 69.4% leave-one-out agreement, outperforming the human annotators' inner agreement rate by 2.4%. They also analyze design choices such as judge model, baseline model, prompt template, and evaluation-set size, and they release a public development set, evaluation code, and a private leaderboard.

Significance. If the main claims hold, HREF would be a useful community resource: it is the largest task-oriented instruction-following benchmark with a private test set, uses open-weight models as judge and baseline for reproducibility and contamination resistance, and provides per-category results that can guide model development. The experimental design has notable strengths: a diverse model pool of 32 models, explicit length-controlled response sampling, four human annotations per instance, and an annotator group distinct from the response writers. The paper also releases code and data, and its development-set-to-evaluation-set correlation check is a sensible validation step. However, the headline reliability numbers are weakened by the fact that the composite method is selected and evaluated on the same 438-instruction set, so the quantitative claims require additional out-of-sample or bias-corrected evidence before they can be accepted at face value.

major comments (3)

[Section 2.1 and Table 2] The headline composite agreement is an in-sample maximum. The composite method is constructed by selecting, for each of the 11 categories, the evaluation method with the highest agreement in Table 2, and the same 438-instruction human agreement set is then used to report the 69.4% composite agreement and the 2.4% margin over human annotators in Section 3.1. With roughly 40 instructions per category (Figure 5, right), the per-category differences among methods are within sampling noise, and selecting the maximum per category introduces optimistic selection bias. The paper provides no held-out human preference data, no cross-validation estimate, and no confidence intervals for these differences. Since the 4,258-instruction private evaluation set has no human annotations, the reliability of the composite method on new instructions is not directly measured; the authors should add a bias-corrected estimate, such as leave-one-category-out or a small human-annotated validation sample from the evaluation set, before relying on the 3.2% and 2.4% claims.
[Section 5.2, Table 6 and Table 4] There is an internal inconsistency about whether the development set has human annotations. Table 4 states that the Development set has no human annotations, but Section 5.2 reports 'Human Agreement Rate' for four prompt templates evaluated on the development set. If these rates were computed on the 438-instruction agreement set instead, the text should say so explicitly; if they were computed on the development set, then Table 4 is incorrect. This matters because the prompt-template comparison is used to justify the choice of Prompt A, so the source of the agreement numbers must be unambiguous.
[Section 4.3] The reported development-set-to-evaluation-set correlation (Spearman 0.98, Pearson 0.99) is only a consistency check between two automatic evaluation pipelines; it does not validate that the composite method agrees with human judges on the private evaluation set. The authors should clarify this in the text, since otherwise readers may interpret the high correlation as evidence that the reliability established on the 438-instruction agreement set transfers to the larger private set, which is not directly shown.

minor comments (5)

[Title and Section 1] The paper uses 'Human Response-Guided Evaluation' in the title and abstract but introduces the benchmark as 'Human Reference-guided Evaluation of instruction Following (HREF)' in Section 1; please unify the terminology.
[Table 2 caption] The caption says 'Bold numbers are the highest numbers with Llama-3.1-70B-Instruct for each categories'; this phrasing is ambiguous about whether bold marks the best method overall in each category or the best among the Llama-70B methods.
[Section 5.2] In the sentence 'the strong correlation between our prompt (Prompt D) and AlpacaEval’s prompt (Prompt A)', the prompt labels appear to be swapped relative to Table 6, where Prompt A is the HREF prompt and Prompt D is the AlpacaEval-style prompt.
[Appendix C] The parsing description says 'a' is mapped to 0 and 'b' or 'tie' to 1; please clarify whether a tie is deliberately treated as a win in the expected-win-rate computation, consistent with Section 4.2, or whether this mapping is only for the agreement-rate calculation.
[Throughout] There are several typos and grammatical errors, including 'using by using' (Section 1), 'defination' (Appendix B.2), 'poolly' (Section 4.3), 'migrate such risk' (Section 6), and 'Appendex' (Section 4.3 caption); a careful proofreading pass is recommended.

Circularity Check

1 steps flagged · score 4.0 of 10

Per-category composite method is selected on the 438-instruction human agreement set and its 69.4% agreement is then reported on the same set, making the composite's headline gains partly in-sample selection artifacts rather than held-out predictions.

fitted input called prediction [Section 2.1.2 / Section 2.2 / Section 3.1, Table 2]
"To enable such a comparison, we built a dataset with human annotated preferences ... We refer to this dataset as the human agreement set and it is a subset of the final dataset described in Section 4. ... Composite select the best method from LLM-as-a-Judge, LLM-as-a-Judge with human response, and embedding-based methods for each category. ... Bold numbers are the highest numbers with Llama-3.1-70B-Instruct for each categories, and we choose their corresponding methods to form the final composite method. ..."

The per-category method is fitted as the maximum-agreement entry on the 438-instruction human agreement set (Table 2), and the composite's 69.4% 'All' agreement and the 2.4% margin over annotator inner agreement are then computed on that same set. Because the composite applies an argmax over candidate methods in each category, its aggregate agreement on the fitting set is mathematically guaranteed to be at least as high as every single method's aggregate, so the reported superiority over Llama-3.1-70B with human reference (67.9%) is partly a selection artifact, not an out-of-sample estimate. No held-out human-annotated data validates the composite: Table 4 marks the 430-instruction development set and the 4,258-instruction private evaluation set as unannotated ('✗').

full rationale

The core comparison of evaluation methods is grounded in external human preference annotations collected from a separate annotator group (Section 2.1.2), so the central observation that human-written references improve LLM-judge agreement is an empirical, falsifiable measurement rather than a self-derivation. There is no load-bearing self-citation chain or imported uniqueness theorem. The main circularity-adjacent problem is confined to the composite construction: the per-category method (and thereby the 69.4% composite agreement reported in Section 3.1, as well as the Llama-3.1-70B judge choice revisited in Section 5.1) is selected and then evaluated on the same 438-instance human agreement set, so part of the reported gain over single methods is guaranteed by the argmax operation. An additional internal inconsistency weakens the independent evidence: Section 5.2 reports 'Human Agreement Rate' for prompts compared 'on our development set' (Table 6), while Table 4 lists the development set as having no annotations ('✗'), removing a would-be external check of the prompt design. Because the main direction of the derivation is externally grounded and only the composite's headline margin is in-sample-fitted, the partial circularity score is moderate rather than extreme.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No fitted model parameters are in play; instead, the evaluation pipeline itself contains design choices (judge model, per-category method selection, baseline model, tie handling) that are selected using the same human agreement data on which reliability is reported. These choices function as free parameters of the benchmark and should be validated on held-out data. The axioms list the assumptions about human annotations, task coverage, and contamination that the benchmark's validity depends on.

free parameters (2)

Per-category evaluation method selection = Best method per category from Table 2 (e.g., Llama-70B judge with human reference for CQA, RoBERTa embedding for Fact…
The best method for each of the 11 categories is chosen based on LOO agreement on the same 438-instruction agreement set used to report final agreement; no held-out validation of this selection is described, so it acts as a fitted component of HREF.
Judge model choice = Llama-3.1-70B-Instruct
Selected because it had the highest agreement in Table 2 on the same data; this is a design parameter fitted to the agreement set. Section 5.1 only varies the baseline model, not the judge selection.

assumptions (5)

domain assumption Human preference annotations from 16 screened annotators, with 4 annotations per instance, are an unbiased gold standard for instruction-following quality.
All agreement rates and HREF's reliability are measured against these annotations (Section 2.1.2). If the annotator pool is unrepresentative, the benchmark inherits that bias.
domain assumption The 11-category InstructGPT taxonomy covers the relevant space of instruction-following tasks and is balanced enough for per-category method selection.
The paper builds tasks from this taxonomy (Section 2.1.1) and selects methods per category. With roughly 40 instructions per category in the agreement set, category-level estimates are noisy.
standard math Leave-One-Out agreement on 4 annotations, with random tie-breaking for modes, reliably measures method quality.
Algorithm 1 defines the metric. Random choice among tied modes adds noise that is averaged but not quantified.
domain assumption Human-written references from No Robots and hired experts are high quality and independent of the preference annotators.
Section 2.1.1. If references are systematically different in style or quality, the improvement from adding them may not transfer.
domain assumption Using open-weight Llama models as judge and baseline prevents test-set contamination through API-based distillation.
Section 5.2. Assumes the private test set stays private and that no other contamination path exists.

how reviews work

0 comments

Cite this review

Pith. "Pith review of HREF: Human Response-Guided Evaluation of Instruction Following in Language Models." pith.science (2026). https://pith.science/paper/2RONHCQL

@misc{pith2026241215524,
  author       = {Pith},
  title        = {Pith review of: HREF: Human Response-Guided Evaluation of Instruction Following in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RONHCQL}},
  note         = {Machine review of arXiv:2412.15524}
}

read the original abstract

Evaluating the capability of Large Language Models (LLMs) in following instructions has heavily relied on a powerful LLM as the judge, introducing unresolved biases that deviate the judgments from human judges. In this work, we reevaluate various choices for automatic evaluation on a wide range of instruction-following tasks. We experiment with methods that leverage human-written responses and observe that they enhance the reliability of automatic evaluations across a wide range of tasks, resulting in up to a 3.2% improvement in agreement with human judges. We also discovered that human-written responses offer an orthogonal perspective to model-generated responses in following instructions and should be used as an additional context when comparing model responses. Based on these observations, we develop a new evaluation benchmark, Human Response-Guided Evaluation of Instruction Following (HREF), comprising 4,258 samples across 11 task categories with a composite evaluation setup, employing a composite evaluation setup that selects the most reliable method for each category. In addition to providing reliable evaluation, HREF emphasizes individual task performance and is free from contamination. Finally, we study the impact of key design choices in HREF, including the size of the evaluation set, the judge model, the baseline model, and the prompt template. We host a live leaderboard that evaluates LLMs on the private evaluation set of HREF.

Figures

Figures reproduced from arXiv: 2412.15524 by the authors.

**Figure 2.** The distribution of length difference between sampled model responses and the base line model responses. The distribution is symmetrical. the remaining 3 scientific text understanding tasks, we hire human experts to write instructions and associated responses. We ended up with 438 pairs where all 11 categories are reasonably represented (See [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Human Agreement Rate using model-generated v.s. human-written responses. Human response outperforms model response for LLM-based evaluation methods but underperforms for embedding-based evaluation methods. LLM-as-a-Judge with human response is similar to LLM-as-a-Judge except that it embeds human-written response into the prompt and instructs the judge to refer to it. See Appendix C for the prompt template we use. … view at source ↗

Figures from the paper (12 more)

**Figure 4.** Figure 4: Judges Preference Between Model-Generated Responses v.s. Human-Written Responses. Model-generated responses are in great favor of all the judges. 3.1 MAIN RESULTS [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Task Categorical Distribution of the three subsets in HREF. Left: evaluation set; Middle: development set; Right: human agreement set. understanding tasks are exclusive to in the evaluation set of HREF and can be considered held-out tasks. We generate a baseline model …

**Figure 6.** Figure 6: P-values of paired T-test on annotations across 13 models on the evaluation and development set. Evaluation set on the left; Development set on the right. We show the average and 90th and 80th quantile p-values from doing paired t-test among all model pairs among 13 mo…

**Figure 7.** Figure 7: Length Bias Rate of Different LLM Judges. It is clear that Llama-3.1-70B Instruct has the least length bias, and such bias is further reduced when using human-written responses as additional context. Relevance of HREF. As the size of the model pool and the strength of …

**Figure 8.** Figure 8: Impact of Changing Baseline Model. The average human agreement rates of various evaluators using two different baseline models. We observe very similar trends when using Llama3.1-405B-Instruct-FP8 and GPT-4-Turbo as the baseline model. A Less Costly Judge Model. Refer…

**Figure 9.** Figure 9: Prompt Template For LLM-as-a-Judge with Human Response. The prompt template we use to prompt our judge model Llama-3.1-70B-Instruct to give the preference between two model responses along with human reference. Note that we intentionally transform the guidelines we giv…

**Figure 10.** Figure 10: Prompt Template For LLM-as-a-Judge. The prompt template we use to prompt our judge model Llama-3.1-70B-Instruct to give the preference between two model responses without a reference. Note that we intentionally transform the guidelines we give to the human annotators …

**Figure 11.** Figure 11: Prompt Template with demonstration examples replaced. A modified version of the prompt template we use to prompt our judge model Llama-3.1-70B-Instruct to give the preference between two model responses with a reference. We replace the demonstrations examples with the…

**Figure 12.** Figure 12: Prompt Template with demonstration examples removed. A modified version of the prompt template we use to prompt our judge model Llama-3.1-70B-Instruct to give the preference between two model responses with a reference. We removes the demonstration examples. 24 [PITH…

**Figure 13.** Figure 13: Prompt Template from AlpacaEval. A modified version of the prompt template we use to prompt our judge model Llama-3.1-70B-Instruct to give the preference between two model responses with a reference. We adopt the exactly prompt that AlpacaEval uses for their judge LLM…

**Figure 14.** Figure 14: Guideline for Human Annotator. The guideline we provide for the human annotators. A modified version from Li et al. (2023). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

**Figure 15.** Figure 15: Annotation Website. The main pages of the website we build for collecting human annotations. The website framework is adopted from Miranda et al. (2024). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 3 canonical work pages

[1]

Phi-3 technical report: A highly capable language model locally on your phone

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

arXiv 2024
[2]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[3]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

arXiv 2023
[4]

Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.\ 65--72, 2005

2005
[5]

Pythia: A suite for analyzing large language models across training and scaling

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

2023
[6]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005
[7]

Gonzalez, Ion Stoica, and Eric P

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ URL https://lmsys.org/blog/2023-03-30-vicuna/

2023
[8]

Gonzalez, and Ion Stoica

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024

2024

Show all 38 references

[9]

Free dolly: Introducing the world’s first truly open instruction-tuned llm

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Company Blog of Databricks, 2023

2023
[10]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[11]

Length-controlled alpacaeval: A simple way to debias automatic evaluators

Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

2024 arXiv
[12]

Prolific first, 2014

Prolific First. Prolific first, 2014. URL https://www.prolific.com/

2014
[13]

Koala: A dialogue model for academic research

Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https://bair.berkeley.edu/blog/2023/04/03/koala/

2023
[14]

Olmo: Accelerating the science of language models

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

2024 arXiv
[15]

Mistral 7b

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

arXiv
[16]

T " ulu 3: Pushing frontiers in open language model post-training

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

2024 arXiv
[17]

From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

2024 arXiv
[18]

Hashimoto

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

2023
[19]

Wildbench: Benchmarking llms with challenging tasks from real users in the wild

Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770, 2024

2024 arXiv
[20]

Rouge: A package for automatic evaluation of summaries

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

2004
[21]

Roberta: A robustly optimized BERT pretraining approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/abs/1907.11692

1907 arXiv
[22]

Hybrid preferences: Learning to route instances for human vs

Lester James V Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Hybrid preferences: Learning to route instances for human vs. ai feedback. arXiv preprint arXiv:2410.19133, 2024

arXiv
[23]

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022

2022
[24]

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.\ 311--318, 2002

2002
[25]

Instruction tuning with gpt-4

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

2023 arXiv
[26]

Rush, and Thomas Wolf

Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https://huggingface.co/datasets/HuggingFaceH4/no_robots, 2023

2023
[27]

Llama 2: Open foundation and fine-tuned chat models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[28]

Self-instruct: Aligning language models with self-generated instructions

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022

2022 arXiv
[29]

Wizard LM : Empowering large pre-trained language models to follow complex instructions

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizard LM : Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. ...

2024
[30]

Yi: Open foundation models by 01

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652, 2024

2024 arXiv
[31]

Bertscore: Evaluating text generation with bert

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

1904 arXiv
[32]

Wildchat: 1m chatgpt interaction logs in the wild

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470, 2024

2024 arXiv
[33]

Judging llm-as-a-judge with mt-bench and chatbot arena

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 0 46595--46623, 2023

2023
[34]

Lima: Less is more for alignment

Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

2024
[35]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[36]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[37]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[38]

Hide and Seek

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Phi-3 technical report: A highly capable language model locally on your phone

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

arXiv 2024

[2] [2]

Gpt-4 technical report

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[3] [3]

Qwen technical report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

arXiv 2023

[4] [4]

Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.\ 65--72, 2005

2005

[5] [5]

Pythia: A suite for analyzing large language models across training and scaling

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

2023

[6] [6]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

arXiv 2005

[7] [7]

Gonzalez, Ion Stoica, and Eric P

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90\ URL https://lmsys.org/blog/2023-03-30-vicuna/

2023

[8] [8]

Gonzalez, and Ion Stoica

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024

2024