Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

Atla Selene Mini: A General Purpose Evaluation Model

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces an 8-billion-parameter open-weights judge model that outperforms existing small judges and GPT-4o-mini on the average of 11 evaluation benchmarks, and claims the top spot among 8B generative models on RewardBench.

desk verdict Useful open judge model, but the headline benchmark superiority is a point estimate within noise and the abstract's RewardBench claim contradicts Table 1. read the letter →

arxiv 2501.17195 v1 pith:YOK4N5EJ submitted 2025-01-27 cs.CL cs.AI

Andrei Alexandru , Antonia Calvi , Henry Broomfield , Jackson Golden , Kyle Dai , Mathias Leys , Maurice Burger , Max Bartolo
show 4 more authors
Roman Engeler Sashank Pisupati Toby Drane Young Sun Park
This is my paper · ORCID
classification cs.CLcs.AI
keywords LLM-as-a-judgesmalllanguagemodeldirectpreferenceoptimizationsyntheticdatacurationevaluationbenchmarksRewardBenchpromptrobustnessopen-weight
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a capable, general-purpose LLM-as-a-judge can be built at 8 billion parameters rather than requiring frontier-scale models. It presents Selene Mini, a judge fine-tuned from Llama 3.1 8B Instruct, claims it beats the best existing small-language-model judges and GPT-4o-mini on the average of 11 evaluation benchmarks, and claims the top score among 8B generative models on RewardBench. The wider importance is practical: if true, reliable automated evaluation can be run locally, cheaply, and with open weights, and the main lever is data curation and a hybrid training objective rather than model size. The paper also reports gains in zero-shot agreement with human expert labels on finance and medical datasets and stable performance when prompt formatting changes.

What carries the argument

The load-bearing mechanism is the data-and-loss recipe, not a new architecture. Each training point pairs a chosen evaluation, which argues for the ground-truth label or a correct score, with a rejected evaluation arguing for a wrong label or a score two points off; both are written as chain-of-thought critiques with a final judgment. The model is trained with $L_{\text{DPO+NLL}} = L_{\text{DPO}} + \alpha\,L_{\text{NLL}}$, where the extra negative log-likelihood term is applied only to chosen responses. Before training, a reward model filters low-quality raw examples and a prompted consistency checker removes synthetic critiques whose reasoning contradicts their assigned judgment. Around 70 percent of training pairs use the critique-plus-judgment format and 30 percent use judgments only, following the baseline judge recipe.

What would settle it

Check the 11 benchmark test sets for exact or near-duplicate overlap with the 16 training datasets; if any of MT-Bench, RewardBench, FLASK, or the others appear in training, the out-of-distribution premise fails. Then rerun the full 11-benchmark comparison several times with different seeds or checkpoints to see whether the roughly 0.007 average margin over the closest baseline survives run-to-run noise.

Watch

Extended reading notes

Core claim

Selene Mini is a fine-tuned Llama 3.1 8B Instruct model trained on 16 public datasets that were augmented with synthetic chosen and rejected critiques, then filtered for quality. On the paper's headline comparison, it scores 0.756 on the unweighted average across 11 benchmarks, ahead of SFR-LLaMA-3.1-8B-Judge (0.749) and GPT-4o-mini (0.743). It also reports the highest score among 8B generative models on RewardBench, surpassing GPT-4o and specialized judges in that comparison. The authors attribute the improvement to their curation pipeline: reward-model filtering of raw data, a critique-consistency checker, and a DPO objective that adds a negative log-likelihood term on chosen responses so the margin over rejected critiques is widened while good critiques become more probable.

Load-bearing premise

The central claim depends on treating an unweighted average of scores on 11 chosen benchmarks as the definition of general-purpose evaluation quality, and on those benchmarks being genuinely outside the 16 training datasets.

Editorial extensions

If this is right

  • An 8B open-weights judge can match or exceed proprietary mini judges on general evaluation tasks, lowering the cost and latency of automated evaluation.
  • The same model can plausibly serve as a reward signal for preference optimization, given its top ranking among 8B generative models on RewardBench.
  • Practitioners should be able to vary prompt templates in production without retraining, since Selene Mini's score stays roughly stable across the tested formats.
  • The zero-shot gains on finance and medical expert-labeled data suggest the training recipe transfers beyond academic benchmarks into regulated, domain-specific settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 'best overall' claim is an equal-weight average over the 11 benchmarks; a user who weights absolute scoring, classification, and pairwise tasks differently could see a different ordering, and the paper itself notes that practitioners prefer absolute scoring.
  • The 0.007 gap in overall average over the closest baseline is small enough that run-to-run variance or a different benchmark mix could plausibly change the rank order, so the stability of the headline result is not yet established.
  • Because the weights are open, the community can rerun the evaluations and extend them to new domains; that independent evidence, rather than the paper's own runs, may settle how general the capability really is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces Atla Selene Mini, an 8B-parameter Llama-3.1-based language-model judge trained on a curated mixture of 16 public datasets augmented with synthetically generated chosen/rejected chain-of-thought critiques, filtered by a reward model and consistency checks, and optimized with a DPO+NLL objective. The central claim is that the resulting model is a general-purpose evaluator that outperforms existing small language-model judges and GPT-4o-mini on an equal-weight average over 11 benchmarks covering absolute scoring, pairwise preference, and classification, and is the highest-scoring 8B generative model on RewardBench. Additional claims concern zero-shot agreement on finance and medical expert labels, robustness to prompt formatting, and top ranking in the authors' community Judge Arena.

Significance. If fully substantiated, an open-weights 8B judge that matches or beats GPT-4o-mini and specialized judges on a broad benchmark suite would be practically significant, lowering the cost of automated evaluation and enabling local deployment. The paper's strengths are its reproducible release of weights, explicit ablations of reward-model filtering and dataset inclusion, and a reasonably detailed curation pipeline. The main weakness is that the headline comparative claim rests on a 0.007 average margin with no statistical uncertainty, and some abstract-level factual claims are contradicted by the paper's own tables. The contribution is primarily empirical; the method itself is a combination of known techniques.

major comments (4)
  1. [Abstract; §3.1, Tables 1 and 3] The abstract's RewardBench claim is contradicted by the paper's own data. Table 1 reports RewardBench scores of 0.688 for Selene Mini and 0.689 for SFR-LLaMA-3.1-8B-Judge, so Selene Mini is not the highest-scoring 8B generative model on RewardBench among the models listed. Similarly, §3.1 states that Selene Mini beats GPT-4o on RewardBench, EvalBiasBench, and Auto-J, but Table 3 shows GPT-4o at 0.765, 0.932, and 0.769 versus Selene Mini's 0.688, 0.900, and 0.732, respectively. These claims need to be corrected or removed; as written, the headline superiority statement is not supported.
  2. [§3.1, Table 1] The overall comparison is a single-run point estimate without error bars, confidence intervals, or significance tests, and the margin over SFR-LLaMA-3.1-8B-Judge is 0.007 (0.756 vs 0.749). Moreover, several baseline rows marked with a dagger are taken from external technical reports, so the comparison mixes evaluation harnesses, prompts, and parsing conventions. Since a small perturbation in one benchmark could flip the ranking, the claim that Selene Mini outperforms the best SLMJs is not established. Please report per-run variance or repeated evaluations with different seeds, and either reproduce all baselines under identical conditions or restrict the comparative claim to the subset that was run in-house.
  3. [§2.1, §3.1, Appendix A] The out-of-distribution status of the 11 evaluation benchmarks is asserted but not verifiable. The training mixture is described only as 16 public datasets inspired by FLAMe, and Appendix A contains an embedding visualization rather than a dataset list. Without enumerating the training datasets and, ideally, reporting overlap or contamination checks against MT-Bench, RewardBench, FLASK, HHH, and the other evaluation benchmarks, the OOD premise cannot be assessed. Please disclose the composition of the training mix and any decontamination procedure.
  4. [§3.3, Abstract] The Judge Arena evidence is self-referential: the authors developed the arena and report an early snapshot of their own model as top-ranking. This is not independent validation. At minimum, state the number of votes, the evaluation procedure, and the relationship between the authors and the platform, or treat the arena result as anecdotal rather than as part of the headline evidence.
minor comments (5)
  1. [§3.2.2] The text says 'six different prompt formats' but lists only five (original, markdown, JSON, PrePair, and simplified instructions); please correct the count or add the missing format.
  2. [Tables 1 and 3] The column label 'RewardB' is ambiguous; consider writing 'RewardBench' in full for readability.
  3. [References] Reference [25] contains a typo in 'Foundation'; please fix it.
  4. [Appendix C, Figure 6] The filtering ablation results in Figure 6 are reported without error bars or sample sizes; adding them would strengthen the conclusion that the effects are dataset-dependent.
  5. [§2.2] The rejected-judgment sampling for absolute-scoring tasks is described only for a 1–5 scale; please clarify how it generalizes to other numeric scales used in the evaluation suite.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Selene Mini's headline results are external benchmark evaluations, and the Judge Arena self-citation is peripheral.

full rationale

The paper's central claim, that Selene Mini outperforms other SLMJs and GPT-4o-mini on an average of 11 benchmarks, is supported by Table 1, which reports externally defined metrics (Pearson correlation for absolute scoring, accuracy for pairwise and classification tasks). The training procedure, DPO+NLL on 16 public datasets augmented with synthetic critiques, is described independently of the evaluation targets; no fitted parameter or loss term is defined in terms of the 11 benchmark scores. The only self-referential element is the Judge Arena [12], a platform the authors developed and cite for a preliminary ranking claim; that claim is presented as preliminary and does not enter the derivation of the benchmark results. The paper does not enumerate the 16 training datasets, so the 'out-of-distribution' assertion cannot be independently checked, but an unverified contamination claim is not a circularity reduction. No equation, fitted value, or self-citation carries the load of the benchmark comparisons, so no circular step can be exhibited.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical rather than derivational, so the main loaded assumptions are training hyperparameters, dataset-inclusion decisions, and the representativeness of the chosen benchmarks. No new theoretical entities are introduced; the released model is an artifact, not an invented entity.

free parameters (6)
  • Learning rate = 1e-7
    Selected from {5.5e-8, 1e-7, 7e-7} in Section 2.4; affects the final model.
  • RPO alpha = 1.0
    Weight on the NLL term in Eq. (1); swept over {0.5, 1} in Section 2.4.
  • Weight decay = 0.1
    Swept over {0.01, 0.1} in Section 2.4.
  • ArmoRM filtering thresholds = Dataset-dependent, values not reported
    Section 2.3 sets thresholds by ablation for four large datasets; exact values are not given.
  • Rejected judgment offset = 2 points
    Section 2.2 samples rejected absolute scores 2 points from ground truth; a design choice influencing preference pairs.
  • Chain-of-thought example ratio = 0.7
    Section 2 sets 70% CoT versus 30% judgment-only training pairs, following prior work.
assumptions (5)
  • domain assumption The DPO plus NLL loss in Eq. (1) is a valid training objective for judge models.
    The paper cites prior work for the loss and does not re-derive it; its suitability for evaluation training is assumed.
  • domain assumption Human preference labels in the 16 public datasets are reliable ground truth for evaluation quality.
    Section 2.1 builds chosen/rejected pairs from these labels without validating them independently.
  • domain assumption ArmoRM reward scores can select high-quality training data.
    Section 2.3 uses ArmoRM filtering on four datasets and relies on its scores correlating with data quality.
  • ad hoc to paper Synthetic critiques that argue for a given judgment carry valid supervision.
    Section 2.2 generates critiques with an unspecified model; no independent evidence is given that this improves human agreement beyond the paper's own benchmarks.
  • ad hoc to paper The 11 evaluation benchmarks are out-of-distribution relative to the 16 training datasets.
    Section 3.1 asserts OOD status, but the training data are not enumerated and no contamination analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Atla Selene Mini: A General Purpose Evaluation Model." pith.science (2026). https://pith.science/paper/YOK4N5EJ

@misc{pith2026250117195,
  author       = {Pith},
  title        = {Pith review of: Atla Selene Mini: A General Purpose Evaluation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOK4N5EJ}},
  note         = {Machine review of arXiv:2501.17195}
}
read the original abstract

We introduce Atla Selene Mini, a state-of-the-art small language model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that outperforms the best SLMJs and GPT-4o-mini on overall performance across 11 out-of-distribution benchmarks, spanning absolute scoring, classification, and pairwise preference tasks. It is the highest-scoring 8B generative model on RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To achieve this, we develop a principled data curation strategy that augments public datasets with synthetically generated critiques and ensures high quality through filtering and dataset ablations. We train our model on a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and produce a highly promptable evaluator that excels in real-world scenarios. Selene Mini shows dramatically improved zero-shot agreement with human expert evaluations on financial and medical industry datasets. It is also robust to variations in prompt format. Preliminary results indicate that Selene Mini is the top-ranking evaluator in a live, community-driven Judge Arena. We release the model weights on HuggingFace (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage widespread community adoption.

Figures

Figures reproduced from arXiv: 2501.17195 by the authors.

Figure 1
Figure 1. Atla Selene Mini outperforms current state-of-the-art SLMJs: a) Overall task-average performance, comparing Atla Selene Mini (black) with the best and most widely used SLMJs. b) Breakdown of performance by task type and benchmark – see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data curation strategy: The process of transforming a candidate dataset (left) into the final training mix (right). Yellow boxes indicate filtering steps, purple represents synthetic generation of chosen and rejected pairs (blue and red) for preference optimization, and red circles highlight ablation-informed decisions, such as reward thresholds and dataset inclusion. 30% with judgments only. Once curated, we ran ab… view at source ↗
Figure 3
Figure 3. Real-world evaluation: a) Performance on domain-specific industry benchmarks of Atla Selene Mini (black) compared to base model (orange) measured in accuracy. Trained model shows higher expert agreement on FinanceBench, a financial benchmark, and CRAFT-MD, a medical dataset. b) Performance on RewardBench of Atla Selene Mini compared to base model, when prompt format is changed. Trained model shows consistent improve… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Training dataset map: Topic-stratified, two-dimensional embedding representation of Atla Selene Mini’s training dataset generated using Nomic Atlas [33]. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Example data point: Training example from FEEDBACKCOLLECTION [34], including the reference response, which is an optional field for Atla Selene Mini. This instance uses a similar prompt template to [10]. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Reward model filtering: Effects of reward model (RM) filtering on single dataset ablations. Bars show difference on accuracy (black) and Pearson correlation (green) metrics between RM-filtered and random subsets of data. We observed that effects were dataset-dependent,…
Figure 7
Figure 7. Figure 7: JSON prompt template PrePair You are a helpful assistant in evaluating the quality of the responses for a given instruction. Your goal is to select the best response for the given instruction. Select Response A or Response B, that is better for the given instruction. T…
Figure 8
Figure 8. Figure 8: PrePair prompt template 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Markdown prompt template Simplified instructions You are tasked with evaluating two responses — Response A and Response B — to determine which one better follows the given instruction. Both responses come from different AI chatbots. **You must pick one. Do not say both…
Figure 10
Figure 10. Figure 10: Simplified instructions prompt template 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Matched two-pass experiments show human revisers improve on objective and subjective tasks, while LLM self-revision yields near-zero information gain on objective tasks and negative information gain on subjective task...

  2. Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Persistent SAEs learn per-feature persistence coefficients from reconstruction, splitting features into fast local detectors and slow topic-tracking states that retain prompt-injection signals over long contexts.

  3. Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

    cs.CL 2026-03 conditional novelty 6.0 of 10

    JudgeBiasBench measures 12 LLM-judge bias types under a 4-dimension taxonomy, and bias-aware GRPO/InfoNCE training cuts BSR substantially while preserving general evaluation performance.

  4. ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments

    cs.CL 2025-05 conditional novelty 6.0 of 10

    ReliableEval estimates the minimum number of meaning-preserving prompt resamplings needed to make an LLM evaluation reliable, and applies it to show frontier LLMs are notably prompt-sensitive.

  5. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    cs.LG 2026-02 conditional novelty 4.0 of 10

    A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.

  6. Reward Reasoning Model

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Reward models that think before judging, trained via reinforcement learning without human-written reasoning traces, outperform standard reward models and improve with more test-time compute.

Reference graph

Works this paper leans on

34 extracted references · 18 canonical work pages · cited by 6 Pith papers

  1. [1]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  2. [2]

    Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    arXiv 2023

  3. [3]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594, 2024

    arXiv 2024

  4. [4]

    Offsetbias: Leveraging debiased data for tuning evaluators, 2024

    Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024

    work page 2024

  5. [5]

    Self-preference bias in llm-as-a-judge, 2024

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge, 2024

    work page 2024

  6. [6]

    Judging the judges: Evaluating alignment and vulnerabilities in llms-as- judges, 2024

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the judges: Evaluating alignment and vulnerabilities in llms-as- judges, 2024

    work page 2024

  7. [7]

    Foundational autoraters: Taming large language models for better automatic evaluation

    Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.10817, 2024

    arXiv 2024

  8. [8]

    Flow judge: An open small language model for llm system evaluations

    Flow AI. Flow judge: An open small language model for llm system evaluations. https: //www.flow-ai.com/blog/flow-judge , 2024

    work page 2024

Show all 34 references
  1. [9]

    Glider: Grading llm interactions and decisions using explainable ranking

    Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang, Bartosz Mielczarek, Anand Kan- nappan, and Rebecca Qian. Glider: Grading llm interactions and decisions using explainable ranking. arXiv preprint arXiv:2412.14140, 2024

    2024 arXiv

  2. [10]

    Direct judgement preference optimization

    Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement preference optimization. arXiv preprint arXiv:2409.14664, 2024

    2024 arXiv

  3. [11]

    Prometheus 2: An open source lan- guage model specialized in evaluating other language models

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source lan- guage model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024

    2024 arXiv

  4. [12]

    Judge arena: Benchmarking llms as evaluators

    Kyle Dai, Maurice Burger, Roman Engeler, Max Bartolo, Clémentine Fourrier, Toby Drane, Mathias Leys, and Jackson Golden. Judge arena: Benchmarking llms as evaluators. https: //huggingface.co/blog/arena-atla, 2024

    2024

  5. [13]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733, 2024

    2024 arXiv

  6. [14]

    Interpretable preferences via multi-objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024

    2024 arXiv

  7. [15]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  8. [16]

    Flask: Fine-grained language model evaluation based on alignment skill sets, 2024

    Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets, 2024

    2024

  9. [17]

    The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models, 2024

    Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko...

    2024

  10. [18]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024

    2024

  11. [19]

    A critical evaluation of evaluations for long-form question answering, 2023

    Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. A critical evaluation of evaluations for long-form question answering, 2023

    2023

  12. [20]

    A general language assistant as a laboratory for alignment, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam Mc...

    2021

  13. [21]

    Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan

    Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization, 2024

    2024

  14. [22]

    Generative judge for evaluating alignment, 2023

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment, 2023

    2023

  15. [23]

    Infobench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuan- sheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601, 2024

    2024 arXiv

  16. [24]

    Minicheck: Efficient fact-checking of llms on grounding documents, 2024

    Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents, 2024

    2024

  17. [25]

    Craft-md: A conversational evaluation framework for comprehensive assessment of clinical llms

    Shreya Johri, Jaehwan Jeong, Benjamin A Tran, Daniel I Schlessinger, Shannon Wongvibulsin, Zhuo Ran Cai, Roxana Daneshjou, and Pranav Rajpurkar. Craft-md: A conversational evaluation framework for comprehensive assessment of clinical llms. In AAAI 2024 Spring Symposium on Clin...

    2024

  18. [26]

    Financebench: A new benchmark for financial question answering

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

    2023 arXiv

  19. [27]

    Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates, 2024

    Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates, 2024

    2024

  20. [28]

    Does prompt formatting have any impact on llm performance?, 2024

    Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on llm performance?, 2024

    2024

  21. [29]

    The comparative trap: Pairwise comparisons amplifies biased preferences of llm evaluators, 2024

    Hawon Jeong, ChaeHun Park, Jimin Hong, Hojoon Lee, and Jaegul Choo. The comparative trap: Pairwise comparisons amplifies biased preferences of llm evaluators, 2024

    2024

  22. [30]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132, 2024

    2024 arXiv

  23. [31]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  24. [32]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025

  25. [33]

    Nomic atlas

    Nomic. Nomic atlas. https://atlas.nomic.ai/. Accessed: 2024-01-21

    2024

  26. [34]

    "Dear Readers, <omitted for conciseness> P.S. No garden gnomes were harmed in the writing of this book

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models, 2024. 10 Appendices A Training dataset embedding Fig...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.