REVIEW 41 cited by
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available at https://github.com/prometheus-eval/prometheus-eval.
Forward citations
Cited by 41 Pith papers
-
Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence
Apodex Discovery introduces the TRACES benchmark of 17 executable, verifiable environments with hidden outcomes and the HDS6 process-evaluation metric, reporting early gains for environment-equipped agents on AAV caps...
-
SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning
SERPO co-evolves per-question grading rubrics, Good-Normal-Bad response archives, and policy parameters so a language model can train itself at inference time without labels, gaining up to 20.6 points on open-ended me...
-
Autoregressive Modeling of Film with Applications in Video Montage
An autoregressive transformer with an explicit cut token and footage-constrained decoding edits raw video into sequences that people rate as better than two prior automated editing methods.
-
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
JETTS, a new benchmark, shows LLM-as-judges are competitive in response reranking, worse than process reward models in beam search, and ineffective as critique providers for refinement.
-
Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
Latent DPO trains only a small preference encoder per user, reducing LLM personalization training time by 80-90% with comparable alignment quality.
-
From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
Using per-example in-context demonstrations built from historical same-source human MQM ratings makes an LLM judge dramatically better at fine-grained MT evaluation on WMT'23 and WMT'24.
-
Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software
REAG and a confidence-calibrated cascade generate context-aware test oracles for LLM-based software and produce statistically controlled verdict reliability, demonstrated on a production nutrition advisory app.
-
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution
SkillTV-Bench provides a multi-domain, skill-aware trajectory verification benchmark, and SkillTV-Evolve's evolved JudgeSkill improves an agent judge's accuracy by 14.8 points on a held-out set.
-
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Distilling agreement-filtered multi-LRM jury annotations into Gemma-3-12B improves MQM translation quality evaluation from 52.63% to 55.03% average segment-level accuracy, approaching closed LRMs.
-
Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs
Multi-modal RAG (text plus UI screenshots) with reward-based polishing generates acceptance criteria from user stories that three industry experts rated near 4/5 on relevance, correctness, and understandability.
-
Interpretable Mnemonic Generation for Kanji Learning via Expectation-Maximization
An EM-type algorithm jointly learns interpretable mnemonic rules and learner/kanji traits, yielding modest cold-start gains over fine-tuning for kanji mnemonic generation.
-
Hierarchical Memory Organization for Wikipedia Generation
Generating Wikipedia articles with factoid memory units organized into a hierarchical outline improves informativeness, verifiability, and citation coverage over RAG and STORM baselines.
-
FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes
A new India-focused benchmark shows that popular LLMs exhibit measurable negative bias against marginalized Indian identities and frequently reinforce caste, religion, region, and tribe stereotypes.
-
MinosEval: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLMs
MinosEval improves open-ended QA evaluation by sorting questions into factoid and non-factoid and applying tailored scoring, outperforming baselines on four datasets.
-
Outside Knowledge Conversational Video (OKCV) Dataset -- Dialoguing over Videos
OKCV is a new human-annotated video dialogue dataset where answering questions requires both visual grounding in the video and external knowledge.
-
Improving Fairness of Large Language Models in Multi-document Summarization
FairPO combines document-set perturbation with DPO-style preference tuning and corpus-level dynamic weighting to improve both summary-level and corpus-level fairness in multi-document summarization.
-
DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi-Dimensional Bias Analysis
Caste-based stereotypes are measurably present in widely used LLMs, with the largest bias appearing when Dalits and Shudras are compared with dominant castes.
-
Safety Degradation in AI Agents
Adding retrieval to aligned LLMs degrades safety: refusal rates fall, bias and harmfulness rise, and prompt-based mitigation only partially restores alignment.
-
Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models
A fully automatic LLM evaluation framework where all evaluated models serve as judges for one another reaches 97% Spearman correlation with human preference rankings while keeping cost sub-quadratic.
-
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines
A new dataset of production-style prompts with matching assertion criteria, and a benchmark where fine-tuned 7-8B models beat GPT-4o at generating those criteria.
-
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
Persona-judge applies speculative decoding between two preference-prompted copies of the same LLM to achieve training-free personalized alignment.
-
Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
In healthcare LLM evaluation, multiple-choice accuracy and open-ended task scores correlate only weakly, and the paper's proposed Relaxed Perplexity metric aims to improve open-ended factuality scoring but rests on an...
-
Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering
Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.
-
SedarEval: Automated Evaluation using Self-Adaptive Rubrics
A benchmark and judge model that uses per-question custom rubrics to score LLM outputs, claiming better alignment with human grading than GPT-4.
-
CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues
An LLM-based pipeline adds synthetic misunderstandings, non-understandings, and vaguely related questions plus repair responses to MultiWOZ dialogues, and the released CoPrUS-MultiWOZ dataset preserves task performanc...
-
Copyright-Protected Language Generation via Adaptive Model Fusion
CP-Fuse adaptively fuses two models trained on disjoint data to suppress verbatim reproduction of memorized text without a measured utility drop.
-
Self-Generated Critiques Boost Reward Modeling for Language Models
Self-generated, label-filtered critiques jointly trained with reward prediction improve preference accuracy and data efficiency without a stronger teacher model.
-
Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering
On a new 3000-item audio-only benchmark stripped of text-solvable questions, the best AI system scores 58.33%, random guessing scores 25.46%, and all 36 competing systems fail the same 233 items.
-
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Judge upgrades are not interchangeable: only Qwen3 1.7B→4B yields robust adjacent gains, MiniMax adjacent releases do not, and stronger judges reduce but do not remove bias or correlated jury errors.
-
Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
Hybrid RAG over UK public health guidance sharply raises MCQA accuracy and free-form faithfulness, letting smaller open models match larger closed models without retrieval.
-
BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges
BACON calibrates multiple AI judges against a small human-labeled sample, then uses cross-fitted outcome models and augmented estimating equations to produce calibrated summary estimates and item-level surrogate scores.
-
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation
PrefBERT, a 150M-parameter reward model trained on human quality ratings, outperforms ROUGE-L and BERTScore as a GRPO reward signal for open-ended long-form generation.
-
A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations
A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.
-
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
An open-source LLM judge for science answers becomes less optimistic after supervised fine-tuning plus contrastive reinforcement learning on benign and adversarially perturbed answers.
-
Trillion 7B Technical Report
Trillion-7B pairs Korean documents with English documents during pretraining and lets them attend to each other, claiming competitive Korean performance with only about 10% multilingual tokens.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
-
Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Styles
Multi-LLM response agreement plus focal loss in an auxiliary BERT confidence model improves LLM calibration across 12 models and four prompt styles.
-
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
A label taxonomy and answer-first generation strategies help RAG developers build evaluation datasets whose question mix matches real usage.
-
FHIR-RAG-MEDS: Integrating HL7 FHIR with Retrieval-Augmented Large Language Models for Enhanced Medical Decision Support
FHIR-RAG-MEDS integrates HL7 FHIR patient summaries into a RAG system and reports improved guideline-based recommendation quality over bare medical LLMs across four clinical domains.
-
Atla Selene Mini: A General Purpose Evaluation Model
The paper presents Selene Mini, an 8B open-weights judge model that reports state-of-the-art average scores across 11 LLM evaluation benchmarks, with gains on medical and financial expert agreement.
-
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Trained reference-free judge models (10M–184M params) achieve 0.747 Pearson correlation with ground-truth quality on QA, matching or exceeding reference-based evaluators for decentralized LLM inference.
Discussion (0). Continue with ORCID to comment.