Pith. sign in

REVIEW 4 major objections 6 minor 46 references

SedarEval: Automated Evaluation using Self-Adaptive Rubrics

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Per-question scoring rubrics, with explicit credit and deduction points, let a trained evaluator language model match human graders more closely than GPT-4 does.

desk verdict The self-adaptive rubric idea is real and the benchmark is a genuine asset, but the central claim of beating GPT-4 is contradicted by the paper's own Table 1 and the key comparison table is missing. read the letter →

arxiv 2501.15595 v1 pith:LZWWCXRH submitted 2025-01-26 cs.CV

Zhiyuan Fan , Weinong Wang , Xing Wu , Debing Zhang This is my paper · ORCID
classification cs.CV
keywords LLM-as-a-judgeself-adaptiverubricsevaluatorlanguagemodelautomatedevaluationhuman-AIconsistencybenchmarkconstructiondirectpreferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard LLM-as-judge approach fails because it scores every answer with the same generic rubric, so the judge has to supply the actual assessment criteria from its own knowledge. The proposed fix is a self-adaptive rubric: for each of 1,000 benchmark questions, a human-written guide lists the primary and secondary scoring points, penalty deductions, and background knowledge, turning evaluation into a rule-following task. Using these rubrics, the authors train a compact evaluator language model and filter its training data with a 'Human-AI Consistency' step that keeps only examples where GPT-4's chain-of-thought score agrees with the human score. They report that this evaluator matches human grading with higher Pearson correlation, better good-same-bad ranking agreement, and higher accuracy than GPT-4 on the same data, at lower cost and without closed-source APIs. If true, automated evaluation could shift from generic preference-matching to transparent, question-specific scoring.

What carries the argument

The load-bearing mechanism is the self-adaptive rubric: a per-question scoring guide, written by human annotators, that decomposes the mark into primary and secondary credit points, explicit deduction points for factual errors, and background context, so scoring becomes an instruction-following exercise rather than a reasoning exercise left to the judge. Alongside it, the Human-AI Consistency filter (Equation 2) selects only training samples where GPT-4's chain-of-thought final score matches the human score, aligning synthetic reasoning traces with human judgment. The evaluator LM itself is trained by SFT from a pretrained base with a mix of general instruction data and evaluator-format data, rather than by continual fine-tuning of a chat model, which the paper argues preserves general ability while adding scoring skill. For automatic rubric generation, the paper uses SFT followed by Direct Preference Optimization (DPO) to align generated rubrics with human-written ones.

What would settle it

Compare the trained evaluator LM's scores with human scores on models held out from training (the model-level out-of-distribution condition of Section 6.3); the paper's own Table 1 reports Pearson 0.380 for its XD model versus 0.822 for GPT-4 in that condition, so reproducing this number would directly test whether the claimed superiority over GPT-4 extends beyond the training distribution.

Watch

Extended reading notes

Core claim

The central discovery is that question-specific 'self-adaptive rubrics' — structured lists of scoring points, penalty points, and background knowledge written for each individual question — make an LLM judge's scores agree with human graders more closely than the same judge using a one-size-fits-all rubric, and that a compact evaluator LM trained to follow these rubrics can exceed GPT-4's agreement with human scores on the SedarEval benchmark (1,000 questions across long-tail knowledge, math, coding, and logic). Building on this, the paper introduces a Human-AI Consistency filter that discards synthetic chain-of-thought scoring samples unless GPT-4's final score equals the human score, and shows this filter improves both evaluation accuracy and the model's general capability. The paper presents this as evidence that evaluation can be made more precise, stable, interpretable, and cheaper than relying on proprietary judges.

Load-bearing premise

The paper's result depends on human-written rubrics and human scores being the correct ground truth, and on the Human-AI Consistency filter not quietly discarding difficult cases so that agreement with humans is higher simply because the training data was easier.

Editorial extensions

If this is right

  • A trained evaluator LM with self-adaptive rubrics could replace closed-source judge APIs for internal model development, cutting cost and avoiding leakage of proprietary evaluation data.
  • Each score comes with a diagnostic breakdown of which rubric points the model earned and lost, so model developers can see which skills need work rather than receiving a single number.
  • The reported scaling law suggests that a few hundred rubricated questions are enough for a stable ranking of models, so the per-round cost of benchmarking new models stays modest.
  • Rubric generation can be automated through SFT followed by DPO, allowing the paradigm to be extended to new tasks without hand-writing every rubric.
  • Direct-assessment scores themselves can be converted into pairwise comparisons, so the same benchmark supports both absolute scoring and head-to-head model comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Human-AI Consistency filter is a general data-selection idea: any pair of graders (two LLMs, or one LLM and humans) could be filtered by agreement, which may reduce noise in other judge-training pipelines without needing a proprietary teacher.
  • The paper's own model-level out-of-distribution table reports a Pearson correlation of 0.380 for its evaluator versus 0.822 for GPT-4, so the claimed superiority appears to be tied to the training distribution; a direct extension would be to test whether adding more model diversity to the training pool closes that gap.
  • Because each rubric item maps to a specific deduction or credit, an adversarially modified rubric could be used to verify whether the judge truly follows the rubric point by point rather than pattern-matching the overall response.
  • For subjective or creative writing tasks, where the paper acknowledges graders disagree, the self-adaptive rubric idea generalizes to user- or group-specific rubrics, turning evaluation into a configurable process rather than a single global standard.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a self-adaptive rubric paradigm for LLM-as-judge evaluation, in which each benchmark question is paired with a hand-written rubric specifying scoring points, penalty points, and background knowledge. It introduces SedarEval, a 1,000-question benchmark across eight categories, collects responses from 20 LLMs with human scores, and trains an evaluator LM using GPT-4 chain-of-thought traces filtered by a proposed Human-AI Consistency strategy. The central claim is that the resulting evaluator LM surpasses GPT-4 in model-level and question-level Pearson correlation, GSB, and ACC agreement with human grading. The paper also studies training-phase choices, automatic rubric generation, and joint versus expert training. The dataset release and the general idea of fine-grained per-question rubrics are potentially useful, but the headline empirical claim is not supported by the presented evidence.

Significance. If the paradigm and the benchmark are valid, the per-question rubric idea is a genuine contribution to interpretable LLM evaluation: it makes the scoring process more transparent and could reduce dependence on proprietary judges. The effort to build 1,000 rubric-annotated questions and collect human scores across 20 models is substantial, and the authors state that the dataset is released. The scaling-law analysis in Section 6.1 and the component ablation in Table 3 are informative. However, the central contribution, namely demonstrating superiority over GPT-4 and over general-rubric paradigms, is not established: the only direct GPT-4 comparison in the manuscript contradicts the headline claim, and the promised general-rubric comparison is missing. As it stands, the paper cannot support its stated main conclusion.

major comments (4)
  1. [§1, Abstract, §6.3 (Table 1)] The central claim that the evaluator LM 'surpasses GPT-4 in model-level and question-level Pearson correlation, GSB, and ACC metrics' is contradicted by the paper's own Table 1. In that table, XD attains question-level Pearson 0.738 versus GPT-4's 0.822 and model-level Pearson 0.380 versus GPT-4's 0.822; the GSB and ACC/ACC(t) columns are identical for the two models. No row or column of Table 1 shows XD exceeding GPT-4. The headline claim is therefore unsupported by the one quantitative comparison the manuscript actually reports.
  2. [§6.6] The dedicated comparison against the general-rubric paradigm, which is needed to support the abstract's claim of a 'higher concordance rate with human grading results than other paradigms,' is missing: the text refers to 'Table ??'. This comparison is the load-bearing experiment for the paper's central contribution, and the reader cannot verify it from any reported table or appendix. The missing table cannot be reconstructed from the numbers given elsewhere.
  3. [§4.2, §6.4] The Human-AI Consistency strategy filters the training data by retaining only GPT-4 scoring traces whose final scores match human scores exactly (Equation 2). Because the evaluator LM is trained on this filtered subset and then evaluated on human-agreement metrics, the reported agreement may in part reflect the selection criterion itself rather than the self-adaptive rubric paradigm. The text states that comparisons against raw chain-of-thought data and rejection sampling were performed, but Table 2 does not report a raw-CoT baseline, so the marginal effect of the filter versus the selection bias cannot be assessed from the manuscript.
  4. [§3.2] The dataset construction retains only questions with large score variance across five probe LLMs and requires unanimous annotator agreement. This is reasonable for building a discriminative benchmark, but it means the SedarEval test distribution is a curated subset rather than a random sample of realistic questions. Consequently, the out-of-distribution generalization claims in §6.3 and the stability analysis in §6.1 should be qualified, and ideally the evaluator should also be tested on an unfiltered question set before claiming that the paradigm generalizes to arbitrary evaluation tasks.
minor comments (6)
  1. [§4.1, Equation 1] Equation 1 is algebraically incorrect as printed: C(n,2) equals n!/(2!(n-2)!) = n(n-1)/2, not that quantity minus n. The expression (n^2 - 3n)/2 is the number of pairwise comparisons after an additional subtraction of n, so the equality in the displayed formula should be stated as C(n,2) - n, not C(n,2).
  2. [§3.1] The text says a typical self-adaptive rubric is 'illustrated in Table 3,' but Table 3 is the ablation study; the intended reference is probably Figure 1 or Figure 5.
  3. [Conclusion and Limitations] The benchmark is called SedarEval in the title, abstract, and Section 3, but the Conclusion and Limitations sections refer to it as INSDA; this inconsistency should be reconciled.
  4. [§5.2] The metric ACC_t is defined in Equation 6 with a tolerance epsilon, but the value of epsilon is never specified, making the reported ACC(t) numbers in Tables 1-3 non-reproducible.
  5. [§6.3, Table 1] The GPT-4 row in Table 1 reports identical values for the question-level and model-level columns across all metrics, which is surprising and should be explained, since the two evaluation dimensions are supposed to be different.
  6. [Throughout] Several cross-references are unresolved, including 'Appendix ??' in Section 3.2, 'Table ??' in Section 6.6, and 'Appendix D' in Section 6.6; these missing references should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluator LM is trained on external human labels and evaluated on held-out questions/models; the main issues are missing and contradictory evidence, not circularity.

full rationale

The paper's derivation chain is a supervised learning pipeline: human annotators write rubrics and scores; GPT-4 chain-of-thought traces are filtered by agreement with human scores (Eq. 2); an evaluator LM is trained on the filtered traces; and performance is reported as agreement with human scores on held-out questions or held-out models (Sec. 6.3). The ground truth human scores are external to the model and to the rubrics, so the final agreement metric is not defined in terms of the model's own outputs. The Human-AI Consistency filter uses human scores as training supervision, which is standard practice and does not make the held-out evaluation circular. I find no self-citation chain, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation that carries the argument. The genuine concerns are empirical and presentational: the promised comparison against the general-rubric paradigm in Sec. 6.6 is a literal missing table ('Table ??'), and Table 1 in Sec. 6.3 shows XD model-level Pearson 0.380 versus GPT-4 0.822, which contradicts the abstract and Sec. 1 claim that the model 'surpasses GPT-4 in model-level and question-level Pearson correlation.' These are serious correctness/completeness problems, but they are not circularity: the reported numbers are still computed against independent human labels rather than being forced by construction. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on human-annotated rubrics and scores as ground truth, the selection of high-variance questions, and the use of GPT-4-generated reasoning traces filtered for agreement as training data. The rubric point values are hand-chosen per question, constituting many free parameters. No new entities are introduced. If the rubrics are not objective or the selection biases the benchmark, the claimed improvement in human concordance is not meaningful.

free parameters (3)
  • Per-question rubric point values = various (e.g., +3, +2, -1)
    Each question's rubric assigns hand-chosen weights to scoring and penalty points. These determine the final score and are not derived from data; they are ad hoc human choices for each of the 1,000 questions.
  • ACC_t tolerance epsilon = not specified
    Equation 6 defines a weaker-threshold accuracy ACC_t with an unspecified epsilon. Without its value, the metric is not reproducible.
  • Maximum iterations n in Algorithm 1 = not specified
    The self-refinement loop for automatic rubric generation (Algorithm 1) has an unspecified maximum iteration count.
assumptions (4)
  • domain assumption Human annotators' scores are the ground truth for evaluating evaluator LMs.
    The entire benchmark treats human grading as correct; no inter-annotator agreement is reported, and the rubric construction process is assumed to yield objective criteria.
  • domain assumption Keeping only questions with high score variance across 5 LLMs yields a representative benchmark.
    Section 3.2 states they delete questions where models perform similarly. This selection may bias the benchmark toward questions that discriminate between current models rather than reflecting all real-world questions.
  • ad hoc to paper GPT-4 chain-of-thought reasoning, filtered by exact agreement with human scores, is a valid training signal.
    Section 4.2's Human-AI Consistency relies on this assumption; GPT-4's reasoning steps are not validated against human explanations, only the final score is compared.
  • ad hoc to paper GPT-4 can reliably score the match between generated rubrics and ground truth rubrics.
    Section 5.2 uses GPT-4 as a judge for rubric quality (Eq. 8), assuming it correctly identifies matching and opposite items.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SedarEval: Automated Evaluation using Self-Adaptive Rubrics." pith.science (2026). https://pith.science/paper/LZWWCXRH

@misc{pith2026250115595,
  author       = {Pith},
  title        = {Pith review of: SedarEval: Automated Evaluation using Self-Adaptive Rubrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZWWCXRH}},
  note         = {Machine review of arXiv:2501.15595}
}
read the original abstract

The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator's analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach. We release our dataset at https://github.com/wwn1233/sedareval.

Figures

Figures reproduced from arXiv: 2501.15595 by the authors.

Figure 1
Figure 1. Automated evaluation pipeline using self-adaptive rubrics.This pipeline dynamically adjusts the evaluation [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Consistency of model rankings as n increases. After n reaches approximately 300, the consistency sta￾bilizes with only minor fluctuations. stabilize due to the broader coverage of the distri￾bution. 6.2 Score Distribution Shift [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Data distribution comparison of different data. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Benchmark multi-level classification system. To maintain conciseness, we have detailed the hierarchy [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: An illustrative example of the format used in self-adaptive rubrics. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card

    2024

  4. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  5. [5]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. https://arxiv.org/abs/2308.07201 Chateval: Towards better llm-based evaluators through multi-agent debate . Preprint, arXiv:2308.07201

    arXiv 2023

  6. [6]

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

    2023

  7. [7]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132

    arXiv 2024

  8. [8]

    Daniel Deutsch, Rotem Dror, and Dan Roth. 2021. A statistical analysis of summarization evaluation metrics using resampling methods. Transactions of the Association for Computational Linguistics, 9:1132--1146

    work page 2021

Show all 46 references
  1. [9]

    Markus Freitag, David Grangier, and Isaac Caswell. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.5 BLEU might be guilty but references are not innocent . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61--71, Online...

    2020 doi

  2. [10]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  3. [11]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36

    2024

  4. [12]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024 a . https://arxiv.org/abs/2310.08491 Prometheus: Inducing fine-grained evaluation capability in language models . Preprint,...

    2024 arXiv

  5. [13]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024 b . https://arxiv.org/abs/2405.01535 Prometheus 2: An open source language model specialized in evaluating other language model...

    2024 arXiv

  6. [14]

    Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. arXiv preprint arXiv:2107.10821

    2021 arXiv

  7. [15]

    Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332

    2021 arXiv

  8. [16]

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023 a . https://arxiv.org/abs/2310.05470 Generative judge for evaluating alignment . Preprint, arXiv:2310.05470

    2023 arXiv

  9. [17]

    Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. https://doi.org/10.18653/v1/D19-1623 Deep reinforcement learning with distributional semantic rewards for abstractive summarization . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pr...

    2019 doi

  10. [18]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 b . Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval

    2023

  11. [19]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  12. [20]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 a . Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

    2023 arXiv

  13. [21]

    Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023 b . https://doi.org/10.18653/v1/2023.acl-long.228 Revisiting the gold standard: Grounding summarization evaluation with ro...

    2023 doi

  14. [22]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.741 FA ct S core: Fine-grained atomic evaluation of factual precision in long form text generatio...

    2023 doi

  15. [23]

    Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019. https://arxiv.org/abs/1904.03971 Jointly measuring diversity and quality in text generation models . Preprint, arXiv:1904.03971

    2019 arXiv

  16. [24]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  17. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

    2002

  18. [26]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fan...

    2019 arXiv

  19. [27]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  20. [28]

    Ehud Reiter. 2018. https://doi.org/10.1162/coli_a_00322 A structured review of the validity of BLEU . Computational Linguistics, 44(3):393--401

    2018 doi

  21. [29]

    Julian Risch, Timo M \"o ller, Julian Gutsch, and Malte Pietsch. 2021. https://doi.org/10.18653/v1/2021.mrqa-1.15 Semantic answer similarity for evaluating question answering models . In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149--157,...

    2021 doi

  22. [30]

    Natalie Schluter. 2017 a . https://aclanthology.org/E17-2007 The limits of automatic summarisation according to ROUGE . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pages 41--45, Valen...

    2017

  23. [31]

    Natalie Schluter. 2017 b . The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 41--45. Association for Computational Linguistics

    2017

  24. [32]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. https://arxiv.org/abs/1909.08053 Megatron-lm: Training multi-billion parameter language models using model parallelism . Preprint, arXiv:1909.08053

    2020 arXiv

  25. [33]

    Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. https://aclanthology.org/2006.amta-papers.25 A study of translation edit rate with targeted human annotation . In Proceedings of the 7th Conference of the Association for Machine Translation ...

    2006

  26. [34]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  27. [35]

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Characterizing the confidence of large language model-based automatic evaluation metrics. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: ...

    2024

  28. [36]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  29. [37]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  30. [38]

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. https://arxiv.org/abs/2306.05087 Pandalm: An automatic evaluation benchmark for llm instruction tuning optimiz...

    2024 arXiv

  31. [39]

    Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928

    2023 arXiv

  32. [40]

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414

    2022 arXiv

  33. [41]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  34. [42]

    Ying Zhang and Stephan Vogel. 2004. Measuring confidence intervals for the machine translation evaluation metrics. In Proceedings of the 10th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages

    2004

  35. [43]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  36. [44]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  37. [45]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364

    2023 arXiv

  38. [46]

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. https://arxiv.org/abs/2310.17631 Judgelm: Fine-tuned large language models are scalable judges . Preprint, arXiv:2310.17631

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.