Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training turns a small open LLM into a reliable science-answer judge.

desk verdict Useful corpus and an honest empirical observation, but the robustness claim is in-distribution and needs human validation before it can carry the paper. read the letter →

arxiv 2505.14279 v2 pith:WKD3OVPK submitted 2025-05-20 cs.CL cs.AI

Jennifer D'Souza , Hamed Babaei Giglou , Quentin Münch This is my paper · ORCID
classification cs.CLcs.AI
keywords LLM-as-a-judgescientificquestionansweringoptimismbiasreinforcementlearningadversarialevaluationrubric-basedopen-sourceLLMcontrastivepreferenceoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

YESciEval claims that a small open-source language model can be made into a reliable judge of scientific question-answering without any human labels or proprietary model calls. The authors pair a nine-rubric scoring scheme with adversarial edits of model-written answers, then fine-tune Llama 3.1 8B by supervised learning followed by Contrastive Preference Optimization. Their central result is that this alignment stops the optimism bias of vanilla LLM judges: the evaluator keeps high scores for ordinary syntheses while dropping to about 1 on extreme perturbations and about 3 on subtle ones. If the claim holds, cheap open-source evaluation of science answers becomes practical and transparent, with direct consequences for how science search engines and research tools are assessed.

What carries the argument

The load-bearing machinery is the contrastive preference pair $(y_{\text{good}}, y_{\text{bad}})$ built from rubric scores: for adversarial data, desirable scores are defined as 1 for extreme and $\le 3$ for subtle perturbations across all nine rubrics, so any evaluation above that threshold becomes the bad sample. The nine rubrics—cohesion, conciseness, readability, coherence, integration, relevancy, correctness, completeness, informativeness—are scored on a 1–5 Likert scale and organized into linguistic, logical, and content dimensions. Training applies QLoRA supervised fine-tuning on benign evaluations, then Contrastive Preference Optimization (CPO), a DPO variant with hard negatives plus a behavior-cloning negative-log-likelihood term, to make the policy prefer good evaluations over bad ones. This contrast between benign and rubric-specific adversarial examples is what teaches the judge to penalize quality deterioration rather than to flatter every answer.

What would settle it

Take the aligned judge and give it a benign synthesis with one unrelated sentence appended at the end, the exact correctness/extreme perturbation, then ask human raters to score only the factual correctness of the original content; if most humans rate it above 1 while the judge assigns 1, the rubric-specific discrimination claim fails. A broader version scores a random sample of the adversarial test set by human raters on all nine rubrics and measures how often the human score disagrees with the paper's assumed 1/≤3 labels.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that optimism bias in LLM-as-a-judge evaluation of scienceQ&A is not an incurable property of small models: after QLoRA-based supervised fine-tuning on benign rubric scores and reinforcement learning (CPO) on preference pairs built from benign and adversarially perturbed responses, a Llama 3.1 8B evaluator stabilized as a reliable judge. It assigned relatively high scores to benign syntheses, scored extreme adversarial variants around 1, and subtle variants around 3 across the nine rubrics. The paper identifies SFT alone as insufficient (it amplifies optimism) and RL on adversarial data alone as overcorrecting; only the joint benign-plus-adversarial RL setting produces the desired discrimination, and it does so without human annotation or proprietary model calls.

Load-bearing premise

The training assumes that every hand-designed adversarial edit makes the response genuinely bad on the rubric it targets, so the correct score is exactly 1 for extreme edits and at most 3 for subtle edits across all nine rubrics, rather than merely adding a recognizable textual cue.

Editorial extensions

If this is right

  • A cost-free, open-source evaluator for scienceQ&A becomes feasible: the aligned Llama 3.1 8B judge can replace proprietary judge calls and human annotation in the evaluation loop.
  • Optimism bias is not an unavoidable failure of small evaluators; the same SFT-plus-RL recipe can be applied to other open-source LLMs and to other rubric-based evaluation tasks.
  • Because the correct score for each adversarial perturbation is known by construction, preference data for RLAIF-style training can be generated without human raters.
  • The released datasets and the 48 judge configurations give downstream researchers a fixed benchmark for comparing any new evaluator against non-proprietary judgments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's robustness is bounded by its perturbation inventory: an evaluator trained on sports-sentence injections and sentence shuffles may learn those specific cues rather than a general capacity for critical judgment, so performance on unseen adversarial styles remains an open question.
  • The same rubric-adversarial recipe could transfer to other NLG tasks such as summarization, translation, or long-form QA, where optimism bias is also reported; a cheap experiment would be to train separate judges on each task's rubrics and adversarial heuristics.
  • A stronger validation would compare the aligned judge's scores against human ratings on the same adversarial instances, since the paper deliberately avoids human references and the assumed 1/≤3 labels are untested against human judgment.
  • In production, the aligned judge's low scores could serve as a cheap filter before human review: extreme cases are clear rejects, while subtle cases near 3 can be routed to closer inspection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces YESciEval, a framework for evaluating science question-answering outputs using a nine-rubric Likert-scale scheme, with adversarial perturbations designed per rubric, and an alignment pipeline (SFT followed by Contrastive Preference Optimization) applied to Llama-3.1-8B to reduce the optimism bias of LLM-as-a-judge systems. The authors release datasets, evaluation scores, and trained model checkpoints, and report that the SFT+RL(benign+adversarial) model assigns high scores to benign syntheses, scores near 1 for extreme adversarial variants, and scores near 3 for subtle adversarial variants, claiming that this demonstrates robustness as a scientific QA judge.

Significance. If the central claim were fully supported, the contribution would be practically valuable: a small open-source judge that does not require proprietary APIs or human annotation would lower the cost of evaluating scientific QA systems, and the released datasets and code would support reproducibility and further research. The nine-rubric scheme is a reasonable synthesis of prior evaluation rubrics, and the decision to evaluate and release multiple LLM-as-a-judge configurations (48 configurations across four generators and evaluators) is a commendable engineering contribution. However, the paper's primary evidence for the robustness claim is an evaluation that is in-distribution with respect to the adversarial perturbation heuristics used to construct the reinforcement learning labels, and there is no human gold-standard validation of the final aligned judge. The strengths of the released artifacts do not by themselves establish that the trained judge has learned rubric-faithful evaluation skill rather than surface-level cues of the specific perturbation types.

major comments (4)
  1. [Section 6 / Figure 4; Section 5] The RQ3 result is an in-distribution evaluation: the RL training data in Section 5 defines desirable scores as 1 for extreme and ≤3 for subtle adversarial variants across the nine rubrics, and the test adversarial sets in Section 4.4 are generated with the same rubric-level heuristics (e.g., appending sports news for Correctness/Relevancy, swapping the last two sentences for Cohesion, removing connectors for Integration). The green line in Figure 4 therefore demonstrates that the model reproduces the label mapping it was trained to reproduce, which does not separate genuine rubric-based judgment from shortcut learning of edit patterns. To support the claim that the model is a robust evaluator, the paper needs either a human gold-standard evaluation of the final aligned judge on the test sets, or an out-of-distribution evaluation with perturbation types or naturally degraded responses not seen during RL training.
  2. [Section 4.4 / Appendix B.3] The perturbation heuristics are assumed to degrade exactly the targeted rubric, but this assumption is not validated. For example, swapping the last two sentences may preserve cohesion in many scientific summaries, and appending a sentence from another synthesis in the same domain may be off-topic for Relevancy without being a factual error for Correctness. Since the RL labels in Section 5 assign scores of 1 or ≤3 to all such variants, the model can be trained to emit arbitrary low scores for texts containing certain surface edits rather than to judge rubric-specific quality. The paper states in Section 4.4 that 'these perturbations systematically degrade response quality,' but no evidence (human ratings, pilot studies, or per-instance checks) is provided to verify that each heuristic affects the intended rubric and no other.
  3. [Appendix C; Section 6] The manual inspection reported in Appendix C was performed only on the vanilla Llama 8B evaluator, not on the SFT+RL models whose behavior is central to RQ3. Consequently, the qualitative claim in Section 6 that the final model 'stabilized as a robust evaluator' rests entirely on automated score comparisons without any human validation of the trained judge's outputs. In addition, the quantitative comparisons in Tables 7–14 and Figure 4 report mean scores without error bars, confidence intervals, or significance tests, leaving it unclear whether the differences between the green and other lines are robust to sampling variability.
  4. [Section 9] The Limitations section appropriately acknowledges that 'broader generalization remains an open question,' but the abstract and introduction claim a robust and cost-free evaluation framework without this scope restriction. The central claim should be explicitly limited to the specific perturbation heuristics used in the paper unless additional experiments (human validation or out-of-distribution adversarial sets) are provided. As written, the external validity of the robustness result is not established.
minor comments (8)
  1. [Section 5, CPO equation] The CPO loss equation is typeset with unclear alignment between the subscript annotations and the terms; please rewrite it so that L_prefer and L_NLL are clearly identified as the two additive terms.
  2. [Figure 4 caption] The figure legend does not clearly map all colored lines to the model variants described in the text; please ensure that the blue, yellow, red, and green lines are labeled consistently with the caption and that the 'SFT (adversarial) + RL (benign+adversarial)' label used in Tables 15–16 is reconciled with the naming convention used elsewhere.
  3. [Section 4.5, counts] The evaluation counts are consistent (348×3×9×4 = 37,584 for ORKGSyn and 73×3×9×4 = 7,884 for BioASQ), but the text says 'resulted in 37,584 evaluation scores for ORKGSyn' while also listing '9,396 evaluations per model'; please clarify that the former is the total across all four evaluators.
  4. [Section 5, Data Preparation for RL] The sentence 'We impose a 100-per-rubrics, per-LLMeval threshold (for ORKGSyn, this threshold is set to 500)' is unclear because the threshold seems to vary by dataset but not by model; please specify whether it is per rubric per model or per rubric overall, and explain the rationale for the ORKGSyn value.
  5. [Appendix G] There is a typo: 'The fine-tuning threats each LLMgen' should be 'treats'; also, the phrase 'SF Tdataset stats' is missing a space.
  6. [References] In the Introduction, citations to BLEU and ROUGE use keys (2002a) and (2004a), while the reference list entries are (Papineni et al., 2002b) and (Lin, 2004b); please harmonize the citation keys.
  7. [Appendix B.3, manual cleanup] The appendix notes that the Llama 8B and 70B models frequently appended phrases like 'Here is a redundant sentence...' and that 'We manually revised over 100 responses generated by the Llama 8B model and 18 by the Llama 70B'; this manual cleanup contradicts the abstract's claim that the approach is 'independent of ... human feedback', and the paper should acknowledge this dependence in the main text.
  8. [Section 1, 'cost-free'] The footnote clarifies that 'cost-free' excludes compute, but the abstract and introduction state the approach is 'cost-free' without this caveat; please qualify the claim in the abstract.

Circularity Check

1 steps flagged · score 6.0 of 10

Apparent robustness is in-distribution: RL labels and RQ3 evaluation share the same perturbation taxonomy, so the trained judge is rewarded for reproducing its own target scores.

  1. fitted input called prediction [Section 5 (Data Preparation for RL), Section 4.4 (Adversarial Corpus), Section 6 (RQ3)]
    "we define desirable scores as 1 for extreme and ≤ 3 for subtle adversarial variants across the nine rubrics. The adversarial perturbation heuristics, for both variants, were designed at the fine-grained rubric level. ... When RL was trained on both benign and adversarial samples (green), Llama 8B stabilized as a robust evaluator ... scoring around 1 in extreme cases and around 3 in subtle cases, demonstrating rubric-specific discrimination."

    The RQ3 headline behavior is the RL target: the same subtle/exreme perturbation taxonomy defines both the preference labels in Section 5 and the test conditions in Section 4.4 and Figure 4. Reporting observed scores of about 1 and about 3 as evidence of 'robustness' measures how well the model reproduces the label rule it was trained to satisfy, not agreement with an independent standard. The paper's own observation that RL on adversarial data alone collapses benign scores to about 3 shows that the ≤3 threshold directly shapes behavior. Appendix C inspects only the vanilla Llama 8B, so the trained judge is never validated against human judgments.

full rationale

The strongest circularity is in the RQ3 claim. The RL preference data define 'desirable scores as 1 for extreme and ≤ 3 for subtle adversarial variants across the nine rubrics,' and the adversarial test sets are constructed with the same rubric-level heuristics described in Section 4.4 (for example, appending sports news for Relevancy and Correctness, swapping the last two sentences for Cohesion, removing connectors for Integration). The green line in Figure 4 then reports scores close to 1 on extreme and close to 3 on subtle cases; this is an in-distribution match to the training signal, so it does not by itself demonstrate that the model learned general rubric-faithful evaluation rather than surface cues such as sports sentences, tweets, redundant phrasing, or shuffled sentence order. The paper even reports that RL with only adversarial data pushed benign scores down to about 3, which is direct evidence that the ≤3 threshold, rather than an independent notion of quality, was driving the score level. No human validation of the final aligned judge is provided: Appendix C manually analyzes only vanilla Llama 8B, and the perturbation heuristics are not shown to consistently degrade the targeted rubric (for instance, swapping the last two sentences may preserve cohesion in many scientific summaries). The framework still has independent content in the benign SFT data, the released datasets, and the comparisons against vanilla models and larger LLMs, so the circularity is partial rather than total. The core 'robust evaluator' result, however, reduces in part to fitting the RL label rule, yielding a score of 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a small set of hand-set thresholds and several domain assumptions. No new theoretical entities are introduced. The most load-bearing item is the mapping between perturbation type and expected rubric score, which drives both RL training and the evaluation of the resulting judge.

free parameters (2)
  • Adversarial expected-score thresholds = extreme = 1, subtle <= 3
    Hand-set in Section 5 to define good and bad labels for RL. These thresholds are applied across all nine rubrics even though the perturbations differ in type and severity.
  • Per-rubric sampling caps = 100 per rubric per evaluator (ORKGSyn: 500)
    Data curation choice in Section 5 that controls the composition and size of the RL training set.
assumptions (4)
  • ad hoc to paper The rubric-based adversarial perturbations degrade exactly the quality dimension they target.
    Appendix B.3 assumes a one-to-one mapping between edit type and rubric. Many edits, such as inserting sports news, simultaneously affect relevance, coherence, correctness, and informativeness, while a sentence shuffle may not reduce cohesion to the forced minimum.
  • domain assumption LLM-generated labels from vanilla evaluators are a valid training signal for SFT.
    Section 5 uses y = LLMeval outputs from four LLMs as target ratings for supervised fine-tuning, without human adjudication of those ratings.
  • domain assumption The ORKGSyn and BioASQ paper-to-question relevance mappings are correct.
    The evaluation correctness depends on the abstracts being the right evidence for each question, but the paper does not audit these mappings and inherits them from prior datasets.
  • domain assumption No human feedback is needed to establish evaluator reliability.
    The framework explicitly avoids human labels and instead relies on adversarial construction, as stated in Sections 1 and 7.2. This is a substantive premise because the paper's central claim is about reliable evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering." pith.science (2026). https://pith.science/paper/WKD3OVPK

@misc{pith2026250514279,
  author       = {Pith},
  title        = {Pith review of: YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WKD3OVPK}},
  note         = {Machine review of arXiv:2505.14279}
}
read the original abstract

Large Language Models (LLMs) drive scientific question-answering on modern search engines, yet their evaluation robustness remains underexplored. We introduce YESciEval, an open-source framework that combines fine-grained rubric-based assessment with reinforcement learning to mitigate optimism bias in LLM evaluators. We release multidisciplinary scienceQ&A datasets, including adversarial variants, with evaluation scores from multiple LLMs. Independent of proprietary models and human feedback, our approach enables scalable, cost-free evaluation. By advancing reliable LLM-as-a-judge models, this work supports AI alignment and fosters robust, transparent evaluation essential for scientific inquiry.

Figures

Figures reproduced from arXiv: 2505.14279 by the authors.

Figure 1
Figure 1. Examples from two domains in the YESciEval science Q&A dataset. Orange boxes show LLM input: a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. YESciEval LLM-as-a-Judge Alignment: Supervised fine-tuning of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Heatmaps depicting agreement for synthesis [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Evaluation of synthesis across different models and fine-tuning strategies on BioASQ and ORKGSynthesis [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Number of Questions per Research Field on the ORKGSyn Dataset. The y-axis represents the "Research [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Number of Reference Abstracts per Question on the BioASQ dataset [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: BLEU metric results. mance, Mistral-Large achieves the highest average BLUE score on the BioASQ dataset, while on the ORKGSynthesis dataset, Meta’s 70B LLM per￾forms best. Recall-Oriented Understudy for Gisting Evalua￾tion (ROUGE). ROUGE (Lin, 2004b) is a recall￾based …
Figure 8
Figure 8. Figure 8: ROUGE-1 metric results. across the two datasets. On the BioASQ dataset, Mistral-Large and Qwen2.5-72B demonstrate a no￾table correlation with a ROUGE-1 score of 0.58, suggesting shared thematic elements or vocabulary usage in their outputs. Additionally, a ROUGE￾L scor…
Figure 10
Figure 10. Figure 10: METEOR metric results. earlier metrics. Conversely, the LLaMA-3.1-8B model achieved comparable scores to Mistral￾Large and Qwen2.5-72B, suggesting it generates re￾sponses that align well with unigram matches. For the ORKGSynthesis dataset, the scores were more uniform…
Figure 12
Figure 12. Figure 12: BERTScore metric results. the underlying semantic content remains closely aligned. This observation underscores the poten￾tial of embedding-based metrics to provide a more nuanced evaluation of language model outputs, par￾ticularly in tasks where paraphrasing and crea…
Figure 13
Figure 13. Figure 13: WER metric results by reference length in the WER formula leads to more significant deviations when Qwen2.5-72B’s outputs are compared against shorter references. These findings underscore the metric’s dependence on the relative length of candidate and reference texts…
Figure 15
Figure 15. Figure 15: WMD metric results using SPECTER2 as embeddings. candidate and reference texts, while higher scores denote greater dissimilarity. Analysis of the WMD scores across datasets highlights several trends. As expected, LLaMA-3.1-70B exhibits poor per￾formance on the BioASQ …
Figure 14
Figure 14. Figure 14: WMD metric results using SciBERT as em￾beddings. The results of WMD using SPECTER2 embed￾dings are represented in [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 16
Figure 16. Figure 16: MoverScore metric results MoverScore produces a value between 0 and 1, where higher scores correspond to greater sim￾ilarity between the candidate and reference texts. Across both datasets as shown in [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: The y format for fine-tuning models. preparation for RLAIF for the subtle adversarial set. Essentially, training the model exclusively on adversarial behavior leads to confusion when it en￾counters benign examples. To solve this, the SFT (benign) + RL (benign + advers…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DeepResearch$^{\text{Eco}}$: A Recursive Agentic Workflow for Complex Scientific Question Answering in Ecology

    cs.AI 2025-07 conditional novelty 5.0 of 10

    A recursive agentic pipeline for literature synthesis showing a 21-fold source increase and 14.9-fold density gain when depth and breadth are raised.

Reference graph

Works this paper leans on

238 extracted references · 75 canonical work pages · cited by 1 Pith paper

  1. [1]

    These alterations mimic realistic errors that may go unnoticed in automated evaluations

    Subtle Adversarial Dataset: Here, reference texts are minimally altered, making it chal- lenging for models to detect changes. These alterations mimic realistic errors that may go unnoticed in automated evaluations

  2. [2]

    Research Fields

    Extreme Adversarial Dataset: This dataset involves substantial modifications to reference texts, making the adversarial setting appar- ent and straightforward for models to identify. The evaluations should result in significantly lower scores. Nine evaluation criteria are systematically targeted during adversarial dataset creation, with distinct manipulat...

    2024

  3. [3]

    – Extreme: Remove the last sentence and append a sentence from an unrelated sports news article

    Completeness: Is the answer a comprehen- sive encapsulation of the relevant information in the provided abstracts? – Subtle: Remove the last sentence from the synthesis. – Extreme: Remove the last sentence and append a sentence from an unrelated sports news article

  4. [4]

    In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650–1669

    Longeval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 1650–1669. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Confere...

    arXiv 2021

  5. [5]

    however",

    Integration: Are the sources structurally and linguistically well-integrated, using appropri- ate markers of provenance/quotation and logi- cal connectors for each reference? – Subtle: Remove the first logical connec- tor (e.g., "however", "therefore"). – Extreme: Remove all logical connectors

  6. [6]

    In Proceedings of the 37th International Conference on Neural In- formation Processing Systems, NIPS ’23, Red Hook, NY , USA

    Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural In- formation Processing Systems, NIPS ’23, Red Hook, NY , USA. Curran Associates Inc. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of ...

    2016

  7. [7]

    – Extreme: Append a sentence from an unrelated sports news article

    Coherence: Are the ideas connected soundly and logically? – Subtle: Append a sentence from a differ- ent synthesis paragraph within the same domain. – Extreme: Append a sentence from an unrelated sports news article

  8. [8]

    In Proceedings of the 57th annual meeting of the as- sociation for computational linguistics, pages 1085– 1097

    Generating natural language adversarial ex- amples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the as- sociation for computational linguistics, pages 1085– 1097. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In P...

    arXiv 2022

Show all 238 references
  1. [9]

    measured the correlation between an LLM judge and human evaluators in open-domain in- struction following for NLG. Early works on using LLMs as evaluators of LLM-generated text em- phasized pairwise evaluation along a single dimen- sion of ‘preference’ in determining which res...

    2023

  2. [13]

    – Extreme: Append a sentence from an unrelated sports news article

    Relevancy: Is the information in the answer relevant to the problem? – Subtle: Append a sentence from a differ- ent synthesis paragraph within the same domain. – Extreme: Append a sentence from an unrelated sports news article

  3. [14]

    – Extreme: Append a sentence from an unrelated sports news article

    Correctness: Is the information in the answer a correct representation of the content of the provided abstracts? – Subtle: Append a sentence from a differ- ent synthesis paragraph within the same domain. – Extreme: Append a sentence from an unrelated sports news article

  4. [16]

    – Extreme: Append a sentence from an unrelated sports news article

    Informativeness: Is the answer a useful and 0 2 4 6 8 10 12 1 2 3 4 5 6 7 9 11 12 13 15 16 17 18 20 21 22 23 25 26 27 28 31 32 33 34 36 40 6 9 4 3 2 1 1 1 1 1 2 1 1 3 3 1 5 1 2 1 1 1 4 1 2 1 1 2 11 Number of Occurrences Number of Reference Abstracts Figure 6: Number of Referen...

  5. [18]

    – Extreme: Randomly shuffle all sen- tences

    Cohesion: Are the sentences connected ap- propriately to make the resulting synthesis cohesive? – Subtle: Swap the positions of the last two sentences. – Extreme: Randomly shuffle all sen- tences

  6. [20]

    – Extreme: Append a sentence from an informal tweet

    Readability: Does the answer follow appro- priate style and structure conventions for aca- demic writing, particularly for readability? – Subtle: Append a snippet from a casual blog post. – Extreme: Append a sentence from an informal tweet

  7. [21]

    Here is a redundant sentence

    Conciseness: Is the answer short and clear, without redundant statements? – Subtle: Use the LLM to generate a re- dundant version of the last sentence and append it to the response. – Extreme: Append a redundant version after every sentence in the original re- sponse. For the ...

    2020

  8. [23]

    Coherence 4.95 4.91 4.73 5.00 4.95 3.00 4.91

  9. [24]

    Cohesion 4.95 4.91 4.68 4.95 4.95 3.00 4.68

  10. [25]

    Completeness 4.41 4.33 4.27 4.91 4.23 3.00 4.32

  11. [26]

    Conciseness 4.95 4.91 4.68 4.45 4.77 3.00 3.82

  12. [27]

    Correctness 4.95 4.92 4.64 4.73 5.00 3.00 5.00

  13. [28]

    Informativeness 4.82 4.82 4.86 5.00 4.68 3.00 5.00

  14. [29]

    Integration 4.91 4.91 4.55 4.77 4.95 3.00 4.59

  15. [30]

    Readability 4.95 4.91 4.73 4.64 4.95 3.00 4.55

  16. [31]

    Relevancy 4.95 4.83 4.86 5.00 5.00 3.00 5.00 extreme

  17. [32]

    Coherence 2.50 3.00 3.00 4.64 3.55 1.00 1.00

  18. [33]

    Cohesion 4.05 4.55 3.82 4.77 4.93 3.00 4.41

  19. [34]

    Completeness 2.68 3.31 3.27 4.64 3.82 1.05 1.00

  20. [35]

    Conciseness 1.41 1.00 1.50 2.27 4.77 1.00 1.00

  21. [36]

    Correctness 3.91 3.90 3.73 4.68 3.59 1.00 1.00

  22. [37]

    Informativeness 3.59 4.20 3.68 4.68 3.41 1.09 1.00

  23. [38]

    Integration 3.73 4.08 3.18 4.77 4.82 3.00 3.14

  24. [39]

    Readability 2.05 2.15 2.55 4.14 1.89 1.00 1.00

  25. [40]

    Relevancy 2.64 3.40 3.55 4.86 2.77 1.00 1.00 subtle

  26. [41]

    Coherence 3.82 4.10 4.14 4.82 4.86 2.91 2.73

  27. [42]

    Cohesion 4.05 4.25 4.23 4.68 4.82 3.00 4.64

  28. [43]

    Completeness 4.14 4.09 3.82 4.82 4.36 3.00 3.91

  29. [44]

    Conciseness 3.09 2.60 2.41 3.68 4.55 2.41 2.27

  30. [45]

    Correctness 4.23 4.50 4.14 4.64 4.86 3.00 2.82

  31. [46]

    Informativeness 4.18 4.40 4.23 4.95 4.59 3.00 2.77

  32. [47]

    Integration 4.09 4.27 3.86 4.64 4.91 3.00 4.64

  33. [48]

    Readability 3.14 2.92 3.45 4.18 4.18 2.18 1.82

  34. [49]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.32 4.20 4.36 5.00 4.77 2.77 2.73 Table 7: BioASQ dataset detailed evaluation results of various LLMeval over LLMgen = LLaMA-3.1-8B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  35. [50]

    Coherence 2.91 4.90 2.91 4.91 3.64 2.55 3.09

  36. [51]

    Cohesion 2.91 4.90 2.86 4.86 4.05 2.32 2.86

  37. [52]

    Completeness 2.50 3.29 2.73 4.64 3.86 2.45 2.64

  38. [53]

    Conciseness 2.91 4.80 3.36 4.23 3.73 2.00 2.50

  39. [54]

    Correctness 3.00 3.73 3.18 4.86 3.91 2.91 3.09

  40. [55]

    Informativeness 2.77 4.36 3.00 4.95 3.27 2.50 2.95

  41. [56]

    Integration 2.86 4.55 2.77 4.77 3.86 1.95 2.82

  42. [57]

    Readability 2.91 4.90 3.05 4.41 3.23 2.05 2.86

  43. [58]

    Relevancy 2.95 3.80 3.23 4.95 3.73 2.50 3.09 extreme

  44. [59]

    Coherence 1.50 2.09 2.14 3.59 2.73 1.00 1.00

  45. [60]

    Cohesion 2.50 3.92 2.64 4.50 3.45 2.27 2.68

  46. [61]

    Completeness 1.73 2.36 2.32 4.18 2.59 1.09 1.00

  47. [62]

    Conciseness 1.50 1.17 1.55 2.41 3.91 1.18 1.00

  48. [63]

    Correctness 2.32 3.27 2.50 3.91 2.95 1.00 1.00

  49. [64]

    Informativeness 2.09 3.18 2.41 3.95 2.05 1.00 1.00

  50. [65]

    Integration 2.23 3.38 2.32 4.45 3.14 1.91 2.14

  51. [66]

    Readability 1.50 1.75 1.50 3.68 2.05 1.00 1.00

  52. [67]

    Relevancy 1.45 2.73 2.41 3.95 2.00 1.00 1.00 subtle

  53. [68]

    Coherence 1.95 3.40 2.32 4.41 3.09 2.45 1.91

  54. [69]

    Cohesion 2.50 3.33 2.45 4.64 3.55 2.32 2.95

  55. [70]

    Completeness 2.59 3.36 2.95 4.77 3.59 2.64 2.27

  56. [71]

    Conciseness 1.86 1.86 1.64 3.23 3.82 1.64 1.32

  57. [72]

    Correctness 2.27 3.50 2.82 4.55 3.50 3.00 2.18

  58. [73]

    Informativeness 2.32 3.82 2.59 4.55 3.14 2.68 2.18

  59. [74]

    Integration 2.45 3.83 2.50 4.36 3.41 1.86 2.86

  60. [75]

    Readability 1.91 1.92 2.41 4.05 3.27 1.68 1.50

  61. [76]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 2.36 3.67 2.68 4.82 3.14 2.41 2.09 Table 8: BioASQ dataset detailed evaluation results of various LLMeval over LLMgen = LLaMA-3.1-70B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  62. [77]

    Coherence 5.00 5.00 4.95 4.95 5.00 3.00 4.95

  63. [78]

    Cohesion 5.00 5.00 4.91 4.95 5.00 3.00 4.45

  64. [79]

    Completeness 4.27 4.00 4.36 4.73 4.27 3.00 4.05

  65. [80]

    Conciseness 5.00 4.91 4.91 4.55 4.86 3.00 4.45

  66. [81]

    Correctness 4.95 5.00 4.91 4.82 4.91 3.00 4.91

  67. [82]

    Informativeness 4.68 4.83 4.91 4.82 4.86 3.00 4.95

  68. [83]

    Integration 4.91 5.00 4.86 4.77 4.91 3.00 4.86

  69. [84]

    Readability 5.00 5.00 5.00 4.45 4.86 3.00 4.86

  70. [85]

    Relevancy 5.00 5.00 5.00 4.91 5.00 3.00 4.91 extreme

  71. [86]

    Coherence 2.45 2.50 2.82 4.36 3.32 1.00 1.00

  72. [87]

    Cohesion 4.09 4.46 3.77 4.73 4.86 3.00 4.59

  73. [88]

    Completeness 2.32 3.10 2.68 3.95 3.91 1.00 1.00

  74. [89]

    Conciseness 2.27 1.60 2.09 3.27 4.77 1.09 1.09

  75. [90]

    Correctness 3.68 3.70 3.77 4.36 3.36 1.00 1.00

  76. [91]

    Informativeness 3.41 3.60 3.50 4.41 2.32 1.00 1.00

  77. [92]

    Integration 3.59 4.00 3.23 4.68 4.95 3.00 3.14

  78. [93]

    Readability 1.91 1.75 2.23 4.23 2.05 1.00 1.00

  79. [94]

    Relevancy 2.18 2.70 3.59 4.68 2.55 1.00 1.00 subtle

  80. [95]

    Coherence 3.64 4.10 4.09 4.77 4.91 3.00 3.00

  81. [96]

    Cohesion 4.18 4.45 4.09 4.68 4.86 3.00 4.59

  82. [97]

    Completeness 3.64 4.10 3.86 4.95 4.27 3.00 3.59

  83. [98]

    Conciseness 2.82 2.70 2.36 3.86 4.59 2.59 2.14

  84. [99]

    Correctness 4.23 4.30 4.00 4.59 4.73 3.00 3.14

  85. [100]

    Informativeness 4.14 4.50 4.18 4.82 4.64 3.00 3.18

  86. [101]

    Integration 4.00 4.55 4.00 4.86 4.95 3.00 4.45

  87. [102]

    Readability 2.55 2.45 3.36 4.09 4.23 2.05 1.91

  88. [103]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.23 4.50 4.27 4.86 4.95 2.95 3.00 Table 9: BioASQ dataset detailed evaluation results of various LLMeval over LLMgen = Qwen2.5-72B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  89. [104]

    Coherence 4.91 5.00 4.95 5.00 5.00 3.00 4.91

  90. [105]

    Cohesion 4.91 4.92 4.77 5.00 4.95 3.00 4.73

  91. [106]

    Completeness 4.27 4.00 4.41 4.57 4.32 3.00 4.36

  92. [107]

    Conciseness 4.95 4.67 4.95 4.59 4.73 2.86 3.77

  93. [108]

    Correctness 4.95 5.00 4.82 4.86 4.95 3.00 4.95

  94. [109]

    Informativeness 4.64 4.75 4.91 4.91 4.73 3.00 4.91

  95. [110]

    Integration 4.86 4.83 4.64 5.00 4.95 3.00 4.68

  96. [111]

    Readability 4.95 4.83 5.00 4.50 4.75 2.91 4.86

  97. [112]

    Relevancy 4.91 4.93 4.95 5.00 5.00 3.00 5.00 extreme

  98. [113]

    Coherence 2.27 2.09 3.14 4.41 3.50 1.00 1.00

  99. [114]

    Cohesion 3.95 4.42 3.77 4.68 4.91 3.00 4.18

  100. [115]

    Completeness 2.32 2.45 2.91 4.18 3.68 1.00 1.00

  101. [116]

    Conciseness 1.95 1.38 1.82 3.09 4.45 1.09 1.09

  102. [117]

    Correctness 3.68 3.45 3.73 4.55 3.41 1.00 1.00

  103. [118]

    Informativeness 3.45 3.18 3.73 4.45 3.00 1.05 1.00

  104. [119]

    Integration 3.59 3.77 3.09 4.68 4.82 2.91 3.00

  105. [120]

    Readability 2.05 2.15 2.27 4.00 2.14 1.00 1.00

  106. [121]

    Relevancy 2.32 2.92 3.64 4.73 3.32 1.00 1.00 subtle

  107. [122]

    Coherence 3.77 3.80 4.14 4.91 4.91 2.95 3.05

  108. [123]

    Cohesion 4.00 4.79 4.27 4.77 4.86 3.00 4.41

  109. [124]

    Completeness 3.86 4.20 3.50 4.91 4.50 3.00 3.68

  110. [125]

    Conciseness 2.91 3.15 2.82 3.91 4.91 2.59 2.27

  111. [126]

    Correctness 4.23 4.27 4.14 4.77 4.68 3.00 3.23

  112. [127]

    Informativeness 4.18 4.20 4.32 4.86 4.59 2.95 3.14

  113. [128]

    Integration 3.91 4.09 3.77 4.59 4.95 3.00 4.45

  114. [129]

    Readability 2.82 2.54 3.73 4.36 4.59 2.27 1.95

  115. [130]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.27 4.18 4.41 5.00 5.00 2.95 2.95 Table 10: BioASQ dataset detailed evaluation results of various LLMeval over LLMgen = Mistral-Large. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  116. [131]

    Coherence 4.74 4.95 4.78 4.97 4.95 3.00 4.98

  117. [132]

    Cohesion 4.72 4.93 4.71 4.97 4.88 3.00 4.95

  118. [133]

    Completeness 4.28 4.40 4.52 4.50 4.59 3.00 4.76

  119. [134]

    Conciseness 4.74 4.71 4.70 4.18 4.77 3.00 4.03

  120. [135]

    Correctness 4.86 4.97 4.79 4.83 4.91 3.00 4.94

  121. [136]

    Informativeness 4.57 4.93 4.77 4.97 4.89 3.06 4.97

  122. [137]

    Integration 4.70 4.90 4.68 4.89 4.93 3.00 4.91

  123. [138]

    Readability 4.77 4.94 4.86 4.41 4.85 3.00 4.86

  124. [139]

    Relevancy 4.70 4.92 4.81 4.99 4.93 3.02 4.96 extreme

  125. [140]

    Coherence 2.58 3.42 3.35 4.27 4.81 1.04 1.00

  126. [141]

    Cohesion 3.76 4.33 3.44 4.85 4.78 3.00 4.41

  127. [142]

    Completeness 2.63 2.94 3.56 3.69 4.59 1.00 1.00

  128. [143]

    Conciseness 1.11 1.05 1.57 1.70 4.64 1.00 1.00

  129. [144]

    Correctness 3.81 4.09 3.98 4.24 4.82 1.01 1.00

  130. [145]

    Informativeness 3.62 4.26 3.80 4.78 4.82 1.03 1.00

  131. [146]

    Integration 3.37 3.90 3.16 4.72 4.86 2.73 3.10

  132. [147]

    Readability 2.20 2.13 2.93 3.77 4.76 1.01 1.00

  133. [148]

    Relevancy 2.78 3.51 3.58 4.99 4.81 1.02 1.00 subtle

  134. [149]

    Coherence 4.21 4.70 4.51 4.81 4.95 3.00 3.44

  135. [150]

    Cohesion 3.98 4.49 3.96 4.90 4.84 3.00 4.52

  136. [151]

    Completeness 4.10 4.23 3.92 4.99 4.67 3.00 4.20

  137. [152]

    Conciseness 2.71 2.36 2.50 3.62 4.70 2.47 2.38

  138. [153]

    Correctness 4.48 4.78 4.44 4.57 4.93 3.00 3.37

  139. [154]

    Informativeness 4.35 4.79 4.50 4.90 4.87 3.03 3.38

  140. [155]

    Integration 4.01 4.52 4.02 4.92 4.91 2.99 4.72

  141. [156]

    Readability 3.22 2.88 3.79 4.49 4.82 2.85 2.98

  142. [157]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.35 4.70 4.56 4.99 4.96 2.99 3.54 Table 11: ORKGSynthesis dataset detailed evaluation results of variousLLMeval over LLMgen = LLaMA-3.1-8B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  143. [158]

    Coherence 4.77 4.97 4.78 5.00 4.88 3.00 4.98

  144. [159]

    Cohesion 4.76 4.95 4.65 4.99 4.87 3.00 4.98

  145. [160]

    Completeness 4.32 4.46 4.54 4.61 4.69 3.00 4.75

  146. [161]

    Conciseness 4.77 4.75 4.65 4.15 4.73 2.99 4.05

  147. [162]

    Correctness 4.89 4.96 4.83 4.80 4.92 3.00 4.92

  148. [163]

    Informativeness 4.65 4.92 4.77 4.98 4.87 3.01 4.95

  149. [164]

    Integration 4.76 4.90 4.56 4.92 4.92 3.00 4.96

  150. [165]

    Readability 4.80 4.98 4.90 4.41 4.89 3.00 4.94

  151. [166]

    Relevancy 4.73 4.90 4.75 4.99 4.93 3.03 4.94 extreme

  152. [167]

    Coherence 2.66 3.39 3.33 4.31 4.82 1.01 1.00

  153. [168]

    Cohesion 3.80 4.30 3.56 4.88 4.87 3.00 4.46

  154. [169]

    Completeness 2.74 2.97 3.56 3.63 4.56 1.00 1.00

  155. [170]

    Conciseness 1.09 1.02 1.62 2.29 4.56 1.00 1.00

  156. [171]

    Correctness 3.90 4.07 3.85 4.25 4.89 1.02 1.00

  157. [172]

    Informativeness 3.76 4.30 3.87 4.69 4.81 1.02 1.00

  158. [173]

    Integration 3.50 4.03 3.27 4.74 4.88 2.83 3.07

  159. [174]

    Readability 2.16 2.03 2.95 3.85 4.71 1.00 1.00

  160. [175]

    Relevancy 2.80 3.67 3.70 4.92 4.86 1.00 1.00 subtle

  161. [176]

    Coherence 4.16 4.72 4.52 4.80 4.89 3.00 3.42

  162. [177]

    Cohesion 4.03 4.56 4.05 4.85 4.88 3.00 4.60

  163. [178]

    Completeness 4.13 4.37 4.13 4.98 4.66 2.99 4.49

  164. [179]

    Conciseness 2.51 2.07 2.29 3.44 4.77 2.22 2.20

  165. [180]

    Correctness 4.35 4.76 4.31 4.50 4.92 3.00 3.38

  166. [181]

    Informativeness 4.32 4.85 4.41 4.87 4.91 3.00 3.46

  167. [182]

    Integration 4.08 4.49 3.98 4.93 4.90 3.00 4.87

  168. [183]

    Readability 3.31 3.09 3.76 4.42 4.77 2.90 2.95

  169. [184]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.31 4.63 4.50 5.00 4.85 2.98 3.48 Table 12: ORKGSynthesis dataset detailed evaluation results of various LLMeval over LLMgen = LLaMA-3.1- 70B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  170. [185]

    Coherence 4.86 4.99 4.87 5.00 4.93 3.01 5.00

  171. [186]

    Cohesion 4.86 4.98 4.78 4.99 4.90 3.00 5.00

  172. [187]

    Completeness 4.47 4.65 4.62 4.76 4.70 3.00 4.93

  173. [188]

    Conciseness 4.87 4.81 4.72 4.30 4.81 3.00 4.13

  174. [189]

    Correctness 4.90 4.98 4.86 4.89 4.94 3.00 4.98

  175. [190]

    Informativeness 4.78 4.98 4.91 4.97 4.93 3.06 5.00

  176. [191]

    Integration 4.86 4.97 4.72 4.91 4.94 3.00 4.98

  177. [192]

    Readability 4.89 4.97 4.90 4.44 4.87 3.00 4.98

  178. [193]

    Relevancy 4.83 4.96 4.91 5.00 4.94 3.05 4.98 extreme

  179. [194]

    Coherence 2.74 3.75 3.54 4.47 4.91 1.04 1.00

  180. [195]

    Cohesion 3.94 4.44 3.70 4.91 4.88 3.00 4.51

  181. [196]

    Completeness 2.90 2.91 3.78 3.81 4.65 1.01 1.00

  182. [197]

    Conciseness 1.81 1.31 1.90 2.87 4.69 1.01 1.00

  183. [198]

    Correctness 3.84 4.15 4.01 4.31 4.92 1.01 1.00

  184. [199]

    Informativeness 3.84 4.51 3.98 4.78 4.81 1.00 1.00

  185. [200]

    Integration 3.82 4.34 3.60 4.86 4.89 2.89 3.52

  186. [201]

    Readability 2.38 2.16 3.07 4.18 4.76 1.00 1.00

  187. [202]

    Relevancy 3.06 3.70 3.83 4.98 4.88 1.00 1.00 subtle

  188. [203]

    Coherence 4.23 4.85 4.63 4.84 4.93 3.00 3.41

  189. [204]

    Cohesion 4.16 4.61 4.11 4.96 4.92 3.00 4.83

  190. [205]

    Completeness 4.40 4.50 4.30 4.99 4.73 3.00 4.79

  191. [206]

    Conciseness 3.13 2.82 2.68 4.50 4.83 2.89 2.97

  192. [207]

    Correctness 4.48 4.83 4.51 4.60 4.93 3.00 3.39

  193. [208]

    Informativeness 4.46 4.90 4.70 4.91 4.87 3.04 3.49

  194. [209]

    Integration 4.21 4.68 4.06 4.96 4.92 3.00 4.90

  195. [210]

    Readability 3.71 3.21 3.86 4.53 4.82 2.91 2.98

  196. [211]

    Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

    Relevancy 4.42 4.73 4.66 4.97 4.92 3.00 3.49 Table 13: ORKGSynthesis dataset detailed evaluation results of various LLMeval over LLMgen = Qwen2.5-72B. Set Rubrics M1 M2 M3 M4 M5 M6 M7 benign

  197. [212]

    Coherence 4.79 4.95 4.83 5.00 4.92 3.00 4.98

  198. [213]

    Cohesion 4.78 4.93 4.76 4.99 4.91 2.99 4.98

  199. [214]

    Completeness 4.39 4.50 4.61 4.62 4.81 3.00 4.86

  200. [215]

    Conciseness 4.82 4.80 4.71 4.26 4.75 2.98 4.07

  201. [216]

    Correctness 4.87 4.97 4.87 4.89 4.95 3.00 4.93

  202. [217]

    Informativeness 4.70 4.94 4.85 4.97 4.92 3.03 4.95

  203. [218]

    Integration 4.77 4.92 4.75 4.90 4.95 3.00 4.97

  204. [219]

    Readability 4.84 4.91 4.90 4.34 4.83 2.99 4.93

  205. [220]

    Relevancy 4.70 4.92 4.83 5.00 4.94 3.01 4.90 extreme

  206. [221]

    Coherence 2.58 3.50 3.47 4.34 4.88 1.00 1.00

  207. [222]

    Cohesion 3.72 4.29 3.62 4.87 4.90 2.99 4.60

  208. [223]

    Completeness 2.76 2.76 3.61 3.67 4.58 1.00 1.00

  209. [224]

    Conciseness 1.77 1.27 1.71 2.88 4.73 1.02 1.00

  210. [225]

    Correctness 3.82 4.06 3.92 4.21 4.91 1.00 1.00

  211. [226]

    Informativeness 3.69 4.43 3.90 4.83 4.87 1.00 1.00

  212. [227]

    Integration 3.52 4.09 3.37 4.79 4.85 2.94 3.34

  213. [228]

    Readability 2.36 2.10 2.97 3.84 4.62 1.01 1.00

  214. [229]

    Relevancy 2.89 3.50 3.76 4.95 4.83 1.00 1.00 subtle

  215. [230]

    Coherence 4.17 4.76 4.45 4.78 4.97 3.00 3.33

  216. [231]

    Cohesion 3.99 4.53 3.98 4.92 4.92 3.00 4.49

  217. [232]

    Completeness 4.05 4.38 4.11 5.00 4.66 3.00 4.64

  218. [233]

    Conciseness 2.98 2.66 2.60 4.22 4.85 2.90 2.89

  219. [234]

    Correctness 4.36 4.78 4.45 4.50 4.92 3.00 3.35

  220. [235]

    Informativeness 4.32 4.85 4.50 4.91 4.95 3.02 3.39

  221. [236]

    Integration 4.05 4.50 4.03 4.90 4.89 3.00 4.91

  222. [237]

    Readability 3.43 3.02 3.82 4.61 4.85 2.86 2.97

  223. [238]

    Relevancy 4.30 4.68 4.52 4.99 4.96 3.00 3.42 Table 14: ORKGSynthesis dataset detailed evaluation results of various LLMeval over LLMgen = Mistral-Large. Set LLMgen LLMeval Rubric Rating Rationale 1 subtle LLaMA-3.1-8BLLaMA-3.1-8B Readability 4 The synthesis follows academic wr...

    2024

  224. [1992]

    distance

    is a similarity metric grounded in the concept of minimum edit distance, measuring the number of edits (substitutions, insertions, and deletions) re- quired to transform a candidate text into reference text. Unlike character-level edit distance, WER operates at the word level,...

    2019

  225. [2018]

    In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896

    Generating natural language adversarial ex- amples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, ...

    2018 arXiv

  226. [2019]

    Transactions of the Association for Com- putational Linguistics, 7:249–266

    Coqa: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249–266. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2023. G...

    2023 arXiv

  227. [2021]

    In International Conference on Learning Representations

    Measuring massive multitask language under- standing. In International Conference on Learning Representations. Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in ...

    2017

  228. [2022]

    Advances in neural in- formation processing systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002a. Bleu: a method for automatic eval- uation of machine translation. In...

    2023

  229. [2023]

    In The Twelfth Inter- national Conference on Learning Representations

    Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth Inter- national Conference on Learning Representations. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, an...

    2024 arXiv

  230. [2024]

    for summarization tasks also align with ours. However, GPTScore also includes dialogue rubrics aligned with instruction-following chat-based eval- uations such as likeability, flexibility, inquisitive- ness, and engagement, which are out of scope for scienceQ&A. Open-ended sto...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.