REVIEW 5 cited by
BARTScore: Evaluating Generated Text as Text Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
Forward citations
Cited by 5 Pith papers
-
C3-Bench: A Context-Aware Change Captioning Benchmark
C3-Bench supplies a multi-domain dataset and LLM-based evaluation protocol that exposes systematic failures in existing change captioning models outside their training regimes.
-
Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages
LLM-as-judge metrics (DeepSeek-V3 most of all) correlate best with human ratings across six Indian languages, though all segment-level correlations are low and many differences lack confidence intervals.
-
AllSummedUp: un framework open-source pour comparer les metriques d'evaluation de resume
On SummEval, LLM-based summary evaluators are expensive and unstable, and several published correlations did not reproduce when using open-weight models.
-
QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting
A contextual bandit that chooses among five query-rewrite strategies, conditioned on 17 linguistic features, reduces LLM hallucination on QA benchmarks and beats static prompting and no-rewrite baselines.
-
Beyond ROUGE: N-Gram Subspace Features for LLM Hallucination Detection
Singular values of label-grouped n-gram frequency tensors are used as MLP features for hallucination detection, with reported gains on HaluEval that rely on label-aware grouping.
Discussion (0). Sign in to comment.