RECORD
$\texttt{COSMIC}$: Mutual Information for Task-Agnostic Summarization Evaluation
Computation and Language · Submitted 29 Feb 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Assessing the quality of summarizers poses significant challenges. In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$ highlight the competitive performance of $\texttt{COSMIC}$.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Assessing the quality of summarizers poses significant challenges. | background 0.97 | 0.05 | 0.02 | asserted 1.00 |
| s1 | In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. | contribution 0.96 | 0.84 | 0.16 | asserted 0.99 |
| s2 | We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. | contribution 0.76 | 0.94 | 0.17 | asserted 1.00 |
| s3 | We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. | contribution 0.51 | 0.96 | 0.54 | asserted 1.00 |
| s4 | Comparative analyses against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$ highlight the competitive performance of $\texttt{COSMIC}$. | result 0.98 | 0.92 | 0.21 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Assessing the quality of summarizers poses significant challenges. In response, we propose a novel task-oriented evaluation approach that assesses summarizers based on their capacity to produce summaries that are useful for downstream tasks, while preserving task outcomes. We theoretically establish a direct relationship between the resulting error probability of these tasks and the mutual information between source texts and generated summaries. We introduce $\texttt{COSMIC}$ as a practical implementation of this metric, demonstrating its strong correlation with human judgment-based metrics and its effectiveness in predicting downstream task performance. Comparative analyses against established metrics like $\texttt{BERTScore}$ and $\texttt{ROUGE}$ highlight the competitive performance of $\texttt{COSMIC}$.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites · 26 sentences without a confident label
Uses the method or tool of 10 papers
-
We rely on the KNIFE estimator [ref] to estimate the differential entropy of the embeddings and then mutual information.
A Differential Entropy Estimator for Training Neural Networks · Estimating MI from samples · 1.00
-
Therefore, following prior research [ref] we estimate Arimoto information [ref] based on the KNIFE estimator [ref] .
Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information · Estimating MI from samples · 0.99
-
Therefore, following prior research [ref] we estimate Arimoto information [ref] based on the KNIFE estimator [ref] .
A Theory of Usable Information Under Computational Constraints · Estimating MI from samples · 0.99
Uses data or values from 2 papers
-
While the SummEval [ref] dataset provides only a very limited number of human judgments, it contains enough aligned source, and AI-generated summaries to evaluate the MI metric.
SummEval: Re-evaluating Summarization Evaluation · Experimental Settings · 0.98
-
We provide additional experiments in French and Spanish using the MLSUM [ref] and XLSUM [ref] datasets.
MLSUM: The Multilingual Summarization Corpus · Experimental Settings · 0.99
Cites as background 5 papers
-
[ref] first proposed a new information-theoretic metric to compare two text distributions based on embedding clustering; more recent work showed that the crucial element was the clustering step [ref] .
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation · Related Work · 0.88
-
[ref] first proposed a new information-theoretic metric to compare two text distributions based on embedding clustering; more recent work showed that the crucial element was the clustering step [ref] .
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers · Related Work · 0.88
-
Another line of research focuses on reconstruction-based metrics, such as BLANC [ref] and Information Difference [ref] , the former evaluating the actual reconstruction error of the source text with and without a summary, and the latter evaluating the information gain obtained when conditioning the generative model on the summary.
Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation · Introduction · 0.93
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.