RECORD
What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization
Computation and Language · Submitted 12 May 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise--the disagreement between model and metric defined candidate rankings--minimized.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE). To address this, recent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness. While effective, much of this work has focused on how to generate and optimize these sets. Less is known about why one setup is more effective than another. In this work, we uncover the underlying characteristics of effective sets. For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for calibration fine-tuning. Each selection strategy targets distinct aspects of the sets, such as lexical diversity or the size of the gap between positive and negatives. On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise--the disagreement between model and metric defined candidate rankings--minimized. Code to create, select, and optimize calibration sets is available at https://github.com/griff4692/calibrating-summaries
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE). | background 1.00 | 0.04 | 0.02 | asserted 0.99 |
| s1 | To address this, recent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness. | background 1.00 | 0.03 | 0.02 | asserted 1.00 |
| s2 | While effective, much of this work has focused on how to generate and optimize these sets. | background 0.66 | 0.05 | 0.02 | asserted 0.98 |
| s3 | Less is known about why one setup is more effective than another. | gap 1.00 | 0.04 | 0.02 | asserted 0.98 |
| s4 | In this work, we uncover the underlying characteristics of effective sets. | contribution 0.99 | 0.75 | 0.09 | asserted 0.99 |
| s5 | For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for calibration fine-tuning. | method 0.99 | 0.95 | 0.03 | asserted 0.99 |
| s6 | Each selection strategy targets distinct aspects of the sets, such as lexical diversity or the size of the gap between positive and negatives. | method 0.97 | 0.76 | 0.03 | asserted 0.98 |
| s7 | On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise--the disagreement between model and metric defined candidate rankings--minimized. | result 1.00 | 0.97 | 0.91 | asserted 1.00 |
| s8 | Code to create, select, and optimize calibration sets is available at https://github.com/griff4692/calibrating-summaries | contribution 0.84 | 0.52 | 0.02 | asserted 0.98 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE). To address this, recent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness. While effective, much of this work has focused on how to generate and optimize these sets. Less is known about why one setup is more effective than another. In this work, we uncover the underlying characteristics of effective sets. For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for calibration fine-tuning. Each selection strategy targets distinct aspects of the sets, such as lexical diversity or the size of the gap between positive and negatives. On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise--the disagreement between model and metric defined candidate rankings--minimized. Code to create, select, and optimize calibration sets is available at https://github.com/griff4692/calibrating-summaries
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Uses its method or tool 1 paper
-
We compute a different random sample for each test case to encourage diversity, as in [ref] .
Generating EDU Extracts for Plan-Guided Summary Re-Ranking · Guiding GPT with EDU Plans · 0.81
What it cites · 3 sentences without a confident label
Builds on 1 paper
-
[ref] extend the work of [ref] and run a broad sweep of loss functions and candidate generation methods for two-step relevance calibration while establishing state of the art performance (ROUGE) across single document corpora.
Calibrating Sequence likelihood Improves Conditional Language Generation · Related Work · 0.87
Uses the method or tool of 4 papers
-
As in [ref] , we calibrate for relevance by learning to rank model-generated summaries (post-, pre- weights).
Calibrating Sequence likelihood Improves Conditional Language Generation · Relevance Calibration · 1.00
-
We fine-tune () two state of the art long-document language models: LongT5 [ref] and PRIMERA [ref] , on each corpus before decoding 10 candidates with diverse beam search [ref] with diversity penalty of $1.0$, as in [ref] .
Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models · Relevance Calibration · 1.00
-
As in [ref] , we use contrastive learning to minimize the latent distance between pairs of positive summaries vis-a-vis negative ones:
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning · Faithfulness Calibration · 1.00
Cites as support 1 paper
-
This is validated by previous work which shows a decrease in relevance when models are trained to be more faithful [ref] .
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data · Relevance and Faithfulness at Odds · 0.99
Cites as background 1 paper
-
Yet, much of this work has addressed how—how to generate candidates [ref] and how to define effective calibration objectives [ref] .
Calibrating Sequence likelihood Improves Conditional Language Generation · Introduction · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.