An Automated Length-Aware Quality Metric for Summarization

Andrew D. Foland

REVIEW 4 major objections 4 minor 16 references

An Automated Length-Aware Quality Metric for Summarization

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes NOIR, a reference-free, parameter-free metric that scores summary quality by the ratio of token-length compression to semantic retention.

desk verdict Novel ratio-of-logs summary metric with a clean distributional result, but the score blows up for near-verbatim copies and the human-correlation evidence is much too thin; worth refereeing, not adopting. read the letter →

arxiv 2507.07653 v1 pith:QAJ3KBZA submitted 2025-07-10 cs.CL

Andrew D. Foland This is my paper · ORCID

classification cs.CL

keywords summarizationevaluationsemanticembeddingsreference-freemetriccompression-recalltradeoffNOIRsentencehumancorrelation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Summarization quality is usually judged by comparing a generated summary to human-written reference summaries, which is slow and measures agreement with a reference rather than skill at condensing the original text. This paper proposes NOIR (NOrmed Index of Retention), a metric that instead compares the summary directly to the parent document: it takes the ratio of the logarithm of the token-compression factor to the logarithm of the semantic similarity between document and summary. Because the ratio cancels the unknown number of compression steps, NOIR has no free parameters and needs no reference summary, only a text and a candidate summary plus an embedding model that maps identical meaning to 1 and unrelated text to 0. The paper presents evidence that NOIR tracks how well a summarizer trades length for meaning, separates true summaries from random shorter text, and correlates with human rank ordering of summaries more strongly than standard metrics such as ROUGE. The payoff, if the metric holds up, is a fast automated quality signal for guiding or filtering LLM-generated summaries.

What carries the argument

The load-bearing object is the idealized multiplicative summarization model: each summarization step multiplies the token count by a factor $k$ and the semantic similarity by a factor $D$, so that after $N$ steps $T_N=T_0 k^N$ and $D_N=D^N$. From these observables the ratio $M_{\rm NOIR}=\ln(T_N/T_0)/\ln(D_N) = \ln k/\ln D$ extracts a parameter-free quality score without knowing $k$ or $N$. The semantic degradation $D$ is measured in practice by cosine similarity between a sentence-embedding vector of the parent text and that of the summary; a heuristic random-walk argument, comparing $\cos(\sqrt{x})$ with an exponential decay, is used to motivate treating cosine loss as multiplicative over the relevant range. The metric's continuity, its sign (negative when the candidate is longer than the parent), and its embedder-independence under the stated 0-to-1 conditions all follow from this form.

What would settle it

A direct check is to summarize the same set of documents once, twice, and three times with one summarizer, measure D_1, D_2, and D_3 by embedding cosine similarity, and test whether D_2 is close to $D_1^{2}$ and D_3 close to $D_1^{3}$; if the multiplicative law fails in the 0.2-to-1.0 similarity range the paper uses, then the log-ratio no longer equals ln k / ln D and NOIR loses its claimed interpretation.

Watch

Extended reading notes

Core claim

The central claim is that the token-length and semantic-retention tradeoff of a summarizer can be expressed by a single scalar, NOIR, defined as $M_{\rm NOIR} = \ln(T_N/T_0)/\ln(D_N)$, where $T_N/T_0$ is the fraction of tokens left after summarization and $D_N$ is the cosine similarity between the semantic embeddings of the original text and the summary. The paper models an idealized summarizer that multiplies token count by $k$ and semantic similarity by $D$ at each step, so after $N$ steps $T_N = T_0 k^N$ and $D_N = D^N$; the log ratio then equals $\ln k / \ln D$, a quantity that is monotone in semantic retention and compression and does not require knowing $N$ or the per-step factors. The metric measures the summarizer's quality directly: the mean NOIR of 4.55 for the paper's hierarchical summaries implies a per-halving semantic retention factor of about 0.86, matching the observed falloff of similarity with compression on the dataset. The paper also reports that NOIR values are well separated from randomly paired text and summaries, that the score is largely unchanged across two different sentence embedders, and that it has a rank correlation of 0.36 with human quality judgments, higher than the reference-based ROUGE metric in the cited comparison.

Load-bearing premise

The metric's interpretation depends on the premise that semantic similarity falls off multiplicatively as a text is repeatedly summarized, so that after N steps the remaining similarity is D^N for a fixed per-step factor D, making the ratio of logarithms a stable per-step tradeoff rather than an artifact of summary depth.

Editorial extensions

If this is right

With only a text and a candidate summary, NOIR computes a quality score in seconds, so summaries can be ranked automatically without waiting for human judgments.
NOIR rewards compression explicitly; a candidate longer than the parent text receives a negative score even if it preserves meaning, something reference-recall metrics do not express.
Across hierarchical summaries (summary of a summary), NOIR remains well separated from random pairings, with a mean of 4.55 in the paper's dataset, so it can evaluate multi-stage condensation.
Swapping the embedding model from all-MiniLM-L6-v2 to all-mpnet-base-v2 changes absolute NOIR values but preserves ranking (rank correlation 0.55), indicating the metric is not tied to one embedder.
A threshold on NOIR can be used to filter automatically generated summaries, feeding only high-quality candidates into training data, user-facing outputs, or reference pools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Left implicit in the paper is the possibility that the same log-ratio could serve as a reward signal for training a summarizer, not only as a post-hoc filter, since NOIR is a scalar computed from the summary alone.
The construction transfers naturally to other compression tasks that pair a parent object with a condensed child, such as headline, subtitle, or caption generation, whenever a trusted similarity measure exists.
The paper's own plot of $\cos(\sqrt{x})$ against an exponential suggests the multiplicative-motivation weakens below a similarity of about 0.2, so extremely short summaries (one sentence or a few words) are the most likely place for NOIR to under- or over-rate quality.
Because the paper's dataset is English news, Wikipedia, and fiction, multilingual behavior and domain-specific behavior remain open; a multilingual embedder would allow a direct test of NOIR's generality.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about arXiv:2507.07653. First, the core idea is genuinely new: NOIR = ln(compression)/ln(cosine-similarity) is a one-line, reference-free metric that I haven't seen elsewhere, and it separates true summaries from random pairings cleanly on a 456-text dataset (4.55 vs 0.35). Second, the metric has a mathematical singularity that makes it behave badly in a regime that actually matters: as similarity D approaches 1, ln(D) goes to 0- and NOIR goes to +infinity. A "summary" that just deletes 10% of tokens with D=0.99 scores about 10.5, double the mean for real summaries. Figure 4 in the paper shows that faithful summaries have similarities near 1, so this is not a far-fetched corner case. The metric's own motivation is D/k, but the log-ratio is not monotone in that quantity. It rewards barely-compressed near-copies more than genuinely compressed summaries. That's a load-bearing flaw for a metric that claims to capture the compression-retention tradeoff.

What the paper does well: it releases code and data, checks that the embedding doesn't trivially encode length, and shows that a second embedder gives qualitatively similar results. The distributional separation is convincing and cheap to reproduce.

The soft spots beyond the singularity: the human evaluation is one evaluator, fifteen items, rank correlation 0.36, with no significance test. At n=15 that rho is not statistically distinguishable from zero. The paper overstates "correlates to human perception." Also, the "no free parameters" claim is a bit overstated: p=1 is chosen from the same data in Figure 10, and the 0.66 coefficient is fitted to justify the multiplicative Ansatz, though the paper is transparent about both.

I'd send this to a serious referee, because the idea is novel and the distributional result is real. But the authors need to address the high-D singularity, either by restricting the metric's domain, modifying the formula, or showing empirically that the problematic regime doesn't occur for real summaries. And they need a real human study with multiple annotators and significance testing. Without those, the metric is not ready to be used as a general evaluation tool.

Who should read it: anyone working on summarization evaluation or LLM-output filtering will want to know this paper exists, but they should not adopt NOIR as-is.

Referee Report

4 major / 4 minor

Summary. The paper proposes NOIR, a reference-free summarization quality metric defined as M_NOIR = ln(T_N/T_0) / ln(D_N), where T_N/T_0 is the token compression ratio and D_N is the cosine similarity between embeddings of the original text and the summary. The authors argue that this ratio captures the compression/retention tradeoff without free parameters, and they present experiments with two embedding models, a dataset of 456 texts with hierarchical summaries, and a small human study. The main empirical findings are that true summaries have NOIR values well separated from random text–summary pairings and that the metric varies only modestly with the choice of embedding model when the embedder satisfies zero/one anchor conditions. The paper also reports a rank correlation of 0.36 with a single human evaluator on 15 items.

Significance. If the central claims held, NOIR would be a useful, inexpensive, fully automatic alternative to reference-based summarization metrics such as ROUGE and BERTScore. The paper ships reproducible code and data, reports machine-checkable distributional results, and demonstrates a clear separation between genuine summaries and random pairings, which is a genuine strength. The claim of a parameter-free, theoretically motivated scalarization of compression versus retention is, however, undermined by a domain pathology in the formula, and the human-correlation evidence is too weak to support the stated conclusion. The idea of comparing the candidate summary to the parent text and explicitly rewarding compression is worth pursuing, but the specific metric as defined is not a stable measure of the tradeoff it is intended to quantify.

major comments (4)

[§4.1, Eq. (3)] The metric M_NOIR = ln(T_N/T_0) / ln(D_N) has a singularity as D_N approaches 1 from below. For any summary with compression ratio r = T_N/T_0 < 1 and D_N just below 1, the denominator tends to 0− while the numerator is finite and negative, so M_NOIR tends to +∞. A near-verbatim excerpt with r = 0.9 and D = 0.99 receives M ≈ 10.5, more than twice the reported genuine-summary mean of 4.55 in Figure 8, and a r = 0.5, D = 0.99 excerpt receives M ≈ 69. Meanwhile a genuinely compressed summary with r = 0.5 and D = 0.9 has D/k = 9 but M ≈ 6.6, while the near-copy with r = 0.9 and D = 0.99 has D/k ≈ 1.98 but M ≈ 10.5. Thus NOIR is not monotone in the quantity D/k that the paper itself uses to motivate the metric, and it can rank low-compression near-copies above genuine summaries. This contradicts the stated goal of rewarding summarization skill and makes the central claim that NOIR captures the recall-compression tradeoff questionable for a relevant regime, since Figure 4 shows faithful summaries have similarities near 1.
[§4.3, Figure 2] The multiplicative degradation assumption D_N = D^N, on which the interpretation of the log-ratio rests, is not established. The paper supports it with a heuristic random-walk argument and a least-squares fit of cos(√x) to e^(−0.66x), where 0.66 is itself a fitted constant. A fit over a finite range of a function that resembles an exponential is not a derivation of multiplicative degradation, and the paper acknowledges that the model is only a motivation. This matters because in §6.1 the mean NOIR of 4.55 is translated into a per-halving semantic degradation factor of 0.86, an interpretation that is meaningful only if repeated summarization really degrades similarity multiplicatively. If the true degradation is non-multiplicative, then the ratio of logarithms does not correspond to a stable per-step quality measure, and the claim that NOIR is a theoretically grounded, parameter-free metric is substantially weakened. The authors should either validate the multiplicative model directly on hierarchical summarization chains or present the metric without the per-step interpretation.
[§6.3, Figure 12] The human evaluation is far too weak to support the claim that NOIR 'correlates with human perception of summarization quality.' The study uses 15 items, a single evaluator, and reports a rank correlation of 0.36 with no significance test, no confidence interval, and no inter-annotator agreement because there is only one annotator. For n = 15, a Spearman correlation of 0.36 is not statistically significant at conventional levels. The claim that this correlation is 'significantly larger' than values reported for ROUGE is therefore not supported. A credible validation would require a substantially larger set of summaries, multiple annotators, an explicit significance test or confidence interval, and ideally a preregistered analysis. Without such evidence, the human-correlation contribution listed in the Introduction is not established.
[§6.2.1, Eq. (5)] The paper repeatedly claims that NOIR has 'no free parameters' (Abstract, Introduction, Section 4.1, Conclusion), but the generalized expression in Eq. (5) introduces a power p, and the value p = 1 is selected empirically by maximizing the separation between the random and true-summary distributions on the same dataset used for evaluation. Choosing a hyperparameter on the evaluation set is a form of model selection, and it undercuts the parameter-free claim. Even if p = 1 is motivated by the form of Eq. (3), the empirical peak at 1.0 reported in Figure 10 is not independent evidence. The authors should either remove the claim of being parameter-free, or demonstrate that the choice of p is fixed a priori and that the conclusions are robust to reasonable variations in p.

minor comments (4)

[Abstract, §1] There are multiple typographical errors throughout, including 'NOrmed' instead of 'Normed', 'sumarization' in the Abstract, 'explcitly' in Section 2, 'previsouly' in Section 5, and 'cmopression' and 'write' in Figure 11. These should be corrected.
[§6.3] The text states 'The resulting rank correlations are shown in figure .' but the figure number is missing; it should refer to Figure 12.
[§6.2.1, Eq. (5)] The parenthetical remark that the compression factor is 'nearly always negative' is incorrect: T_N/T_0 is a positive ratio less than 1 for summarization, so it is the logarithm that is negative, not the ratio itself.
[Figure 12] The caption 'Human rank-ordering of test-summary pair summarization quality as a function of NOIR metric rank for the pair' is unclear about which variable is plotted on each axis; the figure should be labeled more explicitly.

Circularity Check

1 steps flagged · score 2.0 of 10

Metric is mostly self-contained; minor in-sample choice of power p slightly undercuts the 'no free parameters' claim.

fitted input called prediction [Section 6.2.1, Eq. (5), Figure 10; cf. Section 6.1, Figure 8]
"We evaluate the case where the NOIR expression is modified by raising the numerator to a power: MNOIR = − ln | TN/T0 |^p / ln D (5) ... We evaluate the separation of the random NOIR distribution and the summary NOIR distribution ... The resulting distribution, shown in figure 10, has a broad maximum, peaking at 1.0. We conclude that 1.0, as motivated, is empirically an acceptable point of operation."

The paper advertises an 'objective, automated, reference-free, no-free-parameters summarization metric' (Contributions) and later reports that true summaries are separated from random pairings (Section 6.1, Figure 8). But the power p in Eq. (5) is selected on the same dataset to maximize that separation. Choosing p to optimize the in-sample separation and then citing the resulting separation as evidence that NOIR 'definitively distinguishes' summaries is self-confirmation: the evaluation statistic is used as a model-selection criterion. The effect is mild because p=1 was also the a priori motivated value and the optimum is broad, and the human-correlation result (Section 6.3) is independent; nevertheless, the 'no free parameters' claim is not cleanly supported.

full rationale

Most of the derivation is self-contained rather than circular. NOIR is defined directly from observables in Eq. (3) via the idealized multiplicative model of Eqs. (1)-(2); the human-correlation result (Section 6.3), the embedder-robustness check (Section 6.2.2), and the null-distribution separation (Figure 7) are external demonstrations, not constructed from the metric's inputs. The only mild circularity is in Section 6.2.1: the power p in Eq. (5) is selected by maximizing separation between true and random summaries on the same dataset used to report that separation as evidence of utility (Section 6.1, Figure 8). This is an in-sample choice, so the 'no free parameters' claim is slightly overstated. It is not load-bearing for the central human-correlation claim, and the coefficient 0.66 used to motivate the multiplicative ansatz does not enter the final metric. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result; score 2 reflects the one minor self-confirmation rather than genuine derivation-by-construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The NOIR metric does not introduce new physical or conceptual entities. Its central load-bearing assumptions are the multiplicative-degradation Ansatz, the embedding-based similarity measure, the token-count proxy for compression, and the absence of length correlations in the embedding. The p power is technically a free parameter in a generalized form, selected via data, while the 0.66 coefficient is a motivation-only fit.

free parameters (2)

power p in generalized NOIR expression = 1.0
Equation (5) introduces a power p on the compression ratio. The paper selects p=1 based on the empirical separation of summary versus random distributions in Figure 10, despite claiming the metric has no free parameters.
exponential approximation coefficient 0.66 = 0.66
Fitted by least squares in Figure 2 to approximate cos(sqrt(x)) by e^{-0.66x}. Used only to motivate the multiplicative-degradation Ansatz; it does not appear in the final NOIR formula.

assumptions (4)

ad hoc to paper Semantic degradation under repeated summarization is multiplicative, so D_N = D^N for each summarization step.
Section 4.1 defines an idealized summarizer with constant per-step degradation D. Section 4.3 supports this with a random-walk heuristic and a fitted exponential approximation. The meaningfulness of the log-ratio metric depends on this assumption.
domain assumption Cosine similarity of sentence embeddings provides a valid semantic degradation measure, with D=1 for identical meaning and D=0 for unrelated texts.
Section 4.2 states these requirements, and Sections 5.2 and 5.6 provide empirical evidence for the two chosen embedders. The metric's semantic component relies entirely on this property.
domain assumption Token count as measured by tiktoken is an adequate proxy for text length and compression.
Section 5.4 acknowledges that different tokenizers give slightly different counts but assumes the differences cancel in compression ratios. This underpins the length component of the metric.
domain assumption Embedding vector components are not significantly correlated with input length, so observed cosine differences reflect semantics rather than length encoding.
Section 5.7.2 tests component-wise correlations with normalized length and finds no strong patterns. This supports the claim that NOIR is not confounded by length artifacts in the embedding.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Automated Length-Aware Quality Metric for Summarization." pith.science (2026). https://pith.science/paper/QAJ3KBZA

@misc{pith2026250707653,
  author       = {Pith},
  title        = {Pith review of: An Automated Length-Aware Quality Metric for Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QAJ3KBZA}},
  note         = {Machine review of arXiv:2507.07653}
}

read the original abstract

This paper proposes NOrmed Index of Retention (NOIR), a quantitative objective metric for evaluating summarization quality of arbitrary texts that relies on both the retention of semantic meaning and the summary length compression. This gives a measure of how well the recall-compression tradeoff is managed, the most important skill in summarization. Experiments demonstrate that NOIR effectively captures the token-length / semantic retention tradeoff of a summarizer and correlates to human perception of sumarization quality. Using a language model-embedding to measure semantic similarity, it provides an automated alternative for assessing summarization quality without relying on time-consuming human-generated reference summaries. The proposed metric can be applied to various summarization tasks, offering an automated tool for evaluating and improving summarization algorithms, summarization prompts, and synthetically-generated summaries.

Figures

Figures reproduced from arXiv: 2507.07653 by the authors.

**Figure 2.** Demonstration of similarity of cos √ x and e −0.66x over the range from cos √ x = 1 to cos √ x = 0.2. The coefficient value of 0.66 is found by a least-squares fit to the cosine over the range. The functional closeness motivates the choice of cosine similarity as a multiplicative semantic degradation measure. 4.5 Summarization Summarization is a common NLP task. Given the recent interest in large language models (LL… view at source ↗

**Figure 3.** Distribution of semantic similarity for random pairings of paragraphs with the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Distribution of cosine similarity for pairings of paragraphs with their summaries. The distribution is peaked [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Distribution of cosine similarity for paragraphs and their paraphrases, as a function of normalized paraphrase [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Observed correlations between 368 individial embedding vector components and the normalized length of the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Distribution of MNOIR quality for random pairings of paragraphs with summaries of other paragraphs. The mean value is 0.35, close to 0 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Distribution of MNOIR quality for summaries, summaries of summaries, and summaries of summaries of summaries. Also shown is a gaussian fit to the distribution. The fitted mean value is 4.55±0.06, with a Gaussian σ of 2.08. It is well-separated from the values for rando…

**Figure 9.** Figure 9: Distribution of MNOIR against input text length (in tokens). In addition to the scatterplot, the average y value (MNOIR value) is shown, together with its standard error on the mean. Also shown is a linear fit to the trend. The fitted slope is 0.0003±0.0005, or at most…

**Figure 10.** Figure 10: MNOIR distribution separation (in standard deviations of the distribution) between random pairings of text, and true summaries, as a function of the power p applied to the denominator of NOIR. The separation peaks at 1.0, the originally-motivated value of p, and we co…

**Figure 11.** Figure 11: Analysis of the summary dataset using another embedder, [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

**Figure 12.** Figure 12: Human rank-ordering of test-summary pair summarization quality as a function of NOIR metric rank for the [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 6 canonical work pages

[1]

The Current State of Summarization

Fabian Retkowski. The Current State of Summarization. arXiv e-prints , art. arXiv:2305.04853, May

arXiv
[4]

neurips.cc/paper_files/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf

URL https://proceedings. neurips.cc/paper_files/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf. Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Lo...

work page 2021
[7]

doi:10.18653/v1/2020.eval4nlp-1.2

Association for Computational Linguistics. doi:10.18653/v1/2020.eval4nlp-1.2. URL https://aclanthology. org/2020.eval4nlp-1.2. Xiaobo Guo and Soroush V osoughi. Length does matter: Summary length can bias summarization metrics. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Languag...

doi:10.18653/v1/2020.eval4nlp-1.2 2020
[8]

doi:10.18653/v1/2023.emnlp-main.984

Association for Computational Linguistics. doi:10.18653/v1/2023.emnlp-main.984. URL https://aclanthology.org/2023.emnlp-main.984. Kyunghyun Cho Felix Hill and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Lingui...

work page doi:10.18653/v1/2023.emnlp-main.984 2023
[10]

org/abs/2002.10957

URL https://arxiv. org/abs/2002.10957. Ilya Sutskever, Oriol Vinyals, and Quoc V . Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215,

arXiv 2002
[12]

URLhttps://arxiv.org/abs/2210

doi:10.48550/ARXIV .2210.07316. URLhttps://arxiv.org/abs/2210. 07316. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space,

arXiv doi:10.48550/arxiv.2210.07316
[14]

doi:10.3115/v1/D14-1162

Association for Computational Linguistics. doi:10.3115/v1/D14-1162. URL https://aclanthology.org/D14-1162. 14 NOIR Metric A PREPRINT Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805,

arXiv
[17]

Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR, abs/1608.04207,

arXiv

Show all 16 references

[19]

Feifan Liu and Yang Liu

URL https://arxiv.org/abs/2004.09297. Feifan Liu and Yang Liu. Exploring correlation between rouge and human evaluation on meeting summaries. IEEE Transactions on Audio, Speech, and Language Processing , 18(1):187–196,

2004 arXiv
[2010]

doi:10.1109/TASL.2009.2025096. 15

2009
[2013]

GloVe: Global vectors for word representation

Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1...

2014
[2014]

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers

URL http://arxiv.org/abs/1409.3215. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316,

arXiv
[2019]

URL http://arxiv.org/abs/1907.11692. P. Krapivsky. https://mathoverflow.net/questions/320630/,

1907 arXiv
[2020]

Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon

URL https://arxiv.org/abs/2005.03724. Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. Fill in the BLANC: Human-free quality estimation of document summaries. In Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, and Eduard Hovy, editors,Proceedings of the First Workshop ...

2005 arXiv
[2021]

doi:10.1162/tacl_a_00373

ISSN 2307-387X. doi:10.1162/tacl_a_00373. URL https://doi.org/10.1162/tacl_ a_00373. Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July

doi
[2023]

Alexander R

doi:10.48550/arXiv.2305.04853. Alexander R. Fabbri, Wojciech Kry´sci´nski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. SummEval: Re-evaluating Summarization Evaluation. Transactions of the Association for Computational Linguistics , 9:391–409, April

doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

The Current State of Summarization

Fabian Retkowski. The Current State of Summarization. arXiv e-prints , art. arXiv:2305.04853, May

arXiv

[2] [4]

neurips.cc/paper_files/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf

URL https://proceedings. neurips.cc/paper_files/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf. Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Lo...

work page 2021

[3] [7]

doi:10.18653/v1/2020.eval4nlp-1.2

Association for Computational Linguistics. doi:10.18653/v1/2020.eval4nlp-1.2. URL https://aclanthology. org/2020.eval4nlp-1.2. Xiaobo Guo and Soroush V osoughi. Length does matter: Summary length can bias summarization metrics. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Languag...

doi:10.18653/v1/2020.eval4nlp-1.2 2020

[4] [8]

doi:10.18653/v1/2023.emnlp-main.984

Association for Computational Linguistics. doi:10.18653/v1/2023.emnlp-main.984. URL https://aclanthology.org/2023.emnlp-main.984. Kyunghyun Cho Felix Hill and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Lingui...

work page doi:10.18653/v1/2023.emnlp-main.984 2023

[5] [10]

org/abs/2002.10957

URL https://arxiv. org/abs/2002.10957. Ilya Sutskever, Oriol Vinyals, and Quoc V . Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215,

arXiv 2002

[6] [12]

URLhttps://arxiv.org/abs/2210

doi:10.48550/ARXIV .2210.07316. URLhttps://arxiv.org/abs/2210. 07316. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space,

arXiv doi:10.48550/arxiv.2210.07316

[7] [14]

doi:10.3115/v1/D14-1162

Association for Computational Linguistics. doi:10.3115/v1/D14-1162. URL https://aclanthology.org/D14-1162. 14 NOIR Metric A PREPRINT Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805,

arXiv

[8] [17]

Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR, abs/1608.04207,

arXiv