Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts

Matthias Samwald; Milad Moradi

REVIEW 3 major objections 6 minor 16 references

Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen BERT model tops biomedical summarizers.

desk verdict A simple, reproducible BERT-based biomedical summarizer with a modest ROUGE gain; the evaluation is thin but the core claim is plausible and worth refereeing. read the letter →

arxiv 1908.02286 v2 pith:WMSE6WDM submitted 2019-08-06 cs.CL

Milad Moradi , Matthias Samwald This is my paper · ORCID

classification cs.CL

keywords BERTbiomedicaltextsummarizationcontextualizedembeddingssentenceclusteringextractiveROUGEunsuperviseddomainknowledge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a general-purpose pretrained language model, used with no fine-tuning and no curated biomedical resources, can summarize biomedical articles better than generic statistical summarizers and summarizers that explicitly exploit biomedical knowledge. The proposed method maps each sentence to a vector by averaging BERT token embeddings, clusters the sentence vectors with hierarchical clustering, and extracts top-scoring sentences from each cluster. On 300 biomedical journal articles with abstracts as gold summaries, the method reports ROUGE-1 of 0.7639 and ROUGE-2 of 0.3481, higher than all four comparison systems at p<0.05. If the result holds, domain-specific summarization would no longer require expensive ontology or knowledge-base construction.

What carries the argument

The central object is the contextualized sentence embedding: each sentence is represented by the average of the BERT token embeddings of its words, so that sentences sharing context are nearby in vector space. Around this, the method builds an agglomerative hierarchical clustering, with the number of clusters $K$ set to 4, and a within-cluster score that ranks sentences by their average similarity to other sentences in the same cluster. This machinery converts summarization into a coverage problem: clusters stand for topics or contexts, and each cluster contributes sentences proportional to its size, forcing the summary to represent the document's main ideas rather than one dominant block of text.

What would settle it

Re-run the same pipeline on a corpus whose gold summaries are human-annotated extractive sentence selections known to appear verbatim in the source articles, and compare against the same four baseline methods. If the BERT-based method no longer achieves the reported margin or loses significance, the paper's advantage would be an artifact of abstract-based evaluation rather than genuine summarization ability. A cheaper check on the same data is to count how many abstract sentences appear verbatim in the body text; if the fraction is low, ROUGE cannot support the extractive claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that deep contextualized representations extracted from BERT carry enough semantic information about biomedical sentences that a simple pipeline—average token vectors, cluster sentences, pick representative sentences per cluster—outperforms summarizers built on curated biomedical concepts and probabilistic concept heuristics. The paper reports the highest scores at $K=4$ clusters using Euclidean distance, with ROUGE-1 0.7639 and ROUGE-2 0.3481, and states the improvement over comparison methods is significant at p<0.05. This matters because it suggests the context captured by a general language model can substitute for manually built domain knowledge in this task.

Load-bearing premise

The evaluation assumes that an article's abstract is an appropriate gold-standard extractive summary, meaning the best summary can be produced by selecting whole sentences from the main text and that ROUGE overlap with the abstract measures real summarization quality; if abstracts are not sentence-level selections from the body text, the scores lose their meaning.

Editorial extensions

If this is right

Biomedical summarization can be built without maintaining ontologies or knowledge bases, lowering the cost of domain-specific systems.
A frozen pretrained language model can serve as a reusable feature extractor for other specialized text domains.
Clustering contextual representations before sentence selection should improve coverage in extractive summaries of long documents, because each cluster forces representation of a distinct topic area.
The reported scores set a useful reference point that future biomedical summarizers, whether knowledge-based or neural, should be measured against on this corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same clustering pipeline might transfer directly to other technical domains such as legal or financial text, where curated knowledge is expensive; nothing in the method is biomedical-specific, though the paper does not test this.
Fine-tuning BERT on biomedical text, which the authors list as future work, could push the scores higher, making the current result a lower bound for contextualized models on this task.
The reported advantage may be sensitive to the clustering parameter $K$, which the paper fixes at 4; an automatic way to choose $K$ per document would be a natural testable extension.
Because the gold summaries are abstracts, the comparison may favor methods that select sentences close to the abstract's wording; a human evaluation of informativeness would test whether the ROUGE gain is a real quality gain.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, it does something clean and useful: it takes BERT as a fixed feature extractor, averages token embeddings to get sentence vectors, clusters them hierarchically, and picks representative sentences per cluster. No fine-tuning, no UMLS, no domain knowledge bases. That is a genuinely practical shift for biomedical summarization, even if the components are all off the shelf. Second, the empirical claim is plausible but not nailed down. The ROUGE gains over the knowledge-based baselines are real in the reported table (R-1 0.7639 vs 0.7501 for CIBS), but the evaluation has soft spots that a referee would need to see addressed.

What the paper does well: the method is simple enough to reproduce from the description, code and data are promised on GitHub, and the hyperparameter K and compression rate are chosen on a separate development set, so there is no obvious circular fitting of the test set. The comparison against older knowledge-based summarizers is a reasonable first benchmark, and the authors are honest that they only report BERT-Large because it scored higher than BERT-Base.

Now the soft spots, in proportion. The biggest is the gold-standard issue the stress-test note raises: using abstracts as model summaries for extractive summarization of full texts. That is a common convention in this subfield, and the high ROUGE-1 scores (0.76) suggest the abstracts are not wildly disjoint from the article bodies, but the paper does not report any overlap statistic or oracle bound. A referee should ask for verbatim-overlap numbers or a small human evaluation. This is a real gap, but it is addressable, and it does not sink the paper. Second, no variance or exact p-values are reported, just a Wilcoxon claim at p<0.05. That is a reporting failure, not a conceptual one. Third, the baseline set is dated and non-neural; a 2019 paper should at least compare against a simpler neural extractive baseline like sentence-position or TF-IDF with the same clustering. The margin over CIBS is about one ROUGE point, so stronger baselines might shrink it.

Who is this for? Someone building a low-cost summarizer for biomedical text who wants to skip ontology engineering. The paper is not a new method, but it is a solid, reproducible application with a clear message. It deserves a serious referee, not a desk reject.

Recommendation: send it to review, with requests for variance, an overlap analysis of abstracts vs. body text, and at least one stronger baseline. The central argument holds up conditionally; these are fixable issues.

Referee Report

3 major / 6 minor

Summary. The paper proposes an unsupervised extractive summarizer for biomedical articles. It represents each sentence by averaging BERT's contextualized token embeddings, groups sentences with agglomerative hierarchical clustering, and selects the most central sentences from each cluster, with the number of sentences per cluster proportional to cluster size. The method is evaluated on 300 BioMed Central articles using abstracts as gold summaries and ROUGE-1/2, reporting better scores than four baselines (CIBS, Bayesian summarizer, SUMMA, TexLexAn) and claiming statistical significance with a Wilcoxon signed-rank test at p<0.05. Code and data are made publicly available.

Significance. If the reported results are valid, the paper makes a useful contribution to biomedical text summarization by showing that a general-purpose pre-trained model (BERT) used purely as a feature extractor can outperform knowledge-based summarizers that rely on domain-specific resources such as UMLS. The algorithmic pipeline is simple, clearly described, and reproducible from the released artifacts. The claimed advantage would support a broader shift from domain-specific feature engineering to contextualized representations. However, the evaluation relies on an unexamined assumption about the suitability of abstracts as extractive gold summaries, and the statistical reporting is incomplete, so the central claim is currently not fully verified.

major comments (3)

[Section 3.1] The abstract-as-gold-standard assumption is load-bearing but untested. The system extracts sentences from the article body, while abstracts in BioMed Central articles are often structured, compressed, and may not have verbatim counterparts in the main text. The authors note that this convention is widely adopted in biomedical summarization, but that shows custom, not validity. Please add (a) a verbatim-overlap statistic between abstract unigrams/bigrams and main-text sentences, (b) an oracle upper bound on ROUGE for extractive sentence selection on this corpus, and (c) ideally a small human evaluation. Without this evidence, the reported ROUGE differences in Table 2 (e.g., R-1 0.7639 vs. 0.7501) could reflect incidental lexical overlap rather than better summarization quality.
[Section 3.3, Table 2] The significance claim is underreported. The text states that the BERT-based summarizer can significantly (p<0.05) improve performance according to a Wilcoxon signed-rank test, but no p-values, test statistics, or confidence intervals are given. Since the absolute differences in ROUGE scores are small, please report the full test results for each ROUGE metric, state whether the test is one- or two-sided, and clarify that the comparison is performed on per-document ROUGE scores.
[Sections 3.1 and 3.2] It is unclear whether Table 1 reports results on the development set or the evaluation set. The ROUGE-1 score for the BERT-based summarizer in Table 2 (0.7639) exceeds the maximum in Table 1 (0.7607), which suggests that Table 1 may be tuning on the development set and Table 2 on the evaluation set. Please state explicitly which corpus is used in each table, and describe how the value K=4 was selected. If K were chosen using the same corpus on which Table 2 is computed, the comparison would be affected by selection bias, and the reported advantage would not be a fair held-out result.

minor comments (6)

[Section 3.3] The text reads 'Wilcoxon signed-rank text'; this should be 'Wilcoxon signed-rank test'.
[Section 2.2] The description of BERT feature extraction is underspecified: please state which layer (or pooled output) is used for the token embeddings and whether the sentence representation is the average over all layers or a specific layer.
[Section 2.3] The linkage criterion for the agglomerative clustering is described implicitly as average linkage; please name it explicitly and define the distance between clusters in the text or in a formula.
[Section 3.1] Please clarify whether ROUGE-1 and ROUGE-2 are reported as F-measure, recall, or precision, as the ROUGE toolkit can output all three and the values are not comparable across these variants without clarification.
[Section 2.4, Equation (3)] In Equation (3), the symbol N is used for the summary size; please define N explicitly as the number of sentences in the final summary and explain how the compression rate of 0.3 is converted into N.
[Section 3.2] The paper states that both BERT-Base and BERT-Large are used to assess model-size impact, but only BERT-Large results are reported; please include a sentence or a small table showing BERT-Base results, or explicitly state that the effect is negligible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BERT-based summarizer is evaluated against external ROUGE metrics, hyperparameters are chosen on a separate development set, and no fitted value or self-citation chain forces the reported results.

full rationale

The paper's derivation chain is self-contained and externally anchored. BERT is used as a fixed, pre-trained feature extractor to map sentences to contextualized vectors, and the clustering and sentence-selection steps operate only on those vectors and on the input document's sentence set. The key evaluation measures, ROUGE-1 and ROUGE-2, compare the system summary against article abstracts, which are external to the method and not used as training or fitting targets. The clustering parameter K is selected on a separate development corpus (100 articles) and then applied to the evaluation corpus (300 articles), so the reported Table 2 scores are not fitted by construction. The WCS scoring in Eq. (4) and the cluster-size allocation in Eq. (3) determine which sentences are selected, but nothing in these equations is defined in terms of the abstract or the final ROUGE score. The self-citations to prior biomedical summarizers [2,3,6] are used to support the convention that abstracts can serve as model summaries and to supply comparison systems; this is a customary evaluation practice and not load-bearing for the BERT method's internal derivation. Even if the abstract-as-gold-standard assumption is questionable, that concern applies symmetrically to all compared systems and is a validity risk, not circular reasoning. The paper therefore exhibits no step where a prediction reduces by construction to an input, a fitted parameter, or a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method relies on standard assumptions about BERT embeddings and abstract-based evaluation. No new entities or physical constants are introduced. The main free parameters are hyperparameters chosen on a development corpus; none of them are fitted to the test-set ROUGE scores.

free parameters (4)

Number of clusters K = 4
Selected as the best value on the development corpus (Table 1). The paper varies K in [2,12] and reports the highest ROUGE at K=4.
Compression rate = 0.3
Set to 0.3 in all experiments, with no sensitivity analysis or principled derivation.
BERT model size = BERT-Large
The paper reports only BERT-Large results because 'the scores are higher than those of BERT-Base' (Section 3.2).
Pairwise distance measure = Euclidean distance
Best results in Table 1 use Euclidean distance over cosine similarity; the paper tries both and reports the better one.

assumptions (4)

domain assumption Sentences close in BERT embedding space share similar context and are appropriate to cluster together.
Invoked in Section 2.3, where the paper states that nearby sentences in vector space can share similar context. This is the core modeling assumption.
domain assumption The abstract of a biomedical article is a valid gold-standard summary for evaluating extractive summarization.
Invoked in Section 3.1; the evaluation corpus is built by using abstracts as model summaries, and the results are judged against this target.
domain assumption NLTK sentence tokenization is reliable for biomedical full-text articles.
Used in preprocessing (Section 2.1); errors in sentence splitting would propagate through embedding and clustering.
domain assumption ROUGE-1 and ROUGE-2 measure summary quality sufficiently for this comparison.
The paper uses only ROUGE unigram and bigram overlap, without human evaluation or other metrics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts." pith.science (2026). https://pith.science/paper/WMSE6WDM

@misc{pith2026190802286,
  author       = {Pith},
  title        = {Pith review of: Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WMSE6WDM}},
  note         = {Machine review of arXiv:1908.02286}
}

read the original abstract

In recent years, summarizers that incorporate domain knowledge into the process of text summarization have outperformed generic methods, especially for summarization of biomedical texts. However, construction and maintenance of domain knowledge bases are resource-intense tasks requiring significant manual annotation. In this paper, we demonstrate that contextualized representations extracted from the pre-trained deep language model BERT, can be effectively used to measure the similarity between sentences and to quantify the informative content. The results show that our BERT-based summarizer can improve the performance of biomedical summarization. Although the summarizer does not use any sources of domain knowledge, it can capture the context of sentences more accurately than the comparison methods. The source code and data are available at https://github.com/BioTextSumm/BERT-based-Summ.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

[1]

Recent automatic text summarization techniques: a survey,

M. Gambhir and V. Gupta, "Recent automatic text summarization techniques: a survey," Artificial Intelligence Review, vol. 47, pp. 1 - 66, 2016. K Cosine Similarity Euclidean distance R-1 R-2 R-1 R-2 2 0.7441 0.3329 0.7548 0.3373 3 0.7479 0.3361 0.7576 0.3397 4 0.7501 0.3394 0.7607 0.3459 5 0.7472 0.3359 0.7568 0.3402 6 0.7425 0.3317 0.7532 0.3365 7 0.7380...

work page 2016
[2]

CIBS: A biomedical text summarizer using topic -based sentence clustering,

M. Moradi, "CIBS: A biomedical text summarizer using topic -based sentence clustering," Journal of Biomedical Informatics, vol. 88, pp. 53 -61, 2018/12/01/ 2018

work page 2018
[3]

Different approaches for identifying important concepts in probabilistic biomedical text summarization,

M. Moradi and N. Ghadiri, "Different approaches for identifying important concepts in probabilistic biomedical text summarization," Artificial Intelligence in Medicine, vol. 84, pp. 101-116, 2018

work page 2018
[4]

Text summarization in the biomedical domain: a systematic review of recent research,

R. Mishra, J. Bian, M. Fiszman, C. R. Weir, S. Jonn alagadda, J. Mostafa , et al. , "Text summarization in the biomedical domain: a systematic review of recent research," Journal of biomedical informatics, vol. 52, pp. 457-467, 2014

work page 2014
[5]

A semantic graph-based approach to bi omedical summarisation,

L. Plaza, A. Díaz, and P. Gervás, "A semantic graph-based approach to bi omedical summarisation," Artificial intelligence in medicine, vol. 53, pp. 1-14, 2011

work page 2011
[6]

Quantifying the informativeness for biomedical literature summarization: An itemset mining method,

M. Moradi and N. Ghadiri, "Quantifying the informativeness for biomedical literature summarization: An itemset mining method," Computer Methods and Programs in Biomedicine, vol. 146, pp. 77-89, 2017

work page 2017
[7]

Frequent Itemsets as Meaningful Events in Graphs for Summarizing Biomedical Texts,

M. Moradi, "Frequent Itemsets as Meaningful Events in Graphs for Summarizing Biomedical Texts," in 2018 8th International Conference on Computer and Knowledge Engineering (ICCKE), 2018, pp. 135-140

work page 2018
[8]

Co ncept-based single - and multi-document biomedical text summarization,

M. Moradi, "Co ncept-based single - and multi-document biomedical text summarization," Isfahan University of Technology, 2017

work page 2017

Show all 16 references

[9]

Application of text mining in the biomedical domain,

W. W. Fleuren and W. Alkema, "Application of text mining in the biomedical domain," Methods, vol. 74, pp. 97-106, 2015

2015
[10]

Word representations: a simple and general method for semi -supervised learning,

J. Turian, L. Ratinov, and Y. Bengio, "Word representations: a simple and general method for semi -supervised learning," presented at the Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, 2010

2010
[11]

Semi -supervised sequence tagging with bidirectional language models,

M. E. Peters, W. Ammar, C. Bhagavatula, and R. Power, "Semi -supervised sequence tagging with bidirectional language models," arXiv preprint arXiv:1705.00108, 2017

2017 arXiv
[12]

Improving language understanding by generative pre-training,

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Improving language understanding by generative pre-training," URL https://s3 -us-west-2. amazonaws. com/openai-assets/research- covers/languageunsupervised/language understanding paper. pdf, 2018

2018
[13]

Deep contextualized word re presentations,

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee , et al. , "Deep contextualized word re presentations," arXiv preprint arXiv:1802.05365, 2018

2018 arXiv
[14]

Bert: Pre -training of deep bidirectional transformers for language understanding,

J. Devlin, M. -W. Chang, K. Lee, and K. Toutanova, "Bert: Pre -training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[15]

Looking for a few good metrics: Automatic summarization evaluation -how many samples are enough?,

C.-Y. Lin, "Looking for a few good metrics: Automatic summarization evaluation -how many samples are enough?," in NTCIR, 2004

2004
[16]

SUMMA: A robust and adaptable summarization tool,

H. Saggion, "SUMMA: A robust and adaptable summarization tool," Traitement Automatique des Langues, vol. 49, 2008

2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Recent automatic text summarization techniques: a survey,

M. Gambhir and V. Gupta, "Recent automatic text summarization techniques: a survey," Artificial Intelligence Review, vol. 47, pp. 1 - 66, 2016. K Cosine Similarity Euclidean distance R-1 R-2 R-1 R-2 2 0.7441 0.3329 0.7548 0.3373 3 0.7479 0.3361 0.7576 0.3397 4 0.7501 0.3394 0.7607 0.3459 5 0.7472 0.3359 0.7568 0.3402 6 0.7425 0.3317 0.7532 0.3365 7 0.7380...

work page 2016

[2] [2]

CIBS: A biomedical text summarizer using topic -based sentence clustering,

M. Moradi, "CIBS: A biomedical text summarizer using topic -based sentence clustering," Journal of Biomedical Informatics, vol. 88, pp. 53 -61, 2018/12/01/ 2018

work page 2018

[3] [3]

Different approaches for identifying important concepts in probabilistic biomedical text summarization,

M. Moradi and N. Ghadiri, "Different approaches for identifying important concepts in probabilistic biomedical text summarization," Artificial Intelligence in Medicine, vol. 84, pp. 101-116, 2018

work page 2018

[4] [4]

Text summarization in the biomedical domain: a systematic review of recent research,

R. Mishra, J. Bian, M. Fiszman, C. R. Weir, S. Jonn alagadda, J. Mostafa , et al. , "Text summarization in the biomedical domain: a systematic review of recent research," Journal of biomedical informatics, vol. 52, pp. 457-467, 2014

work page 2014

[5] [5]

A semantic graph-based approach to bi omedical summarisation,

L. Plaza, A. Díaz, and P. Gervás, "A semantic graph-based approach to bi omedical summarisation," Artificial intelligence in medicine, vol. 53, pp. 1-14, 2011

work page 2011

[6] [6]

Quantifying the informativeness for biomedical literature summarization: An itemset mining method,

M. Moradi and N. Ghadiri, "Quantifying the informativeness for biomedical literature summarization: An itemset mining method," Computer Methods and Programs in Biomedicine, vol. 146, pp. 77-89, 2017

work page 2017

[7] [7]

Frequent Itemsets as Meaningful Events in Graphs for Summarizing Biomedical Texts,

M. Moradi, "Frequent Itemsets as Meaningful Events in Graphs for Summarizing Biomedical Texts," in 2018 8th International Conference on Computer and Knowledge Engineering (ICCKE), 2018, pp. 135-140

work page 2018

[8] [8]

Co ncept-based single - and multi-document biomedical text summarization,

M. Moradi, "Co ncept-based single - and multi-document biomedical text summarization," Isfahan University of Technology, 2017

work page 2017