Pith. sign in

REVIEW 4 major objections 8 minor 26 references

Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles

T0 review · 4 major / 8 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a human-authored Norwegian news summarisation dataset and argues, based on automatic and human evaluation, that it is a challenging LLM benchmark.

desk verdict New, genuinely useful Norwegian summarization benchmark, but the 'challenging' claim needs a human-human calibration before it is robust. read the letter →

arxiv 2501.07718 v1 pith:CJXDJRE2 submitted 2025-01-13 cs.CL

Samia Touileb , Vladislav Mikhailov , Marie Kroka , Lilja {O}vrelid , Erik Velldal This is my paper · ORCID
classification cs.CL
keywords abstractivesummarisationNorwegianBokmålNynorskLLMbenchmarkhumanevaluationROUGE-LBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a dataset of 63 Norwegian news articles, each paired with three human-written gold summaries in Bokmål and three in Nynorsk, for a total of six reference summaries per article. The authors position it as the first freely available, manually curated Norwegian news summarisation dataset created from scratch, rather than translated from English. They evaluate nine open-weight language models zero-shot with ROUGE-L and BERTScore, aggregating over the three references and twelve prompts. Scores separate the models, with the best reaching 30.57 ROUGE-L overall and the weakest at roughly 9–10, and a human preference study selects the human-written summary in 138 of 146 pairwise comparisons. The paper's conclusion is that the dataset provides a challenging LLM benchmark for Norwegian summarisation.

What carries the argument

The carrying object is the multi-reference annotation protocol in both Norwegian written standards. Three journalists with news-writing backgrounds each authored one summary per article in their preferred standard (two in Bokmål, one in Nynorsk), and a second round of translation by other annotators produced the counterpart standard, so every reference was seen by two native speakers. For evaluation, the paper aggregates with a maximum over the three gold references per prompt and then maximizes BERTScore across twelve prompts before averaging over documents, a procedure chosen to reduce prompt sensitivity and to accommodate legitimate diversity across human summaries.

What would settle it

Resample the 63 articles with replacement and compute bootstrap confidence intervals for each model's mean ROUGE-L; if the intervals for the top-scoring and lowest-scoring models overlap, the claim that the benchmark cleanly separates Norwegian summarisation capabilities is not supported.

Watch

Extended reading notes

Core claim

The paper claims that a small corpus of human-authored summaries is sufficient to expose current LLMs' weaknesses in Norwegian summarisation. In the automatic evaluation, no open model approaches the human references: the strongest model reaches 33.76 ROUGE-L and 70.90 BERTScore on Bokmål and 30.57 ROUGE-L and 70.4 BERTScore overall, while the weakest models fall to about 9–10 ROUGE-L. Most models score higher in Bokmål than in Nynorsk, with ROUGE-L differences up to around ten points. In the human evaluation, readers chose the human-authored summary in 138 of 146 comparisons; the observed model failures include copying the beginning of the source, inventing quotes, confusing entities, and producing repetitive text. Together these results support the paper's abstract, which states that the dataset is a challenging LLM benchmark for Norwegian summarisation capabilities.

Load-bearing premise

The load-bearing assumption is that the 63 news articles taken from the EDEN dev and test splits are a representative and large enough sample of Norwegian news for the reported model scores and ranking to be meaningful.

Editorial extensions

If this is right

  • Norwegian summarisation research gets a native, human-authored reference standard in both Bokmål and Nynorsk, so models no longer need to be evaluated against machine-translated foreign news.
  • The uniformly low scores of the nine open LLMs indicate large headroom for improving Norwegian summarisation systems.
  • The three-reference design and prompt-max aggregation make the benchmark less dependent on a single summary style, so a model is rewarded for matching any of the three human authors.
  • The human evaluation documents concrete failure modes—source copying, invented quotes, entity confusion, and repetition—that can guide targeted model improvements.
  • Future comparisons of additional models against this benchmark will be able to separate model families by capability more cleanly than the existing sparse Norwegian resources allowed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: with only 63 source documents, resampling or a larger extension is needed before the reported ranking, including the roughly 20-point ROUGE-L gap between best and worst models, is treated as stable.
  • Inference: because all source articles are Bokmål originals and the Nynorsk summaries are translations, Nynorsk scores probably measure a joint translation-summarisation ability rather than pure Nynorsk summarisation.
  • Inference: the max-over-references scoring means the published numbers are optimistic upper bounds relative to a single-reference evaluation, which should be kept in mind when comparing models across benchmarks with different reference counts.
  • Inference: the documented failure classes suggest the same dataset could be reused as a faithfulness and factuality diagnostic for Norwegian generation beyond summarisation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper introduces NorSumm, a new benchmark dataset of human-authored summaries for Norwegian news articles. The dataset is built from 63 news documents from the EDEN corpus. Three journalism-trained native speakers each wrote one summary per document in their preferred written variety (Bokmål or Nynorsk), and the summaries were translated into the other variety by a second annotator, yielding six summaries per document. The authors evaluate nine open LLMs in a zero-shot setting using 12 prompts and report ROUGE-L and BERTScore, finding that Viking-13B achieves the highest scores, and they conduct a human preference study in which human-authored summaries are preferred in 138 of 146 comparisons. The paper argues that the low automatic scores show that the dataset is a challenging benchmark.

Significance. If the calibration issue is addressed, this is a valuable resource: it is the first freely available manually curated Norwegian summarization dataset, created with professional annotators, covering both official written varieties, and with intentionally diverse gold summaries that reflect the non-uniqueness of summaries. The inclusion of open Norwegian LLMs and a public evaluation framework supports reproducibility. The paper's main substantive claim—that the benchmark challenges current open models—is plausible but currently under-supported because the automatic scores are not calibrated against a human-human baseline and the small evaluation sample lacks uncertainty estimates.

major comments (4)
  1. [Section 5, Table 4; Section 4] The central claim that the dataset is challenging rests on the low absolute scores in Table 4, but the paper never calibrates these scores against a human upper bound. Because the three gold summaries are intentionally diverse and the protocol takes the maximum over references and prompts, a human summary would also be expected to score well below 100 on ROUGE-L and BERTScore. I request a leave-one-out human-human baseline: for each document and each human summary, compute the max score against the other two human references (or the mean), and report this alongside Table 4. Without this, the 'relatively low performance scores' can be read as an artifact of reference diversity and metric sensitivity rather than task difficulty.
  2. [Section 4, Performance Metrics] The aggregation rule is ambiguous: the text says (i) for each prompt compute the maximum over the three references, and (ii) 'maximize the BERTScore across all prompts and average the resulting ROUGE-L and BERTScore values'. It is unclear whether ROUGE-L is taken from the prompt that maximizes BERTScore or maximized separately; this can change the reported numbers. Please specify the exact algorithm and release the per-document scores. Because Table 4 is based on 63 documents with no variance or significance testing, the small differences between models (e.g., Viking-7B vs Viking-13B) cannot be distinguished from noise.
  3. [Section 6] The human evaluation is reported as 138 vs 8 preferences, but the paper does not state how many evaluators participated, how many judgments each evaluator made, how items were sampled across models and prompts, or whether the 146 responses are independent. Without these details and a statistical test, the preference result is difficult to interpret. Please report the number of unique evaluators, the number of comparisons per model, and at least a sign test or confidence interval.
  4. [Table 3] Prompt 6 in the Bokmål block is written in Nynorsk ('Heile artikkelen', 'Hovudpunkt') and prompt 6 in the Nynorsk block is written in Bokmål ('Hele artikkelen', 'Hovedpunkter'). If the actual prompts used in the evaluation match the table, this is a language mismatch that should be corrected; if the table is inaccurate, the discrepancy undermines reproducibility. Please either fix the prompts or clarify the actual wording.
minor comments (8)
  1. [Abstract] The phrase 'both of the written variants' should be 'both written variants'.
  2. [Section 3, Data sources] 'EDEN was chosen due to its high-quality' should be 'due to its high quality'.
  3. [Section 3, Generation and evaluation] 'We provide more details about the resulting dataset bellow' should be 'below'.
  4. [Table 2] The total average '102,17' uses a decimal comma while the text uses decimal points; please harmonize the decimal separator.
  5. [Section 5] The statement that 'all LLMs achieve acceptable performance' is in tension with the later characterization of 'relatively low performance scores'; please choose a consistent interpretation.
  6. [Section 6] 'led us to finding' should be 'led us to find'.
  7. [Section 3, Dataset statistics] The text says 'Outliers are observed in NN-A2, NN-A2, and NB-A1'; the duplicate 'NN-A2' should be corrected.
  8. [Table 1] The source article spells the minister's name 'Brekk' while the summaries use 'Brekke'; please clarify whether this is a typo in the original article or an intentional correction in the summaries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: benchmark scores are measured against independently authored human summaries, and the main claim does not reduce to any fitted input or self-citation.

full rationale

The paper's central claim, that the dataset is a challenging LLM benchmark for Norwegian summarisation, is supported by measured ROUGE-L and BERTScore results on nine open models (Table 4) and by a separate human preference experiment (Section 6). The gold summaries are human-authored, not generated or fitted by any model, and no parameter is fitted to the evaluation data and then renamed as a prediction. Several cited resources come from overlapping author groups, including EDEN (Touileb et al., 2024), the NORA models (Samuel et al., 2025), and the noreval framework, but these citations are resource credits rather than load-bearing arguments: the models have open weights, the metrics are standard external measures, and the results include fully external baselines such as Mistral-7B and Falcon-7B. The absence of a human-human automatic-score calibration is a legitimate evaluation-robustness concern, but it is not a circularity: low scores would be a weak or incomplete measure of challenge, not a restatement of the dataset's definition. No equation in the paper reduces to its own inputs, and no uniqueness theorem or prior result by the same authors is invoked to force the main conclusion. The derivation chain is therefore self-contained with respect to the circularity patterns considered.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about data representativeness, annotation quality, and metric validity rather than on fitted parameters. We list the design choice of a 700-character summary limit as a hand-set constraint and the domain assumptions about EDEN and evaluation metrics.

free parameters (1)
  • Maximum summary length (700 characters)
    Annotator guideline chosen by the authors to constrain summary length; it shapes all summaries and thus the benchmark, but is not fitted to data.
assumptions (4)
  • domain assumption EDEN dev/test splits are a suitable source of high-quality Norwegian news articles
    Section 3, Data sources: the dataset is built entirely from 63 EDEN articles, so its representativeness depends on EDEN's quality.
  • domain assumption ROUGE-L and BERTScore are valid automatic metrics for summary quality
    Section 4: used as performance metrics without validation for Norwegian; standard practice but an assumption.
  • domain assumption Human evaluator preferences reflect summary quality better than automatic metrics
    Section 6: manual evaluation with three criteria and priority ordering is taken as ground truth.
  • domain assumption Translation between Bokmål and Nynorsk preserves meaning and naturalness
    Section 3, Generation and evaluation: two annotators translated all summaries, with no external quality check on translations beyond annotator self-report.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles." pith.science (2026). https://pith.science/paper/CJXDJRE2

@misc{pith2026250107718,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CJXDJRE2}},
  note         = {Machine review of arXiv:2501.07718}
}
read the original abstract

We introduce a dataset of high-quality human-authored summaries of news articles in Norwegian. The dataset is intended for benchmarking the abstractive summarisation capabilities of generative language models. Each document in the dataset is provided with three different candidate gold-standard summaries written by native Norwegian speakers, and all summaries are provided in both of the written variants of Norwegian -- Bokm{\aa}l and Nynorsk. The paper describes details on the data creation effort as well as an evaluation of existing open LLMs for Norwegian on the dataset. We also provide insights from a manual human evaluation, comparing human-authored to model-generated summaries. Our results indicate that the dataset provides a challenging LLM benchmark for Norwegian summarisation capabilities

Figures

Figures reproduced from arXiv: 2501.07718 by the authors.

Figure 1
Figure 1. Box plots of summary lengths authored by three different annotators (referred to as A1, A2, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Screenshot of the interface used during human evaluation. We present a news article on top, and two suggestions for summaries. The goal for the evaluator is to choose the summary they prefer based on simple criteria (see §6). els we have evaluated here. However, we chose to rely exclusively on open-source models with Norwegian language support to ensure accessibil￾ity and reproducibility for future research. 6 Human… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 9 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B : an open large language model with state-of-the-art performance

    2023

  4. [4]

    Pierre Colombo, Maxime Peyrard, Nathan Noiry, Robert West, and Pablo Piantanida. 2023. https://doi.org/10.18653/v1/2023.findings-ijcnlp.16 The glass ceiling of automatic evaluation in natural language generation . In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 178--183, Nusa Dua, Bali. Association for Comp...

    doi:10.18653/v1/2023.findings-ijcnlp.16 2023

  5. [5]

    Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2022. A survey of natural language generation. ACM Computing Surveys, 55(8):1--38

    work page 2022

  6. [6]

    Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text summarization: A comprehensive survey. Expert systems with applications, 165:113679

    2021

  7. [7]

    Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. https://doi.org/10.18653/v1/P19-1102 Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074--1084, Florence, Italy. Association for C...

    doi:10.18653/v1/p19-1102 2019

  8. [8]

    Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 S umm E val: Re-evaluating summarization evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

    doi:10.1162/tacl_a_00373 2021

Show all 26 references
  1. [9]

    Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2023. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. Journal of Artificial Intelligence Research, 77:103--166

    2023

  2. [10]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28

    2015

  3. [11]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  4. [12]

    Fredrik J rgensen, Tobias Aasmoe, Anne-Stine Ruud Husev g, Lilja vrelid, and Erik Velldal. 2019. Norne: Annotating named entities for norwegian. arXiv preprint arXiv:1911.12146

    2019 arXiv

  5. [13]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  6. [14]

    Rafael Dueire Lins, Hilario Oliveira, Luciano Cabral, Jamilson Batista, Bruno Tenorio, Rafael Ferreira, Rinaldo Lima, Gabriel de Fran c a Pereira e Silva, and Steven J Simske. 2019. The cnn-corpus: A large textual corpus for single-document extractive summarization. In Proceed...

    2019

  7. [15]

    Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang

    Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang. 2024. http://arxiv.org/abs/2312.01314 Nlebench+norglm: A comprehensive empirical analysis and benchmark dataset for generative language models in norwegian

    2024 arXiv

  8. [16]

    Sheng Lu, Hendrik Schuff, and Iryna Gurevych. 2024. https://doi.org/10.18653/v1/2024.naacl-long.325 How are prompts different in terms of sensitivity? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

    2024 doi

  9. [17]

    Petter M hlum, Dag Trygve Truslew Haug, Tollef Emil J rgensen, Andre K sen, Anders N klestad, Egil R nningstad, Per Erik Solberg, Erik Velldal, and Lilja vrelid. 2022. Narc--norwegian anaphora resolution corpus. In International Conference on Computational Linguistics (ICCL)(C...

    2022

  10. [18]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natu...

    2018 doi

  11. [19]

    Lilja vrelid and Petter Hohle. 2016. Universal dependencies for norwegian. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1579--1585

    2016

  12. [20]

    Rush, Sumit Chopra, and Jason Weston

    Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. https://doi.org/10.18653/v1/D15-1044 A neural attention model for abstractive sentence summarization . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379--389, Lisbon, Po...

    2015 doi

  13. [21]

    David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja vrelid, Lucas Georges Gabriel Charpentier, and Andrey Kutuzov. 2025. Small Languages, Big Models: A Study of Continual Training on Languages of Norway

    2025

  14. [22]

    Per Erik Solberg, Arne Skj rholt, Lilja vrelid, Kristin Hagen, and Janne Bondi Johannessen. 2014. The norwegian dependency treebank

    2014

  15. [23]

    Samia Touileb, Jeanett Murstad, Petter M hlum, Lubos Steskal, Lilja Charlotte Storset, Huiling You, and Lilja vrelid. 2024. https://aclanthology.org/2024.lrec-main.488 EDEN : A dataset for event detection in N orwegian news . In Proceedings of the 2024 Joint International Conf...

    2024

  16. [24]

    Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind Your Format: Towards Consistent Evaluation of In-context Learning Improvements . arXiv preprint arXiv:2401.06766

    2024 arXiv

  17. [25]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  18. [26]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr BERTScore: Evaluating Text Generation with BERT . In International Conference on Learning Representations (ICLR)

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.