{"id":"145af517-4b70-4891-8f16-c44fec0c488f","arxiv_id":"2501.07718","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new Norwegian summarization benchmark, NorSumm, contains 63 news articles each with three human-authored summaries in Bokmål and Nynorsk, and open LLMs perform poorly on it.","lead":"This paper presents a new dataset of Norwegian news articles with human-written summaries in both Bokmål and Nynorsk, designed to test how well AI summarizers handle Norwegian. The authors show that today's open language models score low on this benchmark and that human readers prefer the human-written summaries in nearly all comparisons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low automatic scores may reflect reference diversity rather than task difficulty; no human-human ROUGE-L/BERTScore baseline is reported to calibrate Table 4.","rationale":"The reader's weakest assumption concerned sample size and stability of rankings. That is a valid concern about the benchmark's reliability, but it does not directly test whether the dataset is 'challenging'—the central claim in the abstract. The more load-bearing gap is the absence of a human-human baseline for the automatic metrics. The paper explicitly infers challenge from low ROUGE-L/BERTScore values, yet with three deliberately diverse, bullet-format references and max-over-reference aggregation, low scores could simply reflect the metric's inability to credit paraphrase and content selection. The human evaluation provides some support, but it is small and unbalanced, so it cannot independently validate the numbers in Table 4. The proposed test is cheap, uses only existing data, and would settle whether the automatic scores are meaningful. If the test shows human-human scores are similar to model scores, the paper's primary quantitative argument for 'challenging' fails, and the claim would need to be re-based on the human evaluation with appropriate caveats. If the test shows a clear human-model gap, the concern is resolved and the dataset's value is confirmed. No change to the reader's CONDITIONAL verdict is needed, but the condition should include reporting this calibration analysis.","tokens_in":12064,"tokens_out":10033,"duration_ms":96252,"concrete_test":"Compute, for each of the 63 documents, the leave-one-out human-human ROUGE-L and BERTScore: for each gold summary, take its maximum score against the other two gold summaries (mirroring the model's max-over-references), then average over documents and separately over BM and NN. Compare to Table 4. Also report pairwise human-human scores without the max as a measure of reference dispersion. If the human oracle scores are statistically indistinguishable from the best model scores (e.g., within 5 ROUGE-L points), the 'challenging' claim is not supported by the automatic metrics; if human-human scores are substantially higher, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main quantitative evidence for the central claim is the 'relatively low performance scores' in Section 5, but these scores are never calibrated against a human upper bound. Section 4 aggregates by taking, for each document, the maximum score over the three gold references and over 12 prompts. The three gold summaries are intentionally diverse (Section 3), so pairwise overlap between human summaries may also be low; if so, low model ROUGE-L/BERTScore would be an artifact of reference diversity and metric sensitivity, not of task difficulty. The paper reports no inter-annotator agreement or leave-one-out human-human score. The human evaluation (138 vs 8) supports the challenge claim, but it is small, not balanced by model, and does not calibrate the automatic metrics used in the headline table. Thus the central claim currently rests on an unmeasured assumption: that low scores mean a hard task, rather than a hard metric.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NorSumm, a new benchmark dataset of human-authored summaries for Norwegian news articles. The dataset is built from 63 news documents from the EDEN corpus. Three journalism-trained native speakers each wrote one summary per document in their preferred written variety (Bokmål or Nynorsk), and the summaries were translated into the other variety by a second annotator, yielding six summaries per document. The authors evaluate nine open LLMs in a zero-shot setting using 12 prompts and report ROUGE-L and BERTScore, finding that Viking-13B achieves the highest scores, and they conduct a human preference study in which human-authored summaries are preferred in 138 of 146 comparisons. The paper argues that the low automatic scores show that the dataset is a challenging benchmark.","tokens_in":12149,"tokens_out":4938,"duration_ms":44932,"significance":"If the calibration issue is addressed, this is a valuable resource: it is the first freely available manually curated Norwegian summarization dataset, created with professional annotators, covering both official written varieties, and with intentionally diverse gold summaries that reflect the non-uniqueness of summaries. The inclusion of open Norwegian LLMs and a public evaluation framework supports reproducibility. The paper's main substantive claim—that the benchmark challenges current open models—is plausible but currently under-supported because the automatic scores are not calibrated against a human-human baseline and the small evaluation sample lacks uncertainty estimates.","major_comments":[{"comment":"The central claim that the dataset is challenging rests on the low absolute scores in Table 4, but the paper never calibrates these scores against a human upper bound. Because the three gold summaries are intentionally diverse and the protocol takes the maximum over references and prompts, a human summary would also be expected to score well below 100 on ROUGE-L and BERTScore. I request a leave-one-out human-human baseline: for each document and each human summary, compute the max score against the other two human references (or the mean), and report this alongside Table 4. Without this, the 'relatively low performance scores' can be read as an artifact of reference diversity and metric sensitivity rather than task difficulty.","section":"Section 5, Table 4; Section 4"},{"comment":"The aggregation rule is ambiguous: the text says (i) for each prompt compute the maximum over the three references, and (ii) 'maximize the BERTScore across all prompts and average the resulting ROUGE-L and BERTScore values'. It is unclear whether ROUGE-L is taken from the prompt that maximizes BERTScore or maximized separately; this can change the reported numbers. Please specify the exact algorithm and release the per-document scores. Because Table 4 is based on 63 documents with no variance or significance testing, the small differences between models (e.g., Viking-7B vs Viking-13B) cannot be distinguished from noise.","section":"Section 4, Performance Metrics"},{"comment":"The human evaluation is reported as 138 vs 8 preferences, but the paper does not state how many evaluators participated, how many judgments each evaluator made, how items were sampled across models and prompts, or whether the 146 responses are independent. Without these details and a statistical test, the preference result is difficult to interpret. Please report the number of unique evaluators, the number of comparisons per model, and at least a sign test or confidence interval.","section":"Section 6"},{"comment":"Prompt 6 in the Bokmål block is written in Nynorsk ('Heile artikkelen', 'Hovudpunkt') and prompt 6 in the Nynorsk block is written in Bokmål ('Hele artikkelen', 'Hovedpunkter'). If the actual prompts used in the evaluation match the table, this is a language mismatch that should be corrected; if the table is inaccurate, the discrepancy undermines reproducibility. Please either fix the prompts or clarify the actual wording.","section":"Table 3"}],"minor_comments":[{"comment":"The phrase 'both of the written variants' should be 'both written variants'.","section":"Abstract"},{"comment":"'EDEN was chosen due to its high-quality' should be 'due to its high quality'.","section":"Section 3, Data sources"},{"comment":"'We provide more details about the resulting dataset bellow' should be 'below'.","section":"Section 3, Generation and evaluation"},{"comment":"The total average '102,17' uses a decimal comma while the text uses decimal points; please harmonize the decimal separator.","section":"Table 2"},{"comment":"The statement that 'all LLMs achieve acceptable performance' is in tension with the later characterization of 'relatively low performance scores'; please choose a consistent interpretation.","section":"Section 5"},{"comment":"'led us to finding' should be 'led us to find'.","section":"Section 6"},{"comment":"The text says 'Outliers are observed in NN-A2, NN-A2, and NB-A1'; the duplicate 'NN-A2' should be corrected.","section":"Section 3, Dataset statistics"},{"comment":"The source article spells the minister's name 'Brekk' while the summaries use 'Brekke'; please clarify whether this is a typo in the original article or an intentional correction in the summaries.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid resource paper with a careful annotation design, and I think the dataset itself is a useful contribution. The main risk is overclaiming from uncalibrated automatic metrics. If the authors add a human-human calibration baseline and clarify the evaluation protocol, the paper will be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution — a manually authored Norwegian summarization benchmark with three gold summaries per article in both Bokmål and Nynorsk, written by journalism-trained native speakers. That genuinely fills a gap; prior Norwegian summarization resources were machine-translated. The dataset is small (63 EDEN dev/test articles), but the annotation process is careful and transparent, with detailed guidelines and dual annotation via translation. Credit for shipping it openly.\n\nThe evaluation section is the soft spot. Table 4 shows open LLMs scoring low (ROUGE-L 9–34), and the paper reads this as evidence that the benchmark is challenging. That inference is under-supported because there is no human-human upper bound. The three gold summaries are intentionally diverse, and the paper reports no inter-annotator agreement or leave-one-out human ROUGE/BERTScore. Low automatic scores could partly reflect reference diversity and metric sensitivity rather than task difficulty. The human evaluation (138/146 preferring human summaries) does support the challenging-benchmark claim, but it is small, not balanced by model, and does not calibrate the automatic metrics. So the central claim leans on an unmeasured assumption.\n\nAlso minor: the aggregation takes the max over 12 prompts, which can inflate scores, and there are no confidence intervals or significance tests for a 63-document sample. Model rankings could shift. The BM vs NN comparison is interesting but also underpowered. And the paper calls the scores 'acceptable' in one sentence and 'relatively low' in the next — the framing is a bit inconsistent.\n\nNone of this invalidates the dataset. For Norwegian and low-resource summarization benchmarking, this is a useful instrument, and the authors are appropriately candid about the small scale and the difficulty of the annotation task. The paper deserves a serious referee; I would recommend conditional acceptance with a request for a human-human calibration (even a small leave-one-out analysis) and variance reporting, plus a softer claim in the abstract if the calibration shows high human diversity.","headline":"New, genuinely useful Norwegian summarization benchmark, but the 'challenging' claim needs a human-human calibration before it is robust.","tokens_in":12727,"tokens_out":1968,"would_cite":true,"duration_ms":19280,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a human-authored Norwegian news summarisation dataset and argues, based on automatic and human evaluation, that it is a challenging LLM benchmark.","keywords":["abstractive summarisation","Norwegian","Bokmål","Nynorsk","LLM benchmark","human evaluation","ROUGE-L","BERTScore"],"falsifier":"Resample the 63 articles with replacement and compute bootstrap confidence intervals for each model's mean ROUGE-L; if the intervals for the top-scoring and lowest-scoring models overlap, the claim that the benchmark cleanly separates Norwegian summarisation capabilities is not supported.","tokens_in":11820,"feed_emoji":"🗞️","tokens_out":11105,"duration_ms":98832,"temperature":0.7,"pith_summary":"This paper introduces a dataset of 63 Norwegian news articles, each paired with three human-written gold summaries in Bokmål and three in Nynorsk, for a total of six reference summaries per article. The authors position it as the first freely available, manually curated Norwegian news summarisation dataset created from scratch, rather than translated from English. They evaluate nine open-weight language models zero-shot with ROUGE-L and BERTScore, aggregating over the three references and twelve prompts. Scores separate the models, with the best reaching 30.57 ROUGE-L overall and the weakest at roughly 9–10, and a human preference study selects the human-written summary in 138 of 146 pairwise comparisons. The paper's conclusion is that the dataset provides a challenging LLM benchmark for Norwegian summarisation.","feed_headline":"Human-written summaries beat LLMs 138 to 8 in Norwegian news","feed_subtitle":"The first human-written Norwegian news summarisation benchmark gives LLMs clear headroom in both Bokmål and Nynorsk.","key_machinery":"The carrying object is the multi-reference annotation protocol in both Norwegian written standards. Three journalists with news-writing backgrounds each authored one summary per article in their preferred standard (two in Bokmål, one in Nynorsk), and a second round of translation by other annotators produced the counterpart standard, so every reference was seen by two native speakers. For evaluation, the paper aggregates with a maximum over the three gold references per prompt and then maximizes BERTScore across twelve prompts before averaging over documents, a procedure chosen to reduce prompt sensitivity and to accommodate legitimate diversity across human summaries.","core_discovery":"The paper claims that a small corpus of human-authored summaries is sufficient to expose current LLMs' weaknesses in Norwegian summarisation. In the automatic evaluation, no open model approaches the human references: the strongest model reaches 33.76 ROUGE-L and 70.90 BERTScore on Bokmål and 30.57 ROUGE-L and 70.4 BERTScore overall, while the weakest models fall to about 9–10 ROUGE-L. Most models score higher in Bokmål than in Nynorsk, with ROUGE-L differences up to around ten points. In the human evaluation, readers chose the human-authored summary in 138 of 146 comparisons; the observed model failures include copying the beginning of the source, inventing quotes, confusing entities, and producing repetitive text. Together these results support the paper's abstract, which states that the dataset is a challenging LLM benchmark for Norwegian summarisation capabilities.","pith_inferences":["Inference: with only 63 source documents, resampling or a larger extension is needed before the reported ranking, including the roughly 20-point ROUGE-L gap between best and worst models, is treated as stable.","Inference: because all source articles are Bokmål originals and the Nynorsk summaries are translations, Nynorsk scores probably measure a joint translation-summarisation ability rather than pure Nynorsk summarisation.","Inference: the max-over-references scoring means the published numbers are optimistic upper bounds relative to a single-reference evaluation, which should be kept in mind when comparing models across benchmarks with different reference counts.","Inference: the documented failure classes suggest the same dataset could be reused as a faithfulness and factuality diagnostic for Norwegian generation beyond summarisation."],"forward_implications":["Norwegian summarisation research gets a native, human-authored reference standard in both Bokmål and Nynorsk, so models no longer need to be evaluated against machine-translated foreign news.","The uniformly low scores of the nine open LLMs indicate large headroom for improving Norwegian summarisation systems.","The three-reference design and prompt-max aggregation make the benchmark less dependent on a single summary style, so a model is rewarded for matching any of the three human authors.","The human evaluation documents concrete failure modes—source copying, invented quotes, entity confusion, and repetition—that can guide targeted model improvements.","Future comparisons of additional models against this benchmark will be able to separate model families by capability more cleanly than the existing sparse Norwegian resources allowed."],"supporting_citations":[{"why":"Supplies the 30 dev and 33 test news articles from EDEN that make up the benchmark documents.","marker":"(Touileb et al., 2024)"},{"why":"Defines ROUGE-L, the primary metric used for automatic evaluation.","marker":"(Lin, 2004)"},{"why":"Defines BERTScore, the second automatic metric reported alongside ROUGE-L.","marker":"(Zhang et al., 2020)"},{"why":"Provides the relevance, consistency, and fluency criteria used in the human preference evaluation.","marker":"(Fabbri et al., 2021)"},{"why":"Introduces the Norwegian LLM baselines (NorBLOOM and NorMistral) evaluated in the zero-shot experiment.","marker":"(Samuel et al., 2025)"},{"why":"Source of the Mistral-7B baseline included in the model comparison.","marker":"(Jiang et al., 2023)"},{"why":"Source of the falcon-7b baseline included in the model comparison.","marker":"(Almazrouei et al., 2023)"}],"fun_headline_variants":["Humans beat LLMs 138–8 on Norwegian summarisation","Open LLMs can't match humans on Norwegian summaries (138–8)","Norwegian summarisation: humans outscore LLMs 138 to 8","New Norwegian summarisation benchmark: LLMs trail humans 138–8","Bokmål, Nynorsk: human-written summaries beat LLMs 138–8"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 63 news articles taken from the EDEN dev and test splits are a representative and large enough sample of Norwegian news for the reported model scores and ranking to be meaningful.","fun_headline_variants_meta":{"raw":{"variants":["Humans beat LLMs 138–8 on Norwegian summarisation","Open LLMs can't match humans on Norwegian summaries (138–8)","Norwegian summarisation: humans outscore LLMs 138 to 8","New Norwegian summarisation benchmark: LLMs trail humans 138–8","Bokmål, Nynorsk: human-written summaries beat LLMs 138–8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000905,"raw_usage":{"total_tokens":3846,"prompt_tokens":850,"completion_tokens":2996,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2897}},"tokens_in":466,"tokens_out":2996,"duration_ms":22082,"temperature":1.0,"reasoning_tokens":2897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:35:49.820526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Resample the 63 articles with replacement and compute bootstrap confidence intervals for each model's mean ROUGE-L; if the intervals for the top-scoring and lowest-scoring models overlap, the claim that the benchmark cleanly separates Norwegian summarisation capabilities is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 30 dev and 33 test news articles from EDEN that make up the benchmark documents."},{"cited_title":"Weinberger, and Yoav Artzi","cited_arxiv_id":null,"evidence_quote":"Defines BERTScore, the second automatic metric reported alongside ROUGE-L."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Norwegian LLM baselines (NorBLOOM and NorMistral) evaluated in the zero-shot experiment."}],"review_version":1}