REVIEW 3 major objections 4 minor 7 references
DpgMedia2019: A Dutch News Dataset for Partisanship Detection
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper presents DpgMedia2019, a Dutch news dataset with more than 100,000 publisher-level partisanship labels and 766 article-level labels, built to support automated partisan-news detection in Dutch.
desk verdict A genuine and honest dataset release — first Dutch partisan news corpus — but alpha 0.18 and Spearman 0.21 mean the labels are weakly validated and the claims need scoping. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-layer labeling pipeline. The first layer uses an audience-based partisanship score: each publisher's score is the average of its subscribers' self-reported positions on a five-point left-right scale, and a threshold turns those scores into binary partisan or non-partisan publisher labels that propagate to all articles from that publisher. The second layer is a subscriber survey in which readers rate each article's bias intensity, polarity, and target; a reliability score based on free-text answers, filters for too-few annotations, and majority voting convert the raw ratings into article-level partisan labels. The paper then compares the two layers with a Spearman correlation, reporting 0.21, as a check on the outer labeling assumption.
What would settle it
Take the 766 article-level labeled articles and compare each majority label with its publisher-level partisan/non-partisan label; if agreement is no better than always predicting non-partisan (the 74% base rate), the publisher-level layer does not carry the signal the dataset claims.
Extended reading notes
Core claim
The central claim is that DpgMedia2019 fills that gap. It contains 103,812 unique articles from 11 Dutch publishers, split almost evenly into 52,873 publisher-level partisan labels and 50,939 non-partisan labels, plus 766 crowd-annotated articles of which 201 are partisan. Publisher partisanship was derived by averaging the self-reported political leaning of readers per publisher; de Volkskrant, Trouw, and Het Parool were treated as partisan and the remaining eight as non-partisan. The article-level labels came from an internal subscriber survey, filtered by a reliability score, with majority voting across at least three annotations per article. The authors present the dataset as a foundation for a partisan-news detector, to be used, for example, by training on one label layer and testing on the other.
Load-bearing premise
The load-bearing premise is that a publisher's subscriber-averaged political leaning tracks the partisanship of its individual articles closely enough to label more than 100K articles with more signal than noise; the paper's own check of this premise is a Spearman correlation of 0.21.
Editorial extensions
If this is right
- A Dutch partisan-news detector can be trained on the 100K publisher-level labels and evaluated on the 766 article-level labels, giving an in-language benchmark.
- The publisher-level portion can serve as unlabeled data in semi-supervised learning while the article-level portion provides supervision.
- Because the raw survey responses are released, alternative label-aggregation schemes can be tested against the reported majority-vote labels.
- The reported low correlation between label layers means any model trained on publisher labels should be validated on article labels before being trusted.
Reading between the lines
- The near-chance gap in the publisher-article comparison suggests the publisher-level label is better treated as a weak prior than as a gold label; article-level evaluation may be necessary for credible detector claims.
- With only 26% of article-level examples partisan, accuracy is a misleading metric; precision, recall, and calibration by annotator self-reported leaning would be more informative.
- The regional publishers show a spread in partisan percentages, so the dataset could support studies of regional variation in Dutch news partisanship rather than only national titles.
- The released raw ratings could be re-analyzed with models that treat each annotator's political leaning as a parameter, which might yield more consistent labels than the current majority vote.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents DpgMedia2019, a Dutch news dataset with two components: a publisher-level part of 103,812 articles from 11 DPG Media publishers, labeled partisan/non-partisan on the basis of the average self-reported political leaning of subscribers (Table 2), and an article-level part of 766 (elsewhere 776) articles labeled by subscribers through an internal survey that asked about perceived bias intensity, polarity, and pro/anti entities (Section 2.2, Appendix). The paper documents the collection pipeline, post-hoc filtering steps intended to improve annotator reliability, the final Krippendorff's alpha of 0.18, and a Spearman correlation of 0.21 between publisher-level partisanship scores and per-publisher article-level partisan percentages (Section 3). The authors also discuss limitations and suggest applications such as training a partisan news detector, and they state that raw survey data are released.
Significance. If the labels were valid, this would be the first Dutch-language benchmark for partisan news detection, and the combination of one hundred thousand publisher-level articles with a smaller article-level set would be a useful resource. The paper is transparent about its process and releases raw survey responses, which is a genuine strength: future work can re-derive labels with better methods. However, the advertised contribution is a dataset with partisanship labels, and the paper's own validation evidence is weak: the publisher-level label paradigm is supported only by a statistically insignificant Spearman correlation over 11 publishers, and the article-level labels rest on an inter-rater agreement of alpha = 0.18 after multiple post-hoc filtering steps. The resource is valuable as a collection with raw annotations, but the central claim of a reliable partisanship-labeled dataset is not yet established.
major comments (3)
- [Section 3, Tables 2 and 7] The publisher-level labeling paradigm is validated only by a Spearman correlation of 0.21 computed over 11 publishers between the absolute audience-based partisanship scores and the per-publisher percentage of article-level partisan labels. With n = 11, this correlation is not statistically significant, and the raw numbers in Tables 2 and 7 show why: the publisher scores lie in a narrow band from -0.64 to 0.07, and the per-publisher partisan rates are nearly flat (VK 27.1% versus AD 24.1%; regional publishers from 0% to 41.2%). This does not support the assumption that partisan publishers publish more partisan articles in this dataset, so the 103,812 publisher-level labels may encode publisher identity rather than article content. The authors should either supply independent validation (for example, a random sample of publisher-labeled articles judged by experts, or a transfer experiment showing article-level signal) or explicitly reframe the large component as an article collection with publisher-leaning scores rather than a partisanship-labeled benchmark.
- [Section 2.2.2, Table 1] The article-level labels rest on very low inter-rater agreement: Krippendorff's alpha rises only from 0.142 to 0.180 after four filtering steps, including discarding 'uninterested' and 'unreliable' annotators, discarding articles with fewer than three annotations, majority voting, and removing articles where half or more annotations were 'impossible to decide'. The reliability score in equation (1) and the threshold on it are chosen post hoc based on the alpha they produce, so the final agreement estimate is not an independent measure of label quality. With alpha = 0.18, majority-voted labels from three or more untested annotators are close to chance-level agreement, and the paper provides no confidence interval, no per-class agreement, and no validation against expert labels. The authors should provide such evidence, or clearly restrict the claim of article-level partisanship labeling to the raw annotations rather than to the derived 766 labels.
- [Section 2.1.1, Table 2] The binary partition of publishers into partisan and non-partisan is not given a principled criterion. The computed scores are all slightly left-leaning (from -0.64 to 0.07), and the cutoff that places Het Parool (-0.434) into the partisan set and de Gelderlander (-0.245) into the non-partisan set is introduced as a decision without stating a rule or a pre-registered threshold. Because this cutoff determines the labels for all 103,812 publisher-level articles, its arbitrariness is load-bearing for the dataset's central claim. The paper should either present the continuous publisher scores as the primary publisher-level signal, with the binary labels clearly marked as one possible discretization, or justify the cutoff with an external criterion.
minor comments (4)
- [Abstract, Table 1, Section 2.2.2] The number of article-level labeled articles is given as 776 in the abstract but 766 in Table 1 and Section 2.2.2; this inconsistency should be corrected.
- [Throughout] There are several typographical errors: 'Krippendorf' should be 'Krippendorff', 'Spearsman' should be 'Spearman', 'dateset' in the Table 1 caption should be 'dataset', 'drabantsdagblad' in Table 2 should be 'brabantsdagblad', and publisher names are not consistently capitalized (for example 'destem' vs 'de Stem').
- [Table 8] The article length statistics do not state the unit of measurement; the authors should specify whether the values are word counts or character counts, and they should note that the publisher-level and article-level distributions have different standard deviations (387.5 vs 275.1) even though the means are close.
- [Section 2.2 and Appendix] The survey question Q1 asks about 'bevooroordeeld' (biased) rather than 'partisan' specifically, while the dataset is described throughout as a partisanship dataset; the paper should clarify the intended relationship between perceived bias and partisanship, since the two concepts are not identical.
Circularity Check
No circularity found: the dataset labels are constructed and documented rather than derived from a fitted prediction, and the weak 0.21 correlation is an acknowledged validity check rather than an output forced by construction.
full rationale
The paper makes no derived prediction; it constructs and documents a dataset. Publisher-level partisanship scores are computed from annotators' self-reported political leanings averaged per subscribed publisher, while article-level labels are content annotations by readers. These quantities are not defined in terms of one another. Section 3 explicitly tests the labeling assumption by computing a Spearman correlation of 0.21 between absolute publisher partisanship and per-publisher percentages of article-level partisan labels, and the paper acknowledges this low correlation and its causes. No step reduces a stated result to its own input by construction. The only metric that is used to select a filter and then reported afterward is Krippendorff's alpha in the annotator filtering step, but the paper reports both the initial and final alpha values and does not present the final alpha as an independent validation of the labels; it is transparently part of the dataset construction process. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The possible overlap in the subscriber pool supplying both publisher scores and article annotations is a validity concern, not circularity, especially since the paper investigates annotator bias and reports a low cross-measure correlation.
Assumptions & free parameters
free parameters (5)
- Partisan publisher cutoff =
three left-leaning publishers: VK, Trouw, Parool
- Likert binarization threshold =
ratings 1-2 non-partisan, 3-5 partisan
- Minimum annotations per article =
3
- Impossible-to-decide removal threshold =
>=50% impossible-to-decide annotations
- Reliability score threshold =
unknown (selected to maximize Krippendorff's alpha)
assumptions (4)
- domain assumption Partisan publishers publish more partisan articles and non-partisan publishers more non-partisan articles.
- domain assumption Average self-reported political leaning of a publisher's subscribers reflects the publisher's partisanship.
- domain assumption Free-text comments about bias indicate annotator reliability.
- domain assumption Subscribers who pay for news are competent and have read the articles.
Cite this review
Pith. "Pith review of DpgMedia2019: A Dutch News Dataset for Partisanship Detection." pith.science (2026). https://pith.science/paper/IULQS3PB
@misc{pith2026190802322,
author = {Pith},
title = {Pith review of: DpgMedia2019: A Dutch News Dataset for Partisanship Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/IULQS3PB}},
note = {Machine review of arXiv:1908.02322}
}
read the original abstract
We present a new Dutch news dataset with labeled partisanship. The dataset contains more than 100K articles that are labeled on the publisher level and 776 articles that were crowdsourced using an internal survey platform and labeled on the article level. In this paper, we document our original motivation, the collection and annotation process, limitations, and applications.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Pew Research Center. 2018 a . https://www.pewresearch.org/global/fact-sheet/news-media-and-political-attitudes-in-netherlands/ News media and political attitudes in the netherlands
work page 2018
-
[4]
Pew Research Center. 2018 b . https://www.pewresearch.org/global/2018/01/11/publics-globally-want-unbiased-news-coverage-but-are-divided-on-whether-their-news-media-deliver/#maps Public globally want unbiased news coverage
work page 2018
-
[5]
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval-2019 Task 4: Hyperpartisan News Detection . In Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval 2019) . Association for Computational Linguistics
work page 2019
-
[6]
Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018. http://aclweb.org/anthology/D18-1388 Multi-view models for political ideology detection of news articles . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3518--3527. Association for Computational Linguistics
work page 2018
-
[7]
Martin Potthast, Johannes Kiesel, Kevin Reinartz, Ja\-nek Bevendorff, and Benno Stein. 2018. https://aclanthology.info/papers/P18-1022/p18-1022 A Stylometric Inquiry into Hyperpartisan and Fake News . In 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018) , pages 231--240. Association for Computational Linguistics
work page 2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.