REVIEW 4 major objections 6 minor 33 references
The 2021 Tokyo Olympics Multilingual News Article Dataset
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper introduces OG2021, a manually annotated multilingual news dataset of 10,940 articles covering 1,350 sub-events of the 2021 Tokyo Olympics, and argues it fills a gap in benchmarks for multilingual news clustering in…
desk verdict A genuinely new multilingual news clustering benchmark with a documented pipeline, but the missing inter-annotator agreement means the gold-standard labels are not yet demonstrated to be reliable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a three-stage annotation pipeline. Retrieval first filters articles by language, publication window, and required Wikipedia concepts (Olympic Games, Japan, plus one of seven specified sports) through a news-aggregation service. Automatic clustering then runs an online single-pass algorithm that represents each article by a multilingual sentence-embedding vector, a set of extracted named entities, and its publication datetime, and assigns it to a cluster only if content similarity, entity overlap, and temporal proximity all exceed tuned thresholds. Manual annotation completes the pipeline: human annotators review every automatic cluster, following criteria based on who, what, where, when, and optionally how an event unfolded, and are allowed to merge, split, relocate, or delete articles. The cluster IDs produced by this combined process are what makes the dataset a benchmark rather than just a collection of articles.
What would settle it
Re-annotate a random sample of the 1,350 clusters with independent annotators using the paper's stated criteria, then compute pairwise agreement between the original cluster IDs and the re-annotation, for example with BCubed F1 or the adjusted Rand index. Low agreement would show that the dataset's labels do not provide a stable benchmark for clustering.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that a dense, multilingual, real-time news stream can be captured and manually curated into a usable benchmark. The dataset contains 10,940 articles in English, Portuguese, Spanish, French, Russian, German, Slovenian, Arabic, and Chinese, published between July 1 and August 14, 2021, sourced from 1,918 publishers. The articles are grouped into 1,350 clusters corresponding to Olympic sub-events, with the largest cluster, the opening ceremony, containing 499 articles and about 28% of clusters containing articles in at least two languages. The paper reports that the cluster labels were produced by automatic online clustering followed by a full manual review in which annotators merged, split, relocated, or removed clusters according to journalistic who/what/where/when/how criteria, so the final cluster IDs represent the intended ground truth for evaluating clustering algorithms.
Load-bearing premise
The cluster labels are trustworthy: the manual annotators, using machine translation when needed, consistently applied the who/what/where/when/how criteria so that articles about the same sub-event receive the same cluster ID, but the paper reports no inter-annotator agreement or error analysis to back that consistency.
Editorial extensions
If this is right
- A clustering algorithm can now be scored against 1,350 event clusters in a setting where many sub-events occur simultaneously and in similar locations, which is a harder test than most existing news benchmarks provide.
- Because about 28% of clusters contain at least two languages, the dataset supports evaluating cross-lingual event clustering rather than only monolingual grouping.
- The per-language cluster distributions and language co-occurrence statistics give a concrete profile of how reporting on the same event differs across languages and scripts.
- The same retrieval-and-annotation pipeline could be rerun on future Olympics or other time-dense global events to produce comparable benchmarks and track how event coverage evolves.
- Researchers can use the dataset not only for clustering evaluation but also for studying the dynamics of the 2021 Olympics, including media perspectives and the challenges of COVID-era competition.
Reading between the lines
- The paper leaves implicit that its annotation criteria define an operational notion of 'event' centered on who/what/where/when with 'how' optional; that protocol could be extracted and reused for building event datasets on other topics.
- A natural testable extension is to measure inter-annotator agreement on a sample of clusters; the paper reports no such agreement, so its absence is the main open question for label reliability.
- The public version of the dataset omits article bodies, so users who cannot access the research version would evaluate on titles, URLs, and metadata alone; this split itself could be studied as a low-resource clustering scenario.
- Because the pipeline depends on a multilingual sentence-embedding model, benchmark scores will likely be sensitive to that model's language coverage; comparing alternative embedding models on this dataset would reveal which language pairs drive clustering errors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces OG2021, a multilingual news article dataset covering the 2021 Tokyo Olympics, containing 10,940 articles from 1,918 publishers in nine languages, grouped into 1,350 event clusters. The authors describe a three-stage pipeline: retrieval of articles via the Event Registry API using criteria on language, publication date, and Wikipedia concepts; automatic online clustering of articles into event clusters; and manual annotation and evaluation where annotators review, merge, split, relocate, and delete clusters according to criteria based on the journalistic questions who, what, where, when, and how. The paper reports descriptive statistics and technical validation of the dataset, and releases public and research versions through CLARIN.SI. The intended contribution is a benchmark for evaluating online multilingual news clustering algorithms in a high-frequency event setting.
Significance. If the cluster labels are reliable, OG2021 fills a genuine gap in multilingual news event clustering benchmarks, especially for temporally dense and simultaneous sub-events. The paper's strengths include transparent retrieval criteria, public release on a recognized repository, and basic technical validation such as language distributions, cluster-size statistics, and language co-occurrence. The main limitation is that the benchmark value rests entirely on the manual cluster labels, and the paper provides no evidence of their reliability beyond a description of the annotation procedure. Addressing this omission would make the dataset a valuable community resource.
major comments (4)
- [Manual annotation and evaluation] The paper does not report the number of annotators, their language coverage, how disagreements were resolved, or any inter-annotator agreement statistic. Since the manual cluster labels are the core benchmark contribution, the absence of such evidence means the central claim that articles sharing a cluster ID report on the same sub-event is unverified. I recommend adding an inter-annotator agreement study, even on a subsample, and a detailed description of the annotation team and disagreement resolution process.
- [News cleanup] The text states that articles are split into seven datasets, one per sports concept, but does not specify how articles that mention multiple of the sports in the retrieval criteria are assigned. If such articles appear in more than one sports dataset, they may be processed multiple times and could appear multiple times in the final CSV, which would affect the reported article count, cluster count, and cluster-size statistics. The authors should clarify the assignment rule and confirm that the final dataset contains unique article IDs with no duplicates.
- [Manual annotation and evaluation] Because annotators started from clusters produced by an automatic clustering algorithm, the final cluster IDs may inherit the algorithm's biases. The paper mentions that annotators could merge, split, relocate, and eliminate clusters, but it reports no statistics on how often these actions were taken. Without information on, for example, the number of clusters merged or split, or the number of articles relocated, it is impossible to judge how much the manual stage changed the automatic output, and thus how independent the final labels are from the automatic initialization. I request that these statistics be reported, or that the authors provide other evidence of the manual stage's substantial contribution.
- [Automatic news clustering] The clustering thresholds for content similarity, entity overlap, and temporal proximity are said to be chosen to optimize precision while maintaining recall, but the actual threshold values are not reported. Since these thresholds determine the initial clusters that annotators review, their absence impairs reproducibility of the pipeline. I recommend reporting the exact thresholds or providing a configuration file in the released code.
minor comments (6)
- [Manual annotation and evaluation] Typo: 'the annotators are tasks to group articles' should be 'the annotators are tasked to group articles'.
- [Table 1] The entry for Miranda et al. lists the time range as 'Dec 2013 - Avg 2015'; 'Avg' should likely be 'Aug'.
- [Background & Summary] The phrase 'it's news article metadata' should be 'its news article metadata'.
- [Figure 5] The language co-occurrence matrix is visually dense; consider providing the values in a table or with larger font to improve readability.
- [Data Records] The description of the BODY variable states it is 'truncated' but then says the public version does not include it at all; this is confusing and should be clarified as 'available only in the research version'.
- [Manual annotation criteria] The phrase 'The intentionally ambiguous criterion' is unclear; it likely means 'flexible' or 'subjective' rather than 'ambiguous'.
Circularity Check
No significant circularity: the dataset labels are the product of manual annotation, not a fitted prediction or self-citation reduction.
full rationale
The paper's central claim is the creation of OG2021, a manually annotated multilingual news dataset. The derivation chain is a data-construction pipeline rather than a predictive argument. Automatic clustering (citing the first author's algorithm [25]) is used only to produce an initial cluster structure, and the paper explicitly states that annotators could relocate articles, merge, split, and eliminate clusters: 'Annotators had the flexibility to remove specific articles from the dataset, relocate articles between clusters, merge, divide, and eliminate entire clusters.' The final cluster IDs are therefore not equivalent to the automatic clustering output by construction; no fitted parameter is renamed as a prediction, and the paper does not evaluate the authors' own clustering algorithm against the dataset. The self-citations to Event Registry, wikification, and the clustering algorithm are tool citations, not load-bearing justifications of the dataset's validity, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The absence of inter-annotator agreement statistics is a reliability limitation, not evidence of circularity; it concerns the quality of the human ground truth rather than a reduction of the claimed result to its own inputs.
Assumptions & free parameters
free parameters (1)
- Automatic clustering thresholds (content similarity, entity overlap, temporal proximity) =
not reported
assumptions (3)
- domain assumption Event Registry retrieval with the required Wikipedia concepts reliably returns articles about the 2021 Olympics and the selected sports.
- domain assumption The manual annotation criteria based on who, what, where, when, and how produce consistent event groupings.
- domain assumption The automatic clustering algorithm's content embeddings, named entities, and publication dates provide useful initial clusters that can be corrected by human review.
Cite this review
Pith. "Pith review of The 2021 Tokyo Olympics Multilingual News Article Dataset." pith.science (2026). https://pith.science/paper/YSLLM2R4
@misc{pith2026250206648,
author = {Pith},
title = {Pith review of: The 2021 Tokyo Olympics Multilingual News Article Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/YSLLM2R4}},
note = {Machine review of arXiv:2502.06648}
}
read the original abstract
In this paper, we introduce a dataset of multilingual news articles covering the 2021 Tokyo Olympics. A total of 10,940 news articles were gathered from 1,918 different publishers, covering 1,350 sub-events of the 2021 Olympics, and published between July 1, 2021, and August 14, 2021. These articles are written in nine languages from different language families and in different scripts. To create the dataset, the raw news articles were first retrieved via a service that collects and analyzes news articles. Then, the articles were grouped using an online clustering algorithm, with each group containing articles reporting on the same sub-event. Finally, the groups were manually annotated and evaluated. The development of this dataset aims to provide a resource for evaluating the performance of multilingual news clustering algorithms, for which limited datasets are available. It can also be used to analyze the dynamics and events of the 2021 Tokyo Olympics from different perspectives. The dataset is available in CSV format and can be accessed from the CLARIN.SI repository.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Newsweeder: Learning to filter netnews
Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pages 331–339. Morgan Kaufmann, 1995
work page 1995
-
[2]
Reuters-21578 text categorization collection, 1997
David Lewis. Reuters-21578 text categorization collection, 1997
work page 1997
-
[3]
Antonio Gulli. AG news. http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html,
-
[4]
Practical solutions to the problem of diagonal dominance in kernel document clustering
Derek Greene and Pádraig Cunningham. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd International Conference on Machine learning (ICML), pages 377–384. ACM Press, 2006
work page 2006
-
[5]
RCV1: A new benchmark collection for text categoriza- tion research
David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. RCV1: A new benchmark collection for text categoriza- tion research. Journal of Machine Learning Research, 5:361–397, 2004
work page 2004
-
[6]
TDT2 multilanguage text version 4.0, 2001
Charles Wayne, George R Doddington, Jonathan G Fiscus, Mark Liberman, Jennifer Alabiso, David Graff, and Christopher Cieri. TDT2 multilanguage text version 4.0, 2001. 10 The 2021 Tokyo Olympics Multilingual News Article Dataset
work page 2001
-
[7]
NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles
Jeppe Nørregaard, Benjamin D Horne, and Sibel Adalı. NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 630–638. Association for the Advancement of Artificial Intelligence (AAAI), 2019
work page 2018
-
[8]
Zero-shot learning for cross- lingual news sentiment classification
Andraž Pelicon, Marko Pranji´c, Dragana Miljkovi´c, Blaž Škrlj, and Senja Pollak. Zero-shot learning for cross- lingual news sentiment classification. Applied sciences (Basel, Switzerland), 10:5993, 2020
work page 2020
Show all 33 references
-
[9]
NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles
Maurício Gruppi, Benjamin D Horne, and Sibel Adalı. NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. Preprint at https://arxiv.org/abs/2003.08444, 2020
2019 arXiv
-
[10]
The POLUSA dataset: 0.9M political news articles balanced by time and outlet popularity
Lukas Gebhard and Felix Hamborg. The POLUSA dataset: 0.9M political news articles balanced by time and outlet popularity. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 467–468. ACM, 2020
2020
-
[11]
Sculpting Data for ML: The first act of Machine Learning
Rishabh Misra and Jigyasa Grover. Sculpting Data for ML: The first act of Machine Learning. Independently Published, 2021
2021
-
[12]
News category dataset
Rishabh Misra. News category dataset. Preprint at https://arxiv.org/abs/2209.11429, 2022
2022 arXiv
-
[13]
Efficient discrimination between closely related languages
Jörg Tiedemann and Nikola Ljubeši´c. Efficient discrimination between closely related languages. In Proceedings of COLING 2012, pages 2619–2634, 2012
2012
-
[14]
Creating a persian-english comparable corpus
Homa Baradaran Hashemi, Azadeh Shakery, and Heshaam Faili. Creating a persian-english comparable corpus. In Multilingual and Multimodal Information Access Evaluation, Lecture notes in computer science, pages 27–39. Springer Berlin Heidelberg, 2010
2010
-
[15]
A multilingual simplified language news corpus
Renate Hauser, Jannis Vamvas, Sarah Ebling, and Martin V olk. A multilingual simplified language news corpus. In Proceedings of the 2nd Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI) within the 13th Language Resources and Evaluation Confere...
2022
-
[16]
MassiveSumm: a very large-scale, very multilingual, news summarisation dataset
Daniel Varab and Natalie Schluter. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150–10161. Association for Computational Linguistics, 2021
2021
-
[17]
CC- news-en: A large english news corpus
Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, and Alistair Moffat. CC- news-en: A large english news corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM, 2020
2020
-
[18]
A diverse multilingual news headlines dataset from around the world
Felix Leeb and Bernhard Schölkopf. A diverse multilingual news headlines dataset from around the world. Preprint at https://arxiv.org/abs/2403.19352, 2024
2024 arXiv
-
[19]
Event registry: learning about world events from news
Gregor Leban, Blaz Fortuna, Janez Brank, and Marko Grobelnik. Event registry: learning about world events from news. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 2014
2014
-
[20]
Gdelt: Global data on events, location, and tone, 1979–2012
K Leetaru and P A Schrodt. Gdelt: Global data on events, location, and tone, 1979–2012. ISA Annual Convention, 2013
1979
-
[21]
Multilingual clustering of streaming news
Sebastião Miranda, Art¯urs Znotin, š, Shay B Cohen, and Guntis Barzdins. Multilingual clustering of streaming news. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4535–4544. Association for Computational Linguistics, 2018
2018
-
[22]
News across languages - cross-lingual document similarity and event tracking
Jan Rupnik, Andrej Muhic, Gregor Leban, Primoz Skraba, Blaz Fortuna, and Marko Grobelnik. News across languages - cross-lingual document similarity and event tracking. Journal of Artificial Intelligence Research, 55:283–316, 2016
2016
-
[23]
Annotating documents with relevant wikipedia concepts
Janez Brank, Gregor Leban, and Marko Grobelnik. Annotating documents with relevant wikipedia concepts. In Proceedings of Slovenian KDD Conference on Data Mining and Data Warehouses (SiKDD), 2017
2017
-
[24]
Semantic annotation of documents based on wikipedia concepts
Janez Brank, Gregor Leban, and Marko Grobelnik. Semantic annotation of documents based on wikipedia concepts. Informatica, 42, 2018
2018
-
[25]
News stream clustering using multilingual language models
Erik Novak. News stream clustering using multilingual language models. In The Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD), 2021
2021
-
[26]
Sentence-BERT: Sentence embeddings using siamese BERT-networks
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)...
2019
-
[27]
Augmented SBERT: Data augmenta- tion method for improving bi-encoders for pairwise sentence scoring tasks
Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmenta- tion method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computat...
2021
-
[28]
WikiNEu- Ral: Combined neural and knowledge-based silver data creation for multilingual NER
Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Cecconi, and Roberto Navigli. WikiNEu- Ral: Combined neural and knowledge-based silver data creation for multilingual NER. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2521–25...
2021
-
[29]
The news articles reporting on the 2021 tokyo olympics data set OG2021 (public), 2024
Erik Novak, Erik Calcina, Dunja Mladeni´c, and Marko Grobelnik. The news articles reporting on the 2021 tokyo olympics data set OG2021 (public), 2024. Slovenian language resource repository CLARIN.SI
2021
-
[30]
The news articles reporting on the 2021 tokyo olympics data set OG2021 (research), 2024
Erik Novak, Erik Calcina, Dunja Mladeni´c, and Marko Grobelnik. The news articles reporting on the 2021 tokyo olympics data set OG2021 (research), 2024. Slovenian language resource repository CLARIN.SI
2021
-
[31]
A comparison of extrinsic clustering evaluation metrics based on formal constraints
Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval, 12:461–486, 2009
2009
-
[32]
The code for creating the OG2021 dataset, 2024
Erik Novak, Matevž Matjašec, and Erik Calcina. The code for creating the OG2021 dataset, 2024. 12
2024
-
[2004]
Accessed: 2024-08-05
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.