Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Characterizing Linguistic Shifts in Croatian News via Diachronic Word Embeddings

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Croatian news language drifted positive across 25 years

desk verdict Useful resource and solid descriptive shifts, but the sentiment-positivity claim is too brittle to carry the paper. read the letter →

arxiv 2506.13569 v1 pith:RRHA5TOT submitted 2025-06-16 cs.CL

David Dukić , Ana Barić , Marko Čuljak , Josip Jukić , Martin Tutek This is my paper · ORCID
classification cs.CL
keywords diachronicwordembeddingssemanticchangeCroatiannewscorpussentimentdriftskip-gramProcrustesalignmentCOVID-19terminologyEUaccessionvocabulary
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether word meanings in Croatian news shifted measurably over the past 25 years and whether those shifts change how sentiment classifiers behave. By training skip-gram embeddings on 9.5 million articles split into five-year periods, it finds that words tied to COVID-19, EU accession, and technology visibly change their nearest neighbors over just a few years. Its headline quantitative result is that embeddings from later periods encode a more positive stance: transferring aligned embeddings from 2020–2024 into earlier periods moves predicted sentiment upward, even though mental-health studies report worsening mood over the same years. If the finding holds, it means distributional semantics of news became systematically more positive, which would bias any sentiment model trained on one period and applied to another.

What carries the argument

Two devices carry the argument. First, per-period skip-gram (SGNS) embeddings projected into a shared space by recursive Procrustes alignment, with the cumulative shift score $D_c = \sum_{i=1}^4 \frac{1 - \cos(v_i, v_{i+1})}{2}$ for each word; this measures how much a word's neighborhood changes across neighboring periods. Second, a cross-period sentiment transfer quantity: for a classifier $C_i$ trained on period $i$ embeddings, $d_{i\leftarrow j} = \bar{s}_{i\leftarrow j} - \bar{s}_{i\leftarrow i}$, the change in average predicted sentiment when aligned embeddings from period $j$ are substituted for those from $i$. The sign and size of $d_{i\leftarrow j}$ across the $5\times 5$ grid of periods is the evidence for sentiment drift, cross-checked against a Croatian SentiLex lexicon and BERTić-based label-ratio analysis.

What would settle it

Recompute $d_{i\leftarrow j}$ after normalizing all embeddings to unit norm and using a held-out test set per period; if the positivity increase disappears or flips sign, the claim fails. The paper's own Figure 2 already shows the quantity is not antisymmetric (on STONE, $d_{1\leftarrow 5} = -0.09$ while $-d_{5\leftarrow 1} = 0.41$), so a checker can test whether the apparent drift is a property of the target period alone.

Watch

Extended reading notes

Core claim

The paper's central claim is that short-span diachronic word embeddings trained on a 3.7-billion-word Croatian newswire corpus capture real linguistic shifts, and that one of those shifts is an increase in encoded positivity. Topic-wise, nouns like maska 'mask', varijanta 'variant', fond 'fund', and inteligencija 'intelligence' acquire new dominant neighbors at identifiable moments, such as 2020 for the pandemic and 2017 for AI. Sentiment-wise, the transfer quantity $d_{i\leftarrow j} = \bar{s}_{i\leftarrow j} - \bar{s}_{i\leftarrow i}$ is positive when $j$ is later than $i$: classifiers fed aligned embeddings from a later period rate texts more positively, and this is corroborated by an increased share of positive labels in both headline and article data. The paper presents this increase in news positivity as an empirical contrast to studies reporting declining mental health, and leaves the explanation open, suggesting reactions to negativity, diversification of content, or polarization as candidate drivers.

Load-bearing premise

The sentiment-drift result assumes that the numbers measuring sentiment change really reflect a change in how positive the news language became, rather than an artifact of aligning the vector spaces or of the classifier reacting differently to newer word patterns.

Editorial extensions

If this is right

  • Sentiment classifiers trained on recent Croatian news will over-estimate positivity when applied to older articles, and vice versa; longitudinal NLP analysis on Croatian data needs period-aware embeddings.
  • Topic detection from neighbor shifts can pinpoint when a public event changed a word's meaning at five-year resolution, without hand-labeled historical data.
  • The positive drift in news embeddings, if real, means media discourse and mental-health indicators can move in opposite directions, so news positivity should not be read as population mood.
  • The method transfers to other lower-resourced languages that have a news archive of comparable size, because it does not require century-scale corpora.
  • The released embeddings of the Croatian corpus give other researchers a resource for historical text processing and further diachronic studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The asymmetry of $d_{i\leftarrow j}$ in Figure 2 suggests the metric conflates classifier sensitivity to distribution shift with genuine positivity; a symmetric, normalization-invariant measure would be needed to confirm the direction.
  • One testable extension is to apply the same transfer procedure to an artificially controlled corpus where sentiment is kept constant; if $d$ still trends positive, the artifact is in the alignment and classifier pipeline, not in the language.
  • The authors' proposed explanations (reaction to negativity, diversification, polarization) are mutually distinguishable in data: polarization predicts rising variance and extreme labels, diversification predicts more neutral-but-positive topics, and a reaction predicts correlation with mental-health indices.
  • The five-year window is coarse for events like COVID; the paper reports two-year windows failed, but a period-size ablation could identify the minimal corpus size at which shifts become reliable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper trains SGNS word embeddings on the TakeLab Retriever Croatian news corpus, split into five five-year periods (2000–2024), and uses Procrustes alignment to study semantic change. For topic analysis, it computes a cumulative cosine-distance shift score for manually curated words related to COVID-19, Croatia's EU accession, and technology, and reports qualitative neighbor changes. For sentiment, it proposes a transfer quantity d_{i<-j} = s_{i<-j} - s_{i<-i}, measuring how much a logistic-regression sentiment classifier trained on period i changes its predictions when given aligned embeddings from period j. The paper claims that embeddings from post-2020 are more positive, in contrast to studies reporting declining mental health, and that short-span Croatian news embeddings detect topical linguistic shifts.

Significance. If established, the paper would be a useful contribution to diachronic semantics for a language with scarce historical corpora, and the public release of the embeddings and code would benefit follow-up work. The topic-analysis framework extends earlier work by Hamilton et al. to a short, modern news corpus, and the sentiment-transfer experiment asks a practically relevant question about downstream classifier behavior. However, the sentiment claim currently rests on a transfer metric that is confounded by cross-period differences in embedding quality and vector norms, and the topical-shift analysis lacks quantitative baselines. Both issues are addressable with additional controls, so the contribution is potentially publishable after revision.

major comments (3)
  1. [Section 3.4, Figure 2] The quantity d_{i<-j} = s_{i<-j} - s_{i<-i} is not shown to isolate sentiment drift. SGNS vectors are not norm-normalized, Procrustes alignment is orthogonal, and a logistic regression on averaged vectors is sensitive to feature scale; since periods differ sharply in corpus size and embedding quality (Table 1: 53M vs 1.75B words; Table 2: contrastive spread 0.08 vs 0.27), a classifier trained on period i can change predictions because of norm or quality differences rather than semantic content. The asymmetry in Figure 2 (e.g., d_{1<-5} = -0.09 vs d_{5<-1} = -0.41 on STONE) further indicates that the metric is not a coherent symmetric property of the two periods. Please report norm statistics per period, add a norm-normalization control, analyze a common-vocabulary subset, and either justify the directional interpretation or use a symmetrized measure.
  2. [Section 4.3, Figure 3] The BERTić label-distribution analysis is not an independent control for the sentiment-shift claim. A single classifier trained on the same label sources (STONE and 24sata) is applied across all periods, so the observed increase in predicted positive labels could reflect the classifier's sensitivity to the distributional differences documented in Table 2 rather than a genuine sentiment shift. The paper does not show that BERTić's decision boundary is stable across periods, nor does it calibrate the per-period predictions. Please add per-period calibration or a control that does not reuse the same trained classifier, and temper the claim that Figure 3 corroborates the sentiment-shift result.
  3. [Section 4.2, Table 3] The topical-shift analysis is presented as evidence that the embeddings capture major topics, but the supporting evidence is qualitative: top-shift words are selected by cumulative D_c, yet no baseline or significance test is provided. Without a control group of non-topic words, a permutation test over D_c, or an evaluation of the curated topic lists against a null distribution, the observed shifts could be driven by corpus-size or embedding-quality effects rather than by the specified topics. Please add such a quantitative baseline and report effect sizes relative to it.
minor comments (5)
  1. [Section 3.3] The formula for D_c should be written unambiguously as D_c = sum_{i=1}^{4} (1 - cos(v_i, v_{i+1})) / 2, and the notation for cosine distance should be defined.
  2. [Table 2] The dagger annotation reports p < 0.001 but no statistical test is described; please specify the test procedure and the number of samples used.
  3. [Section 4.3] The sentence about observing a similar trend using the SentiLex lexicon is not backed by any displayed result; either report the analysis or remove the claim.
  4. [Limitations] The limitations section acknowledges that varying article counts 'could bias the results,' but this concern is not applied to the sentiment-transfer metric; please add an explicit limitation about norm comparability and classifier sensitivity.
  5. [Section 3.1] There is a typo in 'split the corpus into5five-year periods'; it should read 'into five five-year periods.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements whose conclusions do not reduce to their inputs by construction.

full rationale

The paper's central claims are empirical rather than derived from an input assumption that already contains them. The sentiment-shift quantity d_i<-j = s_i<-j - s_i<-i is an operational definition of a measurement: a classifier trained on period i embeddings is applied to Procrustes-aligned embeddings from period j, and the difference in average predicted sentiment is reported. No parameter is fitted to the target claim and then renamed as a prediction. The corpus from Dukic et al. (2024) is a self-cited resource, but it is an independently available dataset, and its use is resource attribution rather than load-bearing argumentation. The alignment and embedding-training methodology follow Hamilton et al. (2016b), an external prior method, and the additional BERTic-based label-distribution check in Figure 3 is a second, separate model applied to the same data, not a restatement of the word-embedding classifier's outputs. Concerns about embedding-norm comparability, corpus-size asymmetry, and the non-antisymmetry of d_i<-j are validity or robustness risks that could undermine the interpretation of the measurement, but they do not make the derivation circular: the result is not true by definition, and it could in principle have gone the other way. The limitations section explicitly acknowledges that uneven period sizes may bias results, which further confirms that the authors treat the finding as contingent rather than tautological. Overall, no step in the paper reduces to a self-definition, a fitted input called a prediction, or a load-bearing self-citation chain.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on standard distributional semantics assumptions plus several manual thresholds. No new entities are introduced. The sentiment claim additionally assumes that cross-period classifier transfer is a valid measure of sentiment drift, which is the most fragile assumption.

free parameters (4)
  • Period length = 5 years
    Choice of time window; affects detected shift granularity. Authors report two-year periods were too fine-grained (Limitations).
  • SGNS hyperparameters = vector_size=300, window=4, negative=5, sample=1e-5, epochs=5
    Copied from Hamilton et al. (2016b); influence embedding quality and shift magnitudes (Appendix C).
  • Minimum word frequency for shift analysis = 1,000 occurrences over 25 years
    Filter in Appendix A that determines which words are considered for shift scores.
  • Neighbor thresholds = 1,000 nearest neighbors; top 20; at least 20 occurrences per period
    Manual thresholds for neighbor inspection in Appendix A; shape the reported nearest-neighbor tables.
assumptions (5)
  • domain assumption Distributional hypothesis: co-occurrence statistics capture word meaning
    The entire embedding method relies on this; invoked in Sections 1-3 via Mikolov et al. (2013).
  • domain assumption Procrustes alignment preserves semantic correspondences across time periods
    Used in Sections 3.3 and 3.4 to map embeddings into a shared space; standard but unproven here.
  • domain assumption TakeLab Retriever corpus represents Croatian online news for 2000-2024
    The corpus is introduced in Dukić et al. (2024); representativeness is not externally validated.
  • domain assumption Classifier transfer differences d_i<-j reflect sentiment drift rather than alignment artifacts
    Section 3.4; no normalization or control for norms, so this is an unverified assumption.
  • ad hoc to paper Manually curated topic word lists do not bias shift detection
    Section 3.3: topic words are selected semi-automatically based on expected events, without a control list.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Linguistic Shifts in Croatian News via Diachronic Word Embeddings." pith.science (2026). https://pith.science/paper/RRHA5TOT

@misc{pith2026250613569,
  author       = {Pith},
  title        = {Pith review of: Characterizing Linguistic Shifts in Croatian News via Diachronic Word Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRHA5TOT}},
  note         = {Machine review of arXiv:2506.13569}
}
read the original abstract

Measuring how semantics of words change over time improves our understanding of how cultures and perspectives change. Diachronic word embeddings help us quantify this shift, although previous studies leveraged substantial temporally annotated corpora. In this work, we use a corpus of 9.5 million Croatian news articles spanning the past 25 years and quantify semantic change using skip-gram word embeddings trained on five-year periods. Our analysis finds that word embeddings capture linguistic shifts of terms pertaining to major topics in this timespan (COVID-19, Croatia joining the European Union, technological advancements). We also find evidence that embeddings from post-2020 encode increased positivity in sentiment analysis tasks, contrasting studies reporting a decline in mental health over the same period.

Figures

Figures reproduced from arXiv: 2506.13569 by the authors.

Figure 1
Figure 1. Linguistic shifts in Croatian news outlets over [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Sentiment shift between periods. Each cell [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Change of predicted sentiment ratios when [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 22 canonical work pages

  1. [1]

    Samar Al-Saqqa and Arafat Awajan. 2020. https://doi.org/10.1145/3388218.3388229 The use of Word2vec model in sentiment analysis: A survey . In Proceedings of the 2019 International Conference on Artificial Intelligence, Robotics and Control, AIRC '19, page 39–43, New York, NY, USA. Association for Computing Machinery

    work page arXiv 2020

  2. [2]

    Ana Bari \'c , Laura Majer, David Duki \'c , Marijana Grbe s a-zenzerovi \'c , and Jan Snajder. 2023. https://doi.org/10.18653/v1/2023.bsnlp-1.10 Target two birds with one ST o N e: Entity-level sentiment and tone analysis in C roatian news headlines . In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023), pages 78...

    work page doi:10.18653/v1/2023.bsnlp-1.10 2023

  3. [3]

    Michel Br \'e al. 1904. Essai de s \'e mantique (science des significations). Hachette

    work page 1904

  4. [4]

    Walter Cullen, Gautam Gulati, and Brendan D Kelly. 2020. Mental health in the COVID-19 pandemic. QJM: An International Journal of Medicine, 113(5):311--312

    work page 2020

  5. [5]

    David Duki \'c , Marin Petri c evi \'c , Sven \'C urkovi \'c , and Jan S najder. 2024. TakeLab Retriever : AI -driven search engine for articles from Croatian news outlets. arXiv preprint arXiv:2411.19718

    arXiv 2024

  6. [6]

    Darja Fi s er and Nikola Ljube s i \'c . 2019. Distributional modelling for semantic shift detection. International Journal of Lexicography, 32(2):163--183

    work page 2019

  7. [7]

    Goran Glava s , Jan S najder, and Bojana Dalbelo Ba s i \'c . 2012. https://aclanthology.org/W12-0501/ Experiments on hybrid corpus-based sentiment lexicon acquisition . In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, pages 1--9, Avignon, France. Association for Computational Linguistics

    work page 2012

  8. [8]

    Hamilton, Jure Leskovec, and Dan Jurafsky

    William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016 a . https://doi.org/10.18653/v1/D16-1229 Cultural shift or linguistic drift? comparing two computational measures of semantic change . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116--2121, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/d16-1229 2016

Show all 30 references
  1. [9]

    Hamilton, Jure Leskovec, and Dan Jurafsky

    William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016 b . https://doi.org/10.18653/v1/P16-1141 Diachronic word embeddings reveal statistical laws of semantic change . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2016 doi

  2. [10]

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. https://doi.org/10.5281/zenodo.1212303 spaCy : Industrial -strength natural language processing in Python

    2020 doi

  3. [11]

    Vedrana Jankovi \'c , Jan S najder, and Bojana Dalbelo Ba s i \'c . 2011. Random indexing distributional semantic models for Croatian language. In Text, Speech and Dialogue, pages 411--418. Springer

    2011

  4. [12]

    Sinan Kurtyigit, Maike Park, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. https://doi.org/10.18653/V1/2021.ACL-LONG.543 Lexical semantic change discovery . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the...

    2021 doi

  5. [13]

    Nikola Ljube s i \'c and Davor Lauc. 2021. https://aclanthology.org/2021.bsnlp-1.5/ BERT i \'c - the transformer language model for B osnian, C roatian, M ontenegrin and S erbian . In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 37--42, Ki...

    2021

  6. [14]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26

    2013

  7. [15]

    Fabian Pedregosa, Ga \"e l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and 1 others. 2011. Scikit-learn : Machine learning in Python . Journal of machine learning research, 12...

    2011

  8. [16]

    Andraž Pelicon, Marko Pranjić, Dragana Miljković, Blaž Škrlj, and Senja Pollak. 2020. https://www.mdpi.com/2076-3417/10/17/5993 Zero-shot learning for cross-lingual news sentiment classification . MDPI

    2020

  9. [17]

    Radim R eh u r ek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora . In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks , pages 45--50, Valletta, Malta. ELRA. http://is.muni.cz/publication/884893/en

    2010

  10. [18]

    Diego Rozado, Russell Hughes, and Jamin Halberstadt. 2022. https://doi.org/10.1371/journal.pone.0276367 Longitudinal analysis of sentiment and emotion in news media headlines using automated labelling with transformer language models . PLoS One, 17(10):e0276367

    2022 doi

  11. [19]

    Vsevolod Rozanov, Tanja Fran c i s kovi \'c , Igor Marini \'c , Maria-Magdalena Macarenco, Marina Letica-Crepulja, Lana Mu z ini \'c , Ruwan Jayatunge, Merike Sisask, Jan Vevera, Brenda Wiederhold, and 1 others. 2019. Mental health consequences of war conflicts. Advances in ps...

    2019

  12. [20]

    Dominik Schlechtweg, Anna H \"a tty, Marco Del Tredici, and Sabine Schulte im Walde. 2019. https://doi.org/10.18653/v1/P19-1072 A wind of change: Detecting and evaluating lexical semantic change across times and domains . In Proceedings of the 57th Annual Meeting of the Associ...

    2019 doi

  13. [21]

    Dominik Schlechtweg, Frank D Zamora-Reina, Felipe Bravo-Marquez, and Nikolay Arefyev. 2024. Sense through time: Diachronic word sense annotations for word sense induction and lexical semantic change detection. Language Resources and Evaluation, pages 1--35

    2024

  14. [22]

    Peter H Sch \"o nemann. 1966. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31(1):1--10

    1966

  15. [23]

    Jan S najder, B Dalbelo Ba s i \'c , and Marko Tadi \'c . 2008. Automatic acquisition of inflectional lexica for morphological normalisation. Information Processing & Management, 44(5):1720--1731

    2008

  16. [24]

    Jan S najder, Sebastian Pad \'o , and Z eljko Agi \'c . 2013. https://aclanthology.org/P13-2137/ Building and evaluating a distributional memory for C roatian . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),...

    2013

  17. [25]

    Stephen Ullman. 1962. An introduction to the science of meaning. New York: Barnes & Nobel

    1962

  18. [26]

    Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. 2022. A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731--5780

    2022

  19. [27]

    Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4):e1253

    2018

  20. [28]

    Leo Zuanovic, Mladen Karan, and Jan S najder. 2014. Experiments with neural word embeddings for croatian. In Proceedings of the 9th Language Technologies Conference, pages 69--72

    2014

  21. [29]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.