{"id":"6bb8fd8c-6d83-4509-9648-71db4a0f8792","arxiv_id":"2506.13569","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Croatian news embeddings from 2000 to 2024 reveal semantic shifts tied to COVID-19, EU accession, and AI, and indicate a positivity drift in recent-period embeddings on sentiment classifiers.","lead":"The authors trained word embeddings on 3.7 billion words of Croatian news from 2000 to 2024, split into five-year periods, and measured how word meanings shifted. They report that recent-period embeddings push sentiment classifiers toward more positive predictions, a result they contrast with studies of declining mental health.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sentiment-shift claim rests on a transfer metric that is not antisymmetric and ignores embedding-norm/quality differences; the 'later periods more positive' result needs a normalization and common-vocabulary control before it is established.","rationale":"The paper's most valuable contributions are the corpus, the released embeddings, and the qualitative topic-shift evidence in Tables 3 and 5, which are internally consistent and plausible. The central quantitative claim is the sentiment drift, and Section 3.4's design does not rule out obvious confounds: unnormalized embedding norms, large vocabulary and quality differences across periods, and a transfer metric that is demonstrably non-antisymmetric in Figure 2. The reader's weakest_assumption identifies the same load-bearing concern, and the concrete normalization/antisymmetry test would settle whether the positivity finding is real or an artifact. I therefore keep the conditional verdict: the paper should be revised with these controls, not rejected, because the corpus contribution and qualitative analyses are useful regardless of the sentiment-drift outcome.","tokens_in":10934,"tokens_out":5180,"duration_ms":54769,"concrete_test":"Recompute the Section 3.4 experiment with (a) all word embeddings L2-normalized to unit norm before averaging, (b) a fixed vocabulary of words present in all five periods, and (c) the antisymmetry error d_i<-j + d_j<-i reported for every ordered pair. If the monotone increase in Figure 2 disappears or shrinks materially, the central claim is a norm/quality artifact rather than a linguistic shift; if it persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline finding—'embeddings from post-2020 encode increased positivity' (Section 4.3)—is supported only by the transfer quantity d_i<-j = s_i<-j - s_i<-i. For this quantity to isolate sentiment drift, Procrustes-aligned embeddings from period j must be comparable to period-i embeddings in the space where classifier C_i was trained. Two conditions are unchecked. First, SGNS vectors are not norm-normalized, and Procrustes is orthogonal, so any systematic difference in vector norms across periods is preserved; since a logistic regression on averaged vectors is sensitive to feature scale, a period with larger-norm embeddings can shift predictions without any semantic change. The paper reports no norm statistics or normalization control. Second, the metric is not a coherent property of the target period: in Figure 2 (STONE), d_1<-5 = -0.09 whereas -d_5<-1 = 0.41; if d were measuring symmetric drift between periods, these would agree. This asymmetry indicates classifier-specific sensitivity rather than a linguistic property. Corpus-size and embedding-quality differences (Table 1: 53M vs 1.75B words per period; Table 2: contrastive spread rises from 0.08 to 0.27) compound the issue: the period-1 classifier is trained on much noisier embeddings, so feeding it later-period vectors can change predictions for reasons unrelated to sentiment. Figure 3's BERTić label distributions are not an independent control, since the same classifier is applied across periods and may be biased toward its training distribution. Until the transfer experiment is repeated with normalized embeddings, a fixed common vocabulary, and an antisymmetry report, the positivity claim remains an artifact candidate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains SGNS word embeddings on the TakeLab Retriever Croatian news corpus, split into five five-year periods (2000–2024), and uses Procrustes alignment to study semantic change. For topic analysis, it computes a cumulative cosine-distance shift score for manually curated words related to COVID-19, Croatia's EU accession, and technology, and reports qualitative neighbor changes. For sentiment, it proposes a transfer quantity d_{i<-j} = s_{i<-j} - s_{i<-i}, measuring how much a logistic-regression sentiment classifier trained on period i changes its predictions when given aligned embeddings from period j. The paper claims that embeddings from post-2020 are more positive, in contrast to studies reporting declining mental health, and that short-span Croatian news embeddings detect topical linguistic shifts.","tokens_in":11282,"tokens_out":5296,"duration_ms":52628,"significance":"If established, the paper would be a useful contribution to diachronic semantics for a language with scarce historical corpora, and the public release of the embeddings and code would benefit follow-up work. The topic-analysis framework extends earlier work by Hamilton et al. to a short, modern news corpus, and the sentiment-transfer experiment asks a practically relevant question about downstream classifier behavior. However, the sentiment claim currently rests on a transfer metric that is confounded by cross-period differences in embedding quality and vector norms, and the topical-shift analysis lacks quantitative baselines. Both issues are addressable with additional controls, so the contribution is potentially publishable after revision.","major_comments":[{"comment":"The quantity d_{i<-j} = s_{i<-j} - s_{i<-i} is not shown to isolate sentiment drift. SGNS vectors are not norm-normalized, Procrustes alignment is orthogonal, and a logistic regression on averaged vectors is sensitive to feature scale; since periods differ sharply in corpus size and embedding quality (Table 1: 53M vs 1.75B words; Table 2: contrastive spread 0.08 vs 0.27), a classifier trained on period i can change predictions because of norm or quality differences rather than semantic content. The asymmetry in Figure 2 (e.g., d_{1<-5} = -0.09 vs d_{5<-1} = -0.41 on STONE) further indicates that the metric is not a coherent symmetric property of the two periods. Please report norm statistics per period, add a norm-normalization control, analyze a common-vocabulary subset, and either justify the directional interpretation or use a symmetrized measure.","section":"Section 3.4, Figure 2"},{"comment":"The BERTić label-distribution analysis is not an independent control for the sentiment-shift claim. A single classifier trained on the same label sources (STONE and 24sata) is applied across all periods, so the observed increase in predicted positive labels could reflect the classifier's sensitivity to the distributional differences documented in Table 2 rather than a genuine sentiment shift. The paper does not show that BERTić's decision boundary is stable across periods, nor does it calibrate the per-period predictions. Please add per-period calibration or a control that does not reuse the same trained classifier, and temper the claim that Figure 3 corroborates the sentiment-shift result.","section":"Section 4.3, Figure 3"},{"comment":"The topical-shift analysis is presented as evidence that the embeddings capture major topics, but the supporting evidence is qualitative: top-shift words are selected by cumulative D_c, yet no baseline or significance test is provided. Without a control group of non-topic words, a permutation test over D_c, or an evaluation of the curated topic lists against a null distribution, the observed shifts could be driven by corpus-size or embedding-quality effects rather than by the specified topics. Please add such a quantitative baseline and report effect sizes relative to it.","section":"Section 4.2, Table 3"}],"minor_comments":[{"comment":"The formula for D_c should be written unambiguously as D_c = sum_{i=1}^{4} (1 - cos(v_i, v_{i+1})) / 2, and the notation for cosine distance should be defined.","section":"Section 3.3"},{"comment":"The dagger annotation reports p < 0.001 but no statistical test is described; please specify the test procedure and the number of samples used.","section":"Table 2"},{"comment":"The sentence about observing a similar trend using the SentiLex lexicon is not backed by any displayed result; either report the analysis or remove the claim.","section":"Section 4.3"},{"comment":"The limitations section acknowledges that varying article counts 'could bias the results,' but this concern is not applied to the sentiment-transfer metric; please add an explicit limitation about norm comparability and classifier sensitivity.","section":"Limitations"},{"comment":"There is a typo in 'split the corpus into5five-year periods'; it should read 'into five five-year periods.'","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of a CL journal and the resource release is a genuine asset. The main risk is over-interpretation of the sentiment-transfer result; the asymmetry and missing norm/quality controls are concrete and fixable, so I would not recommend rejection if the authors add the requested controls."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean application of the Hamilton et al. pipeline to Croatian news, and the real value is the released embeddings and the well-illustrated topical shifts. The sentiment-positivity headline, however, does not survive close reading.\n\nWhat's actually new: first diachronic embedding study for Croatian, on a 25-year, 3.7B-word corpus split into five-year periods. The method is standard—SGNS with Procrustes alignment—but the language, the corpus, and the descriptive neighbor analyses are new. The COVID/EU/technology examples (maska, varijanta, fond, inteligencija) are convincing and well-presented. The authors also report intrinsic quality checks on two Croatian benchmarks, which is more than many diachronic papers do.\n\nWhere it gets soft: the sentiment shift experiment. The metric d_i<-j = s_i<-j - s_i<-i is supposed to isolate semantic drift, but it doesn't account for embedding norm or quality differences across periods. Early-period embeddings are visibly worse (contrastive spread 0.08 in period 1 vs 0.27 in period 5), and Procrustes is orthogonal, so it preserves norm differences. A logistic regression on averaged vectors will respond to that. More damning, the quantity isn't antisymmetric: in Figure 2 (STONE), d_1<-5 = -0.09 while -d_5<-1 = 0.41. If this were a property of the period pair, those should match. The asymmetry means the result is partly an artifact of classifier sensitivity. The BERTić label distributions in Figure 3 are not an independent control, since the same classifier is applied to all periods. So the claim that 'embeddings from later periods are more positive' is not established. The authors' own limitations section mentions period length and lemmatizer noise, but not this norm/antisymmetry issue.\n\nThe topical-shift analysis is qualitative, but that's an honest descriptive result, not a flaw. The sentiment part needs normalization, a common-vocabulary control, and an antisymmetry check before I'd trust it.\n\nFor who: anyone working on diachronic semantics in low-resource or Slavic languages will find the resource and the neighbor tables useful. The sentiment claim should be read as a hypothesis, not a finding.\n\nRecommendation: this deserves peer review—the resource is real and the descriptive analysis is solid—but the sentiment experiment needs either a redesign or a heavy caveat. I'd send it out, with the expectation of major revision.","headline":"Useful resource and solid descriptive shifts, but the sentiment-positivity claim is too brittle to carry the paper.","tokens_in":11807,"tokens_out":2766,"would_cite":true,"duration_ms":31103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Croatian news language drifted positive across 25 years","keywords":["diachronic word embeddings","semantic change","Croatian news corpus","sentiment drift","skip-gram","Procrustes alignment","COVID-19 terminology","EU accession vocabulary"],"falsifier":"Recompute $d_{i\\leftarrow j}$ after normalizing all embeddings to unit norm and using a held-out test set per period; if the positivity increase disappears or flips sign, the claim fails. The paper's own Figure 2 already shows the quantity is not antisymmetric (on STONE, $d_{1\\leftarrow 5} = -0.09$ while $-d_{5\\leftarrow 1} = 0.41$), so a checker can test whether the apparent drift is a property of the target period alone.","tokens_in":10782,"feed_emoji":"📰","tokens_out":4512,"duration_ms":43287,"temperature":0.7,"pith_summary":"This paper asks whether word meanings in Croatian news shifted measurably over the past 25 years and whether those shifts change how sentiment classifiers behave. By training skip-gram embeddings on 9.5 million articles split into five-year periods, it finds that words tied to COVID-19, EU accession, and technology visibly change their nearest neighbors over just a few years. Its headline quantitative result is that embeddings from later periods encode a more positive stance: transferring aligned embeddings from 2020–2024 into earlier periods moves predicted sentiment upward, even though mental-health studies report worsening mood over the same years. If the finding holds, it means distributional semantics of news became systematically more positive, which would bias any sentiment model trained on one period and applied to another.","feed_headline":"Croatian news language drifted positive across 25 years","feed_subtitle":"Embeddings from 9.5M articles also capture shifts tied to COVID-19, EU accession, and AI.","key_machinery":"Two devices carry the argument. First, per-period skip-gram (SGNS) embeddings projected into a shared space by recursive Procrustes alignment, with the cumulative shift score $D_c = \\sum_{i=1}^4 \\frac{1 - \\cos(v_i, v_{i+1})}{2}$ for each word; this measures how much a word's neighborhood changes across neighboring periods. Second, a cross-period sentiment transfer quantity: for a classifier $C_i$ trained on period $i$ embeddings, $d_{i\\leftarrow j} = \\bar{s}_{i\\leftarrow j} - \\bar{s}_{i\\leftarrow i}$, the change in average predicted sentiment when aligned embeddings from period $j$ are substituted for those from $i$. The sign and size of $d_{i\\leftarrow j}$ across the $5\\times 5$ grid of periods is the evidence for sentiment drift, cross-checked against a Croatian SentiLex lexicon and BERTić-based label-ratio analysis.","core_discovery":"The paper's central claim is that short-span diachronic word embeddings trained on a 3.7-billion-word Croatian newswire corpus capture real linguistic shifts, and that one of those shifts is an increase in encoded positivity. Topic-wise, nouns like maska 'mask', varijanta 'variant', fond 'fund', and inteligencija 'intelligence' acquire new dominant neighbors at identifiable moments, such as 2020 for the pandemic and 2017 for AI. Sentiment-wise, the transfer quantity $d_{i\\leftarrow j} = \\bar{s}_{i\\leftarrow j} - \\bar{s}_{i\\leftarrow i}$ is positive when $j$ is later than $i$: classifiers fed aligned embeddings from a later period rate texts more positively, and this is corroborated by an increased share of positive labels in both headline and article data. The paper presents this increase in news positivity as an empirical contrast to studies reporting declining mental health, and leaves the explanation open, suggesting reactions to negativity, diversification of content, or polarization as candidate drivers.","pith_inferences":["The asymmetry of $d_{i\\leftarrow j}$ in Figure 2 suggests the metric conflates classifier sensitivity to distribution shift with genuine positivity; a symmetric, normalization-invariant measure would be needed to confirm the direction.","One testable extension is to apply the same transfer procedure to an artificially controlled corpus where sentiment is kept constant; if $d$ still trends positive, the artifact is in the alignment and classifier pipeline, not in the language.","The authors' proposed explanations (reaction to negativity, diversification, polarization) are mutually distinguishable in data: polarization predicts rising variance and extreme labels, diversification predicts more neutral-but-positive topics, and a reaction predicts correlation with mental-health indices.","The five-year window is coarse for events like COVID; the paper reports two-year windows failed, but a period-size ablation could identify the minimal corpus size at which shifts become reliable."],"forward_implications":["Sentiment classifiers trained on recent Croatian news will over-estimate positivity when applied to older articles, and vice versa; longitudinal NLP analysis on Croatian data needs period-aware embeddings.","Topic detection from neighbor shifts can pinpoint when a public event changed a word's meaning at five-year resolution, without hand-labeled historical data.","The positive drift in news embeddings, if real, means media discourse and mental-health indicators can move in opposite directions, so news positivity should not be read as population mood.","The method transfers to other lower-resourced languages that have a news archive of comparable size, because it does not require century-scale corpora.","The released embeddings of the Croatian corpus give other researchers a resource for historical text processing and further diachronic studies."],"supporting_citations":[{"why":"Supplies the diachronic embedding methodology and the statistical laws of semantic change that this paper applies to Croatian news.","marker":"Hamilton et al., 2016b"},{"why":"Provides the distinction between cultural shift and linguistic drift and the measure used to identify which words shift.","marker":"Hamilton et al., 2016a"},{"why":"Defines the skip-gram with negative sampling (SGNS) method used to train all period embeddings.","marker":"Mikolov et al., 2013"},{"why":"Provides the TakeLab Retriever corpus of 9.5 million Croatian news articles that the whole study depends on.","marker":"Dukić et al., 2024"},{"why":"Introduces the orthogonal Procrustes alignment that places embeddings from different periods into a common space.","marker":"Schönemann, 1966"},{"why":"Contributes the STONE dataset of Croatian news headlines used for sentiment classification and drift measurement.","marker":"Barić et al., 2023"},{"why":"Contributes the 24sata dataset of full Croatian news articles used as the second sentiment benchmark.","marker":"Pelicon et al., 2020"},{"why":"Provides the SentiLex lexicon for Croatian used to confirm the positivity trend independently of the classifiers.","marker":"Glavaš et al., 2012"},{"why":"Supplies the BERTić transformer model used to assign sentiment labels to unlabeled articles for validation.","marker":"Ljubešić and Lauc, 2021"},{"why":"Documents declining mental health over the same period, forming the contrast that makes the positivity finding noteworthy.","marker":"Cullen et al., 2020"}],"fun_headline_variants":["Croatian news embeddings show positive drift over 25 years","Positive sentiment rises in Croatian news, word models show","25-year Croatian news analysis finds growing positivity","Word embeddings track Croatian news' positive turn since 2000","Croatian news language gets more positive, 9.5M articles show"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sentiment-drift result assumes that the numbers measuring sentiment change really reflect a change in how positive the news language became, rather than an artifact of aligning the vector spaces or of the classifier reacting differently to newer word patterns.","fun_headline_variants_meta":{"raw":{"variants":["Croatian news embeddings show positive drift over 25 years","Positive sentiment rises in Croatian news, word models show","25-year Croatian news analysis finds growing positivity","Word embeddings track Croatian news' positive turn since 2000","Croatian news language gets more positive, 9.5M articles show"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2949,"prompt_tokens":874,"completion_tokens":2075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":1991}},"tokens_in":490,"tokens_out":2075,"duration_ms":16767,"temperature":1.0,"reasoning_tokens":1991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:29:49.869518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute $d_{i\\leftarrow j}$ after normalizing all embeddings to unit norm and using a held-out test set per period; if the positivity increase disappears or flips sign, the claim fails. The paper's own Figure 2 already shows the quantity is not antisymmetric (on STONE, $d_{1\\leftarrow 5} = -0.09$ while $-d_{5\\leftarrow 1} = 0.41$), so a checker can test whether the apparent drift is a property of the target period alone.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the skip-gram with negative sampling (SGNS) method used to train all period embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the 24sata dataset of full Croatian news articles used as the second sentiment benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BERTić transformer model used to assign sentiment labels to unlabeled articles for validation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents declining mental health over the same period, forming the contrast that makes the positivity finding noteworthy."}],"review_version":1}