Pith. sign in

REVIEW 2 major objections 5 minor 24 references

What Makes You CLIC: Detection of Croatian Clickbait Headlines

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned BERTić beats all LLMs at Croatian clickbait detection

desk verdict Useful new dataset for Croatian clickbait, but the headline comparison needs error bars before I would trust the BERTić-vs-LLM gap. read the letter →

arxiv 2507.14314 v2 pith:HAIMJTWG submitted 2025-07-18 cs.CL

Marija An{j̣elić , Dominik Šipek , Laura Majer , Jan Šnajder This is my paper
classification cs.CL
keywords clickbaitdetectionCroatianNLPCLICdatasetBERTićin-contextlearningfine-tunedtransformerslinguisticmarkersnewsheadlines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces CLIC, the first clickbait-detection dataset for Croatian, built from about 2,900 headlines sampled from 30 outlets over 24 years. Its core finding is that clickbait is pervasive in this sample: 52.84% of headlines were labeled clickbait by five annotators each. The paper then asks which tool catches it best in a low-resource language, and answers that a fine-tuned language-specific transformer (BERTić, 77.6% F1) outperforms every general LLM tested, including GPT-4.1 and Mistral 7B with engineered prompts (best LLM at 73.0% F1). It also shows that clickbait is linguistically marked—by pronouns, determiners, particles, and interjections—and that even simple TF-IDF classifiers beat random, confirming the task is largely a lexical one.

What carries the argument

The work rests on two objects. First, the CLIC corpus: 2,907 Croatian news headlines, each annotated by five volunteers without seeing the article text, spanning mainstream and fringe outlets from 2000 to 2024. Second, BERTić, an Electra-based transformer pre-trained on Bosnian, Croatian, Montenegrin, and Serbian; fine-tuning it on CLIC yields the 77.6% F1. For the LLM comparison, the prompt components—Definition, Features, and Examples—are isolated so the paper can attribute gains to each, and Features plus Examples emerges as the strongest configuration.

What would settle it

Re-annotate a random sample of CLIC headlines with the full article visible and measure agreement with the existing labels; if agreement drops below about $\kappa = 0.4$ or the clickbait rate shifts significantly, the headline-only ground truth and its conclusions are not robust. Alternatively, an LLM prompt tuned on the public training split that exceeds 77.6% F1 on the same test split would directly contradict the paper's comparative claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes two things: clickbait is not rare in Croatian online media, and the best detector is not a general LLM. Annotators marked 1,536 of 2,907 headlines as clickbait (52.84%), with moderate agreement (Fleiss $\kappa = 0.53$), and fine-tuned BERTić reached 77.6% F1 on a held-out test set. The strongest LLM, Mistral 7B prompted with explicit clickbait features plus few-shot examples, reached only 73.0% F1, and most prompt configurations fell well short. The paper further demonstrates that adding a features list to LLM prompts consistently helps, and that clickbait headlines skew toward pronouns, determiners, particles, and interjections while non-clickbait headlines carry more proper nouns and numbers.

Load-bearing premise

The ground truth is built on headline-only judgments, so if article context changes what should count as clickbait, the labels and every model comparison inherit that bias.

Editorial extensions

If this is right

  • Any future model for Croatian clickbait detection can be benchmarked directly against CLIC and the 77.6% F1 BERTić result.
  • Practitioners working on low-resource South Slavic languages should prefer fine-tuned language-specific transformers over general LLMs for this classification task.
  • LLM users can expect the biggest F1 gains from spelling out clickbait features and giving few-shot examples, not from switching models.
  • Simple TF-IDF classifiers already beat random, so lightweight lexical detection is a viable baseline for Croatian.
  • The stable ~50% clickbait rate over 2000–2024 contradicts the assumption that clickbait is a newly worsening phenomenon, at least in this sample.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If annotators were given the article text, the clickbait rate and the model ranking could shift; measuring that shift is a direct next experiment the paper leaves open.
  • The same fine-tuned-versus-ICL comparison can be run on other South Slavic languages, such as Slovenian or Bulgarian, to test whether BERTić's edge is regional or Croatian-specific.
  • Because the dataset includes fringe outlets, one could test whether clickbait prevalence correlates with outlet type, which the paper does not analyze.
  • Cross-lingual transfer—training on CLIC and testing on Italian or Romanian clickbait corpora—would reveal whether the linguistic markers found here are universal or language-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces CLIC, a Croatian clickbait detection dataset of 2,907 headlines sampled from 30 outlets over 2000–2024, each annotated by five volunteers (Fleiss' κ=0.53). It compares fine-tuned Transformer models (BERT, BERTić) and TF-IDF-based ML baselines against four LLMs under zero- and few-shot in-context learning with English and Croatian prompts, and reports that BERTić reaches 77.6% F1, ahead of the best LLM (Mistral 7B with features and examples) at 73% F1. The paper also presents linguistic analyses (PoS overrepresentation, dataset cartography, error analysis).

Significance. If the comparative claim holds, this is a solid resource contribution and a useful data point for less-resourced languages, backed by a publicly available dataset and a systematic prompt-grid evaluation. The dataset itself is valuable and likely citable, and the linguistic analysis is a plausible addition. The main comparative result, however, is not yet supported because it lacks uncertainty quantification and the baseline comparison is methodologically mismatched.

major comments (2)
  1. [Section 4, Table 2] The central comparison between BERTić (F1=0.78) and the best LLM (Mistral 7B, FE, English, F1=0.73) is reported from a single run, with no standard deviations, confidence intervals, or significance tests. Since BERTić fine-tuning is stochastic, LLM decoding can be stochastic, and the best LLM number is selected from a 56-configuration grid, the 4.6-point gap may be within run-to-run variation. Please report averaged results over multiple seeds with standard deviations (or confidence intervals) and run a paired significance test (e.g., McNemar or bootstrap) for the BERTić-vs-best-LLM comparison. Without this, the abstract's claim that fine-tuned models deliver better results than general LLMs is not established at the reported precision.
  2. [Section 4, baselines paragraph and Tables 1–2] The paper states that all tested ML methods and LLMs outperform the majority and random baselines, but the majority baseline is reported as accuracy (0.55) while model results are reported as F1. On the test set, a majority-class predictor would have F1 ≈ 0.69 (assuming the test split is balanced as in the full dataset), which is higher than several F1 scores reported in Table 2 (e.g., English Phi 3.5 DFE 0.57; many Croatian configurations below 0.60). The comparison is therefore between different metrics, and the claim that all LLMs outperform the majority baseline is not supported. Please report the majority baseline in terms of F1 (and, if desired, also accuracy) and revise the claim accordingly.
minor comments (5)
  1. [Abstract and Section 6] The wording 'finetuned models deliver better results than general LLMs' is only supported by BERTić; the fine-tuned BERT achieves F1=0.714, which is below the best LLM's 0.73. Please rephrase to 'a fine-tuned BERTić' or 'the best fine-tuned model' to avoid overgeneralization.
  2. [Section 3] The aggregation of the five annotations into a single label is not described explicitly; with three labels (clickbait, not clickbait, invalid), a 2-2-1 split is possible, so the paper should state whether the final label is a majority vote and how such ties are resolved.
  3. [Section 5] The statement 'Chi-squared tests confirm statistical significance for most differences' is not supported by the reported data: Table 3 does not include p-values, and no details on the chi-square test or multiple-comparison correction are given. Please add p-values or a note about correction.
  4. [Table 3] The entry 'Interjection∞' is ambiguous; the caption should clarify that the ratio is clickbait-to-non-clickbait frequency and that an infinite value means the tag never appears in non-clickbait headlines.
  5. [Section 4] The phrase 'the full prompts are available in Appendix 5' is inaccurate because the prompts appear in Table 5, not in a separately numbered appendix; please renumber or rephrase.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose claims rest on held-out evaluation against human annotations, not on a derivation that reduces to its own inputs.

full rationale

The paper makes two central empirical claims: that roughly half of sampled Croatian headlines are clickbait, and that a fine-tuned BERTić model (77.6% F1) outperforms the best in-context-learning LLM configuration (73% F1). Both claims are supported by a supervised evaluation protocol: a human-annotated dataset (Section 3), an 80-10-10 train/validation/test split (Section 4), and held-out F1 scores reported in Tables 1 and 2. No fitted parameter is used to define the clickbait labels; the labels come from five independent annotators per headline, and the model predictions are compared against those labels on a test set untouched during training. The only self-citation, TakeLab Retriever (Dukić et al., 2024), is used as a data-sampling tool and is not load-bearing for any claimed result; BERTić is an externally published pretrained model by different authors, not a result of the present paper. The acknowledged limitation in Section 7 that headline-only annotation may misclassify sensational-but-accurate headlines is a validity concern about the ground truth, not a circularity in the derivation, and it affects all compared models equally. The skeptic's concern about missing uncertainty quantification is a rigor issue about whether the 4.6-point gap is stable, but it does not make the comparison circular. The derivation chain is therefore self-contained with respect to the empirical claims made.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The empirical claims rest on the annotation design and evaluation protocol rather than on free theoretical parameters. The main unverified inputs are the validity of headline-only clickbait annotation, the use of majority labels under moderate agreement, and the assumption that a random split avoids leakage.

assumptions (3)
  • domain assumption Annotating headlines without article text yields valid clickbait labels.
    Section 3 states annotators saw no article, unlike several prior studies, to mimic reader behavior; the paper's own limitations note this can misclassify sensational-but-accurate headlines.
  • domain assumption Majority vote of five annotators is ground truth despite moderate agreement.
    Section 3 reports Fleiss kappa of 0.53, yet majority labels are used as training and evaluation targets throughout the paper.
  • domain assumption A random 80/10/10 split is a valid evaluation protocol.
    Section 4 uses a single random split; no temporal or outlet-based split is used, so stylistic leakage across time or outlets is not controlled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Makes You CLIC: Detection of Croatian Clickbait Headlines." pith.science (2026). https://pith.science/paper/HAIMJTWG

@misc{pith2026250714314,
  author       = {Pith},
  title        = {Pith review of: What Makes You CLIC: Detection of Croatian Clickbait Headlines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAIMJTWG}},
  note         = {Machine review of arXiv:2507.14314}
}
read the original abstract

Online news outlets operate predominantly on an advertising-based revenue model, compelling journalists to create headlines that are often scandalous, intriguing, and provocative -- commonly referred to as clickbait. Automatic detection of clickbait headlines is essential for preserving information quality and reader trust in digital media and requires both contextual understanding and world knowledge. For this task, particularly in less-resourced languages, it remains unclear whether fine-tuned methods or in-context learning (ICL) yield better results. In this paper, we compile CLIC, a novel dataset for clickbait detection of Croatian news headlines spanning a 20-year period and encompassing mainstream and fringe outlets. We fine-tune the BERTi\'c model on this task and compare its performance to LLM-based ICL methods with prompts both in Croatian and English. Finally, we analyze the linguistic properties of clickbait. We find that nearly half of the analyzed headlines contain clickbait, and that finetuned models deliver better results than general LLMs.

Figures

Figures reproduced from arXiv: 2507.14314 by the authors.

Figure 1
Figure 1. Dataset cartography visualization showing the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Percentage of news headlines classified as [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [1]

    Amol Agrawal. 2016. https://doi.org/10.1109/NGCT.2016.7877426 C lickbait detection using deep learning . In 2016 2nd International Conference on Next Generation Computing Technologies (NGCT), pages 268--272

    work page arXiv 2016

  2. [2]

    Liliana Alves, Nuno Antunes, Olga Agrici, Carlos Sousa, and Célia Ramos. 2016. https://doi.org/10.21664/2238-8869.2016v5i2.p196-213 C lick B ait: Y ou W on’t B elieve W hat H appens N ext! Journal of Social, Technological and Environmental Science, 5:196--213

    work page arXiv 2016

  3. [3]

    Valentina Apresjan and Alexander Orlov. 2022. https://doi.org/https://doi.org/10.1016/j.pragma.2022.02.003 P ragmatic mechanisms of manipulation in R ussian online media: H ow clickbait works (or does not) . Journal of Pragmatics, 195:91--108

    doi:10.1016/j.pragma.2022.02.003 2022

  4. [4]

    8 A mazing S ecrets for G etting M ore C licks

    Prakhar Biyani, Kostas Tsioutsiouliklis, and John Blackmer. 2016. https://doi.org/10.1609/aaai.v30i1.9966 "8 A mazing S ecrets for G etting M ore C licks": D etecting C lickbaits in N ews S treams U sing A rticle I nformality . Proceedings of the AAAI Conference on Artificial Intelligence, 30

    work page doi:10.1609/aaai.v30i1.9966 2016

  5. [5]

    Jonas Nygaard Blom and Kenneth Reinecke Hansen. 2015. https://doi.org/https://doi.org/10.1016/j.pragma.2014.11.010 C lick bait: F orward-reference as lure in online news headlines . Journal of Pragmatics, 76:87--100

    doi:10.1016/j.pragma.2014.11.010 2015

  6. [6]

    Mark Bronakowski, Mahmood Al-khassaweneh, and Ali Al Bataineh. 2023. https://doi.org/10.3390/app13042456 A utomatic D etection of C lickbait H eadlines U sing S emantic A nalysis and M achine L earning T echniques . Applied Sciences, 13(4)

    doi:10.3390/app13042456 2023

  7. [7]

    Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly. 2016. http://arxiv.org/abs/1610.09786 S top C lickbait: D etecting and P reventing C lickbaits in O nline N ews M edia . CoRR, abs/1610.09786

    work page Pith review arXiv 2016

  8. [8]

    Abhijnan Chakraborty, Rajdeep Sarkar, Ayushi Mrigen, and Niloy Ganguly. 2017. http://arxiv.org/abs/1709.02957 T abloids in the E ra of S ocial M edia? U nderstanding the P roduction and C onsumption of C lickbaits in T witter

    work page Pith review arXiv 2017

Show all 24 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. http://arxiv.org/abs/1810.04805 BERT : P re-training of D eep B idirectional T ransformers for L anguage U nderstanding

    2019 arXiv

  2. [10]

    David Dukić, Marin Petričević, Sven Ćurković, and Jan Šnajder. 2024. http://arxiv.org/abs/2411.19718 Takelab retriever: AI - D riven S earch E ngine for A rticles from C roatian N ews O utlets

    2024 arXiv

  3. [11]

    Maik Fr \"o be, Benno Stein, Tim Gollub, Matthias Hagen, and Martin Potthast. 2023. https://doi.org/10.18653/v1/2023.semeval-1.312 S em E val-2023 T ask 5: C lickbait S poiling . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 227...

    2023 doi

  4. [12]

    Raluca-Andreea G \^i nga and Ana Sabina Uban. 2024. https://doi.org/10.18653/v1/2024.nlp4pi-1.17 S ci T ech B ait RO : C lick B ait D etection for R omanian S cience and T echnology N ews . In Proceedings of the Third Workshop on NLP for Positive Impact, pages 188--201, Miami,...

    2024 doi

  5. [13]

    Grammarist. 2023. https://grammarist.com/new-words/clickbait/ Clickbait . Accessed: 2025-04-29

    2023

  6. [14]

    Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Gupta, and Vasudeva Varma. 2020. https://doi.org/10.18653/v1/2020.coling-main.425 Predicting clickbait strength in online social media . In Proceedings of the 28th International Conference on Computational Linguistics, pages 4835-...

    2020 doi

  7. [15]

    Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koychev. 2017. https://doi.org/10.26615/978-954-452-049-6_045 We built a fake news / click bait filter: What happened next will blow your mind! In Proceedings of the International Conference Recent Advances in Natural La...

    2017 doi

  8. [16]

    Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2024. http://arxiv.org/abs/arXiv:2404.11553 L anguage R anker: A M etric for Q uantifying LLM P erformance A cross H igh and L ow- R esource L anguages

    2024 arXiv

  9. [17]

    Nikola Ljube s i \'c and Davor Lauc. 2021. https://aclanthology.org/2021.bsnlp-1.5 BERT i \'c - the transformer language model for B osnian, C roatian, M ontenegrin and S erbian . In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 37--42, Kiy...

    2021

  10. [18]

    Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. 2018. https://aclanthology.org/C18-1127 Crowdsourcing a large corpus of clickbait on T witter . In Proceedings of the 27th Inter...

    2018

  11. [19]

    Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. https://doi.org/10.1007/978-3-319-30671-1_72 C lickbait D etection . volume 9626, pages 810--817

    2016 doi

  12. [20]

    Ayla Rigouts Terryn and Miryam de Lhoneux. 2024. https://aclanthology.org/2024.humeval-1.2/ E xploratory S tudy on the I mpact of E nglish B ias of G enerative L arge L anguage M odels in D utch and F rench . In Proceedings of the Fourth Workshop on Human Evaluation of NLP Sys...

    2024

  13. [21]

    Daniel Russo, Oscar Araque, and Marco Guerini. 2024. https://aclanthology.org/2024.clicit-1.90/ T o C lick I t or N ot to C lick I t: An I talian D ataset for N eutralising C lickbait H eadlines . In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC...

    2024

  14. [22]

    Smith, and Yejin Choi

    Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.746 Dataset cartography: Mapping and diagnosing datasets with training dynamics . In Proceedings of the 2020 C...

    2020 doi

  15. [23]

    Veronika Vincze and Martina Katalin Szab \'o . 2020. https://aclanthology.org/2020.rdsm-1.6 Automatic detection of H ungarian clickbait and entertaining fake news . In Proceedings of the 3rd International Workshop on Rumours and Deception in Social Media (RDSM), pages 58--69, ...

    2020

  16. [24]

    Wang, Ye Wang, Yun Li, Yunhao Yuan, and Jipeng Qiang

    Yi Zhu, H. Wang, Ye Wang, Yun Li, Yunhao Yuan, and Jipeng Qiang. 2023. https://api.semanticscholar.org/CorpusID:259187872 C lickbait D etection via L arge L anguage M odels . ArXiv, abs/2306.09597

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.