From BERT to Qwen: Hate Detection across architectures

Ariadna Mon; Jon Lecumberri; Sa\'ul Fenollosa

REVIEW 3 major objections 6 minor 18 references

From BERT to Qwen: Hate Detection across architectures

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fine-tuned 0.5-billion-parameter decoder-only LLM slightly edges out compact encoder transformers on balanced hate-speech detection.

desk verdict Pre-split oversampling contaminates the validation split, so Qwen's reported edge may be memorization, not architecture; the rest is a clear but routine benchmark. read the letter →

arxiv 2507.10468 v1 pith:KCLTPPW4 submitted 2025-07-14 cs.CL cs.LG

Ariadna Mon , Saúl Fenollosa , Jon Lecumberri This is my paper · ORCID

classification cs.CLcs.LG

keywords hatespeechdetectionencoder-onlytransformersdecoder-onlyLLMszero-shotpromptingfew-shotfine-tuningQwenGemma

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether small decoder-only language models can rival fine-tuned encoder-only transformers at real-world hate speech detection. The authors compare DistilBERT and Twitter-RoBERTa with Gemma-3-1B and Qwen1.5-0.5B on a balanced, curated corpus of online sentences, under zero-shot, few-shot, and fine-tuned regimes. Their central finding is that a fine-tuned Qwen, evaluated without in-context examples after weight updates, reaches the highest average accuracy and F1, slightly ahead of both encoder baselines; base Gemma performs poorly in zero-shot but gains 0.15–0.20 F1 from eight-shot prompting. The upshot is that compact decoder-only models can match or exceed encoder-only classifiers once they receive task-specific weight updates, without sacrificing the prompt-based flexibility that makes them easy to adapt to new tasks.

What carries the argument

The paper's engine is a controlled comparison between two transformer families: encoder-only transformers (bidirectional masked-language models such as DistilBERT and Twitter-RoBERTa, fine-tuned with a task head) and decoder-only autoregressive LLMs (Gemma-3-1B and Qwen1.5-0.5B, prompted with zero or eight in-context examples and, for Qwen, also fine-tuned). The load-bearing setup is the shared balanced validation split, five-seed averaging, and identical evaluation protocol, which is what lets the authors attribute observed differences to architecture and inference regime rather than to data artifacts.

What would settle it

Re-run the comparison with Qwen fine-tuned under two explicitly stated setups — a standard classification head and a causal language-model prompt objective — using the same splits and seeds; if the second setup does not beat DistilBERT and RoBERTa, the architecture-level conclusion fails. Alternatively, a matched-parameter encoder with roughly 0.5B parameters trained on the same balanced data would show whether the edge comes from the decoder architecture or simply from larger model size.

Watch

Extended reading notes

Core claim

On a version of the curated hate-speech dataset rebalanced to 50/50 labels and split 70/30, the authors fine-tuned two encoder-only transformers and one decoder-only LLM, and prompted a second decoder-only LLM in zero- and few-shot modes. Across five random seeds, fine-tuned Qwen1.5-0.5B evaluated without in-context examples achieved the highest average accuracy and F1, slightly beating fine-tuned DistilBERT and Twitter-RoBERTa, both of which stayed above 0.80 F1 on both classes. Unadapted Gemma-3-1B scored poorly in zero-shot, but its F1 jumped by 0.15–0.20 when eight balanced in-context examples were prepended, confirming that few-shot prompting can partially replace gradient updates.

Load-bearing premise

The comparison assumes Qwen was fine-tuned in the same task-specific way as the encoders, but the paper never states what kind of output layer or objective Qwen was fine-tuned with, so the reported edge may partly reflect that unspecified setup.

Editorial extensions

If this is right

A 0.5-billion-parameter decoder-only model, after task-specific fine-tuning, becomes a credible alternative to compact encoders for hate-speech classification.
Few-shot prompting can lift an initially poor decoder-only model by 0.15–0.20 F1, making in-context examples a partial substitute for gradient updates.
Without either fine-tuning or few-shot examples, small decoder-only LLMs are not reliable zero-shot hate-speech classifiers.
Since fine-tuned Qwen tops both encoders, decoder-only fine-tuning does not inherently sacrifice discriminative accuracy while keeping prompt-based flexibility.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The encoder baselines have about 66M and 125M parameters while Qwen has 0.5B, so the reported edge compares architecture and scale together; a matched-size encoder trained on the same data would isolate the architectural contribution.
If Qwen's fine-tuning used a causal language-model objective rather than a classification head, the result would partly measure the prompting interface; replicating with both setups would separate the two effects.
The paper's single-corpus scope leaves open whether the Qwen edge transfers to other languages or platforms; testing on imbalanced in-the-wild streams would be a natural next step.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the one thing to know: the stress-test note is right. In Section 4.1 the authors balance the data by random undersampling and oversampling, and only then, in Sections 4.3/5, split 70/30 into training and validation. Oversampling duplicates minority examples, so identical text can land in both splits. The validation set is therefore not 'untouched', and Qwen FT ZS's 'highest overall accuracy and F1' could be inflated by memorized duplicates. Because Qwen is the largest fine-tuned model, it may benefit most. This is a load-bearing flaw, not a cosmetic one.

Credit where it's due: the question is practical and clearly posed—can a 0.5B decoder match compact encoders on hate speech? The paper is the first to measure Qwen-0.5B and Gemma-3-1B on Mody et al. across zero-shot, few-shot, and fine-tuned regimes, so it is a new measurement. Five seeds with error bars is good practice, and the limitations paragraph is honest about single-corpus and runtime environment issues.

Now the soft spots, beyond contamination. Qwen's fine-tuning setup is underspecified: Table 1 gives epochs, batch size, learning rate, but not whether it's a classification head or a causal LM objective with a prompt. Without that, even a clean split wouldn't tell you what 'slightly edging out' means. There are no significance tests, so the claimed edge is just a mean difference across five seeds. Figure 3 shows only seed 42 despite the text saying curves are averaged. No code or data are released, so the contamination can't be checked or corrected by readers. And the conclusion generalizes from Gemma-only few-shot to decoders generally, which outruns the evidence.

The paper is coherent and readable, and the authors seem to have done the work carefully up to the split, but the split mistake is fundamental. As is, the central result is unsupported. Who gets value from this? Practitioners who want to pick a small model for hate detection would get useful guidance only after the experiment is rerun with a leakage-free split. I would not send this to peer review in its current form; the fix requires rerunning everything. If a corrected version appears with code, clear Qwen training details, and significance tests, it's worth a short review. For now, desk reject or return for major revision.

Referee Report

3 major / 6 minor

Summary. The paper compares hate-speech detection across two encoder-only transformers (DistilBERT, Twitter-RoBERTa) and two decoder-only LLMs (Qwen1.5-0.5B, Gemma-3-1B) on Mody et al.'s curated hate-speech dataset. The authors rebalance the label distribution, split the data 70/30, fine-tune the encoders and Qwen, evaluate Gemma in zero-shot and few-shot modes, and report accuracy and F1 metrics averaged over five seeds. The central claim, stated in Section 5, is that fine-tuned Qwen evaluated in zero-shot settings delivers the highest overall accuracy and F1, slightly edging out both encoder-only baselines, while Gemma's few-shot prompting substantially improves over its zero-shot performance.

Significance. If the central result is valid, it is a practically useful finding: a compact 0.5B decoder-only model can match or slightly beat fine-tuned encoder baselines on a hate-speech benchmark after task-specific fine-tuning. The paper has strengths: it uses an external benchmark, evaluates multiple model families under several inference regimes, reports results across five random seeds, and includes explicit limitations about single-corpus generalization. However, the manuscript currently does not support the central architecture comparison because of a likely validation-set contamination issue, an underspecified Qwen fine-tuning setup, and the absence of statistical significance testing. These are fixable, but they are load-bearing for the paper's main claim.

major comments (3)

[Section 4.1 and Section 5] The validation split is very likely contaminated by the pre-split oversampling step. Section 4.1 states that after removing long entries the authors 'addressed label imbalance by randomly undersampling the majority class and oversampling the minority class until both hateful and non-hateful classes were equally represented' and only then, per Sections 4.3 and 5, split the data 70/30. If oversampling duplicates minority-class examples, exact copies of the same text can appear in both the training and validation splits, so the validation set is not 'untouched' as claimed in Section 5. Since the largest fine-tuned model (Qwen-0.5B) may benefit most from memorizing duplicated training instances, the reported 'slightly edging out' result could reflect memorization rather than architectural quality. The authors should fix this by performing augmentation after splitting (or by deduplicating text across splits), and they should state explicitly how oversampling was implemented and whether any duplicated examples cross the train/validation boundary.
[Section 4.3 and Table 1] The Qwen fine-tuning procedure is not specified in enough detail to support the architecture comparison. Table 1 reports only epochs, batch size, and learning rate. The manuscript does not state whether Qwen was fine-tuned with a classification head, a causal language-modeling objective with a prompt template, a sequence-classification wrapper, or some other setup, nor does it describe the inference-time decoding or scoring rule used to assign hate/non-hate labels in the zero-shot fine-tuned evaluation. Without these details, the comparison between Qwen FT ZS and the fine-tuned encoders is not a controlled comparison of architectures; it may instead measure differences in task formulation and output layer. The authors should report the exact fine-tuning objective, model head, prompt format, and label extraction procedure.
[Section 5 and Figure 5] The central claim that Qwen FT ZS 'slightly edges out' the encoder baselines is not supported by any statistical test. Figure 5 shows error bars of one standard deviation across five seeds, but the text does not report confidence intervals, paired significance tests, or effect sizes. With only five seeds and likely overlapping error bars, the observed differences may be within noise. The authors should add appropriate significance tests (e.g., paired bootstrap or randomized permutation tests on the validation set) or at least report per-seed scores and confidence intervals so that the reader can judge whether Qwen's advantage is reliable.

minor comments (6)

[Section 3] The dataset statistics are inconsistent: 371,452 + 80,250 = 451,702, not 451,709, and the stated 82%/18% proportions do not match these counts exactly. Please correct the numbers or clarify rounding.
[Section 5 and Figure 3] The text says that validation accuracy and loss curves are 'averaged across seeds,' but Figure 3's caption explicitly says 'seed 42.' Please clarify whether the figure shows a single seed or the seed-averaged curves, and label it accordingly.
[Section 5] Few-shot evaluation is described as applied to Gemma only, yet the paper's hypotheses and related work emphasize few-shot prompting for decoder-only models in general. Please explain why Qwen was not evaluated in the few-shot setting, or add that evaluation.
[References] Reference [15] is cited for DistilBERT, but the listed work is Yang et al.'s XLNet paper. The DistilBERT architecture should be cited to Sanh et al. (2019), 'DistilBERT, a distilled version of BERT.'
[Keywords and Figure 4] The keyword 'Transfomers' contains a typo, and Figure 4 uses 'No-hate' while the rest of the paper uses 'non-hateful.' Please make terminology consistent.
[Figure 1] Figure 1's caption mentions two panels, 'Label Distribution Balanced' and 'Content Length Distribution,' but the panels are not labeled in the caption. Please add axis labels and panel identifiers to make the figure self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical benchmark result measured on an external validation split, with no fitted quantity or self-citation chain defining the outcome.

full rationale

The paper's central result—that fine-tuned Qwen evaluated in zero-shot mode slightly outperforms encoder-only baselines—is an empirical measurement on a validation split of an external dataset (Mody et al.'s curated hate speech corpus). There is no derivation chain in which a defined quantity is shown to equal its own input. The model hyperparameters (epochs, batch size, learning rate) were chosen before evaluation, and none of the reported metrics are computed from a fitted parameter that was itself fit to the validation outcome. The references cited are external prior works, and there is no load-bearing self-citation or imported uniqueness theorem. The acknowledged limitations (single corpus, inconsistent runtime environments) are threats to generalization and comparability, not circularity. The pre-split oversampling concern raised in the skeptic note is a potential data-leakage/correctness issue, but it does not make the result equivalent to its inputs by construction; it would affect the validity of the comparison rather than the circularity of the derivation. Therefore, no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

All results rest on external pre-trained checkpoints and a single labeled corpus; the paper contributes a comparison, not a derivation, so the ledger contains design choices and background assumptions rather than fitted physical parameters.

free parameters (8)

token length cutoff = 500 tokens
Entries exceeding 500 tokens are removed in preprocessing (Section 4.1); cutoff chosen by hand, affects training distribution.
class balance target = 50/50
Undersampling and oversampling are applied until both classes are equally represented (Section 4.1); the validation set is then balanced, not representative of the original 82/18 distribution.
DistilBERT training epochs = 2
Table 1; chosen by hand.
RoBERTa training epochs = 2
Table 1; chosen by hand.
Qwen training epochs = 3
Table 1; chosen by hand.
batch sizes = 128 encoders, 64 Qwen
Table 1; chosen by hand.
learning rate = 2e-5
Table 1; chosen by hand for all fine-tuned models.
few-shot example count k = 8
Section 5, 4 hate and 4 no-hate examples; chosen by hand.

assumptions (4)

domain assumption Mody et al. dataset labels are correct ground truth
The entire evaluation treats corpus labels as ground truth (Section 3).
domain assumption Pre-trained checkpoints encode general language ability and are used as provided
Section 4.2; results depend on these external models and their pretraining distributions.
domain assumption Balanced validation performance averaged over five seeds is a valid proxy for real-world hate speech detection
Section 5; no significance tests or skewed-distribution evaluation are reported.
ad hoc to paper Fine-tuning Qwen for classification is directly comparable to fine-tuning encoders
Section 4.3 and Table 1 list only epochs, batch size, and learning rate; no fine-tuning objective or prompt format is given, yet the comparison assumes parity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From BERT to Qwen: Hate Detection across architectures." pith.science (2026). https://pith.science/paper/KCLTPPW4

@misc{pith2026250710468,
  author       = {Pith},
  title        = {Pith review of: From BERT to Qwen: Hate Detection across architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KCLTPPW4}},
  note         = {Machine review of arXiv:2507.10468}
}

read the original abstract

Online platforms struggle to curb hate speech without over-censoring legitimate discourse. Early bidirectional transformer encoders made big strides, but the arrival of ultra-large autoregressive LLMs promises deeper context-awareness. Whether this extra scale actually improves practical hate-speech detection on real-world text remains unverified. Our study puts this question to the test by benchmarking both model families, classic encoders and next-generation LLMs, on curated corpora of online interactions for hate-speech detection (Hate or No Hate).

Figures

Figures reproduced from arXiv: 2507.10468 by the authors.

**Figure 2.** highlights the key differences between our two model families. Encoder-only Transformers like BERT are trained with a masked-language-model objective, allowing each token to attend to both its left and right context; this bidirectional design makes them well suited for fine-tuning on classification tasks. In contrast, decoder-only models use a causal attention mask, predicting one token at a time based solely on pre… view at source ↗

**Figure 5.** Final validation metrics for all models: Qwen base ZS, Qwen FT ZS, Gemma ZS, Gemma FS, DistilBERT FT, and RoBERTa FT. while base Gemma performs poorly in ZS mode. However, when Gemma is supplied with FS, its performance jumps by over 0.15–0.20 F1 ( [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗

Figures from the paper (2 more)

**Figure 3.** Figure 3: Validation accuracy and loss curves for DistilBERT vs. RoBERTa (seed 42). For decoder-only models (Qwen, Gemma) we considered two evaluation settings on the same validation split: zeroshot (ZS), no labelled examples provided in the prompt before inference, and few-sho…

**Figure 4.** Figure 4: Base Gemma performance on the validation set under zero-shot (ZS) vs. few-shot (FS) prompting across three metrics (averaged over five seeds). Finally, [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

[1]

Bert: Pre-training of deep bidirectional transform- ers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transform- ers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

arXiv 2018
[2]

Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model

H. Saleh, A. Alhothali, and K. Moria, “Detec- tion of hate speech using bert and hate speech word embedding with deep model,” arXiv preprint arXiv:2111.01515, 2021

work page Pith review arXiv 2021
[3]

Fine-tuning bert for sentiment analysis: A practical guide

H. Amit, “Fine-tuning bert for sentiment analysis: A practical guide.” Medium, Jan 2024

work page 2024
[4]

Language models are few-shot learn- ers,

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...

work page 2020
[5]

A cu- rated hate speech dataset (version 1),

D. Mody, Y . Huang, and T. E. A. de Oliveira, “A cu- rated hate speech dataset (version 1),” 2022

work page 2022
[6]

Challenges and limitations of zero-shot and few-shot learning in large language models,

C. Luca, “Challenges and limitations of zero-shot and few-shot learning in large language models,” inTrends, Applications, and Challenges of Large Language Mod- els, pp. 131–151, IGI Global, 2023

work page 2023
[7]

Hatespeechbert: Retraining bert for automatic hate speechdetection,

A. Ishaq, K. Malik, and A. Zafar, “Hatespeechbert: Retraining bert for automatic hate speechdetection,” SSRN Electronic Journal, 01 2023

work page 2023
[8]

Now you see me: Vision-enhanced bert for obfuscated text abuse detection,

D. Zhou, “Now you see me: Vision-enhanced bert for obfuscated text abuse detection,” 2023. CS224N Custom Project, Natural Language Processing with Deep Learning

work page 2023

Show all 18 references

[9]

Metahate: A dataset for unifying efforts on hate speech detection,

P. Piot, P. Mart´ın-Rodilla, and J. Parapar, “Metahate: A dataset for unifying efforts on hate speech detection,” Proceedings of the International AAAI Conference on Web and Social Media , vol. 18, p. 2025–2039, May 2024

2025
[10]

Efficient hate speech detection: Evaluating 38 models from traditional methods to transformers,

M. Abusaqer, J. Saquer, and H. Shatnawi, “Efficient hate speech detection: Evaluating 38 models from traditional methods to transformers,” in ACMSE 2025: Proceedings of the 2025 ACM Southeast Conference , pp. 203–214, 05 2025

2025
[11]

Enhancing hate speech detection in the digital age: A novel model fusion approach leveraging a comprehensive dataset,

W. Sharif, S. Abdullah, S. Iftikhar, D. Al-Madani, and S. Mumtaz, “Enhancing hate speech detection in the digital age: A novel model fusion approach leveraging a comprehensive dataset,” IEEE Access , vol. 12, pp. 27225–27236, 2024

2024
[12]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural infor- mation processing systems, vol. 30, 2017

2017
[13]

DistilBERT-base-uncased

Hugging Face, “DistilBERT-base-uncased.” https://huggingface.co/distilbert/ distilbert-base-uncased, 2019

2019
[14]

Twitter-RoBERTa-base-offensive

CardiffNLP, “Twitter-RoBERTa-base-offensive.” https://huggingface.co/cardiffnlp/ twitter-roberta-base-offensive , 2022

2022
[15]

Xlnet: Generalized autoregressive pretraining for language understanding,

Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. Salakhutdi- nov, and Q. V . Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in Neural Information Processing Systems , NeurIPS 2019, 2019

2019
[16]

Tweeteval: Unified benchmark and comparative evaluation for tweet classification,

F. Barbieri, J. Camacho-Collados, L. Neves, and L. Espinosa-Anke, “Tweeteval: Unified benchmark and comparative evaluation for tweet classification,” arXiv preprint arXiv:2010.12421, 2020

2010 arXiv
[17]

Gemma 3,

Gemma Team, “Gemma 3,” 2025

2025
[18]

Qwen techni- cal report,

J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. ...

2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Bert: Pre-training of deep bidirectional transform- ers for language understanding,

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transform- ers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

arXiv 2018

[2] [2]

Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model

H. Saleh, A. Alhothali, and K. Moria, “Detec- tion of hate speech using bert and hate speech word embedding with deep model,” arXiv preprint arXiv:2111.01515, 2021

work page Pith review arXiv 2021

[3] [3]

Fine-tuning bert for sentiment analysis: A practical guide

H. Amit, “Fine-tuning bert for sentiment analysis: A practical guide.” Medium, Jan 2024

work page 2024

[4] [4]

Language models are few-shot learn- ers,

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...

work page 2020

[5] [5]

A cu- rated hate speech dataset (version 1),

D. Mody, Y . Huang, and T. E. A. de Oliveira, “A cu- rated hate speech dataset (version 1),” 2022

work page 2022

[6] [6]

Challenges and limitations of zero-shot and few-shot learning in large language models,

C. Luca, “Challenges and limitations of zero-shot and few-shot learning in large language models,” inTrends, Applications, and Challenges of Large Language Mod- els, pp. 131–151, IGI Global, 2023

work page 2023

[7] [7]

Hatespeechbert: Retraining bert for automatic hate speechdetection,

A. Ishaq, K. Malik, and A. Zafar, “Hatespeechbert: Retraining bert for automatic hate speechdetection,” SSRN Electronic Journal, 01 2023

work page 2023

[8] [8]

Now you see me: Vision-enhanced bert for obfuscated text abuse detection,

D. Zhou, “Now you see me: Vision-enhanced bert for obfuscated text abuse detection,” 2023. CS224N Custom Project, Natural Language Processing with Deep Learning

work page 2023