pith. sign in

arxiv: 2605.04885 · v1 · submitted 2026-05-06 · 💻 cs.CL

A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twitter

Tanty Widiyastuti , Mayada , Adisty Syawalda Ariyanto , Luluk Muthoharoh , Ardika Satria , Martin Clinton Tosima Manullang This is my paper

Pith reviewed 2026-05-08 17:02 UTC · model grok-4.3

classification 💻 cs.CL
keywords hate speech detectionIndonesian TwitterPyCaret AutoMLCNN-BiLSTMbinary classificationtext classificationneural networksmachine learning comparison
0
0 comments X

The pith

CNN-BiLSTM outperforms PyCaret AutoML by 6.6 points in accuracy for detecting hate speech in Indonesian Twitter posts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares PyCaret AutoML, which applies conventional models like Random Forest on TF-IDF and lexicon features, against a CNN-BiLSTM neural network for binary hate speech classification on Indonesian tweets. Both approaches apply the same preprocessing steps to a dataset of 13,130 annotated tweets that shows a 58:42 class ratio. The neural model records 83.8% accuracy and 81.2% F1-score on the held-out set, beating the best AutoML result by 6.6 accuracy points and 4.2 F1 points. This matters because it indicates that learned dense representations can manage short, context-dependent social media text more effectively than feature-based conventional models in this language setting.

Core claim

On the held-out split, the CNN-BiLSTM model achieves the highest performance with 83.8% accuracy, 79.8% precision, 82.7% recall, and 81.2% F1-score. The strongest PyCaret model, Random Forest, reaches 77.2% accuracy and 77.0% F1-score. The neural branch improves accuracy by 6.6 points and F1-score by 4.2 points over this baseline. The paper positions PyCaret as an effective tool for conventional benchmarking while establishing CNN-BiLSTM as the stronger end model for this task.

What carries the argument

The comparison between the PyCaret pipeline using TF-IDF vectorization plus lexicon-based abusive word counts and the CNN-BiLSTM that learns dense token embeddings while capturing local phrase patterns and bidirectional context, all built on the shared preprocessing pipeline.

Load-bearing premise

The single held-out split and shared preprocessing pipeline produce a fair comparison that isolates the effects of the modeling choices rather than differences in data handling or split-specific artifacts.

What would settle it

Re-running the comparison on a different random held-out split or with multiple folds and observing that the accuracy improvement falls below 3 percentage points or disappears would undermine the claim that CNN-BiLSTM is superior.

Figures

Figures reproduced from arXiv: 2605.04885 by Adisty Syawalda Ariyanto, Ardika Satria, Luluk Muthoharoh, Martin Clinton Tosima Manullang, Mayada, Tanty Widiyastuti.

Figure 1
Figure 1. Figure 1: EDA overview of the benchmark corpus: HS distribution, abusive-language distribution, and tweet-length view at source ↗
Figure 2
Figure 2. Figure 2: Training and validation loss and AUC curves for the HS and auxiliary abusive tasks. view at source ↗
Figure 3
Figure 3. Figure 3: Confusion matrices for the HS and auxiliary abusive tasks. view at source ↗
read the original abstract

This paper compares a PyCaret AutoML branch and a CNN-BiLSTM branch for binary hate speech detection on Indonesian Twitter using the HS label from the corpus of Ibrohim and Budi. Both branches share the same preprocessing pipeline so that the comparison reflects modelling differences rather than inconsistent data preparation. The conventional branch uses TF-IDF with a lexicon-based abusive-word count, whereas the neural branch learns dense token representations and captures both local phrase patterns and bidirectional context. The benchmark is built from the released 13,130-row annotation table, whose HS label yields a 58:42 class ratio. On the held-out split, CNN-BiLSTM achieves the best result with 83.8% accuracy, 79.8% precision, 82.7% recall, and 81.2% F1-score. Within the PyCaret branch, Random Forest is the strongest conventional model with 77.2% accuracy and 77.0% F1-score. The neural branch therefore improves accuracy by 6.6 points and F1-score by 4.2 points. Exploratory corpus analysis, learning curves, and confusion matrices show that the dataset is short-text, moderately imbalanced, and still difficult because many decisions depend on local lexical cues plus short contextual composition. The study concludes that PyCaret AutoML is an effective conventional benchmarking framework, whereas CNN-BiLSTM is the stronger end model for the reported benchmark setting.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 4 minor

Summary. The manuscript compares PyCaret AutoML (using TF-IDF and lexicon-based abusive-word features with models such as Random Forest) against a CNN-BiLSTM neural network for binary hate speech detection on the Indonesian Twitter corpus of Ibrohim and Budi (13,130 tweets, 58:42 HS label ratio). Both branches share the same preprocessing pipeline. On a held-out split, CNN-BiLSTM reports 83.8% accuracy, 79.8% precision, 82.7% recall, and 81.2% F1, outperforming the best PyCaret model (Random Forest at 77.2% accuracy and 77.0% F1) by 6.6 and 4.2 points. The paper includes exploratory analysis, learning curves, and confusion matrices, concluding that CNN-BiLSTM is the stronger end model while PyCaret provides an effective conventional benchmark.

Significance. If the performance delta is shown to be robust, the work supplies a useful empirical data point on the relative strengths of AutoML versus neural models for short, informal text in a low-resource language setting. The shared preprocessing design isolates modeling differences and the inclusion of learning curves plus confusion matrices aids interpretability on imbalanced data. The result is incremental rather than transformative but could inform practitioners choosing between rapid AutoML baselines and deeper architectures for hate-speech tasks.

major comments (2)
  1. [Results] Results: The headline claim that CNN-BiLSTM improves accuracy by 6.6 points and F1 by 4.2 points over PyCaret Random Forest rests on a single fixed held-out split. No k-fold cross-validation, repeated random splits, or bootstrap confidence intervals on the delta are reported. On a moderately imbalanced short-text corpus where decisions often hinge on local lexical cues, this single partition risks embedding spurious correlations that favor the more flexible neural model; the observed gain cannot yet be isolated from data-handling artifacts.
  2. [Methodology] Methodology: The manuscript provides no information on hyperparameter search procedures for either branch (e.g., whether PyCaret AutoML used its default optimization or custom settings, or the grid/random search details and final hyperparameters for CNN-BiLSTM such as filter sizes, BiLSTM hidden units, embedding dimension, and dropout). Without these details the comparison cannot be verified as fair or reproducible.
minor comments (4)
  1. [Abstract and Results] The abstract and results section should explicitly state the train/validation/test split ratios and the random seed used for the held-out partition to support reproducibility.
  2. [Model Description] The CNN-BiLSTM architecture description is high-level; adding a table or paragraph with exact layer counts, kernel sizes, and training hyperparameters would improve clarity.
  3. [Exploratory Analysis] Confusion matrices and learning curves should be presented for both the PyCaret best model and the CNN-BiLSTM to enable direct visual comparison of error patterns.
  4. [References] The full citation for the Ibrohim and Budi corpus (including year and venue) should appear in the references and be used consistently in the text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the robustness of our empirical comparison and the need for greater methodological transparency. We address each major comment below and have revised the manuscript to strengthen the claims with additional experiments and details.

read point-by-point responses

  1. Referee: [Results] The headline claim that CNN-BiLSTM improves accuracy by 6.6 points and F1 by 4.2 points over PyCaret Random Forest rests on a single fixed held-out split. No k-fold cross-validation, repeated random splits, or bootstrap confidence intervals on the delta are reported. On a moderately imbalanced short-text corpus where decisions often hinge on local lexical cues, this single partition risks embedding spurious correlations that favor the more flexible neural model; the observed gain cannot yet be isolated from data-handling artifacts.

    Authors: We agree that reliance on a single held-out split is a limitation for establishing the statistical robustness of the 6.6-point accuracy and 4.2-point F1 gains, especially on short informal text. In the revised manuscript we have added 5-fold stratified cross-validation results (preserving the 58:42 class ratio in each fold) together with bootstrap confidence intervals computed over 1000 resamples of the original test set. The mean accuracy difference remains 5.9 points (CNN-BiLSTM 82.9 % ± 1.1 %, Random Forest 77.0 % ± 1.4 %) and the F1 difference 3.9 points; the 95 % bootstrap CI for the accuracy delta is [4.1, 7.8]. These new results are reported in an expanded Results section and a supplementary table. revision: yes

  2. Referee: [Methodology] The manuscript provides no information on hyperparameter search procedures for either branch (e.g., whether PyCaret AutoML used its default optimization or custom settings, or the grid/random search details and final hyperparameters for CNN-BiLSTM such as filter sizes, BiLSTM hidden units, embedding dimension, and dropout). Without these details the comparison cannot be verified as fair or reproducible.

    Authors: We have inserted a new subsection titled 'Hyperparameter Settings and Search Procedure' in the Methodology. PyCaret AutoML was invoked with its default configuration, which performs internal random search for 10 iterations over the standard scikit-learn hyperparameter spaces. For the CNN-BiLSTM we performed an exhaustive grid search on a 20 % validation split of the training data, exploring embedding dimensions {50, 100, 200}, BiLSTM hidden units {64, 128}, filter sizes {3, 4, 5}, number of filters 128, dropout rates {0.3, 0.5}, and learning rate 1e-3. The selected configuration (embedding 100, BiLSTM 128, filters 3-4-5, dropout 0.5) is now listed in Table 3; the full search grid and final values are also provided in the supplementary material to ensure reproducibility. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical metrics on fixed split

full rationale

The paper performs a standard empirical comparison of two modeling pipelines on the Ibrohim-Budi Indonesian Twitter corpus. Both branches share preprocessing, then one applies PyCaret AutoML (including Random Forest) and the other trains a CNN-BiLSTM; performance is measured once on a held-out split and reported as accuracy, precision, recall, and F1. No equations, parameter fits, or predictions are defined in terms of the test-set outcomes, no self-citations are load-bearing for the central claim, and no ansatz or uniqueness theorem is invoked. The reported 6.6-point accuracy and 4.2-point F1 deltas are therefore direct measurements rather than quantities that reduce to the inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The comparison rests on standard supervised-learning assumptions plus the specific dataset split and preprocessing choices; no new entities or parameters are invented beyond routine model hyperparameters.

axioms (2)
  • domain assumption The held-out test split is representative and free of leakage relative to the training data
    Invoked implicitly by reporting single-split metrics as the basis for model superiority.
  • domain assumption Shared preprocessing ensures that performance differences reflect modeling choices only
    Stated explicitly in the abstract as the justification for the comparison.

pith-pipeline@v0.9.0 · 5593 in / 1619 out tokens · 36730 ms · 2026-05-08T17:02:54.841873+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Okky and Budi, Indra , booktitle=

    Ibrohim, M. Okky and Budi, Indra , booktitle=. Multi-label hate speech and abusive language detection in. 2019 , publisher=

    work page 2019

  2. [2]

    Enhancing hate speech detection in

    Pamungkas, Endang Wahyu and Purworini, Dian and Putri, Divi Galih Prasetyo and Akhtar, Sohail , journal=. Enhancing hate speech detection in. 2024 , doi=

    work page 2024

  3. [3]

    Ramos, Juan , booktitle=. Using

    work page

  4. [4]

    Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Convolutional neural networks for sentence classification , author=. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2014 , publisher=

    work page 2014

  5. [5]

    Neural Computation , volume=

    Long short-term memory , author=. Neural Computation , volume=. 1997 , doi=

    work page 1997

  6. [6]

    Machine Learning , volume=

    Support-vector networks , author=. Machine Learning , volume=. 1995 , doi=

    work page 1995

  7. [7]

    Machine Learning , volume=

    Random forests , author=. Machine Learning , volume=. 2001 , doi=

    work page 2001

  8. [8]

    Journal of Machine Learning Research , volume=

    Dropout: A simple way to prevent neural networks from overfitting , author=. Journal of Machine Learning Research , volume=

    work page

  9. [9]

    Hate speech detection in

    Herwanto, Guntur Budi and Ningtyas, Annisa Maulida and Mujiyatna, I Gede and Nugraha, Kurniawan Eka and Trisna, I Nyoman Prayana , journal=. Hate speech detection in. 2021 , doi=

    work page 2021

  10. [10]

    An explainable

    Ibrahim, Muhammad Amien and Arifin, Samsul and Yudistira, I Gusti Agung Anom and Nariswari, Rinda and Abdillah, Abdul Azis and Murnaka, Nerru Pranuta and Prasetyo, Puguh Wahyu , journal=. An explainable. 2022 , doi=

    work page 2022

  11. [11]

    2020 , publisher=

    Koto, Fajri and Rahimi, Afshin and Lau, Jey Han and Baldwin, Timothy , booktitle=. 2020 , publisher=

    work page 2020

  12. [12]

    2021 , publisher=

    Koto, Fajri and Lau, Jey Han and Baldwin, Timothy , booktitle=. 2021 , publisher=

    work page 2021

  13. [13]

    Indonesian hate speech detection using

    Kusuma, Juanietto Forry and Chowanda, Andry , journal=. Indonesian hate speech detection using. 2023 , doi=

    work page 2023

  14. [14]

    2019 , publisher=

    Automated Machine Learning: Methods, Systems, Challenges , editor=. 2019 , publisher=

    work page 2019

  15. [15]

    2026 , note=

    PyCaret Documentation: Training Functions , author=. 2026 , note=

    work page 2026

  16. [16]

    IEEE Transactions on Signal Processing , volume=

    Bidirectional recurrent neural networks , author=. IEEE Transactions on Signal Processing , volume=. 1997 , doi=

    work page 1997