REVIEW 3 major objections 4 minor 15 references
Adversarial Robustness in Smishing Detection: A Comparative Analysis of Adversarial Fragility in Classical vs. Transformer-Based Detection Systems
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Smishing-model robustness is set by architecture, not by clean-text accuracy.
desk verdict A transparent, well-written empirical comparison of classical vs transformer smishing detectors, but the headline robustness gap is inflated by mismatched attack protocols across tiers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by the Robustness Degradation Ratio, $\mathrm{RDR} = (F1_{\text{clean}} - F1_{\text{adv}})/F1_{\text{clean}}$, which measures relative fragility by normalizing the adversarial F1 drop against each model's own clean-text baseline. The second load-bearing mechanism is the contrast between two feature pipelines: TF-IDF n-gram surface matching in the classical tier and subword tokenization with multilingual embeddings in the transformer tier. To make the transformer evaluation worst-case, the paper uses attention-guided targeting, where the mean final-layer attention weight per token ranks token importance and the top-K tokens are perturbed, with K = 3, 6, 10 for low, medium, and high intensity. The same three attack types (character obfuscation, structural/spacing perturbation, and code-switching) are applied to both tiers.
What would settle it
Apply one matched attack protocol to all five models, with the same perturbation types, the same number of modified tokens, and the same intensity levels, and compare worst-case RDR. If classical models then stay below about 0.35 or transformers rise above about 0.9, the claimed architectural boundary is an artifact of unequal attack protocols.
Extended reading notes
Core claim
The paper finds evidence of a distinct architectural boundary in adversarial smishing detection: under character obfuscation, structural perturbation, and code-switching, classical lexical models show near-catastrophic degradation, with worst-case RDR up to 0.988, while transformers remain substantially more resilient, with worst-case RDR up to 0.351. It further claims that clean-text performance is not a reliable predictor of adversarial robustness: XLM-RoBERTa has the highest clean F1 (0.9565) yet degrades more than mBERT under attention-guided high-intensity attacks (0.351 versus 0.322). The explanation offered is that TF-IDF n-gram surface matching breaks when surface forms are perturbed, whereas subword tokenization and multilingual pretraining preserve the discriminative signal. Statistical tests (Mann-Whitney U, Cliff's d = 0.85, Friedman rank ordering) are used to argue that these patterns come from architecture rather than sampling.
Load-bearing premise
The comparison assumes that the generic black-box attacks applied to classical models and the attention-guided attacks applied to transformers are equally severe, so the RDR gap is caused by architecture rather than by the different ways the attacks were generated.
Editorial extensions
If this is right
- Deploying TF-IDF-based smishing detectors in low-resource, code-switched settings without input normalization risks near-total failure under character obfuscation and spacing changes, with worst-case RDR up to 0.988.
- Transformer models are not immune, but their worst-case degradation stays near 0.35 even under attention-guided attacks, with structural perturbation as their most dangerous attack type.
- Clean-text F1 cannot serve as a proxy for adversarial robustness, since XLM-RoBERTa beats mBERT on clean text yet degrades more under high-intensity attacks.
- Code-switching is the weakest adversarial lever against transformers (RDR below 0.05) and the strongest against Random Forest (RDR around 0.70), revealing monolingual lexical concentration as the key classical vulnerability.
- Statistical tests support architecture as the cause of the separation, though the small malicious test set means the RDR values are indicative of comparative robustness rather than precise deployment-time estimates.
Reading between the lines
- Inference: If the robustness gap holds under matched attack protocols, then architecture-specific preprocessing, such as Unicode normalization and character n-gram features for lexical models and attention-dispersing training for transformers, should be prioritized over generic adversarial training in low-resource SMS benchmarks.
- Inference: Because multilingual pretraining nearly neutralizes code-switching (RDR below 0.05), fine-tuning a monolingual model on the same corpus and comparing code-switching RDR would isolate how much of the transformer advantage comes from cross-lingual alignment rather than subword tokenization.
- Inference: The paper's attention-as-importance assumption could be tested immediately by comparing attention-guided targeting against random top-K token targeting; if random targeting produces comparable degradation, the transformer tier's worst-case fragility may be understated.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper evaluates the adversarial robustness of five smishing detection models — Random Forest, XGBoost, CNN+BiLSTM, mBERT, and XLM-RoBERTa — under three attack strategies (character obfuscation, structural perturbation, code-switching) at three intensity levels, using the Robustness Degradation Ratio (RDR) computed on a merged English-Swahili dataset of 27,037 messages. The authors report a sharp architectural boundary: classical models degrade catastrophically (RDR up to 0.988) while transformers remain comparatively robust (RDR up to 0.351), with structural perturbation being the transformers' worst-case attack. They also report a within-tier robustness inversion (XLM-RoBERTa degrades more than mBERT despite a higher clean-text baseline) and use Mann-Whitney U, Cliff's delta, and Friedman tests to argue that the differences are attributable to architecture rather than sampling.
Significance. If the central claim were established, the paper would make a useful contribution to smishing detection and to the broader literature on architecture-dependent adversarial robustness in low-resource NLP: it offers a concrete RDR-based evaluation methodology, explicit attack taxonomies, and evidence that clean-text F1 does not predict adversarial robustness. The manuscript is commendably transparent about its main methodological choice (attention-guided attacks for transformers after generic attacks produced negligible degradation) and about its limitations (small malicious test set, synthetic attacks, attention as an imperfect importance proxy). However, the cross-tier comparison — the basis for the headline architectural boundary — is confounded by the different attack protocols used for the two tiers, and the statistical validation does not properly account for the non-independence of observations. The paper's strengths (clear tables, internal consistency, explicit limitations) do not outweigh the need for a matched-protocol or severity-calibrated comparison.
major comments (3)
- [§5.3–5.4, Table 4, §6.4] The cross-tier comparison that grounds Hypothesis 1 is confounded by the attack protocol. Section 5.3 states that classical models receive generic black-box Gemini perturbations, while Section 5.4 states that transformers receive attention-guided top-K targeting (K=3, 6, 10) because preliminary experiments on the same transformer models showed RDR < 0.03 under generic attacks. The RDR values in Tables 2–4 therefore compare performance under two different threat models (black-box vs. white-box token selection) with different, unquantified perturbation budgets: no per-message substitution counts are reported for the classical attacks, whereas K is defined only for the transformer tier. The Mann-Whitney U and Cliff's delta analysis in Section 6.4 inherits this mismatch, so it tests the combined effect of architecture and protocol, not architecture alone. To establish the claimed architectural boundary, the authors need either a matched-protocol condition (e.g., identical substitution counts and identical access across tiers, or a per-message budget reported for both tiers) or an explicit severity calibration that justifies comparing the two protocols; without this, the near-catastrophic classical RDR values may reflect larger edit counts rather than architectural fragility.
- [§6.4, Eqs. (3)–(4)] The Mann-Whitney U test treats each model-attack combination as an independent observation (n1=9, n2=6), but these observations are not independent: repeated observations come from the same model and from the same attack family (character obfuscation, structural, code-switching). The independence assumption is therefore violated, so the reported p=0.0024 is overconfident. Similarly, the Friedman test of Equation (5) uses only n=3 repeated measures, yielding chi-square=6.0 and p=0.050 for all classical models; the paper acknowledges the marginal p-value, but the accompanying claim that the rank ordering 'confirms' the non-linear degradation threshold as an architectural property goes beyond what this test can support. A clustered or mixed-effects analysis, or a per-model summary statistic entered once per model, would be more appropriate for the cross-tier comparison.
- [§5.1, Tables 2–3, §6.3] The malicious test partition contains approximately 71 messages, so the F1 values in Tables 2 and 3 are based on very small counts; a single misclassification can shift F1 by more than the differences underlying the within-tier robustness inversion (e.g., mBERT vs. XLM-RoBERTa under high-intensity character obfuscation: 0.1515 vs. 0.2326). The paper notes this limitation in Section 5.1 and lists bootstrap confidence intervals as future work, but the claims in Section 6.3 (e.g., 'that inversion supports the hypothesis...') are presented without any uncertainty quantification. Because RDR is a ratio of two noisy F1 values, the authors should report confidence intervals or exact counts, at least for the high-intensity comparisons that drive RQ3, before interpreting small RDR differences as evidence of an inversion.
minor comments (4)
- [§2] There are multiple spacing and punctuation errors in the literature review (e.g., 'obfuscationintroduces', 'perturbationsmanipulate', and a missing space after 'while maintaining human readability [3,7]; Conventional systems...'); please proofread the text.
- [Table 1 caption] The caption reads 'T able 1' instead of 'Table 1'.
- [§5.5] The quality-control description says prompts included 'safeguards preserving phone numbers and names', but it does not specify how these safeguards were implemented or verified; please clarify whether they were enforced by the generation prompt, by post-hoc filtering, or by manual inspection.
- [§7] The mechanistic explanation for XLM-RoBERTa's higher degradation invokes 'sharper attention concentration', but no attention-concentration metric (e.g., entropy of the attention distribution) is reported; please add a quantitative measure or soften this claim.
Circularity Check
No significant circularity; the adversarial-robustness comparison is empirically self-contained.
full rationale
The paper's derivation chain is empirical and self-contained: RDR is defined by Eq. (2) directly from measured clean and adversarial F1 values (Tables 1-3), and the cross-tier comparison is a statistical comparison of those externally measured numbers. No fitted parameter is renamed as a prediction, and no quantity in the RDR definition is set equal to the architectural-boundary conclusion. The only self-citations ([8] and [9], with author Ndibwile) appear in the literature review and research gap as background on Swahili smishing classification, and they are not load-bearing for the adversarial-robustness comparison. The transformer protocol was selected after preliminary experiments showed generic attacks gave RDR below 0.03 (Section 5.4); while this creates a severity-matching concern for the cross-tier comparison, it is an experimental-design limitation, not a circular derivation, and the paper itself flags related limitations in Section 9 (small malicious test set, attention as an imperfect proxy, synthetic attack generation). The central statistical tests (Mann-Whitney U, Cliff's delta, Friedman) operate on the reported RDR values and do not reduce to the hypothesis under test by construction. Therefore no significant circularity is present.
Assumptions & free parameters
free parameters (2)
- Attention-guided top-K token counts =
K = 3 (low), K = 6 (medium), K = 10 (high)
- Gemini per-intensity substitution counts =
not reported
assumptions (4)
- domain assumption The merged 27,037-message corpus with binary labels faithfully represents smishing traffic.
- domain assumption RDR observations are independent for the Mann-Whitney U and Friedman tests.
- domain assumption Final-layer attention weights identify tokens that are most influential to model decisions.
- domain assumption Gemini API-generated perturbations preserve intent, readability, and adversarial validity.
Cite this review
Pith. "Pith review of Adversarial Robustness in Smishing Detection: A Comparative Analysis of Adversarial Fragility in Classical vs. Transformer-Based Detection Systems." pith.science (2026). https://pith.science/paper/FGGAPZG4
@misc{pith2026260812889,
author = {Pith},
title = {Pith review of: Adversarial Robustness in Smishing Detection: A Comparative Analysis of Adversarial Fragility in Classical vs. Transformer-Based Detection Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/FGGAPZG4}},
note = {Machine review of arXiv:2608.12889}
}
read the original abstract
Smishing detection systems are commonly trained and evaluated on clean, monolingual text. In low-resource settings, however, attackers frequently circumvent these systems through character obfuscation, cross-lingual code-switching, and structural perturbation. This study evaluates adversarial robustness for five model architectures: three classical lexical models (Random Forest, XGBoost, CNN+BiLSTM) and two multilingual transformers (mBERT, XLM-RoBERTa), using a dataset of 27,037 messages. Classical models are subjected to black-box generic attacks, while transformers are evaluated with attention-guided targeting. Each model is tested across three attack types and intensity levels, with performance measured by the Robustness Degradation Ratio (RDR). The results reveal a distinct architectural boundary: classical models experience near-catastrophic failure under character obfuscation and structural perturbation (RDR up to 0.988), whereas transformers demonstrate significantly greater resilience (RDR up to 0.351), with structural perturbation representing their most pronounced vulnerability. Effect-size analysis (Cliff's d) indicates a substantial difference between the two model categories. Within the transformer group, XLM-RoBERTa, despite achieving a higher clean-text baseline, exhibits greater degradation than mBERT. These findings demonstrate that clean-text performance is not a reliable predictor of adversarial robustness. Statistical validation using Mann-Whitney U and Friedman tests confirms that these patterns are attributable to model architecture rather than sampling. The results underscore the necessity for architecture-specific defences and frame smishing detection as an adversarial cybersecurity challenge rather than a static classification task.
Figures
Reference graph
Works this paper leans on
-
[1]
International Journal of Research and Innovation in Applied Science10, 1921–1939 (Nov 2025)
Asheshemi, N., Daniel, O., Micheal, O.: Adversarial Robustness in Natural Lan- guage Processing: An Empirical Analysis of Machine Learning Model Vulnerabil- ities to Adversarial Attacks. International Journal of Research and Innovation in Applied Science10, 1921–1939 (Nov 2025)
work page 1921
-
[2]
In: 2023 17th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)
Bajaj, A., Vishwakarma, D.K.: Deceiving Deep Learning-based Fraud SMS Detec- tion Models through Adversarial Attacks. In: 2023 17th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS). pp. 327–332 (Nov 2023)
work page 2023
-
[3]
https://doi.org/10.48550/arXiv.2106.09898
Boucher, N., Shumailov, I., Anderson, R., Papernot, N.: Bad Characters: Imper- ceptible NLP Attacks (Dec 2021). https://doi.org/10.48550/arXiv.2106.09898
-
[4]
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised Cross-lingual Representation Learning at Scale. pp. 8440–8451 (2020)
work page 2020
-
[5]
Procedia Computer Science245, 1165–1174 (2024)
Haizam, M.N.B., Zulkipli, N.H.B.N.: Analysing The Impact of Smishing Attack in Public Announcement System on Mobile Phone. Procedia Computer Science245, 1165–1174 (2024)
work page 2024
-
[6]
Computer Networks255, 110892 (Dec 2024)
Hosseinpour, S., Shakibian, H.: Complex-network based model for SMS spam fil- tering. Computer Networks255, 110892 (Dec 2024)
work page 2024
-
[7]
In: Proceedings 2019 Network and Distributed System Security Symposium
Li, J., Ji, S., Du, T., Li, B., Wang, T.: TextBugger: Generating Adversarial Text Against Real-world Applications. In: Proceedings 2019 Network and Distributed System Security Symposium. Internet Society, San Diego, CA (2019)
work page 2019
-
[8]
IEEE Access10, 83061– 83074 (2022)
Mambina, I.S., Ndibwile, J.D., Michael, K.F.: Classifying Swahili Smishing Attacks for Mobile Money Users: A Machine-Learning Approach. IEEE Access10, 83061– 83074 (2022)
work page 2022
Show all 15 references
-
[9]
IEEE Access12, 25164– 25175 (2024)
Mambina, I.S., Ndibwile, J.D., Uwimpuhwe, D., Michael, K.F.: Uncovering SMS Spam in Swahili Text Using Deep Learning Approaches. IEEE Access12, 25164– 25175 (2024)
2024
-
[10]
Morris, J.X., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (Oct 2020), arXiv:2005.05909 [cs]
2020 arXiv
-
[11]
Pires, T., Schlinger, E., Garrette, D.: How Multilingual is Multilingual BERT? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4996–5001. Association for Computational Linguistics, Florence, Italy (2019)
2019
-
[12]
Mesopotamian Journal of CyberSecurity5(2), 590–605 (Jun 2025)
Sankaine, L., Ndia, J.G., Kaburu, D.: An English-Swahili Email Spam De- tection Model for Improved Accuracy Using Convolutional Neural Networks. Mesopotamian Journal of CyberSecurity5(2), 590–605 (Jun 2025)
2025
-
[13]
International Journal of Innovative Technology and Exploring Engineering14(5), 28–33 (Apr 2025)
Shaikh, A., Shaikh, M., Srivaramangai R.: Smishing Detection: Combating SMS Phishing Attacks by Utilizing Machine-Learning Algorithms. International Journal of Innovative Technology and Exploring Engineering14(5), 28–33 (Apr 2025)
2025
-
[14]
In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Tan,S.,Joty,S.:Code-MixingonSesameStreet:DawnoftheAdversarialPolyglots. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3596–3616. Association for Computational Linguistics
2021
-
[15]
Computers & Security154, 104443 (Jul 2025)
Xu, H., Qadir, A., Sadiq, S.: Malicious SMS detection using ensemble learning and SMOTE to improve mobile cybersecurity. Computers & Security154, 104443 (Jul 2025)
2025
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.