{"id":"495c42dc-7ea0-4402-9135-4296f7dff134","arxiv_id":"2608.12889","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Classical smishing detectors degrade near-catastrophically under character and spacing attacks (RDR up to 0.988) while transformers stay below 0.36, showing architecture rather than clean-text accuracy drives adversarial robustness.","lead":"This paper tests how five SMS phishing (smishing) detectors hold up when attackers obfuscate characters, shuffle spacing, or switch between English and Swahili. It finds classical word-based models collapse hard while multilingual transformers degrade far less, and clean-test accuracy does not predict attack resilience.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-tier RDR comparison uses mismatched attack protocols (generic black-box vs. attention-guided top-K with unquantified budgets), so the architectural boundary may be an artifact of attack strength rather than model class.","rationale":"The reader's weakest_assumption is precisely the load-bearing concern, and the paper's own text supports it. Section 5.4 reveals that generic attacks on transformers were ineffective, and the protocol was switched to attention-guided targeting; Section 5.3 leaves the classical attack budget unspecified. This makes the headline numerical contrast (RDR up to 0.988 vs. 0.351) uninterpretable as an architectural comparison. Secondary issues reinforce the need for conditional revision: about 71 malicious test messages with no confidence intervals, non-independent observations in the Mann-Whitney test, and no released code or data. These do not independently decide the verdict because the descriptive gap is large, but they are consistent with CONDITIONAL rather than ACCEPT. I recommend no change to the reader's verdict: the paper should be revised with matched threat models, reported edit budgets, and uncertainty quantification before the central claim is accepted.","tokens_in":9683,"tokens_out":9630,"duration_ms":104510,"concrete_test":"Run a crossed protocol check on the same 80/20 split: (A) identical generic Gemini perturbations with exactly K = 3, 6, and 10 edits per message for all five models; (B) importance-targeted top-K attacks with the same K (TF-IDF feature weight for Random Forest and XGBoost, gradient saliency for CNN+BiLSTM, attention weight for mBERT and XLM-RoBERTa). If the classical-vs-transformer worst-case RDR separation in Tables 2-4 is reproduced under both (A) and (B), the boundary is protocol-independent; if the gap shrinks or reverses under either protocol, the headline claim is an artifact of unequal attack strength.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that architecture determines robustness rests on a comparison between two protocols that are not severity-matched. Section 5.3 attacks classical models with generic black-box Gemini perturbations, but Section 5.4 attacks transformers with attention-guided top-K targeting (K = 3, 6, 10) because, as the paper states, generic attacks on transformers produced RDR < 0.03. The protocols differ in threat-model access (black-box vs. white-box token selection) and, more importantly, in perturbation budget: K is defined only for transformers, and Section 5.3 reports no per-message substitution counts for classical models. The RDR values in Tables 2-4 are therefore not directly comparable, and the Mann-Whitney U / Cliff's delta analysis in Section 6.4 inherits the mismatch: it compares numbers produced under different attack rules, so it tests the combined effect of architecture and protocol, not architecture alone. The transformer results could also understate true fragility because attention weight is an acknowledged imperfect proxy for causal importance; a stronger targeting strategy might raise transformer RDR. Conversely, the classical near-catastrophic results could reflect much larger edit counts than K = 10. The paper's own future-work list includes a random-token baseline, but not a matched cross-tier budget. Without a matched-protocol comparison, the claimed architectural boundary is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates the adversarial robustness of five smishing detection models — Random Forest, XGBoost, CNN+BiLSTM, mBERT, and XLM-RoBERTa — under three attack strategies (character obfuscation, structural perturbation, code-switching) at three intensity levels, using the Robustness Degradation Ratio (RDR) computed on a merged English-Swahili dataset of 27,037 messages. The authors report a sharp architectural boundary: classical models degrade catastrophically (RDR up to 0.988) while transformers remain comparatively robust (RDR up to 0.351), with structural perturbation being the transformers' worst-case attack. They also report a within-tier robustness inversion (XLM-RoBERTa degrades more than mBERT despite a higher clean-text baseline) and use Mann-Whitney U, Cliff's delta, and Friedman tests to argue that the differences are attributable to architecture rather than sampling.","tokens_in":9867,"tokens_out":6350,"duration_ms":57115,"significance":"If the central claim were established, the paper would make a useful contribution to smishing detection and to the broader literature on architecture-dependent adversarial robustness in low-resource NLP: it offers a concrete RDR-based evaluation methodology, explicit attack taxonomies, and evidence that clean-text F1 does not predict adversarial robustness. The manuscript is commendably transparent about its main methodological choice (attention-guided attacks for transformers after generic attacks produced negligible degradation) and about its limitations (small malicious test set, synthetic attacks, attention as an imperfect importance proxy). However, the cross-tier comparison — the basis for the headline architectural boundary — is confounded by the different attack protocols used for the two tiers, and the statistical validation does not properly account for the non-independence of observations. The paper's strengths (clear tables, internal consistency, explicit limitations) do not outweigh the need for a matched-protocol or severity-calibrated comparison.","major_comments":[{"comment":"The cross-tier comparison that grounds Hypothesis 1 is confounded by the attack protocol. Section 5.3 states that classical models receive generic black-box Gemini perturbations, while Section 5.4 states that transformers receive attention-guided top-K targeting (K=3, 6, 10) because preliminary experiments on the same transformer models showed RDR < 0.03 under generic attacks. The RDR values in Tables 2–4 therefore compare performance under two different threat models (black-box vs. white-box token selection) with different, unquantified perturbation budgets: no per-message substitution counts are reported for the classical attacks, whereas K is defined only for the transformer tier. The Mann-Whitney U and Cliff's delta analysis in Section 6.4 inherits this mismatch, so it tests the combined effect of architecture and protocol, not architecture alone. To establish the claimed architectural boundary, the authors need either a matched-protocol condition (e.g., identical substitution counts and identical access across tiers, or a per-message budget reported for both tiers) or an explicit severity calibration that justifies comparing the two protocols; without this, the near-catastrophic classical RDR values may reflect larger edit counts rather than architectural fragility.","section":"§5.3–5.4, Table 4, §6.4"},{"comment":"The Mann-Whitney U test treats each model-attack combination as an independent observation (n1=9, n2=6), but these observations are not independent: repeated observations come from the same model and from the same attack family (character obfuscation, structural, code-switching). The independence assumption is therefore violated, so the reported p=0.0024 is overconfident. Similarly, the Friedman test of Equation (5) uses only n=3 repeated measures, yielding chi-square=6.0 and p=0.050 for all classical models; the paper acknowledges the marginal p-value, but the accompanying claim that the rank ordering 'confirms' the non-linear degradation threshold as an architectural property goes beyond what this test can support. A clustered or mixed-effects analysis, or a per-model summary statistic entered once per model, would be more appropriate for the cross-tier comparison.","section":"§6.4, Eqs. (3)–(4)"},{"comment":"The malicious test partition contains approximately 71 messages, so the F1 values in Tables 2 and 3 are based on very small counts; a single misclassification can shift F1 by more than the differences underlying the within-tier robustness inversion (e.g., mBERT vs. XLM-RoBERTa under high-intensity character obfuscation: 0.1515 vs. 0.2326). The paper notes this limitation in Section 5.1 and lists bootstrap confidence intervals as future work, but the claims in Section 6.3 (e.g., 'that inversion supports the hypothesis...') are presented without any uncertainty quantification. Because RDR is a ratio of two noisy F1 values, the authors should report confidence intervals or exact counts, at least for the high-intensity comparisons that drive RQ3, before interpreting small RDR differences as evidence of an inversion.","section":"§5.1, Tables 2–3, §6.3"}],"minor_comments":[{"comment":"There are multiple spacing and punctuation errors in the literature review (e.g., 'obfuscationintroduces', 'perturbationsmanipulate', and a missing space after 'while maintaining human readability [3,7]; Conventional systems...'); please proofread the text.","section":"§2"},{"comment":"The caption reads 'T able 1' instead of 'Table 1'.","section":"Table 1 caption"},{"comment":"The quality-control description says prompts included 'safeguards preserving phone numbers and names', but it does not specify how these safeguards were implemented or verified; please clarify whether they were enforced by the generation prompt, by post-hoc filtering, or by manual inspection.","section":"§5.5"},{"comment":"The mechanistic explanation for XLM-RoBERTa's higher degradation invokes 'sharper attention concentration', but no attention-concentration metric (e.g., entropy of the attention distribution) is reported; please add a quantitative measure or soften this claim.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The protocol mismatch is acknowledged in the manuscript itself (Section 5.4), which is to the authors' credit but also means the limitation is not hidden. I recommend requesting a matched-protocol experiment (e.g., applying identical substitution-count attacks to both tiers, or reporting classical per-message budgets and applying attention-guided selection to the classical tier via proxy importance measures). The statistical test issue (non-independent observations) is fixable by a clustered analysis or by using one summary statistic per model. The paper's scope is appropriate for a venue interested in empirical security evaluations, but the headline claim needs to be reworked to reflect the protocol-dependent nature of the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is an honest and clearly written empirical study, and it gives the low-resource smishing community something useful—a first systematic comparison of classical vs transformer detectors under three attack families at graded intensities, in a small English-Swahili corpus. The RDR framing is sensible, the raw tables are there, and the performance-robustness decoupling (XLM-R has higher clean F1 but degrades more than mBERT under targeted attack) is a genuinely interesting data point. The authors also deserve credit for stating their limitations plainly, including the small test set and the attention-proxy caveat.\n\nThat said, the paper's central claim—that architecture, not attack protocol, drives the robustness gap—is not actually supported by the experiments as designed. Classical models are hit with generic black-box Gemini perturbations; transformers get attention-guided top-K targeting, and the authors tell you why: generic attacks produced RDR below 0.03 on transformers. That means the two tiers were evaluated under different threat models and, more importantly, under different, uncalibrated perturbation budgets. The RDR values in Tables 2–4 are therefore not directly comparable. The Mann-Whitney and Cliff's delta in Section 6.4 inherit the mismatch; they compare numbers produced under different attack rules, so the test is not isolating architecture. The stress-test note is right: a matched-protocol comparison (same edit budgets, random-token baseline) is the missing control. The paper itself lists a random-token baseline as future work, but not matched cross-tier budgets.\n\nOther soft spots are smaller. The malicious test set is about 71 messages; no confidence intervals, and the Friedman test uses three attack types as repeated measures, yielding marginal p=0.050. The Mann-Whitney treats model-attack combinations as independent when they share the same models and test set. None of this is fatal to the descriptive patterns, but it rules out the stronger language in the abstract about statistical validation confirming architectural causation.\n\nBottom line: a solid empirical starting point with an overclaimed headline. The right fix is revision, not rejection: rerun with matched attack budgets and bootstrap CIs, release the code and data, and re-frame the conclusion as a descriptive finding. I'd send it to peer review, and I'd expect a serious referee to push on exactly this protocol mismatch before acceptance.","headline":"A transparent, well-written empirical comparison of classical vs transformer smishing detectors, but the headline robustness gap is inflated by mismatched attack protocols across tiers.","tokens_in":10439,"tokens_out":3008,"would_cite":false,"duration_ms":29422,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Smishing-model robustness is set by architecture, not by clean-text accuracy.","keywords":["smishing detection","adversarial robustness","Robustness Degradation Ratio","character obfuscation","structural perturbation","code-switching","transformer robustness","classical machine learning"],"falsifier":"Apply one matched attack protocol to all five models, with the same perturbation types, the same number of modified tokens, and the same intensity levels, and compare worst-case RDR. If classical models then stay below about 0.35 or transformers rise above about 0.9, the claimed architectural boundary is an artifact of unequal attack protocols.","tokens_in":9441,"feed_emoji":"🛡️","tokens_out":7926,"duration_ms":68143,"temperature":0.7,"pith_summary":"This paper tries to establish that in low-resource smishing detection, model architecture is the dominant determinant of adversarial fragility. Classical lexical models (Random Forest, XGBoost, CNN+BiLSTM) collapse under character obfuscation and structural perturbation, with worst-case RDR up to 0.988, while transformer models (mBERT, XLM-RoBERTa) stay far more resilient, with worst-case RDR up to 0.351. It also claims that clean-text accuracy does not predict robustness, because XLM-RoBERTa outperforms mBERT on clean text yet degrades more under high-intensity targeted attacks. A sympathetic reader would care because smishing detection in code-switched, low-resource settings is an adversarial problem, and defenses must be tailored to architecture.","feed_headline":"Architecture, not accuracy, predicts smishing-model survival","feed_subtitle":"Classical models lose up to 99 percent of detection quality under attack; transformers lose at most 35 percent.","key_machinery":"The argument is carried by the Robustness Degradation Ratio, $\\mathrm{RDR} = (F1_{\\text{clean}} - F1_{\\text{adv}})/F1_{\\text{clean}}$, which measures relative fragility by normalizing the adversarial F1 drop against each model's own clean-text baseline. The second load-bearing mechanism is the contrast between two feature pipelines: TF-IDF n-gram surface matching in the classical tier and subword tokenization with multilingual embeddings in the transformer tier. To make the transformer evaluation worst-case, the paper uses attention-guided targeting, where the mean final-layer attention weight per token ranks token importance and the top-K tokens are perturbed, with K = 3, 6, 10 for low, medium, and high intensity. The same three attack types (character obfuscation, structural/spacing perturbation, and code-switching) are applied to both tiers.","core_discovery":"The paper finds evidence of a distinct architectural boundary in adversarial smishing detection: under character obfuscation, structural perturbation, and code-switching, classical lexical models show near-catastrophic degradation, with worst-case RDR up to 0.988, while transformers remain substantially more resilient, with worst-case RDR up to 0.351. It further claims that clean-text performance is not a reliable predictor of adversarial robustness: XLM-RoBERTa has the highest clean F1 (0.9565) yet degrades more than mBERT under attention-guided high-intensity attacks (0.351 versus 0.322). The explanation offered is that TF-IDF n-gram surface matching breaks when surface forms are perturbed, whereas subword tokenization and multilingual pretraining preserve the discriminative signal. Statistical tests (Mann-Whitney U, Cliff's d = 0.85, Friedman rank ordering) are used to argue that these patterns come from architecture rather than sampling.","pith_inferences":["Inference: If the robustness gap holds under matched attack protocols, then architecture-specific preprocessing, such as Unicode normalization and character n-gram features for lexical models and attention-dispersing training for transformers, should be prioritized over generic adversarial training in low-resource SMS benchmarks.","Inference: Because multilingual pretraining nearly neutralizes code-switching (RDR below 0.05), fine-tuning a monolingual model on the same corpus and comparing code-switching RDR would isolate how much of the transformer advantage comes from cross-lingual alignment rather than subword tokenization.","Inference: The paper's attention-as-importance assumption could be tested immediately by comparing attention-guided targeting against random top-K token targeting; if random targeting produces comparable degradation, the transformer tier's worst-case fragility may be understated."],"forward_implications":["Deploying TF-IDF-based smishing detectors in low-resource, code-switched settings without input normalization risks near-total failure under character obfuscation and spacing changes, with worst-case RDR up to 0.988.","Transformer models are not immune, but their worst-case degradation stays near 0.35 even under attention-guided attacks, with structural perturbation as their most dangerous attack type.","Clean-text F1 cannot serve as a proxy for adversarial robustness, since XLM-RoBERTa beats mBERT on clean text yet degrades more under high-intensity attacks.","Code-switching is the weakest adversarial lever against transformers (RDR below 0.05) and the strongest against Random Forest (RDR around 0.70), revealing monolingual lexical concentration as the key classical vulnerability.","Statistical tests support architecture as the cause of the separation, though the small malicious test set means the RDR values are indicative of comparative robustness rather than precise deployment-time estimates."],"supporting_citations":[{"why":"Establishes the Swahili smishing classification task and machine-learning baseline this study extends.","marker":"[8]"},{"why":"Supplies the deep-learning approach to Swahili SMS spam that anchors the CNN+BiLSTM classical tier.","marker":"[9]"},{"why":"Provides the multilingual mBERT model and its subword tokenisation used in the transformer tier.","marker":"[11]"},{"why":"Provides the XLM-RoBERTa model and cross-lingual representation learning that the comparison relies on.","marker":"[4]"},{"why":"Documents adversarial attacks that deceive deep-learning fraud SMS detectors, motivating the attack evaluation.","marker":"[2]"},{"why":"Defines imperceptible character-level NLP attacks such as homoglyphs and non-printable characters used as character obfuscation.","marker":"[3]"},{"why":"Supplies a method for generating character-level adversarial text against real-world applications, one source of the obfuscation attack class.","marker":"[7]"},{"why":"Introduces the adversarial polyglot concept underlying the code-switching attack strategy.","marker":"[14]"}],"fun_headline_variants":["Architecture, not accuracy, decides smishing survival","Classical smishing models fail hard; transformers hold","Character attacks crush classical, not transformer, smishing","Smishing robustness: architecture trumps clean performance","Transformers resilient, classical fragile in smishing attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the generic black-box attacks applied to classical models and the attention-guided attacks applied to transformers are equally severe, so the RDR gap is caused by architecture rather than by the different ways the attacks were generated.","fun_headline_variants_meta":{"raw":{"variants":["Architecture, not accuracy, decides smishing survival","Classical smishing models fail hard; transformers hold","Character attacks crush classical, not transformer, smishing","Smishing robustness: architecture trumps clean performance","Transformers resilient, classical fragile in smishing attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2161,"prompt_tokens":1048,"completion_tokens":1113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":1038}},"tokens_in":664,"tokens_out":1113,"duration_ms":9149,"temperature":1.0,"reasoning_tokens":1038,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:14:19.938953+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply one matched attack protocol to all five models, with the same perturbation types, the same number of modified tokens, and the same intensity levels, and compare worst-case RDR. If classical models then stay below about 0.35 or transformers rise above about 0.9, the claimed architectural boundary is an artifact of unequal attack protocols.","supporting_citations":[{"cited_title":"IEEE Access10, 83061– 83074 (2022)","cited_arxiv_id":null,"evidence_quote":"Establishes the Swahili smishing classification task and machine-learning baseline this study extends."},{"cited_title":"IEEE Access12, 25164– 25175 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the deep-learning approach to Swahili SMS spam that anchors the CNN+BiLSTM classical tier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multilingual mBERT model and its subword tokenisation used in the transformer tier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the XLM-RoBERTa model and cross-lingual representation learning that the comparison relies on."},{"cited_title":"In: 2023 17th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)","cited_arxiv_id":null,"evidence_quote":"Documents adversarial attacks that deceive deep-learning fraud SMS detectors, motivating the attack evaluation."},{"cited_title":"In: Proceedings 2019 Network and Distributed System Security Symposium","cited_arxiv_id":null,"evidence_quote":"Supplies a method for generating character-level adversarial text against real-world applications, one source of the obfuscation attack class."},{"cited_title":"In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","cited_arxiv_id":null,"evidence_quote":"Introduces the adversarial polyglot concept underlying the code-switching attack strategy."}],"review_version":1}