{"id":"8d3f4849-0e17-4995-b54b-a6f66dc985f1","arxiv_id":"2505.18233","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A four-stream fusion model for SMS smishing detection reports 97.89% accuracy and 0.963 F1 on 84,000 messages, but its smishing labels were created partly from a keyword list.","lead":"Researchers built a smishing detector that combines four kinds of text clues: country tags, structural patterns, character-level style, and phrase meaning. It reports about 98% accuracy on 84,000 SMS messages, but the labels were created in part from a keyword rule, so the benchmark should be read with caution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label-feature circularity: smishing labels are assigned using the authors' own keyword list [7], and the same smishing-like phrase tags are fed into the contextual feature stream, so the headline accuracy may measure detection of the labeling rule rather than of smishing.","rationale":"The reader's weakest_assumption identifies exactly the risk that matters: the smishing labels are generated by the authors' own keyword list [7], and the contextual stream uses similar phrase tags, so features and labels share a source. I agree with this diagnosis. The paper's strongest claim is not just that the four streams fuse well, but that the result detects evasive smishing; that claim requires labels that are valid independent of the features. The paper gives no evidence for label validity: no human annotation, no external ground-truth comparison, no error analysis, and no baseline against the keyword rule. Without this, the reported 97.89% accuracy is internally consistent but externally unsupported. The ablation results (Table 1) still show that the fused model improves over each single stream on the same labeling; that is a real, if modest, contribution. But the headline 'detecting evasive smishing' is conditioned on label quality. I also note the paper lacks error bars, external baselines, and release of code or full data, which the reader correctly flagged. My check is concrete: re-label a held-out sample independently and see whether the model's performance transfers. If it does not, the verdict should move from conditional to reject; if it does, the conditional verdict can be lifted. Since the reader already arrived at conditional, I recommend no change to the verdict.","tokens_in":4027,"tokens_out":1817,"duration_ms":22278,"concrete_test":"Take a random sample of 2,000 messages labeled smishing by the keyword rule [7] and obtain independent labels through human annotation or original dataset ground truth (e.g., Smishtank or ExAIS labels). Retrain the full model and the Contextual Phrase stream alone on the original features with these independently labeled messages held out as the test set, and report accuracy/F1. If performance collapses toward chance, or if it is no better than a trivial classifier that checks for the presence of keywords from [7], the circularity concern is confirmed. As a secondary check, compare the reported 97.89% accuracy against a keyword-match baseline using list [7] on the same split; the headline claim stands only if the fused model clearly exceeds that baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that the four-stream fusion detects evasive smishing at 97.89% accuracy, 0.963 F1, and 99.73% AUC—depends entirely on the validity of the 24,086 smishing labels. Section 3 states that spam messages were 'further classified as smishing if they contained frequent smishing words [7]', where [7] is the authors' own GitHub keyword list. The same kind of phrase evidence is then encoded as [smishing_like] tags in the Contextual Phrase stream (Section 3.4) and used as a model feature. This creates a circular setup: the label rule and the feature construction are derived from the same unvalidated keyword/phrase source. A model trained on these labels can achieve high accuracy by learning to recognize the presence of those keywords or their embeddings, which is not the same as detecting evasive smishing. The paper provides no evaluation of the keyword rule itself, no external baseline against an independent labeling standard, and no human validation of a sample. Because the abstract and conclusion frame the result as robust region-aware phishing detection, this circularity is load-bearing: if the labels are not a valid proxy, the reported numbers describe detection of a keyword rule, not of evasive smishing. The internal ablation (each stream's contribution) remains meaningful as a comparison among streams on the same labels, but it cannot rescue the external claim without independent label validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a multi-signal smishing detection model that fuses four feature streams: country-specific NER-based semantic tagging, structural pattern tagging, character-level CharCNN, and contextual phrase embeddings from DistilBERT. The authors merge and relabel five public SMS datasets, reporting 24,086 smishing samples, and claim the combined model achieves 97.89% accuracy, 0.963 F1, and 99.73% AUC, outperforming each individual stream. An ablation study is described in which removing any stream degrades performance. The paper also outlines a modular deployment vision with lightweight components for mobile devices.","tokens_in":4322,"tokens_out":4556,"duration_ms":41206,"significance":"If the central claim holds, the paper provides a useful demonstration that fusing heterogeneous feature streams improves SMS phishing detection over single-signal approaches, and the modular architecture is practically appealing for mobile deployment. The authors deserve credit for curating a large combined corpus from five public sources and for constructing a clean, interpretable multi-channel pipeline with an internally consistent ablation ordering. However, the significance is limited by the unvalidated label-generation rule, which creates a potential circularity between features and labels; the external validity of the reported accuracy numbers therefore remains unestablished.","major_comments":[{"comment":"The smishing labels are defined by checking spam messages for 'frequent smishing words [7],' where reference [7] is the authors' own GitHub keyword list, and the same style of phrase evidence is later reintroduced as [smishing_like] tags in the Contextual Phrase stream. This creates a circular benchmark: the label rule and a predictive feature derive from the same unvalidated source. Consequently, the headline 97.89% accuracy may simply measure detection of the keyword rule rather than of evasive smishing. The authors must validate the keyword-based labeling against an independent standard (e.g., human annotation of a random sample, or an external smishing dataset with its own ground truth) and demonstrate that the model also succeeds on labels not derived from the phrase list.","section":"Section 3 (label generation) and Section 3.4"},{"comment":"The performance differences between individual streams are small (96.34%–96.62% accuracy), and the paper reports no confidence intervals, cross-validation, or statistical significance tests. The claims that 'each stream added value' and that ablation caused 'notable performance drops' are unquantified. Please report mean and standard deviation over multiple train/test splits, provide the exact per-ablation accuracies, and include a significance test (e.g., paired bootstrap or McNemar's test) to support the fusion benefit.","section":"Section 4 and Table 1"},{"comment":"The merging of five datasets and the relabeling procedure are described in a single sentence, with no details on the class distribution after merging, deduplication, or how the 80/20 split was performed relative to dataset boundaries. Without this information, the risk of train/test contamination from duplicate messages across datasets is not addressed. Please provide per-dataset counts, a deduplication process, and a description of how the split prevents leakage from overlapping sources.","section":"Section 3 (dataset construction)"},{"comment":"The title claims detection of 'evasive' smishing, but the test set consists of spam messages that contain frequent smishing keywords, and there is no evaluation against adversarial obfuscations, rephrased attacks, or real-world evasive messages. The claim of robustness to evasive attacks is therefore not supported by the current experiments. The authors should either temper the scope of the claim or include an evaluation on a separate evasive-smishing test set.","section":"Title and Section 4"}],"minor_comments":[{"comment":"The phrase 'In such, we present' is grammatically awkward; it should be 'In this work, we present.'","section":"Abstract"},{"comment":"The description 'appending the detected country name to each message' is ambiguous: it is unclear whether the country name is added as a discrete token that the classifier treats as a feature, or as part of the NER tag. Please clarify the exact preprocessing step.","section":"Section 3.1"},{"comment":"The 'curated benign and suspicious phrases' used to generate [legitimate_like] and [smishing_like] tags are not listed or referenced. A sample list and the curation method (e.g., manual review or frequency analysis) are needed for reproducibility.","section":"Section 3.4"},{"comment":"AUC is reported as a percentage (e.g., 99.73%). Please state whether this is AUC-ROC or AUC-PRC, and ensure the metric is defined identically across all rows.","section":"Table 1"},{"comment":"Reference [7] is a GitHub repository. The paper should state explicitly whether the keyword list and the relabeled dataset are publicly released, and under what license, so that the results can be independently reproduced.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The label circularity concern is substantial and load-bearing: the keyword list used to define ground truth is the same style of evidence injected as a feature. In a standard journal review, I would require independent label validation before accepting the reported accuracy. However, the internal ablation is meaningful, so the paper can be salvaged with the requested revisions. As a 2-page WiSec poster, the brevity is understandable, but the methodological gaps remain important for archival quality."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's main contribution is a multi-stream fusion for smishing detection: NER-based country tags, structural regex tagging, a CharCNN, and DistilBERT phrase embeddings, trained on a merged 84k-message corpus. The ablation showing each stream helps—with the character and structural streams most valuable—is a legitimate internal result. The authors are also honest that the contextual stream alone is weak (88.29% accuracy), which gives the ensemble story some credibility.\n\nBut the headline numbers are not independently supported. The core problem is that smishing labels come from the authors' own GitHub keyword list (ref [7]), and the contextual stream uses similar hand-curated phrase tags ([smishing_like]). Features and labels share the same source, so 97.89% accuracy, 0.963 F1, and 99.73% AUC likely describe detection of the labeling rule, not of evasive smishing. There is no external baseline, no comparison to a simple keyword classifier, no cross-validation, no error bars, and no released data. The internal ablation is meaningful on the authors' labels, but the external claim is circular.\n\nWhat is actually new is the specific combination of those four channels and the effort to assemble a relatively large relabeled corpus. That's a legitimate moderate extension of existing spam/phishing filters, not a paradigm shift. The modular design (lightweight structural and character channels, heavier semantic channel) is a practical touch worth keeping.\n\nThe soft spots are real and load-bearing. If the keyword rule is a valid proxy, the approach might work; if not, the results describe a self-fulfilling match. The paper does not provide evidence for the rule's validity, no human validation sample, and no comparison against the keyword rule itself. That is the single biggest fix needed.\n\nFor a poster, this is acceptable as a proposal, not as a finished result. I would send it to peer review with the expectation of heavy revision: validate or replace the label rule, add an external baseline, release code/data, and report error bars. The topic matters, and the ablation has some value, but the headline claim needs to earn its way back in.","headline":"A plausible ensemble for smishing with a real ablation, but the headline accuracy is defined by the authors' own keyword list, making the central claim circular; useful as a work-in-progress poster, not as a reliable detector.","tokens_in":4886,"tokens_out":2265,"would_cite":false,"duration_ms":22394,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fusing semantic, structural, character-level, and phrase-context signals into one architecture detects evasive smishing at 97.89% accuracy, with F1 0.963 and AUC 99.73%, beating each single-stream model.","keywords":["smishing","SMS phishing","multi-signal learning","character-level CNN","named entity recognition","contextual embeddings","mobile security","phishing detection"],"falsifier":"Annotate a random sample of the 24,086 smishing-labeled messages with independent human judges who do not see the keyword list, then compare agreement and measure the model's accuracy on messages containing none of the keyword-smishing words; if performance collapses on keyword-free messages or human agreement is low, the labels—and therefore the 97.89% claim—are artifacts of the keyword rule.","tokens_in":3765,"feed_emoji":"📱","tokens_out":6465,"duration_ms":57732,"temperature":0.7,"pith_summary":"The paper argues that smishing—phishing sent by SMS—evades single-stream detectors because it is short, informal, regionally adapted, and rich in structural tricks. To catch it, the authors build a unified model that fuses four independent views: country-aware semantic tags, structural placeholders for URLs and phone numbers, character-level style, and contextual phrase embeddings. On a relabeled corpus of more than 84,000 messages, the fused model reaches 97.89% accuracy, a 0.963 F1 score, and 99.73% AUC, beating each stream on its own. The practical point is that no one feature type is enough; the combination, and especially the character and structural channels, carries the performance. The stated aim is a modular, region-aware detector that can partly run on-device.","feed_headline":"Four-channel fusion catches smishing at 97.89% accuracy","feed_subtitle":"A model combining country tags, structural placeholders, character style, and phrase context beats any single stream alone.","key_machinery":"The load-bearing machinery is a four-stream feature fusion pipeline. Each message is processed in parallel: (1) an NER semantic stream that replaces countries, organizations, and currencies with abstract tags and adds the detected country name; (2) a regex structural stream that replaces URLs, emails, and phone numbers with placeholders; (3) a character-level CNN (CharCNN) over dense character embeddings; and (4) a DistilBERT contextual stream, with curated phrases tagged [legitimate_like] or [smishing_like], whose [CLS] embedding feeds a CNN. Each stream's vector is reduced with Truncated SVD, the four reduced vectors are concatenated, and a multi-layer perceptron with dropout and attention classifies the result. The mechanism that carries the argument is this late-fusion design: it lets each channel contribute a different kind of evidence instead of forcing one model to learn all cues.","core_discovery":"The paper's central claim is that a four-channel fusion architecture reliably detects evasive smishing and outperforms every single-feature model trained on the same data. In their experiments, the combined model reaches 97.89% accuracy, 0.963 F1, and 99.73% AUC; individually the best stream (character-level CharCNN) reaches 96.62% accuracy, and the contextual phrase stream lags at 88.29%. The authors attribute the gain to complementary signals: NER-based country tags catch region-specific scams, regex structural tags capture intent patterns such as embedded URLs and phone numbers, CharCNN captures obfuscation and emoji, and phrase embeddings catch repeated benign-sounding phishing language. Ablations show measurable drops when any stream is removed, especially character and structural channels.","pith_inferences":["Editorial inference: the per-stream accuracies all cluster near 96%, which suggests the four datasets share strong base-rate or style signals; a harder test would be cross-dataset evaluation where training and test messages come from different sources.","Editorial inference: because the label keyword list is also used to create the [smishing_like] phrase tags, the contextual stream may be partly circular; a cleaned re-run that removes that overlap could lower the fused score and reveal how much of the gain is genuine.","Editorial inference: the country NER results, such as 'UK' appearing predominantly in smishing, point toward a testable regional-tuning claim: a country-specific model that conditions on the detected country should beat the fusion on regional benchmarks."],"forward_implications":["Deploying the fused model in a real SMS gateway should yield higher recall on evasive, region-specific smishing than any single-stream filter it replaces.","Lightweight components such as structural regex tagging and CharCNN can run on-device, while semantic processing is handled in the cloud, so the modular design is compatible with mobile deployment.","The feature streams are modular: replacing the NER tagger or the phrase list lets an operator adapt the same architecture to a new country or language without retraining the whole model.","Ablation results imply that defensive systems should not rely on contextual embeddings alone; character-level and structural cues contribute the largest robustness gains.","Because each stream remains interpretable through country tags, structural placeholders, and phrase tags, the model can give human-readable reasons for flagging a message."],"supporting_citations":[{"why":"Supplies the ExAIS_SMS indigenous dataset that contributes labeled messages to the corpus.","marker":"[1]"},{"why":"Provides the SMS Spam Collection, whose validity is discussed, as a source of spam messages for relabeling.","marker":"[6]"},{"why":"Defines the frequent-smishing-word keyword list used to relabel spam messages as smishing, the source of the labels.","marker":"[7]"},{"why":"Supplies the Real-time SMS dataset used as a source of smishing samples.","marker":"[10]"},{"why":"Supplies the Super dataset, a large corpus split into spam and non-spam, used in the relabeling process.","marker":"[14]"},{"why":"Supplies the Smishtank phishing SMS dataset used as a source of smishing samples.","marker":"[17]"}],"fun_headline_variants":["Fused signals catch evasive smishing at 97.89% accuracy","Multi-channel smishing detector beats all single streams","Region-aware smishing model hits 99.73% AUC","Four-signal fusion defeats evasive SMS phishing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the smishing labels are valid: messages were relabeled as smishing when spam messages contained frequent smishing words from the authors' own keyword list [7], and the same kind of phrase evidence is later injected as [smishing_like] tags in the contextual stream; if that keyword rule is not a valid proxy for smishing, the reported accuracy measures detection of the rule rather than of evasive smishing.","fun_headline_variants_meta":{"raw":{"variants":["Fused signals catch evasive smishing at 97.89% accuracy","Multi-channel smishing detector beats all single streams","Region-aware smishing model hits 99.73% AUC","Four-signal fusion defeats evasive SMS phishing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1297,"prompt_tokens":840,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":456,"tokens_out":457,"duration_ms":3685,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:40:18.601203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a random sample of the 24,086 smishing-labeled messages with independent human judges who do not see the keyword list, then compare agreement and measure the model's accuracy on messages containing none of the keyword-smishing words; if performance collapses on keyword-free messages or human agreement is low, the labels—and therefore the 97.89% claim—are artifacts of the keyword rule.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ExAIS_SMS indigenous dataset that contributes labeled messages to the corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SMS Spam Collection, whose validity is discussed, as a source of spam messages for relabeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the frequent-smishing-word keyword list used to relabel spam messages as smishing, the source of the labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Real-time SMS dataset used as a source of smishing samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Super dataset, a large corpus split into spam and non-spam, used in the relabeling process."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Smishtank phishing SMS dataset used as a source of smishing samples."}],"review_version":1}