{"id":"60352f8d-8b39-412b-b88e-69ff62600c7d","arxiv_id":"2501.00260","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A Naive Bayes classifier with NoSlang dictionary normalization detects smishing messages at 96.2 percent accuracy on a hand-built subset of the SMS Spam Collection.","lead":"This dissertation applies text normalization and a Naive Bayes classifier to detect smishing (SMS phishing) messages, reporting 96.2 percent overall accuracy on a custom dataset. It is a small, incremental contribution within an established line of SMS spam and smishing detection research.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 96.20% accuracy is measured against ham only: arithmetic in §4.2 implies the 456 non-smishing spam messages were dropped, so the claim overstates real-world smishing detection.","rationale":"The reader's conditional verdict focused on manual smishing labels being noisy and unreleased. I agree that is a problem, but the more fundamental issue is the missing control class. The dataset arithmetic is objective: the original SMS Spam Collection has 4,827 ham and 747 spam; the final dataset has 4,807 ham and 362 smishing, with 71 Pinterest messages added. This implies 456 original spam messages are unaccounted for and were almost certainly dropped rather than kept as negative examples. If those 456 messages are absent, the experiment measures smishing-vs-ham discrimination, not smishing detection in a realistic SMS stream where ordinary spam is the most confusable category. This is not a disagreement with external consensus; it is an internal validity gap in the evaluation. The missing-class concern is more load-bearing than label noise because even a perfectly labeled smishing set would not fix it: the classifier is never tested against the closest non-smishing category. The concrete check would settle the issue by rerunning the same pipeline with the 456 non-smishing spam included and with cross-validation. If the 96.20% accuracy survives that test, the conditional acceptance is justified. If it does not, the claim should be rejected or heavily qualified. Since the reader already set CONDITIONAL, I do not move the verdict; I only sharpen the condition that must be met.","tokens_in":33625,"tokens_out":7035,"duration_ms":73019,"concrete_test":"Reconstruct the dataset per §4.2: start with SMS Spam Collection v.1, apply the author's manual smishing criteria to label each of the 747 spam messages as smishing or non-smishing, add the 71 Pinterest messages, and release the resulting 362-message smishing list. Then rerun the same Naive Bayes + NoSlang preprocessing pipeline with (a) the non-smishing spam retained as a third class or as negative examples and (b) 5-fold cross-validation instead of one 90/10 split. Report per-class precision/recall and overall accuracy on the full stream. If accuracy or smishing F1 drops materially below 96.20%, or if ordinary spam is frequently classified as smishing, the headline claim does not transfer to real SMS traffic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the normalization-plus-Naive-Bayes pipeline detects smishing at 96.20% accuracy. Section 4.2 builds the dataset from SMS Spam Collection v.1 (4,827 ham, 747 spam) plus 71 Pinterest smishing messages, and reports a final dataset of 4,807 ham / 362 smishing = 5,169 messages. The arithmetic forces the conclusion that roughly 456 original spam messages that were not manually labeled smishing were removed from the evaluation. The paper never states what happened to them. Because the paper itself defines smishing as a subset of spam (§1.1), ordinary spam (offers, subscriptions, advertisements) is the closest non-target category in real SMS traffic and the most likely source of false smishing alarms. By deleting those 456 messages, the evaluation reduces the problem to smishing-vs-ham, not smishing-vs-everything-else. The reported TPR/TNR therefore cannot support the conclusion that the model 'detects smishing' in a real stream; at best it separates smishing from ham. A secondary but related weakness is the unvalidated, unreleased smishing label set: there is no labeling criterion or inter-annotator agreement, so the selected 362 messages cannot be audited. Even if every label is correct, the missing-class design remains. The fragility is compounded by a single 90/10 split in which the smishing test set is only 35 messages, making the 97.14% TPR equal to 34/35.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript, a Master's dissertation, proposes a content-based smishing detection model. The pipeline first tokenizes, lowercases, normalizes slang/abbreviations via the NoSlang dictionary, removes stop words, and stems, then applies a Naive Bayes classifier to label messages as smishing or ham. Experiments on a custom dataset built from the SMS Spam Collection v.1 plus 71 Pinterest smishing messages report an overall accuracy of 96.20%, a smishing true positive rate of 97.14%, a ham true negative rate of 96.12%, and an 8% accuracy improvement attributable to preprocessing and normalization. The dissertation also contains a broad survey of mobile phishing attacks and defense mechanisms.","tokens_in":33864,"tokens_out":4972,"duration_ms":47094,"significance":"If the experimental result is valid, the paper would offer a simple, privacy-preserving preprocessing contribution to smishing detection and one of the few dedicated smishing datasets. The approach is falsifiable and uses an external dictionary and a held-out test split, so there is no circular derivation. However, the evaluation design has a missing-class problem, the smishing label set is not auditable, and the performance estimate rests on a single small test split; these issues must be resolved before the claimed accuracy can be considered established. The comprehensive survey is a useful reference for researchers entering the area, but the experimental contribution is the central element that needs strengthening.","major_comments":[{"comment":"The dataset arithmetic is inconsistent with the evaluation. Starting from 4,827 ham and 747 spam messages and adding 71 Pinterest smishing messages, the paper reports a final dataset of 4,807 ham and 362 smishing messages (5,169 total). This implies that 456 original spam messages that were not manually labeled as smishing, and 20 original ham messages, were removed from the evaluation without any stated reason. Because the paper defines smishing as a subset of spam (Sections 1.1 and 1.3), ordinary spam is the closest non-target class in real SMS traffic and the most likely source of false alarms. By deleting those 456 messages, the evaluation reduces the problem to smishing-versus-ham rather than smishing-versus-everything-else, so the reported 96.20% accuracy does not support the abstract's claim that the model detects smishing in a real message stream. Please report the exact dataset construction steps, state what happened to the excluded messages, and evaluate the model on a test set that includes ordinary spam as a negative class.","section":"Section 4.2, Dataset Used"},{"comment":"The experimental protocol is fragile: a single 90/10 split produces a test set with only 35 smishing messages (362 × 0.10), so the reported TPR of 97.14% corresponds to 34 correct classifications out of 35. One misclassification changes the TPR by roughly 2.9 percentage points, and the claimed 8% normalization gain (88.20% to 96.20%) is based on this single split with no cross-validation, no repeated runs, and no error bars. Please provide K-fold cross-validation or repeated random splits with confidence intervals, and report the raw confusion matrix counts.","section":"Section 4.3, Results and Discussions"},{"comment":"The smishing labels were created manually by the author from the SMS Spam Collection's spam messages and Pinterest images, but no labeling criteria, annotation guidelines, or inter-annotator agreement are provided, and the labeled dataset is not released. Without an explicit and reproducible labeling rule, the 362 smishing messages cannot be audited, and the reported TPR/TNR may not transfer to other SMS corpora. Please state the labeling rule (e.g., presence of a request to disclose credentials or click a link combined with a sense of urgency), and consider releasing the labeled data or a random sample for independent inspection.","section":"Section 4.2, Dataset Used"},{"comment":"The statement that the proposed model achieves the 'highest classification accuracy and True positive rate among all the schemes discussed' is not supported by the comparison in Table 4.8, because the accuracy numbers are drawn from different datasets and experimental setups, such as S-Detector on a Korean dataset and SMSAssassin on a crowdsourced 4,318-message corpus. Accuracy values across different test sets are not directly comparable; the table should be either restricted to re-implementations on the same data or clearly labeled as indicative only and excluded from the conclusion.","section":"Table 4.8 and Section 5.1"}],"minor_comments":[{"comment":"The word probability estimates use unsmoothed counts (w_ham divided by total ham messages), which can assign zero probability to unseen words and destabilize classification with a small smishing vocabulary; consider adding Laplace smoothing and discussing its effect.","section":"Section 3.1.2, Algorithm 2"},{"comment":"The title promises 'prevention,' but the proposed model only classifies messages as smishing or ham; no blocking, alerting, or quarantine mechanism is described or evaluated. Please adjust the title or add a prevention component to match the stated scope.","section":"Title and Section 3.1"},{"comment":"The claim that 'Vishing has a comparatively higher success rate than other mobile phishing methodologies' is presented without a supporting citation; please add a reference for this assertion.","section":"Section 2.4.1.2"},{"comment":"The tables report only word probabilities without the underlying raw counts, so the reader cannot assess the stability of the probability estimates or the effect of the 90/10 split; please include the count information or specify the number of documents containing each term.","section":"Tables 4.3–4.6"},{"comment":"The paper does not explain why the number of ham messages decreased from 4,827 in the original corpus to 4,807 in the final dataset; please clarify whether 20 ham messages were removed and, if so, why.","section":"Section 4.2, Dataset Used"},{"comment":"The literature review is disproportionately long relative to the experimental contribution, and it contains many reproduced statistics without recent sources; condensing this material and focusing on directly related smishing detection works would improve the manuscript's focus.","section":"Chapter 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an M.Tech dissertation and is being considered in a form that retains thesis-style structure and a very long literature review. The core experimental claim is not yet supported because of the missing-class evaluation and the unvalidated manual labels; both are fixable with additional experiments and reporting. The comparison table's claim of superiority over prior work is also not defensible as written. Given the journal context, I recommend major revision rather than rejection, but the authors should be asked to either re-run the evaluation including ordinary spam as a negative class or substantially soften the claims, and to provide cross-validated performance estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a 2018 M.Tech dissertation posted to arXiv in 2024. It proposes a straightforward content-based smishing detector: NoSlang normalization plus Naive Bayes. The reported 96.20% accuracy is not credible as stated, because the evaluation quietly removes the non-smishing spam messages. Section 4.2 starts with 4,827 ham and 747 spam from SMS Spam Collection v.1, adds 71 Pinterest smishing messages, and ends with 4,807 ham and 362 smishing. The arithmetic forces 456 original spam messages to be dropped, and the paper never mentions them. Since the paper itself defines smishing as a subset of spam (Section 1.1), those 456 spam messages are exactly the nearest non-target class in real SMS traffic. The test therefore measures smishing-vs-ham, not smishing-vs-everything-else. The TPR/TNR numbers cannot support the conclusion that the model detects smishing in a real stream.\n\nWhat is worth credit: the paper is clearly written, the preprocessing pipeline is described in enough detail (tokenization, lowercasing, NoSlang lookup, stop-word removal, stemming) to reimplement, and the use of an external slang dictionary is a sensible idea that directly extends Almeida et al. The author also compares against prior smishing detectors and reports both with and without normalization, which helps isolate the effect. The claim that normalization improves accuracy from 88.20% to 96.20% is the kind of ablation that is useful, even if the absolute numbers are inflated by the missing-class design.\n\nThe soft spots beyond the dropped spam are the manual smishing labels created by the author with no stated criteria or inter-annotator agreement, labels not released; a single 90/10 split with no cross-validation or error bars; and the smishing test set being only 35 messages, making the 97.14% TPR equal to 34/35. The circularity burden is low: word probabilities are estimated on a training split and evaluated on a holdout, and NoSlang is external. The novelty is modest — this is a routine extension of cited work.\n\nBottom line: the paper is a competent student dissertation, not a research breakthrough. The central empirical conclusion is unsupported as stated because of the dropped class. With the label set and evaluation fixed, the result would be a minor contribution. I would not send this to peer review as is; a serious referee would spend most of the report on the dataset design. It could be useful as a teaching example of evaluation pitfalls, but I would not cite it.\n\nRecommendation: desk reject for a top venue; if a venue handles incremental work and the author releases the smishing message list and runs a proper evaluation with all spam retained, it might become a workshop-level paper.","headline":"A competent student dissertation whose headline 96.2% accuracy is inflated by silently dropping all non-smishing spam from the evaluation, leaving a smishing-vs-ham test that does not transfer to real SMS traffic.","tokens_in":34445,"tokens_out":2294,"would_cite":false,"duration_ms":22514,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that replacing SMS slang, abbreviations, and short forms with standard words before classification is the key to detecting smishing, and that this preprocessing step lifts Naive Bayes accuracy on its custom dataset from…","keywords":["smishing detection","SMS phishing","text normalization","Naive Bayes","spam filtering","content-based analysis","NoSlang dictionary","SMS security"],"falsifier":"Train and test the exact Phase 1-to-Phase 2 pipeline on an independently labeled corpus of current real smishing messages and recent ham SMS, using the same 90/10 split. If overall accuracy does not approach 96.20%, or if the normalization stage fails to reproduce the eight-point gain over the no-normalization baseline, the central claim fails. A cheaper check is to have two independent annotators relabel the 362 smishing messages in the paper's dataset and measure label agreement; low agreement would mean the accuracy numbers are not trustworthy.","tokens_in":33354,"feed_emoji":"📱","tokens_out":5323,"duration_ms":49631,"temperature":0.7,"pith_summary":"Smishing is SMS-based phishing: a text message engineered to steal credentials or install malware. The paper claims that such messages can be reliably distinguished from ordinary ham messages by content alone if the text is first cleaned and normalized. It proposes a two-phase model, where phase one tokenizes, lowercases, expands slang and abbreviations using the NoSlang dictionary, removes stop words, and stems; phase two feeds the normalized text to a Naive Bayes classifier. On a manually assembled dataset of 5,169 English messages, the model reaches 96.20% overall accuracy, with 97.14% smishing recall and 96.12% ham recall, beating its own no-normalization baseline by eight points. A reader would care because it suggests a simple, privacy-preserving, content-only route to blocking smishing messages before a user ever clicks a malicious link.","feed_headline":"Smishing detector hits 96.2% by normalizing SMS slang","feed_subtitle":"Normalizing SMS slang lifts accuracy by eight points over the no-normalization baseline.","key_machinery":"The load-bearing mechanism is the preprocessing and normalization pipeline of Phase 1, centered on lookup in the NoSlang lingo dictionary, an online slang and abbreviation-to-standard dictionary. Every token is lowercased; if it appears in the dictionary it is expanded to its standard form, otherwise kept as is; stop words are removed using the NLTK stop-word list; and stemming merges inflected variants. Phase 2 then computes word-level ham and smishing probabilities from the normalized training corpus and classifies a message with the Naive Bayes posterior rule, choosing smishing whenever the smishing probability exceeds the ham probability. The paper reports that normalization raises the smishing probability of terms such as 'call' from 0.443425 to 0.464832, illustrating the mechanism by which standardization concentrates probability mass on discriminative words.","core_discovery":"The paper's central discovery is that normalizing noisy SMS text before classification is what drives detection performance. Trained without preprocessing and normalization, the Naive Bayes classifier achieves 88.20% accuracy; with the normalization stage it achieves 96.20% accuracy, with a true positive rate of 97.14% and a true negative rate of 96.12%. The improvement happens because terms like 'call', 'claim', and 'offer' acquire sharper smishing-versus-ham probability estimates after slang and short forms are expanded to standard forms. The paper further claims that this normalization component is what distinguishes its model from earlier content-based smishing detectors in its comparison table.","pith_inferences":["Beyond the paper: the labeling protocol is unspecified, so if the 362 smishing labels were built from obviously fraudulent spam plus Pinterest examples, the model may be learning a dated and narrow dialect of smishing language rather than a general one.","Beyond the paper: a testable extension is to run the same pipeline on an independently labeled corpus of current real-world smishing messages; the claim would be much stronger if the eight-point normalization gain reproduces there.","Beyond the paper: an adversarial consequence the author does not explore is that attackers can evade the classifier by writing smishing messages in normalized, ham-like language, since the classifier relies on content words alone.","Beyond the paper: the dataset is class-imbalanced with 4,807 ham versus 362 smishing messages, so a trivial ham-only classifier already scores about 93% accuracy; the metrics that matter are smishing recall and the gain over that ham-only baseline."],"forward_implications":["If the claimed numbers hold, a content-only pipeline with no URL analysis, no blacklists, and no access to sender identity can block smishing messages with 96.20% accuracy and a 97.14% smishing detection rate.","The eight-point gain from 88.20% to 96.20% implies that slang normalization, not just feature selection or classifier choice, is the main driver of performance on this dataset.","Because classification uses only the message text after local preprocessing, the scheme can run on-device and preserve user privacy, consistent with the paper's stated privacy objective.","With 96.12% ham accuracy, roughly 3.88% of legitimate messages are flagged as smishing, so a phone with heavy legitimate SMS traffic would see about 1 in 26 ham messages blocked or quarantined.","Since the paper treats smishing as a subset of spam, the same normalized Naive Bayes pipeline should apply to general SMS spam filtering, though the paper does not test that extension."],"supporting_citations":[{"why":"Provides the SMS Spam Collection v.1 dataset of 5,574 labeled ham and spam messages from which the paper manually extracts its smishing set.","marker":"[114]"},{"why":"Supplies the NoSlang lingo dictionary used in Phase 1 to replace slang and abbreviations with standard forms, the load-bearing step of the model.","marker":"[113]"},{"why":"S-Detector is the content-based Naive Bayes smishing detection baseline that the paper extends by adding text normalization.","marker":"[7]"},{"why":"SMSAssassin is the Bayesian-learning SMS spam filter used as a comparison baseline for the classification results.","marker":"[80]"},{"why":"Almeida et al. provide the prior text-normalization-and-semantic-indexing result that motivates the paper's claim that normalizing short forms improves SMS classification.","marker":"[83]"},{"why":"Lee et al. supply the cloud-based smishing detection scheme that appears in the paper's comparative accuracy table.","marker":"[87]"},{"why":"Pinterest smishing message images are the source of the 71 additional smishing messages added to the dataset.","marker":"[118]"}],"fun_headline_variants":["Normalizing SMS slang boosts smishing detection to 96.2%","Slang-aware smishing filter hits 96.2% accuracy","Smishing detection improved by 8 points via slang normalization","SMS slang normalization sharpens smishing detection to 96.2%","How normalizing SMS slang lifts smishing detection to 96.2%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire result rests on the manually built smishing label set: 362 messages called smishing were chosen by one person from a public spam corpus plus 71 Pinterest images, with no stated labeling rule or cross-check, so the 96.20% accuracy could reflect how well the model learns those particular labels rather than smishing in general.","fun_headline_variants_meta":{"raw":{"variants":["Normalizing SMS slang boosts smishing detection to 96.2%","Slang-aware smishing filter hits 96.2% accuracy","Smishing detection improved by 8 points via slang normalization","SMS slang normalization sharpens smishing detection to 96.2%","How normalizing SMS slang lifts smishing detection to 96.2%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2875,"prompt_tokens":836,"completion_tokens":2039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":1945}},"tokens_in":452,"tokens_out":2039,"duration_ms":12547,"temperature":1.0,"reasoning_tokens":1945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:54:13.967450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train and test the exact Phase 1-to-Phase 2 pipeline on an independently labeled corpus of current real smishing messages and recent ham SMS, using the same 90/10 split. If overall accuracy does not approach 96.20%, or if the normalization stage fails to reproduce the eight-point gain over the no-normalization baseline, the central claim fails. A cheaper check is to have two independent annotators relabel the 362 smishing messages in the paper's dataset and measure label agreement; low agreement would mean the accuracy numbers are not trustworthy.","supporting_citations":[{"cited_title":"(2011, March)","cited_arxiv_id":null,"evidence_quote":"SMSAssassin is the Bayesian-learning SMS spam filter used as a comparison baseline for the classification results."},{"cited_title":"A., Silva, T","cited_arxiv_id":null,"evidence_quote":"Almeida et al. provide the prior text-normalization-and-semantic-indexing result that motivates the paper's claim that normalizing short forms improves SMS classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Lee et al. supply the cloud-based smishing detection scheme that appears in the paper's comparative accuracy table."}],"review_version":1}