Pith. sign in

REVIEW 3 major objections 4 minor 24 references

Detecting Voice Phishing with Precision: Fine-Tuning Small Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Expert rules beat chain-of-thought in small-model phishing detection

desk verdict Worth a serious referee, but the headline 'comparable to GPT-4' is shakier than the abstract suggests; the core finding about criteria vs. CoT is solid within the paper's own setup. read the letter →

arxiv 2506.06180 v1 pith:UIHQ27VL submitted 2025-06-06 cs.CL

Ju Yong Sim , Seong Hwan Kim This is my paper
classification cs.CL
keywords voicephishingdetectionsmalllanguagemodelsfine-tuningLlama3chain-of-thoughtpromptengineeringadversarialtestdatasetKoBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper develops a voice phishing detector by fine-tuning Llama3-8B, a small open-source language model, and argues that providing 11 handcrafted voice-phishing evaluation criteria in the prompt is more effective than using chain-of-thought prompting. The authors construct their own adversarial test set of 58 transcripts that resemble phishing but are not, in order to expose performance differences between models. Their main result is that the fine-tuned model with criteria-only prompts reaches 94.64% accuracy on this adversarial set at a block length of 500, close to the 99.11% of a GPT-4o-based detector, while the chain-of-thought fine-tuned variant reaches only 81.25% at the same block length. The paper concludes that injecting human expert knowledge into prompts helps small language models more than reasoning chains in this task.

What carries the argument

The mechanism is a two-stage detector: transcripts are split into fixed-length blocks, and a fine-tuned Llama3-8B model scores each block's voice-phishing likelihood (0–10) based on a prompt that includes 11 voice-phishing evaluation criteria condensed from a crime-stage analysis; the weighted average of block scores is compared to a threshold learned on validation data. The criteria, written with logical operators such as "and" and "or," are the carrier of the paper's central claim—they encode human expert knowledge and are what the authors show gives the accuracy boost.

What would settle it

Replicate the paper's fine-tuning and prompting setup, then evaluate on an independently collected, externally labeled set of real voice-phishing calls and near-fraud non-phishing calls; if the criteria-only fine-tuned Llama3 no longer beats the CoT variant or no longer approaches GPT-4o, the central claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a small language model (Llama3-8B, Korean variant) fine-tuned with prompts containing 11 handcrafted voice-phishing evaluation criteria—without any chain-of-thought—achieves the best accuracy among small models on an adversarial test set and is comparable to a GPT-4o-based detector. On the adversarial set, Llama3-FT-Cri reaches 94.64% at block length 500, while Llama3-FT-CoT reaches 81.25% and GPT4o-Cri reaches 99.11%. The paper interprets this as evidence that supplying domain knowledge in the prompt is more beneficial than CoT reasoning for small language models in voice phishing detection.

Load-bearing premise

The entire comparison rests on the adversarial test set being a fair and representative challenge; that set was written by the authors themselves, with no independent expert review or human-label audit reported.

Editorial extensions

If this is right

  • A small, locally runnable model with expert-criteria prompts can deliver near-GPT-4 performance on voice-phishing detection, reducing operational cost and privacy exposure.
  • Chain-of-thought prompting, at least in this form, does not help small LMs on this task and can even hurt; adding CoT on top of criteria reduces accuracy.
  • The adversarial dataset, if adopted by others, provides a more demanding benchmark than ordinary conversations for comparing voice-phishing detectors.
  • Block-length sensitivity differs by model: GPT-4 improves with longer blocks, while fine-tuned small models with criteria peak at intermediate lengths, suggesting different optimal deployment settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • For other fraud-detection tasks with scarce positive examples, hand-authoring a small set of expert rules in the prompt may be a cheaper and more effective fine-tuning strategy than reasoning-style distillation.
  • If the adversarial test set were independently labeled by external experts or drawn from real-world misclassified calls, the paper's headline comparison would be put on firmer ground; as it stands, the comparison depends on the authors' own construction.
  • A natural extension is to test whether the criteria-prompt advantage persists on longer, multi-speaker, or code-switched transcripts, or when the model is distilled further to a mobile-scale size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper fine-tunes Llama-3-Korean-Bllossom-8B as a voice phishing (VP) detector, comparing prompts that include 11 expert-derived VP evaluation criteria, Chain-of-Thought (CoT), both, or neither. The authors construct an adversarial test set (Dataset G, 58 non-VP transcripts written to be confusable with VP) and a set of 35 synthetic VP transcripts (Dataset H), and evaluate accuracy on normal and adversarial test sets at block lengths from 100 to 2500. On the adversarial set, the fine-tuned model with criteria only (Llama3-FT-Cri) reaches 94.64% at block length 500, versus 81.25% for Llama3-FT-CoT at the same block length and 99.11% for GPT4o-Cri at block length 1500 (Table 5). The paper concludes that incorporating human expert knowledge into the prompt is more effective than CoT for small LMs and that the fine-tuned Llama3-8B is comparable to a GPT-4-based detector.

Significance. If the results held as stated, the paper would provide a practical recipe for building privacy-preserving, low-cost VP detectors on small open-source models, and the release of the dataset and source code (GitHub link in the introduction) is a valuable contribution. The comparison between expert criteria and CoT for small LMs is a useful empirical datapoint. However, the central claims are currently supported by a single table measured on an author-constructed benchmark with small margins, no confidence intervals, and a teacher-student setup that partially inherits the teacher's behavior. The significance is therefore conditional on additional validation of the benchmark and the statistical robustness of the headline numbers.

major comments (3)
  1. [Sections 3.2, 3.3, and Table 5] The adversarial test set is entirely authored and labeled by the same team: Dataset G (58 transcripts) was written to be 'highly likely to be misclassified as VP' and Dataset H (35 synthetic VP transcripts) was manually created, with no external expert review, inter-rater agreement, or human-label audit reported. Since the headline margins are small (5 transcripts separate Llama3-FT-Cri at 94.64% from GPT4o-Cri at 99.11%, and 15 transcripts separate it from Llama3-FT-CoT at 81.25%), even a handful of mislabeled or ambiguous examples could reverse or erase the main conclusions. The paper should report an independent annotation of Dataset G and H (e.g., Cohen's kappa with external raters) or, at minimum, a per-transcript error analysis showing that the decisions are robust to plausible label noise. Without this, the central comparison is not separable from the authors' own construction.
  2. [Section 4.2 and Abstract] The student model is fine-tuned on GPT-4o's VP likelihood scores generated with the same criteria prompt used by the GPT-4o detector. Consequently, the comparison between Llama3-FT-Cri and GPT4o-Cri is largely a distillation-gap measurement, not an independent test of whether the SLM's criteria-based reasoning matches a proprietary LLM's capability. The abstract's claim that the fine-tuned SLM is 'comparable to that of a GPT-4-based VP detector' should be qualified as 'comparable to a detector whose labels were used to train it' or supported by an additional experiment where the SLM is fine-tuned on manually verified labels (even on a subset). As written, the claim risks overstating the independence of the result.
  3. [Section 5.2 and Table 5] The headline accuracies are selected as the best over a grid of block lengths (100 to 2500) for each scheme, and the second-stage threshold λ is fit to the validation set. No confidence intervals, bootstrap estimates, or significance tests are reported, despite the test set containing only 112 transcripts. The 4.47%p gap between Llama3-FT-Cri (94.64%) and GPT4o-Cri (99.11%) corresponds to 5 transcripts; the 13.39%p gap over Llama3-FT-CoT corresponds to 15 transcripts. The paper should report exact binomial confidence intervals or a bootstrap over transcripts, and should justify the block-length selection procedure (e.g., a single pre-registered block length or a multiple-testing correction) so that the reader can assess whether the reported superiority is likely to replicate.
minor comments (4)
  1. [Various] The paper contains numerous typos and inconsistent notation: 'Exsiting' in Section 2.1, 'latent sementic analysis' in the Introduction, 'creteria' in Table 2, 'Appendx' in Section 3.2, 'accross' in Section 5.1, 'block-lenghts' in Fig. 4, and inconsistent capitalization of 'Llama3' vs 'LLaMA3' across tables and text. A thorough copyedit is needed.
  2. [Section 6] The conclusion says 'significantly fewer parameters than the Llama3B model,' but the paper uses the 8B model; this appears to be a typo for 'Llama3-8B' or 'Llama 3 8B'.
  3. [Section 5] The evaluation reports only accuracy. Given the class imbalance (VP to non-VP ratio about 1:4.4) and the thresholded second-stage decision, reporting precision, recall, and F1 for the VP class would help interpret the practical utility of the detector, especially on the adversarial set where false positives and false negatives may have very different costs.
  4. [Table 3 and Section 4.2] The description of which transcripts compose the adversarial test set is somewhat indirect: the text says Dataset G 'along with portions of datasets A and H' constitutes the adversarial set, but Table 3 only lists 54 VP and 58 non-VP adversarial transcripts. The paper should state explicitly how many transcripts come from each source and whether the 54 VP transcripts are the same across the normal and adversarial test sets, to avoid ambiguity in reproducing the numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity found; the central comparisons are empirical measurements on held-out human-labeled transcripts, not quantities forced by the training procedure.

full rationale

The paper's derivation chain is empirical rather than definitional. The fine-tuned SLM is trained on GPT-4o-generated likelihood labels (Section 4.2), and the paper later compares this student against GPT-4o on the adversarial test set (Table 5). This is a knowledge-distillation setup, not a circular reduction: the student is not guaranteed to match the teacher, and the test labels are human-assigned VP/non-VP judgments, not the teacher's likelihood scores. The claim that Llama3-FT-Cri is 'comparable to' GPT-4o is a measured outcome, not a fitted parameter renamed as a prediction. Similarly, the comparison between Cri and CoT prompts is an empirical result on the same test set; the teacher's own score ordering may be inherited, but the student's accuracy is not equal to the teacher's by construction. The author-constructed adversarial dataset (Dataset G) and synthetic VP transcripts (Dataset H) raise legitimate external-validity concerns: the benchmark is not independently labeled or audited, and the criteria used in the prompt were developed from the same domain literature that informed Dataset H. However, this is a data-construction and evaluation-validity issue, not a formal circularity: no equation or definition makes the reported accuracy equivalent to the training objective or to the prompt criteria. The self-citation to the authors' prior work [17] for the block-based prompting method is provenance, not load-bearing evidence for the central claim that criteria-based prompting beats CoT for SLMs. Accordingly, no circular step can be exhibited, and the paper should receive a low circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

All load-bearing inputs are human choices: the 11 hand-authored criteria, the GPT-4o teacher labels, and the author-constructed benchmark transcripts. None of these three is externally validated. The only numeric fitted parameter is the decision threshold λ; block length is a best-of-grid selection. No invented entities in the sense of new theoretical constructs are introduced.

free parameters (3)
  • decision threshold λ = not reported numerically; selected to maximize validation accuracy
    Section 4.4: 'We select λ such that maximizing the accuracy evaluated on the validation dataset of the 2nd stage.' This fitted knob directly sets the headline accuracy numbers.
  • block length = 500 for headline FT-Cri result; 1500 for headline GPT4o-Cri result; grid 100 to 2500
    Six block lengths are tested and the best per scheme is highlighted in Tables 4 and 5, so the reported maxima incorporate best-of-grid selection.
  • 11 VP evaluation criteria = n/a (hand-authored sentences)
    The criteria are human-designed from a crime-stage taxonomy [22] and condensed from the authors' prior 19-criteria prompt [17]; they are the main treatment variable, chosen by hand rather than learned.
assumptions (5)
  • domain assumption GPT-4o's block-level VP likelihood scores are a sufficiently reliable teacher signal for supervised fine-tuning
    Section 4.2: 'we assign the labeling task to GPT-4o... the student model is fine-tuned with the prompt and the labels obtained in the knowledge extraction.' Systematic teacher errors propagate to the student and into the reported accuracies.
  • domain assumption The 11 criteria, derived from a crime-stage analysis [22], capture the discriminative content of Korean VP calls
    Section 4.1 describes condensing 8 VP types and their 3 to 8 stages into 11 criteria; the validity of the criteria as a detection signal is assumed, not measured.
  • domain assumption Dataset A transcripts (Financial Supervisory Service, KBS) are authentic VP calls with correct binary labels
    Section 3.1; test-set ground truth is taken as given from public portals, and manual preprocessing 'filled in missing parts using reasonable estimates' (Section 3.4), which can alter content.
  • domain assumption Accuracy on the held-out test set (54 VP, 216 normal, 58 adversarial) is a meaningful and sufficiently powered metric
    No confidence intervals or significance tests are reported; with 112 adversarial samples a 4 percentage point gap is roughly 4 to 5 transcripts.
  • domain assumption Splitting transcripts into blocks preserves enough context for detection
    Section 4.1 acknowledges sentences can be split across blocks and assumes the LM completes them with marginal loss; this assumption is load-bearing for the block-length dependence results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Voice Phishing with Precision: Fine-Tuning Small Language Models." pith.science (2026). https://pith.science/paper/UIHQ27VL

@misc{pith2026250606180,
  author       = {Pith},
  title        = {Pith review of: Detecting Voice Phishing with Precision: Fine-Tuning Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UIHQ27VL}},
  note         = {Machine review of arXiv:2506.06180}
}
read the original abstract

We develop a voice phishing (VP) detector by fine-tuning Llama3, a representative open-source, small language model (LM). In the prompt, we provide carefully-designed VP evaluation criteria and apply the Chain-of-Thought (CoT) technique. To evaluate the robustness of LMs and highlight differences in their performance, we construct an adversarial test dataset that places the models under challenging conditions. Moreover, to address the lack of VP transcripts, we create transcripts by referencing existing or new types of VP techniques. We compare cases where evaluation criteria are included, the CoT technique is applied, or both are used together. In the experiment, our results show that the Llama3-8B model, fine-tuned with a dataset that includes a prompt with VP evaluation criteria, yields the best performance among small LMs and is comparable to that of a GPT-4-based VP detector. These findings indicate that incorporating human expert knowledge into the prompt is more effective than using the CoT technique for small LMs in VP detection.

Figures

Figures reproduced from arXiv: 2506.06180 by the authors.

Figure 1
Figure 1. The overall process of our VP detector [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Fine-tuning process. 4.2 Fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Accuracy across varying block-length for GPT4o, Llama3-Base, Llama3-FT, KoBERT schemes except for Plain and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Accuracy across varying block-length for GPT4o, Llama3-Base, Llama3-FT, KoBERT schemes except for Plain and [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 14 canonical work pages

  1. [1]

    Lisa C Adams, Daniel Truhn, Felix Busch, Felix Dorfner, Jawed Nawabi, Marcus R Makowski, and Keno K Bressem. 2024. Llama 3 challenges proprietary state-of-the-art large language models in radiology board–style examination questions. Radiology 312, 2 (2024), e241191

    work page 2024

  2. [2]

    AI Hub [n. d.]. AI Hub. https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&dataSetSn=98. Accessed: 2024-05-14

    work page 2024

  3. [3]

    Milandu Keith Moussavou Boussougou and Dong-Joo Park. 2022. Exploiting Korean Language Model to Improve Korean V oice Phishing Detection. KIPS Transactions on Software and Data Engineering 11, 10 (2022), 437–446

    work page 2022

  4. [4]

    ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, and KyungTae Lim. 2024. Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean. arXiv preprint arXiv:2403.10882 (2024)

    arXiv 2024

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  6. [6]

    Federal Trade Commission [n. d.]. Consumer Sentinel Network. https://public.tableau.com/app/profile/federal.trade.commission/viz/FraudReports/ FraudFacts. Accessed: 2024-08-30

    work page 2024

  7. [7]

    Financial Supervisory Service [n. d.]. V oice Phishing Experience Center. https://www.fss.or.kr/fss/bbs/B0000206/list.do?menuNo=200690. Accessed: 2024-05-14

    work page 2024

  8. [8]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071 (2022)

    arXiv 2022

Show all 24 references
  1. [9]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  2. [10]

    KBS [n. d.]. V oice Phishing Scenario. https://news.kbs.co.kr/datafile/2023/03/20230322_FqpJp9.pdf. Accessed: 2024-05-14

    2023

  3. [11]

    Jeong-Wook Kim, Gi-Wan Hong, and Hangbae Chang. 2021. V oice recognition and document classification-based data analysis for voice phishing detection. Human-centric Comput. Inf. Sci 11 (2021)

    2021

  4. [12]

    Korea Communication Commission [n. d.]. Wise User. https://www.wiseuser.go.kr/edu_list.do?boardtypecode=5255&boardno=0&sorting=0. Accessed: 2024-07-10

    2024

  5. [13]

    Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. arXiv preprint arXiv:2212.08410 (2022)

    2022 arXiv

  6. [14]

    National Institute of Korean Language [n. d.]. Dialogue Corpus transcription 2020 (Ver. 1.3). https://kli.korean.go.kr/corpus/main/requestMain.do? lang=en. Accessed: 2024-05-14

    2020

  7. [15]

    National Intelligence Service [n. d.]. The 8 Major Types of V oice Phishing Scams. https://www.nis.go.kr/CM/1_2_5/view.do?seq=12&currentPage=10. Accessed: 2024-07-10

    2024

  8. [16]

    J. Seo. 2022. Current status, type, trend, and response implications of voice phishing. Korean Social Trends 2022 (2022), 307–315

    2022

  9. [17]

    Ju Yong Sim and Seong Hwan Kim. 2024. V oice Phishing Detection Scheme Using a GPT-3.5-based Large Language Model. Journal of Korean Institute of Information Scientists and Engineers 51, 1 (2024), 67–77. doi:10.5626/JOK.2024.51.1.67

    2024 doi

  10. [18]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  11. [19]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, et al. 2024. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with...

    2024 arXiv

  12. [20]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  13. [21]

    Tiankai Yang, Yi Nian, Shawn Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan Rossi, Kaize Ding, et al . 2024. AD-LLM: Benchmarking Large Language Models for Anomaly Detection. arXiv preprint arXiv:2412.11142 (2024)

    2024

  14. [22]

    Haesung Yoon, Yeongsil Jeon, Jeongmin Lee, and Gyehwan Kim. 2023. A study on the response plan for each stage of voice phishing. Korean Institute of Criminology and Justice

    2023

  15. [23]

    Seunguk Yu, Yejin Kwon, Minju Kim, and Kiseong Lee. 2024. Korean V oice Phishing Detection Applying NER With Key Tags and Sentence-Level N-Gram. IEEE Access 12 (2024), 52951–52962. doi:10.1109/ACCESS.2024.3387027

    2024

  16. [24]

    How would you send the money?

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. 2024. Accessing GTP-4 level mathematical olympiad solutions via monte carlo tree self-refine with Llama-3 8B. arXiv preprint arXiv:2406.07394 (2024). A Samples of Adversarial Test Dataset Table 6 shows part...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.