REVIEW 3 major objections 5 minor 1 cited by
To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read LLM majority voting helps only for evenly matched models
desk verdict A clear but thin empirical study: the finding that majority-voting ensembles help only when components are comparable is plausible and standard, but the post hoc subensemble selection on the same test set undercuts the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is majority voting over three component configurations: a prompt-based ensemble (one LLM, multiple prompts), a model-based ensemble (multiple LLMs, one prompt), and a hybrid (multiple LLMs, multiple prompts). Majority voting corrects individual errors when the voters are roughly equal and make independent mistakes, but a lone strong voter cannot rescue an ensemble if it is outvoted by several weaker ones, and a weak prompt cannot be outvoted by one strong prompt. The paper uses this asymmetry to explain why ensembling only helps under performance parity.
What would settle it
Take a fresh random sample of several thousand URLs from PhishStorm, run the same five LLMs and three prompts, and find a model-based or hybrid ensemble that includes a clearly dominant model and still beats that dominant model's individual accuracy; if such a case exists, the paper's no-ensemble-when-dominant rule would be overturned.
Extended reading notes
Core claim
On a balanced 1,000-URL subset of PhishStorm, GPT-4 is the strongest individual classifier (0.946 accuracy, 0.943 F1 with the one-shot prompt) and LLaMA 2 the weakest (0.830 accuracy, 0.797 F1 with two-shot). Prompt-based ensembling improves or matches the best single prompt for most models, but for GPT-3.5-Turbo it lowers accuracy because two of its three prompts are noticeably weaker than its one-shot prompt. Model-based and hybrid ensembles that include GPT-4 never beat GPT-4 alone; when GPT-4 and LLaMA are removed (and in the hybrid also GPT-3.5-Turbo), leaving models of similar strength, the ensembles improve over their individual members. The paper's central claim is that majority voting across LLMs or prompts is valuable precisely when the components perform at equivalent levels, and is not recommended when one component dominates.
Load-bearing premise
The accuracy gaps between models and prompts, measured once on a single balanced 1,000-URL subset with no confidence intervals or repeated sampling, are stable enough to support a general rule about when ensembling helps.
Editorial extensions
If this is right
- A practitioner with one clearly dominant LLM should skip ensembling and deploy that model, saving compute and latency.
- Prompt-based ensembling is a reasonable default only when a model's prompt variants show similar accuracy; variable prompts can drag the vote down.
- Hybrid ensembles should be built from models and prompts preselected for comparable performance, not from the full available pool.
- Before production deployment, a small validation sample should be used to compare the best single component against the proposed ensemble, since the ensemble can underperform the best member.
- The finding extends the logic of majority-vote theory to LLM text classification: component parity is a precondition, not an optional virtue.
Reading between the lines
- Inference: the parity condition is probably not specific to phishing URLs, because the voting mechanics described do not depend on URL features; the same test on any LLM text classification task should reproduce the pattern.
- Inference: the paper leaves 'comparable performance' unquantified; a practical threshold could be derived by simulating majority votes from individual accuracies and locating the gap at which the ensemble crosses the best member.
- Inference: a natural extension is weighted voting, where each model's vote is scaled by its estimated reliability; the paper's result predicts this would restore ensemble gains when components differ, because the dominant model would no longer be outvoted.
- Inference: the single-subset measurement makes the finding a hypothesis about stability; repeated resampling from PhishStorm would show whether small accuracy gaps between middle-tier models are real or noise.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates three majority-voting ensemble strategies for phishing URL detection with LLMs: prompt-based ensembling (multiple prompts, one model), model-based ensembling (one prompt, multiple models), and hybrid ensembling (multiple prompts and models). Using a balanced 1,000-URL subset of the PhishStorm dataset, the authors evaluate five LLMs (GPT-3.5-Turbo, GPT-4, Gemini 1.0 Pro, PaLM 2, LLaMA 2) with zero-shot, one-shot, and two-shot prompts. Their main conclusion is that ensemble strategies improve accuracy only when the individual components have comparable performance; when one model or prompt is substantially better, the ensemble does not beat that best single component. The paper further recommends that, in practice, one should assess individual and ensemble performance on a portion of data before deployment.
Significance. If the central claim were established, this would be practically useful guidance for a topic of current interest: when to use majority-voting ensembles of LLMs rather than picking the best single model. The paper has notable strengths: it evaluates three clearly specified ensemble strategies against individual baselines on an external, balanced dataset (PhishStorm), reports the failure of naive full-model ensembles, and its conclusion is falsifiable in principle. However, the evidential core of the conditional recommendation is not yet demonstrated out-of-sample: the 'equivalent performance' sub-ensembles in Sections 5.4 and 5.5 are selected post hoc from the same 1,000-URL test set on which they are evaluated, and no confidence intervals or significance tests are provided. The paper also lacks a quantitative definition of 'equivalent performance.' Because the main claim depends on these load-bearing methodological choices, the contribution as currently presented is not yet solid, though it could become a useful empirical study after revision.
major comments (3)
- [Sections 5.4 and 5.5] The central evidence for the 'ensemble only when components are equivalent' rule comes from sub-ensembles constructed by removing GPT-4, LLaMA 2, and later GPT-3.5-Turbo based on their performance on the same 1,000-URL evaluation set, after which the resulting ensembles are scored on that same set. This selection-evaluation overlap means the observed gains for 'similarly performing' models (Figures 6 and 8) may be selection artifacts rather than evidence for the general rule. Section 6 recommends using a portion of the data to assess components before deployment, but the experiments do not implement this recommendation. Please either split the data into a model-selection set and a held-out evaluation set, or clearly label the post hoc analysis as exploratory and validate the 'equivalent-performance' rule prospectively on independent data.
- [Sections 5.2 and 5.3] The reported differences that motivate the central recommendation are likely within sampling noise. With 1,000 balanced URLs, the standard error of an accuracy near 0.87 is about 0.011, so GPT-3.5-Turbo's accuracies of 0.854, 0.879, and 0.856 (Section 5.2) are not clearly separated, and the 'variable performance' label is not statistically supported. The paper reports no confidence intervals, no significance tests, and no repeated sampling anywhere. Please add uncertainty quantification (e.g., Wilson intervals, bootstrap, or a paired significance test) for the key individual-versus-ensemble and prompt-versus-prompt comparisons, and temper the conclusions accordingly.
- [General methodology] The paper does not define what counts as 'equivalent' or 'similar' performance. The selection of Gemini, PaLM 2, and GPT-3.5-Turbo in Section 5.4 and of Gemini and PaLM 2 in Section 5.5 appears to rely on inspection of the same test-set accuracies, but no criterion is stated. Without an operational definition (for example, accuracy within a specified tolerance, or an equivalence test), the central conditional recommendation cannot be applied by practitioners or tested by future work. Please state a quantitative rule for component equivalence and apply it consistently.
minor comments (5)
- [Section 5.1] The full text of the three prompts is only described in one line each; please include the exact prompt templates in an appendix or in Figure 2 so that the experiments are reproducible.
- [Section 5.1] No generation parameters (e.g., temperature, max tokens, decoding strategy) are reported for the LLM queries; please specify these or state that default values were used.
- [Section 5.2] The F1-score calculation is not specified (micro vs. macro averaging); please state the averaging method used.
- [Section 5.1] There is a typographical spacing issue in the heading 'Evaluation Framework' that appears as 'Evaluation F ramework' in the text; please correct it.
- [Section 6] The recommendation to 'use a portion of the data to assess the performance of both individual models and their ensembles before deploying' is sensible but is not followed in the experiments themselves; please address this discrepancy explicitly, for example by acknowledging that the experiments are exploratory and by providing a prospective validation protocol.
Circularity Check
No circularity: empirical benchmark comparison with post-hoc subset selection (a validity concern, not a definitional one).
full rationale
The paper is an empirical benchmarking study with no derivation chain, no fitted parameters, and no first-principles result. The central claim—that majority-voting ensembles help only when components have equivalent performance—is an inductive summary of measured accuracies on a 1,000-URL balanced subset of PhishStorm, an external benchmark. None of the seven circularity patterns applies: no quantity is defined in terms of the target result (ensemble outputs are computed from raw LLM responses, not from the final conclusion); no parameter is fitted and then renamed as a prediction; the authors' self-citations [28,29] concern prompt engineering versus fine-tuning and are not used to justify the ensemble conclusions; no uniqueness theorem is invoked; and the majority-voting definitions are standard. The main validity concern is out-of-sample generalization, not circularity: the 'similarly performing' subensembles in Sections 5.4 and 5.5 are selected post hoc from the same test set on which they are then evaluated, and Section 6's recommendation to reserve data before deployment is not followed in the experiments. That is a selection-bias/overfitting risk that could weaken the recommendation, but it does not make the reported numbers equivalent to the inputs by construction. Score 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Majority voting improves accuracy when voters have comparable, at least partially independent error rates.
- domain assumption The 1,000-sample balanced PhishStorm subset is representative of phishing URL detection performance.
- ad hoc to paper The three prompt templates (zero-shot, one-shot, two-shot) represent meaningful diversity for prompt-based ensembling.
- domain assumption A single response per model-prompt pair characterizes that pair's performance.
Cite this review
Pith. "Pith review of To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models." pith.science (2026). https://pith.science/paper/IVLXORGS
@misc{pith2026241200166,
author = {Pith},
title = {Pith review of: To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/IVLXORGS}},
note = {Machine review of arXiv:2412.00166}
}
read the original abstract
The effectiveness of Large Language Models (LLMs) significantly relies on the quality of the prompts they receive. However, even when processing identical prompts, LLMs can yield varying outcomes due to differences in their training processes. To leverage the collective intelligence of multiple LLMs and enhance their performance, this study investigates three majority voting strategies for text classification, focusing on phishing URL detection. The strategies are: (1) a prompt-based ensemble, which utilizes majority voting across the responses generated by a single LLM to various prompts; (2) a model-based ensemble, which entails aggregating responses from multiple LLMs to a single prompt; and (3) a hybrid ensemble, which combines the two methods by sending different prompts to multiple LLMs and then aggregating their responses. Our analysis shows that ensemble strategies are most suited in cases where individual components exhibit equivalent performance levels. However, when there is a significant discrepancy in individual performance, the effectiveness of the ensemble method may not exceed that of the highest-performing single LLM or prompt. In such instances, opting for ensemble techniques is not recommended.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Large Multimodal Agents for Accurate Phishing Detection with Enhanced Token Optimization and Cost Reduction
Using both URLs and screenshots with API-based multimodal LLMs reaches 93-94% accuracy on a 1,000-site phishing test set, and a URL-first agentic cascade cuts API costs 2.6-4.2x with only minor accuracy loss.
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit
Abu-Nimeh, S., Nappa, D., Wang, X., Nair, S.: A comparison of machine learning techniques for phishing detection. In: Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit. pp. 60–69 (2007)
work page 2007
-
[2]
arXiv preprint arXiv:2303.08774 (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[3]
Advances in Engineering Software 173, 103288 (Nov 2022)
Ahammad, S.H., Kale, S.D., Upadhye, G.D., Pande, S.D., Babu, E.V., Dhu- mane, A.V., Bahadur, M.D.K.J.: Phishing URL detection using machine learn- ing methods. Advances in Engineering Software 173, 103288 (Nov 2022). https://doi.org/10.1016/j.advengsoft.2022.103288, https://www.sciencedirect. com/science/article/pii/S0965997822001892
arXiv 2022
-
[4]
arXiv preprint arXiv:2305.10403 (2023)
Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.: Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023)
arXiv 2023
-
[5]
Advances in neural information processing systems 33, 1877–1901 (2020)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
2020
-
[6]
ACM Transactions on Intelligent Systems and Technology (2023)
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (2023)
work page 2023
-
[7]
In: International work- shop on multiple classifier systems
Dietterich, T.G.: Ensemble methods in machine learning. In: International work- shop on multiple classifier systems. pp. 1–15. Springer (2000)
2000
-
[8]
Nature Machine Intelligence 5(3), 220–235 (2023)
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.M., Chen, W., et al.: Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5(3), 220–235 (2023)
work page 2023
Show all 35 references
-
[9]
arXiv preprint arXiv:2005.13012 (2020)
Gonz´ alez-Carvajal, S., Garrido-Merch´ an, E.C.: Comparing bert against traditional machine learning text classification. arXiv preprint arXiv:2005.13012 (2020)
2020 arXiv
-
[10]
In: 2013 international conference on control communication and computing (ICCC)
James, J., Sandhya, L., Thomas, C.: Detection of phishing urls using machine learning techniques. In: 2013 international conference on control communication and computing (ICCC). pp. 304–309. IEEE (2013)
2013
-
[11]
In: CHI Conference on Human Factors in Computing Systems Extended Abstracts
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., Cai, C.J.: Promptmaker: Prompt-based prototyping with large language models. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts. pp. 1– 8 (2022)
2022
-
[12]
In: Security and Privacy in Communication Networks: 13th International Conference, SecureComm 2017, Niagara Falls, ON, Canada, October 22–25, 2017, Proceedings 13
Jiang, J., Chen, J., Choo, K.K.R., Liu, C., Liu, K., Yu, M., Wang, Y.: A deep learning based online malicious url and dns detection scheme. In: Security and Privacy in Communication Networks: 13th International Conference, SecureComm 2017, Niagara Falls, ON, Canada, October 22...
2017
-
[13]
arXiv preprint arXiv:2307.10169 (2023)
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., McHardy, R.: Chal- lenges and applications of large language models. arXiv preprint arXiv:2307.10169 (2023)
2023 arXiv
-
[14]
Pattern Analysis & Applications 6, 22–31 (2003) 14 F
Kuncheva, L.I., Whitaker, C.J., Shipp, C.A., Duin, R.P.: Limits on the majority vote accuracy in classifier fusion. Pattern Analysis & Applications 6, 22–31 (2003) 14 F. Trad and A. Chehab
2003
-
[15]
https://doi.org/10.48550/arXiv.1802.03162, http://arxiv.org/abs/1802.03162, arXiv:1802.03162 [cs]
Le, H., Pham, Q., Sahoo, D., Hoi, S.C.H.: URLNet: Learning a URL Rep- resentation with Deep Learning for Malicious URL Detection (Mar 2018). https://doi.org/10.48550/arXiv.1802.03162, http://arxiv.org/abs/1802.03162, arXiv:1802.03162 [cs]
-
[16]
International Journal of Computer Applications 181(23), 45– 47 (Oct 2018)
Mahajan, R., Siddavatam, I.: Phishing Website Detection using Machine Learn- ing Algorithms. International Journal of Computer Applications 181(23), 45– 47 (Oct 2018). https://doi.org/10.5120/ijca2018918026, http://www.ijcaonline. org/archives/volume181/number23/mahajan-2018-i...
2018 doi
-
[17]
IEEE Transactions on Network and Service Management 11(4), 458–471 (2014)
Marchal, S., Fran¸ cois, J., State, R., Engel, T.: Phishstorm: Detecting phishing with streaming analytics. IEEE Transactions on Network and Service Management 11(4), 458–471 (2014)
2014
-
[18]
In: International Conference on Data Intelligence and Cognitive Informatics
Marvin, G., Hellen, N., Jjingo, D., Nakatumba-Nabende, J.: Prompt engineering in large language models. In: International Conference on Data Intelligence and Cognitive Informatics. pp. 387–402. Springer (2023)
2023
-
[19]
In: Proceedings of the ACL 2010 conference short papers
Moore, R.C., Lewis, W.: Intelligent selection of language model training data. In: Proceedings of the ACL 2010 conference short papers. pp. 220–224 (2010)
2010
-
[20]
In: Futuristic Trends in Networks and Computing Technologies: Se- lect Proceedings of Fourth International Conference on FTNCT 2021
Pawar, C.S., Makwana, A.: Comparison of bert-base and gpt-3 for marathi text classification. In: Futuristic Trends in Networks and Computing Technologies: Se- lect Proceedings of Fourth International Conference on FTNCT 2021. pp. 563–574. Springer (2022)
2022
-
[21]
IEEE Access (2024)
Raiaan, M.A.K., Mukta, M.S.H., Fatema, K., Fahad, N.M., Sakib, S., Mim, M.M.J., Ahmad, J., Ali, M.E., Azam, S.: A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access (2024)
2024
-
[22]
Internet of Things and Cyber- Physical Systems (2023)
Ray, P.P.: Chatgpt: A comprehensive review on background, applications, key chal- lenges, bias, ethics, limitations and future scope. Internet of Things and Cyber- Physical Systems (2023)
2023
-
[23]
Expert Systems with Applications 117, 345–357 (Mar 2019)
Sahingoz, O.K., Buber, E., Demir, O., Diri, B.: Machine learning based phish- ing detection from URLs. Expert Systems with Applications 117, 345–357 (Mar 2019). https://doi.org/10.1016/j.eswa.2018.09.029, https://www.sciencedirect. com/science/article/pii/S0957417418306067
2019 doi
-
[24]
In: 2020 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI)
Sern, L.J., David, Y.G.P., Hao, C.J.: Phishgan: Data augmentation and identifica- tion of homoglyph attacks. In: 2020 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI). pp. 1–6. IEEE (2020)
2020
-
[25]
Sheng, S., Wardman, B., Warner, G., Cranor, L., Hong, J., Zhang, C.: An empirical analysis of phishing blacklists (2009)
2009
-
[26]
arXiv preprint arXiv:2312.11805 (2023)
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalk- wyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
2023 arXiv
-
[27]
arXiv preprint arXiv:2307.09288 (2023)
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
2023 arXiv
-
[28]
arXiv preprint arXiv:2403.17787 (2024)
Trad, F., Chehab, A.: Evaluating the efficacy of prompt-engineered large multi- modal models versus fine-tuned vision transformers in image-based security appli- cations. arXiv preprint arXiv:2403.17787 (2024)
2024 arXiv
-
[29]
Machine Learning and Knowledge Extrac- tion 6(1), 367–384 (2024)
Trad, F., Chehab, A.: Prompt engineering or fine-tuning? a case study on phishing detection with large language models. Machine Learning and Knowledge Extrac- tion 6(1), 367–384 (2024)
2024
-
[30]
Advances in neural information pro- cessing systems 30 (2017) Assessing Majority Voting Strategies for Phishing Detection with LLMs 15
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems 30 (2017) Assessing Majority Voting Strategies for Phishing Detection with LLMs 15
2017
-
[31]
arXiv preprint arXiv:2206.07682 (2022)
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al.: Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022)
2022 arXiv
-
[32]
Computer Networks 178, 107275 (Sep 2020)
Wei, W., Ke, Q., Nowak, J., Korytkowski, M., Scherer, R., Wo´ zniak, M.: Accurate and fast URL phishing detector: A convolutional neu- ral network approach. Computer Networks 178, 107275 (Sep 2020). https://doi.org/10.1016/j.comnet.2020.107275, https://www.sciencedirect. com/s...
2020
-
[33]
arXiv preprint arXiv:2303.10420 (2023)
Ye, J., Chen, X., Xu, N., Zu, C., Shao, Z., Liu, S., Cui, Y., Zhou, Z., Gong, C., Shen, Y., et al.: A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420 (2023)
2023 arXiv
-
[34]
Neurocomputing 557, 126708 (2023)
Zhao, B., Jin, W., Del Ser, J., Yang, G.: Chatagri: Exploring potentials of chatgpt on cross-linguistic agricultural text classification. Neurocomputing 557, 126708 (2023)
2023
-
[35]
Human-centric Computing and Information Sciences 7(1), 17 (Jun 2017)
Zouina, M., Outtaj, B.: A novel lightweight URL phishing detection system using SVM and similarity index. Human-centric Computing and Information Sciences 7(1), 17 (Jun 2017). https://doi.org/10.1186/s13673-017-0098-1, https://doi. org/10.1186/s13673-017-0098-1
2017 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.