{"id":"62729be8-3a1f-419c-ad88-b65a5b8536b0","arxiv_id":"2412.00166","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Majority-voting ensembles of LLMs for phishing URL detection improve on the best single model only when ensemble members have comparable performance.","lead":"Researchers tested three ways to combine large language models by majority vote for phishing URL detection: voting across prompts, voting across models, and a hybrid of both. They found that ensemble methods only help when the individual models or prompts perform at similar levels, and that a clearly stronger single model beats the ensemble.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'equivalent-performance' subensembles are chosen post hoc from the same 1,000-URL test set on which they are evaluated, so the central conditional recommendation has not been tested out of sample.","rationale":"The reader's weakest assumption identifies the lack of confidence intervals and repeated sampling as the main risk. My concern is sharper and more structural: the paper selects which components count as 'equivalent' by looking at the same 1,000-URL set used to evaluate the resulting ensemble, so the positive evidence for the central claim is not an out-of-sample test. This is a form of selection on the test set, which can inflate apparent gains even if all point estimates are exactly as reported. The concern does not overturn the paper's plausible and internally consistent argument, and the reader's CONDITIONAL verdict already captures the need for stronger evidence, so no verdict change is needed. I credit the paper for clear experimental organization, direct comparisons, and an explicit recommendation to evaluate components before deployment; the issue is that the executed experiments do not follow that recommended protocol.","tokens_in":8142,"tokens_out":4237,"duration_ms":41748,"concrete_test":"Hold out a test set from the 1,000 URLs before any model selection. Pre-register a selection rule, e.g., from 600 training/validation URLs, include a model or prompt in the 'equivalent-performance' ensemble iff its validation accuracy is within 0.03 of the best component's validation accuracy. Then evaluate the selected ensembles and the best single component on the held-out 400 URLs, repeating over 100 random splits. If the selected equivalent-performance ensemble does not beat the best single component on held-out data in at least a majority of splits, the central recommendation lacks out-of-sample support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that majority voting helps only when components are comparable and is not recommended when one component dominates. The 'comparable components' branch is supported mainly by the post hoc subensembles in Sections 5.4 and 5.5: the full five-model ensemble fails to beat GPT-4, then GPT-4 and LLaMA 2 (and later GPT-3.5-Turbo) are removed, and the remaining 'similarly performing' models show improvement. The removal criterion is read off the very 1,000-URL evaluation set on which the resulting ensemble is then scored. Because component selection and evaluation share the same sample, the observed gain for equivalent-performance ensembles may be selection bias rather than evidence for the general rule. Section 6 recommends reserving a portion of data to assess components before deployment, but the experiments do not do this; no holdout is used. The noise level aggravates the problem: on 1,000 balanced URLs the standard error of an accuracy near 0.87 is about 0.011, so prompt-level gaps such as GPT-3.5's 0.854 vs 0.879 (Section 5.2) are within sampling noise, and the 'variability' used to label components as dissimilar is itself uncertain. Therefore, the paper's strongest claim—ensemble only when components are equivalent—is plausible but has not been demonstrated out-of-sample; its evidential core is circular with respect to the test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates three majority-voting ensemble strategies for phishing URL detection with LLMs: prompt-based ensembling (multiple prompts, one model), model-based ensembling (one prompt, multiple models), and hybrid ensembling (multiple prompts and models). Using a balanced 1,000-URL subset of the PhishStorm dataset, the authors evaluate five LLMs (GPT-3.5-Turbo, GPT-4, Gemini 1.0 Pro, PaLM 2, LLaMA 2) with zero-shot, one-shot, and two-shot prompts. Their main conclusion is that ensemble strategies improve accuracy only when the individual components have comparable performance; when one model or prompt is substantially better, the ensemble does not beat that best single component. The paper further recommends that, in practice, one should assess individual and ensemble performance on a portion of data before deployment.","tokens_in":8435,"tokens_out":3528,"duration_ms":30840,"significance":"If the central claim were established, this would be practically useful guidance for a topic of current interest: when to use majority-voting ensembles of LLMs rather than picking the best single model. The paper has notable strengths: it evaluates three clearly specified ensemble strategies against individual baselines on an external, balanced dataset (PhishStorm), reports the failure of naive full-model ensembles, and its conclusion is falsifiable in principle. However, the evidential core of the conditional recommendation is not yet demonstrated out-of-sample: the 'equivalent performance' sub-ensembles in Sections 5.4 and 5.5 are selected post hoc from the same 1,000-URL test set on which they are evaluated, and no confidence intervals or significance tests are provided. The paper also lacks a quantitative definition of 'equivalent performance.' Because the main claim depends on these load-bearing methodological choices, the contribution as currently presented is not yet solid, though it could become a useful empirical study after revision.","major_comments":[{"comment":"The central evidence for the 'ensemble only when components are equivalent' rule comes from sub-ensembles constructed by removing GPT-4, LLaMA 2, and later GPT-3.5-Turbo based on their performance on the same 1,000-URL evaluation set, after which the resulting ensembles are scored on that same set. This selection-evaluation overlap means the observed gains for 'similarly performing' models (Figures 6 and 8) may be selection artifacts rather than evidence for the general rule. Section 6 recommends using a portion of the data to assess components before deployment, but the experiments do not implement this recommendation. Please either split the data into a model-selection set and a held-out evaluation set, or clearly label the post hoc analysis as exploratory and validate the 'equivalent-performance' rule prospectively on independent data.","section":"Sections 5.4 and 5.5"},{"comment":"The reported differences that motivate the central recommendation are likely within sampling noise. With 1,000 balanced URLs, the standard error of an accuracy near 0.87 is about 0.011, so GPT-3.5-Turbo's accuracies of 0.854, 0.879, and 0.856 (Section 5.2) are not clearly separated, and the 'variable performance' label is not statistically supported. The paper reports no confidence intervals, no significance tests, and no repeated sampling anywhere. Please add uncertainty quantification (e.g., Wilson intervals, bootstrap, or a paired significance test) for the key individual-versus-ensemble and prompt-versus-prompt comparisons, and temper the conclusions accordingly.","section":"Sections 5.2 and 5.3"},{"comment":"The paper does not define what counts as 'equivalent' or 'similar' performance. The selection of Gemini, PaLM 2, and GPT-3.5-Turbo in Section 5.4 and of Gemini and PaLM 2 in Section 5.5 appears to rely on inspection of the same test-set accuracies, but no criterion is stated. Without an operational definition (for example, accuracy within a specified tolerance, or an equivalence test), the central conditional recommendation cannot be applied by practitioners or tested by future work. Please state a quantitative rule for component equivalence and apply it consistently.","section":"General methodology"}],"minor_comments":[{"comment":"The full text of the three prompts is only described in one line each; please include the exact prompt templates in an appendix or in Figure 2 so that the experiments are reproducible.","section":"Section 5.1"},{"comment":"No generation parameters (e.g., temperature, max tokens, decoding strategy) are reported for the LLM queries; please specify these or state that default values were used.","section":"Section 5.1"},{"comment":"The F1-score calculation is not specified (micro vs. macro averaging); please state the averaging method used.","section":"Section 5.2"},{"comment":"There is a typographical spacing issue in the heading 'Evaluation Framework' that appears as 'Evaluation F ramework' in the text; please correct it.","section":"Section 5.1"},{"comment":"The recommendation to 'use a portion of the data to assess the performance of both individual models and their ensembles before deploying' is sensible but is not followed in the experiments themselves; please address this discrepancy explicitly, for example by acknowledging that the experiments are exploratory and by providing a prospective validation protocol.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical question, but the central claim is not yet supported by the experimental design. The post hoc selection of sub-ensembles on the same evaluation set is the main methodological obstacle; it should be fixable within the scope of the manuscript by adding a holdout split or cross-validation, reporting uncertainty, and defining 'equivalent performance' before the claims can be accepted. The paper may be better suited to a security-focused venue or an applied NLP venue than to a general AI journal, though this is not a reason to reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper tests three majority-voting strategies for phishing URL detection with LLMs: prompt-based, model-based, and hybrid. The headline finding—ensembles help only when the individual components are roughly equal in performance, and don't beat a much better single model—is consistent with standard ensemble theory and is a sensible operational heuristic. The paper is clearly written, the strategies are well defined, and the figures make the pattern easy to see.\n\nWhat's genuinely new is the application to contemporary LLMs on a phishing task. That's a modest contribution, since the strategies are direct applications of majority voting [7,14] and the paper cites those sources. The empirical work is a reasonable first pass, but it has three soft spots.\n\nFirst, the 'equivalent performance' claim rests on post hoc removal of GPT-4, LLaMA, and later GPT-3.5, based on their performance on the same 1,000-URL test set on which the remaining ensemble is evaluated. That's selection bias, not out-of-sample evidence. The paper actually recommends holding out data to check components before deployment, but the experiments never do that. Second, the sample is tiny: 1,000 balanced URLs, no repeated runs, no confidence intervals or significance tests. Many of the prompt-level gaps are within sampling noise, so the 'variability' used to decide which components are comparable is itself uncertain. Third, no code or data are released, so the numbers can't be checked.\n\nNone of this means the central claim is wrong. For the reasons in the cited ensemble literature, it's likely right. But this paper's empirical demonstration is too weak to carry the general recommendation. The right fix is to re-run with a pre-registered component-selection rule, a separate holdout, error bars, and a public artifact.\n\nWho should read it? Practitioners who want a quick rule of thumb and don't already know the ensemble literature. It deserves a serious referee, because the question is practical and the paper is honest, but it needs substantial revision before it can be trusted as a result.\n\nRecommendation: send to peer review, but expect heavy revision and a request for a proper holdout and released code.","headline":"A clear but thin empirical study: the finding that majority-voting ensembles help only when components are comparable is plausible and standard, but the post hoc subensemble selection on the same test set undercuts the evidence.","tokens_in":8934,"tokens_out":2546,"would_cite":false,"duration_ms":22185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM majority voting helps only for evenly matched models","keywords":["ensemble learning","majority voting","large language models","phishing detection","prompt engineering","text classification","URL classification","LLM ensemble"],"falsifier":"Take a fresh random sample of several thousand URLs from PhishStorm, run the same five LLMs and three prompts, and find a model-based or hybrid ensemble that includes a clearly dominant model and still beats that dominant model's individual accuracy; if such a case exists, the paper's no-ensemble-when-dominant rule would be overturned.","tokens_in":7928,"feed_emoji":"🗳️","tokens_out":5547,"duration_ms":45114,"temperature":0.7,"pith_summary":"The paper asks when majority-voting ensembles of large language models actually improve text classification, using phishing URL detection as the testbed. It defines three voting schemes—several prompts voted on by one model, several models voting on one prompt, and a hybrid of both—and runs them across five LLMs and three prompt styles. The consistent finding is that an ensemble gains over its best single member only when the members are close in performance; when one model or prompt is clearly stronger, majority voting lands at or below that strongest component. If this holds, teams should measure individual components before paying the extra compute cost of ensembling.","feed_headline":"LLM majority voting helps only for evenly matched models","feed_subtitle":"On phishing URLs, voting across five LLMs and three prompts improves accuracy only when the models or prompts are comparable.","key_machinery":"The central mechanism is majority voting over three component configurations: a prompt-based ensemble (one LLM, multiple prompts), a model-based ensemble (multiple LLMs, one prompt), and a hybrid (multiple LLMs, multiple prompts). Majority voting corrects individual errors when the voters are roughly equal and make independent mistakes, but a lone strong voter cannot rescue an ensemble if it is outvoted by several weaker ones, and a weak prompt cannot be outvoted by one strong prompt. The paper uses this asymmetry to explain why ensembling only helps under performance parity.","core_discovery":"On a balanced 1,000-URL subset of PhishStorm, GPT-4 is the strongest individual classifier (0.946 accuracy, 0.943 F1 with the one-shot prompt) and LLaMA 2 the weakest (0.830 accuracy, 0.797 F1 with two-shot). Prompt-based ensembling improves or matches the best single prompt for most models, but for GPT-3.5-Turbo it lowers accuracy because two of its three prompts are noticeably weaker than its one-shot prompt. Model-based and hybrid ensembles that include GPT-4 never beat GPT-4 alone; when GPT-4 and LLaMA are removed (and in the hybrid also GPT-3.5-Turbo), leaving models of similar strength, the ensembles improve over their individual members. The paper's central claim is that majority voting across LLMs or prompts is valuable precisely when the components perform at equivalent levels, and is not recommended when one component dominates.","pith_inferences":["Inference: the parity condition is probably not specific to phishing URLs, because the voting mechanics described do not depend on URL features; the same test on any LLM text classification task should reproduce the pattern.","Inference: the paper leaves 'comparable performance' unquantified; a practical threshold could be derived by simulating majority votes from individual accuracies and locating the gap at which the ensemble crosses the best member.","Inference: a natural extension is weighted voting, where each model's vote is scaled by its estimated reliability; the paper's result predicts this would restore ensemble gains when components differ, because the dominant model would no longer be outvoted.","Inference: the single-subset measurement makes the finding a hypothesis about stability; repeated resampling from PhishStorm would show whether small accuracy gaps between middle-tier models are real or noise."],"forward_implications":["A practitioner with one clearly dominant LLM should skip ensembling and deploy that model, saving compute and latency.","Prompt-based ensembling is a reasonable default only when a model's prompt variants show similar accuracy; variable prompts can drag the vote down.","Hybrid ensembles should be built from models and prompts preselected for comparable performance, not from the full available pool.","Before production deployment, a small validation sample should be used to compare the best single component against the proposed ensemble, since the ensemble can underperform the best member.","The finding extends the logic of majority-vote theory to LLM text classification: component parity is a precondition, not an optional virtue."],"supporting_citations":[{"why":"supplies the PhishStorm dataset from which the 1,000-URL balanced test subset is drawn","marker":"[17]"},{"why":"supplies the majority-vote accuracy limits that motivate when voting can help","marker":"[14]"},{"why":"supplies the general ensemble-methods background the three strategies are built on","marker":"[7]"},{"why":"identifies GPT-4, the highest-performing member in the model-based and hybrid ensembles","marker":"[2]"},{"why":"identifies LLaMA 2, the lowest-performing member whose removal enables the balanced ensembles","marker":"[27]"},{"why":"identifies PaLM 2 as one of the mid-tier models in the balanced subset","marker":"[4]"},{"why":"identifies Gemini 1.0 Pro, a mid-tier model with low prompt sensitivity","marker":"[26]"},{"why":"identifies GPT-3.5-Turbo, the model whose variable prompt performance motivates the parity conclusion","marker":"[33]"}],"fun_headline_variants":["Voting only helps when LLMs are evenly matched","Don't ensemble unless your LLMs are equals","Majority voting fails when one LLM dominates","Ensemble LLMs only if they're similarly strong","For phishing, vote only among equal-strength LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gaps between models and prompts, measured once on a single balanced 1,000-URL subset with no confidence intervals or repeated sampling, are stable enough to support a general rule about when ensembling helps.","fun_headline_variants_meta":{"raw":{"variants":["Voting only helps when LLMs are evenly matched","Don't ensemble unless your LLMs are equals","Majority voting fails when one LLM dominates","Ensemble LLMs only if they're similarly strong","For phishing, vote only among equal-strength LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1200,"prompt_tokens":929,"completion_tokens":271,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":545,"tokens_out":271,"duration_ms":2743,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:51:53.016017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of several thousand URLs from PhishStorm, run the same five LLMs and three prompts, and find a model-based or hybrid ensemble that includes a clearly dominant model and still beats that dominant model's individual accuracy; if such a case exists, the paper's no-ensemble-when-dominant rule would be overturned.","supporting_citations":[{"cited_title":"IEEE Transactions on Network and Service Management 11(4), 458–471 (2014)","cited_arxiv_id":null,"evidence_quote":"supplies the PhishStorm dataset from which the 1,000-URL balanced test subset is drawn"},{"cited_title":"Pattern Analysis & Applications 6, 22–31 (2003) 14 F","cited_arxiv_id":null,"evidence_quote":"supplies the majority-vote accuracy limits that motivate when voting can help"}],"review_version":1}