REVIEW 4 major objections 6 minor 23 references
Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read 47% of phishing emails hide text in images
desk verdict Useful prevalence and filter-score baseline, but the co-occurrence 'significance' claim is untested and Table 1 has inconsistent CIs; conditionally acceptable after fixes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by three instruments. First, a manual classification scheme that labels each email for ten body-obfuscation techniques using three cues - discrepancies between the raw source and the rendered output, suspicious character usage, and deviations from the RFC 5322, MIME RFC 2045/2046, and W3C HTML structural standards. Second, a normalized co-occurrence matrix that reports, for each pair of techniques, the proportion of emails containing one that also contain the other, used to identify non-random strategic pairings. Third, a multilinear regression of the SpamAssassin score on ten binary technique-presence predictors, with significance derived from 50,000 bootstrap resamples of the 386-email corpus rather than from parametric assumptions; the regression's role is to estimate each technique's association with the filter score while controlling for the presence of the other techniques.
What would settle it
Replicate the full pipeline on a comparable corpus from a second reporting platform in another country or period and score it with a second filter generation: if Text in Image and Base64 Encoding no longer appear in tens of percent of emails, or if their regression coefficients no longer sit below zero with p < 0.05 under a newer SpamAssassin rule set, then the prevalence baseline and the claimed evasion association are properties of this corpus and this configuration, not of phishing in general.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a three-part empirical baseline for body obfuscation in current phishing emails: a prevalence ranking of ten techniques (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, Excluded Text/Multipart Abuse 21.0%, Invisible Text 15.3%, Long Invisible Random Strings 15.1%, Capitalized HTML Tags 5.2%, Word Splitting with HTML Tags 4.2%, Homograph Attack 0.3%, Payload in Attachment 0.3%); normalized pairwise co-occurrence values that reveal deliberate layering, most notably Excluded Text with Text in Image at 0.33 and Long Invisible Random Strings with Invisible Text at 0.31, which the authors read as non-random, synergistic combinations; and a multilinear regression ($R^2 = 0.486$, $p < 0.001$, bootstrap-derived $p$-values) in which Base64 Encoding (coefficient $\approx -0.97$, $p < 0.001$) and Text in Image (coefficient $\approx -0.29$, $p = 0.046$) are associated with reduced SpamAssassin scores while Invalid HTML (coefficient $\approx 1.11$, $p < 0.001$) is associated with increased scores. The central message is that obfuscation is not a grab-bag of isolated tricks: attackers combine techniques in recognizable pairs aimed at different layers of automated analysis, and at least some of those techniques correlate with the score reductions that determine whether mail is blocked.
Load-bearing premise
The entire study treats 386 emails user-reported to the French SignalSpam platform during 2024 as a representative slice of phishing, and every prevalence, co-occurrence, and score figure would inherit the skew if those reporting habits or that year's campaigns are unrepresentative - a limitation the paper's Section 5.2 itself concedes.
Editorial extensions
If this is right
- Defenses that rely only on scanning the visible text of an email body will miss roughly half of current phishing emails, since nearly 47% render the lure as an image with little machine-readable text.
- Detection logic should treat empirically observed combinations - such as multipart-structure abuse paired with image-based text, or invalid HTML wrapped around hidden-text techniques - as risk signals in their own right, not just as individual indicators.
- For the tested SpamAssassin configuration, penalizing structural anomalies (invalid HTML) appears to be a working defensive tactic, while image-based and Base64-obfuscated content are the places where score reductions point to a likely gap.
- Because body obfuscation explains about half of the score variation (R^2 = 0.486), header features, link reputation, and other factors still carry the other half, so body-only features cannot stand alone in detection.
- The technique mix and the filter responses are moving targets, so repeating this measurement on fresh corpora is needed to keep the prevalence baseline and the score associations current.
Reading between the lines
- The co-occurrence results suggest a layering hypothesis the paper does not fully pursue: if attackers systematically pair parser-disrupting techniques (invalid HTML) with content-hiding ones (images, invisible text), then models trained on cleaned corpora will systematically underestimate evasion on real traffic, because exactly the corrupted messages that curation removes are where the combinatio
- The near-zero prevalence of homograph attacks and payload-in-attachment (one email each) could be an artifact of this single reporting pipeline rather than a true decline; a direct test would re-measure the same ten techniques on corpora from industry telemetry or from non-French platforms to see whether the prevalence ranking is stable across sources.
- Because only SpamAssassin was measured, the natural next experiment is to run the same 386 emails through a newer SpamAssassin rule set and through machine-learning-based commercial filters; if the negative coefficients for Base64 and text-in-image flip sign, that would show how strongly apparent evasion is an artifact of a specific filter generation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a quantitative measurement study of body obfuscation techniques in phishing emails. The authors manually labeled 386 user-reported phishing emails (SignalSpam, 2024) for the presence of ten obfuscation techniques, report prevalence estimates (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, etc.), analyze pairwise co-occurrences via a normalized conditional-probability matrix, and fit a multilinear regression of SpamAssassin scores on binary technique indicators. The abstract claims three contributions: prevalence baselines, statistically significant co-occurrences revealing strategic layering, and regression evidence that Base64 Encoding and Text in Image are associated with lower spam scores while Invalid HTML is associated with higher scores. The paper stresses that these associations are specific to the tested SpamAssassin configuration and not causal.
Significance. If the results hold, this paper would provide a useful empirical baseline for email body obfuscation tactics, a scarce resource in the phishing literature. The manually labeled corpus of 386 verified phishing emails with ten technique labels is a potentially valuable asset, and the prevalence estimates offer concrete priorities for detection engineering. The regression analysis against a standard, externally defined filter (SpamAssassin) is a reproducible approach, and the use of bootstrap resampling to assess coefficient stability is methodologically sensible. The finding that Invalid HTML is penalized while Base64 Encoding is associated with reduced scores is a specific, falsifiable observation. The paper also has a clearly written Limitations section (Section 5.2) that acknowledges dataset specificity and the associative nature of the regression. However, two load-bearing issues need to be addressed: the co-occurrence analysis claims statistical significance without performing any significance test, and the regression table contains an internal inconsistency in the reported confidence interval and p-value for Homograph Attack.
major comments (4)
- [Section 4.2] The paper's abstract and Section 5.1 claim 'statistically significant pairwise co-occurrences' and 'statistically significant co-occurrence patterns,' but Section 4.2 presents no statistical test to support these claims. The normalized co-occurrence matrix in Figure 2 is described as the proportion of emails containing technique i that also contain technique j (i.e., P(j|i)), yet no comparison is made to the values expected under independence, and no p-values, confidence intervals, or effect sizes are reported for any pair. For example, the observed value of 0.33 for the Excluded Text / Text in Image pair must be evaluated against the marginal prevalence of Excluded Text (~21%), which gives an expected conditional probability of about 0.21 under independence; without a test, one cannot conclude that 0.33 is evidence of a deliberate synergistic pairing. The conclusion in Section 5.1 that these patterns 'provide compelling evidence' of layered strategies is therefore not justified by the analysis as presented. I recommend adding a formal independence test (e.g., Fisher's exact test or chi-square test per pair) with multiple-comparison correction, and reporting effect sizes such as the ratio of observed to expected co-occurrence.
- [Section 4.2 / Figure 2] The orientation of the normalized co-occurrence matrix is ambiguous. The text says the matrix depicts 'the proportion of emails containing technique i that also contain technique j,' but the figure does not indicate which axis corresponds to i and which to j, nor whether entries are P(j|i) or P(i|j). This ambiguity is consequential: for a pair with unequal marginal prevalences like Excluded Text (21%) and Text in Image (47%), the value 0.33 could be read as P(Excluded Text | Text in Image) or P(Text in Image | Excluded Text), and these have very different interpretations. The authors should clearly label the rows and columns of Figure 2 and state the exact direction of the conditional probability used in all reported values.
- [Table 1] There is an internal inconsistency in the regression results. For Homograph Attack, the reported 95% confidence interval is [-2.94, -0.62], which lies entirely below zero, yet the reported bootstrap p-value is 0.480 (non-significant). A 95% confidence interval that excludes zero should correspond to a two-sided p-value below 0.05 under standard bootstrap procedures, so at least one of these reported quantities is wrong. Similarly, the Text in Image coefficient has a 95% CI of [-0.62, 0.01], which includes zero, while the reported p-value is 0.046; the text states the intervals lie 'entirely or almost entirely below zero,' but a confidence interval containing zero is conventionally interpreted as non-significant at the 5% level. The authors should clarify the bootstrap method (e.g., percentile-based vs. bias-corrected) and correct the inconsistent entries, since these underpin the abstract's claim of statistically significant associations with filter scores.
- [Section 3.1] The sample size justification uses Cochran's formula for simple random sampling from a well-defined population, but the actual corpus is a convenience sample of user-reported emails from SignalSpam in 2024. No random selection procedure or sampling frame is described, so the claimed margin of error of ±5% at 95% confidence does not follow, and the prevalence estimates in Section 4.1 should be presented as descriptive of this particular corpus rather than as statistically generalizable to all phishing emails. The Limitations section (5.2) appropriately acknowledges dataset specificity, but the methodological framing in Section 3.1 overstates the inferential strength of the prevalence claims. I recommend revising the sampling discussion to distinguish the target population from the obtainable sample and to avoid implying that the computed sample size guarantees representativeness.
minor comments (6)
- [Section 3.2] The manual labeling process is described as 'meticulously inspected by a trained analyst' and 'a subset was checked by an other specialist,' but no inter-rater reliability statistic (e.g., Cohen's kappa) is reported. Given that the entire analysis depends on these binary labels, quantifying labeling agreement would strengthen confidence in the measurements.
- [Section 4.1] In the Base64 Encoding paragraph, the phrase 'even-tough it's scare (detailed in 4.2)' contains typos and unclear wording; it should read 'even though it is scarce' or similar.
- [Abstract] The phrase 'with multipart abusing' should be 'with multipart abuse' for grammatical correctness.
- [Table 1] Several technique names in Table 1 contain stray spaces, e.g., 'T ext In Image,' 'W ord Splitting with HTML T ags,' and 'Capitalized HTML T ags.' These should be corrected to match the nomenclature used in the body text.
- [Section 4.3] The description of the bootstrap procedure says 'with minor random replacements,' which is vague. Standard bootstrap resampling draws samples of size n with replacement from the original data; if a different procedure was used (e.g., residual bootstrap or wild bootstrap), this should be stated precisely so the results are reproducible.
- [Section 4.2] The sentence 'Given the ten distinct obfuscation techniques identified highly present co-occurrences between specific pairs may indicate non-random, deliberate combinations' is grammatically incomplete and should be rephrased for clarity.
Circularity Check
No circularity: the paper is an empirical measurement using external SpamAssassin scores and manual labels, with no fitted parameter defined in terms of the outcomes it predicts.
full rationale
The paper's derivation chain is entirely empirical and self-contained against its own inputs. Prevalence counts (Section 4.1) come from manual labeling of 386 emails; there is no equation that defines a technique in terms of its measured prevalence. The co-occurrence analysis (Section 4.2) computes conditional proportions from the same manual labels; while the claim of 'significant' co-occurrence is statistically under-supported (no independence test), that is a correctness/rigor concern, not a circularity concern because the quantities are not defined by the conclusions drawn from them. The regression analysis (Section 4.3) uses SpamAssassin scores generated by an external, independently executed filter ('spamassassin <path to mail> '), with technique presences as binary predictors. No parameter is fitted to a subset and then renamed as a prediction of a closely related quantity; the reported coefficients are direct associations, and the paper explicitly labels them as associations rather than causal effects. There is no reliance on a self-citation for a load-bearing premise, no imported uniqueness theorem, and no ansatz smuggled in via citation. The methodology's manual labeling scheme is based on observed patterns and established principles, but that is standard feature construction for measurement, not circular reasoning. Thus the paper's central claims are empirical baselines and statistical associations, and none reduce by construction to their inputs.
Assumptions & free parameters
assumptions (4)
- domain assumption The 386 SignalSpam-reported emails are representative of phishing emails generally.
- domain assumption Manual labeling by trained analysts correctly identifies the ten obfuscation techniques.
- domain assumption SpamAssassin v4.0.1 with rules dated 06/04/2025 is a meaningful baseline for filter-evasion effectiveness.
- domain assumption The ten technique categories are an adequate enumeration of body obfuscation methods.
Cite this review
Pith. "Pith review of Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact." pith.science (2026). https://pith.science/paper/SXJWOUKO
@misc{pith2026250620228,
author = {Pith},
title = {Pith review of: Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact},
year = {2026},
howpublished = {\url{https://pith.science/paper/SXJWOUKO}},
note = {Machine review of arXiv:2506.20228}
}
abstract
Phishing attacks frequently use email body obfuscation to bypass detection filters, but quantitative insights into how techniques are combined and their impact on filter scores remain limited. This paper addresses this gap by empirically investigating the prevalence, co-occurrence patterns, and spam score associations of body obfuscation techniques. Analysing 386 verified phishing emails, we quantified ten techniques, identified significant pairwise co-occurrences revealing strategic layering like the presence of text in images with multipart abuse, and assessed associations with antispam scores using multilinear regression. Text in Image (47.0%), Base64 Encoding (31.2%), and Invalid HTML (28.8%) were highly prevalent. Regression (R${}^2$=0.486, p<0.001) linked Base64 Encoding and Text in Image with significant antispam evasion (p<0.05) in this configuration, suggesting potential bypass capabilities, while Invalid HTML correlated with higher scores. These findings establish a quantitative baseline for complex evasion strategies, underscoring the need for multi-modal defences against combined obfuscation tactics.
Figures
Reference graph
Works this paper leans on
-
[1]
APWG: Phishing activity trends report: 3rd quarter 2024 (2024)
work page 2024
-
[2]
Baker, E., Cartier, M.: Phishing trends report (updated for 2025) (2025)
work page 2025
-
[3]
In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS)
Champa, A.I., Rabbi, F., Zibran, M.F.: Why Phishing Emails Escape Detection: A Closer Look at the Failure Points. In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS). pp. 1–6. IEEE, San Antonio, TX, USA (Apr 2024). https://doi.org/10.1109/ISDFS60797.2024.10527344
arXiv 2024
-
[4]
Cohen, W.W.: Enron email dataset (2015)
work page 2015
-
[5]
Cormack: 2007 TREC public spam corpus agreement (2007)
work page 2007
-
[6]
In: Proceedings of TREC 2006 (2006)
Cormack, G.: TREC 2006 spam track overview. In: Proceedings of TREC 2006 (2006)
work page 2006
-
[7]
Cybersecurity and Infrastructure Security Agency (CISA), National Security Agency (NSA), Federal Bureau of Investigation (FBI), Multi-State Information Sharing and Analysis Center (MS-ISAC): Phishing guidance: Stopping the at- tack cycle at phase one (2023), https://www.cisa.gov/sites/default/files/ publications/Phishing-Infographic.pdf
work page 2023
-
[8]
Department of Physics & Astronomy, U.o.B.C.: PhishTank (2017)
work page 2017
Show all 23 references
-
[9]
In: Under- standing Social Engineering Based Scams, pp
Dhiman, M., Yen, T.F., Jakobsson, M.: Obfuscation in spam and scam. In: Under- standing Social Engineering Based Scams, pp. 63–68. Springer (2016)
2016
-
[10]
Chapman and Hall/CRC, 0 edn
Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman and Hall/CRC, 0 edn. (May 1994).https://doi.org/10.1201/9780429246593
1994 doi
-
[11]
Request for Comments RFC 2045, Inter- net Engineering Task Force (Nov 1996).https://doi.org/10.17487/RFC2045
Freed, N., Borenstein, N.S.: Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies. Request for Comments RFC 2045, Inter- net Engineering Task Force (Nov 1996).https://doi.org/10.17487/RFC2045
1996 doi
-
[12]
G. V. Cormack: 2005 TREC public spam corpus (2005)
2005
-
[13]
getastra.com/blog/security-audit/phishing-attack-statistics/ (2025), ac- cessed: 2025-04-17
Getastra: 81 phishing attack statistics 2025: The ultimate insight.https://www. getastra.com/blog/security-audit/phishing-attack-statistics/ (2025), ac- cessed: 2025-04-17
2025
-
[14]
https://keepnetlabs.com/blog/top-phishing-statistics-and-trends-you-must-know (2025)
Keepnet Labs: Top phishing statistics and trends you must know. https://keepnetlabs.com/blog/top-phishing-statistics-and-trends-you-must-know (2025)
2025
-
[15]
Kosinski, M.: What is phishing?https://www.ibm.com/think/topics/phishing (2024), accessed: 2025-04-17
2024
-
[16]
Maurer, M.E.: The phishload phishing test database (2012)
2012
-
[17]
Proceedings on Engineering Sciences 3, 25–32 (03 2021)
Nanjundeswaraswamy,D.,Divakara,S.:Determinationofsamplesizeandsampling methods in applied research. Proceedings on Engineering Sciences 3, 25–32 (03 2021). https://doi.org/10.24874/PES03.01.003
2021 doi
-
[18]
Nazario, J.: Nazario phishing corpus (2007)
2007
-
[19]
Resnick, P.: Internet Message Format. No. RFC 5322 (Oct 2008).https://doi. org/10.17487/RFC5322, https://datatracker.ietf.org/doc/rfc5322
2008 doi
-
[20]
IEEE Access 10,65703–65727(2022)
Salloum, S., Gaber, T., Vadera, S., Shaalan, K.: A Systematic Literature Review on Phishing Email Detection Using Natural Language Processing Techniques. IEEE Access 10,65703–65727(2022). https://doi.org/10.1109/ACCESS.2022.3183083
2022
- [21]
-
[22]
SpamAssassin Project: SpamAssassin public mail corpus (2006)
2006
-
[23]
com/business/resources/reports/dbir/
Verizon: 2024 data breach investigations report (2024), https://www.verizon. com/business/resources/reports/dbir/
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.