Pith. sign in

REVIEW 4 major objections 6 minor 23 references

Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read 47% of phishing emails hide text in images

desk verdict Useful prevalence and filter-score baseline, but the co-occurrence 'significance' claim is untested and Table 1 has inconsistent CIs; conditionally acceptable after fixes. read the letter →

arxiv 2506.20228 v1 pith:SXJWOUKO submitted 2025-06-25 cs.CR

classification cs.CR
keywords phishingemailbodyobfuscationspamfilterevasiontext-in-imageBase64encodingmultipartabuseco-occurrenceanalysisAssassin
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Phishing emails hide malicious content in the message body to slip past filters, but how often, and in what combinations, has not been measured quantitatively. This paper tries to establish that baseline by manually inspecting 386 verified phishing emails and scoring each with the open-source filter SpamAssassin. It claims that text rendered inside images (47.0%) and Base64-encoded content (31.2%) are the most common obfuscations, that certain pairs such as image-based text combined with multipart-structure abuse co-occur more often than chance would predict, and that within this filter configuration Base64 and text-in-image are associated with lower spam scores while invalid HTML is associated with higher ones. If these numbers hold, defenders get a quantitative target list: where to invest in OCR, decoding, and tolerant HTML parsing, and which layered combinations to treat as especially suspicious. The authors frame the score results as associations in one filter configuration, not as proof of evasion.

What carries the argument

The argument is carried by three instruments. First, a manual classification scheme that labels each email for ten body-obfuscation techniques using three cues - discrepancies between the raw source and the rendered output, suspicious character usage, and deviations from the RFC 5322, MIME RFC 2045/2046, and W3C HTML structural standards. Second, a normalized co-occurrence matrix that reports, for each pair of techniques, the proportion of emails containing one that also contain the other, used to identify non-random strategic pairings. Third, a multilinear regression of the SpamAssassin score on ten binary technique-presence predictors, with significance derived from 50,000 bootstrap resamples of the 386-email corpus rather than from parametric assumptions; the regression's role is to estimate each technique's association with the filter score while controlling for the presence of the other techniques.

What would settle it

Replicate the full pipeline on a comparable corpus from a second reporting platform in another country or period and score it with a second filter generation: if Text in Image and Base64 Encoding no longer appear in tens of percent of emails, or if their regression coefficients no longer sit below zero with p < 0.05 under a newer SpamAssassin rule set, then the prevalence baseline and the claimed evasion association are properties of this corpus and this configuration, not of phishing in general.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a three-part empirical baseline for body obfuscation in current phishing emails: a prevalence ranking of ten techniques (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, Excluded Text/Multipart Abuse 21.0%, Invisible Text 15.3%, Long Invisible Random Strings 15.1%, Capitalized HTML Tags 5.2%, Word Splitting with HTML Tags 4.2%, Homograph Attack 0.3%, Payload in Attachment 0.3%); normalized pairwise co-occurrence values that reveal deliberate layering, most notably Excluded Text with Text in Image at 0.33 and Long Invisible Random Strings with Invisible Text at 0.31, which the authors read as non-random, synergistic combinations; and a multilinear regression ($R^2 = 0.486$, $p < 0.001$, bootstrap-derived $p$-values) in which Base64 Encoding (coefficient $\approx -0.97$, $p < 0.001$) and Text in Image (coefficient $\approx -0.29$, $p = 0.046$) are associated with reduced SpamAssassin scores while Invalid HTML (coefficient $\approx 1.11$, $p < 0.001$) is associated with increased scores. The central message is that obfuscation is not a grab-bag of isolated tricks: attackers combine techniques in recognizable pairs aimed at different layers of automated analysis, and at least some of those techniques correlate with the score reductions that determine whether mail is blocked.

Load-bearing premise

The entire study treats 386 emails user-reported to the French SignalSpam platform during 2024 as a representative slice of phishing, and every prevalence, co-occurrence, and score figure would inherit the skew if those reporting habits or that year's campaigns are unrepresentative - a limitation the paper's Section 5.2 itself concedes.

Editorial extensions

If this is right

  • Defenses that rely only on scanning the visible text of an email body will miss roughly half of current phishing emails, since nearly 47% render the lure as an image with little machine-readable text.
  • Detection logic should treat empirically observed combinations - such as multipart-structure abuse paired with image-based text, or invalid HTML wrapped around hidden-text techniques - as risk signals in their own right, not just as individual indicators.
  • For the tested SpamAssassin configuration, penalizing structural anomalies (invalid HTML) appears to be a working defensive tactic, while image-based and Base64-obfuscated content are the places where score reductions point to a likely gap.
  • Because body obfuscation explains about half of the score variation (R^2 = 0.486), header features, link reputation, and other factors still carry the other half, so body-only features cannot stand alone in detection.
  • The technique mix and the filter responses are moving targets, so repeating this measurement on fresh corpora is needed to keep the prevalence baseline and the score associations current.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The co-occurrence results suggest a layering hypothesis the paper does not fully pursue: if attackers systematically pair parser-disrupting techniques (invalid HTML) with content-hiding ones (images, invisible text), then models trained on cleaned corpora will systematically underestimate evasion on real traffic, because exactly the corrupted messages that curation removes are where the combinatio
  • The near-zero prevalence of homograph attacks and payload-in-attachment (one email each) could be an artifact of this single reporting pipeline rather than a true decline; a direct test would re-measure the same ten techniques on corpora from industry telemetry or from non-French platforms to see whether the prevalence ranking is stable across sources.
  • Because only SpamAssassin was measured, the natural next experiment is to run the same 386 emails through a newer SpamAssassin rule set and through machine-learning-based commercial filters; if the negative coefficients for Base64 and text-in-image flip sign, that would show how strongly apparent evasion is an artifact of a specific filter generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper presents a quantitative measurement study of body obfuscation techniques in phishing emails. The authors manually labeled 386 user-reported phishing emails (SignalSpam, 2024) for the presence of ten obfuscation techniques, report prevalence estimates (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, etc.), analyze pairwise co-occurrences via a normalized conditional-probability matrix, and fit a multilinear regression of SpamAssassin scores on binary technique indicators. The abstract claims three contributions: prevalence baselines, statistically significant co-occurrences revealing strategic layering, and regression evidence that Base64 Encoding and Text in Image are associated with lower spam scores while Invalid HTML is associated with higher scores. The paper stresses that these associations are specific to the tested SpamAssassin configuration and not causal.

Significance. If the results hold, this paper would provide a useful empirical baseline for email body obfuscation tactics, a scarce resource in the phishing literature. The manually labeled corpus of 386 verified phishing emails with ten technique labels is a potentially valuable asset, and the prevalence estimates offer concrete priorities for detection engineering. The regression analysis against a standard, externally defined filter (SpamAssassin) is a reproducible approach, and the use of bootstrap resampling to assess coefficient stability is methodologically sensible. The finding that Invalid HTML is penalized while Base64 Encoding is associated with reduced scores is a specific, falsifiable observation. The paper also has a clearly written Limitations section (Section 5.2) that acknowledges dataset specificity and the associative nature of the regression. However, two load-bearing issues need to be addressed: the co-occurrence analysis claims statistical significance without performing any significance test, and the regression table contains an internal inconsistency in the reported confidence interval and p-value for Homograph Attack.

major comments (4)
  1. [Section 4.2] The paper's abstract and Section 5.1 claim 'statistically significant pairwise co-occurrences' and 'statistically significant co-occurrence patterns,' but Section 4.2 presents no statistical test to support these claims. The normalized co-occurrence matrix in Figure 2 is described as the proportion of emails containing technique i that also contain technique j (i.e., P(j|i)), yet no comparison is made to the values expected under independence, and no p-values, confidence intervals, or effect sizes are reported for any pair. For example, the observed value of 0.33 for the Excluded Text / Text in Image pair must be evaluated against the marginal prevalence of Excluded Text (~21%), which gives an expected conditional probability of about 0.21 under independence; without a test, one cannot conclude that 0.33 is evidence of a deliberate synergistic pairing. The conclusion in Section 5.1 that these patterns 'provide compelling evidence' of layered strategies is therefore not justified by the analysis as presented. I recommend adding a formal independence test (e.g., Fisher's exact test or chi-square test per pair) with multiple-comparison correction, and reporting effect sizes such as the ratio of observed to expected co-occurrence.
  2. [Section 4.2 / Figure 2] The orientation of the normalized co-occurrence matrix is ambiguous. The text says the matrix depicts 'the proportion of emails containing technique i that also contain technique j,' but the figure does not indicate which axis corresponds to i and which to j, nor whether entries are P(j|i) or P(i|j). This ambiguity is consequential: for a pair with unequal marginal prevalences like Excluded Text (21%) and Text in Image (47%), the value 0.33 could be read as P(Excluded Text | Text in Image) or P(Text in Image | Excluded Text), and these have very different interpretations. The authors should clearly label the rows and columns of Figure 2 and state the exact direction of the conditional probability used in all reported values.
  3. [Table 1] There is an internal inconsistency in the regression results. For Homograph Attack, the reported 95% confidence interval is [-2.94, -0.62], which lies entirely below zero, yet the reported bootstrap p-value is 0.480 (non-significant). A 95% confidence interval that excludes zero should correspond to a two-sided p-value below 0.05 under standard bootstrap procedures, so at least one of these reported quantities is wrong. Similarly, the Text in Image coefficient has a 95% CI of [-0.62, 0.01], which includes zero, while the reported p-value is 0.046; the text states the intervals lie 'entirely or almost entirely below zero,' but a confidence interval containing zero is conventionally interpreted as non-significant at the 5% level. The authors should clarify the bootstrap method (e.g., percentile-based vs. bias-corrected) and correct the inconsistent entries, since these underpin the abstract's claim of statistically significant associations with filter scores.
  4. [Section 3.1] The sample size justification uses Cochran's formula for simple random sampling from a well-defined population, but the actual corpus is a convenience sample of user-reported emails from SignalSpam in 2024. No random selection procedure or sampling frame is described, so the claimed margin of error of ±5% at 95% confidence does not follow, and the prevalence estimates in Section 4.1 should be presented as descriptive of this particular corpus rather than as statistically generalizable to all phishing emails. The Limitations section (5.2) appropriately acknowledges dataset specificity, but the methodological framing in Section 3.1 overstates the inferential strength of the prevalence claims. I recommend revising the sampling discussion to distinguish the target population from the obtainable sample and to avoid implying that the computed sample size guarantees representativeness.
minor comments (6)
  1. [Section 3.2] The manual labeling process is described as 'meticulously inspected by a trained analyst' and 'a subset was checked by an other specialist,' but no inter-rater reliability statistic (e.g., Cohen's kappa) is reported. Given that the entire analysis depends on these binary labels, quantifying labeling agreement would strengthen confidence in the measurements.
  2. [Section 4.1] In the Base64 Encoding paragraph, the phrase 'even-tough it's scare (detailed in 4.2)' contains typos and unclear wording; it should read 'even though it is scarce' or similar.
  3. [Abstract] The phrase 'with multipart abusing' should be 'with multipart abuse' for grammatical correctness.
  4. [Table 1] Several technique names in Table 1 contain stray spaces, e.g., 'T ext In Image,' 'W ord Splitting with HTML T ags,' and 'Capitalized HTML T ags.' These should be corrected to match the nomenclature used in the body text.
  5. [Section 4.3] The description of the bootstrap procedure says 'with minor random replacements,' which is vague. Standard bootstrap resampling draws samples of size n with replacement from the original data; if a different procedure was used (e.g., residual bootstrap or wild bootstrap), this should be stated precisely so the results are reproducible.
  6. [Section 4.2] The sentence 'Given the ten distinct obfuscation techniques identified highly present co-occurrences between specific pairs may indicate non-random, deliberate combinations' is grammatically incomplete and should be rephrased for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical measurement using external SpamAssassin scores and manual labels, with no fitted parameter defined in terms of the outcomes it predicts.

full rationale

The paper's derivation chain is entirely empirical and self-contained against its own inputs. Prevalence counts (Section 4.1) come from manual labeling of 386 emails; there is no equation that defines a technique in terms of its measured prevalence. The co-occurrence analysis (Section 4.2) computes conditional proportions from the same manual labels; while the claim of 'significant' co-occurrence is statistically under-supported (no independence test), that is a correctness/rigor concern, not a circularity concern because the quantities are not defined by the conclusions drawn from them. The regression analysis (Section 4.3) uses SpamAssassin scores generated by an external, independently executed filter ('spamassassin <path to mail> '), with technique presences as binary predictors. No parameter is fitted to a subset and then renamed as a prediction of a closely related quantity; the reported coefficients are direct associations, and the paper explicitly labels them as associations rather than causal effects. There is no reliance on a self-citation for a load-bearing premise, no imported uniqueness theorem, and no ansatz smuggled in via citation. The methodology's manual labeling scheme is based on observed patterns and established principles, but that is standard feature construction for measurement, not circular reasoning. Thus the paper's central claims are empirical baselines and statistical associations, and none reduce by construction to their inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The analysis rests on the representativeness of the corpus, the reliability of manual labeling, and the use of one SpamAssassin configuration as a proxy for filter evasion. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption The 386 SignalSpam-reported emails are representative of phishing emails generally.
    Section 3.1 presents the corpus as a statistically grounded sample, but no random sampling from the reported pool is described; the paper's Limitations section notes generalizability needs validation.
  • domain assumption Manual labeling by trained analysts correctly identifies the ten obfuscation techniques.
    Section 3.2 describes manual inspection with a subset checked by a second analyst, but reports no inter-rater reliability or detailed annotation rules.
  • domain assumption SpamAssassin v4.0.1 with rules dated 06/04/2025 is a meaningful baseline for filter-evasion effectiveness.
    Section 4.3 uses a single filter configuration and acknowledges results may differ for other systems.
  • domain assumption The ten technique categories are an adequate enumeration of body obfuscation methods.
    The classification scheme was developed from observed patterns (Section 3.2) and may omit unseen techniques.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact." pith.science (2026). https://pith.science/paper/SXJWOUKO

@misc{pith2026250620228,
  author       = {Pith},
  title        = {Pith review of: Measuring Modern Phishing Tactics: A Quantitative Study of Body Obfuscation Prevalence, Co-occurrence, and Filter Impact},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SXJWOUKO}},
  note         = {Machine review of arXiv:2506.20228}
}
abstract

Phishing attacks frequently use email body obfuscation to bypass detection filters, but quantitative insights into how techniques are combined and their impact on filter scores remain limited. This paper addresses this gap by empirically investigating the prevalence, co-occurrence patterns, and spam score associations of body obfuscation techniques. Analysing 386 verified phishing emails, we quantified ten techniques, identified significant pairwise co-occurrences revealing strategic layering like the presence of text in images with multipart abuse, and assessed associations with antispam scores using multilinear regression. Text in Image (47.0%), Base64 Encoding (31.2%), and Invalid HTML (28.8%) were highly prevalent. Regression (R${}^2$=0.486, p<0.001) linked Base64 Encoding and Text in Image with significant antispam evasion (p<0.05) in this configuration, suggesting potential bypass capabilities, while Invalid HTML correlated with higher scores. These findings establish a quantitative baseline for complex evasion strategies, underscoring the need for multi-modal defences against combined obfuscation tactics.

Figures

Figures reproduced from arXiv: 2506.20228 by the authors.

Figure 1
Figure 1. Prevalence of identified body obfuscation techniques in the corpus (N=386). the <img> tag referencing the image file. This method fundamentally circum￾vents conventional text-based analysis pipelines, including keyword filtering, reg￾ular expression matching, and Natural Language Processing (NLP) techniques commonly employed by anti-spam and anti-phishing systems. Effective defense against this technique necessitate… view at source ↗
Figure 2
Figure 2. Normalized co-occurrence matrix of obfuscation techniques [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 20 canonical work pages

  1. [1]

    APWG: Phishing activity trends report: 3rd quarter 2024 (2024)

  2. [2]

    Baker, E., Cartier, M.: Phishing trends report (updated for 2025) (2025)

  3. [3]

    In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS)

    Champa, A.I., Rabbi, F., Zibran, M.F.: Why Phishing Emails Escape Detection: A Closer Look at the Failure Points. In: 2024 12th International Symposium on Digital Forensics and Security (ISDFS). pp. 1–6. IEEE, San Antonio, TX, USA (Apr 2024). https://doi.org/10.1109/ISDFS60797.2024.10527344

  4. [4]

    Cohen, W.W.: Enron email dataset (2015)

  5. [5]

    Cormack: 2007 TREC public spam corpus agreement (2007)

  6. [6]

    In: Proceedings of TREC 2006 (2006)

    Cormack, G.: TREC 2006 spam track overview. In: Proceedings of TREC 2006 (2006)

  7. [7]

    Cybersecurity and Infrastructure Security Agency (CISA), National Security Agency (NSA), Federal Bureau of Investigation (FBI), Multi-State Information Sharing and Analysis Center (MS-ISAC): Phishing guidance: Stopping the at- tack cycle at phase one (2023), https://www.cisa.gov/sites/default/files/ publications/Phishing-Infographic.pdf

  8. [8]

    Department of Physics & Astronomy, U.o.B.C.: PhishTank (2017)

Show all 23 references
  1. [9]

    In: Under- standing Social Engineering Based Scams, pp

    Dhiman, M., Yen, T.F., Jakobsson, M.: Obfuscation in spam and scam. In: Under- standing Social Engineering Based Scams, pp. 63–68. Springer (2016)

  2. [10]

    Chapman and Hall/CRC, 0 edn

    Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Chapman and Hall/CRC, 0 edn. (May 1994).https://doi.org/10.1201/9780429246593

  3. [11]

    Request for Comments RFC 2045, Inter- net Engineering Task Force (Nov 1996).https://doi.org/10.17487/RFC2045

    Freed, N., Borenstein, N.S.: Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies. Request for Comments RFC 2045, Inter- net Engineering Task Force (Nov 1996).https://doi.org/10.17487/RFC2045

  4. [12]

    G. V. Cormack: 2005 TREC public spam corpus (2005)

  5. [13]

    getastra.com/blog/security-audit/phishing-attack-statistics/ (2025), ac- cessed: 2025-04-17

    Getastra: 81 phishing attack statistics 2025: The ultimate insight.https://www. getastra.com/blog/security-audit/phishing-attack-statistics/ (2025), ac- cessed: 2025-04-17

  6. [14]

    https://keepnetlabs.com/blog/top-phishing-statistics-and-trends-you-must-know (2025)

    Keepnet Labs: Top phishing statistics and trends you must know. https://keepnetlabs.com/blog/top-phishing-statistics-and-trends-you-must-know (2025)

  7. [15]

    Kosinski, M.: What is phishing?https://www.ibm.com/think/topics/phishing (2024), accessed: 2025-04-17

  8. [16]

    Maurer, M.E.: The phishload phishing test database (2012)

  9. [17]

    Proceedings on Engineering Sciences 3, 25–32 (03 2021)

    Nanjundeswaraswamy,D.,Divakara,S.:Determinationofsamplesizeandsampling methods in applied research. Proceedings on Engineering Sciences 3, 25–32 (03 2021). https://doi.org/10.24874/PES03.01.003

  10. [18]

    Nazario, J.: Nazario phishing corpus (2007)

  11. [19]

    Resnick, P.: Internet Message Format. No. RFC 5322 (Oct 2008).https://doi. org/10.17487/RFC5322, https://datatracker.ietf.org/doc/rfc5322

  12. [20]

    IEEE Access 10,65703–65727(2022)

    Salloum, S., Gaber, T., Vadera, S., Shaalan, K.: A Systematic Literature Review on Phishing Email Detection Using Natural Language Processing Techniques. IEEE Access 10,65703–65727(2022). https://doi.org/10.1109/ACCESS.2022.3183083

  13. [21]

    Shahriar, S., Mukherjee, A., Gnawali, O.: Improving Phishing Detection Via Psy- chological Trait Scoring (2022).https://doi.org/10.48550/ARXIV.2208.06792

  14. [22]

    SpamAssassin Project: SpamAssassin public mail corpus (2006)

  15. [23]

    com/business/resources/reports/dbir/

    Verizon: 2024 data breach investigations report (2024), https://www.verizon. com/business/resources/reports/dbir/

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.