{"id":"7b3b42e8-53fd-427f-a49f-1c1cb374319c","arxiv_id":"2506.20228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A study of 386 verified phishing emails quantifies ten body-obfuscation techniques, their co-occurrence patterns, and their associations with SpamAssassin scores, with text-in-image and Base64 encoding being most prevalent.","lead":"This paper counts how often phishing emails hide their words using ten tricks, such as putting text into images or encoding it in Base64, based on 386 real phishing emails. It also checks which tricks are combined and how they affect a popular spam filter's score, giving defenders a quantitative baseline for building better detectors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'significant co-occurrences' claim is unsupported: §4.2 presents only a normalized conditional-probability matrix with no independence test, so the strategic-layering result may reflect base rates.","rationale":"The paper's strongest claim bundles three quantitative results: prevalence estimates, co-occurrence patterns, and SpamAssassin score associations. The prevalence baseline is descriptive and its representativeness is already acknowledged in Section 5.2, making it less decisive as an internal flaw. The regression analysis has a visible internal inconsistency in Table 1 — for example, Homograph Attack is reported with p=0.480 yet a 95% CI of [-2.94, -0.62] that excludes zero — but this could in principle be repaired by recomputation and does not necessarily invalidate the overall direction of the Base64/Text-in-Image findings. The co-occurrence claim is the least secure because the paper explicitly asserts statistical significance without ever performing a significance test. Section 4.2 presents only conditional proportions, and the conclusion of 'deliberate, synergistic pairings' is drawn from visual inspection of those proportions. This is an internal gap between claim and evidence, not a disagreement with external consensus or a limitation that can be deferred to future work. A Fisher exact test on the raw pair counts is the natural, decisive check. If the pairings fail that test, the abstract's phrase 'identified significant pairwise co-occurrences' must be retracted or weakened, and the contribution should be reframed as descriptive. If they pass, the concern is resolved. The reader's conditionality already anticipates this issue in the rationale, though the reader's formal weakest-assumption statement emphasizes corpus representativeness; my concern is thus partially aligned with the reader. The appropriate verdict remains conditional: the paper is promising and the dataset is valuable, but the co-occurrence analysis must be made statistically rigorous before the central claim can be accepted as stated.","tokens_in":10518,"tokens_out":4779,"duration_ms":51678,"concrete_test":"Re-run the §4.2 analysis by constructing 2x2 contingency tables for all 45 unordered technique pairs and applying Fisher's exact test (or chi-square with continuity correction where expected counts permit), with Benjamini-Hochberg FDR control across the 45 tests. Report unadjusted p-values and adjusted q-values for the pairs currently highlighted as significant, especially Excluded Text × Text in Image, Long Invisible Random Strings × Invisible Text, and Invalid HTML × Text in Image. If, after multiple-testing correction, the q-values for these pairs are ≥0.05, the strategic-layering claim is not supported; if they remain below 0.05, the claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second contribution, and a headline result in the abstract, is the claim of 'significant pairwise co-occurrences' revealing strategic layering (e.g., Text in Image with Excluded Text/Multipart Abuse). But §4.2 provides no statistical test to support this. The normalized co-occurrence matrix in Figure 2 is defined as the proportion of emails containing technique i that also contain technique j, i.e., P(j|i). A value like 0.33 for Excluded Text given Text in Image cannot be interpreted without comparing it to the value expected under independence. Given the marginal prevalence of Excluded Text is about 21%, the expected P(Excluded Text | Text in Image) under independence is roughly 0.21; the observed 0.33 may or may not be significantly different, but the paper never reports a p-value, confidence interval, or effect size for any pair. The conclusion that 'these patterns provide quantitative evidence that phishing obfuscation often involves deliberate, synergistic pairings' is therefore not justified by the analysis presented. This is load-bearing because one of the three central claims is specifically the co-occurrence/strategic-layering result. If an appropriate test shows the observed conditional probabilities are consistent with independence, that contribution collapses to descriptive observation, even though the prevalence and regression findings might still stand. The fix is straightforward and the paper should be revised to include the missing test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a quantitative measurement study of body obfuscation techniques in phishing emails. The authors manually labeled 386 user-reported phishing emails (SignalSpam, 2024) for the presence of ten obfuscation techniques, report prevalence estimates (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, etc.), analyze pairwise co-occurrences via a normalized conditional-probability matrix, and fit a multilinear regression of SpamAssassin scores on binary technique indicators. The abstract claims three contributions: prevalence baselines, statistically significant co-occurrences revealing strategic layering, and regression evidence that Base64 Encoding and Text in Image are associated with lower spam scores while Invalid HTML is associated with higher scores. The paper stresses that these associations are specific to the tested SpamAssassin configuration and not causal.","tokens_in":10774,"tokens_out":6353,"duration_ms":64980,"significance":"If the results hold, this paper would provide a useful empirical baseline for email body obfuscation tactics, a scarce resource in the phishing literature. The manually labeled corpus of 386 verified phishing emails with ten technique labels is a potentially valuable asset, and the prevalence estimates offer concrete priorities for detection engineering. The regression analysis against a standard, externally defined filter (SpamAssassin) is a reproducible approach, and the use of bootstrap resampling to assess coefficient stability is methodologically sensible. The finding that Invalid HTML is penalized while Base64 Encoding is associated with reduced scores is a specific, falsifiable observation. The paper also has a clearly written Limitations section (Section 5.2) that acknowledges dataset specificity and the associative nature of the regression. However, two load-bearing issues need to be addressed: the co-occurrence analysis claims statistical significance without performing any significance test, and the regression table contains an internal inconsistency in the reported confidence interval and p-value for Homograph Attack.","major_comments":[{"comment":"The paper's abstract and Section 5.1 claim 'statistically significant pairwise co-occurrences' and 'statistically significant co-occurrence patterns,' but Section 4.2 presents no statistical test to support these claims. The normalized co-occurrence matrix in Figure 2 is described as the proportion of emails containing technique i that also contain technique j (i.e., P(j|i)), yet no comparison is made to the values expected under independence, and no p-values, confidence intervals, or effect sizes are reported for any pair. For example, the observed value of 0.33 for the Excluded Text / Text in Image pair must be evaluated against the marginal prevalence of Excluded Text (~21%), which gives an expected conditional probability of about 0.21 under independence; without a test, one cannot conclude that 0.33 is evidence of a deliberate synergistic pairing. The conclusion in Section 5.1 that these patterns 'provide compelling evidence' of layered strategies is therefore not justified by the analysis as presented. I recommend adding a formal independence test (e.g., Fisher's exact test or chi-square test per pair) with multiple-comparison correction, and reporting effect sizes such as the ratio of observed to expected co-occurrence.","section":"Section 4.2"},{"comment":"The orientation of the normalized co-occurrence matrix is ambiguous. The text says the matrix depicts 'the proportion of emails containing technique i that also contain technique j,' but the figure does not indicate which axis corresponds to i and which to j, nor whether entries are P(j|i) or P(i|j). This ambiguity is consequential: for a pair with unequal marginal prevalences like Excluded Text (21%) and Text in Image (47%), the value 0.33 could be read as P(Excluded Text | Text in Image) or P(Text in Image | Excluded Text), and these have very different interpretations. The authors should clearly label the rows and columns of Figure 2 and state the exact direction of the conditional probability used in all reported values.","section":"Section 4.2 / Figure 2"},{"comment":"There is an internal inconsistency in the regression results. For Homograph Attack, the reported 95% confidence interval is [-2.94, -0.62], which lies entirely below zero, yet the reported bootstrap p-value is 0.480 (non-significant). A 95% confidence interval that excludes zero should correspond to a two-sided p-value below 0.05 under standard bootstrap procedures, so at least one of these reported quantities is wrong. Similarly, the Text in Image coefficient has a 95% CI of [-0.62, 0.01], which includes zero, while the reported p-value is 0.046; the text states the intervals lie 'entirely or almost entirely below zero,' but a confidence interval containing zero is conventionally interpreted as non-significant at the 5% level. The authors should clarify the bootstrap method (e.g., percentile-based vs. bias-corrected) and correct the inconsistent entries, since these underpin the abstract's claim of statistically significant associations with filter scores.","section":"Table 1"},{"comment":"The sample size justification uses Cochran's formula for simple random sampling from a well-defined population, but the actual corpus is a convenience sample of user-reported emails from SignalSpam in 2024. No random selection procedure or sampling frame is described, so the claimed margin of error of ±5% at 95% confidence does not follow, and the prevalence estimates in Section 4.1 should be presented as descriptive of this particular corpus rather than as statistically generalizable to all phishing emails. The Limitations section (5.2) appropriately acknowledges dataset specificity, but the methodological framing in Section 3.1 overstates the inferential strength of the prevalence claims. I recommend revising the sampling discussion to distinguish the target population from the obtainable sample and to avoid implying that the computed sample size guarantees representativeness.","section":"Section 3.1"}],"minor_comments":[{"comment":"The manual labeling process is described as 'meticulously inspected by a trained analyst' and 'a subset was checked by an other specialist,' but no inter-rater reliability statistic (e.g., Cohen's kappa) is reported. Given that the entire analysis depends on these binary labels, quantifying labeling agreement would strengthen confidence in the measurements.","section":"Section 3.2"},{"comment":"In the Base64 Encoding paragraph, the phrase 'even-tough it's scare (detailed in 4.2)' contains typos and unclear wording; it should read 'even though it is scarce' or similar.","section":"Section 4.1"},{"comment":"The phrase 'with multipart abusing' should be 'with multipart abuse' for grammatical correctness.","section":"Abstract"},{"comment":"Several technique names in Table 1 contain stray spaces, e.g., 'T ext In Image,' 'W ord Splitting with HTML T ags,' and 'Capitalized HTML T ags.' These should be corrected to match the nomenclature used in the body text.","section":"Table 1"},{"comment":"The description of the bootstrap procedure says 'with minor random replacements,' which is vague. Standard bootstrap resampling draws samples of size n with replacement from the original data; if a different procedure was used (e.g., residual bootstrap or wild bootstrap), this should be stated precisely so the results are reproducible.","section":"Section 4.3"},{"comment":"The sentence 'Given the ten distinct obfuscation techniques identified highly present co-occurrences between specific pairs may indicate non-random, deliberate combinations' is grammatically incomplete and should be rephrased for clarity.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and under-studied problem, and the corpus and labeling effort are valuable. The main technical weaknesses are fixable: adding proper significance tests to the co-occurrence analysis, correcting the Table 1 inconsistency, and softening the sampling-based generalizability claims. I would encourage the authors to consider releasing the labeled dataset (with PII removed) as a community resource, which would substantially increase the paper's impact. The manuscript is within the scope of the journal and, after revision, could be a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paper gives the anti-phishing community a useful quantitative baseline: ten body obfuscation techniques, their prevalence in 386 verified phishing emails from SignalSpam, and a multilinear regression against SpamAssassin scores. The prevalence numbers (47% text-in-image, 31% Base64, etc.) and the finding that Base64 and text-in-image are associated with lower scores while invalid HTML is penalized are credible and useful. The manual labeling is careful, with cross-checking, and the corpus is real. The measurements themselves appear sound.\n\nThe soft spots are real, though. The abstract and Section 4.2 claim 'statistically significant pairwise co-occurrences' and 'strategic layering,' but there is no statistical test anywhere. The normalized matrix is just conditional probabilities P(j|i). A 0.33 for Excluded Text given Text in Image needs to be compared to the base rate of Excluded Text (about 0.21) under independence; the paper never reports a p-value, CI, or effect size for any pair. So the co-occurrence contribution currently reduces to descriptive observation. That is load-bearing because it is one of the paper's three stated contributions. The fix is straightforward: run a proper independence test (Fisher's exact or chi-square) on each pair, correct for multiple comparisons, and report which pairs actually exceed chance. If the patterns survive, the conclusion stands; if not, the prevalence and regression findings still stand, but the 'strategic layering' claim needs to be dropped.\n\nThe regression section is more solid, but Table 1 has internal inconsistencies. For Homograph Attack, the coefficient is -0.91, p=0.480, but the 95% CI is [-2.94, -0.62]—a CI that excludes zero cannot coexist with a non-significant p-value. For Text in Image, the CI [-0.62, 0.01] just barely includes zero while p=0.046; that's borderline and should be explained. These look like transcription or rounding errors, but the table needs to be fixed and the analysis re-run.\n\nThe bigger external-validity question is the single-source, single-year, user-reported corpus. The paper acknowledges this in Section 5.2, so it is a known limitation rather than a hidden one. I would not call it fatal; the prevalence baseline is honestly labeled as descriptive of this corpus.\n\nThis paper is for anti-phishing researchers and defense engineers who need a baseline against which to compare detection results. It deserves a serious referee. I'd send it to peer review with a request for revision: add the missing co-occurrence independence tests, fix Table 1, and temper the abstract until those are done. After that, it would be a solid reference.","headline":"Useful prevalence and filter-score baseline, but the co-occurrence 'significance' claim is untested and Table 1 has inconsistent CIs; conditionally acceptable after fixes.","tokens_in":11316,"tokens_out":2962,"would_cite":true,"duration_ms":28322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"47% of phishing emails hide text in images","keywords":["phishing","email body obfuscation","spam filter evasion","text-in-image","Base64 encoding","multipart abuse","co-occurrence analysis","SpamAssassin"],"falsifier":"Replicate the full pipeline on a comparable corpus from a second reporting platform in another country or period and score it with a second filter generation: if Text in Image and Base64 Encoding no longer appear in tens of percent of emails, or if their regression coefficients no longer sit below zero with p < 0.05 under a newer SpamAssassin rule set, then the prevalence baseline and the claimed evasion association are properties of this corpus and this configuration, not of phishing in general.","tokens_in":1969,"feed_emoji":"🎣","tokens_out":8634,"duration_ms":159834,"temperature":0.7,"pith_summary":"Phishing emails hide malicious content in the message body to slip past filters, but how often, and in what combinations, has not been measured quantitatively. This paper tries to establish that baseline by manually inspecting 386 verified phishing emails and scoring each with the open-source filter SpamAssassin. It claims that text rendered inside images (47.0%) and Base64-encoded content (31.2%) are the most common obfuscations, that certain pairs such as image-based text combined with multipart-structure abuse co-occur more often than chance would predict, and that within this filter configuration Base64 and text-in-image are associated with lower spam scores while invalid HTML is associated with higher ones. If these numbers hold, defenders get a quantitative target list: where to invest in OCR, decoding, and tolerant HTML parsing, and which layered combinations to treat as especially suspicious. The authors frame the score results as associations in one filter configuration, not as proof of evasion.","feed_headline":"47% of phishing emails hide text in images","feed_subtitle":"A 386-email study ties Base64 and image-embedded text to lower filter scores and maps how obfuscation tricks combine.","key_machinery":"The argument is carried by three instruments. First, a manual classification scheme that labels each email for ten body-obfuscation techniques using three cues - discrepancies between the raw source and the rendered output, suspicious character usage, and deviations from the RFC 5322, MIME RFC 2045/2046, and W3C HTML structural standards. Second, a normalized co-occurrence matrix that reports, for each pair of techniques, the proportion of emails containing one that also contain the other, used to identify non-random strategic pairings. Third, a multilinear regression of the SpamAssassin score on ten binary technique-presence predictors, with significance derived from 50,000 bootstrap resamples of the 386-email corpus rather than from parametric assumptions; the regression's role is to estimate each technique's association with the filter score while controlling for the presence of the other techniques.","core_discovery":"On the paper's own terms, the discovery is a three-part empirical baseline for body obfuscation in current phishing emails: a prevalence ranking of ten techniques (Text in Image 47.0%, Base64 Encoding 31.2%, Invalid HTML 28.8%, Excluded Text/Multipart Abuse 21.0%, Invisible Text 15.3%, Long Invisible Random Strings 15.1%, Capitalized HTML Tags 5.2%, Word Splitting with HTML Tags 4.2%, Homograph Attack 0.3%, Payload in Attachment 0.3%); normalized pairwise co-occurrence values that reveal deliberate layering, most notably Excluded Text with Text in Image at 0.33 and Long Invisible Random Strings with Invisible Text at 0.31, which the authors read as non-random, synergistic combinations; and a multilinear regression ($R^2 = 0.486$, $p < 0.001$, bootstrap-derived $p$-values) in which Base64 Encoding (coefficient $\\approx -0.97$, $p < 0.001$) and Text in Image (coefficient $\\approx -0.29$, $p = 0.046$) are associated with reduced SpamAssassin scores while Invalid HTML (coefficient $\\approx 1.11$, $p < 0.001$) is associated with increased scores. The central message is that obfuscation is not a grab-bag of isolated tricks: attackers combine techniques in recognizable pairs aimed at different layers of automated analysis, and at least some of those techniques correlate with the score reductions that determine whether mail is blocked.","pith_inferences":["The co-occurrence results suggest a layering hypothesis the paper does not fully pursue: if attackers systematically pair parser-disrupting techniques (invalid HTML) with content-hiding ones (images, invisible text), then models trained on cleaned corpora will systematically underestimate evasion on real traffic, because exactly the corrupted messages that curation removes are where the combinatio","The near-zero prevalence of homograph attacks and payload-in-attachment (one email each) could be an artifact of this single reporting pipeline rather than a true decline; a direct test would re-measure the same ten techniques on corpora from industry telemetry or from non-French platforms to see whether the prevalence ranking is stable across sources.","Because only SpamAssassin was measured, the natural next experiment is to run the same 386 emails through a newer SpamAssassin rule set and through machine-learning-based commercial filters; if the negative coefficients for Base64 and text-in-image flip sign, that would show how strongly apparent evasion is an artifact of a specific filter generation."],"forward_implications":["Defenses that rely only on scanning the visible text of an email body will miss roughly half of current phishing emails, since nearly 47% render the lure as an image with little machine-readable text.","Detection logic should treat empirically observed combinations - such as multipart-structure abuse paired with image-based text, or invalid HTML wrapped around hidden-text techniques - as risk signals in their own right, not just as individual indicators.","For the tested SpamAssassin configuration, penalizing structural anomalies (invalid HTML) appears to be a working defensive tactic, while image-based and Base64-obfuscated content are the places where score reductions point to a likely gap.","Because body obfuscation explains about half of the score variation (R^2 = 0.486), header features, link reputation, and other factors still carry the other half, so body-only features cannot stand alone in detection.","The technique mix and the filter responses are moving targets, so repeating this measurement on fresh corpora is needed to keep the prevalence baseline and the score associations current."],"supporting_citations":[{"why":"Names and provides the filter under test - the SpamAssassin project and its corpus - whose scores are the dependent variable in the effectiveness analysis.","marker":"[22]"},{"why":"Supplies the bootstrap resampling method (50,000 resamples) used to derive the p-values and confidence intervals for the regression coefficients without parametric assumptions.","marker":"[10]"},{"why":"Supplies the Cochran sample-size formula that fixes roughly 384 emails as the sample needed for a ±5% margin at 95% confidence, justifying the 386-email corpus size.","marker":"[17]"},{"why":"Documents homograph obfuscation and its bypass of digest and signature-based detection, anchoring the taxonomy's rarest technique and motivating the study of evasion.","marker":"[9]"},{"why":"Provides MIME RFC 2045/2046, the standard defining multipart/alternative structure whose misuse is classified as Excluded Text (Multipart Abuse).","marker":"[11]"},{"why":"Provides RFC 5322, the message-format standard against which structural anomalies such as invalid HTML are defined in the classification scheme.","marker":"[19]"}],"fun_headline_variants":["Half of phishing emails hide text in images","Study: base64 and image text lower spam scores","Phishing obfuscation: ten tricks mapped, pairings found","386 phishing emails reveal layered obfuscation combos","Image text and Base64 tied to phishing filter evasion"],"cache_read_input_tokens":13440,"weakest_assumption_plain":"The entire study treats 386 emails user-reported to the French SignalSpam platform during 2024 as a representative slice of phishing, and every prevalence, co-occurrence, and score figure would inherit the skew if those reporting habits or that year's campaigns are unrepresentative - a limitation the paper's Section 5.2 itself concedes.","fun_headline_variants_meta":{"raw":{"variants":["Half of phishing emails hide text in images","Study: base64 and image text lower spam scores","Phishing obfuscation: ten tricks mapped, pairings found","386 phishing emails reveal layered obfuscation combos","Image text and Base64 tied to phishing filter evasion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1914,"prompt_tokens":1086,"completion_tokens":828,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":702,"tokens_out":828,"duration_ms":9135,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:53:35.562169+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replicate the full pipeline on a comparable corpus from a second reporting platform in another country or period and score it with a second filter generation: if Text in Image and Base64 Encoding no longer appear in tens of percent of emails, or if their regression coefficients no longer sit below zero with p < 0.05 under a newer SpamAssassin rule set, then the prevalence baseline and the claimed evasion association are properties of this corpus and this configuration, not of phishing in general.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Names and provides the filter under test - the SpamAssassin project and its corpus - whose scores are the dependent variable in the effectiveness analysis."},{"cited_title":"Proceedings on Engineering Sciences 3, 25–32 (03 2021)","cited_arxiv_id":null,"evidence_quote":"Supplies the Cochran sample-size formula that fixes roughly 384 emails as the sample needed for a ±5% margin at 95% confidence, justifying the 386-email corpus size."},{"cited_title":"In: Under- standing Social Engineering Based Scams, pp","cited_arxiv_id":null,"evidence_quote":"Documents homograph obfuscation and its bypass of digest and signature-based detection, anchoring the taxonomy's rarest technique and motivating the study of evasion."},{"cited_title":"Request for Comments RFC 2045, Inter- net Engineering Task Force (Nov 1996).https://doi.org/10.17487/RFC2045","cited_arxiv_id":null,"evidence_quote":"Provides MIME RFC 2045/2046, the standard defining multipart/alternative structure whose misuse is classified as Excluded Text (Multipart Abuse)."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides RFC 5322, the message-format standard against which structural anomalies such as invalid HTML are defined in the classification scheme."}],"review_version":1}