Pith. sign in

REVIEW 2 major objections 5 minor 14 references

Data interference: emojis, homoglyphs, and issues of data fidelity in corpora and their results

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read Emojis and homoglyphs are routinely mis-tokenised by corpus tools, and preprocessing them before corpus compilation restores fidelity to source data.

desk verdict A solid, reproducible audit of how three corpus tools mangle emojis and homoglyphs; the preprocessing recipe works, but the 'no information loss' claim oversells the plain-text and VRT outputs. read the letter →

arxiv 2507.01764 v1 pith:55IY4NRO submitted 2025-07-02 cs.CL

classification cs.CL
keywords tokenisationemojishomoglyphsUTF-8datafidelitycorpuslinguisticsUnicodenormalizationsourcesofinterference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that emojis and homoglyphs are not neutral content in corpora: when corpus tools tokenise raw UTF-8 text, they routinely split emoji sequences into invalid fragments, merge emojis with adjacent words, and treat visually identical homoglyphs as different characters. The consequence is a corpus that is unfaithful to what was actually written, so quantitative findings such as token and type counts, collocations, keywords, and n-grams rest on phantom or missing elements rather than on the language under study. The paper's remedy is a preprocessing pipeline that retokenises emojis by adding whitespace, transliterates them into bracketed Unicode short names, and normalises homoglyphs with Unicode NFKC before the corpus is compiled. On the paper's evidence, preprocessed corpus files reproduce the source data faithfully across three popular corpus tools, whereas non-preprocessed files generate mismatches in every tool. A sympathetic reader would care because the claim, if right, gives corpus linguists a practical, tool-independent way to make analyses repeatable and to keep qualitative objects like identity-relevant emoji sequences visible.

What carries the argument

The key machinery is the distinction between human-centred and computer-centred representations of text, together with two Unicode mechanisms. For emojis, the relevant mechanism is codepoint sequencing: a single visible emoji can be an emoji character, an emoji sequence joined by zero-width joiner (ZWJ) and variation selectors, or a sequence with skin-tone modifiers, so tokenisation at the codepoint level fragments what a reader sees as one unit. For homoglyphs, the mechanism is Unicode character equivalence and Normalization Form KC (NFKC), which performs compatibility decomposition followed by canonical composition, mapping compatibility variants such as mathematical double-struck capitals and ligatures onto their base characters. The preprocessing pipeline operationalises these mechanisms by retokenising emojis with added whitespace, replacing each emoji with a curly-bracketed Unicode short name so it remains a single token, and running every token through NFKC normalisation with a second tokenisation pass, since one homoglyph can expand into several tokens; metadata in the corpus files preserves the original forms.

What would settle it

Re-run the emoji test-file procedure with a newly released Unicode emoji, for example one added in Unicode 16.0, passing it through the same preprocessing pipeline and the same three corpus tools, and count how many new emojis survive as single tokens; any newly added emoji that is split or dropped would show that the ground-truth list, not the pipeline, is the limiting factor.

Watch

Extended reading notes

Core claim

The central claim is that incorrect tokenisation of UTF-8 features, especially emoji sequences and homoglyphs, creates 'sources of interference' that make researchers analyse an object that is not the language they set out to study. Emojis are counted by humans as single pictographs, but many are encoded as sequences of multiple codepoints joined by zero-width joiners, variation selectors, and skin-tone modifiers; the three corpus tools tested (AntConc, LancsBoxX, and SketchEngine) split these sequences into components, drop the joiners, or fuse emojis with neighbouring text, producing invalid emoji types and missing valid ones. Homoglyphs, by contrast, are visually identical characters with different codepoints, and without normalisation they survive as distinct types, so queries for the ordinary character miss them and strings such as a decomposed accent plus letter are counted separately from the precomposed character. The paper shows that preprocessing, separating emojis with whitespace, replacing each emoji with its official Unicode short name enclosed in curly brackets, and applying NFKC normalisation to homoglyphs, eliminates these mismatches: in the natural-language test corpus, all mismatches were found only in non-preprocessed files, while preprocessed files faithfully represented emojis and homoglyphs as they appear in the source data. The conclusion is that preprocessing, guided by linguistic purpose and with metadata preserving original forms, restores fidelity and makes results repeatable regardless of which corpus tool is used.

Load-bearing premise

The load-bearing premise is that the emoji inventory and NFKC normalisation used to build the test files define what a corpus should contain, and that the English-based, non-Latin-script-excluded analysis stands in for all Unicode text; if the inventory is incomplete, a linguistic question needs a different normalisation, or non-Latin scripts behave differently, the claim that preprocessing restores fidelity weakens.

Editorial extensions

If this is right

  • Any corpus compiled from raw, unpreprocessed text containing emojis or homoglyphs will, in at least one of the three tested tools, contain phantom types, split sequences, or missing elements, so token and type counts cannot be trusted without a fidelity check.
  • Preprocessed corpus files, plain text with custom token definitions or pre-tagged XML/VRT, reproduce the source data's emoji and homoglyph content across tools, so researchers can keep the tool they prefer and still obtain comparable results.
  • Downstream methods that rely on tokens and types inherit the interference: if a sequence such as the Transgender Flag emoji is split into two components, the original emoji disappears from the corpus and becomes unavailable to qualitative and quantitative analysis.
  • Normalising homoglyphs with NFKC unifies characters that are indistinguishable to a reader, so one query retrieves all equivalent spellings, but because normalisation is destructive, the original forms must be preserved in metadata.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same test-file method could be reused as a systematic fidelity audit for any tokeniser, including large language model tokenisers, since the paper notes that incorrect tokenisation also degrades LLM output but does not test its pipeline there.
  • The logic extends to ligatures, accented characters, zero-width spaces, and non-Latin scripts, which the paper lists as additional potential interference but does not measure; equivalent test files for those classes would show whether preprocessing generalises.
  • If fidelity means matching a human-centred reading, then preprocessing choices such as transliterating the emoji ™ to its short name or to the string TM are linguistic decisions, not neutral technical fixes, and different research questions would justify different normalisations.
  • The paper's tool-independence guarantee is bounded by the tools and token-definition settings it tested, so a fourth tool that cannot handle curly-bracketed tokens would require additional preprocessing decisions rather than automatically working.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies how emojis and homoglyphs in UTF-8 source texts are tokenised by three corpus tools (AntConc, LancsBoxX, SketchEngine). It defines 'sources of interference' as mismatches between source data and corpus wordlists, and shows—with artificial test files and a roughly 200,000-token Instagram corpus—that un-preprocessed data produces many spurious or unrecognised emoji and homoglyph types, while a preprocessing pipeline that adds whitespace around emojis, transliterates them to CLDR short names, and applies NFKC normalisation to homoglyphs removes the observed mismatches. The paper concludes that such preprocessing restores corpus fidelity and permits tool-independent repeatable analysis without information loss.

Significance. The strongest contribution is the concrete, reproducible documentation of tokeniser behaviour: test files are fully specified, tool versions and settings are given, and the real-data sample shows hundreds of mismatches in raw corpora that disappear after preprocessing. This is practically useful for corpus linguists and for tool developers. The paper also usefully separates human-centred and computer-centred views of emoji identity. However, the strength of the fidelity/no-loss conclusion is not matched by the validation design, because the source-data 'ground truth' is defined by the same Python libraries used for preprocessing (Sections 3.3 and 4.3). With revised, more modest claims, the paper would be a solid methodological contribution.

major comments (2)
  1. [§4.3, Table 12; §3.3] The evaluation of preprocessing is circular with respect to the chosen libraries. The source-data emoji count uses the Python `emoji` library, the same library used to retokenise and transliterate, and the homoglyph count uses `unicodedata` NFKC, the same normalisation applied in preprocessing. Any emoji absent from the library's 5,034-entry list, or any character whose NFKC mapping is not the one intended by the analyst, is invisible in both the source-data count and the preprocessed wordlist. Consequently the claim that preprocessed corpora 'faithfully represent' both features (Section 4.3) and that preprocessing 'does not introduce any additional or unwanted linguistic-related value' (Section 5) is stronger than the evaluation can support. The authors should validate against an independent source—for example, the official Unicode emoji list plus manual inspection of characters not covered by the library—or explicitly restrict the claim to 'the features recognised by the preprocessing library'.
  2. [§5; §3.3] The conclusion that the pipeline operates 'without any information loss' is contradicted by the described pipeline. NFKC is acknowledged to be destructive (Section 2.3), and the XML metadata in Example (4) that would preserve the original homoglyph forms is only present in the .xml format; the .txt (AntConc) and .vrt (SkE) preprocessed corpus files do not contain that metadata. A researcher using the .txt or .vrt versions cannot reconstruct, for instance, the distinction between the double-struck 𝕃 and the ordinary L, or between decomposed and precomposed é. The claim should be limited to 'no loss beyond what the metadata format captures', or the pipeline should be changed so that every distributed format carries the metadata needed for reversal.
minor comments (5)
  1. [§3.1, Table 7] Please clarify whether the 5,034-item test set is the union of Unicode emoji and aliases; if aliases are included as separate lines, this should be stated in the caption of Table 7 and in Section 4.1, since 'all current Unicode emojis' and 'the library's emoji plus aliases' are different populations.
  2. [Table 12] The column heading 'Homoglyphs (as recognised by the library unicodedata)' is imprecise: `unicodedata` does not provide homoglyph detection; it provides character data and normalisation. Please state that the count is of characters whose NFKC form differs from the original, or otherwise describe the counting procedure exactly.
  3. [§3.3] The statement that 'the use of the library emoji guarantees that emoji characters and sequences are correctly interpreted' is too absolute; the library is a finite and versioned list and can be incomplete or outdated. Please rephrase as 'are interpreted according to the library's list'.
  4. [§5] The sentence beginning 'its complexity and the increasing number of characters have not kept pace with advances in former' contains a typo: 'advances in former' should presumably be 'advances in the latter' or similar. Please revise.
  5. [§3.2] The 200,000-token cutoff is described as arbitrary; please state explicitly whether the real-data sample is intended as illustrative rather than representative, since the generalised 'ensures fidelity' wording in the conclusion goes beyond what one convenience sample can demonstrate.

Circularity Check

1 steps flagged · score 2.0 of 10

Fidelity validation is closed-loop: the same Python emoji/unicodedata libraries define preprocessing and the source-data ground truth, but the paper's direct tool-behavior observations are independent.

  1. self definitional [Section 4.3 (Tables 12–13) and Sections 3.1/3.3]
    "Emojis and homoglyphs mismatches were only found in the non preprocessed corpora, with the preprocessed ones (both pre-tagged and plain-text versions) faithfully representing both features as they are included in the source data. ... comparison against the list of emojis and homoglyphs effectively present in the source data. ... the total number of emojis as recognised by the library emoji; and the total number of homoglyphs (as recognised by the library unicodedata)."

    The fidelity check compares corpus wordlists against a source-data emoji/homoglyph list that is itself built from the same Python libraries used in preprocessing: emoji for emoji identification/transliteration and unicodedata/NFKC for homoglyph normalization. Any emoji absent from emoji's 5,034-entry list, or any character whose NFKC normalization is contextually inappropriate, is invisible on both sides of the comparison, so the evaluation cannot detect its own blind spots. The conclusion that preprocessing 'faithfully represents' the source data is therefore self-referential rather than independently established. The paper's documented observations of AntConc, LancsBoxX, and SketchEngine splitting behaviour are unaffected and remain independent evidence.

full rationale

No fitted parameters are renamed as predictions, no derivation is shown to reduce to its own inputs, and no load-bearing argument rests on self-citation: the paper reports direct, reproducible observations of how three corpus tools tokenize emojis and homoglyphs, and those observations (Tables 7–10, 13–14) are independent of the preprocessing libraries. The only notable circular aspect is the validation design for the fidelity claim: the 'source data' emoji and homoglyph counts are defined by the same Python emoji and unicodedata libraries that define the preprocessing, so the check is closed-loop and cannot reveal omissions or normalization losses in those libraries. The paper even acknowledges that the transliteration-before-NFKC ordering is 'arbitrary' and that NFKC is 'destructive', and the Section 5 claim of 'without any information loss' is stronger than the .txt/.vrt files support since those files lack the XML metadata said to allow reconstruction. These are overclaims and a mild self-referential evaluation, not a derivation that reduces by construction, hence score 2.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim relies on operational choices rather than free parameters fitted to data. The key assumptions are about the emoji library as ground truth, the use of NFKC as the normalization standard, and the human-centred view of emoji sequences. These are stated transparently but not independently validated.

free parameters (4)
  • Delimiter characters for emoji transliteration = {, }, ^
    Chosen because they do not appear in the source data; used to keep transliterated emojis as single tokens in AntConc's Token Definition (Section 3.3).
  • Order of transliteration relative to NFKC normalization = transliteration before NFKC
    Arbitrary choice based on querying the emoji ™ as {trade^mark} rather than TM; affects 31 emojis (22 unique and 9 variants) that could be normalized (Section 3.3).
  • Token count cutoff for corpus selection = 200,000
    Set arbitrarily to keep the corpus manually inspectable; determined the 13-brand subset (Section 3.2).
  • Unicode normalization form = NFKC
    Chosen to unify homoglyphs despite being destructive; the paper argues metadata preserves original forms (Section 2.3).
assumptions (4)
  • domain assumption NFKC normalization, combined with metadata preservation, is an appropriate way to represent homoglyphs without meaningful information loss.
    Paper adopts NFKC in Section 2.3 despite acknowledging it is destructive; the correctness of this trade-off for corpus linguistics is assumed.
  • domain assumption The Python 'emoji' library (version 2.12.1) provides an accurate and complete list of emojis and CLDR short names.
    Used as the reference for test files and for counting emojis in source data (Sections 3.1, 4.3); no independent verification is provided.
  • domain assumption Human-centred counting, where a ZWJ sequence such as 🏳️🌈 counts as a single emoji, is the correct linguistic ground truth.
    Defines fidelity in Section 2.2; the paper's claim that tools are wrong depends on this perspective.
  • ad hoc to paper The tokenization settings chosen for each tool (e.g., AntConc's Token Definition categories) are representative of how the tools are typically used.
    AntConc required custom settings (Table 11); results are specific to these configurations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data interference: emojis, homoglyphs, and issues of data fidelity in corpora and their results." pith.science (2026). https://pith.science/paper/55IY4NRO

@misc{pith2026250701764,
  author       = {Pith},
  title        = {Pith review of: Data interference: emojis, homoglyphs, and issues of data fidelity in corpora and their results},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/55IY4NRO}},
  note         = {Machine review of arXiv:2507.01764}
}
read the original abstract

Tokenisation - "the process of splitting text into atomic parts" (Brezina & Timperley, 2017: 1) - is a crucial step for corpus linguistics, as it provides the basis for any applicable quantitative method (e.g. collocations) while ensuring the reliability of qualitative approaches. This paper examines how discrepancies in tokenisation affect the representation of language data and the validity of analytical findings: investigating the challenges posed by emojis and homoglyphs, the study highlights the necessity of preprocessing these elements to maintain corpus fidelity to the source data. The research presents methods for ensuring that digital texts are accurately represented in corpora, thereby supporting reliable linguistic analysis and guaranteeing the repeatability of linguistic interpretations. The findings emphasise the necessity of a detailed understanding of both linguistic and technical aspects involved in digital textual data to enhance the accuracy of corpus analysis, and have significant implications for both quantitative and qualitative approaches in corpus-based research.

Figures

Figures reproduced from arXiv: 2507.01764 by the authors.

Figure 1
Figure 1. Graph showing computers characters decoding (adapted from Zappavigna & [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Examples of compatibility composites (Whistler, 2023) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Emoji+homoglyphs tokenisation in AntConc and LancsBoxX [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 13 canonical work pages

  1. [1]

    running word

    Introduction Corpus linguistics – and quantitative approaches to language data more in general – relies on the ability of computers to count elements in a corpus. Computers may not however always handle this task in the same way a human would, due to variables such as defining what constitutes an element 6 with corpus linguistics commonly setting a “runni...

  2. [2]

    a standard in corpus building

    Humans and computers reading digital textual data Textual data is read by computers, like any other content in digital format, as sequences of bits – 0s and 1s 6 subsequently interpreted according to sets of rules called ‘encodings’ that allow the contents to be rendered. In corpus linguistics, encodings are essential since they ensure that texts are cons...

  3. [3]

    Face with Spiral Eyes

    ‘emoji sequence’ refers to (c); e.g. the emoji 😵‍💫 (“Face with Spiral Eyes”) is inter6 nally composed of the two emoji characters 😵 and 💫 (“Dizzy”) joined by the Zero6Width Joiner component (see further below)

  4. [4]

    Different ‘emoji images’ may however appear to different readers (see further below)

    ‘glyph’ refers to (a); in this paper it is a synonym of emoji, as the three emojis reported in Example (3). Different ‘emoji images’ may however appear to different readers (see further below)

  5. [5]

    Face with Crossed6Out Eyes

    ‘emoji character’ refers to (b); e.g. the two emojis 😵 (“Face with Crossed6Out Eyes”) and 🤹 (“Person Juggling”) are internally represented by a single emoji character each (see Table 3)

  6. [6]

    They Edited Out her Nip Nops

    References Alsulami, A. (2019). A Sociolinguistic Analysis of the Use of Arabizi in Social Media Among Saudi Arabians. International Journal of English Linguistics, 9(6), 257. https:// doi.org/10.5539/ijel.v9n6p257 Andrade, B., Morais, R., & Soares De Lima, E. (2024). The Personality of Visual Ele6 ments: A Framework for the Development of Visual Identity...

  7. [7]

    character[s] that can be used to modify the appearance of a preceding emoji

    ‘emoji modifier’ refers to those “character[s] that can be used to modify the appearance of a preceding emoji” (Davis & Holbrook, 2023, Emoji modifiers section); e.g. in the emoji sequence (“Woman Juggling: Medium Skin Tone”) the Medium Skin Tone is assigned through the Fitzpatrick modifier8 🏽 (“Medium Skin Tone”)

  8. [8]

    displayed as […] capital A..Z character[s] with a special display

    ‘emoji component’ refers to those characters appearing in sequences but not appearing in emoji keyboards as separate choices. Examples of components are the Zero6Width Joiner (ZWJ; codepoint: U+200D), ‘Variation Selectors’, and ‘Regional Indicators’. The latter, if split as separate characters, are “displayed as […] capital A..Z character[s] with a specia...

Show all 14 references
  1. [9]

    artificial data

    Methodology Corpus tools often use custom rules for processing text files, which can affect the handling of UTF68 features like emojis and homoglyphs. This paper stems from the ob6 servation of how three corpus tools (AntConc, LancsBoxX, and SketchEngine) operate differently o...

  2. [10]

    which corpus tool is best

    Consequences of SoI in corpora and their results It is important to remark once again that the aim of the paper is not to suggest “which corpus tool is best”, but rather to discuss a number of their functionalities in relation to the treatment of SoI in corpus files, and eluci...

  3. [12]

    perceived

    Conclusions Despite UTF68 being the most widely used encoding on the web and the preferred choice in corpus linguistics, its complexity and the increasing number of characters have not kept pace with advances in former; and issues related to characters reading identified 20 ye...

  4. [45]

    emoji+homoglyph

    and q (SkE freq.: 35; test file freq.: 37) indicate that the former does not apply any normalisation to characters from blocks Latin Extended6D and Latin Extended6F; Table 10 contains the c and q homoglyphs that SkE does not normalise. Non6normalised homoglyphs Codepoint Block...

  5. [558]

    https://doi.org/10.1109/ASONAM.2016.7752290 Moore, A., & Rayson, P. (2022). PyMUSAS: Python Multilingual Ucrel Semantic Analysis System (0.3.0) [Computer software]. https://github.com/ucrel/pymusas preprint 24 of 26 Data interference | Di Cristofaro, 2025 Moran, S., & Cysouw, ...

  6. [2023]

    becoming pragmatically unmarked

    6, whereas Unix6based ones default to UTF68 (Gries, 2016: 111). Due to these complexities, the data and scripts used in this paper are processed on a Unix6based system with UTF68 encoding and LF for line breaks7 to minimise potential technical issues. 2.2. Emojis As a result o...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.