grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP

Ashini Kavindya; Izzath Nisfer; Menan Velayuthan; Ovindu Atukorala; Purushoth Velayuthan

REVIEW 4 major objections 5 minor 4 references

grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counting visual characters instead of raw code points changes measured OCR error in Tamil and Sinhala.

desk verdict A genuinely useful open-source metric library whose central segmentation claim is undercut by an internal pipeline contradiction and lack of validation; worth refereeing but not yet citable as evidence. read the letter →

arxiv 2607.22456 v1 pith:S47D2X7V submitted 2026-07-24 cs.CL

Izzath Nisfer , Ashini Kavindya , Ovindu Atukorala , Purushoth Velayuthan , Menan Velayuthan This is my paper · ORCID

classification cs.CL

keywords graphemeclustersUnicodeTamilSinhalaOCRevaluationcharactererrorratechrF++zerowidthjoiner

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces grapheme-kit, a Python library that rebuilds standard lexical metrics — Levenshtein and Hamming distance, Jaro similarity, CER, chrF/chrF++, and CharBLEU — so they operate on grapheme clusters (user-perceived characters) rather than raw Unicode code points. Its core claim is that for scripts like Tamil and Sinhala, where one visible character is usually several code points, code-point-based metrics systematically overstate errors. In an OCR case study, grapheme-CER drops Tamil's error rate from 5.48 to 0.62 and Sinhala's from 3.80 to 0.92, while grapheme-chrF++ rises by up to 24.94 points; Latin and Arabic controls barely move. The library also fixes a segmentation flaw: Zero-Width-Joiner chains that form a single consonant cluster (like 'ஸ்ரீ' and 'ශ්රී') are folded into one grapheme, where four existing libraries split them. The benefit, if the paper is right, is that evaluation of low-resource and complex-script NLP finally measures what readers actually see.

What carries the argument

The load-bearing component is the Graphemizer, a segmentation layer that first NFC-normalizes text and then applies a recursive rule: any consonant, virama, and vowel modifier joined by Zero Width Joiner (or handled by script-specific normalization) is folded into a single grapheme cluster, no matter how many ZWJs are in the chain. This fixes cases like 'ශ්රී' and 'ஸ்ரீ', which standard Unicode segmentation splits into multiple clusters. All distance, similarity, and evaluation metrics are then recomputed over the resulting cluster sequence, so a multi-code-point grapheme costs one edit operation rather than several. The library also provides composition/decomposition between clusters and th

What would settle it

Show native Tamil and Sinhala readers a set of words containing ZWJ-linked sequences like 'ஸ்ரீ' and 'ශ්රී' and ask them to mark character boundaries; if a substantial fraction treats the ZWJ sequence as two characters, or conversely if the library leaves a sequence split that readers perceive as one, the central claim collapses. Alternatively, a labeled OCR error corpus with human judgment of which outputs are 'correct' would settle whether grapheme-CER tracks perceived quality better than code-point CER.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the unit of evaluation matters as much as the metric itself. When a dependent vowel sign is misrecognized by OCR, code-point evaluation counts one extra edit per code point, even though the reader sees a single corrupted character. By redefining the lexical unit as the grapheme cluster, the paper shows that Tamil OCR error falls from 5.48 to 0.62 CER and Sinhala from 3.80 to 0.92, and that grapheme-chrF++ improves by 24.94 and 12.43 points respectively, while languages where graphemes map 1:1 to code points (Dutch, French, Arabic) show no change. A controlled perturbation study reinforces the mechanism: deleting a single code point (corrupting

Load-bearing premise

The entire metric suite inherits the correctness of the Graphemizer rule that folds every ZWJ-joined consonant, virama, and vowel modifier into one cluster; if that rule over-merges or under-merges relative to what native readers treat as a character, every reported CER and chrF number inherits the same error, and the paper itself acknowledges this dependency in its Limitations section.

Editorial extensions

If this is right

For OCR, ASR, and MT evaluation in abugida and Indic scripts, reported error rates will shift substantially once grapheme clusters replace code points; the paper shows this directly for OCR.
Existing results on Tamil and Sinhala measured with code-point CER or chrF may have painted systems as worse than they are; re-evaluating with grapheme metrics could change model rankings and comparisons.
Grapheme-aware tokenization and evaluation can be unified around the same segmentation, reducing fragmentation in downstream NLP pipelines.
The ZWJ-folding rule likely generalizes to other Brahmic scripts with similar orthographic patterns, since the paper attributes the errors to ZWJ handling.
The controlled perturbation results imply that edit-based metrics (CER) and n-gram overlap metrics (chrF++) behave differently under grapheme units; a normalization strategy for grapheme-chrF remains an open issue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the ZWJ-folding rule matches human character judgments — which the paper does not test directly — then similar error-reduction effects should appear in Malayalam, Telugu, and other Brahmic scripts with comparable grapheme inflation; the paper reports OCR only for a subset.
Grapheme-CER's sharper penalty for a corrupted-but-single grapheme (codepoint deletion, +0.60) implies that a system that drops a vowel sign is judged as making a full character error; whether that matches reader perception of 'almost right' spellings is an empirical question the paper leaves open.
The grapheme-chrF 'unforgiving' behavior on clean substitutions suggests that simply switching the unit does not make chrF monotone in perceived quality; a normalized or length-adjusted variant may be needed before grapheme-chrF can be used for system ranking.
Because the library exposes composition/decomposition only for Tamil and Sinhala, the strongest testable extension is to add ZWJ-aware segmentation for other Indic scripts and check whether the OCR-case-study pattern repeats.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a real, pip-installable library with sensible metric extensions and an honest perturbation study, but the paper's core segmentation claim has a load-bearing internal inconsistency, and the OCR numbers inherit it. Worth engaging, but only after the segmentation path is clarified and validated.

What is actually new: the segmentation layer for Tamil and Sinhala — grouping ZWJ-linked consonant clusters in Sinhala and, allegedly, virama sequences in Tamil that off-the-shelf segmenters split. That is real engineering, and the comparison against four existing libraries (Table 1) is useful. The metric extensions themselves (CER, chrF++, charBLEU on grapheme tokens) are straightforward compositions of sacrebleu and textdistance; the novelty is the unit choice, not the math. The DEMETR-style perturbation study correctly isolates unit-choice effects, and the paper candidly reports that grapheme-chrF behaves counterintuitively on clean substitutions (§4.2). Credit also for shipping code and data and for the explicit limitations section.

The soft spots are real and, in one case, serious. First: the segmentation mechanism is internally inconsistent. §3.2 says the graphemizer's correction is to recursively fold ZWJ-linked sequences into one cluster. §A.1 says Tamil normalization removes ZWJ/ZWNJ. If that runs before graphemization, the ZWJ rule can never fire for Tamil. And Table 2's Tamil example contains no ZWJ at all — the claimed single cluster 'ஸ்ரீ' cannot be produced by the stated rule. Either the code has an undocumented Tamil-specific virama rule, or the normalization is bypassed in the experiments; the paper does not say. Every headline number — the Table 4 CER drop, the chrF++ gains, the inflation ratios — is computed over clusters from this pipeline, so they all inherit the ambiguity. Second, no human segmentation judgments are reported; the only evidence for 'correct' segmentation is the library's own output. Third, the conclusion says the metrics were demonstrated across MT, ASR, and OCR, but only OCR was run. Fourth, the grapheme n in chrF++ is 'tuned' without saying how or on what; part of the difference may reflect n relative to unit length, not unit choice alone. The PDF rendering garbles the tables and code examples that carry the claims, which doesn't help.

None of this kills the artifact. If the code works as the README says, it is a useful tool for Indic NLP evaluation. But the 'more faithful' framing is not yet supported. For peer review: send it out, but require the authors to fix the pipeline description, validate segmentation against human judgments (or at least a linguistically explicit rule), add error bars to the OCR numbers, and cut the MT/ASR claims. I would not cite it as evidence for grapheme-level evaluation until that is done.

Referee Report

4 major / 5 minor

Summary. grapheme-kit is a Python library that shifts lexical distance, similarity, and evaluation metrics from Unicode code points to grapheme clusters. It introduces a Graphemizer intended to correct Tamil and Sinhala cluster segmentation (particularly ZWJ-linked sequences), plus vowel-consonant composition/decomposition utilities. The empirical section reports an OCR case study across 12 languages showing large gains for grapheme-level CER and chrF++ in complex scripts (e.g., Tamil CER 5.48→0.62, Sinhala 3.80→0.92) and a DEMETR-style controlled perturbation study aimed at isolating unit-choice effects. The paper concludes that grapheme-level metrics provide a more faithful evaluation of complex scripts.

Significance. The library addresses a real gap in multilingual NLP tooling, and the open-source release, documentation, and CLI are concrete practical contributions. The controlled perturbation design is a appropriate way to separate unit choice from edit-distance mechanics, and the paper is unusually candid about its limitations. However, the central empirical claim that grapheme-level evaluation is 'more faithful' is load-bearing on the correctness of the Graphemizer, which is not validated against any gold standard. One described preprocessing path also appears internally inconsistent with the stated ZWJ rule. If segmentation correctness is established, the contribution is solid; as it stands, the headline numbers inherit a significant unresolved ambiguity.

major comments (4)

[§3.2, Table 2, Tables 4–5, Limitations] The claim that grapheme-kit produces 'accurate grapheme clusters' and that Table 4's large improvements reflect more faithful scoring presupposes segmentation correctness. No human annotation, orthographic gold standard, or independent segmenter comparison is provided for Tamil/Sinhala segmentation. The Limitations section explicitly concedes that 'the accuracy of our evaluation metrics depends on the underlying grapheme segmentation library.' Because Table 3's inflation ratios and Tables 4–5 are all computed with the same segmenter, systematic over-merging would inflate both the apparent script complexity and the apparent CER/chrF improvements. Please validate the segmenter against native-speaker/orthographic judgments or an independent implementation, and report disagreement rates per script.
[§3.1, §A.1, §3.2, Table 2] The processing pipeline is internally inconsistent. §3.1 says raw text passes through normalization before Graphemizer, and §A.1 says Tamil normalization 'remove[s] Zero Width Joiner (ZWJ) and Zero Width Non-Joiner (ZWNJ) characters'; yet §3.2 says Graphemizer's core correction is to 'detect[] ZWJ linked sequences' and fold them into single clusters. A ZWJ-triggered rule cannot fire after ZWJ has been stripped. Moreover, Table 2's Tamil codepoint row for 'Sri Lanka' contains no ZWJ at all, so the output 'ஸ்ரீ' as a single cluster cannot be produced by the stated ZWJ rule. Please clarify the actual code path: whether normalization is bypassed in experiments, applied conditionally, or whether an undocumented Tamil-specific virama-combining rule exists. The headline numbers need to be rerun or justified under the clarified pipeline.
[§5 (Conclusion)] The Conclusion states that 'multilingual case studies across machine translation, automatic speech recognition, and optical character recognition' demonstrate the benefits of grapheme-level evaluation. The paper only reports an OCR case study (§4.1) and synthetic perturbations (§4.2); no MT or ASR experiments are described anywhere. This overclaim should be removed, or the corresponding experiments added.
[§4.2, Table 5, Abstract] The stronger conclusion that grapheme-level metrics are 'more faithful' for complex scripts requires external validation beyond internal metric behavior. The controlled perturbation study shows that grapheme-CER and char-CER respond differently to codepoint-level vs. grapheme-level deletions, and it is a good diagnostic. But it does not demonstrate which metric's response better matches human perception of OCR errors, translation quality, or reading difficulty. I recommend either adding a human correlation/error-judgment study, or rephrasing the claims as demonstrating sensitivity to unit choice rather than overall faithfulness.

minor comments (5)

[§3.2, §3.5] Typos and rendering artifacts: 'libarary' in §3.2, and the code examples in §3.5 contain LaTeX artifacts such as '/quotesingle.ts1' that make them unreadable. Please regenerate the examples or show clean terminal output.
[Table 1] The table layout is visually mangled by combining marks and spacing in the Tamil/Sinhala examples. Use explicit codepoint notation (U+xxxx) or a monospaced font with clear cell separation so the comparisons are interpretable.
[§4.1] The grapheme inflation ratio is defined informally. Please state the exact formula (e.g., mean over sentences of graphemes/codepoints, or pooled over the corpus) and report the standard deviation; this would strengthen Table 3.
[Table 5] Only mean values over 12 languages are reported. Since the Arabic and Latin controls behave very differently, provide per-language or at least per-group breakdowns so the reader can verify which scripts drive the reported deltas.
[References] The reference to Elkhayati et al. contains a typo ('Y oussfi'). Also, Figure 2 is referenced but does not appear in the text as rendered; ensure all figures are included in the submission.

Circularity Check

1 steps flagged · score 4.0 of 10

Grapheme-level metric 'improvements' in Table 4 are a definitional consequence of the unit choice; segmentation correctness for Tamil/Sinhala is unvalidated, but the paper is transparent about the mechanism.

self definitional [Section 4.1, 'Script complexity' paragraph (after Table 3)]
"Such scripts are expected to benefit most from grapheme-level metrics, as a single grapheme spanning multiple Unicode code points may otherwise incur multiple edit operations under codepoint-level evaluation."

The 'benefit' of grapheme-level metrics is stated as a direct consequence of defining the metric over grapheme clusters: one multi-codepoint grapheme is scored as one unit. Table 4's lower grapheme-CER and higher grapheme-chrF++ for Tamil/Sinhala therefore follow from the unit definition rather than from an independent measurement of 'faithfulness.' The paper presents this expected mechanism as the experimental demonstration, so the supporting evidence reduces to the definition of the metric. The controlled perturbation study isolates the unit effect but does not validate that the chosen grapheme boundaries match human perception.

full rationale

The core library contribution—segmentation and metrics over grapheme units—is a tooling contribution, not a derived scientific prediction. The paper explicitly says the OCR improvement is 'expected' from grapheme inflation, and the controlled perturbations show the differences 'arise from the choice of evaluation unit rather than the edit-distance computation itself,' which is transparent. Thus the headline numbers are partly definitional rather than an independent empirical validation. The self-citation to Velayuthan and Sarveswaran (2025) is not load-bearing for the metric results. However, the paper's own Limitations concede that 'the accuracy of our evaluation metrics depends on the underlying grapheme segmentation library,' and no human segmentation judgments are reported. There is also an internal tension: Section A.1 removes ZWJ/ZWNJ in Tamil normalization, while Section 3.2's ZWJ-folding rule is described as the mechanism for Sinhala; Table 2's Tamil example contains no ZWJ, so the Tamil clustering rule is not fully specified. These are validity risks rather than circularity, but they compound the definitional nature of the demonstration.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests mostly on engineering choices (segmentation rules, normalization mappings) rather than fitted parameters; the only data-influenced knob is the 'tuned' grapheme n=3. The interpretive claim 'more faithful' rests on the domain assumption that Unicode extended grapheme clusters match user-perceived characters. No invented entities: 'grapheme inflation ratio' is a descriptive statistic, not a postulated object.

free parameters (3)

chrF/grapheme n-gram order n = n = 3
Table 4 caption: 'chrF++ at tuned grapheme n=3'. The grapheme variant's n-gram order is chosen, not derived, and 'tuned' implies selection on data; tuning details (dev set, grid) are not given.
ZWJ-folding rule = Fold all ZWJ-joined consonant-virama-vowel chains into one cluster
Hand-designed rule in §3.2 ('regardless of how many ZWJs are involved in the chain'). A modeling choice that shapes every downstream metric and all case-study numbers.
Language-specific normalization rule sets = Tamil: ZWJ/ZWNJ removal, reversed-vowel reordering; Sinhala: confusion-set mapping
Hand-crafted transformation rules in §A.1 that define canonical equivalence before segmentation; no evaluation of their correctness beyond a few examples.

assumptions (5)

domain assumption Unicode extended grapheme clusters equal user-perceived characters (graphemes)
Used throughout §1 and §3; the interpretive bridge for 'more faithful'. Standard in UAX #29 work but a modeling assumption, especially for Tamil/Sinhala where akshara-level conventions may differ.
domain assumption The 'grapheme' library's base segmentation is correct; only ZWJ/ZWNJ handling needs patching
§3.2: Graphemizer 'builds on top of grapheme library'; all non-ZWJ behavior is inherited without re-verification in the paper.
standard math NFC normalization is an appropriate canonical form before grapheme processing
§A.1, first normalization stage; standard Unicode-defined behavior.
domain assumption Tesseract + Noto-font OCR errors are representative of real complex-script error distributions
§4.1: the entire generalization of 'more faithful evaluation' rests on this single OCR pipeline.
domain assumption FLORES+ sentences are representative multilingual test text
§4.1: the only corpus used, 1,012 sentences per language, 12 languages.

how reviews work

0 comments

Cite this review

Pith. "Pith review of grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP." pith.science (2026). https://pith.science/paper/S47D2X7V

@misc{pith2026260722456,
  author       = {Pith},
  title        = {Pith review of: grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S47D2X7V}},
  note         = {Machine review of arXiv:2607.22456}
}

read the original abstract

Existing lexical distance, similarity, and evaluation metrics operate on Unicode code points, which can misrepresent errors in writing systems where a single grapheme is represented by multiple Unicode code points. We introduce grapheme-kit, an open-source Python library that extends these metrics to operate on grapheme clusters instead. The library also provides improved grapheme processing for Tamil and Sinhala, including accurate grapheme cluster identification and grapheme composition/decomposition utilities. Through an OCR case study, we demonstrate that grapheme-level metrics provide a more faithful evaluation of complex scripts.

Figures

Figures reproduced from arXiv: 2607.22456 by the authors.

**Figure 1.** , a substantial number of languages exhibit a high proportion of graphemes composed of multiple Unicode code points. This disparity between writing systems that predominantly represent a character using a single Unicode code point and those that require multiple Unicode code points introduces additional computational overhead and inefficiencies in NLP systems (Petrov et al., 2023). Grapheme clusters (referred to … view at source ↗

**Figure 2.** Overview of the grapheme-kit library architec [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith

[2016]

In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers) , pages 1715–1725, Berlin, Germany

Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers) , pages 1715–1725, Berlin, Germany. Association for Computational Lin- guistics. Mirac Suzgun, Stuart Shieber, and Dan Jurafsky. 2024. string2string: A modern python library for s...

2024
[2021]

Preprint, arXiv:2006.14799

Evaluation of text generation: A survey . Preprint, arXiv:2006.14799. Nisansa De Silva. 2019. Survey on publicly available sin- hala natural language processing tools and research. arXiv preprint arXiv:1906.02358. Lu Dong, Zhi-Qiang Guo, Chao-Hong Tan, Y a-Jun Hu, Yuan Jiang, and Zhen-Hua Ling. 2022. Neural grapheme-to-phoneme conversion with pre-trained ...

arXiv 2006
[2024]

Scientific Reports, 14

Multilingual end-to-end asr for low-resource turkic languages with common alphabets . Scientific Reports, 14. Maharaj Brahma, NJ Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Mor- phtok: Morphologically grounded tokenization for indian languages. arXiv preprint arXiv:2504.10335. ...

arXiv 2025
[2025]

In Pro- ceedings of the 31st International Conference on Com- putational Linguistics, pages 5987–5996, Abu Dhabi, UAE

Egalitarian language representation in lan- guage models: It all begins with tokenizers . In Pro- ceedings of the 31st International Conference on Com- putational Linguistics, pages 5987–5996, Abu Dhabi, UAE. Association for Computational Linguistics. A Text Pre-processing A.1 Normalization grapheme-kit provides a language aware normal- ization module for...

2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [2016]

In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers) , pages 1715–1725, Berlin, Germany

Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers) , pages 1715–1725, Berlin, Germany. Association for Computational Lin- guistics. Mirac Suzgun, Stuart Shieber, and Dan Jurafsky. 2024. string2string: A modern python library for s...

2024

[2] [2021]

Preprint, arXiv:2006.14799

Evaluation of text generation: A survey . Preprint, arXiv:2006.14799. Nisansa De Silva. 2019. Survey on publicly available sin- hala natural language processing tools and research. arXiv preprint arXiv:1906.02358. Lu Dong, Zhi-Qiang Guo, Chao-Hong Tan, Y a-Jun Hu, Yuan Jiang, and Zhen-Hua Ling. 2022. Neural grapheme-to-phoneme conversion with pre-trained ...

arXiv 2006

[3] [2024]

Scientific Reports, 14

Multilingual end-to-end asr for low-resource turkic languages with common alphabets . Scientific Reports, 14. Maharaj Brahma, NJ Karthika, Atul Singh, Devaraj Adiga, Smruti Bhate, Ganesh Ramakrishnan, Rohit Saluja, and Maunendra Sankar Desarkar. 2025. Mor- phtok: Morphologically grounded tokenization for indian languages. arXiv preprint arXiv:2504.10335. ...

arXiv 2025

[4] [2025]

In Pro- ceedings of the 31st International Conference on Com- putational Linguistics, pages 5987–5996, Abu Dhabi, UAE

Egalitarian language representation in lan- guage models: It all begins with tokenizers . In Pro- ceedings of the 31st International Conference on Com- putational Linguistics, pages 5987–5996, Abu Dhabi, UAE. Association for Computational Linguistics. A Text Pre-processing A.1 Normalization grapheme-kit provides a language aware normal- ization module for...

2022