{"id":"8291d27d-f6f7-410f-884f-effd2f629ff4","arxiv_id":"2607.22456","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new open-source library computes distance, similarity, and evaluation metrics on grapheme clusters rather than Unicode code points, and corrects ZWJ/ZWNJ segmentation for Tamil and Sinhala.","lead":"grapheme-kit is a Python library that measures text errors and similarity using whole visible characters (graphemes) instead of raw Unicode symbols, and it fixes common mis-splits in Tamil and Sinhala text. An OCR experiment across 12 languages shows the new metrics change error scores dramatically for scripts that pack several Unicode symbols into one visible character.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Segmentation correctness is unvalidated and the Tamil path is internally inconsistent: §A.1 strips ZWJ before the §3.2 ZWJ-triggered folding rule can run, yet Table 2’s Tamil example contains no ZWJ; all headline OCR metrics inherit this uncertainty.","rationale":"The reader identified segmentation correctness as the weakest assumption; my stress-test concurs and sharpens it with a concrete internal inconsistency. The paper’s central value proposition—grapheme-level metrics are more faithful—requires that the segmenter matches user-perceived units and that normalization does not conflate distinct strings. The §A.1 Tamil ZWJ-removal rule and the §3.2 ZWJ-based folding rule are contradictory as written, and Table 2’s Tamil example lacks the very ZWJ that the rule is said to require. This is load-bearing because the entire evaluation pipeline consumes Graphemizer’s output; if ZWJ removal precedes segmentation, the claimed Tamil behavior cannot arise from the named mechanism, and if it does arise, it must come from an undocumented rule. Either way, the headline OCR numbers are not anchored to a clear, reproducible description. I do not raise this as evidence of fraud or sloppiness in the artifact: the library is public, the perturbation study is informative, and the large CER reductions are partly definitional, as the paper acknowledges. But the paper offers no human-segmentation validation, and its own Limitations text concedes dependence on the segmentation library. Thus the ‘more faithful’ claim is under-supported and the mechanism description is ambiguous. The appropriate verdict remains CONDITIONAL: the artifact is real and checkable, but the evaluation claims need repair and direct segmentation validation before being relied on. My read does not change the reader’s verdict, hence UNCHANGED.","tokens_in":10261,"tokens_out":8516,"duration_ms":89347,"concrete_test":"Run the released package on the exact codepoint sequences from Table 2 and Table 1 for Tamil and Sinhala, with and without the §A.1 normalization enabled: (i) verify whether ‘ஸ்ரீ’ and ‘ශ්‍රී’ are emitted as single clusters; (ii) trace whether normalization removes ZWJ before Graphemizer; (iii) collect human boundary annotations from native speakers on 200–500 sample strings and compare boundary precision/recall against grapheme-kit, the baseline grapheme library, and Unicode UAX #29 InCB segmentation. If the Tamil path never depends on a stripped ZWJ and human agreement is high, the concern is resolved; if the described mechanism does not reproduce the claimed clusters, or human boundaries disagree on ZWJ chains, the Table 4 improvements must be re-qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—Table 4’s OCR improvements (Tamil CER 5.48→0.62; Sinhala 3.80→0.92) and the ‘more faithful’ framing—rests on segmentation correctness. The paper’s own Limitations section concedes ‘the accuracy of our evaluation metrics depends on the underlying grapheme segmentation library.’ No human segmentation judgments are reported. Worse, the described pipeline is internally inconsistent: §3.1 routes raw text through language-specific normalization before Graphemizer, and §A.1 says Tamil normalization ‘remove[s] Zero Width Joiner (ZWJ) and Zero Width Non-Joiner (ZWNJ) characters’; yet §3.2 says Graphemizer’s core correction is to ‘detect[] ZWJ linked sequences and recursively fold[] every consonant, virama, and vowel modifier joined by ZWJ into a single grapheme cluster.’ A ZWJ-triggered rule cannot fire after ZWJs have been stripped. Table 2’s Tamil codepoint row for ‘Sri Lanka’ contains no ZWJ at all (‘ஸ’, ‘்’, ‘ர’, ‘ீ’, …), so the claimed output ‘ஸ்ரீ’ as one cluster cannot be produced by the stated ZWJ rule. Either the shipped code contains an undocumented Tamil-specific virama-combining rule, or the normalization is bypassed in the experiments; the paper does not say which. Since Tables 4–5 and the perturbation study are computed over clusters produced downstream of this stage, every headline number inherits the ambiguity. Table 3’s inflation ratios are computed with the same segmenter, so over-merging would amplify the apparent benefit as well.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"grapheme-kit is a Python library that shifts lexical distance, similarity, and evaluation metrics from Unicode code points to grapheme clusters. It introduces a Graphemizer intended to correct Tamil and Sinhala cluster segmentation (particularly ZWJ-linked sequences), plus vowel-consonant composition/decomposition utilities. The empirical section reports an OCR case study across 12 languages showing large gains for grapheme-level CER and chrF++ in complex scripts (e.g., Tamil CER 5.48→0.62, Sinhala 3.80→0.92) and a DEMETR-style controlled perturbation study aimed at isolating unit-choice effects. The paper concludes that grapheme-level metrics provide a more faithful evaluation of complex scripts.","tokens_in":10588,"tokens_out":5548,"duration_ms":61556,"significance":"The library addresses a real gap in multilingual NLP tooling, and the open-source release, documentation, and CLI are concrete practical contributions. The controlled perturbation design is a appropriate way to separate unit choice from edit-distance mechanics, and the paper is unusually candid about its limitations. However, the central empirical claim that grapheme-level evaluation is 'more faithful' is load-bearing on the correctness of the Graphemizer, which is not validated against any gold standard. One described preprocessing path also appears internally inconsistent with the stated ZWJ rule. If segmentation correctness is established, the contribution is solid; as it stands, the headline numbers inherit a significant unresolved ambiguity.","major_comments":[{"comment":"The claim that grapheme-kit produces 'accurate grapheme clusters' and that Table 4's large improvements reflect more faithful scoring presupposes segmentation correctness. No human annotation, orthographic gold standard, or independent segmenter comparison is provided for Tamil/Sinhala segmentation. The Limitations section explicitly concedes that 'the accuracy of our evaluation metrics depends on the underlying grapheme segmentation library.' Because Table 3's inflation ratios and Tables 4–5 are all computed with the same segmenter, systematic over-merging would inflate both the apparent script complexity and the apparent CER/chrF improvements. Please validate the segmenter against native-speaker/orthographic judgments or an independent implementation, and report disagreement rates per script.","section":"§3.2, Table 2, Tables 4–5, Limitations"},{"comment":"The processing pipeline is internally inconsistent. §3.1 says raw text passes through normalization before Graphemizer, and §A.1 says Tamil normalization 'remove[s] Zero Width Joiner (ZWJ) and Zero Width Non-Joiner (ZWNJ) characters'; yet §3.2 says Graphemizer's core correction is to 'detect[] ZWJ linked sequences' and fold them into single clusters. A ZWJ-triggered rule cannot fire after ZWJ has been stripped. Moreover, Table 2's Tamil codepoint row for 'Sri Lanka' contains no ZWJ at all, so the output 'ஸ்ரீ' as a single cluster cannot be produced by the stated ZWJ rule. Please clarify the actual code path: whether normalization is bypassed in experiments, applied conditionally, or whether an undocumented Tamil-specific virama-combining rule exists. The headline numbers need to be rerun or justified under the clarified pipeline.","section":"§3.1, §A.1, §3.2, Table 2"},{"comment":"The Conclusion states that 'multilingual case studies across machine translation, automatic speech recognition, and optical character recognition' demonstrate the benefits of grapheme-level evaluation. The paper only reports an OCR case study (§4.1) and synthetic perturbations (§4.2); no MT or ASR experiments are described anywhere. This overclaim should be removed, or the corresponding experiments added.","section":"§5 (Conclusion)"},{"comment":"The stronger conclusion that grapheme-level metrics are 'more faithful' for complex scripts requires external validation beyond internal metric behavior. The controlled perturbation study shows that grapheme-CER and char-CER respond differently to codepoint-level vs. grapheme-level deletions, and it is a good diagnostic. But it does not demonstrate which metric's response better matches human perception of OCR errors, translation quality, or reading difficulty. I recommend either adding a human correlation/error-judgment study, or rephrasing the claims as demonstrating sensitivity to unit choice rather than overall faithfulness.","section":"§4.2, Table 5, Abstract"}],"minor_comments":[{"comment":"Typos and rendering artifacts: 'libarary' in §3.2, and the code examples in §3.5 contain LaTeX artifacts such as '/quotesingle.ts1' that make them unreadable. Please regenerate the examples or show clean terminal output.","section":"§3.2, §3.5"},{"comment":"The table layout is visually mangled by combining marks and spacing in the Tamil/Sinhala examples. Use explicit codepoint notation (U+xxxx) or a monospaced font with clear cell separation so the comparisons are interpretable.","section":"Table 1"},{"comment":"The grapheme inflation ratio is defined informally. Please state the exact formula (e.g., mean over sentences of graphemes/codepoints, or pooled over the corpus) and report the standard deviation; this would strengthen Table 3.","section":"§4.1"},{"comment":"Only mean values over 12 languages are reported. Since the Arabic and Latin controls behave very differently, provide per-language or at least per-group breakdowns so the reader can verify which scripts drive the reported deltas.","section":"Table 5"},{"comment":"The reference to Elkhayati et al. contains a typo ('Y oussfi'). Also, Figure 2 is referenced but does not appear in the text as rendered; ensure all figures are included in the submission.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I agree with the conditional assessment. The most serious issue is the normalization/ZWJ inconsistency: if normalization is applied as described, the segmenter's stated mechanism is dead code, and Table 2's Tamil example cannot be produced by the stated rule. This must be clarified and the segmentation validated before the paper can be accepted. The library itself is a useful systems contribution, especially if the overclaims in the conclusion are corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real, pip-installable library with sensible metric extensions and an honest perturbation study, but the paper's core segmentation claim has a load-bearing internal inconsistency, and the OCR numbers inherit it. Worth engaging, but only after the segmentation path is clarified and validated.\n\nWhat is actually new: the segmentation layer for Tamil and Sinhala — grouping ZWJ-linked consonant clusters in Sinhala and, allegedly, virama sequences in Tamil that off-the-shelf segmenters split. That is real engineering, and the comparison against four existing libraries (Table 1) is useful. The metric extensions themselves (CER, chrF++, charBLEU on grapheme tokens) are straightforward compositions of sacrebleu and textdistance; the novelty is the unit choice, not the math. The DEMETR-style perturbation study correctly isolates unit-choice effects, and the paper candidly reports that grapheme-chrF behaves counterintuitively on clean substitutions (§4.2). Credit also for shipping code and data and for the explicit limitations section.\n\nThe soft spots are real and, in one case, serious. First: the segmentation mechanism is internally inconsistent. §3.2 says the graphemizer's correction is to recursively fold ZWJ-linked sequences into one cluster. §A.1 says Tamil normalization removes ZWJ/ZWNJ. If that runs before graphemization, the ZWJ rule can never fire for Tamil. And Table 2's Tamil example contains no ZWJ at all — the claimed single cluster 'ஸ்ரீ' cannot be produced by the stated rule. Either the code has an undocumented Tamil-specific virama rule, or the normalization is bypassed in the experiments; the paper does not say. Every headline number — the Table 4 CER drop, the chrF++ gains, the inflation ratios — is computed over clusters from this pipeline, so they all inherit the ambiguity. Second, no human segmentation judgments are reported; the only evidence for 'correct' segmentation is the library's own output. Third, the conclusion says the metrics were demonstrated across MT, ASR, and OCR, but only OCR was run. Fourth, the grapheme n in chrF++ is 'tuned' without saying how or on what; part of the difference may reflect n relative to unit length, not unit choice alone. The PDF rendering garbles the tables and code examples that carry the claims, which doesn't help.\n\nNone of this kills the artifact. If the code works as the README says, it is a useful tool for Indic NLP evaluation. But the 'more faithful' framing is not yet supported. For peer review: send it out, but require the authors to fix the pipeline description, validate segmentation against human judgments (or at least a linguistically explicit rule), add error bars to the OCR numbers, and cut the MT/ASR claims. I would not cite it as evidence for grapheme-level evaluation until that is done.","headline":"A genuinely useful open-source metric library whose central segmentation claim is undercut by an internal pipeline contradiction and lack of validation; worth refereeing but not yet citable as evidence.","tokens_in":11201,"tokens_out":4552,"would_cite":false,"duration_ms":51074,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Counting visual characters instead of raw code points changes measured OCR error in Tamil and Sinhala.","keywords":["grapheme clusters","Unicode","Tamil","Sinhala","OCR evaluation","character error rate","chrF++","zero width joiner"],"falsifier":"Show native Tamil and Sinhala readers a set of words containing ZWJ-linked sequences like 'ஸ்ரீ' and 'ශ්රී' and ask them to mark character boundaries; if a substantial fraction treats the ZWJ sequence as two characters, or conversely if the library leaves a sequence split that readers perceive as one, the central claim collapses. Alternatively, a labeled OCR error corpus with human judgment of which outputs are 'correct' would settle whether grapheme-CER tracks perceived quality better than code-point CER.","tokens_in":10074,"feed_emoji":"🔤","tokens_out":5113,"duration_ms":54007,"temperature":0.7,"pith_summary":"The paper introduces grapheme-kit, a Python library that rebuilds standard lexical metrics — Levenshtein and Hamming distance, Jaro similarity, CER, chrF/chrF++, and CharBLEU — so they operate on grapheme clusters (user-perceived characters) rather than raw Unicode code points. Its core claim is that for scripts like Tamil and Sinhala, where one visible character is usually several code points, code-point-based metrics systematically overstate errors. In an OCR case study, grapheme-CER drops Tamil's error rate from 5.48 to 0.62 and Sinhala's from 3.80 to 0.92, while grapheme-chrF++ rises by up to 24.94 points; Latin and Arabic controls barely move. The library also fixes a segmentation flaw: Zero-Width-Joiner chains that form a single consonant cluster (like 'ஸ்ரீ' and 'ශ්රී') are folded into one grapheme, where four existing libraries split them. The benefit, if the paper is right, is that evaluation of low-resource and complex-script NLP finally measures what readers actually see.","feed_headline":"Grapheme metrics slash reported OCR error for Tamil and Sinhala","feed_subtitle":"Counting visual characters instead of raw code points reveals OCR is far more accurate than standard metrics suggest.","key_machinery":"The load-bearing component is the Graphemizer, a segmentation layer that first NFC-normalizes text and then applies a recursive rule: any consonant, virama, and vowel modifier joined by Zero Width Joiner (or handled by script-specific normalization) is folded into a single grapheme cluster, no matter how many ZWJs are in the chain. This fixes cases like 'ශ්රී' and 'ஸ்ரீ', which standard Unicode segmentation splits into multiple clusters. All distance, similarity, and evaluation metrics are then recomputed over the resulting cluster sequence, so a multi-code-point grapheme costs one edit operation rather than several. The library also provides composition/decomposition between clusters and th","core_discovery":"On its own terms, the paper's discovery is that the unit of evaluation matters as much as the metric itself. When a dependent vowel sign is misrecognized by OCR, code-point evaluation counts one extra edit per code point, even though the reader sees a single corrupted character. By redefining the lexical unit as the grapheme cluster, the paper shows that Tamil OCR error falls from 5.48 to 0.62 CER and Sinhala from 3.80 to 0.92, and that grapheme-chrF++ improves by 24.94 and 12.43 points respectively, while languages where graphemes map 1:1 to code points (Dutch, French, Arabic) show no change. A controlled perturbation study reinforces the mechanism: deleting a single code point (corrupting","pith_inferences":["If the ZWJ-folding rule matches human character judgments — which the paper does not test directly — then similar error-reduction effects should appear in Malayalam, Telugu, and other Brahmic scripts with comparable grapheme inflation; the paper reports OCR only for a subset.","Grapheme-CER's sharper penalty for a corrupted-but-single grapheme (codepoint deletion, +0.60) implies that a system that drops a vowel sign is judged as making a full character error; whether that matches reader perception of 'almost right' spellings is an empirical question the paper leaves open.","The grapheme-chrF 'unforgiving' behavior on clean substitutions suggests that simply switching the unit does not make chrF monotone in perceived quality; a normalized or length-adjusted variant may be needed before grapheme-chrF can be used for system ranking.","Because the library exposes composition/decomposition only for Tamil and Sinhala, the strongest testable extension is to add ZWJ-aware segmentation for other Indic scripts and check whether the OCR-case-study pattern repeats."],"forward_implications":["For OCR, ASR, and MT evaluation in abugida and Indic scripts, reported error rates will shift substantially once grapheme clusters replace code points; the paper shows this directly for OCR.","Existing results on Tamil and Sinhala measured with code-point CER or chrF may have painted systems as worse than they are; re-evaluating with grapheme metrics could change model rankings and comparisons.","Grapheme-aware tokenization and evaluation can be unified around the same segmentation, reducing fragmentation in downstream NLP pipelines.","The ZWJ-folding rule likely generalizes to other Brahmic scripts with similar orthographic patterns, since the paper attributes the errors to ZWJ handling.","The controlled perturbation results imply that edit-based metrics (CER) and n-gram overlap metrics (chrF++) behave differently under grapheme units; a normalization strategy for grapheme-chrF remains an open issue."],"fun_headline_variants":["Grapheme clusters cut reported Tamil OCR error from 5.48 to 0.62","Grapheme-level metrics reveal true OCR accuracy for Tamil and Sinhala","Count graphemes, not code points, for fair OCR evaluation","Grapheme metrics: Tamil OCR error 5.48→0.62, Sinhala 3.80→0.92","Why code-point metrics mislead on OCR for Tamil and Sinhala"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire metric suite inherits the correctness of the Graphemizer rule that folds every ZWJ-joined consonant, virama, and vowel modifier into one cluster; if that rule over-merges or under-merges relative to what native readers treat as a character, every reported CER and chrF number inherits the same error, and the paper itself acknowledges this dependency in its Limitations section.","fun_headline_variants_meta":{"raw":{"variants":["Grapheme clusters cut reported Tamil OCR error from 5.48 to 0.62","Grapheme-level metrics reveal true OCR accuracy for Tamil and Sinhala","Count graphemes, not code points, for fair OCR evaluation","Grapheme metrics: Tamil OCR error 5.48→0.62, Sinhala 3.80→0.92","Why code-point metrics mislead on OCR for Tamil and Sinhala"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":2803,"prompt_tokens":651,"completion_tokens":2152,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":395,"tokens_out":2152,"duration_ms":20383,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:40:28.421314+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Show native Tamil and Sinhala readers a set of words containing ZWJ-linked sequences like 'ஸ்ரீ' and 'ශ්රී' and ask them to mark character boundaries; if a substantial fraction treats the ZWJ sequence as two characters, or conversely if the library leaves a sequence split that readers perceive as one, the central claim collapses. Alternatively, a labeled OCR error corpus with human judgment of which outputs are 'correct' would settle whether grapheme-CER tracks perceived quality better than code-point CER.","supporting_citations":[],"review_version":1}