Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Tokenization Matters: Improving Zero-Shot NER for Indic Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that SentencePiece tokenization consistently outperforms Byte-Pair Encoding for zero-shot named entity recognition in low-resource Indic languages because it preserves entity boundaries better than frequency-based merging.

desk verdict Sensible question, broken execution: the paper never explains how a BPE-pretrained IndicBERT can be fine-tuned with a SentencePiece tokenizer, so the headline zero-shot gains are uninterpretable as submitted. read the letter →

arxiv 2504.16977 v1 pith:OFIOFJXD submitted 2025-04-23 cs.CL cs.AI

Priyaranjan Pattnayak , Hitesh Laxmichand Patel , Amit Agarwal This is my paper · ORCID
classification cs.CLcs.AI
keywords tokenizationnamedentityrecognitionzero-shotcross-lingualtransferlow-resourceIndiclanguagesSentencePiecebytepairencodingBERTmorphologicalpreservation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tokenization choice is often treated as an implementation detail, but this paper argues it determines whether zero-shot named entity recognition works at all in low-resource Indic languages. The authors compare BPE, SentencePiece, and character-level tokenization, fine-tuning IndicBERT on Hindi and Bengali and transferring zero-shot to Assamese, Oriya, Marathi, Manipuri, Santali, and Sindhi. They report that SentencePiece consistently beats BPE in zero-shot transfer, with F1 scores of 88.38% versus 0.00% on Assamese and 81.09% versus 67.79% on Marathi, and that BPE often degrades to predicting only the non-entity class. The reason, they argue, is that SentencePiece's character-aware subword segmentation preserves morphological structure and entity boundaries, while BPE's frequency-based merging fragments entities when it encounters unseen inflections. If correct, the result gives practitioners a simple, low-cost change—swap the tokenizer—that materially improves zero-shot NER for morphologically rich and under-resourced languages.

What carries the argument

The central object is the match between subword segmentation granularity and named-entity boundary preservation inside a pretrained multilingual transformer. SentencePiece is a whitespace-independent subword tokenizer that can split at character or subword boundaries, which the paper argues retains morphemes in inflectional and agglutinative languages; BPE is a frequency-based byte-pair encoder that merges the most frequent character sequences and produces a compact vocabulary at the cost of over-segmenting rare, unseen, or inflected forms. This machinery carries the argument because the zero-shot results line up with it: SentencePiece keeps entity labels on recognizable subwords, while BPE's merges push entity tokens into fragments the classifier maps to the non-entity class.

What would settle it

Check the released fine-tuning code or checkpoints to see how SentencePiece token IDs are mapped to IndicBERT's pretrained embedding matrix; if the embeddings for SentencePiece tokens are randomly initialized rather than derived from the BPE embeddings, then the reported zero-shot gains could be an artifact of re-training an embedding layer, not a property of SentencePiece itself.

Watch

Extended reading notes

Core claim

The central claim is that SentencePiece is a consistently better tokenization strategy than BPE for named entity recognition in low-resource Indic languages, particularly in zero-shot cross-lingual settings, because it better preserves entity consistency. The paper supports this with intrinsic metrics (tokenization efficiency, OOV rates, morphological preservation) on the FLORES-200 subset and with extrinsic fine-tuning of IndicBERT on Hindi and Bengali followed by zero-shot transfer to six unseen languages. The zero-shot results show SentencePiece at 88.38% F1 on Assamese where BPE gets 0.00%, 81.08% on Oriya where BPE gets 0.00%, 51.98% on Manipuri versus 9.34%, and 33.28% on Sindhi versus 20.69%; BPE also shows a pattern of predicting only the non-entity class in the zero-shot setting. The paper acknowledges BPE's compactness and slightly higher precision on same-language tests, but argues that this compactness is what causes entity fragmentation and poor generalization, making SentencePiece the more effective default for low-resource Indic NER.

Load-bearing premise

The load-bearing assumption is that IndicBERT, pretrained with a BPE tokenizer and embedding matrix, can be meaningfully fine-tuned with a SentencePiece tokenizer, even though the paper never states how new SentencePiece tokens get their embeddings or whether the embeddings are re-initialized.

Editorial extensions

If this is right

  • Swapping BPE for SentencePiece when fine-tuning a multilingual transformer like IndicBERT should improve zero-shot NER on closely related unseen languages, with gains on the order of tens of F1 points.
  • BPE's failure mode of predicting only the non-entity class on some unseen languages means compact tokenizers can silently make a deployed NER system output nothing useful in new languages.
  • Character-level tokenization is not practical for NER in these languages because it expands sequence lengths and vocabulary drastically, as measured by high vocabulary compression ratios.
  • SentencePiece yields a precision-recall trade-off: slightly lower precision than BPE on same-language tests but higher entity recall, which is the property that transfers to unseen languages.
  • Even with SentencePiece, extremely low-resource and script-distant languages like Sindhi and Santali remain hard, so tokenizer choice alone is not enough for those cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the paper's intended mechanism—entity-boundary preservation—could be tested more directly by measuring how often each tokenizer splits inside a gold entity span; the paper's intrinsic metrics do not include such a span-aware measure.
  • We infer that the same comparison could be run on part-of-speech tagging or chunking to see whether SentencePiece's advantage is specific to named entities or generalizes to all sequence-labeling tasks, a test the paper does not perform.
  • We infer that the reported BPE collapse to all non-entity labels on Assamese and Oriya might reflect a decision-boundary threshold rather than tokenization alone; checking the predicted label distribution's entropy under each tokenizer would clarify the cause.
  • We infer that the paper's negative OOV rates and very high character-level OOV numbers come from defining OOV as a word split into multiple subwords, which inverts the usual definition; a reader should treat the intrinsic ranking as method-specific.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper compares Byte-Pair Encoding (BPE), SentencePiece, and character-level tokenization for named entity recognition (NER) in low-resource Indic languages. The intrinsic evaluation uses FLORES-200 subsets for Manipuri, Santali, Assamese, and Sindhi, reporting tokenization efficiency, OOV rates, vocabulary compression, and morphological preservation. The extrinsic evaluation fine-tunes IndicBERT on Hindi and Bengali NER data and reports zero-shot transfer to Assamese, Oriya, Marathi, Sindhi, Santali, and Manipuri. The paper's central claim is that SentencePiece consistently outperforms BPE in zero-shot NER because it better preserves entity boundaries and morphology, while character-level tokenization is impractical and is excluded after intrinsic analysis.

Significance. If the result were established, it would be practically useful: tokenizer choice is often treated as a default in multilingual NER, and evidence that SentencePiece preserves entity structure better than BPE in zero-shot settings would guide low-resource Indic pipelines. The paper also addresses an under-explored question and contributes small manually annotated test sets for three extremely low-resource languages. However, the significance is conditional on the validity of the comparison, and the current manuscript does not establish that validity. The paper makes concrete, falsifiable claims, but the experimental protocol and reported numbers contain load-bearing gaps and contradictions.

major comments (4)
  1. [Section III-D] The SentencePiece fine-tuning protocol is unspecified and likely invalid. IndicBERT was pretrained with a BPE-based tokenizer and a matching token-embedding matrix. Fine-tuning 'using the two tokenization strategies' with a SentencePiece tokenizer requires one of the following: re-initializing the embedding matrix for a new vocabulary (losing pretrained representations), mapping SentencePiece pieces to existing BPE embeddings (undefined for most pieces), or pretraining a new model from scratch (which would change the comparison). None of these options is described, and no code or checkpoints are released. The reported zero-shot gap, e.g., 0.00% F1 for BPE versus 88.38% for SentencePiece on Assamese in Table VII, may therefore reflect an implementation artifact rather than a property of tokenization. This issue is load-bearing for the central claim.
  2. [Table IV / Section IV-A] Table IV reports negative OOV rates for SentencePiece (-341.59 to -680.12) and Character-Level (-4042.75 to -7235.11), while the prose in Section IV-A states that SentencePiece has OOV rates between 4.34% and 7.81% and that Character-Level has the highest OOV at 40.70% to 50.32%. These two sets of numbers are mutually inconsistent, and negative percentages are meaningless for the OOV-rate definition given in Section III-B1. Since the intrinsic comparison is used to justify excluding character-level tokenization and to motivate SentencePiece, this contradiction undermines the intrinsic analysis and must be corrected.
  3. [Table VII] The zero-shot results are reported as single runs without variance, label distributions, or confusion analyses. The BPE row for Assamese reports F1 = 0.00% with accuracy = 91.22%, which could be consistent with an all-'O' prediction only if the test set entity proportion is about 8.8%, but no such statistics are given. The claim that BPE 'completely fails' on Assamese and Oriya may be an artifact of tokenizer/label misalignment or of a single unlucky run. The comparison needs multiple seeds, per-entity results, and an explicit statement of what the BPE model actually predicts.
  4. [Section III-B1 / Table V] The morphological preservation metric is never operationalized. Table V gives only qualitative statements such as 'BPE can't preserve morphemes in agglutinative languages' and 'SentencePiece can preserve morphemes in low-resource languages,' with no quantitative definition, annotation scheme, or inter-annotator agreement. Since morphological preservation is one of the two intrinsic pillars used to justify the choice of SentencePiece, this metric requires a concrete, reproducible definition.
minor comments (6)
  1. [Section IV-C1 / Figure 3] The text states that B-ORG recall 'improved from 80.99% (BPE) to 80.86% (SentencePiece)', which is a decrease, not an improvement; and it says BPE has higher B-LOC precision at 84.61% compared to 84.72% with SentencePiece, but 84.72% is in fact higher. These numeric inconsistencies should be fixed.
  2. [Throughout] The language name is spelled inconsistently as both 'Santali' and 'Santhali'; please standardize.
  3. [Section III-D] The text contains the typo 'Interannotaor agreement' for 'Interannotator agreement'.
  4. [References] References [27] and [30] appear to describe the same paper ('When every token counts: Optimal segmentation for low-resource language models'); this duplication should be resolved.
  5. [Section III-F] The implementation section says model checkpoints and tokenized datasets are 'stored for reproducibility', but no code, checkpoints, or data links are provided; please make these available or state explicitly what is released.
  6. [All experiments] No random seeds or number of runs are reported, so the consistency claims in the abstract and conclusion are not statistically supported.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim is an empirical comparison with independent zero-shot test languages; no equation, fit, or self-citation forces the result.

full rationale

The paper's central claim — SentencePiece outperforms BPE for zero-shot Indic NER — is supported by an extrinsic evaluation in which IndicBERT is fine-tuned on Hindi/Bengali and evaluated on unseen languages (Assamese, Oriya, Marathi, Sindhi, Santali, Manipuri). These test languages are not used to fit any parameter, and the tokenizers are standard algorithms (BPE, SentencePiece, character-level), not quantities defined in terms of the outcome. The intrinsic morphological-preservation assessment is qualitative, but it only motivates the exclusion of character-level tokenization and does not enter the downstream F1 calculation. The paper contains self-citations, but none is load-bearing: [3], [5], [10], [15], [17], [23] are contextual or prior-work references, not evidence for the empirical result. The under-specified SentencePiece fine-tuning protocol in Section III-D (how a BPE-pretrained embedding matrix is adapted) and the contradictory OOV numbers in Table IV are serious reproducibility/validity concerns, but they are experimental confounds and internal inconsistencies, not reductions of the conclusion to its inputs. No equation or fitted parameter makes the claimed prediction equivalent by construction, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the implicit ability to adapt a pretrained model to a foreign tokenizer, on label compatibility between manually annotated and existing datasets, and on the validity of intrinsic tables that contain impossible negative OOV rates. No free parameters are disclosed, but the tokenizer configurations and annotation details that would be needed to audit these assumptions are also missing.

assumptions (3)
  • domain assumption IndicBERT can be fine-tuned with a non-native SentencePiece tokenizer without losing its pretrained representation benefits.
    Section III-D states 'We fine-tune IndicBERT ... using the two tokenization strategies' but never describes vocabulary alignment, embedding re-initialization, or training from scratch. If the SentencePiece variant requires a new embedding matrix, the comparison is not fine-tuning and the pretraining advantage is confounded.
  • domain assumption The hand-annotated Sindhi, Santali, and Manipuri datasets use the same entity label set and annotation conventions as Naamapadam.
    Section III-B2 introduces the 200-sentence datasets and reports inter-annotator F1 above 72%, but provides no label schema or guidelines; label mismatch would invalidate zero-shot comparisons.
  • ad hoc to paper The OOV rate values in Table IV, though negative, can be interpreted as the prose describes.
    Table IV lists negative OOV rates for SentencePiece and Character-Level, which contradict the positive percentages in Section IV-A. The paper offers no explanation; any interpretation is ad hoc.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tokenization Matters: Improving Zero-Shot NER for Indic Languages." pith.science (2026). https://pith.science/paper/OFIOFJXD

@misc{pith2026250416977,
  author       = {Pith},
  title        = {Pith review of: Tokenization Matters: Improving Zero-Shot NER for Indic Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OFIOFJXD}},
  note         = {Machine review of arXiv:2504.16977}
}
read the original abstract

Tokenization is a critical component of Natural Language Processing (NLP), especially for low resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character Level tokenization strategies using IndicBERT for NER tasks in low resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties tokenization efficiency, out of vocabulary (OOV) rates, and morphological preservation as well as extrinsic downstream performance, including fine tuning and zero shot cross lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low resource Indic Languages, particularly in zero shot cross lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low resource and morphologically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low resource Indic NLP applications.

Figures

Figures reproduced from arXiv: 2504.16977 by the authors.

Figure 1
Figure 1. Tokenization Efficiency (Avg. Tokens per Sentence) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Vocabulary Compression Ratio TABLE V MORPHOLOGICAL PRESERVATION ANALYSIS Method Findings BPE (Simple) Can’t preserve morphemes in agglutinative languages SentencePiece Can preserve morphemes in low-resource languages Character-Level Over-fragments words, causing semantic loss tive languages, failing to maintain structure. SentencePiece, with its flexible subword units, better preserves morphology across diverse scri… view at source ↗
Figure 3
Figure 3. Precision and Recall across Entity Categories for IndicBERT fine [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of Zero-Shot Cross-Lingual Performance between Sen [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MVTamperBench: Evaluating Robustness of Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    The paper builds a 17,435-clip video tampering benchmark and finds that many vision-language models score near trivial-guessing levels, while model size shows no clear link to robustness.

  2. Hybrid AI for Responsive Multi-Turn Online Conversations with Novel Dynamic Routing and Feedback Adaptation

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A hybrid chatbot that routes easy queries to canned responses and complex queries to RAG reports 95% accuracy and 180ms latency on an internal support dataset.

  3. Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems

    cs.IR 2025-05 conditional novelty 4.0 of 10

    A reranker fine-tuned on hard negatives selected by two cosine-distance criteria outperforms older negative sampling methods on enterprise and domain-specific retrieval benchmarks.

  4. SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use

    cs.CL 2025-05 conditional novelty 4.0 of 10

    A new cross-lingual benchmark shows large language models comply with explicit requests to use swear words far more often in Indic languages than in English, revealing a safety alignment gap.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages · cited by 4 Pith papers

  1. [27]

    When every token counts: Optimal segmentation for low-resource language models,

    V . Dewangan, G. Suri, and R. Sonavane, “When every token counts: Optimal segmentation for low-resource language models,” in LoResLM Workshop, 2025. [Online]. Available: https://aclanthology.org/2025. loreslm-1.24/

    work page 2025

  2. [30]

    When every token counts: Optimal segmentation for low-resource language models,

    G. Suri, V . Dewangan, and R. Sonavane, “When every token counts: Optimal segmentation for low-resource language models,” arXiv preprint arXiv:2412.06926 , 2024. [Online]. Available: https: //arxiv.org/pdf/2412.06926

    arXiv 2024

  3. [1]

    Census of india 2011: Data on language and mother tongue,

    I. Office of the Registrar General & Census Commissioner, “Census of india 2011: Data on language and mother tongue,” https://censusindia. gov.in/2011census/C-16/DDW-C16-STMT-MDDS-0000.XLSX, accessed: 2025-03-09

    work page 2011

  4. [2]

    Eighth schedule to the constitution of india,

    C. of India, “Eighth schedule to the constitution of india,” https://en. wikipedia.org/wiki/Eighth Schedule to the Constitution of India, ac- cessed: 2025-03-09

    work page 2025

  5. [3]

    Review of reference generation methods in large language models,

    P. Pattnayak, A. Agarwal, B. Kumar, Y . Bangera, S. Panda, T. Kumar, and H. L. Patel, “Review of reference generation methods in large language models,” Journal ID, vol. 9339, p. 1263

    work page

  6. [4]

    Retrofitting language models with dynamic tokenisation,

    D. Feher, “Retrofitting language models with dynamic tokenisation,” Cambridge MLMI , 2024. [Online]. Available: https://www.mlmi.eng. cam.ac.uk/files/2023-2024/feher retrofitting 2024 0.pdf

    work page 2024

  7. [5]

    Improving clinical question answering with multi-task learning: A joint approach for answer extraction and medical categorization,

    P. Pattnayak, H. L. Patel, A. Agarwal, B. Kumar, S. Panda, and T. Kumar, “Improving clinical question answering with multi-task learning: A joint approach for answer extraction and medical categorization,” 2025. [Online]. Available: https://arxiv.org/abs/2502.13108

    arXiv 2025

  8. [6]

    Neural machine translation of rare words with subword units,

    R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL) , p. 1715–1725, 2016

    work page 2016

Show all 33 references
  1. [7]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , p. 4171–4186, 2019

    2019

  2. [8]

    Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages,

    S. Doddapaneni, R. Aralikatte, G. Ramesh, S. Goyal, M. M. Khapra, A. Kunchukuttan, and P. Kumar, “Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages,” 2023. [Online]. Available: https://arxiv.org/abs/2212.05409

    2023 arXiv

  3. [9]

    Un- supervised cross-lingual representation learning at scale,

    A. Conneau, K. Khandelwal, N. Goyal, V . Chaudhary, G. Wenzek, F. Guzm ´an, E. Grave, M. Ott, L. Zettlemoyer, and V . Stoyanov, “Un- supervised cross-lingual representation learning at scale,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguist...

    2020

  4. [10]

    Enhancing document ai data generation through graph-based synthetic layouts,

    A. Agarwal, H. Patel, P. Pattnayak, S. Panda, B. Kumar, and T. Kumar, “Enhancing document ai data generation through graph-based synthetic layouts,” arXiv preprint arXiv:2412.03590 , 2024

    2024 arXiv

  5. [11]

    Fs-dag: Few shot domain adapting graph networks for visually rich document understanding,

    A. Agarwal, S. Panda, and K. Pachauri, “Fs-dag: Few shot domain adapting graph networks for visually rich document understanding,” in Proceedings of the 31st International Conference on Computational Linguistics: Industry Track . Abu Dhabi, UAE: Association for Computational L...

    2025

  6. [12]

    Continuous spiking graph neural networks,

    N. Yin, M. Wan, L. Shen, H. L. Patel, B. Li, B. Gu, and H. Xiong, “Continuous spiking graph neural networks,” arXiv preprint arXiv:2404.01897, 2024

    2024 arXiv

  7. [13]

    Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,

    T. Kudo and J. Richardson, “Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text processing,” Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , p. 66–71, 2018

    2018

  8. [14]

    A survey of cross-lingual word embedding models,

    S. Ruder, I. Vuli ´c, and A. Søgaard, “A survey of cross-lingual word embedding models,” Journal of Artificial Intelligence Research , vol. 66, p. 673–717, 2021

    2021

  9. [15]

    Survey of large multimodal model datasets, application categories and taxonomy,

    P. Pattnayak, H. L. Patel, B. Kumar, A. Agarwal, I. Banerjee, S. Panda, and T. Kumar, “Survey of large multimodal model datasets, application categories and taxonomy,” arXiv preprint arXiv:2412.17759 , 2024

    2024 arXiv

  10. [16]

    Tokenizers for african languages,

    G. Ndomba, M. Mswahili, and Y . Jeong, “Tokenizers for african languages,” IEEE Access , 2024. [Online]. Available: https://ieeexplore. ieee.org/abstract/document/10815724/

    2024

  11. [17]

    Mvtamperbench: Evaluating ro- bustness of vision-language models,

    A. Agarwal, S. Panda, A. Charles, B. Kumar, H. Patel, P. Pattnayak, T. H. Rafi, T. Kumar, and D.-K. Chae, “Mvtamperbench: Evaluating ro- bustness of vision-language models,” arXiv preprint arXiv:2412.19794 , 2024

    2024 arXiv

  12. [18]

    Hybrid machine learning and deep learning approaches for insult detection in roman urdu text,

    N. Hussain, A. Qasim, G. Mehak, and O. Kolesnikova, “Hybrid machine learning and deep learning approaches for insult detection in roman urdu text,” AI, 2025. [Online]. Available: https://www.mdpi. com/2673-2688/6/2/33

    2025

  13. [19]

    Tokenization standards for linguistic integrity: Turkish as a benchmark,

    M. Bayram, A. Fincan, A. G ¨um¨us ¸, and S. Karakas ¸, “Tokenization standards for linguistic integrity: Turkish as a benchmark,” arXiv preprint arXiv:2502.07057, 2025. [Online]. Available: https://arxiv.org/ pdf/2502.07057

    2025 arXiv

  14. [20]

    Pseudo-labelling based boot- strapping for semi supervised learning,

    O. Olaleye, H. L. Patel, and T. Sheng, “Pseudo-labelling based boot- strapping for semi supervised learning,” Feb. 2025, uS Patent App. 18/237,234

    2025

  15. [21]

    Leveraging cumeta for enhanced document classification in cursive languages with transformer stacking,

    M. Shahid, M. Iqbal, and M. Umair, “Leveraging cumeta for enhanced document classification in cursive languages with transformer stacking,” Multimedia Tools and Applications, 2025. [Online]. Available: https://link.springer.com/article/10.1007/s11042-025-20681-w

    2025 doi

  16. [22]

    Augmented input representations in sequence generation models for decipherment and translation,

    N. Kambhatla, “Augmented input representations in sequence generation models for decipherment and translation,” SFU Summit , 2024. [Online]. Available: https://summit.sfu.ca/ flysystem/fedora/2025-02/ etd23279.pdf

    2024

  17. [23]

    Llm for barcodes: Generating diverse synthetic data for identity documents,

    H. L. Patel, A. Agarwal, B. Kumar, K. Gupta, and P. Pattnayak, “Llm for barcodes: Generating diverse synthetic data for identity documents,” arXiv preprint arXiv:2411.14962 , 2024

    2024 arXiv

  18. [24]

    Indicnlp corpus: Monolingual corpora and word embeddings for indic languages,

    D. Kakwani, A. Varma, A. Kunchukuttan, M. M. Khapra, P. Kumar, and K. Shashi, “Indicnlp corpus: Monolingual corpora and word embeddings for indic languages,” Proceedings of the 12th Language Resources and Evaluation Conference (LREC) , p. 1173–1182, 2020

    2020

  19. [25]

    A survey on recent approaches for natural language pro- cessing in low-resource scenarios,

    M. A. Hedderich, D. Klakow, G. Glava ˇs, O. Rohanian, J. Risch, and A. Bharadwaj, “A survey on recent approaches for natural language pro- cessing in low-resource scenarios,” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational ...

    2021

  20. [26]

    Named entity recognition for indian languages,

    A. Kumar, P. Mehta, and P. Bhattacharyya, “Named entity recognition for indian languages,” Proceedings of the 2022 Conference of the European Chapter of the Association for Computational Linguistics (EACL), p. 376–387, 2022

    2022

  21. [28]

    Ner-roberta: Fine- tuning roberta for named entity recognition (ner) within low-resource languages,

    A. Abdullah, S. Abdulla, and D. Toufiq, “Ner-roberta: Fine- tuning roberta for named entity recognition (ner) within low-resource languages,” arXiv preprint arXiv:2412.15252, 2024. [Online]. Available: https://arxiv.org/abs/2412.15252

    2024 arXiv

  22. [29]

    Adaptive subword tokenization for low- resource nlp: Balancing efficiency and generalization,

    Y . Wang, X. Jin, Y . Sun et al., “Adaptive subword tokenization for low- resource nlp: Balancing efficiency and generalization,” in ACL 2022 , 2022

    2022

  23. [31]

    No language left behind: Scaling human-centered machine translation,

    M. R. Costa-Juss‘a, J. Cross, O. C ¸ elebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard et al. , “No language left behind: Scaling human-centered machine translation,” arXiv preprint arXiv:2207.04672 , 2022

    2022 arXiv

  24. [32]

    Naamapadam: A large-scale named entity annotated data for indic languages,

    A. Mhaske, H. Kedia, S. Doddapaneni, M. M. Khapra, P. Kumar, R. M. V , and A. Kunchukuttan, “Naamapadam: A large-scale named entity annotated data for indic languages,” 2023. [Online]. Available: https://arxiv.org/abs/2212.10168

    2023 arXiv

  25. [33]

    Multiclass text classi- fications of sindhi newspaper articles,

    S. Kumar and R. Vavekanand, “Multiclass text classi- fications of sindhi newspaper articles,” Preprints, 2025. [Online]. Available: https://www.preprints.org/frontend/manuscript/ d40099f1eed56b67c6f65d138e209557/download pub

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.