Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training LLMs to reverse-map characters inside tokens improves character-position prediction, raising Chinese spelling correction position accuracy from 79.45% to 87.52%.

desk verdict Novel reverse-token-decomposition training idea with genuine CSC gains, but the headline PPA result is confounded by extra training data and test-set-informed token pruning. read the letter →

arxiv 2411.17679 v5 pith:XXTNMKMB submitted 2024-11-26 cs.CL

Zhu Xu , Zhiqiang Zhao , Zihan Zhang , Yuchi Liu , Quanwei Shen , Fei Liu , Yu Kuang , Jian He
show 1 more author
Conglin Liu
This is my paper · ORCID
classification cs.CL
keywords TIPAMTIPAtokeninternalstructurecharacterpositionpredictionChinesespellingcorrectionreverseByte-PairEncodingmultilingualawareness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models trained with byte-pair encoding tend to lose the ability to say which character sits at which position inside a token. This paper proposes a training method, Token Internal Position Awareness (TIPA), that builds a dataset from the tokenizer's own vocabulary: for every UTF-8 token, the model practices outputting a JSON map of character positions in reverse order. A sentence-level extension, MTIPA, practices the same reverse mapping on sampled sentences. On a position-aware Chinese spelling correction task, TIPA raises position prediction accuracy from 79.45% to 84.72% and MTIPA to 87.52% on a 7B model. The paper also reports gains in traditional spelling correction, multilingual character counting, and instruction following, without degrading general benchmarks.

What carries the argument

The load-bearing object is the reverse position mapping $D_t = \{(i, c_i) \mid i = n, \dots, 1\}$ over the tokenizer's vocabulary. TIPA converts each UTF-8-representable token into an exercise: input the token, output a JSON object whose keys are positions in descending order and whose values are the characters at those positions. That structure forces the model to state the token's length as its first output and to attend to character order without left-to-right sequence clues. MTIPA applies the same mapping to roughly 10% of the training sentences, extending the exercise to multi-token context. Both variants are trained through low-rank adaptation or full-parameter supervised fine-tuning, leaving the tokenizer and architecture unchanged, so the method adds no inference latency.

What would settle it

Re-run the TIPA experiments with the pruning step restricted to tokens from Wang271K and CSCD-NS training data only, excluding every token that appears in LEMON; if the LEMON character-level F1 gains largely disappear while the CSCD-NS gains persist, the LEMON result is better explained by token overlap in the TIPA dataset than by improved position awareness.

Watch

Extended reading notes

Core claim

The central claim is that an LLM's poor handling of characters inside tokens is a trainable gap rather than an architectural ceiling. Given a token $t = c_1 \dots c_n$, TIPA defines a reverse position mapping $D_t = \{(i, c_i) \mid i = n, n-1, \dots, 1\}$ and trains the model to produce this mapping from the token alone. The reverse order is deliberate: the first number the model must emit is the token length, so length and position are learned together. The paper presents this as the mechanism behind the measured improvements, including the finding that reverse ordering outperforms forward ordering on traditional Chinese spelling correction at the 7B scale, and that MTIPA's sentence-level training yields the largest position-accuracy gains.

Load-bearing premise

The LEMON generalization results depend on the assumption that building the TIPA training data from tokens appearing in the LEMON test set does not leak test-set information; if LEMON-specific tokens teach the model patterns it is later tested on, the reported LEMON gains could overstate genuine character-position learning, even though the CSCD-NS comparisons rest on separate data.

Editorial extensions

If this is right

  • Position-aware Chinese spelling correction becomes practical: a 7B model can emit error locations directly, reducing output tokens per correction while reaching 87.52% position prediction accuracy on CSCD-NS.
  • Character-structure training can be folded into general supervised fine-tuning without sacrificing standard benchmarks: Llama-3.1-8B keeps its GSM8K and MMLU scores while improving on instruction following and character-level tasks.
  • The reverse-order construction matters more at scale; at 7B, reverse TIPA outperforms forward TIPA on traditional CSC, suggesting the length-explicit first output is the active ingredient.
  • TIPA's benefit is not Chinese-only; average TyDi QA F1 across nine languages rises from 47.85 to 52.81, with the largest gains in Finnish, Indonesian, and Korean.
  • Because the method does not change the tokenizer or architecture, existing subword-based models can adopt it as a fine-tuning stage with no inference-time cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's Section 8 limitation about out-of-vocabulary words is testable directly: remove a subset of tokens from the TIPA training vocabulary, then measure position prediction on those held-out words; if accuracy collapses, the method's generalization relies on exact token memorization rather than a character-position rule.
  • An implication the authors leave implicit is that reverse-position training may also soften the related reversal curse seen in LLMs, since it forces the model to reproduce content in the opposite order from familiar left-to-right text; this can be tested on synthetic fact-reversal tasks.
  • The reported near-doubling of distinct-character counting (9.29% to 18.74%) suggests TIPA sharpens attention to rare token constituents; a natural extension is to check whether gains scale with token length and are strongest for tokens that appear whole in pretraining.
  • The position-aware output format could transfer to other character-critical tasks such as code error localisation, OCR post-correction, or watermark steganography, where the model needs to name a character's position rather than rewrite the whole string.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TIPA and MTIPA, two training schemes that teach LLMs to predict character positions within tokens (TIPA) and within sentences (MTIPA) using reverse-order character-to-position mappings derived from the tokenizer vocabulary. The authors evaluate the methods on a redefined Chinese Spelling Correction task that requires outputting the positions of misspelled characters (Experiment 1), on traditional CSC without explicit position prediction (Experiment 2), and on general multilingual/character-level benchmarks via full-parameter SFT of Llama-3.1-8B (Experiment 3). The headline results are large PPA gains on CSCD-NS test (Pure-SFT-7B 79.45% vs. TIPA-7B 84.72% vs. MTIPA-7B 87.52%) and smaller but positive gains on traditional CSC and LEMON, with no degradation on standard benchmarks.

Significance. If the causal claim is supported, TIPA would be a cheap, architecture-agnostic way to improve character-level abilities of subword-tokenized LLMs, with clear practical value for Chinese spelling correction and potentially for other character-sensitive tasks. The paper ships an open-source implementation and training code, and it includes a useful forward-vs-reverse ordering control (Table 5) as well as a broad set of downstream evaluations, including multilingual and instruction-following benchmarks. However, the current experimental design does not isolate the proposed mechanism from two alternative explanations: selection of TIPA training tokens using the evaluation sets, and additional optimization steps for the TIPA/MTIPA models. The PPA metric used for the headline claim is also a precision-like measure that is gameable. These issues do not necessarily invalidate the method, but they need to be fixed before the stated causal conclusions are justified.

major comments (4)
  1. [Section 5.3, Appendix A.5] The TIPA dataset is built by deduplicating tokens 'appearing in Wang271K, CSCD-NS, and LEMON', where CSCD-NS includes the test split used in Table 4 and LEMON is the held-out test set for Table 6. Selecting which tokens receive reverse-character supervision using the evaluation texts is a form of test-set-dependent training-data selection. The Appendix A.5 response ('tokens are known in the vocabulary... we can train the tokenizer's vocabulary without obtaining any datasets') does not address this: the experiments do not train on the full vocabulary, only on a 24,994-token subset chosen with knowledge of the test texts. This weakens the generalization claims for LEMON and, to a lesser extent, for CSCD-NS. Please re-run the key comparisons with an unpruned TIPA dataset (as described in A.2.1) or with pruning based only on the training split, and report whether the PPA/F1 gains persist.
  2. [Table 12 and Section 5.4] The headline PPA comparison in Table 4 is confounded by training budget: Pure-SFT-7B is trained for 188,340 batches, TIPA-7B for 203,960, and MTIPA-7B for 222,780, so the treated models receive roughly 8-18% more optimization steps. While Table 5 provides a forward-vs-reverse ordering control for traditional CSC, there is no matched control for Experiment 1 that isolates the reverse character-prediction task from simply having more training data/steps. Please add a compute- and data-matched control, e.g., Pure-SFT trained for the same number of batches as MTIPA, or an additional SFT dataset of matched size, to show that the PPA gains are due to the TIPA/MTIPA task rather than to extra gradient updates.
  3. [Section 4.1, PPA definition] The Position Prediction Accuracy metric is defined as PPA = |{hat c = x_hat i}| / |hat P|, where the denominator counts only positions that the model chose to predict. This is a precision-style metric: a model that predicts only easy positions, or predicts no positions at all, can achieve high PPA while missing actual errors. Since PPA is the headline metric in Table 4 (79.45% to 87.52%), the metric itself is load-bearing for the paper's main claim. Please report a recall-oriented metric (e.g., fraction of true error positions predicted), the average number of predictions per sentence, or NESSA/SA as the primary evidence. If PPA is kept, its behavior under empty or trivial predictions should be explicitly analyzed.
  4. [Section 5.5.1 and Table 5] The text states that 'the reverse TIPA construction (←) consistently outperforms the forward version (→)', but Table 5 shows that for the 1.5B model, TIPA(→)-1.5B has a higher character-level correction F1 (64.55%) than TIPA(←)-1.5B (64.36%), and the sentence-level metrics are nearly identical. The word 'consistently' is therefore inaccurate; the claim should be restricted to the 3B and 7B scales, or the 1.5B result should be discussed as an exception. This is not a fatal issue, but it affects the interpretation of the ordering rationale in Section 3.1.
minor comments (5)
  1. [Abstract and Section 5.3] The abstract says TIPA trains on 'the tokenizer's vocabulary', but in Experiments 1 and 2 the TIPA dataset is pruned to only 24,994 tokens appearing in the training/evaluation corpora. Please clarify in the abstract or methodology that the reported results use pruned TIPA unless stated otherwise.
  2. [Section 5.5] The evaluation in Experiment 2 excludes samples where the model output length differs from the source length. This filtering can bias results toward models that produce equal-length outputs, which is itself one of the behaviors TIPA is supposed to improve. Please report the fraction of excluded samples per model, or use a metric that does not require filtering.
  3. [Figures 3, 4, 5, 6] Several figures are corrupted in the manuscript PDF: the axis labels and legend text are replaced by strings such as '/uni00000013' and similar escape sequences. As a result, the analysis in Section 6.1 and 6.2 cannot be read by a human reviewer. Please regenerate these figures with standard text rendering.
  4. [Section 3.1, Algorithm 1] Equation (3) defines the TIPA dataset over all UTF-8 tokens, but Algorithm 1 includes a pruning step that is not reflected in the equation or in Algorithm 2. Formalizing the pruned dataset would help avoid confusion about what is actually trained on.
  5. [Table 4] The caption states that GPT-4o's advantage stems from 'character-level tokenization', but the paper does not describe GPT-4o's tokenizer or provide evidence for this claim. Please either cite a source or soften the explanation.

Circularity Check

1 steps flagged · score 6.0 of 10

TIPA's headline PPA gains are scored on the same token-position labels that TIPA was trained on, so the Table 4 result reduces to a memorization check rather than an independent prediction.

  1. fitted input called prediction [Section 5.3 (Training Details), Equation (3), Section 4.1 (PPA), Table 4; Appendix A.5]
    "For TIPA integration, we generated a TIPA dataset by performing set deduplication on tokens appearing in Wang271K, CSCD-NS, and LEMON ... However, this does not involve including the test set in the training set, because the tokens are known in the vocabulary, and we can train the tokenizer's vocabulary without obtaining any datasets."

    CSCD-NS and LEMON include the evaluation splits used in Tables 4 and 6, so the TIPA training set contains every token (and hence every character-position mapping) appearing in the test sentences. Equation (3) defines each training example as (t, D_t) with D_t = {(i, c_i)}; the PPA metric in Section 4.1 credits the model when its predicted incorrect character at predicted position i equals the source character x_i. For any test token t, x_i is exactly c_i from D_t, so the scored 'prediction' is identical to the training label the model memorized for that token. The reported Table 4 gain (79.45 to 87.52) therefore reduces to a memorization check on the tokenizer vocabulary, not a prediction about unseen character-position structure.

full rationale

The paper has no mathematical derivation chain, and there is no load-bearing self-citation or imported uniqueness theorem, so the classic circularity failure modes do not directly apply. The central circularity is empirical: the TIPA training targets are reverse character-position mappings of the tokenizer vocabulary, and the headline PPA metric scores exactly those mappings on the test set. Since the training set was pruned using tokens appearing in CSCD-NS and LEMON, and since all vocabulary tokens are anyway included in the unpruned construction, every test sentence's tokens have their complete position-to-character mapping present as a trained label. Thus the reported PPA improvement is substantially forced by training on the evaluated quantity itself. This is why the result is not a clean generalization claim, despite the paper's Appendix A.5 assertion that no dataset is needed: the assertion only confirms the overlap, since the full vocabulary includes all test tokens. A separate, non-circularity confound is that TIPA/MTIPA receive more optimization steps than Pure-SFT (Table 12: 203,960 and 222,780 batches vs 188,340), so compute differences are an alternative explanation. The score is 6 rather than 8 or 10 because the model still has to localize which positions are erroneous and compose token-internal offsets into full-sentence coordinates; the training data does not literally contain the final JSON correction outputs for the test sentences, so the derivation is not fully equivalent to its inputs by definition. However, the character-at-position component of every test token was directly trained, making the central position-prediction claim partially circular by construction.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The central claim depends on the transfer assumption and on the pruning choice that includes test-set tokens.

free parameters (8)
  • MTIPA sampling ratio r = 10%
    Chosen by hand to balance additional data and training efficiency (Section 3.2).
  • LoRA rank = 16
    Fixed hyperparameter for LoRA (Appendix A.2.2).
  • LoRA alpha = 16
    Fixed hyperparameter for LoRA (Appendix A.2.2).
  • LoRA dropout = 0.05
    Fixed hyperparameter for LoRA (Appendix A.2.2).
  • Learning rate (LoRA experiments) = 1e-4
    Fixed hyperparameter (Appendix A.2.2).
  • Learning rate (full-parameter SFT) = 5e-6
    Fixed hyperparameter (Appendix A.2.3).
  • Epochs (Experiment 2) = 6 trained, results reported at epoch 3
    Best epoch selected on validation; post hoc model selection (Table 13).
  • TIPA token pruning threshold = 24,994 tokens
    Prune to tokens appearing in CSC datasets, including LEMON tokens (Section 5.3).
assumptions (4)
  • standard math UTF-8 tokens can be unambiguously decomposed into their constituent characters.
    Algorithm 1 decomposes each token into characters using UTF-8 decoding, assuming the tokenizer's byte sequences align with character boundaries.
  • domain assumption Reverse character prediction on isolated tokens transfers to sentence-level position prediction in context.
    The paper assumes the learned token-level position signal generalizes to full-sentence tasks (Experiments 1 and 2).
  • domain assumption Including tokens from the evaluation datasets in the TIPA training set does not leak task-relevant information.
    Section 5.3 prunes TIPA tokens using LEMON, and Appendix A.5 argues this is harmless because the vocabulary is known a priori; this assumption is load-bearing for the LEMON generalization claim.
  • domain assumption The defined metrics (PPA, SA, SAIP, NESSA, CP, CR, CF1) measure the intended abilities.
    PPA divides by the number of predicted positions only, so a model that predicts fewer positions is rewarded; the paper does not report position recall.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning." pith.science (2026). https://pith.science/paper/XXTNMKMB

@misc{pith2026241117679,
  author       = {Pith},
  title        = {Pith review of: Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXTNMKMB}},
  note         = {Machine review of arXiv:2411.17679}
}
read the original abstract

Tokenization methods like Byte-Pair Encoding (BPE) enhance computational efficiency in large language models (LLMs) but often obscure internal character structures within tokens. This limitation hinders LLMs' ability to predict precise character positions, which is crucial in tasks like Chinese Spelling Correction (CSC) where identifying the positions of misspelled characters accelerates correction processes. We propose Token Internal Position Awareness (TIPA), a method that significantly improves models' ability to capture character positions within tokens by training them on reverse character prediction tasks using the tokenizer's vocabulary. Experiments demonstrate that TIPA enhances position prediction accuracy in LLMs, enabling more precise identification of target characters in original text. Furthermore, when applied to downstream tasks that do not require exact position prediction, TIPA still boosts performance in tasks needing character-level information, validating its versatility and effectiveness.

Figures

Figures reproduced from arXiv: 2411.17679 by the authors.

Figure 1
Figure 1. (a) demonstrates LLMs’ inability to perform [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of TIPA and MTIPA. TIPA enhances character-level structure awareness per token. MTIPA [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. (Experiment 1)Training dynamics showing character-level metrics and position accuracy. MTIPA￾7B achieves the highest performance ceiling. degradation These results suggest that TIPA’s benefits are not limited to Chinese-specific tasks or LoRA fine￾tuning, but represent a general improvement in LLMs’ character-level understanding. 7 Conclusion We introduced Token Internal Position Aware￾ness (TIPA) and Multi-Token In… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: (Experiment 1)Comparison of position pre [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 5
Figure 5. Figure 5: Token length distribution in the pruned token [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: (Experiment 2)Training and validation loss [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters

    cs.CL 2025-06 conditional novelty 6.0 of 10

    LLMs reconstruct token spelling in intermediate and later transformer layers rather than reading it directly from embeddings.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288

    arXiv 2023

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  5. [5]

    Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939

    arXiv 2024

  6. [6]

    Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 8:454--470

    2020

  7. [7]

    Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73--91

    2022

  8. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 44 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. Exams: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. arXiv preprint arXiv:2011.03080

    2020 arXiv

  3. [11]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  4. [12]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  5. [13]

    Yong Hu, Fandong Meng, and Jie Zhou. 2024. Cscd-ns: a chinese spelling check dataset for native speakers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146--159

    2024

  6. [14]

    John D Hunter. 2007. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(03):90--95

    2007

  7. [15]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  8. [16]

    Itay Itzhak and Omer Levy. 2022. Models in a spelling bee: Language models implicitly learn the character composition of tokens. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p...

    2022

  9. [17]

    Myeongjun Jang, Dohyung Kim, Deuk Sin Kwon, and Eric Davis. 2022. Kobest: Korean balanced evaluation of significant tasks. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3697--3708

    2022

  10. [18]

    Lai Jiang, Hongqiu Wu, Hai Zhao, and Min Zhang. 2024. Chinese spelling corrector is just a language learner. In Findings of the Association for Computational Linguistics ACL 2024, pages 6933--6943

    2024

  11. [19]

    Ayush Kaushal and Kyle Mahowald. 2022. What do tokens know about their characters and how do they know it? arXiv preprint arXiv:2206.02608

    2022 arXiv

  12. [20]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2024

  13. [21]

    Jiahao Li, Quan Wang, Zhendong Mao, Junbo Guo, Yanyan Yang, and Yongdong Zhang. 2022. Improving chinese spelling check by character pronunciation prediction: The effects of adaptivity and granularity. arXiv preprint arXiv:2210.10996

    2022 arXiv

  14. [22]

    Kunting Li, Yong Hu, Liang He, Fandong Meng, and Jie Zhou. 2024. C-llm: Learn to check chinese spelling errors character by character. arXiv preprint arXiv:2406.16536

    2024 arXiv

  15. [23]

    Zihong Liang, Xiaojun Quan, and Qifan Wang. 2023. Disentangled phonetic representation for chinese spelling correction. arXiv preprint arXiv:2305.14783

    2023 arXiv

  16. [24]

    Linfeng Liu, Hongqiu Wu, and Hai Zhao. 2024. Chinese spelling correction as rephrasing language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18662--18670

    2024

  17. [25]

    Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149--5152. IEEE

    2012

  18. [26]

    Rico Sennrich. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909

    2015 arXiv

  19. [27]

    Andrew Shin and Kunitake Kaneko. 2024. Large language models lack understanding of character composition of words. arXiv preprint arXiv:2405.11357

    2024 arXiv

  20. [28]

    Changxuan Sun, Linlin She, and Xuesong Lu. 2024. Two issues with chinese spelling correction and a refinement solution. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196--204

    2024

  21. [29]

    Avijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu, and Jay Pujara. 2023. Learn your tokens: Word-pooled tokenization for language modeling. arXiv preprint arXiv:2310.11628

    2023 arXiv

  22. [30]

    Yuen-Hsien Tseng, Lung-Hao Lee, Li-Ping Chang, and Hsin-Hsi Chen. 2015. Introduction to sighan 2015 bake-off for chinese spelling check. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 32--37

    2015

  23. [31]

    A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  24. [32]

    Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2020. Neural machine translation with byte-level subwords. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9154--9160

    2020

  25. [33]

    Dingmin Wang, Yan Song, Jing Li, Jialong Han, and Haisong Zhang. 2018. A hybrid approach to automatic corpus generation for chinese spelling check. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2517--2527

    2018

  26. [34]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical met...

    2020

  27. [35]

    Hongqiu Wu, Shaohua Zhang, Yuchen Zhang, and Hai Zhao. 2023. Rethinking masked language modeling for chinese spelling correction. arXiv preprint arXiv:2305.17721

    2023 arXiv

  28. [36]

    Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. 2013. Chinese spelling check evaluation at sighan bake-off 2013. In Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing, pages 35--42

    2013

  29. [37]

    Nan Xu and Xuezhe Ma. 2024. Llm the genius paradox: A linguistic and math expert's struggle with simple word-based counting problems. arXiv preprint arXiv:2410.14166

    2024 arXiv

  30. [38]

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291--306

    2022

  31. [39]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  32. [40]

    Fran c ois Yergeau. 2003. Utf-8, a transformation format of iso 10646. Technical report

    2003

  33. [41]

    Xunjian Yin and Xiaojun Wan. 2023. A comprehensive evaluation and analysis study for chinese spelling check. arXiv preprint arXiv:2307.13655

    2023 arXiv

  34. [42]

    Junjie Yu and Zhenghua Li. 2014. Chinese spelling error detection and correction based on language model, pronunciation, and shape. In Proceedings of The Third CIPS-SIGHAN Joint Conference on Chinese Language Processing, pages 220--223

    2014

  35. [43]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372

    2024 arXiv

  36. [44]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.