{"id":"77abbd6c-aed5-4e4b-81c7-8d57a9660389","arxiv_id":"2411.17679","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"TIPA and MTIPA fine-tune LLMs on reverse character-position prediction using the tokenizer's own vocabulary, improving Chinese spelling correction and character-level benchmarks.","lead":"The paper introduces TIPA, a fine-tuning method that trains LLMs to spell out the characters inside their own tokenizer's vocabulary in reverse order, which improves character-position accuracy and Chinese spelling correction. A variant, MTIPA, extends this to whole sentences, gaining more position accuracy at the cost of extra training time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline PPA gains in Table 4 are not isolated from two confounds: TIPA/MTIPA receive more training batches than Pure-SFT, and the TIPA token set is pruned using the evaluation sets. A compute- and data-matched control is needed before the causal claim is supported.","rationale":"The reader's weakest assumption is the LEMON test-set leakage from TIPA pruning. I agree that is a real flaw, but the more load-bearing issue for the paper's headline claim is that the TIPA/MTIPA condition differs from Pure-SFT in two ways simultaneously: more training batches and test-set-based token pruning. Either alone could explain the Table 4 improvement. The paper's only matched internal comparison, TIPA-forward versus TIPA-reverse in Table 5, addresses ordering but not the presence of the auxiliary task, and it is reported for traditional CSC, not for the position-prediction setup of Table 4. Therefore a compute- and data-matched control is the key missing experiment. The concern is not an internal inconsistency; the method is well specified. It is a threat to causal attribution: the observed gains may be due to additional optimization steps or to teaching exactly the tokens that occur in the evaluation sets, rather than to reverse character prediction as such. This supports the reader's CONDITIONAL verdict: the idea is plausible, but the headline evidence is not yet clean, and the requested control experiments would settle whether the central claim holds.","tokens_in":22371,"tokens_out":12730,"duration_ms":117763,"concrete_test":"Re-run Experiment 1 with two changes: (i) prune TIPA tokens using only the training splits (Wang271K + CSCD-NS train), excluding CSCD-NS test and LEMON; and (ii) train a Pure-SFT-7B control for the same total number of batches as MTIPA-7B (222,780) with the same optimizer and LoRA settings. If MTIPA-7B no longer exceeds compute-matched Pure-SFT-7B on CSCD-NS test PPA by a substantial margin, the central causal claim is unsupported. An additional arm using an equally sized forward-mapping auxiliary task would further isolate the reverse-order mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: TIPA/MTIPA's reverse character prediction training improves character position prediction, with PPA rising from 79.45% to 87.52% on CSCD-NS test (Table 4). For this claim to hold, the only systematic difference between Pure-SFT-7B and TIPA/MTIPA-7B must be the reverse prediction task. Two features of the experimental design violate this. First, Section 5.3 constructs the TIPA dataset by deduplicating tokens 'appearing in Wang271K, CSCD-NS, and LEMON'. CSCD-NS includes the test split used in Table 4, and LEMON is the held-out test set for Table 6. The token subset taught by TIPA is therefore selected using the evaluation texts. The Appendix A.5 response ('tokens are known in the vocabulary... we can train the tokenizer's vocabulary without obtaining any datasets') does not address selection bias: the experiments do not train on the full vocabulary, only on tokens chosen with test-set knowledge. Second, Table 12 reports 188,340 batches for Pure-SFT-7B versus 203,960 for TIPA-7B and 222,780 for MTIPA-7B, so the treated models receive 8-18% more optimization steps. Table 5 provides a forward-vs-reverse control for traditional CSC, but no matched control exists for the headline PPA result in Table 4. The observed PPA gains could therefore reflect extra training data/steps or test-token selection rather than reverse character prediction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TIPA and MTIPA, two training schemes that teach LLMs to predict character positions within tokens (TIPA) and within sentences (MTIPA) using reverse-order character-to-position mappings derived from the tokenizer vocabulary. The authors evaluate the methods on a redefined Chinese Spelling Correction task that requires outputting the positions of misspelled characters (Experiment 1), on traditional CSC without explicit position prediction (Experiment 2), and on general multilingual/character-level benchmarks via full-parameter SFT of Llama-3.1-8B (Experiment 3). The headline results are large PPA gains on CSCD-NS test (Pure-SFT-7B 79.45% vs. TIPA-7B 84.72% vs. MTIPA-7B 87.52%) and smaller but positive gains on traditional CSC and LEMON, with no degradation on standard benchmarks.","tokens_in":22717,"tokens_out":3867,"duration_ms":36281,"significance":"If the causal claim is supported, TIPA would be a cheap, architecture-agnostic way to improve character-level abilities of subword-tokenized LLMs, with clear practical value for Chinese spelling correction and potentially for other character-sensitive tasks. The paper ships an open-source implementation and training code, and it includes a useful forward-vs-reverse ordering control (Table 5) as well as a broad set of downstream evaluations, including multilingual and instruction-following benchmarks. However, the current experimental design does not isolate the proposed mechanism from two alternative explanations: selection of TIPA training tokens using the evaluation sets, and additional optimization steps for the TIPA/MTIPA models. The PPA metric used for the headline claim is also a precision-like measure that is gameable. These issues do not necessarily invalidate the method, but they need to be fixed before the stated causal conclusions are justified.","major_comments":[{"comment":"The TIPA dataset is built by deduplicating tokens 'appearing in Wang271K, CSCD-NS, and LEMON', where CSCD-NS includes the test split used in Table 4 and LEMON is the held-out test set for Table 6. Selecting which tokens receive reverse-character supervision using the evaluation texts is a form of test-set-dependent training-data selection. The Appendix A.5 response ('tokens are known in the vocabulary... we can train the tokenizer's vocabulary without obtaining any datasets') does not address this: the experiments do not train on the full vocabulary, only on a 24,994-token subset chosen with knowledge of the test texts. This weakens the generalization claims for LEMON and, to a lesser extent, for CSCD-NS. Please re-run the key comparisons with an unpruned TIPA dataset (as described in A.2.1) or with pruning based only on the training split, and report whether the PPA/F1 gains persist.","section":"Section 5.3, Appendix A.5"},{"comment":"The headline PPA comparison in Table 4 is confounded by training budget: Pure-SFT-7B is trained for 188,340 batches, TIPA-7B for 203,960, and MTIPA-7B for 222,780, so the treated models receive roughly 8-18% more optimization steps. While Table 5 provides a forward-vs-reverse ordering control for traditional CSC, there is no matched control for Experiment 1 that isolates the reverse character-prediction task from simply having more training data/steps. Please add a compute- and data-matched control, e.g., Pure-SFT trained for the same number of batches as MTIPA, or an additional SFT dataset of matched size, to show that the PPA gains are due to the TIPA/MTIPA task rather than to extra gradient updates.","section":"Table 12 and Section 5.4"},{"comment":"The Position Prediction Accuracy metric is defined as PPA = |{hat c = x_hat i}| / |hat P|, where the denominator counts only positions that the model chose to predict. This is a precision-style metric: a model that predicts only easy positions, or predicts no positions at all, can achieve high PPA while missing actual errors. Since PPA is the headline metric in Table 4 (79.45% to 87.52%), the metric itself is load-bearing for the paper's main claim. Please report a recall-oriented metric (e.g., fraction of true error positions predicted), the average number of predictions per sentence, or NESSA/SA as the primary evidence. If PPA is kept, its behavior under empty or trivial predictions should be explicitly analyzed.","section":"Section 4.1, PPA definition"},{"comment":"The text states that 'the reverse TIPA construction (←) consistently outperforms the forward version (→)', but Table 5 shows that for the 1.5B model, TIPA(→)-1.5B has a higher character-level correction F1 (64.55%) than TIPA(←)-1.5B (64.36%), and the sentence-level metrics are nearly identical. The word 'consistently' is therefore inaccurate; the claim should be restricted to the 3B and 7B scales, or the 1.5B result should be discussed as an exception. This is not a fatal issue, but it affects the interpretation of the ordering rationale in Section 3.1.","section":"Section 5.5.1 and Table 5"}],"minor_comments":[{"comment":"The abstract says TIPA trains on 'the tokenizer's vocabulary', but in Experiments 1 and 2 the TIPA dataset is pruned to only 24,994 tokens appearing in the training/evaluation corpora. Please clarify in the abstract or methodology that the reported results use pruned TIPA unless stated otherwise.","section":"Abstract and Section 5.3"},{"comment":"The evaluation in Experiment 2 excludes samples where the model output length differs from the source length. This filtering can bias results toward models that produce equal-length outputs, which is itself one of the behaviors TIPA is supposed to improve. Please report the fraction of excluded samples per model, or use a metric that does not require filtering.","section":"Section 5.5"},{"comment":"Several figures are corrupted in the manuscript PDF: the axis labels and legend text are replaced by strings such as '/uni00000013' and similar escape sequences. As a result, the analysis in Section 6.1 and 6.2 cannot be read by a human reviewer. Please regenerate these figures with standard text rendering.","section":"Figures 3, 4, 5, 6"},{"comment":"Equation (3) defines the TIPA dataset over all UTF-8 tokens, but Algorithm 1 includes a pruning step that is not reflected in the equation or in Algorithm 2. Formalizing the pruned dataset would help avoid confusion about what is actually trained on.","section":"Section 3.1, Algorithm 1"},{"comment":"The caption states that GPT-4o's advantage stems from 'character-level tokenization', but the paper does not describe GPT-4o's tokenizer or provide evidence for this claim. Please either cite a source or soften the explanation.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has the right idea and the open-source artifacts are a plus, but the current evidence does not yet support the headline causal claim. The two confounds I identified (test-set-dependent pruning of the TIPA token set and unequal training budgets) are fixable with additional experiments, and the PPA metric issue is fixable by reporting recall-based metrics. I would like to see the revised experiments before making a final accept/reject decision. The Appendix A.5 paragraph should be rewritten; as it stands, it denies a leakage problem that is actually present in the experimental protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2411.17679. First, the core idea is genuinely new: train an LLM to reverse-decompose the tokenizer's own vocabulary tokens into character-position pairs, then mix that data into fine-tuning. Second, the headline gains in Table 4 are not yet cleanly attributable to that idea, because the treated models see more training data and the TIPA token set is pruned using the test sets.\n\nWhat's actually good: the method is cheap, architecture-agnostic, and doesn't change the tokenizer. The reverse ordering has a sensible rationale: it forces the model to output the token length first. The gains on Chinese spelling correction, especially the position-based task, are large. The authors include a forward-vs-reverse control in Experiment 2, which is the right kind of control and mostly supports reverse ordering at 3B and 7B, though not at 1.5B. They also ship code and mention an open-source model. That is real evidence.\n\nNow the soft spots, in order of severity.\n\nFirst is the matched-training confound. Table 12 shows Pure-SFT-7B gets 188,340 batches while TIPA-7B gets 203,960 and MTIPA-7B 222,780. The treated models see 8-18% more optimization steps. The forward-vs-reverse comparison is matched, but the headline PPA gain of 79.45 to 87.52 is not. That gain could partially reflect more training, not reverse prediction.\n\nSecond is the token pruning. Section 5.3 constructs the TIPA dataset from tokens in Wang271K, CSCD-NS, and LEMON, and LEMON is a held-out test set. The appendix's reply — that all tokens are in the vocabulary so no test text is seen — is technically true but misses the point: the selection of which tokens to train on used the test set's token distribution. Since the goal is vocabulary-wide awareness, this matters for the generalization claims on LEMON.\n\nThird, Table 7 has identical GSM8K and MMLU numbers for both models. That looks like a reporting error and undermines confidence in the other benchmark numbers.\n\nThe PPA metric's denominator counts only predicted positions, so a model could game it by predicting few positions. The other metrics, SA and NESSA, also improve, so the conclusion is not entirely hostage to that, but the metric should be reported alongside recall.\n\nWho is this for? People working on character-level understanding, tokenizer internals, and Chinese spelling correction. It deserves a serious referee: the idea is novel, the experiments are extensive, and the flaws are fixable with a compute-matched control and a train-only token pruning. I would engage with it and ask for a revision.\n\nRecommendation: send to peer review, but require the matched control and a corrected Table 7 before accepting.","headline":"Novel reverse-token-decomposition training idea with genuine CSC gains, but the headline PPA result is confounded by extra training data and test-set-informed token pruning.","tokens_in":23287,"tokens_out":3181,"would_cite":true,"duration_ms":26637,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training LLMs to reverse-map characters inside tokens improves character-position prediction, raising Chinese spelling correction position accuracy from 79.45% to 87.52%.","keywords":["TIPA","MTIPA","token internal structure","character position prediction","Chinese spelling correction","reverse character prediction","Byte-Pair Encoding","multilingual character awareness"],"falsifier":"Re-run the TIPA experiments with the pruning step restricted to tokens from Wang271K and CSCD-NS training data only, excluding every token that appears in LEMON; if the LEMON character-level F1 gains largely disappear while the CSCD-NS gains persist, the LEMON result is better explained by token overlap in the TIPA dataset than by improved position awareness.","tokens_in":22156,"feed_emoji":"🔤","tokens_out":13060,"duration_ms":102556,"temperature":0.7,"pith_summary":"Large language models trained with byte-pair encoding tend to lose the ability to say which character sits at which position inside a token. This paper proposes a training method, Token Internal Position Awareness (TIPA), that builds a dataset from the tokenizer's own vocabulary: for every UTF-8 token, the model practices outputting a JSON map of character positions in reverse order. A sentence-level extension, MTIPA, practices the same reverse mapping on sampled sentences. On a position-aware Chinese spelling correction task, TIPA raises position prediction accuracy from 79.45% to 84.72% and MTIPA to 87.52% on a 7B model. The paper also reports gains in traditional spelling correction, multilingual character counting, and instruction following, without degrading general benchmarks.","feed_headline":"Reverse character maps lift spelling fix positions to 87 percent","feed_subtitle":"Teaching a 7B model the tokenizer's own character order boosts position prediction from 79.45 to 87.52 percent.","key_machinery":"The load-bearing object is the reverse position mapping $D_t = \\{(i, c_i) \\mid i = n, \\dots, 1\\}$ over the tokenizer's vocabulary. TIPA converts each UTF-8-representable token into an exercise: input the token, output a JSON object whose keys are positions in descending order and whose values are the characters at those positions. That structure forces the model to state the token's length as its first output and to attend to character order without left-to-right sequence clues. MTIPA applies the same mapping to roughly 10% of the training sentences, extending the exercise to multi-token context. Both variants are trained through low-rank adaptation or full-parameter supervised fine-tuning, leaving the tokenizer and architecture unchanged, so the method adds no inference latency.","core_discovery":"The central claim is that an LLM's poor handling of characters inside tokens is a trainable gap rather than an architectural ceiling. Given a token $t = c_1 \\dots c_n$, TIPA defines a reverse position mapping $D_t = \\{(i, c_i) \\mid i = n, n-1, \\dots, 1\\}$ and trains the model to produce this mapping from the token alone. The reverse order is deliberate: the first number the model must emit is the token length, so length and position are learned together. The paper presents this as the mechanism behind the measured improvements, including the finding that reverse ordering outperforms forward ordering on traditional Chinese spelling correction at the 7B scale, and that MTIPA's sentence-level training yields the largest position-accuracy gains.","pith_inferences":["The paper's Section 8 limitation about out-of-vocabulary words is testable directly: remove a subset of tokens from the TIPA training vocabulary, then measure position prediction on those held-out words; if accuracy collapses, the method's generalization relies on exact token memorization rather than a character-position rule.","An implication the authors leave implicit is that reverse-position training may also soften the related reversal curse seen in LLMs, since it forces the model to reproduce content in the opposite order from familiar left-to-right text; this can be tested on synthetic fact-reversal tasks.","The reported near-doubling of distinct-character counting (9.29% to 18.74%) suggests TIPA sharpens attention to rare token constituents; a natural extension is to check whether gains scale with token length and are strongest for tokens that appear whole in pretraining.","The position-aware output format could transfer to other character-critical tasks such as code error localisation, OCR post-correction, or watermark steganography, where the model needs to name a character's position rather than rewrite the whole string."],"forward_implications":["Position-aware Chinese spelling correction becomes practical: a 7B model can emit error locations directly, reducing output tokens per correction while reaching 87.52% position prediction accuracy on CSCD-NS.","Character-structure training can be folded into general supervised fine-tuning without sacrificing standard benchmarks: Llama-3.1-8B keeps its GSM8K and MMLU scores while improving on instruction following and character-level tasks.","The reverse-order construction matters more at scale; at 7B, reverse TIPA outperforms forward TIPA on traditional CSC, suggesting the length-explicit first output is the active ingredient.","TIPA's benefit is not Chinese-only; average TyDi QA F1 across nine languages rises from 47.85 to 52.81, with the largest gains in Finnish, Indonesian, and Korean.","Because the method does not change the tokenizer or architecture, existing subword-based models can adopt it as a fine-tuning stage with no inference-time cost."],"supporting_citations":[{"why":"Supplies the CSCD-NS dataset and its native-speaker error annotations used in the main experiments.","marker":"Hu et al., 2024"},{"why":"Provides the LEMON benchmark whose seven domains test zero-shot cross-domain generalization in Experiment 2.","marker":"Wu et al., 2023"},{"why":"Supplies the Wang271K pseudo-corpus used with CSCD-NS for training.","marker":"Wang et al., 2018"},{"why":"Defines the C-LLM character-level tokenization approach and the data-split design that TIPA extends to position prediction.","marker":"Li et al., 2024"},{"why":"Identifies Qwen2.5-7B, the base model for the LoRA experiments.","marker":"Yang et al., 2024"},{"why":"Provides the LoRA training method used for TIPA and MTIPA fine-tuning.","marker":"Hu et al., 2021"},{"why":"Provides the Llama-3.1 tokenizer vocabulary extracted to build the TIPA dataset for full-parameter SFT.","marker":"Grattafiori et al., 2024"},{"why":"Supplies the tulu-3-sft-mixture dataset used alongside the TIPA dataset for general-model training.","marker":"Lambert et al., 2024"}],"fun_headline_variants":["Teaching LLMs token internals boosts spelling fix accuracy to 87%","Inner-token position training lifts LLM spelling fixes to 87%","TIPA: Reverse character maps sharpen spelling correction to 87%","LLMs learn token anatomy for precise spelling fixes (87%)","From 79% to 87%: TIPA teaches token-internal positions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The LEMON generalization results depend on the assumption that building the TIPA training data from tokens appearing in the LEMON test set does not leak test-set information; if LEMON-specific tokens teach the model patterns it is later tested on, the reported LEMON gains could overstate genuine character-position learning, even though the CSCD-NS comparisons rest on separate data.","fun_headline_variants_meta":{"raw":{"variants":["Teaching LLMs token internals boosts spelling fix accuracy to 87%","Inner-token position training lifts LLM spelling fixes to 87%","TIPA: Reverse character maps sharpen spelling correction to 87%","LLMs learn token anatomy for precise spelling fixes (87%)","From 79% to 87%: TIPA teaches token-internal positions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000746,"raw_usage":{"total_tokens":3278,"prompt_tokens":848,"completion_tokens":2430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2335}},"tokens_in":464,"tokens_out":2430,"duration_ms":14477,"temperature":1.0,"reasoning_tokens":2335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:49:57.464995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the TIPA experiments with the pruning step restricted to tokens from Wang271K and CSCD-NS training data only, excluding every token that appears in LEMON; if the LEMON character-level F1 gains largely disappear while the CSCD-NS gains persist, the LEMON result is better explained by token overlap in the TIPA dataset than by improved position awareness.","supporting_citations":[{"cited_title":"Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D","cited_arxiv_id":null,"evidence_quote":"Supplies the tulu-3-sft-mixture dataset used alongside the TIPA dataset for general-model training."}],"review_version":1}