Mark My Words: A Robust Multilingual Model for Punctuation in Text and Speech Transcripts

Ashwin Sankar; Raj Dabre; Sidharth Pulipaka; Sparsh Jain

REVIEW 4 major objections 5 minor 20 references

Mark My Words: A Robust Multilingual Model for Punctuation in Text and Speech Transcripts

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single model, Cadence, outperforms prior punctuation-restoration systems across 22 Indian languages and English.

desk verdict A genuinely useful model and corpus for Indic punctuation, but the SOTA claim is not yet supported because the test set was filtered by an LLM judge and compared only against two old baselines. read the letter →

arxiv 2506.03793 v1 pith:PZU4JQC6 submitted 2025-06-04 cs.CL

Sidharth Pulipaka , Sparsh Jain , Ashwin Sankar , Raj Dabre This is my paper · ORCID

classification cs.CL

keywords punctuationrestorationmultilingualNLPIndiclanguagesspeechtranscriptssequencetaggingmaskednexttokenpredictionlow-resourceLLMadaptation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that one relatively small pretrained language model, adapted into a bidirectional token classifier, can restore punctuation in all 22 scheduled Indian languages plus English, and that it beats the previous state of the art on both clean written text and spontaneous speech transcripts. If true, this collapses a patchwork of language-specific punctuation systems into a single checkpoint, which matters because unpunctuated ASR output and web-crawled text are a bottleneck for downstream translation, summarization, and speech synthesis. The authors build Cadence by taking a 1-billion-parameter causal language model, switching its attention to bidirectional, pre-training it with a masked next-token-prediction objective on Indic web data, and fine-tuning it to label each token with one of 30 punctuation classes.

What carries the argument

Masked Next Token Prediction (MNTP): after converting the causal attention of Gemma-3-1B to bidirectional, the model is pre-trained to predict a masked token from the representation of the immediately preceding unmasked token. This objective trains the model to use local left-context to decide what comes next, which is exactly the skill punctuation restoration needs, since a punctuation mark is predicted from the token it follows. The second mechanism is a four-phase curriculum over languages, moving from English to high-resource, then low-resource Indic languages, then all 23 languages together, which lets the model transfer knowledge to scarce-data languages. Fine-tuning then replaces the language-modeling head with a linear classifier over 30 punctuation labels, trained with weighted sampling to upweight low-resource languages.

What would settle it

Take the full held-out set before the judge filters out low-scoring sentences, have native speakers of a sample of the 22 languages score a random subset, then compute Cadence's macro-F1 and the baselines' macro-F1 on that unfiltered subset; if the performance gap over the baselines shrinks or disappears, the SOTA claim depends on the automated judge's filtering.

Watch

Extended reading notes

Core claim

Cadence establishes a new state of the art for punctuation restoration in Indic languages across both formal text and extempore speech transcripts. On the 14 languages where the earlier IndicPunct system applies, Cadence scores higher on every shared test set, and it extends coverage to the remaining eight scheduled languages and English. The central result is that a generalist, non-autoregressive model trained with a curriculum over languages, then fine-tuned on a mixture of verified web text and LLM-punctuated speech transcripts, transfers well to low-resource languages and even to a completely unseen language. The model's overall macro-F1 on focus punctuation labels is 0.79 on written text and 0.62 on spontaneous transcripts, with the gap reflecting the inherent difficulty of disfluent speech.

Load-bearing premise

The benchmark is built by keeping only held-out test sentences that an AI judge scores at 4.5 or higher on punctuation quality, and if that judge's preferences line up with how large language models punctuate, the reported F1 scores—and the state-of-the-art claim—could be higher than they would be against a strictly human-graded test set.

Editorial extensions

If this is right

Downstream Indic NLP pipelines—machine translation, text-to-speech, and ASR post-processing—can adopt a single Cadence checkpoint instead of maintaining language-specific punctuation models.
The model covers Indic-specific marks such as the Devanagari danda, the Urdu full stop, Arabic commas, and the Santali mucaad, so those writing systems get punctuated in their own conventions, not translated English punctuation.
Reported gains over baselines hold on spontaneous speech transcripts, meaning ASR output with false starts and repetitions is already better served than by prior tools.
Low-resource languages like Bodo, Dogri, and Santali receive usable punctuation restoration despite tiny training corpora, thanks to cross-lingual transfer during curriculum pre-training.
The zero-shot Bhojpuri result suggests the same model can be dropped into additional unseen Indic languages without retraining, though the absolute score is modest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same bidirectional MNTP recipe could be applied to other token-level tasks for low-resource languages—named-entity recognition, part-of-speech tagging, or inverse text normalization—reusing Cadence's pre-training rather than starting from scratch.
The 0.17 F1 gap between written text and spontaneous transcripts suggests the bottleneck is disfluency structure, not punctuation knowledge; a model explicitly trained on disfluency-tagged utterances might close much of that gap.
Because the test set was filtered by an automated punctuation judge, an independent human-annotated gold set on a random sample would test whether the SOTA claim survives a judge-agnostic evaluation.
The strong Santali result despite low data hints that script-specific punctuation conventions can be learned from very few examples when the model already shares structure with related languages; extending the label set to other tribal languages' marks is a natural next step.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: Cadence is a real contribution—a 30-label, 23-language punctuation restoration model built by adapting Gemma-3-1B with bidirectional attention and MNTP, plus a new aggregated training corpus. The modeling and data work are solid and useful. The evaluation, however, is the weak link. The headline numbers in Table 3 come from a test set filtered by Gemini-2.5-Flash to keep only instances scoring 4.5/5 on punctuation quality, and there are no unfiltered results, no human validation, no error bars, and no contamination check against FineWeb-2. That doesn't kill the paper, but it means 'state of the art' is not established. The margins over IndicPunct and DMP might partly reflect the judge's taste aligning with the LLM-generated training data.

What's genuinely new: the corpus aggregation across Sangraha, IndicVoices, Cosmopedia, and IndicCorp-v2; the 22-language coverage; and the 30-label space including danda, Urdu marks, and Santali mucaad. The curriculum pretraining with MNTP and the weighted sampling are sensible choices. The per-language analysis and zero-shot results on Bhojpuri and Manipuri are interesting. The paper is honest about limitations: it admits spontaneous speech lags, rare marks are hard, and data volume alone doesn't explain performance.

The main soft spot is the test-set filtering. Section 4.3 is transparent, but it introduces selection bias. If Gemini's notion of 'good punctuation' aligns with the Llama-3-405B punctuation used to create IndicVoices and Cosmopedia, filtering to high scores selects for the training distribution. The paper doesn't report how many instances were removed per language or domain, nor does it compare on the unfiltered set. A human-annotated subset, even a few hundred sentences per language, would settle this. Also, only two baselines, neither recent, and no significance tests. That said, the architecture is sound and the model likely does help in practice; the magnitude of the SOTA claim is just not proven.

Who this is for: NLP researchers working on Indic MT/TTS/ASR pipelines. They'll find Cadence useful regardless of the exact F1. It deserves a serious referee, but the evaluation needs to be strengthened before the SOTA claim is accepted.

Referee Report

4 major / 5 minor

Summary. The paper presents Cadence, a punctuation-restoration model for English and 22 Indian languages. It starts from Gemma-3-1B-Pretrain, converts the attention to bidirectional, continues pretraining on FineWeb-2 Indic data with a Masked Next Token Prediction objective under a four-phase curriculum, and fine-tunes on a mixture of Sangraha-Verified, IndicVoices-ST, translated Cosmopedia, and IndicCorp-v2 to classify 30 punctuation labels per token. Evaluation uses macro-F1 on held-out splits of the four fine-tuning corpora plus BPCC, after filtering test instances with Gemini-2.5-Flash to keep only those scoring at least 4.5 on a punctuation-quality rubric. The authors report substantial gains over IndicPunct and DeepMultilingualPunctuation on a reduced set of 'focus labels', plus zero-shot results on Bhojpuri and low-resource results on Manipuri, and they release the model.

Significance. The main contribution is a single open-source model covering 23 languages with 30 punctuation classes, including Indic-specific marks such as the danda and Urdu full stop, which is more useful than the previous 14-language IndicPunct. The architecture choice, adapting a pretrained decoder into a bidirectional MNTP tagger, is a reasonable and reproducible design, and the release of Cadence is a practical asset for Indic NLP pipelines. The analysis of performance versus training-data size and the identification of persistent weaknesses on spontaneous speech are informative. However, the central SOTA claim is only as strong as the evaluation, and the evaluation has three load-bearing weaknesses: an LLM-filtered test set without unfiltered results or human validation, only two pre-LLM baselines, and no statistical significance or error-bar analysis. With those issues addressed, the empirical claim would be credible; as it stands, the SOTA claim is not fully supported.

major comments (4)

[Section 4.3] The held-out test set is filtered by Gemini-2.5-Flash using a 1-5 punctuation-quality rubric, and only instances scoring 4.5 or higher are retained. The paper does not report how many instances were removed per language or domain, does not report scores on the unfiltered set, and provides no human-agreement analysis for the judge. Because the training data include Llama-3-405B-generated punctuation (Section 3.1, IndicVoices-ST and translated Cosmopedia), an LLM judge may share stylistic preferences with the model's training distribution, so filtering can select examples on which Cadence is unusually strong. Please report filter statistics, re-run the comparison on the unfiltered held-out set, and provide a human-validated subset or a detailed disagreement analysis.
[Section 5.1 and Table 3] The SOTA comparison is against IndicPunct (2022) and DeepMultilingualPunctuation (2021), both with small label spaces and no LLM-based component, and the paper presents no error bars, confidence intervals, or significance tests. The claim of new state-of-the-art is therefore based on a single run against two old baselines, and the reported margins (e.g., 0.78 vs 0.26 on Cosmopedia) may reflect evaluation-protocol differences rather than model quality. Add at least one recent multilingual or LLM-based baseline, and report bootstrap confidence intervals or multi-seed variance for Cadence and the baselines.
[Table 3, Overall row] The aggregate scores are not computed over the same language sets. Cadence supports all 23 languages, whereas IndicPunct and DeepMultilingualPunctuation support only subsets; cells marked 'x' or '–' are excluded differently for each model. The headline '0.76 vs 0.54' on IndicCorp-v2 therefore does not compare like with like. Report averages over the common subset of languages or use per-language paired differences with significance tests.
[Sections 3.1 and 4.1] Test instances are held out from the same four corpora used for fine-tuning, but the paper does not describe the split procedure, any deduplication, or whether documents or paragraphs overlapping between train and test were removed. In addition, continual pretraining uses FineWeb-2 (Section 4.1.1), and no contamination check against the test corpora is reported, so memorization cannot be ruled out. Please specify the splitting and deduplication protocol and, if feasible, report a contamination analysis.

minor comments (5)

[Table 3] The column headers ('Formal Extempore S IC C BPCC IV S IC C IV IC BPCC') are difficult to parse; provide a legend or separate sub-tables so it is clear which columns correspond to each model and corpus.
[Section 4.2.3] There is an unresolved cross-reference 'detailed in Table ??' that should read Table 2.
[Throughout] The model name is styled inconsistently as 'GEMMA 3-1B-PRETRAIN', 'GEMMA -3-1B', and 'Gemma-3-1B-Pretrain'; use the official capitalization consistently.
[Section 5.3] The text reports 'Sanskrit (75k, 0.2966)' and 'Sindhi (58k, 0.2966)', but Table 4 lists written all-labels scores of 0.21 and 0.29 for these languages; align the prose and table values.
[Section 7] The Limitations section discusses training-data quality but does not mention that the evaluation test set itself was LLM-filtered; this is a separate limitation that should be acknowledged.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical SOTA claim is measured against fixed held-out instances; the LLM-filtered test set is a validity risk, not a derivation-level circularity.

full rationale

The paper's central claim is an empirical comparison, not a formal derivation from assumptions. Cadence is built from external components (Gemma 3 1B pretrained, FineWeb-2, the MNTP objective from BehnamGhader et al. 2024) and evaluated on fixed held-out test instances from IndicCorp-v2, Sangraha-Verified, translated Cosmopedia, IndicVoices, and BPCC. No reported F1 score is fed back into the training objective, the architecture design, or the test-set construction. The main evaluation concern is that Section 4.3 filters the test set with Gemini-2.5-Flash while some training labels were generated with Llama-3-405B, which could bias the benchmark toward LLM punctuation style; this is an external-validity and fairness caveat, not a circular derivation, because the filter is not Cadence and the benchmark is not defined in terms of Cadence's outputs. The self-citations (e.g., IndicVoices-ST and Sangraha) are data-source references rather than load-bearing theorems or fitted parameters, and the Limitations section's acknowledgement of data-quality challenges reinforces that the authors treat the empirical gaps as open rather than as consequences of the method. Accordingly, no step in the claimed derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a manually designed training pipeline with several hand-chosen hyperparameters and a test set curated by an LLM judge. None of these choices are ablated or validated against human annotation, so the reported SOTA numbers are sensitive to these choices.

free parameters (6)

MNTP masking ratios = 0.30 (Phase 1), 0.25 (Phase 2), 0.15 (Phase 3), 0.25 (Phase 4)
Chosen by hand without ablations; directly affects the continual pretraining and thus model quality.
Test set quality threshold = 4.5 (on Gemini 1-5 rubric)
Chosen to filter test instances; this post-hoc selection affects the benchmark and can inflate scores.
Label space of 30 punctuation classes = 30 classes including rare combinations
Manually defined; includes combinations like ').' and '।' that are rare, affecting macro F1.
Curriculum phase assignment = English only; 13 mid-high resource; 9 low-resource; all 23
Selected by hand with no ablation; influences transfer and final performance.
Focus labels subset = 9 labels (., ,, :, ?, danda, Urdu full stop, Urdu question mark, Arabic comma, Santali mucaad)
Hand-defined subset used for baseline comparison; results on focus labels are higher than all labels.
Weighted sampling weights = Not specified
The paper states weighted sampling for low-resource languages but does not give the sampling weights or algorithm.

assumptions (6)

domain assumption Punctuation restoration is best framed as token-level sequence tagging
Paper uses this framing without comparing to generation-based or other formulations.
domain assumption LLM-generated punctuation (LLAMA-3-405B) is a valid source of ground-truth labels
Used to create training data from IndicVoices-ST and translated Cosmopedia; may encode LLM punctuation style rather than human norms.
domain assumption Gemini-2.5-Flash's punctuation quality scores are a reliable evaluation standard
Test set is filtered based on this judge; no human agreement or validation is reported.
domain assumption Macro F1 across punctuation labels is the appropriate evaluation metric
The paper does not justify excluding the O (no punctuation) class or choosing macro over micro F1.
ad hoc to paper Curriculum learning phases improve multilingual transfer
No ablation or baseline comparison is provided to show that the four-phase curriculum is beneficial.
domain assumption GEMMA-3-1B-PRETRAIN is a suitable base for all 22 Indic languages
Some languages (e.g., Manipuri) are not well supported by the tokenizer, so the model may struggle; this is acknowledged but the assumption is still baked in.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mark My Words: A Robust Multilingual Model for Punctuation in Text and Speech Transcripts." pith.science (2026). https://pith.science/paper/PZU4JQC6

@misc{pith2026250603793,
  author       = {Pith},
  title        = {Pith review of: Mark My Words: A Robust Multilingual Model for Punctuation in Text and Speech Transcripts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZU4JQC6}},
  note         = {Machine review of arXiv:2506.03793}
}

read the original abstract

Punctuation plays a vital role in structuring meaning, yet current models often struggle to restore it accurately in transcripts of spontaneous speech, especially in the presence of disfluencies such as false starts and backtracking. These limitations hinder the performance of downstream tasks like translation, text to speech, summarization, etc. where sentence boundaries are critical for preserving quality. In this work, we introduce Cadence, a generalist punctuation restoration model adapted from a pretrained large language model. Cadence is designed to handle both clean written text and highly spontaneous spoken transcripts. It surpasses the previous state of the art in performance while expanding support from 14 to all 22 Indian languages and English. We conduct a comprehensive analysis of model behavior across punctuation types and language families, identifying persistent challenges under domain shift and with rare punctuation marks. Our findings demonstrate the efficacy of utilizing pretrained language models for multilingual punctuation restoration and highlight Cadence practical value for low resource NLP pipelines at scale.

Figures

Figures reproduced from arXiv: 2506.03793 by the authors.

**Figure 1.** Overview of our training methodology. Stage-1: Modify causal attention to bidirectional attention. Stage-2: Pre-train with Masked Next Token Prediction Objective. Stage-3: Train for punctuation restoration, as a token-level classification task. Figure inspired from BehnamGhader et al., 2024. under-resourced Indic languages. Cadence supports English and all 22 scheduled languages of India: Assamese, Bengali, Bodo, Do… view at source ↗

**Figure 2.** Prompt For Punctuation Restoration C Prompt used for LLM as a Judge The datasets we have used for training contain webscraped text (Sangraha-verified, IndicCorpV2) and also synthetically punctuated text (IndicVoices). As a result punctuations may not always be correct. Ensuring a high quality test set becomes important to accurately assess our model and compare with existing models. We have employed Gemini-2.5- Fla… view at source ↗

**Figure 3.** The LLM-as-a-Judge prompt, outlining the comprehensive rubric used for evaluating punctuation and [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

Accuracy: Punctuation must conform to the grammatical rules of the specified input lan- guage (lang)

work page
[2]

Readability: Sentence clarity should be im- proved using appropriate punctuation (e.g., commas, periods, question marks)

work page
[3]

Consistency: The punctuation style should align with any provided reference text

work page
[4]

punctuated_text

Preservation of Structure: Word order and sentence construction must remain unaltered; only punctuation is to be adjusted. To accommodate linguistic diversity, particularly the varied sentence terminators across Indian lan- guages (e.g., period vs. danda), the prompt requires the input language (lang) and its corresponding sen- tence terminator (terminato...

work page
[5]

**Accuracy:** Ensure punctuation is applied correctlybased on the language's grammatical rules

work page
[6]

**Readability:** Improve sentence clarity by insertingappropriate punctuation marks (commas, periods,question marks, etc.)

work page
[7]

**Consistency:** Follow the punctuation style observedin the provided reference text

work page
[8]

punctuated_text

**Preservation of Structure:** Do not alter word order orintroduce new elements—only punctuation should beadjusted. Reference Information: - Language of the text: {lang} - Sentence terminator for {lang}: {terminator} Output Format: Provide only the punctuated text in JSON format with thestructure: json{ "punctuated_text": "Your punctuated text here" } Fig...

work page

Show all 20 references

[9]

Correct use of period for German

**Sentence Termination (Score: 1-5):** * Is appropriate sentence-ending punctuation used (e.g., '.', '?', '!', ' 。 ', '؟¿...?', '¡...!', etc.) according to the identified language's standard practice? * Is the type of terminator suitable for the sentence's function (declarativ...
[10]

Correct comma usage for French clauses

**Intra-Sentence Separation (Commas, Etc.) (Score: 1-5):** ' ,'،、 ') used correctly to separate clauses, list items,introductory elements, etc., according to the identified language's grammatical and stylistic rules? * Are there missing or extraneous separators based on that l...
[11]

", '...', « ... », „

**Quotation/Speech Marks (Score: 1-5):** * Are quotation marks or guillemets (e.g., "...", '...', « ... », „..." ) used correctly for direct speech, titles, or otherquoted elements according to the standard style of the identified language? * Are they properly paired and neste...
[12]

Incorrect use of apostrophe for English possessive

**Contraction/Possessive/Joining Markers (Apostrophes, Hyphens, Etc.) (Score: 1-5):** * Are apostrophes, hyphens, or other language-specific markers used correctly for contractions, possessives,compound words, case endings, or similar functions *if applicable* in the identifie...
[13]

Colon used correctly before list in English

**Other Punctuation (Colons, Semicolons, Dashes, Etc.) (Score: 1-5):** * Assess the use of any other punctuation present (e.g., colons ':', semicolons ';', dashes '–'/'—', ellipses '...', brackets '()'/ '[]') according to the identified language's standard usage. * Are they us...
[14]

Sentence start capitalized correctly

**Capitalization (Score: 1-5):** * Is capitalization used correctly according to the identified language's rules? (Consider: Sentence start, proper nouns, titles, language-specific rules like all nouns in German, etc.) * Comment: [Explain based on the specific capitalization r...
[15]

Identify Language: First, determine the primary language of the sentence below
[16]

Evaluate: Evaluate it strictly based on the criteria in the multilingual rubric, applying the rules and conventions standard to the *identified language*

Analyze Sentence: Carefully analyze the sentence provided.3. Evaluate: Evaluate it strictly based on the criteria in the multilingual rubric, applying the rules and conventions standard to the *identified language*. Focus *only* on punctuation and standard capitalization rules...
[17]

Then, provide a score (1-5) and a brief comment for *each* numbered evaluation category in the rubric, justifyingyour assessment based on the identified language's norms

Provide Scores & Comments: Fill in the **Identified Language** and **Confidence**. Then, provide a score (1-5) and a brief comment for *each* numbered evaluation category in the rubric, justifyingyour assessment based on the identified language's norms. Ensure scores are numer...
[18]

Overall Assessment: Calculate an **Overall Score (1-5)** reflecting the average or holistic quality,ensure this is also a numeric integer or float
[19]

Corrected Sentence: Provide a **Corrected Sentence**
[20]

Identified_Language

IMPORTANT: Respond *only* with a single valid JSON object. The JSON object must contain keys corresponding exactly to the rubric sections: "Identified_Language", "Confidence","Sentence_Termination", "Intra_Sentence_Separation", "Quotation_Speech_Marks", "Contraction_Possessive...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Accuracy: Punctuation must conform to the grammatical rules of the specified input lan- guage (lang)

work page

[2] [2]

Readability: Sentence clarity should be im- proved using appropriate punctuation (e.g., commas, periods, question marks)

work page

[3] [3]

Consistency: The punctuation style should align with any provided reference text

work page

[4] [4]

punctuated_text

Preservation of Structure: Word order and sentence construction must remain unaltered; only punctuation is to be adjusted. To accommodate linguistic diversity, particularly the varied sentence terminators across Indian lan- guages (e.g., period vs. danda), the prompt requires the input language (lang) and its corresponding sen- tence terminator (terminato...

work page

[5] [5]

**Accuracy:** Ensure punctuation is applied correctlybased on the language's grammatical rules

work page

[6] [6]

**Readability:** Improve sentence clarity by insertingappropriate punctuation marks (commas, periods,question marks, etc.)

work page

[7] [7]

**Consistency:** Follow the punctuation style observedin the provided reference text

work page

[8] [8]

punctuated_text

**Preservation of Structure:** Do not alter word order orintroduce new elements—only punctuation should beadjusted. Reference Information: - Language of the text: {lang} - Sentence terminator for {lang}: {terminator} Output Format: Provide only the punctuated text in JSON format with thestructure: json{ "punctuated_text": "Your punctuated text here" } Fig...

work page