{"id":"94dc95bd-3b41-40dd-8539-641ca2d8b5ef","arxiv_id":"1908.03734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Supervised and unsupervised stemming reduced Telugu broadcast news ASR word error rate from 18.69% to 17.93% and 16.97%, respectively, with Kneser-Ney smoothing.","lead":"This paper reports that splitting Telugu words into roots and suffixes before training a broadcast-news speech recognizer, using either grammar rules or an unsupervised stemmer adapted from Hindi, lowers word error rate by about one percent absolute. It is a small empirical contribution to low-resource ASR, with several reporting gaps that limit how much weight the numbers can carry.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported WER gains rest on a test-set-leaked evaluation: Table 10's 18.69/17.93/16.97 exactly match Table 8's 'with test data' SGMM Kneser-Ney columns, so the 0.76%/0.94% claim compares LMs trained on the test transcript.","rationale":"The reader's conditional verdict is appropriate. I identify a more specific, evidence-based problem: the numbers used for the headline improvement are drawn from the 'with test data' evaluation condition, where test transcripts are part of the LM training text. This makes the absolute WERs misleading and the 0.76%/0.94% deltas non-representative of real ASR performance. The reader's recombination concern is also valid and remains unresolved; together they mean the paper's central claim cannot be accepted as stated. Because Table 8 also contains 'without test data' results that show improvements in the same direction, the proper outcome is not rejection but a required re-analysis: report the held-out results and specify the recombination/scoring procedure. This keeps the verdict at CONDITIONAL (UNCHANGED).","tokens_in":11150,"tokens_out":8864,"duration_ms":90033,"concrete_test":"Cross-reference Table 10 rows 1-3 against Table 8: verify that 18.69, 17.93, 16.97 are the SGMM/Kneser-Ney 'with test data' entries. Then rerun the same system in the 'without test data' condition and recompute the accuracy deltas; if the conclusion's 0.76%/0.94% values disappear or change, the claim is an artifact of test data leakage. Additionally, score the with-test-data outputs after deterministically recombining stem+suffix tokens into surface words to check whether the reported WERs are on the intended word units.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section V quotes accuracy improvements of 0.76% and 0.94%. The only numbers that produce those deltas are the SGMM/Kneser-Ney WERs in Table 10: 18.69 (no stemming), 17.93 (supervised split), 16.97 (unsupervised split). Comparing Table 10 with Table 8 shows these three values are exactly the 'with test data' columns for SGMM and Kneser-Ney. In that protocol the LM training text includes the transcripts of the 5h30m test set, so OOVs are artificially zero and test n-grams are directly scored. A WER comparison under this condition does not measure how stemming helps recognize unseen words; it measures behavior when the test text is known. The conclusions do not report the valid held-out condition ('without test data'), where Table 8 SGMM/Kneser-Ney WERs are 53.21, 49.98, 43.21 and the absolute improvements are very different. Compounding this, even in the with-test-data numbers the paper never specifies how the emitted stem+suffix tokens are recombined into surface words before WER is computed (Section I only says the recognized text 'is combined to form a meaningful new word'). If WER is scored on split tokens, the comparisons are not on the same word units.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses out-of-vocabulary (OOV) words in Telugu broadcast-news ASR by building language models from stemmed text. The authors collect 65 hours of Telugu broadcast audio, transcribe it, build HMM and SGMM acoustic models with Kaldi, and evaluate Witten-Bell, Kneser-Ney, and other smoothing techniques. They then apply a supervised stemming method using Telugu grammar rules and an unsupervised stemming algorithm adapted from a Hindi stemmer, and report that the unsupervised method improves ASR word accuracy by 0.94% over the supervised method and by 0.76% over the unstemmed baseline. The paper also reports perplexity and OOV statistics under several training conditions and claims that Kneser-Ney smoothing performs best on pre-processed data.","tokens_in":11454,"tokens_out":3393,"duration_ms":35414,"significance":"If the reported results were obtained from a valid held-out evaluation, the paper would be a useful practical contribution: it provides a new Telugu broadcast-news corpus, a phoneme set, and a systematic comparison of smoothing techniques for Telugu, and it demonstrates that morphological segmentation can reduce OOVs for a morphologically rich language. The use of standard tools (Kaldi, SRILM, cmuclmtk) and the availability of detailed corpus statistics are strengths. However, the central empirical claim is undermined by the evaluation protocol described below, so the significance currently lies more in the assembled resources and baseline comparisons than in the headline accuracy improvement.","major_comments":[{"comment":"The headline improvements of 0.76% and 0.94% in Section V are derived from the 'with test data' condition, not from a held-out evaluation. The WERs in Table 10 (18.69, 17.93, 16.97) exactly match the SGMM/Kneser-Ney 'with test data' columns in Table 8. In this protocol the LM training text includes the transcripts of the test set, so OOVs are artificially zero and test n-grams are directly scored; such a comparison cannot measure how stemming helps recognize unseen words. Under the valid 'without test data' condition in Table 8, the SGMM/Kneser-Ney WERs are 53.21, 49.98, and 43.21, giving improvements of 3.23 and 6.77 percentage points. The abstract and conclusions must be rewritten to report the held-out numbers, and the 'with test data' condition should be presented only as an oracle or removed.","section":"Section V and Table 10 vs. Table 8"},{"comment":"The manuscript never specifies how recognized root-word and suffix/prefix tokens are recombined into surface words before WER is computed. Section I says only that the recognized text 'is combined to form a meaningful new word.' If WER is scored on split tokens, the baseline and stemmed systems are compared on different word units, making the WER difference meaningless; if recombination is performed, its errors and the recombination algorithm must be described. Without this information, the measured WER improvement cannot be interpreted.","section":"Section I and WER computation"},{"comment":"The unsupervised stemmer is described as requiring a 'stem frequency threshold' and a 'suffix frequency threshold,' but these values are never reported. The segmentation produced by the algorithm depends directly on these thresholds, so the experimental result is not reproducible without them. If the thresholds were chosen by inspecting the test set, the comparison would be circular; even if not, the sensitivity of the downstream WER to these parameters is unknown. The authors should report the threshold values and, ideally, a small sensitivity analysis.","section":"Section IV.B, unsupervised stemmer thresholds"}],"minor_comments":[{"comment":"The test-set duration is inconsistent: Tables 2, 3, and 7 state '2-HOUR 30 MIN' of test data, while Section IV and Table 8 refer to '5 hours 30 minutes' of test speech. Please clarify the actual test set size.","section":"Section III vs. Table 8"},{"comment":"The baseline ASR accuracy is reported as 81.01% in Section V, but Table 10 gives a WER of 18.69% for the unstemmed condition, which implies an accuracy of 81.31%. These numbers should be reconciled.","section":"Section V"},{"comment":"Several cross-references appear as 'Error! Reference source not found.' (e.g., before Tables 2 and 7), and Figure 1 is cited but not shown. These placeholders should be fixed before publication.","section":"Throughout"},{"comment":"There are typos such as 'knser-ney' and 'Kneser -ney' in Table 8 and 'un supervised' in the Section IV heading; a careful proofread is needed.","section":"Table 8 and Section IV"}],"recommendation":"major_revision","confidential_remarks":"The core problem is that the paper's main accuracy claim is based on a test-set-leaked evaluation condition, and the valid held-out results in Table 8 are dramatically different. This is not a matter of degree of smoothing; it is a mismatch between the reported conclusion and the experimental setup. The good news is that the held-out numbers already exist in Table 8, so the authors can correct the presentation by reporting those and by providing the missing stemmer thresholds and recombination details. I would not reject outright because the underlying data and method may be salvageable, but the current version as written is misleading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: the paper's main claim—0.76% and 0.94% WER improvements from supervised and unsupervised stemming—comes from a test-set-leaked evaluation. The numbers in Table 10 (18.69, 17.93, 16.97) are exactly the SGMM/Kneser-Ney columns in Table 8 under the condition 'with test data,' where the LM training text includes the test transcript. That condition does not measure how stemming helps with unseen words; it measures fit to known test text. The 'without test data' columns tell a different story (WER 53.21 → 49.98 → 43.21), and those are the ones that should have been used.\n\nWhat the paper does well: it documents a real Telugu broadcast-news ASR effort—65 hours of speech, a 1.9M-word text corpus, and a systematic comparison of standard smoothing techniques (Witten-Bell and Kneser-Ney come out ahead). The supervised stemming is based on explicit Telugu morphology, and the unsupervised stemmer is a direct adaptation of Pandey and Siddiqui's Hindi method [36], which the authors acknowledge. For someone working on low-resource Indic ASR, the corpus details and morphological tables are useful.\n\nSoft spots, in proportion. The leakage issue is not minor; it invalidates the paper's central claim as stated. On top of that: the stem and suffix frequency thresholds for the unsupervised stemmer are never reported; there are no error bars or significance tests; the baseline accuracy appears as both 81.01% (Section V) and 81.31% (implied by Table 10); and the paper never explains how recognized stem+suffix tokens are recombined into surface words before WER is computed. If scoring was done on split tokens, the WER numbers are not comparable across conditions. These are all fixable in principle, but they need to be addressed before the numbers can be trusted.\n\nWho is this for? Maybe someone looking for baseline Telugu ASR numbers or an example of morphological LM preprocessing. But as it stands, the reported gains are not evidence for the conclusion. A serious editor would be justified in sending this back for major revision or rejecting it. I'd tell the authors to redo the evaluation on the held-out condition, report the thresholds, and describe the recombination step.","headline":"The reported 0.76%/0.94% WER gains come from a test-set-leaked evaluation, so the paper's central claim does not hold up as stated.","tokens_in":11979,"tokens_out":3567,"would_cite":false,"duration_ms":35298,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Splitting Telugu words into stems and suffixes before language modeling reduces ASR word errors.","keywords":["Telugu ASR","unsupervised stemming","language modeling","out-of-vocabulary words","smoothing techniques","Witten-Bell","Kneser-Ney","broadcast news transcription"],"falsifier":"Decode the same 5-hour-30-minute test set under all three vocabulary conditions and score word error twice: once on raw decoder tokens and once after stem-suffix recombination, with test transcripts excluded from language-model training. If the split vocabulary does not lower the raw token error rate, the reported gains come from the unmeasured recombination step rather than from stemming itself.","tokens_in":10970,"feed_emoji":"🗣️","tokens_out":9764,"duration_ms":100796,"temperature":0.7,"pith_summary":"The paper's central claim is that Telugu's heavy inflectional morphology is better handled by building the language model from split tokens—stems plus suffixes or prefixes—rather than from whole surface words. Using grammar-based supervised rules to split 26,213 unique words reduces the vocabulary from 2,37,994 to 2,01,657 tokens, and an unsupervised statistical stemmer reduces it further to 1,96,435. With smaller split vocabularies, the decoder encounters fewer out-of-vocabulary words, and the paper reports word-accuracy gains of about 0.76 percentage points from supervised stemming and a further 0.94 points from unsupervised stemming. The result matters because it offers a way to improve Telugu broadcast-news transcription without collecting new speech data, using only text preprocessing and language-model smoothing.","feed_headline":"Splitting Telugu words into stems and suffixes trims ASR errors","feed_subtitle":"Grammar-based and statistical splitting cut out-of-vocabulary words and lift recognition accuracy by nearly two points.","key_machinery":"The load-bearing object is a split-token vocabulary: before language-model training, each inflected Telugu word is represented as a root or stem plus its suffix or prefix, and the n-gram model assigns probabilities to those pieces as separate tokens. Supervised splitting follows explicit Telugu grammar rules, such as sixteen nominal case-marker inflections and twenty-one suffixes that attach to verbs ending in the \"uu\" sound. Unsupervised splitting, adapted from an existing Hindi stemmer, builds a bipartite graph of all possible prefix and suffix splits in the corpus, keeps only prefixes and suffixes that occur with enough distinct partners, and iteratively prunes low-frequency vertices until a stable stem-suffix inventory remains. The smoothing method completes the mechanism by giving nonzero probability to unseen stem-suffix combinations; Witten-Bell is best for unsplit text and Kneser-Ney is best after splitting.","core_discovery":"The paper finds that for Telugu broadcast-news ASR, training the n-gram language model on stem and suffix tokens instead of whole inflected words reduces out-of-vocabulary words and improves recognition accuracy. In experiments with a subspace Gaussian mixture model acoustic model, word error rate falls from 18.69% with no stemming to 17.93% after grammar-based supervised splitting and to 16.97% after unsupervised statistical splitting. Among five smoothing techniques, Witten-Bell works best on unsplit text, while Kneser-Ney works best on pre-processed, split text. The authors attribute the improvements to the lower out-of-vocabulary rate that split vocabularies produce, and they also report further gains when unsplit text is added alongside the unsupervised-stemmed corpus.","pith_inferences":["A natural extension, not tested in the paper, is to vary the stem-frequency and suffix-frequency thresholds of the unsupervised stemmer; the reported 0.94-point gain is likely threshold-dependent and might be larger on other corpora.","The paper's row showing 16.42% WER for \"without splitting + unsupervised stemming\" suggests that a hybrid vocabulary—unsplit words plus discovered stem-suffix units—may be the most practical configuration, though the paper does not analyze this combination in depth.","If the unmeasured recombination step can be made explicit and robust, the same split-vocabulary recipe should transfer to Kannada, Tamil, and Malayalam, which share Telugu's agglutinative suffix patterns.","A direct way to test the mechanism would be to hold the acoustic model fixed and compare perplexity on held-out Telugu text for split versus unsplit vocabularies, isolating whether the gains come from language-model coverage rather than decoder behavior."],"forward_implications":["Telugu broadcast-news ASR can be improved without new speech data by reusing an existing text corpus and a statistical stemmer.","Splitting the vocabulary shrinks the language model's unique-word count, reducing the computational burden caused by Telugu's many morphological variants.","The unsupervised stemmer transfers from Hindi to Telugu, suggesting that the approach may generalize to other suffix-heavy Indic languages.","Kneser-Ney smoothing should be preferred once the training text is stem-split, while Witten-Bell remains the better choice for unsplit text.","The per-stage accuracy gains compound: supervised stemming and then unsupervised stemming together reduce word error by roughly 1.7 percentage points in the paper's main comparison."],"supporting_citations":[{"why":"Supplies the prior Hindi ASR adaptation that motivates applying unsupervised splitting to Telugu.","marker":"[3]"},{"why":"Provides the unsupervised Hindi stemmer whose bipartite-graph pruning algorithm the paper adapts.","marker":"[36]"},{"why":"Supplies an earlier unsupervised Telugu stemmer cited as background for statistical segmentation of Telugu words.","marker":"[35]"},{"why":"Provides the Telugu morphological generator behind the supervised grammar-based split rules.","marker":"[30]"},{"why":"Introduces the Kneser-Ney smoothing method that performs best on the stem-split text corpora.","marker":"[22]"},{"why":"Introduces the Witten-Bell smoothing method that performs best when the language model is built from unsplit text.","marker":"[26]"},{"why":"Supplies the speech-recognition toolkit used to build the HMM and SGMM acoustic models that produce the reported word-error rates.","marker":"[15]"}],"fun_headline_variants":["Unsupervised stemming lifts Telugu ASR accuracy","Stem-based language model cuts Telugu OOV words, boosts ASR","Unsupervised splitting of Telugu stems trims out-of-vocabulary words","Telugu ASR: unsupervised stemming reduces word errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reported gain rests on the assumption that the decoder's separate stem and suffix tokens can be recombined into the intended surface word, a step the paper describes only in passing in Section I and never measures.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised stemming lifts Telugu ASR accuracy","Stem-based language model cuts Telugu OOV words, boosts ASR","Unsupervised splitting of Telugu stems trims out-of-vocabulary words","Telugu ASR: unsupervised stemming reduces word errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001771,"raw_usage":{"total_tokens":6957,"prompt_tokens":889,"completion_tokens":6068,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":5994}},"tokens_in":505,"tokens_out":6068,"duration_ms":41999,"temperature":1.0,"reasoning_tokens":5994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:03:25.479645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode the same 5-hour-30-minute test set under all three vocabulary conditions and score word error twice: once on raw decoder tokens and once after stem-suffix recombination, with test transcripts excluded from language-model training. If the split vocabulary does not lower the raw token error rate, the reported gains come from the unmeasured recombination step rather than from stemming itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prior Hindi ASR adaptation that motivates applying unsupervised splitting to Telugu."},{"cited_title":"https://www.iitm.ac.in/donlab/tts/downloads/cls/cls_v2.1.6.pdf","cited_arxiv_id":null,"evidence_quote":"Provides the unsupervised Hindi stemmer whose bipartite-graph pruning algorithm the paper adapts."},{"cited_title":"https://baraha.com/v10/index.php","cited_arxiv_id":null,"evidence_quote":"Supplies an earlier unsupervised Telugu stemmer cited as background for statistical segmentation of Telugu words."},{"cited_title":"Improved Hindi broadcast ASR by adapting the language model and pronunciation model using a priori syntactic and morphophonemic knowledge,","cited_arxiv_id":null,"evidence_quote":"Provides the Telugu morphological generator behind the supervised grammar-based split rules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Kneser-Ney smoothing method that performs best on the stem-split text corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Witten-Bell smoothing method that performs best when the language model is built from unsplit text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the speech-recognition toolkit used to build the HMM and SGMM acoustic models that produce the reported word-error rates."}],"review_version":1}