REVIEW 3 major objections 5 minor 32 references
IMS-Speech: A Speech to Text Tool
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read One generic speech recognizer, trained on many public corpora, matches or beats task-specific systems on English and German benchmarks.
desk verdict Useful ASR tool paper with competitive results on standard benchmarks, but the 'generic' claim outruns the evidence since all test sets are within training domains. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The system combines two trained components: a speech-activity detector that cuts an upload into short speech segments, and an end-to-end recognizer that maps each segment's audio features to a transcription. The recognizer is an attention-based encoder-decoder network trained with a hybrid objective that adds Connectionist Temporal Classification (CTC) loss to the usual attention loss, so the model learns the monotonic alignment between audio and text. Outputs are sub-word units, character-level pieces that can be combined to form any word, which keeps the vocabulary compact and avoids out-of-vocabulary failures. The generic behavior comes from the training mixture: about 2,277 hours of English and 3,797 hours of German covering read, spontaneous, mixed, and distant speech, with the German data augmented to ten times its original size.
What would settle it
Take audio recorded in conditions absent from the training corpora, such as telephone codec speech, overlapping multi-speaker conversations, or background music, and transcribe it with the public IMS-Speech service. If word error rates on those clips are substantially higher than the clean read and spontaneous speech benchmarks reported in Table 2, the claim of generic competitive accuracy across diverse conditions is falsified for exactly those conditions.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that pooling heterogeneous public speech data into one model does not cost accuracy: the generic systems achieve WER of 3.8% on WSJ eval'92, 4.4% on LibriSpeech test-clean, and 17.4% on AMI IHM, alongside German results of 11.1% on Tuda-De dev and 6.7% on Verbmobil 1 dev. Against the best published specialised systems cited in the paper, the generic model is slightly worse on some English benchmarks but clearly better on the German ones. The authors attribute the occasional advantage to better generalization from a larger amount of training data, and they take the overall pattern as evidence that a single generic service can provide transcriptions of competitive accuracy across tasks and conditions.
Load-bearing premise
The claim that the models are generic is tested only on corpora that also supplied training data, so accuracy on genuinely unseen recording conditions is not demonstrated; if users upload audio unlike those corpora, the reported word error rates may not hold.
Editorial extensions
If this is right
- A non-specialist researcher can upload English or German audio and receive a usable transcript without building, training, or tuning an ASR pipeline.
- On the German benchmarks Tuda-De and Verbmobil, the generic model beats the published task-specific results, suggesting that scale and diversity of training data can compensate for lack of task-specific adaptation.
- The beam-width experiments show that reducing search width from 20 to 5 raises WER only slightly, from 12.0% to 13.7% on Tuda-De, while cutting computation roughly in half, so users can trade a little accuracy for speed.
- Batched GPU inference processes one second of audio in about 0.7 seconds at the full beam width, making web-based use practical.
Reading between the lines
- Because the reported test sets all come from corpora used in training, the strongest unmeasured implication is whether the model generalizes to genuinely new recording conditions; a held-out evaluation with far-field, telephone, or overlapping speech would settle that.
- If the generic claim holds, a practical downstream consequence is that large archives of oral history, interviews, and lecture recordings in English and German could be transcribed at scale without per-collection model training.
- The comparison with a commercial speech API suggests open generic models are approaching commercial services on clean read and spontaneous speech; the same comparison on noisy or distant audio would show where the boundary lies.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents IMS-Speech, a web-based speech-to-text tool for English and German, built on Kaldi for speech activity detection and feature extraction and on ESPnet for end-to-end ASR with CTC/attention joint training and subword outputs. The English acoustic model is trained on about 2,277 hours and the German model on about 3,797 hours of public speech corpora, with external language models trained on the corresponding training transcriptions and additional German text. The paper reports word error rates on standard benchmarks (WSJ, LibriSpeech, TED-LIUM, AMI, Tuda-De, Verbmobil), compares against published state-of-the-art numbers and against Google's speech API on a subset of utterances, and shows batched GPU inference speedups in Table 3. The central claim is that these generically trained models achieve competitive accuracy across a diverse set of tasks and conditions.
Significance. If the claims are supported, the tool would be a useful free academic resource for non-specialist researchers who need transcriptions of spoken-language materials. The paper's main contribution is engineering: a reproducible pipeline combining a pretrained Kaldi SAD model, ESPnet-based end-to-end acoustic modeling, and external subword language models over a large multilingual training corpus, wrapped in a web interface. The reported results show that a single generic system can reach competitive WERs on several benchmark test sets, and Table 3 usefully documents a substantial real-time-factor improvement from GPU batching at equal WER. These strengths are tempered by the fact that every evaluation set shares its recording conditions with the training data, so the 'generic, cross-condition' generalization claim is not directly evidenced.
major comments (3)
- [§5.1, Tables 1 and 2] The central claim that the systems are generic and achieve 'competitive accuracy on a diverse set of tasks and conditions' is not fully supported by the reported evaluations, because every evaluation set in Table 2 comes from a corpus that also appears in the training list in Table 1: WSJ eval'92, LibriSpeech test-clean/test-other, TED-LIUM 3 test, AMI IHM/SDM/MDM eval, Tuda-De dev/test, and Verbmobil 1 dev/test. The external language model is additionally trained on 'transcriptions from the training speech databases' (Section 4), so the textual and acoustic conditions of the test sets have been seen during training. Held-out speakers within the same corpora demonstrate within-domain speaker generalization, not the ability to handle genuinely new recording conditions. The conclusion's explanation that gains come 'possibly due to better generalization from larger amount of training data' is likewise an untested hypothesis. The authors should either narrow the claim to 'competitive on the domains represented in the training data' or provide an out-of-domain evaluation on a corpus whose recording conditions, microphones, and speaking styles are absent from the training set.
- [§5.2, Tables 4 and 5] The Google API comparison is not a clean apples-to-apples comparison because the evaluation excludes all utterances for which Google produced digit-containing transcriptions or empty transcriptions. The reported coverage is 93% (LibriSpeech test-clean), 92% (test-other), 94% (Common Voice), 85% (Tuda-De), and only 53% (Verbmobil 1 test), so a large fraction of the hardest or most domain-specific utterances is removed. Since numeric transcriptions are a natural source of mismatch between systems, selectively filtering them can bias the WER gap in favor of IMS-Speech. The finding that Google has higher WER on the remaining utterances may still be informative, but the text should report the proportion of excluded utterances for each system separately, present results with standard numeric normalization, or include the full set to make the comparison interpretable.
- [§5.1, Table 2] The 'state of the art' column mixes results from heterogeneous sources that were not necessarily produced under the same evaluation conditions, decoding setups, or training-data restrictions. For example, the TED-LIUM 3 test reference (6.7% WER) and the CAPIO LibriSpeech references come from systems with different architectures and possibly different external resources, while the AMI numbers point to Kaldi recipe logs. Because the manuscript uses the comparison to support the 'compete with task-specific systems' claim, the text should at least state, for each reference, whether the compared system uses task-specific adaptation, additional training data, or different scoring, or should accompany the comparison with confidence intervals or multiple evaluation runs so that small WER differences (e.g., WSJ 3.8 vs. 3.5) are not overinterpreted.
minor comments (5)
- [§5.2] The section heading contains a typo: 'Comparisions' should be 'Comparisons'.
- [§4] In the sentence 'External LM for the German system is trained on all transcriptions form the training speech databases', 'form' should be 'from'.
- [§5.2] The sentence 'The numbers suggest that that Google API models may be optimized...' contains a doubled 'that'; it should read 'that Google API models'.
- [§3, Table 3] Table 3 reports WER and real-time factor for beam widths 5, 10, 15, and 20, but the text does not state whether these runs use the same batch size on GPU and CPU or whether WER differences across beam widths are statistically meaningful; a sentence clarifying the evaluation setup would help.
- [References] Reference [13] appears as 'V ery deep convolutional networks...' due to a typographical artifact; please correct the spacing.
Circularity Check
No significant circularity; the paper is a system description with direct empirical WER measurements on standard benchmarks.
full rationale
The paper reports an ASR tool and evaluates its performance by measuring word error rate on established test sets (WSJ eval'92, LibriSpeech test-clean/test-other, TED-LIUM 3 test, AMI eval splits, Tuda-De dev/test, Verbmobil 1 dev/test). There is no derivation chain in which a predicted quantity is constructed from its own definition, and no fitted parameter is renamed as a prediction. Training details (network architecture, CTC/attention loss, LM scaling factors, subword vocabulary size) are standard model-fitting choices tuned on validation data, which is normal empirical practice rather than circular reasoning. The claim that the systems are 'generic' and 'competitive' is a generalization from measured results, not a quantity derived from the model's own equations. The concern that all test sets come from corpora used in training is a legitimate limitation about external validity and generalization to unseen conditions, but it is a correctness or evaluation-design issue, not circularity within the paper's logic. Citations to Kaldi, ESPnet, and other toolkits are external software components, not self-citations carrying unverified premises. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (6)
- Sub-word vocabulary size =
100
- CTC weight lambda =
0.5
- LM scaling factor gamma (English) =
0.5
- LM scaling factor gamma (German) =
1.1
- Beam size =
20
- Maximum speech segment duration =
30 seconds
assumptions (4)
- domain assumption Word error rate (WER) is a valid and sufficient metric for transcription accuracy.
- domain assumption The pretrained Kaldi SAD model, trained on Fisher-English, generalizes to the German and English recordings uploaded by users.
- domain assumption The test sets used in Table 2 are comparable despite coming from different sources with potentially different scoring protocols.
- domain assumption The multi-corpus training data is representative of the diverse conditions the tool targets.
Cite this review
Pith. "Pith review of IMS-Speech: A Speech to Text Tool." pith.science (2026). https://pith.science/paper/3GTNNPFZ
@misc{pith2026190804743,
author = {Pith},
title = {Pith review of: IMS-Speech: A Speech to Text Tool},
year = {2026},
howpublished = {\url{https://pith.science/paper/3GTNNPFZ}},
note = {Machine review of arXiv:1908.04743}
}
read the original abstract
We present the IMS-Speech, a web based tool for German and English speech transcription aiming to facilitate research in various disciplines which require accesses to lexical information in spoken language materials. This tool is based on modern open source software stack, advanced speech recognition methods and public data resources and is freely available for academic researchers. The utilized models are built to be generic in order to provide transcriptions of competitive accuracy on a diverse set of tasks and conditions.
Reference graph
Works this paper leans on
-
[1]
P OVEY, D., A. G HOSHAL , G. B OULIANNE , L. B URGET , O. G LEMBEK , N. G OEL , M. H ANNEMANN , P. M OTLICEK , Y. Q IAN , P. S CHWARZ ET AL .: The Kaldi speech recognition toolkit. In Proc. of ASRU. 2011
work page 2011
- [2]
-
[3]
W AIBEL , A., T. H ANAZAWA, G. H INTON , K. S HIKANO , and K. J. L ANG : Phoneme recognition using time-delay neural networks . In Readings in speech recognition . 1990
work page 1990
-
[4]
G HAHREMANI , P., V. M ANOHAR , D. P OVEY , and S. K HUDANPUR : Acoustic Modelling from the Signal Domain Using CNNs . In Proc. of Interspeech. 2016
work page 2016
-
[5]
IEEE Transactions on Information Theory , 1967
V ITERBI , A.: Error bounds for convolutional codes and an asymptotically optimum de- coding algorithm. IEEE Transactions on Information Theory , 1967
work page 1967
-
[6]
B AHDANAU , D., K. C HO, and Y. B ENGIO : Neural machine translation by jointly learn- ing to align and translate . arXiv preprint arXiv:1409.0473, 2014
arXiv 2014
-
[7]
G RAVES, A., S. F ERNÁNDEZ , F. G OMEZ , and J. S CHMIDHUBER : Connectionist tempo- ral classification: labelling unsegmented sequence data wi th recurrent neural networks . In Proc. of ICML. 2006
work page 2006
-
[8]
S ENNRICH , R., B. H ADDOW , and A. B IRCH : Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015
arXiv 2015
Show all 32 references
-
[9]
I RIE , R
Z EYER , A., K. I RIE , R. S CHLÜTER , and H. N EY: Improved training of end-to-end at- tention models for speech recognition . arXiv preprint arXiv:1805.03294, 2018
2018 arXiv
-
[10]
arXiv preprint arXiv:1804.10959, 2018
K UDO , T.: Subword Regularization: Improving Neural Network Transla tion Models with Multiple Subword Candidates. arXiv preprint arXiv:1804.10959, 2018
2018 arXiv
-
[11]
M ILLER , and K
C IERI , C., D. M ILLER , and K. W ALKER : The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text. In LREC. 2004
2004
-
[12]
P EDDINTI , D
K O, T., V. P EDDINTI , D. P OVEY , M. L. S ELTZER , and S. K HUDANPUR : A study on data augmentation of reverberant speech for robust speech r ecognition. In Proc. of IEEE ICASSP. 2017
2017
-
[13]
S IMONYAN , K. and A. Z ISSERMAN : V ery deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[14]
F ERNÁNDEZ , and J
G RAVES, A., S. F ERNÁNDEZ , and J. S CHMIDHUBER : Bidirectional LSTM networks for improved phoneme classification and recognition . In International Conference on Artificial Neural Networks , pp. 799–804. Springer, 2005
2005
-
[15]
H OCHREITER , S. and J. S CHMIDHUBER : Long short-term memory. Neural computation, 9(8), pp. 1735–1780, 1997
1997
-
[16]
D.: ADADELTA: an adaptive learning rate method
Z EILER , M. D.: ADADELTA: an adaptive learning rate method . arXiv preprint arXiv:1212.5701, 2012
2012 arXiv
-
[17]
K INGMA , D. P. and J. B A: Adam: A method for stochastic optimization . arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[18]
M ILDE , B. and A. K ÖHN : Open Source Automatic Speech Recognition for German . In Proc. of ITG. 2018
2018
-
[19]
C HEN , D
P ANAYOTOV, V., G. C HEN , D. P OVEY , and S. K HUDANPUR : Librispeech: an ASR cor- pus based on public domain audio books . In Proc. of IEEE ICASSP . 2015
2015
-
[20]
G ODFREY , J. J., E. C. H OLLIMAN , and J. M CDANIEL : SWITCHBOARD: Telephone speech corpus for research and development . In Proc. of IEEE ICASSP . 1992
1992
-
[21]
N GUYEN , S
H ERNANDEZ , F., V. N GUYEN , S. G HANNAY , N. T OMASHENKO , and Y. E STEVE : TED- LIUM 3: twice as much data and corpus repartition for experim ents on speaker adapta- tion. arXiv preprint arXiv:1805.04699, 2018
2018 arXiv
-
[22]
Language Resources and Evaluation , 2007
C ARLETTA , J.: Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus . Language Resources and Evaluation , 2007
2007
-
[23]
P AUL, D. B. and J. M. B AKER : The design for the Wall Street Journal-based CSR corpus. In Proc. of the workshop on Speech and Natural Language . 1992
1992
-
[24]
M ILDE , A
R ADECK -A RNETH , S., B. M ILDE , A. L ANGE , E. G OUVÊA , S. R ADOMSKI , M. M ÜHLHÄUSER , and C. B IEMANN : Open source german distant speech recognition: Corpus and acoustic model . In Text, Speech, and Dialogue. 2015
2015
-
[25]
S TEGEN , and T
K ÖHN , A., F. S TEGEN , and T. B AUMANN : Mining the Spoken Wikipedia for Speech Data and Beyond. In Proc. of LREC. 2016
2016
-
[26]
Springer Sci- ence & Business Media, 2013
W AHLSTER , W.: V erbmobil: foundations of speech-to-speech translation . Springer Sci- ence & Business Media, 2013
2013
-
[27]
B URGER , S. and F. S CHIEL : RVG 1 – A Database for Regional Variants of Contemporary German. In Proc. of LREC. 1998
1998
-
[28]
H ESS , W. J., K. J. K OHLER , and H.-G. T ILLMANN : The Phondat-verbmobil speech corpus. In F ourth European Conference on Speech Communication and Technology. 1995
1995
-
[29]
C HAN , W. and I. L ANE : Deep recurrent neural networks for acoustic modelling . arXiv preprint arXiv:1504.01482, 2015
2015 arXiv
-
[30]
H AN, K. J., A. C HANDRASHEKARAN , J. K IM, and I. L ANE : The CAPIO 2017 conver- sational speech recognition system. arXiv preprint arXiv:1801.00059, 2017
2017 arXiv
-
[31]
L ANGE , R
G AIDA , C., P. L ANGE , R. P ETRICK , P. P ROBA, A. M ALATAWY, and D. S UENDERMANN -O EFT : Comparing open-source speech recognition toolkits . Tech. Rep., DHBW Stuttgart, 2014
2014
-
[32]
B UI, and W
D ERNONCOURT , F., T. B UI, and W. C HANG : A Framework for Speech Recognition Benchmarking. Proc. of Interspeech, 2018
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.