{"id":"3ea6eaf5-34d9-4338-a1e3-c35b8c809021","arxiv_id":"1908.02119","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper reports a 100% command completion rate for a small vocabulary Indonesian speech recognizer built with HTK, but no valid held-out evaluation supports that number.","lead":"This paper reviews a practical HTK recipe for small vocabulary speech recognition and illustrates it with an Indonesian voice-controlled home automation switch. It reports 100% command completion, but the evaluation is anecdotal and the refinement loop retrains on test errors.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% completion rate is not an independent evaluation: Section IV.G adds each failed utterance to the training database and retests it, so the reported accuracy measures memorization rather than generalization.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I would name: the 100% completion rate depends on a clean separation between training and test data, and Section IV.G removes that separation by design. I checked the full text for an independent validation protocol and found none. Section IV.F describes testing with HVite but gives no utterance count, no speaker list, no train/test split, and no confusion matrix. Section IV.G then explicitly describes retraining on errors, and Section IV.H describes deployment only functionally. The tutorial content, including grammar design, contraction handling, matched recording conditions, volume scaling, noise labeling, and HInit/HRest training, is coherent and could be practically useful; I am not treating disagreement with mainstream ASR practice as an error. The reject-level problem is narrower: the paper makes a quantitative empirical claim without a valid evaluation of it. A held-out session split or a frozen-model fresh deployment test would settle whether the procedure generalizes or merely memorizes. I agree with the reader's verdict and see no reason to adjust it.","tokens_in":6653,"tokens_out":4304,"duration_ms":47253,"concrete_test":"Require release of the annotated command database and definition of a fixed, disjoint evaluation split by recording session (e.g., all commands from session 1 for training, session 2 held out). Re-run the exact HTK pipeline described in Section IV without applying Section IV.G refinement, then compute command completion rate on the held-out session. If data cannot be released, run a fresh deployment with a frozen model and record results on a prespecified number of previously unheard utterances per command, reporting N and per-command errors. If 100% is not achieved on unseen data, the abstract's accuracy claim should be relabeled as training-set performance, not system accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the HTK-based SVASR achieved a 100% command completion rate. The load-bearing condition for that claim is that reported accuracy is measured on utterances that were not used to construct the recognizer. Section IV.G ('Refining') violates that condition: 'If in online testing an error occurs in the sentence that is spoken, then the sentence is taken and labelled and added to the database, then retest the sentence again.' Every observed error is re-inserted into the training database, the acoustic model is retrained, and the same utterance is retested until it is recognized. The paper also permits expanding the pronunciation dictionary with the wrongly detected phoneme sequence, so the lexicon itself is adapted to the test items. The final '100% command completion rate' therefore describes a model that has already seen the test items, not a system's ability to generalize to unseen commands. No held-out partition, cross-validation, per-condition accuracy breakdown, or sample size is reported, and no data or code are provided to allow an external check. Without an evaluation that freezes the acoustic model and dictionary before testing on previously unseen recordings, the empirical conclusion is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a practical procedure for building a small-vocabulary automatic speech recognition (SVASR) system with the Hidden Markov Model Toolkit (HTK). The procedure covers grammar and pronunciation dictionary design, recording a training database under conditions matched to the target environment, MFCC feature extraction with optional PCA, HMM-GMM acoustic model training, online testing with HVite, a refinement step that retrains on newly observed errors, and deployment via an Arduino-based electrical switch. The method is illustrated with an Indonesian-language voice-controlled electrical switch (IDSwitch) system. The central empirical claim, stated in the abstract and conclusion, is a 100% command completion rate for this system.","tokens_in":6967,"tokens_out":3020,"duration_ms":30784,"significance":"If the central claim were validated, the paper would offer a useful practical recipe for rapidly building small-vocabulary ASR systems in moderately noisy environments. The paper gives explicit credit for several practical ideas: representing noise as a word in the dictionary, scaling microphone volume to reduce silence-detector false triggers, modeling cross-word contractions via dictionary entries, and recording training data with the same program (HVite) used for testing. These are concrete and potentially helpful for practitioners. However, the reported 100% command completion rate is not supported by a valid evaluation. The refinement procedure described in Section IV.G explicitly adds each failed test utterance to the training database, retrains the model, and retests the same utterance until it is recognized; the paper also permits adding the wrongly detected phoneme sequence to the dictionary. Consequently, the reported accuracy measures memorization of the test items rather than generalization to unseen commands.","major_comments":[{"comment":"The central claim of 100% command completion rate is invalid as an accuracy measurement because the evaluation is circular. Section IV.G states that when an error occurs in online testing, the sentence is taken, labelled, added to the database, and then retested after retraining. This means the utterances used to measure the final accuracy were included in the training set of the model that produced the reported result. The same section also permits adding wrongly detected phoneme sequences to the pronunciation dictionary, so the lexicon is adapted to the test items as well. The reported 100% therefore reflects the system's ability to memorize the specific test sentences, not its ability to generalize to unseen commands. To support the claim, the authors must freeze the acoustic model and dictionary after training and then evaluate on a separate set of previously unseen recordings, with no refinement applied after the evaluation begins.","section":"Section IV.G"},{"comment":"The evaluation is under-specified in load-bearing ways. No number of test utterances, number of speakers, number of repetitions per command, or per-condition accuracy breakdown is given. The deployment section mentions a 'frame probability' threshold that revalidates recognition results, but the threshold value is never reported, so the reader cannot know how many ambiguous results were discarded before computing the 100% completion rate. Without these details, even a held-out evaluation would not be interpretable. The authors should report the test protocol, the test set size, the threshold used, and accuracy per command and per noise condition.","section":"Section IV.F and IV.H"},{"comment":"The feature extraction parameters as written are internally inconsistent with the stated 16 kHz sampling rate. The text says 'windowing of 250 ms (400 samples)' and 'striding of 100ms (160 samples) and overlapping of 150 ms (240 samples)'. At 16 kHz, 400 samples correspond to 25 ms, not 250 ms, and 160 samples correspond to 10 ms, not 100 ms. This appears to be a factor-of-ten error that affects the reproducibility of the described procedure. The authors should correct the stated window length, frame shift, and overlap values to be consistent with the sampling rate.","section":"Section IV.D"}],"minor_comments":[{"comment":"The refinement step is described twice (Section III.F 'Refinement' and Section IV.G 'Refining') with slightly different wording; the relationship between these descriptions should be clarified to avoid confusion about when refinement is applied.","section":"Section III.F vs IV.G"},{"comment":"Table II has duplicate row numbers ('5' appears twice) and skips numbers 3 and 4. The table should be renumbered consecutively.","section":"Table II"},{"comment":"The caption contains a typo: 'Pronounciation' should be 'Pronunciation'.","section":"Figure 3"},{"comment":"The caption text 'contains only one single sensor' presumably means 'sensor' rather than 'censor'.","section":"Figure 5 caption"},{"comment":"The biography says 'he thought computer science' but should read 'he taught computer science'.","section":"Author biography"},{"comment":"Several references are incomplete or lack page numbers (e.g., [11] and [14]); the authors should verify that all citations follow a consistent format.","section":"References"}],"recommendation":"reject","confidential_remarks":"The circular evaluation in Section IV.G is explicit and not a subtle methodological ambiguity: the text says failed test sentences are added to the training database and retested until recognized. This cannot be repaired by local edits; it requires a new experimental design with held-out test data. The paper also lacks basic statistics on the evaluation and contains a factual error in the feature-extraction parameters. I recommend rejection, though a substantially revised version that removes the unsupported 100% claim and reports a proper held-out evaluation could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading this one. First, it is a genuinely useful practical tutorial on building small-vocabulary HTK recognizers. Second, its headline 100% accuracy claim is unsupported, and the reason is visible in the paper itself: Section IV.G describes retraining on utterances that fail in online testing, then retesting them until they pass. The reported accuracy is a measure of memorization, not generalization.\n\nGive credit where it is due. The paper is written by someone who has clearly spent time fighting HTK in real conditions. The advice on matching recording and test environments, scaling microphone volume to avoid false silence-detector triggers, labeling noise as a separate class, and handling Indonesian cross-word contractions with dictionary entries is practical and specific. The step-by-step walkthrough of grammar, dictionary, recording, labeling, MFCC extraction, and HMM-GMM training with HInit/HRest is a coherent field guide. The author also cites the standard references. For a practitioner who wants a quick start with HTK on a small vocabulary in a noisy room, this is a reasonable place to look.\n\nNow the soft spots, and the biggest one is load-bearing. The 100% command completion rate is not an independent evaluation. Section IV.G explicitly says that when an online test error occurs, the sentence is taken, labeled, added to the database, and the system is retrained and retested until it works. The paper also allows adding the wrongly detected phoneme sequence to the dictionary. That means the acoustic model and lexicon are being adapted to the specific test sentences. No held-out partition, cross-validation, sample size, speaker count, or threshold value is reported. No code or data is provided. The claim in the conclusion that the system “achieved 100% command recognition rate” therefore collapses once you read the refinement section.\n\nOther issues are minor by comparison. The noise-condition table has inconsistent numbering, and the paper does not compare against prior Indonesian SVASR work, which is fine for a tutorial but weak for a research paper. The free parameters are not systematically explored; the volume scaling factor c and the frame-probability threshold are mentioned but never analyzed.\n\nWho is this paper for? Someone who wants to replicate the practical steps could benefit from it, but as a research claim it should not survive peer review. The author is not being dishonest; the refinement step is described in the open. The problem is that the conclusion overclaims. A reader with a proper held-out evaluation would be in a much better position.\n\nRecommendation: desk reject from a research venue, but consider inviting the author to resubmit to a practice-and-experience track or to revise with a real evaluation. If the accuracy claim were replaced with a transparent description of the refinement process and its limits, the tutorial would be a solid engineering report.","headline":"A practical HTK tutorial with real field experience, but its 100% accuracy claim is circular because Section IV.G retrains on test errors.","tokens_in":7401,"tokens_out":1841,"would_cite":false,"duration_ms":22989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a practical HTK workflow for small-vocabulary speech recognition and a 100% command completion rate on an Indonesian voice-controlled electrical switch.","keywords":["speech recognition","HTK","small-vocabulary ASR","home automation","Indonesian language","noise robustness","MFCC","hidden Markov models"],"falsifier":"Record a fresh set of Indonesian switch commands from the same speaker in the same room with the same microphone, run the finished HTK system on them without adding any of them to the training database or dictionary, and count completions. A completion rate below 100% would show that the reported figure reflects memorized training examples rather than generalization to new utterances.","tokens_in":6477,"feed_emoji":"🗣️","tokens_out":9258,"duration_ms":86228,"temperature":0.7,"pith_summary":"This paper lays out a practical, step-by-step recipe for building a small-vocabulary speech recognizer with the Hidden Markov Model Toolkit (HTK) and applies it to an Indonesian voice-controlled electrical switch. The central idea is to make training and testing share the same acoustic world: training audio is recorded through the same HVite decoder, the same Bluetooth microphone, and the same room noise that will be present during live use. Two practical fixes carry much of the weight: scaling down microphone volume so ordinary noises do not trigger HTK's silence detector, and adding labeled noise patterns to the training database as dictionary words. The paper reports a 100% command completion rate for the resulting switch-control system. If the recipe works as described, it offers a fast, low-cost route to accurate command recognition for small vocabularies in moderately noisy homes.","feed_headline":"A voice-controlled house switch completes 100% of commands","feed_subtitle":"An HTK recipe trains Indonesian commands in the same noisy room where the switch will listen.","key_machinery":"The load-bearing mechanism is a matched-condition training loop built around HTK's HVite program: HVite records the training audio, HCopy extracts MFCC_0_D_A features, HTK's HInit and HRest train phone-level HMM-GMM acoustic models, and HVite decodes live commands. Because the same decoder, microphone, room, and gain setting appear in training and testing, the model does not have to generalize across acoustic conditions. Supporting mechanisms are the volume-scaling rule $\\hat{x}(t)=c x(t)$ with $0<c<1$, labeled NOISE entries in the pronunciation dictionary, and contraction entries such as NUM_3_4 to prevent word deletion.","core_discovery":"The core claim is that HTK, used with the right data-collection discipline, is enough to build a noise-tolerant small-vocabulary recognizer quickly. The decisive move is to record the training database with the same program that will perform live testing, HVite, under the same microphone, room, and gain conditions, instead of relying on clean speech corpora recorded in quiet studios. The paper further shows that reducing microphone gain by a factor $c$ with $0<c<1$ prevents everyday noises from being misread as words, and that including noise segments in the training database as explicitly labeled entries such as NOISE makes the recognizer robust to them. Indonesian cross-word contractions are handled by adding combined dictionary entries such as NUM_3_4. With this workflow, the paper finds that the Indonesian switch recognizer completed 100% of spoken commands, whether spoken as a switch number or as a named switch.","pith_inferences":["A fair multi-speaker evaluation would probably lower the completion rate, because the procedure as written ties the model to one speaker, one microphone, and one room; extending it would require per-speaker retraining or speaker adaptation.","The dictionary-entry fix for contractions is a workaround that grows combinatorially with vocabulary size, so a larger Indonesian recognizer would need a real lexicon or language model rather than more dictionary entries.","Because the underlying HTK algorithms are standard, the recipe's real contribution is data-collection discipline; a direct test would compare this matched-condition loop against a clean-trained baseline on the same held-out commands to quantify the gain."],"forward_implications":["A small-vocabulary recognizer for Indonesian can be built and deployed on an ordinary PC or smartphone with low-cost tools and a cheap Bluetooth microphone.","Recording training audio through the same decoder used for testing, in the same room, is the main route to reliable command recognition under noise.","Lowering microphone gain is enough to stop everyday noises from triggering HTK's silence detector and creating spurious words.","Adding dictionary entries for cross-word contractions removes a common word-deletion error in Indonesian digit sequences.","Residual online errors can be corrected by recording the failed sentence, labeling it, retraining, and adding variant pronunciations to the dictionary."],"supporting_citations":[{"why":"Supplies the HTK program suite, HMM training commands, and HVite decoder that the entire procedure is built on.","marker":"[1]"},{"why":"Defines the HMM-GMM framework used for the phone-level acoustic models.","marker":"[13]"},{"why":"Defines the MFCC parameterization that HCopy computes for the feature files.","marker":"[10]"},{"why":"Provides the phoneme list and 16 kHz sampling convention adopted for the Indonesian dictionary.","marker":"[6]"},{"why":"Motivates the noise-robustness problem that the matched-condition procedure targets.","marker":"[4]"},{"why":"Supports the claim that PCA improves recognition for noise-distorted input.","marker":"[12]"},{"why":"Supplies Praat for phoneme-level labeling of the training recordings.","marker":"[8]"},{"why":"Supplies SFS as the labeling format that converts conveniently into HTK's format.","marker":"[7]"}],"fun_headline_variants":["HTK beats noise: 100% Indonesian voice switch accuracy","Train with HVite, run in same room: 100% command success","Quiet the mic, keep noise in training: 100% home commands","Match testing and training: HTK switch nails 100% of commands"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100% completion rate assumes the test commands were not also training examples; the paper's refinement step, however, adds each failed sentence to the training database and retrains until it is recognized.","fun_headline_variants_meta":{"raw":{"variants":["HTK beats noise: 100% Indonesian voice switch accuracy","Train with HVite, run in same room: 100% command success","Quiet the mic, keep noise in training: 100% home commands","Match testing and training: HTK switch nails 100% of commands"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2730,"prompt_tokens":863,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":1788}},"tokens_in":479,"tokens_out":1867,"duration_ms":13277,"temperature":1.0,"reasoning_tokens":1788,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:52:45.528009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a fresh set of Indonesian switch commands from the same speaker in the same room with the same microphone, run the finished HTK system on them without adding any of them to the training database or dictionary, and count completions. A completion rate below 100% would show that the reported figure reflects memorized training examples rather than generalization to new utterances.","supporting_citations":[{"cited_title":"A Tutorial on Hidden Markov Mo dels and Selected Applications in Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the HMM-GMM framework used for the phone-level acoustic models."},{"cited_title":"Speaker-Independent Phon e Recognition Using Hidden Markov Models,","cited_arxiv_id":null,"evidence_quote":"Provides the phoneme list and 16 kHz sampling convention adopted for the Indonesian dictionary."},{"cited_title":"Robust Sp eech Recognition for Car Environment Noise,","cited_arxiv_id":null,"evidence_quote":"Motivates the noise-robustness problem that the matched-condition procedure targets."},{"cited_title":"PCA-Based Speech E nhancement for Distorted Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that PCA improves recognition for noise-distorted input."},{"cited_title":"Speak and unSpea k with Praat,","cited_arxiv_id":null,"evidence_quote":"Supplies Praat for phoneme-level labeling of the training recordings."},{"cited_title":"The SPAR Speech Fil ing System,","cited_arxiv_id":null,"evidence_quote":"Supplies SFS as the labeling format that converts conveniently into HTK's format."}],"review_version":1}