{"id":"8ae902c5-23f4-4a18-aa97-01d92d4a98b2","arxiv_id":"2506.16174","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On one Finnish rap song, Faster Whisper and YouTube auto-captions both mishear lyrics, and vocal isolation does not reliably reduce errors, with no statistically meaningful winner.","lead":"This paper compares two speech-to-text tools on a Finnish rap song, counting hallucinations and mishearings against the official lyrics. It finds both tools make errors and that stripping out the music does not clearly help, but the sample is far too small to draw a real conclusion.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 8's summary counts are internally inconsistent: the per-lyric verdicts above it give B 4/7, A 2/7, and one draw, not B 3/7; the reported quantitative outcome is unsupported as printed.","rationale":"The reader's verdict of REJECT is correct. However, the single most load-bearing concern is simpler than the reader's named weakest assumption: the Section 8 aggregate count does not match the per-lyric verdicts in the same section. The central quantitative claim is the winner count, and that count is internally inconsistent; correcting it would move B from 3 to 4 wins, which only strengthens the already non-significant preference for preprocessing in this dataset. The paper is transparent about n=1, hand-picked lyrics, and an informal metric, and it claims no statistical confidence, so the stakes are low; still, a reported count cannot be accepted if it contradicts the table it summarizes. If the count were fixed, the remaining concern would be the undefined hallucination/mishearing split in Section 6.1 step 1 and the unblinded judge, which makes the Levenshtein totals non-reproducible. Those issues are secondary because any fix to the count requires the same subjective judgments; the whole evaluation procedure would benefit from a pre-registered error definition and a second blind annotator, especially since Section 3.3 already states a 'gut feeling' that YouTube would win.","tokens_in":8409,"tokens_out":3900,"duration_ms":44629,"concrete_test":"Recompute the winner counts from the Section 8 per-lyric verdicts only, manually: assign each of Lyrics #1-#7 a win using the verdict text and Levenshtein values stated, then compare the tally to the paragraph's 'won twice'/'won three times' sentence. If the recomputation returns B=4, A=2, draw=1, the summary sentence is wrong and must be corrected before any conclusion about preprocessing can be drawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Tallying the seven verdicts in Section 8 from the displayed strings and stated Levenshtein distances: Lyric #1 B wins, #2 A wins, #3 draw, #4 B wins, #5 A wins, #6 B wins, #7 B wins. This yields A=2, B=4, draw=1. The paragraph immediately below says A won twice and B three times out of seven; with only one draw these numbers do not sum to seven either. The discrepancy is not resolved by the paper's 'no statistical confidence' caveat, because the summary number is contradicted by the evidence printed in the same section. A secondary but related weakness is Section 6.1's winner rule: step 1 requires the first author to classify each difference as hallucination or mishearing without an operational definition, and Section 3.3 records a prior 'gut feeling' that YouTube is better, so the per-lyric tallies are not independently reproducible. But the arithmetic contradiction can be established without appealing to that subjectivity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two automatic speech-to-text systems, Faster Whisper (referred to as 'Faster Whisperer') and YouTube's internal speech-to-text, on the task of transcribing a Finnish rap song, 'Pantterinousut'. The authors define an informal error metric that distinguishes hallucinations from mishearings, evaluate both systems on seven selected lyric fragments, and also test three preprocessing approaches (ICA, center cancellation, and a LALAL.AI stem splitter) aimed at isolating vocals before transcription. The main reported findings are that YouTube and Faster Whisper perform similarly on the raw audio (YouTube 2 wins, Faster Whisper 3 wins, 2 draws), that the LALAL.AI-preprocessed input changes the tally (raw audio 2 wins, preprocessed 3 wins, with a draw), and that, given the small sample, there is no statistical confidence in either comparison.","tokens_in":8590,"tokens_out":4808,"duration_ms":52764,"significance":"As a small, informal case study, the paper has some strengths: it is transparent about its methodology, includes the commands and code used, discloses the outputs for each lyric, and explicitly acknowledges the lack of statistical power. The authors also name the song and provide the reference URL, so the experiment is in principle reproducible. However, the contribution is limited by the subjective error metric, the very small sample size (one song, seven lyric fragments), and the judge being the first author, who states a prior 'gut feeling' in Section 3.3. The internal arithmetic inconsistency in Section 8 undermines the one quantitative summary. If the tally and metric issues are fixed, the paper could serve as a useful data point for informal comparisons of ASR on Finnish rap, but in its current form the reported counts are not trustworthy.","major_comments":[{"comment":"The reported Levenshtein distances are not consistent with the displayed strings. In Lyric #3, the paper reports Faster Whisperer mishearing 'sallii' as 'salliin' with distance 2, but that substitution is a single insertion. In Lyric #5, the printed distance totals for Faster Whisperer (3) and YouTube (4) appear too small: a direct edit count of 'Tordaita alas, niinku katu hauskaa' against 'Portaita alas, niin kuin katuhaukka' is considerably larger, even allowing for the paper's half-distance rule. Because the winner rule in Section 6.1 relies on these distances, the alignment used for each distance must be shown.","section":"Section 6.2, Lyrics #3 and #5"}],"minor_comments":[{"comment":"The paper consistently writes 'Faster Whisperer' where the model is actually 'Faster Whisper'; the reference in [3] and the OpenAI Whisper reference in [6] use the correct name.","section":"Throughout"},{"comment":"Approaches #1 (ICA) and #2 (center cancellation) are described as failures, but no quantitative or qualitative evidence is given for that assessment; at least a brief description of what the resulting audio sounded like would help the reader.","section":"Section 7"},{"comment":"The phrase 'automatic text-to-speech translation' should read 'speech-to-text'.","section":"Section 3.3"},{"comment":"The 'Tordaita alas, niinku katu hauskaa' extraction is called an example of AI hallucination in Section 5, but in Section 6.2 Lyric #5 it is classified as 'hallucinates twice and mishears one word,' which is a contradiction in the labeling.","section":"Section 5 and Section 6.2"},{"comment":"There are several typos: 'Levenshstein' appears multiple times, 'Pantteri nousuista' in Lyric #7 of Section 8 is later spelled 'Pantteri nousuista' (inconsistent use of 'Pantteri' vs 'Pantteri'), and the reference to 'ref. 23' in Section 3.3 does not point to the caption reference.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is openly informal and is probably more of a fun technical blog post than a journal paper. The arithmetic error in Section 8 is easy to fix, but the subjectivity of the error metric is a more structural problem; even with a corrected count, the paper's central claims depend on a classification that only the first author can reproduce. If the journal is not willing to consider such an informal case study, the scope may be a separate reason to reject, but for the content itself I would ask for the corrections and re-labeling described in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a fun, honest case study, not a serious scientific evaluation, and its main quantitative conclusion doesn't survive contact with its own numbers.\n\nWhat's actually new here is narrow but real: a concrete data point comparing Faster Whisper and YouTube ASR on a Finnish rap song, plus an explicit test of three preprocessing approaches. The LALAL.AI vocal isolation did change outputs, sometimes improving and sometimes worsening the transcription, and the authors report that mixed outcome without overselling it. They also deserve credit for disclosing n=1, hand-picked lyrics, the first author as judge, and the informal error function up front. That is more transparency than many papers show.\n\nThe soft spot is load-bearing. In Section 8, tallying the seven listed per-lyric verdicts gives Method B four wins, Method A two wins, and one draw. The text immediately below says A won twice, B won three times, and \"the draw happened twice.\" That is not a rounding difference; it is an arithmetic contradiction, and this tally is the entire basis for the preprocessing conclusion. The authors' \"no statistical confidence\" caveat does not fix it.\n\nA second weakness is the error metric in Section 6.1. Step 1 asks the first author to classify each difference as hallucination or mishearing without an operational definition, and Section 3.3 records a prior gut feeling that YouTube is better. Combined with the ad hoc halving rule for phonemically close single-character changes, the per-lyric tallies are not independently reproducible. The authors admit the error function is informal, which helps, but it still means the reported winner counts are subjective.\n\nWho is this for? A reader curious about how off-the-shelf ASR handles stylized Finnish rap, or someone looking for a cautionary example of evaluation pitfalls. It is not a paper that advances methodology or provides statistically meaningful evidence. The citation pattern is actually fine, with relevant references to Whisper, FastICA, and Finnish speech models.\n\nIf this crossed my desk at a serious journal, I would desk reject: the main result as printed is internally inconsistent, and the measurement procedure is not reproducible. It could be acceptable at a workshop or as a non-archival case study after the tally is corrected and the classification rule is tightened. As is, it does not deserve referee time.","headline":"A self-aware, n=1 case study whose one quantitative claim is internally contradicted by its own table, so the preprocessing result is unsupported as printed.","tokens_in":9114,"tokens_out":2841,"would_cite":false,"duration_ms":35052,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two speech-to-text engines trade wins on a Finnish rap song, with no clear champion and mixed results from stripping out the music.","keywords":["AI hallucination","automatic speech recognition","Faster Whisper","Finnish rap","speech-to-text","Levenshtein distance","stem separation","YouTube captions"],"falsifier":"Re-tally Section 8's per-lyric verdicts: the winners listed are B, A, draw, B, A, B, B, which is four wins for B and two for A—not the three B wins stated in the summary; recounting the table decides whether the headline count is a typo or a different counting rule.","tokens_in":8198,"feed_emoji":"🎤","tokens_out":7115,"duration_ms":73904,"temperature":0.7,"pith_summary":"The paper tries to measure how often two automatic speech-to-text systems—Whisper-derived Faster Whisper and YouTube's internal caption engine—hallucinate or mishear Finnish rap lyrics from a single song. The authors build an informal error metric that counts hallucinations as worse than mishearings and adjusts Levenshtein edit distance for phonemically close single-character swaps. Their verdict is that neither system is clearly better, and that preprocessing the audio with a stem splitter to isolate vocals improves some lyric extracts while worsening others. The point matters because Finnish is under-served by speech-to-text services, and rap over synth music is a stress case for automatic subtitling.","feed_headline":"Rap lyrics expose no clear winner in speech-to-text duel","feed_subtitle":"One-song test: YouTube captions and Faster Whisper trade wins, and vocal-only preprocessing helps only some lines.","key_machinery":"The load-bearing object is the informal human-assisted error function of Section 6.1. For each difference between the transcribed line and the written Finnish lyrics, the first author first classifies the difference as a hallucination or a mishearing; hallucinations are penalized most heavily, mishearings are scored by Levenshtein string edit distance, and a single-character substitution with low phonemic distance (like b↔p or t↔d) counts as half an error. This metric decides every per-lyric verdict and the overall winner count. The other machinery is the additive signal model audio(t) = vocals(t) + instrumental(t), used to frame three preprocessing attempts—independent component analysis, center-channel cancellation, and LALAL.AI stem separation—of which only the last produced a usable vocal track.","core_discovery":"On the paper's own terms, the central discovery is that a large local Whisper-based model and YouTube's proprietary speech-to-text engine are roughly matched on Finnish rap: in the seven sampled lyrics, YouTube won twice, Faster Whisper won three times, and two lines were perfect for both. When vocals were separated from the instrumental with a stem splitter, the paper reports that pre-processed audio won three times and raw audio twice, but it also notes the result is not statistically meaningful and that preprocessing helps some lines and hurts others. The authors explicitly state that it is not clear that YouTube's algorithm performs better. The paper's own per-lyric table in Section 8 actually lists four wins for the pre-processed method and two for raw audio, with one draw, so the stated 3-out-of-7 summary is internally inconsistent; either way, the conclusion of no clear winner remains the intended claim.","pith_inferences":["Editorial inference: a natural extension the authors do not pursue is to replace the binary phonemic halving rule with a continuous phoneme-confusion score, such as one based on IPA feature distances, which would make the metric less sensitive to the judge's choices.","Editorial inference: the internal tally discrepancy (three vs four B wins) suggests that the counting rule may not be fully specified; a re-analysis that states exactly how multi-word hallucinations and multiple mishearings within one lyric are counted would decide whether the headline number is a typo.","Editorial inference: the same paired-comparison protocol could be run on a dozen Finnish rap tracks and scored with a McNemar-style test of the per-lyric winners, turning the authors' 'gut feeling' into a statistical statement.","Editorial inference: since preprocessing artifacts are a plausible cause of the mixed results, a testable extension would compare several stem splitters on the same track and check whether any splitter's vocal isolation quality predicts ASR improvement line by line."],"forward_implications":["If the claim holds, automatic subtitling of Finnish rap cannot yet rely on a single off-the-shelf model; hallucinations and mishearings will occur in roughly equal measure across engines.","Preprocessing by stem separation is not a guaranteed fix: it can remove background music artifacts but can also introduce new mishearings or hallucinations on other lines.","The metric's penalty structure matters: a system with fewer hallucinations can be declared the winner even when its total edit distance is larger, so hallucination rate, not just word accuracy, drives the comparison.","Because the sample is one song, the paper's own conclusion is that stronger claims would require more songs and an automated version of the error function."],"supporting_citations":[{"why":"Supplies the test audio and the reference lyrics that every transcription is scored against.","marker":"ref. 1."},{"why":"The Faster Whisper standalone build is the local ASR engine used for both raw and pre-processed audio runs.","marker":"ref. 3."},{"why":"Defines the Whisper encoder-decoder architecture and log-spectrogram preprocessing that Faster Whisper reimplements.","marker":"ref. 6."},{"why":"The model-selection rule of thumb that leads the authors to choose the Whisper large model for Finnish.","marker":"ref. 9."},{"why":"LALAL.AI is the stem splitter whose vocal-only output defines the pre-processed method in the comparison.","marker":"ref. 12."},{"why":"The claimed human difficulty in distinguishing b/p and t/d motivates halving Levenshtein distance for phonemically close substitutions.","marker":"ref. 21."}],"fun_headline_variants":["Finnish rap: no winner in AI speech-to-text faceoff","Whisper and YouTube captions tie on Finnish rap","Finnish rap stumps AI speech-to-text, no clear winner","AI hallucination in Finnish rap: no winner"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scoreboard rests on one person's hand-labeled judgment of whether each wrong word is a hallucination or a mishearing, with no operational rule given for telling the two apart.","fun_headline_variants_meta":{"raw":{"variants":["Finnish rap: no winner in AI speech-to-text faceoff","Whisper and YouTube captions tie on Finnish rap","Finnish rap stumps AI speech-to-text, no clear winner","AI hallucination in Finnish rap: no winner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000758,"raw_usage":{"total_tokens":3333,"prompt_tokens":877,"completion_tokens":2456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2387}},"tokens_in":493,"tokens_out":2456,"duration_ms":17863,"temperature":1.0,"reasoning_tokens":2387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:44:48.564194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-tally Section 8's per-lyric verdicts: the winners listed are B, A, draw, B, A, B, B, which is four wins for B and two for A—not the three B wins stated in the summary; recounting the table decides whether the headline count is a typo or a different counting rule.","supporting_citations":[],"review_version":1}