{"id":"b564ac4c-8e4f-4c3d-abff-ebd792d5f746","arxiv_id":"2506.22858","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Whisper with overlapping 5-second context windows and embedded entity tags improves named entity recognition and formatting on Spoken Wikipedia, with the largest gains on numerical entities.","lead":"This paper trains Whisper, a speech recognition model, with extra audio context on both sides of each 30-second segment, plus special tags around names and numbers. The result is better recognition and formatting of entities in long audio, especially for numbers, compared to a tag-fine-tuned baseline without the extra context.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference uses 35s left+mid windows while training uses 40s left+mid+right, so the model is evaluated without the right context it was trained on, undermining the claim that extended windows drive the reported gains.","rationale":"The reader's weakest_assumption focuses on the confound of the extra positional-embedding pretraining (Section 4.3) as an alternative explanation for the reported gains. While that is a valid confound, I identify a more direct flaw in the central mechanism: the paper trains on 40s chunks that include a right (future) context window, but Section 4.5 states that inference uses only 35s chunks (left+mid), with no right context. This means the model is evaluated without a component of the input it was trained on, creating a train/test distribution shift. The shift is especially problematic for the claimed boundary-handling behavior: the model is explicitly trained to delay or shift entities whose end falls into the right window, but at inference the right audio is absent, so such entities may be misaligned or omitted. The paper never reports WER for the windowed model, so we cannot check whether this mismatch harms general transcription quality. This concern is load-bearing because it directly undermines the interpretation that 'extended semantic windows' (both left and right) drive the improvements; at best, only the left context is used at test time. I still recommend CONDITIONAL rather than REJECT because the empirical improvements may survive the mismatch, but the authors must address it by evaluating with the full 40s input and providing WER. I disagree with the reader's choice of weakest_assumption because the inference mismatch is more fundamental and is not subsumed by the embedding-pretraining confound.","tokens_in":10095,"tokens_out":8286,"duration_ms":92373,"concrete_test":"Run Whisper-puncted-windowed inference on the Spoken Wikipedia test set in two configurations: (a) the reported 35s left+mid input, and (b) the full 40s left+mid+right input using a 5s look-ahead buffer (feasible for offline processing). Keep the same left-transcript prompting. Compute NER F1, entity CER, Jaro-Winkler, and WER on the central 30s for both. If (b) significantly outperforms (a), the published evaluation under-reports the method and the right-context training is essential; if (a) is comparable or better, the right-context training does not transfer to inference and the mismatch should be disclosed. Also report WER for (a) to check that the model does not have worse general transcription quality than the baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5 describes an inference procedure that feeds 35-second audio segments (left + mid) and prompts with the left window transcript, with a 5-second stride providing left context only. However, Section 4.1 and Section 4.4 train the model on 40-second chunks containing both left and right context (5s+30s+5s), with loss masked to the mid window. Consequently, the model is trained with the right (future) 5 seconds of audio as input but evaluated without it. This train/inference mismatch has two direct consequences: (1) The reported entity improvements cannot be attributed to 'extended semantic windows' as implemented at test time, because the deployed input does not include the right context that the training exploited; the 'effective semantic window' is actually 35s (left+mid), not 40s. (2) The mismatch may introduce systematic errors—e.g., the model was trained to delay or shift entities whose end falls in the right window, but at inference that right audio is absent, so such entities may be misaligned or dropped. The paper does not report WER for Whisper-puncted-windowed, so it is impossible to detect whether this mismatch degrades overall transcription accuracy. Without an ablation that separates the windowing mechanism from the right-context training (e.g., training a model on 40s chunks with right context but evaluating on 40s input), the headline claim is not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ASR fine-tuning approach for entity-aware transcription: Whisper-puncted-windowed is trained on 40-second chunks (5s left, 30s middle, 5s right), with a masked loss over the central 30 seconds, NER tags embedded in transcripts, and boundary-crossing entities shifted entirely into the right-hand chunk. A baseline Whisper-puncted model is fine-tuned on 30-second tagged chunks. On the Spoken Wikipedia dataset, the windowed model reports higher seqeval F1 for most entity types and lower CER for numerical entities, and the authors conclude that extended semantic windows improve ASR performance. The paper also introduces entity-formatting metrics for numerical and textual entities and releases code and models.","tokens_in":10425,"tokens_out":5005,"duration_ms":49502,"significance":"If the attribution to the extended context window were cleanly established, this would be a useful contribution: the boundary-shift idea is practical and clearly described, the per-entity-type evaluation with separate CER and Jaro-Winkler metrics is informative, and the public release of code and fine-tuned models supports reproducibility. The central comparison, however, is currently confounded by several simultaneous changes between baseline and proposed model, and the inference procedure does not match the training setup. As a result, the headline quantitative claims are not yet supported, although the underlying approach is worth investigating further with additional ablations and reporting.","major_comments":[{"comment":"Section 4.5 states that during inference the model processes 35-second left+mid audio segments (30 seconds for the first segment), while Sections 4.1 and 4.4 train on 40-second left+mid+right chunks with loss masked to the mid window. The deployed setup therefore removes the right (future) 5 seconds of audio that the model saw at training time, so the effective semantic window at inference is 35 seconds, not the 40 seconds claimed in the abstract. Moreover, training shifts entities crossing the mid-right boundary entirely into the right window (Figure 1), teaching the model to delay their emission; at inference that right audio is absent, which can cause those entities to be dropped or misaligned. Because Section 6 does not report WER for Whisper-puncted-windowed, the reader cannot tell whether this mismatch degrades overall transcription. The paper should either evaluate with a 40-second input that provides the right context, or add an ablation trained and evaluated with the 35-second left+mid input used at inference, and should report WER for the proposed model.","section":"Section 4.5 vs. Sections 4.1 and 4.4"},{"comment":"The comparison between Whisper-puncted and Whisper-puncted-windowed is confounded: the windowed model differs from the baseline not only in the 10 extra seconds of audio, but also in newly added positional embeddings pretrained on 100 hours of concatenated VoxPopuli audio with inserted silences, in three extra special tokens (<|left|>, <|mid|>, <|right|>), and in a different prompting structure. The models are also trained for different numbers of epochs (100 versus 150) with checkpoint selection reported at different epochs. Without ablations that isolate the context window from these other changes, the reported gains cannot be attributed specifically to the extended semantic window, and the introduction's claim that the results underscore the value of extended semantic windows is stronger than the experiments support.","section":"Section 4.3 and Section 5"},{"comment":"Section 6.1 reports WER only for the unmodified Whisper-Medium model (38%) and for Whisper-puncted (26%); no WER is reported for Whisper-puncted-windowed. Since NER F1 and formatting CER are computed on the model's own transcripts, a change in overall transcription quality could inflate or deflate the semantic metrics independently of the windowing mechanism. Reporting WER for the proposed model, and ideally entity-conditioned WER, is necessary to interpret the improvements in Tables 1 and 2.","section":"Section 6.1 and Tables 1-2"},{"comment":"The claimed improvements \"across all metrics\" are not supported by the tables. In Table 2, Jaro-Winkler scores are identical for PERSON (0.75/0.75), ORG (0.71/0.71), EVENT, LAW, FAC, PROD, LANG, and W_OF_ART, and in Table 1 DATE F1 is unchanged at 0.65. Section 6.2 itself acknowledges only \"marginal gains\" for textual entities. The abstract and introduction should be revised to claim improvements only for the specific entity types and metrics where they are observed, and the paper should include significance tests or confidence intervals, especially for low-count types such as MONEY (109 instances) and QUANTITY (351 instances).","section":"Tables 1 and 2 and Abstract/Introduction"}],"minor_comments":[{"comment":"The text refers to \"Whisper-puncted-segmented\" but the model under discussion is called \"Whisper-puncted-windowed\" elsewhere; the naming should be made consistent.","section":"Section 4.3"},{"comment":"Equation (7) defines the mask using tmid and tright but never defines tleft; the paper should define all boundary positions used in the mask and clarify whether the mask includes the special boundary tokens themselves.","section":"Section 4.4"},{"comment":"The description of inference for the first segment (30 seconds with no left context) and for subsequent segments (35 seconds with left context) does not state how these two conditions are pooled in the reported metrics; this should be clarified.","section":"Section 4.5"},{"comment":"The citation \"Jannet et al., 2017\" in the related work should be \"Ben Jannet et al., 2017\" to match the reference list entry.","section":"Section 2"},{"comment":"The column headers \"W-punct\" and \"W-pwindow\" are not expanded anywhere; adding a note in the table captions would improve readability.","section":"Tables 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the conditional assessment in the reader's report: the central problem is attribution, not circularity. The train/inference mismatch is real and the missing WER for the proposed model makes the semantic improvements hard to interpret. These issues are addressable within the scope of the manuscript, so I would not recommend rejection if the author can supply the missing ablations and metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper has a sensible, practical idea—train an ASR model on overlapping 40-second chunks with a masked loss over the central 30 seconds and teach it to defer boundary-spanning entities to the right chunk—and it ships code and models. But the evaluation as written doesn't support the headline claim. The comparison is confounded and, more importantly, the inference procedure drops the right-context that training used.\n\nWhat's new: the specific combination of overlapping windows with masked mid loss and entity reassignment is new, as far as I know. The tag-embedding approach itself is from prior work (Yadav, Gaido), but the windowing and boundary handling are the contribution. The paper also does something right in evaluating numerical entity formatting separately from textual entities using CER and Jaro-Winkler. That's appropriate and useful.\n\nWhere it falls down: First, WER is never reported for the proposed model. If the windowing hurts raw transcription, the entity gains are not meaningful. Second, the baseline Whisper-puncted uses 30-second chunks, while the windowed model gets a longer encoder (40s), pretrained positional embeddings on VoxPopuli, three extra special tokens, and 150 epochs of training instead of 100. There's no ablation to isolate the effect of the windowing. Third, and most serious, Section 4.5 describes inference with 35-second left+mid segments, while training uses left+mid+right. So the model is evaluated without the future context it was trained on. That makes the '40-second effective semantic window' claim inaccurate at test time; the effective window is 35 seconds. The stress-test note flagged this and it holds up. Fourth, there are no error bars or significance tests, and checkpoint selection appears to be test-based.\n\nOn the positive side, the numerical entity improvements (e.g., MONEY CER 0.42 to 0.11) are large and consistent with the boundary-reassignment mechanism. Textual entities are mostly unchanged, which is at least honest in the tables. The paper is clearly written and the artifacts are public.\n\nWho should read it: people working on ASR post-processing or end-to-end entity recognition will find the windowing idea worth knowing. It deserves a serious referee, but a reviewer should ask for the missing WER, an ablation that separates the context window from the other changes, and either a fix to the train/inference mismatch or an explanation for why it doesn't matter.","headline":"A practical windowing idea for ASR entity formatting, but the main comparison is confounded and the inference procedure drops the right context used in training.","tokens_in":10915,"tokens_out":3946,"would_cite":false,"duration_ms":41091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving a speech recognizer five extra seconds of audio on each side of its training chunk, while evaluating only the central part, improves how it recognizes and formats names and numbers.","keywords":["automatic speech recognition","named entity recognition","entity formatting","overlapping context windows","Whisper fine-tuning","long-form transcription","spoken Wikipedia","sequence labeling"],"falsifier":"Train a control model on the same 40-second inputs and masked central-window loss, but with the left and right 5-second overlap regions replaced by silence; if its entity F1 and numeric character error rate match Whisper-puncted-windowed, the reported gains come from longer input or the embedding warm-up, not from surrounding semantic context.","tokens_in":9910,"feed_emoji":"🎙️","tokens_out":10729,"duration_ms":101541,"temperature":0.7,"pith_summary":"This paper claims that a speech recognizer can be made better at transcribing named entities and numbers by training it on audio chunks that include five extra seconds of context on each side. The proposal, applied to the Whisper-Medium model, uses a 40-second input window while computing the prediction loss only on the central 30 seconds, so inference does not see longer chunks than usual. Entities that would be split by a window boundary are moved entirely into the right-hand chunk, and entity-type tags are embedded in the training text so the model learns both recognition and type-specific formatting. On the Spoken Wikipedia corpus, the windowed model improves named-entity F1 and numeric formatting over the same model trained on plain 30-second chunks. If the effect holds, long-form ASR and spoken-language understanding can gain semantic accuracy without a more expensive inference pipeline.","feed_headline":"Overlapping audio windows improve named-entity tagging in ASR","feed_subtitle":"Sliding 5 seconds of extra audio around each 30-second chunk improves entity recognition and formatting.","key_machinery":"The load-bearing mechanism is the 'effective semantic window': a 40-second training clip formed by sliding 5-second overlaps onto both sides of a 30-second target chunk. The model's encoder is extended with new positional-embedding rows, initialised to match the distribution of the original embeddings and warmed up on 100 hours of concatenated VoxPopuli audio, and the decoder is trained with a masked cross-entropy loss that counts only tokens in the central 30-second span, signalled by the special tokens <|left|>, <|mid|>, and <|right|>. Entities crossing a boundary are reassigned to the right-hand chunk, which teaches the model to defer a prediction until the full entity is visible. This combination allows longer context to inform each prediction without changing the inference-time chunking.","core_discovery":"The central discovery is that a fixed-length ASR model can be taught to use a wider semantic context at no inference cost by decoupling the input window from the prediction window. Whisper-puncted-windowed is trained on 40-second clips (left 5 seconds, middle 30 seconds, and right 5 seconds), with cross-entropy loss masked so only tokens between <|mid|> and <|right|> count, and with boundary-spanning entities shifted to the right-hand chunk. Compared with Whisper-puncted, a Whisper-Medium model fine-tuned on the same entity-tagged transcripts in plain 30-second chunks, this model reports higher named-entity recognition F1 for most entity types (e.g., PERSON 0.50 to 0.65, GPE 0.61 to 0.71, CARDINAL 0.95 to 0.98) and lower character error rates on numeric entities (e.g., MONEY 0.42 to 0.11, PERCENT 0.32 to 0.12, PHONE_NUM 0.35 to 0.15). The paper interprets these gains as evidence that the 10 extra seconds of overlapping context let the decoder recognize entity boundaries and formats that are ambiguous in isolation.","pith_inferences":["A testable extension would be to apply the same sliding-window masking to entity-aware speech translation, where the encoder context may benefit from future audio before a spoken entity is complete.","The paper leaves open whether the gains come from the overlapping context itself or from the extra 100-hour embedding warm-up; an ablation with the same 40-second inputs and the overlap regions replaced by silence would separate the two.","The boundary-reassignment rule could be generalised to any multi-token unit, such as dates or other numerical expressions, wherever the decoder needs to see the end of a unit before committing.","Because the method changes only the training data layout and loss mask, it could be combined with decoder prompting or language-model rescoring to further reduce residual punctuation errors inside entities."],"forward_implications":["If the reported gains are real, the same model can be deployed on long-form audio with its standard strided inference, so better entity recognition does not require a larger or slower recognizer.","The entity-reassignment rule should remove the characteristic error in which a monetary or numeric entity is split across chunks and mis-tagged as two entities.","Masking the loss to the central window turns the overlap region into free context for every training example, which could be applied to other sequence-to-sequence ASR models with fixed positional embeddings.","Embedding entity tags in the training transcript teaches the decoder to emit type-specific punctuation and casing, giving downstream text-based NER a cleaner input."],"supporting_citations":[{"why":"Provides the Whisper-Medium sequence-to-sequence model that both training variants fine-tune and extend.","marker":"Radford et al., 2022"},{"why":"Supplies the Spoken Wikipedia corpus with aligned audio, text, and timestamps used for training and evaluation.","marker":"Baumann et al., 2018"},{"why":"Supplies the 100 hours of VoxPopuli audio used to warm up the newly added positional embedding rows for 40-second inputs.","marker":"Wang et al., 2021"},{"why":"Supplies the F1 evaluation metric used for named entity recognition.","marker":"Nakayama, 2018"},{"why":"Establishes the entity-tag embedding approach in transcripts that the paper adapts to ASR.","marker":"Gaido et al., 2023"},{"why":"Provides the named-entity tagger used to produce the 18 entity labels for the reverse-normalisation step.","marker":"Honnibal et al., 2020"}],"fun_headline_variants":["Overlapping context windows improve ASR entity tagging","Extra audio context improves ASR named-entity accuracy","Sliding context windows enhance ASR entity recognition","5-second context overlap lifts ASR entity formatting","ASR entity recognition improves with overlapping context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumption that the extra positional-embedding rows, warmed up on only 100 hours of concatenated audio, genuinely enable the model to exploit the 10 extra seconds of audio rather than merely pass them through unprocessed.","fun_headline_variants_meta":{"raw":{"variants":["Overlapping context windows improve ASR entity tagging","Extra audio context improves ASR named-entity accuracy","Sliding context windows enhance ASR entity recognition","5-second context overlap lifts ASR entity formatting","ASR entity recognition improves with overlapping context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3700,"prompt_tokens":1002,"completion_tokens":2698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":2627}},"tokens_in":618,"tokens_out":2698,"duration_ms":20027,"temperature":1.0,"reasoning_tokens":2627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:57:10.665586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control model on the same 40-second inputs and masked central-window loss, but with the left and right 5-second overlap regions replaced by silence; if its entity F1 and numeric character error rate match Whisper-puncted-windowed, the reported gains come from longer input or the embedding warm-up, not from surrounding semantic context.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the F1 evaluation metric used for named entity recognition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the entity-tag embedding approach in transcripts that the paper adapts to ASR."}],"review_version":1}