{"id":"3b548740-7ae6-4a3c-b448-051de8a3f263","arxiv_id":"2506.14434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Dynamic right-context chunked attention masking lets a single zipformer ASR model cover streaming and non-streaming use, nearly closing the accuracy gap with a modest latency increase.","lead":"This paper trains one zipformer speech-recognition model that can be used both for streaming (audio processed as it arrives) and offline (full audio available) by allowing the model to look at a configurable number of future frames during decoding. Evaluations on LibriSpeech and a large in-house conversational dataset show the streaming version can nearly match the offline accuracy when given enough future context.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central result depends on an unspecified per-block allocation of right-context frames across zipformer's six frame rates; until the mask construction is pinned down, the Table 3 convergence to non-streaming cannot be independently verified.","rationale":"The reader's weakest assumption correctly identifies the unstated per-layer distribution of right-context frames. I agree that this is load-bearing: the central claim is not just that right context helps ASR, but that dynamic right-context training in zipformer yields a single model whose streaming WER converges to non-streaming WER, and the mechanism is claimed to depend on the multi-scale stack. Since each of the six blocks operates at a different frame rate, the same nominal count of right-context frames corresponds to different amounts of future audio unless the mask is designed per block. The paper does not give that design. This is not merely a formatting omission: a factor-of-16 ambiguity at the 6.25 Hz block changes the effective receptive field enough to alter the attention pattern in the deepest layers, which are the most downsampled and the largest-dimensional. The reported convergence in Tables 3 and 4 could in principle survive any reasonable allocation, but without the allocation the reader cannot tell; the ablation in Appendix A compares RC-0, RC-64, RC-128, and RC-0-64-128-256, but again does not report how frames are mapped across blocks. The paper has real strengths: it uses large in-house data, reports a server-client latency/accuracy tradeoff in Table 5, and includes a small-data tuning experiment; the tables are broadly internally consistent. Those strengths support a conditional acceptance, not a rejection. The concrete check—release the exact mask code and compare a temporal-context-matched allocation—would settle whether the multi-scale advantage is a genuine property of the architecture or of an unstated configuration.","tokens_in":997,"tokens_out":2033,"duration_ms":156094,"concrete_test":"Ask the authors to release the exact mask-generation routine, then compute the effective future-context duration in seconds at each of the six encoder rates for RC=64 input frames. Re-run the Table 3 LargeRC-64 evaluation under a matched temporal allocation in which every block receives the same 0.64 s future horizon; if the average WER shifts by more than about 0.2 WER or the gap to LargeNS (8.26%) widens materially, the central convergence claim is configuration-dependent rather than robust to the unspecified detail.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that a chosen right-context value (0/64/128/256 frames, stated as 10 ms per frame) is translated into attention masks in a well-defined way inside zipformer's six encoder blocks at 50/25/12.5/6.25/12.5/25 Hz. The paper never specifies this mapping. At the 6.25 Hz block, '64 frames' could mean 0.64 s of future audio if counted at input rate, or 10.24 s if counted at that block's rate—a 16x difference. The claimed multi-scale advantage (Section 2.3, Figure 3) and the central convergence result (Table 3: 8.31% at RC=64 vs. 8.26% non-streaming; Table 4: 8.2% vs. 8.2%) depend on this allocation. The small-data tuning in Appendix A does not report the allocation either. If the mask is applied with equal input-frame counts at every block, low-rate blocks receive a much shorter future horizon in seconds than high-rate blocks; if it is scaled per block, the opposite can occur. Without this detail, the result cannot be re-implemented or independently checked, so it is unclear whether the reported convergence is a property of the proposed algorithm or of an unstated mask configuration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes training a single Zipformer-based ASR model with chunked attention masking that dynamically incorporates right-context frames, sampled per training batch from {0, 64, 128, 256}, so that one model can serve both streaming and non-streaming use cases. At inference the user chooses the number of decoding right-context frames to trade accuracy against latency. The authors compare their right-context-trained model with a no-right-context streaming baseline and a non-streaming model on LibriSpeech and on a 12,468-hour in-house conversational corpus, using both simulated streaming and a production server-client setup. The main reported result is that the right-context streaming model approaches non-streaming accuracy (e.g., Table 4 average WER 8.2% at 64 decoding right-context frames versus 8.2% for the non-streaming model), with only a small latency penalty.","tokens_in":11542,"tokens_out":5690,"duration_ms":54167,"significance":"If the central result is reproducible, the paper offers a practical low-cost method for unifying streaming and non-streaming ASR in production: a training-time masking modification with no extra parameters, plus a tunable inference-time latency-accuracy knob. The evaluation is industrially strong, using a large in-house corpus, diverse domain and accent test sets, and concurrent server-client latency/RTFX measurements. The small-data tuning study in Appendix A is a useful ablation. The main weakness is that the proposed masking is not specified precisely enough for independent implementation, and the headline 7.9% relative improvement is not directly supported by the reported tables.","major_comments":[{"comment":"The paper never specifies how a chosen number of right-context frames is allocated across the six Zipformer encoder blocks, which operate at frame rates of 50, 25, 12.5, 6.25, 12.5, and 25 Hz. The description of the attention mask in Equation (2) and Figure 2 is conceptual; it does not state whether the right-context frame count is applied at the input frame rate, at each block's own frame rate, or via some per-block scaling. This matters because a frame at the 50 Hz block corresponds to 20 ms of audio, while a frame at the 6.25 Hz block corresponds to 160 ms. The ambiguity is compounded by the statement in Section 3.2.3 that 'each frame corresponds to 10 ms in both training and decoding' and by Appendix A, which says RC-64 corresponds to 1.28 seconds of context (implying 20 ms per frame). Depending on the interpretation, the future acoustic horizon seen by low-rate blocks can differ by a factor of 8 or more from that seen by high-rate blocks. Because the reported convergence of the streaming model to non-streaming WER (Table 3: 8.31% at RC=64 versus 8.26% non-streaming) depends on this allocation, the result cannot be independently verified or reimplemented without the missing detail.","section":"Section 2.3 and Section 3.2.3"},{"comment":"The abstract claims a 'relative 7.9%' word error reduction, but that number is not directly reproducible from the reported results. From Table 3, the average WER of LargeRC-0-64-128-256 with 64 decoding right-context frames is 8.31% versus 10.34% for the baseline at 0 right-context frames, a relative reduction of 19.6%. From Table 4, the same model improves from 9.0% to 8.2%, a relative reduction of 8.9%. The 7.9% figure does not correspond to any comparison shown in the paper, and the authors should either point to the exact source of this number or correct the abstract.","section":"Abstract and Tables 3-4"},{"comment":"The RTFX values in Table 5 show no variation with the number of decoding right-context frames (e.g., 82.65 and 82.66 at concurrency 100 for RC=0 and RC=64) even though the reported final-chunk latency increases from 1.41 s to 1.47 s over the same range. Since RTFX is defined in Section 3.3 as duration divided by inference time, an increase in per-request inference time should reduce RTFX unless the metric is aggregated in some concurrency-dependent way that is not described. The metric definition and the numbers in Table 5 need to be reconciled.","section":"Table 5"}],"minor_comments":[{"comment":"The horizontal axis label contains a typo: 'No of right-context frame' should be 'Number of right-context frames'.","section":"Figure 3"},{"comment":"The text says the Libri Baseline WER decreases from 3.33% to 2.83% as decoding right-context increases from 0 to 256, but Table 2 lists 2.81% at 256 right-context frames; please correct the inconsistency.","section":"Section 4.1, Table 2"},{"comment":"The training setup says chunk size varies among [16, 32, 64] frames, while evaluation uses a chunk size of 32 frames; the paper should clarify whether the model was trained with all three chunk sizes and whether the server-client evaluation also fixed the chunk size or used the 500 ms client stream as the chunk boundary.","section":"Section 3.2.3"},{"comment":"The appendix states that all models trained with right-context outperform the baseline, but the figure labels (Figure 4) are cramped and hard to read; providing exact WER values in a table or in the text would improve reproducibility.","section":"Appendix A"},{"comment":"The abstract says 'reduces word error by relative 7.9%' but the standard term is 'word error rate' (WER); please use the full term.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system paper without code release. The central concern is reproducibility: the per-block right-context allocation in the Zipformer encoder must be specified before the claimed convergence can be assessed as a property of the proposed algorithm rather than of an unstated implementation choice. The headline 7.9% figure should also be reconciled with Tables 3 and 4. I would be inclined to accept after these points are addressed with a precise masking description and corrected reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper delivers a credible recipe for training one zipformer that can serve both streaming and non-streaming ASR by randomly varying right-context during training and choosing it at decode time. The direction of the result is supported by the tables; the biggest problem is that the paper never specifies how a right-context count becomes attention masks across zipformer's six blocks, which makes the central claim hard to verify independently.\n\nWhat's new: prior variable-attention-masking work has been on conformer/transformer transducers. Applying the idea to zipformer and studying the accuracy-latency trade-off at production scale (12.5k hours, server-client, concurrency up to 300) is a genuine contribution. The small-data tuning in Appendix A is the right way to pick the RC set, and the latency/RTFX table is useful.\n\nSoft spots: the missing per-block allocation is load-bearing, not minor. Zipformer stacks run at 50/25/12.5/6.25/12.5/25 Hz. Whether '64 right-context frames' means 64 frames at every block's rate or 64 input-rate frames changes the future horizon by 16x at the lowest block. The paper's claim that zipformer benefits more than conformer 'due to its multi-scale nature' is not established because Figure 3 compares different models trained with different recipes, not a controlled ablation. Also, the abstract's 7.9% relative improvement does not appear in Tables 3 or 4 (Table 4 gives 8.9%, Table 3 gives ~19.6% depending on comparison), so the headline number is sloppy. No code or data is released, which makes the missing mask spec harder to fill in.\n\nThat said, the core empirical phenomenon seems real: in Table 3 the RC model at 64 decoding frames averages 8.31% versus 8.26% for the non-streaming model, and in Table 4 both hit 8.2%. Those numbers are close enough that the method likely does what it claims, but the paper as written leaves the algorithm underspecified.\n\nWho this is for: ASR practitioners maintaining separate streaming/offline models, and researchers working on unified streaming models. It deserves peer review. A serious referee should ask for the mask construction, a block-wise scaling ablation, and a consistent headline statistic. I'd send it to review; with those details fixed it could be a solid conference paper.","headline":"A practical unified streaming/non-streaming zipformer recipe whose central mask detail is left unspecified; the convergence result looks real but needs a precise mask spec to be verified.","tokens_in":12082,"tokens_out":4281,"would_cite":true,"duration_ms":40665,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single zipformer ASR model trained with dynamic right-context chunked masking can serve both streaming and non-streaming use; at 64 future frames it reaches 8.31% WER versus 8.26% for a non-streaming model.","keywords":["automatic speech recognition","streaming ASR","non-streaming ASR","zipformer","chunked attention masking","right context","latency-accuracy tradeoff","word error rate"],"falsifier":"Re-run the small-data training experiment with two different per-layer allocations of the right-context frames, for example assigning all right-context only to the 50 Hz encoder blocks versus only to the 6.25 Hz block, and compare WER for each decoding right-context setting; the central claim requires the dynamic right-context model to keep its improvement under both allocations, and this would reveal whether the missing per-layer allocation detail is load-bearing.","tokens_in":11075,"feed_emoji":"🎙️","tokens_out":13804,"duration_ms":111349,"temperature":0.7,"pith_summary":"There is a long-standing trade-off in speech recognition: streaming models read audio as it arrives and are less accurate, while offline models wait for the whole utterance and are more accurate, so products typically train and serve both. This paper tries to establish that a single zipformer-based recognizer can cover both modes by training with a chunked attention mask that randomly grants the encoder 0, 64, 128, or 256 future frames per batch. At inference the operator chooses how many future frames to allow, trading a little latency for accuracy. On a 12,468-hour conversational corpus, the large model with 64 decoding right-context frames averages 8.31% WER versus 8.26% for the non-streaming model, and in a server-client setup both average 8.2%; the paper reports a 7.9% relative WER reduction over the no-right-context baseline. The paper also argues that the zipformer's multi-scale encoder makes right-context far more effective than in a conformer, which is why the unification works.","feed_headline":"Training with future audio closes the streaming accuracy gap","feed_subtitle":"With 64 extra frames of future audio, streaming WER hits 8.2%, equal to the offline model.","key_machinery":"The central object is a binary attention mask $M$ applied inside self-attention: $\\mathrm{softmax}(M(QK^\\top)/\\sqrt{d})V^\\top$. The mask selects, for each query frame, which key and value frames are visible. The paper changes $M$ so that, beyond the usual left-context and chunk boundaries, a variable number of future frames is visible; during training that number is randomly chosen per mini-batch from {0, 64, 128, 256}, and during decoding it is set freely. The zipformer, a multi-scale transformer encoder whose six blocks run at frame rates from 50 Hz to 6.25 Hz, is what makes this mask effective: the right-context reaches across several temporal resolutions, so fewer future frames carry more information. The mask is the mechanism that lets one set of weights interpolate between causal streaming behavior and near-offline behavior without retraining.","core_discovery":"The central claim is that a single zipformer-based speech recognizer, trained with dynamic right-context chunked attention masking, can serve both streaming and non-streaming applications, and that the streaming accuracy gap closes as the decoding right-context grows. On a 12,468-hour conversational corpus, the model trained with right-context randomly drawn from {0, 64, 128, 256} frames per batch averages 8.31% WER at 64 decoding right-context frames, versus 8.26% for the non-streaming model; in a production server-client setup, both average 8.2%. The paper further claims that the effect is stronger for zipformer than for conformer because zipformer's six encoder blocks operate at different frame rates, from 50 Hz down to 6.25 Hz and back, so future context is exploited more efficiently.","pith_inferences":["If the convergence is real, it suggests the streaming/non-streaming WER gap for transducer models is mostly a missing-future-context problem, not a fundamental limitation of causal architectures; other multi-scale causal encoders could adopt the same trick.","The per-batch randomization of right-context may act as a form of context dropout, making the model robust across latency regimes; a natural extension is to sample right-context from a continuous distribution or tie it to chunk size during training.","A testable extension is to apply the same dynamic right-context mask in a decoder-only or attention-decoder ASR model; if the benefit transfers, the method is not specific to zipformer or RNN-T.","The paper's conformer comparison implies the advantage comes from multi-scale frame rates; adding a downsampling/upsampling path to a conformer might close the gap, which would isolate the mechanism."],"forward_implications":["A deployment can be switched from streaming to near-offline accuracy by raising the decoding right-context, without retraining, so one model artifact serves both modes.","Training cost and maintenance drop because the separate offline model can be retired; the paper reports the streaming model at 256 right-context frames closely approaching its non-streaming counterpart on LibriSpeech (2.43% versus 2.38% test-clean).","The latency-accuracy knob is continuous in inference: at concurrency 100, final-chunk latency rises only from 1.41 to 1.47 seconds when the decoding right-context goes from 0 to 64 frames, with RTFX essentially unchanged.","Multi-scale encoders such as zipformer, rather than fixed-rate conformers, are the better substrate for unified streaming and non-streaming ASR, because they convert right-context into larger WER gains per frame."],"supporting_citations":[{"why":"Supplies the zipformer encoder, its six-block multi-scale frame-rate design, and the medium/large training recipes on which all the paper's models are built.","marker":"(Yao et al., 2023)"},{"why":"Defines the conformer architecture used as the main non-zipformer baseline and supplies the attention masking context the paper modifies.","marker":"(Gulati et al., 2020)"},{"why":"Introduced the chunked attention masking that confines each frame's receptive field during self-attention, which the right-context mask extends.","marker":"(Shi et al., 2021)"},{"why":"Also cited for chunked attention masking in real-time streaming transformer transducers, supporting the masked training setup.","marker":"(Chen et al., 2021)"},{"why":"Establishes variable attention masking for transformer transducers; the paper contrasts its fixed right-context range and streaming setup against this line of work.","marker":"(Swietojanski et al., 2023)"},{"why":"Presents a prior unified streaming and non-streaming conformer approach with dynamic chunk convolution, the main alternative the paper distinguishes itself from.","marker":"(Li et al., 2023)"},{"why":"Earlier one-model streaming and non-streaming transformer transducer with layer-dependent context, used as a point of difference for the per-batch right-context strategy.","marker":"(Tripathi et al., 2020)"},{"why":"Provides the pruned RNN-T loss used to train the conformer and zipformer transducer models in the experiments.","marker":"(Kuang et al., 2022)"}],"fun_headline_variants":["One ASR model, both streaming and offline, with future frames","Dynamic right-context closes the streaming ASR gap","Zipformer unifies streaming and offline speech recognition","Future audio lets streaming ASR match offline accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the variable right-context frames are distributed across the six zipformer encoder blocks in a way that realizes the multi-scale benefit, but the paper never specifies that per-layer distribution; if a different distribution changes the WER gains, the central claim does not stand as stated.","fun_headline_variants_meta":{"raw":{"variants":["One ASR model, both streaming and offline, with future frames","Dynamic right-context closes the streaming ASR gap","Zipformer unifies streaming and offline speech recognition","Future audio lets streaming ASR match offline accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2590,"prompt_tokens":929,"completion_tokens":1661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1597}},"tokens_in":545,"tokens_out":1661,"duration_ms":11513,"temperature":1.0,"reasoning_tokens":1597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:52:13.337443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the small-data training experiment with two different per-layer allocations of the right-context frames, for example assigning all right-context only to the 50 Hz encoder blocks versus only to the 6.25 Hz block, and compare WER for each decoding right-context setting; the central claim requires the dynamic right-context model to keep its improvement under both allocations, and this would reveal whether the missing per-layer allocation detail is load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the chunked attention masking that confines each frame's receptive field during self-attention, which the right-context mask extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes variable attention masking for transformer transducers; the paper contrasts its fixed right-context range and streaming setup against this line of work."}],"review_version":1}