{"id":"5b763745-9548-409c-97cc-c74d6831ca98","arxiv_id":"2507.18263","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Locate-and-Focus localizes the audio span of a terminology in an utterance and uses the located clip, a matched audio replacement, and a special <Term> cue to make speech LLMs translate the terminology correctly.","lead":"This paper introduces Locate-and-Focus, a method that improves how speech translation models handle rare terms by first finding the audio clip where a term is spoken and then feeding the model the term's translation alongside that clip. It reports large gains in term translation success on English-Chinese and English-German benchmarks while keeping overall translation quality roughly unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The end-to-end TSR gain is measured with a retrieval pool that is guaranteed to contain every test terminology, so the result demonstrates oracle-dictionary performance rather than robustness to missing KB entries; a held-out-term evaluation is needed.","rationale":"The paper is a careful empirical study with several genuinely supporting pieces: the Sliding Retrieval comparison against pooling baselines (Table 3 and Appendix B), ablations isolating Audio Replacement and Tag Cue, human verification of located clips (88-94%), and a latency measurement showing that retrieval overhead is small. I read the central claim charitably: given a supplied terminology KB, Locate-and-Focus improves TSR and roughly preserves BLEU. The weakest point in that argument is that the KB is not merely supplied; it is constructed from the exact test transcripts used for evaluation. This makes the end-to-end setting a closed-vocabulary oracle retrieval task: Hits@N and TSR are computed on a pool that contains the correct answer for every evaluation term. The method's two mechanisms (localization and knowledge association) are tested only under coverage=1.0. If coverage is lower, retrieval cannot supply the knowledge, and the locate step has nothing to locate; the paper's own Limitations section concedes this dependence, but no experiment measures the slope. This is a generalization and correctness-risk concern rather than an internal inconsistency, and it does not require assuming ill intent. The reader's CONDITIONAL verdict already captures the need for artifacts and a more rigorous protocol; I would keep the verdict conditional and add this held-out-terminology experiment to the acceptance criteria.","tokens_in":18542,"tokens_out":9502,"duration_ms":110456,"concrete_test":"Run a held-out-terminology evaluation on the released data: split the test terms into in-KB and out-of-KB sets, remove the out-of-KB triplets from the retrieval pool, and compare TSR on both sets. Also rerun MuST-C and MSLT end-to-end with a pool built only from CoVoST2 training splits, so that the test terms are not guaranteed to be present. If out-of-KB TSR drops to the no-context baseline (roughly 25-35 in Table 2), the reported gains are contingent on complete dictionary coverage; if it degrades gracefully, the closed-vocabulary concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is not any single equation but the evaluation protocol that makes the retrieval problem closed-vocabulary by construction. In Section 4, the knowledge base P is built from the same transcripts used for evaluation: for CoVoST2, from both train and test splits; for MuST-C and MSLT, from the test split alone (Table 1). The method then retrieves from this pool and reports Hits@N and TSR on the same items. Thus every terminology that must be translated is guaranteed to have a correct triplet (x, c, y) with TTS audio and golden translation in the pool at inference. This means the reported end-to-end TSR (65.5-77.4 across settings) is a conditional success rate given that the dictionary contains the answer, not a success rate under incomplete or incorrect knowledge. If a term is absent from the KB, there is no candidate to retrieve and no knowledge to focus on; the authors acknowledge this dependence in the Limitations section, but no experiment quantifies the degradation. Because the central contribution is clip-level localization plus knowledge association, the externally valid claim depends on KB coverage, and the paper currently supplies only the coverage=1.0 point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Locate-and-Focus, a method for terminology translation in speech-to-text LLMs. The method first runs a sliding-window audio retriever over the input utterance to locate speech clips that contain terminology and to retrieve matching knowledge triplets (term transcript, TTS audio, translation) from a knowledge base. It then feeds the utterance together with the retrieved triplets to a speech LLM, replacing the TTS audio in each triplet with the located utterance clip (Audio Replacement) and training the model to emit a <Term> tag before translating a term (Tag Cue). The authors build a terminology-translation dataset from CoVoST2, MuST-C, and MSLT with LLM-assisted extraction and manual verification, covering English-to-Chinese and English-to-German. Experiments in oracle and end-to-end knowledge settings report large Term Success Rate (TSR) gains over SALM and Retrieval-and-Demonstration baselines, with comparable BLEU, and ablations indicate that each component contributes. The paper also reports retrieval Hits@N, a manual evaluation of located clips, inference latency, and general-domain BLEU on standard CoVoST2.","tokens_in":18740,"tokens_out":6426,"duration_ms":70170,"significance":"If the results hold, the paper offers a practical and reasonably novel approach to terminology translation in speech LLMs. The sliding-window clip-level retrieval is a sensible way to localize terminology in the audio stream, and the audio-replacement idea is an elegant mechanism for making dictionary knowledge acoustically consistent with the utterance. The paper is thorough in its ablations, model-size scaling, latency measurements, and manual verification of located clips. The main empirical claim, however, is currently demonstrated under a closed-vocabulary evaluation protocol: the retrieval pool is constructed from the same test transcripts used for evaluation, so every target term is guaranteed to have a correct triplet in the pool. The reported end-to-end TSR is therefore a conditional success rate at dictionary coverage equal to 1.0, and the paper does not quantify how performance degrades when the knowledge base is incomplete or contains incorrect entries. Because the method's central value proposition depends on the knowledge base, this external-validity gap is load-bearing and needs to be addressed with additional experiments.","major_comments":[{"comment":"The end-to-end evaluation is closed-vocabulary by construction. For MuST-C and MSLT, the retrieval pool P is built from the test split itself; for CoVoST2, from train and test splits. Consequently, every terminology that must be translated is guaranteed to appear in P with a correct transcript, TTS audio, and golden translation at inference time. The reported end-to-end TSR (65.5-77.4 across settings) is a conditional success rate given full KB coverage, not a measure of robustness under incomplete or incorrect knowledge. The Limitations section acknowledges the reliance on predefined terminologies, but no experiment quantifies the degradation when a term is absent from the KB. Please add a held-out-term evaluation or a KB-coverage sweep (e.g., randomly dropping 10%, 25%, 50% of test terms from the pool) and report the resulting TSR and BLEU.","section":"Section 4, Table 1; Section 5.1"},{"comment":"The ablation claims are not supported by variance estimates or significance tests. For example, in the CoVoST2 English-to-Chinese end-to-end setting, full Locate-and-Focus achieves 65.53 TSR, w/o Audio Replacement 63.49, w/o Tag Cue 63.73, and w/o Replacement and Cue 62.95; the oracle-setting gaps are similarly small (e.g., 90.13 vs 89.67 vs 89.00). With single runs and no error bars, these differences may be within noise. Please report results over multiple random seeds with standard deviations, or at least provide bootstrap confidence intervals and a paired significance test for the key comparisons in Table 2.","section":"Section 5.1, Table 2"},{"comment":"The Oracle Knowledge Setting for the SALM baseline is under-specified. The paper states that SALM is provided with 'the translations of the top 50 most frequent terms,' but it does not state whether term frequency is computed on the training split, the test split, or the retrieval pool. If the frequency list is computed on the test split, SALM's oracle performance reflects test-set statistics and gives the baseline an unfair advantage; if computed on training data, the setup is not truly 'ground truth knowledge without irrelevant noise.' Please specify the exact construction of the frequency list in both the oracle and end-to-end settings.","section":"Section 5, Setups"},{"comment":"There is an inconsistency in how the located clip s is extracted. Equation (1) identifies the subsequence of length |c| with maximum pooled-cosine similarity, which would make s exactly as long as the KB clip c. Appendix A.1, however, states that 'we focus on the hidden state with the highest similarity' and segment 'the relevant portion,' suggesting a frame-level selection rather than a full-window selection. Because Audio Replacement and the reported clip-localization accuracy in Section 5.2 depend on the exact segmentation rule, please clarify the precise algorithm used to produce s.","section":"Section 3.1, Eq. (1); Appendix A.1"}],"minor_comments":[{"comment":"It is not stated whether the standard CoVoST2 evaluation is performed with or without retrieval knowledge and whether <Term> tags are stripped before computing BLEU. Please clarify the inference protocol and the post-processing of tag tokens.","section":"Section 5.4, Table 5"},{"comment":"The Term Success Rate metric is defined only as 'the proportion of terminologies accurately translated within an utterance.' Please specify the matching criterion (exact substring match with the reference translation, normalized match, etc.) and how the <Term> tag is handled in TSR computation, especially for inflected target languages like German.","section":"Section 5, Metrics"},{"comment":"The manual annotation process is described as requiring three-expert agreement, but no inter-annotator agreement statistic (e.g., Cohen's kappa) is reported. Please add agreement numbers to support the claim of high annotation quality.","section":"Section 4, D.1"},{"comment":"The table header and text use 'BELU' instead of 'BLEU'; please fix this typo.","section":"Appendix B.2, Table 10"},{"comment":"The instruction template for SALM includes an 'Audio: <audio>...</audio>' field even though SALM is described as a text-only term-list method. Please ensure the template matches the actual input format used in the SALM experiments.","section":"Appendix D.2"},{"comment":"The sentence 'we are the first end-to-end terminology translation method that retrieves and fully utilizes multi-modal fined-granularity multi-modal fine-grained knowledge' contains a repeated and malformed phrase ('fined-granularity multi-modal fine-grained'); please rephrase.","section":"Section 3, last paragraph"}],"recommendation":"major_revision","confidential_remarks":"The main risk is external validity rather than internal correctness. The closed-vocabulary evaluation is the single most important issue: the authors should either add a held-out-terms experiment or explicitly reframe the contribution as 'oracle-dictionary performance' and study sensitivity to KB coverage. If they also add variance estimates and clarify the SALM oracle setup, the paper would be a solid contribution. I do not see a fundamental flaw in the method itself, so rejection is not warranted at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read of arXiv:2507.18263. The paper has a real contribution: a sliding-window, clip-level retriever for terminology localization in speech translation, plus audio replacement and a learned <Term> tag. The ablations are clean and the gains are large: end-to-end TSR on CoVoST2 EN-ZH goes from 32.93 (Retrieve-and-Demonstrate) to 65.53, with each component contributing. The dataset construction, with LLM extraction, TTS, ASR filtering, and human review, is careful. The human check that located clips actually contain the terminology, with 88-94% success, is good evidence the localization works. Latency numbers are reassuring. This is a useful recipe.\n\nNow the soft spots, in order of importance.\n\nThe evaluation is closed-vocabulary. The knowledge base pool is built from the same test transcripts and TTS clips that are being retrieved (Table 1). Every target term is guaranteed a correct triplet in the pool. So the reported TSR is a conditional success rate given that the dictionary contains the answer, not a rate under incomplete or noisy knowledge. The stress-test note is right about this. The Limitations section acknowledges reliance on predefined terminologies, but no experiment quantifies degradation when a term is missing. That limits the external validity of the central claim.\n\nSecond, there are no error bars or significance tests anywhere. TSR differences of a few points between ablations could be noise. The oracle setting also leaves ambiguity: SALM is given the top-50 frequent terms, but it's not clear how that budget was chosen, and retrieval methods get top-5. That's a fair comparison only if the budgets are matched.\n\nThird, the \"first end-to-end terminology translation method that retrieves multi-modal fine-grained knowledge\" is too strong given the Retrieve-and-Demonstrate baseline already does end-to-end retrieval. The novelty is real, but it's incremental over that line.\n\nWhat holds up: sliding retrieval versus pooling is a clear win across model sizes and datasets. The ablations support the design. The central mechanism is plausible.\n\nThis paper deserves a serious referee. It's a solid empirical contribution for people working on speech LLMs and terminology-aware ST. The main thing a referee should push for is a held-out-term evaluation, where the KB is built from training data only, or at least a sensitivity analysis over KB coverage. Also, code and data are promised, not delivered; that matters because the dataset is self-built.\n\nMy recommendation: send it to peer review, but require the held-out-term experiment and proper statistics before acceptance.","headline":"A genuinely useful recipe for clip-level terminology retrieval in speech translation, with the caveat that the reported gains assume the dictionary already contains every term.","tokens_in":19329,"tokens_out":3165,"would_cite":true,"duration_ms":30410,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Direct speech translation misses terms; Locate-and-Focus finds them first, then translates them.","keywords":["terminology translation","speech translation","speech language models","retrieval-augmented translation","speech clip localization","sliding window retrieval","audio replacement","term success rate"],"falsifier":"Take the end-to-end setup and remove one term from the retrieval pool while keeping the rest; if Term Success Rate does not fall for that term substantially more than for pool-covered terms, the gain is not attributable to retrieval. Alternatively, replace the pool's synthetic audio with a different speaker's pronunciation of the same term but keep the located clip replaced: if the method's advantage over the audio-replacement ablation disappears, the benefit is specifically the acoustic match, not the presence of the transcript and translation.","tokens_in":18301,"feed_emoji":"🎙️","tokens_out":7071,"duration_ms":63316,"temperature":0.7,"pith_summary":"Direct speech translation often mangles names, places, and domain terms, and existing knowledge-injection methods drown the model in irrelevant context or mismatched speakers. Locate-and-Focus claims to fix this in two moves: it first uses a sliding-window speech retriever to find the exact audio clip where the term is spoken, then feeds the model a knowledge triplet in which that located clip replaces the dictionary's synthetic audio, so the utterance and the knowledge share acoustics. A special <Term> tag is inserted before the term's translation during training, acting as a self-reminder to consult the triplet. On CoVoST2, MuST-C, and MSLT, the method raises Term Success Rate from roughly 24–53% to 65–77% in the end-to-end setting while keeping BLEU comparable to translation-training baselines.","feed_headline":"Locating a term's audio clip lifts speech translation accuracy","feed_subtitle":"Sliding-window retrieval plus an audio anchor raises term success on three datasets while keeping overall BLEU.","key_machinery":"The load-bearing object is the translation knowledge triplet $K=(x,c,y)$—term transcript, term speech clip, term translation—and the operation that reshapes it. Sliding Retrieval encodes the clip $c$ and utterance $u$ with the same speech encoder, then takes $\\max_i \\text{Cosine}(\\text{MaxPool}(z_c), \\text{MaxPool}(z_u^i))$ over windows $z_u^i$ of length $|c|$, which both scores the term's presence and returns the located subsequence $s$. Audio Replacement swaps $c$ for $s$ to form $K'=(x,s,y)$, giving utterance and knowledge identical acoustics; Tag Cue trains the model to predict <Term> before each terminology translation. The contrastive loss on the speech encoder and the next-token loss on the speech LLM (via LoRA) train the two steps sequentially.","core_discovery":"The paper establishes that clip-level localization is the missing step in knowledge-augmented direct speech translation. Given an utterance and a pool of term triplets (transcript, speech clip, translation), a max-pooled cosine similarity between the utterance embedding and each short clip—computed over every sliding window of the clip's length—both identifies the terms present and pinpoints the acoustic span containing them. Substituting that located span for the pool's synthetic clip creates an acoustic anchor shared by the utterance and the injected knowledge, and training the model to emit <Term> before translating the term lets it attend to that knowledge at the right moment. In the oracle setting the method reaches 86–96% Term Success Rate across six dataset–direction pairs; in the end-to-end setting it outperforms both the collect-and-integrate and retrieve-and-demonstrate baselines by large margins (e.g., 65.5 vs. 28.2 and 32.9 TSR on CoVoST2 EN–ZH), with BLEU gains rather than losses on most pairs.","pith_inferences":["Not claimed in the paper: the same locate-then-focus recipe could transfer to other span-level speech tasks such as hotword customization in ASR or keyword spotting, where the acoustic anchor can replace the dictionary audio.","This goes beyond the paper: the method's reliance on a complete, acoustically matched knowledge base is a boundary condition—if the pool is built from the test transcripts themselves, in-the-wild gains will shrink whenever a term is missing or pronounced differently; a testable extension is to measure TSR when the pool is contaminated with distractor terms.","The top-5 sweet spot suggests an optimizable tradeoff between recall and noise; a dynamic similarity threshold, rather than a fixed $k$, could further improve end-to-end TSR.","The <Term> tag effectively teaches the model when to consult the retrieved knowledge—an attention-control mechanism that could be reused in other retrieval-augmented generation settings."],"forward_implications":["In end-to-end use, Term Success Rate rises to 65–77% across three datasets (from 24–53% at baseline), and the method beats both the collect-and-integrate and retrieve-and-demonstrate baselines on TSR in every tested setting.","General translation quality does not suffer: BLEU on the standard CoVoST2 test set (43.48 EN–ZH) is close to translation-training (43.64) and above the other knowledge-based methods.","Clip localization is robust across encoder scales: Sliding Retrieval with Whisper-base, -small, and -medium consistently outperforms pooling-based retrieval, reaching about 60% Hits@1 and 85% Hits@10 on CoVoST2 with the medium encoder.","Each component contributes: removing Audio Replacement and Tag Cue together drops oracle TSR (e.g., 86.14 vs. 94.09 on MuST-C EN–ZH), and removing Sliding Retrieval degrades end-to-end TSR.","Retrieval adds negligible latency (0.217 ms vs. 621.951 ms for translation itself), so the method is compatible with real-time constraints."],"supporting_citations":[{"why":"SALM baseline: supplies the Collect-and-Integrate comparison that Locate-and-Focus must beat on TSR and BLEU.","marker":"Chen et al., 2024"},{"why":"Retrieval-and-Demonstration baseline: retrieves utterance-translation pairs via in-context learning, and provides the observation that same-modality retrieval outperforms cross-modal retrieval.","marker":"Li et al., 2024a"},{"why":"CoVoST2: supplies the training split and in-domain test set used for both retriever and speech LLM training.","marker":"Wang et al., 2020"},{"why":"MuST-C: out-of-domain test set measuring whether terminology gains generalize beyond the training distribution.","marker":"Cattoni et al., 2021"},{"why":"MSLT: the other out-of-domain test set for English-to-Chinese and English-to-German.","marker":"Federmann and Lewis, 2016, 2017"},{"why":"Whisper-medium: the speech encoder whose hidden states drive both Sliding Retrieval and the located clip extraction.","marker":"Radford et al., 2023"},{"why":"Qwen2-Audio-Instruct: the speech LLM that is fine-tuned with LoRA and performs the actual translation.","marker":"Chu et al., 2024"},{"why":"LoRA: the parameter-efficient fine-tuning method used to train the speech LLM.","marker":"Hu et al., 2022"},{"why":"Term Success Rate: the metric that defines the paper's primary measure of terminology translation quality.","marker":"Semenov et al., 2023"}],"fun_headline_variants":["Pinpointing term audio clips boosts speech translation accuracy","Audio anchoring lifts terminology success in speech translation","Sliding-window clip retrieval sharpens speech translation terms","Term clip localization powers better speech translation","Locating terms in audio improves translation fidelity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The knowledge base must already contain the term with a correct transcript, a usable audio clip, and a correct translation, and the evaluation pool is built from the very transcripts being tested, so the retriever never faces genuinely out-of-vocabulary terms.","fun_headline_variants_meta":{"raw":{"variants":["Pinpointing term audio clips boosts speech translation accuracy","Audio anchoring lifts terminology success in speech translation","Sliding-window clip retrieval sharpens speech translation terms","Term clip localization powers better speech translation","Locating terms in audio improves translation fidelity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000155,"raw_usage":{"total_tokens":1193,"prompt_tokens":905,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":218}},"tokens_in":521,"tokens_out":288,"duration_ms":3864,"temperature":1.0,"reasoning_tokens":218,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:16:38.945252+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the end-to-end setup and remove one term from the retrieval pool while keeping the rest; if Term Success Rate does not fall for that term substantially more than for pool-covered terms, the gain is not attributable to retrieval. Alternatively, replace the pool's synthetic audio with a different speaker's pronunciation of the same term but keep the located clip replaced: if the method's advantage over the audio-replacement ablation disappears, the benefit is specifically the acoustic match, not the presence of the transcript and translation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MSLT: the other out-of-domain test set for English-to-Chinese and English-to-German."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Term Success Rate: the metric that defines the paper's primary measure of terminology translation quality."}],"review_version":1}