{"id":"932cd22f-8aaa-43b2-aba4-6f32e07fa535","arxiv_id":"2606.13507","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"An audio-LLM is trained with Rank-to-Distill pseudo-labels to filter speech pairs, improving end-to-end S2ST performance over unfiltered baselines on CVSS-C and SpeechMatrix.","lead":"The paper describes training an audio large language model to decide which noisy speech pairs to keep or drop for speech-to-speech translation training. A two-stage Rank-to-Distill process generates pseudo-labels from a lightweight ranker and then distills them into the audio-LLM, leading to reported gains of up to 1.4 ASR-BLEU on two datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Pseudo-label quality from the lightweight ranker is unverified and could embed systematic bias into audio-LLM supervision","rationale":"The reader's weakest_assumption matches the load-bearing step exactly. Because the supplied review was abstract-only, the absence of any direct validation of the ranker labels is the clearest internal risk; downstream gains alone do not rule out bias propagation. No other technical inconsistency is visible from the given material.","tokens_in":1633,"tokens_out":383,"duration_ms":14443,"concrete_test":"On a 200-pair held-out subset of CVSS-C, collect human keep/drop annotations; compute precision, recall, and F1 for both the lightweight ranker and the trained audio-LLM against these labels. If the audio-LLM F1 does not exceed the ranker F1 by >0.10 or if both are <0.65, the pseudo-label supervision is too noisy to support the headline gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the Rank-to-Distill pipeline: a lightweight ranker first produces keep/drop pseudo-labels from noisy speech pairs, then these labels supervise an audio-LLM that is later used for final filtering. For the reported +1.4 ASR-BLEU gains on CVSS-C and SpeechMatrix to be attributable to improved selection rather than other factors, the ranker labels must be sufficiently accurate and unbiased. No evidence is supplied that the ranker itself correlates with human judgments, with semantic consistency metrics, or with downstream S2ST performance when used directly. If the ranker systematically errs (e.g., on prosody, speaker overlap, or cross-lingual misalignment), those errors are distilled into the audio-LLM, so the final model cannot be said to “jointly capture acoustic fidelity and cross-lingual semantic consistency.”","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a Rank-to-Distill pipeline for filtering noisy data in speech-to-speech translation (S2ST) training. A lightweight ranker first assigns keep/drop pseudo-labels to speech pairs, which are then used to train an audio large language model (audio-LLM) to make similar decisions directly from audio input. The trained audio-LLM is applied to filter training data for end-to-end S2ST models. Experiments on the CVSS-C and SpeechMatrix datasets demonstrate consistent improvements, with gains of up to +1.4 ASR-BLEU compared to training on unfiltered data.","tokens_in":1830,"tokens_out":494,"duration_ms":16505,"significance":"Should the approach prove robust, it provides a scalable method to improve the quality of large-scale mined S2ST corpora by leveraging audio-LLMs to capture both acoustic and semantic aspects without requiring manual labels. This could have practical impact on training more reliable S2ST systems from noisy web-mined data.","major_comments":[{"comment":"The lightweight ranker is central to generating supervision for the audio-LLM, yet no details are provided on its training procedure, architecture, or validation against human judgments or semantic metrics. Without this, it is unclear whether the pseudo-labels introduce systematic bias, undermining the claim that the audio-LLM jointly captures acoustic fidelity and cross-lingual semantic consistency.","section":"Rank-to-Distill pipeline description"},{"comment":"The reported +1.4 ASR-BLEU improvement on CVSS-C and SpeechMatrix is presented without ablations isolating the effect of the audio-LLM filtering from the ranker or other training factors. Additionally, no error analysis or comparison to using the ranker directly for filtering is included, making it difficult to attribute gains specifically to the proposed method.","section":"Experimental results"}],"minor_comments":[{"comment":"The abstract mentions 'up to +1.4 ASR-BLEU' but the full results section should clarify if this is the maximum across multiple runs or specific configurations.","section":null},{"comment":"Notation for the keep/drop decisions and the audio-LLM input format could be more explicitly defined for reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback, which identifies key areas for improving the clarity and rigor of our manuscript on the Rank-to-Distill pipeline. We respond point-by-point to the major comments below.","responses":[{"response":"We agree that the manuscript would benefit from expanded details on the lightweight ranker. In the revised version, we will add a dedicated subsection describing its architecture, training procedure (including data, objectives, and hyperparameters), and any validation performed using semantic metrics. We will also discuss potential systematic biases in the pseudo-labels and their relation to the audio-LLM's ability to capture acoustic fidelity and semantic consistency.","revision_made":"yes","referee_comment":"The lightweight ranker is central to generating supervision for the audio-LLM, yet no details are provided on its training procedure, architecture, or validation against human judgments or semantic metrics. Without this, it is unclear whether the pseudo-labels introduce systematic bias, undermining the claim that the audio-LLM jointly captures acoustic fidelity and cross-lingual semantic consistency."},{"response":"The request for ablations and comparisons is reasonable. While our results show consistent gains from audio-LLM filtering over unfiltered data, we will add a direct comparison of ranker-based filtering versus the distilled audio-LLM in the revision. We will also incorporate error analysis on the types of noise reduced. However, a complete set of ablations isolating every training factor is not feasible with current resources, so we will provide the most informative comparisons possible from existing pipeline outputs.","revision_made":"partial","referee_comment":"The reported +1.4 ASR-BLEU improvement on CVSS-C and SpeechMatrix is presented without ablations isolating the effect of the audio-LLM filtering from the ranker or other training factors. Additionally, no error analysis or comparison to using the ranker directly for filtering is included, making it difficult to attribute gains specifically to the proposed method."}],"tokens_in":1334,"tokens_out":427,"duration_ms":25615,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is a two-stage filtering method for mined speech-to-speech translation data. A lightweight ranker first assigns keep/drop pseudo-labels to speech pairs, then those labels train an audio-LLM to make the same decisions directly from the raw audio. Experiments on CVSS-C and SpeechMatrix show consistent gains over unfiltered training, up to +1.4 ASR-BLEU.\n\nWhat is new is the direct application of an audio-LLM to paired speech for the keep/drop task, with the goal of capturing both acoustic fidelity and cross-lingual semantics. The setup is practical because it scales without manual labels. The reported improvements on two standard corpora are a concrete result that engineers working on S2ST data pipelines can test.\n\nThe soft spot is exactly the one flagged in the stress-test note. The abstract gives no evidence that the ranker’s pseudo-labels are reliable. There is no human validation, no correlation with semantic metrics, and no comparison of the ranker alone versus the distilled LLM. If the ranker systematically misses misalignment or prosody issues, those errors get passed to the audio-LLM and the final gains cannot be cleanly attributed to better filtering.\n\nThe paper also lacks model details, ablations, and error analysis, so the +1.4 figure is hard to interpret. These are real gaps, but they are the kind that referees can ask the authors to close.\n\nThis is for people building end-to-end S2ST systems who need better ways to clean large mined corpora. A reader focused on practical data filtering will find the pipeline idea and the dataset results useful. The work has enough experimental grounding on public data to deserve peer review, even though the label-quality question needs attention.","headline":"The paper shows a rank-to-distill pipeline that lets an audio-LLM filter noisy S2ST pairs and reports +1.4 ASR-BLEU gains, but supplies no check on whether the initial ranker labels are accurate or unbiased.","tokens_in":2294,"tokens_out":446,"would_cite":false,"duration_ms":16753,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"An audio large language model trained on ranker pseudo-labels can select clean speech pairs directly from raw audio to improve end-to-end speech-to-speech translation.","keywords":["speech-to-speech translation","data filtering","audio language models","pseudo-labeling","end-to-end S2ST","Rank-to-Distill","CVSS-C","SpeechMatrix"],"falsifier":"An end-to-end S2ST model trained on data filtered by the audio-LLM shows no gain or a drop in ASR-BLEU relative to the same model trained on the original unfiltered corpus.","tokens_in":2532,"feed_emoji":"🎙️","tokens_out":676,"duration_ms":22920,"temperature":0.7,"pith_summary":"Large mined corpora for speech-to-speech translation contain noise, misalignment, and semantic errors that degrade performance. The paper demonstrates a two-stage process that first generates keep-or-drop labels with a lightweight ranker and then distills those labels into an audio large language model. The resulting model decides which paired speech samples to retain using only the audio input. If successful, this removes the need for manual annotation while preserving both acoustic quality and cross-lingual meaning. Experiments on standard benchmarks show that speech-to-speech models trained on the selected data outperform those trained on unfiltered data.","feed_headline":"Audio-LLM learns to drop noisy speech pairs and lift S2ST scores","feed_subtitle":"Ranker-generated labels train the model to keep clean audio-only pairs, delivering up to 1.4 ASR-BLEU gain on standard benchmarks.","key_machinery":"The Rank-to-Distill strategy, in which a lightweight ranker first produces keep/drop pseudo-labels that then supervise an audio large language model to predict the same labels from raw paired speech.","core_discovery":"The paper claims that an audio large language model, trained to output keep/drop decisions on paired speech after supervision from a lightweight ranker, jointly models acoustic fidelity and cross-lingual semantic consistency and thereby produces filtered datasets that yield higher ASR-BLEU scores in end-to-end speech-to-speech translation.","pith_inferences":["The same distillation pattern could be applied to clean training data for other speech tasks such as recognition or synthesis.","If the audio-LLM generalizes beyond the ranker that trained it, the approach may reduce dependence on any single ranking heuristic.","The filtered subsets could serve as higher-quality starting points for further self-supervised pre-training of speech models."],"forward_implications":["Training data filtered by the audio-LLM produces consistent gains over unfiltered baselines on CVSS-C and SpeechMatrix.","The largest observed gain reaches +1.4 ASR-BLEU for end-to-end S2ST.","The audio-LLM captures both acoustic fidelity and cross-lingual semantic consistency in its keep/drop decisions.","The method scales filtering to large mined corpora without requiring manual labels."],"fun_headline_variants":["Audio-LLM learns keep drop decisions from ranker on speech pairs","Two-stage ranker distills labels for audio LLM data filtering","Audio large language model selects clean speech pairs for S2ST","Rank-to-distill enables audio LLM filtering of noisy speech data","Audio LLM captures acoustic fidelity and semantic consistency for pairs"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The lightweight ranker produces sufficiently reliable keep/drop pseudo-labels that can serve as supervision for the audio-LLM without introducing systematic bias into the final selection decisions.","fun_headline_variants_meta":{"raw":{"variants":["Audio-LLM learns keep drop decisions from ranker on speech pairs","Two-stage ranker distills labels for audio LLM data filtering","Audio large language model selects clean speech pairs for S2ST","Rank-to-distill enables audio LLM filtering of noisy speech data","Audio LLM captures acoustic fidelity and semantic consistency for pairs"]},"model":"grok-4.3","cost_usd":0.004346,"raw_usage":{"total_tokens":2143,"prompt_tokens":594,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":43462000,"prompt_tokens_details":{"text_tokens":594,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1466,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":594,"tokens_out":83,"duration_ms":10743,"temperature":1.0,"reasoning_tokens":1466,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T06:58:51.203696+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An end-to-end S2ST model trained on data filtered by the audio-LLM shows no gain or a drop in ASR-BLEU relative to the same model trained on the original unfiltered corpus.","supporting_citations":[],"review_version":1}