{"id":"004404ce-e6bc-4db9-ae56-b857245a01da","arxiv_id":"2501.09169","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"StyleTSE extracts target speech from mixtures using natural-language speaking style descriptions, optionally combined with reference audio, trained on the new TextrolMix dataset.","lead":"This paper presents StyleTSE, a model that separates one person's voice from a mixed recording using a written description of how that person speaks, not just a sample of their voice. The authors also introduce TextrolMix, a dataset of more than 120,000 two-talker mixtures paired with natural language style descriptions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Text-only gains may reflect speaker-identity leakage rather than style understanding; the paper never isolates same-speaker mixtures, so the 'beyond speaker identity' claim is not yet empirically secured.","rationale":"I have no additional concerns beyond this one; the paper's ablations and two-stage training are reasonable, and the dataset is a useful contribution. The requested same-speaker and mismatched-text analyses are cheap and would directly test the central claim.","tokens_in":82,"tokens_out":5756,"duration_ms":79817,"concrete_test":"Using the released TextrolMix metadata, split the text-only test results into (a) mixtures where target and interference share the same TextrolSpeech speaker ID and (b) mixtures with different speaker IDs, then recompute SI-SDRi for each subset. If the same-speaker subset is within roughly 1 dB of the overall 16.41 dB, the model is genuinely extracting by style; if it drops substantially (for example, more than 4 dB) or cannot be computed, the aggregate result is largely explained by speaker-identity leakage. A complementary sanity check is to permute the text prompts between target and interference at inference; text-only performance should collapse if the text semantics are actually being used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that StyleTSE separates by speaking style, not speaker identity, and the headline 16.41 dB text-only SI-SDRi is cited as evidence. The most load-bearing problem is that TextrolMix may let the text clue act as a speaker-ID proxy. TextrolSpeech is a TTS corpus with a limited speaker pool (dynamic mixing reports only 2.45 alternative interferences per target), and its style attributes such as gender, accent, pitch, and emotion may be strongly correlated with particular TTS speakers. A BERT [CLS] embedding of 'male, shocked, slow' could then encode a specific speaker rather than a general style, making the task ordinary speaker-conditioned extraction in disguise. Section II explicitly allows same-speaker target/interference pairs, which is exactly the regime where style, not identity, must do the work, but Table I reports only aggregate results and no same-speaker breakdown. Without that split, or a mismatched-text control, the paper has not demonstrated that text is being used for how the speech sounds rather than who produced it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StyleTSE, a target speech extraction model that conditions a SepFormer separation network on natural-language descriptions of speaking style, optionally combined with an audio reference clue through a gated fusion network. The authors also introduce TextrolMix, a two-talker mixture dataset built from TextrolSpeech, in which each mixture has a target utterance, an interference utterance that differs in at least one style attribute, and text/audio clues. The reported results show text-only SI-SDRi of 16.41 dB and audio+text SI-SDRi of 16.84 dB on the TextrolMix test set, outperforming an LLM-TSE replication and a fine-tuned AudioSep baseline. The paper claims that this demonstrates target speech extraction based on how speech sounds, not only on who is speaking.","tokens_in":8047,"tokens_out":4164,"duration_ms":43483,"significance":"If the results are robust, the paper makes a useful contribution: it introduces a public text-guided TSE dataset, proposes a practical bi-modality clue architecture, and reports strong quantitative gains. The two-stage training and dynamic mixing strategy is also a reasonable engineering contribution. The public dataset and generation scripts are a positive feature that should help reproducibility. However, the central claim that StyleTSE separates by speaking style rather than speaker identity is not yet empirically secured: the paper lacks same-speaker evaluation, mismatched-text controls, and architecture-matched baselines, and it reports no variance across runs.","major_comments":[{"comment":"Section II states that TextrolMix mixtures can include utterances from the same speaker if they exhibit distinguishable style attributes, but Table I reports only aggregate results and attribute-class results, with no breakdown by same-speaker versus different-speaker mixtures. Since TextrolSpeech is a TTS corpus and style attributes such as gender, accent, pitch, and emotion may be strongly correlated with particular TTS speakers, the text-only 16.41 dB SI-SDRi result could be driven by speaker-identity leakage rather than by understanding the text description of speaking style. The claim that the model extracts speech based on 'how they are speaking' requires a same-speaker condition; please report SI-SDRi and PESQ separately for same-speaker and different-speaker test mixtures.","section":"Section II, Table I"},{"comment":"The experiments include no control condition with mismatched text, random text, or no text clue. To establish that the text semantics are actually used for extraction, the authors should compare the text-clue condition against a mismatched-text condition (e.g., a text description of the non-target speaker) and a random-text condition under the same mixture and backbone. If mismatched text does not substantially degrade performance, the model is likely relying on speaker-identity cues rather than on the meaning of the natural-language description. This control is load-bearing for the paper's central 'beyond speaker identity' claim.","section":"Section IV-B, Table I"},{"comment":"The baseline comparison is confounded by architecture: the LLM-TSE replication uses a Conv-TasNet-based SpeakerBeam architecture, while StyleTSE uses SepFormer. The reported gain of roughly 4-6 dB SI-SDRi could reflect the stronger separation backbone rather than the effectiveness of text-guided style conditioning. Please add an architecture-matched ablation, such as the same SepFormer-based model trained with audio-only enrollment, or a text-conditioned model with the same backbone as LLM-TSE, to isolate the contribution of the text clue. In addition, no error bars or repeated-run statistics are reported; small differences in Table I and Table II (for example 15.35 vs 15.10 dB for long vs mid text clues) cannot be interpreted without variance information.","section":"Section IV-B, Table I"},{"comment":"The dynamic mixing description appears inconsistent with Section II. Section II says that each target utterance is paired with an interference utterance that differs in at least one style attribute, but Section III-C says 'DM pairs utterances such that the interference shares the same style attributes as the predefined ones, preserving clue effectiveness.' Please clarify whether the dynamic-mixing interference shares or differs from the target's attributes, and quantify how many attributes typically differ. This is important because the construction rule determines whether the text clue is discriminative at training time and whether the average of 2.45 alternative interferences per target preserves the intended clue semantics.","section":"Section III-C"}],"minor_comments":[{"comment":"Figure 1 contains several typographical artifacts in labels, including 'Es0mate', 'A8en0on', and 'So#max'; these should be corrected.","section":"Figure 1"},{"comment":"Reference [1] is rendered as 'V oiceFilter' with an erroneous space, and reference [21] contains a stray 'Y ' in the author list; these should be fixed.","section":"References"},{"comment":"The row label 'Gated w/o attpool' should be written consistently as 'Gated w/o AttPool' or 'Gated w/o attention pooling' to match the text in Section IV-C.","section":"Table II"},{"comment":"The paper says training signals are truncated to 3 seconds, but TextrolMix utterances are 3-15 seconds; please clarify how truncation interacts with the dataset construction and whether the test set is truncated in the same way.","section":"Section IV-A"},{"comment":"The statement that audio-only performance 'was similar to the speaker identity class (±0.05 dB SI-SDRi), thus not reported separately' is surprising given the large attribute-conditioned differences in the lower half of Table I; a brief discussion of why the speaker-identity class is the relevant comparison would help the reader.","section":"Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the dataset release is a positive step, but the missing same-speaker split and the missing mismatched-text control are exactly the experiments that would make the 'beyond speaker identity' claim convincing. I would be willing to accept after those controls are added and the baseline comparison is made architecture-matched."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on target speech extraction. The clear contribution is TextrolMix: a public, 157-hour two-talker dataset with varied-length natural language style descriptions and metadata, plus generation scripts. That alone is practically useful, since prior text-guided TSE datasets were private or single-attribute. The model work is also solid: gated bi-modality fusion and attention pooling both earn their keep in the ablation, and the two-stage training handles missing modalities sensibly. The gains over the LLM-TSE replication (16.41 vs 10.92 dB SI-SDRi text-only) are large, and fine-tuning AudioSep on TextrolMix also improves it, which suggests the dataset is doing real work.\n\nThe soft spots are about evidence, not execution. There are no error bars or repeated-run variance, so we don't know how stable the 16 dB figure is. The baselines use different backbones (Conv-TasNet vs SepFormer), so some of the gain may be architecture, not the text clue. And the paper never reports a no-text or random-text control, so the marginal contribution of the text is not isolated.\n\nThe bigger issue is the 'beyond speaker identity' claim. TextrolMix deliberately allows same-speaker pairs, but Table I only gives aggregate results. If style attributes like pitch, emotion, and gender are correlated with specific TTS speakers in TextrolSpeech, a BERT embedding of 'male, shocked, slow' could simply be encoding speaker identity. The paper needs a same-speaker vs different-speaker breakdown, or a mismatched-text experiment, to show the model is actually using how the speech sounds rather than who produced it. The stress-test note is right that this is missing; it is not fatal, but it is load-bearing for the title.\n\nCircularity is not a serious concern here—it is an empirical system trained on a held-out split of a self-built dataset. The construction rules do make the task easier by guaranteeing attribute separability, which limits generality to real-world mixtures, but the paper acknowledges this in its design and the dataset is still a valid capability demonstration.\n\nWho is this for? Speech separation researchers, especially those working on clue-conditioned extraction or dataset construction. It deserves a serious referee. I would ask for variance estimates, a same-speaker split, and a control text condition, but I would not desk-reject it.\n\nRecommendation: send to peer review, with the caveat that the 'beyond speaker identity' title needs stronger empirical support.","headline":"Plausible and useful text-guided TSE system with a strong new dataset, but the paper never shows same-speaker mixtures, so the 'beyond speaker identity' claim remains a gap.","tokens_in":8586,"tokens_out":1401,"would_cite":true,"duration_ms":15828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StyleTSE demonstrates that a natural-language description of speaking style is sufficient to extract a target voice from a two-talker mixture, outperforming the previous text-guided baseline by about 5 dB SI-SDRi.","keywords":["target speech extraction","text-guided separation","speaking style","TextrolMix","natural language clue","gated fusion","SepFormer","SI-SDRi"],"falsifier":"Build mixtures where the target and interference carry identical style-attribute annotations (same emotion, pitch, gender, accent, and tempo, differing only in speaker identity) and test StyleTSE; if SI-SDRi does not fall sharply from the reported 16 dB range, the model is exploiting dataset conventions or hidden identity cues rather than the text description.","tokens_in":7655,"feed_emoji":"🎙️","tokens_out":8042,"duration_ms":72825,"temperature":0.7,"pith_summary":"Target speech extraction normally needs a speaker-specific clue such as an enrollment audio snippet, and fails when no such clue has been collected. This paper argues that a natural-language description of how a person speaks—for example 'shocked pitch and slow speaking rate'—can serve as the clue instead, provided the described style separates the target from the interferer. To make that test possible, the authors build TextrolMix, 120,000 two-talker mixtures (157 hours) with six annotated style attributes per utterance, and StyleTSE, a SepFormer-based extraction model whose gated bi-modality clue network reads audio and text. On TextrolMix, text-only guidance reaches 16.41 dB SI-SDRi with dynamic mixing, and audio-plus-text reaches 16.84 dB, against 10.92 dB for the replicated LLM-TSE baseline. The paper claims this extends target speech extraction beyond speaker identity to how the speech is delivered.","feed_headline":"Text alone pulls one voice out of a two-talker mix","feed_subtitle":"Style descriptions replace enrollment audio, lifting extraction quality about 5 dB over the prior text-only baseline.","key_machinery":"The central object is the bi-modality clue network working together with the TextrolMix construction rule. The clue network encodes the text with a frozen BERT model, encodes the reference audio with a trainable encoder whose output is attention-pooled across time frames, and combines the two vectors with a sigmoid-gated weighted sum; the fused vector is then expanded and added to the chunked mixture representation at every IntraTransformer block of the SepFormer backbone. The dataset rule pairs each target utterance with an interference utterance that differs in at least one of six style attributes (speaker identity, emotion, pitch, gender, accent, tempo) and requires any reference audio to share an attribute with the target but not with the non-targets, which guarantees that the text clue has a real separator to point at. The gated fusion and attention pooling are the components that the ablations identify as responsible for the gains.","core_discovery":"StyleTSE establishes that target speech extraction can be driven by a style description rather than a speaker identity: the model conditions a SepFormer separation backbone on a 768-dimensional BERT [CLS] embedding of the text clue and an attention-pooled embedding of the audio clue, then fuses the two with a learned gate and injects the result into the chunked mixture representation inside each IntraTransformer block. With text-only clues on TextrolMix the model reaches 16.41 dB SI-SDRi; with a style-matched audio clue added it reaches 16.84 dB, while the replicated LLM-TSE baseline reaches 10.92 dB. The authors attribute the gains to the dataset construction rule (target and interference differ in at least one of six style attributes), to attention pooling over the reference audio, and to two-stage training with dynamic mixing. The paper frames the result as showing that text guidance works not only for who is speaking but for how they are speaking.","pith_inferences":["If the result generalizes beyond TextrolMix, the practical interface for extraction could shift from 'provide a recording of the speaker' to 'say which style you want,' which matters for hearables, meeting assistants, and archival audio where enrollment samples do not exist.","The dataset construction guarantees a text-discoverable separator, so the reported 16 dB figures likely represent an upper bound for style-guided extraction; real-world performance depends on whether the target and interference actually differ in a describable attribute.","The same gated-fusion pattern could be applied to other query types, such as typed descriptions in general sound-separation systems, where text and audio tags currently compete rather than cooperate.","A direct next test would be zero-shot evaluation on naturally recorded (non-TTS) two-talker conversations with human-written style descriptions, which would reveal how much of the result relies on TextrolSpeech's synthesized voices and attribute labels."],"forward_implications":["Text-only clues suffice for target speech extraction on TextrolMix, reaching 16.41 dB SI-SDRi with dynamic mixing and staying above 15 dB across long, mid, and short text descriptions.","Adding a reference audio clue that shares a style attribute with the target improves extraction to 16.84 dB SI-SDRi, so the same model covers classical enrollment-based TSE and text-only TSE.","Because the model is trained with missing modalities, audio-only inputs perform at roughly the speaker-identity level (around 15.7 dB SI-SDRi in the ablation), meaning a single system can serve both settings.","Replacing gated fusion with averaging or concatenation costs about 1 dB, and removing attention pooling in favor of average pooling costs about 2 dB, identifying the mechanism behind the improvement."],"supporting_citations":[{"why":"Baseline text-guided target speech extraction model that the paper replicates on TextrolMix; its 10.92 dB SI-SDRi is the number StyleTSE is compared against.","marker":"[11]"},{"why":"SepFormer, the transformer-based separation architecture that StyleTSE adapts for clue-conditioned extraction.","marker":"[13]"},{"why":"TextrolSpeech, the source corpus whose utterances and style-attribute labels are used to build TextrolMix.","marker":"[15]"},{"why":"LibriMix mixing procedure that defines loudness normalization and the mixture SNR distribution used by TextrolMix.","marker":"[17]"},{"why":"Frozen BERT model that converts text descriptions into the [CLS] embedding used as the text clue.","marker":"[21]"},{"why":"Exformer approach for expanding and injecting the clue embedding into the chunked mixture representation.","marker":"[22]"}],"fun_headline_variants":["Style text alone extracts target voice from mixture","How they speak, not who: text-guided speech extraction","Text clues upstage audio enrollment for target extraction","StyleTSE: text descriptions drive speech separation","From identity to style: text-only clues for TSE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every TextrolMix mixture has a text-discoverable separator: the target and interference differ in at least one of the six annotated style attributes, and any reference audio shares an attribute only with the target.","fun_headline_variants_meta":{"raw":{"variants":["Style text alone extracts target voice from mixture","How they speak, not who: text-guided speech extraction","Text clues upstage audio enrollment for target extraction","StyleTSE: text descriptions drive speech separation","From identity to style: text-only clues for TSE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":1961,"prompt_tokens":885,"completion_tokens":1076,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1002}},"tokens_in":501,"tokens_out":1076,"duration_ms":11199,"temperature":1.0,"reasoning_tokens":1002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:02.560464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build mixtures where the target and interference carry identical style-attribute annotations (same emotion, pitch, gender, accent, and tempo, differing only in speaker identity) and test StyleTSE; if SI-SDRi does not fall sharply from the reported 16 dB range, the model is exploiting dataset conventions or hidden identity cues rather than the text description.","supporting_citations":[{"cited_title":"Attention is all you need in speech separation,","cited_arxiv_id":null,"evidence_quote":"SepFormer, the transformer-based separation architecture that StyleTSE adapts for clue-conditioned extraction."},{"cited_title":"Textrolspeech: A text style control speech corpus with codec language text-to-speech models,","cited_arxiv_id":null,"evidence_quote":"TextrolSpeech, the source corpus whose utterances and style-attribute labels are used to build TextrolMix."},{"cited_title":"Semi- supervised time domain target speaker extraction with attention,","cited_arxiv_id":null,"evidence_quote":"Exformer approach for expanding and injecting the clue embedding into the chunked mixture representation."}],"review_version":1}