{"id":"1308bd52-3a4d-4813-acd0-685467ebbb16","arxiv_id":"2504.17782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ClearSep iteratively decomposes naturally mixed audio into label-specific tracks, filters them by remix consistency, and uses them to train a query-based sound separation model, reporting modest SDR improvements on AudioCaps, AudioSet, and ESC-50.","lead":"This paper trains a sound separation model on naturally mixed audio by iteratively decomposing recordings into individual labeled tracks, filtering those tracks by remix consistency, and feeding them back into training. The gains over the strongest baseline on standard benchmarks are small and within one standard deviation, and the real-world evaluation uses the same metric that selected the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Remix-based filtering certifies reconstruction, not source isolation; mask-sum degeneracy lets non-separating tracks pass the data engine.","rationale":"The reader identified the same load-bearing weakness: Re-SDR measures remix consistency, which is a necessary but not sufficient condition for correct separation, and the paper provides no validation that it tracks true separation quality. My analysis sharpens this by showing the failure is not merely a missing validation but a mathematical degeneracy of mask-based separation: any mask set summing to one reconstructs the mixture exactly. This directly threatens the data-engine filtering step and the real-world evaluation in Table 4, both of which are central to the paper's claim. The main benchmark results in Table 1 are still informative because they use ground-truth SDRi on synthetic mixtures, and the iterative improvements in Table 2 are suggestive, but they do not by themselves establish that the filtering mechanism selects high-quality single-source tracks. The manuscript also contains an unreconciled numerical inconsistency in Section 4.5.2 (text reports R6 SDRi of 9.51, Table 6 shows 9.32) and promises code/models without releasing artifacts, which further supports a conditional rather than accept verdict. Since the concern matches the reader's weakest assumption and the recommended conditional stance already reflects the need for validation, the verdict should remain unchanged.","tokens_in":16811,"tokens_out":5886,"duration_ms":65399,"concrete_test":"Build 1,000 labeled synthetic two-source mixtures from isolated AudioSet clips with known ground-truth sources, run ClearSep on each, and compute per-sample ground-truth SDRi alongside Re-SDR. Report the rank correlation and the fraction of tracks passing the Re-SDR > 15 dB threshold whose ground-truth SDRi is below 0 dB; also measure the sum of the predicted masks across tracks. If high Re-SDR coincides with near-unity mask sums and low track-specific SDRi, the filter is certifying reconstruction fidelity, not separation quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pipeline's central step is the remix-based filter: tracks are admitted when Re-SDR and Re-SISDR (Eq. 12-15) exceed thresholds, and Table 4 uses the same metrics as real-world evidence. These metrics only measure whether the separated tracks sum back to the original mixture. Because separation is mask-based (Eq. 4), if the predicted masks sum to the all-ones mask, the remix equals the input regardless of how spectrogram energy is divided among tracks: one track carrying the whole mixture and all other tracks silent gives Re-SDR = Re-SISDR = infinity. Thus a track can pass the filter while containing every sound event in the clip, not just its target. The paper never validates Re-SDR against ground-truth SDR on labeled or synthetic mixtures, so Section 3.3.1's admission thresholds and the real-world results in Table 4 rest on an unvalidated proxy. This is the load-bearing step for the claim that naturally mixed audio can be converted into clean single-source training data; if the filter is selecting reconstruction fidelity rather than isolation, the data engine's benefit and the real-world evaluation are both unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ClearSep extends query-based universal sound separation with an iterative data engine. Starting from a CLAPSep-style model, it uses AudioSet category labels as positive and negative text queries to separate naturally mixed clips into per-event tracks, filters the resulting tracks with remix-based Re-SDR and Re-SISDR thresholds, adds the accepted tracks to the training set under two strategies (Independent Track Training and Self-Separate Training), and retrains the model in a two-stage loop. It also proposes silence augmentation so that the model outputs silence when the query label is absent. Experiments report improvements over CLAPSep on AudioCaps, AudioSet, and ESC-50, large gains in Silent-SDR on AudioCaps, and better Re-SDR/Re-SISDR on natural AudioSet mixtures. The central claim is that naturally mixed audio can be converted into usable single-source training data without manual supervision, yielding state-of-the-art separation performance.","tokens_in":17041,"tokens_out":5657,"duration_ms":61632,"significance":"If the data-engine claim holds, this is a genuinely useful contribution: it offers an automated way to turn weakly labeled natural recordings into single-source training material and an unsupervised monitor for separation quality. The pipeline is concrete, the writing is clear, and the silence-augmentation result in Table 3 is a striking, falsifiable positive result. However, the reported benchmark deltas are small relative to the per-sample standard deviations, and both the data-engine filter and the real-world evaluation rest on a remix-consistency proxy that has not been validated against ground-truth isolation. The significance of the paper therefore depends on additional validation rather than being established by the current evidence.","major_comments":[{"comment":"Re-SDR and Re-SISDR certify remix consistency, not source isolation. Because each track is reconstructed by applying a mask to the mixture spectrogram (Eq. 4), any set of masks that sums to the all-ones mask will remix exactly to the input; one track carrying the full mixture and all other tracks silent achieves perfect Re-SDR and Re-SISDR. The admission thresholds in §4.1 and Appendix A.2 therefore do not establish that an accepted track contains only its target event, and Table 4 evaluates real-world performance with the same unvalidated proxy. The paper should validate Re-SDR/Re-SISDR against ground-truth SDR on labeled or synthetic mixtures, or add a target-specific isolation check that is not satisfied by the mask-sum degeneracy.","section":"§3.3.1 and Appendix B.3, Eqs. (12)–(15)"},{"comment":"The headline improvements are small relative to the reported standard deviations: for example, 11.23±5.07 versus 10.63±5.17 dB SDRi on AudioSet and 14.16±6.29 versus 13.44±6.43 dB SDRi on ESC-50. No significance tests, paired statistics, or confidence intervals are reported. Because the same evaluation items are scored across models, paired tests (for example, Wilcoxon signed-rank or bootstrap on per-sample deltas) would settle whether the differences are reliable rather than within run-to-run variability.","section":"Tables 1 and 2"},{"comment":"The iteration comparison changes both data composition and data volume (5,800 → 9,429 → 10,288 hours of tracks) and adds an extra fine-tuning stage at each iteration. The reported monotone improvement is therefore not a clean test of the data engine's quality filtering: it could be driven by larger data volume or longer training. A matched control that uses an equal number of unfiltered or randomly selected separated tracks is needed to attribute the gain to the proposed filtering mechanism.","section":"Table 2"}],"minor_comments":[{"comment":"The sentence 'surpassing ClearSep (16.764 dB)' should read 'surpassing CLAPSep (16.764 dB)' to match the table row.","section":"§4.4 and Table 4"},{"comment":"The definition of the silent reference is malformed: 'Xsilent =′' should use a proper mathematical notation for the zero signal.","section":"Appendix B.2"},{"comment":"The equations mix |·|² and ‖·‖² notation for signal energy; please use consistent norm notation throughout.","section":"Appendix B, Eqs. (6)–(15)"},{"comment":"Calling Re-SDR and Re-SISDR the 'first' unsupervised evaluation metrics for sound separation is overstated given remix-consistency ideas in earlier unsupervised separation literature; please qualify this claim.","section":"Abstract and Contributions"},{"comment":"The state-of-the-art claim is supported only by a small baseline set; OmniSep and GASS are cited in the text but not compared in Table 1, so the SOTA claim should be narrowed or the missing comparisons added.","section":"Table 1 and Related Work"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the remix-metric degeneracy: Re-SDR and Re-SISDR are used both as the data-engine admission criterion and as the real-world evaluation metric, yet they only measure reconstruction fidelity. The paper's proof burden is to show that the proxy tracks true isolation, for example by validating it against ground-truth SDR on synthetic mixtures and by reporting paired significance tests for the small benchmark deltas. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on ClearSep. The genuinely new thing is the full pipeline: using a strong label-queried separator to decompose naturally mixed AudioSet clips into per-label tracks, admitting tracks only when remix metrics exceed thresholds, then retraining with independent-track and self-separate objectives, with silence augmentation thrown in. CLAPSep already exists, and MixIT used remix consistency, but nobody has packaged this particular iterative data-engine loop. The paper does a decent job of isolating effects: ablations over thresholds, silence rates, and iterations, and the monotonic gain in Table 2 is suggestive.\n\nNow the soft spots. The load-bearing concern is the remix-based filter. Re-SDR and Re-SISDR only measure whether separated tracks sum back to the input. With mask-based separation, if the masks sum to one, the remix is perfect no matter how the energy is distributed. One track carrying the whole mixture and the rest silent passes with infinite Re-SDR. The paper never validates these metrics against ground-truth SDR on labeled or synthetic mixtures, so both the admission threshold (Sec 3.3.1) and the real-world evaluation (Table 4) rest on an unverified proxy. This is a genuine weakness, and the stress-test note holds up.\n\nOther issues are smaller. The external benchmark deltas are consistently positive but only around +0.5 to +0.9 dB SDRi, with standard deviations above 4 dB; no significance tests are reported. Given large test sets, those deltas are probably real, but the paper should say so. There's also a numerical inconsistency in Table 6: the text says R6 reaches an SDRi of 9.51, a 0.53 dB improvement over R1, but the table lists R6 as 9.32 (and 8.98 + 0.53 = 9.51). The claim of the 'first' unsupervised metric is also overbroad; remix consistency has been used in MixIT and elsewhere. And for all the 'code and models will be released' talk, there are no artifacts or links to checkpoints.\n\nOverall: the central idea is plausible and the experiments are extensive enough to warrant a serious referee. The paper should not be accepted as is. I'd want a validation study of Re-SDR against ground-truth SDR on synthetic mixtures, significance tests, correction of the numbers, and code/checkpoints. If the remix filter turns out to track true separation quality, this becomes a useful contribution to data-centric sound separation. Right now it's a promising but unproven recipe.","headline":"ClearSep's data engine idea is worth a serious referee, but the remix filter is unvalidated, the benchmark gains are small, and one table contradicts itself.","tokens_in":17585,"tokens_out":5408,"would_cite":true,"duration_ms":48266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ClearSep converts naturally mixed audio into single-source training tracks, beating artificial-mixture training on sound separation benchmarks.","keywords":["universal sound separation","query-based source separation","natural audio","data engine","remix consistency","self-supervised training","silence augmentation","AudioSet"],"falsifier":"On a set of synthetic mixtures whose true sources are known, compute Re-SDR and Re-SISDR for the data engine's accepted tracks and compare them with true SDRi: if many accepted tracks score above the 15 dB threshold while their true separation SDRi is near zero, or if a one-track-copies-mixture solution passes the filter, the remix proxy is not measuring separation.","tokens_in":16564,"feed_emoji":"🔊","tokens_out":7067,"duration_ms":63167,"temperature":0.7,"pith_summary":"The paper claims that naturally mixed audio, which normally cannot be used for supervised sound separation because no clean reference tracks exist, can be turned into high-quality single-source training data by an iterative data engine. The engine separates each mixture into label-specific tracks, keeps only tracks whose remix closely reconstructs the original audio, and retrains the separation model on the accepted tracks. This loop is claimed to improve query-based separation beyond training on the same 5,800 hours of artificially mixed audio, with Table 1 reporting 11.23 dB SDRi on AudioSet versus 10.63 dB for the strongest baseline. The relevance is practical: real-world audio separation currently lacks ground truth, so a way to mine natural recordings for training would remove the main bottleneck to scaling.","feed_headline":"Natural mixes can replace synthetic ones for training sound separation","feed_subtitle":"Remix-consistency filtering lifts AudioSet SDRi to 11.23 dB, beating 5,800 hours of synthetic-mixture training.","key_machinery":"The load-bearing object is the iterative data engine with remix-based filtering. Re-SDR and Re-SISDR compare the original mixture $X$ with the remix $\\bar X = \\sum_i \\hat X_i$ of the separated tracks, computing signal-to-distortion ratios exactly as in standard SDR and SISDR but with the remix standing in for the ground-truth source. These two metrics are simultaneously an unsupervised evaluation, a data filter that admits tracks only above threshold, and a diagnostic signal for the loop; mutually exclusive class labels act as negative queries so tracks are forced apart, and silence augmentation trains the model to output zero for labels absent from the mixture.","core_discovery":"The central discovery claim is that remix consistency is a usable, unsupervised signal for curating training data from real-world mixtures. ClearSep uses mutually exclusive AudioSet labels as positive and negative queries, separates each natural clip into tracks, and admits a track into the training set only when Re-SDR and Re-SISDR between the original mixture and the sum of the separated tracks exceed thresholds (15 dB for self-separate training, 10 dB for independent-track training). Iterating this data-engine/training loop raised AudioSet SDRi from 9.56 dB after the first pass to 10.45 dB after the third, and the final model reached 11.23 dB SDRi on AudioSet and 14.16 dB on ESC-50 with positive-plus-negative queries, surpassing CLAPSep trained on the same 5,800 hours. The paper also claims that silence augmentation makes absent-query outputs nearly silent, with Silence-SDR jumping to 107.09 dB for positive-plus-negative queries.","pith_inferences":["A natural extension would be to validate the Re-SDR thresholds against known ground truth on synthetic mixtures; if the thresholds correlate strongly with true SDRi, the data engine could be trusted on unlabeled audio.","Because Re-SDR only checks that tracks sum to the mixture, a degenerate separation that puts the entire mix in one track and silence in the others would pass the filter; adding a per-track energy or sparsity prior, or a cross-track disjointness term, would close that loophole.","The same remix-consistency criterion could be applied beyond labels to caption-based queries, where the mutually exclusive negative-query structure is less explicit, by clustering captions or using an ontology to define exclusions.","If the loop can bootstrap from a weak initial separator, the approach points toward purely self-supervised dataset curation for general audio, not just the AudioSet label set."],"forward_implications":["Label-queried separation can be scaled directly from raw, naturally mixed audio: the data engine added thousands of hours of separated tracks across iterations without human annotation of clean sources.","Training on single-source separated tracks outperforms training on mixed tracks at equal scale, supporting the decision to decompose rather than keep mixtures.","Iterating the data-engine/training loop yields steady gains: AudioSet SDRi rises from 9.56 to 10.45 dB across three iterations.","Silence augmentation gives the model a usable “not present” behavior, which is what makes separated tracks clean enough to trust as training data."],"supporting_citations":[{"why":"Supplies the CLAPSep architecture ClearSep builds on and the strongest label- and caption-queried baselines it is compared with.","marker":"Ma et al. (2024)"},{"why":"Provides AudioSet, the source of naturally mixed audio and the category labels used as mutually exclusive queries.","marker":"Gemmeke et al. (2017)"},{"why":"AudioSep is the language-queried separation baseline that ClearSep is measured against.","marker":"Liu et al. (2023)"},{"why":"Provides CLAP, the frozen audio-text encoder that produces query embeddings and hierarchical audio features.","marker":"Wu et al. (2023)"},{"why":"Supplies Silent-SDR and Silent-SISDR, the purity metrics used to evaluate absent-query behavior.","marker":"Wang et al. (2024)"},{"why":"MixIT is the prior unsupervised approach to separating unlabeled mixtures that the data engine extends conceptually.","marker":"Wisdom et al. (2020)"},{"why":"Defines SISDR, which underlies the training objective and the construction of Re-SISDR.","marker":"Le Roux et al. (2019)"},{"why":"Defines SDR, the measurement basis for both the training objective and Re-SDR.","marker":"Vincent et al. (2006)"}],"fun_headline_variants":["Remix consistency lets real-world audio train separation models","No synthetic mixes needed: ClearSep learns from natural audio","ClearSep: separating real-world sounds without synthetic data","Iterative remix filtering boosts sound separation to 11.23 dB SDRi","Natural audio alone trains state-of-the-art sound separator"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loop assumes that if the separated tracks add back up to the original mixture, then each track has isolated its own sound event; a degenerate output where one track carries the entire mixture and all others are silent would also remix perfectly, so the filter's quality signal is only as good as that assumption.","fun_headline_variants_meta":{"raw":{"variants":["Remix consistency lets real-world audio train separation models","No synthetic mixes needed: ClearSep learns from natural audio","ClearSep: separating real-world sounds without synthetic data","Iterative remix filtering boosts sound separation to 11.23 dB SDRi","Natural audio alone trains state-of-the-art sound separator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00098,"raw_usage":{"total_tokens":4152,"prompt_tokens":928,"completion_tokens":3224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3140}},"tokens_in":544,"tokens_out":3224,"duration_ms":22084,"temperature":1.0,"reasoning_tokens":3140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:32:20.414686+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a set of synthetic mixtures whose true sources are known, compute Re-SDR and Re-SISDR for the data engine's accepted tracks and compare them with true SDRi: if many accepted tracks score above the 15 dB threshold while their true separation SDRi is near zero, or if a one-track-copies-mixture solution passes the filter, the remix proxy is not measuring separation.","supporting_citations":[{"cited_title":"F., Ellis, D","cited_arxiv_id":null,"evidence_quote":"Provides AudioSet, the source of naturally mixed audio and the category labels used as mutually exclusive queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SISDR, which underlies the training objective and the construction of Re-SISDR."}],"review_version":1}