{"id":"2b7f6c57-22de-45dd-9142-2886fd0fa6b6","arxiv_id":"2411.08375","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A playback-and-record method creates a realistic two-speaker training set that yields up to 1.65 dB SI-SDR improvement over synthetic training.","lead":"The authors build a speaker separation training set by playing individual TIMIT recordings through loudspeakers, recording them alone to get references, then playing two together and recording the mixture. Training a recurrent model on this realistic dataset improved separation quality by 1.65 dB SI-SDR over a synthetic dataset in their test recordings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth recordings are separate takes from the mixture; if they are not true mixture components, the 1.65 dB SI-SDR gain is uninterpretable.","rationale":"The reader's weakest assumption is exactly the separate-take ground-truth problem, and I agree it is load-bearing. If the solo recordings are not the true components of the mixture, both training and evaluation are corrupted, so the central claim is unverified. However, this is not a fatal flaw in the method per se; it is an unvalidated assumption that can be checked with a residual test. Since the reader already issued a CONDITIONAL verdict, my read does not change that verdict. The internal contradiction in the learning-curve minima (0.23 vs 0.26) and the same-room test set are additional concerns, but the ground-truth consistency issue is more fundamental because it directly undermines the meaning of the reported SI-SDR gain.","tokens_in":14843,"tokens_out":3771,"duration_ms":43239,"concrete_test":"Select a held-out set of 50 mixtures. For each, align gts1 and gts2 to rMix (using known audio latency or cross-correlation) and compute the residual e = rMix - (gts1 + gts2). Report the residual energy relative to each source energy and the SI-SDR of the residual. If the residual is within, say, 10 dB of either source energy, the ground truths are not the actual mixture components and the reported 1.65 dB improvement cannot be attributed to speaker separation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the assumption in Section 4.4 and Algorithm 1 that the solo recordings gts1 and gts2 are exactly the components inside the later-recorded mixture rMix. These are separate takes: the loudspeaker, room state, amplifier, and microphone noise can differ between the solo and joint playback. If rMix is not well approximated by gts1 + gts2, the training targets are inconsistent with the input mixture, and the model is trained to reconstruct signals that are not actually present in the mixture. The same separate-take procedure is used to construct the test references, so the reported SI-SDR is computed against these same solo recordings. A model that simply learns to output a reverberant solo-like signal could therefore appear to improve SI-SDR without actually separating the two sources in the mixture. The paper never reports a check of the identity rMix ≈ gts1 + gts2, so the headline 1.65 dB improvement may reflect consistency with the recording protocol rather than true separation accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for constructing a 'realistic' two-speaker speech-separation training set, called Realistic_TIMIT_2mix, by playing TIMIT utterances through loudspeakers in a room and recording the resulting mixture with a microphone, while obtaining ground truths by playing each speaker file alone and recording it in a separate take, all using MATLAB's AudioPlayerRecorder full-duplex function (Algorithm 1). The authors train two copies of the same BGRU deep attractor network from their prior work [32], one on this realistic set and one on a synthetic TIMIT mixture set, and compare SI-SDR on LibriMix and on their own real-mixture test recordings. They report that the model trained on the realistic set outperforms the synthetic-trained model by 1.65 dB on real mixtures and by 1.25 dB on LibriMix, and that its performance remains more stable as microphone distance varies.","tokens_in":14978,"tokens_out":6504,"duration_ms":66511,"significance":"If the acquisition procedure genuinely yields the true components of the recorded mixtures as ground truths, the dataset and the full-duplex recording protocol would be a practical contribution to speaker separation, targeting a real bottleneck in the field. The LibriMix result is a useful control: because LibriMix is an external synthetic corpus, the improvement there is not trivially explained by matching the recording setup. However, the validity of the central contribution depends on an unverified assumption that the separately recorded solo takes are exactly the components inside the jointly recorded mixture, and the real-mixture evaluation shares the training room and hardware. With explicit validation of the additive-component identity, independent test recordings, and statistical reporting, the result could be significant; in its current form the evidence is suggestive but not conclusive.","major_comments":[{"comment":"The dataset construction assumes that the solo recordings gts1 and gts2 (Algorithm 1, lines 5-10) are exactly the source components inside the mixture rMix (line 11). These are separate playbacks: any change in loudspeaker response, amplifier gain, room impulse response, or microphone noise between takes makes the training targets inconsistent with the input mixture. The paper reports no check of the identity rMix ≈ gts1 + gts2 (e.g., residual energy after time alignment), even though this identity is the central assumption that makes the ground truths usable. Without it, the 1.65 dB SI-SDR gain in Table 2 can reward a model for emitting a plausible solo-like reverberant signal rather than actually separating the sources present in the mixture. Please add an explicit validation of the additive-component assumption, or modify the acquisition procedure so that the mixture and the components are recorded simultaneously with known capture paths.","section":"Section 4.4 / Algorithm 1"},{"comment":"The real-mixture test set was recorded by the authors using the same procedure described in Algorithm 1, in the same room, with the same hardware as the training set. The comparison in Table 2 therefore measures within-protocol consistency rather than generalization to other real acoustic environments, which is the conclusion drawn in the abstract ('real-world scenarios'). The paper should evaluate on independently recorded real mixtures, for example the REAL-M corpus [16] or recordings made in different rooms with different microphones, and report the resulting domain gap. The LibriMix result in Table 1 is a helpful external check, but LibriMix is a synthetic mixture set and does not by itself establish real-world generalization.","section":"Section 6, real-mixture test set"},{"comment":"The paper reports only point estimates of SI-SDR, with no number of test mixtures, no confidence intervals, and no variance across training runs. The 1.65 dB difference that supports the main claim could plausibly fall within run-to-run seed variance for a model of this type. Please report mean ± standard deviation over at least three training runs for each condition, state the test-set size, and, if feasible, a paired significance test. This is necessary to make the headline quantitative claim reliable.","section":"Section 6, Tables 1-2 and Figure 6"}],"minor_comments":[{"comment":"The learning-curve description is internally inconsistent: the text states that the synthetic model reaches its minimum (0.23) at epoch 157 and the realistic model at 0.26 at epoch 143, then says the realistic model has the lowest minimum and that the synthetic model converged faster. The numbers imply the opposite on both counts; please correct the prose or the numbers.","section":"Section 5.2, learning-curve paragraph"},{"comment":"The dataset itself is not made available; no URL, repository, or data-access statement is provided. For a contribution whose main deliverable is a dataset, a download link and a data sheet are important for reproducibility and for the community to adopt the method.","section":"Section 4.4 / Algorithm 1"},{"comment":"The real-mixture test procedure is underspecified: the paper should state how many real test mixtures were recorded, from which TIMIT subset the held-out utterances were drawn, and how speaker overlap with the training set was avoided beyond saying 'files not used for training.'","section":"Section 6"},{"comment":"The claim that this is 'the first' real speaker-separation training set with ground truths is asserted repeatedly, but the related-work discussion does not systematically rule out earlier corpora with close-microphone or multi-channel recordings that provide both mixtures and per-speaker tracks; please narrow the claim or add explicit comparisons with such datasets.","section":"Sections 2 and 4"},{"comment":"The LibriMix test configuration should be specified in detail, including the exact version (e.g., Libri2Mix, clean/noisy/reverb), sampling rate, and mixture construction, because the 1.25 dB gain in Table 1 can be interpreted only with that context.","section":"Section 6, LibriMix test set"},{"comment":"Figure numbering is inconsistent: the scenario diagram appears under a caption labeled 'Figure (1)' that duplicates the earlier audio-card diagram, and subsequent figures are numbered out of sequence; please renumber all figures in order of appearance.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The main technical risk is the unverified additive-component assumption behind the ground truths; a residual check or a changed acquisition design is essential before the dataset claim can be accepted. The authors should also consider that the claim of being 'first' needs more careful positioning, and that releasing the dataset will materially affect the contribution's impact. The evaluation with a single model from the authors' own prior work weakens the title's generality, so either a second architecture should be added or the claims should be narrowed. The inconsistency in the learning-curve paragraph suggests that the results section should be carefully rechecked before revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the recording protocol is a neat, practical idea, but the paper never verifies that the solo recordings are actual components of the mixtures, so the headline 1.65 dB gain is not yet interpretable.\n\nThe genuinely new thing is the dataset construction procedure. They use MATLAB's AudioPlayerRecorder in full-duplex mode to play each TIMIT utterance alone through a loudspeaker and record it as the ground truth, then play the two utterances together on separate channels and record the mixture. That yields paired real recordings, Realistic_TIMIT_2mix, without asking a speaker to repeat a sentence. Training a BGRU deep-attractor network on this dataset improves SI-SDR by 1.65 dB on real test mixtures over the same model trained on synthetic TIMIT mixtures, and the distance-robustness experiment shows the real-trained model stays above 8 dB while the synthetic one degrades. That is a credible existence proof that real training data can help a separation model in the same room.\n\nThe soft spots are load-bearing. The ground truths are separate takes from the mixture: first they record gts1 alone, then gts2 alone, then rMix from both speakers. The paper assumes these solo recordings are exactly the components in rMix, but it never checks rMix ≈ gts1 + gts2. Differences in loudspeaker response, amplifier state, background noise, or room conditions between the takes would make the training targets inconsistent with the mixture, and the same solo recordings are used as test references. Second, the real test set is recorded in the same room and with the same hardware as the training set, so there is no evidence of generalization to other environments. Third, the 'first realistic training set' claim is contradicted by the paper's own citations: CHiME-5 and Mixer6 are real recordings with per-speaker reference channels. Fourth, the learning-curve paragraph contains a small contradiction: it reports minima of 0.23 (synthetic) and 0.26 (realistic) but then says the realistic curve is the lowest; that is backwards. Also there are no error bars, no dataset release, and only one baseline model, which happens to be the authors' own.\n\nOn the positive side, the SI-SDR tables are internally consistent, the protocol is detailed enough to reproduce, and the authors are explicit about hardware and recording conditions. The flaws are all addressable: verify the additive identity, record a held-out room, release the dataset, and fix the claim.\n\nRecommendation: send to peer review. A serious referee could help turn this into a solid contribution. I would not cite it as-is, but I would read the revision.","headline":"The recording protocol is a neat practical idea, but the paper never verifies that the solo recordings are true components of the mixtures, so the headline 1.65 dB gain is not yet interpretable.","tokens_in":15524,"tokens_out":4401,"would_cite":false,"duration_ms":36412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real recorded training data beats synthetic mixes by 1.65 dB.","keywords":["speaker separation","training dataset","ground truth","realistic mixtures","full-duplex recording","BGRU deep attractor","SI-SDR","TIMIT"],"falsifier":"Record the same two speakers with close microphones at the same time as the room microphone captures the mixture, then compare each speaker's close-mic signal, after time alignment, with the separately recorded solo take; if they differ by more than the noise floor, the solo takes are not the true mixture components, and the reported gain is not a clean measure of separation accuracy.","tokens_in":14630,"feed_emoji":"🎧","tokens_out":7884,"duration_ms":76418,"temperature":0.7,"pith_summary":"Speaker separation models usually train on synthetic mixtures made by digitally adding two clean speech signals, so they do not see the reverberation, device response, and latency that a real microphone introduces. This paper claims to build the first real training set with ground truths, Realistic_TIMIT_2mix, by using a full-duplex audio player/recorder to capture each speaker alone and then capture the two speakers together under the same conditions. The authors train the same BGRU deep attractor network on this recorded set and on a synthetic set made from the same TIMIT files. On real test mixtures the realistic-trained model reaches 8.66 dB SI-SDR versus 7.01 dB for the synthetic-trained model, a gain of 1.65 dB, and it degrades far less when the distance between microphone and speakers is varied.","feed_headline":"Real recorded training data beats synthetic mixes by 1.65 dB","feed_subtitle":"Recording mixtures and solo ground truths through the same microphone lifts separation accuracy on real audio.","key_machinery":"The key object is a full-duplex audio playback/recording routine (MATLAB's AudioPlayerRecorder with an ASIO driver) that can write and read audio samples at the same time, letting the same hardware path first record each speaker alone and then record the simultaneous two-speaker mixture. Overrun and underrun counts are checked on every take so that a take with lost samples is redone, which is the step that makes the solo recordings usable as ground truths. The evaluation model is a deep attractor network: log-magnitude spectrogram frames pass through four bidirectional gated recurrent unit (BGRU) layers and a fully connected layer that embeds each frame in a 20-dimensional space, and frames are assigned to the nearest speaker center to build masks that reconstruct the separated speech.","core_discovery":"The central claim is that a recorded, non-synthetic two-speaker separation corpus with clean ground-truth stems is feasible, and that supervised training on it transfers to real recordings better than training on digitally mixed speech. The construction records each TIMIT utterance alone through a loudspeaker and microphone to form the ground truth, then plays two utterances on two channels simultaneously and records the resulting mixture, retrying whenever the routine reports overrun or underrun sample loss. This yields Realistic_TIMIT_2mix with 30/10/5 hours of training/validation/test data. With identical hyperparameters, the BGRU deep attractor model trained on this corpus beats the synthetic-trained model by 1.65 dB SI-SDR on real mixtures and by 1.25 dB on synthetic LibriMix mixtures, and its performance stays above 8 dB SI-SDR as microphone–speaker distance varies while the synthetic model drops toward 2 dB.","pith_inferences":["The paper does not test transfer to a different room or a different loudspeaker; a plausible extension is measuring how much of the 1.65 dB gain survives when the trained model is evaluated on recordings made elsewhere.","Because ground truths are recorded in separate takes, any drift between takes enters as a corrupted target; this could be tested by recording close-microphone references simultaneously with the mixture and checking whether the solo takes match them after alignment.","Adding real noise and reverberation to the recorded-set construction, as the paper's outlook suggests, would test whether the same full-duplex recipe can replace digitally noisy synthetic data.","A stronger evaluation would compare the realistic-trained model against a model trained on a large synthetic noisy/reverberant corpus evaluated on the same real test set, isolating the contribution of the recording method from the contribution of training data size."],"forward_implications":["Models trained on recorded mixtures should separate real two-speaker audio more accurately than models trained on digitally summed clean speech.","The distance experiment implies that a fixed recording distance of 2 meters teaches features that keep separation above 8 dB SI-SDR across 0.5 to 3 meters in the same room.","The recording recipe can be reused to create paired mixture/ground-truth sets for other corpora, enabling supervised separation training in environments where digital mixing is unrealistic.","The 1.25 dB improvement on synthetic LibriMix indicates the benefit is not just matching the recording setup; the realistic-trained model also separates computer-mixed speech better."],"supporting_citations":[{"why":"Establishes the motivating failure mode: models trained on synthetic mixtures perform poorly on real mixtures, which is the gap the proposed dataset targets.","marker":"[16]"},{"why":"Supplies the synthetic LibriMix test material used to compare the two trained models.","marker":"[19]"},{"why":"Provides the TIMIT utterances that are played and recorded to create ground truths and mixtures.","marker":"[30]"},{"why":"Provides the four criteria used to select which utterance pairs form the mixture list.","marker":"[31]"},{"why":"Supplies the BGRU deep attractor network used as the base model for both training runs.","marker":"[32]"},{"why":"Defines the SI-SDR metric that quantifies separation quality in the reported comparisons.","marker":"[34]"}],"fun_headline_variants":["Recorded training data beats synthetic by 1.65 dB in speaker separation","Training on realistic recordings lifts speaker separation by 1.65 dB","Realistic recorded dataset improves speaker separation by 1.65 dB","Real recording training data outperforms synthetic by 1.65 dB"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a speaker's solo recording, made in a separate playback pass, is exactly the signal that appears inside the two-speaker mixture; if the loudspeaker, the room, or the timing behaves differently during the joint pass, the training targets do not match the mixture.","fun_headline_variants_meta":{"raw":{"variants":["Recorded training data beats synthetic by 1.65 dB in speaker separation","Training on realistic recordings lifts speaker separation by 1.65 dB","Realistic recorded dataset improves speaker separation by 1.65 dB","Real recording training data outperforms synthetic by 1.65 dB"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000919,"raw_usage":{"total_tokens":3929,"prompt_tokens":918,"completion_tokens":3011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2934}},"tokens_in":534,"tokens_out":3011,"duration_ms":19645,"temperature":1.0,"reasoning_tokens":2934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:39:13.782848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the same two speakers with close microphones at the same time as the room microphone captures the mixture, then compare each speaker's close-mic signal, after time alignment, with the separately recorded solo take; if they differ by more than the noise floor, the solo takes are not the true mixture components, and the reported gain is not a clean measure of separation accuracy.","supporting_citations":[{"cited_title":"REAL-M: Towards Speech Separation on Real Mixtures","cited_arxiv_id":"2110.10812","evidence_quote":"Establishes the motivating failure mode: models trained on synthetic mixtures perform poorly on real mixtures, which is the gap the proposed dataset targets."},{"cited_title":"DARPA TIMIT acoustic -phonetic continous speech corpus CD -ROM. NIST speech disc 1 -1.1,","cited_arxiv_id":null,"evidence_quote":"Provides the TIMIT utterances that are played and recorded to create ground truths and mixtures."},{"cited_title":"Building Corpora for Single-Channel Speech Separation Across Multiple Domains","cited_arxiv_id":"1811.02641","evidence_quote":"Provides the four criteria used to select which utterance pairs form the mixture list."},{"cited_title":"Improving deep attractor network by BGRU and GMM for speech separation,","cited_arxiv_id":null,"evidence_quote":"Supplies the BGRU deep attractor network used as the base model for both training runs."},{"cited_title":"SDR – HALF-BAKED OR WELL DONE?,","cited_arxiv_id":null,"evidence_quote":"Defines the SI-SDR metric that quantifies separation quality in the reported comparisons."}],"review_version":1}