{"id":"fd4980e6-5e20-4182-a67a-c7d693d3cffc","arxiv_id":"2412.18710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Simi-SFX conditions a DDSP-based sound effect synthesizer on per-class normalized Mahalanobis distances in CLAP embedding space to enable continuous timbre control and interpolation.","lead":"A new conditioning method for sound effect synthesis uses normalized Mahalanobis distances in a pre-trained audio embedding space as a [0,1] control vector for timbre, combined with a DDSP decoder with transient synthesis. The paper reports synthesis quality comparable to existing models and a regression-based controllability measure, but the controllability evaluation is partly circular.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The controllability evidence rests on a training-aligned embedding metric and lacks a perceptual check; the reported R^2 values do not fully support 'fine-grained timbre control.'","rationale":"The reader's identified weakest assumption — possible circularity between the fine-tuning objective and the controllability metric — is a genuine concern, but the paper partially addresses it by using PANNs (rather than CLAP) for evaluation in Section 4.3. However, the fine-tuning section never explicitly names the embedding model for the measured score, leaving the independence unverifiable without code or a clarifying statement. My stress-test agrees that the regression evidence is not fully convincing for the 'fine-grained' claim, especially because the paper's own analysis shows the effective control range is narrow (0.8–1.0). This does not change the overall verdict: the synthesis quality results (FAD/LSD) are reasonable, and the controllability issue is addressable with additional experiments and more measured language. Therefore the reader's CONDITIONAL verdict remains appropriate.","tokens_in":14106,"tokens_out":6551,"duration_ms":59222,"concrete_test":"Conduct a MUSHRA-style listening test: for several test sounds, generate variants at c = 0, 0.2, 0.4, 0.6, 0.8, 1.0 for a target class while holding loudness and spectral centroid fixed, and ask expert listeners to rate each variant's perceived timbral similarity to the target class. If the perceptual ordering does not match the intended monotonic relationship, the machine-metric R^2 does not reflect genuine timbre control. Additionally, re-run the fine-tuning and evaluation using a third, never-used embedding model (e.g., VGGish or a different CLAP checkpoint) and compare the resulting R^2 values; a drop below 0.3 would indicate that the reported controllability is largely specific to the embedding used during fine-tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 'effective and fine-grained timbre control' is supported mainly by the OLS R^2 values in Section 5.2 (0.4774 for Footstep-set, 0.6041 for Impact-set). The load-bearing problem is that the fine-tuning objective in Section 3.3 minimizes L1+L2 error between the input similarity score and 'the score measured from the generated audio' — the same kind of normalized Mahalanobis-distance measurement later used as the evaluation metric in Section 4.3. The paper does not explicitly state which embedding model is used for the measured score during fine-tuning; if it is the same CLAP-based MDN used for conditioning, then the PANNs-based evaluation is the only independent check, and the moderate R^2 values already suggest limited cross-model generalization. Even if the fine-tuning uses CLAP and the evaluation uses PANNs, both are embedding-space distance proxies, not perceptual timbre judgments. Furthermore, the paper's own regression plots and conclusion state that the effective control range is concentrated in 0.8–1.0, so 'fine-grained' control across the full [0,1] range is not demonstrated. The headline claim is therefore stronger than the evidence supports, and the evaluation pipeline needs an independent, ideally perceptual, validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Simi-SFX, a DDSP-based sound effect synthesizer conditioned on a normalized Mahalanobis-distance similarity vector (MDN) computed from CLAP embeddings. An MLP plus FiLM layer projects this vector into the decoder, which drives noise and transient synthesizers. After main reconstruction training, the conditioning layers are fine-tuned by sampling uniform pseudo-scores in [0,1] and minimizing an L1+L2 loss against a similarity score measured from the generated audio. The authors introduce two Freesound-derived datasets (Footstep-set and Impact-set), report FAD/LSD reconstruction results against DDSP, NoisebandNet, and ICGAN, and evaluate controllability with OLS regression of the measured normalized Mahalanobis distance against the interpolated conditioning score. They report mean R^2 values of 0.4774 (Footstep-set) and 0.6041 (Impact-set), which they interpret as evidence of strong correlation and fine-grained timbre control, and they demonstrate a qualitative timbre interpolation example.","tokens_in":14378,"tokens_out":8544,"duration_ms":78912,"significance":"If the controllability claim survives scrutiny, the contribution has genuine practical merit: a compact [0,1] conditioning vector is an intuitive interface, the model is lightweight and trains in hours on a single GPU, the datasets are released on Zenodo, and the project page provides audio examples. The reconstruction comparison is a credible baseline study with multiple seeds and shows the proposed model is competitive with NoisebandNet while clearly improving over DDSP and ICGAN on the proposed datasets. However, the paper's distinctive claim of fine-grained, full-range timbre control currently rests on a regression metric whose independence from the fine-tuning objective is not demonstrated, on moderate R^2 values, and on the authors' own observation that effective control is concentrated near 0.8-1.0. The evaluation would be substantially strengthened by a fully specified third-party embedding metric or by perceptual judgments, and by an ablation that isolates the conditioning path.","major_comments":[{"comment":"The independence of the controllability evaluation is not established, and this is load-bearing for the central claim. Equation (5) is minimized using \"the score measured from the generated audio,\" but the paper does not state which embedding model, class statistics, and normalization are used for that measurement. Section 4.3 says the evaluation deliberately switches to a PANNs-based embedding extractor \"to minimize potential biases,\" which only makes sense if the fine-tuning measurement used a different pipeline; the reader cannot verify this. Moreover, the PANNs model used in §4.3 is itself fine-tuned to classify the model's synthesized sounds, so it is not an independent pre-trained evaluator. Please specify the exact fine-tuning measurement (embedding model, MD computation, normalization), and report R^2 with an embedding model that is neither the conditioning model nor fine-tuned on the generated test sounds.","section":"§3.3 and §4.3"},{"comment":"The reported effect sizes do not support the abstract's \"fine-grained timbre control.\" Mean R^2 values of 0.4774 and 0.6041, even if reliable, leave a large fraction of the variance unexplained, and the regression is performed on the log-transformed model in Eq. (6), so the reported R^2 is a fit in log space; no back-transformed or distribution-free measure is given. The authors also state that separation is concentrated in the range 0.8-1.0 and that values below 0.8 have limited influence on timbre, which is an explicit limitation on controllability. To support the headline claim, the manuscript needs either a demonstration of usable control across the whole [0,1] interval (for example, per-category monotonicity, perceptual ratings at several score values, or a remapped control curve) or a more modest claim restricted to boundary and extreme timbre separation.","section":"§5.2 and Conclusion"},{"comment":"No ablation isolates the contribution of the similarity conditioning. The regression experiment tests the full model after fine-tuning, and the fine-tuning objective is precisely to make the measured MDN match the conditioning score, so the reported R^2 is at least partly a self-consistency result. The paper does not compare against the same architecture with the similarity conditioning removed, with random or shuffled conditioning vectors, or with class-label conditioning under the same decoder and loss. Such an ablation is needed to show that the MDN vector, rather than the decoder capacity or the fixed spectral centroid and loudness features (which are held constant during interpolation), is responsible for the observed timbre shifts.","section":"§4.2, §4.3, §5.2"},{"comment":"The qualitative timbre interpolation example does not compensate for the weak quantitative controllability evidence. Figure 7 shows spectrograms for one pair of channels but provides no listening ratings, no independent acoustic measurements, and no test of whether listeners can perceive the interpolated steps. Since the paper claims a practical fader/knob control on a [0,1] score, the authors should provide listening results or ABX discrimination at several interpolation positions, including positions below 0.8, where their own analysis indicates limited control.","section":"§5.2 and §6"}],"minor_comments":[{"comment":"Equation (2) should be parenthesized as (MD - MD_min)/(MD_max - MD_min); as printed, the normalization is ambiguous.","section":"Eq. (2)"},{"comment":"In Eq. (3), n is used both as the output sample index and as the summation variable for frames, which makes the equation unreadable; please use m for the frame index.","section":"Eq. (3)"},{"comment":"There are several typos: \"of of\" in the training paragraph, \"Eletrical\" in the affiliation, and \"Y unyi Liu\" in the author byline.","section":"§3.2 and author byline"},{"comment":"The mean R^2 is reported only as a dataset-level number; a per-category table with standard deviations or confidence intervals would help identify which classes drive the effect and whether the result is uniform or dominated by a few categories.","section":"§5.2"},{"comment":"Figures 8 and 9 are placed in the appendix but are the central evidence in Section 5.2; they should be integrated into the main text or referenced consistently with their appendix location.","section":"Figures 8 and 9"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major-revision manuscript. The reconstruction results and the released datasets are useful contributions, and the paper fits the journal's scope. The central controllability claim is currently overextended relative to the evidence, but the missing specification of the fine-tuning measurement and the lack of an independent evaluation are fixable within the manuscript's scope. I do not recommend rejection, but the authors should be required to either provide the independent evaluation or explicitly narrow the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid incremental paper. The normalized Mahalanobis-distance conditioning vector is a real idea, the sparse-peak transient loss is a sensible tweak, and the two released datasets (Footstep-set, Impact-set) are reusable. The reader's circularity concern is mostly answered, though not entirely: the controllability regression in Section 4.3 deliberately uses PANNs instead of CLAP, so the reported R² values are a cross-model test, not a self-fulfilling fit to the training objective.\n\nThat said, the soft spots are genuine. Both CLAP and PANNs are embedding-distance proxies, not perceptual judgments. R² of 0.48 and 0.60 is not 'strong correlation' (Section 5.2), and the paper's own conclusion admits the effective control range is concentrated at 0.8–1.0. 'Fine-grained timbre control across the full [0,1] range' is not demonstrated, so the abstract overstates. The fine-tuning loss (Eq. 5) also uses a measured similarity score, but the paper never says which embedding model computes that score; this should be clarified. If the fine-tuning uses CLAP, the PANNs evaluation is a decent independent check. If it uses PANNs, the regression is largely fitting the objective, and the reader's circularity concern would land squarely.\n\nSynthesis quality is comparable to NoisebandNet, slightly worse FAD on footsteps but overlapping error bars, similar LSD. The paper is honest about the range limitation and provides a demo website. The biggest missing piece is a listening test; 'timbre control' should be validated perceptually, not only through embedding distances.\n\nWho is this for: people working on DDSP-based sound effects, controllable Foley synthesis, or anyone who wants those two datasets. It deserves serious peer review; the flaws are addressable with a clarification, a perceptual evaluation, per-category confidence intervals, and a more measured abstract. I would send it to review with a request for major revision.","headline":"A useful incremental contribution with a genuine new conditioning vector and released datasets, but the 'fine-grained' claim is over-sold and the controllability evidence needs a perceptual check.","tokens_in":14890,"tokens_out":3345,"would_cite":false,"duration_ms":30705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A normalized Mahalanobis-distance similarity vector gives fine-grained, fader-like timbre control in a lightweight DDSP sound-effect synthesizer.","keywords":["sound effect synthesis","timbre control","similarity conditioning","Mahalanobis distance","DDSP","transient synthesis","timbre interpolation","controllable audio generation"],"falsifier":"Re-run the interpolation test and measure the generated audio with a different pre-trained embedding model, or with human perceptual similarity ratings, instead of the metric used to compute fine-tuning losses. If the OLS $R^{2}$ falls well below the reported 0.4774 and 0.6041, the apparent timbre control is specific to the training metric rather than a general acoustic property.","tokens_in":13869,"feed_emoji":"🎛️","tokens_out":4704,"duration_ms":44977,"temperature":0.7,"pith_summary":"This paper tries to establish that a compact similarity vector, computed as the normalized Mahalanobis distance of an audio embedding to each sound class, is an effective and intuitive conditioning signal for synthesizing sound effects with controllable timbre. The authors build this into a lightweight DDSP-based synthesizer with a transient model for impulsive sounds like footsteps and impacts, and they introduce two datasets, Footstep-set and Impact-set, to test it. They report that regression of measured timbre similarity against the injected conditioning value gives mean $R^{2}$ values of 0.4774 for footsteps and 0.6041 for impacts, which they interpret as strong, fine-grained control. If correct, this makes timbre control as simple as turning a knob, without requiring text prompts or one-hot class labels.","feed_headline":"A similarity score becomes a fader for sound-effect timbre","feed_subtitle":"In a DDSP synthesizer, a normalized distance to each class drives footstep and impact timbre, enabling interpolation and knob-like control.","key_machinery":"The central object is the normalized Mahalanobis distance (MDN): for each sound, the Mahalanobis distance of its CLAP embedding to the mean and covariance of each class's embedding cluster, min-max normalized per channel to [0,1]. This vector is smoothed by an MLP and injected into the DDSP decoder through FiLM layers, and the conditioning layers are later fine-tuned with uniformly sampled pseudo-scores using an L1 and L2 regression loss against scores measured from the generated audio. A transient synthesizer based on the inverse discrete cosine transform of sinusoids, regularized by an L2 loss on a harmonic-percussive separated peak signal, carries the impulsive content that the noise filter alone cannot represent.","core_discovery":"The central claim is that a normalized Mahalanobis distance (MDN) vector—the per-class Mahalanobis distance of a CLAP audio embedding, min-max scaled to [0,1]—serves as a continuous, bounded descriptor of timbre that a DDSP decoder can learn to follow. The paper demonstrates this through synthesis of footsteps and impact sounds with FAD and LSD values comparable to a larger filterbank model, and through an OLS regression showing an exponential relationship between the interpolated conditioning score and the measured normalized Mahalanobis distance of the generated audio. A fine-tuning stage trains the conditioning layers to match uniformly sampled pseudo-scores, and the authors show timbre interpolation between classes, such as board and gravel footsteps. They also report a limitation: the effective control range is concentrated near similarity scores of 0.8 to 1.0, leaving the lower range less influential.","pith_inferences":["If the same embedding family is used for both the fine-tuning loss and the controllability evaluation, the reported R^2 values may partly measure the model's fit to that specific metric rather than generalizable timbre control; a cross-embedding or human perceptual test would separate these.","The similarity vector behaves like a soft multi-class membership, so sampling uniformly from the score space could synthesize novel timbres that do not correspond to any single recording in the dataset.","The observed concentration of control in the 0.8–1.0 range suggests an asymmetric mapping; a monotonic rescaling of the score during fine-tuning, such as sampling from a beta distribution, could spread effective control more evenly.","Because the method only needs class-level embedding statistics, it could be applied to fine-grained subcategories within a single broad class, enabling control over subtle material or surface differences beyond the two datasets introduced here."],"forward_implications":["Timbre can be steered continuously between classes by interpolating the similarity vector, enabling sounds that mix characteristics of two categories, such as footsteps on board versus gravel.","The guiding vector is derived from pre-trained embeddings rather than text prompts or one-hot labels, so the method extends to any dataset where class-level embedding clusters can be estimated.","The effective control range is concentrated near similarity scores of 0.8 to 1.0, so flattening the score distribution or mapping scores to a more uniform scale could extend granularity across the full [0,1] range.","Synthesis quality is comparable to NoisebandNet while using the smaller noise synthesizer of the original DDSP, suggesting a practical lightweight configuration for impulsive sound effects."],"supporting_citations":[{"why":"Supplies the CLAP pre-trained embedding space from which class clusters and Mahalanobis distances are computed.","marker":"[39]"},{"why":"Defines the Mahalanobis distance used to construct the similarity score for each class.","marker":"[40]"},{"why":"Provides the base DDSP framework with noise synthesizer and reverb that this work builds on.","marker":"[17]"},{"why":"DDSP-SFX introduces the transient synthesis approach for rigid-body impact sounds that this architecture extends.","marker":"[27]"},{"why":"NoisebandNet serves as the main comparison baseline for synthesis quality using large filterbanks.","marker":"[28]"},{"why":"ICGAN is the comparison baseline for implicit conditioning with continuous vectors and motivates the bounded similarity-score design.","marker":"[31]"},{"why":"Frechet Audio Distance is the objective metric used to evaluate synthesis quality against the reference datasets.","marker":"[36]"},{"why":"PANNs embeddings are used in the controllability evaluation, replacing CLAP to reduce potential bias.","marker":"[38]"},{"why":"FiLM conditioning layers project the smoothed similarity vector onto the decoder, enabling feature-wise modulation.","marker":"[44]"}],"fun_headline_variants":["Similarity score turns into a timbre knob for sound effects","Mahalanobis distance becomes a fader for timbre synthesis","Similarity-based DDSP lets you dial in footstep and impact timbre","Similarity knob drives footstep timbre from board to gravel"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the embedding-based Mahalanobis-distance metric used to measure timbre similarity in the controllability test is independent of the metric the model was trained to match during fine-tuning, and the paper does not specify which embedding model is used for the fine-tuning measurement.","fun_headline_variants_meta":{"raw":{"variants":["Similarity score turns into a timbre knob for sound effects","Mahalanobis distance becomes a fader for timbre synthesis","Similarity-based DDSP lets you dial in footstep and impact timbre","Similarity knob drives footstep timbre from board to gravel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2303,"prompt_tokens":946,"completion_tokens":1357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":1282}},"tokens_in":562,"tokens_out":1357,"duration_ms":9964,"temperature":1.0,"reasoning_tokens":1282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:33:03.011701+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the interpolation test and measure the generated audio with a different pre-trained embedding model, or with human perceptual similarity ratings, instead of the metric used to compute fine-tuning losses. If the OLS $R^{2}$ falls well below the reported 0.4774 and 0.6041, the apparent timbre control is specific to the training metric rather than a general acoustic property.","supporting_citations":[{"cited_title":"Large-Scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies the CLAP pre-trained embedding space from which class clusters and Mahalanobis distances are computed."},{"cited_title":"DDSP: Differentiable Digital Signal Processing,","cited_arxiv_id":null,"evidence_quote":"Provides the base DDSP framework with noise synthesizer and reverb that this work builds on."},{"cited_title":"DDSP-SFX: Acoustically-guided sound effects generation with differ- entiable digital signal processing,","cited_arxiv_id":null,"evidence_quote":"DDSP-SFX introduces the transient synthesis approach for rigid-body impact sounds that this architecture extends."},{"cited_title":"NoiseBand- Net: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks,","cited_arxiv_id":null,"evidence_quote":"NoisebandNet serves as the main comparison baseline for synthesis quality using large filterbanks."},{"cited_title":"ICGAN: An implicit condition- ing method for interpretable feature control of neural au- dio synthesis,","cited_arxiv_id":null,"evidence_quote":"ICGAN is the comparison baseline for implicit conditioning with continuous vectors and motivates the bounded similarity-score design."},{"cited_title":"Fr ´echet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms,","cited_arxiv_id":null,"evidence_quote":"Frechet Audio Distance is the objective metric used to evaluate synthesis quality against the reference datasets."},{"cited_title":"PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recogni- tion,","cited_arxiv_id":null,"evidence_quote":"PANNs embeddings are used in the controllability evaluation, replacing CLAP to reduce potential bias."},{"cited_title":"FiLM: visual reasoning with a gen- eral conditioning layer,","cited_arxiv_id":null,"evidence_quote":"FiLM conditioning layers project the smoothed similarity vector onto the decoder, enabling feature-wise modulation."}],"review_version":1}