{"id":"2f668c60-79bd-4cd6-a9ea-fc2ea3065ef9","arxiv_id":"2505.10885","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Bengali deepfake audio dataset is introduced, but its size figures contradict each other and the paper provides no detector evaluation to substantiate the benchmark claim.","lead":"The authors built a Bengali deepfake audio dataset by synthesizing speech with a VITS text-to-speech model and collecting real Bengali utterances from existing corpora. The paper claims this is the first public resource of its kind, but the dataset statistics are internally inconsistent and no deepfake detection benchmark is actually run.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Common Voice subset's fake audio is generated by a single-speaker VITS trained on SUST, so fake clips use a different voice than their paired real clips; detection can succeed on speaker identity alone, which breaks the benchmark claim.","rationale":"The reader's weakest assumption is precisely the most load-bearing concern: for the Common Voice subset, the fake audio is not a deepfake of the paired real speaker but synthetic speech in the SUST speaker's voice. This follows directly from the paper's own description of the VITS model as single-speaker and trained solely on SUST. The benchmark claim fails because a large subset can be solved by speaker verification rather than by detecting synthetic artifacts. The numerical inconsistencies in Table 2 and the text add a second, independent layer of unreliability, but the speaker confound is the more fundamental scientific flaw. My concrete test would settle the matter by measuring whether speaker identity alone separates the classes on the Common Voice subset; if it does, the dataset as described cannot support its stated benchmark purpose. Since the reader's verdict is REJECT and my analysis confirms the same critical flaw, no adjustment to the verdict is needed.","tokens_in":6550,"tokens_out":5673,"duration_ms":56224,"concrete_test":"Download the Hugging Face dataset and compute a standard speaker embedding (e.g., ECAPA-TDNN from SpeechBrain) for every real and fake utterance. On the Common Voice subset only, train a logistic regression or cosine-similarity classifier to distinguish real from fake using embedding similarity to the SUST male voice prototype. If the resulting equal error rate is below 1 percent (or AUC is above 0.99), the labels are trivially separable by speaker identity, confirming that the fake audio does not preserve the target speaker's voice. Also verify with a listening check: for 10 random Common Voice pairs, if the real and fake clips have different speakers, the pairing is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BanglaFake is a valid Bengali deepfake audio benchmark. That requires each fake utterance to be a synthetic version of the corresponding real utterance (same speaker, same content), so that discrimination must rely on synthesis artifacts rather than on speaker identity. The paper's own methodology violates this for the Common Voice portion. Section 4.2 states that the fake audio for all sources is 'generated using our model', where the model is a VITS trained on SUST (Section 4.1), and Sections 3.1 and 6 state that the model is a single male speaker. A single-speaker VITS is a TTS system that always produces the SUST speaker's voice, not the voice of a Common Voice speaker. Therefore, for the 2,260 to 2,797 Common Voice real/fake pairs, the real clip is speaker X and the 'fake' clip is the SUST male speaker. A detector can classify these pairs almost perfectly using an off-the-shelf speaker embedding, without ever detecting a deepfake artifact. This confound affects roughly 18 to 23 percent of the dataset's real samples, and the paper neither reports nor controls for it. Because the stated purpose is a deepfake detection benchmark, this is a load-bearing flaw. Additionally, the reported counts are internally inconsistent: Table 2's rows sum to 12,797 real and 12,797 fake, while the abstract, Table 1, and the Total row claim 12,260 and 13,260; the text says 2,260 Common Voice samples but the per-speaker rows sum to 2,797. These discrepancies mean the exact dataset composition is not reliably specified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BanglaFake, a Bengali audio dataset claimed to contain 12,260 real and 13,260 synthetic utterances, with real speech from the SUST TTS corpus and Mozilla Common Voice and synthetic speech generated by a VITS model trained on the SUST corpus. The authors report a mean opinion score evaluation with 30 native Bengali listeners and a t-SNE visualization of MFCC features, and they claim that the dataset is the first public Bengali deepfake audio benchmark.","tokens_in":6936,"tokens_out":6841,"duration_ms":68421,"significance":"If the dataset were constructed as claimed, it would fill a real gap in low-resource audio forensics, and the public release of the dataset and code is a positive feature. However, the current manuscript does not establish that the fake clips are deepfakes of the paired real clips for the Common Voice subset, and the internal dataset statistics are inconsistent. The significance is therefore conditional on a substantial reconstruction of the dataset and its evaluation.","major_comments":[{"comment":"The Common Voice portion of the dataset does not implement a deepfake task. The VITS model was trained only on the SUST corpus (Section 4.1) and is a single-speaker male model (Table 1 and Section 6), yet Section 4.2 states that the fake audio paired with Common Voice real audio was 'generated using our model.' Consequently, for every Common Voice pair, the real clip is from one of five Common Voice speakers while the fake clip is the SUST male voice. A detector can separate these pairs by speaker identity alone, for example with any off-the-shelf speaker embedding, without detecting synthesis artifacts. This affects the 2,260 (or 2,797, see next comment) Common Voice samples and invalidates the benchmark claim for that subset.","section":"§4.2, §3.1, Table 1, §6"},{"comment":"The reported dataset sizes are internally inconsistent. Table 2's rows sum to 12,797 real and 12,797 fake, but the row labeled 'Total' says 13,260; the abstract and Table 1 say 12,260 real and 13,260 fake; and Section 4.2 says 2,260 Common Voice samples while the five Common Voice rows in Table 2 sum to 2,797. The released dataset cannot be checked or reproduced against these numbers, and the discrepancy affects the central quantitative claims of the paper.","section":"Table 2; Abstract; Table 1; §4.2"},{"comment":"The only quantitative evaluation is a t-SNE plot of MFCCs for 1,000 randomly selected pairs. t-SNE overlap is not a detection metric: it depends on hyperparameters and initialization, and it cannot be used to estimate error rates. No deepfake detector baseline (for example, wav2vec 2.0 or LFCC-LCNN with an EER) is reported, so the paper provides no evidence that the dataset is usable as a benchmark. Moreover, if the Common Voice fakes come from a different speaker, the t-SNE plot is not even informative about synthesis artifacts.","section":"§5.2"},{"comment":"The MOS evaluation rates only the synthesized audio and does not compare it against the corresponding real audio, nor does it report inter-rater variability, so it does not by itself support the claim that real and fake audio are hard to distinguish. The text is also internally inconsistent: it says 10 sentences were selected for evaluation, but then states that participants listened to five sets of clips.","section":"§5.1"}],"minor_comments":[{"comment":"The dataset name is spelled 'BangalFake' in the abstract and introduction, while the title and the rest of the paper use 'BanglaFake'; this should be made consistent.","section":"Abstract and Introduction"},{"comment":"In the sentence 'Additionally, we include2,260 text and real audio samples', a space is missing between 'include' and '2,260'; the manuscript needs a careful proofreading pass.","section":"§4.2"},{"comment":"Reference [16] and reference [25] are the same Common Voice paper, and reference [17] and reference [19] are the same VITS paper; these duplicates should be consolidated.","section":"References"},{"comment":"The t-SNE visualization lacks the hyperparameters used (perplexity, number of iterations, random state) and the MFCC configuration, so the figure is not reproducible.","section":"Figure 2"},{"comment":"The dataset release description does not provide file counts, exact license terms, total duration, or a train/validation/test split, which makes it difficult for readers to verify the 'publicly available' claim.","section":"§3.5"},{"comment":"The conclusion says the deepfake audio was generated for a 'single speaker (male)', while Table 1 reports seven real speakers; the relationship between these numbers should be clarified, especially after the Common Voice discussion.","section":"§6"}],"recommendation":"reject","confidential_remarks":"The central failure is the un-matched speaker design in the Common Voice subset, which means the benchmark claim cannot be supported without regenerating that portion of the dataset. The internal count inconsistencies compound the problem, and the evaluation lacks any detector baseline. If the authors rebuild the dataset with proper speaker-conditioned synthesis or voice conversion, correct all counts, and add a real detection baseline, a new submission could be reconsidered, but the current release should not be cited as a valid deepfake benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The gap is real: Bengali has no public deepfake audio corpus, and this paper tries to fill it with a Hugging Face dataset and a VITS-based generation pipeline. That the authors shipped the data and code is to their credit, and the MOS evaluation with 30 native speakers is standard practice. If the dataset exists as described, it is a useful raw resource for Bengali speech research.\n\nBut the central benchmark claim does not hold. The fake audio is always generated by a single-speaker VITS model trained on SUST, and Table 1 confirms the fake side is one male voice. The real side is drawn from 7 SUST speakers plus 5 Common Voice speakers. So every real/fake pair uses a different voice. For the Common Voice subset specifically, the fake clips are the SUST male voice, not the voice of the paired Common Voice speaker. That is not a deepfake of the target speaker; it is a different speaker entirely. A detector solving this task could achieve near-perfect accuracy with an off-the-shelf speaker embedding, without ever learning synthesis artifacts. The paper's t-SNE plot, which claims overlap between real and fake, is odd under this reading; if the voices are different, the features should separate by identity. No detector baseline is reported, so the benchmark utility is asserted rather than demonstrated.\n\nThe internal statistics also do not add up. The abstract and Table 1 state 12,260 real and 13,260 fake. Table 2's rows sum to 12,797 real and 12,797 fake, while its Total row says 13,260. Section 4.2 says 2,260 Common Voice samples, but the Common Voice rows sum to 2,797. These are not cosmetic; they mean the dataset composition cannot be trusted from the paper alone.\n\nMinor point: the MOS is based on 10 sentences and 30 listeners, which is fine for a quick quality check but does not validate the dataset's detection difficulty.\n\nNet assessment: the paper is not incoherent, and the authors clearly put effort into collecting real data and generating synthetic audio. But the load-bearing design choice — one TTS voice for all fakes — breaks the deepfake-detection premise. The corpus might still be useful for speaker-identity detection or as a general Bengali synthetic-speech resource, but it cannot be called a deepfake benchmark without either re-generating fakes with same-speaker voice conversion or multi-speaker TTS, or reframing the task entirely.\n\nWho is this for? Audio forensics researchers working on low-resource languages. They would learn about the existence of the corpus and its limitations, but should not use it as a benchmark until the confound is addressed and the counts are corrected.\n\nFor peer review: send it out. A serious referee will quickly identify the confound, and the community needs a documented record of why this kind of dataset construction fails. But the expected outcome should be rejection or major revision, not acceptance.","headline":"A genuinely novel resource undermined by a speaker-identity confound and inconsistent numbers; the dataset idea is worth something, but the paper as written does not support a deepfake benchmark.","tokens_in":7390,"tokens_out":2714,"would_cite":false,"duration_ms":28595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces BanglaFake, which it presents as the first publicly available Bengali deepfake audio dataset, with 12,260 real and 13,260 synthetic utterances, and argues it can serve as a benchmark for detector training and…","keywords":["Bengali deepfake audio","deepfake audio detection","text-to-speech","VITS","low-resource language","speaker verification","speech forensics","mean opinion score"],"falsifier":"Run a speaker-embedding comparison on the additional corpus pairs: if the cosine similarity between a fake clip and its paired real clip is much lower than the similarity between the fake clip and a reference clip from the main corpus's speaker, the fake class is carrying a different speaker's identity, and the benchmark is testing speaker discrimination rather than deepfake detection. A listener study asking whether each fake clip sounds like the person in the paired real clip would also settle it.","tokens_in":6376,"feed_emoji":"🎙️","tokens_out":8439,"duration_ms":74912,"temperature":0.7,"pith_summary":"This paper introduces BanglaFake, a new audio dataset meant to close a gap: Bengali has almost no public resources for training and testing deepfake-audio detectors, even though voice spoofing is a known risk to speaker-verification systems. The dataset contains 12,260 real Bengali utterances and 13,260 synthetic utterances generated by a VITS-based end-to-end text-to-speech model trained from scratch on a phonetically balanced Bengali speech corpus. The authors evaluate the fake audio with 30 native Bengali listeners, reporting a Robust-MOS of 3.40 for naturalness and 4.01 for intelligibility, and use t-SNE on MFCC features to show that real and fake samples overlap substantially. If the dataset holds up as described, it would give low-resource Bengali speech forensics a common benchmark and a training resource that did not previously exist.","feed_headline":"13,260 Bengali deepfake audio clips now form a public benchmark","feed_subtitle":"Low-resource Bengali speech forensics gets its first open real-versus-synthetic benchmark.","key_machinery":"The load-bearing component is VITS, a named end-to-end text-to-speech architecture that combines a conditional variational autoencoder with adversarial training: a posterior encoder extracts latent variables from a target spectrogram, a normalizing flow and stochastic duration predictor model phoneme timing, and an adversarial neural vocoder decoder converts the latent variables into raw waveforms. The authors train VITS on the phonetically balanced Bengali corpus, then use it to synthesize the fake half of the dataset. The dataset's standardized text-audio metadata layout makes the pairs directly usable by common TTS and detection pipelines, and the evaluation rests on MOS listening tests and t-SNE visualization of MFCCs.","core_discovery":"The paper's central claim is that BanglaFake is the first publicly available Bengali deepfake audio dataset and a usable benchmark for detecting synthetic speech in Bengali. It pairs real recordings from two sources—a phonetically balanced Bengali speech corpus and a crowd-sourced multilingual corpus—with deepfake clips generated from the same text by the authors' VITS text-to-speech model. The authors report that human listeners rate the synthetic speech as reasonably natural (Robust-MOS 3.40) and highly intelligible (4.01), and that t-SNE projections of MFCC features place many real and fake samples in overlapping regions, which they interpret as evidence that the fakes are hard to distinguish from real speech.","pith_inferences":["Because the text-to-speech model was trained on a single male voice from the main corpus, the fake clips paired with the other corpus's speakers are likely spoken in that male voice rather than in the paired speaker's voice; if so, a detector trained on these pairs could succeed by spotting speaker mismatch, not synthesis artifacts.","The same confound means the t-SNE overlap should not be read solely as high naturalness; it may partly reflect domain or speaker differences between the two real-speech sources and the synthetic clips.","A stronger test of the benchmark would add matched-speaker synthesis (same target voice for real and fake) or voice conversion, and report detection accuracy separately for the matched and mismatched subsets.","A quick validation experiment would be to ask listeners or a speaker-embedding model whether each fake clip sounds like the paired real clip's speaker; if it does not, the dataset documents voice replacement rather than speaker-preserving deepfake."],"forward_implications":["A public Bengali real/fake corpus makes it possible to train supervised deepfake detectors where none could be trained before in this language.","The dataset gives speaker-verification researchers a low-resource test set for spoofing countermeasures, extending standard countermeasure research to Bengali.","The reported MOS scores suggest synthetic clips are natural enough that simple energy- or artifact-based detectors will not trivially separate the classes.","The overlap seen in t-SNE motivates detectors that use learned representations rather than raw MFCCs.","The dataset structure supports cross-lingual comparison and zero-shot detection experiments, since the metadata and format match existing TTS pipelines."],"supporting_citations":[{"why":"Supplies the phonetically balanced Bengali corpus that provides 10,000 real utterances and the training data for the VITS model.","marker":"[15]"},{"why":"Defines VITS, the end-to-end text-to-speech architecture used to generate the deepfake utterances.","marker":"[19]"},{"why":"Supplies the additional real utterances and paired text used to generate the remaining fake clips.","marker":"[25]"},{"why":"Defines the standardized text-audio metadata layout in which the dataset is structured.","marker":"[18]"},{"why":"Defines Mean Opinion Score, the subjective evaluation method used to rate naturalness and intelligibility.","marker":"[26]"},{"why":"Provides t-SNE, the dimensionality-reduction technique used to visualize real/fake separability.","marker":"[27]"},{"why":"Represents a major spoofing-detection challenge that motivates the need for such datasets and frames the gap BanglaFake fills.","marker":"[9]"},{"why":"Represents a leading audio deepfake detection benchmark that frames the high-resource focus BanglaFake aims to counter.","marker":"[10]"}],"fun_headline_variants":["BanglaFake: 13,260 synthetic Bengali clips for deepfake detection","First Bengali deepfake audio dataset: 13,260 fakes for benchmarking","Bengali deepfake detection gets open benchmark with 13K clips","BanglaFake benchmark: 13,260 deepfake audio clips for research","Spot AI-voiced Bengali: new 13,260-clip deepfake benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity rests on the assumption that the fake clips are deepfakes of the paired real clips' speakers, but the model was trained on a single voice from the main corpus, so the additional corpus pairs may present a different speaker as fake.","fun_headline_variants_meta":{"raw":{"variants":["BanglaFake: 13,260 synthetic Bengali clips for deepfake detection","First Bengali deepfake audio dataset: 13,260 fakes for benchmarking","Bengali deepfake detection gets open benchmark with 13K clips","BanglaFake benchmark: 13,260 deepfake audio clips for research","Spot AI-voiced Bengali: new 13,260-clip deepfake benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2514,"prompt_tokens":821,"completion_tokens":1693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":1590}},"tokens_in":437,"tokens_out":1693,"duration_ms":13824,"temperature":1.0,"reasoning_tokens":1590,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:01:12.136608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a speaker-embedding comparison on the additional corpus pairs: if the cosine similarity between a fake clip and its paired real clip is much lower than the similarity between the fake clip and a reference clip from the main corpus's speaker, the fake class is carrying a different speaker's identity, and the benchmark is testing speaker discrimination rather than deepfake detection. A listener study asking whether each fake clip sounds like the person in the paired real clip would also settle it.","supporting_citations":[{"cited_title":"Zmm-tts: Zero-shot multi- lingual and multispeaker speech synthesis conditioned on self- supervised discrete speech representations,","cited_arxiv_id":null,"evidence_quote":"Supplies the phonetically balanced Bengali corpus that provides 10,000 real utterances and the training data for the VITS model."},{"cited_title":"Comparative Analysis of Transfer Learning in Deep Learning Text-to-Speech Models on a Few-Shot, Low-Resource, Customized Dataset","cited_arxiv_id":"2310.04982","evidence_quote":"Defines VITS, the end-to-end text-to-speech architecture used to generate the deepfake utterances."},{"cited_title":"Deepfakes: What are they and why would i make one?","cited_arxiv_id":null,"evidence_quote":"Represents a major spoofing-detection challenge that motivates the need for such datasets and frames the gap BanglaFake fills."},{"cited_title":"Dataset pruning for resource-constrained spoofed audio detection,","cited_arxiv_id":null,"evidence_quote":"Represents a leading audio deepfake detection benchmark that frames the high-resource focus BanglaFake aims to counter."}],"review_version":1}