{"id":"0580d48c-83bf-4f0e-a881-398bf24c6229","arxiv_id":"2412.09032","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Speech-Forensics combines multi-span partial forgeries with per-span algorithm labels, and the TEST network detects authenticity, localizes fake segments, and identifies synthesis algorithms simultaneously.","lead":"This paper introduces Speech-Forensics, a dataset of authentic, fully synthetic, and partially spliced speech where multiple segments are generated by different deepfake algorithms, plus a model called TEST that detects, localizes, and attributes those forgeries in one pass. A smart generalist might care because speech deepfakes are a growing disinformation and identity-theft risk, and few tools currently reveal where and how a clip was manipulated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies a train/validation/test split for Speech-Forensics, so the reported mAP, EER, and F1 numbers cannot be verified as held-out test results; this is the most load-bearing gap.","rationale":"The reader's weakest_assumption concerned MFA alignment accuracy propagating into span labels. While that is a legitimate dataset-quality issue, it is not the most load-bearing problem for the central claim. Even if MFA timestamps are perfectly accurate, the paper's failure to state how the dataset is split into training and test sets makes the reported performance numbers unverifiable. A model evaluated on its own training data could easily achieve very low EER and high mAP, especially with a dense-prediction architecture that can memorize label boundaries. The absence of any test-set definition also blocks external comparison: a researcher cannot reproduce the results or determine whether the model generalizes. The reader's concern about MFA is partially valid—if the labels are noisy, the numbers could be optimistic in a different way—but the lack of a split is more fundamental because it precedes any labels. The sign error in Eq. (6) is a concrete internal inconsistency that warrants correction, but it is a typo-level issue that may not affect the released code. Therefore, the appropriate verdict is CONDITIONAL: the paper should be accepted only after the authors provide a clear data split and confirm that the reported metrics are computed on a held-out test set. This does not change the overall direction of the reader's verdict, but it shifts the emphasis from alignment accuracy to evaluation protocol transparency.","tokens_in":12581,"tokens_out":5237,"duration_ms":58209,"concrete_test":"Inspect the released GitHub repository for a fixed split of LJ Speech clip IDs into train/val/test sets. If none exists, ask the authors to provide the exact split (or a seed-based random split) and rerun TEST on that split, recomputing the four headline metrics. Additionally, verify the inference formula in the code: if predictions compute end = t + d_e rather than t - d_e, correct Eq. (6) and confirm the metrics are unchanged.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim—TEST reaching 83.55% mAP, 5.25% utterance-level EER, 1.07% segment-level EER, and 92.19% F1—depends on evaluation on a test set that the paper never defines. Section 3 describes constructing 4,323 NER-based and 3,129 antonym-based forged samples from the LJ Speech corpus, but nowhere states how these samples are partitioned into training, validation, and test subsets. Section 5.2 gives optimizer settings, but no split ratio, no random seed, and no cross-validation scheme. Without a defined split, the model could be evaluated on the same data used for training, which would inflate all metrics and make the headline results unverifiable. The GitHub repository may contain a split, but the manuscript does not reference one, and the reader cannot check from the paper alone. This is distinct from MFA alignment noise: even with perfect labels, an undefined evaluation protocol prevents any independent confirmation of the method's claimed performance. A secondary issue is that Eq. (6) states e_t = t - d_t^e for the span end, but if d_t^e is the positive distance from timestamp t to the end, the correct formula is e_t = t + d_t^e; this sign error, if present in code, would break inference, so it should be resolved in a revision.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Speech-Forensics, a dataset of partially forged English speech built from LJ Speech by splicing TTS/VC-synthesized segments at MFA-aligned timestamps, with multiple forged spans per utterance and algorithm-level labels. It also proposes TEST, a temporal localization network using LSTM-Transformer blocks and a feature pyramid to predict dense span proposals and algorithm probabilities, enabling simultaneous authenticity detection, localization, and algorithm recognition. On this dataset, TEST reports an mAP of 83.55%, an utterance-level EER of 5.25%, a segment-level EER of 1.07%, and an F1 score of 92.19%.","tokens_in":12869,"tokens_out":3462,"duration_ms":33017,"significance":"If the reported results are reliable, Speech-Forensics fills a gap by providing multi-span, algorithm-labeled forgeries for comprehensive forensic analysis, and TEST demonstrates that a single model can accomplish three tasks without complex post-processing. The release of code and data is a concrete strength that supports reproducibility. However, the missing definition of the evaluation split and an apparent sign error in the localization formula currently prevent the quantitative claims from being verified, and the lack of cross-dataset evaluation limits the generality of the model claims.","major_comments":[{"comment":"The paper never specifies how the Speech-Forensics dataset is divided into training, validation, and test sets. Section 3.2 reports 4,323 NER-based and 3,129 antonym-based forged samples plus bonafide samples, and Section 5.2 gives optimizer settings, but no split ratio, random seed, or cross-validation scheme is stated. The headline results in Tables 4 and 5 (mAP 83.55%, utterance EER 5.25%, segment EER 1.07%, F1 92.19%) are therefore not verifiable as held-out test results. Please define the split explicitly (including the number of utterances per subset and the method used), or provide a reference to a fixed split in the repository that is described in the paper.","section":"Section 3.2 and Section 5.2"},{"comment":"Equation (6) defines the end of a forged span as e_t = t - d_t^e. Since d_t^e is described in Section 4.1 as the distance from the timestamp to the offset, the correct formula should be e_t = t + d_t^e when d_t^e is a positive distance. The current sign would place e_t before t, making the localization output invalid. Please correct the equation and clarify the sign convention of d_t^e.","section":"Section 4.4, Eq. (6)"},{"comment":"The model is evaluated only on the self-built Speech-Forensics dataset, and comparisons with RawNet2, AASIST, and WBD are performed only on this dataset. To substantiate the claim that TEST is a comprehensive and generalizable solution, please also report results on established partial-forgery benchmarks such as PartialSpoof or ADD 2023 Track 2. This would also address the concern that the model and dataset are from the same group and that metrics could be inflated by dataset-specific patterns.","section":"Section 5.4, Table 4"},{"comment":"The ground truth span timestamps are derived from Montreal Forced Aligner (MFA) alignments without any reported validation of alignment accuracy. Since the segment-level EER and F1 are computed against these labels, systematic MFA errors would directly compromise the reported numbers. Please provide a reliability assessment, such as a manual inspection of a random sample of spliced segments, or a comparison with an alternative aligner, and discuss the impact of alignment errors on the dataset labels.","section":"Section 3.1"}],"minor_comments":[{"comment":"The hyperparameter θ in the masked difference convolution is not discussed in the experimental setup; please state its value or explain how it is chosen.","section":"Section 4.2, Eq. (4)"},{"comment":"The name 'wavLM' appears in Table 5, which should be capitalized as 'WavLM' for consistency with the rest of the paper.","section":"Section 5.1"},{"comment":"The table uses abbreviations such as 'Det.', 'Loc.', and 'Reg.' in column headers; please spell them out or add a note to improve readability.","section":"Table 1"},{"comment":"The segment-level EER is described as evaluating 'each 0.01s audio segment', but it is unclear whether this refers to a fixed evaluation grid or to the frameshift of the feature extractor; please clarify the exact evaluation protocol.","section":"Section 5.3"},{"comment":"The abstract states an EER of 5.25% without defining EER; please add a brief parenthetical definition in the abstract or introduce the metric earlier.","section":"Abstract and Section 5.3"},{"comment":"The paper claims the method requires 'no complex post-processing' but uses Non-Maximum Suppression at inference; please clarify that NMS is considered a simple and standard post-processing step.","section":"Sections 1 and 5.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are valuable, and the multi-span, algorithm-labeled design addresses a real gap in the literature. The main blocking issue is the undefined data split, which makes the central quantitative claim unverifiable; if the GitHub repository contains the split, the authors must describe it in the paper. The sign error in Eq. (6) is likely fixable, but it needs a clear correction. I would also encourage the authors to add cross-dataset experiments, as the current evaluation is entirely internal to their new benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for the dataset. Speech-Forensics is the first public benchmark I know of that gives you multiple forged segments per sample with per-span synthesis algorithm labels, a real gap relative to PartialSpoof and ADD 2023. The pipeline is sensible: NER/antonym replacement, full-utterance synthesis, MFA alignment, then splicing. TEST is a fairly direct port of ActionFormer-style dense prediction to audio, not a big methodological leap, but the combination of detection, localization, and attribution in one forward pass is a serviceable baseline and it beats the older detectors on their own data.\n\nThe soft spots are real. The most load-bearing: the paper never defines a train/validation/test split. Section 3 describes constructing 4,323 NER and 3,129 antonym samples, and Section 5.2 gives optimizer settings, but no split ratio, no seed, no cross-validation scheme. Without that, the reported 83.55% mAP, 1.07% segment EER, and 92.19% F1 cannot be checked as held-out numbers by any reader. This is a reporting gap, not necessarily fraud -- the GitHub repo may have a split -- but it needs to be fixed before the numbers mean anything.\n\nSecond, Eq. (6) has e_t = t - d_t^e where d_t^e is defined as a positive distance to the span end. That should be t + d_t^e. If the code follows the paper, inference would produce nonsense, so the authors need to clarify which is correct.\n\nMinor: NMS is used in Section 5.2, which contradicts the 'no complex post-processing' claim. And there are no error bars or significance tests; with a single-speaker dataset, some of the differences in Table 5 could easily be noise. The lack of evaluation on PartialSpoof or ADD 2023 Track 2 also weakens the claim that the method transfers.\n\nOn balance: the dataset contribution is genuinely new and worth a serious referee. The evaluation protocol needs to be pinned down, and the sign error resolved. I'd send it to review with a request for major revision, and I'd want the split and external benchmark results in the revision.\n\nWho benefits: people working on audio deepfake detection and localization who want a multi-span, algorithm-annotated benchmark. The method is a serviceable baseline, not a breakthrough.\n\nRecommendation: engage with it, but get the split and sign error addressed.","headline":"A genuinely useful multi-span partial forgery dataset, undermined by an underspecified evaluation protocol that makes the headline numbers unverifiable as written.","tokens_in":13379,"tokens_out":2573,"would_cite":true,"duration_ms":24857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single network, TEST, can detect fake speech, locate every forged segment, and identify the synthesis algorithm that made each one, with an utterance-level EER of 5.25% and a segment-level EER of 1.07% on the new Speech-Forensics dataset.","keywords":["synthetic speech detection","deepfake audio","partial forgery localization","temporal action localization","speech forensics dataset","synthesis algorithm recognition","self-supervised speech features","multi-span forgery"],"falsifier":"Take a random sample of Speech-Forensics forged clips, manually mark the exact splice points by listening and inspecting spectrograms, and compare those boundaries against the paper's automated alignment timestamps; if the median offset is large (say above 50 ms) or systematically biased, the segment-level EER and F1 would need to be recomputed on corrected labels.","tokens_in":12392,"feed_emoji":"🎙️","tokens_out":9257,"duration_ms":84965,"temperature":0.7,"pith_summary":"Synthetic speech detection today is fragmented: most datasets support either utterance-level real/fake decisions or single-region localization, and few record which synthesis algorithm produced a forgery. This paper tries to close that gap by building Speech-Forensics, a dataset whose partially forged clips contain multiple tampered spans produced by different text-to-speech and voice-conversion algorithms, each span labeled with timestamps and the algorithm that made it. On top of the dataset, the paper proposes TEST, a temporal localization network that in one forward pass decides whether an utterance is authentic, where every forged segment sits, and which algorithm generated each segment. The reported numbers are an average mAP of 83.55%, an utterance-level EER of 5.25%, a segment-level EER of 1.07%, and a 92.19% F1 score. If the claim holds, forensic audio analysis can move from separate detectors and fragile post-processing to a single dense-prediction model.","feed_headline":"One network detects, locates, and names fake speech in one pass","feed_subtitle":"Flags forged audio, finds every tampered span, names each algorithm—92% F1 on a new benchmark.","key_machinery":"The machinery is an anchor-free dense prediction head on a temporal feature pyramid. Rather than proposing candidate segments, the network gives every timestamp a distance to the start and end of the forged interval containing it plus a per-algorithm class score; timestamps are mapped back to real time via $t = \\lfloor s_i/2\\rfloor + \\tau s_i$, and redundant spans are removed by non-maximum suppression. The backbone combines LSTM and Transformer blocks (R-Transformer) at L pyramid scales, and the embedding uses masked difference convolution on top of handcrafted (LFCC/MFCC) or self-supervised (Wav2Vec2/WavLM) features. This turns localization and algorithm recognition into one dense regression and classification problem, which is what lets the model avoid complex post-processing.","core_discovery":"The central claim is that a single model can perform three forensic tasks on the same audio at once: authenticity detection, multiple fake-segment localization, and synthesis-algorithm recognition. The paper introduces Speech-Forensics, built from a public single-speaker corpus by replacing named entities or adjectives with antonyms, synthesizing the full edited transcript, and using automated audio-text alignment timestamps to splice the synthesized phrases back into the original clip; clips can have several forged spans generated by different algorithms from a set of twelve TTS and VC systems. The accompanying TEST network treats the problem as dense temporal prediction: each time step on a multi-scale feature pyramid estimates its distance to the onset and offset of a forged span and a probability distribution over algorithms, and non-maximum suppression assembles the final spans. On this dataset TEST reports a mean average precision (mAP) of 83.55% averaged over temporal intersection-over-union thresholds, an utterance-level EER of 5.25%, a segment-level EER of 1.07%, and an F1 score of 92.19%, with self-supervised features (Wav2Vec2 and WavLM) clearly outperforming handcrafted cepstral features.","pith_inferences":["If the alignment assumption holds, the same named-entity and antonym-splicing pipeline could scale to other voices and languages, but generalization across speakers and recording conditions remains untested; a speaker-disjoint split would show whether the model learns forgeries or speaker artifacts.","The reported near-perfect segment-level localization may partly reflect acoustic discontinuities at splice points, so a harder test is whether performance survives seamless editing, e.g., splicing with cross-fades or vocoder-domain insertion.","A natural stress test is cross-dataset evaluation: training on Speech-Forensics and evaluating on PartialSpoof or ADD2023 would reveal whether the joint representation transfers or simply memorizes the dataset's alignment pipeline."],"forward_implications":["A forensic analyst could get a complete verdict on a single recording—authentic or foraged, exactly which intervals, and which generator made each interval—from one model output rather than a pipeline of separate tools.","Training directly on multi-span, multi-algorithm partial forgeries appears to be necessary: RawNet2 and AASIST, built for fully fake utterances, degrade sharply on this dataset, while TEST keeps segment-level EER near 1%.","Better self-supervised representations transfer to the combined task; with WavLM-Large, the reported segment EER drops to 0.43%, suggesting the joint formulation continues to improve as embeddings improve.","The dense-prediction formulation is not tied to a particular feature extractor, so it can be re-run with newer self-supervised models or phonetic features as they appear."],"supporting_citations":[{"why":"Supplies the PartialSpoof dataset and the multiple-forged-region localization setting that Speech-Forensics extends with algorithm labels.","marker":"[Zhang et al., 2021]"},{"why":"Defines the ADD2023 tasks of manipulation region location and deepfake algorithm recognition that the proposed model unifies.","marker":"[Yi et al., 2023]"},{"why":"Introduces the anchor-free dense regression formulation that the prediction head adapts to speech.","marker":"[Yang et al., 2020]"},{"why":"ActionFormer supplies the temporal pyramid and transformer-based localization design that TEST builds on.","marker":"[Zhang et al., 2022]"},{"why":"Wav2Vec2 is the self-supervised feature extractor whose representations carry the reported main results.","marker":"[Baevski et al., 2020]"},{"why":"WavLM is the other self-supervised extractor and gives the best results in the feature comparison.","marker":"[Chen et al., 2022]"},{"why":"RawNet2 is a baseline that fails on partial forgeries, motivating the need for the new dataset and model.","marker":"[Tak et al., 2021]"},{"why":"WBD is the boundary-modeling baseline whose segment-level localization is outperformed by TEST.","marker":"[Cai et al., 2023a]"}],"fun_headline_variants":["One network detects, locates, and names synthetic speech","All-in-one forensics: spot, trace, and identify fake audio","One pass: detect fake speech, find every edit, name the tool","Triple task net: detect, localize, and attribute synthetic speech","New dataset and net tackle all fake speech tasks at once"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's ground-truth span boundaries come from automated alignment between audio and text, so any systematic misalignment in those timestamps would be baked into the labels and inflate the reported localization accuracy.","fun_headline_variants_meta":{"raw":{"variants":["One network detects, locates, and names synthetic speech","All-in-one forensics: spot, trace, and identify fake audio","One pass: detect fake speech, find every edit, name the tool","Triple task net: detect, localize, and attribute synthetic speech","New dataset and net tackle all fake speech tasks at once"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1551,"prompt_tokens":998,"completion_tokens":553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":614,"tokens_out":553,"duration_ms":6350,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:18.544755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of Speech-Forensics forged clips, manually mark the exact splice points by listening and inspecting spectrograms, and compare those boundaries against the paper's automated alignment timestamps; if the median offset is large (say above 50 ms) or systematically biased, the segment-level EER and F1 would need to be recomputed on corrected labels.","supporting_citations":[{"cited_title":"Revisiting anchor mechanisms for temporal action localization","cited_arxiv_id":null,"evidence_quote":"Introduces the anchor-free dense regression formulation that the prediction head adapts to speech."},{"cited_title":"Actionformer: Localizing moments of actions with transformers","cited_arxiv_id":null,"evidence_quote":"ActionFormer supplies the temporal pyramid and transformer-based localization design that TEST builds on."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","cited_arxiv_id":null,"evidence_quote":"Wav2Vec2 is the self-supervised feature extractor whose representations carry the reported main results."},{"cited_title":"End-to-end anti-spoofing with rawnet2","cited_arxiv_id":null,"evidence_quote":"RawNet2 is a baseline that fails on partial forgeries, motivating the need for the new dataset and model."}],"review_version":1}