{"id":"bd167393-4ad5-4f81-83f3-d81800e0a02c","arxiv_id":"2501.06514","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new open-set neural codec source tracing benchmark and dataset shows strong in-distribution classification and OOD detection, but poor generalization to unseen real audio.","lead":"This paper introduces a new benchmark and dataset, ST-Codecfake, for tracing which neural codec generated a deepfake audio clip, including open-set and real-world conditions. It finds that current models classify known codecs and detect new ones well, but fail badly on unseen real speech.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Threshold-based F1 may understate models' ability to handle unseen real audio; a real-vs-fake AUC is needed.","rationale":"The paper's main contribution is a benchmark and dataset, and the negative result on unseen real audio is the key finding. My stress test focuses on how that finding is measured. F1 is threshold-dependent, and the paper itself notes that unseen real samples receive low-confidence logits, so the low F1 may reflect threshold placement rather than discriminative failure. The paper uses threshold-free metrics for OOD detection but not for unseen real classification, creating an asymmetry. A binary real-vs-fake AUC on the same sets would settle this. A secondary concern is that the unseen real sets differ in domain; adding a held-out same-corpus real set would disambiguate 'unseen' from 'out-of-domain'. The reader's CONDITIONAL verdict remains appropriate: the benchmark is valuable, but the central claim needs a threshold-free robustness check before full acceptance.","tokens_in":9845,"tokens_out":7703,"duration_ms":73805,"concrete_test":"Compute the binary AUC for real vs fake on each unseen-real test set using the class-0 softmax score of W2V2-AASIST: 19LA real vs 19LA fake, ITW real vs ITW fake, and NCSSD real vs NCSSD Mimi/SNAC reconstructions (available from the 'different source' conditions). If AUC is high while F1 is low, Table 4's headline low F1 is a threshold artifact; if AUC is also low, the robustness failure is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NCST models 'lack robustness in classifying unseen real audio' rests on the F1 scores in Table 4. F1 requires a decision threshold on the real class, and the paper's own Section 4.2 explains that the model assigns low-confidence logits to both unseen real and unseen fake samples while ID samples get excessively high-confidence logits. If the threshold is inherited from the dev set, unseen real samples can be rejected as non-real even when the model has good real-vs-fake discriminative information. The paper evaluates OOD fake with threshold-free metrics (AUC, FPR95, EER) but evaluates unseen real only with threshold-based F1. This asymmetry means the headline conclusion may conflate poor calibration with an inability to classify. Specifically, the low 4.77% F1 on NCSSD real and 16.94% on 19LA real could arise because the real-class score is low but still higher for real audio than for codec-reconstructed audio. Without a real-vs-fake ROC/AUC on these unseen real sets, the claim is not fully supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript defines a new task, Neural Codec Source Tracing (NCST), in which a model must classify audio by the neural codec used to reconstruct it, detect out-of-distribution (OOD) novel codecs, and attribute the back-end of audio language model (ALM) speech. The authors construct the ST-Codecfake dataset, containing bilingual real audio and fake audio reconstructed with 11 neural codecs, and evaluate three baselines (Mel-LCNN, AASIST, W2V2-AASIST) under closed-set, OOD, robustness, and ALM back-end conditions. The central empirical finding is that the models achieve near-perfect in-distribution classification (99.99% average F1) and strong OOD detection (best AUC 97.54%), but perform poorly at classifying unseen real audio from ASVspoof2019LA, In-the-Wild, and NCSSD, with the best model reaching only 16.94%, 77.01%, and 4.77% F1, respectively. The paper concludes that current source-tracing models lack robustness in classifying unseen real audio and that future work should focus on improving real-class robustness.","tokens_in":10024,"tokens_out":4350,"duration_ms":43468,"significance":"If the reported findings hold, the paper makes a useful contribution by formalizing the NCST task, releasing a large codec-based source-tracing dataset (ST-Codecfake), and showing a striking generalization gap to unseen real audio. The use of threshold-free OOD metrics (AUC, FPR95, EER), multiple OOD scenarios (different method/configuration/source), three diverse baseline architectures, and public code and data are clear strengths. The negative result on unseen real audio is important for the community because it indicates that strong in-distribution and OOD performance does not imply the ability to recognize real speech in the open-set setting. However, the strength of this central claim is currently limited by the threshold-based F1 metric used for unseen-real evaluation and by the very small test set for one ALM back-end, so the main conclusion needs additional validation before it can be accepted as stated.","major_comments":[{"comment":"The \"Unseen Real\" robustness evaluation is reported exclusively via F1-score, which requires a decision threshold, but the manuscript does not specify how thresholds are selected for these unseen sets. Moreover, §4.2 explicitly states that the model assigns low-confidence logits to both unseen real and unseen fake samples while ID samples receive excessively high-confidence logits. This implies that the low F1 values (e.g., 4.77% on NCSSD real, 16.94% on 19LA real) may reflect an inappropriate threshold inherited from the development set rather than a lack of discriminative ranking between real and fake audio. To support the claim that models \"lack robustness in classifying unseen real audio,\" please report a threshold-free measure such as real-vs-fake ROC-AUC or average precision on these unseen-real sets, and discuss whether the intended claim is about ranking ability or about decision-making under a fixed threshold.","section":"§4.1, Table 4"},{"comment":"The ALM back-end detection experiment uses only 18 SpeechGPT-Gen samples and 4,436 VALL-E samples, yet Table 6 reports per-model F1 scores for these categories (e.g., 99.35–99.48% for SpeechGPT-Gen) without confidence intervals or significance tests. With 18 samples, a single misclassification changes the F1 by several percentage points, so these numbers are not reliable evidence of accurate back-end detection. Please provide the sample count for each tested condition, bootstrap confidence intervals, or a statistical comparison, and temper the corresponding conclusions in §4.3 and the abstract.","section":"§2.4, Table 6"},{"comment":"All audio is downsampled to 16 kHz and trimmed/padded to 4 seconds before feature extraction, even though several codecs (Mimi, EnCodec, FACodec, SNAC, WavTokenizer, DAC, AudioDec) operate natively at 24 kHz or 44 kHz. The benchmark's validity depends on codec-specific artifacts surviving this resampling and length normalization, but the paper provides no analysis of whether codec traces are preserved. Without such validation, the reported ID and OOD results may be confounded by the preprocessing pipeline rather than reflecting codec identity. Please add an experiment or analysis demonstrating that the 16-kHz/4-s representation retains codec-specific information (e.g., by comparing ID classification with and without downsampling for a subset of codecs).","section":"§3.3, implementation details"}],"minor_comments":[{"comment":"There are several typos and inconsistencies: \"categoriary\" should be \"category,\" \"ALSHELL\" should be \"AISHELL-3,\" and the dataset name is inconsistently capitalized as ST-Codecfake vs. ST-codecfake.","section":"§1 and §2.1"},{"comment":"The column heading \"A VG\" should be \"AVG\".","section":"Tables 3–4"},{"comment":"The text refers to the NCSSD dataset using reference [19], but the cited title is \"Generative expressive conversational speech synthesis\"; please clarify the relationship between NCSSD and this reference.","section":"§2.1"},{"comment":"The labels \"T3-T5\" appear in Figure 2 but are never defined in the text; please explain what they represent.","section":"Figure 2"},{"comment":"The introduction criticizes threshold-based F1 as overly subjective, but the manuscript never describes the threshold-selection procedure used to compute the F1 scores in Table 4; please clarify how the thresholds are set for both ID and unseen-real evaluations.","section":"§4.1, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The dataset and benchmark are potentially valuable for the audio deepfake detection community, and the central negative result about unseen real audio is likely correct. However, the paper currently supports that claim with threshold-based F1 alone, which is arguably not the right metric for the stated open-set robustness problem, and the ALM back-end claim rests on a very small speech sample. These are fixable with additional experiments rather than a fundamental flaw. I would also gently note that the paper relies on several of the authors' prior works (e.g., Codecfake, NSD) and would benefit from an independent baseline or external dataset to strengthen the generality of the findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the paper gives the field a genuinely new evaluation setup for open-set neural codec source tracing, and the headline claim—that models fail on unseen real audio—is plausible but under-supported because all the evidence on that point is threshold-based F1, with no rank-based metric to separate calibration from discrimination.\n\nWhat's new: the ST-Codecfake dataset (235k bilingual samples, 11 codecs) and a protocol that goes beyond ADD2023T3's single-OOD setup. The OOD conditions include multiple novel codecs, non-codec fakes, and configuration/source variation, which is a real improvement. The finding that AASIST and W2V2-AASIST get 99.99% F1 in-domain but collapse to 16.94% F1 on 19LA real, 77.01% on ITW real, and 4.77% on NCSSD real is cleanly demonstrated across three baselines. That's a useful alarm.\n\nWhat's done well: the benchmark design is thoughtful, the OOD metrics are threshold-free (AUC, FPR95, EER), and data and code are public. Self-citation is present but not excessive; their prior work genuinely supplies the dataset and the strongest baseline.\n\nSoft spots: the main one is the unseen-real evaluation. The paper's own Section 4.2 says the model gives low-confidence logits to both unseen real and unseen fake samples, while ID samples get high confidence. If the decision threshold is carried over from the dev set, low F1 on unseen real may just mean the threshold rejects anything with low confidence. Without a real-vs-fake AUC or other rank-based measure on those sets, we can't tell whether the model has no discriminative information or just bad calibration. That's a load-bearing gap for the central claim.\n\nThe ALM back-end detection is much weaker: SpeechGPT-Gen has only 18 test samples, VALL-E 4,436, and there are no error bars or significance tests. Comparing backends (Moshi 70.74% vs VALL-E 99.35%) on these sizes is not meaningful. Calling this 'interpretable' oversells it; it's just backend classification.\n\nThe dataset label assumption—that codec artifacts dominate source and resampling effects—isn't directly tested, but the high OOD AUC suggests the model does learn something codec-specific, so that's not a central concern.\n\nFor whom: anyone building or evaluating audio deepfake detectors, especially in the codec/ALM space. The dataset is likely to become a reference resource. I'd bring it to a reading group focused on deepfake detection.\n\nRecommendation: send to peer review. It's a solid benchmark paper with a reproducible resource. The revision should add real-vs-fake ranking metrics on the unseen real sets, expand the tiny ALM sets or drop them, and report variance.","headline":"Useful new open-set codec source tracing benchmark; the unseen-real claim needs a ranking metric to fully land.","tokens_in":10608,"tokens_out":4121,"would_cite":true,"duration_ms":38430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that neural codec source tracing models, while nearly perfect on in-distribution classification and out-of-distribution detection, perform very poorly on unseen real audio, with the best model reaching only 16.94% F1 on…","keywords":["neural codec source tracing","open-set recognition","audio deepfake detection","out-of-distribution detection","ST-Codecfake","audio language model"],"falsifier":"A direct test: train W2V2-AASIST on ST-Codecfake with the real class oversampled to match the majority fake class and with an additional held-out speaker set from VCTK and AISHELL, then measure F1 on the 19LA, ITW, and NCSSD real subsets. If F1 climbs above 90% on all three, the paper's attribution of failure to real-class underrepresentation and insufficient real training data is confirmed; if it stays near the reported 16.94%, 77.01%, and 4.77%, the failure is due to corpus shift or codec-specific overfitting rather than class imbalance.","tokens_in":9650,"feed_emoji":"🎙️","tokens_out":6498,"duration_ms":53895,"temperature":0.7,"pith_summary":"This paper defines the Neural Codec Source Tracing (NCST) task: given a piece of audio, identify which of several known neural codecs reconstructed it, while also flagging audio from unknown codecs and from non-codec synthesis. To support the task, the authors built the ST-Codecfake dataset of about 235,000 bilingual samples synthesized by 11 neural codecs plus audio-language-model test material, and they set up a three-part benchmark covering in-distribution classification, open-set out-of-distribution detection, and behavior on unseen real audio, changed codec configurations, and changed source domains. The central finding is an asymmetry: the best NCST models reach 99.99% F1 on in-distribution classification and 97.54% AUC on OOD detection, but collapse on unseen real audio, with the best model scoring only 16.94%, 77.01%, and 4.77% F1 on three real-speech test sets. The paper argues that current source tracing systems therefore cannot be trusted to attribute or reject audio in open-set conditions, and identifies the likely cause as too little real training data and underweighting of the real class.","feed_headline":"Source tracing models ace fake audio, flop on new real audio","feed_subtitle":"Best model drops to 16.9% F1 on one real-speech set after scoring 99.99% in distribution.","key_machinery":"The machinery is the ST-Codecfake dataset and the accompanying evaluation protocol. ST-Codecfake pairs 23,228 real utterances from VCTK and AISHELL-3 with reconstructions from six in-distribution codecs (Mimi, SpeechTokenizer, FunCodec, EnCodec, FACodec, SNAC) and five out-of-distribution codecs (WavTokenizer, AcademicCodec, DAC, AudioDec, SoundStream), totalling 235,736 samples. The protocol then tests a seven-class classifier (real plus six codecs) under three axes of generalization failure: unseen real audio from different corpora, different generation configurations of the same codec, and different source domains; it also measures OOD detection with threshold-free metrics (AUC, FPR95, EER) and runs a back-end detection test on audio-language-model outputs. The three baselines, Mel-LCNN, AASIST, and W2V2-AASIST, are standard anti-spoofing models repurposed with a seven-head classifier; W2V2-AASIST is the strongest and is the model used for the OOD experiments.","core_discovery":"The paper's central claim is that open-set neural codec source tracing, as formalized by the new task, is not yet solved: the models that nearly perfectly classify the seven in-distribution classes (real plus six codecs) and that detect out-of-distribution codecs with high AUC systematically misclassify real audio they have not been trained on. The evidence is Table 4, where every baseline model drops sharply on the 19LA real subset, the In-the-Wild real subset, and the NCSSD set, with W2V2-AASIST, the best model overall, at 16.94%, 77.01%, and 4.77% F1 respectively. The paper frames this as a failure of the real category to generalize and shows that the same high-confidence logits that make OOD detection easy are what push unseen real audio outside the real class. The authors also show that the NCST model can often identify the back-end codec of audio-language-model outputs, which is the promised interpretability benefit.","pith_inferences":["The 4.77% F1 on the NCSSD real subset, far below chance for a seven-way classifier, suggests that studio-recorded dialogue speech is being assigned to codec classes with high confidence; it would be worth measuring whether the model's per-sample confidence correlates with codec artifacts or with speaker characteristics.","The benchmark's OOD detection treats all non-codec fake audio and all unseen codecs as a single 'other' class; a stricter test would require per-codec open-set identification, where the model must say which known codec is closest, not just whether it is out-of-distribution.","The ALM back-end detection results, with Mini-Omni at 100% F1, likely reflect that some ALMs directly expose the codec's residual artifacts; if so, NCST could be inverted as a tool for auditing which codec a deployed ALM uses, which the paper only hints at."],"forward_implications":["Source tracing in open-set conditions is not achieved by current models; any deployed NCST system must treat unseen real audio as an open problem rather than as an in-distribution class.","The high OOD detection scores (up to 97.54% AUC) mean that logits-based confidence is enough to tell known codecs from novel ones, so back-end codec detection for ALM audio can be built on top of existing in-distribution classifiers.","Because the failure on unseen real audio is attributed to data imbalance and real-class underrepresentation, adding more diverse real speech and reweighting the real class is a concrete next step that could close most of the gap.","The dataset and code release make it possible for other groups to reproduce the benchmark and to test whether alternative training strategies fix the real-class collapse."],"supporting_citations":[{"why":"Supplies the W2V2-AASIST baseline and the NSD out-of-distribution method that the benchmark compares against.","marker":"[8]"},{"why":"Provides the Codecfake dataset configurations (C3-1 to C4-3) and the VALL-E fake audio used for robustness and back-end tests.","marker":"[10]"},{"why":"Defines the Mimi codec (ID 1) and supplies Moshi ALM audio for the back-end detection experiment.","marker":"[12]"},{"why":"Defines the SNAC codec (ID 6) used as an in-distribution class and for the different-source experiment.","marker":"[14]"},{"why":"Supplies the ASVspoof2019 LA real and fake test sets used as unseen real and OOD audio.","marker":"[17]"},{"why":"Supplies the In-the-Wild real and fake test sets representing complex acoustic environments.","marker":"[18]"},{"why":"Supplies the NCSSD real audio used as unseen studio dialogue and as instruction audio for ALM generation.","marker":"[19]"},{"why":"Defines the LCNN back-end for the Mel-spectrogram baseline model.","marker":"[31]"},{"why":"Defines the AASIST architecture that two of the three baseline models are built on.","marker":"[32]"}],"fun_headline_variants":["Open-set tracing: strong on fakes, flops on unseen real","Near-perfect fake tracing, near-zero unseen real","Models excel on known codecs, miss new real speech","Fakes: near-perfect. Unseen real: near zero","Open-set attribution: high on fakes, low on unseen real"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity depends on the ST-Codecfake labels: that audio reconstructed by a given neural codec carries artifacts specific to that codec, and that downsampling to 16 kHz and trimming or padding to 4 seconds preserves those artifacts well enough for a classifier to learn them.","fun_headline_variants_meta":{"raw":{"variants":["Open-set tracing: strong on fakes, flops on unseen real","Near-perfect fake tracing, near-zero unseen real","Models excel on known codecs, miss new real speech","Fakes: near-perfect. Unseen real: near zero","Open-set attribution: high on fakes, low on unseen real"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00197,"raw_usage":{"total_tokens":7688,"prompt_tokens":931,"completion_tokens":6757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":6670}},"tokens_in":547,"tokens_out":6757,"duration_ms":43417,"temperature":1.0,"reasoning_tokens":6670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:57:42.620489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: train W2V2-AASIST on ST-Codecfake with the real class oversampled to match the majority fake class and with an additional held-out speaker set from VCTK and AISHELL, then measure F1 on the 19LA, ITW, and NCSSD real subsets. If F1 climbs above 90% on all three, the paper's attribution of failure to real-class underrepresentation and insufficient real training data is confirmed; if it stays near the reported 16.94%, 77.01%, and 4.77%, the failure is due to corpus shift or codec-specific overfitting rather than class imbalance.","supporting_citations":[{"cited_title":"ASVspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,","cited_arxiv_id":null,"evidence_quote":"Supplies the W2V2-AASIST baseline and the NSD out-of-distribution method that the benchmark compares against."},{"cited_title":"Add 2023: the second audio deep- fake detection challenge,","cited_arxiv_id":null,"evidence_quote":"Provides the Codecfake dataset configurations (C3-1 to C4-3) and the VALL-E fake audio used for robustness and back-end tests."}],"review_version":1}